Delphi Methods

What are Delphi methods?

The Delphi method is a forecasting process framework based on the results of several rounds of questionnaires sent to a panel of experts (i.e. people who have expertise in an area requiring decision making). Several rounds of questionnaires are sent out, and the anonymous responses are aggregated and shared with the group after each round. There is a minimum of two rounds and in the second round (and later rounds) of the survey, the results from the previous round are given as feedback. It is used where there is limited data or to gain insight into future trends. Long-range predications (20-30 years) may only have expert opinions as a source of information.   


There are three types of Delphi methods/survey: 

  1. Policy (where there is a need to devise a strategy to address a specific problem; 
  2. Classical (used to forecast future trends) 
  3. Decision-making (used to achieve better decision making). 


It is a systematic interactive way of gaining opinions or forecasts from a panel of anonymous participants with relevant expertise to answer two or more sequential surveys to identify consensus or convergence. The methods can be used to identify issues or outcomes of importance to stakeholder groups, allowing participants to reconsider their own views in the second survey round in light of those of other participants. The process is intended to to lead to convergence towards identifying preliminary sets of core outcome domains. In healthcare, it is a good means to collate consensus on outcomes or care that is consistent with professional knowledge. The idea is that respondents can learn from views of others without being influenced by who speaks the loudest at a meeting (or has more prestige). Ideally, dissenters from consensus should be given the opportunity to explain their viewpoint as this is useful intelligence and can help debate and make expert's tacit knowledge more explicit.


The Delphi method is a type of consensus method which does not require face to face meetings. It can bring together and synthesize the knowledge of a group of geographically scattered participants that never meet.  A large number of individuals across diverse locations and areas of expertise can be included anonymously, thus avoiding domination of the consensus process by one or a few experts.


Summary of Steps

  1. Define the aim and scope of the study
  2. Design questionnaire (literature review, identify driving forces, trends)
  3. Select the experts
  4. Draft the survey and test the draft survey (~10 participants)
  5. Finalise the survey
  6. Invite the participants to take part
  7. Send out the first round of the questionnaire
  8. Analyse responses from round 1 questionnaire and alter in light of round 1 responses and include new information arising from answers
  9. Prepare and send out the second round questionnaire - request justification of answers falling outside of interquartile range established in round 1
  10. Analyse responses from round 2 questionnaire
  11. Repeat previous steps 8 and 9 if you're doing subsequent rounds - typically you continue until consensus has been reached.
  12. Carry out analysis and report results


Recruitment

Panel sizes are usually guided by practicality or question scope. Typically this is between 10 and 100.   However, there is some evidence that 15-30 participants are ideal as more than 30 are not seen to improve the results (Keeney, Hasson & McKenna, 2011).  Participants are not sampled for representativeness but are viewed as 'experts' drawn from different stakeholder groups to allow comparison and consideration of different perspectives.  


It is worth deciding who your stakeholders are. Do you have different groups of stakeholders? Then ensure you have invited as many as you can. How they're chosen should be recorded: was it their willingness? their knowledge of a particular area?  


Participants are invited through email, letter or through social media, such as through Facebook group pages.  All participants should understand the purpose of the study and that his/her expertise should be made available in different rounds.   Anonymity and feedback are key.  All responses are anonymous and all panel members should communicate individually with the researcher.  In terms of feedback, the results collected and collated from the previous round is returned to the participants in a series of rounds where they are asked to review the outcome of the previous round until consensus is reached.  Typically they receive the median response, the highest and lowest ratings, their response and a summary of commentaries. 


Delphi methods is an iterative process and efforts should be taken to reduce drop outs after the first round.  There will be drop outs and care should be taken to ensure that drop outs are not due to people thinking that their responses are not being acknowledged.  


Questionnaire Design & Administration

The subject should really be one where there is a lack of data or limited evidence. Care should be taken in formulating the statements. This can be done by taking them from the literature and/or using a working group to formulate topics.  Ideally the same person should administer the questionnaire and feed back results.

A questionnaire is required for Round 1,which is then adapted for Round 2 and onwards. The questionnaire can be conducted online - each participant is sent a link or they can be sent paper versions. Similar to questionnaires used in surveys, the questionnaire should be pre-tested and piloted before being used.


You can use a Likert scale or other rating methods to capture opinions on the statements (this can help with the analysis by enabling you to calculate means, modes and medians). Questions can be repeated in the following round(s). Usually in Round 1, you would ask participants to rate and propose additional outcomes or issues and then combine these into questions or statements to test with the wider group in the next round.  This occurs in what is know as 'Conventional Delphi' design, where participants have the opportunity to suggest alternatives.  (A 'Modified Delphi' design has the initial alternatives already selected before presenting them to the panel).  Conventional method overcomes the researcher imposing his/her preconceptions on respondents by enabling contribution and not just being reliant on predetermined literature based statements.  Researchers need to be facilitators in Delphi methods not contributors. 


The second round questionnaire also gives participants the opportunity to see how the rest of the group prioritised areas and to see if they want to change their opinion on the basis of the group consensus.  Consensus does not mean 100% agreement.  Delphi consensus typically ranges between 55% to 100%, with 70% considered to be the standard (Vernon, 2009). 


The recommended number of rounds is two or three (Boulkedid et al, 2011).  However there is little evidence on the optimal number.  Ideally, you continue until you reach a consensus or 'point of diminishing returns'.  Stopping too early may lead to results that are invalid, too long may induce participant fatigue and cause them to drop out. 


It is important to have short periods between rounds as this improves the quality of the quality. Ideally 1 to 3 weeks between rounds. Participants need to be kept anonymous so that they are not identifiable to each other (this could influence/bias their responses).  


Some studies using Delphi Methods have a physical meeting of experts between rounds.  However, this goes against the Delphi procedure, which is avoidance of situations where a panel member may dominate the consensus process.  A physical meeting could be used at the end to resolve any uncertainties.  The meeting will need to be well-structured with a moderator to contain influence of dominant personalities. 


Analysis of responses

Results are presented statistically. Typically, the median response and interquartile ranges are presented for each question.  Participants whose responses fall outside the interquartile range are asked to review their opinion in light of the group consensus.  Members who maintain an estimate outside this interquartile range can be asked to give a justification for their viewpoint. New medians and interquartile ranges can be calculated at each round. The questionnaire can circulate until consensus is reached. Usually this is 2 to 3 rounds and no more than 4 rounds.


In each round, each participant should receive  the median response, the highest and lowest ratings, their response and a summary of commentaries.



Are Delphi Methods quantitative or qualitative?

Although questionnaires are used, the Delphi method is essentially a qualitative method as it is used to gain insights to form a consensus. It can also explore an area of future thinking that goes beyond the currently known or believed.


Further Reading

Delphi Methods at www.healthknowledge.org.uk

Boulkedid et al.  Using and reporting the Delphi Method for selecting healthcare quality indicators: a systematic review.  PLos One 6(6):e20476. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0020476

De Meyrick J (2003) "The Delphi method and health research", Health Education, Vol. 103 Issue: 1, pp.7-16, https://doi.org/10.1108/09654280310459112

Keeney, S., Hasson, F. & McKenna, H. (2011). The Delphi technique in nursing and health research. Oxford, United Kingdom: Wiley-Blackwell.

Vernon, W.  (2009).  A Delphi technique: A review. International Journal of Therapy and Rehabilitation.  16(2), 69-76.