Delphi Method

The Delphi method is a structured consensus building technique that uses multiple rounds of questionnaires administered to a panel of experts, with controlled feedback between rounds, to converge toward group agreement on a topic where evidence is uncertain, incomplete, or contested. Unlike a face to face committee meeting, the Delphi method operates anonymously, reducing the scope for dominant individuals to disproportionately influence the group and allowing each panelist to reconsider their position in light of the group's aggregated responses rather than who gave them.

Many research and clinical questions cannot be answered by empirical evidence alone. When developing clinical practice guidelines, identifying research priorities, defining outcome measures, or establishing diagnostic criteria, expert judgment is essential. However, unstructured expert panels are vulnerable to groupthink, anchoring effects, and domination by the most vocal or senior members. The Delphi method addresses these problems by structuring the process of expert consultation, promoting independent judgment while allowing iterative refinement. It is widely used in health research to develop consensus statements, clinical practice guidelines, core outcome sets, and reporting guidelines.

The process typically begins with the selection of a panel of individuals with relevant knowledge and experience. There is no agreed optimal panel size: the number is not based on statistical power and is often a pragmatic choice, and published panels range from about a dozen experts to well over a hundred participants. Panel composition should balance expertise, geographic diversity, disciplinary perspectives, and stakeholder representation (including, in many applications, patient or consumer representatives).

In the first round, panelists respond to a questionnaire that may include rating scales (such as a 9 point Likert scale rating the importance or appropriateness of each item), ranking tasks, or open ended questions. The questionnaire is based on a preliminary list of items generated from a literature review, qualitative research, or initial brainstorming.

After the first round, the research team analyzes responses and prepares a summary, typically including the median (or mean) rating for each item, the interquartile range (IQR) as a measure of dispersion, and the percentage of panelists who rated the item at each level. This summary is fed back to panelists at the beginning of the second round, along with each panelist's own previous response. Panelists can then revise their ratings in light of the group's aggregated opinions.

This process of rating, feedback, and re rating continues for two to four rounds, or until a predefined criterion for consensus is met. Common consensus criteria include a threshold percentage of panelists agreeing (for example, 70% or more of panelists rating an item 7 to 9 on a 9 point scale) combined with a narrow interquartile range (IQR of 2 or less). Items that meet the consensus threshold are retained; items that clearly do not are discarded; and items with uncertain consensus may be discussed in a face to face meeting or additional round.

The label "modified Delphi" covers many variations, such as starting from a list of items drawn from the literature instead of open questions, or adding a face to face meeting, and studies use it inconsistently, so a report should say exactly what was changed. In core outcome set development, a common approach is an anonymous Delphi survey followed by a face to face consensus meeting, as described in the handbook of the COMET (Core Outcome Measures in Effectiveness Trials) Initiative.

The Delphi method is appropriate when empirical evidence is insufficient to resolve a question, when the question involves value judgments or priorities that benefit from diverse perspectives, and when a structured process is needed to generate group consensus. Common applications include developing clinical practice guidelines, identifying core outcome sets, setting research priorities, defining diagnostic or classification criteria, and establishing reporting standards. It is not appropriate when high quality empirical evidence directly answers the question, when the topic requires real time deliberation and negotiation, or when the panel cannot be composed of individuals with genuine expertise or relevant experience.

A common mistake is selecting a panel that lacks diversity of perspective, producing "consensus" that reflects a narrow viewpoint rather than a genuine integration of varied expertise. Setting vague or post hoc consensus criteria allows researchers to manipulate results by adjusting thresholds after seeing the data. Providing feedback that reveals individual identities, even indirectly (through comments that are attributable), undermines the anonymity that protects against social pressure. Conducting too few rounds may not allow sufficient convergence, while too many rounds (more than four) can cause panelist fatigue and dropout. Dropout is not neutral: if the panelists who leave held different views from those who stay, agreement among the remaining panelists can rise without anyone changing their rating, a form of attrition bias. Finally, failing to report the Delphi methodology transparently, including panel composition, response rates, consensus criteria, and the number of rounds, makes it impossible for readers to evaluate the rigor of the process.

A core outcome set development project for pediatric asthma clinical trials uses a three round Delphi process with a 28 member panel comprising 12 pulmonologists, 6 primary care physicians, 4 nurses, and 6 parent representatives from four countries. Round 1 presents 47 potential outcomes identified from a systematic review, rated on a 9 point scale. After three rounds, 8 outcomes achieve consensus (defined as 70% or more panelists rating 7 to 9 and fewer than 15% rating 1 to 3): exacerbation frequency, symptom free days, FEV1, rescue inhaler use, quality of life, school absence, nocturnal awakening, and hospitalizations. The final core outcome set is published with full reporting of ratings and consensus trajectories.