Methods for analyzing data from Delphi panels: Some evidence from a forecasting study

Traditional assumptions about the proper methods for analyzing a Delphi study may be inappropriate, and the use of robust estimates of location as summaries of expert opinion yield better forecasts than nonrobust measures.

Abstract

Delphi and other methods of using expert opinion to generate forecasts can be a useful tool for planning, impact assessment, and policy analysis. Unfortunately, little is known about the accuracy of forecasts produced using these methods, so their utility is limited at present. Based on the logic of the Delphi method, I suggest that: 1) forecast accuracy should increase across rounds of a Delphi iteration, 2) there is a positive correlation between a panelist's uncertainty about a forecast and his or her shift in forecast from round to round, 3) forecasts weighted by self-reported confidence will be more accurate than unweighted forecasts, and 4) the use of robust estimates of location as summaries of expert opinion yield better forecasts than nonrobust measures. A Delphi experiment provides little support to any of these hypotheses. This finding suggests that traditional assumptions about the proper methods for analyzing a Delphi study may be inappropriate.