By Benedikt Steiner.
How can the Delphi method be modified to provide data aggregation and visualisation in real time? Which aspects of the Delphi method are preserved and which are changed? How does such a modified method work best?
A brief overview of the Delphi method
The Delphi method is a structured elicitation process that invites experts to explore complex, uncertain or contested topics. It aims to make the assumptions, expectations, and uncertainties of the experts involved explicit.
Key characteristics include:
- anonymity of participants, reducing social pressure and dominance effects
- iterative assessment, allowing experts to reflect and revise their judgments
- controlled feedback, showing aggregated group responses
- aggregation, rather than forced agreement.
Unlike workshops or focus groups, traditional studies using the Delphi method do not rely on real-time interactive discussion, but instead gather participants’ views in an independent, sequential manner across structured rounds of questioning and feedback.
The Delphi method is described in more detail by Dmitry Khodyakov in his i2Insights contribution, Generating evidence using the Delphi method.
Durvey – a modified Delphi method
The modifications colleagues and I developed for the Delphi method are centred around providing a web-based research environment that automates data aggregation and visualisation in real time, reducing the administrative workload for researchers. The platform enables continuous iteration, so that as soon as new responses are submitted, group statistics and feedback are updated. This has implications for other aspects of the Delphi method, which are described below.
The method retains the following strengths of the Delphi method: anonymity, iterative reflection and controlled feedback. Durvey allows researchers to control when the experts see the updated results, for example, making it possible for expert participants—after submitting their judgments anonymously—to immediately gain access to:
- aggregated group responses (eg., central tendencies and distributions)
- measures of dispersion that make uncertainty and disagreement visible
- their own position in relation to the group statistics including visual elements (eg., boxplots)
- anonymised qualitative comments provided by participants.
Researchers using Durvey to elicit expert opinion have flexibility in how they use the method, especially determining:
- when and how often experts can provide additional responses after seeing their input in the aggregated group responses
- when and whether experts can see the comments made by other respondents
- whether aggregated responses are available when the processes begins (with very few respondents) or if there is a pre-phase, where aggregated results are not shown until a specified number of expert participants have completed the survey.
The key difference from the traditional Delphi method is the modification of “rounds.” In the traditional Delphi method, participants do not see the aggregated responses until everyone has responded. In Durvey, participants can potentially gain access to the aggregated group results immediately after submitting their own.
Depending on how the researchers have set up Durvey, the experts can potentially refine their own ratings or comments whenever they feel their perspective should be adjusted. It is also possible for the researchers to limit the ability of expert participants to respond again until everyone has responded, keeping the idea of rounds.
In any case, each expert participant will respond to a slightly different set of aggregated data, depending on when they enter the process. Because aggregated results can change over time, the information contexts for different participants are likely to be different. On the other hand, being able to see the aggregated results can lead to faster response times and a shorter overall process.
What works best for the Duvey method
Real-time aggregation works best with structured quantitative inputs, such as:
- Likert scale ratings
- probability estimates
- desirability or feasibility ratings
- rankings or prioritizations
- multiple-choice selections.
These allow for instant calculation of statistical summaries such as medians, interquartile ranges, or consensus measures. Quantitative responses form the backbone of real-time metrics. Qualitative inputs are typically in the form of comments which explain or expand on the ratings.
Depending on the survey settings, these comments can be made visible to other participants, helping to surface the reasoning behind individual positions. Participants may also be able to interact with comments through likes, making it easier to highlight arguments that resonate most strongly with the group as “important.”
The researchers in charge of the study generally undertake deeper qualitative analysis (eg., thematic coding) either after the study or periodically during the study. Therefore, although comments appear immediately, qualitative interpretation is not available in real-time.
Concluding questions
Does the Durvey method look like it could be useful for you? What aspects are most appealing? Are there aspects that are of concern or that you want to know more about?
To find out more:
Durvey website. (Online): https://www.durvey.org.
Use of Artificial Intelligence (AI) Statement: Generative AI (Anthropic’s Claude Sonnet 4.6 and OpenAI’s Chat GPT) was used in the drafting and editing of this contribution. All frameworks, arguments, and ideas are the author’s own, developed independently of AI assistance. AI-generated text was reviewed, revised, and approved by the author prior to submission. (For i2Insights policy on artificial intelligence please see https://i2insights.org/contributing-to-i2insights/guidelines-for-authors/#artificial-intelligence.)
Biography: Benedikt Steiner PhD is Co-Founder of Beleo Labs GmbH and affiliated with Foresight Research, based in Munich, Germany. His work focuses on advancing the integration of research into practice by developing digital solutions that support evidence-based decision-making. His main areas of interest include implementation science, knowledge translation, innovation processes, and the use of Delphi methods to bridge the gap between academic research and real-world application.