Judgment – Integration and Implementation Insights

Generating evidence using the Delphi method

November 14, 2023October 17, 2023

By Dmitry Khodyakov

What is Delphi? How has the Delphi method stood up over time? How can the best of Delphi be adapted to new circumstances and problems?

The Delphi method is a group-based process for eliciting and aggregating opinion on a topic with a goal of exploring the existence of consensus among a diverse group of handpicked experts. The Delphi method was developed at the RAND Corporation in the early 1950s to obtain a reliable expert consensus, which is often used as a substitute for empirical evidence when it does not exist.

The four key characteristics of the Delphi method are:

anonymity,
iterative data collection,
participant feedback, and
statistical determination of group response.

As a result, Delphi has become best practice for quantifying the results of group elicitation processes.

Ten dialogue methods for integrating judgments

June 21, 2023August 2, 2022

By David McDonald, Gabriele Bammer and Peter Deane

authors_david-mcdonald_gabriele-bammer_peter-deane — 1. David McDonald (biography)
2. Gabriele Bammer (biography)
3. Peter Deane (biography)

What formal dialogue methods can assist researchers in synthesising judgments about a complex societal or environmental issue when a range of parties with different perspectives are involved? How can researchers decide which methods will be most suitable for their purposes?

We review ten dialogue methods. Our purpose is not to describe the dialogue methods in detail, but instead to review the circumstances in which each method is likely to be most useful in a research context, bearing in mind that most methods a) were not developed for research, b) can be applied flexibly and c) have evolved into different variations. The methods are clustered into six groups:

Judgment and decision making with unknown states and outcomes

April 15, 2022February 15, 2022

By Michael Smithson

What issues arise for effective judgments, predictions, and decisions when decision makers do not know all the potential starting positions, available alternatives and possible outcomes?

A shorthand term for this collection of possible starting points (also known as prior states), alternatives, and outcomes is “sample space.” Here I elucidate why sample space is important and how judgments and decisions can be influenced when it is incomplete.

Why is sample space important?

When it comes to dealing with unknowns, economists and others traditionally distinguish between “risk” (where probabilities can be assigned to every possible outcome) and “uncertainty” (where the probabilities are vague or unknown). Both of those versions of unknowns assume that decision makers know everything about the sample space.

Problem framing and co-creation

August 24, 2022October 27, 2016

By Graeme Nicholas

How can people with quite different ways of ‘seeing’ and thinking about a problem discover and negotiate these differences?

A key element of co-creation is joint problem definition. However, problem definition is likely to be a matter of perspective, or a matter of how each person involved ‘frames’ the problem. Differing frames are inevitable when participants bring their differing expertise and experience to a problem. Methods and processes to support co-creation, then, need to manage the coming together of people with differing ways of framing the problem, so participants can contribute to joint problem definition.

Co-creation without systems thinking can be dangerous

August 24, 2022July 7, 2016

By Gerald Midgley

Why does the theory and practice of co-creation need to be informed by systems thinking? Co-creation without a thorough understanding of systems thinking can be deeply problematic. Essentially, we need a theory and practice of systemic co-creation.

Three key things happen in any co-creation:

It is necessary for a diversity of perspectives to engage.
There is the synergistic innovation that results from this engagement.
The innovation is meaningful in a context of use.