Methods – Integration and Implementation Insights

Generating evidence using the Delphi method

November 14, 2023October 17, 2023

By Dmitry Khodyakov

What is Delphi? How has the Delphi method stood up over time? How can the best of Delphi be adapted to new circumstances and problems?

The Delphi method is a group-based process for eliciting and aggregating opinion on a topic with a goal of exploring the existence of consensus among a diverse group of handpicked experts. The Delphi method was developed at the RAND Corporation in the early 1950s to obtain a reliable expert consensus, which is often used as a substitute for empirical evidence when it does not exist.

The four key characteristics of the Delphi method are:

anonymity,
iterative data collection,
participant feedback, and
statistical determination of group response.

As a result, Delphi has become best practice for quantifying the results of group elicitation processes.

Making the Nominal Group Technique more accessible

September 5, 2023

By Jason Olsen

Looking to gain real insights from those with lived experience about a specific topic? Interested in a low-cost method that fosters equal participation and discussion over participant domination in a research focus group? Want to know about modifications to make pan-disability (ie., working with participants with different impairments) research focus groups more inclusive?

The Nominal Group Technique developed by Ven and Delbecq (1972) has been used for more than 50 years. Key to its success is the posing of a single unambiguous and unbiased question about a problem that can generate a wide range of answers. The process structures the meeting to enable critical dimensions of the question to be identified, ranked and rated in a way that:

limits the influence of the researcher leading the project, as well as the influence of attendees,
allows participants to clarify the question’s dimensions and gaps,
increases the likelihood of equal participation for all group members,
affords equal influence to different, and potentially conflicting, values and ideas.

Keyword quiz: an icebreaker method for interdisciplinary teams

April 11, 2023March 14, 2023

By Sebastian Rogga and Anton Parisi

authors_sebastian-rogga_anton-parisi — 1. Sebastian Rogga (biography)
2. Anton Parisi (biography)

How can members of interdisciplinary teams quickly gain a better understanding of each other’s thematic preferences and skills in a way that is also engaging and fun?

We have developed a “keyword quiz” icebreaker method to facilitate exchange between members of interdisciplinary teams, especially between people who are not complete strangers to each other but are collaborating in a project context for the first time.

In brief, the idea is to communicate each member’s scientific profile based on keywords from publications that the team members have published and that they have selected based on specific categories.

The keywords of a publication are presented visually to the whole group and the team members then guess, in the form of a quiz, which team member published the associated publication.

After the author has been revealed, they share the reasons for choosing the publication.

Seven methods for mapping systems

February 28, 2023

By Pete Barbrook-Johnson and Alexandra S. Penn

authors_pete-barbrook-johnson_alexandra-penn — 1. Pete Barbrook-Johnson (biography)
2. Alexandra S. Penn (biography)

What are some effective approaches for developing causal maps of systems in participatory ways? How do different approaches relate to each other and what are the ways in which systems maps can be useful?

Here we focus on seven system mapping methods, described briefly in alphabetical order.

1. Bayesian Belief Networks: a network of variables representing their conditional dependencies (ie., the likelihood of the variable taking different states depending on the states of the variables that influence them). The networks follow a strict acyclic structure (ie., no feedbacks), and nodes tend to be restricted to maximum two incoming arrows. These maps are analysed using the conditional probabilities to compute the potential impact of changes to certain variables, or the influence of certain variables given an observed outcome.

Externalizing implicit expectations and assumptions in transdisciplinary research

August 3, 2023January 17, 2023

By Verena Radinger-Peer, Katharina Gugerell and Marianne Penker

authors_verena-radinger-peer_katharina-gugerell_marianne-penker — 1. Verena Radinger-Peer (biography)
2. Katharina Gugerell (biography)
3. Marianne Penker (biography

How can implicit expectations and assumptions of team members in transdisciplinary research collaborations be identified?

We used Q-methodology as a tool to make diverse expectations and perceptions of transdisciplinary research collaborations tangible and thus negotiable.

Q-methodology is an established explorative, semi-quantitative method for investigating distinctive viewpoints of a given population based on inverted factor analysis. While we do not explain Q methodology here, it is increasingly used and we refer those who want to find out more to Watts and Stenner (2012).

One disadvantage of the Q-method is the amount of time and effort that has to be invested in developing the Q-statements. Here we offer the statements we developed through an extensive process in our study for others to use either in their own Q methodology or in surveys.

Improving cross-disciplinary collaboration with strategy knotworking and ecocycle planning

December 13, 2022November 22, 2022

By Nancy White

How can cross-disciplinary teams improve their project results and cross-team learning, especially when they are part of a portfolio of funded projects?

I have worked with cross disciplinary teams in international agriculture development, ecosystems management and mental health. For the most part, these are externally funded initiatives and have requirements both for results (application of the work) and for cross-team learning. Often there is not useful clarity about how funder and grantee agendas work in sync. And there is rarely opportunity or support for shared optimization and exploration across different portfolios of funded work.

I have used the six knotworking questions plus ecocycle planning from Liberating Structures to make it possible for a group to look back critically, assess the current state, and prospectively generate options to move forward.

Dealing with differences in interests through principled negotiation

November 15, 2022

By Gabriele Bammer

gabriele-bammer_nov-2021 — Gabriele Bammer (biography)

How can the interests of a diverse group of researchers and stakeholders tackling a complex societal problem be understood and managed?

Interests arise when a person has a stake in something and stands to gain or lose depending on what happens to that something:

researchers commonly have a stake in advancing their work and careers,
stakeholders affected by a societal problem generally have a stake in improving the problem, and
stakeholders in a position to do something about a problem generally have a stake in improving outcomes for the problem through their sphere of influence.

Interests relate not only to personal conditions or stakes (self-interest), but also to principles such as reducing inequities and promoting justice.

Creative destruction

November 29, 2022November 8, 2022

By Keith McCandless

author_keith-mccandless — Keith McCandless (biography)

My favorite part of working with groups is helping people notice and stop counterproductive behavior. We all have self-limiting individual and group behaviors. Of course, they are easier to spot in others than in ourselves. So, finding seriously fun ways to help people discover for themselves what they can stop doing is important.

I use an activity called TRIZ from Liberating Structures. The purpose of TRIZ is to:

Make it possible to speak the unspeakable and get skeletons out of the closet
Make space for innovation
Lay the ground for creative destruction by doing the hard work in a fun way
TRIZ may be used before or in place of visioning sessions
Build trust by acting all together to remove barriers.

Multidisciplinary perspectives on unknown unknowns

June 21, 2023September 6, 2022

By Gabriele Bammer

This is part of a series of occasional “synthesis blog posts” drawing together perspectives on related topics across i2Insights contributions.

How can different disciplines and practitioners enhance the ability to understand and manage unknown unknowns, also referred to as deep uncertainty?

Seventeen blog posts have addressed these issues, covering:

how unknown unknowns can be understood
exploiting unknown unknowns
accepting unknown unknowns
reducing unknown unknowns.

Ten dialogue methods for integrating judgments

June 21, 2023August 2, 2022

By David McDonald, Gabriele Bammer and Peter Deane

authors_david-mcdonald_gabriele-bammer_peter-deane — 1. David McDonald (biography)
2. Gabriele Bammer (biography)
3. Peter Deane (biography)

What formal dialogue methods can assist researchers in synthesising judgments about a complex societal or environmental issue when a range of parties with different perspectives are involved? How can researchers decide which methods will be most suitable for their purposes?

We review ten dialogue methods. Our purpose is not to describe the dialogue methods in detail, but instead to review the circumstances in which each method is likely to be most useful in a research context, bearing in mind that most methods a) were not developed for research, b) can be applied flexibly and c) have evolved into different variations. The methods are clustered into six groups:

Understanding diversity primer: 3. Perceptions of good research

June 23, 2022May 5, 2022

By Gabriele Bammer

primer_diversity_3

How do different perceptions arise of what makes for ‘good’ research? How can researchers come to understand such differences and their impacts on how problems are framed, understood and responded to, as well as how they affect the ability of those contributing to the research to work together?

Differences arise because training in a discipline involves inculcating a specific way of investigating the world, including which types of questions are worth addressing; legitimate ways of gathering, analysing and interpreting data; standards for validation; and the role of values in the research process. Educating someone in a discipline aims to make the discipline’s specific approach to research ingrained and tacit.

The MATRICx: Measuring motivation in science teams

April 19, 2022April 5, 2022

By Gaetano R. Lotrecchiano

gaetano-lotrecchiano — Gaetano R. Lotrecchiano (biography)

What motivates scientists to work in teams? How can we measure motivation? Why should we be concerned about motivation in science teams?

Six domains of motivation for collaboration

Scientists and science stakeholders draw on different motivations to collaborate. The literature has discussed these motivations in different ways:

1. Advancing Science: Motivations to contribute to an agenda or the progression of research and science.

2. Building Relationships: Motivations to utilize resources and/or knowledge to establish or expand connections and one’s network of collaborators.