Ten insights on the interplay between evidence and policy

By Kat Smith and Paul Cairney

authors_kat-smith_paul-cairney
1. Kat Smith (biography)
2. Paul Cairney (biography)

How can we improve the way we think about the relationship between evidence and policy? What are the key insights that existing research provides?

1. Evidence does not tell us what to do

It helps reduce uncertainty, but does not tell us how we should interpret problems or what to do about them.

2. There is no such thing as ‘the evidence’

Instead, there is a large number of researchers with different backgrounds, making different assumptions, asking different questions, using different methods, and addressing different problems.

Read moreTen insights on the interplay between evidence and policy

Considering uncertainty, awareness and ambiguity as a three-dimensional space

By Fabio Boschetti

author-fabio-boschetti
Fabio Boschetti (biography)

The concept of unknown unknowns highlights the importance of introspection in assessing knowledge. It suggests that finding our way in the set of known-knowns, known-unknowns, unknown-knowns and unknown-unknowns, reduces to asking:

  1. how uncertain are we? and
  2. how aware are we of uncertainty?

When a problem involves a decision-making team, rather than a single individual, we also need to ask:

  1. how do context and perception affect what we know?

Read moreConsidering uncertainty, awareness and ambiguity as a three-dimensional space

How can resilience benefit from planning?

By Pedro Ferreira

author - pedro-ferreira
Pedro Ferreira (biography)

Improved resilience can contribute to the ability to deal with unknown unknowns. Dealing with uncertainty is also at the core of every planning activity. The argument put forward here is that planning processes should be considered a cornerstone for any given resilience approach. An outline of planning and resilience is given, before presenting fundamental aspects of planning that should be strengthened within a resilience strategy.

Planning

From attempting to do as much as possible within a day’s work, to launching rockets into space or managing a nation, everything requires planning.

Read moreHow can resilience benefit from planning?

Yin-yang thinking – A solution to dealing with unknown unknowns?

By Christiane Prange and Alicia Hennig

author - christiane prange
Christiane Prange (biography)

Sometimes, we wonder why decisions in Asia are being made at gargantuan speed. How do Asians deal with uncertainty arising from unknown unknowns? Can yin-yang thinking that is typical for several Asian cultures provide a useful answer?

Let’s look at differences between Asian and Western thinking first. Western people tend to prefer strategic planning with linear extrapolation of things past. The underlying mantra is risk management to buffer the organization and to protect it from harmful consequences for the business. But juxtaposing risk and uncertainty is critical. Under conditions of uncertainty, linearity is at stake and risk management limited.

Read moreYin-yang thinking – A solution to dealing with unknown unknowns?

Looking in the right places to identify “unknown unknowns” in projects

Author - Tyson R. Browning
Tyson R. Browning (biography)

By Tyson R. Browning

Unknown unknowns pose a tremendous challenge as they are essentially to blame for many of the unwelcome surprises that pop up to derail projects. However, many, perhaps even most, of these so-called unknown unknowns were actually knowable in advance, if project managers had merely looked in the right places.

For example, investigations following major catastrophes (such as space shuttle disasters, train derailments, and terrorist attacks), and project cost and schedule overruns, commonly identify instances where a key bit of knowledge was in fact known by someone working on that project—but failed to be communicated to the project’s top decision makers. In other cases, unknown unknowns emerge from unforeseen interactions among known elements of complex systems, such as product components, process activities, or software systems.

Read moreLooking in the right places to identify “unknown unknowns” in projects

Designing scenarios to guide robust decisions

By Bonnie McBain

Bonnie McBain (biography)

What makes scenarios useful to decision makers in effectively planning for the future? Here I discuss three aspects of scenarios:

  • goals;
  • design; and,
  • use and defensibility.

Goals of scenarios

Since predicting the future is not possible, it’s important to know that scenarios are not predictions. Instead, scenarios stimulate thinking and conversations about possible futures.

Read moreDesigning scenarios to guide robust decisions

Managing uncertainty in decision making: What can we learn from economics?

By Siobhan Bourke and Emily Lancsar

Siobhan Bourke (biography)

How can researchers interested in complex societal and environmental problems best understand and deal with uncertainty, which is an inherent part of the world in which we live? Accidents happen, governments change, technological innovation occurs making some products and services obsolete, markets boom and inevitably go bust. How can uncertainty be managed when all possible outcomes of an action or decision cannot be known? In particular, are there lessons from the discipline of economics which have broader applicability?

Read moreManaging uncertainty in decision making: What can we learn from economics?

What every interdisciplinarian should know about p values

By Alice Richardson

Alice Richardson (biography)

In interdisciplinary research it’s common for at least some data to be analysed using statistical techniques. Have you been taught to look for ‘p < 0.05’ meaning that there is a less than 5% probability that the finding occurred by chance? Do you look askance at your statistician colleagues when they tell you it’s not so simple? Here’s why you need to believe them.

The whole focus on p < 0.05 to the exclusion of all else is a historical hiccup, based on a throwaway line in a manual for research workers. That manual was produced by none other than R.A. Fisher, giant of statistical inference and inventor of statistical methods ranging from the randomised block design to the analysis of variance. But all he said was that “[p = 0.05] is convenient to take … as a limit in judging whether a deviation is to be considered significant or not.” Convenient, nothing more!

Read moreWhat every interdisciplinarian should know about p values

Conceptual modelling of complex topics: ConML as an example / Modelado conceptual de temas complejos: ConML como ejemplo

By Cesar Gonzalez-Perez

cesar-gonzalez-perez
Cesar Gonzalez-Perez (biography)

A Spanish version of this post is available

What are conceptual models? How can conceptual modelling effectively represent complex topics and assist communication among people from different backgrounds and disciplines?

This blog post describes ConML, which stands for “Conceptual Modelling Language”. ConML is a specific modelling language that was designed to allow researchers who are not expert in information technologies to create and develop their own conceptual models. It is useful for the humanities, social sciences and experimental sciences.

Read moreConceptual modelling of complex topics: ConML as an example / Modelado conceptual de temas complejos: ConML como ejemplo

Uncertainty in participatory modeling – What can we learn from management research?

By Antonie Jetter

antonie-jetter
Antonie Jetter (biography)

I frequently struggle to explain how participatory modeling deals with uncertainty. I found useful guidance in the management literature.

After all, participatory modeling projects and strategic business planning have one commonality – a group of stakeholders and decision-makers aims to understand and ultimately influence a complex system. They do so in the face of great uncertainty that frequently cannot be resolved – at least not within the required time frame. Businesses, for example, have precise data on customer behavior when their accountants report on annual sales. However, by this time, the very precise data is irrelevant because the opportunity to influence the system has passed.

Two key lessons from the management literature deal with the nature of uncertainty and responding to four major types of uncertainty.

Read moreUncertainty in participatory modeling – What can we learn from management research?

Making predictions under uncertainty

By Joseph Guillaume

Joseph Guillaume (biography)

Prediction under uncertainty is typically seen as a daunting task. It conjures up images of clouded crystal balls and mysterious oracles in shadowy temples. In a modelling context, it might raise concerns about conclusions built on doubtful assumptions about the future, or about the difficulty in making sense of the many sources of uncertainty affecting highly complex models.

However, prediction under uncertainty can be made tractable depending on the type of prediction. Here I describe ways of making predictions under uncertainty for testing which conclusion is correct. Suppose, for example, that you want to predict whether objectives will be met. There are two possible conclusions – Yes and No, so prediction in this case involves testing which of these competing conclusions is plausible.

Read moreMaking predictions under uncertainty

In praise of multidisciplinarity

By Gabriele Bammer

gabriele-bammer
Gabriele Bammer (biography)

What characterizes multidisciplinary research? When is it most appropriate? What does it take to do it well? Multidisciplinarity often gets a bad rap, being seen as less sophisticated than interdisciplinarity and transdisciplinarity. But does it have its own important role in dealing with complex social and environmental problems?

Multidisciplinary research has two primary characteristics:

  1. different disciplines independently shine their light on a particular problem, and
  2. synthesis happens at the end and can be undertaken by anyone.

Unlike interdisciplinary and transdisciplinary research, there is no attempt to agree upfront on either a problem definition or on how the different perspectives will be brought together.

Read moreIn praise of multidisciplinarity