Unknowns – Page 3 – Integration and Implementation Insights

Blackboxing unknown unknowns through vulnerability analysis

August 11, 2023November 19, 2019

By Joseph Guillaume

Author - Joseph Guillaume — Joseph Guillaume (biography)

What’s a productive way to think about undesirable outcomes and how to avoid them, especially in an unpredictable future full of unknown unknowns? Here I describe the technique of vulnerability analysis, which essentially has three steps:

Step 1: Identify undesirable outcomes, to be avoided
Step 2: Look for conditions that can lead to such outcomes, ie. vulnerabilities
Step 3: Manage the system to mitigate or adapt to vulnerable conditions.

The power of vulnerability analysis is that, by starting from outcomes, it avoids making assumptions about what led to the vulnerabilities. The causes of the vulnerabilities are effectively a ‘black box’, in other words, they do not need to be understood in order to take effective action. The vulnerability itself is either a known known or a known unknown. The causes of the vulnerability, on the other hand, can be unknown unknowns.

Looking in the right places to identify “unknown unknowns” in projects

August 11, 2023November 5, 2019

Author - Tyson R. Browning — Tyson R. Browning (biography)

By Tyson R. Browning

Unknown unknowns pose a tremendous challenge as they are essentially to blame for many of the unwelcome surprises that pop up to derail projects. However, many, perhaps even most, of these so-called unknown unknowns were actually knowable in advance, if project managers had merely looked in the right places.

For example, investigations following major catastrophes (such as space shuttle disasters, train derailments, and terrorist attacks), and project cost and schedule overruns, commonly identify instances where a key bit of knowledge was in fact known by someone working on that project—but failed to be communicated to the project’s top decision makers. In other cases, unknown unknowns emerge from unforeseen interactions among known elements of complex systems, such as product components, process activities, or software systems.

With the right mindset and toolset, we can shine a light into the right holes to uncover the uncertainties that could affect a project’s success.

Creative writing as a journey into the unknown unknown

August 20, 2024October 22, 2019

By Lelia Green

Author - Lelia Green — Lelia Green (biography)

Do you use writing as a means of accessing your unconscious knowledge and understanding? The electric experience of things falling into place is a well-recorded outcome of ‘writing to find out what you want to say.’ E. L. Doctorow is credited with saying that writing a novel is “like driving at night. You never see further than your headlights, but you can make the whole trip that way” (no formal reference identifiable, but see Quotation Celebration). There is a sense of allowing the unfolding journey to deliver you to your destination, and experiencing the energy rush when you arrive. It’s a matter of relinquishing control and being open to the unexpected.

Australian novelist Helen Milton Bastow described an equivalent revelation. A fiction writer, Bastow had avoided researching Indigenous stories about the places of her childhood in case they overly influenced her own story telling. One day, however, she was on an eco-trail in the Coorong, South Australia, when she happened upon an information display about the local Indigenous dreamtime:

Managing innovation dilemmas: Info-gap theory

August 11, 2023October 8, 2019

By Yakov Ben-Haim

Author - Yakov Ben-Haim — Yakov Ben-Haim (biography)

To use or not to use a new and promising but unfamiliar and hence uncertain innovation? That is the dilemma facing policy makers, engineers, social planners, entrepreneurs, physicians, parents, teachers, and just about everybody in their daily lives. There are new drugs, new energy sources, new foods, new manufacturing technologies, new toys, new pedagogical methods, new weapon systems, new home appliances and many other discoveries and inventions.

Furthermore, the innovation dilemma occurs even when a new technology is not actually involved. The dilemma arises from new attitudes, like individual responsibility for the global environment, or new social conceptions, like global allegiance and self-identity transcending all nation-states. Even the enthusiastic belief in innovation itself as the source of all that is good and worthy entails a dilemma of innovation.

An innovation’s newness and the uncertainty of its promise for improvement is the source of the dilemma. Tomorrow we will understand the innovation better, its dangers and its benefits, but today we must decide.

What do you know? And how is it relevant to unknown unknowns?

August 11, 2023September 24, 2019

By Matthew Welsh

Author - Matthew Welsh — Matthew Welsh (biography)

How can we distinguish between knowledge and ignorance and our meta-knowledge of these – that is, whether we are aware that we know or don’t know any particular thing? The common answer is the 2×2 trope of: known knowns; unknown knowns; known unknowns; and unknown unknowns.

For those interested in helping people navigate a complex world, unknown unknowns are perhaps the trickiest of these to explain – partly because the moment you think of an example, the previously “unknown unknown” morphs into a “known unknown”.

My interest here is to demonstrate that this 2×2 division of knowledge and ignorance is far less crisp than we often assume.

This is because knowledge is not something that exists in the world but rather in individual minds. That is, whether something is ‘known’ depends not on whether someone, somewhere, knows it; but on whether this person, here-and-now does.

How can we know unknown unknowns?

December 13, 2021September 10, 2019

By Michael Smithson

In a 1993 paper, philosopher Ann Kerwin elaborated a view on ignorance that has been summarized in a 2×2 table describing crucial components of metacognition (see figure below). One margin of the table consisted of “knowns” and “unknowns”. The other margin comprised the adjectives “known” and “unknown”. Crosstabulating these produced “known knowns”, “known unknowns”, “unknown knowns”, and unknown unknowns”. The latter two categories have caused some befuddlement. What does it mean to not know what is known, or to not know what is unknown? And how can we convert either of these into their known counterparts?

Accountability and adapting to surprises

July 10, 2026August 27, 2019

By Patricia Hirl Longstaff

Image of Patricia Hirl Longstaff — Patricia Hirl Longstaff (biography)

We have all been there: something bad happens and somebody (maybe an innocent somebody) has their career ruined in order to prove that the problem has been fixed. When is blame appropriate? When is the blame game not only the wrong response, but damaging for long-term decision making?

In a complex and adapting world, errors and failure are not avoidable. The challenges decision-makers and organizations face are sometimes predictable but sometimes brand new. Adapting to surprises requires more flexibility, fewer unbreakable rules, more improvisation and deductive tinkering, and a lot more information about what’s going right and going wrong. But getting there is not easy because this challenges some very closely held assumptions about how the world works and our desire to control things.

Let’s not kid ourselves. Sometimes people do really dumb things that they should be blamed for. What we need is to be more discriminating about when finding blame and accountability is appropriate.

Designing scenarios to guide robust decisions

April 4, 2023July 2, 2019

By Bonnie McBain

What makes scenarios useful to decision makers in effectively planning for the future? Here I discuss three aspects of scenarios:

goals;
design; and,
use and defensibility.

Goals of scenarios

Since predicting the future is not possible, it’s important to know that scenarios are not predictions. Instead, scenarios stimulate thinking and conversations about possible futures.

Key goals and purposes of scenarios can be any of the following:

Managing uncertainty in decision making: What can we learn from economics?

August 11, 2023June 4, 2019

By Siobhan Bourke and Emily Lancsar

authors_siobhan-bourke_emily-lancsar — 1. Siobhan Bourke (biography)
2. Emily Lancsar (biography)

How can researchers interested in complex societal and environmental problems best understand and deal with uncertainty, which is an inherent part of the world in which we live? Accidents happen, governments change, technological innovation occurs making some products and services obsolete, markets boom and inevitably go bust. How can uncertainty be managed when all possible outcomes of an action or decision cannot be known? In particular, are there lessons from the discipline of economics which have broader applicability?

While uncertainty is often discussed alongside risk, a fundamental difference between uncertainty and risk is that risk involves events with known probabilities (or probabilities based on reliable empirical evidence), whereas under uncertainty probabilities are unknown and reflect an individual’s subjective belief concerning the likelihood of a given outcome. Given the subjectivity, that likelihood can differ from person to person. It can also involve a perceived zero probability in the case of unforeseen events (or ‘unknown unknowns’).

What every interdisciplinarian should know about p values

April 15, 2022April 30, 2019

By Alice Richardson

In interdisciplinary research it’s common for at least some data to be analysed using statistical techniques. Have you been taught to look for ‘p < 0.05’ meaning that there is a less than 5% probability that the finding occurred by chance? Do you look askance at your statistician colleagues when they tell you it’s not so simple? Here’s why you need to believe them.

The whole focus on p < 0.05 to the exclusion of all else is a historical hiccup, based on a throwaway line in a manual for research workers. That manual was produced by none other than R.A. Fisher, giant of statistical inference and inventor of statistical methods ranging from the randomised block design to the analysis of variance. But all he said was that “[p = 0.05] is convenient to take … as a limit in judging whether a deviation is to be considered significant or not.” Convenient, nothing more!

Knowledge asymmetry in interdisciplinary collaborations and how to reduce it

January 21, 2026February 26, 2019

By Max Kemman

How can tasks and goals among partners in a collaboration be effectively negotiated, especially when one party is dependent on the deliverables of another party? How does knowledge asymmetry affect such negotiations? What is knowledge asymmetry anyway and how can it be dealt with?

What is knowledge asymmetry?

My PhD research involves historians who are dependent on computational experts to develop an algorithm or user interface for historical research. They therefore needed to be aware of what the computational experts were doing.

Managing deep uncertainty: Exploratory modeling, adaptive plans and joint sense making

April 15, 2022August 15, 2017

By Jan Kwakkel

How can decision making on complex systems come to grips with irreducible, or deep, uncertainty? Such uncertainty has three sources:

Intrinsic limits to predictability in complex systems.
A variety of stakeholders with different perspectives on what the system is and what problem needs to be solved.
Complex systems are generally subject to dynamic change, and can never be completely understood.

Deep uncertainty means that the various parties to a decision do not know or cannot agree on how the system works, how likely various possible future states of the world are, and how important the various outcomes of interest are.