Looking in the right places to identify “unknown unknowns” in projects

Author - Tyson R. Browning
Tyson R. Browning (biography)

By Tyson R. Browning

Unknown unknowns pose a tremendous challenge as they are essentially to blame for many of the unwelcome surprises that pop up to derail projects. However, many, perhaps even most, of these so-called unknown unknowns were actually knowable in advance, if project managers had merely looked in the right places.

For example, investigations following major catastrophes (such as space shuttle disasters, train derailments, and terrorist attacks), and project cost and schedule overruns, commonly identify instances where a key bit of knowledge was in fact known by someone working on that project—but failed to be communicated to the project’s top decision makers. In other cases, unknown unknowns emerge from unforeseen interactions among known elements of complex systems, such as product components, process activities, or software systems.

With the right mindset and toolset, we can shine a light into the right holes to uncover the uncertainties that could affect a project’s success. Various tools for directed recognition can help us convert unknown unknowns (unk unks) to known unknowns, as depicted below (figure adapted from Browning and Ramasesh 2015).

directed recognition for converting unknown unknowns to known unknowns

To be clear, I’m referring to a particular context, a project, which is a “temporary endeavor undertaken to create a unique product, service, or result” (Project Management Institute 2017: 4)—although I would expect many of the concepts discussed in this post to apply more generally. In projects, a subset of uncertainties have the potential to have a positive or negative effect on its success. Positive uncertainties present opportunities, while negative ones (threats) present risks. Many formal techniques exist for managing project risks, but all of these begin by identifying the risks in the first place—ie., by rendering them as known unknowns.

Where to look?

In projects, unknown unknowns can emerge from at least six complex systems: the project’s desired result, the work done to get it (process), the people and teams doing the work (organization), the resources and tools they’re using (tools), the project’s requirements and objectives (goals), and the project’s environment (context). Each of these systems, the first five of which we could also call project subsystems, involves a complex network of related elements. These systems are also related with each other. Many of these systems have been studied in isolation; much less have they been studied in tandem. These six systems provide a minimal starting point for searching for project risks and opportunities.

What makes unknown unknowns more likely?

Six factors increase the likelihood of unknown unknowns in projects:

  1. Complexity stems from the constituent elements of a system and their interactions. Complexity increases with the number, variety, internal complexity and lack of robustness of its elements, and with the number, variety, criticality and internal complexity of the relationships among these elements.
  2. Complicatedness is observer-dependent. It depends on project participants’ abilities to understand and anticipate the project, which depends on the intuitiveness of the project’s structure, organization, and behavior; its newness or novelty; how easy it is to find necessary elements and identify cause and effect relationships; and the participants’ aptitudes and experiences.
  3. Dynamism is a system’s propensity to change. Unknown unknowns are more likely to emerge from fast-changing systems.
  4. Equivocality refers to imprecise information. In projects, this may manifest as an aspect of poor communication. It clouds judgment and inhibits decision making.
  5. Mindlessness refers to perceptive barriers that interfere with the recognition of unknown unknowns, such as an overreliance on past experiences and traditions, the inability to detect weak signals, and ignoring input that is inconvenient or unappealing. It includes individual biases and inappropriate filters such as denial and dismissal.
  6. Project pathologies represent structural or behavioral conditions that allow unknown unknowns to remain hidden, including unclear expectations among stakeholders and dysfunctional cultures, such as shooting messengers, covering up failures, discouraging new ideas, and making some topics taboo for discussion.

Knowing places and causes is a good start

When we consider how each of these six factors can affect each of the six project subsystems, we get a 6×6 grid of places to start looking for lurking unknown unknowns. If we save records from past projects, we do not have to start with a blank sheet—but it is just a better starting point, because it is also dangerous to rely completely on historical data. But how do we plumb these 36 places?

Shining the light: tools for directed recognition

Here are eleven types of lights to use for detecting unknown unknowns and converting them into known unknowns:

  1. Decompose the project: Model the project’s subsystems. Understand their structures, how their elements relate to one another, and the sub-factors of complexity.
  2. Analyze scenarios: Construct several different future outlooks and explore their ramifications.
  3. Use checklists: Codify learning from past projects.
  4. Scrutinize plans: Independently review a project’s work plans, schedules, resources, budgets, etc.
  5. Use “long interviews” (Mullins 2007) with project stakeholders, subject matter experts, and other participants to uncover lurking problems and issues. Such interviews probe deep and wide and ask ‘out-of-the box’ questions to help managers identify latent needs that project stakeholders are unable or unlikely to articulate readily.
  6. Pick up weak signals: Weak signals often come in subtle forms, such as a realization that no one in the organization has a complete understanding of a project, unexplained behaviors, or confusing outcomes. Recognizing and interpreting weak signals requires scanning local and extended networks, mobilizing search parties, testing multiple hypotheses, and probing for further clarity.
  7. Mine data: Electronic data mining can be a particularly powerful tool for extracting implicit, previously unknown, and potentially useful information. By simultaneously reviewing data from multiple projects, data mining could enable project managers to identify the precursors of potential problems.
  8. Communicate frequently and effectively
  9. Balance local autonomy and central control. Allow bad news to travel ‘up’ in the organization structure. Provide emergency channels through any bureaucracy.
  10. Incentivize discovery. Reward the messenger.
  11. Cultivate an alert culture. Educate about unknown unknowns, where they tend to lurk, and why.

I always ask my project and risk management students a question. Given two very similar projects—one with a list of 100 risks identified, and the other with no risks identified—which project is riskier? Unfortunately, many executives and managers seem to get this backwards in practice. Individuals and cultures that prefer (or are incentivized) to ignore uncertainties and risks fuel the delusion and deception problems (eg., Flyvbjerg, et al. 2009) that plague many projects and other endeavors. Many unknown unknowns remain so because of our own lack of will to find and face them. The approaches outlined above should be part of the due diligence of any complex project’s planning and execution.

To find out more:
Browning, T. R. and Ramasesh, R. V. (2015). Reducing Unwelcome Surprises in Project Management. MIT Sloan Management Review, 56, 3: 53-62. (Online): http://sloanreview.mit.edu/x/56319 (The ideas and some of the wording in this blog post come from this article.)

Ramasesh, R. V. and Browning, T. R. (2014). A Conceptual Framework for Tackling Knowable Unknown Unknowns in Project Management. Journal of Operations Management, 32, 4: 190-204. (Online) (DOI): http://dx.doi.org/10.1016/j.jom.2014.03.003

Flyvbjerg, B., Garbuio, M. and Lovallo, D. (2009). Delusion and Deception in Large Infrastructure Projects: Two Models for Explaining and Preventing Executive Disaster. California Management Review, 51, 2: 170-193

Mullins, J. W. (2007). Discovering ‘Unk-Unks’: How Innovators Identify the Critical Things They Don’t Even Know that They Don’t Know. MIT Sloan Management Review, 48, 4: 17-21

Project Management Institute. (2017). A Guide to the Project Management Body of Knowledge, 6th ed., Newtown Square: PA, United States of America

Biography: Tyson R. Browning PhD is Professor of Operations Management in the Neeley School of Business at Texas Christian University in Fort Worth, Texas, USA. His primary research is on managing complex projects. Previously, he worked for Lockheed Martin, Honeywell Space Systems, and Los Alamos National Laboratory. He is currently co-Editor-in-Chief of the Journal of Operations Management.

This blog post is part of a series on unknown unknowns as part of a collaboration between the Australian National University and Defence Science and Technology.

For the five other blog posts already published in this series, see: https://i2insights.org/tag/partner-defence-science-and-technology/

Scheduled blog posts in this series:
November 19: Blackboxing unknown unknowns through vulnerability analysis by Joseph Guillaume
December 3: Yin-yang thinking – A solution to dealing with unknown unknowns? by Christiane Prange and Alicia Hennig
TBA: Detecting non-linear change ‘inside-the-system’ and ‘out-of-the-blue’ by Susan van ‘t Klooster and Marjolijn Haasnoot
January 28, 2020: How can resilience benefit from planning? by Pedro Ferreira
February 11, 2020: Why do we protect ourselves from unknown unknowns? by Bem Le Hunte

Creative writing as a journey into the unknown unknown

By Lelia Green

Author - Lelia Green
Lelia Green (biography)

Do you use writing as a means of accessing your unconscious knowledge and understanding? The electric experience of things falling into place is a well-recorded outcome of ‘writing to find out what you want to say.’ E. L. Doctorow is credited with saying that writing a novel is “like driving at night. You never see further than your headlights, but you can make the whole trip that way” (no formal reference identifiable, but see Quotation Celebration). There is a sense of allowing the unfolding journey to deliver you to your destination, and experiencing the energy rush when you arrive. It’s a matter of relinquishing control and being open to the unexpected. Continue reading

Managing innovation dilemmas: Info-gap theory

By Yakov Ben-Haim

Author - Yakov Ben-Haim
Yakov Ben-Haim (biography)

To use or not to use a new and promising but unfamiliar and hence uncertain innovation? That is the dilemma facing policy makers, engineers, social planners, entrepreneurs, physicians, parents, teachers, and just about everybody in their daily lives. There are new drugs, new energy sources, new foods, new manufacturing technologies, new toys, new pedagogical methods, new weapon systems, new home appliances and many other discoveries and inventions.

Furthermore, the innovation dilemma occurs even when a new technology is not actually involved. The dilemma arises from new attitudes, like individual responsibility for the global environment, or new social conceptions, like global allegiance and self-identity transcending all nation-states. Even the enthusiastic belief in innovation itself as the source of all that is good and worthy entails a dilemma of innovation. Continue reading

What do you know? And how is it relevant to unknown unknowns?

By Matthew Welsh

Author - Matthew Welsh
Matthew Welsh (biography)

How can we distinguish between knowledge and ignorance and our meta-knowledge of these – that is, whether we are aware that we know or don’t know any particular thing? The common answer is the 2×2 trope of: known knowns; unknown knowns; known unknowns; and unknown unknowns.

For those interested in helping people navigate a complex world, unknown unknowns are perhaps the trickiest of these to explain – partly because the moment you think of an example, the previously “unknown unknown” morphs into a “known unknown”.

My interest here is to demonstrate that this 2×2 division of knowledge and ignorance is far less crisp than we often assume.

This is because knowledge is not something that exists in the world but rather in individual minds. That is, whether something is ‘known’ depends not on whether someone, somewhere, knows it; but on whether this person, here-and-now does. Continue reading

How can we know unknown unknowns?

By Michael Smithson

Michael Smithson
Michael Smithson (biography)

In a 1993 paper, philosopher Ann Kerwin elaborated a view on ignorance that has been summarized in a 2×2 table describing crucial components of metacognition (see figure below). One margin of the table consisted of “knowns” and “unknowns”. The other margin comprised the adjectives “known” and “unknown”. Crosstabulating these produced “known knowns”, “known unknowns”, “unknown knowns”, and unknown unknowns”. The latter two categories have caused some befuddlement. What does it mean to not know what is known, or to not know what is unknown? And how can we convert either of these into their known counterparts? Continue reading

Accountability and adapting to surprises

By Patricia Hirl Longstaff

Image of Patricia Hirl Longstaff
Patricia Hirl Longstaff (biography)

We have all been there: something bad happens and somebody (maybe an innocent somebody) has their career ruined in order to prove that the problem has been fixed. When is blame appropriate? When is the blame game not only the wrong response, but damaging for long-term decision making?

In a complex and adapting world, errors and failure are not avoidable. The challenges decision-makers and organizations face are sometimes predictable but sometimes brand new. Adapting to surprises requires more flexibility, fewer unbreakable rules, more improvisation and deductive tinkering, and a lot more information about what’s going right and going wrong. But getting there is not easy because this challenges some very closely held assumptions about how the world works and our desire to control things. Continue reading