Looking in the right places to identify “unknown unknowns” in projects

By Tyson R. Browning

Unknown unknowns pose a tremendous challenge as they are essentially to blame for many of the unwelcome surprises that pop up to derail projects. However, many, perhaps even most, of these so-called unknown unknowns were actually knowable in advance, if project managers had merely looked in the right places.

For example, investigations following major catastrophes (such as space shuttle disasters, train derailments, and terrorist attacks), and project cost and schedule overruns, commonly identify instances where a key bit of knowledge was in fact known by someone working on that project—but failed to be communicated to the project’s top decision makers. In other cases, unknown unknowns emerge from unforeseen interactions among known elements of complex systems, such as product components, process activities, or software systems.

With the right mindset and toolset, we can shine a light into the right holes to uncover the uncertainties that could affect a project’s success. Various tools for directed recognition can help us convert unknown unknowns (unk unks) to known unknowns, as depicted below (figure adapted from Browning and Ramasesh 2015).

To be clear, I’m referring to a particular context, a project, which is a “temporary endeavor undertaken to create a unique product, service, or result” (Project Management Institute 2017: 4)—although I would expect many of the concepts discussed in this post to apply more generally. In projects, a subset of uncertainties have the potential to have a positive or negative effect on its success. Positive uncertainties present opportunities, while negative ones (threats) present risks. Many formal techniques exist for managing project risks, but all of these begin by identifying the risks in the first place—ie., by rendering them as known unknowns.

Where to look?

In projects, unknown unknowns can emerge from at least six complex systems: the project’s desired result, the work done to get it (process), the people and teams doing the work (organization), the resources and tools they’re using (tools), the project’s requirements and objectives (goals), and the project’s environment (context). Each of these systems, the first five of which we could also call project subsystems, involves a complex network of related elements. These systems are also related with each other. Many of these systems have been studied in isolation; much less have they been studied in tandem. These six systems provide a minimal starting point for searching for project risks and opportunities.

What makes unknown unknowns more likely?

Six factors increase the likelihood of unknown unknowns in projects:

1. Complexity stems from the constituent elements of a system and their interactions. Complexity increases with the number, variety, internal complexity and lack of robustness of its elements, and with the number, variety, criticality and internal complexity of the relationships among these elements.
2. Complicatedness is observer-dependent. It depends on project participants’ abilities to understand and anticipate the project, which depends on the intuitiveness of the project’s structure, organization, and behavior; its newness or novelty; how easy it is to find necessary elements and identify cause and effect relationships; and the participants’ aptitudes and experiences.
3. Dynamism is a system’s propensity to change. Unknown unknowns are more likely to emerge from fast-changing systems.
4. Equivocality refers to imprecise information. In projects, this may manifest as an aspect of poor communication. It clouds judgment and inhibits decision making.
5. Mindlessness refers to perceptive barriers that interfere with the recognition of unknown unknowns, such as an overreliance on past experiences and traditions, the inability to detect weak signals, and ignoring input that is inconvenient or unappealing. It includes individual biases and inappropriate filters such as denial and dismissal.
6. Project pathologies represent structural or behavioral conditions that allow unknown unknowns to remain hidden, including unclear expectations among stakeholders and dysfunctional cultures, such as shooting messengers, covering up failures, discouraging new ideas, and making some topics taboo for discussion.

Knowing places and causes is a good start

When we consider how each of these six factors can affect each of the six project subsystems, we get a 6×6 grid of places to start looking for lurking unknown unknowns. If we save records from past projects, we do not have to start with a blank sheet—but it is just a better starting point, because it is also dangerous to rely completely on historical data. But how do we plumb these 36 places?

Shining the light: tools for directed recognition

Here are eleven types of lights to use for detecting unknown unknowns and converting them into known unknowns:

1. Decompose the project: Model the project’s subsystems. Understand their structures, how their elements relate to one another, and the sub-factors of complexity.
2. Analyze scenarios: Construct several different future outlooks and explore their ramifications.
3. Use checklists: Codify learning from past projects.
4. Scrutinize plans: Independently review a project’s work plans, schedules, resources, budgets, etc.
5. Use “long interviews” (Mullins 2007) with project stakeholders, subject matter experts, and other participants to uncover lurking problems and issues. Such interviews probe deep and wide and ask ‘out-of-the box’ questions to help managers identify latent needs that project stakeholders are unable or unlikely to articulate readily.
6. Pick up weak signals: Weak signals often come in subtle forms, such as a realization that no one in the organization has a complete understanding of a project, unexplained behaviors, or confusing outcomes. Recognizing and interpreting weak signals requires scanning local and extended networks, mobilizing search parties, testing multiple hypotheses, and probing for further clarity.
7. Mine data: Electronic data mining can be a particularly powerful tool for extracting implicit, previously unknown, and potentially useful information. By simultaneously reviewing data from multiple projects, data mining could enable project managers to identify the precursors of potential problems.
8. Communicate frequently and effectively
9. Balance local autonomy and central control. Allow bad news to travel ‘up’ in the organization structure. Provide emergency channels through any bureaucracy.
10. Incentivize discovery. Reward the messenger.
11. Cultivate an alert culture. Educate about unknown unknowns, where they tend to lurk, and why.

I always ask my project and risk management students a question. Given two very similar projects—one with a list of 100 risks identified, and the other with no risks identified—which project is riskier? Unfortunately, many executives and managers seem to get this backwards in practice. Individuals and cultures that prefer (or are incentivized) to ignore uncertainties and risks fuel the delusion and deception problems (eg., Flyvbjerg, et al. 2009) that plague many projects and other endeavors. Many unknown unknowns remain so because of our own lack of will to find and face them. The approaches outlined above should be part of the due diligence of any complex project’s planning and execution.

To find out more:
Browning, T. R. and Ramasesh, R. V. (2015). Reducing Unwelcome Surprises in Project Management. MIT Sloan Management Review, 56, 3: 53-62. (Online): http://sloanreview.mit.edu/x/56319 (The ideas and some of the wording in this blog post come from this article.)

Ramasesh, R. V. and Browning, T. R. (2014). A Conceptual Framework for Tackling Knowable Unknown Unknowns in Project Management. Journal of Operations Management, 32, 4: 190-204. (Online) (DOI): http://dx.doi.org/10.1016/j.jom.2014.03.003

References:
Flyvbjerg, B., Garbuio, M. and Lovallo, D. (2009). Delusion and Deception in Large Infrastructure Projects: Two Models for Explaining and Preventing Executive Disaster. California Management Review, 51, 2: 170-193

Mullins, J. W. (2007). Discovering ‘Unk-Unks’: How Innovators Identify the Critical Things They Don’t Even Know that They Don’t Know. MIT Sloan Management Review, 48, 4: 17-21

Project Management Institute. (2017). A Guide to the Project Management Body of Knowledge, 6th ed., Newtown Square: PA, United States of America

Biography: Tyson R. Browning PhD is Professor of Operations Management in the Neeley School of Business at Texas Christian University in Fort Worth, Texas, USA. His primary research is on managing complex projects. Previously, he worked for Lockheed Martin, Honeywell Space Systems, and Los Alamos National Laboratory. He is currently co-Editor-in-Chief of the Journal of Operations Management.

This blog post is part of a series on unknown unknowns as part of a collaboration between the Australian National University and Defence Science and Technology.

For the five other blog posts already published in this series, see: https://i2insights.org/tag/partner-defence-science-and-technology/

Scheduled blog posts in this series:
November 19: Blackboxing unknown unknowns through vulnerability analysis by Joseph Guillaume
December 3: Yin-yang thinking – A solution to dealing with unknown unknowns? by Christiane Prange and Alicia Hennig
TBA: Detecting non-linear change ‘inside-the-system’ and ‘out-of-the-blue’ by Susan van ‘t Klooster and Marjolijn Haasnoot
January 28, 2020: How can resilience benefit from planning? by Pedro Ferreira
February 11, 2020: Why do we protect ourselves from unknown unknowns? by Bem Le Hunte