Looking in the right places to identify “unknown unknowns” in projects

Author - Tyson R. Browning
Tyson R. Browning (biography)

By Tyson R. Browning

Unknown unknowns pose a tremendous challenge as they are essentially to blame for many of the unwelcome surprises that pop up to derail projects. However, many, perhaps even most, of these so-called unknown unknowns were actually knowable in advance, if project managers had merely looked in the right places.

For example, investigations following major catastrophes (such as space shuttle disasters, train derailments, and terrorist attacks), and project cost and schedule overruns, commonly identify instances where a key bit of knowledge was in fact known by someone working on that project—but failed to be communicated to the project’s top decision makers. In other cases, unknown unknowns emerge from unforeseen interactions among known elements of complex systems, such as product components, process activities, or software systems.

With the right mindset and toolset, we can shine a light into the right holes to uncover the uncertainties that could affect a project’s success. Various tools for directed recognition can help us convert unknown unknowns (unk unks) to known unknowns, as depicted below (figure adapted from Browning and Ramasesh 2015).

directed recognition for converting unknown unknowns to known unknowns

To be clear, I’m referring to a particular context, a project, which is a “temporary endeavor undertaken to create a unique product, service, or result” (Project Management Institute 2017: 4)—although I would expect many of the concepts discussed in this post to apply more generally. In projects, a subset of uncertainties have the potential to have a positive or negative effect on its success. Positive uncertainties present opportunities, while negative ones (threats) present risks. Many formal techniques exist for managing project risks, but all of these begin by identifying the risks in the first place—ie., by rendering them as known unknowns.

Where to look?

In projects, unknown unknowns can emerge from at least six complex systems: the project’s desired result, the work done to get it (process), the people and teams doing the work (organization), the resources and tools they’re using (tools), the project’s requirements and objectives (goals), and the project’s environment (context). Each of these systems, the first five of which we could also call project subsystems, involves a complex network of related elements. These systems are also related with each other. Many of these systems have been studied in isolation; much less have they been studied in tandem. These six systems provide a minimal starting point for searching for project risks and opportunities.

What makes unknown unknowns more likely?

Six factors increase the likelihood of unknown unknowns in projects:

  1. Complexity stems from the constituent elements of a system and their interactions. Complexity increases with the number, variety, internal complexity and lack of robustness of its elements, and with the number, variety, criticality and internal complexity of the relationships among these elements.
  2. Complicatedness is observer-dependent. It depends on project participants’ abilities to understand and anticipate the project, which depends on the intuitiveness of the project’s structure, organization, and behavior; its newness or novelty; how easy it is to find necessary elements and identify cause and effect relationships; and the participants’ aptitudes and experiences.
  3. Dynamism is a system’s propensity to change. Unknown unknowns are more likely to emerge from fast-changing systems.
  4. Equivocality refers to imprecise information. In projects, this may manifest as an aspect of poor communication. It clouds judgment and inhibits decision making.
  5. Mindlessness refers to perceptive barriers that interfere with the recognition of unknown unknowns, such as an overreliance on past experiences and traditions, the inability to detect weak signals, and ignoring input that is inconvenient or unappealing. It includes individual biases and inappropriate filters such as denial and dismissal.
  6. Project pathologies represent structural or behavioral conditions that allow unknown unknowns to remain hidden, including unclear expectations among stakeholders and dysfunctional cultures, such as shooting messengers, covering up failures, discouraging new ideas, and making some topics taboo for discussion.

Knowing places and causes is a good start

When we consider how each of these six factors can affect each of the six project subsystems, we get a 6×6 grid of places to start looking for lurking unknown unknowns. If we save records from past projects, we do not have to start with a blank sheet—but it is just a better starting point, because it is also dangerous to rely completely on historical data. But how do we plumb these 36 places?

Shining the light: tools for directed recognition

Here are eleven types of lights to use for detecting unknown unknowns and converting them into known unknowns:

  1. Decompose the project: Model the project’s subsystems. Understand their structures, how their elements relate to one another, and the sub-factors of complexity.
  2. Analyze scenarios: Construct several different future outlooks and explore their ramifications.
  3. Use checklists: Codify learning from past projects.
  4. Scrutinize plans: Independently review a project’s work plans, schedules, resources, budgets, etc.
  5. Use “long interviews” (Mullins 2007) with project stakeholders, subject matter experts, and other participants to uncover lurking problems and issues. Such interviews probe deep and wide and ask ‘out-of-the box’ questions to help managers identify latent needs that project stakeholders are unable or unlikely to articulate readily.
  6. Pick up weak signals: Weak signals often come in subtle forms, such as a realization that no one in the organization has a complete understanding of a project, unexplained behaviors, or confusing outcomes. Recognizing and interpreting weak signals requires scanning local and extended networks, mobilizing search parties, testing multiple hypotheses, and probing for further clarity.
  7. Mine data: Electronic data mining can be a particularly powerful tool for extracting implicit, previously unknown, and potentially useful information. By simultaneously reviewing data from multiple projects, data mining could enable project managers to identify the precursors of potential problems.
  8. Communicate frequently and effectively
  9. Balance local autonomy and central control. Allow bad news to travel ‘up’ in the organization structure. Provide emergency channels through any bureaucracy.
  10. Incentivize discovery. Reward the messenger.
  11. Cultivate an alert culture. Educate about unknown unknowns, where they tend to lurk, and why.

I always ask my project and risk management students a question. Given two very similar projects—one with a list of 100 risks identified, and the other with no risks identified—which project is riskier? Unfortunately, many executives and managers seem to get this backwards in practice. Individuals and cultures that prefer (or are incentivized) to ignore uncertainties and risks fuel the delusion and deception problems (eg., Flyvbjerg, et al. 2009) that plague many projects and other endeavors. Many unknown unknowns remain so because of our own lack of will to find and face them. The approaches outlined above should be part of the due diligence of any complex project’s planning and execution.

To find out more:
Browning, T. R. and Ramasesh, R. V. (2015). Reducing Unwelcome Surprises in Project Management. MIT Sloan Management Review, 56, 3: 53-62. (Online): http://sloanreview.mit.edu/x/56319 (The ideas and some of the wording in this blog post come from this article.)

Ramasesh, R. V. and Browning, T. R. (2014). A Conceptual Framework for Tackling Knowable Unknown Unknowns in Project Management. Journal of Operations Management, 32, 4: 190-204. (Online) (DOI): http://dx.doi.org/10.1016/j.jom.2014.03.003

Flyvbjerg, B., Garbuio, M. and Lovallo, D. (2009). Delusion and Deception in Large Infrastructure Projects: Two Models for Explaining and Preventing Executive Disaster. California Management Review, 51, 2: 170-193

Mullins, J. W. (2007). Discovering ‘Unk-Unks’: How Innovators Identify the Critical Things They Don’t Even Know that They Don’t Know. MIT Sloan Management Review, 48, 4: 17-21

Project Management Institute. (2017). A Guide to the Project Management Body of Knowledge, 6th ed., Newtown Square: PA, United States of America

Biography: Tyson R. Browning PhD is Professor of Operations Management in the Neeley School of Business at Texas Christian University in Fort Worth, Texas, USA. His primary research is on managing complex projects. Previously, he worked for Lockheed Martin, Honeywell Space Systems, and Los Alamos National Laboratory. He is currently co-Editor-in-Chief of the Journal of Operations Management.

This blog post is part of a series on unknown unknowns as part of a collaboration between the Australian National University and Defence Science and Technology.

For the five other blog posts already published in this series, see: https://i2insights.org/tag/partner-defence-science-and-technology/

Scheduled blog posts in this series:
November 19: Blackboxing unknown unknowns through vulnerability analysis by Joseph Guillaume
December 3: Yin-yang thinking – A solution to dealing with unknown unknowns? by Christiane Prange and Alicia Hennig
TBA: Detecting non-linear change ‘inside-the-system’ and ‘out-of-the-blue’ by Susan van ‘t Klooster and Marjolijn Haasnoot
January 28, 2020: How can resilience benefit from planning? by Pedro Ferreira
February 11, 2020: Why do we protect ourselves from unknown unknowns? by Bem Le Hunte

The role of persistence in influencing policy with research

By David McDonald

Author - David McDonald
David McDonald (biography)

Seeking to influence policy with our research is difficult. Sometimes we feel that it is too hard, we are not achieving our goals fast enough, and we really should give up and find easier ways of operating. However, persistence, rather than giving up, seems to be a characteristic of those of us working in this domain!

What do we mean by persistence? A good dictionary definition is ‘continuing firmly, especially despite obstacles and protests’. Does that sound familiar: facing obstacles to doing high-quality implementation work, and protests from colleagues who do not share our perceptions of the value of working in this manner? Continue reading

Creative writing as a journey into the unknown unknown

By Lelia Green

Author - Lelia Green
Lelia Green (biography)

Do you use writing as a means of accessing your unconscious knowledge and understanding? The electric experience of things falling into place is a well-recorded outcome of ‘writing to find out what you want to say.’ E. L. Doctorow is credited with saying that writing a novel is “like driving at night. You never see further than your headlights, but you can make the whole trip that way” (no formal reference identifiable, but see Quotation Celebration). There is a sense of allowing the unfolding journey to deliver you to your destination, and experiencing the energy rush when you arrive. It’s a matter of relinquishing control and being open to the unexpected. Continue reading

10 tips for next generation interdisciplinary research

By Rachel Kelly

Author - Rachel Kelly
Rachel Kelly (biography)

Can we develop a shared understanding on how to engage in an interdisciplinary setting that will be useful in addressing current and future grand challenges?

Advice provided by interdisciplinary experts from 25 countries, across all continents, and with over 240 years cumulative experience (Kelly, et al., 2019) is combined here into succinct guidance that aims to empower researchers wishing to engage in interdisciplinary endeavors. The ten tips are also summarized in the figure below (focused on socio-ecological researchers). Continue reading

How can we know unknown unknowns?

By Michael Smithson

Michael Smithson
Michael Smithson (biography)

In a 1993 paper, philosopher Ann Kerwin elaborated a view on ignorance that has been summarized in a 2×2 table describing crucial components of metacognition (see figure below). One margin of the table consisted of “knowns” and “unknowns”. The other margin comprised the adjectives “known” and “unknown”. Crosstabulating these produced “known knowns”, “known unknowns”, “unknown knowns”, and unknown unknowns”. The latter two categories have caused some befuddlement. What does it mean to not know what is known, or to not know what is unknown? And how can we convert either of these into their known counterparts? Continue reading

Learning from interdisciplinary and transdisciplinary research ‘failures’

By Dena Fam and Michael O’Rourke

Dena Fam
Dena Fam (biography)

What makes interdisciplinary and transdisciplinary research challenging? What can go wrong and lead to failure? What has your experience been?

Modes of research that involve the integration of different perspectives, such as interdisciplinary and transdisciplinary research, are notoriously challenging for a host of reasons. Interdisciplinary research requires the combination of insights from different academic disciplines and it is common that these:

  • bear the stamp of different epistemologies; and,
  • involve different types of data collected using different methods in the service of different explanations.

Continue reading