Blackboxing unknown unknowns through vulnerability analysis

By Joseph Guillaume

Author - Joseph Guillaume
Joseph Guillaume (biography)

What’s a productive way to think about undesirable outcomes and how to avoid them, especially in an unpredictable future full of unknown unknowns? Here I describe the technique of vulnerability analysis, which essentially has three steps:

  • Step 1: Identify undesirable outcomes, to be avoided
  • Step 2: Look for conditions that can lead to such outcomes, ie. vulnerabilities
  • Step 3: Manage the system to mitigate or adapt to vulnerable conditions.

The power of vulnerability analysis is that, by starting from outcomes, it avoids making assumptions about what led to the vulnerabilities. The causes of the vulnerabilities are effectively a ‘black box’, in other words, they do not need to be understood in order to take effective action. The vulnerability itself is either a known known or a known unknown. The causes of the vulnerability, on the other hand, can be unknown unknowns.

We can of course partially open the black box and try to construct an understanding of those causes – turning them into known unknowns. Mitigating a vulnerability relies on having sufficient knowledge to anticipate and counter it. We can, however, also recognise that the black box remains partially unopened – that the vulnerability might occur for reasons we have not anticipated, but we can still monitor to check whether the vulnerability is occurring and adapt accordingly.

Let’s take an example investment decision. We want to store water from wet times to prepare for drought, and we are considering two options: “surface storage” in a dam and “managed aquifer recharge”, involving storing water underground, as groundwater. We want to make our decision based on outcomes – we want to choose the option that provides the greatest net benefit.

There’s a lot we know about costs and benefits of each option. They both have capital costs – to build the dam and infrastructure to infiltrate or inject water underground. They both have maintenance costs. Benefits come from having water available when needed, and a key advantage of storing water underground is that it reduces evaporation – we expect that this means there will be more water available for dry years, which translates to better socio-economic outcomes.

These costs and benefits are uncertain but vulnerability analysis gives us a way of thinking through them. Suppose we had decided to invest in managed aquifer recharge. It would be undesirable if surface storage then turned out to be better value – there would be a “crossover” in our preferred option. What are then our vulnerabilities?

If we look at each of our costs and benefits in turn, surface storage would have the advantage if its costs were lower than expected and benefits higher, and vice versa for managed aquifer recharge. If the cost of infiltration infrastructure rises, or the price of an irrigated crop falls, managed aquifer recharge may no longer be worthwhile. We can investigate how much of a change would cause this crossover to occur. If we look at both uncertainties at once, even small changes in infrastructure cost and crop price may cause a crossover – managed aquifer recharge is even less viable. These vulnerabilities become scenarios we can discuss within our investment planning process (for a description of scenarios, see Bonnie McBain’s blog post). (Info-gap theory as described in Yakov Ben-Haim’s blog post does something similar to vulnerability analysis.)

At this stage, we have not needed to know why the infrastructure cost would rise or crop price would fall. Both remain unknown unknown black boxes. But we can add information if we have it: we might be able to get a fixed price contract for the infrastructure, and we might be able to use price forecasts to evaluate how worried we should be about that vulnerability. And importantly, we can do this while only partially opening the black box, by identifying the vulnerabilities introduced by our new information. A fixed price contract can be associated with a black box probability that the contractor will not finish the job. Our price forecast can be accompanied by an error or a probability distribution with unknown unknown drivers, used for example to maximise expected utility (for a description of expected utility see the blog post by Siobhan Bourke and Emily Lancsar).

Using vulnerability analysis to work backwards from outcomes provides a powerful way of working with unknown unknowns, gradually identifying known unknowns as we come across them, while making the best use of what we consider known knowns.

What has your experience been with vulnerability analyses? Have you seen them used to blackbox unknown unknowns in practice?

To find out more:

Arshad, M., Guillaume, JHA. and Ross, A. (2014). Assessing the Feasibility of Managed Aquifer Recharge for Irrigation under Uncertainty. Water, 6 (9): 2748–69. (Online) (DOI):

Guillaume JHA, Arshad M, Jakeman AJ, Jalava M, Kummu M (2016) Robust Discrimination between Uncertain Management Alternatives by Iterative Reflection on Crossover Point Scenarios: Principles, Design and Implementations. Environmental Modelling & Software, 83: 326–43. (Online) (DOI):

Biography: Joseph Guillaume PhD is a DECRA (Discovery Early Career Researcher Award) Research Fellow in the Fenner School of Environment & Society at the Australian National University in Canberra, Australia. He is an integrated modeller with a particular interest in uncertainty and decision support. Application areas have focussed primarily on water resources. Ongoing work involves providing a synthesis of the many ways we communicate about uncertainty, and their implications for modelling and decision support.

This blog post is part of a series on unknown unknowns as part of a collaboration between the Australian National University and Defence Science and Technology.

For the six other blog posts already published in this series, see:

Scheduled blog posts in this series:
December 3: Yin-yang thinking – A solution to dealing with unknown unknowns? by Christiane Prange and Alicia Hennig
January 14, 2020: Detecting non-linear change ‘inside-the-system’ and ‘out-of-the-blue’ by Susan van ‘t Klooster and Marjolijn Haasnoot
January 28, 2020: How can resilience benefit from planning? by Pedro Ferreira
February 11, 2020: Why do we protect ourselves from unknown unknowns? by Bem Le Hunte

What motivates researchers to become transdisciplinary and what are the implications for career development?

By Maria Helena Guimarães, Olivia Bina and Christian Pohl

Author - Maria Helena Guimarães
Maria Helena Guimarães (biography)

If disciplines shape scientific research by forming the primary institutional and cognitive units in academia, how do researchers start being interested in and working with a transdisciplinary approach? How does this influence their career development?

Author - Olivia Bina
Olivia Bina (biography)

We interviewed 12 researchers working in Switzerland who are part of academia and identify as ‘transdisciplinarians’.

Author - Christian Pohl
Christian Pohl (biography)

They described seven types of motivations:

  1. Individual ethics, especially a desire to improve society and contribute to the advancement of the common good.
  2. Concern about real-world problems, particularly a desire to engage with societal issues that do not primarily emerge from disciplinary journals or academic discourse alone.
  3. Search for fulfillment, especially the possibility of making a difference in their own lives and those of others.
  4. Wanting to bring together theoretical and practical perspectives, as well as communities undertaking complementary but independent work.
  5. Realising that individual disciplines do not provide sufficient insights to deal with complex problems and wanting to go beyond them.
  6. Wanting to step “out of the box” and being attracted to transdisciplinarity as a transgressive and risk-taking activity.
  7. Desire to be reflective, connected to a range of research interests and to connect across a range of fields.

The following quotations from the interviewees illustrate these motivations.

quote number one - guimaraes post - motivations-TD-researchers

Career development

Most interviewees defined their trajectory in transdisciplinarity (ie., their entry into transdisciplinarity and their continuing development within it) as something that just happened and was linked to a specific way of perceiving science.

Most did not present a well-established career path in transdisciplinarity. Further they considered that the CV (curriculum vitae) profiles that they had developed so far meant that well-established career paths were not attainable.

Those who were professors described a dual role, one focused on disciplinary excellence and the other on transdisciplinarity. A few of the young researchers aimed at developing a career in academia and were therefore intermingling their transdisciplinary activities with disciplinary knowledge production.

Generally, professors and supervisors did not advise young researchers to move into transdisciplinarity because of its impact on career advancement or of the lack of supervision for inter- and trans- disciplinary research compared with disciplinary projects. Most of the challenges described were related to the lack of recognition for transdisciplinarity within the academic system. Therefore, independent of the individual’s age or years spent working in academia, the general perception was a lack of opportunities to progress in academia.

Within this discussion, some interviewees suggested creating a discipline, a field, or a community focused on the formalization of transdisciplinarity, as well as on its development. Others viewed this track as a threat to the inherent openness of transdisciplinarity to other disciplines.

The following quotations capture the above points.

Do these experiences match your own? What other motivations for entering transdisciplinarity are you aware of? What impacts on career development have you seen?

To find out more:
Guimarães M. H., Pohl C., Bina O. and Varanda M. (2019). Who is doing inter- and transdisciplinary research, and why? An empirical study of motivations, attitudes, skills, and behaviours. Futures, 112, 102441. Online open-access (DOI):

Biography: Maria Helena Guimarães PhD is a postdoctoral researcher at the Institute of Mediterranean Agricultural and Environmental Sciences (ICAAM) in Évora University, Portugal. In her research group, she coordinates the line of research dedicated to transdisciplinary processes and co-construction of knowledge. Her research interest is the practical application of knowledge co-construction and the use of systems thinking for the sustainable management of natural resources..

Biography: Olivia Bina PhD is Principal Researcher at the Institute of Social Sciences, University of Lisbon in Portugal and Fellow of the World Academy of Art and Science. She coordinates the Urban Transitions Hub at her Institute. Her research focuses on change and sustainable futures, on the critique of “green” growth and the limits to growth, on connectedness between humans and nature, and notions of scarcity. .

Biography: Christian Pohl PhD is co-director of the Transdisciplinarity Lab of the Department of Environmental Systems Science (USYS TdLab) at ETH Zurich in Switzerland. He completed his habilitation at the University of Bern. His research interest is the theory and practice of transdisciplinary research as a means for sustainable development.

Looking in the right places to identify “unknown unknowns” in projects

Author - Tyson R. Browning
Tyson R. Browning (biography)

By Tyson R. Browning

Unknown unknowns pose a tremendous challenge as they are essentially to blame for many of the unwelcome surprises that pop up to derail projects. However, many, perhaps even most, of these so-called unknown unknowns were actually knowable in advance, if project managers had merely looked in the right places.

For example, investigations following major catastrophes (such as space shuttle disasters, train derailments, and terrorist attacks), and project cost and schedule overruns, commonly identify instances where a key bit of knowledge was in fact known by someone working on that project—but failed to be communicated to the project’s top decision makers. In other cases, unknown unknowns emerge from unforeseen interactions among known elements of complex systems, such as product components, process activities, or software systems. Continue reading

The role of persistence in influencing policy with research

By David McDonald

Author - David McDonald
David McDonald (biography)

Seeking to influence policy with our research is difficult. Sometimes we feel that it is too hard, we are not achieving our goals fast enough, and we really should give up and find easier ways of operating. However, persistence, rather than giving up, seems to be a characteristic of those of us working in this domain!

What do we mean by persistence? A good dictionary definition is ‘continuing firmly, especially despite obstacles and protests’. Does that sound familiar: facing obstacles to doing high-quality implementation work, and protests from colleagues who do not share our perceptions of the value of working in this manner? Continue reading

Creative writing as a journey into the unknown unknown

By Lelia Green

Author - Lelia Green
Lelia Green (biography)

Do you use writing as a means of accessing your unconscious knowledge and understanding? The electric experience of things falling into place is a well-recorded outcome of ‘writing to find out what you want to say.’ E. L. Doctorow is credited with saying that writing a novel is “like driving at night. You never see further than your headlights, but you can make the whole trip that way” (no formal reference identifiable, but see Quotation Celebration). There is a sense of allowing the unfolding journey to deliver you to your destination, and experiencing the energy rush when you arrive. It’s a matter of relinquishing control and being open to the unexpected. Continue reading

Participatory research and power

By Diana Rose

Diana Rose
Diana Rose (biography)

Can even the most well-designed participatory research really level the power relations between researchers and the relevant community? The key issues are who sets the research agenda, who drives the research process and governs it, and who interprets information. In all these aspects of research, the aim is for the community to no longer be ‘subjects’ but equal partners.

In this blog post, I outline challenges to achieving this mission, so that we can be realistic about what’s involved in trying to achieve equal partnerships. The difficulties identified are not proposed as tensions to be ‘solved’ but as dilemmas that can be articulated so as better to facilitate good practice, not reach an unattainable perfect state. Continue reading