By Joseph Guillaume

What’s a productive way to think about undesirable outcomes and how to avoid them, especially in an unpredictable future full of unknown unknowns? Here I describe the technique of vulnerability analysis, which essentially has three steps:

• Step 1: Identify undesirable outcomes, to be avoided
• Step 2: Look for conditions that can lead to such outcomes, ie. vulnerabilities
• Step 3: Manage the system to mitigate or adapt to vulnerable conditions.

The power of vulnerability analysis is that, by starting from outcomes, it avoids making assumptions about what led to the vulnerabilities. The causes of the vulnerabilities are effectively a ‘black box’, in other words, they do not need to be understood in order to take effective action. The vulnerability itself is either a known known or a known unknown. The causes of the vulnerability, on the other hand, can be unknown unknowns.

We can of course partially open the black box and try to construct an understanding of those causes – turning them into known unknowns. Mitigating a vulnerability relies on having sufficient knowledge to anticipate and counter it. We can, however, also recognise that the black box remains partially unopened – that the vulnerability might occur for reasons we have not anticipated, but we can still monitor to check whether the vulnerability is occurring and adapt accordingly.

Let’s take an example investment decision. We want to store water from wet times to prepare for drought, and we are considering two options: “surface storage” in a dam and “managed aquifer recharge”, involving storing water underground, as groundwater. We want to make our decision based on outcomes – we want to choose the option that provides the greatest net benefit.

There’s a lot we know about costs and benefits of each option. They both have capital costs – to build the dam and infrastructure to infiltrate or inject water underground. They both have maintenance costs. Benefits come from having water available when needed, and a key advantage of storing water underground is that it reduces evaporation – we expect that this means there will be more water available for dry years, which translates to better socio-economic outcomes.

These costs and benefits are uncertain but vulnerability analysis gives us a way of thinking through them. Suppose we had decided to invest in managed aquifer recharge. It would be undesirable if surface storage then turned out to be better value – there would be a “crossover” in our preferred option. What are then our vulnerabilities?

If we look at each of our costs and benefits in turn, surface storage would have the advantage if its costs were lower than expected and benefits higher, and vice versa for managed aquifer recharge. If the cost of infiltration infrastructure rises, or the price of an irrigated crop falls, managed aquifer recharge may no longer be worthwhile. We can investigate how much of a change would cause this crossover to occur. If we look at both uncertainties at once, even small changes in infrastructure cost and crop price may cause a crossover – managed aquifer recharge is even less viable. These vulnerabilities become scenarios we can discuss within our investment planning process (for a description of scenarios, see Bonnie McBain’s blog post). (Info-gap theory as described in Yakov Ben-Haim’s blog post does something similar to vulnerability analysis.)

At this stage, we have not needed to know why the infrastructure cost would rise or crop price would fall. Both remain unknown unknown black boxes. But we can add information if we have it: we might be able to get a fixed price contract for the infrastructure, and we might be able to use price forecasts to evaluate how worried we should be about that vulnerability. And importantly, we can do this while only partially opening the black box, by identifying the vulnerabilities introduced by our new information. A fixed price contract can be associated with a black box probability that the contractor will not finish the job. Our price forecast can be accompanied by an error or a probability distribution with unknown unknown drivers, used for example to maximise expected utility (for a description of expected utility see the blog post by Siobhan Bourke and Emily Lancsar).

Using vulnerability analysis to work backwards from outcomes provides a powerful way of working with unknown unknowns, gradually identifying known unknowns as we come across them, while making the best use of what we consider known knowns.

What has your experience been with vulnerability analyses? Have you seen them used to blackbox unknown unknowns in practice?

To find out more:

Arshad, M., Guillaume, JHA. and Ross, A. (2014). Assessing the Feasibility of Managed Aquifer Recharge for Irrigation under Uncertainty. Water, 6 (9): 2748–69. (Online) (DOI): http://dx.doi.org/10.3390/w6092748

Guillaume JHA, Arshad M, Jakeman AJ, Jalava M, Kummu M (2016) Robust Discrimination between Uncertain Management Alternatives by Iterative Reflection on Crossover Point Scenarios: Principles, Design and Implementations. Environmental Modelling & Software, 83: 326–43. (Online) (DOI): http://dx.doi.org/10.1016/j.envsoft.2016.04.005

Biography: Joseph Guillaume PhD is a DECRA (Discovery Early Career Researcher Award) Research Fellow in the Fenner School of Environment & Society at the Australian National University in Canberra, Australia. He is an integrated modeller with a particular interest in uncertainty and decision support. Application areas have focussed primarily on water resources. Ongoing work involves providing a synthesis of the many ways we communicate about uncertainty, and their implications for modelling and decision support.

This blog post is part of a series on unknown unknowns as part of a collaboration between the Australian National University and Defence Science and Technology.

For the six other blog posts already published in this series, see: https://i2insights.org/tag/partner-defence-science-and-technology/

Scheduled blog posts in this series:
December 3: Yin-yang thinking – A solution to dealing with unknown unknowns? by Christiane Prange and Alicia Hennig
January 14, 2020: Detecting non-linear change ‘inside-the-system’ and ‘out-of-the-blue’ by Susan van ‘t Klooster and Marjolijn Haasnoot
January 28, 2020: How can resilience benefit from planning? by Pedro Ferreira
February 11, 2020: Why do we protect ourselves from unknown unknowns? by Bem Le Hunte