By Tuomas J. Lahtinen, Joseph H. A. Guillaume, Raimo P. Hämäläinen
How can we identify and evaluate decision forks in a modelling project; those points where a different decision might lead to a better model?
Although modellers often follow so called best practices, it is not uncommon that a project goes astray. Sometimes we become so embedded in the work that we do not take time to stop and think through options when decision points are reached.
One way of clarifying thinking about this phenomenon is to think of the path followed. The path is the sequence of steps actually taken in developing a model or in a problem solving case. A modelling process can typically be carried out in different ways, which generate different paths that can lead to different outcomes. That is, there can be path dependence in modelling.
Recently, we have come to understand the importance of human behaviour in modelling and the fact that modellers are subject to biases. Behavioural phenomena naturally affect the problem solving path. For example, the problem solving team can become anchored to one approach and only look for refinements in the model that was initially chosen. Due to confirmation bias, modelers may selectively gather and use evidence in a way that supports their initial beliefs and assumptions. The availability heuristic is at play when modellers focus on phenomena that are easily imaginable or recalled. Moreover particularly in high interest cases strategic behaviour of the project team members can impact the path of the process.
Taking a path perspective means engaging in reflection on the path taken, and awareness that the modelling path can matter. Even if a perfect path does not exist or cannot be found, a poor path or possibilities to improve a planned path can often be identified.
The problem solvers’ choices at decisions forks define the path. Paths are influenced by phenomena that:
- influence the choices made at each fork,
- give reasons to redirect the path from the route that was previously chosen,
- make it difficult to change the path taken.
The phenomena can be classified according to their origin. The path might be taken:
- through deliberate thinking (or lack thereof)
- as a result of the processes, methods, and approaches used
- as a reflection of preferences and motives (possibly hidden)
- through intuitive reasoning, based on tacit knowledge or biases
- as a result of the system of problem solving and system under study, and emergent phenomena arising from them.
The framework described in the table below covers a broad range of phenomena to be aware of when reflecting on a path. For more detail, including links to existing techniques to address the phenomena, see Lahtinen et al. (2017).
The path perspective challenges modellers to navigate their paths in a reflective mode. Critical and possibly hidden forks can easily exist in complex environmental problems.
Having explicit criteria for the success of the modelling project can help to keep track of the path. Resources should also be reserved to enable possible redirecting or restarting of the project. In important problems one could consider having two independent parallel problem solving processes.
Do you have any stories to share, where any of the phenomena described had an influence on the choices made, gave reason to redirect the path, or made it difficult to change the path taken?
Reference:
Lahtinen, T. J., Guillaume, J. H. A. and Hämäläinen, R. P. (2017). Why pay attention to paths in the practice of environmental modelling? Environmental Modelling and Software, 92: 74–81. Online (DOI): 10.1016/j.envsoft.2017.02.019
Biography: Tuomas Lahtinen is a doctoral student in the Systems Analysis Laboratory, Aalto University, Finland. He works on various topics related to Behavioural Operations Research, including path dependence, decision analysis, environmental portfolio decision making, behavioural experiments, and the practice of modelling. He is a board member in the Finnish Operations Research Society.
Biography: Joseph Guillaume is a Postdoctoral Researcher with the Water and Development Research Group at Aalto University, Finland. He is a transdisciplinary modeller with a particular interest in uncertainty and decision support. Application areas have focussed primarily on water resources, including rainfall-runoff modelling, hydro-economic modelling, ecosystem health, global water scarcity and global food security. Ongoing work involves providing a synthesis of the many ways we communicate about uncertainty, and their implications for modelling and decision support. He is member of the Core Modeling Practices pursuit funded by the National Socio-Environmental Synthesis Center (SESYNC).
Biography: Raimo P. Hämäläinen is a professor emeritus in the Systems Analysis Laboratory, Aalto University, Finland. He has published extensively on decision and game theory, environmental decision making, as well as developed widely used decision support software. His recent interests include behavioural issues in modelling and systems intelligence in social interaction. He is the recipient of the Edgeworth Pareto Award of the International Society on Multiple Criteria Decision Making. He is the chair of the working group on Behavioural Operations Research of the Association of the European Operational Research Societies.
I think your analysis is of the model development process vs the content of the model itself, right? A knowledge of human behaviours and decision making obviously impact both. One of the key variables missing from your analysis is the experience level of the modeller which influences all aspects of their decision making, but particularly their use of heuristics, which have evolved to be adaptive/effective. Be careful not to cast heuristics in a negative light… in the main, they support effective decision making under uncertainty and complexity which no amount of deliberate analysis will solve. See, for example, Gary Klein’s work on recognition-primed decision making (RPD; e.g. ‘Sources of Power’ 1999).
Thanks for your stimulating comments.
While the focus is on assessing the model development process, that necessarily includes decisions about the content of the model, so we don’t consider the two separable.
Your note on the experience level of the modeler opens an interesting theme. There are lots of studies on best practices but we do not know of research literature on the impact of experience. Experience is most often likely to have a positive impact. However, it is also easy to envision problems related to experience. An experienced modeler may have become stuck with his or her own approach, which creates a risk of the hammer-and-nail syndrome. An experienced modeler is seen as an authority. In a modelling team this can hinder open reflection and result in groupthink.
The role of heuristics is also an interesting question which has been studied surprisingly little in the modelling literature. The theme is broad. Heuristics can relate to, e.g. modelling, procedure, or collection of data. The term can refer to practical rules of thumb, mental shortcuts, or even unconscious cognitive processes. In the paper we try to avoid presenting heuristics in a negative light. We write “In complex problems one may need to adhere to heuristics, i.e. mental shortcuts or practical rules of thumb. The preferred heuristics are likely to vary across modelers. Heuristics can be appropriate and useful when applied in the right context (see, e.g. Gigerenzer and Todd, 1999; Keller and Katsikopoulos, 2016).”
The interaction of heuristics and the level of experience raises interesting research questions as well. The combination of strong experience and use of heuristics could possibly be terrific or disastrous.
Generally, the role of human behavior, including the use of heuristics, in modelling processes is a very interesting theme, which has recently received a strong interest. If you want to learn more, visit http://bor.aalto.fi/. It is a collaboration platform for researchers interested in Behavioural Operational Research. It is also the home page of The EURO working group on Behavioural OR.
Best Regards,
Tuomas, Joseph and Raimo
References and further reading:
Gigerenzer, G., and Todd P.M. (1999). Simple heuristics that make us smart. Oxford University Press.
Hämäläinen, R.P., Luoma, J., and Saarinen, E. (2013). On the Importance of Behavioral Operational Research: The Case of Understanding and Communicating about Dynamic Systems. European Journal of Operational Research, 228(3): 623-634. http://dx.doi.org/10.1016/j.ejor.2013.02.001
Keller, N., and Katsikopoulos, K.V. (2016). On the role of psychological heuristics in operational research; and a demonstration in military stability operations. European Journal of Operational Research 249(3): 1063-1073. http://dx.doi.org/10.1016/j.ejor.2015.07.023