By Denis Karcher and Chris Cvitanovic
2. Chris Cvitanovic (biography)
How well do researchers achieve the research impacts they aim for? And if there is a mismatch, does it matter?
Together with colleagues (Karcher et al., 2021), we systematically searched for and reviewed nearly 400 studies that described goals and outcomes that were claimed for knowledge exchange at the science-policy interface. Although our focus was on the environmental sciences, the results may be more widely useful.
Big ambitions
The eight top goals that studies described for their knowledge exchange activities were:
1. Usability, eg., that the interaction with policy makers and/or the knowledge created were credible, legitimate, relevant, and timely (458 references).
2. Social outcomes, eg., the creation of new networks, raising awareness of the issue, developing mutual understanding, and building trust (226 references).
3. Use, eg., that the knowledge was considered in the decision-making process leading to ‘evidence-informed’ policy (215 references).
4. Policy or societal impact, eg., longer-term influence on policy, as well as impacts on democracy, economy or the overall well-being of society (186 references).
5. Process, ingredients, eg., the process ensured accountability, facilitation quality, fairness, respect (101 references).
6. Quality and outlook, eg., self-evaluation and implications for the future (creating new projects, etc.) (43 references).
7. Ecological impact, eg., maintained or improved state of the environment (43 references).
8. Products, eg., reports, tools, or models jointly developed through the process (33 references).
Diverse achievements
However, the outcomes that the same studies reported achieving were rather different in order and quantity, as shown in the figure below.
The order of the top eight claimed outcomes was:
1. Social outcomes (220 references).
2. Usability (164 references).
3. Products (92 references).
4. Use (73 references).
5. Process, ingredients (60 references).
6. Policy or societal impact (56 references).
7. Quality and outlook (41 references).
8. Ecological impact (9 references).
Divergence between goals and achievements
We found that usability, social outcomes and the actual use of knowledge are the main goals of knowledge exchange, followed by the last of these leading to policy or societal impact. However, claimed achievements were, apart from usability, more frequently related to social outcomes, process components and the creation of products. Reasons for that difference between ambitions and achievements might be that:
- having big ambitions works to generate funding (aiming high demonstrates broader relevance and shows big potential of a project);
- knowledge use and impact are more difficult to achieve than outcomes such as joint products or a fair and inclusive process (eg., because policy actors have more than just one project to consider when taking decisions); and/or,
- knowledge use and policy or societal impact require long time spans to become evident and this does not usually occur by the time academic papers are published.
So what?
Our findings indicate the need to appreciate the diverse impacts of knowledge exchange activities, be they social outcomes, good products or process components. Therefore, we encourage researchers to include those factors directly in their planning to better align goals and evidence. This would give clearer expectations to both researchers and policy makers (and all other societal actors involved in knowledge exchange, eg., stakeholders, non-governmental organisations (NGOs), local/Indigenous communities) before the start of an interaction. It would also have practical, academic and personal benefits, as follows:
- So what on a practical scale: if researchers do not appreciate the diversity of achievable impacts, efforts to plan impact will be too narrowly focused and will not demonstrate the full range of potential benefits. For example, aiming for good processes in the first place should be enough to justify projects as it can lay the foundations for deeper and more enduring collaborations which in time achieve other goals.
- So what on an academic scale: broader sets of metrics need to be established to monitor and measure the full spectrum of impacts over time. As a community, researchers need to develop, apply, test and refine methods (mostly qualitative) and indicators for measuring the diversity of impacts. Progress may require (and also affect) academic rewards for good practice related to the knowledge exchange process and engagement of policy makers and other non-academic actors.
- So what on a personal level: Academics are under pressure to demonstrate impact when they apply for promotion and tenure. Therefore, universities and other organisations need to appreciate the diversity of potential impacts so that efforts to achieve successful knowledge exchange can be made visible, supported, and rewarded.
Do our findings of a mismatch between goals and reported outcomes in knowledge exchange resonate with your experience? Do you have anything to add to the practical, academic and personal benefits of overcoming the mismatch? Do you think the mismatch can be overcome without institutional reform in how knowledge exchange work is incentivised, planned, and funded?
To find out more:
Karcher, D. B., Cvitanovic, C., Colvin, R. M., van Putten, I. E. and Reed, M. S. (2021). Is this what success looks like? Mismatches between the aims, claims, and evidence used to demonstrate impact from knowledge exchange processes at the interface of environmental science and policy. Environmental Science and Policy, 125: 202-218.
Biography: Denis Karcher is a PhD candidate in Science Communication at the Centre for the Public Awareness of Science at the Australian National University in Canberra. His focus is on the marine science-policy interface. More specifically, he investigates how knowledge exchange between actors from research, policy and society can be improved to ensure evidence-informed coastal and ocean management.
Biography: Chris Cvitanovic PhD is a transdisciplinary marine scientist in the Centre for the Public Awareness of Science at the Australian National University in Canberra. His research is focused on improving the relationship between science, policy and practice to enable evidence-informed decision-making for sustainable ocean futures.