By Denis Karcher and Chris Cvitanovic
2. Chris Cvitanovic (biography)
How well do researchers achieve the research impacts they aim for? And if there is a mismatch, does it matter?
Together with colleagues (Karcher et al., 2021), we systematically searched for and reviewed nearly 400 studies that described goals and outcomes that were claimed for knowledge exchange at the science-policy interface. Although our focus was on the environmental sciences, the results may be more widely useful.
Big ambitions
The eight top goals that studies described for their knowledge exchange activities were:
1. Usability, eg., that the interaction with policy makers and/or the knowledge created were credible, legitimate, relevant, and timely (458 references).
2. Social outcomes, eg., the creation of new networks, raising awareness of the issue, developing mutual understanding, and building trust (226 references).
3. Use, eg., that the knowledge was considered in the decision-making process leading to ‘evidence-informed’ policy (215 references).
4. Policy or societal impact, eg., longer-term influence on policy, as well as impacts on democracy, economy or the overall well-being of society (186 references).
5. Process, ingredients, eg., the process ensured accountability, facilitation quality, fairness, respect (101 references).
6. Quality and outlook, eg., self-evaluation and implications for the future (creating new projects, etc.) (43 references).
7. Ecological impact, eg., maintained or improved state of the environment (43 references).
8. Products, eg., reports, tools, or models jointly developed through the process (33 references).
Diverse achievements
However, the outcomes that the same studies reported achieving were rather different in order and quantity, as shown in the figure below.
The order of the top eight claimed outcomes was:
1. Social outcomes (220 references).
2. Usability (164 references).
3. Products (92 references).
4. Use (73 references).
5. Process, ingredients (60 references).
6. Policy or societal impact (56 references).
7. Quality and outlook (41 references).
8. Ecological impact (9 references).
Divergence between goals and achievements
We found that usability, social outcomes and the actual use of knowledge are the main goals of knowledge exchange, followed by the last of these leading to policy or societal impact. However, claimed achievements were, apart from usability, more frequently related to social outcomes, process components and the creation of products. Reasons for that difference between ambitions and achievements might be that:
- having big ambitions works to generate funding (aiming high demonstrates broader relevance and shows big potential of a project);
- knowledge use and impact are more difficult to achieve than outcomes such as joint products or a fair and inclusive process (eg., because policy actors have more than just one project to consider when taking decisions); and/or,
- knowledge use and policy or societal impact require long time spans to become evident and this does not usually occur by the time academic papers are published.
So what?
Our findings indicate the need to appreciate the diverse impacts of knowledge exchange activities, be they social outcomes, good products or process components. Therefore, we encourage researchers to include those factors directly in their planning to better align goals and evidence. This would give clearer expectations to both researchers and policy makers (and all other societal actors involved in knowledge exchange, eg., stakeholders, non-governmental organisations (NGOs), local/Indigenous communities) before the start of an interaction. It would also have practical, academic and personal benefits, as follows:
- So what on a practical scale: if researchers do not appreciate the diversity of achievable impacts, efforts to plan impact will be too narrowly focused and will not demonstrate the full range of potential benefits. For example, aiming for good processes in the first place should be enough to justify projects as it can lay the foundations for deeper and more enduring collaborations which in time achieve other goals.
- So what on an academic scale: broader sets of metrics need to be established to monitor and measure the full spectrum of impacts over time. As a community, researchers need to develop, apply, test and refine methods (mostly qualitative) and indicators for measuring the diversity of impacts. Progress may require (and also affect) academic rewards for good practice related to the knowledge exchange process and engagement of policy makers and other non-academic actors.
- So what on a personal level: Academics are under pressure to demonstrate impact when they apply for promotion and tenure. Therefore, universities and other organisations need to appreciate the diversity of potential impacts so that efforts to achieve successful knowledge exchange can be made visible, supported, and rewarded.
Do our findings of a mismatch between goals and reported outcomes in knowledge exchange resonate with your experience? Do you have anything to add to the practical, academic and personal benefits of overcoming the mismatch? Do you think the mismatch can be overcome without institutional reform in how knowledge exchange work is incentivised, planned, and funded?
To find out more:
Karcher, D. B., Cvitanovic, C., Colvin, R. M., van Putten, I. E. and Reed, M. S. (2021). Is this what success looks like? Mismatches between the aims, claims, and evidence used to demonstrate impact from knowledge exchange processes at the interface of environmental science and policy. Environmental Science and Policy, 125: 202-218.
Biography: Denis Karcher is a PhD candidate in Science Communication at the Centre for the Public Awareness of Science at the Australian National University in Canberra. His focus is on the marine science-policy interface. More specifically, he investigates how knowledge exchange between actors from research, policy and society can be improved to ensure evidence-informed coastal and ocean management.
Biography: Chris Cvitanovic PhD is a transdisciplinary marine scientist in the Centre for the Public Awareness of Science at the Australian National University in Canberra. His research is focused on improving the relationship between science, policy and practice to enable evidence-informed decision-making for sustainable ocean futures.
I haven’t seen an investigation of impact goals and outcomes before, thank you for this contribution. I agree especially with the third bullet about time required for longer term outcomes. In my impact practice I encourage researchers to differentiate between outcomes (ie short term outcomes) and impacts (ie longer term impacts) as the former (ie usability, products and use) precede downstream impacts. One question though about your assertion that impact is increasingly being required for tenure and promotion. We don’t see this yet in Canada. Where are you seeing it and are there standards of excellence by which these impact related tenure and promotion decisions are being made?
Thanks for your comment and good to hear that some of the findings resonate with you! I totally agree with the realistic planning and communication of short-term and long-term goals/impacts. I found this work by Cooke and colleagues very helpful https://doi.org/10.1139/er-2020-0045.
Regarding impacts in relation to tenure and promotion, of course every country and even every university is different. We know of some in Australia where it is included in promotion packages (“Public policy activities and evidence of impact”). But even for promotion it’s largely anecdotal without standards (or commonly accepted metrics) for excellence. Some studies have shown the importance of showing impact for promotion. For example, https://doi.org/10.1016/j.marpol.2014.10.026 showing that it is important to career progression, but less so than traditional academic metrics. Big funding applications (like those to the Australian Research Council) look at impact (e.g., research impact, policy advice) as part of the Research Opportunity and Performance Evidence Statement. Also, works by Mark Reed might be helpful here, talking of university impact cultures https://doi.org/10.3389/frsus.2021.662296, or research impact planning https://doi.org/10.1332/174426418X15326967547242.
Hi Denis/Chris,
Thanks very much for highlighting this important work on the gaps between goals and outcomes for knowledge exchange activities.
One of the (many) aims of transdisciplinary or co-produced research is to narrow the gap between research goals and user needs by involving the user (community, policymaker, local authority or company) in the entire research process from design to implementation and beyond. Was there any evidence in your review of studies that co-produced research led to less mismatches between knowledge transfer goals and outcomes?
Best wishes,
Paul
Hi Paul,
Thanks for your comment and thoughtful question. The short answer is no. That is largely because our study did not intend to investigate that much detail. Hence, the connection between a strategy (e.g. co-production) and the goals/outcomes is not causal.
When looking at co-production as “co-occurring” (≠ causing) with certain outcomes (or mismatches), it is apparent that co-production brought slightly more diverse outcomes than other approaches, but not less mismatches. And all that wouldn’t be statistically significant.
In sum, a comprehensive study that compares different knowledge exchange strategies, their effectiveness, and mismatches would be a great way to progress research.
Best, Denis
Hi Denis,
Thanks very much for your reply.
I would fully agree that a study which looks at the impact of knowledge exchange strategies, including co-production, in terms of research uptake and utilisation would be very valuable. A much stronger evidence base to illustrate the value of such strategies under different contexts is needed.
Best wishes,
Paul
Thanks for sharing these ‘pithy’ insights about the research impact process. I am part of a team launching a Community Science program in the Shoalhaven area of Australia and your article will be an excellent conversation starter for this group as it attempts to better match aspirations and outcomes.
Thanks a lot for your comment and positive feedback!