Considerations for choosing frameworks to assess research impact

By Elena Louder, Carina Wyborn, Christopher Cvitanovic and Angela T. Bednarek

What should you take into account in selecting among the many frameworks for evaluating research impact?

In our recent paper (Louder et al., 2021) we examined the epistemological foundations and assumptions of several frameworks and drew out their similarities and differences to help improve the evaluation of research impact. In doing so we identified four key principles or ‘rules of thumb’ to help guide the selection of an evaluation framework for application within a specific context.

1. Be clear about underlying assumptions of knowledge production and definitions of impact

Clarifying from the start how research activities are intended to achieve impact is an important pre-cursor to designing an evaluation. Furthermore, defining what you mean by impact is an important first step in selecting indicators to know if you’ve achieved it.

For example, a research organization should be clear up front whether changes in attitude, problem framing, and/or relationships count as impact. This must involve outlining why certain activities are expected to contribute to impact, and what those impacts might look like. If, for instance, it is assumed that interactions between stakeholders lead to improved relationships, indicators can usefully be developed to evaluate the nature, frequency, quality, etc., of interactions.

This epistemological clarity helps define what counts as impact, and what counts as robust evidence of that impact.

2. Attempt to measure intermediate and process-related impacts

Whether this means expanding the definition of impact, or evaluating quality, or ‘contribution to impact,’ select indicators that capture nuanced changes in problem framing, understanding, or mindsets.

Evaluations should at least partially attempt to capture the ‘below the tip of the iceberg’ knowledge co-production activities. This could be done by focusing at least part of an evaluation on measuring perspectives of participants (via interview or survey) regarding changes such as increased capacity, changes in expertise and knowledge, and shifts in how a problem is understood or framed.

Attention to such intermediate impacts is important as they may serve as building blocks for end-of-process outcomes, and also enable the evaluation of ‘progress makers’ along a theory of change to identify if a project is tracking towards intended outcomes.

3. Balance emergent and expected outcomes

While it is important to be clear on expectations and aspirations, evaluations should have at least some open-ended component which captures unexpected outcomes, both positive and negative. This could be implemented through crafting at least part of an evaluation in an open-ended manner.

For example, rather than rubrics with pre-determined criteria, ask instead – what changed? who changed? how do you know? Such an open-ended approach allows unexpected outcomes to surface.

4. Balance indicators that capture nuance and those that simplify

Evaluations which assign numerical scores to impact may be extremely useful for project managers and large research organizations. However, aggregated scores can sometimes overshadow conceptual changes in the way a problem is framed, or subtle changes resulting from knowledge co-production. Over-emphasis on simple evaluations can also lead to ‘gaming the indicators,’ and provide perverse incentives to tailor research to meet the indicators.

While indicators that can be quantitatively scored (for a hypothetical example, assigning 1-10 scores on dimensions like suitable context, legitimacy and relevance, project outputs) may be easy to use, especially for comparing different research projects, such an approach might not register why or how changes occurred.

The same is true for the number of indicators – fewer indicators may make evaluation simpler and more convenient, where more indicators may deliver more detailed information. This tension must be considered when designing an evaluation.

Closing questions

While these four considerations were derived from a review of frameworks used in the environmental sciences, how well can they be applied in other domains and disciplines? Within other domains and disciplines, are there additional considerations that must be accounted for? Are there other key considerations that you would add based on your experiences of impact evaluation?

To find out more see:
Louder, E., Wyborn, C., Cvitanovic, C. and Bednarek, A. T. (2021). A synthesis of the frameworks available to guide evaluations of research impact at the interface of environmental science, policy and practice. Environmental Science and Policy, 116: 258-265. (Online – open access) (DOI): https://doi.org/10.1016/j.envsci.2020.12.006

Biography: Elena Louder is a PhD student in the department of Geography, Development and Environment at the University of Arizona, Tucson, USA. Her research interests include political ecology, the politics of renewable energy development, knowledge co-production, and biodiversity conservation.

Biography: Carina Wyborn PhD is an interdisciplinary social scientist with a background in science and technology studies, and human ecology. She is based at the Institute for Water Futures at The Australian National University in Canberra, where she researches the science, policy, and politics of environmental futures and capacities that enable future-oriented decision making, in the context of uncertainty.

Biography: Chris Cvitanovic PhD is a transdisciplinary marine scientist at The Australian National University in Canberra. His research is focused on improving the relationship between science, policy and practice to enable evidence-informed decision-making for sustainable ocean futures.

Biography: Angela Bednarek PhD directs the Evidence Project at The Pew Charitable Trusts in Washington DC in the USA. The Evidence Project is a cross-cutting initiative aimed at increasing the use of evidence in policy and practice by marshalling funders, practitioners, scholars, and others to demonstrate effective practice and spur systemic changes in research and evidence use infrastructure.