Research impact – Integration and Implementation Insights

How to systematically design transdisciplinary project evaluation

February 9, 2022February 8, 2022

By Emilia Nagy and Martina Schäfer

authors_emilia-nagy_martina-schafer — 1. Emilia Nagy (biography)
2. Martina Schäfer (biography; photo source: Landtag Brandenburg)

How can the formative, ie. process, evaluation of transdisciplinary research projects best incorporate the likely link between process and outcomes in such research? What are some useful approaches for developing an effective evaluation plan with a lens of impact orientation?

We describe how to systematically formulate criteria and indicators for the evaluation of transdisciplinary projects by combining:

impactful research practices (Lux et al., 2019)
impact heuristics (Schäfer et al., 2021)
theory-of-change method (Belcher et al., 2019).

The combination of these approaches provides a strong foundation for impact orientation in all project phases.

Overcoming the mismatch between goals and outcomes in knowledge exchange

January 11, 2022

By Denis Karcher and Chris Cvitanovic

authors_denis-karcher_christopher-cvitanovic — 1. Denis Karcher (biography)
2. Chris Cvitanovic (biography)

How well do researchers achieve the research impacts they aim for? And if there is a mismatch, does it matter?

Together with colleagues (Karcher et al., 2021), we systematically searched for and reviewed nearly 400 studies that described goals and outcomes that were claimed for knowledge exchange at the science-policy interface. Although our focus was on the environmental sciences, the results may be more widely useful.

Big ambitions

The eight top goals that studies described for their knowledge exchange activities were:

1. Usability, eg., that the interaction with policy makers and/or the knowledge created were credible, legitimate, relevant, and timely (458 references).

Five insights on achieving research impact

July 8, 2021June 1, 2021

authors_niki-ellis_anne-maree-dowd_tamika-heiden_gabriele-bammer — 1. Niki Ellis (biography)
2, Anne-Maree Dowd (biography)
3. Tamika Heiden (biography)
4, Gabriele Bammer (biography)

By Niki Ellis, Anne-Maree Dowd, Tamika Heiden and Gabriele Bammer

What does it take for research to be impactful? How should research impact be assessed? How much responsibility for impact should rest with researchers and how much with government, business and/or community partners?

We present five key insights based on our experience in achieving research impact in Australia:

Planning for impact is essential
Quality relationships trump all other factors
Assessment of research contributions should be tailored to the type of research and based on team, not individual, performance
Researchers alone cannot be responsible for achieving impact
Be open to continual learning.

Investing in change through research funding

April 27, 2021

By Petra Lundgren

author_petra-lundgren — Petra Lundgren (biography)

How do funders think about investing in research that is intended to lead to change?

This blog post is written from the perspective of a research funder. More specifically it is based on reflections and lessons learned during five years managing and directing strategic research programs at a not-for-profit foundation, investing in science that would benefit the health and resilience of Australia’s Great Barrier Reef.

Our funding mandate was to include research in a larger body of work towards a broader vision of change. This therefore provided the basis of my work and helped me shape the view that the funder has a big and critical role to play.

In the research that my organisation funded, it was important to both define and deliver impact beyond that of classic academic achievement.

Considerations for choosing frameworks to assess research impact

July 8, 2021March 16, 2021

By Elena Louder, Carina Wyborn, Christopher Cvitanovic and Angela T. Bednarek

authors_elena-louder_carina-wyborn_christopher-cvitanovic_angela-t-bednarek — 1. Elena Louder (biography)
2. Carina Wyborn (biography)
3. Christopher Cvitanovic (biography)
4. Angela Bednarek (biography)

What should you take into account in selecting among the many frameworks for evaluating research impact?

In our recent paper (Louder et al., 2021) we examined the epistemological foundations and assumptions of several frameworks and drew out their similarities and differences to help improve the evaluation of research impact. In doing so we identified four key principles or ‘rules of thumb’ to help guide the selection of an evaluation framework for application within a specific context.

1. Be clear about underlying assumptions of knowledge production and definitions of impact

Clarifying from the start how research activities are intended to achieve impact is an important pre-cursor to designing an evaluation. Furthermore, defining what you mean by impact is an important first step in selecting indicators to know if you’ve achieved it.

Five organizational features for successful interdisciplinary research

May 4, 2022January 26, 2021

By Jessica Blythe and Chris Cvitanovic

mosaic_authors_jessica-blythe_chris-cvitanovic — 1. Jessica Blythe (biography)
2. Chris Cvitanovic (biography)

How can significant challenges associated with doing interdisciplinary research be overcome? What are the best ways to build institutional capacity and structures that support interdisciplinary research?

We have identified five key organizational features that enable successful interdisciplinary research. These are based on an evaluation of the Centre for Marine Socioecology in Tasmania, Australia, which brings together disciplinary expertise in physics, law, economics, biology, sociology and governance. We obtained perspectives across all disciplines and career stages from PhD students to the leadership team.

The five organizational features, also shown in the figure below, are:

Providing a richer assessment of research influence and impact

October 17, 2022April 14, 2020

By Gabriele Bammer

author - gabriele bammer — Gabriele Bammer (biography)

How can we affirm, value and capitalise on the unique strengths that each individual brings to interdisciplinary and transdisciplinary research? In particular, how can we capture diversity across individuals, as well as the richness and distinctness of each individual’s influence and impact?

In the course of writing ten reflective narratives (nine single-authored and one co-authored), eleven of us stumbled on a technique that we think could have broader utility in assessing influence and impact, especially in research but also in education (Bammer et al., 2019).

A framework to evaluate the impacts of research on policy and practice

July 5, 2022March 3, 2020

By Laura Meagher and David Edwards

authors_laura-meagher_david-edwards — 1. Laura Meagher (biography)
2. David Edwards (biography)

What is meant by impact generation and how can it be facilitated, captured and shared? How can researchers be empowered to think beyond ‘instrumental’ impact and identify other changes generated by their work? How can the cloud of complexity be dispersed so that numerous factors affecting development of impacts can be seen? How can a way be opened for researchers to step back and reflect critically on what happened and what could be improved in the future? How can research teams and stakeholders translate isolated examples of impact and causes of impact into narratives for both learning and dissemination?

Tracking stakeholder engagement and research impact

September 16, 2021February 12, 2019

By Cathy Day

Is there an easy and efficient way to keep track of stakeholder engagement and research impact?

My colleagues and I have developed a system with two components: (1) noting engagement and impact soon after they occur and (2) recording them in a way that enables the information to be extracted for whatever purpose is required. I describe the tracking spreadsheet, the recording process we use and then how the spreadsheet is used for reporting.

Developing a ‘capabilities approach’ for measuring social impact

August 11, 2021November 27, 2018

By Daniel J. Hicks

daniel-hicks — Daniel J. Hicks (biography)

Why do familiar metrics of impact often seem “thin” or to miss the point of research designed to address real-world problems? Is there a better way to measure the social impact of research?

In a recent paper (Hicks et al., 2018), my coauthors and I identified a key limitation with current metrics and started to look at how concepts from philosophy — specifically, ethics — can help us explain the goals of our research, and potentially lead to better metrics.

What’s the problem?

To understand the limitations of current metrics for measuring the social impact of research, it is useful to understand two distinctions, between resources and goals and between inward-facing and outward-facing goals for research.

Assessing research contribution claims: The “what else test”

April 17, 2020August 28, 2018

By Jess Dart

In situations where multiple factors, in addition to your research, are likely to have caused an observed policy or practice change, how can you measure your contribution? How can you be sure that the changes would not have happened anyway?

In making contribution claims there are three levels of rigour, each requiring more evaluation expertise and resourcing. These are summarised in the table below. The focus in this blog post is on the basic or minimum level of evaluation and specifically on the “what else test.”

Is it legitimate for transdisciplinary research to set out to change society?

November 9, 2021March 15, 2018

By Antonietta Di Giulio and Rico Defila

An unspoken and unchallenged assumption underpinning much discourse about transdisciplinary research is that it must change society.

The assumption goes beyond whether research should contribute to change, or whether research impacts developments in society, or whether research should investigate societal problems and provide solutions, or anything similar – it is that research should actively and intentionally be transformative. This generally goes hand-in-hand with a deep conviction that researchers are entitled to actually change society according to what they believe to be right. For many this conviction allows researchers to impose their interventions and solutions on other societal actors by, if necessary, being manipulative.