Research impact

How to systematically design transdisciplinary project evaluation

By Emilia Nagy and Martina Schäfer

authors_emilia-nagy_martina-schafer
1. Emilia Nagy (biography)
2. Martina Schäfer (biography; photo source: Landtag Brandenburg)

How can the formative, ie. process, evaluation of transdisciplinary research projects best incorporate the likely link between process and outcomes in such research? What are some useful approaches for developing an effective evaluation plan with a lens of impact orientation?

We describe how to systematically formulate criteria and indicators for the evaluation of transdisciplinary projects by combining:

  • impactful research practices (Lux et al., 2019)
  • impact heuristics (Schäfer et al., 2021)
  • theory-of-change method (Belcher et al., 2019).

The combination of these approaches provides a strong foundation for impact orientation in all project phases.

Read more

Overcoming the mismatch between goals and outcomes in knowledge exchange

By Denis Karcher and Chris Cvitanovic

authors_denis-karcher_christopher-cvitanovic
1. Denis Karcher (biography)
2. Chris Cvitanovic (biography)

How well do researchers achieve the research impacts they aim for? And if there is a mismatch, does it matter?

Together with colleagues (Karcher et al., 2021), we systematically searched for and reviewed nearly 400 studies that described goals and outcomes that were claimed for knowledge exchange at the science-policy interface. Although our focus was on the environmental sciences, the results may be more widely useful.

Big ambitions

The eight top goals that studies described for their knowledge exchange activities were:

1. Usability, eg., that the interaction with policy makers and/or the knowledge created were credible, legitimate, relevant, and timely (458 references).

Read more

Five insights on achieving research impact

authors_niki-ellis_anne-maree-dowd_tamika-heiden_gabriele-bammer
1. Niki Ellis (biography)
2, Anne-Maree Dowd (biography)
3. Tamika Heiden (biography)
4, Gabriele Bammer (biography)

By Niki Ellis, Anne-Maree Dowd, Tamika Heiden and Gabriele Bammer

What does it take for research to be impactful? How should research impact be assessed? How much responsibility for impact should rest with researchers and how much with government, business and/or community partners?

We present five key insights based on our experience in achieving research impact in Australia:

  1. Planning for impact is essential
  2. Quality relationships trump all other factors
  3. Assessment of research contributions should be tailored to the type of research and based on team, not individual, performance
  4. Researchers alone cannot be responsible for achieving impact
  5. Be open to continual learning.

Read more

Investing in change through research funding

By Petra Lundgren

author_petra-lundgren
Petra Lundgren (biography)

How do funders think about investing in research that is intended to lead to change?

This blog post is written from the perspective of a research funder. More specifically it is based on reflections and lessons learned during five years managing and directing strategic research programs at a not-for-profit foundation, investing in science that would benefit the health and resilience of Australia’s Great Barrier Reef.

Our funding mandate was to include research in a larger body of work towards a broader vision of change. This therefore provided the basis of my work and helped me shape the view that the funder has a big and critical role to play.

In the research that my organisation funded, it was important to both define and deliver impact beyond that of classic academic achievement.

Read more

Considerations for choosing frameworks to assess research impact

By Elena Louder, Carina Wyborn, Christopher Cvitanovic and Angela T. Bednarek

authors_elena-louder_carina-wyborn_christopher-cvitanovic_angela-t-bednarek
1. Elena Louder (biography)
2. Carina Wyborn (biography)
3. Christopher Cvitanovic (biography)
4. Angela Bednarek (biography)

What should you take into account in selecting among the many frameworks for evaluating research impact?

In our recent paper (Louder et al., 2021) we examined the epistemological foundations and assumptions of several frameworks and drew out their similarities and differences to help improve the evaluation of research impact. In doing so we identified four key principles or ‘rules of thumb’ to help guide the selection of an evaluation framework for application within a specific context.

1. Be clear about underlying assumptions of knowledge production and definitions of impact

Clarifying from the start how research activities are intended to achieve impact is an important pre-cursor to designing an evaluation. Furthermore, defining what you mean by impact is an important first step in selecting indicators to know if you’ve achieved it.

Read more

Five organizational features for successful interdisciplinary research

By Jessica Blythe and Chris Cvitanovic

mosaic_authors_jessica-blythe_chris-cvitanovic
1. Jessica Blythe (biography)
2. Chris Cvitanovic (biography)

How can significant challenges associated with doing interdisciplinary research be overcome? What are the best ways to build institutional capacity and structures that support interdisciplinary research?

We have identified five key organizational features that enable successful interdisciplinary research. These are based on an evaluation of the Centre for Marine Socioecology in Tasmania, Australia, which brings together disciplinary expertise in physics, law, economics, biology, sociology and governance. We obtained perspectives across all disciplines and career stages from PhD students to the leadership team.

The five organizational features, also shown in the figure below, are:

Read more

Providing a richer assessment of research influence and impact

By Gabriele Bammer

author - gabriele bammer
Gabriele Bammer (biography)

How can we affirm, value and capitalise on the unique strengths that each individual brings to interdisciplinary and transdisciplinary research? In particular, how can we capture diversity across individuals, as well as the richness and distinctness of each individual’s influence and impact?

In the course of writing ten reflective narratives (nine single-authored and one co-authored), eleven of us stumbled on a technique that we think could have broader utility in assessing influence and impact, especially in research but also in education (Bammer et al., 2019).We wrote the reflective narratives for a special issue on Julie Thompson Klein’s academic work, in our case especially her impact in Australia and New Zealand. (A taste of Klein’s work can be found in her contributions to this i2Insights blog.)

The process of writing the reflective narratives was unstructured, with each of us writing as much or as little as we wished on whatever aspect of Klein’s work had resonated with us.

Read more

A framework to evaluate the impacts of research on policy and practice

By Laura Meagher and David Edwards

authors_laura-meagher_david-edwards
1. Laura Meagher (biography)
2. David Edwards (biography)

What is meant by impact generation and how can it be facilitated, captured and shared? How can researchers be empowered to think beyond ‘instrumental’ impact and identify other changes generated by their work? How can the cloud of complexity be dispersed so that numerous factors affecting development of impacts can be seen? How can a way be opened for researchers to step back and reflect critically on what happened and what could be improved in the future? How can research teams and stakeholders translate isolated examples of impact and causes of impact into narratives for both learning and dissemination?

We have developed a framework to evaluate research impact in a way that addresses these questions. It has been piloted on 12 case studies led by Forest Research, a government research agency in UK (Edwards and Meagher 2019) and is likely to be useful to researchers more generally, perhaps especially but not exclusively those in applied fields. To date the framework has been found to be user-friendly and fit for purpose.

Read more

Tracking stakeholder engagement and research impact

By Cathy Day

Cathy Day (biography)

Is there an easy and efficient way to keep track of stakeholder engagement and research impact?

My colleagues and I have developed a system with two components: (1) noting engagement and impact soon after they occur and (2) recording them in a way that enables the information to be extracted for whatever purpose is required. I describe the tracking spreadsheet, the recording process we use and then how the spreadsheet is used for reporting.

Read more

Developing a ‘capabilities approach’ for measuring social impact

By Daniel J. Hicks

daniel-hicks
Daniel J. Hicks (biography)

Why do familiar metrics of impact often seem “thin” or to miss the point of research designed to address real-world problems? Is there a better way to measure the social impact of research?

In a recent paper (Hicks et al., 2018), my coauthors and I identified a key limitation with current metrics and started to look at how concepts from philosophy — specifically, ethics — can help us explain the goals of our research, and potentially lead to better metrics.

What’s the problem?

To understand the limitations of current metrics for measuring the social impact of research, it is useful to understand two distinctions, between resources and goals and between inward-facing and outward-facing goals for research.

Read more

Assessing research contribution claims: The “what else test”

By Jess Dart

Jess Dart (biography)

In situations where multiple factors, in addition to your research, are likely to have caused an observed policy or practice change, how can you measure your contribution? How can you be sure that the changes would not have happened anyway?

In making contribution claims there are three levels of rigour, each requiring more evaluation expertise and resourcing. These are summarised in the table below. The focus in this blog post is on the basic or minimum level of evaluation and specifically on the “what else test.”

Read more

Is it legitimate for transdisciplinary research to set out to change society?

By Antonietta Di Giulio and Rico Defila

antonietta-di-giulio
Antonietta Di Giulio (biography)
rico-defila
Rico Defila (biography)

An unspoken and unchallenged assumption underpinning much discourse about transdisciplinary research is that it must change society.

The assumption goes beyond whether research should contribute to change, or whether research impacts developments in society, or whether research should investigate societal problems and provide solutions, or anything similar – it is that research should actively and intentionally be transformative. This generally goes hand-in-hand with a deep conviction that researchers are entitled to actually change society according to what they believe to be right. For many this conviction allows researchers to impose their interventions and solutions on other societal actors by, if necessary, being manipulative.

Read more