Evaluation – Integration and Implementation Insights

What transdisciplinary researchers should know about evaluation: Origins and current state

June 16, 2023September 13, 2022

By Wolfgang Beywl and Amy Gullickson

authors_wolfgang-beywl_amy-gullickson — 1. Wolfgang Beywl (biography)
2. Amy Gullickson (biography)

Efforts to develop evaluation in transdisciplinary research have mostly been conducted without reference to the evaluation literature, effectively re-inventing and re-discussing key concepts. What do transdisciplinary researchers need to know to build on the in-depth knowledge available in evaluation science?

Here we add to other key contributions about evaluation in i2Insights, especially:

Belcher and colleagues, who provide a tool for evaluating transdisciplinary research
Nagy and Schäfer, who describe how to systematically design transdisciplinary project evaluation
Meagher and Edwards, who provide a framework to evaluate the impacts of research on policy and practice
Louder and colleagues, who discuss ways of choosing a framework to assess research impact.

We focus specifically on the origins and the current state of the evaluation field.

How to systematically design transdisciplinary project evaluation

February 9, 2022February 8, 2022

By Emilia Nagy and Martina Schäfer

authors_emilia-nagy_martina-schafer — 1. Emilia Nagy (biography)
2. Martina Schäfer (biography; photo source: Landtag Brandenburg)

How can the formative, ie. process, evaluation of transdisciplinary research projects best incorporate the likely link between process and outcomes in such research? What are some useful approaches for developing an effective evaluation plan with a lens of impact orientation?

We describe how to systematically formulate criteria and indicators for the evaluation of transdisciplinary projects by combining:

impactful research practices (Lux et al., 2019)
impact heuristics (Schäfer et al., 2021)
theory-of-change method (Belcher et al., 2019).

The combination of these approaches provides a strong foundation for impact orientation in all project phases.

Eleven success factors for transdisciplinary real-world labs

August 3, 2023January 18, 2022

By Niko Schäpke, Oskar Marg, Matthias Bergmann, Franziska Stelzer and Daniel J. Lang

authors_niko-schapke_oskar-marg_matthias-bergmann_franziska-stelzer_daniel-lang — 1. Niko Schäpke; 2. Oskar Marg; 3. Matthias Bergmann; 4. Franziska Stelzer; 5. Daniel Lang (biographies)

What is required for transdisciplinary real-world laboratories (labs) to successfully tackle and achieve long-term societal change? How can they make the change process transferable? What is required of the societal and scientific actors?

We discuss eleven success factors to facilitate successful transdisciplinary collaboration and to achieve desired societal effects. These are based on an accompanying research project, which supported and observed several real-world labs, aiming to develop overarching insights on methods and success factors. The accompanying research project also provided consultancy to strengthen the implementation activities of the real-world labs and supported the networking of the labs, including with labs in other countries.

The factors presented below should be seen as generic recommendations to increase the likelihood of success. As each lab has a unique character, factors should be considered, deliberated and addressed for each lab on an individual basis.

Success factor 1: Find the right balance between scientific and societal goals

Overcoming the mismatch between goals and outcomes in knowledge exchange

January 11, 2022

By Denis Karcher and Chris Cvitanovic

authors_denis-karcher_christopher-cvitanovic — 1. Denis Karcher (biography)
2. Chris Cvitanovic (biography)

How well do researchers achieve the research impacts they aim for? And if there is a mismatch, does it matter?

Together with colleagues (Karcher et al., 2021), we systematically searched for and reviewed nearly 400 studies that described goals and outcomes that were claimed for knowledge exchange at the science-policy interface. Although our focus was on the environmental sciences, the results may be more widely useful.

Big ambitions

The eight top goals that studies described for their knowledge exchange activities were:

1. Usability, eg., that the interaction with policy makers and/or the knowledge created were credible, legitimate, relevant, and timely (458 references).

Stakeholder engagement primer: 9. Evaluating engagement

January 11, 2022December 9, 2021

By Gabriele Bammer

primer_stakeholder-engagement_9

How can the effectiveness of stakeholder engagement be judged? How can the outcomes be assessed? How much effort should go into such evaluation?

How and when to evaluate

In any stakeholder engagement there is no shortage of aspects that could be evaluated. The challenge is aligning the audience for the evaluation, the key issues to be assessed and the available resources.

For example, if a research team is interested in learning from what went wrong in a stakeholder engagement (for instance, if a stakeholder stopped participating or became hostile) and has no money set aside for evaluation, it might rely on self-reflection and anecdotal evidence to figure out what happened.

A tool for transdisciplinary research planning and evaluation

June 23, 2023September 2, 2021

By Brian Belcher, Rachel Claus, Rachel Davel, Stephanie Jones and Daniela Pinto

authors_brian-belcher_rachel-claus_rachel-davel_stephanie-jones_daniela-pinto — 1. Brian Belcher; 2. Rachel Claus; 3. Rachel Davel; 4. Stephanie Jones; 5. Daniela Pinto (biographies)

What are the characteristics of high-quality transdisciplinary research? As research approaches increasingly cross disciplinary bounds and engage stakeholders in the research process to more effectively address complex problems, traditional academic research assessment criteria are insufficient and may even constrain transdisciplinary research development and use. There is a need for appropriate principles and criteria to guide transdisciplinary research practice and evaluation.

In response to this need, Belcher et al. (2016) developed the Transdisciplinary Research Quality Assessment Framework based on a systematic review of literature that discussed the definition and measurement of research quality for inter- and trans- disciplinary research. Through applied testing in case study evaluations, we have revised and refined the framework to improve its utility.

Principles and Criteria

The tool provides guiding principles and corresponding criteria for transdisciplinary research design and implementation.

Theory of Change in a nutshell

September 23, 2021August 24, 2021

By Heléne Clark

author_helene-clark — Heléne Clark (biography)

How can you plan to make change happen or evaluate the effectiveness of actions you took? How can you link desired long-term goals with all the conditions that must be in place? How can you map out a step-by-step pathway that highlights your assumptions and expectations?

Theory of Change (ToC) is a graphic and narrative explanation of how and why a change process is expected to happen in a particular context.

At its heart, Theory of Change spells out initiative or program logic. It defines long-term goals and then maps backward to identify changes thought to be necessary to the goal that need to happen earlier (preconditions).

Theory of Change purports to explain change process in diagrammatically modeling all the causal linkages in an initiative, ie., its shorter-term, intermediate, and longer-term outcomes.

Considerations for choosing frameworks to assess research impact

July 8, 2021March 16, 2021

By Elena Louder, Carina Wyborn, Christopher Cvitanovic and Angela T. Bednarek

authors_elena-louder_carina-wyborn_christopher-cvitanovic_angela-t-bednarek — 1. Elena Louder (biography)
2. Carina Wyborn (biography)
3. Christopher Cvitanovic (biography)
4. Angela Bednarek (biography)

What should you take into account in selecting among the many frameworks for evaluating research impact?

In our recent paper (Louder et al., 2021) we examined the epistemological foundations and assumptions of several frameworks and drew out their similarities and differences to help improve the evaluation of research impact. In doing so we identified four key principles or ‘rules of thumb’ to help guide the selection of an evaluation framework for application within a specific context.

1. Be clear about underlying assumptions of knowledge production and definitions of impact

Clarifying from the start how research activities are intended to achieve impact is an important pre-cursor to designing an evaluation. Furthermore, defining what you mean by impact is an important first step in selecting indicators to know if you’ve achieved it.

Addressing societal challenges: From interdisciplinarity to research portfolios analysis

March 9, 2021February 18, 2021

By Ismael Rafols

author_ismael-rafols — Ismael Rafols (biography)

How can knowledge integration for addressing societal challenges be mapped, ‘measured’ and assessed?

In this blog post I argue that measuring averages or aggregates of ‘interdisciplinarity’ is not sufficiently focused for evaluating research aimed at societal contributions. Instead, one should take a portfolio approach to analyze knowledge integration as a systemic process over research landscapes; in particular, focusing on the directions, diversity and synergies of research trajectories.

There are two main reasons:

‘Measuring’ interdisciplinarity: from indicators to indicating

August 18, 2023February 9, 2021

By Ismael Rafols

Indicators of interdisciplinarity are increasingly requested. Yet efforts to make aggregate indicators have repeatedly failed due to the diversity and ambiguity of understandings of the notion of interdisciplinarity. What if, instead of universal indicators, a contextualised process of indicating interdisciplinarity was used?

In this blog post I briefly explore the failure of attempts to identify universal indicators and the importance of moving from indicatORS to indicatING. By this I mean: An assessment of specific interdisciplinary projects or programs for indicating where and how interdisciplinarity develops as a process, given the particular understandings relevant for the specific policy goals.

This reflects the notion of directionality in research and innovation, which is gaining hold in policy.

Acknowledging and responding to criticisms of interdisciplinarity / Reconnaître et répondre aux critiques de l’interdisciplinarité

August 3, 2023July 14, 2020

By Romain Sauzet

A French version of this post is available

author_romain-sauzet — Romain Sauzet (biography)

What are the core arguments that critics of interdisciplinarity employ? Which of these criticisms can help to clarify what interdisciplinarity is and what it isn’t?

While some of the criticisms of interdisciplinarity stem from a general misunderstanding of its purpose or from a bad experience, others seem well-founded. Thus, while some must be rejected, others should be accepted.

I outline five different types of criticisms drawn from three main sources:
(1) academic writings (see reference list), (2) an empirical survey on interdisciplinarity (Sauzet 2017), (3) informal discussions. These criticisms extend the ideas presented in an earlier blog post, Why We Should Not Ignore Interdisciplinarity’s Critics by Rick Szostak. I reflect on how interdisciplinarity could be improved by attending to key criticisms.

Providing a richer assessment of research influence and impact

August 11, 2023April 14, 2020

By Gabriele Bammer

author - gabriele bammer — Gabriele Bammer (biography)

How can we affirm, value and capitalise on the unique strengths that each individual brings to interdisciplinary and transdisciplinary research? In particular, how can we capture diversity across individuals, as well as the richness and distinctness of each individual’s influence and impact?

In the course of writing ten reflective narratives (nine single-authored and one co-authored), eleven of us stumbled on a technique that we think could have broader utility in assessing influence and impact, especially in research but also in education (Bammer et al., 2019).We wrote the reflective narratives for a special issue on Julie Thompson Klein’s academic work, in our case especially her impact in Australia and New Zealand. (A taste of Klein’s work can be found in her contributions to this i2Insights blog.)

The process of writing the reflective narratives was unstructured, with each of us writing as much or as little as we wished on whatever aspect of Klein’s work had resonated with us.