Evaluation

A framework for considering context in evaluation of transdisciplinary research projects

By Julia Schegg, Rea Pärli, Manuel Fischer and Eva Lieberherr.

authors_schegg_parli_fischer_lieberherr
1. Julia Schegg (biography)
2. Rea Pärli (biography)
3. Manuel Fischer (biography)
4. Eva Lieberherr (biography)

How do contextual factors influence the effects of transdisciplinary research projects? How can assessment of transdisciplinary research move from only considering outcome effectiveness to understanding the reasons behind how and why something works?

Our framework considers both context and the effects of transdisciplinary research projects, as summarised in the figure below. Each aspect of the framework is described in more detail.

Context

We based the conceptualization of context for any given problem on three established frameworks from policy studies, emphasising system conditions, public action resources and events.

Read more

Six key steps for stakeholder engagement

By Khara Grieger, Kimberly Bourne, Alison Deviney and Nourou Barry.

authors_grieger_bourne_deviney_barry
1. Khara Grieger (biography)
2. Kimberly Bourne (biography)
3. Alison Deviney (biography)
4. Nourou Barry (biography)

How can you systematically plan stakeholder engagement? What are the key issues that need to be considered? What guiding questions can help? 

STEP 1: Identify and clarify engagement goals

Spend time at the beginning of the engagement process to clearly identify why you are engaging with stakeholders. Common goals include sharing knowledge or information; collecting insights, perspectives, or information from stakeholders; and co-creating or co-designing solutions. Other potential goals may include building trust and improving transparency, enhancing collaborations and partnerships, and improving the implementation of decisions.

Read more

A self-assessment checklist to improve interdisciplinarity in research projects / Un questionnaire d’auto-évaluation pour améliorer les projets de recherche interdisciplinaire

By Flore Nonchez.

A French version of this post is available

flore-nonchez
Flore Nonchez (biography)

How can researchers improve the quality of their interdisciplinary proposals? What would help in clarifying project formulation, enriching description, maximizing relevance, facilitating workplan implementation (by anticipating possible difficulties) and enhancing impact?

The self-assessment checklist provided below aims to help project leaders and their research teams systematically consider the specific interdisciplinary aspects of an interdisciplinary research project, whatever their original discipline or experience of interdisciplinarity.

Read more

Transdisciplinary research with and for artificial intelligence

By Florian Keil, Melina Stein and Flurina Schneider.

authors_florian-keil_melina-stein_flurina-schneider
1. Florian Keil’s biography
2. Melina Stein (biography)
3. Flurina Schneider (biography)

Is artificial intelligence, a technology aggressively advertised as the ultimate cure-all, fundamentally incompatible with transdisciplinarity and its decades-old insight that the “wicked” problems of the real world do not lend themselves to one-dimensional solutions? Should transdisciplinary research outright reject a technology that is already undermining efforts to achieve social and environmental justice? Or can artificial intelligence actually support transdisciplinary research when used responsibly?

Using artificial intelligence in transdisciplinary research requires a critical mindset

Read more

Evaluation criteria for transdisciplinary research

This i2Insights contribution has been retracted.

For an excellent framework on evaluation criteria for transdisciplinary research, see:

Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R. and Zornes, D. A. (2016). Defining and Assessing Research Quality in a Transdisciplinary Context. Research Evaluation, 25, 1–17. (Online – open access) (DOI): https://doi.org/10.1093/reseval/rvv025

with an updated and refined version available at: Transdisciplinary-Research-Quality-Assessment-Framework-2.0.pdf. 

 

What transdisciplinary researchers should know about evaluation: Origins and current state

By Wolfgang Beywl and Amy Gullickson

authors_wolfgang-beywl_amy-gullickson
1. Wolfgang Beywl (biography)
2. Amy Gullickson (biography)

Efforts to develop evaluation in transdisciplinary research have mostly been conducted without reference to the evaluation literature, effectively re-inventing and re-discussing key concepts. What do transdisciplinary researchers need to know to build on the in-depth knowledge available in evaluation science?

Here we add to other key contributions about evaluation in i2Insights, especially:

Read more

How to systematically design transdisciplinary project evaluation

By Emilia Nagy and Martina Schäfer

authors_emilia-nagy_martina-schafer
1. Emilia Nagy (biography)
2. Martina Schäfer (biography; photo source: Landtag Brandenburg)

How can the formative, ie. process, evaluation of transdisciplinary research projects best incorporate the likely link between process and outcomes in such research? What are some useful approaches for developing an effective evaluation plan with a lens of impact orientation?

We describe how to systematically formulate criteria and indicators for the evaluation of transdisciplinary projects by combining:

  • impactful research practices (Lux et al., 2019)
  • impact heuristics (Schäfer et al., 2021)
  • theory-of-change method (Belcher et al., 2019).

The combination of these approaches provides a strong foundation for impact orientation in all project phases.

Read more

Eleven success factors for transdisciplinary real-world labs

By Niko Schäpke, Oskar Marg, Matthias Bergmann, Franziska Stelzer and Daniel J. Lang

authors_niko-schapke_oskar-marg_matthias-bergmann_franziska-stelzer_daniel-lang
1. Niko Schäpke; 2. Oskar Marg; 3. Matthias Bergmann; 4. Franziska Stelzer; 5. Daniel Lang (biographies)

What is required for transdisciplinary real-world laboratories (labs) to successfully tackle and achieve long-term societal change? How can they make the change process transferable? What is required of the societal and scientific actors?

We discuss eleven success factors to facilitate successful transdisciplinary collaboration and to achieve desired societal effects. These are based on an accompanying research project, which supported and observed several real-world labs, aiming to develop overarching insights on methods and success factors. The accompanying research project also provided consultancy to strengthen the implementation activities of the real-world labs and supported the networking of the labs, including with labs in other countries.

The factors presented below should be seen as generic recommendations to increase the likelihood of success. As each lab has a unique character, factors should be considered, deliberated and addressed for each lab on an individual basis.

Read more

Overcoming the mismatch between goals and outcomes in knowledge exchange

By Denis Karcher and Chris Cvitanovic

authors_denis-karcher_christopher-cvitanovic
1. Denis Karcher (biography)
2. Chris Cvitanovic (biography)

How well do researchers achieve the research impacts they aim for? And if there is a mismatch, does it matter?

Together with colleagues (Karcher et al., 2021), we systematically searched for and reviewed nearly 400 studies that described goals and outcomes that were claimed for knowledge exchange at the science-policy interface. Although our focus was on the environmental sciences, the results may be more widely useful.

Big ambitions

The eight top goals that studies described for their knowledge exchange activities were:

1. Usability, eg., that the interaction with policy makers and/or the knowledge created were credible, legitimate, relevant, and timely (458 references).

Read more

Stakeholder engagement primer: 9. Evaluating engagement

By Gabriele Bammer

primer_stakeholder-engagement_9

How can the effectiveness of stakeholder engagement be judged? How can the outcomes be assessed? How much effort should go into such evaluation?

How and when to evaluate

In any stakeholder engagement there is no shortage of aspects that could be evaluated. The challenge is aligning the audience for the evaluation, the key issues to be assessed and the available resources.

For example, if a research team is interested in learning from what went wrong in a stakeholder engagement (for instance, if a stakeholder stopped participating or became hostile) and has no money set aside for evaluation, it might rely on self-reflection and anecdotal evidence to figure out what happened.

Read more

A tool for transdisciplinary research planning and evaluation

By Brian Belcher, Rachel Claus, Rachel Davel, Stephanie Jones and Daniela Pinto

authors_brian-belcher_rachel-claus_rachel-davel_stephanie-jones_daniela-pinto
1. Brian Belcher; 2. Rachel Claus; 3. Rachel Davel; 4. Stephanie Jones; 5. Daniela Pinto (biographies)

What are the characteristics of high-quality transdisciplinary research? As research approaches increasingly cross disciplinary bounds and engage stakeholders in the research process to more effectively address complex problems, traditional academic research assessment criteria are insufficient and may even constrain transdisciplinary research development and use. There is a need for appropriate principles and criteria to guide transdisciplinary research practice and evaluation.

In response to this need, Belcher et al. (2016) developed the Transdisciplinary Research Quality Assessment Framework based on a systematic review of literature that discussed the definition and measurement of research quality for inter- and trans- disciplinary research. Through applied testing in case study evaluations, we have revised and refined the framework to improve its utility.

Principles and Criteria

The tool provides guiding principles and corresponding criteria for transdisciplinary research design and implementation. Four principles comprise the Quality Assessment Framework:

Read more

Theory of Change in a nutshell

By Heléne Clark

author_helene-clark
Heléne Clark (biography)

How can you plan to make change happen or evaluate the effectiveness of actions you took? How can you link desired long-term goals with all the conditions that must be in place? How can you map out a step-by-step pathway that highlights your assumptions and expectations?

Theory of Change (ToC) is a graphic and narrative explanation of how and why a change process is expected to happen in a particular context.

At its heart, Theory of Change spells out initiative or program logic. It defines long-term goals and then maps backward to identify changes thought to be necessary to the goal that need to happen earlier (preconditions).

Read more