By Emilia Nagy and Martina Schäfer
2. Martina Schäfer (biography; photo source: Landtag Brandenburg)
How can the formative, ie. process, evaluation of transdisciplinary research projects best incorporate the likely link between process and outcomes in such research? What are some useful approaches for developing an effective evaluation plan with a lens of impact orientation?
We describe how to systematically formulate criteria and indicators for the evaluation of transdisciplinary projects by combining:
- impactful research practices (Lux et al., 2019)
- impact heuristics (Schäfer et al., 2021)
- theory-of-change method (Belcher et al., 2019).
The combination of these approaches provides a strong foundation for impact orientation in all project phases. The additional value for formative evaluation is that it combines the quality criteria for effective processes or activities in transdisciplinary research projects with impact indicators.
Impactful research practices
In transdisciplinary research, societal effects can evolve not only from relevant project results but also from the activities carried out during the transdisciplinary research processes, especially when these are done well. In research with colleagues (Lux et al., 2019), we have suggested that there are five major areas where research practices are most likely to be impactful:
- problem relevance – ie. everything that facilitates a better understanding of the problem situation and the application context
- connectivity – ie. meeting the needs and expectations of current target groups, as well as others who may need to be involved
- roles and responsibilities – ie. clarity about the roles, functions and tasks of each partner, which may change in different phases of the project. Particularly important are responsibility for knowledge integration and the role of intermediaries in supporting the transfer of knowledge to the field or comparable problem contexts
- interests and concerns – ie. transparency about underlying interests and concerns, avoiding hidden agendas and objectives that are not shared
- collaborative culture – ie. positive and inspiring formal and informal interactions.
Impact heuristics
Impact heuristics distinguish between different orders of societal effects depending on their temporal and spatial distance from the research processes and their results (Schäfer et al., 2021). They are:
- first-order effects: direct effects within the duration and the spatial scope of a research project, such as learning and capacity building, formation of networks, improving the situation in the respective field of action, and increase in reputation
- second-order effects: effects beyond the project but within the close temporal or spatial context of the project, such as institutionalisation of transformative approaches, establishment of project-related products or infrastructure, transfer of project results to other spatial contexts
- third-order effects: changes beyond the temporal or spatial context of the project in the entire field of action or problem field, such as influence on public discourse, influence on law and regulation, further structural effects.
Theory of change method
The Theory of Change method is useful for establishing shared hypotheses for a change process by planning backwards starting from a long-term shared project vision and identifying the conditions that need to be in place for the intended effects to occur (Belcher et al., 2019). Applying the theory of change method results in a flow diagram containing sets of activities, outputs and effects organized by non-linear impact-pathways.
A systematic evaluation framework
To combine the three components described above, we, as project evaluators, usually start with a workshop to define the theory of change, including as many project members (scientists as well as practitioners) as possible. From the flowchart of impacts, participants select particularly relevant impact pathways, with the starting point being a research activity.
As the formative evaluation team, we suggest indicators and monitoring questions for the intended impacts on first-, second- and third- order effects. The project team assesses the suggested indicators for suitability, feasibility and manageability.
After the joint adaptation of the set of indicators and their operationalisation via monitoring questions, data collection and ongoing reflection about the process can be started. At this stage particular attention is paid to the five impactful research practices.
Our experience so far has shown that this systematic approach to formative evaluation, which defines indicators beyond the boundaries of the project duration and project context, supports projects in their impact-oriented research activities.
What do you think? Have you developed other ways for improving formative evaluation of transdisciplinary research? Would the approach we have developed be useful in your research too?
References:
Belcher, B. M., Claus, R., Davel, R. and Ramirez, L. F. (2019). Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects. Environmental Science and Policy, 101: 192-203. (Online) (DOI): https://doi.org/10.1016/j.envsci.2019.08.013
Lux, A., Schäfer, M., Bergmann, M., Jahn, T., Marg, O., Nagy, E., Ransiek, A. and Theiler, L. (2019). Societal effects of transdisciplinary sustainability research – How can they be strengthened during the research process? Environmental Science and Policy, 101: 183-191. (Online). (DOI): https://doi.org/10.1016/j.envsci.2019.08.012
Schäfer, M., Bergmann, M. and Theiler, L. (2021). Systematizing societal effects of transdisciplinary research. Research Evaluation, 30, 4: 484–499. (Online) (DOI): https://doi.org/10.1093/reseval/rvab019
Biography: Emilia Nagy researches on transdisciplinarity at the Center for Technology and Society (ZTG) at Technische Universität Berlin in Germany. Her focus is on increasing the societal effects of transdisciplinary sustainability research.
Biography: Martina Schäfer PhD is the Scientific Director of the Center for Technology and Society (ZTG) of Technische Universität Berlin in Germany. She has coordinated inter- and transdisciplinary research projects in sustainable regional development, sustainable consumption and sustainable land use. One of her research foci is reflection on methods for inter- and transdisciplinary cooperation as well as the societal impact of this research mode.