By Emilia Nagy and Martina Schäfer
2. Martina Schäfer (biography; photo source: Landtag Brandenburg)
How can the formative, ie. process, evaluation of transdisciplinary research projects best incorporate the likely link between process and outcomes in such research? What are some useful approaches for developing an effective evaluation plan with a lens of impact orientation?
We describe how to systematically formulate criteria and indicators for the evaluation of transdisciplinary projects by combining:
- impactful research practices (Lux et al., 2019)
- impact heuristics (Schäfer et al., 2021)
- theory-of-change method (Belcher et al., 2019).
The combination of these approaches provides a strong foundation for impact orientation in all project phases. The additional value for formative evaluation is that it combines the quality criteria for effective processes or activities in transdisciplinary research projects with impact indicators.
Impactful research practices
In transdisciplinary research, societal effects can evolve not only from relevant project results but also from the activities carried out during the transdisciplinary research processes, especially when these are done well. In research with colleagues (Lux et al., 2019), we have suggested that there are five major areas where research practices are most likely to be impactful:
- problem relevance – ie. everything that facilitates a better understanding of the problem situation and the application context
- connectivity – ie. meeting the needs and expectations of current target groups, as well as others who may need to be involved
- roles and responsibilities – ie. clarity about the roles, functions and tasks of each partner, which may change in different phases of the project. Particularly important are responsibility for knowledge integration and the role of intermediaries in supporting the transfer of knowledge to the field or comparable problem contexts
- interests and concerns – ie. transparency about underlying interests and concerns, avoiding hidden agendas and objectives that are not shared
- collaborative culture – ie. positive and inspiring formal and informal interactions.
Impact heuristics
Impact heuristics distinguish between different orders of societal effects depending on their temporal and spatial distance from the research processes and their results (Schäfer et al., 2021). They are:
- first-order effects: direct effects within the duration and the spatial scope of a research project, such as learning and capacity building, formation of networks, improving the situation in the respective field of action, and increase in reputation
- second-order effects: effects beyond the project but within the close temporal or spatial context of the project, such as institutionalisation of transformative approaches, establishment of project-related products or infrastructure, transfer of project results to other spatial contexts
- third-order effects: changes beyond the temporal or spatial context of the project in the entire field of action or problem field, such as influence on public discourse, influence on law and regulation, further structural effects.
Theory of change method
The Theory of Change method is useful for establishing shared hypotheses for a change process by planning backwards starting from a long-term shared project vision and identifying the conditions that need to be in place for the intended effects to occur (Belcher et al., 2019). Applying the theory of change method results in a flow diagram containing sets of activities, outputs and effects organized by non-linear impact-pathways.
A systematic evaluation framework
To combine the three components described above, we, as project evaluators, usually start with a workshop to define the theory of change, including as many project members (scientists as well as practitioners) as possible. From the flowchart of impacts, participants select particularly relevant impact pathways, with the starting point being a research activity.
As the formative evaluation team, we suggest indicators and monitoring questions for the intended impacts on first-, second- and third- order effects. The project team assesses the suggested indicators for suitability, feasibility and manageability.
After the joint adaptation of the set of indicators and their operationalisation via monitoring questions, data collection and ongoing reflection about the process can be started. At this stage particular attention is paid to the five impactful research practices.
Our experience so far has shown that this systematic approach to formative evaluation, which defines indicators beyond the boundaries of the project duration and project context, supports projects in their impact-oriented research activities.
What do you think? Have you developed other ways for improving formative evaluation of transdisciplinary research? Would the approach we have developed be useful in your research too?
References:
Belcher, B. M., Claus, R., Davel, R. and Ramirez, L. F. (2019). Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects. Environmental Science and Policy, 101: 192-203. (Online) (DOI): https://doi.org/10.1016/j.envsci.2019.08.013
Lux, A., Schäfer, M., Bergmann, M., Jahn, T., Marg, O., Nagy, E., Ransiek, A. and Theiler, L. (2019). Societal effects of transdisciplinary sustainability research – How can they be strengthened during the research process? Environmental Science and Policy, 101: 183-191. (Online). (DOI): https://doi.org/10.1016/j.envsci.2019.08.012
Schäfer, M., Bergmann, M. and Theiler, L. (2021). Systematizing societal effects of transdisciplinary research. Research Evaluation, 30, 4: 484–499. (Online) (DOI): https://doi.org/10.1093/reseval/rvab019
Biography: Emilia Nagy researches on transdisciplinarity at the Center for Technology and Society (ZTG) at Technische Universität Berlin in Germany. Her focus is on increasing the societal effects of transdisciplinary sustainability research.
Biography: Martina Schäfer PhD is the Scientific Director of the Center for Technology and Society (ZTG) of Technische Universität Berlin in Germany. She has coordinated inter- and transdisciplinary research projects in sustainable regional development, sustainable consumption and sustainable land use. One of her research foci is reflection on methods for inter- and transdisciplinary cooperation as well as the societal impact of this research mode.
Hi Emilia and Martina,
great to read about your work on using ToCs for impact evaluation purposes while at Eawag so far we have used it more for joint planning and visioning with ITD teams. I would be curious to learn more about how you exactly derive the indicators for the milestones and interventions and how you combine quantitative and qualitative indicators in a meaningful and complementary way? In addition, how long does it take from your experience to gather all the necessary information for assessing the impact in the end (e.g. for a project vs. a program)?
Thanks in advance for your insights!
Lisa
Dear Lisa,
Thank you very much for the questions! We have been following the project at EAWAG with great attention and are very much looking forward to exchanging ideas with you.
Your questions deal with our major challenges. While it is relatively easy to formulate indicators for the quality of the research process and results (outputs), it is much more challenging for the impacts. For the research process, we usually refer to the quality criteria of TD research for the definition of the indicators, the outputs can be described by subject-specific criteria.
For the definition of the impact criteria, we first collect intended impacts in three impact orders (according to temporal and spatial distance from the project context) in a ToC workshop. In a second step, we prioritise the impact pathways that emerge from the project activities and lead to impacts of higher orders. Pragmatic considerations also play a role in the prioritisation. High priority has often been given to pathways that can be influenced by the project and where context conditions are favourable.
In the next step, the projects define indicators along the prioritised impact pathways. For the definition of indicators, the systematic differentiation of frequently used impact categories (such as ‘learning effects’, ‘capacity building’, ‘network effects’, etc.) in subcategories is very helpful (Schäfer, Bergmann and Theiler, 2021). For example, network effects can be documented by collecting data in the subcategories “new strategic contacts” and “overcoming reservations and building trusting relationships”. Counting new strategic contacts provides relatively simple quantitative data. However, the number of new contacts says little about the quality of the new strategic relationships. This data can be usefully extended with the impact quality “building trusting relationships”, which in our experience is important for most implementation oriented projects.
Schäfer, Bergmann and Theiler (2021) describe 19 subcategories for societal effects (first order: 9 / second order: 6/ third order: 4). We usually aim at defining indicators at least for all subcategories of the first order in order to capture the variety of effects a project can achieve already during the ongoing research process. Indicators such as ‘willingness to continue activities after the end of the research project’ or ‘efforts of transferring knowledge and experiences to further contexts’ indicate 2nd order effects.
After agreeing on a set of indicators, we support the research teams in setting up a plan how to collect the necessary data. Since the projects often have limited resources for the evaluation, pragmatic considerations as easy access to data often play a critical role for the choice of indicators.
In our case, the period of data collection is limited to three years due to the duration of the project funding. It would make sense to conduct another survey about 2 years after the end of the project to be able to detect 2nd order effects in more detail.
We would be very happy to exchange ideas with you on the definition of the indicators.
Best wishes,
Emilia and Martina
Dear Dena,
thank you so much for your comment! We are very pleased that you find our approach inspiring.
We usually run at least three workshops for each project: One at the beginning, one in the middle and one at the end of the project. The exact timing of the workshops also depends on the current needs of the projects.
We had the opportunity to apply the described evaluation concept in a consortium that initiates projects to trigger socio-technical innovation processes in a specific region. We support 4-5 projects from the initial idea till finalization of the project. The projects have been embedded in a common innovation strategy and are continuously accompanied by the evaluation team. We are aware that this is a particularly privileged situation for transdisciplinary research.
In this consortium, we conducted the first ToC workshops already in the application phase or in the early project phase. During the lifetime of the projects, we held regular reflection meetings with the projects and invited them to workshops. These reflective workshops served to collect data and to review the theory of change in the projects. At the end of the projects, we are going to hold a ToC workshop with each project. In this workshop, we are going to reflect together on which impacts have already been achieved and which can still be expected, as well as on the enabling and hindering conditions of effectiveness.
We will be happy to explain other aspects if you have more questions.
Best wishes,
Emilia and Martina
Thank-you Emilia and Martina:
This is inspiring work. I wondered whether you have any thoughts on the “readiness” requirements, especially around the Impactful Research Practices. In our experience, organizational jealousies and budget control mind-sets can get in the way of a horizontal decision-making. Have you encountered such challenges?
With regards to the Impact Heuristics, the gradient of effects reminds me of the continuum in Outcome Mapping, especially the notion of loss of attribution. Acknowledging that impact can only barely be connected to activities remains a tough sell with many funders. I wonder if / how you have faced this challenge and accommodated it.
In our work we mentor project teams to design their own evaluations of research (in tandem with communication planning). We are increasing leaning on Developmental Evaluation, where experimental work means that it is challenging to develop a good Theory of Change early on. Perhaps this sounds familiar?
I look forward to exploring your references.
Kind regards, Ricardo
Dear Ricardo,
Thank you very much for the stimulating questions! We have also encountered similar problems in our work. Our project framework does not allow us to address each of these issues with the same intensity. Still, we are happy to describe our thoughts on the three points.
Our projects were preceded by participatory planning. In this phase, it was commonly agreed by the stakeholders that cooperative partnership among the project partner was a key component. All project members joined the project with a certain commitment to this preconception. Another special feature of our projects is that they are building up new value chains. If these can be maintained, all partners will benefit in the longer term. Nevertheless, this conceptual foundation of the projects cannot prevent partners from going their own way when their (often economic) interests “tempt” them to do so. So yes, some partners are able to jeopardize a horizontal or cooperative decision-making.
So far, we have little experience with funders’ reaction to these outcomes. However, we think it is important to make them aware of the qualitative impacts that can be achieved in the course of the project (e.g. better mutual understanding between science and practice or between different groups of practitioners, capacity building and new networks). As far as we have experienced it, these qualitative impacts may pave the way for more far-reaching impacts, such as changes in organizational or individual practices, the introduction of new regulations, and so on. And yes, there is some work to be done to clarify that there are obstacles to clearly attribute impacts to project activities.
In our experience, projects also react with reluctance when it comes to defining impact indicators. They cannot and do not want to commit to providing credible impact data. The priority goal of projects is to safely complete their processes and deliver their products. Our evaluation starts at this point: We raise awareness among projects of their responsibility for their contribution to the changes on a bigger scale a project can bring about.
We have experienced that project teams find it very helpful to reflect together on intended impacts in the beginning of the project. It was relatively easy for the projects to define immediate and visionlike impacts. The participants had the most difficulties in describing complete impact pathways between these distant impact orders. Even if the impact pathways could not have been clearly defined, the projects benefited from the workshop: The different perspectives on project goals and project impacts became visible. On this basis, the participants were able to develop a common understanding of objectives and impacts.
I hope that our answers have given you an insight into our experiences.
Best wishes,
Emilia and Martina
Thank you Emilia and Martina for your interesting post!
what a very thorough way of thinking about TD project evaluation. I was wondering how you decide when these formative evaluation workshops take place throughout the developmental stages of the TD project?
Thanks so much for your very insightful post
Dena
Thank you for this insightful work. May I ask why “planning backwards ” and is the practice in terms of formulating theories of change in transdisplinary research? What is the rationale?