By Brian Belcher, Rachel Claus, Rachel Davel, Stephanie Jones and Daniela Pinto
What are the characteristics of high-quality transdisciplinary research? As research approaches increasingly cross disciplinary bounds and engage stakeholders in the research process to more effectively address complex problems, traditional academic research assessment criteria are insufficient and may even constrain transdisciplinary research development and use. There is a need for appropriate principles and criteria to guide transdisciplinary research practice and evaluation.
In response to this need, Belcher et al. (2016) developed the Transdisciplinary Research Quality Assessment Framework based on a systematic review of literature that discussed the definition and measurement of research quality for inter- and trans- disciplinary research. Through applied testing in case study evaluations, we have revised and refined the framework to improve its utility.
Principles and Criteria
The tool provides guiding principles and corresponding criteria for transdisciplinary research design and implementation. Four principles comprise the Quality Assessment Framework:
- Relevance, which refers to the appropriateness of the problem positioning, objectives, and research approach for intended users;
- Credibility, which pertains to rigour of the design and research process to produce dependable and defensible conclusions;
- Legitimacy, which refers to the perceived fairness and representativeness of the research process; and
- Positioning for use, which assesses the degree to which research is likely to be taken up and used.
The criteria provide a checklist for planning and a scoring system for evaluation (see the figure below for the list of criteria).
Using the Tool
The Quality Assessment Framework was designed to be applied at any stage of a research intervention (eg., project, program). For planning, the Quality Assessment Framework can help support comprehensive thinking on how to design and implement a research project to (co)produce knowledge that will be useful and used, and that will contribute to a process of change. The criteria and associated guidance:
- encourage a holistic and in-depth understanding of the problem context;
- facilitate the integration of diverse perspectives and opportunities for a genuine collaborative research process, while ensuring transparency and ethical conduct;
- identify opportunities to build knowledge, skills, and relationships, and to influence the attitudes and behaviours of stakeholders of the project or program; and
- tailor outputs for uptake and use in the respective research/program context and beyond.
For monitoring and adaptive management, the criteria guide reflection on progress and help identify gaps and opportunities to make adjustments. For evaluation, the framework provides a comprehensive assessment of a research project or program with respect to its purpose. The criteria help guide research projects aiming for impact, highlighting critical elements of successful research design and implementation. This can help evaluators identify strengths and weaknesses in research design and implementation for future learning. It also provides a systematic way to compare projects.
The Quality Assessment Framework was designed for multiple users, including research funders and research managers assessing proposals; researchers designing and planning a research project; and, research evaluators like ourselves who apply the tool to learn lessons about effective research practice.
In evaluation, each criterion should be assessed considering the purpose of the project. Document review, interviews, and survey data can serve as evidence to inform the scoring process. Criteria are scored using a three-point Likert scale:
- 2 = criterion is fully satisfied
- 1 = criterion is partially satisfied
- 0 = criterion is not at all satisfied
The scores of one or multiple projects can be mapped in a diagram, as shown in the figure below for three projects (A, B, and C). Please note that in this figure, the original principles and criteria were used – these have since been revised, as shown in the figure above.
Contributions Welcome!
We encourage broader testing and application of the Quality Assessment Framework in various transdisciplinary contexts. We welcome suggestions from the transdisciplinary research community to further improve the framework to comprehensively capture the unique characteristics of transdisciplinary research. What other criteria do you think would be useful to integrate?
To find out more:
Sustainability Research Effectiveness (Online): https://researcheffectiveness.ca/
For early access to our two pre-crafted contributions for the 2021 International Transdisciplinarity Conference, see 1) the Integration and Implementation Sciences (i2S) YouTube channel at: https://youtu.be/YdY8UadvERc and 2) our Prezi at https://prezi.com/p/52ap5yagsezm/qaf/. The conference will be held online from September 13–17, 2021. (Online): https://en.transdisciplinarity.ch/tdnet_en/events/itd-21
Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R. and Zornes, D. A. (2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, 25, 1: 1-17. (Online – open access): https://doi.org/10.1093/reseval/rvv025
Belcher, B. M., Claus, R., Davel, R. and Jones, S. M. (2021). Evaluating and improving the contributions of university research to social innovation. Social Enterprise Journal. Online (open access): https://doi.org/10.1108/SEJ-10-2020-0099
Brian Belcher PhD is the Ashoka Chair in Research Effectiveness and professor in the College of Interdisciplinary Studies at Royal Roads University in Victoria, Canada. He leads the Sustainability Research Effectiveness Program, developing theory, methodology, and methods for evaluating research in complex transdisciplinary contexts. This work helps to demonstrate the societal value and impact of research and learns lessons to improve future research. He is also a Senior Associate Scientist with the Centre for International Forestry Research and the Consortium Research Program on Forests, Trees and Agroforestry.
Rachel Claus MSc is a research assistant with the Sustainability Research Effectiveness Program at Royal Roads University in Victoria, Canada. Her research expertise and interests are in research design for impact and theory-based research evaluation methods. She has five years of experience developing and applying theories of change to projects and programs to support monitoring and evaluation strategies to optimize effectiveness.
Rachel Davel MDev is a research assistant with the Sustainability Research Effectiveness Program at Royal Roads University in Victoria, Canada. With fours years of experience in theory-based research evaluation methods, her current work focuses on impact assessment and research effectiveness to understand how research and development projects contribute to societal change.
Stephanie Jones MSc is a consultant with the Center for International Forestry Research, a non-profit research organisation headquartered in Bogor, Indonesia, working on a program-level impact evaluation of the Forests, Trees and Agroforestry research program. She is based in Vancouver, Canada. She is also an ongoing collaborator with the Sustainability Research Effectiveness Program at Royal Roads University in Victoria, Canada.
Daniela Pinto MA is a research assistant with the Sustainability Research Effectiveness Program at Royal Roads University in Victoria, Canada. She has more than 15 years of experience monitoring multi-country, multi-stakeholder projects and programs focused on human rights, gender equality, and environmental protection.