By Gabriele Bammer
How do different perceptions arise of what makes for ‘good’ research? How can researchers come to understand such differences and their impacts on how problems are framed, understood and responded to, as well as how they affect the ability of those contributing to the research to work together?
Differences arise because training in a discipline involves inculcating a specific way of investigating the world, including which types of questions are worth addressing; legitimate ways of gathering, analysing and interpreting data; standards for validation; and the role of values in the research process. Educating someone in a discipline aims to make the discipline’s specific approach to research ingrained and tacit.
While this makes for relatively smooth operation of discipline-based research, it is the source of hidden stumbling blocks when investigators from multiple disciplines seek to work together, leading to what Hubbs and colleagues (2021) have called the “problem of unacknowledged differences.” In research groups comprising members from different disciplines, these unacknowledged differences can come into play when deciding how the problem will be addressed, as well as in how research group members perceive the value of what colleagues from other disciplines have to offer.
The Toolbox Dialogue Initiative has drawn on the philosophy of science to provide a practical way for research groups to develop an understanding of the different perceptions of their members. This involves group members filling out a self-completion survey, where they rate ― from strongly agree to strongly disagree ― five or six (generally) statements about research. The statements serve as prompts that disclose key features of each researcher’s assumptions about what makes research good. The responses are not scored, but instead are used to stimulate dialogue at a facilitated workshop that brings these differences into the light.
In order to understand differences in perceptions of good research, the survey focuses on six areas:
- motivation
- methodology
- confirmation
- objectivity
- values
- reductionism–emergence.
Motivation
Differences in motivation concern the perceived relative importance of basic and applied research.
Examples of prompt statements:
- Knowledge generated by research is valuable even if it has no application
- The principal value of research stems from the potential application of the knowledge gained.
Methodology
Differences in methodology concern research goals, such as whether research should be hypothesis driven and whether the aim is to describe or predict. Differences also include approaches to data acquisition, for example, whether it is experimental or observational, and whether quantitative or qualitative data are seen as more useful.
Examples of prompt statements:
- Scientific research (applied or basic) must be hypothesis driven
- In my disciplinary research, I employ primarily qualitative methods.
Confirmation
Differences relevant to confirmation concern what processes are required for the research results to add to the storehouse of knowledge. Different disciplines put different emphases on, for example:
- measurement validity ie., determining whether what was measured was what was intended
- replication ie., the original team and other researchers being able to repeat the findings
- triangulation ie., getting the same results using unrelated methods
- generalisability ie., finding the same results in other settings and/or with other groups of research subjects.
Examples of prompt statements:
- There are strict requirements for determining when empirical data confirm a tested hypothesis
- Unreplicated results can be validated if confirmed by a combination of several different methods.
Objectivity
Differences in objectivity concern “whether a fully objective world exists independent of any perspective, or whether the world is to some extent constructed by those who investigate it” (Eigenbrode et al., p. 59).
Examples of prompt statements:
- Scientific research aims to identify facts about a world independent of the investigators
- The subject of my research is a human construction.
Values
Differences here concern whether research can be value-neutral or whether research is “infused with values” (Eigenbrode at al., p. 59). Proponents of the second view argue that values can affect the topics chosen, how results are implemented, whose perspectives are taken into account in the research and much more.
Examples of prompt statements:
- Determining what constitutes acceptable validation of research data is a value issue
- Allowing values to influence scientific research is advocacy.
Reductionism–emergence
Differences here concern whether the world can be fully understood by reducing it to its component parts, which are then studied (reductionism) or whether a systems view is required that focuses on context, interconnections among components and emergent properties (ie., new properties or behaviours displayed by an entity that its parts do not have).
Examples of prompt statements:
- The world under investigation is fully explicable as the assembly of its constituent parts
- The world under investigation must be explained in terms of the emergent properties arising from the interactions of its individual components.
Differences in perceptions of good research and how problems are framed, understood and responded to
How problems are framed is affected by different perceptions of what makes for good research; consider, for example, a reductionist approach that examines a component of the problem and focuses on finding a cause, versus an approach that looks for interconnections and emergence. Similarly, whether research is seen as legitimate will influence its contribution to understanding the problem and valid responses to the problem. For example, if qualitative research is seen as flawed, its findings may be rejected, and if potential responses are thought to result from research that is biased by values, they may be ignored.
Differences in perceptions of good research and how well those contributing to the research work together
One determinant of how well those contributing work together is whether they can come to an understanding and agreement about their different perceptions of what makes for good research. This is influenced both by the uptake of their own contributions, as well as what they think of the contributions of others.
The ability to work together will be enhanced when everyone feels their own contributions are understood, valued and used. This may involve agreeing to disagree and to incorporate different perspectives of what is considered to be good research.
Concluding notes
The topics and prompts discussed here are taken from the Scientific Research Tool Instrument which was developed early in the life of the Toolbox Dialogue Initiative and which has been used in around one-third of the dialogues organised and facilitated under the auspices of the Initiative. Nowadays the Toolbox Dialogue Initiative works with teams to tailor the topics and prompts to their needs, which can go beyond what makes research good. The process also goes beyond understanding differences to finding ways to accommodate them.
Although the focus here is on researchers, stakeholders can also be involved as they also often have specific views on what constitutes good research.
Anything to add?
Do you have additional perspectives to share about perceptions of what makes for good research, as well as how you learnt about different perceptions? What was the impact of these perceptions on approaching problems, as well as on the ability to work together?
Examples and lessons are particularly welcome. Is there anything you wish you had known when you were starting out?
If you are new to this topic, is there anything else on understanding different perspectives about good research that would be useful?
Sources and references:
The main sources are from the Toolbox Dialogue Initiative (http://tdi.msu.edu), and especially the references listed below, which also cite the work of others who influenced this approach.
Eigenbrode, S. D., O’Rourke, M., Wulfhorst, J. D., Althoff, D. M., Goldberg, C. S., Merrill, K., Morse, W., Nielsen-Pincus, M., Stephens, J., Winowiecki, L. and Bosque-Pérez, N. A. (2007). Employing Philosophical Dialogue in Collaborative Science. BioScience, 57, 1: 55-64. (Online – Open access): https://academic.oup.com/bioscience/article/57/1/55/224519
Hubbs, G., O’Rourke, M. and Orzack, S. H. (Eds.). (2020). The Toolbox Dialogue Initiative: The Power of Cross-Disciplinary Practice. CRC Press: Florida, United States of America. (Online – book description): https://www.taylorfrancis.com/books/toolbox-dialogue-initiative-graham-hubbs-michael-rourke-steven-hecht-orzack/e/10.1201/9780429440014
Hubbs, G., O’Rourke, M. and Orzack, S. H. (2021). Responding to Unacknowledged Disciplinary Differences with the Toolbox Dialogue Method. Integration and Implementation Insights, March. (Online): https://i2insights.org/2021/03/02/toolbox-dialogue-method/
Biography: Gabriele Bammer PhD is Professor of Integration and Implementation Sciences (i2S) at the National Centre for Epidemiology and Population Health at The Australian National University in Canberra. i2S provides theory and methods for tackling complex societal and environmental problems, especially for synthesis of disciplinary and stakeholder knowledge, understanding and managing diverse unknowns, and providing integrated research support for policy and practice change. She is also a member of blog partner PopulationHealthXchange.
The Understanding Diversity Primer comprises the following blog posts:
Published:
1. Why diversity? (April 21, 2022)
2. Mental models (April 28, 2022)
This blog post:
3. Perceptions of good research (May 5, 2022)
Still to come:
4. Power (May 12, 2022)
5. Values (May 19, 2022)
6. Interests (May 26, 2022)
7. Culture (June 2, 2022)
8. Personality (June 9, 2022)
9. Team roles (June 16, 2022)
10. Advanced considerations (June 23, 2022)
Thanks Gabriele, I wanted to make a couple more points about dealing with differences in perceptions. Over my career I’ve been fortunate to have worked with collaborators from a variety of disciplines with methodological expertise and orientations both similar and different to mine. Like myself, they happen to be personalities more open to diversity and innovation, knowing that different approaches – justified by credible sources – can be valuable and trustworthy, and also they have built their successes on this principle. In some cases where people can perceive differences negatively (eg. interpretivist being less valid than positivist), I think this can happen when people are not familiar with or convinced of the benefits of a particular approach, or they have personally had a negative experience (being rejected) from using that approach.
I think people’s positive or negative experiences play a big part in what shapes people’s perceptions of what makes good research, as are some of the factors that have helped us collaborate such as:
1) sharing a broader systems thinking approach, regardless of different orientations;
2) making a habit of monitoring evolving trends in research approaches discussed from reputable sources (eg. methodology journals, discussion from thought leaders in particular approaches) and using this knowledge as an evidence-base to justify research design choices, not just to support the quality of research output but in our everyday team communications;
3) demonstrating personal integrity and encouraging mutual respect, that we have all hard earned reputations in a particular area/expertise.
Terrific – thanks.
Thanks Gabriele. My research over the years has uncovered the increased need for transdisciplinary researchers and participants to unlearn methodologies and approaches, sometimes deeply ingrained from their disciplinary training and home cultures. Although the unlearning process can be difficult in many ways, this appears to be a prerequisite to enabling contributions to ‘good research’ – which they define as the team’s co-designed approach best suited to the challenge of addressing the transdisciplinary project’s aims and outcomes.
In some cases, this involves disruptive research methods emerging from academics’ interactions with industry and policy. For example, mixed method user experience research with the tech industry or program evaluation and implementation studies from policy analysis.
In a TD (transdisciplinary) context, ‘good research’ is not necessarily that which is methodologically or epistemologically sound (as suggested in this post, perceptions of these differ across disciplines and contexts). Here it may be less about knowing the methods and focusing more on how best to adapt and merge methods or approaches suited to the specific project and ultimately, to empower the problem solving capacity of the team addressing the problems.
Thanks Faye. That’s a useful addition to the points raised in the blog post. There’s some interesting thinking (and gathering of experience) to do about whether complex societal and environmental problems are most effectively addressed by using different disciplinary approaches side-by-side or whether there needs to be merging of approaches or some of both. (My experience is ‘some of both’.)
The point of the blog post is not actually to address that question, but rather to point to an explanation of why people’s perceptions of good research differ and why some unlearning may be necessary when people from different disciplines seek to work together. Your comment then moves the discussion forward to how those differences – once identified – are accommodated and/or managed.
In addition to hearing more about your experience, it would be most helpful to hear from the Toolbox Dialogue Initiative folks and others how they’ve dealt with the differences in perceptions of what makes research good..