By Gabriele Bammer
How do you write-up the methods section for research synthesizing knowledge from different disciplines and stakeholders to improve understanding about a complex societal or environmental problem?
In research on complex real-world problems, the methods section is often incomplete. An agreed protocol is needed to ensure systematic recording of what was undertaken. Here I use a checklist to provide a first pass at developing such a protocol specifically addressing how knowledge from a range of disciplines and stakeholders is brought together.
KNOWLEDGE SYNTHESIS CHECKLIST
1. What did the synthesis of disciplinary and stakeholder knowledge aim to achieve, which knowledge was included and how were decisions made?
What was the purpose of the knowledge synthesis?- Which disciplines and stakeholders were involved and what knowledge did they contribute?
- Who was responsible for making decisions about the knowledge synthesis?
2. What were the salient characteristics of the problem, what knowledge was relevant and how was the collaboration structured?
Problem definition
- Was a systems approach used for defining the problem? If so, was it explicit or implicit?
- If explicit
Which aspects of the system were considered and how was the approach taken decided?
Was a formal problem structuring method used? If so, how was it chosen? - If implicit
Can you describe it in retrospect?
Was there a guiding mental model? How did this come to be the one used?
Problem framing
- How did you describe the problem? How was that description arrived at?
- Were any metaphors used in the problem description?
- Did different groups involved in investigating the problem frame it differently? If so, why and how?
Scoping and boundary setting around disciplinary and stakeholder knowledge
- How did you think about which disciplines and stakeholders could be relevant?
- How did you decide which disciplines and stakeholders to include?
- For the disciplines included, how did you decide which knowledge was relevant and why?
- For the stakeholders included, how did you decide which perspectives were relevant and why?
- How were the experts who contributed the disciplinary and stakeholder knowledge chosen?
Values
- How did you think about values that influence the way the problem is understood and acted on?
- Were value conflicts evident in the project? If so, how did you deal with them?
Collaboration
- What productive differences did those involved bring to the project?
- How were these identified and ‘harnessed’?
- Were there differences that got in the way of working together?
- How were they identified and managed?
- How were project participants rewarded for their contributions?
3. How was the disciplinary and stakeholder knowledge synthesised, by whom and when?
Dialogue
- Was dialogue part of the knowledge synthesis process? If so, were the methods used formal or informal?
- If formal:
- Which methods were chosen and why?
- If informal:
- How was the dialogue managed? Was this based on any principles or guidelines?
Modelling
- Was modelling part of the knowledge synthesis process? If so, which methods were chosen and why?
Common metric (eg., dollar value, ecosystem service, global footprint)
- Was a common metric used as part of the knowledge synthesis process? Which common metric was chosen and why? How was the common metric operationalized?
Other
- Were other methods used to achieve knowledge synthesis? If so, which methods were chosen and why?
Multiple methods
- Was more than one method used? If so, how did they interact?
Who and when
- Who was involved in the knowledge synthesis process? What was/were their position/s on the team?
- At what particular time points in the life cycle of the project was knowledge synthesis undertaken? How and why were these time points chosen?
4. How was context taken into account?
Big picture context
- How did you identify and take into account relevant aspects of big-picture context – such as the history of the problem, cultural factors, geographical location or political circumstances?
Authorisation
- What gave the knowledge synthesis legitimacy?
Institutional facilitators and barriers
- For the organisations involved, which elements of their structure and culture supported the knowledge synthesis?
- For the organisations involved, which elements of structure and culture were barriers to knowledge synthesis?
- How were institutional facilitators and barriers identified? How were the facilitators exploited? How were the barriers overcome?
5. How well did the knowledge synthesis work?
- Overall, how well did the knowledge synthesis work?
- Was sufficient flexibility and iteration built into the process?
- What are the criteria for making these assessments?
For each aspect of the knowledge synthesis covered in the questions above (eg., problem framing or authorization):
- Was it addressed adequately, given the constraints of the project?
- Were any notable trade-offs made? Were they worthwhile?
- Did something that was excluded or missed turn out to be critical?
- Were there any unintended beneficial or adverse consequences?
- What are the criteria for making these assessments?
For each method, concept or process used in the knowledge synthesis:
- Was the method, concept or process appropriate?
- Would some other method, concept or process have been better?
- What are the criteria for making these assessments?
CONTEXT AND CONCLUSIONS
This checklist tackles the first of three primary domains for documenting the methods used in research on complex societal and environmental problems:
- synthesis of disciplinary and stakeholder knowledge
- understanding and managing unknowns
- providing integrated research support (bringing together both what is known and unknowns) for policy and practice change.
These are described in more detail in Bammer (2013).
The rationale for developing an agreed protocol for writing the methods section in papers abut tackling complex real world problems is described in an earlier blog post: The ‘methods section’ in research publications on complex problems – Purpose.
Does this protocol work for you? Do any questions need further explanation? What adaptations would you advise?
Acknowledgement:
Melissa Robson-Williams and Bruce Small made useful suggestions on an earlier draft of the checklist.
Reference:
Bammer, G. (2013). Disciplining Interdisciplinarity: Integration and Implementation Sciences for Researching Complex Real-World Problems. ANU Press: Canberra, Australia. Online: http://press.anu.edu.au?p=222171
Biography: Gabriele Bammer PhD is a professor at The Australian National University in the Research School of Population Health’s National Centre for Epidemiology and Population Health. She is developing the new discipline of Integration and Implementation Sciences (i2S) to improve research strengths for tackling complex real-world problems through synthesis of disciplinary and stakeholder knowledge, understanding and managing diverse unknowns and providing integrated research support for policy and practice change.
Gabriele–as usual you have posted a most useful article. It is one that could be applied to foresight studies which demand knowledge synthesis. Thank you for your work. Jim
Great list – You might add something about how the resulting/synthesized knowledge should be more systemic than the starting knowledge. Too often, especially in the academic world, there is “cherry picking” of concepts so that the resulting knowledge is little (if any better) than the starting knowledge. In contrast, I believe we can accelerate the advancement of knowledge to better understand and resolve the wicked complex problems by more effectively synthesizing our knowledge. https://www.researchgate.net/publication/271525902_Existing_and_emerging_methods_for_integrating_theories_within_and_between_disciplines
Good point – thanks!
Thank you for this information. Such as great resource for researchers….
Thank-you Gabriele for this well-structured, comprehensive (if exhausting) checklist. Some comments from dealing with practical issues and trying to find and implement resolutions:
1.
The decision making is necessarily iterative – when the conclusions/recommendations are reached collaborators/stakeholders and even the diverse “virtue signallers” who emerge, will respond, in general prompting further review and adjustment.
2.
Explicit statements of scope and boundaries is crucial (stakeholders’ objectives, interests, capabilities etc, resources that may be deployed and timelines). Further iteration may respond to the need for adjustment of scopes.
3.
Explicit statement is crucial of how the value of “difficult to compare” objectives (and their costs and benefits of achieving them) is required – again, further iteration may respond to the need for adjustment of comparative value parameters.
Thanks. Absolutely agree with all your comments, and especially on iteration. Part of the point is to help funders, research leaders, researchers and stakeholders understand that to do knowledge synthesis well requires a substantial commitment of time and other resources.
It is a very useful checklist for researchers. Thanks.