A checklist for documenting knowledge synthesis

Community member post by Gabriele Bammer

Gabriele Bammer (biography)

How do you write-up the methods section for research synthesizing knowledge from different disciplines and stakeholders to improve understanding about a complex societal or environmental problem?

In research on complex real-world problems, the methods section is often incomplete. An agreed protocol is needed to ensure systematic recording of what was undertaken. Here I use a checklist to provide a first pass at developing such a protocol specifically addressing how knowledge from a range of disciplines and stakeholders is brought together.

KNOWLEDGE SYNTHESIS CHECKLIST

1. What did the synthesis of disciplinary and stakeholder knowledge aim to achieve, which knowledge was included and how were decisions made?

  • tick-box-checkedWhat was the purpose of the knowledge synthesis?
  • tick-box-checkedWhich disciplines and stakeholders were involved and what knowledge did they contribute?
  • tick-box-checkedWho was responsible for making decisions about the knowledge synthesis?

2. What were the salient characteristics of the problem, what knowledge was relevant and how was the collaboration structured?

Problem definition

  • tick-box-checkedWas a systems approach used for defining the problem? If so, was it explicit or implicit?
  • If explicit
    tick-box-checkedWhich aspects of the system were considered and how was the approach taken decided?
    tick-box-checkedWas a formal problem structuring method used? If so, how was it chosen?
  • If implicit
    tick-box-checkedCan you describe it in retrospect?
    tick-box-checkedWas there a guiding mental model? How did this come to be the one used?

Problem framing

  • tick-box-checkedHow did you describe the problem? How was that description arrived at?
  • tick-box-checkedWere any metaphors used in the problem description?
  • tick-box-checkedDid different groups involved in investigating the problem frame it differently? If so, why and how?

Scoping and boundary setting around disciplinary and stakeholder knowledge

  • tick-box-checkedHow did you think about which disciplines and stakeholders could be relevant?
  • tick-box-checkedHow did you decide which disciplines and stakeholders to include?
  • tick-box-checkedFor the disciplines included, how did you decide which knowledge was relevant and why?
  • tick-box-checkedFor the stakeholders included, how did you decide which perspectives were relevant and why?
  • tick-box-checkedHow were the experts who contributed the disciplinary and stakeholder knowledge chosen?

Values

  • tick-box-checkedHow did you think about values that influence the way the problem is understood and acted on?
  • tick-box-checkedWere value conflicts evident in the project? If so, how did you deal with them?

Collaboration

  • tick-box-checkedWhat productive differences did those involved bring to the project?
  • tick-box-checkedHow were these identified and ‘harnessed’?
  • tick-box-checkedWere there differences that got in the way of working together?
  • tick-box-checkedHow were they identified and managed?
  • tick-box-checkedHow were project participants rewarded for their contributions?

3. How was the disciplinary and stakeholder knowledge synthesised, by whom and when?

Dialogue

  • tick-box-checkedWas dialogue part of the knowledge synthesis process? If so, were the methods used formal or informal?
  • If formal:
  • tick-box-checkedWhich methods were chosen and why?
  • If informal:
  • tick-box-checkedHow was the dialogue managed? Was this based on any principles or guidelines?

Modelling

  • tick-box-checkedWas modelling part of the knowledge synthesis process? If so, which methods were chosen and why?

Common metric (eg., dollar value, ecosystem service, global footprint)

  • tick-box-checkedWas a common metric used as part of the knowledge synthesis process? Which common metric was chosen and why? How was the common metric operationalized?

Other

  • tick-box-checkedWere other methods used to achieve knowledge synthesis? If so, which methods were chosen and why?

Multiple methods

  • tick-box-checkedWas more than one method used? If so, how did they interact?

Who and when

  • tick-box-checkedWho was involved in the knowledge synthesis process? What was/were their position/s on the team?
  • tick-box-checkedAt what particular time points in the life cycle of the project was knowledge synthesis undertaken? How and why were these time points chosen?

4. How was context taken into account?

Big picture context

  • tick-box-checkedHow did you identify and take into account relevant aspects of big-picture context – such as the history of the problem, cultural factors, geographical location or political circumstances?

Authorisation

  • tick-box-checkedWhat gave the knowledge synthesis legitimacy?

Institutional facilitators and barriers

  • tick-box-checkedFor the organisations involved, which elements of their structure and culture supported the knowledge synthesis?
  • tick-box-checkedFor the organisations involved, which elements of structure and culture were barriers to knowledge synthesis?
  • tick-box-checkedHow were institutional facilitators and barriers identified? How were the facilitators exploited? How were the barriers overcome?

5. How well did the knowledge synthesis work?

  • tick-box-checkedOverall, how well did the knowledge synthesis work?
  • tick-box-checkedWas sufficient flexibility and iteration built into the process?
  • tick-box-checkedWhat are the criteria for making these assessments?

For each aspect of the knowledge synthesis covered in the questions above (eg., problem framing or authorization):

  • tick-box-checkedWas it addressed adequately, given the constraints of the project?
  • tick-box-checkedWere any notable trade-offs made? Were they worthwhile?
  • tick-box-checkedDid something that was excluded or missed turn out to be critical?
  • tick-box-checkedWere there any unintended beneficial or adverse consequences?
  • tick-box-checkedWhat are the criteria for making these assessments?

For each method, concept or process used in the knowledge synthesis:

  • tick-box-checkedWas the method, concept or process appropriate?
  • tick-box-checkedWould some other method, concept or process have been better?
  • tick-box-checkedWhat are the criteria for making these assessments?

CONTEXT AND CONCLUSIONS

This checklist tackles the first of three primary domains for documenting the methods used in research on complex societal and environmental problems:

  1. synthesis of disciplinary and stakeholder knowledge
  2. understanding and managing unknowns
  3. providing integrated research support (bringing together both what is known and unknowns) for policy and practice change.

These are described in more detail in Bammer (2013).

The rationale for developing an agreed protocol for writing the methods section in papers abut tackling complex real world problems is described in an earlier blog post: The ‘methods section’ in research publications on complex problems – Purpose.

Does this protocol work for you? Do any questions need further explanation? What adaptations would you advise?

Acknowledgement:
Melissa Robson-Williams and Bruce Small made useful suggestions on an earlier draft of the checklist.

Reference:
Bammer, G. (2013). Disciplining Interdisciplinarity: Integration and Implementation Sciences for Researching Complex Real-World Problems. ANU Press: Canberra, Australia. Online: http://press.anu.edu.au?p=222171

Biography: Gabriele Bammer PhD is a professor at The Australian National University in the Research School of Population Health’s National Centre for Epidemiology and Population Health. She is developing the new discipline of Integration and Implementation Sciences (i2S) to improve research strengths for tackling complex real-world problems through synthesis of disciplinary and stakeholder knowledge, understanding and managing diverse unknowns and providing integrated research support for policy and practice change. 

5 thoughts on “A checklist for documenting knowledge synthesis

  1. Thank-you Gabriele for this well-structured, comprehensive (if exhausting) checklist. Some comments from dealing with practical issues and trying to find and implement resolutions:

    1.
    The decision making is necessarily iterative – when the conclusions/recommendations are reached collaborators/stakeholders and even the diverse “virtue signallers” who emerge, will respond, in general prompting further review and adjustment.

    2.
    Explicit statements of scope and boundaries is crucial (stakeholders’ objectives, interests, capabilities etc, resources that may be deployed and timelines). Further iteration may respond to the need for adjustment of scopes.

    3.
    Explicit statement is crucial of how the value of “difficult to compare” objectives (and their costs and benefits of achieving them) is required – again, further iteration may respond to the need for adjustment of comparative value parameters.

    • Thanks. Absolutely agree with all your comments, and especially on iteration. Part of the point is to help funders, research leaders, researchers and stakeholders understand that to do knowledge synthesis well requires a substantial commitment of time and other resources.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.