By Gabriele Bammer
How can the effectiveness of stakeholder engagement be judged? How can the outcomes be assessed? How much effort should go into such evaluation?
How and when to evaluate
In any stakeholder engagement there is no shortage of aspects that could be evaluated. The challenge is aligning the audience for the evaluation, the key issues to be assessed and the available resources.
For example, if a research team is interested in learning from what went wrong in a stakeholder engagement (for instance, if a stakeholder stopped participating or became hostile) and has no money set aside for evaluation, it might rely on self-reflection and anecdotal evidence to figure out what happened.
On the other hand, if a funder wants to know whether the time and money invested in stakeholder engagement was worthwhile, an independent expert evaluation might be warranted, with money earmarked for that in the grant. Such an evaluation would involve multiple considerations, including what the funder would find convincing, what was measurable, what the researchers and stakeholders thought was important and what could realistically be undertaken with the available time and budget. Typically budgets for evaluation are tight and circumscribe what can be undertaken.
In aligning the audience for the evaluation with the key issues to be assessed and the available resources, researchers may benefit from asking the following questions and sub-questions.
- Who is the evaluation for?
Sub-questions include:
- Do we need to convince funders that the stakeholder engagement was worth the time and money expended?
- Do we want to improve our effectiveness in engaging stakeholders in future research?
- Do we need to demonstrate to the stakeholders that the time and effort they put into participating in the research was worthwhile?
- What do we want to know?
Sub-questions include:
- Are we more interested in assessing the process or outcomes; or are we interested in both?
- Which aspects of the process are of most interest: eg., defining, identifying or selecting stakeholders; stakeholder engagement processes?
- What outcomes are we most interested in: eg., effects on the research design, conduct, results, interpretation; stakeholder satisfaction?
- Do we want to focus on what went well and/or what went wrong in both process and outcomes?
- Do we want to assess the engagement while the research is in progress, so that we can make adjustments, if necessary, or wait until the research has been completed?
- What resources and skills are available to conduct the evaluation?
Sub-questions include:
- How much funding and time can we set aside for evaluation?
- Does anyone on the research team have expertise in evaluation?
- Do we have funding to hire an independent expert evaluator?
Periodic self-assessment
It is worth undertaking at least a brief review of the stakeholder engagement when planning the research, while it is underway and at the end of the project, using questions such as those below. This can be as simple as an occasional discussion among members of the research team, including stakeholders when appropriate.
Planning the research:
Useful questions for reviewing stakeholder engagement when planning a research project include:
- Why are we engaging stakeholders? What outcomes do we expect and when?
- What are the best processes for achieving those aims? If processes have already been chosen, do they align with those aims?
- What are our indicators of success and failure?
- Are there any data that we can easily collect to measure success? If so, do we need to (and can we) collect baseline data before the research and engagement start?
During the research:
Reviewing stakeholder engagement while the research is underway provides opportunities to assess how well it is working and whether changes are needed. It can also provide useful feedback to stakeholders about how their contributions are being used. Useful questions include:
- What is working well? What is not? Why?
- Are appropriate processes being used?
- Should additional or different stakeholders be engaged?
- Are enough money, time and other resources available for the engagement? Are the costs reasonable? If the answer to either question is no, what can we do about it?
At the end of the research:
It is useful to review both process and outcomes when a project ends, particularly to glean lessons for future research. It is also important to bear in mind that some outcomes may not be fully realized until some time after a project is completed, so that subsequent reviews may also be required. Useful questions at the end of the project include:
- Were the aims of the stakeholder engagement met? Were the expected outcomes achieved? If not, why not?
- Were there any unexpected outcomes or impacts?
- Was the engagement process successful from both the researcher and stakeholder perspectives?
- Were sufficient time, money and other resources allocated to the stakeholder engagement? In relation to what was achieved were the costs of engagement reasonable?
- Are there lessons for future engagement processes?
Anything to add?
What has your experience been with evaluating stakeholder engagement? Do you have other tips for deciding when and how to evaluate? Do you have lessons to share about how to do periodic self-assessment effectively? How did you evaluate successes and failures?
This is the final of nine blog posts setting out a basic set of skills for researchers who want to engage with stakeholders. Are there other skills that you would add as essential? Alternatively, would you omit any of the skills presented?
Sources and Reference:
In addition to my own research and experience, the main source for this blog post is Durham et al. (2014). They also drew on the work of others which is cited in the reference.
Durham E., Baker H., Smith M., Moore E. and Morgan V. (2014). BiodivERsA Stakeholder Engagement Handbook. ERA-NET BiodivERsA: Paris, France. (Online): http://www.biodiversa.org/702
Biography: Gabriele Bammer PhD is Professor of Integration and Implementation Sciences (i2S) at the National Centre for Epidemiology and Population Health at The Australian National University in Canberra. She is also a member of blog partner PopulationHealthXchange.
The Stakeholder Engagement Primer comprises the following blog posts:
Published:
1. a. Why a primer? b. Defining stakeholders (October 14, 2021)
2. Identifying stakeholders (October 21, 2021)
3. Selecting stakeholders (October 28, 2021)
4. Options for engagement (November 4, 2021)
5. Choosing engagement options (November 11, 2021)
6. Making engagement effective (November 18, 2021)
7. Listening and dialogue (November 25, 2021)
8. Generating ideas and reaching agreement (December 2, 2021)
This blog post:
9. Evaluating engagement (December 9, 2021)
Still to come:
10. Advanced skills (December 16, 2021)
Nice piece Gabriele. You’ve moved the argument for stakeholder engagement from “do this because it’s the ethical thing to do” to “think deliberately about what you expect stakeholders to contribute to the project at hand.”
Many funders are imprecise in their rationale for stakeholder engagement, which means they probably won’t invest much in evaluating how well stakeholders were engaged in a particular setting. One exception is PCORI (Patient-Centered Outcomes Research Institute), which has clear and specific expectations for how specific forms of stakeholder engagement will inform the design and implementation of intervention studies, as well as how results will be disseminated and translated in practice. (See p. 2 of https://www.pcori.org/sites/default/files/PCORI-Methodology-Standards.pdf)
Likewise, PCORI expects funded investigators to evaluate the effectiveness of their stakeholder engagement strategies, and promotes best practices learned through the studies they fund. See a recent article by my colleague, Sabina Gesell and her colleagues representing 9 PCORI-funded studies of transitional care interventions. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8263141/
I’d be curious to learn of other funders who are this intentional in their strategies for stakeholder engagement.
Many thanks, Doug, for these insights and references. It’s very useful to know about the work of the Patient-Centered Outcomes Research Institute (PCORI) and their clear-sighted understanding of the challenges of stakeholder engagement.
Indeed it would be great to get views from other funders.
Thanks very much Gabriele for putting together such a wonderful series of blogs on stakeholder engagement. I have really enjoyed reading them over the past 9 weeks and found them hugely informative and valuable.
Thanks! There is still one blog post to come on advanced skills. I am curious about how well you and others think the first nine blog posts cover a set of basics – whether there are other basic skills that should be included or some that I’ve included that could be omitted.
Thanks Gabriele for an interesting take on this! Speaking from the perspective of programmers/funders, I thought you may be interested in seeing how we assess stakeholder engagement post-project in the Biodiversa+ context. We’ve analysed outcomes of funded projects here, looking at both academic products and stakeholder engagement/products for society & policy, using a set of quantitative and qualitative indicators and giving concrete examples of products/outcomes for policy/society. Briefly, our general philosophy in evaluating/analysing projects in terms of stakeholder engagement is related rather to the means put in place to do so (identification and engagement of appropriate stakeholders, in well-thought ways etc. incl. production of stakeholder-intended outputs) rather than judging successful engagement through actual policy/societal impacts achieved. What we mean by this is that there are many decision factors for stakeholders, and that sometimes despite research having put all necessary means in place for knowledge to be taken up, decisions don’t build so much on it for other reasons, which we believe cannot be seen as the accountability of researchers. It’s somewhat inspiredfrom the honest broker position from Pielke’s work. Find out more here: https://www.biodiversa.org/893
Many thanks for these thoughts. Your comments also make me realise that I did not make clear enough what I mean by outcomes of stakeholder engagement. I agree completely that the outcomes of the research itself are dependent on many factors.
Mostly, evaluating the outcomes of the stakeholder engagement involves assessing how many stakeholders participated, how, for how long, etc. (as you have done in your reports), It could also assess insights gained, as this is a major reason for inviting stakeholder participation. Just occasionally, stakeholder engagement opens a door to implementation that would not otherwise have occurred and that should, of course, also be recognised, although it should be seen as the exception, not the rule.
Thanks for alerting readers to your excellent evaluation reports. This is also a good opportunity for me to reiterate my appreciation of the 2014 BiodivERsA Stakeholder Engagement Handbook, which provided material that underpinned several of the primer blog posts (cited in the respective primer posts). It is a tribute to the agencies involved that that they recognised the need for, and funded the production of, this excellent handbook.
Thank you for this post, Dr. Bammer. In my experience, stakeholder engagement is rarely evaluated in a manner as systematic as the one you propose here. The lack of systematic evaluation of stakeholder engagement results in missed opportunities, including failure to detect when engagement is going poorly from the perspective of stakeholders in time to take corrective action, and failure to gather the data that would contribute to an evidence base regarding what engagement strategies succeed in achieving researcher and stakeholder goals. The evaluation framework and questions you propose here move us in a productive direction.
Thanks!