By Gabriele Bammer
How can the effectiveness of stakeholder engagement be judged? How can the outcomes be assessed? How much effort should go into such evaluation?
How and when to evaluate
In any stakeholder engagement there is no shortage of aspects that could be evaluated. The challenge is aligning the audience for the evaluation, the key issues to be assessed and the available resources.
For example, if a research team is interested in learning from what went wrong in a stakeholder engagement (for instance, if a stakeholder stopped participating or became hostile) and has no money set aside for evaluation, it might rely on self-reflection and anecdotal evidence to figure out what happened.
On the other hand, if a funder wants to know whether the time and money invested in stakeholder engagement was worthwhile, an independent expert evaluation might be warranted, with money earmarked for that in the grant. Such an evaluation would involve multiple considerations, including what the funder would find convincing, what was measurable, what the researchers and stakeholders thought was important and what could realistically be undertaken with the available time and budget. Typically budgets for evaluation are tight and circumscribe what can be undertaken.
In aligning the audience for the evaluation with the key issues to be assessed and the available resources, researchers may benefit from asking the following questions and sub-questions.
- Who is the evaluation for?
Sub-questions include:
- Do we need to convince funders that the stakeholder engagement was worth the time and money expended?
- Do we want to improve our effectiveness in engaging stakeholders in future research?
- Do we need to demonstrate to the stakeholders that the time and effort they put into participating in the research was worthwhile?
- What do we want to know?
Sub-questions include:
- Are we more interested in assessing the process or outcomes; or are we interested in both?
- Which aspects of the process are of most interest: eg., defining, identifying or selecting stakeholders; stakeholder engagement processes?
- What outcomes are we most interested in: eg., effects on the research design, conduct, results, interpretation; stakeholder satisfaction?
- Do we want to focus on what went well and/or what went wrong in both process and outcomes?
- Do we want to assess the engagement while the research is in progress, so that we can make adjustments, if necessary, or wait until the research has been completed?
- What resources and skills are available to conduct the evaluation?
Sub-questions include:
- How much funding and time can we set aside for evaluation?
- Does anyone on the research team have expertise in evaluation?
- Do we have funding to hire an independent expert evaluator?
Periodic self-assessment
It is worth undertaking at least a brief review of the stakeholder engagement when planning the research, while it is underway and at the end of the project, using questions such as those below. This can be as simple as an occasional discussion among members of the research team, including stakeholders when appropriate.
Planning the research:
Useful questions for reviewing stakeholder engagement when planning a research project include:
- Why are we engaging stakeholders? What outcomes do we expect and when?
- What are the best processes for achieving those aims? If processes have already been chosen, do they align with those aims?
- What are our indicators of success and failure?
- Are there any data that we can easily collect to measure success? If so, do we need to (and can we) collect baseline data before the research and engagement start?
During the research:
Reviewing stakeholder engagement while the research is underway provides opportunities to assess how well it is working and whether changes are needed. It can also provide useful feedback to stakeholders about how their contributions are being used. Useful questions include:
- What is working well? What is not? Why?
- Are appropriate processes being used?
- Should additional or different stakeholders be engaged?
- Are enough money, time and other resources available for the engagement? Are the costs reasonable? If the answer to either question is no, what can we do about it?
At the end of the research:
It is useful to review both process and outcomes when a project ends, particularly to glean lessons for future research. It is also important to bear in mind that some outcomes may not be fully realized until some time after a project is completed, so that subsequent reviews may also be required. Useful questions at the end of the project include:
- Were the aims of the stakeholder engagement met? Were the expected outcomes achieved? If not, why not?
- Were there any unexpected outcomes or impacts?
- Was the engagement process successful from both the researcher and stakeholder perspectives?
- Were sufficient time, money and other resources allocated to the stakeholder engagement? In relation to what was achieved were the costs of engagement reasonable?
- Are there lessons for future engagement processes?
Anything to add?
What has your experience been with evaluating stakeholder engagement? Do you have other tips for deciding when and how to evaluate? Do you have lessons to share about how to do periodic self-assessment effectively? How did you evaluate successes and failures?
This is the final of nine blog posts setting out a basic set of skills for researchers who want to engage with stakeholders. Are there other skills that you would add as essential? Alternatively, would you omit any of the skills presented?
Sources and Reference:
In addition to my own research and experience, the main source for this blog post is Durham et al. (2014). They also drew on the work of others which is cited in the reference.
Durham E., Baker H., Smith M., Moore E. and Morgan V. (2014). BiodivERsA Stakeholder Engagement Handbook. ERA-NET BiodivERsA: Paris, France. (Online): http://www.biodiversa.org/702
Biography: Gabriele Bammer PhD is Professor of Integration and Implementation Sciences (i2S) at the National Centre for Epidemiology and Population Health at The Australian National University in Canberra. She is also a member of blog partner PopulationHealthXchange.
The Stakeholder Engagement Primer comprises the following blog posts:
Published:
1. a. Why a primer? b. Defining stakeholders (October 14, 2021)
2. Identifying stakeholders (October 21, 2021)
3. Selecting stakeholders (October 28, 2021)
4. Options for engagement (November 4, 2021)
5. Choosing engagement options (November 11, 2021)
6. Making engagement effective (November 18, 2021)
7. Listening and dialogue (November 25, 2021)
8. Generating ideas and reaching agreement (December 2, 2021)
This blog post:
9. Evaluating engagement (December 9, 2021)
Still to come:
10. Advanced skills (December 16, 2021)