Producing evaluation and communication strategies in tandem

Community member post by Ricardo Ramírez and Dal Brodhead

ricardo-ramirez
Ricardo Ramírez (biography)

How can projects produce evaluation and communication strategies in tandem? Why should they even try? A major benefit of helping projects produce evaluation and communication strategies at the same time is that it helps projects clarify their theories of change; it helps teams be specific and explicit about their actions. Before returning to the benefits, let us begin with how we mentor projects to use this approach.

dal-brodhead
Dal Brodhead (biography)

We co-lead DECI (Designing Evaluation and Communication for Impact), an action-research project that provides capacity building to research teams in the global south. We mentor these projects to develop their own evaluation and communication plans, something that we refer to as a hybrid decision-making approach. We recently published an on-line guide (Ramirez and Brodhead 2017) that includes the diagram below which summarizes the steps.

ramirez_mentoring - steps in evaluation and communication planning

(Source: Ramirez and Brodhead 2017)

The steps on the left side of the diagram are derived from Utilization-focused Evaluation, UFE (Patton 2008; Ramirez and Brodhead 2013), an approach to evaluation that emphasizes practical use of the findings and the process. Those on the right come from communication planning. Both share the notion of a readiness assessment, and situational analysis, which are depicted in the middle. The rotating arrows underline the iterative nature of the process.

How to do this?

We support partners at a pace that suits their needs through a process of coaching combined with peer learning. Each step in the hybrid approach includes a set of questions, which challenge the partner team to be clear on outcomes, procedures, stakeholders, networks, assumptions and methodology. On the evaluation side, we ask a set of readiness questions:

  • How capable is the project to work with this approach?
  • Is there a power relationship with funders that allows the team to co-own the evaluation design?
  • Is there an organizational culture that respects learning and adaptation?
  • Is there buy-in from senior management; and,
  • Are there engaged staff and resources able to carry out evaluation plans and implement a communication strategy?

These readiness requirements are sufficiently strategic to the point that we don’t sign a memorandum of understanding until we have most of these questions addressed (Ramírez and Brodhead 2014).

We follow with the main Utilization-focused Evaluation steps and ask who the primary users of the evaluation may be (internal staff, trusted partners, representatives of the funder, etc) and we engage with them to confirm their interest and availability. We also explore their expected uses or purposes. This process begins to show us the internal dynamics of an organization or project, especially the hierarchy, level of trust, and willingness to work in a collaborative manner.

In our experience, most evaluation users are quick to elicit evaluation questions, but they find it more difficult to step back and explore the underlying uses or purposes. The latter are important as a means of mapping how an evaluation will be utilized, rather than having a report sit on a shelf collecting dust. We find that it is also important to link the evaluation uses to the key evaluation questions. A good key evaluation question is clear, linked to the evaluation uses, and based upon this clarity one can easily determine the type of data collection tools and evidence needed.

In tandem with the Utilization-focused Evaluation steps, we begin asking questions about the project’s existing communication practices. Most projects have a communication way of doing things, even though it may be neither explicit nor strategic; yet it is often based on experience and some intuition. We ask about overall communication purposes, and we help the team identify them. The following purposes are common:

  • Communication for networking
  • Communication for active-listening and engagement
  • Communication for knowledge sharing for a community or practice
  • Communication for public relations, for visibility
  • Communication for dissemination of findings and lessons learned
  • Communication for advocacy
  • Communication for policy influence.

Differentiating the purposes and the audiences is helpful, as a means of setting priorities. We encourage the partners to do some ‘audience research’ to confirm each audience preference such as for media channels, methods, timing, etc. For each communication purpose, we explore the best combination and recommend testing of materials and methods. We also remind our partners that often the most effective communication is difficult to plan, but being ready to respond to windows of opportunity is possible and desirable, especially in the policy-making arena.

The benefits

Often, we witness how a hybrid effect begins to unfold. The partner realizes that an evaluation use could focus on the effectiveness of their communication strategy. Conversely, they realize that evaluation findings can often feed into and strengthen a communication strategy. However, most important, is that the process of clarification creates a space for organizational reflection and adaptation. The process begins as a planning process and creates the conditions for adaptive management. This result, in a nutshell, is the benefit we flagged earlier.

These concepts and processes can be useful to practitioners, facilitators, and researchers. Our website (https://evaluationandcommunicationinpractice.net/) contains multiple tools and case studies. We are keen to hear about comparable practices and experiences that others may wish to share.

To find out more:
Ramírez, R. and Brodhead, D. (2017). Evaluation and communication decision-making: A practitioner’s guide. Developing Evaluation and Communication Capacity in Information Society Research, DECI-2 Project. Ontario, Canada. (Online): https://evaluationandcommunicationinpractice.net/e-primer/

References:
Patton, M. Q. (2008). Utilization-focused evaluation. 4th edn. Sage: California, United States of America.

Ramírez, R. and Brodhead, D. (2013). Utilization-focused evaluation: A primer for evaluators. Southbound: Penang, Malaysia. (Online): https://evaluationandcommunicationinpractice.net/knowledgebase/utilization-focused-evaluation-a-primer-for-evaluators/

Ramírez, R. and Brodhead, D. (2014). Readiness and mentoring: Two touchstones for capacity development in evaluation. CDI Conference: Improving the use of M&E processes and findings, 20-21 March 2014, Wageningen, The Netherlands.

Biography: Ricardo Ramírez PhD is a researcher and consultant, based in Guelph, Ontario, Canada and is active in the fields of evaluation, communication for development, rural planning and natural resource management. He is an adjunct professor in the School of Environmental Design and Rural Development, University of Guelph. He is a Credentialed Evaluator (Canadian Evaluation Society) and co-principal investigator of DECI-3: Designing Evaluation and Communication for Impact: an action-research project in evaluation & communication funded by International Development Research Centre, Canada.

Biography: Dal Brodhead has been the CEO of the New Economy Development Group Inc., a value–based consulting firm located in Ottawa, Ontario, Canada since 1990. He brings a strong background in community development, project management, evaluation, and applied research in Canada and internationally. Previously, he held senior posts in various Federal departments, and directed a national research project on regional development for the Economic Council of Canada. Internationally, he has led numerous evaluation and monitoring missions in Asia and Africa with an emphasis upon participatory and recipient-driven and inclusive approaches. He is co-principal investigator of DECI-3: Designing Evaluation and Communication for Impact: an action-research project in evaluation & communication funded by International Development Research Centre, Canada.

Good practices in system dynamics modelling

Community member post by Sondoss Elsawah and Serena Hamilton

sondoss-elsawah
Sondoss Elsawah (biography)

Too often, lessons about modelling practices are left out of papers, including the ad-hoc decisions, serendipities, and failures incurred through the modelling process. The lack of attention to these details can lead to misperceptions about how the modelling process unfolds.

serena-hamilton
Serena Hamilton (biography)

We are part of a small team that examined five case studies where system dynamics was used to model socio-ecological systems. We had direct and intimate knowledge of the modelling process and outcomes in each case. Based on the lessons from the case studies as well as the collective experience of the team, we compiled the following set of good practices for systems dynamics modelling of complex systems. Continue reading

Scaling up amidst complexity

Community member post by Ann Larson

ann-larson
Ann Larson (biography)

How can new or under-utilized healthcare practices be expanded and institutionalized to achieve audacious and diverse global health outcomes, ranging from eliminating polio to reversing the rise in non-communicable diseases? How can complex adaptive systems with diverse components and actors interacting in multiple ways with each other and the external environment best be dealt with? What makes for an effective scale-up effort?

Four in-depth case studies of scale-up efforts were used to explore if there were different pathways to positively change a complex adaptive system. Continue reading

Overturning the design of outcome measures

Community member post by Diana Rose

rose
Diana Rose (biography)

Outcome measures in research about treatment and service provision may not seem a particularly controversial or even exciting domain for citizen involvement. Although the research landscape is changing – partly as a result of engaging stakeholders in knowledge production and its effects – the design of outcome measures has been largely immune to these developments.

The standard way of constructing such measures – for evaluating treatment outcomes and services – has serious flaws and requires an alternative that grounds them firmly in the experiences and situations of the people whose views are being solicited. Continue reading

Whose side are we on and for whom do we write?

Community member post by Jon Warren and Kayleigh Garthwaite

Jon Warren (biography)

In 1967 Howard Becker posed the question – to academics – “Whose side are we on?.

Becker was discussing the question during the time of civil rights, the Vietnam war and widespread social change in the US. He sparked a debate about objectivity and value neutrality which had long featured as part of the social sciences’ methodological foundations and which has implications beyond the social sciences for all academics.

garthwaite
Kayleigh Garthwaite (biography)

What relevance do these ideas have now, in an era when academics and their research are becoming increasingly commodified? Academics are increasingly pressured by their own institutions and fellow professionals to gain more funding, publish more papers and make more impact. Questions of social justice and professional integrity are at risk of being swamped by these forces allied to unscrupulous careerism.

We argue that the question now is not only who academics serve but also who we write for. Continue reading

Pro-active learning to improve interdisciplinary processes

Community member post by Laura R. Meagher

Member of Board of Governors
Laura R. Meagher (biography)

I am a firm believer in looking at interdisciplinary collaboration and knowledge exchange – or impact generation – as processes. If you can see something as a process, you can learn about it. If you can learn about it, you can do it better!

I find that this approach helps people to feel enfranchised, to believe that it is possible for them to open up what might have seemed to be a static black box and achieve understanding of the dynamics of how nouns like ‘interdisciplinarity’ or ‘knowledge exchange’ or ‘research impact’ can actually come to be. Continue reading