Evaluation – Page 3 – Integration and Implementation Insights

Overturning the design of outcome measures

December 2, 2022February 21, 2017

By Diana Rose

Outcome measures in research about treatment and service provision may not seem a particularly controversial or even exciting domain for citizen involvement. Although the research landscape is changing – partly as a result of engaging stakeholders in knowledge production and its effects – the design of outcome measures has been largely immune to these developments.

The standard way of constructing such measures – for evaluating treatment outcomes and services – has serious flaws and requires an alternative that grounds them firmly in the experiences and situations of the people whose views are being solicited.

Pro-active learning to improve interdisciplinary processes

September 15, 2023January 3, 2017

By Laura R. Meagher

Member of Board of Governors — Laura R. Meagher (biography)

I am a firm believer in looking at interdisciplinary collaboration and knowledge exchange – or impact generation – as processes. If you can see something as a process, you can learn about it. If you can learn about it, you can do it better!

I find that this approach helps people to feel enfranchised, to believe that it is possible for them to open up what might have seemed to be a static black box and achieve understanding of the dynamics of how nouns like ‘interdisciplinarity’ or ‘knowledge exchange’ or ‘research impact’ can actually come to be.

In addition to using sources such as master classes, briefing guides or articles, individuals involved in interdisciplinary work can take several (inter-related) pro-active approaches to learning about interdisciplinary processes:

The strength of failing (or how I learned to love ugly babies)

January 21, 2026December 1, 2016

By Randall J. Hunt

randall-hunt — Randall J. Hunt (biography)

How to give others your hard-won insights so that their work can be more informed, efficient, and effective? As I’ve gotten older, it is something that I think about more.

It is widely recognized that the environment is an integrated but also “open” system. As a result, when working with issues relating to the environment we are faced with the unsatisfying fact that we won’t know “truth”. We develop an understanding that is consistent with what we currently know and what we consider state-of-the-practice methods. But, we can never be sure that more observations or different methods would not result in different insights.

The ‘methods section’ in research publications on complex problems – Purpose

September 5, 2024October 11, 2016

By Gabriele Bammer

Do we need a protocol for documenting how research tackling complex social and environmental problems was undertaken?

Usually when I read descriptions of research addressing a problem such as poverty reduction or obesity prevention or mitigation of the environmental impact of a particular development, I find myself frustrated by the lack of information about what was actually done. Some processes may be dealt with in detail, but others are glossed over or ignored completely.

For example, often such research brings together insights from a range of disciplines, but details may be scant on why and how those disciplines were selected, whether and how they interacted and how their contributions to understanding the problem were combined. I am often left wondering about whose job it was to do the synthesis and how they did it: did they use specific methods and were these up to the task? And I am curious about how the researchers assessed their efforts at the end of the project: did they miss a key discipline? would a different perspective from one of the disciplines included have been more useful? did they know what to do with all the information generated?

Should I trust that model?

September 15, 2023October 6, 2016

By Val Snow

How do those building and using models decide whether a model should be trusted? While my thinking has evolved through modelling to predict the impacts of land use on losses of nutrients to the environment – such models are central to land use policy development – this under-discussed question applies to any model.

In principle, model development is a straightforward series of steps:

• Specification: what will be included in the model is determined conceptually and/or quantitatively by peers, experts and/or stakeholders and the underlying equations are decided

• Coding: the concepts and equations are translated into computer code and the code is tested using appropriate software development processes

What makes a translational ecologist? Part 3: Dispositional attributes

June 25, 2025July 4, 2016

By the Translational Ecology Group

This is the third and final blog post considering competencies to make ecologists more effective in informing and supporting policy and practice change (see the right sidebar for links to all four related blog posts on translational ecology). In other words these are the competencies underpinning a new discipline of translational ecology.

The two previous blog posts examined the knowledge and skills required in three major areas:

Socio-ecological systems
Communication across boundaries, with beneficiaries, stakeholders and other scientists
Engagement with beneficiaries, stakeholders and other scientists.

This blog post uses the same three areas to examine the dispositional attributes required.

What are dispositional attributes?

Each person’s internal cognitive and moral qualities are collectively known as dispositions.

Choosing a model: If all you have is a hammer…

September 11, 2023June 2, 2016

By Jen Badham

As a modeller, I often get requests from research or policy colleagues along the lines of ‘we want a model of the health system’. It’s relatively easy to recognise that ‘health system’ is too vague and needs explicit discussion about the specific issue to be modelled. It is much less obvious that the term ‘model’ also needs to be refined. In practice, different modelling methods are more or less appropriate for different questions. So how is the modelling method chosen?

Sometimes the colleague will actually name a technique, for example ‘we want an agent-based model of an intervention to encourage physical activity’. However, this choice may not be informed. It might reflect something that he or she saw in a paper, at a conference, or whatever some previous modeller used. On the other hand, the colleague may have sufficient knowledge to have consciously selected an appropriate method.

From the other direction, many modellers work with only system dynamics or agent-based modelling or some other specific technique.

Can co-creation achieve better outcomes for people and communities?

December 10, 2024May 5, 2016

By Deborah Ghate

The language of ‘co-processes’ is much in vogue in policy, practice and academic communities worldwide. In commerce, product design and politics, the power of the crowd has long been recognised, but can co-processes be harnessed for the public good? The answer, right now, appears to be ‘maybe’.

What are co-processes and what are they for?

The briefest survey of the literature on co-processes confirms there is substantial variation in how they are defined and what methods or techniques they include. A confusing multiplicity of related terms exists—co-construction, co-production, co-design, co-innovation, co-creation—all are in regular use, sometimes interchangeably, and often defined at an unhelpful level of abstraction (for more on this topic see the blog post by Allison Metz on Co-creation, co-design, co-production, co-construction: same or different?). Nevertheless, however we define co-processes, participatory methods, boundary-spanning and inclusivity to varying degrees are foundational principles that can be detected in most accounts. Beyond that, the stated purposes and proposed outcomes vary considerably.

Towards an evaluation framework for participatory modeling

September 5, 2024March 31, 2016

By Miles McNall

What are the results of participatory modeling efforts? What contextual factors, resources and processes contribute to these results? Answering such questions requires the systematic and ongoing evaluation of processes, outputs and outcomes. At present participatory modeling lacks a framework to guide such evaluation efforts. In this post I offer some initial thoughts on the features of this framework.

A first step in developing an evaluation framework for participatory modeling is to establish criteria for processes, outputs, and outcomes. Such criteria would answer a basic question about what it means when we say that a participatory modeling process, output, or outcome is good, worthy, or meritorious.