By Diana Rose
Outcome measures in research about treatment and service provision may not seem a particularly controversial or even exciting domain for citizen involvement. Although the research landscape is changing – partly as a result of engaging stakeholders in knowledge production and its effects – the design of outcome measures has been largely immune to these developments.
The standard way of constructing such measures – for evaluating treatment outcomes and services – has serious flaws and requires an alternative that grounds them firmly in the experiences and situations of the people whose views are being solicited.
Standard measure generation is top-down. It starts from the literature, written by clinicians and academics, and collects together all existing scales on a given topic, also produced by clinicians and academics. This generates a large number of ‘items’ and there follow complex statistical processes to reduce the number of items, to ensure the measure is psychometrically robust and thus to develop a new and hopefully ‘better’ measure of what is being assessed. Occasionally the new scale will be ‘concept checked’ by potential treatment and service users, but that is both uncommon and a very limited role.
The first problem with this is that not much can ever change as new scales are so contingent on what has gone before. But more significantly, this method ensures that the measures assess what clinicians and academics think is important in what is being evaluated. This may be light years away from what is important to those on the receiving end of services. Their insights are rendered entirely invisible by the usual method just described.
My work is mainly in mental health, although the insights described here are transferable. Myself and my team are ourselves mental health service users as well as researchers (we refer to ourselves as user-researchers). We have devised a completely different way of generating outcome measures that has now been successfully used nearly a dozen times.
It starts with qualitative research with people who have used or experienced the treatment or service that we want to evaluate to gain their perspective. We run focus groups and expert panels (here the users are the experts) and these are facilitated by researchers who themselves have experience of what is being assessed. This ‘participatory method’ is an effort to break down the power structures that usually pervade the research endeavour – even though this is never spoken of. We are all part of the same community because we all have experience of mental health challenges and treatment – not exactly the same experiences but with much in common.
From intensive qualitative work like this we gradually build up a mixed-methods measure, constantly checking with participants that it reflects their concerns. When it is finally ready we do indeed undertake psychometric assessment. We are not anti-science!
In my area of mental health we have found that participants with diagnoses of psychosis deliver more robust psychometric results, for example very high test-retest reliability, than are usual for such scales. Interestingly this flies in the face of dictums in psychometric textbooks that ‘subjects’ must be ‘cognitively intact’ for psychometric assessment. It suggests that scales developed by peers are more likely to make sense and are therefore more robust.
Nevertheless, the method is not perfect, with three major problems. First, breaking down power relations in this context is difficult. It can fail completely or give rise to complex issues about reciprocity and degrees of disclosure by user-researchers.
Second, like any participatory method, it relies on ‘gatekeepers’ and crucial groups may be missed – for example those thought to ‘lack capacity to consent’ may be excluded, as may others whom gatekeepers regard as too ‘vulnerable’. The views of such people are crucial yet including them is incredibly hard.
Third, the measures do not please everyone. Out in the wider community most seem to like them, but – by believing we needed to make the measures ‘easy to complete’ – we have been accused of envisioning mental health service users as ‘simple-minded’. This was a wake-up call.
Paradoxes abound but this method does turn the standard on its head and in our view is grounded in and stays closer to the people who all this research is supposed to benefit.
What do you think? Do you have relevant experience to share?
For more information see:
Rose, D., Evans, J., Sweeney, A., and Wykes, T. (2011). A model for developing outcome measures from the perspectives of mental health service users. International Review of Psychiatry, 23, 1: 41-46. Online (DOI): 10.3109/09540261.2010.545990
Biography: Diana Rose PhD is Co-director of the Service User Research Enterprise and Professor of User-Led Research at the Institute of Psychiatry, Psychology and Neuroscience, King’s College London. She has also been treated by mental health services all her adult life and uses that experience to spear-head consumer-led research.