By Daniel J. Hicks
Why do familiar metrics of impact often seem “thin” or to miss the point of research designed to address real-world problems? Is there a better way to measure the social impact of research?
In a recent paper (Hicks et al., 2018), my coauthors and I identified a key limitation with current metrics and started to look at how concepts from philosophy — specifically, ethics — can help us explain the goals of our research, and potentially lead to better metrics.
What’s the problem?
To understand the limitations of current metrics for measuring the social impact of research, it is useful to understand two distinctions, between resources and goals and between inward-facing and outward-facing goals for research.
Resources — such as funding and status — are valuable because they’re useful for pursuing scholarly goals; but they have no value in themselves. The point of getting a large research grant is to spend that money on worthwhile research. Research funds that go unspent are “wasted”; and research funds that are spent on, say, redecorating the principal investigator’s kitchen are “misused.”
Traditional scholarship is focused primarily on producing research that is valuable for other researchers. In our terminology, such scholarship has “inward-facing goals.” But scholarship can also aim to promote non-scholarly concerns or interests, whether by developing new technology or by identifying and critiquing the causes of social inequality. Such scholarship has “outward-facing goals.”
I believe that most researchers have both kinds of goals, though I also recognize that some researchers may have only inward-facing goals.
The problem is that many familiar research metrics attempt to use resources as proxies for goals. For example, it’s much easier to count citations and patents than it is to examine how lab-based experimental research has improved treatments for adolescent depression. When resource-based metrics get baked into institutional reward structures, researchers are incentivized to chase more and more resources, and not necessarily to achieve the intended social goals of research.
What’s needed for better metrics for measuring social impact?
To develop better metrics for outward-facing research, especially social impact, two critical steps are:
- to clarify the variety of outward-facing goals that researchers are pursuing, and
- to identify the variety of scholarly outputs they use in pursuit of those goals.
In other words, what social impacts are researchers trying to have, and how are they trying to have those impacts?
A conceptual framework called the “capabilities approach” can help with step #1. The capabilities approach was developed in the 1980s by development economists and ethicists. The approach attempts to measure societal development by focusing on the variety of things that people “can do and be,” rather than traditional economic indicators such as Gross Domestic Product (GDP).
In other words, the capabilities approach looks at the “goals” that characterize a flourishing society, rather than the resources that a society has. The capabilities approach is behind several familiar development initiatives, including the UN’s Human Development Index.
In the 1990s, ethicist Martha Nussbaum formulated a list of “central human capabilities,” which aims to be a relatively concrete list of the capabilities human beings require for a flourishing life and which is reproduced below (Nussbaum 2000). The list includes items such as adequate nutrition and shelter, but also opportunities for play, artistic expression, and political participation.
Importantly, Nussbaum’s list is designed to be open-ended. For example, her list does not include access to clean water and air. But environmental scientists can — and have — argued that pollution has a pervasive and damaging effect on human flourishing, and thereby make the case that clean water and air should be regarded as human rights (Gleick 1998). These arguments provide a strong case that clean water and air should be regarded as central human capabilities.
My coauthors and I suggest that the outward-facing goals of researchers can often be characterized in terms of central human capabilities.
And central capabilities can provide a shared language in which researchers, administrators, funders, policymakers, and the general public could discuss the intended social impacts of outward-facing research.
Where next?
My coauthors and I have begun to develop these ideas into an empirical research project. So far this has included showing how researchers from different disciplines can approach a single capability — adequate shelter, in our case — from different directions, as well as exploring the use of text mining methods to identify the central capabilities pursued by researchers at our home institution, the University of California, Davis.
The next phase of this project is to explore how capabilities and forms of public engagement vary across disciplines. We believe that institutional data — such as sections of tenure and promotion files, or broader impacts statements in grant proposals — will be invaluable for developing new metrics for outward-facing research. University records are likely to be especially important for humanities and arts disciplines, which are often not adequately covered by STEM-oriented (Science, Technology, Engineering and Mathematics-oriented) research databases. In addition, these publication databases do not track other scholarly outputs, especially public engagement. So far our focus has been on researchers at the University of California, Davis, and the data that might be used to describe their outward-facing goals and public engagement.
Do you agree that developing a ‘capabilities approach’ to the social impact of research has merit? Do you have ideas for other sources of data that might be used to make comparisons across universities, including internationally? How could capabilities-inspired analyses be used by university administrators or funding agencies to recognize and promote outward-facing scholarship without infringing on academic freedom?
To find out more:
Hicks, D. J., Stahmer, C. and Smith, M. (2018). Impacting Capabilities: A Conceptual Framework for the Social Value of Research. Frontiers in Research Metrics and Analytics, 3. (Online – DOI): 10.3389/frma.2018.00024
References:
Gleick, P. H. (1998). The Human Right to Water. Water Policy, 1, 5: 487–503. (Online – DOI): 10.1016/S1366-7017(99)00008-2
Nussbaum, M. C. (2000). Women and Human Development: The Capabilities Approach. Cambridge University Press: New York, United States of America.
Biography: Daniel J. Hicks is a postdoctoral researcher in the Data Science Initiative at the University of California, Davis. He previously worked in research program evaluation in the US federal government as a Science and Technology Policy Fellow with the American Association for the Advancement of Science. His broader research focuses on public scientific controversies, especially related to technology and the environment. His postdoctoral fellowship is funded by a gift from Elsevier. Elsevier played no role in designing or conducting either the study described in this blog post or this blog post.
Dear colleagues, I think that it is acceptable to be non-trivial in discussions on such important issues.
Once a literary hero exclaimed: “Money is not the main thing. The main thing is ideas”. The well-known popularizer of physical science, cosmologist physicist Paul Davies is sure that the main thing is not just ideas. The main thing is breakthrough worldview ideas. In this case, money (financing) is only an inevitable catalyst for breakthrough ideas that drop the first domino. Probably, it is precisely such ideas that are able to radically influence society, to solve its topical problems. However, these ideas suggest the creation of a new classification of research opportunities. Paul Davies writes in the paper «A new theory of cancer. After billions spent for little benefit, it’s time to look at the disease in a different way (2018). https://www.themonthly.com.au/issue/2018/november/1540990800/paul-davies/new-theory-cancer ):
“Unfortunately, there is a widespread belief that if enough money is thrown at the problem a solution will be found. All science requires funding, of course, but the truth is we have to think our way to a solution, not spend our way to one. And we won’t outsmart cancer until we understand what it is. I don’t mean understand it as a disease, but understand it as a biological phenomenon. There are more than a million published research papers on cancer and yet the simple questions, “What is cancer?” and “Why does it exist?”, have no clear answers in the scientific community. Maybe progress against cancer is so slow because we have been thinking about the problem the wrong way?
I first became involved in this subject in 2008 when I received a phone call from Anna Barker, at that time deputy director of the NCI. She perceived the need for some radical new thinking to replace the sameold-same-old mindset of many cancer researchers, and wondered whether bringing in scientists from other disciplines, notably physics with its stunning track record of success, might provide a welcome shakeup of the field. I countered that I knew nothing about cancer, and she replied, “That’s perfect!” Thus began a US$35 million per annum NCI-funded program in physical science and oncology. In all, 12 new Сentres were created across the US, each pairing a physical scientist with an oncologist.
I was chosen to run the Centre at Arizona State University (ASU), and I took seriously Dr Barker’s entreaty to rethink cancer from the bottom up. As a physicist and cosmologist I am used to sidestepping technicalities and searching for deep unifying principles” (p.42).
It is important to note that in this situation, Paul Davies did not go beyond the disciplines. Paul has gone beyond the limits of a disciplinary worldview. Thus, he showed that a worldview transition from disciplinarity to transdisciplinarity is possible without compromising effective interdisciplinary interactions. This circumstance forces us not to stop there. It is important to develop success.
We know that the basis of each discipline is a specific disciplinary worldview. Therefore, in order to develop success, it is necessary to clothe the transdisciplinary worldview in a specific scientific discipline. Depending on the image that is used in such a discipline, there may be two transdisciplinary disciplines.
– The image of the most integrated idealized object, in fact, is the basis of “object transdisciplinarity”. This type of transdisciplinary discipline is characterized by disciplinary and interdisciplinary thinking. Such thinking is the basis of the diverse methods that most transdisciplinary research is carried out today.
– The image of the most generalized system is the basis of “systems transdisciplinarity” (https://i2insights.org/2020/10/ ). This type of transdisciplinary discipline is characterized by system thinking. Such thinking is implemented within the framework of a systems transdisciplinary philosophy, concept and specific methodology for conducting research.
The term “maximum” is a key term that characterizes both transdisciplinary disciplines. The same term characterizes complex multifactorial problems of modern society. In this case, the conclusion arises about the adequacy of the capabilities of transdisciplinary disciplines in relation to rethinking the problems of modern society and the ways to solve them. This conclusion not only supports the idea of developing an “opportunity assessment approach” to the social impact of research, but also identifies the first domino that should fall according to Susan M. Fitzpatrick.
Very valuable. In my experience, many empirical scientists in biomedicine may believe that their contributions to societal goals are what motivates them while every action that contributes to advancement would align with inward looking goals. With time the gravitational pull of the culture can lead one to believe that the inward looking goals ARE how you achieve the outward looking goals. That amassing grants and publications is synonymous with truly advancing knowledge and that accumulating knowledge from purely experimental constructs is akin to the hard work of formulating solutions to the most pressing societal needs. We all know the problems – and we even know what should be done to right the ship and yet everyone (funders, institutions, researchers, professional societies) keep waiting for some unidentified first domino to fall…
Posted by editor from comment made on Listserv For Scientists.
I agree with Steve. Enable the ‘end-user’ of the research output/s to gauge / determine the utility of the research outputs. This could entail, inter alia, enabling the ‘end-user’ to determine the focus of the research; include them as co-researchers etc. Above all, include the ‘end-user’ of the research output as a partner from the outset. Also – design the research process using concepts such as complex adaptive management. I have attempted this and found it to have utility. I have attempted something similar and hope to have a more comprehensive account published next year,
With regard to Snowden’s work. I agree with Steve’s suggestion [there are opportunities to exploit SenseMaker to assist with evaluation]. I have developed a schema that could easily be adapted to measuring / evaluating impact.
Burman, C. J. (2018). Re-contextualizing medical pluralism in South Africa: a research schema for indigenous decision making Systemic Practice and Action Research. Retrieved from https://link.springer.com/article/10.1007/s11213-018-9460-0?wt_mc=Internal.Event.1.SEM.ArticleAuthorOnlineFirst. doi:10.1007/s11213-018-9460-0
I’ve argued for the importance of community-based research in some of my other work. But one advantage of our framework is that it can also be applied to research that’s not community-based. For example, consider Matthew Desmond’s book /Evicted/, which looks at the causes and effects of eviction on low-income families and communities in the US using a combination of ethnography and institutional data. This book directly addresses the capability for adequate shelter (under #2, bodily health), but isn’t community-based research.
I suppose that one has to balance the deficit left by not involving the community against the ease with which exercises of each type can be integrated with one another.
The regularity with which government policy or business strategy finds itself at cross purposes with the priorities of the communities with which they interact suggests to me that the value of community involvement is vastly underrated and its absence diminishes the value of research.
Chris
I’ve tried to contact you on LinkedIn
If I have the wrong Chris Byrman can you please drop me a line at grey@broadleaf.com.au?
Steve
Reply f Steve – email on its way
I suggest that this line of research should be linked with work in International Economics and Organization Science related to the “Capabilities-based View”. Here are some resources:
1) Atlas of Economic Complexity
– interactive web site : http://atlas.cid.harvard.edu
– Part I (pdf) : https://atlas.media.mit.edu/static/pdf/atlas/AtlasOfEconomicComplexity_Part_I.pdf
– Many good academic articles here: http://atlas.cid.harvard.edu/publications
2) Hidalgo, C. (2015). Why information grows: The evolution of order, from atoms to economies. Basic Books.
This article provides an excellent overview and integrative framework for the Capability-based View in Organization Science. It’s reference list is excellent.
– Peng, D. X., Schroeder, R. G. and Shah, R. (2008), Linking routines to operations capabilities: A new perspective, Journal of Operations Management 26(6): 730–748.
Sorry, I’m not sure I understand the connection you see here. Looking at uses of “capabilities” in the /Atlas of Economic Complexity/, it looks like it’s defined as “modularized chunks” of know-how or practical knowledge. But know-how can be a resource, in our sense. Knowing how to balance the books or giving financial advice, say, is very useful knowledge, but it doesn’t describe a goal of research programs (except perhaps in fields like accounting or finance).
Maybe you could clarify the connection as you see it?
I will try to clarify. Please forgive my long reply.
I am suggesting a cross-discipline connection on the very definition and operationalization of the construct “capability”, and specifically how capabilities develop, grow, decline, and transfer/learn; and also how capabilities relate to performance at some level of societal analysis.
In your blog post above, and also Fig. 1 in your paper, you give Nussbaum’s list of central capabilities. I presume that these capabilities are ascribed to individual people and also the smallest social units (a.k.a. focal entity = individuals, families, kin networks, and similar). As I understand your thesis, you claim that a better way of measuring/evaluating social impact of research is by how much they improve these lowest-level capabilities. Do I have that right?
The research I referred to you analyzes capabilities at higher social levels — focal entity = organizations, firms, supply chains, trading networks, countries, and regions. Even so, I argue, the construct of “capability” and how capabilities relate to the performance of the focal social entity are the same.
In all these settings, there often exist external institutions and organizations (‘players”) who have a goal to improve the performance of the focal entities. For these external players, the challenge is how to measure the impact of their work.
In your setting, you are interested in the societal impact of research performed by certain organizations (universities, non-profits, firms, individuals, agencies, etc.) and also funded/sponsored by certain organizations (governments, philanthropies, etc.). “Societal impact” is drawn very broad but also, as I understand you, focusing on only the lowest/smallest societal units (individuals, families, etc.)
I’ll add here that when individuals, families, etc. develop and apply Nussbaum’s capabilities, the are acquiring “modularized chunks” of know-how, practical knowledge, and resources, in the same way that firms and regions do when they develop and apply capabilities. Nussbaum’s capabilities are all described with sentences that start with “Being able to…”, which points to a potentially rich, complicated, and socially situated/constructed set of underlying know-how, practical knowledge, and resources needed to put each of those capabilities into action. Same goes for firms when they try to build capabilities, e.g. Lean Manufacturing.
In the setting of regional development, the external players are governments, large firms, wealthy/powerful families and landowners, bankers, venture capitalists, civil society groups, and so on. They may have general development goals (“prosperity”, “equity/justice”) or more specific goals and visions (“another Silicon Valley”). Their focus is usually on firms, trading networks, resource pools (incl. employees, experienced managers, etc.) and so on.
My main message: “societal impact of research” is structurally similar to “regional development” in that they both need to measure their impact in terms of the growth/development/evolution/transfer of “capabilities” for their respective focal social units. The construct “capabilites” is essentially the same in all of these settings, and would benefit from cross-discipline collaboration and alignment.
I think this ia a move in the right direction. It could go further though and some researchers are already doing this.
The natural destination of efforts to measure social impact is to engage those being impacted in exploring what matters and how much it is being affected by a programme. David Snowden has recently made some changes to Cognitive Edge that will see complexity science methods being more widely used to explore social systems.
Unfortunately, documented cases are few but a flavour of what is involved can be gained from this https://cdn.cognitive-edge.com/wp-content/uploads/sites/7/2021/04/12064432/GH-SenseMaker-brief.pdf and here https://cognitive-edge.wistia.com/medias/nc1oib4v01
Yes, I think I remember seeing this before, maybe shared on the SciSIP list. It’s an intriguing example of how data could be collected for the kind of analysis I’m advocating.
Not just a novel survey method but, handled properly, a means of engaging communities and providing them with an opportunity to have complex insights emerge from the process that would not have seen the light of day otherwise.