By Daniel J. Hicks
Why do familiar metrics of impact often seem “thin” or to miss the point of research designed to address real-world problems? Is there a better way to measure the social impact of research?
In a recent paper (Hicks et al., 2018), my coauthors and I identified a key limitation with current metrics and started to look at how concepts from philosophy — specifically, ethics — can help us explain the goals of our research, and potentially lead to better metrics.
What’s the problem?
To understand the limitations of current metrics for measuring the social impact of research, it is useful to understand two distinctions, between resources and goals and between inward-facing and outward-facing goals for research.
Resources — such as funding and status — are valuable because they’re useful for pursuing scholarly goals; but they have no value in themselves. The point of getting a large research grant is to spend that money on worthwhile research. Research funds that go unspent are “wasted”; and research funds that are spent on, say, redecorating the principal investigator’s kitchen are “misused.”
Traditional scholarship is focused primarily on producing research that is valuable for other researchers. In our terminology, such scholarship has “inward-facing goals.” But scholarship can also aim to promote non-scholarly concerns or interests, whether by developing new technology or by identifying and critiquing the causes of social inequality. Such scholarship has “outward-facing goals.”
I believe that most researchers have both kinds of goals, though I also recognize that some researchers may have only inward-facing goals.
The problem is that many familiar research metrics attempt to use resources as proxies for goals. For example, it’s much easier to count citations and patents than it is to examine how lab-based experimental research has improved treatments for adolescent depression. When resource-based metrics get baked into institutional reward structures, researchers are incentivized to chase more and more resources, and not necessarily to achieve the intended social goals of research.
What’s needed for better metrics for measuring social impact?
To develop better metrics for outward-facing research, especially social impact, two critical steps are:
- to clarify the variety of outward-facing goals that researchers are pursuing, and
- to identify the variety of scholarly outputs they use in pursuit of those goals.
In other words, what social impacts are researchers trying to have, and how are they trying to have those impacts?
A conceptual framework called the “capabilities approach” can help with step #1. The capabilities approach was developed in the 1980s by development economists and ethicists. The approach attempts to measure societal development by focusing on the variety of things that people “can do and be,” rather than traditional economic indicators such as Gross Domestic Product (GDP).
In other words, the capabilities approach looks at the “goals” that characterize a flourishing society, rather than the resources that a society has. The capabilities approach is behind several familiar development initiatives, including the UN’s Human Development Index.
In the 1990s, ethicist Martha Nussbaum formulated a list of “central human capabilities,” which aims to be a relatively concrete list of the capabilities human beings require for a flourishing life and which is reproduced below (Nussbaum 2000). The list includes items such as adequate nutrition and shelter, but also opportunities for play, artistic expression, and political participation.
Importantly, Nussbaum’s list is designed to be open-ended. For example, her list does not include access to clean water and air. But environmental scientists can — and have — argued that pollution has a pervasive and damaging effect on human flourishing, and thereby make the case that clean water and air should be regarded as human rights (Gleick 1998). These arguments provide a strong case that clean water and air should be regarded as central human capabilities.
My coauthors and I suggest that the outward-facing goals of researchers can often be characterized in terms of central human capabilities.
And central capabilities can provide a shared language in which researchers, administrators, funders, policymakers, and the general public could discuss the intended social impacts of outward-facing research.
Where next?
My coauthors and I have begun to develop these ideas into an empirical research project. So far this has included showing how researchers from different disciplines can approach a single capability — adequate shelter, in our case — from different directions, as well as exploring the use of text mining methods to identify the central capabilities pursued by researchers at our home institution, the University of California, Davis.
The next phase of this project is to explore how capabilities and forms of public engagement vary across disciplines. We believe that institutional data — such as sections of tenure and promotion files, or broader impacts statements in grant proposals — will be invaluable for developing new metrics for outward-facing research. University records are likely to be especially important for humanities and arts disciplines, which are often not adequately covered by STEM-oriented (Science, Technology, Engineering and Mathematics-oriented) research databases. In addition, these publication databases do not track other scholarly outputs, especially public engagement. So far our focus has been on researchers at the University of California, Davis, and the data that might be used to describe their outward-facing goals and public engagement.
Do you agree that developing a ‘capabilities approach’ to the social impact of research has merit? Do you have ideas for other sources of data that might be used to make comparisons across universities, including internationally? How could capabilities-inspired analyses be used by university administrators or funding agencies to recognize and promote outward-facing scholarship without infringing on academic freedom?
To find out more:
Hicks, D. J., Stahmer, C. and Smith, M. (2018). Impacting Capabilities: A Conceptual Framework for the Social Value of Research. Frontiers in Research Metrics and Analytics, 3. (Online – DOI): 10.3389/frma.2018.00024
References:
Gleick, P. H. (1998). The Human Right to Water. Water Policy, 1, 5: 487–503. (Online – DOI): 10.1016/S1366-7017(99)00008-2
Nussbaum, M. C. (2000). Women and Human Development: The Capabilities Approach. Cambridge University Press: New York, United States of America.
Biography: Daniel J. Hicks is a postdoctoral researcher in the Data Science Initiative at the University of California, Davis. He previously worked in research program evaluation in the US federal government as a Science and Technology Policy Fellow with the American Association for the Advancement of Science. His broader research focuses on public scientific controversies, especially related to technology and the environment. His postdoctoral fellowship is funded by a gift from Elsevier. Elsevier played no role in designing or conducting either the study described in this blog post or this blog post.