Developing a ‘capabilities approach’ for measuring social impact

Community member post by Daniel J. Hicks

daniel-hicks
Daniel J. Hicks (biography)

Why do familiar metrics of impact often seem “thin” or to miss the point of research designed to address real-world problems? Is there a better way to measure the social impact of research?

In a recent paper (Hicks et al., 2018), my coauthors and I identified a key limitation with current metrics and started to look at how concepts from philosophy — specifically, ethics — can help us explain the goals of our research, and potentially lead to better metrics.

What’s the problem?

To understand the limitations of current metrics for measuring the social impact of research, it is useful to understand two distinctions, between resources and goals and between inward-facing and outward-facing goals for research.

Resources — such as funding and status — are valuable because they’re useful for pursuing scholarly goals; but they have no value in themselves. The point of getting a large research grant is to spend that money on worthwhile research. Research funds that go unspent are “wasted”; and research funds that are spent on, say, redecorating the principal investigator’s kitchen are “misused.”

Traditional scholarship is focused primarily on producing research that is valuable for other researchers. In our terminology, such scholarship has “inward-facing goals.” But scholarship can also aim to promote non-scholarly concerns or interests, whether by developing new technology or by identifying and critiquing the causes of social inequality. Such scholarship has “outward-facing goals.”

I believe that most researchers have both kinds of goals, though I also recognize that some researchers may have only inward-facing goals.

The problem is that many familiar research metrics attempt to use resources as proxies for goals. For example, it’s much easier to count citations and patents than it is to examine how lab-based experimental research has improved treatments for adolescent depression. When resource-based metrics get baked into institutional reward structures, researchers are incentivized to chase more and more resources, and not necessarily to achieve the intended social goals of research.

What’s needed for better metrics for measuring social impact?

To develop better metrics for outward-facing research, especially social impact, two critical steps are:

  1. to clarify the variety of outward-facing goals that researchers are pursuing, and
  2. to identify the variety of scholarly outputs they use in pursuit of those goals.

In other words, what social impacts are researchers trying to have, and how are they trying to have those impacts?

A conceptual framework called the “capabilities approach” can help with step #1. The capabilities approach was developed in the 1980s by development economists and ethicists. The approach attempts to measure societal development by focusing on the variety of things that people “can do and be,” rather than traditional economic indicators such as Gross Domestic Product (GDP).

In other words, the capabilities approach looks at the “goals” that characterize a flourishing society, rather than the resources that a society has. The capabilities approach is behind several familiar development initiatives, including the UN’s Human Development Index.

In the 1990s, ethicist Martha Nussbaum formulated a list of “central human capabilities,” which aims to be a relatively concrete list of the capabilities human beings require for a flourishing life and which is reproduced below (Nussbaum 2000). The list includes items such as adequate nutrition and shelter, but also opportunities for play, artistic expression, and political participation.

Nussbaum’s list of central human capabilities (Nussbaum 2000)

Importantly, Nussbaum’s list is designed to be open-ended. For example, her list does not include access to clean water and air. But environmental scientists can — and have — argued that pollution has a pervasive and damaging effect on human flourishing, and thereby make the case that clean water and air should be regarded as human rights (Gleick 1998). These arguments provide a strong case that clean water and air should be regarded as central human capabilities.

My coauthors and I suggest that the outward-facing goals of researchers can often be characterized in terms of central human capabilities.

And central capabilities can provide a shared language in which researchers, administrators, funders, policymakers, and the general public could discuss the intended social impacts of outward-facing research.

Where next?

My coauthors and I have begun to develop these ideas into an empirical research project. So far this has included showing how researchers from different disciplines can approach a single capability — adequate shelter, in our case — from different directions, as well as exploring the use of text mining methods to identify the central capabilities pursued by researchers at our home institution, the University of California, Davis.

The next phase of this project is to explore how capabilities and forms of public engagement vary across disciplines. We believe that institutional data — such as sections of tenure and promotion files, or broader impacts statements in grant proposals — will be invaluable for developing new metrics for outward-facing research. University records are likely to be especially important for humanities and arts disciplines, which are often not adequately covered by STEM-oriented (Science, Technology, Engineering and Mathematics-oriented) research databases. In addition, these publication databases do not track other scholarly outputs, especially public engagement. So far our focus has been on researchers at the University of California, Davis, and the data that might be used to describe their outward-facing goals and public engagement.

Do you agree that developing a ‘capabilities approach’ to the social impact of research has merit? Do you have ideas for other sources of data that might be used to make comparisons across universities, including internationally? How could capabilities-inspired analyses be used by university administrators or funding agencies to recognize and promote outward-facing scholarship without infringing on academic freedom?

To find out more:
Hicks, D. J., Stahmer, C. and Smith, M. (2018). Impacting Capabilities: A Conceptual Framework for the Social Value of Research. Frontiers in Research Metrics and Analytics, 3. (Online – DOI): 10.3389/frma.2018.00024

References:
Gleick, P. H. (1998). The Human Right to Water. Water Policy, 1, 5: 487–503. (Online – DOI): 10.1016/S1366-7017(99)00008-2

Nussbaum, M. C. (2000). Women and Human Development: The Capabilities Approach. Cambridge University Press: New York, United States of America.

Biography: Daniel J. Hicks is a postdoctoral researcher in the Data Science Initiative at the University of California, Davis. He previously worked in research program evaluation in the US federal government as a Science and Technology Policy Fellow with the American Association for the Advancement of Science. His broader research focuses on public scientific controversies, especially related to technology and the environment. His postdoctoral fellowship is funded by a gift from Elsevier. Elsevier played no role in designing or conducting either the study described in this blog post or this blog post.

11 thoughts on “Developing a ‘capabilities approach’ for measuring social impact

  1. I agree with Steve. Enable the ‘end-user’ of the research output/s to gauge / determine the utility of the research outputs. This could entail, inter alia, enabling the ‘end-user’ to determine the focus of the research; include them as co-researchers etc. Above all, include the ‘end-user’ of the research output as a partner from the outset. Also – design the research process using concepts such as complex adaptive management. I have attempted this and found it to have utility. I have attempted something similar and hope to have a more comprehensive account published next year,

    With regard to Snowden’s work. I agree with Steve’s suggestion [there are opportunities to exploit SenseMaker to assist with evaluation]. I have developed a schema that could easily be adapted to measuring / evaluating impact.

    Burman, C. J. (2018). Re-contextualizing medical pluralism in South Africa: a research schema for indigenous decision making Systemic Practice and Action Research. Retrieved from https://link.springer.com/article/10.1007/s11213-018-9460-0?wt_mc=Internal.Event.1.SEM.ArticleAuthorOnlineFirst. doi:10.1007/s11213-018-9460-0

    • I’ve argued for the importance of community-based research in some of my other work. But one advantage of our framework is that it can also be applied to research that’s not community-based. For example, consider Matthew Desmond’s book /Evicted/, which looks at the causes and effects of eviction on low-income families and communities in the US using a combination of ethnography and institutional data. This book directly addresses the capability for adequate shelter (under #2, bodily health), but isn’t community-based research.

      • I suppose that one has to balance the deficit left by not involving the community against the ease with which exercises of each type can be integrated with one another.

        The regularity with which government policy or business strategy finds itself at cross purposes with the priorities of the communities with which they interact suggests to me that the value of community involvement is vastly underrated and its absence diminishes the value of research.

  2. I suggest that this line of research should be linked with work in International Economics and Organization Science related to the “Capabilities-based View”. Here are some resources:

    1) Atlas of Economic Complexity
    – interactive web site : http://atlas.cid.harvard.edu
    – Part I (pdf) : https://atlas.media.mit.edu/static/pdf/atlas/AtlasOfEconomicComplexity_Part_I.pdf
    – Many good academic articles here: http://atlas.cid.harvard.edu/publications

    2) Hidalgo, C. (2015). Why information grows: The evolution of order, from atoms to economies. Basic Books.

    This article provides an excellent overview and integrative framework for the Capability-based View in Organization Science. It’s reference list is excellent.
    – Peng, D. X., Schroeder, R. G. and Shah, R. (2008), Linking routines to operations capabilities: A new perspective, Journal of Operations Management 26(6): 730–748.

    • Sorry, I’m not sure I understand the connection you see here. Looking at uses of “capabilities” in the /Atlas of Economic Complexity/, it looks like it’s defined as “modularized chunks” of know-how or practical knowledge. But know-how can be a resource, in our sense. Knowing how to balance the books or giving financial advice, say, is very useful knowledge, but it doesn’t describe a goal of research programs (except perhaps in fields like accounting or finance).

      Maybe you could clarify the connection as you see it?

      • I will try to clarify. Please forgive my long reply.

        I am suggesting a cross-discipline connection on the very definition and operationalization of the construct “capability”, and specifically how capabilities develop, grow, decline, and transfer/learn; and also how capabilities relate to performance at some level of societal analysis.

        In your blog post above, and also Fig. 1 in your paper, you give Nussbaum’s list of central capabilities. I presume that these capabilities are ascribed to individual people and also the smallest social units (a.k.a. focal entity = individuals, families, kin networks, and similar). As I understand your thesis, you claim that a better way of measuring/evaluating social impact of research is by how much they improve these lowest-level capabilities. Do I have that right?

        The research I referred to you analyzes capabilities at higher social levels — focal entity = organizations, firms, supply chains, trading networks, countries, and regions. Even so, I argue, the construct of “capability” and how capabilities relate to the performance of the focal social entity are the same.

        In all these settings, there often exist external institutions and organizations (‘players”) who have a goal to improve the performance of the focal entities. For these external players, the challenge is how to measure the impact of their work.

        In your setting, you are interested in the societal impact of research performed by certain organizations (universities, non-profits, firms, individuals, agencies, etc.) and also funded/sponsored by certain organizations (governments, philanthropies, etc.). “Societal impact” is drawn very broad but also, as I understand you, focusing on only the lowest/smallest societal units (individuals, families, etc.)

        I’ll add here that when individuals, families, etc. develop and apply Nussbaum’s capabilities, the are acquiring “modularized chunks” of know-how, practical knowledge, and resources, in the same way that firms and regions do when they develop and apply capabilities. Nussbaum’s capabilities are all described with sentences that start with “Being able to…”, which points to a potentially rich, complicated, and socially situated/constructed set of underlying know-how, practical knowledge, and resources needed to put each of those capabilities into action. Same goes for firms when they try to build capabilities, e.g. Lean Manufacturing.

        In the setting of regional development, the external players are governments, large firms, wealthy/powerful families and landowners, bankers, venture capitalists, civil society groups, and so on. They may have general development goals (“prosperity”, “equity/justice”) or more specific goals and visions (“another Silicon Valley”). Their focus is usually on firms, trading networks, resource pools (incl. employees, experienced managers, etc.) and so on.

        My main message: “societal impact of research” is structurally similar to “regional development” in that they both need to measure their impact in terms of the growth/development/evolution/transfer of “capabilities” for their respective focal social units. The construct “capabilites” is essentially the same in all of these settings, and would benefit from cross-discipline collaboration and alignment.

  3. I think this ia a move in the right direction. It could go further though and some researchers are already doing this.

    The natural destination of efforts to measure social impact is to engage those being impacted in exploring what matters and how much it is being affected by a programme. David Snowden has recently made some changes to Cognitive Edge that will see complexity science methods being more widely used to explore social systems.

    Unfortunately, documented cases are few but a flavour of what is involved can be gained from this http://old.cognitive-edge.com/wp-content/uploads/2015/04/GH-SenseMaker-brief.pdf and here https://cognitive-edge.wistia.com/medias/nc1oib4v01

      • Not just a novel survey method but, handled properly, a means of engaging communities and providing them with an opportunity to have complex insights emerge from the process that would not have seen the light of day otherwise.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.