Providing a richer assessment of research influence and impact

By Gabriele Bammer

author - gabriele bammer
Gabriele Bammer (biography)

How can we affirm, value and capitalise on the unique strengths that each individual brings to interdisciplinary and transdisciplinary research? In particular, how can we capture diversity across individuals, as well as the richness and distinctness of each individual’s influence and impact?

In the course of writing ten reflective narratives (nine single-authored and one co-authored), eleven of us stumbled on a technique that we think could have broader utility in assessing influence and impact, especially in research but also in education (Bammer et al., 2019). We wrote the reflective narratives for a special issue on Julie Thompson Klein’s academic work, in our case especially her impact in Australia and New Zealand. (A taste of Klein’s work can be found in her contributions to this i2Insights blog.)

The process of writing the reflective narratives was unstructured, with each of us writing as much or as little as we wished on whatever aspect of Klein’s work had resonated with us. Contributions ranged from a paragraph to a few pages. In addition to conventional influences, especially through her seminal book (Klein, 1990), our combined reflections illustrated the following:

  • the breadth of Klein’s work that influenced us was noteworthy, with only a few overlaps among the ideas and publications we each drew on
  • six of the reflections described fundamental influences on thinking and orientation that were not easily citeable in our own publications, including finding ‘permission’ in Klein’s work to do things differently
  • two of the reflections showed the practical value of her ideas in making sense of and/or shaping events and circumstances in inter- and trans-disciplinarity in our own institutions
  • two of the reflections illustrated catalytic effects when Klein’s ideas were combined with the ideas of others in particular educational and research settings.

These points illustrate that by each of us focusing as we saw fit, we uncovered dimensions of influence that might otherwise have remained hidden. In addition, we were pleasantly surprised by the greater whole that emerged from the sum of our individual written parts, especially as some of us were not sure we had anything worthwhile to contribute when agreeing to participate. The result reminded us of assembling shards of glazed pottery into a mosaic.

We suggest that there is merit in exploring how reflective narratives could be used more widely in assessing research impact. Three issues that require further consideration are:

  1. selecting contributors
  2. synthesis of the shards into a mosaic
  3. highlighting unique strengths.

Selecting contributors

If the person whose influence is being assessed belongs to a field with a large, active professional association, drawing on members for contributions seems like an obvious step. In Klein’s case, this was not possible as the main professional association – the Association for Interdisciplinary Studies – is small and has no members in Australia and New Zealand. In our case, the original invited contributor (GB) found the additional ten contributors a) by seeking volunteers from attendees at a conference and a workshop where Klein’s work was featured and b) through personal networks.

One of the interesting features of our group is that only four of us knew Klein personally. In addition, we covered a range of ages and career levels, from early career researchers through to professors and contributors who are retired and emeriti. Nevertheless we were not a representative sample.

There will always be challenges in drawing a representative sample from whatever population is chosen. But perhaps seeking a sample that is representative is less important than ensuring that there are enough pieces to assemble a mosaic, recognising that such a mosaic is not only a product of the individuals involved, but also of the moment of writing. The same person writing at a different time, in a different context, may well contribute different shards to the mosaic.

Synthesis of the shards into a mosaic

We found that just producing the shards of reflective narratives was not enough. They have to be assembled into a mosaic that tells a story of the individual’s influence.

How best to achieve such synthesis needs further thought. In our case the driver was an academic publication and several of the group provided valuable insights that guided the synthesis, including the metaphor of the mosaic and links to literature that helped emphasise aspects of Klein’s work.

Highlighting unique strengths

What most appealed to us about this process was its ability to highlight an individual’s distinctive strengths and contributions. In Klein’s case these included the meticulousness and depth of her scholarship, as well as her generosity towards others in the field.

More generally, each of us brings a different set of knowledge, skills and personal qualities to our academic work, which are not captured by metrics such as H factors and the number of publications in high impact factor journals.

For some it is deep knowledge about particular methods, for others the focus is on concepts, still others can extrapolate from myriad cases, and so on. Some are qualitatively skilled, others are outstanding wordsmiths, others are expert at project design and more. Some are skilled in nurturing up-and-coming talent, others in working with senior leaders; some are good at starting projects, others at finishing them; some bring creative thinking, others attention to detail; and the list could go on.

Metrics tend to focus on targets to reach or exceed, rather than affirming, valuing and capitalising on the wide range of individual differences that exist. And it is differences that are critical for good inter- and trans- disciplinary research. It is combining differences that makes for richer understandings of problems and that yields new, creative insights for tackling them.

What do you think? What other benefits and limitations can you see from employing reflective narratives? How do you think this could work in an institutional setting?

To find out more:
Bammer, G., Mitchell, C., Elford, W., Dumaresq, D., van Kerkhoff, L., Small, B., Kaufman Hall, V., Kaufman, S., Browne, C., Brown, V. A. and Blessington, L. (2019). A Rich Mosaic of Impact: Julie Thompson Klein’s Scholarly Influence in Australia and New Zealand. Issues in Interdisciplinary Studies, 37, 2: 144-169. Available online on Researchgate or on request.

Reference:
Klein, J. T. (1990). Interdisciplinarity: History, theory, and practice. Wayne State University Press: Detroit, United States of America.

Biography: Gabriele Bammer PhD is a professor at The Australian National University in the Research School of Population Health’s National Centre for Epidemiology and Population Health. She is developing the new discipline of Integration and Implementation Sciences (i2S) to improve research strengths for tackling complex real-world problems through synthesis of disciplinary and stakeholder knowledge, understanding and managing diverse unknowns, and providing integrated research support for policy and practice change.

Gabriele Bammer is a member of blog partner PopulationHealthXchange, which is in the Research School of Population Health at The Australian National University.

8 thoughts on “Providing a richer assessment of research influence and impact”

  1. Hi Gabriele. As always, I love your thinking. The image of forming shards into a mosaic is very neat. For me, it captures one of the keys to making a positive external contribution – creating a meaningful whole that others can understand and use. The extra thought I would have is the importance of designing the mosaic with an external audience in mind. I suspect we have all been guilty of bringing together shards to create a mosaic whose beauty only we see!

    Reply
    • Thanks Sean. That’s an important point and not easy to achieve. It invites us to think about differences in integrative and sense-making skills and also how not to go beyond what the evidence will support.

      Reply
    • Excellent point, Sean. While not working on a canvas or stucco wall, one of my favorite exercises when teaching students how to write has been drafting the same text for multiple audiences. Following your thinking, they were piecing the same shards into different mosaics. Julie T. Klein

      Reply
  2. Gabriele – thanks – I was impressed by this idea of “finding ‘permission’ in Klein’s work to do things differently” since the first time I read the submission to our Issues in Interdisciplinary Studies journal. The article brings out the more subtle ways that renown scholars like Julie lead by letting others feel permitted to really engage what, previously, they could can only imagine.

    Reply
    • Sven’s comment is also a good opportunity to remind folks to check the AIS website from time to time, for free access to past volumes of the journal and membership access for new ones. The journal is in a period of transition, so it’s also good to have his steady hand as co-editor. Julie T. Klein

      Reply
  3. I appreciate Chris Browne’s new example and the commitment this group of authors and editor made to narrative reflection. Humanities and to an extent social sciences are still marginalized in a narrow regime of “metrics” and “measures.” They have a valuable role to play but a dialogue of qualitative and quantitative perspectives is needed to produce a truly “rich mosaic,” to echo Chris and Gabriele. Thank you, all of you, for your efforts in this direction. Julie Thompson Klein

    Reply
    • Thanks, Julie, for your comment. It’s a timely reminder that both quantitative and qualitative methods have their place. Assessment of influence and impact is much poorer when one is missing.

      It also made me realise a point that is only implicit in the paper. The person whose influence and impact is being assessed is not the right person to organise the narrative reflections. That would put both them and the people they approach in a difficult position. The processes of soliciting and synthesising need to be done by someone who knows the relevant field or fields well and who is in a position to approach a broad network of people, including some who are not close associates of the person under consideration.

      Reply
  4. Gabriele – thanks for sharing your write-up of the paper… The ‘rich mosaic’ metaphor for considering impact is something we should all take away and apply in our daily lives.

    I wanted to share an experience that you can add to your own mosaic of impact, which feels particularly apt given the blog post on “Strategies to deal with forced hostile collaborations(!).” This would not be captured by any citation metric.

    I’ve been working on a project with an international cohort of professional engineers around “technical leadership” since mid-2018. Set up by senior/retired leaders in the field, we committed over 2 years to 6-monthly face-to-face workshops, with monthly virtual meetings, and homework in-between. We were primed with some theory at the face-to-face workshops (such as the Cynefin framework, the Johari window, and an individual 360-degree feedback process within our organisation), and were charged with running experiments of leadership/influence in our technical areas.

    We were encouraged to “produce an output on technical leadership” to mark the end of our experience and ‘graduate’ from the program. Having observed your process for the ‘Julie Thompson Klein article’, I led a reflective narrative process with no less than 18 engineers across 6 countries. To the astonishment of all (including me), the sum was indeed greater than the parts, and the result was — as you describe here — a rich mosaic that highlights each contributor’s unique strengths.

    For me, the ‘synthesis of turning the shards into a mosaic’ was an incredibly rewarding experience, and the authors — many of whom had never published anything — are collectively very proud of this output. The paper was accepted without revision to our international conference (TBA now with COVID-19). Browne, C. et al, 2020, “Experiments in Leading through Influence: Reflections from a Group of Emerging Technical Leaders”, International Council for Systems Engineering International Symposium.

    Reply

Leave a Reply to Sven Arvidson Cancel reply

%d bloggers like this: