What’s Next for the Quantified Scholar? Impact,Metrics,and (Social) Media

The Metrics

Realistically, greater attention to impact is driven, at least partially, by metrics. Let’s face it, business schools are always driven by metrics, whether we like it or not. So if impact can be measured using metrics, business schools are likely to make decisions and allocate resources based on these, and generally take impact more seriously.

In the realm of scholarly impact, the notion of a “quantified” scholar is becoming more pronounced. More and more metrics are being used to measure individual-level impact, led by publishing houses and information companies such as the “Pure” platform of Elsevier, or Publons offered by Clarivate Analytics. Where once we largely evaluated scholarly performance according to the esteem of the publishers or journals we published with, now we also rely on a whole range of much more specific, and individualized, indicators of impact.

Some of these, like journal impact factors, are still measures related to the performance of the publishing outlets. But mostly, the growth in impact metrics focus on individual researchers and individual pieces of work. Just consider that we can now track how many downloads or citations each of our publications achieves on a daily basis. Or that many academic CVs now proudly boast of an individual’s burgeoning h-index (a numerical indicator of how productive and influential a researcher is based on their citations). And our professional rewards—whether in terms of hiring, tenure, promotion, pay, or other decisions—are increasingly influenced by such measures of scholarly impact. But what of real-world impact beyond academia? Are similar types of metrics (and criticisms) emerging to measure social impact?

The (Social) Media

The short answer is yes they are. And, outside of the U.K. government’s approach to measuring impact as part of its regular research assessment of universities, ¹ the main metrics used so far in business schools have tended to focus on media and social media impact. A good illustration is the 2020 Financial Times (FT) listing of 100 articles of “business school research with social impact.” ² This largely relied on Altmetrics, which are quantitative scores of online attention to scholarly works based on tweets and mentions in media, blogs, and policy documents, among other sources.

Many journals, including Business & Society, now post Altmetric scores alongside an article’s download and citation data. Altmetrics has fast become the default measure of impact beyond the scholarly community in the near decade since its inception. However, an increasing number of competitors, such as Kudos—designed to enhance the impact and visibility of journal articles—show that Altmetrics is not the only game in town. But now that the FT and elite journals have turned their attention to Altmetrics, you can be sure that business schools are not far behind. While this may be a positive move to many, what might be the fall-out from this trend?

A Malady?

Measuring impact will always bring with it as many problems as it does solutions. Real impact on the ground is not easily quantifiable and goes far beyond impact in the media or social media. Measures based on quantity of mentions, numbers of followers, or dizzy highs of engagement hardly account for quality of impact. And most metrics based on social media data can relatively easily be gamed by those who have a basic understanding of how an algorithm works.

It is also important to recognize that impact as we currently measure it can be more easily accrued by those in elite institutions and positions of privilege than those elsewhere. In some circumstances, such forms of impact measurement may even exacerbate gender inequality (Fortin et al., 2021). This is because certain scholars have access to the resources and networks of influence to begin with. It is these scholars who are more likely to generate the kind of impacts that get measured which in turn bring rewards to those high-impact scholars and their institutions. Those rewards can then be used to generate even greater amounts of measurable impacts. It is a self-perpetuating cycle that reinforces privilege and quashes access to impact for many.

Our experience with the Thinklist is a good example of this cycle—and of how we might start to break it (Centre for Business, Organisations and Society, 2020). We started the Thinklist in 2018 as a quarterly ranking of the business and society scholars who were making an impact on social media; essentially the social media influencers among our scholarly community. What we quickly realized, though, was that most of the most influential people in our field (as measured by our algorithms) were white men. To our dismay, we (and our critical friends) considered if in developing the Thinklist, we had become part of a broader problem of reinforcing voices of those in privileged positions. And in doing so, were we marginalizing the voices of those who, for a variety of reasons, were struggling to gain cut-through in a social media world dominated by metrics and (Western) masculinity?

Mastering the Metrics and (Social) Media

Last year we shifted our thinking on the Thinklist. Out went the ranking based on measures of social media impact and in came a deliberate attempt to identify voices ripe for amplification. The approach offered a much more inclusive process for identifying individual social media impact, moving beyond the quantified scholar toward the qualified scholar. It was an approach focused on community-building. We have since introduced dedicated lists of women scholars on social media, racialized scholars, and most recently, doctoral researchers. There is much more to come.

The approach is far from perfect. But rather than measuring impact, and therefore reinforcing the views of those that already have impact, we have attempted to carve out a path to give greater visibility to those that might desire impact but face structural obstacles to achieving it. We continue to work with expert curators for each of our quarterly Thinklists; asking those who come from positions of experience, rather than privilege, to identify who and what counts.

So, as we view impact metrics with an increasing dose of caution, the path forward is equivocal. We will probably need to continue to measure impact in order for it to be taken seriously, and social media present another means through which to rank, rate, and reward. But metrics also reinforce inequalities, and run the risk of becoming a pointless “game” of maximizing scores rather than making a real impact.

While we see the growing attention to impact (and to social media as one way that such impact is achieved and measured) as a positive development, we will need to work hard to avoid the negative consequences that the development of new impact metrics can bring. By and large, making an impact requires a skill-set that many of us are yet to master, and there is a real danger that any race for impact will not be a fair one. As business and society scholars, we need to take impact metrics seriously and be mindful of who they will advantage and who they will disadvantage. One of the biggest impacts we could in fact have would be in ensuring that our metrics work for the benefit of all—and that our activities as scholars, supervisors, and editors enable all business and society scholars to make a meaningful impact on the world.