Science evaluation and social transformations

Science has always been evaluated. But over the past few decades the intensity of evaluation has drastically increased, and the forms of evaluation have radically changed. These transformations are observed with increasing concern by many scholars, and even by some funders of research. As a consequence, evaluation practices and their outcomes have themselves been subjected to analysis and critical scrutiny. In a preceding issue of Social Science Information/Information sur les sciences sociales, Olof Hallonsten (2021) developed a ‘historical-sociological argument’ on evaluation practices in order to offer a critical assessment of the current situation. He concluded that an ill-conceived concern about science not being sufficiently productive in supporting economic growth has led to the growth of bureaucratic control and management of science, increasingly based on quantitative performance measurement. His ‘deliberately pointed and provocative’ article found a strong echo. In the first part of this issue, we document the ensuing, rich and varied debate.

The interventions are short and sharp. They speak for themselves and do not require an extended introduction or overview. But given the variable emphasis on topics and the diversity of interpretations, it may be useful to outline the aspects that are being addressed and the sets of questions that are being raised in what follows.

A historico-sociological approach, as proposed by Hallonsten, is interested in understanding the current situation in the light of preceding social transformations. Hallonsten himself focuses on long-term trends such as economization, commodification and democratization. While not necessarily incompatible with such a focus on trends, other scholars prefer to emphasize ideational and/or institutional ruptures. Under this latter angle, the intensification of evaluation and the use of quantitative metrics to construct performance indicators are often related to the politico-economic transformation most commonly referred to as the rise of neo-liberalism. Rather than signaling the primacy of market self-regulation, as early promoters had put it, neo-liberalism is best understood as an ideology emphasizing competitiveness in many walks of life, on the one hand, and as a practice that works with parameters set or licensed by state authorities, on the other. Current practices of science evaluation would then appear to be only one component of a broader socio-political reorientation.

Thus, it becomes worth asking what particular place those practices may have in their socio-political context. From the 1990s onwards, maybe broadly inspired by Michel Foucault’s earlier opening, an important strand of historico-sociological research has investigated the relation between state forms and knowledge forms. In particular, the ‘origins of social policies’, in what became referred to as the early welfare state, were elaborated on the basis of new forms of social knowledge, notably in statistics with probability analyses, from the late 19th century onwards (Desrosières, 1991 and elsewhere; Rueschemeyer and Skocpol, 1996). From the late 20th century onwards, rather than simply dismantling the existing welfare state and its knowledge forms, the so-called neo-liberal state has aimed to enhance governance by new knowledge tools, among which figure prominently quantitative performance indicators made possible by big-data generation through new information technologies.

It would be erroneous, however, to see in this change only an intensification of control in an otherwise unchanged setting. Social change since the heyday of the interventionist welfare state in the 1970s included increasing diversity of view-points and plurality of practices within societies as well as the extension of many social relations across increasingly porous state boundaries. Scholarly self-understandings and audiences for scholarly work have undergone changes also owing to such processes, which are at best indirectly related to economization and democratization. As a consequence, there has been increasing awareness of the variety of ways in which scholarly work is embedded in its societal context.

Science-policy debates have long operated with a distinction between internal and external forms of evaluation, the meaning and significance of which were largely taken for granted. The distinction has at least twofold roots: in politico-philosophical terms, it is based on some concept of autonomy, against the imposition of constraints to knowledge-seeking, be they religiously, politically or otherwise motivated. In socio-theoretical terms, it resides in the view of ‘modern society’ being functionally differentiated into sub-systems operating according to their own logics or codes. While both arguments have their merits, they tend to neglect the complexity of the historical processes of constituting relatively separate, bounded sets of practices. Rather than assuming an overarching rationale or logics, more open concepts such as ‘field’, in Pierre Bourdieu’s approach, or ‘world’, in the pragmatic sociology proposed by Luc Boltanski and Laurent Thévenot, seem more suitable to capture the struggle over the organization of practices and the plurality of criteria for their evaluation.

Furthermore, one will need to consider the conditions of evaluation and its purpose. When the performative metrics focus on citation counts, then the apparent object of evaluation is a research result, namely the one that is cited. Clearly, the indicator does not measure in any direct way the quality of the research result. At best, it refers to its perception and recognition by other researchers, and even this measure is not beyond doubt. Nevertheless, this indicator is now regularly used to assess scholars and to determine decisions within the scholarly profession, such as appointments, promotions, salary increases. Doing so, other tasks within the profession, such as teaching, outreach, service tasks, are de-emphasized or even neglected, even though they are crucial for sustaining the collective endeavor. Yet another dimension is reached when evaluating research proposals. While the evaluation of researchers and research results already raises issues of validity and plurality of criteria, at least it can relate to information from the past. If the activity in question was conceived as a repetition of past activities, then research proposals could be evaluated on similar grounds. However, it is part of the basic understanding of scientific research to explore something that is not yet known. To evaluate future originality and creativity, therefore, requires reflection not only about criteria for evaluation but also about its very condition of possibility. It is a widespread observation that even evaluation by peers, the most accepted procedure within the scholarly community, tends to conservatively give primacy to feasibility over originality and, thus, to undermine the very purpose of scientific research.

All of the following interventions are rather critical of current evaluation practices, which should not be surprising, but in rather different ways. From some of them, one receives an image of science evaluation as a house of cards to which stories have been added after one another without investing in the firmness of the foundations and the solidity of the structure. Or, to use a recently fashionable image, of a bubble that inevitably is bound to burst, and maybe sooner rather than later. From others, in turn, the image that comes to mind is that of a trickling flow of water, small but steady, which is eroding a rock structure that seemed firm. If the latter image suggests that one might stop or reroute the flow and save the rock, the former leaves no other option than to undo and start over again. To decide which image is more appropriate, one could try to sort the arguments in the light of Luc Boltanski’s (2009) distinction between two forms of critique. If the critic observes that a practice is directed towards a valid purpose but fails to achieve it, then the task would be to improve the practice. If, however, the critic concludes that the practice is not pursuing a valid purpose or may even be undermining the pursuit of valid activities of those to be evaluated, then the conclusion would be that such form of evaluation should be abandoned. We hope that the following debate will help seeing more clearly where evaluation fails and what should be done about that.

*

The second part of this issue presents four articles that analyze aspects of recent social and political transformations. Taking up political phenomena that are often discussed under the heading of populism, here with the example of the mobilization for the secession of Catalonia from Spain, Esperança Bielsa focuses on the experiences of the participants in the movement using the sociological thinking on crowds. Similarly addressing social movement activity, Thomas Laux analyzes the conditions for the emergence of a global movement by looking at the protests calling for more resolute action on climate change. Through a comparative study on internet users, Laurent Cordonier, Florian Cafiero and Gérald Bronner try to answer the question why currently widespread conspiracy theories are more successful in some countries than in others. Given the persistent economic stagnation in many Western countries, concern has been rising about limited generational social mobility. In this context, Magda Nico analyzes changing meanings of the concept of social mobility, thus bringing this issue of SSI back to the assessment – though not evaluation – of developments in scientific fields.