Research evaluation and the assessment of public value

A new ‘social contract’ for research

The growing interest in the assessment of the value of research outcomes is linked with a policy practice that has been spreading over the last three decades: funding organisations are increasingly asking for evidence of the results and effects that their ‘research investments’ eventually generate. Although the way in which they do so, and the type of evidence they ask for, varies across countries and organisations, there is a common interest in finding evidence of the economic or social value that research helps generate. As other articles in this special issue show, the arts and humanities (A&H) are not immune to this trend and present distinctive challenges to the assessment of research value.

The growing interest in obtaining evidence of economic or social impact has often been associated with the emergence of a ‘new social contract’ between science and society. The new ‘contract’ would emphasize the need to obtain economic and social benefits from research investments and would not take it for granted that such benefits would naturally follow from academic enquiry and scientific discoveries (Guston, 1992; Martin, 2004; Ravetz, 1988; Rip, 2003). This new environment helps explain the generalised diffusion of research evaluation practices and their institutionalisation (Whitley and Glässer, 2007).The expectation that science would generate social and economic returns is not new. Debates on the social and economic value of research and the extent to which they may guide the work of scientists can be traced to the early 20 th century (Kline, 1995). As Pielke argues, ‘in accepting the importance of science, politicians never deviated from their expectation that science would be instrumental’ (Pielke, 2012). Yet, what changed towards the end of the 20 th century was the way in which these expectations were translated into policy practice. The position that extraneous interference in the work of science in the name of practical application would be counterproductive to this same objective is also very old and was dominant in the decades following the Second World War. It is perhaps best reflected in Mme Curie’s famous quote:

We must not forget that when radium was discovered no one knew that it would prove useful in hospitals. The work was one of pure science. And this is a proof that scientific work must not be considered from the point of view of the direct usefulness of it. It must be done for itself, for the beauty of science and then there is always the chance that a scientific discovery may become like the radium, a benefit for humanity. (Curie, 1921)

From some quarters, the lack of evident economic impact was used to criticise public investment in research and to question whether research generated ‘value for money’. For instance, in the late 1970 s a critique of UK public sector waste targeted UK Government research establishments, arguing that expenditure in such organisations should be set ‘alongside an appraisal of what this expenditure has bought for the community’, and that such appraisal should involve ‘that section of industry most concerned with the end product of research’ (Chapman, 1979: 180).

This view implicitly considered the economic returns to industry as the only valuable result of research and was based on a simple image of knowledge generation and application: research leads to results that are then transferred to industry and turned into economic benefits. From this perspective, many, although not all, research activities in the A&H would not generate ‘value for money’. Indeed, for many years the analysis of how research could generate valuable impacts focused on the economic returns on ‘technology transfer’ achievable through commercial transactions like patent licensing and spin-offs (Radosevich and Kassicieh, 1994).

Partly as a reaction to the emphasis placed on economic returns, some analysts started to focus instead on the contributions of science to the generation of public or social value. Bozeman and Sarewitz (2011) argue that, while the economic and scientific impact of research has received substantial attention leading to the development of sophisticated impact assessment methodologies, science policy research has not focused with equal energy on the problem of the social value of research. This is important because the economic and scientific effects do not capture all the dimensions that are important to policymakers and the public at large. Science policy often explicitly pursues values other than the merely economic or scientific. Such public values can be defined as those ‘providing normative consensus about (1) the rights, benefits, and prerogatives to which citizens should (and should not) be entitled’; (2) the obligations of citizens to society, the state and one another; and (3) the principles on which governments and policies should be based’ (Bozeman, 2007). The ways of measuring such values are ‘as diverse as the aspirations of users, including not only pricing and profits, but status, curiosity and mastery of the physical worlds’ (Bozeman et al., 1999: 9).

The differentiation between economic returns and some wider concept of social value has reached the official policy literature. The Ricci Report on the assessment of the ‘social and environmental impact of European research’ distinguishes social and environmental effects from economic impacts and makes the well-known observation that measurement of the former is much more challenging (European Commission, 2005). As we move into the second decade of the 21 st century it would seem that the notion of research impact has definitely become multifaceted.

Yet, despite repeated warnings from scholars and evaluation experts that research impact assessment should not limit itself to the calculation of economic benefits, this practice has become entrenched. Benneworth and Jongbloed (2010) argue, for instance, that studies of how universities generate value have ‘largely become narrowly understood in terms of universities’ economic contributions’ and that ‘this emergent restrictive definition […] is a worrying development, overlooking the potential of universities’ knowledge in the Humanities, Arts and Social Sciences’. How ‘value for money’ is defined in practice and therefore how the ‘social contract’ is implemented becomes crucial for the future of the A&H. Too narrow a focus on economic returns may lead to a perception of valuable return that is biased against the type of contributions that we may expect from the A&H.

Implications for A&H research

Scholars in the A&H see this new environment with concern, and to a certain extent such concern is justified. A study of the A&H in the UK conducted by the Centre for Business Research at the University of Cambridge found that a large number of A&H academics saw their research as ‘basic’ and that they were more likely than their counterparts in other disciplines to feel that it was of no relevance to external organisations (Hughes et al., 2011). ¹ Such self-perception as ‘basic researchers’ with little applied or economic interest is accompanied by concerns about the way in which A&H scholars are perceived by policy-makers. For instance, Martha Nussbaum, while writing in defence of the A&H and against what she sees as ‘education for growth’, states of the A&H:

Seen by policy-makers as useless frills, at a time when nations must cut away all useless things in order to stay competitive in the global market, they are rapidly losing their place in curricula … nations prefer to pursue short-term profit by the cultivation of the useful and highly applied skills suited to profit making. (Nussbaum, 2012: 2)

Diversity in A&H research and its implications for evaluation

The Centre for Business Research study mentioned above finds important differences within the A&H when it comes, for instance, to the extent and the way in which they relate to research users and beneficiaries outside academia. While academics in the ‘Creative Arts and Media’ see their research as ‘user-inspired’ and applied, are highly connected and engage in relatively high levels of commercialisation, scholars in ‘languages’ tend to see their research as being of little relevance to external organisations. It is likely that further differences would have appeared within these groups had the study descended to a more detailed level of disaggregation.

It could be argued that these internal differences make it difficult to treat the area as a single whole. But internal diversity is not unique to the A&H. There are areas in the ‘hard’ sciences that are highly theoretical and whose link to application can only be long-term and circuitous, while others (say, pharmaceutical research) are explicitly linked with application though a series of formalised and documented steps (Benneworth, 2015) that make it easy to trace the path between research and application. Such diversity makes it difficult to treat as a unit any grouping of disciplines or intellectual endeavours we may come across. We could then debate, for instance, whether we should refer to the social sciences and humanities as a single area or whether we should always and under every circumstance treat them as separate entities.

Yet the research context will often force us to deal with groups we know are not homogeneous; from a practical perspective, the ‘subject’ of any evaluation will have to be defined to suit its specific policy and management objectives and these will, in turn, be framed by the existing, relevant organisational boundaries. In the UK much research has been carried out on the impact of the A&H (Arts & Humanities Research Council, 2012; Hughes et al., 2011; Levitt et al., 2010) not because there is any profound epistemological reason to bring these broad bodies of knowledge together and treat them separately from the social sciences, but because such evaluation research has been funded by the Arts and Humanities Research Council, the other social sciences being funded by a different Council. Our own studies of the social engagement of researchers from the Spanish National Research Council (CSIC) treat the humanities and the social sciences together because CSIC organises these disciplines under the same research area (Olmos et al., 2013). These subjects are internally diverse and the studies show that seldom do such differences fit with established large disciplinary aggregates.

Therefore, regardless of the way in which we may outline our research subject, there is likely to be a great deal of internal diversity. The A&H are not a solid, homogenous block and therefore the threat (perceived or otherwise) posed by the ‘new social contract’ will be felt and interpreted differently within different sections of the A&H research communities. How can we then deal with such internal diversity when analysing the ways in which academic disciplines generate value?

Aggregate quantitative methods aim to uncover common patterns, characteristics and general trends that a diverse population shares, and which we would not have been able to see if we had focused only on the variety of situations and actors and the importance of contextual factors. Quantitative studies can therefore help us find commonalities instead of identifying differences. Several studies have, for instance, found that in many important aspects the social sciences and the humanities are not so different from the natural or hard sciences. Hughes and his colleagues at the Centre for Business Research, University of Cambridge, found that the forms of connection and the external partners with whom such connections were established were similar between the A&H and other disciplines in the ‘hard’ sciences (Hughes et al., 2011). There is no doubt that such broad areas of knowledge are both internally diverse and different across them, and yet aggregate analysis is able to find common characteristics bringing together diverse populations. On the other hand, however, there are whole evaluation ‘schools’ (Pawson and Tilley, 1997) and impact assessment techniques (Spaapen and Dijstelbloem, 2005) that seek to identify in detail how policy mechanisms interact with different contexts in order to generate different types of outcomes.

We therefore encounter a tension between diversity and similarity: depending on our level of analysis, subject definition, and research methods, our assessments of the value generated by scientific work may offer common cross-cutting views or emphasize contextual factors explaining differences down to the group or individual level. By using quantitative techniques that allow us to focus on aggregates we can find common trends and characteristics, while some qualitative studies will highlight differences and the ‘contextual’ conditions that lead to such differences.

This tension between, on the one hand, seeking common causes and generalisable trends, and, on the other, emphasizing the role of contextual conditions and of differences across groups of subjects and even individuals, is crucial to the study of whether and in which ways academic work generates social value. The choice among approaches should be informed by the situation we are trying to analyse and the specific questions we are attempting to answer. If such study is conducted with the purpose of contributing to an evaluation of the performance of an individual, group or organisation, the perspective we will adopt will depend on the objectives of such an evaluation.

The diffuse nature of valorisation processes

There is an already sizeable stream of work that has analysed through case studies and other qualitative methodologies ² the impact of social sciences and humanities research. ³ These studies have found abundant examples of academics engaging with potential users and beneficiaries of their research and remarkable diversity in the ways in which the results of academic research have found practical application. Some are pervasive and have been identified in many studies and evaluations, like, for instance, the work of A&H researchers as curators of museums and exhibitions and their contribution to the production of historical theatre shows and films. Other contributions are less apparent, like the involvement of linguists in police investigations (Olmos et al., 2013), the role of historical research in political debates (Benneworth, in press), and the work of A&H scholars in road safety education projects (Cassity and Ang, 2006). Sometimes, the nature of these contributions has been questioned inside and outside the academic disciplines involved; for instance, the political utilisation of archaeology has a long history that has generated substantial debate and triggered detailed studies (McGuire, 2008).

Together with the different types of use and application comes diversity in the processes of application. Survey-based quantitative studies have confirmed the multidimensional nature of the valorisation processes. The study conducted by the Centre for Business Research surveyed 125,900 academics and 25,015 firms (obtaining a response rate of 17% and 11%, respectively) to find 23 different types of interactions between academics and external organisations classified under four broad activities (people-based, community-based, problem solving and commercialisation) (Abreu et al., 2009).

The pathways through which research results become socially valuable are not only diverse but also diffuse. Instead of being used in a direct and instrumental fashion, research knowledge can shape private and public activities and policies by providing a background of ideas that ‘creep’ (Weiss, 1980) into social and political attitudes and opinions. These diffuse paths contrast with the more straightforward links between research and application that can be found, for instance, in the formally staged process by which biomedical research acquires value (Benneworth, in press). This observation has methodological implications. Processes that can follow different paths at the same time and are diffuse, engaging different participants in different ways and over different periods of time, are difficult to encapsulate within a limited number of quantitative indicators. Enumerating and counting the connections, transactions and instances of use behind the valorisation processes is unlikely to provide a valid approximation to the social and economic value generated through such processes.

Nevertheless, there have been several attempts at developing scales of knowledge use, typically focusing on the ‘utilisation process’ and interpreting it as composed of different stages. The view adopted is that of a process by which a clearly bounded piece of knowledge (a theory, a method…) moves from researchers to users in cumulative steps. ⁴ Yet, such utilisation process is only one of the forms in which the knowledge generated by the A&H creates value; in other words, direct ‘utilisation’ of a specific piece of knowledge is only one among many ‘valorisation processes’. Co-production and iterative processes of knowledge generation and use, together with the multiplicity of pathways and the diffuse nature of other valorisation processes make synthetic measurement of these processes very difficult and partial, when not simply impossible.

The interactive nature of valorisation processes

Despite the variety of channels through which valorisation takes place, there is an issue where substantial consensus is emerging: the importance of social interactions as the main foundation through which research results find applications that generate social value. This argument has been made in different ways and from different perspectives. Landry and his colleagues find that the determinants of knowledge utilisation lie in the linkages between researchers and users (what they label the ‘interaction model’) instead of the nature of the research results (‘science push’), the users’ needs and context (‘demand pull’), or the efforts made to disseminate results (‘dissemination model’). The authors suggest that knowledge utilisation depends on various disorderly interactions occurring between researchers and users rather than on linear sequences beginning with the needs of the researchers or the needs of the users (Landry et al., 2001). From a different perspective, an international team of researchers placed the interactions between researchers and stakeholders at the core of a methodology to identify and trace research impacts (Molas-Gallart and Tang, 2011; Spaapen and van Drooge, 2011). This research found evidence of the indirect and interactive processes of application occurring across scientific disciplines including the social sciences and the humanities.

It has been further argued that the interaction between researcher and non-academic stakeholders is not only crucial for the use of research results but for social research itself (van Langenhove, 2011). From this perspective, researchers who deal with human issues should aim, first and foremost, at introducing change at the local level and therefore they should favour research that is local in scope. Only at later stages there would be room for more indirect contributions through the active dissemination of knowledge; for this reason, the argument continues, the sciences that deal with human affairs need to put more focus on knowledge-brokering and participatory approaches (van Langenhove, 2011). Other authors have placed interaction with society at the centre of what a social scientist should do; researchers dealing with human issues should aim their research at praxis, and therefore social links and interactions become central to the conduct of science (rather than just the ‘dissemination’ of its results) (Flyvbjerg, 2001, 2005). If this is the case, it follows that the analysis of such links should be a core element of research evaluation.

Yet, when evaluation focuses on the interactions between academics and those individuals and groups that the evaluator expects to be among the non-academic users or beneficiaries of the academic’s work, the evaluation will seek to address the processes through which research and its results become socially valuable. A focus on processes requires the use of research methods capable of providing a fine-grained understanding of the ways in which research generates public value, and the contextual factors that facilitate or hinder such processes. This work is very time consuming (and therefore expensive), and yields a type of result (awareness of the importance of specific, context-dependent factors) that is useful to improve practice, to adapt policies, and to tweak implementation routines, but can only provide contributory help to make decisions on the distribution of resources. Further, it cannot be used in evaluations carried out for control purposes: to learn in detail how researchers work and interact with non-academic partners it is necessary to obtain information about collaborative links that are often informal (Schiller, 2010) and may not even be part of the researchers’ formal responsibilities. The evaluator needs to get close to the evaluands and work with them, establishing a level of trust that could not be achieved if the goals of the evaluation were to be controlling or distributive.

Towards an approach for the evaluation of A&H research: Understanding ‘value’ in formative evaluation

Addressing the processes through which research and its results become socially valuable, aligns itself naturally with the ‘improvement’ function of evaluation and with the formative approaches described above. Evaluation practice can support the social engagement of academia, working with researchers to identify instances of interaction and valorisation and, in the process, helping them appreciate such activities and justifying their value (Molas-Gallart and Tang, 2011). Further, in a formative evaluation approach, the analysis of the social value of research will only provide one form of evidence and will contribute to framing the arguments accompanying policy definition and implementation.

In this process, questions about the nature of the effects observed, and in particular about whether a specific effect should be considered ‘valuable’, do not need to be solved a priori. It is not necessary for the evaluator to decide beforehand, whether and how a particular effect should be assessed as positive in itself or compared with other effects. The HERAVALUE project ⁵ has argued that humanities research contributes to social development, and such social development includes aspects like freeing the individual from the quotidian, or helping people develop more fulfilled lives. Not only are these objectives incommensurable, but social development is a political question (Benneworth, 2015); it follows that an analysis of the valorisation of A&H research needs to be explicit about the social development model to which such research may be contributing.

This is not the same, however, as making a preliminary decision about what form of social development is preferable before engaging in an evaluation exercise. As notions of social development, and therefore of social value, can be disputed, an evaluation could do nothing more than contribute to a process of policy development based on argument and discussion. As different publics will disagree on what is considered to be social progress, the value of A&H research cannot be measured in a way that could be consensually considered as objective or fair.

This means that the role of evaluation and impact assessment is not to provide final answers but to contribute to a process of policy development based on discussion and argument (Majone, 1989). Therefore, although evaluation needs to be explicit about the concept or concepts of public value on which a specific assessment is based, these concepts need not and should not be considered unique and beyond debate.

Not only does A&H research evaluation need to address how users and beneficiaries engage in research, but also requires an understanding of how they conceptualise value. Yet, such conceptualisation and engagement will vary across research areas, projects and application contexts. For instance, the way in which research in psychology is likely to generate social value and the nature of such value are different from the form of value generated by historical research. The understanding of these differences, and of how users and beneficiaries conceptualise value in different relevant contexts, needs to become part of any evaluation research.

This further strengthens the view of evaluation we are proposing here: evaluation needs to engage with researchers and ‘stakeholders’, it has to be aware of the context where knowledge is generated and applied, and it needs to focus on the processes that generate and are generated by the interactions between researchers and beneficiaries. Learning about the types of knowledge and other outputs that A&H research produces, the way they are disseminated and applied, and the contexts that enable such application, will clarify the nature of foreseeable or actual impacts and provide a perspective that can then be combined with others in a ‘multiple perspective’ evaluation framework (Kuhlmann, 1998).

Within this approach the ‘expectation of an exact measurement of “objective impacts” of a policy in the sense of immutable truth is neither possible nor wise’ (Kuhlmann, 1998: 138). Indeed, many research evaluation experts have long been advocating a formative approach contributing to policy argument and negotiation. Some funding bodies are starting to support this view. The UK Arts & Humanities Research Council has argued in favour of formative evaluation, noting that ‘too much emphasis has been given to the evaluation of impacts and not enough to the assessment of the process’ by which A&H generates socially valuable outcomes (Arts & Humanities Research Council, 2013).

These are, however, difficult goals to achieve in practice, mainly because they cut against the grain of current trends in evaluation and research management. Controlling and distributive functions are dominant in the current police drive to establish evaluation as an ‘accountability’ tool. Formative approaches emphasizing debate and progressive adaptation are confronted with rigid administrative systems unable to incorporate a formative evaluation approach (Silvani et al., 2005) and focusing on the controlling function of evaluation (Molas-Gallart, 2012). To incorporate formative approaches to the evaluation of A&H research will often require profound changes in research management, going well beyond the boundaries of evaluation practice.