Abstract
Early- and mid-career researchers will shape the future of library and information science (LIS) research and it is crucial they be well placed to engage with the research impact agenda. Their understanding of research impact may influence their capacity to be returned to research excellence framework (REF), the UK’s research quality assessment tool, as well as their ability to access research funding. This article reports the findings of a qualitative study exploring how the research impact agenda is influencing early- and mid-career researcher behaviour. Semi-structured interviews were conducted with 14 early- and mid-career researchers. While enthusiastic about creating lasting impact, participants lack effective institutional support to maximise their own research impact. Participants demonstrate uncertainty about what REF impact is. The authors conclude that while there is evidence LIS academics engage with practice to maximise impact, they lack support in building impact and the discipline needs to do more to create opportunities for the academy and the profession to coalesce to identify objects for and deliver impactful research.
1. Introduction
This article presents the findings of a study designed to gather data about the research practices of early- and mid-career academics in departments of librarianship and information science in the United Kingdom with particular reference to research impact. The study explores the careers of the participants and their engagement with the research excellence framework (REF), which assessed the quality of UK universities’ research output. An impact measurement element was introduced to the REF for the first time in 2014 in response to the Research Councils’ expectation that high-quality research should be able to demonstrate measurable impact. Impact in REF 2014 was defined as ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ (p. 26) [1].
While it is widely acknowledged that the impact of research is now a real priority for UK universities and academics, less is known about how that prioritisation is affecting and influencing institutional and researcher behaviour. A previous project by Marcella et al. [2] explored the research practices of senior researchers, revealing a degree of scepticism or cynicism with regard to the impact agenda. This article reports the findings of a second study, gathering new data this time from early- and mid-career researchers in the United Kingdom, designed to further understanding of the extent to which research impact might be influencing their research behaviour, recognising that these are the research leaders of the future.
Semi-structured interviews were conducted with 14 early- and mid-career academics between May and September 2016. These explored participants’ research career and the impact of their research, their involvement in REF2014, the influence of impact on future research plans, the support provided by participants’ institutions and their personal views on the measures of research excellence. Research findings create a vivid picture of the level of awareness of and interest in research impact among the participant group and how well motivated, informed and supported they feel in their efforts to enhance the impact of their research moving forward.
2. Literature review
REF is the UK’s research assessment framework, which guides the allocation of funding for research activities to higher education institutions based on the quality of staff research outputs. Institutions make submissions in five sections: information on staff, detail of publications and outputs, details of an institution’s approach to enabling impact alongside case studies demonstrating impact, data about doctoral research degrees and research income, and description of the institution’s research environment. Submissions are assessed within 1 of 36 units of assessment by expert panels [1]. Beyond the allocation of research funding, the REF provides a measure of accountability for public fund investment, benchmarking and reputational yardsticks. Introduction of the impact element to the REF 2014 was further proof of a general trend towards enhanced measurement and accountability, ensuring ‘value for money’ from public spending [3].
In the authors’ previous paper [2], impact case studies submitted to REF 2014 were critically analysed to gain understanding on the focus on impact assessment is having on the library and information science (LIS) discipline. Additionally, qualitative research interviews were conducted with established LIS researchers. Individual behaviour was not expected to change but a shift in focus across the discipline to include greater engagement with stakeholders and research beneficiaries in the early stages of research design was expected. The following literature review focuses on key research literature which has appeared since that publication and which discusses the experiences of early- and mid-career researchers.
During REF 2014, impact was assessed by expert panels in two ways: through institutional submissions of impact templates which detailed the whole unit approach to creating impact and in case studies which provided detail of specific examples of impact [4]. This methodology has come under scrutiny since the results of the exercise were published. While it is suggested that: ‘until future research on societal impact has developed reliable and robust methods, it clearly makes sense to have the societal relevance of research qualitatively assessed by expert panels’ (p. 230) [5], others consider the REF 2014 to have been an ineffective tool in the measurement of research quality [6] and posit that impact reward systems may be ‘unintentionally reifying traditional academic elites’ [7].
Critics of this method of impact assessment claim that expert panels fail to distinguish significance of impact from ease of presentation of impact [7], a phenomenon described as ‘impact mercantilism’ where the impact story better told – or sold – was given the highest evaluative scores [8]. It is also suggested that this focus on story-telling has led to less than accurate reflections of research impact [9]. By contrast, another study found that in Panel D, which dealt with LIS research in REF, research income and quantity of grants were in fact the most consistent variables in determining the impact case study scores [10], while Kellard and Sliwa [11] found that those which were scored highly were based on the work of small teams with established researchers, who had been in post for 20+ years, with a solid research base and external funding.
Critics have also labelled this type of research assessment as an institutional exercise in ‘game playing’: a game which is burdensome, costly and time consuming [6]. It was found that in many instances, academics are not prepared, trained or provided with the support required to document non-academic impact [12]. However, alternative measures of research excellence such as citation metrics while perhaps easier to compile and compare, are deemed unreliable, particularly in the Arts and Humanities [13]. For example, publication in high ranking journals does not necessarily equate to a high number of citations [14], and the impact factor of journals is now less strongly tied to citation counts [15].
The effects of the shift to impact assessment on institutional behaviour is thought to be far reaching, with the potential for institutions to focus on ‘REF-able’ work only and undervalue theoretical research [5] leading to an infringement of scholarly choice and an institutional research agenda which is deleterious to the production of new knowledge [16]. The impact of impact assessment on researcher behaviour has also been examined, with studies finding that user participation [17] and communication [18] were becoming increasingly important, albeit largely occurring towards the end of projects rather than in research development and design. With particular focus on LIS researchers, it was found that social media were used to widen knowledge of research and that collaboration with practitioners in conducting and communicating research were strategies used to increase impact [18].
For the REF 2014, early-career researchers (ECRs) were defined as members of academic staff who ‘started their careers as independent researchers on or after 1 August 2009’ (p. 19) [1]. An ‘independent researcher’ was someone who had ‘led or acted as principal investigator or equivalent on a research grant or significant piece of research work’ (p. 19) [1]. Evidence suggests that ECRs were under-represented as key researchers in impact submissions. While this might be deemed unsurprising considering the time required for impact to mature and become verifiable, some argue that the focus on impact weakens already precarious position of ECRs in higher education [10].
It has been argued, for example, that ECRs may be developing different ways of thinking about trust and authority in terms of scholarly communications [19]. Due to the influence of the 2014 REF, it has been found that UK ECRs ‘demonstrate a wider interest in reaching out to the general public and using innovative means (including social media) to do so’ (p. 11) [20]. Earlier research had suggested that ECRs felt they lack connections with non-academic groups that they tended not to regard engagement as a top priority focusing rather on public output and teaching, but that they do intend in future to enhance their public engagement beyond academia [21].
Several studies have also sought to establish how institutions can support ECRs to respond to the impact agenda. It is suggested that research-poor institutions should provide: a supportive and positive research-oriented culture; training for research management skills; access to research networks, communities of practice and research models; and rewards for achievement [22]. Effective mentoring and nurturing was found to be scant, even in research-intensive universities [23], but it is argued that supportive, non-transitory networks are important for ECRs, who typically will have experience of engagement with stakeholders but may lack guidance to build future collaborative partnerships in today’s highly competitive academic environment [24]. Others have suggested that programmes for ECR development should focus on teaching participants how to build a track record of research [25].
Looking forward to REF 2021, new Open Access rules for publications could be beneficial to ECRs as evidence suggests that that open access to publications and datasets creates not only a citation advantage of between 9% and 67% for ECRs but also increases the visibility of and engagement with their research [26].
3. Methodology
The aims of the study were to deepen understanding of both impact as defined in the REF 2014, and of wider impact assessment, on early- and mid-career researcher behaviour in the LIS discipline. An interpretivist approach was taken which allowed the researchers to understand and reflect upon the behaviour of early- and mid-career researchers, with in-depth interviews aiding the collection of rich qualitative data [27]. The interview schedule was evolved from the research design used successfully in the first project [2], amended to reflect a potentially greater need for interviewer explication of terms and to explore to the participants’ research careers to date, their current practices and their future aspirations. The schedule was structured in five sections: (1) eliciting data around participants’ research career and giving examples of the impact of their research; (2) their involvement in REF2014; (3) the influence of impact on their future research plans; (4) research and career support provided by participants’ institutions; and finally, (5) their personal views on the measures of research excellence and the reliability of various measures. The questions were open and sought to prompt extended responses.
Semi-structured interviews were carried out with 14 academics in UK and Irish universities between May and September 2016. Purposive sampling was achieved through web searches of BAILER (British Association of Information and Library Education and Research) institutions staff pages, identifying those whose published biographies indicated they were early- to mid-career researchers. The prospective participants were contacted by e-mail with an invitation to participate in the research and were provided with a paper which reported on the author’s first REF2014 impact project so that they might understand the focus of the present research [2]. From the 19 individuals approached, four failed to respond to the request, while another had retired and was excluded from the pool: all others agreed to be involved.
All 14 interviewees were full-time permanent members of academic staff, carrying out research in LIS. They were evenly split in terms of gender, with seven male and seven female participants. Five respondents described themselves as early career, eight as mid-career and one failed to answer the question; however, at least five of the overall number found it difficult to categorise themselves and this will be discussed further in the results below. The purpose and use of the research was explained to all participants as were the protocols for the execution of the research.
The interviews lasted between 25 and 90 min were recorded and fully transcribed. The research was designed and conducted with the four principles of ethical research conduct: to not cause harm to research participants, to not deceive research participants, for participation in research to be voluntary and for data collected to be confidential and anonymous [27]. All participants were assured both that their own anonymity and that of their institution would be preserved. Interviewees spoke extensively and engaged openly with the research questions: typically, they were self-reflexive and unassuming in their description of their achievements and behaviour. On occasion, where questions might be regarded as sensitive by participants, some displayed a degree of reluctance to elaborate and in such instances their unwillingness was respected by the interviewer and the line of questioning was not pursued so to reduce any potential psychological discomfort.
Thematic analysis was conducted manually using the interview transcripts, identifying the main themes within each set of questions asked of interviewees. In order to assure the rigour of the findings, the transcripts were analysed independently by two of the authors and the themes and findings were refined through comparison of both sets of emerging results.
4. Findings
4.1. Career profiles
Interviewees’ experience in an academic role varied between 4 and 20 years with the majority between 10 and 17 years. Half of the respondents had followed a purely academic path through doctoral studies, research assistant and fellow roles and were now lecturers and readers. The others had moved to an academic role from practice. It is uncertain whether this high proportion of practitioner experience would be typical for the discipline and indeed whether similar proportions would be common in other disciplines. All respondents were engaged in research, but the level varied from ‘I still try to do a bit of research’ to those who were primarily research oriented with a role in research leadership. Two participants were concluding their own doctoral studies.
When asked to state whether they regarded themselves as early or mid-career, interviewees paused to consider, with five interviewees able to respond categorically. Many described themselves as between two categories ‘I’m on the cusp between early and mid’, or were undecided about whether they merited in research terms the designation of mid-career researcher. Others distinguished between their lengths of employment as opposed to their research career duration, in particular those who had moved to academia from practice. Some reflected on their age rather than research trajectory but when prompted could clarify their self-perception around research maturity. Ultimately even the interviewee who at first claimed to be neither, on further reflection indicated ‘I’m certainly mid-career but not necessarily as a researcher so I suppose I would identify more with an ECR’.
Although this leaves five participants as ECRs on their own evaluation, and eight as mid-career and one unclear, the authors would acknowledge that this is a subjective assessment and illustrates the difficulty of self-categorising research status.
4.2. Understanding of research impact
Interviewees were asked to describe what impact meant to them in the context of their own research. There was a recognition that definitions of impact had changed in recent years and that impact has become increasingly important to them as the agenda has changed.
The most commonly expressed form of impact related to research being used in practice, being applied and used by the profession, in industry, in schools and being taken up and disseminated through the work of professional bodies. Industry funding was often cited as evidence of impact. There was some recognition that this related to the discipline and its focus: ‘I know I’m an academic and I’m an academic that primarily teaches a course that’s practitioner based’. However, impact on the public or wider society was also cited by a significant number of respondents.
Despite its reduced emphasis in the REF 2014 definition of impact, respondents also spoke of the continuing importance of the impact of their research on academia, on other academics and the continuing importance of citations in establishing the credentials of one’s research, on invitations to be involved in conferences or on expert panels as markers of ‘the contribution you’re making to knowledge and development in your research area’. One interviewee quoted the REF 2014 definition ‘outside academia’.
A surprisingly small number spoke about impact on policy, while one spoke about impact on specific groups. In two cases, interviewees pragmatically alluded to impact pathways statement being required as part of Research Council funding bids, and the extent to which this has necessitated greater reflection on impact on their part.
In one case an interviewee declared that ‘impact [is] not a driver for my research’ and suggested that the requirement to evidence impact had created a ‘false goal’ for researchers. Others tended to display a positive or pragmatic approach; however, there was recognition in four cases of a tension between theory and contribution to knowledge and practice: this was also framed as the potential for researcher focus on broad philosophic issues of social inclusion being stifled by the need to demonstrate impact: ‘that’s the mistake I made in my early career worrying too much about that – I was too focused on the practical issues and not theoretical enough’.
Respondents were probed further on their understanding of research impact by being asked for a definition. Only one could not provide a definition. While a small number concentrated on academic impact within and beyond the discipline, the majority spoke about impact more broadly: ‘it’s the legacy and … what happens as a result of you conducting your research’. They spoke of the concept of impact beyond academia: to demonstrate to ‘other people and groups that academia is relevant to them and the research that is done can have an impact in their lives in all sorts of ways’.
Two cases spoke of media coverage and press interest as being an indicator of their research impact. While it was acceptable to include this as a supplement to other evidence in an impact case study for REF 2014, some interviewees suggested that ‘academics very often confuse or conflate engagement or media attention with impact but for me it’s something which is a bit more tangible and often that can take a lot longer to generate’. Interviewees who focused on this tangible evidence of influencing policy and debate, noted that for them: ‘impact is when you can demonstrate a change or a shift perhaps in a policy which can be attributed to the research’.
4.2.1. Evidencing impact
When asked to give an example of the impact of their research, most interviewees were able to give an example of concrete evidence quickly and those who could not tended to be very early career: ‘my research is at the very early stages in terms of impact I think that’s probably something which will be evidenced next year’. Another was planning impact into future project design. Two spoke of challenges relating specifically to their doctoral studies: ‘any meaningful impact I don’t know and never actually went back and measured that’.
Four interviewees spoke of impact on practice in libraries and other organisations; ‘that did have a practical impact on how the … library went about their day to day business in terms of better outreach work’; and ‘the work methodology – the models are widely used in industry’. Two respondents had documented evidence of impact on performance data, for example, in schools; ‘there was a measurable difference in performance’ and ‘in terms of the way they are using the teaching to advance education’. Another respondent spoke about working with a professional body to develop guidelines around service delivery.
Five respondents gave examples of impact on policy often in projects working in partnership with or commissioned by government. Another spoke of their impact on legal proceedings, in providing expert evidence and on the formulation of legislation. One interviewee had developed an app and could provide data around usage to demonstrate scale of use. Two spoke of impact on the teaching curriculum and graduate skills, clarifying that this impact met the criteria as it extended beyond their institution as ‘it’s also been the teaching model in a lot of universities’.
When asked to give an example of the kind of evidence they would use to demonstrate impact, examples cited fell into five categories:
Comments in documents, letters and emails from participants, end users, during or after the research constituted the largest single group and a variety of examples were given of ‘testimonials’. These most frequently followed the conclusion of the research but respondents were increasingly trying to gather before and during, as well as after a project.
Interviewees also referred to seeking corroborative evidence from publicly available practitioner papers and other types of reference to the work: ‘the link on the Scottish government website’. Finding this kind of evidence often required the researcher to carry out an extended search and was particularly difficult to uncover from internal documentation. One interviewee described accidentally discovering that their work was being referred to verbally and seeking to make contact to record that this was indeed the case.
Impact on policy might be found in the evidence types (1) and (2) but for three respondents they could identify changes in policy without necessarily having any additional evidence to indicate impact. Two others were asked to give expert input to a policy forming entity.
Four respondents had evidence which showed that real change has taken place in performance or discourse, ‘there’s the actual notable difference in performance which I think is what people actually want to see’. Two of these were however hesitant about drawing a direct causal link.
Four researchers spoke about response to their research either in the form of media presence and coverage or in terms of response in the public domain or in reaction to web based material ‘metrics in terms of downloads and outreach of it internationally’.
Interviewees also spoke about the significant challenges associated with finding evidence and lack of clarity and/or consensus about what form evidence should take: ‘I think the edges are quite blurred’. Three respondents were trying to identify impact that was not being openly acknowledged, for example, on governmental approaches and structures; ‘the study possibly had some influence on the government having knowledge of a problem … I don’t think it necessarily influenced how they handled it’. Interestingly, one respondent spoke about the downsides of very high-profile research: ‘it wasn’t a very pleasant experience, so I deliberately didn’t pursue it any further’.
Gathering testimonials depends on researchers’ capacity to elicit response from the networks of influence, organisations, contacts and people involved in the original research and these had sometimes changed and despite requests no corroboration was forthcoming: ’I couldn’t get formal written confirmation through supporting letters’. Equally verbal references to impact of research might go unrecorded and undetected. Six respondents described the frustration that while there was evidence that change had taken place, they lacked confidence or capacity in demonstrating cause and effect: ‘it’s got to be factual (evidence) – you can’t just say well you know it’s made the world a better place’. One respondent could think of no examples of evidence that might apply and focused more on traditional forms of recognition of research excellence through citation.
Several respondents spoke of future plans for assembling impact evidence more proactively, in particular in doing so throughout the course of a research project in a more systematic and intentional way, eliciting feedback from stakeholders and participants on use, both pre- and post-events. Others were building connections with non-governmental organizations (NGOs) and professional bodies and seeking ways to improve the bibliometrics, citation counts and downloads of their research output.
4.2.2. Involvement in the REF 2014
Of the 14 researchers participating in the study, 9 had outputs submitted to REF 2014 and 4 did not (for one respondent the question was not relevant). Four had been part of a REF 2014 impact case study.
Inevitably, there was some unhappiness for those not submitted, who tended to be unclear why and how that decision had been taken. However, equally frequently, those whose work had been submitted were unclear about the process and there were references to anonymous senior officers with little knowledge of LIS and what constitutes quality in LIS taking decisions: ‘it was a complete cloak of invisibility’. Another respondent reflected: ‘there was an element of your being peer reviewed not by your peers … my work did go in but it was not the work that I would have wanted to represent for myself’. There were also references from respondents to the assessment exercise having been contentious and difficult.
Three interviewees had been involved in writing a case study and the process had been time consuming, frustrating and iterative ‘backwards and forwards, backwards and forwards’. In one case, the respondent was unhappy with the final submission: ‘it certainly wasn’t in its final form as far as I was concerned but it was submitted as if it was’.
In terms of being included in or referred to in one of their institution’s case studies, three were not included, while four did not know whether or not that had been referred to: ’I think it should have done but I’ve no idea if it did. It was … a little bit secretive’. Another four had been referred to but did not know at the time how inclusion had been decided: ‘it was quite mysterious’. Although it was not a question asked by the interviewer, it became clear from the tenor of the responses that many of the interviewees had never seen their institution’s impact cases, either prior to or since submission.
Overall three respondents gave a comprehensive description of the key elements of a REF 2014 impact case study: description of impact which is underpinned by high-quality research, published in the relevant research assessment period, authored by a member of staff of the institution, with evidence of the impact extending significantly beyond academia. Six respondents mentioned none of the above criteria, while five respondents admitted to being hazy around the detail and at best gave a partial description of what was required: ‘I presume some kind of narrative about how the department’s research has impacted more widely. Beyond that I really couldn’t tell you’.
4.2.3. Future research and impact
Respondents were asked whether they were planning ahead for inclusion in a future REF impact case. In all, 10 respondents were seeking to enhance the impact of their own research, while others were focussing primarily on the quality of their publications or securing funding to free up research time. Those who responded positively regarded research impact as important: ‘I’ve just made the assumption that it will be the only safe way to play it’ and ‘if anything impact might be even bigger the next time’.
Six respondents spoke about impact influencing research conceptualisation and design, seeing this as a ‘fundamental part of a new research project design’. Respondents also saw the importance of impact when making an application to funding bodies that require demonstrable pathways to impact as part of a funding bid. Three others were purposively designing research with the intent of ‘directly engaging policy makers and providing evidence to help them’.
Three respondents were very conscious of the need to be actively looking for evidence of impact and recording it and six spoke of the need to keep in touch with practitioners, individuals and organisations involved in their research. They were also engaging with professional bodies and some were volunteering for membership of professional body groups and advisory groups in order to maximise impact. Others were ensuring high levels of external involvement from expert groups. Involvement in writing codes, guidelines, creating tools for use in bringing about improvements and change were also activities thought probably by interviewees to ensure high impact for their research.
4.2.4. Maximising research impact
In terms of the strategies to maximise impact, four themes emerged: (1) dissemination, (2) stakeholder engagement, (3) methodological design and (4) topic selection.
In relation to dissemination strategies, almost all of the interviewees spoke about seeking better ways to disseminate knowledge about or raising awareness of their research through proactive strategies involving a range of traditional and new techniques, such as press releases, events for influencers, social media campaigns, public events and workshops and circulating White Papers to influential organisations and individuals. Dissemination events at the end of projects remained a feature but these were being supplemented by events throughout the duration of the research. Interviewees were ‘trying to … disseminate the results of the research more widely beyond it sitting in a journal paper’.
There was a recognition that their work might not always be very high profile in the United Kingdom and therefore needed to be ‘sold’. The application and significance of the knowledge needed to be highlighted: ‘that it’s a kind of lobbying, saying you can use this, this is of value to you … we can translate that into something practical’. This theme is echoed in the comment of another practitioner who said: ‘I have to get out there to make people aware of the work because it can’t just be left to publication or conference – I actually have to be knocking on people’s doors and saying listen here’s a project we’re doing, you need to know about it’.
Looking beyond traditional means of dissemination also inevitably involved social media, with a number of respondents mentioning online campaigns and websites. However, one respondent recognised the potential limitations in this approach: ‘we’re very good at thinking of ways of pushing it out but … we haven’t really thought through how we’re going to monitor’.
For stakeholder engagement half of the respondents mentioned the importance of working closely with influential stakeholders, as well as with end users of the research. Some emphasised the importance of embedding practitioners at the core of their research and of being conscious of there being a hierarchy of stakeholders from the end user to government: ‘it starts with the individual, whose life you’re impacting on … then it’s the people who need them to function in a different way – the government’. Stakeholders are frequently involved in workshops, advisory groups and so forth: ‘we’re getting immediate feedback so that you’ve got a better chance things will be used’. It was regarded as important to involve stakeholders from the inception of the research, in mapping out the research problem: ‘I think for any research … unless it’s completely blue skies research, it’s probably your starting point. Who is this going to matter to? Who is this going to have an impact on?’
In terms of methodology for four respondents, the impact agenda was influencing their research design: ‘we usually have some sort of participative methodology where we engage stakeholders and end users as well’. One interviewee spoke of ‘hiring a market research company to do a survey of public attitudes’. One respondent made the point that the nature of the vulnerable group involved in their research meant that impact was a priority regardless of REF and that this had impacted on methodological design.
For choice of topic, two participants spoke about impact influencing their research subject focus, so that they were focussing on: ‘the digital stuff, people are interested in digital culture’ and choosing ‘issues that we feel have not yet been considered by the relevant regulators’.
4.2.5. Guidance and support for increasing research impact
When asked if their institution had a research impact strategy, just one respondent was certain that such a strategy existed (from a post-92 institution). However, for four others there was a more hesitant belief that there would be one, while another was unclear if such a strategy existed. Remaining interviewees responded in the negative: ‘I fear that we’re already getting quite well into the next REF period … without a proper strategy’.
Impact evidence gathering was left to individuals, ‘I’m not aware of a proactive stance in terms of gathering data beyond the fact that we are all encouraged to sort of measure, gather data relating to impact’. For six respondents, their institution had a system repository for evidence, most commonly the PURE system: ‘we have to update our own personal records and we have impact sections there’. Not all of those respondents had fully engaged with the system in place. One respondent described additional support from coordinators who manage data relating to research projects. Two others said their institution was planning to introduce impact toolkit … that will hopefully provide practical guidance on how to create impact and measure and collect evidence of impact’.
Six respondents had access to mentoring although usually with a more general research focus. Of the eight who were unaware of an institutional research mentoring system, one described the lack as ‘problematic’.
When asked if research impact featured in their performance appraisal or management review, five respondents responded in the affirmative. Research impact tended not to be part of the career review conversations: ‘It’s a really a funny thing but it’s not … there’s nowhere on that form …’ For others, even research more generally seemed to merit little attention during performance review and where it did the focus was more on funding on and peer-reviewed journals: ‘there’s a lot more monitoring in place to see if we are working towards being submitted or being REF-able [but] it’s on journal articles’.
Nine respondents mentioned institutional impact training events or guidelines although they might not have personally participated. When they do attend they sometimes found the session unilluminating, as one respondent had learned: ‘we don’t have a very clear picture as to how impact is going to be assessed and how our research is going to be assessed for the forthcoming round of REF’.
Respondents were also asked if they had been given any direct personal advice beyond their institution. Nine had been given no such advice. Others mentioned: attendance at a research event where ‘people like myself who had submitted potential case studies were asked to give a bit of overview and we were given direct feedback from members of the advisory group’; in informal conversations with a research mentor; through institutional support and guidance on working with the media; and informal conversations around how to engage beyond academia with colleagues.
4.2.6. Impact as a measure of research quality
The interviewers then asked if the introduction of impact in research assessment had resulted in more impactful research. This is a challenging question and overall the responses given tended to be tentative and vague. Two respondents were unable to say, while a further three were only able to hazard that it might have, in particular for those who have recognised its influence in terms of career progression. One respondent spoke of impact as making ‘people more mindful and more reflective’, while acknowledging ‘there’s a danger that people will focus too much on speaking about what they’re doing and not on actually generating impact’. Four respondents thought that research might become more impactful eventually ‘because it’s been placed as such a key measurement’ and is already influencing ‘the kinds of research activities that people engage in, so that it’s harder now to do blue skies research’. Three interviewees proposed that impact assessment would not increase impactful research: ‘it’s made people more aware of how to evidence and how to … externalise impact … things are no more impactful than they were before’. A third expressed doubts: ‘it feels like it’s an artificial exercise really in people … morphing their research to fit the indicators’. One respondent answered with an unequivocal affirmative: ‘universities need to bridge the gap between them … and how we influence the region and the regional economy’.
When asked if they thought that the introduction of impact to research quality assessment had resulted in ‘game playing’, eight respondents agreed that this had been the case, four said that it was true to an extent and two said that this was not so. Of the 12 respondents who believed that impact assessment had led to game playing, four said they knew how to play the game. Overwhelmingly, the majority observed and spoke with some passion about this phenomenon: ‘its gaming for me writ large … a really expensive example of game theory by accountants’. Another answered equally emphatically ‘one hundred percent yes – I mean it is, it really is.’ As one said, ‘well the whole of REF is a bit of a game play so I don’t know if we need to pick impact out of it’.
Some respondents spoke of game playing at institutional level ‘all the institutions are fighting for the same pot of money, and I suppose it’s inevitable they’re not all going to play by the same rules’. Others focused on individuals ‘it’s about compliance with the rules that are set for us rather than perhaps joining an organic collaboration with the non-academic world’. One respondent described the phenomenon as ‘politics and there’s a lot of that in universities and there’s a lot of bandwagon jumping’. Another made the broader observation ‘I think it’s very similar to a lot of government policies based around metrics and scoring … where the metric becomes more important than the actual thing’. Of the two respondents who said that game playing was not a result, one took care to distinguish their comment as relating to their own behaviour: ‘personally no – I think impact is vital for me and it’s always been part of what I do’. The second saw ‘impact as something that’s really important and I know we’re all doing research but what impact is doing … it’s building connections’.
ECR interviewees tended to rate their capacity for ‘playing the game’ as low: ‘I think the gaming is people way above my grade’. One respondent commented that ‘I don’t think anybody knows exactly how to play’ and reflected on the extent to which decisions about submissions would be taken by universities that might nullify any game playing by individuals. One interviewee took a more altruistic perspective explaining ‘I don’t really want to play the game … I just want to do good research and if impact is part of that then that’s great’. Alternatively, one respondent dismissed game playing because ‘I think it’s the personal academic career game that everybody is probably playing’. Of the four who knew how to play the game one said: ‘I think I do, it sounds quite cynical. I also know people who are making quite good careers out of showing other people how to play the games’.
Finally, interviewees were asked whether they regarded impact as defined by REF to be a reliable measure of research excellence. One respondent agreed unequivocally: ‘I think that it’s good that the importance of impact be increased … the numbers game is not necessarily productive’.
Five respondents agreed broadly that impact was a good measure but hedged their statements with caveats around:
The capacity to measure meaningfully: ‘You could maybe identify that it’s changed practice in governments and in parliaments but how can you possibly measure what impact that’s had on the wider society?’
The negative consequences of overly focussing on impact; ‘I think broadly it is, although the negative repercussions of it are … going for short terms wins instead of long term sustainable things … I think it causes a lot of anxiety’.
That it is one among a number of valid measures, such as: ‘generating PhD students and getting money for research, applying for and getting funding for good research projects and publishing in high quality journals’.
Overall eight interviewees replied in the negative that impact was not a reliable measure. Concerns were expressed around the reliability, objectivity and tangibility of measurement of impact. One interviewee, who felt strongly that research impact was really important, still believed that it was not currently being reliably measured; ‘it’s highly suspect in terms of the evidence that people can gather and the judgements that people will make … it’s very open to personal belief’. Another concurred: ‘it’s an unreliable measure … it’s the direction of travel we’re stuck in but I don’t see that changing anytime soon’. Overall there was a sense that it was something of a movable feast capable of interpretation in a myriad of ways: ‘I feel it’s another slightly confusing goal that will probably change again in the next cycle … I don’t feel … that it’s got fixed meaning’.
Others thought that it excluded valuable research in systematic review, interdisciplinary creativity and blue sky, theoretical research and disadvantaged research in some subject areas: ‘I think the problem [is] … that people who are theorists and … for example historians who do things about the past, things which are very niche, are still doing excellent research it just doesn’t perhaps have the commodification that people who have perhaps more impact-friendly research has’.
The 13 who had expressed doubts about the reliability of impact as a measure of research quality were asked what they thought was a good and reliable measure. Two could not identify a more reliable indicator suggesting that this in itself might be a problem endemic to academia: ‘that’s inevitably part of having a community of peer review researchers because they will all probably have a different view of what quality is, and it’s quite hard to ever imagine there being a consensus’. Nine interviewees identified perceived better alternatives: four respondents answered bibliometrics and citation analysis; four, peer-reviewed output; and one, research funding.
5. Conclusion and recommendations
Throughout the interviews, respondents reflected on how the LIS domain is enmeshed with the professional domain: many had entered academia from practice, many spoke of interacting with the profession as partners, stakeholders, research subjects and users of the research. There was for many a sense of specialness, of vocation sitting in a profession where much of what is done is intrinsically for the benefit of society. On the downside others acknowledged the challenges facing the subject in demonstrating impact, as this benefit can be intangible and not easily measured. This echoes literature which suggests academics are not prepared or trained in documenting non-academic impact [11].
ECRs tended not to be able to provide examples of the impact of their personal research and, in particular, their doctoral studies seem to have not involved any sense of obligation to bring about impact. While almost all participants could give examples of their research impact, fewer felt comfortable that their evidence was completely satisfactory and defensible, particularly in terms of conclusively showing cause and effect. They were almost unanimously purposively designing impact into all stages of future research design and employing strategies designed to maximise and record impact. Enhanced impact strategies included better ‘selling’ of the value of the research to various stakeholders. This was felt to be achievable through: increased stakeholder engagement with influential voices and end users; use of participative methodological design; and choosing topics with a high interest factor. Additionally, more consideration would be given to ‘telling the story’ of impact in response to perceived notions of the evaluative scoring of REF panels [7, 8].
Early- and mid-career researchers demonstrated a range of belief when it came to their views on the reliability of impact as a quality measure and the extent to which it involves game playing. However, on two things there was overwhelming consensus: (1) that research impact is not a reliable measure and needs to be considered alongside other qualitative criteria in evaluating research quality and (2) that institutions in particular and individuals engage in game playing with regard to demonstrating research impact, in line with assertions made in the literature [5, 6]. Their responses ranged from what might be described as the purist or idealist at one end of the spectrum and the realist or cynical at the other. To an extent, their placing on that spectrum did relate to length of tenure, with mid-career tending towards the more realist positions.
Although participants believed that the introduction of impact to research quality assessment had resulted in ‘game playing’, they did not know how to play the game. In the authors’ earlier paper [2], it was hypothesised by the authors that early- and mid-career researchers would learn from the experience of established researchers, whose research is probably to achieve high scoring impact [10]; however, there is little evidence from the present findings that this learning is taking place. ECRs in this study report minimal mentoring, lack institutional support and are not proactively seeking personal advice from others.
The interviewees are thinking about impact and its importance for their future careers. Some spoke with passion about the vulnerable groups with which they worked and their aspirations to make a difference, confirming findings in earlier research which suggest ECRs have a wider interest in reaching out to the general public [11, 20]. However, it is unclear how well they are prepared to create and capture impact under the current REF regime. There was a lack of understanding of how to facilitate impact and a lack of knowledge of the scope and supporting evidencing required for a REF impact case study. Respondents had notable gaps in their understanding of REF impact and describe themselves as not knowing how to play the game in line with findings in earlier research [22, 23]. This is and should be of concern to the discipline, as is the apparent lack of mentoring and personal advice available to early- and mid-career researchers. It paints a compelling picture of a subject facing a dearth of authoritative sources of advice and expertise, where more could and should be done to bring together the knowledge of the discipline as a whole on impact. LIS has the capacity to demonstrate and evidence real impact, but this will only be achieved if development needs of those in the earlier stages of a career in the discipline are addressed.
Equally, it is of some concern that respondents had typically not been engaged in REF 2014 and had not subsequently informed themselves about its outcome. This would suggest a sense of distance, disenchantment and perhaps even indifference. Many appeared to be content not to know what their institution’s strategy was/is in relation to REF, and there was a degree of fatalism and a sense of ennui with ever changing demands. Support for ECRs in the small sample here was not evident, and institutions were not following guidance on how to support ECRs to respond to the impact agenda [22–25].
However, and despite such detachment, the majority of the early- and mid-career researchers who participated in this study have research aspirations specifically with reference to impact. They feel passionately about making a difference, but the research community needs to find ways of making research impact as a concept compelling and engaging and tackle perceptions of it as merely another hoop through which researchers must jump.
The next step for the authors will be to continue this exploration, working with practitioners to determine which data will be of value to them. In a subject with real synergies between academics and their practitioner analogues, if we can fill the gap between what academics want to achieve from impact and what society, commerce, education and the profession need to be achieved that would be impactful, LIS could as a discipline stake a real claim for the value to society of our research. Such a study would underpin a mapping of potentially high-impact, high-value research priorities that the academy might draw on as inspiration for future engagement.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.