Beyond REF 2014: The impact of impact assessment on the future of information research

Abstract

The importance of demonstrating value for money in terms of academic research beyond the walls of institutions grows stronger as demonstrated by the inclusion of impact assessment in the 2014 REF (Research Excellence Framework) exercise for UK higher education institutions (HEIs). To understand if such focus is influencing the library and information science (LIS) discipline, this paper reports a critical examination of impact case studies submitted to REF 2014 under the Communication, Cultural and Media Studies, Library and Information Management Unit of Assessment. Content analysis was conducted on 25 case studies submitted by 14 institutions, establishing the methodologies, impacts, beneficiaries, published outputs and corroborative evidence reported. The implications of impact assessment on future LIS researcher behaviour, in terms of research conceptualisation and design, were explored through nine qualitative telephone interviews. While individual researchers did not anticipate their behaviour to change due to the introduction of impact assessments, there are anticipated changes across the discipline including a greater focus on engaging with stakeholders and research beneficiaries at early stages of research design and an emphasis on mixed methodologies to maximise the power and consequences of research results.

Keywords

Research Excellence Framework impact impact assessment

1. Introduction

Publicly funded bodies are under increasing pressure to demonstrate the effects of their work; as a result, higher education institutions (HEIs) are entering into ‘a new social contract’ [1: 112] between research and society, which requires evidencing of economic, social and cultural value of research through ‘impact assessment’. The assessment of research impact in UK universities is now part of REF: the current national framework exercise replacing RAE (Research Assessment Exercise) which last took place in 2008. The outcomes of REF not only inform funding bodies about their grant allocation, but they provide accountability for the investment of public funds. Internationally, similar research assessment exercises are being piloted, for example, in Australia [2].

An impact component of research assessment in the UK appeared for the first time in REF 2014, executed through institutional submissions of impact templates which detailed the whole unit approach to creating impact, and case studies which provided detail of specific examples of impact. This new inclusion was to allow the funding bodies to: ‘identify and reward the impact that excellent research has had on society and the economy, and to encourage the sector to build on this to achieve the full potential impact across a broad range of research’ [3].

Impact in REF 2014 was defined as ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ [4: 26]. The case studies from REF 2014 are now publicly available, allowing other researchers insight into the approaches taken to impact in terms of its achievement, demonstration and evidencing and these will form the focus of this paper. Case studies were to be no longer than four pages long and include: a summary of the impact; a summary of the links between the research and the impact; evidence of the high quality underpinning research (at least 2*); further details of the impact; and sources to corroborate the impact.

Additionally, the guidance suggested that the REF 2014 panel were looking for both ‘reach’ and ‘significance’ in relation to the impacts described in the case studies. The reach element, which is not necessarily geographical, was expected to show, for example, that the research has extended significantly beyond its original scope and had reached further beneficiaries. The significance element was concerned with the scale of the change or benefit resulting from the research, and to explore what this meant for the beneficiaries. The qualifying period for an impact occurring was between 1 January 2008 and 31 July 2013, and the qualifying period for underpinning research included publications from 1 January 1993 to 31 December 2013. This meant that institutions potentially had two decades of research from which to retro-actively trace impacts. However, a non-linear route to impact was allowed by the REF 2014 impact case studies. For example, it was permissible for the underpinning research to have been published after the impact happened. Essentially, the impact was considered to be linked to a body of research and a research team, rather than to specific outputs.

The weighting of impact assessment in REF 2014 was 20% with recommendations to raise this to 25% for the next assessment which will take place in 2020 or later [5]. HEIs were only required to provide a limited number of case studies per full-time equivalent (FTE) submitted to the exercise (2 impact case studies for up to 14.99 FTE; 3 for 15–24.99 FTE; 4 for 25–34.99 FTE; 5 for 35–44.99 FTE; and 6 plus 1 further for every additional 10 FTE for 45 or more FTE) [5: 28]. Despite this small proportion of required ‘impactful’ research for REF purposes and by REF definitions, the authors would argue that a wider demonstration of ‘value for money’ from research will become more important with research being exposed to greater scrutiny as access to funds becomes increasingly competitive [1]. The somewhat recent introduction of impact to UK research assessment in particular [see for example, 6 –8], means there is much learning to be done in the facilitation of impact and the writing of case studies to showcase this. Guidance exists on the achievement of impact from Research Councils UK and the individual research councils themselves; however, the compiling of a case study in a fluent narrative with corroborative evidence is a new and bureaucratic process and there remain challenges around identifying, measuring and evidencing research impact.

Previous indicative content analysis has been conducted by the research team on ESRC impact case studies in business and management, through exploration of the methodologies used in case studies, the published outputs of the projects, the dissemination activities undertaken and the reported impact [9]. The research findings suggested there can be a variety of approaches taken to enhancing the reach and dissemination of research outputs but a limited range of impacts are reported: largely impact on government, research partners and practitioners [9]. Results from the previous study suggest that impact requirements may influence research design in particular in terms of ‘early and continuous stakeholder engagement’ [9].

For the current research project, systematic content analysis was carried out on the case studies submitted to Unit of Assessment (UOA) 36 – Communication, Cultural and Media Studies, Library and Information Management – with a focus on submissions from the library and information science (LIS) discipline. The research team identified: the common research methodologies employed in reported impact cases; the range of impacts commonly reported in the cases; and how the reported impact is evidenced and corroborated. Subsequently semi-structured research interviews were conducted with key academics who contributed to the relevant case studies. The research team sought to interview at least one member of staff from each participant institution in order to gain reflections on the experience of evidencing impact for REF 2014. The interviews cover specific issues in relation to research direction, research strategy and research design.

It is anticipated that the findings will suggest future research directions in LIS focused more specifically on researching and facilitating quality driven interactions between people and/or organisations and information, as this is where real, demonstrable impact could be achieved. The results will facilitate discussion of the best corroborative evidence for impact evidencing and how this might be effectively gathered throughout a project. Stimulating creativity and openness in the achievement and identification of impact will ensure the longevity of research being funded within the LIS domain. Due to increased international attention on impact, the findings will be of interest beyond the UK and allow researchers to begin developing impact strategies in preparation for this. Development of such strategies would facilitate better engagement with information and knowledge services and increase the uptake of research findings, and ultimately contributing to the sustainability and perceived value of information research.

2. Literature review

2.1. Research metrics

While research impact has been heavily on the minds of researchers in the UK due to its inclusion in the REF 2014 exercise, there are as yet very few substantial academic articles on the REF 2014 impact element. Instead, there has been healthy debate through short opinion pieces and blogs in various fields [10].

Measuring research output more generally is, however, by no means an under-explored area of research. There is said to be an ‘obsession’ with metrics, largely bibliometrics, meaning researchers must both compete to publish in high ranking journals and ensure their work is highly cited in order to prove the worth of the research [11]. Bibliometrics are also said to be ‘at the heart of LIS’ as a method which was developed by scholars in the LIS field to quantitatively analyse documents [12]. However, bibliometrics as applied to research assessment are more problematic. The implied issues associated with this approach are evident – being published in a high ranking journal does not always lead to a large number of citations [11]. Indeed, there is evidence that in the digital age, indicators of journal excellence such as the ‘Impact Factor’ are becoming less strongly tied to the citation count of an individual paper [13]. Papers are becoming more widely available as individual electronic documents rather than being bound to the physical copy of a journal, which means they can be accessed, read and cited based on the merits of the work [13]. Additionally, simple citation counts do not indicate why another researcher has cited a piece of work [11].

It is suggested that there is a perceptible influence of this focus on bibliometric indicators of impact on researcher behaviour. With more researcher time being taken up in the lengthy submission processes of top journals, there is a move towards publishing several smaller papers [11]. With ever increasing focus on submission, review and publication, academics may be spending less time on the research itself as a result of the ‘distorted’ influence of the REF [8, 11, 14] and that the goal-orientated approach of REF may reduce more serendipitous, blue-sky research [8, 10, 15, 16]. Additionally, there is a concern that academics will focus on the research which produces the best ‘story’ for a REF impact case study, rather than tackle important issues which may be too abstract to evidence in an impact case study [17].

Another measurement system gaining popularity is ‘altmetrics’ – which uses indicators of excellence such as evidence of reach, uptake and diffusion of research [11, 18, 19]. Altmetrics are said to complement traditional bibliometric indicators rather than act as an alternative as often the metrics used, usually social web coverage, are not prevalent enough to be used as sole indicator of article impact and use [19]. However, they can give an early indication of uptake of research (through social media stats) [18] and are particularly useful for early career researchers who do not have a large body of work [18] and have not been cited by high numbers of other researchers.

2.2. REF

As outlined earlier, REF is the replacement for RAE. In addition to the inclusion of an impact element, for the LIS discipline, REF 2014 also saw a change in the composition of units of assessment, often making comparisons with the results of RAE 2008 difficult [20]. The 2014 REF exercise received submissions at a disciplinary level which were divided into 36 UOAs, each with their own sub-panel, which sit under one of four main panels, A to D [21].

While in RAE, the LIS discipline was represented largely by sub-panel 37 (library and information management), in REF 2014, the sub-panels were restructured, and library and information management became part of UOA 36 along with former RAE sub-panel 66 (communication, cultural and media studies) [20]. This new composite of former sub-panels into a single UOA was said to pose difficulties for the panel assessors, as despite the commonalities, the two RAE sub-panels have ‘substantive differences of intellectual origins, approach, and interest’ [20: 112]. Additionally, this new composite unit saw a ‘marked reduction’, compared to 2008, of research in information systems: research in this area is presumed to have been submitted to another main panel [20]. This has implications for the use of UOA 36 results as a ‘clear and accurate indicator of the fields assessed’ [20: 112] and the implications for this paper are discussed further in the limitations and further research section below.

There were 160 impact case studies in total submitted to UOA 36. Table 1 provides an overview of the impact rating given overall in panel D and specifically in UOA 36.

Table 1.

Summary of impact rating in REF 2014.

Rating	Main Panel D	UOA 36
4*	36.7%	35.2%
3*	44.4%	42.0%
2*	15.1%	17.0%
1*	3.1%	3.8%
Unclassified	0.7%	2.0%

Panels consisted of both academics and research users in order to gain perspective from outside academia [21]. Overall panel D saw impact across a range of areas: civil society, cultural life, economic prosperity, education, policy making, public discourse, public services. Across panel D, strong impact case studies tended to provide: brief but detailed summaries of activities and impacts (rather than long lists); concrete evidence of claims being made including who benefitted and how; clearly articulated research imperatives in underpinning research which were then evidenced by research outputs and robust data that explicitly reflected the relationship between the research process, finding or product and the impact achieved in the public domain; and clearly identified beneficiaries from research with concrete evidence of reach/significance to support [20]. Less persuasive case studies did not always demonstrate links between the underpinning research and benefits claimed [20].

In the overview report [20] for Main Panel D, published by REF in January 2015, it was reported that in some impact case studies the underpinning research was specifically undertaken to achieve the impacts reported, but there were also other instances of significant impacts not being planned or anticipated when research was first undertaken. However, the panel felt impact has long been an inherent aspect of research in fields represented in UOA 36 – reflecting the maturity and established character of LIS research impact [20]. They did also note, however, that in some cases there was insufficient explanation of the link between the research and the impact and the relationships between impact, knowledge transfer and dissemination [20].

2.3. Studies of REF impact

One of the few published examinations of the impact component of REF is a Higher Education Funding Council for England (HEFCE)-commissioned report by King’s College London on an analysis of the 679 impact case studies [22]. The initial assessment used text mining and qualitative analysis to explore the nature, scale and beneficiaries of impact. The study led to the development of a summary list of the areas and beneficiaries of impact which have been used to code the content of the case studies in this paper.

An earlier study by RAND, commissioned by higher education funding councils, into how HEIs were preparing for the impact component of the REF found that panellists generally felt the impact assessment allowed them to assess the case studies in a fair way, and that they were confident that the criteria were reliable [15, 23]. A separate study by the same group revealed that one effect of being a REF panellist, both for academics and research users, was an increased awareness of the diverse range of impacts generated by academic research [21].

However, academics have raised concerns about differing perceptions of the characteristics of impact both between individual members of REF panels [24], and disparity between the academics preparing the case studies and the panel members assessing them [8]. Academics have also questioned the composition and remit of the panels which consist of a large proportion of ‘powerful’ research users and dominant social groups [10]. These concerns are largely centred on who the impactful research is for – is it for, to take an example from library and information services, service users or for the senior management tasked with delivering those services? What about research which does not ‘chime’ with the expectations of the powerful research users sitting on REF panels [10]?

The RAND survey of academics revealed that the impact exercise increased ability to identify and understand impact and affirmed relationships with stakeholders [15, 23]. In fact, it found that while academics felt the impact exercise was time-consuming, external stakeholders did not find supplying evidence of impact to be particularly onerous [15, 23]. It is clear that the impact element of REF was a burden for HEIs, however. The main issues appeared to be: aforementioned challenges around the definition of impact [8, 15, 23, 24]; using the restrictive case study template and adhering to the rules and eligibility criteria [8, 15, 23]; uncertainty about what the panel will judge to be ‘good’ impact [15, 23]; and adapting to the concept of impact within different disciplines [1].

There have also been some attempts to estimate the cost of preparing the REF impact submissions although it should be noted that these estimates observed a great level of variance between institutions. Writing and training were said to take up the most time, with each case study estimated to cost around £7000 and impact template around £4000, respectively, although there was evidence of economies of scale [15]. While some reports claimed that it was difficult to ‘confidently’ separate the total cost of the inclusion of impact for REF 2014 [25], the estimate is around £55M [23]. In terms of the time cost for preparing the impact case studies, it was reported that a median of three academics were involved in producing a single impact case study, taking 18 months to complete [15].

2.4. Impact of REF on research

There is general consensus that HEIs are changing their practices as a result of the impact agenda [15, 17, 23, 26] with some claiming that it will only change researcher behaviour in the sense that academics will learn how to play the new ‘REF game’ [11, 26]. In other words: ‘it is not what we do that matters but what we are seen to do by those who count or who can be counted’ [26: 250].

Some studies of wider research impact note that applied social sciences are more suited to the type of impact assessment seen in REF 2014 [8, 27] and therefore, the LIS discipline may be seen to have had an easier job than others. For many disciplines, impact in the way it is defined in REF 2014 was difficult to evidence [28].

However, in the LIS domain, traditionally, librarians or other practitioners have contributed to the literature to a greater extent than professionals in other disciplines [29, 30]. These authors operate outside the remit of the REF assessment exercise as they often do not hold academic posts; does this mean that there will be a weaker influence of REF and the impact component introduced for the 2014 exercise than on other disciplines? The declining contributions to literature from librarians and other researchers not eligible for REF assessment suggests that perhaps this will not be the case [29, 31]. It may be that the greater effect will come from dilution of the discipline as research contributions from departments of management, computer science, communication, natural sciences and other social sciences increases [29].

The effect of REF impact implementation and the changing landscape of contributors to the research methods within the LIS domain is as yet unknown, but research methods in the discipline have been advancing over the last 20 years, becoming more varied including increase in qualitative and mixed methodologies [32]. The four most common research methods used as reported in the three journals represented in Chu’s review of LIS research methods (JDoc, JASIS&T and LISR) were ‘theoretical approach’, ‘content analysis’, ‘questionnaire’ and ‘experiment’ [32]. Further findings included the increased use of multiple methods in individual studies and a growth in qualitative research [32].

In his seminal work, Case [33] discusses the common methodologies employed in the LIS discipline. In order to find out things in our discipline there are various ways of measuring and analysing research data, and Case recognises the diversity of these employed within the discipline as the subject matter of studies are also diverse. Examples of methods used in the discipline are given, including: case study; field experiments; surveys (postal, email and web); interviews (extensive and brief); focus groups; discourse analysis; mixed/multiple methods; and meta-analysis. Case suggests these methods largely describe or explore a phenomenon, not explain it, and that in nearly all the studies presented as examples the unit of analysis is individuals.

3. Methodology

In order to deepen understanding of impact as manifest in REF 2014 and investigate the potential impact of impact assessment on researcher behaviour in the LIS discipline, a two-stage research study was designed. The study focused on the REF 2014 Panel D Unit of Assessment 36: Communication, Cultural and Media Studies, Library and Information Management (UOA 36).

3.1. Stage one

Stage one of the research consisted of a quantitative analysis of published impact case studies from UOA 36. This UOA comprised 2% of the case studies submitted overall to REF 2014 (160/6,679). The research team began by downloading all of the impact case studies which were submitted to UOA 36. Initially all BAILER (British Association for Information and Library Education and Research) institution’s submissions were highlighted, as these were assumed to be the institutions which would have case studies most relevant to the LIS discipline. BAILER includes all teaching and research staff in the Information and Library Schools and Departments in the UK and Ireland. There are 20 HEIs who are BAILER members, 15 of these institutions had submitted impact case studies to UOA 36 with a total of 43 case studies submitted overall. On further examination of the summary of impact provided in the case studies, 27 of the cases submitted by BAILER institutions 63% were excluded from analysis as they were more aligned to the media or communications aspect of UOA 36, leaving 37% aligned to the library and information management side of UOA 36. Then the research team examined the titles of all other impact case studies submitted to UOA 36 to identify submissions by non-BAILER HEIs which could be relevant to the LIS discipline. More specifically, this was done by searching for content relevant to the LIS domain, guided by the REF Main Panel D criteria and working methods [34]:

…includes research concerned with the management of information and knowledge in all formats, namely librarianship and information science, archives and records management, and information systems. This may include: research on the generation, dissemination and publication, exploitation and evaluation of information and knowledge; information policy; information media; information literacy; systems thinking; systems development; knowledge management systems; information retrieval; preservation and conservation; impact assessment; digital humanities; and historical and cultural aspects of the disciplines. [34: 83]

This approach resulted in 15 further case studies being initially identified, and nine selected for inclusion in the analysis based on the summary of impact provided in the case study. Overall this resulted in 25 case studies from 14 institutions being included in analysis for stage one of the research. While the number of case studies may appear surprisingly small, this perhaps reflects the extent to which LIS is in fact a very small research community in the UK HE context.

In order to identify the common research methodologies employed in reported impact cases, the range of impacts commonly reported in the cases, and how the reported impact is evidenced and corroborated, an analytical template was developed to use for the coding of each case study. Using Microsoft Excel, a spreadsheet was developed with rows for each case study identified, and columns for each of the features of the case study content being analysed: methodology; impact; beneficiaries; published outputs; and corroborative evidence.

Methodologies were identified inductively from the case studies themselves both through any methodological approaches mentioned in the case study and methodologies reported in the underpinning research outputs listed in each case study. The types of impact in each case study was coded using the list of impacts developed from the HEFCE report [22] on impact case studies across all panels, which was developed using topic modelling based on the ‘details of impact’ section of the case studies (see Appendix 1). Before HEFCE conducted this analysis they undertook reviews of impact frameworks and taxonomies but all were context-specific and conceptual; the resulting list from HEFCE was empirically derived. Equally, the beneficiaries of research were coded using the list of beneficiaries used in the HEFCE report [22] (see Appendix 2). The types of published outputs associated with impact case studies were coded using the output types submitted to REF (see Appendix 3). Finally, corroborative evidence used in the case studies was coded inductively from the content of the case studies.

The analytical template was tested by selecting every fifth case study listed in the excel sheet (i.e. 1, 5, 10, 15…) and the analytical tool was deemed suitable for the purpose of quantitative analysis of the case studies in that it captured all of the data required by the research team. Analysis was then conducted on each case study in turn which involved reading each case study at least twice and consulting the underpinning research associated with each case study.

It is worth mentioning here that the research team were not seeking to judge the quality of impacts stated in case studies, or whether certain impacts could be interpreted as better than others. The research team were interested in describing the impacts presented by researchers submitting to UOA 36 in REF 2014.

3.2. Stage two

Post analysis of the impact case studies, research interviews were conducted with key academics whose impact case studies were analysed in stage one of the research, allowing for open exploration of the process implemented by differing institutions and academics in their first encounter with impact in terms of research assessment. Key academics in this instance are defined as those who were ideally both heavily involved with the writing of the case study and who had led the bulk of the featured impactful research, and were therefore able to comment fully on the process. Initial discussions with the academics mentioned in the relevant case studies were held. Usually the key researcher was nominated by others named in the case study as the ‘authority’ on its development, with one case study requiring two interviewees who were deemed by those involved with the case study to be equally ‘key’. The research team sought to interview at least one member of staff from each institution whose case study had been included in the analysis. There were 14 institutions’ case studies included in the quantitative analysis and interviews were conducted with nine academics from eight of those institutions. Each academic interviewed was the main author of an impact case study. Overall this meant that 57% of the institutions included in the quantitative analysis were represented in the interview stage.

Interviews were conducted between May and June 2015 and were semi-structured. In order to understand research direction, strategy and design in relation to REF and impact assessment, the interview schedule was split into three broad sections: reflections on submitting a case study to REF 2014; the influence impact assessment may have on individual researcher behaviour; and the influence impact assessment may have more holistically on the LIS discipline.

The research interviews were conducted via telephone by one member of the research team. Each interview was recorded with interviewee permission and assurance of anonymity. Interview audio files were then transcribed and coded by the interviewer, and checked for validity by another member of the research team.

4. Findings

4.1. Stage one findings: Case study analysis

Case studies were analysed to identify the methodology, impact, beneficiaries, published outputs and corroborative evidence reported for REF 2014.

4.1.1. Methodologies

From the 25 case studies, 35 ‘methodologies’ were identified. Included in the methodologies are ways which researchers gathered, measured and analysed data. Literature review, theoretical work and interviews were the most commonly used methodologies in research reported in the REF 2014 impact case studies analysed. A summary of all methodologies identified and their frequency is provided in Table 2. The results illustrate the wide diversity of methods used in LIS research, in line with Chu [32] and Case [33].

Table 2.

Summary of methodologies identified in case studies submitted to UOA 36.

Methodology	Case studies (n)	Methodology	Case studies (n)
Literature review	13	Stakeholder analysis	1
Theoretical work	11	Action research	1
Interviews	10	Round table	1
Case Study	9	Impact studies	1
Survey	9	System design	1
Questionnaire	8	System evaluation	1
Qualitative	6	‘Unconference’	1
Domain review/summary	5	Discourse analysis	1
Focus groups	5	Systematic analysis	1
Quantitative	4	Use of advisory panel	1
Observation	4	The Delphi technique	1
Seminar / colloquium	4	Deep log analysis	1
Workshop	4	3D scanning	1
E-research	2	Technology assessment	1
KTPs	2	Text/data mining	1
Desk research	2	Data modelling	1
Discussion forum	2	Algorithm development	1
Feature analysis	1

4.1.2. Research impact

The post hoc compiled list of impacts taken from the HEFCE report [22] (see Appendix 1) contained 60 types of impact. The research team included an ‘other’ code so there were 61 potential types of impact to be coded in the impact case studies. This inclusion was in order to code some of the more LIS-focused categories which were not a natural fit with the high-level REF categories. Multiple types of impact were found in some case studies, and all types of impact where present were counted, rather than assigning only one type of impact to each case study. Twenty-six areas of impact were identified, and the most common areas of impact were historical archives, informing government policy, and cultural and heritage preservation. A summary of the impact identified and their frequency in the case studies is provided in Table 3.

Table 3.

Summary of impact identified in case studies submitted to UOA 36.

Impact	Case studies (n)	Impact	Case studies (n)
Cultural and heritage preservation	7	International development	1
Historical archives	7	Nature and conservation	1
Informing government policy	7	Oil and gas	1
Schools and education	4	Pharmaceuticals	1
Arts and culture	2	Print media and publishing	1
Business and industry	2	Scotland	1
Community and local government	2	Transport	1
Europe	2	Water and flood management	1
Museums and exhibitions	2	Work, labour and employment	1
Software development	2	Other:
Technology commercialisation	2	Libraries	4
Children, young people and families	1	Funding bodies	2
Crime and justice	1	Digital preservation / digital asset management	2
Democracy and political engagement	1	Records management	1
Film and theatre	1	Researchers	1
Healthcare services	1

4.1.3. Beneficiaries

The list of impact taken from the HEFCE report [16] (see Appendix 2) contained 37 beneficiaries of research. The research team again included an ‘other’ code so there were 38 potential beneficiaries to be coded in the impact case studies. Eighteen reported beneficiaries of research were identified in the case studies. Workers, policymakers and businesses/companies were the most commonly cited beneficiaries of research. For the purposes of this study, the category of workers includes library and information practitioners, and companies and businesses were brought together into the same category. A summary of the beneficiaries identified and their frequency in the case studies is provided in Table 4.

Table 4.

Summary of beneficiaries identified in case studies submitted to UOA 36.

Beneficiaries	Case studies (n)	Beneficiaries	Case studies (n)
Workers	15	Teachers	1
Policymakers	12	Writers	1
Companies / businesses	11	Charities	1
Governments	7	Farmers	1
Communities	5	Other:
Curators	4	Organisations	4
Students	3	Museum visitors	2
Museums	3	Professional bodies	2
Citizens	3	Academics	1
Children	1	Broadcasters	1
Patients	1	Library users	1
Schools	1	Data users	1

4.1.4. Published outputs

The list of published outputs used to code the impact case studies was taken from the types of published outputs submitted to REF (see Appendix 3). There are 20 potential types of output listed and the research team included a code for ‘other’, so there were 21 in total. Twelve published outputs were identified in the impact case studies. Journal articles were most common types of output reported in the impact case studies. A summary of the published outputs and their frequency in the case studies is provided in Table 5.

Table 5.

Summary of published outputs identified in case studies submitted to UOA 36.

Published outputs	Case studies (n)	Published outputs	Case studies (n)
Journal article	23	Patent / published patent application	1
Chapter in book	11	Software	1
Research report for external body	11	Website content	1
Conference contribution	9	Other:
Authored book	5	Details of awarded grants	10
Edited book	3	Details of industry sponsored research projects	1
Exhibition	2

4.1.5. Corroborative evidence

Corroborative evidence listed in the impact case studies was inductively coded, resulting in 36 types of corroborative evidence being identified in the case studies. Personal statements and reports were the most commonly cited corroborative evidence in the impact case studies. A summary of the corroborative evidence and their frequency in the case studies is provided in Table 6.

Table 6.

Summary of published corroborative evidence identified in case studies submitted to UA36.

Corroborative evidence	Case studies (n)	Corroborative evidence	Case studies (n)
Personal statement	20	Event website	1
Reports	14	Examples of work conducted in advisory role	1
Academic work citing research	8	Professional body recognition	1
Outputs linked to on website	6	Book review	1
Corroborative statistics	5	Final report review	1
Media coverage	5	Audio transcripts from focus groups	1
Citation in government report	5	Training materials	1
Blog post	4	Conference programme	1
Letter / email confirming role appointment	4	Guidelines	1
Presentation	4	Final recommendations from UN conference	1
Published book information (incl. sales stats)	2	Invitation to present	1
Citation of work by national organisation	2	User survey results	1
Expert witness evidence / commission submission	2	Meeting minutes	1
Evidence of contribution to organisation	2	Business plan	1
PhD Thesis / research in progress	2	Video	1
Award nomination / win	2	Confidential tender	1

4.2. Discussion on stage one findings

The prevalence of both literature review and theoretical work in the methodologies identified in work underpinning the impact case studies submitted to UOA 36 is interesting. This could suggest that reflective work, rather than primary research underpins demonstrable impact, or that this type of research is very prevalent in LIS. It could also be indicative of institutions adherence to the rule of submission to REF 2014 and the definitions given for what could be included in the impact case studies and may therefore not be reflective of all LIS research impact in the UK. On initial contemplation this was a somewhat unexpected outcome from the research findings and would be an area worth revisiting after the next REF exercise, or even earlier, to establish if this is a continuing trend for REF impact, or indeed for LIS research more holistically.

That historical archives, cultural and heritage preservation and informing government policy were areas of LIS research having an impact was not necessarily surprising to the research team given the prominence of information and library focused research funded by the AHRC. The findings are valuable from a general perspective of what the discipline does in terms of its research and where the discipline may go in the future. There does however remain a question as to whether the impact reported in the impact case studies for REF 2014 was impact that was easily demonstrable by the submitting institutions within the confines of the guidelines. It could be that there was research having impact in differing areas but without the evidence and underpinning research outputs to back it up required for a case study submitted to REF 2014.

The range of stakeholders impacted by research from the LIS discipline was unsurprising, but this may evolve in the next REF exercise. Information researchers may also reflect on whether there are any gaps or opportunities arising from the present focus.

Unsurprisingly journal articles were the main outputs listed in the impact case studies, but there was also variety beyond these in the types of published outputs associated with impact reported. The significance of research reports for external organisations is unsurprising given the definitions of impact for REF 2014 and the need to demonstrate impact beyond academia. A further area of research may be in establishing whether commissioned or funded research is more impactful; commissioned research may satisfy a need from a user community, whereas funded research may be an idea spawned by a research team as they perceive a need of a user community. The requirement of funding applicants to map their anticipated impact and evidence impact post study may influence this. Equally commissioned research is undertaken for a real purpose and that purpose remains a necessary priority for the research team, and therefore the link to impact is more immediate.

Finally, in terms of the corroborative evidence the use of personal statements in high volumes was predictable given the retrospective nature of impact in REF 2014 and the relative ease by which these could be obtained. There was a wide range of materials used as corroborative evidence and could possibly reflect a lack of understanding of what was expected by the panels here. The relative significance of media coverage is also interesting as this would evidence the reach more than the impact of research. Going forward into the next REF exercise one would expect to see a change in the types of corroborative evidence used in the impact case studies, potentially with a greater focus on documentary evidence.

Overall there were some surprises when conducting quantitative analysis on the impact case studies, as well as some issues which would merit further exploration as described above. There were a variety of approaches taken to constructing the content of each section, but there remained uniformity to an extent over what was being reported. This could be due to the limitations in understanding of what constituted impact in terms of the REF submission and assessment by the REF panels, and what they expected to be included in the impact case studies. Equally there could have been uncertainty among those tasked with constructing impact case studies, leading to a conservative approach being taken to the content of case studies.

4.3. Stage two findings: Research interviews

Research interviews began with questions on the REF 2014 process and interviewees were asked about their experiences of being involved in a case study for submission to REF 2014.

4.3.1. Case study development

Generally, case studies were identified for development at a departmental or faculty level. More case studies than required for the submission would be identified by highlighting research which demonstrated departmental strengths, different types of impact or had attracted funding. In one instance, research selected for inclusion in a case study was described by the interviewee as ‘self-selecting’ based on a long-standing collaboration with industry organisation. REF coordinators or individuals were appointed as responsible for the case studies in an individual institution and would oversee the process. The majority of interviewees had been involved in the selection process, however one interviewee, somewhat surprisingly, was not involved in the process whatsoever and was unaware their work had eventually been submitted to REF 2014. Two interviewees felt that it was unclear what an impact case study should look like.

When it came to writing the case studies, overall the process tended to be long and iterative. There was often a university or departmental team in place to review drafts of case studies, with each approach reported by interviewees being different depending on team composition and the institutional approach. There were some instances of external review of the case studies as well as internal review. External review was conducted by other academics from either the same or differing disciplines, and internal review was in one instance conducted by the communications office of an institution. Overall there was a sense from interviewees that feedback received internally was opinion led as a lack of understanding of what REF panels would be seeking pervaded. This was described as the ‘blind leading the blind’.

4.3.2. Experiences of case study involvement

One interviewee felt they had developed a greater understanding of what impact means, another felt they were now alert for and recorded impact, with yet another interviewee asking ‘what’s the impact?’ at an early stage in project development. One academic felt the exercise was a useful way to showcase their work to a wider audience, and another felt the exercise had helped them to become less modest and have the ability to write about their work without the perpetual addition of caveats. Others found the process less rewarding; one academic felt they had learnt how difficult it can be to translate impact into something coherent and comprehensible, while a second felt the process taught them to ensure tighter ownership of their research outputs. The issue of using the same measurement tools for different disciplines was raised in terms of impact assessment where, for example, the number of people who have directly benefited from the introduction of a new drug was seen to be much more demonstrable than, say, qualitative research on the information-seeking behaviour of office workers and it was felt by some that the LIS domain would have a harder task evidencing these types of impacts. In terms of the REF 2014 process and impact definitions, one academic felt their institutional approach could have been different in terms of writing the impact case studies, with a more uniform approach to their compilation by professional writers. One interviewee felt that the definition of impact used by REF 2014 was failing to capture a whole area of impact: that of the research process itself.

Going through the process of producing a REF 2014 impact case study had no immediate influence on the behaviour of four interviewees. Two others were now designing impact into research studies. Other interviewees felt the experience had influenced their behaviour in terms of: being able to articulate impact; getting an important tick on their CV; realising the value of dissemination; realising the value of recording how your research has been used in differing contexts; collecting and filing evidence of impact; and becoming more involved in the REF process at a senior level within their institution.

4.3.3. Research direction and design

Eight interviewees felt their future research direction would not alter in order to facilitate impact, with two interviewees highlighting that they did not feel any special impetus to facilitate impact in the first place as their work was already impactful. One interviewee felt they would now question themselves more around why they are interested in conducting a certain piece of research, and one other interviewee felt their research direction would actually alter in order to maximise future impact. Probing deeper, five interviewees did not feel the way they designed research would be affected by the focus on impact in research assessment.

In terms of research stakeholders, one interviewee did feel they would now get people involved in their research at an earlier stage, one interviewee was beginning to think about how they could engage with an impact on funders and policymakers, while another will now identify participants from outside Higher Education and focus on gaining as much demographic information as possible to allow for alternative analyses. Mixed methods research design was highlighted by one interviewee as being particularly useful in maximising impact and therefore a future area of focus. This is in line with the findings from Chu’s study of LIS journal articles on the rise of mixed methodologies and qualitative in published LIS research [32]. Another interviewee indicated there would be a slight change in topic for some of their research because of greater consideration of the ‘why’ question. On a practical level one interviewee would be paying closer attention to the ‘pathways to impact’ on funding application and another would take a more careful approach to publication, choosing carefully where to publish in order to ensure underlying research publications for impact case studies existed. This is in line with the earlier RAND report which found that participants felt that impact should now be approached more strategically [15, 23].

4.3.4. Research at a discipline level

Six interviewees were either not sure or felt there could be the possibility of new areas or topics becoming more attractive to researchers due to perceived high levels of impact attainment in those areas. One interviewee identified a potential change in the precise focus of some people’s work, and another felt there would be a marriage between areas where there exists both a need (in terms of a community) and the expertise (in terms of researchers). There exists a dichotomy as to whether research will become more or less interdisciplinary; one interviewee felt interdisciplinarity would become more prominent, another felt there would be an increased pressure to become more disciplinary in order to ensure impact is achieved among research and practitioner communities. Others questioned how to maintain identity of LIS in such a varied UOA, and stated that there should now be a debate on the direction that information research should take.

Two interviewees felt that methodologies across the discipline would not alter in order to achieve greater impact. Other interviewees identified a variety of ways they could see the discipline changing: there being an increase in participatory design, co-production with communities, active participation with communities, and generally seeing a greater adoption of ethnographic and participant observation methods; an increased openness in general across the discipline in taking the appropriate methodologies and solutions from a wide area; and an increase in mixed methods with a greater need for quantitative data collection techniques alongside the more common qualitative. One interviewee felt there would not be a change to methodologies but there would be a change in the questions asked by researchers, and to whom they were addressed. Another interviewee identified a potential change in attention to and resources for dissemination and outreach activities, but not the methodologies themselves.

Again, two interviewees did not feel that stakeholder or participant identification would change across the discipline to facilitate impact. Other interviewees reiterated the change in relationships with communities, suggesting that relationships would be closer and stakeholders would be engaged with at the research design stage. Recognition was given of the application of research beyond campus walls being the cornerstone of impact by one interviewee, and others reinforced this by suggesting that there could be an increased desire in future to work with significant, well established institutions (beyond academia) and that greater attention will be paid to industry as an audience or active participant in research. One interviewee interestingly pointed out that in order to facilitate impact beyond academia, there may be an increased divergence between the intellectual domain of the discipline and the practical domain of the profession, because of the focus on skills development rather than education by professional bodies.

The management of research projects in order to enable the identification, measurement and evidencing of impact was expected to change. At institutional levels there were discussions ongoing as to whether data collection should be centralised or collected at a departmental level. The resource implications of this were central to the debate. One interviewee identified that there had already been substantial investments at an institutional level to support the identification, measurement and tracking of the impact of research. One interviewee pointed out that the need to give more thought to demonstrating how things have changed as a result of what researchers have done was not necessarily easy or a positive thing. Two interviewees suggested there would be changes to the way data were collected and how evidence of impact was gathered for projects, and two interviewees pointed out that as REF 2014 was about retrospective impact gathering there would be a change to the approach from now on as evidence of impact could be gathered right from the start of a project.

4.3.5. Other changes in research agendas

Three interviewees highlighted concerns surrounding the resource implications of the REF and impact assessment, including the time burden on both institutions and those on the REF panels and another interviewee commented on the burden of reporting. Two interviewees felt there would be a change to the REF ‘goal-posts’ and that where to publish research may be more tightly managed in future. One interviewee anticipated a greater focus on enterprise activity, with another deploring the popularisation of scholarship through the media.

At a departmental level one interviewee felt there was more structured encouragement in prepping for the next REF, and another interviewee felt that the move towards interdisciplinarity and collaborative research would not just be external but between departments of the same institution. The potential for some institutions to become ‘super-schools’ rather than research active because of REF and impact assessment was highlighted by one interviewee. More broadly comments were made by individuals that: impact is being spoken about much more than ever before, showing the importance of the exercise, especially in terms of getting funding; changes in research agendas are more to do with the overall funding environment than the REF exercise; and that REF is narrowing the possibilities of what is acceptable research.

5. Limitations and further research

The interviewees for this research were largely senior researchers. An area of future research may be to establish any influence impact assessment may have on the behaviour of either early career researchers or established researchers who were not included in an impact case study in 2014. For example: will early and mid-career researchers encounter challenges in having their work considered for an impact case study in future REF exercises? Does their relative exclusion from the submitted 2014 case studies mean that they are less informed about REF impact and will struggle in subsequent exercises, or will they have absorbed some of the learning despite being less included? Are these early and mid-career researchers in a better position to respond to the new impact requirement than those senior researchers who had to track impact retrospectively?

The involvement of more interviewees would have strengthened this research in order to allow reflection upon a wider range of experiences. However, it was found that some academics were unwilling to comment in such detail on their case study. This highlights the sensitive nature of the REF impact exercise and the sometimes very personal and private nature of REF impact case studies. It is hoped that a less ‘intrusive’ or broader follow-up study would overcome this issue and foster more participation. The authors were not particularly surprised at this small number of case studies which fit the criteria. As mentioned in the literature review there was a ‘marked reduction’, compared to 2008, of research in information systems [20] submitted in 2014 to the new panel described as including the ‘library and information management’ field. There was also mention of a ‘large volume’ of submitted material which lay beyond the remit of UOA 36, and which was referred to other sub-panels for expert advice [20]. A further area of potential future research would be to cast the net wider by searching for LIS case studies submitted to other REF panels. It is acknowledged that by focusing on UOA 36 in panel D, the authors may have missed the opportunity to include impact case studies on information systems, for example, which could have been submitted to another REF panel. However, the authors chose to focus on the UOA 36 only for the purpose of this paper, as it represents the main body of LIS work and kept the scope of the paper manageable while achieving the necessary depth of investigation. A broader, shallower, study incorporating more case studies which may be considered to belong to the LIS domain would provide an interesting comparison with the current study.

There is also the issue of the limited scope of analysing only the submitted impact case studies and making conclusions about the direction of LIS research. The researchers featured in case studies were the ‘success stories’ which fit the REF criteria and were therefore put forward for the REF 2014 submission. What would be the influence of research which is still impactful, but was not included in REF 2014 as it lacked evidence of impact, corroborative evidence or lack of suitable outputs? What would be the influence of research which is not impactful at all by REF definitions, both on individual researcher behaviour and the research supported by HEIs? These questions are beyond the scope of this paper, but in reviewing the perceptions of those who were included in REF 2014 and analysing the submitted case studies, the authors hope to explore the influence of the ‘impact agenda’. It could be said that the analysis presented here is more telling about this game-playing and the tactics associated with REF, rather than the wider development and nature of information research. Despite this, the authors would argue that there are a number of factors influencing the development of the discipline, and exploring the effects of a new element to national research assessment exercise is certainly a valid endeavour, if for no other reason than to rule out any effect. It is also a fact that the case studies selected by institutions will reflect their estimation of their ‘best’ cases and are likely therefore to send a message to other researchers in the institution about forward trajectory. There is now an opportunity and a necessity for LIS to debate and reconsider the future disciplinary evolution in order to ensure the future strength of the discipline – in a world where impact is likely to become ever more important.

6. Discussion and conclusion

The research has successfully gathered data allowing insight into the creation of LIS case studies for REF, identifying lessons learned by those involved and identifying ways in which research strategies might be influenced in the future. At the outset of this research the team were interested in answering several questions relating to the introduction of impact to research assessment: will the research direction of individual researchers alter in order to facilitate impact? Will new areas or topics become more attractive due to perceived impact attainment in these areas? How will research design be affected by an increased focus on impact in research assessment? Will research methodologies change? Will stakeholders or participants be identified in a different way? Will there be a change to the way projects are managed to enable the identification, measurement and evidencing of impact? The research findings suggest a selection of learnings from the experiences of academics being involved in the development of an impact case study for REF 2014, with no two experiences identical and some being more positive than others. Academics interviewed did not initially predict much change in their individual behaviour. Overall, however, it is clear from the results that LIS researchers involved in the case studies are now more mindful of impact at all stages of the research process and envisage a wide array of future strategies to enhance impact and its evidencing. There was also agreement among participants that there should be a debate among LIS researchers about the discipline’s future direction.

There does appear to have been a major shift in awareness of impact and the requirement for closer relationships with stakeholder communities and research beneficiaries. This is similar to the findings of the RAND report on the perceptions of those assessing REF impact [21], suggesting this is not a discipline-specific finding and, indeed, is not a shift limited to only the academics as this RAND report involved REF panellists. The separate RAND report into preparations for REF impact found similar results among academics involved in preparing case studies, where participants felt that impact should now be approached more strategically [15, 23]. The increased consideration of the audiences of research results is also in line with the findings of the earlier study into ESRC case studies [9].

There was also evidence that preparation of the case studies was onerous, both to individuals and HEIs. The stated intention of incorporating impact into the UK’s national research assessment exercise was to identify and reward impact beyond academia, and to encourage more impactful research [3]. Many of the interviewees stated that their research is already impactful, but that communicating the impact using the template for REF was difficult and time-consuming. Those involved in the RAND study also commented on the difficulty and time-consuming nature of preparing an impact case study [24].

Therefore, in one sense, for the LIS researchers interviewed, the exercise did not particularly lend itself to identifying or encouraging impact. In fact, one interviewee stated that the only thing they learned from the exercise was how to complete a REF impact case study template. However, the effect of going through the REF 2014 process and preparing an impact case study is not to be underestimated. Interviewees indicated that going forward they would now: record evidence of impact; and ask ‘what’s the impact?’ at an early stage of a new project. These findings are also generally in line with those from the RAND report [23].

In writing case studies there appears to have been a structured approach to writing, which required input from academics, external ‘non-experts’ and someone familiar with REF impact requirements. The lack of clarity around definitions of impact caused issues for those compiling the case studies, and in some institutions clearly respondents felt that internal review and advice was lacking and ill-informed. There was also a struggle to fit case studies in to a one-size-fits-all template with page restrictions. As discussed in the literature review, the panel D report that the strong impact results from disciplines in UOA 36 were not entirely unexpected, as they already have an ‘established character of impact’ [20]. However, panel D also found that there was evidence of confused presentations of impact case studies and lack of demonstration of the links between the research and the impact [20], so the point raised by our interviewees about the difficulty in demonstrating certain types of impact may be salient. It is worth bearing in mind here who was interviewed for this research project: established, often leading, researchers whose work was included as an impact case study for REF 2014. However, despite being established researchers, they still encountered challenges in translating their impact for the purposes of REF and using the case study template. Therefore, while LIS as part of UOA 36 has a long tradition of impactful research, evidencing and communicating this impact is still a difficult task, certainly in terms of the criteria and requirements for a REF impact case study. For many of our participants, the REF exercise was akin to ‘the blind leading the blind’.

Finally, it was acknowledged that not all research is impactful as defined by REF, or indeed should be, as there remains room for, and a need for, blue sky, highly theoretical, early exploratory research. Therefore, it will be interesting to see how HEIs support LIS research, even that which is not considered to be impactful enough for a REF impact case study, or which does not meet the criteria for inclusion in other parts of the REF exercise.

It is the view of the authors that while REF impact to date has been understandably predominantly focused on the ways in which LIS research is presented in the REF submissions, that the legacy of REF impact will invariably lead to change in how LIS researchers behave in the future. In particular, given the emphasis placed by the Research Councils on explaining predicted impact in funding bids, future funded research will have the ethos of impact built into research design and execution. It is equally the case that many early and mid-career researchers will aspire to form the nucleus for future impact cases and will seek to learn from and model the behaviour of previously successful researchers. Inevitably, early and mid-career researchers will seek to learn how to ‘play the game’, a game which will vary by discipline and it is hoped that the current paper has helped to shed some light on how the game was played for REF 2014. Future research might usefully explore what lessons early and mid-career researchers are taking from REF impact and how they envisage it affecting and influencing their future research plans. Equally, for LIS as a whole there is a debate to be had as to how the discipline can maximise impact and perceived value in order to build a successful future for the discipline in the context of tightening resources.

The authors acknowledge that not all research will deliver immediate impact and that arguably the most important blue sky, experimental and innovative work might find a more circuitous route to impact through the work of others. In such cases the timeline of achieving impact may be very slow and it is also therefore imperative that the discipline consider and debate how to build and support such blue sky thinking as a base for future applied research.

Footnotes

Appendices

Appendix 3.

Types of published outputs as submitted to REF 2014.

Type	Examples
(Parts of) Books	Authored book
	Edited book
	Chapter in book
Journal articles	Scholarly edition
	Journal article
	Conference contribution
	Working paper
Physical artefacts	Artefact
	Devices and products
Exhibitions and performances	Exhibition
	Performance
Other documents	Patent/published patent application
	Composition
	Design
	Research report for external body
	Confidential report for external body
Digital artefacts	Software
	Website content
	Digital or visual media
	Research datasets and databases

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

References

Molas-Gallart

Research evaluation and the assessment of public value. Arts & Humanities in Higher Education 2014; 14: 111–126.

Morgan Jones

Castle-Clarke

Manville

Gunashekar

Grant

. Assessing Research Impact: An international review of the Excellence in Innovation for Australia Trial, http://www.rand.org/pubs/research_reports/RR278.html (2013, accessed September 2015).

REF 2014. Research Excellence Framework impact pilot exercise Findings of the expert panels, http://www.ref.ac.uk/media/ref/content/pub/researchexcellenceframeworkimpactpilotexercisefindingsoftheexpertpanels/re01_10.pdf (2010, accessed December 2015).

REF 2014. Assessment framework and guidance on submissions, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf (2011, accessed September 2015).

Witty

Encouraging a British Invention Revolution: Sir Andrew Witty’s Review of Universities and Growth London, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/249720/bis-13–1241-encouraging-a-british-invention-revolution-andrew-witty-review-R1.pdf (2013, accessed September 2015).

Oswald

REF should stay out of the game. The Independent, http://www.independent.co.uk/news/education/higher/andrew-oswald-ref-should-stay-out-of-the-game-1827306.html (2009, accessed August 2015).

Dunleavy

The Research Excellence Framework is lumbering and expensive. The Impact Blog, http://blogs.lse.ac.uk/impactofsocialsciences/2011/06/10/ref-alternative-harzing-google-scholar/ (2011, accessed August 2015).

Watermeyer

Issues in the articulation of ‘impact’: the responses of UK academics to ‘impact’ as a new measure of research assessment. Studies in Higher Education 2014; 39: 359–377.

Marcella

Lockerbie

Cameron

. The Challenge of Demonstrating the Impact of Research Beyond Traditional Mechanisms. In: Proceedings of the 14th European Conference on Research Methodology for Business and Management Studies in Valetta, Malta, 11–12th June 2015.

10.

Colley

What (a) to do about ‘impact’: a Bourdieusian critique. British Educational Research Journal 2014; 40: 660–68.

11.

Gruber

Academic sell-out: how an obsession with metrics and rankings is damaging academia. Journal of Marketing for Higher Education 2014; 24: 165–177.

12.

Larivière

The decade of metrics? Examining the evolution of metrics within and outside LIS. Bulletin of the American Society for Information Science and Technology 2012; 38: 12–17.

13.

Lozano

Larivière

Gingras

The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology 2012; 63: 2140–2145.

14.

Murphy

Sage

Perceptions of the UK’s Research Excellence Framework 2014: a media analysis. Journal of Higher Education Policy and Management 2014; 36: 603–615.

15.

Manville

Jones

Henham

Castle-Clarke

Frearson

Gunashekar

Grant

Preparing impact submissions for REF 2014: An evaluation: Approach and evidence, http://www.rand.org/pubs/research_reports/RR726.html (2015, accessed December 2015).

16.

Morgan

Research impact: Income for outcome. Nature 2014; 511: 72–75.

17.

Knowles

Burrows

The impact of impact. Etnográfica. Revista do Centro em Rede de Investigação em Antropologia 2014; 18: 237–254.

18.

Dinsmore

Allen

Dolby

Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLOS Biology 2014; 12: e1002003.

19.

Thelwall

Haustein

Lariviere

Sugimoto

CR.

Do altmetrics work? Twitter and ten other social web services. PLoS One 2013; 8: e64841.

20.

REF 2014. Overview report by Main Panel D and sub-panels 27–36, http://www.ref.ac.uk/media/ref/content/expanel/member/Main%20Panel%20D%20overview%20report.pdf (2015, accessed September 2015).

21.

Manville

Guthrie

Henham

Garrod

Sousa

Kirtley

Castle-Clarke

Ling

Assessing impact submissions for REF 2014, http://www.rand.org/pubs/research_reports/RR1032.html (2015, accessed December 2015).

22.

Higher Education Funding Council of England. The nature, scale and beneficiaries of research impact: an initial analysis of Research Excellence Framework (REF) 2014 impact case studies, http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/Analysis,of,REF,impact/Analysis_of_REF_impact.pdf (2015, accessed September 2015).

23.

Manville

Jones

Frearson

Castle-Clarke

Henham

Gunashekar

Grant

Preparing impact submissions for REF 2014: An evaluation: Findings and observations, http://www.rand.org/pubs/research_reports/RR727.html (2015, accessed December 2015).

24.

Samuel

Derrick

GE.

Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation 2015; 24: 229–241.

25.

Farla

Simmonds

REF Accountability Review: Costs, benefits and burden, http://www.technopolis-group.com/wp-content/uploads/2015/11/REF_costs_review_July_2015.pdf (2015, accessed December 2015).

26.

Khazragui

Hudson

Measuring the benefits of university research: impact and the REF in the UK. Research Evaluation 2015; 24: 51–62.

27.

Upton

Vallance

Goddard

From outcomes to process: evidence for a new approach to research impact assessment. Research Evaluation 2014; 23: 352–365.

28.

Gray

Has the Research Excellence Framework killed creativity?

Journal of Psychiatric and Mental Health Nursing 2015; 22: 155–156.

29.

Walters

Wilder

EI.

Disciplinary, national, and departmental contributions to the literature of library and information science, 2007–2012. Journal of the Association for Information Science and Technology, http://onlinelibrary.wiley.com/doi/10.1002/asi.23448/abstract (2015, accessed 29 September 2015).

30.

Sugimoto

Tsou

Naslund

Hauser

Brandon

Winter

Behles

Finlay

SC.

Beyond gatekeepers of knowledge: Scholarly communication practices of academic librarians and archivists at ARL institutions. College & Research Libraries 2014; 75: 145–161.

31.

Finlay

Tsou

Sugimoto

CR.

Publish or practice? An examination of librarians’ contributions to research. Libraries and the Academy 2013; 13 (4): 403–421.

32.

Chu

Research methods in library and information science: A content analysis. Library and Information Science Research 2015; 37: 36–41.

33.

Case

DO.

Looking for information: A survey of research on information seeking, needs and behavior. Bingley: Emerald Group Publishing, 2012.

34.

REF 2014. Panel criteria and working methods: Part 2D Main Panel D criteria, http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf (2012, accessed December 2015).