Exploring research impact in the assessment of leadership,governance and management research

Abstract

Research is increasingly under the spotlight to demonstrate impact as well as ‘World Class’ quality. Impact measures were introduced into the United Kingdom’s Research Excellence Framework in 2014, and are being adopted in other countries. However, impact is a concept that is both loosely applied and often contested. It needs unpacking to build understanding about how it can be effectively evaluated. This article uses findings from a subsample of 1309 research-based case studies in leadership, governance and management submitted to the Research Excellence Framework. The mixed-method study used Complex Adaptive Systems as a lens to explore perspectives of impact as a consequence of research, as a process and as an emerging concept. We describe some of the rich patterns of impact practices, mechanisms for exchange, connections with context, and types of measures, used to evidence impact. The article helps to illuminate the complexity of impact and implications for its evaluation.

Keywords

impact pathways leadership management research evaluation research impact

Introduction

In a milieu of political and public interest in the effectiveness and value for money of government-funded institutions, academic research is increasingly under the spotlight to demonstrate impact as well as ‘World Class’ quality (Altbach and Salmi, 2011). For the first time, impact measures were introduced into the United Kingdom’s Research Excellence Framework (REF) in 2014 and will carry even more weighting (20 to 25%) in REF 2021. The results of the REF 2014 will govern the award of £27 billion to higher education institutions (HEIs) over the next six years (Hefce, 2012). Other countries globally, such as Australia and Canada, are moving towards national research assessment strategies that include impact (Grant et al., 2010). This paper explores issues to do with the formal external evaluation of research impact, as well as important related issues of evaluation for self-assessment, learning and development.

Assessing the impact of research is in principle a good thing because it values research that has a positive benefit on society. It suggests the highest rewards for research that is capable of addressing ‘grand’ challenges, life-limiting issues or costly problems (Owen et al., 2012). However, impact is a concept that is both loosely applied and often contested (Brewer, 2011; Martin, 2011). It needs unpacking to build understanding about how it can be effectively evaluated in national assessment exercises, HEIs institutional evaluations, and researcher’s self-evaluation and professional development. For example, in REF 2014 decisions were made about the reach and significance of impact by panels of assessors for different discipline-based Units of Assessment (UOA). The results of an independent review of the assessment process concluded that there is a need to clarify the process of assessing different types of impact (RAND Europe, 2015).

Owing to the influencing power of research evaluation (Brewer, 2011; Dill, 1998) it is important to develop evaluative strategies and approaches that are based on evidence of what impact is and how it occurs (Hefce, 2015). Not least because impact has major financial consequences for HEIs, both in terms of the funds they stand to gain for showing impact, and the investment costs involved in capturing and evidencing impact (RAND Europe, 2015). It has also been suggested that impact could be enhanced if HEIs are rewarded for using interdisciplinary approaches to address research issues (Hefce, 2016). In the REF, impact was used to mean the outcomes achieved for beneficiaries, whether they occur in the short or long-term (Hefce, 2012). We use the term here in its broader sense to mean the effects or influence of research on individual people, organizations or institutions and the social or longer-term impact of research.

This article uses findings from a study of impact from leadership, governance and management (LGM) research to illustrate some of the complexity of research impact and the implications for its evaluation. The reason for this focus was that the research funders (the Leadership Foundation for Higher Education) were interested in learning about impact from research in LGM. In the LGM field, there is a need for evidence about the impact of research on large complex institutions, like governments, educational institutions and health care systems (Bevir, 2013; Chadwick, 2010; Hamlin, 2010); as well as impact on the people that work within them (Atwood et al., 2010; Bass and Stogdill, 1990; Kezar et al., 2006) and the indirect impact (Greenhalgh and Fahy, 2015) on end-users and the public (Bolden, 2010). The study describes the extent and diversity of impact from LGM research. The findings, presented in a full report of the study (Morrow, 2016), can inform researchers in the LGM field about how other researchers have achieved and evidenced impact. The focus of this article is to examine the complexity of impact and implications for its evaluation.

Background

In research evaluation and research practice impact from research is most commonly thought of as the consequences of research (Donovan, 2011), such as the effect, change or influence of research on policy (Boaz et al., 2009; Oliver et al., 2014), society (Bornmann, 2013) or the economy (Deloitte Access Economics, 2012). This focuses on the ‘end-goals of research’ and how to capture and measure them (Grant et al., 2010; Guthrie et al., 2013; Hefce, 2015; Morgan Jones and Grant, 2013). Different approaches to evaluating impact include the REF assessment (Hefce, 2012), Payback Framework (Buxton, 2011; Buxton and Hanney, 1996; Donovan and Hanney, 2011) and Research Impact Framework (Kuruvilla et al., 2007) as well as locally developed conceptual frameworks (Banzi et al., 2011). The detail of approaches is important because it defines the specific types of impact that are included for assessment and the criteria by which impact is judged (Greenhalgh et al., 2016; Penfield et al., 2014).

In REF 2014 impact was assessed according to its reach and significance, but these concepts encompass many judgements about the value of different types of impact, benefits and outcomes of research (Carter, 2013). Reach is concerned with the number of individuals or constituencies who have benefitted from the impact (Hefce, 2012). It includes the diversity of those affected and can include secondary benefits. Significance is concerned with how deeply the impact has been felt or how important it was (Hefce, 2012). For example, researchers could show the significance of the impact was substantial for individual beneficiaries concerned in terms of their health, safety or wellbeing, even if relatively few individuals benefitted.

The literature on research impact assessment offers various logic models to explain the link between research and impact (Greenhalgh et al., 2016). Existing models suggest that impact is generated through multiple interrelated processes of knowledge creation, exchange or transfer between researchers and research users (Carter, 2013; Donovan and Hanney, 2011; Kuruvilla et al., 2007). They also suggest that impact is context-specific, to the research context and to the wider political, economic and social context (Guthrie et al., 2013; Penfield et al., 2014). Thus impact is partly dependent on research users and their adaption and use of research evidence or products (Boaz et al., 2009; Boswell, 2009; Oliver et al., 2014). It relates to individual, group and organizational capacity to recognize and value new information, assimilate it and apply it to useful ends (Carter, 2013; Rogers et al., 2005).

From another perspective, increasingly impact is also being approached as the process of how it develops or occurs from research (Milat et al., 2015; Morton, 2015; Nutley and Walter, 2005; Wooding et al., 2007). For example, impact processes or pathways to impact describe how impact might be planned and maximized (LSE, 2011). Research funders now generally require researchers to submit impact plans as part of proposals – for example, applicants to the UK research councils are offered guidance about suggested pathways to impact – which identify areas of practice thought to generate impact (RCUK, 2015). Activities could include, for example: research collaboration (Denicolo, 2014; Sooho and Bozeman, 2005), knowledge exchange through forums for professional debate (Davies et al., 2008; Walshe and Davies, 2013), public engagement and public involvement (Jones et al., 2008; Oakley and Selwood, 2010; Staley et al., 2014), communications and dissemination to different audiences (Gagnon, 2011; Given et al., 2015; Milat et al., 2015; Penfield et al., 2014), or partnerships with research users to implement research evidence in practice (Durlak and DuPre, 2008; Meagher et al., 2008; Rogers et al., 2005). Relatively little is known about the range and effectiveness of different pathways to impact (Upton et al., 2014), including the most effective ways to engage non-academic groups in different research contexts (Meagher et al., 2008; Sarli et al., 2010) and to manage knowledge-based resources (Leitner and Warden, 2004). The aims of the study reported here are explained below.

Aims

In the study we aimed to explore impact as both a consequence and a process of research, but also as an emerging concept (a set of ideas, beliefs and understandings) in research evaluation and research practice. As a concept, we made the assumption that understandings about impact to be shaped by the research system and its component parts, including through discourse (e.g. research evaluation criteria and researcher’s accounts of impact) and knowledge building (e.g. research on impact and reflexivity).

The specific focus on LGM research reflects the remit and interests of the funding organization to increase capacity and excellence in LGM in higher education in the UK. The study aimed to address three research questions:

What is the nature and extent of impact from LGM research?

What are the pathways to impact?

How can impact be evidenced and assessed?

The focus of this article is on illuminating the complexity of impact and implications for its evaluation. The methods we used to explore impact are described below.

Methods

Approach

A mixed-method approach allowed us to explore perspectives, test out our assumptions about impact and generate themes from the impact case study data (Denzin and Lincoln, 2005). The approach involved drawing on systems thinking as a lens, or ontological frame (Dori and Sillitto, 2017), through which to approach impact. We drew on Complex Adaptive Systems (CAS) (Brown, 2008; Dodder and Dare, 2000) specifically, as this branch of systems thinking fits with our assumption that the research system is ‘a complex system of agents based on relationships, emergence, patterns and iterations’ (Lansing, 2003: 186). Local level practices associated with impact appear complex because they are diverse and complex (Chadwick, 2010; Dooley, 1997; Glatter, 2006; Keshavarz et al., 2010). An adaptive research system is capable of conserving information, has the ability to learn, and to organize itself (Lansing, 2003; McDaniel, 2007). In this way, learning is gained through each iteration of the assessment process (REF) and fed back into the system to develop future assessment practices (Hefce, 2015; RAND Europe, 2015).

We argue that as an emerging concept, the meaning of impact is actively developed and enacted through the beliefs and behaviour of funders, evaluators, researchers and other actors in the research system. CAS offered a way to perceive of the data (researcher’s narrative accounts of impact captured through the REF case studies) in a wider context of sense making about the object of enquiry (Dori and Sillitto, 2017). CAS also encouraged us to reflect on our own sets of beliefs, references, and commitments that we were making about impact through the study (Dodder and Dare, 2000). For example, how our interpretive sensibilities, or the worldview we apply to make sense of the data (Yanow and Schwartz-Shea, 2015), is developed through our particular experiences as researchers within a research system, interacting with other actors and actively seeking to understand impact.

A CAS perspective suggested that when viewed as a consequence of research, impact is elusive, changing, time- and place-dependent (Brown, 2008; Dodder and Dare, 2000; Levin, 2003). This corresponded with our experiences as researchers trying to track and ‘pin-down’ impact from our own research. The multiple positive and negative events and effects that occur within the research system (and its interfaces with other systems) suggest that the impact of research from any particular study, such as research on leadership development (Flinn and Mowles, 2014), cannot always be predicted or fully accounted for. The complexity associated with such interrelationships makes it challenging for researchers to capture and evidence the full impact of their research, or to attribute impact back to their research specifically (Upton et al., 2014). The challenge for evaluators is how to define and assess impact if it is elusive, changing, time- and place-dependent. A possible solution to complexity is to identify the most likely pathways to impact in different contexts (Dodder and Dare, 2000; Holland, 1999).

In relation to the issue of how impact develops, CAS provided a lens to explore multiple researcher accounts, non-linear pathways to impact, productive interactions (Molas-Gallart and Tang, 2011; Spaapen and Drooge, 2011) and iterative processes of impact formation (Dodder and Dare, 2000; Harkema, 2003; Holland, 1999), rather than a single solution to research impact. We were aware that in the UK, and internationally, cooperation, coalitions and networks of interaction have emerged around issues of research impact (e.g. Association of Research Managers and Administrators Good Practice Exchanges, Research Impact Summit and London School of Economics Impact blog), which feed back to influence research practice and organizational behaviours (Glatter, 2006; Keshavarz et al., 2010; Levin, 2003). We therefore believe that useful definitions and understandings about impact can be generated from researchers’ accounts of research impact to enable research assessors, HEIs and researchers to adapt to an adaptive research system.

The data

The REF 2014 impact case studies (6679 in total) offered the possibility of building knowledge about research impact by exploring, comparing and identifying patterns in researchers’ narrative accounts of impact and how it was achieved (Denzin and Lincoln, 2005). Case studies have been made publicly available on a searchable web-based database (http://impact.ref.ac.uk/CaseStudies/). The impact case studies are located within particular linguistic, historical and values standpoints (Miles et al., 2013) and there are limitations to the data, which we discuss later. However, the aim of the study was to explore perspectives of impact rather than to make evaluative judgements about specific claims to impact. The data exhibit consistency, as a template of four pages was used for submissions. Case studies contain the following standardized categories of information: Submitting Institution, Unit of Assessment (UOA), Title, Summary of the Impact, Underpinning Research, References to the Research, Details of the Impact, Sources to Corroborate the Impact.

Sample

A purposive sample of LGM case studies was defined using key terms to enable future replication of the sampling process (Ritchie et al., 2013). As the focus was on identifying all LGM case studies, it would not have been appropriate to use a randomized sample (Denzin and Lincoln, 2005). We selected key terms by drawing together research themes from LGM research literature and using web-based searches of 30 LGM research institutions. Table 1 shows the key terms derived from this scoping work. The authors drew on their knowledge of research in the LGM field to independently review each key term, identify any missing terms and remove unnecessary overlapping terms (e.g. using truncation of words) (Denzin and Lincoln, 2005). To perform the searches we used the search function in the REF database. Multiple searches were conducted using our defined key terms. Details of each search and its results were recorded to keep track of the search history, illustrated by Figure 1. Once found, 3874 case studies were downloaded to Microsoft Excel. After 1308 duplicates were removed, 2566 case studies remained for screening.

Table 1.

Defining the leadership, governance and management field.

Keywords and ‘search terms’	No. of case studies
leadership	768
governance	505
management (*Business and Management Studies UOA only)	314
transformative	141
teamwork	22
‘team work’	70
‘organizational development’	20
‘organizational transformation’	2
‘organizational change’	45
‘community development’	80
‘organizational innovation’	3
‘organizational policy’	12
‘community engagement’	137
managerial	85
‘corporate responsibility’	11
entrepreneurship	86
‘human resource’	61
careers	167
business (*Business and Management Studies UOA only)	338
marketing (*Business and Management Studies UOA only)	73
‘business strategy’	35
accountability	201
‘knowledge transfer’ (‘knowledge management’ searched under management)	533
mentor	63
coaching	86
‘strategic change’	14
‘systems change’	2
Total	3874

Notes:

Owing to the very high number of returns for the keywords ‘management’ (2633), ‘business’ (1979) and ‘marketing’ (591) and the high potential for false positives, these keyword searches were limited to case studies returned under the Business and Management Studies Unit of Assessment (UOA) only.

Figure 1.

Search history.

Screening (inclusions/exclusions)

The data were screened to identify LGM case studies and to remove any false positives returned by the searches. The lead researcher read and familiarized herself with the returned case studies (initially reading titles and Summary of the Impact) to build a tentative framework for inclusion and exclusion (Ritchie et al., 2013). As screening progressed, the inclusion criteria were further refined (reviewed by the two co-authors) as part of an iterative process of reading, modifying the criteria and screening (Box 1).

Box 1.

Inclusion/exclusion criteria.

(i) Included: case studies that relate to human dimensions of LGM, research involving the leadership, governance or management of people, communities or organizations, such as individuals, professional groups, communities, marginalized groups or the general public
Excluded: case studies that do not primarily relate to LGM of people, communities or organizations, such as research on: the development or use of physical spaces such as land use, buildings or bridges, water distribution, or conservation areas; the development or use of commodities such as food crops, textiles, biological pesticides, fisheries, construction materials, antibodies or pharmaceuticals; the development of technologies such as astronomical satellites, nanotechnologies, genetic diagnostics, forensic tests, rail network signaling or ticketing technologies, computer software, carbon dating systems, or emission monitoring equipment; the preservation of histories such as artefacts, human remains, sites of historical interest, collections or cultural heritage; and the study of animals, endangered or extinct species, such as their anatomy, physiology, cognition or behaviour
(ii) Included: case studies that describe identifiable research outputs (research evidence, research products or novel ideas)
Excluded: case studies that describe research outputs in very general terms (such as ‘new knowledge’, ‘insights’ or ‘substantial research’)
(iii) Included: case studies that describe impact (e.g. influence, incorporation, uptake or effect) caused by a research output
Excluded: case studies that only describe impact in very general terms (such as ‘widespread dissemination’, ‘diverse benefits’, ‘significant impact’, ‘profound effect’ or ‘changing the discourse’) or case studies that only express an intended impact (such as ‘to inspire’ or ‘to inform public understanding’)

The number of case studies included in the final sample was 1309. Case studies that did not meet inclusion criteria (n = 1257) were excluded with reasons. Data from all included case studies were extracted to new tables in Microsoft Excel for analysis.

Analysis

The assumptions about impact that we explored in the analysis were led by the research questions, informed by a review of the research impact assessment literature, and influenced by drivers of research impact such as criteria used in the REF and the UK funding council’s pathways to impact (summarized in the full report). We provide a summary of how these assumptions were explored in the data in Table 2.

Table 2.

Overview of mixed-method approach.

Research questions	Assumptions about impact explored in the data	Analytic techniques used to explore the REF case studies
What is the nature and extent of impact from LGM research?	Impact can be observed as a consequence of research activities and can be attributed to multiple levels of the system (e.g. research project, organization, region and discipline)	• Identify which HEIs are undertaking LGM research and tabulate all included case studies by HEI (using Submitting Institution) • Identify the distribution of HEIs contributing to impact across the UK by region (analysis of the location of Submitting Institution) • Tabulate all included case studies by discipline (determined by the Unit of Assessment a case study was submitted to)
	Impact displays diversity and contextual-specificity at the local level, different types of impact can be categorized according to effects, influence on different systems or entities (e.g. economic, political, social impact). The extent of impact from research can be indicated by the range of types of impact achieved	• Content analysis of all 1309 included case studies to explore types of impact from LGM research (using Summary of the Impact and Details of the Impact) • Develop a classification of types of impact from LGM research (synthesis of themes emerging from the content analysis of Summary of the Impact and Details of the Impact)
	Impact can be transient, transform or be adapted from one form to another	• Explore any associations between different types of impact identified in the data (text-mining of key words in Details of the Impact)
	The reach of impact can be indicated by the influence or effects on different actors (beneficiaries)	• Explore the range of who ‘beneficiaries’ are in the LGM context (text-mining of key words in context in Details of the Impact) • Frequency analysis of key terms to identify who most often benefits from LGM research (using Details of the Impact)
What are the pathways to impact?	Impact occurs through processes (e.g. interactions, events, practices) that are dynamic and interrelated, not fixed or linear	• Identify themes in the data about types of processes that researchers claim contributed to impact (thematic analysis of Details of the Impact)
	Processes contributing to impact simultaneously affect and are shaped by the system and are interdependent and interlinked with wider contexts such as policy, social or economic climate	• Explore ways that researchers and research connects with the research system and wider context to stimulate impact (thematic analysis of Details of the Impact)
	The influence and effects of research are not only attributable to researchers, impact is affected by other actors (e.g. stakeholders, research users) and agents can be simultaneously members of several systems (e.g. through research networks)	• Explore the main ‘stakeholder’ groups for LGM research (key word in context in Details of the Impact)• Explore how researcher roles and network membership may facilitate impact processes (content analysis of Details of the Impact)
	Impact can be generated through direct and indirect interactions (e.g. collaborations, coalitions, engagement or involvement) between researchers and other actors, through the exchange of knowledge, information or ideas and influence of behaviour	• Identify examples of knowledge exchange, learning or knowledge development between researchers and research users, including public engagement and involvement in the research process itself (identified in Details of the Impact)
	Levels of organization within the system (e.g. research units, departments, universities) can provide resources and structures that support impact to occur	• Identify different mechanisms for exchange between professionals or HEIs (thematic analysis of Details of the Impact)
How can impact be evidenced and assessed?	Impact from research can emerge at any point in the research process and cannot always be predicted. Spread of impact is influenced by levels of organization (e.g. countries and research funding) but impact can cross boundaries (e.g. geographical and disciplinary)	• Use a subsample of case studies (all case studies relating to HEI contexts identified during screening) to focus on identifying average time to impact reported for LGM research and patterns in time to impact for different types of impact• Use a subsample of case studies to focus on exploring the relationship between emergence of different types of impact and the localities and levels at which impact is reported to have occurred (local, regional, national and multinational)
	Impact is non-linear, direct causality cannot always be shown or attributed to specific research activities	• Explore patterns in how impact is evidenced qualitatively through the narrative of the case studies (thematic analysis of types of evidence used in Details of the Impact)
	Different types of information about impact can be collected using different methods and techniques, this information can be used as evidence to support claims to impact	• Identify types of impact evidence and any patterns in the types of evidence reported, note how impact data were captured including use of formal evaluation, observation, impact logs or testimonials (using Details of the Impact)
	Understandings of impact are influenced by the history of the research system (e.g. discipline, methods, fields), understandings of impact can be developed through language, learning, improvisation and self-organization	• Explore LGM impact across different disciplines (using UOA) to identify different types of measures or indicators that are used to quantify different types of impact (using key word in context for the term ‘measure’ and reading surrounding text to identify measures of impact or outcome and the stage of development)

Table 2 shows the analytic techniques that we used to explore the data. We were aware that a previous high level analysis of the REF database had successfully used text mining, albeit to explore a different set of research questions (King’s College London and Digital Science, 2015). Text mining provided us a basic structural building block to conceptual analysis using a range of techniques (e.g. frequency of key terms, codes and coded text, key word in context, near word search functions) to ‘mine’ a vast amount of text-based data in a systematic way. Text mining was supported by use of specialist text analysis software (QDA Miner) which was also used for coding and development of themes using qualitative techniques. The qualitative analyses we undertook included: content analysis (Denzin and Lincoln, 2005) to identify themes across the data, analysis of coded text to explore patterns and associations (Yanow and Schwartz-Shea, 2015), grouping coded data into taxonomies to generate classifications, and synthesis of the data (Miles et al., 2013) (summarized by Table 2). Themes were created by progressively synthesizing information – for example, lists of codes or text – to reduce the volume of information without losing meaning. Within the QDA Miner program we made use of visual displays such as flow diagrams, word clouds, charts and graphs to view and discuss the results and cross-check our interpretations with the original data.

The analytic techniques that we used required the lead researcher to make judgements, for example, about inclusions and meaning (Miles et al., 2013). Using clearly defined inclusions/exclusions and code structures helps to improve the transparency of the analytic process (Denzin and Lincoln, 2005). To check the consistency of application of codes the co-authors reviewed a sample of 20 case studies at screening stage. The analyses were reviewed by the co-authors and other researchers (two anonymous peer reviewers), and circulated for comments and discussion. We have included numerical codes (e.g. frequency and percentages) and key search terms in the findings. Our code frameworks are available for comparison and use in future research studies. The case study examples we have used can be traced back to the original sources by searching the REF database using titles and/or submitting institution. Selected results are presented below.

Results

The results are structured according to our three research questions, about the nature and extent of impact from LGM research, pathways to impact, and ways to evidence and assess research impact. We have selected key findings that particularly support the focus of this article to illustrate the complexity of impact and implications for its evaluation. Figure 2 provides an overview of the components of research impact described below.

Figure 2.

Components of research impact.

The nature and extent of impact from LGM research

Analysis of the LGM case study sample by submitting HEI identified which HEIs in the UK are undertaking LGM research and this detail is provided in a table in the full report (Morrow, 2016). A high proportion of the 154 UK HEIs that submitted to the 2014 REF (86%, n = 131 HEIs) returned LGM case studies. The average number of case studies per HEI was 8.5, with four HEIs submitting 30 or more.

The distribution of HEIs contributing to impact across the UK by region is influenced by the considerable variation in the number of HEIs within different regions (e.g. 4 HEIs in Northern Ireland versus 39 HEIs in London). By region, the high proportion of the included case studies (16%, 203) came from London, followed by the South East (13%) and Scotland (12%).

Analysis of case studies by discipline (determined by the Unit of Assessment a case study was submitted to) showed the majority of LGM case studies were from UOA for Business and management studies (n = 243), Education (n = 108), Politics and international studies (n = 94). However over two-fifths of LGM case studies were returned by other subject areas not often perceived as contributing to LGM research, including: Allied health professions, dentistry, nursing and pharmacy (n = 85), Social work and social policy (n = 80), Psychology, psychiatry and neuroscience (n = 64), Communication, cultural, media, library & information (n = 33), Clinical medicine (n = 31), History (n = 30), and Computer science and informatics (n = 28). These results show that there is substantial LGM research going on in the sector, which has largely been obscured from view until now because it cross-cuts disciplinary boundaries.

Content analysis of all 1309 included Summary of the Impact statements to explore types of impact from LGM research reveals the wide range of types of impact that are associated with LGM research. A detailed synthesis of the types of impact captured within 32 categories is provided in an annex to the full report (Morrow, 2016). Key word searches suggest the most prevalent types of impact reported are: government policy (n = 676, 52% of cases), training (n = 615, 47% of cases), impact on understanding (n = 506, 39% of cases) and strategy (n = 478, 37% of cases).

Results of the thematic analysis of Summary of the Impact data from all 1309 case studies were used to further synthesize the data to develop a classification of types of impact from LGM research, as follows:

Use of evidence (type I impact) – e.g. to inform law, government policy, strategy, regulation, standards, guidelines or recommendations for practice, or priorities for research or practice

Use of research products (type II impact) – e.g. a training programme or course, intervention (programme or initiative), toolkit, model, decision aid, system (operational), support network, technology, measure, benchmark, information resource, research method, participation method, teaching method, visual art, music or fiction

Effect on individuals (type III impact) – e.g. a change in awareness, attitude, understanding (knowledge) or behaviour

Effect on groups/organizations (type IV impact) – e.g. knowledge-transfer (intra-organization, inter-organization or network), organizational development (innovation, new systems or structures), or organizational performance (impact on end-users).

We used this classification to explore the data for any associations between different types of impact identified. Key word searches across all 1309 case studies indicate 45 per cent (n = 2865) of outcomes are type II (research products), and 34 per cent (n = 2178) are type I (use of evidence). Far fewer case studies claimed effect on individuals (16%, n = 989), or effect on groups/organizations (5%, n = 297). These findings echo observations about indirect impact (classified as type III and IV impact here) in the case study data (Greenhalgh and Fahy, 2015). However caution needs to be taken with these results as the analysis is based on text mining rather than more accurate content analysis. We explored time to emergence of different types of impact in a subsample of case studies (described later in the results).

Analysis of key word (beneficiaries) in context identified different types of beneficiaries in the LGM context. Looking at the frequency of key words according to these categories of beneficiaries reveals over half relate to end-users (53%) (Public, Community, Young people, Patient, Student, Service users, Older people, End-user) and nearly a third relate to research users (27%) (Professionals, Academics, Managers, Teachers, Leaders, Governors, Policymaker). Key terms relating to researchers or stakeholders appear far less often (9 and 3% respectively).

Pathways to impact

Thematic analysis of Details of the Impact for all 1309 case studies identified eight interrelated themes in the data about types of processes that researchers claim contributed to impact. These are explained below.

Researcher impact skills

Researcher subject expertise and roles were described as contributing to impact in the majority of case studies (n = 955) (determined by analysis of the key words ‘role’ and ‘expertise’ in context in Details of the Impact). Ability to collaborate was clearly an important advantage for case study authors and most (n = 820 case studies) were able to attract and work with other researchers within the limitations of research funding or research design (identified using the terms ‘partner’ and ‘collaborator’ in Underpinning research). Researchers suggested they had an influencing role in the decision-making of stakeholders through appointment to an advisory or special interest group (n = 54). Authors claimed that researcher skills and knowledge of research users, through researcher engagement or collaboration (n = 87), helped to understand user needs, generate useful research outputs or led to more active involvement in implementation of outputs with research users. Researcher skills to work with public or users was described in fewer case studies (n = 25 had used some form of public or user involvement). Case studies that did, emphasized researcher’s skills in selection or recruitment of public or user representatives to be involved in the research. Research design skills were a more implicit theme of achieving impact. For example, researchers were able to generate impact through their knowledge of approaches to knowledge translation, participation or collaborative research. Skills in impact planning was expressed in terms of: researchers predicting the types of benefit and beneficiaries of the research, identifying target audiences and research users, setting objectives for impact, planning opportunities to maximize impact or attracting/allocating resources for impact activities. Skills in participatory working were described as researchers: being skilled in shared decision-making, co-design or co-production, or having the know-how to work with particular client groups, such as young people or refugee communities. Researcher ‘communication skills’ (key term) were consistently described as contributing to impact (n = 249) particularly at dissemination stage. In most case studies researchers had used their ‘authority’ or ‘expertise’ to influence policy, practice or research. Skills in ‘public speaking’, confidence in the research and ability to ‘explain’ or ‘translate’ technical or complex information, were the skills that authors described as contributing to impact. Skills and knowledge in implementation was described in relation to ‘implementation of evidence’ or research outputs (n = 434) with stakeholders and research users. A small number of case studies (n = 10) described how researchers had put systems in place to capture impact as it happened, approaches included monitoring changes in routinely collected data and planned researcher user and stakeholder feedback.

Connecting with context

When we explored the ways that researchers connect with the research system and wider context to stimulate impact we found consistent patterns in how researchers used context to lever change. Most notably authors described how favourable conditions for collaboration occurred, such as winning competitive funding or professional/research groups recognizing the need to address identified research ‘problems’ ‘questions’ or ‘issues’ (n = 882 case studies). Socio-economic-political factors tended to be important influences at research design stage but could also provide momentum for research that was underway. Political, professional or public agendas had helped to promote research studies by directly linking research outputs to political figures or campaigns by professional bodies on issues, such as anti-smoking and alcohol responsibility. Other case study authors described how linking the research to public or user ‘interest’ (n = 278) such as media campaigns or consumer groups helped to stimulate change, such as safety in the workplace and employee wellbeing. Wider ‘strategies’, ‘agendas’ or ‘plans’ for change were described as a way of generating impact (n = 134) by identifying national policies or recommendations that required research users to act, such as health or social care providers to take action on waiting times. Some case study authors reported using indicated outcomes, such as estimates of the scale of impact or numbers of potential beneficiaries (e.g. population prevalence) to gain support or to attract resources for the research.

Designing research for impact

Choices about research approach were described as influencing impact. For example the case study data suggests that researchers using a critical approach (research that engages with political or moral questions about who stands to lose or gain from the research or its findings) or action orientated research methods (such as Participative Action Research) may be more likely to achieve impact in organizations. For example, a case study from Queen Mary University of London was designed to specifically address under-representation of women in union leadership. Examples of planning impact involved: activities to predict the types of benefit and beneficiaries of the research, identifying target audiences and research users, setting objectives for impact, planning opportunities to maximize impact and allocating resources for impact activities. For example in one case study of research on micro-financial institutions at the University of Salford, impact was supported through the use of two Knowledge Transfer Partnerships (KTPs). The KTPs enabled sharing of knowledge between the researchers and stakeholders to improve outcomes for end-users of financial services. Planning for progressive or incremental impact through roll-out, phases or levels of the research were ways to spread impact. For example research from Coventry University on organizational change and leadership and their impact on organizational wellbeing has been progressed at four strategic levels: on management and leadership practice generally, direct influence on management and leadership education and practice, direct impact in organizations that have used evidence to improve business practice, and commercial impact.

Stakeholder engagement

In our analysis ‘stakeholders’ were perceived as people directly influencing or involved in the research for example ‘policymakers’, ‘commissioners’, ‘regulators’ or ‘collaborators’ (key terms) in the research. Authors reported engaging stakeholders early in the planning of research to support applications for funding and enable the undertaking of the research. For example, in a case study carried out by the Open University Business School researchers built up strong stakeholder engagement at different levels of health care policy and practice to achieve political impact. Participative working stimulated impact through close interaction between researchers and research users with the research. Authors suggested stakeholder engagement allowed researchers to identify the types of research outcomes that might be beneficial in situations where the research users had a good understanding of the challenges or problems faced, and expressed a need for certain types of research evidence, or outputs. Maintaining the engagement of research users was supported by seeing the benefit of early results or the positive gains others have achieved (n = 38). A further factor was the role of research users in implementation of research (n = 64), for example facilitating change management, or creating receptive conditions for implementation of research outputs.

Public or user involvement

Public or user involvement in LGM research was found to include consultation or collaboration with members of the public or end-users (e.g. community members, young people, patients, students, service users, older people) and more active forms of direct involvement. Involving the public or users in deciding the aims or focus of research may build towards targeted impact. For example social work and social policy research by the Social Policy Research Centre (SPRC) has enabled minority ethnic organizations and other local service providers to gather evidence and develop initiatives and practices better adapted to the needs of service users. Public or user involvement can inform the way that research users such as health service providers engage with patients and carers. For example the People in Public Health (PIPH) study undertaken by Leeds Metropolitan University has brought together evidence on rationales for lay engagement, effectiveness and models of support. The research has had an impact on health policy, national networks for public health and public health practice. Public or user involvement in LGM research also includes service user involvement in the design and development of service-focused organizations. One example is health and social care research undertaken by Anglia Ruskin University involving peer led self-help groups and peer led Citizen Research Groups. Impact is evidenced by national and local guidelines, national and local training initiatives, the sustained commissioning of two service user or citizen research groups and related service improvements, increased social capital and skills for the citizens involved.

Mechanisms for exchange

Thematic analysis of Summary of the Impact statements identified 20 types of mechanisms for exchange which authors reported to have contributed to impact. Over two-thirds of the case studies describe mechanisms for stimulating interest in the research e.g. influencing (16.9%, n = 221), debate (33%, n = 432) discussion (21.4%, n = 280). Nearly a third of the case studies identify mechanisms for keying in to the interests of stakeholders, research users, researchers or end-users e.g. engagement (31.8%, n = 416), civic engagement (1.1%, n = 14) or user led (0.2%, n = 2). Nearly a third of the case studies describe mechanisms for spreading knowledge e.g. informing (18.2%, n = 238), demonstrating (9.8%, n = 128) or knowledge transfer (10.4%, n = 136). Nearly a third of the case studies describe mechanisms for building knowledge e.g. consultation (14.8%, n = 194), co-creation (0.6%, n = 8) or shared decision-making (16.1%, n = 211). Less than a fifth of case studies describe mechanisms for validating knowledge/impact e.g. feedback (15.1%, n = 198). Just over a tenth of the case studies describe mechanisms for creating spaces for exchange e.g. through discourse (5.3%, n = 69), linking (3.7%, n = 49) and connection (3.8%, n = 50). Few case studies describe mechanism for perspective sharing e.g. interaction (6.5%, n = 85) or knowledge exchange (7.4%, n = 97), or mechanisms for spreading expertise e.g. shared learning (3.8%, n = 50) or secondment (1%, n = 13). These findings reveal the mechanisms through which impact might be achieved and the different functions that mechanisms serve.

Developing impactful outputs

Research output factors that may support impact include stakeholder backing, for example research outputs were advocated or funded by senior stakeholders (n = 77). It was also suggested that research outputs were considered safe, economical or acceptable to the public or users (n = 29). Some authors described how research outcomes had utility or were ‘fit for purpose’ for example research outputs met research users’ needs in an effective and efficient way (n = 159). Other authors reported that research outputs aligned with strategies/plans in the research context or research users’ plans for change (n = 14). Some researchers had developed outputs with research users (n = 112) to produce content, formats or language that was acceptable and accessible to target groups. Research outputs that were accessible to research users (e.g. short guides or web-based applications) and were supported by further information or sources (e.g. reference databases) helped to support uptake. It was advantageous for assessing impact if research outputs clearly led to detectable change, such as implementation of new systems in organizations. Framing or translating research outputs in ways that resonate with research users supported impact through improved uptake and implementation of research outputs. For example a case study of business owner-managers research at the University of Leeds examined how managers acquired knowledge and identified ways that they could learn more effectively by framing practical education and business support in ways that appealed to the research users. In some case studies, communicating research involved in-depth work to develop research outputs that connect with the language, communicative resources or modes of communication of research users or end-users, such as research with young people to improve sexual health.

Implementing and evaluating outcomes

For most case studies implementation involved dissemination activities such as the development of summaries, briefings or web-based information about the research. Implementing research outputs included producing or adapting research outputs to support uptake by specific groups of research users. Researchers using applied research methods were the most likely group of researchers to make use of an implementation strategy to achieve impact, or to use a planned separate phase or follow-up to the research. An example case study from Aston University illustrates how implementation of a change management model using a set of rules and guidelines has been implemented to change management initiatives through applied research. In the case studies outcome assessment involved planning to capture, record, monitor or evaluate the outcomes of the research, which encouraged change by focusing stakeholders or research users on particular outcomes.

How impact from LGM research might be evidenced and assessed

Patterns in time to impact were explored in a subsample of 20 LGM case studies relating to research in higher education contexts selected during the screening process by reading case study titles and/or Summary of the Impact. The low number of LGM case studies relating to higher education contexts is unsurprising because of the focus of REF ‘beyond academia’. In this subsample impact on HEIs included: improvements in leadership skills and behaviours, equality and diversity in leadership roles, coordination and leadership capacity development, management development, change management, organizational improvement, and new LGM research methods. Impact at the end-user level was described as including effects on: student experience, student engagement, student retention, student learning or attainment, staff and student well-being, staff skills or knowledge, and staff commitment or employee engagement. Average time to impact was 7 years. The earliest reported impact (mean average 5 years) related to the use of evidence (type I impact). It generally took longer (averaging 8.3 years but with a notable peak at 6 years) to develop research products, such as training programmes or interventions, and to see the effects of research on individuals (8 years), groups or organizations (7.3 years).

Analysis of spread of impact in the higher education subsample shows that the first reported place of impact was most often at a national level (e.g. impact on national policy or programmes) with subsequent spread of impact towards the local (university) level.

Across the data case study authors used a number of techniques to evidence LGM impact through the narrative of the case studies. These techniques were to:

Show causality, if the research design permitted researchers to control for variables and randomization

Use measures and indicators of research outcomes used in the research to evidence impact

Address issues of attribution and contribution in relation to the specifics of the research context and outcomes

Emphasize progression or spread of impact from the research

Show systematic capture of impact information

Present a tailored account of how impact occurred using an active authorship style

Patterns in the types of evidence reported, such as how impact data were captured, show authors draw upon many types of evidence, captured using different techniques. In general types of evidence of impact included:

Evidence-based: impact was rigorously evaluated and the research has consistently been shown to achieve the impact

Research-based: impact is based on sound theory and a growing body of empirical research to corroborate what has been achieved

Expert review: results of an independent critical review or expert opinion corroborate the impact

User-review: the research has been used or applied by research users and users report the research to have a benefit or impact

Self-report: robust researcher evaluation, audit, structured reflexivity or stakeholder feedback indicate positive findings about impact from the research

Informed opinion: favourable views, comments, testimonials or opinions about the impact of the research from those that have experience of the research, its outcomes or outputs.

By exploring LGM impact across different disciplines we identified 145 different types of measures or indicators used to quantify different types of impact in different research contexts, measuring different types of outcomes, and at different stages of use or development. Detail is provided in the full report (Morrow, 2016). In general measures included:

Measures of effect to show changes in performance, productivity, effectiveness, learning, skills or knowledge, decision-making, behaviour, and health or social impact

Measures of importance, including clinical or psychological assessment or diagnosis, inequalities and community needs

Measures of value, such as individual attitude or attributes, experiences, perception, economic, safety, political, social, employee engagement, and participation.

Our discussion of these findings focuses on implications for the evaluation of research impact.

Discussion

Strengths and limitations of the study

The findings illustrate some of the complexity of impact in terms of the contributions of different HEIs, disciplines and regions to impact from the LGM field. They classify the types of impact from LGM research and associations between different types of impact; beneficiaries in the LGM context; types of processes that researchers claim contributed to impact; time to impact and spread of impact. These findings illustrate how UK HE researchers attempt to articulate their claims to impact, including techniques to evidence LGM impact; the main types of evidence reported; and types of measures or indicators used to quantify different types of impact. Unpacking these components of impact can contribute to developing appropriate approaches to its evaluation (Upton et al., 2014).

The study was devised and undertaken in the UK context using data from academic research, which was submitted to REF 2014 under specific rules and conditions. We do not claim all LGM research in UK HEIs is captured here, as not all institutions or eligible units will have submitted to the 2014 REF. Different interpretations and understandings of impact are likely to exist in different contexts internationally.

The case study data provided an element of consistency to the data (because of the required template format), but there are constraints to what is captured because of the four-page length limit, REF inclusion criteria, and the types of research HEIs chose to submit. The accuracy of accounts is likely to be high due to the REF requirement to provide verifiable and auditable sources for underpinning research. However, perhaps not surprisingly there is a bias towards positive accounts of impact, and retrospectively constructed accounts that do not recount the challenges, failures and drawbacks of achieving impact. The data relates to specific research projects and programmes which means it does not fully capture the impact of scholarship (Learmonth et al., 2012), research methods development, or research partnerships, which might have developed over longer periods outside of time limited research studies. The focus on attributing impact to the local level (researcher, project or programme) means the data does not capture activities at the institutional or regional level to support or maximize impact. While there is a high degree of consistency in the case study data about types of impact, the absence of counterfactuals makes showing the effectiveness of different impact processes a challenge.

Adopting a CAS lens helped us to critically reflect on the perspectives of impact that we were taking in the study and the assumptions we were exploring in the data, which supports the views of other researchers (Harkema, 2003; Palmberg, 2009). There is not space in this article to critically discuss the literature on adaptive systems and our main caveat for the CAS approach is that complexity could be used to suggest a lack of consistency, or coherency, in impact, rather than providing the rationale for tailored solutions and measures of success at a local level. However, for us the benefit of thinking through a CAS lens was that it allowed us to seek to understand complexity and encouraged us to question how the data connects with the research system, existing theories and methodologies for assessing impact (Greenhalgh et al., 2016; Morgan Jones and Grant, 2013), as well as wider influences such as policy, social or economic climate (Harkema, 2003). The findings can therefore provide insights into how researchers shape the impact discourse in the context of funding council guidelines, collective expectations, and other features of the research system. Evaluators could use CAS as a reflective framework to consider the evidence and assumptions that underpin impact evaluation and how these effect the research system (Carter, 2013). With support from the UK higher education funding councils, the Leadership Foundation is working with academic learning partners to take the findings from this study forward in developing a research-based toolkit for research leaders and managers (www.lfhe.ac.uk). Building on the findings there are significant practical implications of the findings for three groups of evaluators (research assessors, HEIs and researchers), which are discussed below.

REF and large-scale research assessment exercises

Given the challenges of showing causality between research and impact, assessment panels could consider how to assess different types of evidence of impact, so as not to disadvantage or discourage reporting on impact from the full range of research undertaken by HEIs (Morgan Jones and Grant, 2013). This could include criteria for judging the plausibility of impact pathways, including: impact processes used, mechanisms for exchange employed and connections with context. Evaluators can consider the measures and indicators of LGM research outcomes and how they are used by authors to evidence impact. In a diverse field such as LGM research it is not possible to define a complete set of outcome measures, for the purposes of assessing reach, significance or other measures. Even if it were possible to define such a list, this could unhelpfully limit LGM research, and fail to represent the full extent of actual impact (Hefce, 2015). The findings show some conceptual overlap in the current use of measures in some areas of LGM research. For example, measures of performance, productivity, behaviour and engagement could be further developed as key suites of measures for LGM research. Psychological measures of happiness and well-being are also promising but more diverse in their meaning and use in different areas of LGM research. Measures of learning, experience and perception could also be useful for evidencing types of impact at the end-user level. Findings on impact processes could inform future assessment guidance on how to address issues of attribution and contribution in relation to the specifics of the research context and outcomes. Request to see evidence of progression or spread of impact from the research and the systematic capture of impact information. The findings on impact processes raise questions about whether evaluators should only focus on impact as the consequence of research, or reward institutions that support researchers to plan and integrate impact in their practice.

HEI’s institutional impact evaluations

Findings on types of impact could be used by HEIs to evaluate their contribution to impact from LGM research. Recognizing the many disciplines that contribute to LGM research could open up new possibilities for interprofessional collaboration and future case studies. For example it could be advantageous for HEIs to cluster teaching and research themes across disciplinary boundaries by using LGM as a theme of institutional research. Information about LGM research impact could inform impact targets, for example HEIs could consider engagement of different stakeholders and effects on different groups of beneficiaries. Themes about the types of evidence researchers used to substantiate their claims to impact can guide HEIs in their work to capture and record impact. HEIs can use findings about impact processes to inform impact strategies and support researchers to integrate impact in their practice. Findings on how researchers constructed claims to impact can inform future submissions to institutional research assessment exercises. HEIs can support the development and evaluation of institution-wide mechanisms for exchange, rather than focusing narrowly on capturing research outputs. Involvement of non-academic groups in research emerges as a key issue for the future development of impact in HEIs.

Researcher self-evaluation of impact and professional development

Researchers can use these findings to extend their understandings of what impact is and how it might be achieved, or to acquire ‘impact literacy’. They can use findings on the knowledge and skills authors said contributed to achieving impact to self-evaluate personal knowledge and skills for impact. The findings suggest impact emerges from relationships between multiple individuals each with their own roles, skills, knowledge, abilities and interests. This means researchers should evaluate their own performance towards impact (e.g. as part of performance review) but also consider their relationships with collaborators, stakeholders, professional support staff, teaching staff, students and so on, to identify and build relationships which support productive interactions (Molas-Gallart and Tang, 2011; Spaapen and Drooge, 2011). Researchers can contribute to a growing evidence base for impact by planning to evaluate impact as part of research proposals and including impact in their reporting.

Future research

The findings on impact presented here are derived from researchers’ accounts and our interpretation of them. Further exploration of multiple perspectives and vantage points in the research system (and beyond it) is needed to unpack the complexity of impact. For example, research could explore impact as a structure of power, governance or accountability; as an indicator of research system adaption to change; as a strategy to close the research-practice gap; or as a developing field of research on research. Research could examine these or other perspectives of impact from the vantage points of funders, assessors, stakeholders, beneficiaries and so on. Our findings relate to impact in the LGM context, future research could test the transferability to other fields and to support interdisciplinary research (Hefce, 2016). Research on the contribution of public or user involvement to impact is needed to inform future approaches to public engagement.

Conclusion

This study of a subsample of REF impact case studies identifies the contributions of UK higher education institutions to a range of types of impact from LGM research, timescales to impact and spread. Systems thinking was a useful lens for critically exploring what impact is, how it occurs and how its meaning develops. Evaluators could use CAS as a reflective framework to consider the evidence and assumptions that underpin impact evaluation and how these may affect the research system. Author’s accounts of the causal pathways to impact reveal intricate patterns of impact processes, mechanisms for exchange, and connections with the wider contexts of research. Evaluators developing REF and other large-scale research assessment exercises can consider ways to judge the plausibility of impact pathways to encourage reporting on impact from the full range of research approaches. HEI’s can draw on the types of evidence researchers used to substantiate their claims to impact to inform institutional impact evaluations and to embed impact in research strategies and practices. We have since produced ‘The Research Leaders Impact Toolkit’ to support HEIs with their impact work (available from www.lfhe.ac.uk/impacttoolkit). As part of their professional development researchers can self-evaluate their roles, skills, knowledge and research relationships against those reported to generate impact. Future research could explore impact from other perspectives and vantage points and could test the transferability of these findings to other fields.

Footnotes

Acknowledgements

Grateful thanks to all those who have contributed to this research, in particular the two anonymous external reviewers of the study and the three anonymous reviewers of this article.

Conflict of interest

None declared.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the United Kingdom Higher Education Funding Councils through the Leadership Foundation for Higher Education.

Elizabeth M. Morrow is an independent researcher based in Northern Ireland. She is interested in the power and politics of research practice.

Helen Goreham is Research Manager at the Leadership Foundation for Higher Education.

Fiona Ross CBE is Professor of Health Research at Kingston University and St George’s, University of London and former Director of Research at the Leadership Foundation for Higher Education.

References

Altbach

Salmi

(eds) (2011) The Road to Academic Excellence: The Making of World-Class Research Universities. Washington, DC: World Bank Publications.

Atwood

Mora

Kaplan

(2010) Learning to lead: Evaluating leadership and organizational learning. Leadership & Organization Development Journal 31(7): 576–95.

Banzi

Moja

Pistotti

et al . (2011) Conceptual frameworks and empirical approaches used to assess the impact of health research: An overview of reviews. Health Research Policy and Systems 9(1): 26.

Bass

Stogdill

(1990) Handbook of Leadership, Vol. 11. New York: Free Press.

Bevir

(2013) Governance: A Very Short Introduction. Oxford: Oxford University Press.

Boaz

Fitzpatrick

Shaw

(2009) Assessing the impact of research on policy: A literature review. Science and Public Policy 36(4): 255–70.

Bolden

(2010) Leadership, management and organisational development. In: Gold

Thorpe

Mumford

(eds) Gower Handbook of Leadership and Management Development. Farnham: Gower.

Bornmann

(2013) What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology 64(2): 217–33.

Boswell

(2009) The Political Uses of Expert Knowledge: Immigration Policy and Social Research. Cambridge: Cambridge University Press.

10.

Brewer

(2011) The impact of impact. Research Evaluation 20(3): 255–6.

11.

Brown

(2008) The use of complex adaptive systems as a generative metaphor in an action research study of an organisation. The Qualitative Report 13(3): 416–31.

12.

Buxton

(2011) The payback of ‘Payback’: Challenges in assessing research impact. Research Evaluation 20(3): 259–60.

13.

Buxton

Hanney

(1996) How can payback from health services research be assessed? Journal of Health Services Research & Policy 1: 35–43.

14.

Carter

(2013) The impact of impact on universities: Skills, resources and organisational structures. In: Dean et al . (eds) 7 Essays on Impact. DESCRIBE Project Report for Jisc. University of Exeter. Available at: https://www.exeter.ac.uk/media/universityofexeter/research/ourresearchexcellence/describeproject/pdfs/2013_06_04_7_Essays_on_Impact_FINAL.pdf

15.

Chadwick

(2010) Creating order out of chaos: A leadership approach. Association of periOperative Registered Nurses 91(1): 154–70.

16.

Davies

Nutley

Walter

(2008) Why ‘knowledge–transfer’ is misconceived for applied social research. Journal of Health Services Research & Policy 13: 188–90.

17.

Deloitte Access Economics (2012) Extrapolated Returns on Investment in NHMRC Medical Research. Canberra: Australian Society for Medical Research.

18.

Denicolo

(2014) Achieving impact in research. SAGE, 166. Available at: https://uk.sagepub.com/en-gb/eur/achieving-impact-in-research/book240291

19.

Denzin

Lincoln

(eds) (2005) The SAGE Handbook of Qualitative Research, 3rd edn. Thousand Oaks, CA: SAGE.

20.

Dill

(1998) Evaluating the ‘Evaluative State’: Implications for research in higher education. European Journal of Education 33(3): 361–77.

21.

Dodder

Dare

(2000) Complex Adaptive Systems and Complexity Theory: Inter-related Knowledge Domains. Cambridge, MA: MIT Press.

22.

Donovan

(2011) State of the art in assessing research impact: Introduction to a special issue. Research Evaluation 20(3): 175–9.

23.

Donovan

Hanney

(2011) The ‘Payback Framework’ explained. Research Evaluation 20(3): 181–3.

24.

Dooley

(1997) A complex adaptive systems model of organization change. Nonlinear Dynamics, Psychology, and Life Sciences 1(1): 69–97.

25.

Dori

Sillitto

(2017) What is a system? An ontological framework. Systems Engineering 20(3): 207–219.

26.

Durlak

DuPre

(2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology 41(3–4): 327–50.

27.

Flinn

Mowles

(2014) A complexity approach to leadership development: Developing practical judgement. Stimulus Paper. London: Leadership Foundation for Higher Education.

28.

Gagnon

(2011) Moving knowledge to action through dissemination and exchange. Journal of Clinical Epidemiology 64(1): 25–31.

29.

Given

Kelly

Willson

(2015) Bracing for impact: The role of information science in supporting societal research impact. Proceedings of the Association for Information Science and Technology 52(1): 1–10.

30.

Glatter

(2006) Leadership and organization in education: Time for a re-orientation? School Leadership & Management 26(1): 69–83.

31.

Grant

Brutscher

Kirk

et al . (2010) Capturing research impacts: A review of international practice. Documented briefing. RAND Corporation. Available at: http://www.hefce.ac.uk/media/hefce/content/pubs/2009/rd2309/rd23_09.pdf

32.

Greenhalgh

(2015) We must do more to tell the full story of impact. Times Higher Education, 1 October.

33.

Greenhalgh

Fahy

(2015) Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK Research Excellence Framework. BioMed Central Medicine 13: 232. DOI: https://doi.org/10.1186/s12916-015-0467-4.

34.

Greenhalgh

Raftery

Hanney

et al . (2016) Research impact: A narrative review. BioMed Cenral Medicine 14: 78. DOI: https://doi.org/10.1186/s12916-016-0620-8.

35.

Guthrie

Wamae

Diepeveen

et al . (2013) Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Arlington, VA: RAND Corporation.

36.

Hamlin

(2010) Evidence-based leadership and management development. Gower Handbook of Leadership and Management Development. Routledge, 197–220.

37.

Harkema

(2003) A complex adaptive perspective on learning within innovation projects. Learning Organization 10(6): 340–6.

38.

Hefce (2012) Research Excellence Framework 2014. Assessment framework and guidance on submissions. July 2011. Available at: http://www.ref.ac.uk/pubs/2011-02/

39.

Hefce (2015) The Metric Tide - Report of the Independent Review of the Role of Metrics in Research Assessment and Management. Higher Education Funding Council for England. Available at: http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/

40.

Hefce (2016) Landscape Review of Interdisciplinary Research in the UK, Higher Education Funding Council for England. Available at: http://www.hefce.ac.uk/pubs/rereports/year/2016/interdis/

41.

Holland

(1999) Emergence: From Chaos to Order. Reading, MA: Perseus Books.

42.

Jones

Wuchty

Uzzi

(2008) Multi-university research teams: Shifting impact, geography, and stratification in science. Science 322(5905): 1259–62.

43.

Keshavarz

Nutbeam

Rowling

et al . (2010) Schools as social complex adaptive systems: A new way to understand the challenges of introducing the health promoting schools concept. Social Science & Medicine 70(10): 1467–74.

44.

Kezar

Carducci

Contreras-McGavin

(2006) Rethinking the ‘L’ word in higher education: The revolution of research on leadership. ASHE Higher Education Report. Chicester, UK: Jossey-Bass.

45.

King’s College London and Digital Science (2015) The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies. Bristol: Hefce.

46.

Kuruvilla

Mays

Walt

(2007) Describing the impact of health services and policy research. Journal of Health Services Research & Policy 12(Suppl. 1): 23–31.

47.

Lansing

(2003) Complex adaptive systems. Annual Review of Anthropology 32: 183–204.

48.

Learmonth

Lockett

Dowd

(2012) Promoting scholarship that matters: The uselessness of useful research and the usefulness of useless research. British Journal of Management 23(1): 35–44.

49.

Leitner

Warden

(2004) Managing and reporting knowledge-based resources and processes in research organisations: Specifics, lessons learned and perspectives. Management Accounting Research 15(1): 33–51.

50.

Levin

(2003) Complex adaptive systems: Exploring the known, the unknown and the unknowable. Bulletin of the American Mathematical Society 40(1): 3–19.

51.

LSE (2011) London School of Economics Public Policy Group. Maximising the impacts of your research: A handbook for social scientists, 1–298. Available at: http://blogs.lse.ac.uk/impactofsocialsciences/the-handbook/

52.

McDaniel

(2007) Management strategies for complex adaptive systems: Sensemaking, learning, and improvisation. Performance Improvement Quarterly 20(2): 21–41.

53.

Martin

(2011) The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation 20(3): 247–54.

54.

Meagher

Lyall

Nutley

(2008) Flows of knowledge, expertise and influence: A method for assessing policy and practice impacts from social science research. Research Evaluation 17: 163–73.

55.

Milat

Bauman

Redman

(2015) A narrative review of research impact assessment models and methods. BioMed Central Medicine 13: 18. DOI: https://doi.org/10.1186/s12961-015-0003-1.

56.

Miles

Huberman

Saldana

(2013) Qualitative Data Analysis. London, UK: SAGE.

57.

Molas-Gallart

Tang

(2011) Tracing ‘productive interactions’ to identify social impacts: An example from the social sciences. Research Evaluation 20(3): 219–26.

58.

Morgan Jones

Grant

(2013) Making the grade: Methodologies for assessing and evidencing research impact. In: Dean et al . (eds) 7 Essays on Impact. DESCRIBE Project Report for Jisc. University of Exeter. Available at: https://www.exeter.ac.uk/media/universityofexeter/research/ourresearchexcellence/describeproject/pdfs/2013_06_04_7_Essays_on_Impact_FINAL.pdf

59.

Morrow

(2016) The impact of higher education leadership, management and governance research: Mining the 2014 Research Excellence Framework impact case studies. Leadership Foundation for Higher Education, London. Available at: https://www.lfhe.ac.uk/en/research-resources/research-hub/2016-research/the-impact-of-higher-education-leadership-governance-and-management-research--final.cfm?utm_source=research&utm_campaign=morrow (accessed 6 September 2017).

60.

Morton

(2015) Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation 24(4): 405–19.

61.

Nutley

Walter

(2005) Assessing the impact of social science research: Conceptual, methodological and practical issues. Research Unit for Research Utilisation School of Management, University of St Andrews. Availale at: https://www.odi.org/sites/odi.org.uk/files/odi-assets/events-documents/4381.pdf

62.

Oakley

Selwood

(2010) Conversations and collaborations: The leadership of collaborative projects between higher education and the arts and cultural sector. Final Report. London: Leadership Foundation for Higher Education.

63.

Oliver

Innvar

Lorenc

et al . (2014) A systematic review of barriers to and facilitators of the use of evidence by policymakers. BioMed Central Medicine 14: 2. DOI: https://doi.org/10.1186/1472-6963-14-2.

64.

Owen

Macnaghten

Stilgoe

(2012) Responsible research and innovation: From science in society to science for society, with society. Science and Public Policy 39(6): 751–60.

65.

Palmberg

(2009) Complex adaptive systems as metaphors for organizational management. Learning Organization 16(6): 483–98.

66.

Penfield

Baker

Scoble

et al . (2014) Assessment, evaluations, and definitions of research impact: A review. Research Evaluation 23(1): 21–32.

67.

RAND Europe (2015) Evaluation of the 2014 Research Excellence Framework. Cambridge: RAND Europe.

68.

RCUK (2015) Research Councils UK: Pathways to impact. Available at: http://www.rcuk.ac.uk/innovation/impacts/ (accessed 6 September 2017).

69.

Ritchie

Lewis

Nicholls

et al . (eds) (2013) Qualitative Research Practice: A Guide for Social Science Students and Researchers. London, UK: SAGE.

70.

Rogers

Medina

Rivera

et al . (2005) Complex adaptive systems and the diffusion of innovations. The Innovation Journal: The Public Sector Innovation Journal 10(3): 1–26.

71.

Sarli

Dubinsky

Holmes

(2010) Beyond citation analysis: A model for assessment of research impact. Journal of the Medical Library Association 98(1): 17–23.

72.

Sooho

Bozeman

(2005) The impact of research collaboration on scientific productivity. Social Studies of Science 35(5): 673–702.

73.

Spaapen

Drooge

(2011) Introducing productive interactions in social impact assessment. Research Evaluation 20(3): 211–18.

74.

Staley

Buckland

Hayes

et al . (2014) ‘The missing links’: Understanding how context and mechanism influence the impact of public involvement in research. Health Expectations 17(6): 755–64.

75.

Upton

Vallance

Goddard

(2014) From outcomes to process: Evidence for a new approach to research impact assessment. Research Evaluation 23(4): 352–65.

76.

Walshe

Davies

(2013) Health research, development and innovation in England from 1988 to 2013: From research production to knowledge mobilization. Journal of Health Service Research Policy 18: 1–12.

77.

Wooding

Nason

Klautzer

et al . (2007) Policy and Practice Impacts of ESRC Funded Research: A Case Study of the ESRC Future of Work Programme, Approach and Analysis. Cambridge: RAND Europe.

78.

Yanow

Schwartz-Shea

(2015) Interpretation and Method: Empirical Research Methods and the Interpretive Turn. London, UK: Routledge.