Abstract
Objectives:
There have been calls for greater exchange between research and practice in health care policy and management, but little empirical research on what commissioners of research and researchers themselves consider appropriate, good quality research knowledge. This paper addresses this gap, considering the views of commissioners and producers of policy and management research in health care and other fields.
Methods:
Qualitative semi-structured interviews with 18 commissioners and producers of research, in central government, the NHS and other commissioning organizations, and in universities and independent sector providers.
Results:
Commissioners and producers agreed that research often fails to fulfil policy-makers’ and managers’ needs, and that greater interaction is required to improve this relationship. However, they offered differing accounts of the nature of research knowledge (as a ‘product’ or a ‘mindset’), and of what constitutes value, validity and originality in research, reflecting the differing priorities and pressures faced by the two groups.
Conclusions:
Efforts to promote greater interaction between research, policy and practice, and more critical, reflexive engagement between policy-makers, managers and researchers, are likely to face significant obstacles given these competing constructions of research knowledge and their reinforcement by divergent priorities.
Introduction
Much attention has been devoted in recent years to the connections – and indeed the frequent lack of connections – between research and practice in public management, including health services management research and policy. The last 10–15 years have seen an expansion in the volume of research relating to the organization and management of public services, prompted initially in the UK by policy-driven efforts to reform the institutions of public services according to evidence of what works. 1 In the UK and other economically developed countries, the drive towards improving links between research and policy has been driven in part by the remarkable ascendancy of evidence-based medicine, 2 which has, in a matter of a few decades, transformed the delivery of health care such that the requirement for robust evidence of efficacy and cost-effectiveness is now central to medical education, 3 resource allocation 4 and professional regulation. 5
There is increasing recognition, however, that the model of evidence-based medicine, and the modes by which it has been implemented, do not provide an unproblematic template for increasing the influence of management research on management practice. Social research in health care is methodologically heterogeneous, and so the epistemological assumptions of the hierarchy of biomedical evidence, which proposes that some forms of research are superior to others, are inappropriate. 6 Social research is also frequently highly contingent upon local circumstance: specifics of context matter in a way in which they do not for most biomedical (or at least pharmaceutical) research. 7 Finally, social research can be used in diverse ways, not just as a straightforward instrumental guide to ‘best practice’, but also more conceptually, to challenge and change ways of thinking and approaches to the day-to-day issues facing health care managers.8,9
This has led several commentators to claim that a new way of conceptualizing the relationship between research and practice is needed in relation to management and other social research, moving away from the linear and simplistic connotations of models of ‘knowledge transfer’. The ideas of ‘mode 2’ knowledge production, in which research knowledge is produced closer to the context of its application, 10 have been incorporated into some health care management research projects,11–13 but others have highlighted challenges with this way of conducting and applying research. 14 Some authors have called for greater cross-pollination between research and practice, through the involvement of decision-makers in research and of researchers in decision-making, 15 or through the introduction of ‘knowledge-brokering’ roles to facilitate dialogue between the two groups. 16 More generally, Davies et al. argue for ‘knowledge interaction’ as a way of understanding the ‘messy engagement of multiple players’ involved in research-knowledge utilization through linkage, exchange and knowledge brokerage, as well as more conventional processes of dissemination. 17
There is an emergent literature on the ways in which such processes give rise to utilization of research knowledge, and some evidence of their effectiveness. 8 However, there has been a surprising absence of study of the connections between the commissioners and producers of social research in health and other public-management fields. Since Caplan's famous characterisation of researchers and policy-makers as two epistemic communities, 9 lacking the will or ability to communicate with one another, research has focused largely on the potential to bridge the gap downstream, between producers and users of research. As those responsible for setting the aims, objectives and terms of research projects at their conception, though, commissioners have a potentially vital upstream role in mediating the gap between research and practice, bringing the views and interests of health care managers (and other practice-based stakeholders, such as health professionals, policymakers and patients) to bear on what research is done. There have been some developments in the way research is commissioned in the UK and elsewhere, with a growth of policy- and practice-led research-funding programmes in health care alongside traditional, academically-oriented streams, such as those offered through block grants and research councils. The National Institute for Health Research's Service Delivery and Organisation stream, for example, consults wide ranging groups, including health care managers, in commissioning practice-oriented academic research. Yet to date, the roles of research commissioners – and the views of both commissioners and producers on the commissioning process – have been neglected in the literature on research-knowledge utilization.
This paper seeks to begin to remedy this gap in the literature. It focuses on the commissioning of research relating to the organization and management of health care and other public services, in the British context. It draws on an interview study of the views and experiences of government- and NHS-based commissioners of research, and the producers of that research (in both universities and independent consultancies). It asks two principal questions:
l What kinds of notions of research knowledge do the roles of these parties, and the expectations of other stakeholders around them, give rise to, and how do these differ between commissioners and producers?
What can be done upstream to mitigate these differences and assist the production of research knowledge that is more readily utilized by health care managers and other stakeholders downstream?
It seeks to answer these questions by analysing in particular respondents’ accounts of the priorities and pressures they face in commissioning and producing research, commissioners’ and producers’ respective conceptualizations of the nature and value of research knowledge, and their ideas about how divergent views on these issues might be reconciled.
Methods
The data presented in this paper derive from a small-scale study from 2007 of the commissioners and producers of research relating to public policy and management in the UK, including commissioners within central government, the NHS and other research-commissioning organizations, and producers in universities and independent-sector consultancies and think tanks. The former group was identified through approaches to research-commissioning organizations, the latter via searches of the academic and grey literatures. Approaches were made to commissioners and providers across the range of public policy and management research, but with a particular emphasis on health care given the recent policy efforts around knowledge translation in this field. Further respondents were recruited through snowball sampling, and recruitment continued until ‘theoretical saturation’ of the issues of interest was reached. Participants were asked through semi-structured interviews about the kinds of policy-and management-related research contracts they commissioned or competed for, the criteria of quality they applied in evaluating research, and the relationship (actual and normative) between research, policy and practice. Participants were drawn from various fields of public service provision, primarily health and social care, but also local government, labour market policy and public management in general.
Eighteen interviews were carried out, lastinga round1.5 hours on average. Nine interviewees worked primarily as university academics, four were researchers in independent sector organizations, and five were commissioners of research. Half of the respondents worked primarily or substantially in health care research. The university providers interviewed were predominantly senior-level (professorial) academics; independent-sector providers had similarly lengthy experience; commissioners tended to have somewhat less experience of commissioning research, but had experience of academic, civil service and public management roles. Interviews were fully transcribed. Analysis adopted the framework approach, 18 which attempts to combine a focus on certain, preconceived themes for analysis with a sensitivity to other, unanticipated issues presented by participants. It involves categorizing data according to these preconceived and inductively generated themes, enabling analysis within and across participants’ interviews. The initial analysis was checked by the other authors, both of whom were involved in the inception of the study, and following critical discussion a final, agreed set of themes and interpretations was reached.
Results
We divide our findings into three subsections. Under ‘competing priorities’, we consider how institutional norms within academia and commissioning organizations give rise to different expectations about the substance, scope and timescales of research, and about what constitutes ‘good quality’ research. Next, under ‘cognitive boundaries’, we explore the consequences of these institutional differences for the ways that the outputs of research are or are not valued, disseminated and utilised. Finally, under ‘mediating the boundaries’, we discuss the prospects for a reconciliation of these divergent understandings of the quality and value of research.
Competing priorities
Participants spoke of a surge in the amount of policy and management research commissioned over the last 10–15 years, relating this both to the rise of the ideas of evidence-based policy and management, and to the succession of research and evaluation programmes accompanying major shifts in health care and other policy. Many noted a positive change in government attitudes towards research, but they also highlighted some of the challenges this brought. In particular, academic researchers noted that the rapid turnover of policies, ministers and civil servants could have disruptive effects. As previous policy was disregarded, so too was previous research, even though for academics, it had not lost its significance:
I'm old enough now to have seen them come round twice and they'll come round once more before I've had it. But the people making the policy often have only been in post a year and they'll be gone in a year. And so their organizational memory's difficult to generate. (I16, university provider)
Providers of research consequently found commissioners with much shorter time horizons than their own, preoccupied with immediate policy priorities subject to change at short notice following a shift in ministerial or civil service personnel. On the other side, research commissioners found academics reluctant to respond appropriately to these pressures, and more inclined to focus on their own agenda than address the questions they had been commissioned to answer. They were also clear on what was creating this tension: the priorities of academics were dominated by academic institutions, particularly the Research Assessment Exercise (RAE) (now succeeded by the similar Research Excellence Framework) by which academic departments were judged and core funding distributed:
Getting universities to work with us in customer-supplier relationships and to really listen to the requirements of the customer [is challenging. …] The provider and the purchaser have information asymmetry and often the provider knows more. […] And I know very well why this happens: one wants to score RAE points as well as serve the needs of the purchaser of government research out of the same pot of money. (I17, commissioner)
University academics recognized these conflicting pressures, and spoke of the challenges of reconciling the RAE and other academic requirements with doing research that would meet commissioners’ needs and inform policy and management:
You're trying to do two things at the same time. You're trying to write a report for commissioned research and you're trying to write an academic paper. And the theory is that you write the report as you're going to write the academic paper and then you just do a bit of fiddling. But actually I think it's quite fun-damental[ly different]. (I6, university provider)
Commissioners felt that the RAE and associated performance management measures in universities were distorting university providers’ priorities, deterring them both from carrying out research with real-world application, and from disseminating this work beyond conventional, RAE-returnable outputs:
To be blunt, academics aren't really very interested in dissemination outside the academic community, are they? (I13, commissioner)
In sum, the exigencies of the institutions of government and academia were acknowledged on both sides as creating strong pressures on the kind of research that was required and valued. Moreover, these institutions seemed to be pulling producers and commissioners in opposite directions, with academics asked to produce very different outputs for their masters in universities and their customers in policy. What was the result in terms of the nature of the research produced and the way in which it was utilized? The next subsection considers this question in detail.
Cognitive boundaries
In general terms, there was dissatisfaction on both sides with the consequences of these pressures for the way in which the current system operated. This applied to the process of policy research commissioning, to the format taken by the product and, most fundamentally, to the nature of the product offered, particularly by university providers. Ultimately, then, this dispute over the nature and value of research knowledge went beyond issues associated with the tendering process and translating findings to cater for the expectations of different episte-mic communities. Rather, it was a question of fundamentally divergent understandings of what constituted good quality, relevant and valid research for the purpose of policy-making.
Research commissioners expressed frustration about the difficulty of managing their contracts with university providers to ensure responsiveness and relevance of research, calling for more of a ‘trade-off’ (I16, commissioner) between quality and timeliness. Academics often craved a rather different model of research commissioning – ‘go away, do good science, tell us at the end when you've finished’ (I3, university provider) – and acknowledged that even policy-oriented research contracts gave them the freedom to be distracted by ‘interesting side issues’. This, along with the competing pressures arising from the RAE and other sources, could indeed result in neglect of what commissioners wanted:
Academics can be like jobbing builders: you go round and you start a job and you get the scaffolding up in one place, just to reassure your client that you've started. And then you go off for six weeks and do something else. (I6, university provider)
Academics and commissioners alike also recognized deficiencies in the communication of research findings across boundaries. Once again, the pressures and norms of academia were seen as resulting in outputs that failed to meet the need of those who had paid for them:
I have seen academic research coming out with post-modernist theory which is entirely unusable in the field. Because it's working in a conceptual frame that nobody understands. […] There are some issues about how the academic world navigates the tension between academic theory and the empirical practicalities of the day job of the clients. (I14, independent-sector provider)
In the main, academics accepted the need to maximize accessibility of their research findings, and favoured more diverse forms of dissemination than they had traditionally practised. However, they were not convinced that what they saw as the preoccupation of commissioners with responsiveness and timeliness of findings was something which would unproblematically improve the appropriateness and relevance of their research to policy-making and management practice. Rather, they articulated a competing vision of how policy-oriented research might contribute to policy and practice.
For a start, academics suggested that a concern for timeliness of research might risk losing the insights that could only come from longer-term study, free of the shackles of short-term policy requirements. The fast moving policy scene, some academics felt, had produced a research-commissioning process which fetishised the novel, neglecting lessons from the past. Government- and NHS-based commissioners, too, were aware of the risk of being driven by the latest ‘fads’ of policy. There was a concern among some to commission ‘topics that are likely to be of enduring interest’ (I13, commissioner), reconciling the shorter-term concerns of their policy-making colleagues with the longer timescales preferred by academics.
However, commissioners pointed out that the desire of academic research providers to carry out work that would make a contribution to the academic literature created its own preoccupations. The commitment to creating ‘new knowledge’, publishable in top-ranking journals and offering a lasting contribution to science, represented a different kind of fetish for novelty, driving academic research providers away from policy-oriented outputs. This was reflected in the way academics discussed their research:
The question we come back to with an assignment is […] ‘Does this contribute to the generation of new knowledge?’ […] We would always try to see things written up in an appropriate way for an academic journal because that enters the permanent literature. In an academic journal or a book or whatever, but seeing it entering the permanent literature, so that in 15 years’ time, when someone does a search on subject X, you want them to be able to find that research. (I7, university provider)
Academic research providers, then, held very different notions of what constituted ‘novelty’ and ‘influence’ – and therefore value – of research from those held by commissioners. Yet, as even some academics acknowledged, this meant risking disregarding the more direct influence that could arise from engaging with the needs and wants of commissioners:
Academics underestimate the difference that they can make. It won't be quoted in the knowledge in the way that someone writing a book after you will refer to what you've done, but your ideas will go in, and somebody somewhere will remember them and say, ‘hang on’. (I9, university provider)
On each side, then, research and its products were valued very differently. For most academic providers, the value of research knowledge was best measured by its scientific novelty, and by lasting contribution to the social scientific literature, through (for example) citations many years later. For commissioners, research knowledge was valuable if it spoke to current issues, and if it influenced practitioners in a more direct – but more diffuse and less measurable – way. The combination of the ill-aligned institutional pressures considered in the previous subsection, and this differing conception of what research knowledge is, and what should be valued, gave rise to a fundamental mismatch of expectations about what research was for, how it should be done and how it should be disseminated.
Mediating the boundaries
The problems in the commissioning, production and utilization of knowledge identified by providers and commissioners were, then, three-fold. There was frequently a lack of effective communication of research findings across this boundary; there were markedly different constructions of what kinds of knowledge (in terms of scope, theoretical content and historical context) were of most relevance and utility to the policy-making process; most fundamentally of all, there was a divergence between how research was valued by the institutions and actors of academia and those of government.
Yet there was also considerable common ground about the need to improve the appropriateness of the knowledge produced:
[It's about] putting it in a context for the policy-maker. And I think often because we don't make that first connection, we launch straight into our research and the policy-maker just goes, ‘but I'm not interested in this’. If my research study takes me three years, it doesn't matter it's taken me three years, as long as when I present it, […] I connect it with debates of the day. (I1, university provider)
What turns people off is research information and papers that are long, laborious, not focused and it switches them off. So it's how we make it accessible for them. (I11, commissioner)
Given this eagerness to improve alignment between research outputs and users’ needs, and to translate the value ascribed to their work by academics into something practically valuable to policy-makers and practitioners, what kinds of approaches to mediating the boundaries might be feasible?
Respondents from both sides highlighted how the commissioning process itself might be improved. Research providers felt they should be involved more closely in the development of calls for research proposals. Commissioners were more concerned that the users of research – policy colleagues and public service managers beyond Whitehall – be involved more closely, to ensure prioritisation of practical over academic concerns. Again, apparent was a divergence of views about the relationship between, and relative importance of, academic integrity and practical relevance:
Sometimes we're asked to do commissioned research where there isn't much academic understanding: [we face] commissioners who don't want to or don't understand the academic basis of issues they're thinking about. (I8, university provider)
We've been trying to do more to get ideas about research priorities from practitioners, particularly the service management practitioners. […] Our peer review, commissioning rules are somewhat dominated by academics. (I13, commissioner)
Some commissioners called for a more mixed economy of research provision, with parallel funding for shorter-term consultancy projects led by independent institutes outside the university sector, alongside longer-term research, which they felt would address concerns around the timeliness, responsiveness, and communication of research. However, for university providers in particular, this did not address the fundamental issue: that academically rigorous research is relevant and valuable to policy-makers and practitioners, if only ways might be found of communicating and mobilising this knowledge more effectively.
In this light, for university respondents a more appropriate response to the dissatisfaction with the current situation was not to supplement the work done (largely) by academics with some kind of ‘research lite’, but to work to ensure that the fruits of research were more appetising for their potential consumers. With these concerns in mind, respondents commented on the importance of building these links by one means or another. The concept of the ‘knowledge broker’ received some support, with commissioners highlighting the potential for expanding their roles in mediating between the research they commissioned and the utilizers of that research within and beyond Whitehall. However, for others, a better model of knowledge brokerage involved more direct interaction between the producers and users of research. Providers in particular stressed the limitations of a process of dissemination that constructed research findings as a ‘product’, an inert, final entity to be applied unproblematically to different settings by a third party:
Some people would argue that it's very important to put in place knowledge brokers who can take the knowledge from research and translate it and present it to practice communities. I [prefer] the idea that you promote interworking, so that people get to know each other. (I7, university provider)
For academics, greater interaction between the producers and commissioners of research was not about ensuring straightforward ‘knowledge transfer’ from the former to the latter, but about bringing the users of research into contact with a distinctive way of thinking about problems. Research knowledge, for providers, did not offer ‘solutions’ to be packaged and delivered to users, but rather, a different way of thinking about problems:
Academics have a role in cross-fertilization. So I can say, ‘have you tried what they're doing here?’ […] or ‘you ought to go and visit such-and-such place’. But that's an informal process. (I14, independent-sector provider)
At the moment, the form of engagement [between research and practice] – this is a caricature – but it's the academic gurus, oracles, and we go and give a shaft of light, or we sing an aria and somehow people will improve what they're doing. That's not my model of how the link between theory and practice should work. (I18, university provider)
What was needed, academics argued, was a model of engagement that encouraged interaction between researchers, policy-makers and practitioners, to facilitate critical engagement with a set of theories and methods, so that research users might benefit from research as a mindset rather than a product.
Research commissioners, however, were largely doubtful about how far such ideas would find a receptive audience among managers and policy-makers. Given competing pressures on their time, this critical interactive model was not one for which they would have time: rather, commissioners envisaged a more conventional division of research labour, with professional researchers to ‘do the thinking’ and provide the insights in a digestible format:
People's main priority is first and foremost sorting out their own local organization. […] It's about the business benefit of research and what research can deliver. I don't think that research is a solution looking for a problem; I think we have to start with the problem and then work back. [… Managers] need to be able to access resources and support according to the needs that they're going to define in their own local organizations. I don't think they need to have the ins and outs of a comprehensive knowledge around research specifically. But they need to know how to access it. (I10, commissioner)
[Public managers are] an exceptionally difficult community to reach in terms of research awareness, research literacy and the whole notion of what comprises evidence-informed practice. And the cultural aspects of how low premiums are placed upon that. […] They're people who culturally choose not to know things. (I13, commissioner)
Agreement about the need for greater interactions across the boundary between research and practice masked, once again, considerable divergence on the nature, validity and value of research knowledge. Among public policy-makers and public managers, commissioners felt that research was unlikely to command sufficient attention for the model of knowledge mobilization favoured by research providers to gain widespread uptake. And furthermore, regardless of the question of what model of knowledge mobilisation is desirable, there remained on both sides more practical hurdles to overcome if greater engagement was to be achieved:
Within government, the constraints have been a huge loss of staff, and reliance on management consultants, which makes it hard to outreach to the academic community. […] Within universities, people are so hard-pressed that they're not going out looking for things to do unless they contribute to the RAE or their own advancement, their own survival. (I9, university provider)
Discussion and conclusion
The previous section highlights radically different constructions of what constitutes appropriate research knowledge from commissioners and providers, and some important consequences of these for the relationship between research, policy and practice. In terms of the first research question – what notions of research knowledge are held by commissioners and producers in their respective roles in the research economy? – this study identified divergent views on the nature of research (a product or a mindset), the value of research (academic versus practical utility) and originality of research (academic versus policy currency). These perspectives seem entrenched by different performance management pressures on each side of the divide, though the academic construction of the value and nature of research also appears to be an integral part of academic professional identity.
This division gives rise to significant problems in the uptake and use of research, and it also poses challenges to some of the ways of increasing uptake that have been suggested. Caplan found that policy-makers were inclined to use academic research conceptually as well as instrumentally. 9 Although the conceptual use of research may involve a less immediate, more diffuse effect on research users’ mindsets, 19 much of what participants on both sides of the divide in this study say suggests that research knowledge is conceived in increasingly instrumental terms. Perhaps influenced by the legacy of evidence-based medicine, and the accompanying bureaucratic approach to implementation of the evidence it produces, 5 there remains a desire for scientific ‘answers’ that might be subjected to a linear, one-way process of transfer into practice. While this is a small-scale study, it covers both sides of the divide and a number of areas of public policy besides health care, and the suggestion from these groups is that pressures of performance management and rapid policy turnover mean that what managers and policy-makers increasingly want from research is ‘solutions’ to implement – and not a dialogical, interactive process of ‘knowledge intermediation’. 17
In terms of the second question posed in this paper – what might be done to assist the production of research that can be utilised readily? – it would seem that efforts to increase the quantity and quality of exchanges between research and practice face profound challenges, at least in the UK context with its strong performance management and wider institutional pressures on both sides of the divide. The desirability of models of ‘linkage and exchange’ between research and practice, such as knowledge brokerage 16 and the ‘organizational excellence’ model of learning from research, 20 is difficult to dispute – at least if we accept that evidence is malleable and context-sensitive, and we expect our public managers to be reflexive decision-makers rather than rule-following apparatchiks. 7 The feasibility of these models, however, seems more questionable. They face a context in which, according to respondents in this study and others,21,22 the pressures which deter practitioners and academic researchers from engaging reflexively are intensifying, and in which demands managers to implement research evidence that offers clear proof of effectiveness and efficiency are likely to redouble. The responses of a few of the providers in this study seem to suggest that because of pressures around research performance management, they are eschewing what Boyer called ‘the scholarship of application’ in preference for an all-encompassing focus on carrying out research, entrenching the division between producers, commissioners and users. 23 Efforts to increase and enrich the interface between health care research and practice, such as CLAHRCs, will need to address these deep-seated, structural obstacles to knowledge exchange if they are to do more than reproduce a one-way, instrumental relationship between research and practice, with all the limitations this implies to the potential utility of social research knowledge. This might require strategies to shift the incentive structures governing the production and dissemination of research to ‘recouple’ the scholarship of research and the scholarship of application. Such strategies might include more creative contract specifications that give greater emphasis to non-academic outputs, or the cultivation of longer-term funding relationships to develop a cyclical bond between research and dissemination. By challenging rather than reinforcing providers’ preoccupation with academic outputs and consumers’ demand for unpro-blematic research ‘products’, such strategies might reconfigure the research-commissioning relationship in a way that is uncomfortable, but potentially beneficial for both sides.
Footnotes
Acknowledgements
The authors would like to thank Alison Seymour, who carried out much of the data collection for this study, the participants for their time and views, and Phil Martin and two anonymous reviewers for their helpful comments. GPM would like to acknowledge the support of NIHR CLAHRC for Leicestershire, Northamptonshire and Rutland in the production of this paper. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.