Abstract
Objectives
Our aim was to evaluate whether the involvement of health care managers in research projects improves the quality and relevance of research, and whether collaboration builds capacity in the managerial community.
Methods
The NIHR Service Delivery and Organization Management Fellowship programme supports the direct involvement of health care managers in research projects. Data were collected from face-to-face interviews with management fellows and chief investigators of research projects at 10 case study sites. Data were analysed thematically using an adapted Kirkpatrick framework for programme evaluation.
Results
Management fellows improved the relevance and quality of research through enhancing its validity, efficiency and credibility. This was achieved by: using their contextual understanding to enable and support access and recruitment participants, data collection tools, processes and analysis; supporting dissemination activities; and undertaking additional work which was complementary to the main project. Capacity was developed through formal courses and exposure to new knowledge, ideas and practices. Factors found to enable or impede improvements in research included management fellows' knowledge and experience of the NHS, their background and personal characteristics, mutual respect, timing and flexibility. Consequences were not always predictable. Costs for management fellows included foregone opportunities, specifically for promoted posts. Researchers reported time-costs associated with administering the fellowship.
Conclusions
Collaborations between managers and researchers can improve research relevance and quality and research capacity development. Factors critical to success relate to the fit between the project and the management fellow and how clearly the purpose is understood.
Introduction
Interest in research collaborations reflects an international policy drive to make health care delivery and policy more evidence-based in the belief that this will improve quality and outcomes.1-4 In 2003, the Journal of Health Services Research & Policy published a supplement on collaborative research, defining it as ‘a deliberate set of interactions and processes designed specifically to bring together those who study societal problems and issues (researchers) and those who act on or within these societal issues and problems (decision-makers, practitioners, citizens)’. 5 At a similar time, Nutley et al. 6 described collaboration as ‘very much in vogue’ and ‘the way forward’ though noted that it was little understood. Such programmes build on empirical studies which suggest that a close relationship between researcher and user is more likely to encourage the uptake of research. 7,8
The number of programmes designed to support collaborative research in health care has grown. Current examples in England are the Collaboration for Leadership in Applied Health Research (CLAHRC) Diffusion Fellowships and the Service Delivery and Organisation (SDO) Management Fellowships. Both initiatives use secondments to academic research projects to address concerns about the translation of research into practice,9,10 though they differ in detail. CLAHRCs attract applicants who are managers or clinicians, whereas the SDO Fellowship programme is designed specifically for NHS managers.
The few evaluations of collaborative research programmes, largely from outside the UK, indicate positive outcomes. 11-13 The impact of context and process on the success of the programmes remains largely unexplored. 6-14 Furthermore, the majority of such programmes focus on clinicians and not managers. Yet, it is argued that managers face particular challenges in using research to inform their practice because of the nature of their decisionmaking, workplace culture and professional skills. 8,15-19 Despite this, managers are increasingly expected to use research evidence in their practice.8,20,21
This paper reports empirical findings from a formative evaluation of a collaborative research programme for health care managers, looking at the impact of the programme and factors influencing success. The SDO Management Fellowship programme is designed to allow ‘a practising manager from a health care organization that is involved or engaged in [a large-scale] research project to become directly involved in the project. 22 Fellowships run for the equivalent of 12 months full-time, typically spread over the life of the research project to which they are seconded. Programme objectives are ‘to improve the quality and relevance of the respective funded research projects through greater managerial involvement; to develop capacity in the managerial community for accessing, appraising and using research evidence; and to encourage greater engagement, linkage and exchange between the research and practice communities in health care management’. 20 This paper focuses on the first two programme objectives: how the contribution of management fellows improves the quality and relevance of the research, and how the collaboration develops capacity in accessing, appraising and using research evidence. It adds to current understanding through its focus on what features appear to enable or impede successful outcomes.
Methods
Fieldwork was undertaken at 10 sites through semi-structured face-to-face interviews conducted with all fellows and chief investigators and co-investigators, if appropriate. One fellow who had left the programme early was included in the evaluation. Interview questions focused on motivations for taking part, arrangements, experiences, lessons learned, and improvements for the programme. The evaluation focused on the programme rather than individual projects.
Data gathering and analysis was shaped by an adapted version of Kirkpatrick's framework for programme evaluation (Figure 1).23,24 The original framework focused on interlinked sets of outcomes: reactions to the programme (for example, did participants like it?); learning gains (for example, did chief investigators learn more about what was relevant to NHS managers?); behaviour change as a consequence of participation in the programme (for example, was what researchers learnt about administering research instruments applied to practice?). We adapted the framework in four ways. First, we recognised that reactions, learning gains and behaviour change are all outcomes or what SDO termed, ‘impacts’, and a key focus of the evaluation. Second, we acknowledged that these impacts could relate to the research project, the fellow and the workplace. Third, these impacts are linked to each other and not arranged hierarchically as in the original framework (e.g. learning might lead directly to outcomes). To illustrate the pathways, what research teams learn from the fellows might directly result in improved research relevance. Finally, the framework makes an explicit link between impacts and the processes and dynamics of the programme and projects (including motivations and set-up) and the wider context. Our analysis identifies processes that enable or hinder outcomes.
Programme Evaluation Framework (adapted from Kirkpatrick)
Data analysis entailed: development of a coding frame, structured by the framework and based on a priori themes derived from the evaluation questions and wider literature and emergent themes identified through discussion; independent coding by the evaluation team member leading on that site, using the coding frame; discussion of case studies, leading to the integration of themes and summary findings. All data were anonymised.
Results
Improving quality and relevance of the research
Did the management fellowships improve the quality and relevance of the research projects, and if so how? Definitions and measurement of quality and relevance are open to interpretation. In the language of the UK Research Excellence Framework, high quality research is that which is new or advances the field (original), is undertaken meticulously (with rigour), and attends to issues of importance (significant). In understanding ‘relevance’, we must ask, ‘relevant to whom?’ In this programme, relevant research would address questions of interest to health services managers. In this context, relevance would seem to overlap with significance. The projects themselves were not part of the evaluation so we did not look specifically at the originality of the research and only at its significance in relation to relevance.
Our focus is thus on aspects of rigour and relevance, although managers and researchers seldom used the terms relevance and quality but instead identified processes and outcomes which they felt contributed to better research.
Improving the quality and relevance of recruitment
Studies were improved by management fellows providing reportedly invaluable assistance in the recruitment of study sites and participants. They did this by being ‘on the inside’, knowing ‘the gatekeepers’ and ‘key informants’ and being able to engage them:
[MF] Susan can open doors that we can't…;does all the arrangements and knows the circulation lists to use to contact relevant people Able to identify the right people to invite and when/how to give the right sort of push. (Co-applicant Hilary)
MF Natalie used her knowledge of the study Trusts to advise the research team on whether the routine data they needed were actually available within the Trust. Based on her networks, she was also able to nudge those who had not been forthcoming. MF Debbie used her clinical expertise to help the research team make ethical and feasible approaches to very ill patients.
Improving the quality and relevance of data collection tools and processes
MFs used contextual understanding to improve the design of data collection tools and processes. They helped researchers develop appropriate questions for example. CI Priscilla spoke of how helpful she had found the the MF's ‘reality check - to have someone to say “you can't ask that stupid question’”. This was echoed elsewhere.
Other CIs reported the MF's contribution to designing data collection processes. For example, one team had decided not to use an online survey as managers told them that because of the quantity of email, they deleted much without reading. The MF argued that an online questionnaire would be opened if ‘it carried a member of staffs name [and if] the subject line &;said something like ‘provide feedback on the management experience’.’ The CI commented:
That might seem like kind of low grade advice but in fact if that contributes both to the design administration and response rate that will be an enormous assistance. (CI Evelyn)
MF Lesley piloted the recruitment process and interview schedule which was described by a project team member as a ‘valuable contribution’ to the research.
Improving the quality and relevance of analysis
It was reported that management fellows made positive contributions to data analysis and interpretation. CIs commented that the fellow provided a ‘reality check’ or ‘sounding board’ for the researchers' analysis:
Sometimes it's about checking, ‘does this make sense?’…; We were able to bounce ideas off her and say&; ‘in this context, does that kind of make sense?’. (CI Melanie).
Other fellows drew on wider networks to help validate emerging findings. MF Rosemary described how she tested an analytical model with colleagues:
There's been a lot of input from here that's channelled via me …;. I go to the consultants' meetings and say ‘I want 20 minutes, this is the model, what'd you think?’ So I'm a kind of conduit.
Rosemary described herself as a ‘clinician with a raft of other clinicians behind me’. The significance of this is captured in CI Ashley's comment that as insiders the MFs are able to validate findings and give them credibility:
Our MFs are not outsiders, they're insiders. …; It gives us an opportunity to&;say that we've not just checked with one person but that one person has also checked&;with consultant colleagues, medical colleagues.
Improving relevance and quality of dissemination
Fellows were reported to have improved the relevance of the research through supporting and offering guidance on dissemination activities. CI Ruth, for example, spoke of how the management fellow helped her understand how to report her research findings to increase their impact on practice:
It made us think more about…;how our outputs would make sense to people who weren't researchers&; Policy makers&;need to have&;very clear focussed nuggets&;that they can take and use very quickly&;because they're working in an environment where actually things change very fast&; Rosemary's made&;us think as a team a bit more about how we might produce much more focussed stuff and how we can deliver takeaway messages.
Many of the collaborations saw dissemination of their project findings as a key contribution of the fellow. However, few of them were at that point:
The test I think in the dissemination then will be how far we were able to communicate in a way that we might not otherwise have been able to do …; . Having Natalie guide us through the different levels and the different groups might be something that we would missed otherwise. (CI Kerry)
Developing capacity in tine managerial community for accessing, appraising and using research evidence
Did the management fellowships develop capacity in the NHS managerial communities, and if so how? The stated aim of the programme relates only to building capacity around accessing, appraising and using research evidence. Reflecting the responses of the participants, attention here is given to the broader contribution of the Fellowship programme to the development of fellows' understanding of research processes and use of research. The programme is concerned with individual MFs and their organisation. This paper limits its focus to the individual and we simply note here that moving capacity development to an organisation level is challenging.
Increasing capacity through formal learning
Fellows reported developing new knowledge and skills about research methods from taking part in formal courses. One of the research projects had run a study day for fellows:
I attended a qualitative research module on the MSc course which was just fantastic. I think it made bits of my grey matter work that hadn't worked for some time in that respect. (MF Laura).
Increasing capacity through exposure to research, new knowledge, ideas and practice
All the fellows reported learning new knowledge and skills through their exposure to the research teams. Typically this related to data collection, analysis and literature reviewing. Future plans for some fellows included co-authoring research papers and involvement in developing research proposals.
Having the opportunity to apply formal learning was important to some MFs. One described how she had progressed from reviewing transcripts to interviewing:
…to actually put some of that learning into practice. The opportunity to go on the module was fantastic but it's kind of better if I can try and implement. (MF Laura)
MFs gained improved access to knowledge and research relevant to their management role. For some this was through researchers loaning books and recommending papers but also through access to new research findings from the projects. MF Debbie, for example, commented that from an organizational perspective, ‘we can be first in there with&; improving the practice in that area’.
Barriers and enablers
There were barriers and enablers both to improving the quality and relevance of the research and to developing capacity for accessing, appraising and using research evidence. Many of the barriers and enablers were common to both objectives. One, knowledge and experience of the NHS, related only to improving the relevance and quality of research.
Knowledge and experience of the NHS
All participants were clear that the MFs' experience and knowledge of the NHS improved the quality and relevance of the research. Fellows provided shortcuts to information and news, and helped interpret this for the research team. CI Martin spoke of how MF Susan's ‘detailed up-to-date knowledge of the ‘nuts and bolts’ of NHS organization, including recent changes’ had provided ‘contextual sensitivity’. Some fellows were particularly valued by the research team because they also had specific knowledge or access to professional networks:
What made [Laura] interesting to us was her role at national level…;Having somebody with that kind of overview and access to high level [practitioners] within the service, I think, made her ideal for the project. (CI Ashley)
Characteristics of fellows
Fellows' characteristics were relevant to the perceived value of their contribution to research. What worked well depended on the needs of the research rather than a simple formula. For example, co-applicant Hilary thought that the ideal was someone at a senior level so that they could work to ‘change the culture in that particular Primary Care Trust plus inform us what the barriers are’. Others saw middle managers as well placed to effect change because they themselves performed the job being researched or directly managed teams who did.
A related issue was whether the fellow had an operational or strategic role. Again, what appears to matter is how the role fits with the needs of the project and with the fellows' aspirations for influencing change. Some CIs found benefits in the operational manager's insight into the realities of doing the jobs whereas others, seeking more high-level change across organizations or sectors, valued a manager in a strategic position.
The wrong fit can also occur when the MF does not have the networks or skills which the researchers assume. For example, CI Sally felt that her MF had neither sway in her NHS organization nor the research skills she had hoped for. In her view this limited the MF's service impact, and put the CI in an unwelcome role as trainer:
We've done a bit of joint analysis…;but this, I suppose if I'm honest, turns out to be a little bit aggravating because you end up in a kind of research trainer role rather than a more collaborative role &;. In a way it's as if I've taken on a novice research associate sometimes.
CI Kerry agreed that his research team ‘didn't think we were in the business of training up someone else to do the research&;That could have been a huge distraction’.
Role was thought to have implications for the feasibility of the fellowship taking place. Some argued that more senior managers were more difficult to recruit to a fellowship. For example, one study failed to recruit their first choice fellow because their more senior role meant they could not be easily substituted in the Trust. However, there was also a sense that operational managers were difficult to recruit for similar reasons. For example, MF Morgan explained that her manager ‘had reservations because&; losing an operations manager for any length of time&;has an impact on the service’.
This leads to a practical concern that support for the fellow from the line manager was important if the fellow was to participate at all. MF Morgan emphasised the importance of ‘making sure their line manager is 100% on side’.
The MF's personal characteristics were also seen to contribute to success. Their willingness to learn and contribute to the research as well as engage and be proactive in disseminating findings helped achieve both objectives. Relevant also was the extent of their on-going connection with workplace and professional colleagues to exchange knowledge and insights. Co-applicant Hilary reported how their MF was:
getting more involved in activities in the workplace …; any kind of engagement events, she will go. She will tell colleagues about the project. She's obviously learning&; as we start the analysis &; so it's feeding into work colleagues.
Mutual respect and being valued
Most fellows spoke of the respect they had for the researchers and welcomed the opportunity to learn from them. For example, MF Morgan was motivated to work with a particular university ‘because they're up there with the top universities.’ MF Natalie spoke of the benefit of working with ‘high calibre&;well respected’ people.
MFs reported feeling supported and valued by the researchers and most researchers were positive about what the MF brought to the research. This attitude built MFs' confidence and made them more inclined to engage, even if anxious at first:
I did feel quite intimidated about coming here…; I was quite anxious that&; I'd say something stupid &; but [the researchers] are so welcoming and so sharing. Robert [Research Fellow]&;he's been an absolutely terrific help. (MF Hannah)
Some MFs had researchers who acted as mentors and others built good relationships with the project team. Co-location of fellow and researchers assisted the flow of information and ideas.
Timing and flexibility
Timing of the fellowships influenced fellows' contribution. In relation to the timing of the fellowship, vis-a-vis the development of the research project, some fellows reported being underemployed while the project team focused on research ethics and governance. Much dissemination activity might occur beyond the fellowship period, although at the time of the evaluation a few were providing feedback to the workplace on preliminary findings. To facilitate the fellow's contribution to dissemination, one team had timed the fellowship to go beyond the formal end of the research project.
Another issue around timing is about the mode of attendance. Most of the fellowships were undertaken on a part-time basis and fellows organised their time around other commitments. A couple of fellowships were full-time. In one case full-time attendance was seen as the way to secure the release of the manger to take up the fellowship as it allowed the employer to appoint someone else to the NHS post on a temporary basis. In the other case, however, there were concerns that being out of the NHS full-time for a year reduced the potential impact of the fellowship on the NHS.
Being able to be flexible helped teams make the most of the opportunity. Projects were adapted to make best use of the fellows' skills and interests, but within the broader logic of the research project and fellowship programme:
I think it certainly developed and changed from what it probably was…; But also I've got a learning framework&;. So what I did was basically take the learning objectives from the application we put in and added a few. (MF Debbie)
The research funder's flexibility was welcomed:
Because the funder was fairly open, it actually gave a lot of scope…;. But it needs a bit of imagination and a bit of flexibility to understand the possibilities rather than trying to define it from the outset. (CI Ruth)
Summary
Barriers and enablers are summarised in Table 1, which also shows the dimension poles and how the ten MFs were located. Of key value was the MFs' knowledge of the NHS. The extent to which this enabled positive outcomes depended in part on how well their particular skills and workplace experience matched the needs of the research. There was no single, generalisable message about which levels and roles best serve projects. What was important was selecting the right person for each project.
Dimensions of the Management Fellowships
Costs
Costs to fellows
Taking part in the fellowships is not without costs. The costs for the fellow were in terms of foregone opportunities, specifically for promotion which could not accommodate the fellowship. This was true of MF Debbie whose dream job became available during the fellowship and to which she was asked to apply. However, she decided not to pursue the opportunity:
[I asked] ‘can I do it half time?’ ‘No, you'd have to give up your research’. Then I went away and thought there's no way I'm going to do that which was hard because it was…;the next step up.
Another MF, facing with a similar situation, decided to leave the fellowship.
Concern was expressed by one fellow who felt that leaving the NHS to undertake her fellowship made her less of an insider, potentially outdating her NHS knowledge and networks. Another concern expressed was that secondment could endanger the return to the MF's previous job.
Costs to projects
The costs to the CIs were largely in terms of time, including advertising and recruitment, chasing contracts and payments:
It's taken an enormous amount of our time to sort that out (CI Priscilla)
When fellows were made redundant by the employing organisation, or simply left (as in two cases), CIs spent time attempting to find ways to allow the fellowship to continue.
Other costs in time were incurred by CIs and members of the project team in supporting and providing training for the MF. Some research projects also reported that their fellow's travel and conference attendance costs had had to be met from the research budget and these had not been included.
Discussion
Management fellowships do impact on the quality and relevance of research, resulting in clear benefits for the project and researchers. These findings resonate with previous studies on collaborative research which identify improvements in research quality in terms of generating better insights and greater relevance 25 and improvements to the researchers' ‘practical and conceptual understanding of their fields’. 21 In contrast with studies reporting that payoffs may not occur in the short term, 26 here there is evidence of significant early benefits arising from these collaborations.
Management fellowships also developed the capacity of the managers who participated. Fellows gained in terms of personal development which accords with Garland et al. 27
The formative evaluation of the programme sought to understand process and contextual issues that led to success. In line with some other studies, soft factors were found to be important, for example trust 5 and mutual respect. 27 Concerns about hostility were found in earlier studies25,27 but not in our responses from MFs about CIs. We did not find a lack of skills on the part of the researchers to engage, nor barriers arising from power differentials. 21 Some of the advice offered in the literature, such as academics being less arrogant, 25 has limited relevance in this context. Commitment to equality and mutual respect were characteristic features of the CI/MF relationship. Reasons for this are uncertain but may relate to the recognised value of managers' input and to many fellows being previously known to the research team or coming with recommendations from people known to the team.
The observed mutual respect supported the development of other factors which assist success: a willingness on the part of the research team to accommodate the fellows' aspirations and a readiness to adapt to changing circumstances on both parts. We agree that ‘the success of collaborative research depends as much or more on the people involved as the processes they put in place’. 5 However, processes can be established that enable the most to be gained from the programme.
The fellows helped to make the research more relevant and the results more credible with their colleagues but only if the fellows fitted the particular project. In judging the suitability of a manager for a Fellowship, CIs need to consider whether they have appropriate connections, seniority, credibility and backing from colleagues. Potential fellows need to think how the fellowship fits in their career plans, what the role will entail, whether the research offers appropriate opportunities for skill development and whether they are likely to succeed in negotiating release for the secondment. The timing of fellowships needs to be matched with the expectation of role. Impact may be limited by project aims and methods predating the fellows or the fellowship ending, prior to dissemination activity.
Failure to articulate clearly the purpose of the programme is liable to impact negatively on its success. We highlighted an absence of definitions of quality, relevance and capacity. The lack of clarity about purpose was clearly confusing for some CIs, who felt they were finding their own way. Not all CIs appeared to see adding to research capacity as part of their role. The programme could therefore benefit from clearer articulation by the funder about what is expected, with quality, relevance and capacity defined. A clearer idea of what to strive for could assist project teams specify the sort of fellow they need and the best timing of the fellowship, and could help to better identify the kind of support and training needed.
Our study has limitations. The evaluation did not aim to test capacity development or provide independent assessment of the impact on research quality and relevance. We relied on proxy, process-based indicators, self-report and triangulation of responses from participants. The potential for over-reporting benefits has been minimised by the guarantee of confidentiality; we heard criticisms of individuals and these observations are echoed in our reporting of barriers and enablers. The evaluation reflected a limited time period and respondents' views may alter over time. This also means that we have been unable to explore events such as dissemination of the projects' final outputs. Further, we focus only on the views of the MFs and senior researchers and not with developing engagement, linkage and exchange between the research and practice communities. 28
Conclusions
The programme goes some way towards achieving its objectives. The MFs helped to improve both the relevance of the research by focusing it on the interests of managers and the quality of the research through improving the validity and credibility. They also helped to ensure that dissemination to managers would be more appropriate. MFs improved their individual capacity to make use of research, mainly though improved understanding of research processes, although perhaps more broadly than originally envisioned. Contextual factors enabled or impeded the process of collaboration: the fit of the MF to the research project and team was a critical factor and the scope to improve the relevance of the research was limited by the timing of the fellowships.
Some of the reported benefits of the programme were serendipitous and this, together with the reported problems around understanding the roles, indicates that a clearer articulation of how the fellowships are expected to work could be beneficial. Contribution analysis or logic modelling29,30 could be useful. For example an explicit aim to improve capacity through fellows developing skills to access and appraise research could identify training to address these needs. Making explicit links between processes and outcomes would enable the success of such programme to be more easily tested and replicated in future. Another would be the need for fellowships to be built into funding applications when research aims are developed.
One significant conceptual challenge is how the micro-level nature of fellowships will impact on the patient care. We have reported evidence of increased capacity of individual fellows but this will make little difference if they are unable to affect change in the workplace. How these collaborations impact on wider workplace practice is a question the evaluation has considered and we will report on in future.
Footnotes
Acknowledgements
This evaluation was funded by the NIHR SDO (project number 09/ 1003/01). The views and opinions expressed are those of the authors and do not necessarily reflect those of SDO programme, NIHR, NHS or the Department of Health.