Abstract
A number of evaluations of models of integrated care have not produced the expected result of reduced hospital admissions, and in some cases have even found people receiving integrated care services using hospitals more than matched controls. We tested three hypotheses for these surprising results with a group of 50 integrated care experts in a seminar: (1) problems with the model; (2) problems of implementation; and (3) problems of evaluation. Our group of experts did not rule out any of these hypotheses and came up with some advice as to manage these issues. For example, model designers should rigorously test the underlying logic; commissioners should seek out advice from experts and patients/professionals; and evaluators should choose outcomes wisely, use mixed methods approaches, and provide regular feedback loops to implementation sites. Evaluating integrated care is a skilled task that requires multiple approaches in terms of the design and implementation of the models. National research funders or other appropriate bodies might consider developing an advisory service to provide support to local systems planning evaluations.
Introduction
Within our own work and elsewhere, we have found a number of curious cases in which the data suggest that people receiving integrated care were using some hospital services more than a carefully selected group of matched controls.1–3 Studies like these stick out, even in an equivocal evidence base that cannot agree whether integrated care initiatives can reduce the use or cost of services in the community and in hospitals.4–7 The curious studies we highlight raise questions about why such unexpected results are still being found decades into the integrated care journey in England, and whether the aims of integration, specifically surrounding reducing hospital admissions and improving system efficiency, will ever be consistently achieved. In this paper, we examine potential flaws in the design, delivery and evaluation of integrated care programmes and make recommendations to improve how integrated care initiatives are planned and evaluated.
We discuss these flaws in the context of three hypotheses. The hypotheses and the recommendations themselves are the culmination of ideas gathered from the integrated care literature base and discussions with 50 integrated care experts who attended a seminar organised by the Nuffield Trust and The Health Foundation in May 2019.
The hypotheses
Hypothesis 1: Integrated service models are poorly designed
Poor design, such as flawed theories of change, could result in integrated care programmes, and indeed other complex interventions, not achieving their desired results. For example, failing to recognise that hospital admission rates are affected by a range of contextual factors, including other health system policies and initiatives, is not wholly amenable to be reduced by the integrated care programme. Other flaws in integrated care models include a failure to fully consider patients’ needs and preferences, which often relate to maintaining access and continuity with particular providers.8–10 These misdirected design principles can limit the partners involved, progress made, and a programme’s ‘success’.
Hypothesis 2: The delivery of the model as intended is too demanding in real-world settings
Integrated care schemes can be difficult to implement as designed. Challenges can include: interventions not being fully implemented (as we outline below in terms of MDTs and primary care), interventions not being appropriately targeted (either at the right group of patients or in terms of the anticipated outcomes), and interventions being implemented and having unexpected (additional) effects; these are described below.
Multidisciplinary teams, which often form part of integrated care models, may fail to deliver because they are lacking a specification of team objectives and clear role delineation. They could also be experiencing relational difficulties, poor team working and poor communication – the phenomena of ‘pseudo-teams’ in health is a well-described problem, 11 but may not be easily visible to evaluators. They could also be facing time pressures and lacking adequate training opportunities.7,10
Complex local and national contexts, including local politics, can also undermine integration efforts. The full engagement of primary care in these models is recognised as particularly challenging due to pressures of time and workload and a lack of standardisation of care and systems between general practices.12,13
Implementation issues also arise when interventions are not appropriately targeted. For example, when an integrated care intervention that aims to reduce hospital admission rates is mainly targeted at individuals at low risk of hospital admission, making it less effective than if it had targeted high-risk populations. 14
The inherent complexity and biomedical nature of some integrated care models may also be causing unexpected dynamics, such as higher admissions, because intensive case management models can sometimes uncover undiagnosed pathology or medicalise aspects of life that individuals had been coping with.
Hypothesis 3: The evaluations of integrated care programmes are challenging
There are a number of challenges to evaluating integrated care programmes including: unrealistic expectations of the evaluation, insufficient time, availability and reliability of outcome data and a focus on asking if integrated care ‘works’ independently of context. Evaluation can be challenging to deliver when it is commissioned (implicitly or explicitly) to prove an intervention works, rather than to find out if it works. Although understandable when expectations are high for health improvements and cost savings as the outcomes of integrated care initiatives, 5 this undermines the scientific principles of evaluation research. This optimism bias can also lead to unrealistic timelines that do not allow sufficient time to accommodate for implementation delays, for changes to occur during implementation and for the full realisation of benefits, which may arise after the intervention’s trial is over.7,15,16 These expectations are also at odds with the trend of marginal gains that are often seen in evaluations, especially in local areas where it is known that usual care (the counterfactual) is already performing well.
Another related challenge is that evaluations of integrated care tend to focus on a limited number of outcomes. The scope of outcomes often exclusively focuses on the available health data and current policy concerns (e.g. emergency admissions), rather than impacts on patient experience and satisfaction4,17 and other linked sectors (e.g. housing). Moreover, measuring genuine changes can be compromised over time by changes in data collection methodology and definitions, a lack of reliable controls, or by differences in coding practices across areas (e.g. zero day cases). 18
Finally, many commissioners of evaluations and evaluators assume that comparative or controlled research designs will provide answers to questions that are often more complex than simply ‘did integration work?’ It is widely acknowledged that the design and delivery of integrated care initiatives are not straightforward, and their evaluations tend to overlook ‘how integration is happening’ and narrowly focus instead on measuring ‘what is being achieved’ (or not). Process evaluations and embedded approaches are able to describe the model over time and at multiple levels, explore the variable penetration and fidelity and undertake the necessary ‘sensemaking’ to understand what, for example, integrated care might mean for the local health economy, as well as examining outcomes.
Which hypothesis explains the confounding results?
No single hypothesis completely explains the perplexing evaluation finding of intervention groups receiving integrated care using hospital services more than carefully matched control groups, rather it is likely that each of these hypotheses could be proved at different points across different integrated care programmes. However, our experts came up with the following advice to help to evolve approaches to evaluating integrated care.
Advice to service model designers
Designers should start by carefully defining the problem and aims of the proposed initiative before assuming that integrated care will fix local issues. Where integration is the answer, designers should clearly describe how the initiative will work, as well as investigate local barriers and needed enablers.12,19–22 Logic models can usefully visualise links between inputs, activities, outputs and outcomes, as well as distinguish evidence-based claims from unjustified assumptions. 23 Narrative approaches describing the theory of change can help make sense of events and data.
An important part of the design process involves co-production with all key stakeholders – including patients and professionals. 15 Evidence seems to suggest that frontline professionals want more opportunities to feedback and influence integrated care models, 10 and where change is forced upon staff delivering interventions they are less likely to support the new activity. 12 Thus, models should be designed in layers: starting with outcomes that patients believe are important and plausible; then, by considering how the behaviours of the systems and professionals will influence implementation; and finally adding in structural considerations. The potential for tensions between conflicting aims for the model need to be acknowledged and managed, for example, improved processes and patient experience could increase costs. 24
Designers should note that substantial time and resources will be needed to ensure that mutual understanding of the initiative extends beyond the core project team. This is especially important where an initiative’s aims are more ambitious and aim to transform the way care is delivered, or where partners span primary, secondary and social care services. 12
Advice to commissioners of evaluation
Evaluations are often commissioned in a policy climate of wanting to show that the solutions work,
15
which can lead both experienced and inexperienced commissioners of evaluation to set unrealistic evaluation timelines and expectations. Instead, commissioners should:
use longer evaluation time cycles, possibly encompassing ‘short’ and ‘long’ packages of work, to allow time for change to happen; talk to local evaluation sites about the potential evaluation enablers and access to data before commissioning the research; request regular formative feedback between evaluators and local sites to inform iterative changes to the model and evaluation approach; encourage evaluation teams to prove they are building on existing evaluations by seeking out or developing a community of evaluators to allow for comparison of integrated care initiatives, reflection and overarching narratives to form (i.e. meta-evaluation); and use co-design with patients and professionals to bring in the expertise of people who have participated in and/or evaluated integrated care first hand to clearly set out in tender documents the research questions, resources available, the purpose of the evaluation, timelines and intellectual property rights.
The specification of evaluation methods in tender documents can be unhelpful, as bidders may bring varied experiences and skills; however, economic analysis and purely quantitative approaches will not necessarily be able to engage with the complexity associated with delivering integrated care services. Instead, evaluators should be encouraged to describe how the model of integrated care is being implemented and unpick the local context influencing the model. 15
Advice to evaluators
Evaluators should ensure that the outcomes are meaningful and align with the initiative’s programme theory. Engaging commissioners, policymakers and governments in the evaluation will help to shift their thinking away from popular yet not always appropriate measures, such as avoided admissions. Consider for example, whether clinical markers of health (such as control of HbA1c in people with diabetes), patient-reported health status or care experiences, staff perceptions, or early markers of change might be more appropriate rather than focusing on emergency admissions.
15
However, where emergency admissions is the right measure, evaluators should:
be explicit about how admissions are counted and coded locally and in any potential control areas (i.e. zero day admissions may or may not count); describe why admissions for certain populations (e.g. older frail people, homeless populations) can be appropriate and necessary; explore through primary qualitative (and additional quantitative) research whether increases in admissions resulted from identifying and addressing – and potentially over-medicalising – unmet need (or some other reason); and explain limitations of the analysis, such as ceiling effects limiting the ability to show any improvement (as standard care is good) or challenges in finding a counterfactual (as many local sites are undertaking integration of some form).
The limitations of some of the existing quantitative approaches highlight the need for more qualitative and mixed-methods work. For example, case studies to examine complexity or participatory process evaluations to co-produce knowledge about the intervention (see, for example, Eyre et al. 25 ). Regardless of study design, a description of the history and context of the starting point of the evaluation are essential information. Primary research might be needed to comprehensively evaluate patient and carer outcomes, but evaluators should be prepared for the greater time and other resource requirements, and the need for research governance.
Evaluators should aim to provide ongoing, accessible feedback to local sites to inform implementation and decision making. 26 It is helpful to describe in detail the work of implementation teams who are delivering the integrated initiative 27 and to articulate the active ingredients, major challenges and quantitative measures, so that the model can be spread. Descriptions should focus on what is happening, rather than what is missing, but should include acknowledging the effect of weak, delayed or changing implementation as a key variable associated with outcomes.
Finally, evaluators should work together to undertake fewer evaluations that are better and deeper, building on what is already known. For example, formally via meta-evaluations or less formally via communities of practice.
Conclusions
This area is beset with complexity that seems to emerge from: uncertainty about the nature of the integrated models and how they work, potential lack of agreement on the objectives of integration, local context and the impact of other features such as the quality of internal or inter-organisational relationships and the ability of local leadership. This complexity is not a reason not to evaluate but it does mean that well-designed approaches are required.
The seminar that produced these conclusions brought together investigators and people responsible for designing models and commissioning evaluations. There was a striking level of agreement about the need for multiple approaches in this area in terms of the design and implementation of the models, the definition of success, the approach to evaluation and the time required to do this.
National research funders or other appropriate bodies might consider developing advisory services to provide support to local systems planning evaluations. Commissioning evaluations is a skilled task, and additional help and guidance for health and social care systems could be beneficial.
Footnotes
Acknowledgements
The hypotheses and recommendations we outline were tested and formed, respectively, from discussions with 50 integrated care experts who attended a seminar organised by the Nuffield Trust and The Health Foundation on 1 May 2019. Attendees represented providers, researchers and commissioners of evaluations. We appreciate their contributions to the event.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
