Abstract
Objectives
To report on selected findings from an evaluation of two consecutive quality improvement campaigns that adapted the Breakthrough Series Collaborative model to the broad topic of reducing demand for hospital care and reflect on lessons learned from their adaption of the model for subsequent collaborative improvement efforts.
Methods
We conducted a series of semi-structured interviews with Campaign sponsors and Collaborative team leaders as part of the broader realist evaluation of the two Campaigns. In addition, follow-up semi-structured interviews with Campaign sponsors and implementers were undertaken three years after the evaluation concluded (ex post evaluation interviews) to understand which adaptions to the Breakthrough Series Collaborative model had been the most influential.
Results
The interviews explored two features that differentiated the Campaigns from other Breakthrough Series Collaboratives. Firstly, the Campaigns enabled a diverse range of improvement solutions to be tested which had implications for the collaborative nature of the learning collaboratives. Secondly, two sequential Campaigns were implemented that incorporated common elements and provided the opportunity for the transfer of knowledge from one Campaign to the next.
Conclusions
Given widespread pressure to transform health care into a learning system, this paper provides a practical example of using cumulative insights to encourage the sustainability of collaborative improvement efforts. These insights centre on the gains from spreading improvement methodology throughout the organization and learning how to select and support successful collaborative teams.
Introduction
The Breakthrough Series (BTS) framework is a change model that builds on collaborative learning to achieve quality improvement in health care. Developed by the Institute for Healthcare Improvement (IHI) in the USA,1–3 its central tenet is the idea that small-scale changes have the potential to accumulate into large effects. A key element of the framework is the Model for Improvement, 4 which incorporates four critical components of successful process improvement: aims that are specific and quantifiable, appropriate measures that can be tracked over time, key changes that will bring about the sought after improvement, and a series of testing cycles (known as Plan Do Study Act or PDSA cycles) during which the key changes are refined and applied by teams to their organizations.1,4
The BTS framework has been used widely to support and accelerate health care improvement. In this paper, we present the experience of Counties Manukau District Health Board (CMDHB), which is one of 20 District Health Boards in New Zealand that are responsible for planning and funding health services for their populations. In 2011, CMDHB launched the first of two consecutive quality improvement campaigns primarily to address a projected increase in demand for hospital bed capacity 5 while also striving to ‘give back’ healthy and well days to the local community. The first 20,000 Days Campaign (October 2011–July 2013) sought to reduce hospital demand, equating to 20,000 bed days by returning 20,000 well and healthy days to the community, 6 followed by the subsequent Beyond 20,000 Days Campaign (July 2013–June 2014), which aimed to continue returning healthy and well days to the population of Counties Manukau. 7
The Campaigns were subject to an independent evaluation, which focused, largely, on how the Campaigns achieved improvements and to provide formative lessons for future campaigns.8,9 In particular, evaluative attention focused on the social processes and mechanisms that enabled (or hindered) the work of a wide diversity of collaborative teams, and it concluded that each Campaign was likely to be moving the system towards better management of demand. However, the scale at which this was happening was likely to be continuously shifting as the work of the most successful collaborative teams spread to more sites.
In this paper, we build on the findings of the independent evaluation of the Campaigns, reflecting on their dynamic and sequential nature, along with the distinctive application of the BTS model, to elaborate on the lessons learned from their adaption of the model for subsequent collaborative improvement efforts.
Methods
Study context
The 20,000 Days and the Beyond 20,000 Days Campaigns have been described in detail elsewhere.6,7 In brief, collaboratives that adopt the BTS framework typically have a single clinical focus in an area where evidence exists regarding best practice but is not widely implemented. 10 In contrast, teams in the CMDHB Campaigns were tasked with generating improvement ideas that linked to the overarching goal of reducing hospital demand. The two Campaigns involved up to 29 Collaboratives comprised of 8–10 members, each testing a diversity of quality improvement solutions that aimed to: (i) reduce hospital admissions and length of stay; (ii) increase patient access to community support; and (iii) reduce harm to patients and subsequent readmissions. Each Collaborative selected a specific measurable aim to be achieved during the Campaign they were part of. In total, 24 teams maintained momentum until the end of the Campaigns (10 teams in the 20,000 Days Campaign and 14 teams in the Beyond 20,000 Days Campaign).
Lessons learned from the 20,000 Days Campaign that informed the implementation of the Beyond 20,000 Days Campaign.
Approach
A realist evaluative approach was adopted to understand more about what worked in the Campaigns and in what contexts. 11 It used a mixed-methods design with three phases of data collection undertaken between March 2013 and November 2014: (1) a review of Campaign planning records from the first Campaign and semi-structured interviews with key informants to develop an initial logic model of the rationale and expectations underpinning the initiative; (2) semi-structured interviews with Campaign sponsors and team leaders, a questionnaire emailed to all participants, and secondary analysis of teams’ dashboards of quantitative outcomes and measures to assess the first Campaign; and (3) semi-structured interviews with team leaders, in-depth case studies of four teams, and a secondary analysis of the overall system-level dashboard to evaluate the second Campaign. Analyses involved an iterative process of testing against the initial logic model to progressively develop a more refined theory about how the Campaigns achieved their results; the details of the evaluation and key findings are reported elsewhere.8,9,12
Following the realist emphasis on understanding how and why people respond to interventions, 11 we were further interested in how the ‘practical wisdom’ derived from operating a diversity of collaborative teams was applied in subsequent Campaigns. The term practical wisdom refers to the dynamic adaptation shown by those involved in the design and implementation of the Campaigns. 13 To build this understanding, we drew on, first, semi-structured interviews with four Campaign sponsors in 2013 and 2014 explaining how they adapted their processes between the Campaigns. These interviews were initially used to develop a logic model of how the Campaigns were expected to achieve their effects for the main evaluation. We revisited these and explored commentary on how processes were adapted over time. Second, we used semi-structured interviews with five Collaborative team leaders in 2013 and 2014 who participated in both Campaigns. Again, these interviews were used as case study material to inform the main evaluation. We revisited these and investigated reflections of what was different between the Campaigns. Third, we carried out follow-up semi-structured interviews with five Campaign sponsors and implementers in 2017. These ex-post evaluation interviews occurred three years after the main evaluative reports. The questions in these interviews were shaped by our in-depth exploration of the adaptions that had occurred between the Campaigns, drawn from the above-mentioned interviews from 2013 and 2014. Specifically, they explored the significant features of the 20,000 Days and Beyond 20,000 Days Campaigns that distinguished them from other BTS Collaboratives, and how subsequent CMDHB initiatives14,15 were influenced by what was learnt from these first two Campaigns.
All interviews lasted between 30 and 45 minutes and were held over the phone or in person. All interview participants provided informed consent either verbally or in writing.
Results
Figure 1 describes the key findings from the main evaluation. The findings confirm that the Campaigns benefited from a culture responsive to change, visible support from senior management and team leaders who motivated, rather than just managed their collaborative teams. Constraints centred on a lack of consensus about what changes would be most likely to reduce hospital demand, and confidence in the measures used by teams to track change. 12

Summary of how the Collaborative teams worked to achieve the Campaigns' outcomes.
As those implementing the Campaigns gained an understanding of what characterized successful and unsuccessful teams, future Campaign activity was adapted in light of these findings. In what follows, we report on two features that dominated the adaptive thinking of those implementing the Campaigns. These features were initially flagged in the main evaluation and selected for in-depth follow-up in the ex post interviews: (1) The implications of testing a diverse range of improvement solutions rather than a single clinical best practice or bundle of best practices; and (2) the ways in which implementation insights transferred from one Campaign to the next.
A diversity of improvement solutions
The CMDHB Campaigns encouraged participants to come up with their own project ideas for redesigning care processes that would reduce hospital demand and keep people well in the community. This resulted in a plethora of different ‘solutions’ that varied not only in the topic area chosen for the intervention but also in the development of process and outcome indicators required to measure progress: … these were very different from standard collaboratives, which … have a single proven methodology and you’re spreading it. These were, in a way, a different beast. They were an innovation collaborative. We were testing many flowers and identifying what worked. (Campaign sponsor, 2017 ex post interview) The tools we learned, and it is hard to reflect on now as they have started to be part of the culture, this is how we test using the PDSA cycles; this is how we measure, these are now part of the air that we breathe. (20,000 Days Collaborative team leader) The challenge which was so different … is this idea that we didn’t have any solutions like the traditional IHI [Breakthrough Series Collaboratives] … We were trying to come up with the solutions as part of the work … (Campaign sponsor, 2017 ex post interview)
Two sequential Campaigns: Learning what works
A second distinctive feature of the CMDHB Campaigns was having two sequential Campaigns with elements that flowed from one to another. This included a shared commitment to managing hospital demand, and a reliance on the same organizational support and branding. The openness to ensuring key lessons were transferred from one Campaign to another is captured by the following reflection: … what we were doing at a higher level in the Campaign was in effect doing a PDSA cycle ourselves. We weren’t sure how this was all going to pan out. We definitely, in the true spirit of improvement, reflected a lot on what had worked and what hadn’t worked in the first Campaign to help inform what we then did in the follow up one. (Campaign sponsor, 2017 ex post interview)
Lessons from teams that succeeded in the first Campaign included the need to support even more robust measurement practices in the second Campaign to ensure the sustainability of improvement efforts in terms of having an intervention picked up and funded as mainstream business. Successful collaborative teams were able to access time-limited funding set up for their improvement ideas. To secure permanent funding, business case requirements in the second Campaign increased efforts by teams to quantitatively link their improvement to the larger Campaign goal of giving back healthy and well days to the local community and thereby demonstrate potential costs and savings. Table 1 provides detail on lessons learned according to different aspects of the Campaigns.
Ex post interviews highlighted several additional emphases for subsequent CMDHB improvement efforts that built on the earlier lessons learned. These included incorporating co-design with patients as a fundamental component of any initiative that seeks to engage effectively with patients and their families, understand what is important to them and design improved services that meet their needs: The routine use of co-design with patients and their families, we’ve learnt that that should just be a standard part of the work now. (Campaign implementer, 2017 ex post interview) What we learnt was that we tried to get more teams to have a more specific understanding of what their problem was and clearer baseline data. (Campaign implementer, 2017 ex post interview). We absolutely, over the whole series of collaboratives, were aiming to build a network, build a movement, build a capacity and capability … (Campaign sponsor, 2017 ex post interview).
Discussion
It is not uncommon to find that successful health care quality improvement initiatives fail to transfer to different settings due to local contextual factors. 13 Moreover, many methodologies do not facilitate the exploration of the contexts underlying how and why quality improvements fail or succeed in different settings. The Breakthrough Series Collaborative has been widely used as a vehicle for improving the quality of health care, using collaborative learning to achieve breakthrough improvements in the quality and safety of health services. Our results highlight what has worked and not worked when the BTS model was applied to the broad topic of managing hospital demand within a New Zealand District Health Board.
Our realist evaluative findings confirmed much that was shown in other research regarding the importance of having the right organizational context for improvement, voluntary participation and the ability to work as a team.16–19 However, what proved of more interest and prompted a further investigation was an examination of the ways in which the evaluation findings influenced subsequent quality improvement efforts at CMDHB. These insights focus on the practical wisdom that was deployed in the design and running of subsequent Campaigns. 13 This practical wisdom involved the dynamic adaption of the BTS model in two key aspects.
First, while the BTS model has been adopted on a large scale by various organizations,20,21 it is relatively unusual to incorporate a diversity of improvement solutions within the scope of a single initiative. In single topic campaigns, information sharing among teams across institutions and within the same institution is expected to drive improvement through a cooperative group effort, or in some circumstances, competition may be the main motivator for change. 22 However, given the diversity of improvement solutions being tested in the CMDHB Campaigns, there was little opportunity for benchmarking between teams. As Øvretveit et al.23 explain, when a broad topic is selected for a quality improvement collaborative, each team is likely to focus on different change ideas and there may not be ‘clear or directly comparable examples of best practice, change concepts, or good research evidence’ (p. 347). What did serve to unify the teams in the CMDHB Campaigns, however, and provide essential peer stimulus was the shared experience of applying the BTS methodology to the broad topic of reducing demand for hospital care. This had benefits for spreading knowledge of quality improvement methodology throughout the organization, creating a network of health professionals with the methodological expertise to bring about sustainable improvements in their practice.
Commentators on quality collaboratives have pointed out that a common set of measures is useful for monitoring and comparing improvement. 23 In traditional Breakthrough Series Collaboratives, the specific content for the collaborative is developed by expert faculty and presented to participants at the start of the initiative as part of a predefined change package that includes collective wisdom around measurement. 1 However, given there was more uncertainty in the CMDHB Campaigns regarding what changes would have the most impact on managing hospital demand than typically experienced in pre-defined single topic campaigns, so the complexity of measurement tasks undertaken by collaborative teams was increased. Adequate time and support are therefore needed to develop appropriate measures to assess the effectiveness of quality improvement collaboratives in such circumstances.
Second, in transitioning between the two Campaigns, there was a concerted attempt to carry forward what had been learned from the first Campaign. This involved building on successes in the 20,000 Days Campaign, as well as learning from failures to ensure that the Beyond 20,000 Days Campaign had the ability to select and support successful collaborative teams in their work to produce sustainable quality improvements in practice. Further, examination of how key learnings continue to influence and shape the design of more recent Campaigns, reinforced the cumulative role of CMDHB’s initiatives for growing new models of care and spreading knowledge of improvement methodology, thereby encouraging the sustainability of collaborative improvement efforts across the sector.
Recognizing that health care systems are complex adaptive systems that alter according to their own localized rhythm, the Campaigns avoided mandating solutions from the top. In this aspect, they match advice to nudge health care systems towards change. 24 Those leading the Campaigns harnessed the creativity of those with an idea for improvement, but also needed to marshal the resulting diversity of ideas into a collective effort that would have a recognizable impact on the Campaigns’ goals. Caution has been raised that allowing a thousand flowers of quality interventions to bloom is not always efficient. 25 However, in this situation any inefficiency from teams that failed was converted into learning for future campaigns.
Footnotes
Acknowledgements
We wish to offer a very appreciative thank you to those involved in both Campaigns who shared their insights and experiences. This paper draws from the findings of two evaluation reports and follow-up that involved a wider team comprising David Mason, Janet McDonald, David Grayson and David Codyre as well as research support provided by Madeline Fuller. We thank them for their insightful contributions.
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DD, JG and LV were employed by the organisation delivering the Campaigns. DD and JG had direct involvement in delivering the Campaigns.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Counties Manukau District Health Board, which commissioned an independent evaluation of the Campaigns. Approval was sought for the publication of these results as part of the contract for services.
