A Structured Process Description of a Pragmatic Implementation Project: Improving Integrated Care for Older Persons in Residential Care Homes

Abstract

Evaluation of the implementation of integrated care can differ from trial-based research due to complexity. Therefore, we examined whether a theory-based method for process description of implementation can contribute to improvement of evidence-based care. MOVIT, a Dutch project aimed at implementing integrated care for older vulnerable persons in residential care homes, was used as a case study. The project activities were defined according to implementation taxonomy and mapped in a matrix of theoretical levels and domains. Project activities mainly targeted professionals (both individual and group). A few activities targeted the organizational level, whereas none targeted the policy level, or the patient, or the “social, political, and legal” domains. However, the resulting changes in care delivery arrangement had consequences for professionals, patients, organizations, and the social, political, and legal domains. A structured process description of a pragmatic implementation project can help assess the fidelity and quality of the implementation, and identify relevant contextual factors for immediate adaptation and future research. The description showed that, in the MOVIT project, there was a discrepancy between the levels and domains targeted by the implementation activities and those influenced by the resulting changes in delivery arrangement. This could have influenced, in particular, the adoption and sustainability of the project.

Keywords

implementation pragmatic process description delivery of health care integrated integrated care older persons homes for the aged frail elderly health services research

Background

There is broad consensus among medical professionals and policy makers that the concept of integrated care offers opportunities in meeting the demands of the growing group of older persons with combined care needs in the population.^1-3 Although interpretation of the concept of integrated care varies, it generally involves integration of cure and care on one hand, and user and provider perspectives on the other.³ Practical applications are often based on the Chronic Care Model.^4-8 In this latter model, improved functional and clinical patient outcomes are achieved through a productive interaction between an informed and activated patient and a proactive care team. This interaction takes place within the context of a supportive community and health care system.^9-11

Implementation often takes the form of complex intervention projects in a system with multiple independent professional disciplines, and organizations with individual objectives, operating and adapting in a changeable environment while performing multiple interventions in various ways, to differing degrees, resulting in nonlinear, disproportionate, and unforeseen (emergent) outcomes.^12-14 Project components can be adapted to the changes that occur and the outcomes that emerge, and can evolve to attain the project objectives.

While comparative trial-based research on the causal mechanisms between the interventions and their outcomes is often preferred, real-life implementation projects can encounter problems meeting the rigorous demands of such an evaluation approach.^13-18 In the present study, the term pragmatic is used to characterize a real-life implementation project with a high degree of adaptiveness which is aimed more at demonstrating the applicability of the approach than the efficacy of an intervention.^19,20 To study complex interventions in the context of evidence-based health care, various (research) conceptual frameworks have been proposed, such as those based on a realist approach, sociological theory, mathematical modeling, program theory, and the theory of complex adaptive systems.^13-15,21,22 Besides these frameworks, approaches with a more descriptive character have also been used, such as those for strategies, determinants, and levels of implementation interventions.^23-25

We propose that a structured process description of pragmatic (adaptive) care implementation projects (which do not meet the requirements of specific [research] conceptual frameworks) can also offer a valuable contribution to the evidence-based development of improved health care. This article describes how, retrospectively, a matrix was developed, using descriptive frameworks, to describe the process of a pragmatic real-life implementation project which does not conform to the research requirements of existing (research) conceptual frameworks.

During the development of the matrix, the MOVIT project was used as a case study. The MOVIT project (as part of a national elderly care improvement program in the Netherlands) aimed to develop and evaluate an implementation strategy for integrated primary care for older persons with complex needs living in residential care homes.²⁶

Toward an ideal of integrated care, the intended improvement in medical care was to coordinate the care provided by general practitioners (GPs), elderly care physicians, pharmacologists, and nursing staff, and to initiate a long-term process so that further steps would be taken in that direction. The general strategy was that, within a larger administrative region, local working groups were initiated and supported per residential home and allowed a large degree of freedom in determining their local priorities and steps toward integrated care. Support was provided for the local groups by information provision, logistical support, and team coaching. At a regional level, educational sessions were organized and support was provided in negotiating facilities and terms.

At the start of the project, the project components were developed based on (a selection of) (1) general implementation theory, (2) a small-scale pilot, and (3) interviews with stakeholders (care financiers, residential care home governors/managers, GPs, nursing staff, pharmacists, and elderly care physicians).²⁷ During the 3-year project, components were adapted according to the obstacles and changes encountered, wherever possible within the constraints of the original project design. All MOVIT components were documented in a toolkit (in Dutch).

Appendix 1 provides additional details on the background, setting, stakeholders, project team, strategy, specific project components, and progress and follow-up of the MOVIT project.

Methods

Development of a New Matrix

A new matrix was constructed combining 2 existing frameworks: (1) “levels of organization influenced by implementation” (originating from Shortell) and (2) “domains of implementation” (originating from Flottorp et al).^24,25 They were combined because individually they specify different aspects of an implementation strategy and together they provide a comprehensive matrix in which defined project activities can be positioned according to their intended target domain and level and, thus, in total providing a structured description of the project. Criteria for the choice of these particular frameworks are their recognition in implementation science and their applicability to the described project.²⁸

Levels of organization influenced by implementation were distinguished by Shortell in a framework for change to address the managerial and organizational challenges facing health care delivery in the United States.²⁵ Although based on the US health care situation, these levels are regarded as generalizable and were recognized as being applicable to our project.²⁸

These levels are (1) policy, (2) organization, (3) group professional, and (4) individual professional. In the new matrix, these levels were listed vertically.

Domains of implementation were established by Flottorp et al after reviewing frameworks listing the determinants of practice that might prevent or enable care improvement.²⁴ These domains were seen as headings under which implementation activities fall that have a common focus. The domains are (1) guideline factors, (2) individual health professional factors, (3) professional interactions, (4) patient factors, (5) incentives and resources, (6) capacity for organizational change, and (7) social, political, and legal factors. In the matrix, these domains were presented horizontally.

Categorizing Components of Case Study MOVIT in the New Matrix

To be able to categorize the evolved MOVIT components, the Cochrane Effective Practice and Organization of Care (EPOC) Review Group Taxonomy 2015 was retrospectively applied.²³ This taxonomy was originally developed in 2002 by the EPOC editorial team as a framework for characterizing implementation interventions and was updated in 2015. However, initially we used the 2002 version, with its comprehensive coverage and international acknowledgment, and converted to the 2015 version when it became available. The complete taxonomy was searched for items closely fitting the MOVIT components to identify the EPOC implementation strategies and delivery arrangements, grouped in their categories and subcategories. A proposal was made by the author and discussed by the project team. When consensus was reached, these were placed in tables (each item with the related MOVIT components and their objectives; see Appendices 2 and 3). Taxonomy items not represented in the project, and MOVIT activities not meeting the definition of an implementation strategy, financial arrangement, or delivery arrangement, are not shown in the tables (see Appendices 2 and 3).

Filling the Matrix

Each of the implementation strategies and financial arrangements was placed in the matrix, according to the level of organization that was targeted and the domain of implementation it influenced. As the changes in care delivery arrangements were neither planned nor initiated by the project team, but were initiated by local MOVIT teams, these were placed in the matrix separately, according to the domains affected and the levels of implementation.

Results

In this section, we present the results of applying the developed matrix to the MOVIT project in figures and text. Appendix 1 provides a narrative description of the project and Appendices 2 and 3 detailed illustrations of applying the described method.

Figure 1 presents the results of mapping the EPOC items present in MOVIT in the matrix of levels and implementation domains: A clustering of implementation activities can be seen (“Financial arrangements” and “Implementation strategies”) mainly at the levels of individual and groups of professionals within the domains of professional individual functioning and group interactions. As part of the project plan, the level of the organization and the domain of guideline formation are targeted to a lesser degree, whereas the policy level, and the “patient,” “capacity to change,” and “social/political/legal” domains are not targeted at all.

Figure 1.

Identified EPOC taxonomy “Financial arrangements” and “Implementation strategies” positioned in the matrix of targeted “Levels of organization” and “Domains of implementation.”

Figure 1 shows that the identified EPOC taxonomy “Financial arrangements” and “Implementation strategies” are positioned in the matrix of targeted “Levels of organization” and “Domains of implementation.” Appendix 2 presents background information on Figure 1, that is, specifying the relation between the EPOC “Implementation strategies” and “Financial arrangements,” and the MOVIT project activities and objectives.

The following example illustrates how this results in the placement of an EPOC item in the matrix. The MOVIT component “Coached local team meetings” closely fits the EPOC definition “Educational outreach visits, or academic detailing.” The last column of Appendix 2 reports the contribution of these meetings to the MOVIT project objectives. In this case, “Team formation. Awareness and knowledge improvement. Translation of general theory to the local situation. Improved care organization.” In Figure 1, the EPOC label “D” for “Educational outreach visits, or academic detailing” is positioned in the matrix having targeted (vertically) the levels of the “individual” professional and the “group” of professionals and (horizontally) the domains of “Individual professional functioning” and “Professional interactions.”

Figure 2 shows the domains and levels of implementation affected by the changed “Delivery arrangements.” The figure shows that, besides the domains of “Development of guideline consensus,” “Functioning of the individual professional,” and “Professional interactions,” the domains of “Patient,” “Incentives and resources,” “Capacity for organizational change,” and the “Social, political, and legal” are also affected. These domains are variously influenced from the level of the individual professional to the policy level through those of the groups of professionals and organizations.

Figure 2.

Identified EPOC taxonomy “Delivery arrangements” positioned in the matrix of influenced “Levels of organization” and “Domains of implementation.”

Figure 2 also shows the changes in delivery arrangements that occurred during the project in the affected levels and domains. It becomes apparent that shared care and comprehensive geriatric assessment by the teams are important elements in the project and that they are associated with role expansion, task shifting, and communication between providers. It can also be seen that these changes in delivery arrangements not only impact individual professionals and groups of professionals regarding their guidelines, functioning, and interactions but also impacts the individual patient, as well as professionals and organizations, regarding incentives and resources, capacity for organizational change, and social, political, and legal matters.

Appendix 3 specifies which changes in delivery arrangements occurred during the project through the activities of the local working groups which fall within defined EPOC “Delivery arrangements.” They represent the contribution made to the overall MOVIT objective of implementing improved integrated care.

As an example, Appendix 3 shows that the MOVIT components “Coached local team meetings,” “Regional educational meetings,” and “Support for clinical improvement plans of local teams” fit into the EPOC delivery arrangement “Shared care.” Figure 2 shows that this delivery arrangement (labeled with “K”) influenced different levels vertically and all of the implementation domains (apart from guideline development) horizontally.

Figure 3 shows the overlap and discrepancy between the levels and domains targeted by “Financial arrangements” and “Implementation strategies,” and the levels and domains affected by the resulting “Delivery arrangements.” It can be seen that there is a large degree of overlap in the domains of individual professional functioning and professional interactions at the individual and group levels, and that the discrepancy is mainly in the capacity for organizational change and social, political, and legal domains, where there are resulting delivery arrangements but no implementation strategies and financial arrangements. Also in the “Patient” domain of implementation, there are changes in delivery arrangements but no implementation activities.

Figure 3.

Identified EPOC taxonomy “Financial arrangements” and “Implementation strategies” and the resulting “Delivery arrangements” shown together in the matrix of “Levels of organization” and “Domains of implementation.”

Discussion

The UK Medical Research Council, in its guidance on developing and evaluating complex interventions, makes clear its preference for systematic experimental evaluation but also states that, as a consequence of practical constraints, less rigorous methods can also offer “useful results.”²⁹ The Council emphasizes that, besides the evaluation of outcomes, a process evaluation can be used to assess the fidelity and quality of implementation, clarify causal mechanisms, and identify contextual factors associated with variation in outcomes.

In the case study of the MOVIT project, the process description matrix shows that the main target was the cooperation of the professionals at an individual and a group level. While the implementation strategies were primarily aimed at these levels and domains, the resulting changes in “Delivery arrangements” had a much broader impact, that is, at the organization and policy level, and in the “Patient,” “Capacity for organizational change,” and “Social, political, and legal” domains. Although some project activities in the narrative project description involved these domains and levels, the fact that they did not meet the definition of the EPOC implementation strategies suggests that they were insufficiently developed.

This process description fits the observation that the professionals and managers directly concerned with the delivery of care adopted the MOVIT approach by participating in the local teams and starting initiatives aimed at the further development of integrated care and translating them to new delivery arrangements. The continuation of the local teams and the starting of new ones (as well as requests for further support/development after project cessation) are indications that the changes in professional cooperation are sustainable. However, the reluctance of managers and governors to commit to further adoption is a matter of concern. It suggests that the overall implementation strategy has failed to bridge the gap between professional motivation and governance adoption, and raises the question whether more implementation strategies aimed at the domains and levels affected by the changed delivery arrangements would have been beneficial for further penetration, adoption, and sustainability. It also raises the question whether more effective involvement of the patient perspective would have helped to bridge this gap.

We note that, in the MOVIT project, the freedom to choose, adapt, and reconfigure interventions by the local teams (contributing to integrated care) can be regarded as a success factor, and insufficient strategic handling of contextual factors as a weakness. Specifically, the matrix (Figure 3) shows that some contextual factors (eg, guideline development, and individual and group functioning of professionals) were well covered by the implementation strategies. However, other contextual factors (eg, incentives and guidelines falling in the domains “Incentives and resources,” “Capacity to change,” and “Political, social, and legal”) were not covered by project activities; because these contextual factors were neglected and might be underdeveloped, they need to be critically reviewed. In brief, some stakeholders experience the consequences of changes without having been consulted, prepared, or compensated in areas which could be essential to them. Generally speaking, this can be considered detrimental to the adoption and sustainability of change.

A more general reflection on the mechanisms of successful implementation based on the MOVIT project is that room for adaptation of the design and the positioning of interventions and contextual influences (eg, financial arrangements) is a bonus. Therefore, any discrepancy between targeted and influenced levels and domains should be carefully reviewed, as this can provide useful clues for the necessary adaptation of the overall implementation strategy. This observation aligns with (1) the conceptual framework of Kitson et al in which (besides the level of evidence) the context in which the evidence is implemented and the method of facilitating the change are considered equally important and (2) the model of Grol et al that proposes that an iterative or cyclic implementation process is preferred for sustainable change in care.^30,31

We have chosen to use the EPOC taxonomy of implementation strategies and care arrangements and to use the frameworks of Flottorp et al²⁴ and Shortell²⁵ for the domains and levels of implementation, respectively. However, other taxonomies and frameworks that we could have used are available; Powell et al provide an overview of both in their publication “Methods to Improve the Selection and Tailoring of Implementation Strategies.”³² Although each has their particular focus and qualities, we think that the choice does not fundamentally affect our matrix. Nevertheless, it is essential that the selected taxonomy is suitable for the context of the project and that the framework encompasses levels and domains of implementation. Although we have not included organizational performance and patient care outcomes, adding these and applying the matrix in the context of, for example, the “Logic Model” can result in more comprehensive evaluation in the context of implementation research.³³

In conclusion, we regard our method as a structured process description which can be used as it stands to be learned from and to improve practice-based projects, or as a basis for more rigorous evaluation. By offering a basis for the assessment of fidelity, quality, and contextual factors, we found that this structured process description can help to use a pragmatic implementation project to make documented, experience-based steps toward improved care organization. For further generalization, more experience with the matrix is required.

Strengths, Limitations, and Future Developments

Studies and publications in medical implementation science are ideally aimed at understanding the underlying processes, and the efficacy of specific interventions and methodology; generally, the most convincing are those based on comparative trial studies. Our retrospective approach of a pragmatic project does not meet these standards. However, a strength of our approach is that a real-life project was used to develop and illustrate the use of a theory-based method that accommodates its full complexity. The matrix itself is a combination of previously described frameworks; we minimized the use of new terminology to avoid adding to the already considerable amount used in implementation literature.³⁴

A methodological aspect of the matrix requiring development is the transparency and robustness of the translation process from project components to the EPOC-defined implementation strategies and arrangements. Also, the alignment of project activities, implementation theory, and the taxonomy used requires attention. For example, our inability to define some of the MOVIT components in the taxonomy could be due to the original choice and design of these activities that were based on a more limited or different scope in 2009 compared with that of the 2015 taxonomy. Better alignment could benefit the outcomes of a project, as well as contributing to scientific progress. Examples of recent work are the assessment of context in care homes and the possibilities that feedback to stakeholders offer (reported by Estabrooks et al), and the realist approach in general and the realist review of effective health care in homes in particular (as presented by Goodman et al).^21,35,36

Conclusion

A matrix was developed as a method for a structured process description of a pragmatic implementation project. This matrix provides a basis for the assessment of the fidelity and quality of the implementation and identification of the contextual factors.

We conclude that valuable steps in health care development can be made by evaluation of the experience gained from pragmatic innovation and implementation projects which, through their complexity and adaptation, defy study by trial. This process can also help to identify areas that require further research. Because this method was developed retrospectively, we aim to test whether this method will help to plan an implementation project in advance.

Footnotes

Appendix 1

Appendix 2

“Effective Practice and Organisation of Care (EPOC) Taxonomy Financial Arrangements and Implementation Strategies” Fitting the MOVIT Project Components and Objectives.

Financial arrangements Changes in how funds are collected, insurance schemes, how services are purchased, and the use of targeted financial incentives or disincentives
Category: Mechanisms for the payment of health services
Subcategory		Definition	MOVIT component	Component objective
A	Method of paying health care organizations	Global budgets, employer-based insurance schemes, line-item budgets; case-based reimbursement; pay for performance; mixed payment	Project team: developing financial constructions with financial and regulatory bodies within national frameworks	Enabling participation of organizations and their employees in the project and sustaining their cooperation and care improvements after the project
B	Payment methods for health workers	Fee-for-services, capitation, salary	Project team: developing financial constructions with financial and regulatory bodies within national frameworks	Enabling participation of self-employed health workers in the project and sustaining their cooperation and care improvements after the project
Implementation strategies Interventions designed to bring about changes in health care organizations, the behavior of health care professionals, or the use of health services by health care recipients
Category: Interventions targeted at health care workers
Subcategory		Definition	MOVIT component	Component objective
C	Educational meetings	Courses, workshops, conferences, or other educational meetings	Regional educational meetings	Knowledge improvement, experience exchange, interdisciplinary interaction, inspiration and motivation, consensus development.
Implementation strategies Interventions designed to bring about changes in health care organizations, the behavior of health care professionals, or the use of health services by health care recipients
Category: Interventions targeted at health care workers
Subcategory		Definition	MOVIT component	Component objective
D	Educational outreach visits or academic detailing	Personal visits by a trained person to health workers in their own settings, to provide information with the aim of changing practice	Coached local team meetings	Team formation Awareness and knowledge improvement Translation of general theory to the local situation Improved care organization
E	Clinical practice guidelines	Clinical guidelines are systematically developed statements to assist health care providers and patients to decide on appropriate health care for specific clinical circumstances	Regional educational meetings and related guideline development by the project team	Combining geriatric knowledge and practical experience in regional consensus guidelines and instruments Development of a sustainable improvement cycle
F	Interprofessional education	Continuing education for health professionals that involves more than one profession in joint, interactive learning	Coached local team meetings Regional educational meetings	Consensus and improved cooperation through a common knowledge base and awareness of respective professional competencies and limitations
G	Local consensus processes	Formal or informal local consensus processes, for example, agreeing a clinical protocol to manage a patient group, adapting a guideline for a local health system, or promoting the implementation of guidelines	Coached local team meetings, regional educational meetings, facilitated clinical improvement plans of local teams	More uniformly and optimized delivery of integrated geriatric care within the local possibilities

Appendix 3

“Effective Practice and Organisation of Care (EPOC) Taxonomy Delivery Arrangements” Identified in the MOVIT Project, With the Related Project Components and Objectives.

Delivery arrangements Changes in how, when and where health care is organized and delivered, and who delivers health care
Category: Who provides care and how the health care workforce is managed
Subcategory		Definition	MOVIT component	Component objective
H	Role expansion or task shifting	Expanding tasks undertaken by a cadre of health workers or shifting tasks from one cadre to another, to include tasks not previously part of their scope of practice	Coached local interdisciplinary team meetings, facilitated clinical improvement plans of local teams, regional educational meetings	Improved use of available competencies and manpower. Decreased frustration from indistinct task assignation.
Category: Coordination of care and management of care processes
Subcategory		Definition	MOVIT activity	MOVIT objective
I	Communication between providers	Systems or strategies for improving the communication between health care providers	Coached local interdisciplinary team meetings, support for clinical improvement plans of local teams, regional educational meetings	Establishing and facilitating communication round daily topics and developing an improvement dialogue.
Category: Coordination of care and management of care processes
Subcategory		Definition	MOVIT activity	MOVIT objective
J	Comprehensive geriatric assessment	A multidimensional interdisciplinary diagnostic process focused on determining a frail older person’s medical, psychological, and functional capability to ensure that problems are identified, quantified, and managed appropriately	Coached local interdisciplinary team meetings, support for clinical improvement plans of local teams, regional educational meetings	Comprehensive geriatric assessment presented as a practical ideal which can be attained via various routes and steps
K	Shared care	Continuing collaborative clinical care between primary and specialist care physicians	Coached local interdisciplinary team meetings, regional educational meetings, support for clinical improvement plans of local teams	Pharmacist, elderly care physician, and general practitioner involved in organization improvement and case-related care
L	Teams	Creating and delivering care through a multidisciplinary team of health care workers	Coached local interdisciplinary team meetings, support for clinical improvement plans of local teams, regional educational meetings	Establishing a team with organizational status and capability based on common case-related relevance and effectivity

Authors’ Note

The lead author (Jacobijn Gussekloo) affirms that this article is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, registered) have been explained. All funding sources and suppliers were independent and had no influence on study design; collection, analyses, and interpretation of data; writing the report; or the decision to submit the paper.

Author Contributions

Gussekloo had full access to all data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Poot, de Waard, Wind, Caljouw, and Gussekloo conceived the study concept and design, performed the analysis and interpretation of the data and critically revised the manuscript for important intellectual content. Poot drafted the manuscript. All authors conform to the 4 criteria of the International Committee of Medical Journal Editors (ICMJE) Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals (ICMJE Recommendations 2013).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: A grant was received from ZonMw, the Dutch Organization for Health Research, the Netherlands (Project: 311060401).

References

Wise

. Services for older people need major change, says report. BMJ. 2014;348:1994.

Haggerty

. Ordering the chaos for patients with multimorbidity. BMJ. 2012;345:5915.

James Lloyd

. Integrated care: a guide for policy makers. Alliance for Health and the Future. http://www.ilcuk.org.uk/index.php/publications/publication_details/integrated_care_-_a_guide_for_policymakers. Published 2006. Accessed June 8, 2017.

Minkman

Schouten

Huijsman

van Splunteren

. Integrated care for patients with a stroke in the Netherlands: results and experiences from a national Breakthrough Collaborative Improvement project. Int J Integr Care. 2005;5:e14.

MacNeil Vroomen

Boorsma

Bosmans

Frijters

Nijpels

van Hout

. Is it time for a change? A cost-effectiveness analysis comparing a multidisciplinary integrated care model for residential homes to usual care. PLoS One. 2012;7(5):e37444.

Boorsma

Frijters

Knol

Ribbe

Nijpels

van Hout

. Effects of multidisciplinary integrated care on quality of care in residential care facilities for elderly people: a cluster randomized trial. CMAJ. 2011;183(11):724-732.

Metzelthin

van Rossum

de Witte

et al . Effectiveness of interdisciplinary primary care approach to reduce disability in community dwelling frail older people: cluster randomised controlled trial. BMJ. 2013;347:5264.

Blom

den Elzen

van Houwelingen

et al . Effectiveness and cost-effectiveness of a proactive, goal-oriented, integrated care model in general practice for older people. A cluster randomised controlled trial: integrated systematic care for older people—the ISCOPE study. Age Ageing. 2016;45(1):30-41.

Bodenheimer

Wagner

Grumbach

. Improving primary care for patients with chronic illness. JAMA. 2002;288(14):1775-1779.

10.

Bodenheimer

Wagner

Grumbach

. Improving primary care for patients with chronic illness: the chronic care model, Part 2. JAMA. 2002;288(15):1909-1914.

11.

Wagner

Glasgow

Davis

et al . Quality improvement in chronic illness care: a collaborative approach. Jt Comm J Qual Improv. 2001;27(2):63-80.

12.

Davy

Bleasel

Liu

Tchan

Ponniah

Brown

. Factors influencing the implementation of chronic care models: a systematic literature review. BMC Fam Pract. 2015;16:102.

13.

Greenwood-Lee

Hawe

Nettel-Aguirre

Shiell

Marshall

. Complex intervention modelling should capture the dynamics of adaptation. BMC Med Res Methodol. 2016;16(1):51.

14.

Rogers

. Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008;14:29-48.

15.

Plsek

Greenhalgh

. Complexity science: the challenge of complexity in health care. BMJ. 2001;323(7313):625-628.

16.

Crow

Gage

Hampson

et al . The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature. Health Technol Assess. 2002;6(32):1-244.

17.

Ham

Walsh

. Making Integrated Care Happen at Scale and Pace: Lessons From Experience. London, England: The King’s Fund. https://www.kingsfund.org.uk/sites/files/kf/field/field_publication_file/making-integrated-care-happen-kingsfund-mar13.pdf. Published 2013. Accessed June 8, 2017.

18.

Vize

. Integrated care: a story of hard won success. BMJ. 2012;344:3529.

19.

Kilbourne

Abraham

Goodrich

et al . Cluster randomized adaptive implementation trial comparing a standard versus enhanced implementation intervention to improve uptake of an effective re-engagement program for patients with serious mental illness. Implement Sci. 2013;8:136.

20.

Zwarenstein

Treweek

Gagnier

et al . Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:2390.

21.

Goodman

Dening

Gordon

et al . Effective health care for older people living and dying in care homes: a realist review. BMC Health Serv Res. 2016;16:269.

22.

Dopson

. The diffusion of medical innovations: can figurational sociology contribute. Organ Stud. 2005;26:1125-1144.

23.

Norwegian Knowledge Centre for the Health Services. EPOC taxonomy. Cochrane Effective Practice and Organisation of Care (EPOC). https://epoc.cochrane.org/epoc-taxonomy. Published 2015. Accessed June 8, 2017.

24.

Flottorp

Oxman

Krause

et al . A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35.

25.

Shortell

. Increasing value: a research agenda for addressing the managerial and organizational challenges facing health care delivery in the United States. Med Care Res Rev. 2004;61:12-30.

26.

The National Care for the Elderly Programme. http://www.beteroud.nl/ouderen/dutch-national-care-programme-for-the-elderly.html. Published 2014. Accessed June 8, 2017.

27.

Grol

Bosch

Hulscher

Eccles

Wensing

. Planning and studying improvement in patient care: the use of theoretical perspectives. Milbank Q. 2007;85(1):93-138.

28.

Proctor

Powell

McMillen

. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

29.

Craig

Dieppe

Macintyre

Michie

Nazareth

Petticrew

. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:1655.

30.

Kitson

Harvey

McCormack

. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7(3):149-158.

31.

Grol

Wensing

Davis

. Improving Patient Care: The Implementation of Change in Health Care. Hoboken, New Jersey, US Wiley-Blackwell; 2013.

32.

Powell

Beidas

Lewis

et al . Methods to improve the selection and tailoring of implementation strategies. J Behav Health Serv Res. 2017;44(2):177-194.

33.

Saunders

Evans

Joshi

. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6(2):134-147.

34.

Damschroder

Aron

Keith

Kirsh

Alexander

Lowery

. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

35.

Estabrooks

Knopp-Sihota

Cummings

Norton

. Making research results relevant and useable: presenting complex organizational context data to nonresearch stakeholders in the nursing home setting. Worldviews Evid Based Nurs. 2016;13(4):270-276.

36.

Hoben

Estabrooks

Squires

Behrens

. Factor structure, reliability and measurement invariance of the Alberta Context Tool and the Conceptual Research Utilization Scale, for German residential long term care. Front Psychol. 2016;7:1339.

37.

Schouten

Hulscher

van Everdingen

Huijsman

Grol

. Evidence for the impact of quality improvement collaboratives: systematic review. BMJ. 2008;336(7659):1491-1494.

38.

Shiell

Hawe

Gold

. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008;336(7656):1281-1283.