Abstract
Evaluation of the implementation of integrated care can differ from trial-based research due to complexity. Therefore, we examined whether a theory-based method for process description of implementation can contribute to improvement of evidence-based care. MOVIT, a Dutch project aimed at implementing integrated care for older vulnerable persons in residential care homes, was used as a case study. The project activities were defined according to implementation taxonomy and mapped in a matrix of theoretical levels and domains. Project activities mainly targeted professionals (both individual and group). A few activities targeted the organizational level, whereas none targeted the policy level, or the patient, or the “social, political, and legal” domains. However, the resulting changes in care delivery arrangement had consequences for professionals, patients, organizations, and the social, political, and legal domains. A structured process description of a pragmatic implementation project can help assess the fidelity and quality of the implementation, and identify relevant contextual factors for immediate adaptation and future research. The description showed that, in the MOVIT project, there was a discrepancy between the levels and domains targeted by the implementation activities and those influenced by the resulting changes in delivery arrangement. This could have influenced, in particular, the adoption and sustainability of the project.
Keywords
Background
There is broad consensus among medical professionals and policy makers that the concept of integrated care offers opportunities in meeting the demands of the growing group of older persons with combined care needs in the population.1-3 Although interpretation of the concept of integrated care varies, it generally involves integration of cure and care on one hand, and user and provider perspectives on the other. 3 Practical applications are often based on the Chronic Care Model.4-8 In this latter model, improved functional and clinical patient outcomes are achieved through a productive interaction between an informed and activated patient and a proactive care team. This interaction takes place within the context of a supportive community and health care system.9-11
Implementation often takes the form of complex intervention projects in a system with multiple independent professional disciplines, and organizations with individual objectives, operating and adapting in a changeable environment while performing multiple interventions in various ways, to differing degrees, resulting in nonlinear, disproportionate, and unforeseen (emergent) outcomes.12-14 Project components can be adapted to the changes that occur and the outcomes that emerge, and can evolve to attain the project objectives.
While comparative trial-based research on the causal mechanisms between the interventions and their outcomes is often preferred, real-life implementation projects can encounter problems meeting the rigorous demands of such an evaluation approach.13-18 In the present study, the term pragmatic is used to characterize a real-life implementation project with a high degree of adaptiveness which is aimed more at demonstrating the applicability of the approach than the efficacy of an intervention.19,20 To study complex interventions in the context of evidence-based health care, various (research) conceptual frameworks have been proposed, such as those based on a realist approach, sociological theory, mathematical modeling, program theory, and the theory of complex adaptive systems.13-15,21,22 Besides these frameworks, approaches with a more descriptive character have also been used, such as those for strategies, determinants, and levels of implementation interventions.23-25
We propose that a structured process description of pragmatic (adaptive) care implementation projects (which do not meet the requirements of specific [research] conceptual frameworks) can also offer a valuable contribution to the evidence-based development of improved health care. This article describes how, retrospectively, a matrix was developed, using descriptive frameworks, to describe the process of a pragmatic real-life implementation project which does not conform to the research requirements of existing (research) conceptual frameworks.
During the development of the matrix, the MOVIT project was used as a case study. The MOVIT project (as part of a national elderly care improvement program in the Netherlands) aimed to develop and evaluate an implementation strategy for integrated primary care for older persons with complex needs living in residential care homes. 26
Toward an ideal of integrated care, the intended improvement in medical care was to coordinate the care provided by general practitioners (GPs), elderly care physicians, pharmacologists, and nursing staff, and to initiate a long-term process so that further steps would be taken in that direction. The general strategy was that, within a larger administrative region, local working groups were initiated and supported per residential home and allowed a large degree of freedom in determining their local priorities and steps toward integrated care. Support was provided for the local groups by information provision, logistical support, and team coaching. At a regional level, educational sessions were organized and support was provided in negotiating facilities and terms.
At the start of the project, the project components were developed based on (a selection of) (1) general implementation theory, (2) a small-scale pilot, and (3) interviews with stakeholders (care financiers, residential care home governors/managers, GPs, nursing staff, pharmacists, and elderly care physicians). 27 During the 3-year project, components were adapted according to the obstacles and changes encountered, wherever possible within the constraints of the original project design. All MOVIT components were documented in a toolkit (in Dutch).
Appendix 1 provides additional details on the background, setting, stakeholders, project team, strategy, specific project components, and progress and follow-up of the MOVIT project.
Methods
Development of a New Matrix
A new matrix was constructed combining 2 existing frameworks: (1) “levels of organization influenced by implementation” (originating from Shortell) and (2) “domains of implementation” (originating from Flottorp et al).24,25 They were combined because individually they specify different aspects of an implementation strategy and together they provide a comprehensive matrix in which defined project activities can be positioned according to their intended target domain and level and, thus, in total providing a structured description of the project. Criteria for the choice of these particular frameworks are their recognition in implementation science and their applicability to the described project. 28
Levels of organization influenced by implementation were distinguished by Shortell in a framework for change to address the managerial and organizational challenges facing health care delivery in the United States. 25 Although based on the US health care situation, these levels are regarded as generalizable and were recognized as being applicable to our project. 28
These levels are (1) policy, (2) organization, (3) group professional, and (4) individual professional. In the new matrix, these levels were listed vertically.
Domains of implementation were established by Flottorp et al after reviewing frameworks listing the determinants of practice that might prevent or enable care improvement. 24 These domains were seen as headings under which implementation activities fall that have a common focus. The domains are (1) guideline factors, (2) individual health professional factors, (3) professional interactions, (4) patient factors, (5) incentives and resources, (6) capacity for organizational change, and (7) social, political, and legal factors. In the matrix, these domains were presented horizontally.
Categorizing Components of Case Study MOVIT in the New Matrix
To be able to categorize the evolved MOVIT components, the Cochrane Effective Practice and Organization of Care (EPOC) Review Group Taxonomy 2015 was retrospectively applied. 23 This taxonomy was originally developed in 2002 by the EPOC editorial team as a framework for characterizing implementation interventions and was updated in 2015. However, initially we used the 2002 version, with its comprehensive coverage and international acknowledgment, and converted to the 2015 version when it became available. The complete taxonomy was searched for items closely fitting the MOVIT components to identify the EPOC implementation strategies and delivery arrangements, grouped in their categories and subcategories. A proposal was made by the author and discussed by the project team. When consensus was reached, these were placed in tables (each item with the related MOVIT components and their objectives; see Appendices 2 and 3). Taxonomy items not represented in the project, and MOVIT activities not meeting the definition of an implementation strategy, financial arrangement, or delivery arrangement, are not shown in the tables (see Appendices 2 and 3).
Filling the Matrix
Each of the implementation strategies and financial arrangements was placed in the matrix, according to the level of organization that was targeted and the domain of implementation it influenced. As the changes in care delivery arrangements were neither planned nor initiated by the project team, but were initiated by local MOVIT teams, these were placed in the matrix separately, according to the domains affected and the levels of implementation.
Results
In this section, we present the results of applying the developed matrix to the MOVIT project in figures and text. Appendix 1 provides a narrative description of the project and Appendices 2 and 3 detailed illustrations of applying the described method.
Figure 1 presents the results of mapping the EPOC items present in MOVIT in the matrix of levels and implementation domains: A clustering of implementation activities can be seen (“Financial arrangements” and “Implementation strategies”) mainly at the levels of individual and groups of professionals within the domains of professional individual functioning and group interactions. As part of the project plan, the level of the organization and the domain of guideline formation are targeted to a lesser degree, whereas the policy level, and the “patient,” “capacity to change,” and “social/political/legal” domains are not targeted at all.
Identified EPOC taxonomy “Financial arrangements” and “Implementation strategies” positioned in the matrix of targeted “Levels of organization” and “Domains of implementation.”
Figure 1 shows that the identified EPOC taxonomy “Financial arrangements” and “Implementation strategies” are positioned in the matrix of targeted “Levels of organization” and “Domains of implementation.” Appendix 2 presents background information on Figure 1, that is, specifying the relation between the EPOC “Implementation strategies” and “Financial arrangements,” and the MOVIT project activities and objectives.
The following example illustrates how this results in the placement of an EPOC item in the matrix. The MOVIT component “Coached local team meetings” closely fits the EPOC definition “Educational outreach visits, or academic detailing.” The last column of Appendix 2 reports the contribution of these meetings to the MOVIT project objectives. In this case, “Team formation. Awareness and knowledge improvement. Translation of general theory to the local situation. Improved care organization.” In Figure 1, the EPOC label “D” for “Educational outreach visits, or academic detailing” is positioned in the matrix having targeted (vertically) the levels of the “individual” professional and the “group” of professionals and (horizontally) the domains of “Individual professional functioning” and “Professional interactions.”
Figure 2 shows the domains and levels of implementation affected by the changed “Delivery arrangements.” The figure shows that, besides the domains of “Development of guideline consensus,” “Functioning of the individual professional,” and “Professional interactions,” the domains of “Patient,” “Incentives and resources,” “Capacity for organizational change,” and the “Social, political, and legal” are also affected. These domains are variously influenced from the level of the individual professional to the policy level through those of the groups of professionals and organizations.
Identified EPOC taxonomy “Delivery arrangements” positioned in the matrix of influenced “Levels of organization” and “Domains of implementation.”
Figure 2 also shows the changes in delivery arrangements that occurred during the project in the affected levels and domains. It becomes apparent that shared care and comprehensive geriatric assessment by the teams are important elements in the project and that they are associated with role expansion, task shifting, and communication between providers. It can also be seen that these changes in delivery arrangements not only impact individual professionals and groups of professionals regarding their guidelines, functioning, and interactions but also impacts the individual patient, as well as professionals and organizations, regarding incentives and resources, capacity for organizational change, and social, political, and legal matters.
Appendix 3 specifies which changes in delivery arrangements occurred during the project through the activities of the local working groups which fall within defined EPOC “Delivery arrangements.” They represent the contribution made to the overall MOVIT objective of implementing improved integrated care.
As an example, Appendix 3 shows that the MOVIT components “Coached local team meetings,” “Regional educational meetings,” and “Support for clinical improvement plans of local teams” fit into the EPOC delivery arrangement “Shared care.” Figure 2 shows that this delivery arrangement (labeled with “K”) influenced different levels vertically and all of the implementation domains (apart from guideline development) horizontally.
Figure 3 shows the overlap and discrepancy between the levels and domains targeted by “Financial arrangements” and “Implementation strategies,” and the levels and domains affected by the resulting “Delivery arrangements.” It can be seen that there is a large degree of overlap in the domains of individual professional functioning and professional interactions at the individual and group levels, and that the discrepancy is mainly in the capacity for organizational change and social, political, and legal domains, where there are resulting delivery arrangements but no implementation strategies and financial arrangements. Also in the “Patient” domain of implementation, there are changes in delivery arrangements but no implementation activities.
Identified EPOC taxonomy “Financial arrangements” and “Implementation strategies” and the resulting “Delivery arrangements” shown together in the matrix of “Levels of organization” and “Domains of implementation.”
Discussion
The UK Medical Research Council, in its guidance on developing and evaluating complex interventions, makes clear its preference for systematic experimental evaluation but also states that, as a consequence of practical constraints, less rigorous methods can also offer “useful results.” 29 The Council emphasizes that, besides the evaluation of outcomes, a process evaluation can be used to assess the fidelity and quality of implementation, clarify causal mechanisms, and identify contextual factors associated with variation in outcomes.
In the case study of the MOVIT project, the process description matrix shows that the main target was the cooperation of the professionals at an individual and a group level. While the implementation strategies were primarily aimed at these levels and domains, the resulting changes in “Delivery arrangements” had a much broader impact, that is, at the organization and policy level, and in the “Patient,” “Capacity for organizational change,” and “Social, political, and legal” domains. Although some project activities in the narrative project description involved these domains and levels, the fact that they did not meet the definition of the EPOC implementation strategies suggests that they were insufficiently developed.
This process description fits the observation that the professionals and managers directly concerned with the delivery of care adopted the MOVIT approach by participating in the local teams and starting initiatives aimed at the further development of integrated care and translating them to new delivery arrangements. The continuation of the local teams and the starting of new ones (as well as requests for further support/development after project cessation) are indications that the changes in professional cooperation are sustainable. However, the reluctance of managers and governors to commit to further adoption is a matter of concern. It suggests that the overall implementation strategy has failed to bridge the gap between professional motivation and governance adoption, and raises the question whether more implementation strategies aimed at the domains and levels affected by the changed delivery arrangements would have been beneficial for further penetration, adoption, and sustainability. It also raises the question whether more effective involvement of the patient perspective would have helped to bridge this gap.
We note that, in the MOVIT project, the freedom to choose, adapt, and reconfigure interventions by the local teams (contributing to integrated care) can be regarded as a success factor, and insufficient strategic handling of contextual factors as a weakness. Specifically, the matrix (Figure 3) shows that some contextual factors (eg, guideline development, and individual and group functioning of professionals) were well covered by the implementation strategies. However, other contextual factors (eg, incentives and guidelines falling in the domains “Incentives and resources,” “Capacity to change,” and “Political, social, and legal”) were not covered by project activities; because these contextual factors were neglected and might be underdeveloped, they need to be critically reviewed. In brief, some stakeholders experience the consequences of changes without having been consulted, prepared, or compensated in areas which could be essential to them. Generally speaking, this can be considered detrimental to the adoption and sustainability of change.
A more general reflection on the mechanisms of successful implementation based on the MOVIT project is that room for adaptation of the design and the positioning of interventions and contextual influences (eg, financial arrangements) is a bonus. Therefore, any discrepancy between targeted and influenced levels and domains should be carefully reviewed, as this can provide useful clues for the necessary adaptation of the overall implementation strategy. This observation aligns with (1) the conceptual framework of Kitson et al in which (besides the level of evidence) the context in which the evidence is implemented and the method of facilitating the change are considered equally important and (2) the model of Grol et al that proposes that an iterative or cyclic implementation process is preferred for sustainable change in care.30,31
We have chosen to use the EPOC taxonomy of implementation strategies and care arrangements and to use the frameworks of Flottorp et al 24 and Shortell 25 for the domains and levels of implementation, respectively. However, other taxonomies and frameworks that we could have used are available; Powell et al provide an overview of both in their publication “Methods to Improve the Selection and Tailoring of Implementation Strategies.” 32 Although each has their particular focus and qualities, we think that the choice does not fundamentally affect our matrix. Nevertheless, it is essential that the selected taxonomy is suitable for the context of the project and that the framework encompasses levels and domains of implementation. Although we have not included organizational performance and patient care outcomes, adding these and applying the matrix in the context of, for example, the “Logic Model” can result in more comprehensive evaluation in the context of implementation research. 33
In conclusion, we regard our method as a structured process description which can be used as it stands to be learned from and to improve practice-based projects, or as a basis for more rigorous evaluation. By offering a basis for the assessment of fidelity, quality, and contextual factors, we found that this structured process description can help to use a pragmatic implementation project to make documented, experience-based steps toward improved care organization. For further generalization, more experience with the matrix is required.
Strengths, Limitations, and Future Developments
Studies and publications in medical implementation science are ideally aimed at understanding the underlying processes, and the efficacy of specific interventions and methodology; generally, the most convincing are those based on comparative trial studies. Our retrospective approach of a pragmatic project does not meet these standards. However, a strength of our approach is that a real-life project was used to develop and illustrate the use of a theory-based method that accommodates its full complexity. The matrix itself is a combination of previously described frameworks; we minimized the use of new terminology to avoid adding to the already considerable amount used in implementation literature. 34
A methodological aspect of the matrix requiring development is the transparency and robustness of the translation process from project components to the EPOC-defined implementation strategies and arrangements. Also, the alignment of project activities, implementation theory, and the taxonomy used requires attention. For example, our inability to define some of the MOVIT components in the taxonomy could be due to the original choice and design of these activities that were based on a more limited or different scope in 2009 compared with that of the 2015 taxonomy. Better alignment could benefit the outcomes of a project, as well as contributing to scientific progress. Examples of recent work are the assessment of context in care homes and the possibilities that feedback to stakeholders offer (reported by Estabrooks et al), and the realist approach in general and the realist review of effective health care in homes in particular (as presented by Goodman et al).21,35,36
Conclusion
A matrix was developed as a method for a structured process description of a pragmatic implementation project. This matrix provides a basis for the assessment of the fidelity and quality of the implementation and identification of the contextual factors.
We conclude that valuable steps in health care development can be made by evaluation of the experience gained from pragmatic innovation and implementation projects which, through their complexity and adaptation, defy study by trial. This process can also help to identify areas that require further research. Because this method was developed retrospectively, we aim to test whether this method will help to plan an implementation project in advance.
Footnotes
Appendix 1
Appendix 2
“Effective Practice and Organisation of Care (EPOC) Taxonomy Financial Arrangements and Implementation Strategies” Fitting the MOVIT Project Components and Objectives.
| Financial arrangements Changes in how funds are collected, insurance schemes, how services are purchased, and the use of targeted financial incentives or disincentives |
||||
| Category: Mechanisms for the payment of health services | ||||
| Subcategory | Definition | MOVIT component | Component objective | |
| A | Method of paying health care organizations | Global budgets, employer-based insurance schemes, line-item budgets; case-based reimbursement; pay for performance; mixed payment | Project team: developing financial constructions with financial and regulatory bodies within national frameworks | Enabling participation of organizations and their employees in the project and sustaining their cooperation and care improvements after the project |
| B | Payment methods for health workers | Fee-for-services, capitation, salary | Project team: developing financial constructions with financial and regulatory bodies within national frameworks | Enabling participation of self-employed health workers in the project and sustaining their cooperation and care improvements after the project |
| Implementation strategies Interventions designed to bring about changes in health care organizations, the behavior of health care professionals, or the use of health services by health care recipients |
||||
| Category: Interventions targeted at health care workers | ||||
| Subcategory | Definition | MOVIT component | Component objective | |
| C | Educational meetings | Courses, workshops, conferences, or other educational meetings | Regional educational meetings | Knowledge improvement, experience exchange, interdisciplinary interaction, inspiration and motivation, consensus development. |
| Implementation strategies Interventions designed to bring about changes in health care organizations, the behavior of health care professionals, or the use of health services by health care recipients |
||||
| Category: Interventions targeted at health care workers | ||||
| Subcategory | Definition | MOVIT component | Component objective | |
| D | Educational outreach visits or academic detailing | Personal visits by a trained person to health workers in their own settings, to provide information with the aim of changing practice | Coached local team meetings | Team formation Awareness and knowledge improvement Translation of general theory to the local situation Improved care organization |
| E | Clinical practice guidelines | Clinical guidelines are systematically developed statements to assist health care providers and patients to decide on appropriate health care for specific clinical circumstances | Regional educational meetings and related guideline development by the project team | Combining geriatric knowledge and practical experience in regional consensus guidelines and instruments Development of a sustainable improvement cycle |
| F | Interprofessional education | Continuing education for health professionals that involves more than one profession in joint, interactive learning | Coached local team meetings Regional educational meetings |
Consensus and improved cooperation through a common knowledge base and awareness of respective professional competencies and limitations |
| G | Local consensus processes | Formal or informal local consensus processes, for example, agreeing a clinical protocol to manage a patient group, adapting a guideline for a local health system, or promoting the implementation of guidelines | Coached local team meetings, regional educational meetings, facilitated clinical improvement plans of local teams | More uniformly and optimized delivery of integrated geriatric care within the local possibilities |
Appendix 3
“Effective Practice and Organisation of Care (EPOC) Taxonomy Delivery Arrangements” Identified in the MOVIT Project, With the Related Project Components and Objectives.
| Delivery arrangements Changes in how, when and where health care is organized and delivered, and who delivers health care |
||||
| Category: Who provides care and how the health care workforce is managed | ||||
| Subcategory | Definition | MOVIT component | Component objective | |
| H | Role expansion or task shifting | Expanding tasks undertaken by a cadre of health workers or shifting tasks from one cadre to another, to include tasks not previously part of their scope of practice | Coached local interdisciplinary team meetings, facilitated clinical improvement plans of local teams, regional educational meetings | Improved use of available competencies and manpower. Decreased frustration from indistinct task assignation. |
| Category: Coordination of care and management of care processes | ||||
| Subcategory | Definition | MOVIT activity | MOVIT objective | |
| I | Communication between providers | Systems or strategies for improving the communication between health care providers | Coached local interdisciplinary team meetings, support for clinical improvement plans of local teams, regional educational meetings | Establishing and facilitating communication round daily topics and developing an improvement dialogue. |
| Category: Coordination of care and management of care processes | ||||
| Subcategory | Definition | MOVIT activity | MOVIT objective | |
| J | Comprehensive geriatric assessment | A multidimensional interdisciplinary diagnostic process focused on determining a frail older person’s medical, psychological, and functional capability to ensure that problems are identified, quantified, and managed appropriately | Coached local interdisciplinary team meetings, support for clinical improvement plans of local teams, regional educational meetings | Comprehensive geriatric assessment presented as a practical ideal which can be attained via various routes and steps |
| K | Shared care | Continuing collaborative clinical care between primary and specialist care physicians | Coached local interdisciplinary team meetings, regional educational meetings, support for clinical improvement plans of local teams | Pharmacist, elderly care physician, and general practitioner involved in organization improvement and case-related care |
| L | Teams | Creating and delivering care through a multidisciplinary team of health care workers | Coached local interdisciplinary team meetings, support for clinical improvement plans of local teams, regional educational meetings | Establishing a team with organizational status and capability based on common case-related relevance and effectivity |
Authors’ Note
The lead author (Jacobijn Gussekloo) affirms that this article is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, registered) have been explained. All funding sources and suppliers were independent and had no influence on study design; collection, analyses, and interpretation of data; writing the report; or the decision to submit the paper.
Author Contributions
Gussekloo had full access to all data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Poot, de Waard, Wind, Caljouw, and Gussekloo conceived the study concept and design, performed the analysis and interpretation of the data and critically revised the manuscript for important intellectual content. Poot drafted the manuscript. All authors conform to the 4 criteria of the International Committee of Medical Journal Editors (ICMJE) Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals (ICMJE Recommendations 2013).
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: A grant was received from ZonMw, the Dutch Organization for Health Research, the Netherlands (Project: 311060401).