Evaluation of the Veterans Health Administration's Specialty Care Transformational Initiatives to Promote Patient-Centered Delivery of Specialty Care: A Mixed-Methods Approach

Abstract

Introduction:

Veteran's Affairs Office of Specialty Care (OSC) launched four national initiatives (Electronic-Consults [e-Consults], Specialty Care Access Networks-Extension for Community Healthcare Outcomes [SCAN-ECHO], Mini-Residencies, and Specialty Care Neighborhood) to improve specialty care delivery and funded a center to evaluate the initiatives.

Methods:

The evaluation, guided by two implementation frameworks, provides formative (administrator/provider interviews and surveys) and summative data (quantitative data on patterns of use) about the initiatives to OSC.

Results:

Evaluation of initiative implementation is assessed through CFIR (Consolidated Framework for Implementation Research)-grounded qualitative interviews to identify barriers/facilitators. Depending on high or low implementation, factors such as receiving workload credit, protected time, existing workflow/systems compatibility, leadership engagement, and access to information/resources were considered implementation barriers or facilitators. Findings were shared with OSC and used to further refine implementation at additional sites. Evaluation of other initiatives is ongoing.

Conclusions:

The mixed-methods approach has provided timely information to OSC about initiative effect and impacted OSC policies on implementation at additional sites.

Introduction

The delivery of patient-centered specialty care is challenging, particularly in a large geographically dispersed integrated healthcare system such as the Veterans Health Administration (VHA). Of the 8.3 million Veterans receiving services annually, nearly 50% see one or more specialist.¹ With specialists located predominantly in urban medical centers, Veterans receiving care in Community-Based Outpatient Clinics (CBOCs), which are often located in nonurban areas, may experience difficulty accessing this care and encounter problems with care coordination. In response to Veterans' needs, the VA's Office of Specialty Care (OSC) sought innovative means to improve access to specialist care by launching four national initiatives in May 2011: (1) Specialty Care Access Networks-Extension for Community Healthcare Outcomes (SCAN-ECHO), the VA adaption of Project Echo²; (2) Specialty Care Mini-Residency Program (SC-MRP); (3) Specialty Care Neighborhoods (SCNs); and (4) Electronic-Consults (e-Consults). The goal of these initiatives was to provide specialty care more efficiently, timely, and locally through enhanced coordination with primary care.

To assess the implementation and impact of these initiatives, the Office of Specialty Care and VA Health Services Research and Development (HSR&D) Quality Enhancement Research Initiative (QUERI) program jointly funded evaluation centers through a competitive Request for Proposal Process. We describe our evaluation approach by using a convergent parallel mixed-methods technique along with initial results from the e-Consults initiative to illustrate our general approach.^3,4

Materials and Methods

Description of Initiatives

The four OSC initiatives spanned a range of models of specialist care delivery. In brief, the SCAN-ECHO, based on the Project Echo model, was developed to improve primary care provider (PCP) clinical expertise in a specific specialty area.² Specialists usually comprised a team (i.e., physician, nurse, pharmacist, and psychologist) meet via multisite videoconferencing with multiple PCPs during each session for case-based learning and knowledge sharing.⁵ The Mini-Residency Program trains PCPs to perform basic musculoskeletal, dermatologic, and/or pain care procedures within primary care settings (e.g., joint injections and skin biopsies). This program comprised in-person training and supervised experiences with clinical experts, which then allow PCPs to care for Veterans locally, without the need for travel.² SCNs are team-based specialty care models that are focused on particular conditions, for example, heart failure, and are designed to support and better coordinate with primary care.^6,7 e-Consults also known as “chart only consults” or “e-referrals,” are used to ask clinical or focused diagnostic questions by a PCP or licensed independent practitioner to a medical specialist or surgeon.⁸ Communication occurs through the VA electronic medical record, and PCPs receive recommendations from specialists without necessitating a patient face-to-face specialist visit. e-Consults can be converted to a face-to-face referral visit if needed.

Conceptual Frameworks

The goals of the evaluation center are to provide recommendations to OSC about initiative progress and outcomes. Our convergent parallel mixed-methods approach, meaning quantitative and qualitative data, is collected simultaneously and then merged, and it is guided by two implementation research/evaluation frameworks: reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) and Consolidated Framework for Implementation Research (CFIR). The evaluation framework, RE-AIM, extends the evaluation of quality initiatives beyond local effectiveness to issues of scalability, willingness of adoption and implementation by users, and long-term sustainability.⁹ CFIR is a framework of the determinants of effective implementation of new evidence-based initiatives, thereby expanding on the “implementation” component of RE-AIM.¹⁰ This combination of frameworks provides a way of organizing data collection and findings.¹⁰

Our approach combines formative evaluation, including administrator/provider cross-sectional interviews and surveys, with summative evaluation by using quantitative data on specialty care initiatives patterns of use. The integration of qualitative and quantitative data allows for a better understanding of the subtleties, as well as cross-validation of findings that emerge throughout this process (Fig. 1).^3,4 Next, we describe in detail this approach.

Fig. 1.

Formative and summative evaluation approach.

Evaluation Approach

Initially, the specialty care initiatives were implemented voluntarily with each medical center able to implement one or more initiative(s). In addition, each medical center and specialty combination is defined as a unique site in our subsequent discussion of the initiatives (i.e., one medical center implementing an initiative for cardiology and nephrology would be counted as two sites). The general formative evaluation approach for each initiative is a developmental sequential integration of data, when one type of methodology to collect data informs the development of another. For example, we use quantitative survey data to inform the development of the qualitative interview guide and then merge the findings from both data types (Appendix 1).³

For each initiative, our evaluation approach is as follows. First, key stakeholders are surveyed, generally the clinician lead at each pilot implementation site funded by OSC. These surveys identify key CFIR constructs associated with implementation, which then guides subsequent data collection efforts. There are 37 constructs within the 5 domains of CFIR, including intervention characteristics, outer setting, inner setting, process of implementation, and individual characteristics.⁸ Survey respondents are asked to rate the importance of the specific key CFIR constructs in terms of whether they believe it has been or will be an important factor in determining initiative implementation success.

We then use administrative data from the VA's Corporate Data Warehouse (CDW) to identify high and low implementation sites for purposive sampling for conducting in-depth interviews of key informants participating in the initiative. CDW is a national repository comprising data from several VHA clinical and administrative systems, and it provides data and tools to support management decision making, performance measurement, and research objectives. High and low implementation sites are categorized based on number of consults or procedures completed at the time of evaluation. By selecting high and low implementation sites, we can determine which CFIR constructs are manifested differently across these sites, thereby identifying the most important and influential constructs in successful implementation. These constructs then become the basis for developing recommendations for improving implementation effectiveness among low-performing sites and for further dissemination of the initiatives in future sites.

Next, a semi-structured interview guide is developed, focusing on stakeholder-identified CFIR constructs, to obtain information about the constructs and other specific implementation issues of interest to OSC (effect of initiatives on quality of care and degree of Patient Aligned Care Team [PACT] integration). The key informant interviews are conducted via telephone with two evaluation team members, one conducting the interview and one taking field notes (written or typed). A rapid analytic approach is then used to incorporate emergent findings that inform data collection for subsequent interviews (e.g., adapting the interview guide).¹¹ The semi-structured interview guide provides the opportunity for rich narratives that may identify key barriers and facilitators to implementation.

The qualitative findings are then used to inform a broader provider quantitative survey, hence a developmental sequential mixed-methods approach.^3,4 The survey measures the extent of initiative implementation, what are barriers and facilitators to implementation, and what the initiative impact has been on providers and patients. A merging data approach is then used to integrate the survey data that complements the qualitative narrative by providing quantitative findings from a more representative sample.^3,4 Survey findings and implementation volume data are compared to corroborate the link between potential implementation determinants and consult volumes and between volumes and provider-reported outcomes such as improvement in access, care coordination, and patient satisfaction.

Data Analysis

Interview data are initially analyzed by using a rapid coding approach at the site level.¹¹ The majority of interview guide questions define specific CFIR constructs allowing researchers to efficiently collect and categorize data. After each interview, the research team reviews, notes, and evaluates the consistency to the predefined CFIR constructs, reaching consensus on all construct codes. The evaluation document includes field notes on each CFIR construct to allow for emerging findings (i.e., CFIR constructs not anticipated).

After code assignments, both team members assign ordinal ratings to each CFIR construct for each respondent.¹⁰ This ordinal rating is used to indicate whether the respondent considers the construct to be manifested negatively or positively, weakly or strongly. The interview team then meets to reach consensus on each rating. When all interviews for a site are completed, they are used to assign an overall site rating of each construct. Senior evaluation team members then review site ratings for each construct as a final check of the codes and ratings. Our coding process involves identification of emergent codes and issues not previously identified. Furthermore, we look for relationships among specific constructs and implementation process patterns. To assess changes in barriers and facilitators to the implementation process, the key informant interviews are conducted at two different time points throughout the evaluation.

Interview data are analyzed by using an iterative, content analysis methodology, where an open coding approach is used.¹² A priori codes are not defined for this portion of the analysis, although many codes address specific issues identified a priori related to the initiative being evaluated.¹² The coding process is organized by using Atlas.ti software, interviews are independently coded, and finally codes are combined.¹³ The analytic team then reviews the documents to clarify meanings of codes with emergent codes added throughout the analysis. The analytic team meets weekly to biweekly to discuss interviews, codes, memos, themes, and emerging conclusions. Members of the multidisciplinary research team, comprising clinicians, implementation scientists, and mixed methodologists, review findings to assess their thoroughness and comprehensiveness.¹² To ensure proper training of the analytic team, members participated in a 2 day, in-person training, covering data collection and analysis of key informant data.

Quantitative analysis occurs simultaneously throughout the evaluation to assess for changes in practice. Workload analyses of the initiatives are done by using CDW data with codes provided from OSC to derive the number of consults or sessions completed at each site and specialty. Other analyses include volume of consults over time, volume and travel cost, volume and travel cost by Veterans Integrated Service Network (VISN), change in wait times, costs for out of system care, health services utilization, and quality of care measures.

Results

We describe initial findings based on the RE-AIM framework. Program reach is steadily increasing over time. As of October 2013, there have been 740,149 e-Consults generated and 3,398 patients have been presented in SCAN-ECHO sessions. Initiative effectiveness has been assessed through semi-structured interviews and provider surveys. For example, we asked PCPs and specialists about their opinion of quality of care provided by e-Consults and SCAN-ECHO and providers shared that both initiatives improved the quality and efficiency of face-to-face consults.

In terms of adoption, e-Consults have been implemented at 121 VHA medical centers for 58 specialties as of October 2013. There are 280 clinics in the VHA implementing SCAN-ECHO for 17 different specialties (including hepatitis C, pain, endocrinology, etc.). SCN is implemented at 25 sites for 22 specialties, and 12 sites are currently implementing SC-MRP for three specialties (musculoskeletal, pain, and dermatology). We are evaluating initiative implementation through CFIR-grounded qualitative interviews to identify barriers and facilitators. Depending on high or low implementation, factors such as receiving workload credit, having protected time, compatibility with existing workflow/systems, leadership engagement, and access to information/resources are considered either barriers or facilitators to implementation.¹⁰ Finally, evaluation of maintenance occurs through 12-month follow-up interviews assessing how the initiatives are functioning since their launch and the site's plans in continuing the program.

Case Study of the e-Consult Evaluation

The e-Consult initiative was implemented at 15 pilot sites (representing 11 out of 21 geographic VHA networks across the country) in the spring of 2011 (Table 1). All sites were funded to support implementation of e-Consults in more than one specialty area.

Table 1.

e-Consult Sites, Specialties, and Counts During Pilot Implementation Period May 2011–May 2012

e-CONSULT SITE	SPECIALTY	e-CONSULT COUNT
1	Neurosurgery	93
2	Pain	278
3	Diabetes	834
4	Diabetes	463
5^a	Oncology	21
6^a	Hematology	261
7	Dementia	10
8	Oncology	243
9	Cardiology	201
10	Diabetes	621
11	Diabetes	68
12	Dementia	12
13	Rheumatology	78
14^b	Liver transplant	151
15^b	Dementia	41

One site implementing e-Consults for two different specialties.

e-Consults, electronic-consults.

The Web-based survey completed by 21 e-Consult clinical leads identified the following 14 key factors that are important to implementation: (1) compatibility; (2) relative priority; (3) goals and feedback; (4) leadership engagement; (5) available resources; (6) champions; (7) executing; (8) reflecting and evaluating; (9) design, quality, and packaging; (10) readiness for implementation; (11) knowledge and beliefs about e-Consults; (12) adaptability; (13) networks and communications; and (14) engaging (Table 2).⁹

Table 2.

The 14 CFIR Constructs Identified by Clinical Leads at the Pilot e-Consult Sites as Important Factors to Implementation of e-Consults

CONSTRUCT	SHORT DESCRIPTION
Compatibility	The degree of tangible fit between meaning and values attached to the intervention by involved individuals, how those align with individuals' own norms, values, and perceived risks and needs, and how the intervention fits with existing workflows and systems.
Relative priority	Individuals' shared perception of the importance of the implementation within the organization.
Goals and feedback	The degree to which goals are clearly communicated, acted on, and fed back to staff and alignment of that feedback with goals.
Leadership engagement	Commitment, involvement, and accountability of leaders and managers with the implementation.
Available resources	The level of resources dedicated for implementation and ongoing operations, including money, training, education, physical space, and time.
Champions	Individuals who dedicate themselves to supporting, marketing, and “driving through” an implementation, overcoming indifference or resistance that the intervention may provoke in an organization.
Executing	Carrying out or accomplishing the implementation according to plan.
Reflecting and evaluating	Quantitative and qualitative feedback about the progress and quality of implementation accompanied with regular personal and team debriefing about progress and experience.
Design, quality, and packaging	Perceived excellence in how the intervention is bundled, presented, and assembled.
Readiness for implementation	Tangible and immediate indicators of organizational commitment to its decisions to implement an intervention.
Knowledge and beliefs about the intervention	Individuals' attitudes toward and value placed on the intervention as well as familiarity with facts, truths, and principles related to the intervention.
Adaptability
Networks and communications	The nature and quality of webs of social networks and the nature and quality of formal and informal communications within an organization.
Engaging	Attracting and involving appropriate individuals in the implementation and use of the intervention through a combined strategy of social marketing, education, role modeling, training, and other similar activities.

CFIR, Consolidated Framework for Implementation Research.

Next, using administrative data from CDW, sites were identified for purposive sampling to conduct in-depth interviews of key informants. Based on the number of e-Consults completed from May 2011 to May 2012, sites were divided into four groups: (1) No utilization; (2) Low utilization (1–19 e-Consults); (3) Intermediate utilization (20–68 e-Consults); and (4) High utilization (69–1,357 e-Consults). Of the initial 15 sites, 0 sites had no utilization and 2 sites had low utilization, averaging 11 consults (SD: 1.4; range 10–12). There were three sites with intermediate utilization averaging 43 consults (SD: 23.6; range 21–68), and 10 sites with high utilization, averaging 322 e-Consults (SD: 244.8; range 78–834). During the pilot implementation period, 3,375 e-Consults were completed and the three specialties with the greatest number of e-Consults were diabetes (1,986), pain management (278), and oncology (264).

Two medical centers were randomly selected from each of the four categories to participate in the semi-structured interviews. We conducted 37 interviews at 8 different sites with primary and specialty care providers and clinic administrators who participated in the initiative. Throughout the interviews, seven factors were identified as facilitators to implementation in high utilization sites and as barriers in low utilization sites and include: (1) adaptability of e-Consult for meeting local needs; (2) compatibility of e-Consults with existing workflows and systems; (3) feedback on progress and effectiveness of implementation of e-Consults; (4) quality of networks and communications among providers; (5) available resources for training; (6) leadership engagement; and (7) access to knowledge and information about e-Consults (Table 3).⁹

Table 3.

Barriers and Facilitators to e-Consult Implementation as Identified Through the Qualitative Interviews

CONSTRUCT	SHORT DESCRIPTION
Adaptability	The degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs.
Compatibility	The degree of tangible fit between meaning and values attached to the intervention by involved individuals, how those align with individuals' own norms, values, and perceived risks needs, and how the intervention fits with existing workflows and systems.
Reflecting and evaluating	Quantitative and qualitative feedback about the progress and quality of implementation accompanied with regular personal and team debriefing about progress and experience.
Networks and communication	The nature and quality of webs of social networks and the nature and quality of formal and informal communications within an organization.
Available resources	The level of resources dedicated for implementation and ongoing operations, including money, training, education, physical space, and time.
Leadership engagement	Commitment, involvement, and accountability of leaders and managers with the implementation.
Access to knowledge and information	Ease of access to digestible information and knowledge about the intervention and how to incorporate it into work tasks.

Next, the provider survey was fielded to primary and specialty care physicians, pharmacists, program managers, project coordinators, and nurses between June 21, 2012 and July 10, 2012 to assess questions related to implementation fidelity, barriers and facilitators to e-Consult implementation, and impact of implementation on patients and staff. A total of 180 surveys from 87 sites were completed and analyzed. Overall, the majority of providers (52.3% of PCPs; 69.7% of specialists) surveyed and interviewed believed that patients preferred fewer face-to-face visits and appreciated less travel and wait time for specialist input. Nearly two-thirds (64.7%) of providers surveyed agreed that e-Consults had increased efficiency. The interviews identified consistent themes in the perception that e-Consults had improved communication among providers.

The majority of providers interviewed believed that e-Consults contributed to increasing PCP knowledge and competencies in patient care and was supported in the provider survey, with agreement among 56.7% at high utilization sites and 28.6% at low utilization sites. Concepts that emerged from the provider survey included the perception that e-Consults have improved access (61.7%), care coordination (60.2%), and quality of care (63%).

Discussion

Our goals are to describe our methods of a large-scale, broad evaluation that meets the needs of operations and research. Evaluation of these initiatives has demonstrated benefits of implementation for both clinicians and patients. In addition, based on these findings, the evaluation team has provided recommendations to VHA operations, which have resulted in refinement of the initiative implementation process. This collaborative evaluation may provide a framework for future partnerships incorporating operational and research objectives.

The Evaluation Center has worked closely with OSC, adjusting timing and content of deliverables to meet operational needs. We have provided feedback on evaluation components to OSC, and they have incorporated our findings into guidance documents for subsequent rollout of the initiatives. In addition, this partnership has evolved. First, the evaluation team has suggested further analytics (e.g., travel distance averted by using e-Consult instead of face-to-face consult) to support guidance to the field with emphasis on meeting the needs of operations (e.g., medical centers and CBOCs) and OSC. Moreover, the Evaluation Center has worked with individual medical centers, addressing specific questions and helping facilitate initiative implementation. For example, the Evaluation Center has incorporated geospatial information systems mapping to analyze the spread of e-Consults. Based on these maps, OSC and medical center operational leaders are promoting e-Consult implementation in specific locations and targeting specific patient population needs.

Our operational stakeholders, OSC, have been involved in all aspects of the evaluation process. They have the opportunity to participate in the interpretation of findings and reporting of results. Evaluation Center members and OSC have presented evaluation findings to individuals in OSC leadership, providing insight into how the initiatives are impacting care in the VHA. In addition, OSC has incorporated evaluation findings to increase acceptability and sustainability of subsequent initiative rollout. Members at each Evaluation Center site contribute and participate through either quantitative and/or qualitative expertise. Weekly qualitative and quantitative calls report progress on individual aspects of the evaluation, and members from one group participate in the other group's meetings to provide updates and discuss analyses that support one another. This close collaboration between groups and across sites allows for all members of our Evaluation Center to be involved and contribute their expertise.

In summary, we describe our evaluation approach guided by iteration and collaboration. To date, we have completed the initial evaluation of e-Consults, SCAN-ECHO, and SCN and the evaluation is planned to continue until September 30, 2015. One potential limitation to our current evaluation is that we have not directly assessed outcomes of the initiatives. Future evaluation efforts will focus on assessing outcome measures for each of the specialty care initiatives. This evaluation meets the needs of both operations and research and provides an opportunity to share ways in which research and operational partnerships may enhance the outcome of the work, as well as the view of each other's needs. This partnership will continue, and evaluation of the initiatives will move toward focusing on sustainability over the next few years.

Conclusions

This partnership has effectively and iteratively informed national implementation and policy of the rollout of the Office of Specialty Care initiatives. Capturing quantitative and qualitative data provided structured guidance, leading to scale-up and spread of the e-Consult initiative from 11 pilot sites to national implementation (e.g., 121 sites for 58 specialties). Thus, the research was actionable and researchers could see the impact of their work (within weeks to months) much faster than that seen in typical research (years to decades). In summary, our findings inform practical aspects of conducting partnered research. Partnered research may have a more timely impact on both operations and policy while at the same time fostering discovery of new and generalizable research knowledge.

Footnotes

Acknowledgments

The study was conducted with support from the U.S. Department of Veteran Affairs, Office of Specialty Care (OSC) and VA Health Services Research and Development (HSR&D) Quality Enhancement Research Initiative (QUERI), and it was undertaken by the Specialty Care Evaluation Center. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs.

Disclosure Statement

No competing financial interests exist.

Appendix 1

ORCA

1.1 Please select the VISN, facility, and division you primarily work in: [Drop down menu w/ VISN]

1.2 What is the main type of service you provide? [Have drop down list of specialty care services and primary care]

1.3 Is any part of your work time spent in a primary care clinic at the VA?

Instructions and introduction

This survey is intended for [primary care providers/specialty care providers]. The purpose is to evaluate progress on the [SCT Initiative: e-Consult/SCAN-ECHO, SC-MRPs, PCM-MCD] and how it is working. The survey is voluntary, and your input is confidential and anonymous. Results of the survey will be used solely to evaluate the [SCT Initiative]. The results will not be used to evaluate you, but summary results will be provided to VA leadership to guide future development and support. Results will be provided only as grouped responses (e.g., average scores for clinics), not findings for specific individuals, and no data from any site with fewer than 10 respondents will be reported to assure confidentiality.

We will field this survey again in ∼1 year, to assess changes over time. We will share summary data with every clinic (or facility, if the clinic has fewer than 10 respondents).

The survey will be open from [open date] until midnight Pacific Time on [close date].

If you have any questions, please contact Christian Helfrich at christian.helfrich@va.gov.

We appreciate your help, thank you.

SECTION 1

Finding: The [SCT Initiative] will improve quality of care for patients with [SCT initiative target condition].

Based on your assessment, please rate the evidence for the statement made earlier, on a scale of 1 to 5, where 1 is very weak evidence and 5 is very strong evidence:

Very weak Weak Neither weak nor strong Strong Very strong

1 2 3 4 5

Now, rate the strength of the evidence basis for this statement based on how you think respected clinical experts in your institution feel:

Very weak Weak Neither weak nor strong Strong Very strong

1 2 3 4 5

For each of the following statements, please rate the strength of your agreement with the statement, from 1 (strongly disagree) to 5 (strongly agree):

The proposed practice changes or guideline implementation:

a) are(is) supported by randomized controlled trials or other scientific evidence from the VA