Implementing and Sustaining Team-Based Telecare for Bipolar Disorder: Lessons Learned from a Model-Guided,Mixed Methods Analysis

Abstract

Background:

Telemental health interventions have empirical support from clinical trials and structured demonstration projects. However, their implementation and sustainability under less structured clinical conditions are not well demonstrated.

Introduction:

We conducted a follow-up analysis of the implementation and sustainability of a clinical video teleconference-based collaborative care model for individuals with bipolar disorder treated in the Department of Veterans Affairs to (a) characterize the extent of implementation and sustainability of the program after its establishment and (b) identify barriers and facilitators to implementation and sustainability.

Materials and Methods:

We conducted a mixed methods program evaluation, assessing quantitative aspects of implementation according to the Reach, Efficacy, Adoption, Implementation, and Maintenance implementation framework. We conducted qualitative analysis of semistructured interviews with 16 of the providers who submitted consults, utilizing the Integrated Promoting Action on Research Implementation in the Health Services implementation framework.

Results:

The program demonstrated linear growth in sites (n = 35) and consults (n = 915) from late 2011 through mid-2016. Site-based analysis indicated statistically significant sustainability beyond the first year of operation. Qualitative analysis identified key facilitators, including consult content, ease of use via electronic health record, and national infrastructure. Barriers included availability of telehealth space, equipment, and staff at the sites, as well as the labor-intensive nature of scheduling.

Discussion:

The program achieved continuous growth over almost 5 years due to (1) successfully filling a need perceived by providers, (2) developing in a supportive context, and (3) receiving effective facilitation by national and local infrastructure.

Conclusion:

Clinical video teleconference-based interventions, even multicomponent collaborative care interventions for individuals with complex mental health conditions, can grow vigorously under appropriate conditions.

Introduction

Clinical video teleconferencing (CVT) has substantial potential to replicate face-to-face care. Research data and clinical experience have demonstrated its utility in a wide variety of mental health conditions.^{3

–11} In aggregate, comparisons to control conditions for several diagnoses demonstrate efficacy, although noninferiority to face-to-face care is yet to be determined.³ Furthermore, data on the acceptability of CVT-based treatments for populations with mental health conditions have been mixed.^4,5,10
–12

Challenges in moving CVT interventions from highly controlled demonstrations and controlled trials into general practice have been recognized since the early days of CVT.^{1,2,13

–17} Factors contributing to this “disjunction of research and practice”¹⁸ include technical, social, organizational, and policy factors,^2,13,15 as well as lack of empirical evidence on outcomes.^18,19 Such factors also affect the adoption and sustainment of pharmacological healthcare innovations generally.²⁰ CVT, however, is a multistakeholder process, where buy-in from a range of parties is necessary to implement and sustain the modality. For instance, Broens et al.² empirically identified five major determinants of telehealth implementation: (1) technology, (2) acceptance, (3) financing, (4) organization, and (5) policy/legislation. The commitment of a variety of stakeholders across determinants must align to implement and sustain telehealth, respectively: information technology staff, providers and the individuals in their care, insurance and other reimbursement mechanisms, organizational resources and leadership priorities, and a supportive or at least permissive policy environment.²

We recently reported the clinical and quality impact of a CVT-based collaborative chronic care model (CCM) for bipolar disorder deployed across multiple regions in the U.S. Department of Veterans Affairs (VA) healthcare system.²¹ CCMs are multicomponent models developed initially for chronic medical illnesses treated in primary care,^22,23 which now also have a strong evidence base across a wide variety of mental health conditions.^24,25 CCMs consist of six elements: (1) work role redesign for anticipatory, continuous care; (2) self-management skill enhancement; (3) provider decision support; (4) information management; (5) access to community resources; and (6) organization and leadership support.²⁶ Previously published analysis of the first 400 participants (November 2011 through May 2014) demonstrated significant improvements in three of four measures of quality of care and, for program completers, significant improvements in bipolar symptoms and mental health quality of life.²¹

To identify factors that impact implementation of CVT modalities in clinical practice, we thus undertook the current quality improvement/program analysis, which focused on two questions: (1) to what degree was this program implemented and subsequently sustained through May 2016 and (2) what are the barriers and facilitators that impacted implementation and sustainment? The analytic plan was guided by two widely used implementation science²⁷ frameworks.

Materials and Methods

This evaluation was judged to be quality improvement/program evaluation by the VA Boston Institutional Review Board and thus, in accordance with the VA Office of Research Oversight procedures, was exempted from Institutional Review Board review. The study took place within the VA healthcare system, which is divided into 18 administrative regions. Healthcare is provided by over 150 separate healthcare systems. Healthcare systems are typically made up of a main medical center plus several community-based clinics that are geographically separate from the main medical center; in total, the VA includes over 1,000 distinct care sites. This report analyzes data at the level of the healthcare system and the individual site (i.e., main medical center or community-based clinic).

Program Description

The Bipolar Disorders Telehealth Program is described in detail in our prior report.²¹ Briefly, Bipolar Telehealth is a CVT-based program currently staffed by 1.1 full-time equivalent (FTE) psychiatrists and 2.6 FTE psychologists located across three sites, with administrative support from the VA National TeleMental Health Center. Sites are recruited via information dissemination to mental health leadership at the national and facility level, including national and regional information calls, plus word-of-mouth.

The target population consists of veterans with known or suspected bipolar disorder without other exclusion criteria. Clinical procedures are based on the Life Goals Collaborative Care Program, a bipolar CCM^28
–30 that is guideline endorsed^31,32 and listed on the Substance Abuse and Mental Health Administration's National Registry of Evidence-Based Programs and Practices (www.samhsa.gov/nrepp). Core components, delivered via CVT, include the following:

• Structured diagnostic assessment, including relevant medical and psychiatric comorbidities (by a psychiatrist or psychologist)

• Psychopharmacologic consultation (by a psychiatrist)

• Life Goals Self-Management Skills Modules²⁸ consisting of six structured, manual-based modules covering goals and values, mania and depression profiles and coping responses, provider visit preparation (by a psychologist)

• Follow-up monitoring, including measurement-based symptom assessment and liaison with primary providers (by a psychologist).

Bipolar Telehealth psychologists conduct Life Goals training while the psychiatrists make medication and monitoring recommendations to the primary mental health provider but do not write orders. Individuals are enrolled for ∼6 months.

Services are accessed via consultation request in the participating site's electronic health record. Clinical communications are made via chart note in the site's electronic health record with additional provider-to-provider communication as needed.

Program Evaluation Plan: Quantitative

The primary goal of the quantitative evaluation was to characterize the extent and pattern of program implementation and sustainment according to the Reach, Efficacy, Adoption, Implementation, and Maintenance (RE-AIM) framework. The RE-AIM framework³³ evaluates implementation according to reach (participation in the intervention, in this case by individuals in treatment), efficacy (clinical impact), adoption (proportion and representativeness of sites that utilize the innovation), implementation (adherence to procedures, in this case by individuals in treatment), and maintenance (sustainment beyond start-up). Specific measures were selected a priori and chosen according to the components of that framework as summarized in Table 1.

Table 1.

Quantitative Program Evaluation Measures Mapped onto Reach, Efficacy, Adoption, Implementation, and Maintenance Components

RE-AIM COMPONENT	APPLICATION TO BIPOLAR TELEHEALTH PROGRAM EVALUATION	PROGRAM EVALUATION MEASURE
Reach	Number and proportion of candidate individuals in care who received Bipolar Telehealth consultation	• Number of consults • Proportion of individuals with bipolar disorder at a participating site who received consults
Efficacy	Clinical impact of Bipolar Telehealth on individuals receiving care	• Health status and care quality (per prior publication)
Adoption	Proportion and representativeness of facilities that utilized Bipolar Telehealth	• Number and proportion of medical centers and regions participating • Geographic distribution of participating sites • Urban/rural classification • Medical center versus community-based clinic
Implementation	Participation in Bipolar Telehealth by individuals receiving care	• Completion of CVT intake interview • Completion of Life Goals Self-Management Skills modules
Maintenance	Sustainment of Bipolar Telehealth beyond start-up	• Accrual of consult requests beyond the first 400 (May 2014 through May 2016) • Subsequent years versus first-year consult numbers in sites with at least 2 years of participation

RE-AIM, Reach, Efficacy, Adoption, Implementation, and Maintenance; CVT, clinical video teleconferencing.

Program Evaluation Plan: Qualitative

The primary goal of the qualitative evaluation was to characterize the perceived usefulness of the program from the consulting provider perspective, and to identify barriers and facilitators to program implementation and sustainability. We analyzed provider interview data according to the Integrated Promoting Action on Research Implementation in the Health Services (i-PARIHS) model.³⁴ i-PARIHS proposes that successful implementation is a function of characteristics of the innovation, the recipients, the inner and outer context, and facilitation efforts to support implementation.

To minimize bias, a quality management nurse from the National TeleMental Health Center, who was not directly involved in the program, conducted 20-min semistructured interviews with 16 providers who had submitted consults to Bipolar Telehealth. Provider sampling was stratified by high- and low-use providers (10 and 6 providers, respectively). At the time of qualitative interviews, 89 providers had submitted consults. High utilizers were defined as those who had submitted five or more consults (n = 19), while low utilizers were defined as those who had submitted two or fewer consults (n = 61). Interviewees were selected randomly from each group. Interviewees were predominantly prescribers (14 of 16 [87.5%]: nine physicians, three nurses, and two doctors of pharmacy; plus one social worker and one psychologist). By comparison, the disciplinary distribution of the overall population of providers submitting consults included 73 (82.0%) prescribers: 60 physicians, 9 nurses, 4 doctors of pharmacy, 1 physician's assistant, 9 social workers or other counselors, and 5 psychologists. Thirteen (81.3%) of interviewees came from healthcare systems above the median split for number of consults.

The interview (available on request) contained 17 open-ended questions covering the following: consultation process; scheduling process; usefulness of the diagnostic, psychopharmacological, and self-management recommendations; and organizational support and infrastructure. The interviewer followed a structured interview tool to ensure consistency and took notes in real time but did not audio record the interview. Interview notes were analyzed by two experienced qualitative analysts who were not part of the program.

Responses were categorized in two dimensions: valence (positive/negative/neutral) and i-PARIHS domain (innovation, recipients, inner context, outer context, and facilitation)³⁴ using a directed content analytic approach,³⁵ and assigning responses to predetermined categories' aspects of the qualitative summary that related to national infrastructure policies and support also incorporated perceptions of the Bipolar Telehealth clinicians, since the consulting providers would have little knowledge of these factors. Table 2 provides a further definition of components and mapping of interview domains onto i-PARIHS components.

Table 2.

Qualitative Interview Themes Mapped onto Integrated Promoting Action on Research Implementation in the Health Services Components

i-PARIHS COMPONENT	BIPOLAR TELEHEALTH PROVIDER INTERVIEW DOMAIN	EXAMPLES OF ITEMS
Innovation	Usefulness of content of Bipolar Telehealth Program consult to the referring providers	• Provider perception of usefulness of: o Expert diagnosis review o Medication recommendations o Life Goals Self-Management Skills Modules in enhancing treatment participation by the individual receiving care
Recipients	Consulting providers, particularly alignment of provider values and attitudes with Bipolar Telehealth approach	• Provider perception of relevance of recommendations to their priority issues • Bipolar Telehealth staff perceptions of responsivity of consulting providers to recommendations
Inner and outer context	Facility infrastructure (inner); national policy and infrastructure support (outer)	• Provider perception of facility support • Bipolar Telehealth staff perceptions of national policy and infrastructure support issues
Facilitation	Support by National TeleMental Health Center and Bipolar Telehealth to make the consult process easy for the clinician and individual receiving consultation	• Consulting provider perception of: o Timeliness of consultation given acuity of needs o Ease of placing consult o Ease of scheduling o Convenience of CVT access points for individual receiving consultation

i-PARIHS, Integrated Promoting Action on Research Implementation in the Health Services.

Results

Quantitative Results

Reach

Figure 1 illustrates the cumulative growth of the program in terms of unique individuals served and number of sites since program inception. Overall, consults on 915 individuals were received across 35 distinct sites (main medical centers or community-based clinics) across 16 healthcare systems through May 2016.

Fig. 1.

Growth of the program.

The proportion of individuals with bipolar disorder who received a Bipolar Telehealth consult varied widely across the 16 healthcare systems (Fig. 2). Not surprisingly, sites that were engaged longer (leftward sites in the graph) tended to achieve greater penetration. Size also had a relationship to penetration, as five of the six sites with at least 20% penetration were classified as large, complex healthcare systems (“level 1”) by the VA; four of these five were also early sites, which may have been responsible for more extensive penetration.

Fig. 2.

Penetration within 16 VA healthcare systems. Sites are displayed in order of first participation in the program, from left to right. That is, Site A was the first site and Site P the most recent site. VA, Veterans Affairs.

Efficacy

As summarized in our prior report,²¹ the program had substantive clinical and quality impact on participants. Among the first 400 individuals evaluated, three of four guideline-based quality measures improved in the year postconsultation compared to the prior year. Specifically, lithium use increased (35.8% vs. 52.0%; McNemar's test p < 0.001), antidepressant use decreased (48.3% vs. 24.8%; McNemar's test p < 0.001), and prazosin use increased in those with comorbid posttraumatic stress disorder (21.9% vs. 35.0%; McNemar's test p < 0.02), while rates of obtaining lithium serum levels in those taking lithium did not change from their high baseline levels (76.4% vs. 75.5%). In addition, individuals who completed the Life Goals modules demonstrated significant improvement compared to baseline in mania (Internal State Scale [ISS]³⁶ Activation Subscale: 201.9 ± 131.4 vs. 163.4 ± 114.3, paired t test p < 0.02), depression (ISS Depression Index: 86.5 ± 56.2 vs. 53.0 ± 47.7; paired t test p < 0.001), and paranoia (ISS Perceived Conflict Subscale 148.1 ± 114 vs. 109.0 ± 90.3, paired t test p < 0.02), as well as mental health quality of life (VR-12 Mental Component Score³⁷: 30.0 ± 14.0 to 36.9 ± 13.5, paired t test p < 0.001). Life Goals completers did not differ from noncompleters on a wide variety of demographic and clinical characteristics.

Adoption

By May 2016, Bipolar Telehealth had been established across 12 of 50 states in 8 of 20 VA administrative regions. This included 35 distinct clinical sites that were part of 16 of 152 healthcare systems (10.5%). Geographically, the sites came predominantly from east of the Mississippi (n = 32, 91%). Ten of 35 sites (28.6%) were classified as rural by VA criteria. Ninety-six of the 915 (10.5%) individuals were classified as rural, reflecting the fact that rural sites were somewhat smaller and engaged in Bipolar Telehealth somewhat later than urban sites.

Figure 3 summarizes the number of individuals seen at each of the 16 healthcare systems. The degree of overall adoption varied widely among healthcare systems, ranging from 3 to 196 consults. As can be seen in Figure 3, in many of the healthcare systems there was substantial penetration of Bipolar Telehealth beyond the main medical center and into community-based clinics. As is expectable, those more recently activated (rightward sites in the graph) submitted fewer consults than healthcare systems engaged earlier (leftward sites). Notably, some earlier sites sent relatively fewer consults (Sites C and K; Site J was designed to consult only on hospitalized individuals with refractory symptoms). Site C lost the telehealth support staff and was not able to replace them. At Site K, the main consulting provider did not agree with the content of several early consultations and ceased participation.

Fig. 3.

Consults across 16 VA healthcare systems. Sites are displayed in order of first participation in the program, from left to right. That is, Site A was the first site and Site P the most recent site.

Implementation

Figure 4 summarizes flow through the program. Of 915 individuals, 675 (73.8%) completed CVT interview. Chart review consultation without interview was provided to 129 (53.75%) of those not interviewed.

Fig. 4.

Flow through the Bipolar Telehealth Program.

Reasons varied for noncompletion of the CVT intake interview (n = 240): 10 (4.2%) were not clinically appropriate, 6 (2.5%) were discontinued by the referring provider, 170 (70.8%) declined or otherwise were unable to be engaged, and 54 (22.5%) were willing but unable. Reasons among this last group included transportation (n = 12, 22.2%), scheduling conflicts (n = 7, 13.0%), or other (n = 35, 64.8%).

Of those who completed CVT intake (n = 675), 452 (67%) began Life Goals modules. Among the remainder, Life Goals was not clinically indicated in 58 (8.6%) and was declined in 116 (17.2%), while 49 (7.2%) were unable to participate for various reasons. Among those for whom Life Goals was not clinically indicated, reasons included not bipolar spectrum (50.8%), too ill to participate (13.8%), or otherwise not clinically appropriate (35.4%). Of those beginning Life Goals modules, 319 (70.6%) completed all six modules, over a mean of 10.4 ± 3.9 sessions. Those who began but did not complete attended 3.8 ± 2.6 sessions.

Maintenance

Since the initial program impact evaluation on the first 400 participants (through May 2014), the program has continued to grow in terms of consults and sites (Fig. 1). As noted above, 2 of the 35 sites (5.7%) are no longer active.

For healthcare systems with at least 2 years' participation by May 2016 (n = 14), we further characterized sustainment by comparing the number of consultations in the first versus subsequent years of participation (including Sites C and K). Median number of consultation increased from 17 to 30 in year 1 versus subsequent years (Wilcoxon signed rank p = 0.0005); nine sites increased their use after Year 1, while five plateaued or decreased. Differences in use were not associated with facility complexity, urban/rural facility setting, overall number of Veterans using general mental health services, or proportion of Veterans residing in rural zip codes.

Qualitative Results

Innovation

Providers who submitted consults generally appreciated that the consultations clarified and supported evidence-based practices. Timeliness of response was by and large acceptable, as was the ease of accessing the completed consult in the electronic health record. There were some instances in which a quicker turnaround would have been appreciated; delays were typically related to scheduling (provider openings and/or availability of the individual). Consultations were seen as useful in treatment planning, particularly if the individual in treatment had difficulty accepting the diagnosis or plan. However, recommendations sometimes caused tension or disagreement between the individual in treatment and provider if the provider did not agree with the recommendations. Providers who submitted consults were quite positive about Life Goals Self-Management Skills Modules, although some noted that the program did not always seem to be integrated into other aspects of treatment. Providers noted overall positive feedback from individuals in treatment, who cited in particular receiving a fresh opinion on diagnosis and treatment options, and support about living with a mental health condition.

Recipients

Quantitative data (above) indicated wide acceptance of the program by consulting providers. Qualitative data confirmed this and also revealed occasional disjunction of consultant/provider opinions. When this did occur, it was sometimes due to disagreement on the content of the recommendations, and sometimes due to concern about how much information the consultant provided to the individual. This sometimes led to raising the individual's expectations for specific treatment from the referring provider, with the latter feeling pressured to implement a particular recommendation.

Inner and outer context

Among inner context factors, availability of telehealth space and equipment varied, as did availability and skill of facility-based telehealth support staff. This was most acute at smaller community-based clinics where multiple programs compete for limited telehealth availability. Local formulary restrictions sometimes limited the ease of access to recommended medications, although it was noted that consultants also helped justify the use of medications that were classified as second- or third-line treatments on local formularies.

Among outer context factors, most obvious was the provision of centralized VA funding and resources to make Bipolar Telehealth available to facilities without the expectation that individual facilities provide financial support. On the contrary, Bipolar Telehealth adoption by individual facilities occurred despite many competing policy initiatives and priorities from the national, regional, and facility levels.

Facilitation

The scheduling process was noted to be labor-intensive, requiring a three-way match among the veteran's schedule, the consultant's schedule, and telehealth room availability. However, consulting providers appreciated not having involvement in scheduling, which was centralized. One site noted the importance of having a facility-based “champion” who both publicized the program locally and managed liaison between facility clinical and telehealth staff for troubleshooting issues.

Discussion

Key Findings in Context

The Bipolar Telehealth Program has experienced linear and continuing growth and good sustainability over almost 5 years of operation. Key facilitators included valued recommendations, ease of use and integration into ongoing workflow via electronic health record for consulting providers, and extensive infrastructure at the national level to support implementation. Notable barriers included the labor-intensive nature of scheduling; variable availability of telehealth space, equipment, and staff at certain sites; and, on occasion, disjunction of opinions of consulting provider and consultant.

Mental health applications have been the most published on telehealth modalities, after store-and-forward applications (teleradiology, telepathology, and teledermatology).¹⁸ Notably, however, we could find no comparable analyses of CVT implementation and sustainability of telemental health interventions. Literature review revealed that adoption rates compared favorably with that of a CVT depression care management intervention in community-based clinics,⁵ and rates of participation in Life Goals modules resembled rates seen in clinical trials of on-site delivery.^28
–30 A recent qualitative implementation analysis of depression care management in an HIV population similarly identified care management and integration of consultation into the electronic health record as facilitators, while difficulty in incorporating depression case-finding into routine work-flow was a barrier.³⁸

Generalizable lessons learned

Several prior analyses and reviews describe themes regarding implementation and sustainability that echo the Bipolar Telehealth experience. One review² identified five factors, including technological support, incorporation into daily work-flow, standardization of processes, policy support, and financing, while another¹⁸ emphasized several “sociolegal” aspects that require organization-level policy support including confidentiality, professional portability, and malpractice. Another review noted the importance of champions who initiate the service, promote the method, and build relationships between providers.¹⁴

From a specifically organizational implementation perspective, others¹³ have identified technical, social, and organizational dimensions of implementation. Technical characteristics must add value to existing work. Social aspects require peer attitudes that align with the innovation. Organizational aspects need to support or at least minimally disrupt organizational priorities.

By any formulation, the goal for adapting an innovation like telehealth is “normalization,”²⁰ which Robert et al. have proposed has four components, illustrated by four queries:

• Interactional workability: Does the innovation fit with the microenvironment of the clinical encounter?

• Relational integration: Does it fit within the network of relationships?

• Skill set workability: Does it fit with the formal and informal division of labor?

• Contextual integration: Does the organization endorse and allocate resources?

To each of these queries, the mixed methods data on Bipolar Telehealth support an affirmative response, predicting continued sustainability and growth—that is, normalization of the process.

Limitations

Most notable among limitations of these analyses is the generalizability of experience in the VA to other healthcare systems. In particular, the availability of national support infrastructure and the absence of billing for services were distinct advantages. However, as healthcare systems consolidate and capitated payments become more widespread, supporting such telehealth efforts will become more feasible. In addition, these analyses were purposely conducted outside of a controlled trial framework, which may have yielded different results. However, this approach increases the generalizability of results and responds to a long-identified need in the field.^1,2

Conclusion: Future Directions

As Robert et al. point out, implementation and sustainability are parts of an ongoing process, not discrete events.²⁰ This report and its prior companion piece²¹ indicate that even complex CCM-type interventions applied to complex populations such as those afflicted with bipolar disorder can have clinical impact and be sustained and grow over years. Continued attention to facilitators and barriers can ensure normalization of CVT-based telehealth as a core clinical process.

Footnotes

Acknowledgments

This work was supported by clinical funds from the Department of Veterans Affairs Office of Telehealth Services. Dr. M.S.B. receives royalties on publications based on the Life Goals model from New Harbinger and Springer. Although this work was funded by clinical funds from the U.S. Department of Veterans Affairs, the views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

Disclosure Statement

Dr. M.S.B. receives royalties from Springer Publishing and New Harbinger Publishing for work related to the collaborative chronic care model for bipolar disorders. No other authors report conflicting interests.

References

May

, Harrison

, Finch

, MacFarlane

, Mair

, Wallace

. Understanding the normalization of telemedicine services through qualitative evaluation. J Am Med Inform Assoc, 2003; 10:596–604.

Broens

, Huis in't Veld

, Vollenbroek-Hutten

, Hermens

, van Halteren

, Nieuwenhuis

. Determinants of successful telemedicine implementations: A literature study. J Telemed Telecare, 2007; 13:303–309.

Bolton

, Dorstyn

. Telepsychology for posttraumatic stress disorder: A systematic review. J Telemed Telecare, 2015; 21:254–267.

Chong

, Moreno

. Feasibility and acceptability of clinic-based telepsychiatry for low-income Hispanic primary care patients. Telemed J E Health, 2012; 18:297–304.

Deen

, Fortney

, Schroeder

. Patient acceptance of and initiation and engagement in telepsychotherapy in primary care. Psychiatr Serv, 2013; 64:380–384.

Fortney

, Pyne

, Kimbrell

, et al. Telemedicine-based collaborative care for posttraumatic stress disorder: A randomized clinical trial. JAMA Psychiatry, 2015; 72:58–67.

Fortney

, Pyne

, Mouden

, et al. Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: A pragmatic randomized comparative effectiveness trial. Am J Psychiatry, 2013; 170:414–425.

Godleski

, Darkins

, Peters

. Outcomes of 98,609 U.S. Department of Veterans Affairs patients enrolled in telemental health services, 2006–2010. Psychiatr Serv, 2012; 63:383–385.

Gros

, Morland

, Greene

, et al. Delivery of evidence-based psychotherapy via video telehealth. J Psychopathol Behav Assess, 2013; 35:506–521.

10.

Hernandez-Tejada

, Zoller

, Ruggiero

, Kazley

, Acierno

. Early treatment withdrawal from evidence-based psychotherapy for PTSD: Telemedicine and in-person parameters. Int J Psychiatry Med, 2014; 48:33–55.

11.

Johansson

, Lindberg

, Soderberg

. Patients' experiences with specialist care via video consultation in primary healthcare in rural areas. Int J Telemed Appl, 2014; 2014:143824.

12.

Rees

, Stone

. Therapeutic alliance in face-to-face versus videoconferenced psychotherapy. Prof Psychol Res Pr, 2005; 36:649–653.

13.

Cresswell

, Sheikh

. Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review. Int J Med Inform, 2013; 82:e73–e86.

14.

Wade

, Eliott

. The role of the champion in telehealth service development: A qualitative analysis. J Telemed Telecare, 2012; 18:490–492.

15.

Wade

, Taylor

, Kidd

, Carati

. Transitioning a home telehealth project into a sustainable, large-scale service: A qualitative study. BMC Health Serv Res, 2016; 16:183.

16.

Brooks

, Turvey

, Augusterfer

. Provider barriers to telemental health: Obstacles overcome, obstacles remaining. Telemed J E Health, 2013; 19:433–437.

17.

Hall

, McGraw

. For telehealth to succeed, privacy and security risks must be identified and addressed. Health Aff (Millwood), 2014; 33:216–221.

18.

Miller

. Solving the disjuncture between research and practice: Telehealth trends in the 21st century. Health Policy, 2007; 82:133–141.

19.

Hebert

, Korabek

, Scott

. Moving research into practice: A decision framework for integrating home telehealth into chronic illness care. Int J Med Inform, 2006; 75:786–794.

20.

Robert

, Greenhalgh

, MacFarlane

, Peacock

. Adopting and assimilating new non-pharmaceutical technologies into health care: A systematic review. J Health Serv Res Policy, 2010; 15:243–250.

21.

Bauer

, Krawczyk

, Miller

, et al. Team-based telecare for bipolar disorder. Telemed J E Health, 2016; 22:855–864.

22.

Von Korff

, Gruman

, Schaefer

, Curry

, Wagner

. Collaborative management of chronic illness. Ann Intern Med, 1997; 127:1097–1102.

23.

Wagner

, Austin

, Von Korff

. Organizing care for patients with chronic illness. Millbank Q, 1996; 74:511–545.

24.

Woltmann

, Grogan-Kaylor

, Perron

, Georges

, Kilbourne

, Bauer

. Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: Systematic review and meta-analysis. Am J Psychiatry, 2012; 169:790–804.

25.

Miller

, Grogan-Kaylor

, Perron

, Kilbourne

, Woltmann

, Bauer

. Collaborative chronic care models for mental health conditions: Cumulative meta-analysis and metaregression to guide future research and implementation. Med Care, 2013; 51:922–930.

26.

Coleman

, Austin

, Brach

, Wagner

. Evidence on the chronic care model in the new millennium. Health Aff (Millwood), 2009; 28:75–85.

27.

Bauer

, Damschroder

, Hagedorn

, Smith

, Kilbourne

. An introduction to implementation science for the non-specialist. BMC Psychol, 2015; 3:32.

28.

Bauer

, McBride

, Williford

, et al. Collaborative care for bipolar disorder: Part I. Intervention and implementation in a randomized effectiveness trial. Psychiatr Serv , 2006; 57:927–936.

29.

Bauer

, McBride

, Williford

, et al. Collaborative care for bipolar disorder: Part II. Impact on clinical outcome, function, and costs. Psychiatr Serv , 2006; 57:937–945.

30.

Simon

, Ludman

, Bauer

, Unutzer

, Opersalski

. Long-term effectiveness and cost of a systematic care program for bipolar disorder. Arch Gen Psychiatry, 2006; 63:500–508.

31.

Group VDMoBDW.. Management of bipolar disorder in adults, version 2.0 (summary). Department of Veterans Affairs and Department of Defense, 2010.

32.

Yatham

, Kennedy

. Canadian network for mood and anxiety treatments (CANMAT) guidelines for the management of patients with bipolar diosrder: Consensus, controversies, and international commentary. Bipolar Disord, 2005; 7(Suppl 3):1–88.

33.

Glasgow

, Vogt

, Boles

. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am J Public Health, 1999; 89:1322–1327.

34.

Harvey

, Kitson

. PARIHS re-visited. Introducing the i-PARIHS framework. In: Harvey

, Kitson

, eds. Implementing evidence-based practice in healthcare: A facilitation guide. London, Unite Kingdom: Routledge, 2015.

35.

Hsieh

, Shannon

. Three approaches to qualitative content analysis. Qual Health Res, 2005; 15:1277–1288.

36.

Bauer

, Crits-Christoph

, Ball

, et al. Independent assessment of manic and depressive symptoms by self-rating. Arch Gen Psychiatry, 1991; 48:807–812.

37.

Selim

, Rogers

, Fleishman

, et al. Updated U.S. population standard for the Veterans RAND 12-item Health Survey (VR-12). Qual Life Res, 2009; 18:43–52.

38.

Drummond

, Painter

, Curran

, et al. HIV patient and provider feedback on a telehealth collaborative care depression intervention. AIDS Care, 2017; 29:290–298.