Abstract
The telehealth field has advanced historic promises to improve access, cost, and quality of care. However, the extent to which it is delivering on its promises is unclear as the scientific evidence needed to justify success is still emerging. Many have identified the need to advance the scientific knowledge base to better quantify success. One method for advancing that knowledge base is a standard telemental health evaluation model. Telemental health is defined here as the provision of mental health services using live, interactive video-teleconferencing technology. Evaluation in the telemental health field largely consists of descriptive and small pilot studies, is often defined by the individual goals of the specific programs, and is typically focused on only one outcome. The field should adopt new evaluation methods that consider the co-adaptive interaction between users (patients and providers), healthcare costs and savings, and the rapid evolution in communication technologies. Acceptance of a standard evaluation model will improve perceptions of telemental health as an established field, promote development of a sounder empirical base, promote interagency collaboration, and provide a framework for more multidisciplinary research that integrates measuring the impact of the technology and the overall healthcare aspect. We suggest that consideration of a standard model is timely given where telemental health is at in terms of its stage of scientific progress. We will broadly recommend some elements of what such a standard evaluation model might include for telemental health and suggest a way forward for adopting such a model.
Introduction
The telehealth field continues to advance on promises to improve quality and utilization of care, particularly to address the unmet healthcare needs of those living in rural, remote, and underserved populations. 1 However, integration of telehealth into the mainstream is still not fully realized, in part because of questions about the quality of scientific literature. 2 Scientifically, the field has evolved from basic pilot and user satisfaction studies to a growing number of outcome-based studies. 3 There is now a large body of research and evaluation literature that includes strong evidence for specific telehealth benefits. However, some consider the sum of that literature inconsistent in terms of how it measures those benefits. 4 –6 This has prompted commentary on the need for more consistent scientific advancement if telehealth is to demonstrate its promise and have increased acceptance and implementation. 2,3 Experts have identified standard benchmarks of success as one means to support this, yet no widely implemented evaluation model exists to our knowledge. 3
This article focuses on the process and feasibility of developing a standard evaluation model to help advance the science and acceptance of the field of telemental health (TMH). Although focused on TMH, the principles outlined throughout the article have implications for the broader telehealth field. This article is not intended as another review of telehealth or TMH research and quality comparisons, but rather as a commentary for how organizations and individuals might think about measuring the impact of broad programs as well as individual projects. The goal is to stimulate discussion about paths forward such that the field may better demonstrate its promise as an efficient, beneficial, and integrated mode of healthcare delivery.
The Current Stage of TMH Evaluation
A review of the TMH evaluation literature suggests several thematic gaps in how the field has attempted to measure its impact. Although these gaps are not unexpected given the general trajectory of scientific advancement, they do pose important areas for focus, as the field heads toward more widespread implementation. The section below highlights how each gap has in part contributed to a lack of a standard evaluation model: • Early studies consisted of local projects that were mostly descriptive, had small sample sizes, and were in the early stages of implementation.
4,7,8
TMH started with federal policy and seed funding to initiate projects, similar to many newly developed healthcare services. Consistent with that development, early data came primarily from pilot and feasibility studies meant to demonstrate clinical feasibility, rather than from rigorous scientific research. This resulted in little incentive or opportunity for efficacy or effectiveness research, and the initial data produced were quite basic and limited in broader applicability. • Measurement of success has been varied and was often defined by the individual goals of specific programs. Early TMH projects needed to produce success data that were meaningful for local environments and funding agencies who initiated the projects, often for very specific goals such as expanding care to rural populations. This led to a situation where local projects evaluated themselves using whatever data collection means were available, and they often created single-purpose evaluation methods and measurement instruments. A result of this localized evaluation focus was an emphasis on satisfaction studies and a limited focus on other clinical outcomes.
6
Satisfaction was often put forth as evaluation of program success, but with no consensus on what outcomes should constitute success. The result of this historical development was an inconsistent methodological approach with no theory-driven or uniform set of parameters to investigate.
3
–5
• Evaluations investigating broader impact beyond one effect are limited. Whether to maintain funding or to meet the needs of a specific population, evaluations have typically focused on only one, often isolated outcome. For example, a project to expand TMH to rural populations may focus on utilization and patient satisfaction to determine success at the exclusion of cost factors, technological usability, and broader feasibility measures. Few studies have looked at these coordinated interactions (e.g., access within the context of cost or the integration of technology and workflow issues within the system), and programmatic success has been determined by culling the individual projects focused on one aspect of the service delivery, such as satisfaction.
6
The outcome of this process is an unsophisticated view of how the various potential outcomes impact and may counteract or otherwise interact with each other. The broad implication is difficulty understanding and precisely reporting how and when the various factors drive success. • The field has generally used a narrow range of evaluation methods. Some large programs such as the Veterans Health Administration, the Department of Defense, and several large university-based programs have emerged; however, the bulk of existing telehealth programs generally do not have the resources or long-term support for sustained, state-of-the-art research including randomized controlled trials (RCTs).
4
This has led to a literature base with few longitudinal studies,
8
studies with methodological flaws,
2,6
and few evaluations of cost (with most that do include cost lacking detail or sophistication in evaluating cost economics).
2,6,8
Most of the RCTs that do exist are focused on a single study and not a program, resulting in incomplete evidence of the comparable effectiveness of TMH in treating the full range of various disorders.
6,9,10
This lack of systematic studies has also resulted in limited evidence of other process outcomes relevant to mental health (e.g., therapeutic alliance, patient comfort with technology).
6
Some have thus cautioned that the current evidence base for effective use of TMH is limited given insufficient numbers of clinical outcome studies across populations and disorders.
7,10,11
Yet others suggest that the state of research provides evidence for reliability of assessment via video-teleconferencing relative to face-to-face care
6,11
and supports a strong hypothesis that TMH is equivalent to face to face therapy and a useful alternative in some situations.
10
Although the latter accurately reflects the state of the field and is assumed for this article, the lack of consistency provides support for those who remain skeptical of its efficacy.
Nothing above should be perceived to suggest that TMH is following a unique evolutionary trajectory. In fact, the development of TMH is not unlike that of many other health fields that began with descriptive studies and have eventually moved to more sophisticated evaluation measures including RCTs with multiple outcomes. The increased emphasis on evidence-based practice in healthcare has led many major funding agencies for mental health and drug abuse research to endorse a stage model for gauging scientific progress of interventions. One well-known model, the Stage Model of Behavioral Therapies, demarcates three stages of scientific process that begins with clinical innovations (e.g., pilot and feasibility testing for new treatments), then advances to efficacy research in the form of RCTs, and continues to controlled effectiveness research in the community that may include cost-effectiveness and implementation issues. 12 The course of scientific progress in TMH evaluation, as outlined above, is consistent with this model.
Furthermore, it makes sense that there has been a push to broaden clinical work before the level of hard scientific data that some would like to demonstrate given TMH's clear promise and established successes. 13 TMH is not unique in having practice precede the full range of scientific investigation. 14 In fact, one source estimates it takes an average of 17 years for new knowledge generated by RCTs to be incorporated into practice. 15 Arguably, TMH is progressing along an appropriate scientific path. However, we believe that increased methodological and evaluation sophistication is required to progress TMH to the next phase of development. A standard evaluation model is one means to get there.
Why Now Is a Good Time
An important first question is whether a standard evaluation model is possible for a field as diverse as TMH. Historically mental health has followed two major evaluation models: Intervention research and program evaluation. 4 Intervention research, most notably in the form of RCTs, is generally concerned with efficacy, effectiveness, or safety of a particular intervention, whereas program evaluation is concerned with overall performance based on stated goals. 4 One can see a natural tension between research in the form of RCTs that aim to narrowly test one specific intervention approach versus another and program evaluation that aims to examine various, more diverse, and sometimes less clearly defined outcomes. TMH is concerned with the nature and quality of the intervention, along with broader benefits such as increasing access and cost efficiency. As such it requires examination of both clinical outcomes and program success. RCTs are the gold standard by which the mental health field measures intervention success, but are these the most appropriate ways to measure TMH success given that it is particularly valuable in rural and underserved areas where little or no care currently exists? It should be noted that whether to characterize TMH as an intervention, a mode of individual service delivery, a technological innovation, or an entire system of care is an important question to consider, but outside the scope of this article. Consideration of this issue is relevant as the field moves forward and may prove important to take it to the next phase.
Other issues exist that may make agreeing on one standard evaluation model difficult. The historical development of the field not only offered little guidance as to precisely what should be evaluated but also how to define what was measured. 3 There remains no formal agreement on what set of outcomes, beyond clinical outcomes, ultimately determine success, nor how to define some of the outcomes being used. For example, questions remain in terms of how to measure the economic impacts of TMH in a standard manner that allows policy makers to make informed and rational decisions given that “fixed and variable costs vary greatly among sites, making generalizability of study results difficult.” 16 TMH also raises expectations that the latest technology can produce cost efficiencies within large healthcare systems and through workflow changes. 2,17 Determining the appropriate parameters to investigate the various ways TMH can impact cost and other outcomes is a necessary step towards adopting a standard evaluation model.
An additional step is reviewing what already exists in the literature and conceptualizing a model that uniquely fits TMH. The idea of a standard evaluation model has precedent in the telemedicine literature. In 1995, Rashid Bashshur originally put forth an evaluation model for telemedicine. 18 That model included three main effects (accessibility, cost, quality) and three essential perspectives (client, provider, society). 18 Bashshur 18 argued that any comprehensive telehealth assessment should encompass all of these elements and their interactions. Bashshur et al. 4 have since discussed a three-dimensional model of evaluation adapted from Avedis Donabedian: In this three-dimensional model, the first dimension includes the three applications of clinical, educational, and public health; the second dimension includes the perspectives of clients, providers, and society; and the third dimension represents the various technologies used. Bashshur et al. 4 created a three-dimensional matrix that can be viewed singly or in various combinations.
Another suggested model is the transaction cost economics model for telemedicine evaluation. 19 According to this model there are three levels (individual, community, and society), three major themes (cost, quality, and access), and three activities/levels of analysis (clinical, educational, and administrative). 19 This model is also put forth as a three-dimensional analysis where any telemedicine evaluation examines at least one of the cells, with additional cells involved as research becomes more complex. 19
The above two examples demonstrate similarity of thought as to how a standard model could look, and possibly what one would address.
Perhaps we are at a crossroads given the state of the field in terms of its growth and integration into the healthcare system, and given the evaluation gaps that exist. The field is providing care and delivering on some of its promise, people are satisfied with the care, and the form of care is continuing to expand. Are we perhaps at a natural stage in the field where we have learned enough to better define what exactly the field should promise to deliver so that we don't overpromise and underdeliver? We concur with others who have suggested that a standard evaluation model is one potential solution. 3
Recommendations for a Standard Evaluation Model
The models presented above both offer plausible solutions, yet have not been implemented in TMH to our knowledge. Understanding that we are not the first to suggest a standard model, we put forth in the following section some ideas for what a model might look like and may be implemented: 1. Theory-driven based on the specific goals of TMH. The field needs to establish a more solid theoretical base for individual researchers to use when formulating specific research questions. This might incorporate aspects of healthcare research, information technology, economics, and further delineation of TMH goals. 2. Establish agreed-upon outcomes and incorporate standard measures. The field needs consensus on a range of outcomes appropriate for TMH that encompass broader domains than mere satisfaction. We suggest a two-tiered approach. The first part is determining what accepted outcomes from the mental health field can be used and use those standardized outcomes when at all possible. The second part is agreeing upon which disease-specific/healthcare outcomes are unique to TMH given the nature of the broader benefits it can offer (e.g., reduced travel time and cost, wait time until appointment and follow-up, life satisfaction benefits to family members). Finally, create specialized instruments or adapt ones already developed and used in other areas of healthcare research to measure the outcomes.
2,3
3. Integrate various forms of evaluation research. The field would further benefit from discussion of how best to incorporate program evaluation and intervention studies into a cohesive framework for measuring TMH success. This includes when and whether RCTs are the best method to fully measure quality of intervention. When not appropriate, agreement is required on when to consider additional forms of research and statistical analysis that can better capture the nature of TMH's strengths while maintaining scientific rigor. 4. Generalize for all types of programs. The evaluation model should include multiple components to cover the broad range of different program needs and agendas, but precise enough to allow comparison against accepted benchmarks of success. Elements of this have already been noted in the literature. Although each model is slightly different, there exists a commonality to suggest that any evaluation model should encompass multiple effects, interactions, and ultimately the flexibility to consider different levels of evaluation to meet varied program goals. This is particularly salient given the often small, nonuniform nature of TMH programs and the need for localized adoption. 5. Improved cost analysis. A critical outcome and driver of research in the current era is healthcare cost. TMH strategies have the potential to demonstrate advantages such as reduced overhead by elimination of physical facilities, more appropriate use of healthcare services through better communication, and improved patient access and treatment compliance with consequent greater costly crisis avoidance. Cost analysis needs to encompass nontraditional cost factors affected by TMH, for example, the potential in reduction of morbidity and mortality related to unnecessary travel for patients in environments such as the battlefield or winter driving conditions in rural areas. To overcome reviews that find lack of evidence for cost-effectiveness,
20
more rigorous economic evaluation is needed. An evaluation model should include multiple levels of economic analysis, emphasize return on investment, and be easily adaptable by payors and funders.
Conclusions
It can be argued that the TMH field is at an evolutionary scientific phase requiring an agreed-upon standard evaluation model to drive its continued growth and acceptance. A standard evaluation model could provide several benefits. First and foremost, acceptance of a standard evaluation model is a step towards greater perception of TMH as an established field. A standard evaluation model allows for determination of TMH's impact beyond clinical outcomes by focusing on what makes TMH unique (e.g., access to care, cost). The mental health field has an established way of evaluating whether treatments improve care (i.e., the scientific method, including a stage process of RCTs). Whether TMH improves clinical outcomes is only part of its unique potential contribution; a standard model focuses broadly on the overall contribution it makes as a form of healthcare provision. Second, a standard evaluation model could stimulate the type of research needed to overcome the methodological flaws pointed out by some authors 3,4,6 and help establish a sounder empirical base. A greater empirical base would make it easier to inform legislators and policy makers to make the necessary regulatory, legal, and infrastructure changes needed for further expansion. 5,13 Third, a standard evaluation model could promote more interagency collaboration and multisite research to address the common issue of sample size. 2,3,8 This could specifically benefit research on rural and underserved populations. Additionally, this could allow researchers and programs to better investigate more diverse populations, disorders, and treatment conditions. Fourth, a standard evaluation model could allow agencies to talk to each other and share successes with a common benchmark. Finally, a standard evaluation model could allow a framework for more multidisciplinary research that integrates measuring the impact of the technology and the overall healthcare aspect. This could provide a more comprehensive evidence base from which to examine the overall impact of the field on the various perspectives (e.g., patient, provider, society) that TMH has the potential to impact.
We recommend beginning this process by convening a group of experts to define the common goals that TMH should meet. This should include discussion about the standard elements that an evaluation model should contain in order to best measure those goals. Several large organizations and infrastructures exist that could then implement and pilot the evaluation model, once the individual components are agreed upon. In addition, any federal agency that distributes grants could identify using the evaluation model as a requirement for any grant funding. Although this will take some time, consistent implementation of a standard evaluation model is an important step forward for the field. This article outlines several elements to consider in creating this standard evaluation model. We hope that our ideas will initiate both debate about and momentum towards establishing a standard evaluation model to measure the overall impact and success of TMH.
Footnotes
Disclosure Statement
No competing financial interests exist.