Building the Evidence Base for Tele-Emergency Care: Efforts to Identify a Standardized Set of Outcome Measures

Introduction

The provision of high-quality emergency care in rural communities is an ever-present challenge. ¹ Rural residents are more likely than those in urban communities to be older ² and to have more risk factors associated with acute conditions. ³ In contrast to urban physicians, rural healthcare professionals are more likely to have general medical training, but lack specialized training in emergency care and extensive experience providing emergency services. ⁴ To address these challenges, some rural hospitals have adopted the use of telehealth technologies in hospital emergency departments (tele-ED). Tele-ED care is usually characterized by the use of video communication devices, along with shared radiological and imaging reports. Telemedicine has been effectively used to support care in a variety of circumstances, including trauma care, emergencies, and disaster management. ⁵ The Institute of Medicine (IOM) identified tele-ED care as a growing area that represents promising applications of telehealth technology in rural areas. ⁶

The live exchange between rural clinicians without extensive emergency medicine experience and distant physicians with specialized training has been demonstrated to increase the likelihood of timely diagnosis and treatment, enhance the skill set of local providers, increase compliance with evidence-based clinical protocols, and reduce transfers for patients who, through telehealth, can be successfully treated at the local hospital. ⁷ In addition, telehealth can reduce travel time and enable patients to remain close to family and other social supports, which have been shown to be of value in supporting a more successful recovery. ^8,9 Telehealth also benefits the community by enabling local hospitals to provide a broader scope of services, increase their reputation in the community, and increase their patient volume. ⁷

A main challenge in expanding the use of telehealth—particularly for emergency care—has been the lack of standardized metrics that can be used to demonstrate its value and to compare its outcomes to those that would have been achieved in the same community and healthcare setting using in-person treatment. ¹⁰ Without a common set of metrics, it is difficult to evaluate and quantify the value of this technology or to generalize the findings beyond the small-scale studies that have been conducted for specific populations, settings, and technologies. Several national organizations—such as the Institute of Medicine (IOM) and the National Advisory Commission on Rural Health and Human Services—have highlighted this challenge and recommended enhanced efforts to develop a set of standardized outcome measures that can be used to build a robust evidence base for telehealth. A telehealth workshop summary published by the IOM also emphasized the need for larger and more rigorous design-control studies to increase statistical power and support the standardization of programs and measures to reduce heterogeneity and facilitate meta-analysis. ¹¹

In an effort to address this limitation, in 2014 the Federal Office of Rural Health Policy (FORHP) in the Health Resources and Services Administration of the U.S. Department of Health and Human Services (HHS) established the Evidence-Based Tele-Emergency Network Grant Program (EB-TNGP). In addition to awarding 3-year grants to entities using telemedicine to deliver emergency care in rural hospitals, the program also sought to identify and test a standardized set of performance measures that could be used to establish an evidence base to demonstrate the effect of telehealth technologies on the provision of hospital-based emergency services in rural communities. Specifically, the program was designed to (1) increase the volume of tele-ED encounters for statistical analysis, (2) test common metrics and protocols across multiple settings and populations, and (3) assess the feasibility of asking providers to collect and report standardized measures as part of regular operating procedures.

In July 2014, FORHP awarded cooperative agreements to six provider networks to promote the use of tele-ED care and to leverage their experience to test the feasibility of collecting and reporting outcome data in diverse provider environments and for different target populations. Two months later, FORHP contracted with Mathematica Policy Research and its partner, the RUPRI Center for Rural Health Policy Analysis at the University of Iowa, to identify, evaluate, and pilot test a standardized set of measures among the EB-TNGP grantees that could provide the foundation for a larger scale assessment of the effectiveness of tele-ED in rural communities. This article summarizes the findings from that project. ^*

Methods

The first phase of the project involved a systematic review of the published literature, ¹² search of measure databases, and review of Web sites for organizations involved in telemedicine or emergency medicine to identify existing measures that could be used to assess the impact of tele-ED in rural hospitals. To narrow the list of measures to those best suited to assess the value of tele-ED care, each measure was assessed against a set of predefined criteria. Building upon the criteria used by the National Quality Forum (NQF), a national quality measurement organization, the following criteria were used to guide the evaluation of the inventory of potential measures to identify a core set of measures as follows: (1) more than half meet the NQF evaluation criteria related to reliability and validity, clearly defined measure specifications, importance, feasibility, and utility to stakeholders and are relevant to the study objectives; (2) include measures from as many IOM quality domains (process, outcomes, access, efficiency, experience, and structure) as possible; (3) be applicable to conditions and populations most commonly treated in rural hospital EDs; (4) measure aspects of care that are likely to be affected by the use of tele-ED technology (relative to face-to-face encounters); (5) be applicable to the face-to-face treatment of patients presenting for similar conditions and procedures for comparative purposes; and (6) be as parsimonious as possible to minimize provider reporting burden and maximize response rates. ¹³

To assess the feasibility of the measures recommended as most appropriate for assessing the impact of tele-ED in rural hospitals, the EB-TNGP grantees and their network hospitals participated in a pilot test. The purpose of the pilot test was threefold: (1) to assess the usability of an Excel-based data collection and reporting tool developed to support measure calculation; (2) to evaluate the clarity of the accompanying documentation, including a user's guide and data specifications; and (3) to assess the feasibility and burden of the data collection process, including the source, availability, and burden of collecting the individual data elements required to calculate the measures. Using the reporting tool, grantees were instructed to collect the required data elements for all patients seeking care in the ED over a 2-week period in August 2015. Following the pilot test, grantees participated in a debriefing interview to provide general feedback on the data collection and reporting process as well as any challenges experienced in collecting the data.

The results from each phase of the study (identification, evaluation, and collection and reporting of measures) were presented to an expert stakeholder group specifically convened for this study. The purpose of the stakeholder group was to provide feedback and guidance on the development of a core set of measures that FORHP could use to build a broad evidence base for assessing the impact of tele-ED in diverse settings and for different patient populations. The stakeholder group brought together the perspectives of a wide range of providers, payers, researchers, and advanced users of this technology.

Results

The environmental scan produced a vast inventory of measures relevant to ED care and a narrower subset of measures relevant to telehealth. As shown in Table 1, the largest number of identified measures came from searching the National Quality Measures Clearinghouse, an initiative of the HHS Agency for Healthcare Research and Quality (AHRQ) that supports a Web-based database of specific evidence-based healthcare quality measures and measure sets. The next largest number of defined measures was identified through a search of the Centers for Medicare & Medicaid Services Web site. The third largest number of defined measures was identified from a list of NQF-endorsed measures. The environmental scan identified a much smaller group of measures related to telehealth, used largely by telehealth advocacy or trade organizations. None of these telehealth measures are currently endorsed by NQF nor used in public reporting.

None of the measures identified in the search are applicable to both ED and telehealth care; those related to the ED setting offered little insight into the marginal impact of the type of technology used and those related to telehealth were indifferent to the provider setting. Nor did the environmental scan identify any clearly defined measures specific to tele-ED care. However, the tele-ED research literature referenced several broadly defined measurement concepts (without clear specifications) related to processes of care, staffing, and facility characteristics (including the features of the telehealth technology used) that could be helpful for assessing the effect of tele-ED care on outcomes.

All identified measures were assessed against the evaluation criteria previously described. In the process of applying these criteria, it was discovered that many of the measures lacked sufficient documentation to determine if they met the scientific criteria to demonstrate reliability or validity. As such, reliability and validity were determined based on those measures that had been endorsed by the NQF or were currently publicly reported, thus indicating that they had undergone developmental testing. In addition, the vast majority of measures meeting these criteria focused on clinical care processes and few were specific to telehealth. Due to the paucity of measures related to outcomes and efficiency domains and those specific to telehealth, the research team determined that it was necessary to borrow several measurement concepts from the tele-ED research literature (without rigorous testing) that could be used to compare emergency care delivered using telehealth technology with services delivered in the absence of this technology.

The resulting set of measures, presented in Table 2, encompassed 12 measures applicable to all presenting conditions or services provided in a rural ED. Only two of the 12 all-condition measures were either endorsed by the NQF or currently publicly reported. Two-thirds of the measures could be calculated using information routinely collected by the hospital including claims data and information captured in the electronic health record (EHR). Half of the all-condition measures related to processes of care and/or access, five could be used directly or indirectly to measure efficiency, and two were associated with outcomes. These measures were vetted with the EB-TNGP grantees to confirm their value to the goals of the study and their potential feasibility, given the available data.

Based on guidance from members of the stakeholder group, a supplemental set of measures was also established, capturing measures related to specific conditions or services that are most commonly treated with tele-ED: cardiovascular disease, stroke, and trauma. All of these measures, presented in Table 3, assess clinical care processes. Although none of the supplemental measures are specific to telehealth, over three-quarters capture concepts that could be affected by the use of tele-ED services. Unlike the all-condition measures, all of the condition-specific measures are related to processes of care, more than half were either endorsed by the NQF or currently in use for public reporting programs, and over three-quarters could be calculated using information from the EHR.

Three of the six EB-TNGP grantees and 14 of their participating rural hospitals participated in a pilot test to assess the feasibility of collecting and reporting the 45 data elements required to calculate these measures. Using an Excel-based data collection and reporting tool, grantees collected the required data for all patients seeking care in the ED, regardless of their presenting illness or condition, type of care received, or outcome of the visit. A total of 3,674 visits were recorded over the 2-week reporting period. The vast majority of data elements were successfully recorded. In addition, most of the data elements were readily available from electronic sources and hospitals already collected many of them as part of routine reporting procedures.

Discussion

In the process of identifying a core set of measures to assess the value of tele-ED services in diverse rural settings, a number of challenges were encountered. First, the environmental scan was unable to identify existing measures specific to the use of telehealth in supporting emergency care. Instead, existing measures were either specific to emergency care but insensitive to telehealth or were sensitive to telehealth but not specific to emergency care. In addition, among measures relevant to telehealth, few had a well-established evidence base and none had gone through the rigorous testing process needed to establish their statistical validity or reliability. Moreover, in almost all cases, measures aimed at telehealth lacked clearly defined specifications, prohibiting the ability to conduct formal measurement testing.

Second, to build a business case for tele-ED, there is a need for measures that can demonstrate the impact of telehealth on clinical outcomes and cost of care. However, the vast majority of the measures identified through the environmental scan, particularly those with a scientific evidence base, focused on assessing processes of care. Few measures assessed other performance domains such as outcomes, efficiency, and patient and provider experience. Identifying measures in these domains poses unique challenges with regard to the lack of available data and data collection burden. Nonetheless, discussions with the grantees and members of the stakeholder group indicated that these measures are critical in establishing the value of tele-ED relative to face-to-face encounters.

An analysis of the pilot test records and qualitative feedback from pilot test participants provided several important lessons about the feasibility and burden of asking hospitals to collect and report the data elements needed to calculate the recommended measures. The volume of ED visits where telemedicine is used was extremely small, making it difficult to assess the relative performance of tele-ED compared with non-tele-ED services. Data collection will therefore need to occur over an extended period to capture a sufficient volume of patients using tele-ED to calculate the measures. During this period, however, the total number of ED visits, in which telehealth is not used, is likely to be large. A sampling strategy used for face-to-face encounters will help reduce the reporting burden while still producing a sufficient sample size of non-tele-ED visits to generate an effective comparison.

Although the vast majority of the data elements required to calculate the tele-ED measures are routinely collected by rural hospitals, a small subset of the data elements is not universally collected. Several hospitals did not document data elements specifically related to the tele-ED consultation, including consultation duration, averted transfers, and distance and mode of transportation that would have been used if the transfers had not been averted. Some small rural hospitals indicated that they do not routinely collect the Emergency Severity Index (ESI), a five-level assessment designed to assess the expected severity of an ED patient's condition, the urgency to treatment, and the level of resources needed to treat the person. When this information was recorded, it was apparent that patients who received tele-ED were more likely to have higher ESI scores than those treated through face-to-face encounters with clinicians. Without ESI scores to assess and control for patients' severity in the ED, it will be difficult to compare the relative benefits of tele-ED services. Providers using telemedicine in the ED are often strong advocates for the use of this technology and, therefore, may be willing to collect and report ESI scores (or comparable measures of severity) if they understand that the information will help establish a business case for the value of, and potentially impact reimbursement for, this technology.

Finally, pilot test participants viewed the required data elements as both valid (i.e., they thought the data elements accurately captured what they are supposed to measure) and reliable (i.e., they thought the data elements will be consistently defined across hospitals). However, grantees cautioned that several elements (including reason for transfer, averted transfer, time to tomography/magnetic resonance imaging procedure and results, ESI, and primary payer) might not be consistently reported across providers because they rely on subjective determination, are inconsistently collected across hospitals, or lack standard measurement specifications. A quantitative analysis to assess the statistical reliability and validity of these data elements may be needed to strengthen the scientific credibility of these measures.

Conclusion

There is considerable interest in whether tele-ED provides comparable care to face-to-face ED care in real-life situations. In 2013, the American Stroke Association (ASA) and American Heart Association released a statement advocating that because of the limited distribution and availability of neurological, neurosurgical, and radiological expertise, the use of telemedicine and telestroke resources and systems should be supported by healthcare institutions, governments, and payers to ensure adequate coverage and care of stroke patients in a variety of settings. ¹⁴ In addition, The ASA and the American Telemedicine Association are currently developing practice guidelines and quality metrics for the use of telemedicine for the treatment of acute stroke. ^15,16

The 12 core measures and 11 supplemental (condition-specific) measures identified through this project have the potential to demonstrate the value of telemedicine for a variety of conditions treated in rural hospital EDs. Feedback from participating grantees and their partner hospitals indicated that most of these data elements can be extracted from an EHR using the developed Excel-based instrument without imposing undue burden on rural providers. Through the widespread use of a common set of feasible and reliable metrics—in diverse settings and for different patient populations—it may be possible to build a business case for the use of telemedicine in emergency care. The measure set identified in this study is a necessary first step in moving toward this longer term goal. The next challenge is to make use of this measure set in well-designed research studies that use appropriate comparison groups to permit the examination of comparative effectiveness of tele-ED.