Utilization and Patient-Reported Outcomes of Direct-to-Consumer Telemedicine During the First 6 Weeks of the COVID-19 Pandemic in the Largest Pediatric Ambulatory Network in New York State

Introduction

At the onset of the novel coronavirus disease (COVID-19) pandemic in March 2020, ambulatory health systems around the world implemented direct-to-consumer telemedicine programs to provide continued access to care without risking COVID exposure. ¹ Prior limitations to telemedicine due to variable insurance coverage and reimbursement rates were largely eliminated, as restrictions were lifted by federal policy, and telemedicine expanded at unprecedented speed and scale across adult ^{2 –4} and pediatric general and subspecialty practices. ^{5 –9}

In the years since the onset of the pandemic, it has become clear that large-scale telemedicine programs can be implemented across a wide variety of circumstances and successfully deliver many forms of health care remotely. ¹⁰ However, details of what specific visit outcomes can be achieved in telemedicine visits, such as laboratory orders, medication prescriptions, or subspecialty referrals, have received little attention. Most pediatric care also requires a parent or caregiver as proxy-reporters, which creates an additional layer of complexity.

There is a growing body of evidence that telemedicine is usable and acceptable, sometimes even preferred, in pediatric care, from the patients' perspective. This has now been well described in numerous subspecialties, including rheumatology, ¹¹ orthopedics, ¹² endocrinology, ⁵ cardiology, ¹³ hematology, ¹⁴ and urology. ¹⁵ However, there have been few reports of pediatric primary care patient perspectives on telemedicine. Studies from the perspective of any one specific specialty cannot be generalized across all specialties. Furthermore, few studies have been large enough to make comparisons across demographic groups of patients. Understanding the nuances of usability and acceptability from the patient perspective across the breadth of the pediatric experience during this unique time period is important because crisis prompted changes in complex systems are largely driven by the system's ability to meet the needs of the given context, rather than patient preference. In addition, technological innovations in health care have a long history of exacerbating existing disparities, ¹⁶ and patient-reported outcomes are a critical measure of this impact.

Therefore, we aimed to (1) describe direct-to-consumer telemedicine utilization, visit outcomes, and caregiver-reported usability in pediatric general and subspecialty ambulatory care practices in the largest integrated health network in New York State during the first 6 weeks of the COVID-19 pandemic; and (2) investigate differences in caregiver-reported usability of telemedicine by individual- and visit-level characteristics.

Methods

We conducted a retrospective cohort study of all ambulatory pediatric telemedicine encounters occurring between March 10, 2020, and April 18, 2020, in a single large, diverse academic health system serving New York City and Long Island. The ambulatory pediatric network includes 21 primary care sites and 30 subspecialty care sites, with an average annual volume of 467,474 visits. Subspecialties include adolescent medicine, allergy/immunology, cardiology, developmental-behavioral, endocrinology, gastroenterology, hematology/oncology, infectious disease, medical genetics, neonatal follow-up, nephrology, neurology, orthopedic surgery, pediatric surgery (general), pulmonology, rheumatology, and urology.

We defined telemedicine as health care delivered in real time using two-way audiovisual technology. Telemedicine encounters were conducted by a primary care physician, subspecialty physician, or advanced care provider (i.e., physician assistant or nurse practitioner). We did not include visits scheduled with a nutritionist, psychologist, social worker, or nurse only. Professional medical interpreter services were available for all telemedicine visits.

We pulled patient and encounter information from our electronic medical record scheduling system and manually reviewed clinical documentation and demographic information from each encounter using a predefined chart review tool.

Sociodemographic data included age, sex, preferred language, race, ethnicity, and insurance type. These data are reported by the patient or parent at the time of initial registration within our health system (and updated as needed). Race categories in this system include American Indian/Alaska Native, Asian, Black/African American, Native Hawaiian/Pacific Islander, White, other, or unknown. Ethnicity categories in this system include non-Hispanic/Latino, Hispanic/Latino, or unknown. Due to the large number of records with unknown race but reported ethnicity and vice versa, we collapsed race and ethnicity into a single race/ethnicity variable according to federal data standards. ¹⁷ Preferred language was categorized as English, Spanish, other, or unknown. Type of insurance was categorized as private, public, self-pay, or other/unknown.

We examined encounter-level characteristics for each visit including specialty, visit type (well/preventative visit, newborn weight check/follow-up, acute visit, or chronic disease management), visit contents (orders placed, such as medications, immunizations, laboratory tests, radiology, procedures, supplies, or referrals), suggested follow-up modality, if any, and any provider-documented technological difficulties experienced during visits. We also examined medical complexity by constructing three variables of health care service utilization in the preceding 12 months: number of emergency department (ED) visits (none vs. any), number of inpatient admissions (none vs. any), and number of subspecialists seen (0–2 vs. 3 or more).

All telemedicine visits with an associated caregiver email address were then included in the survey portion of the study, with the exception of visits in the division of adolescent medicine (due to concerns regarding confidentiality). Caregivers were invited to complete a 23-item English-language electronic survey consisting of an adapted Telehealth Usability Questionnaire (TUQ) and several additional items to assess technical details of the visit (device used, application used, application preferences). The TUQ measures telehealth usability across the following six domains: usefulness, ease of use, interface quality, interaction quality, reliability, and satisfaction. We calculated overall and domain-specific usability scores by averaging responses from the corresponding items and dichotomizing at 5, such that high usability in a given domain indicates selecting “strongly agree/very good” for every item in that domain. ⁴

We used descriptive statistics to characterize sociodemographic data, clinical characteristics, visit characteristics, and survey responses. We used multiple logistic regression to explore visit-level (visit date, visit type, any order placed, visit division) and patient-level (race/ethnicity, language, insurance, age, complexity) predictors of high usability of telemedicine. For analyses of survey data, given the smaller sample size, we further collapsed visit-level data as follows: date of visit was defined as early (3/9–3/29) versus late (3/30–4/18); visit type was categorized as chronic disease management versus all other; pediatric division was categorized as general versus subspecialty; orders and referrals were categorized as any versus none. We collapsed patient-level data as follows: race/ethnicity was categorized as non-Hispanic White, non-Hispanic Asian, non-Hispanic Black, Hispanic, and non-Hispanic other (unknown excluded); preferred language was categorized as English versus not English; insurance was categorized as private, public, or other, and medical complexity was categorized as complex if the patient met at least one of the three measures of medical complexity versus none.

All study procedures were deemed as exempt from institutional review board (IRB) approval by the Northwell Health IRB.

Results

During the 6-week study period from March 8, 2020, to April 18, 2020, there were a total of 3,197 ambulatory pediatric telemedicine visits for 2,967 unique patients residing in New York City and Nassau and Suffolk Counties in New York. Table 1 shows the descriptive statistics of the overall and survey respondent samples.

Telemedicine utilization expanded rapidly, with three telemedicine visits occurring in week 1 of implementation and more than 1,200 visits in week 6 of implementation. Visits were primarily for chronic disease management (53.8%) or acute concerns (26.7%), with a small proportion of visits for well-child care (1.1%), newborn follow-ups (3.1%), establishing new care (9.1%), or other (6.2%). Orders were placed or referrals made in 1,693 (52.9%) visits. Fewer than 1% of visits included a recommendation for same-day, in-person care (0.6%), and only 3.8% of visits documented any technology challenges. Approximately one third of telemedicine visits (30.7%) occurred in general pediatrics, 62.5% were in a medical subspecialty, and 6.9% were in a surgical subspecialty. Patients seen were from all age groups, majority male (53.3%), covered by private insurance (55.4%), and English-speaking (89.2%).

The most common racial/ethnic category was non-Hispanic White (42.5%), followed by non-Hispanic Black or African American (12.6%), Hispanic (10.4%), non-Hispanic Other (10.4%), non-Hispanic Asian (8.7%), non-Hispanic American Indian/Alaskan Native (0.4%), and non-Hispanic Native Hawaiian or Pacific Islander (0.4%). Of the three measures of medical complexity based on health care service utilization in the past 12 months, 28% had at least one ED visit, 17.5% had at least one inpatient admission, and 16.7% had at least three subspecialists seen.

Of the 2,640 caregivers with valid email addresses who received the survey invitation, 441 (17%) responded (Fig. 1). Compared with nonrespondents, survey respondents were more likely to have had a visit for chronic disease management (64.0% respondents, 52.2% nonrespondents; p < 0.01), more likely to have had a telemedicine visit with a medical subspecialist (70.3% respondents, 61.2% nonrespondents; p < 0.01), and less likely to have a child with an ED visit in the last year (23.3% respondents, 29.3% nonrespondents; p = 0.01).

OPEN IN VIEWER

Fig. 1.

Study flow diagram.

Every item of the TUQ had agreement or strong agreement from the majority of caregivers (Fig. 2). Twenty-two percent of respondents selected the most positive response option (strongly agree or very good) for every single TUQ item (overall high usability). Rates of strong agreement for individual TUQ items ranged from 36.5% for “Telehealth is an acceptable way to receive healthcare services” to 60.6% for “Telehealth saved me time traveling to a hospital or specialist clinic.” High usability (strongly agree or very good) across the entire TUQ domain ranged from 31% (Satisfaction and Future Use) to 46.5% (Reliability).

OPEN IN VIEWER

Fig. 2.

Telehealth Usability Questionnaire items and frequency distribution among 441 caregivers of ambulatory pediatric patients receiving care via telemedicine. Frequency distribution of caregiver responses to the Telehealth Usability Questionnaire. The questionnaire consists of six subscales: usefulness, ease of use, interface quality, interaction quality, reliability, and satisfaction.

In multivariable logistic regression, patient race/ethnicity and age were the only significant predictor of high usability. Caregivers of Asian, non-Hispanic patients were less likely to report overall high usability (adjusted odds ratio [aOR] 0.32 [95% confidence interval {CI} 0.10–0.99]), ease of use (aOR 0.39 [95% CI 0.18–0.93]), interface quality (aOR 0.39 [95% CI 0.17–0.93]), and satisfaction and future use (aOR 0.31 [95% CI 0.11–0.87]) compared with caregivers of non-Hispanic White patients. Caregivers of non-Hispanic Black patients were more likely to report high satisfaction and future use than caregivers of non-Hispanic White patients (aOR 1.95 [95% CI 1.05–3.63]). Age was a significant predictor of telehealth usability in three domains. Compared with caregivers of infants younger than 1 year, caregivers of 18- to 21-year-old patients reported lower interaction quality (aOR 0.23 [95% 0.06–0.92]) and lower satisfaction and future use (aOR 0.22 [95% CI 0.05–0.87]); and caregivers of 6- to 11-year-olds reported lower reliability (aOR 0.37 [95% CI 0.14–0.99]). We identified no significant predictors of telehealth usefulness (Table 2).

Discussion

This observational study describes successful implementation of direct-to-consumer telemedicine at the onset of the COVID-19 pandemic in the largest pediatric ambulatory care network in New York State, the first epicenter of the pandemic in the United States. This included acute and chronic care visits across general pediatrics and 17 pediatric subspecialties, in which all the standard tasks of pediatric medical care were accomplished, apart from administration of vaccines. In this context, we also describe and quantify telemedicine usability from the caregiver perspective. This is the largest sample, to the best of our knowledge, of patient-reported telemedicine outcomes in the pediatric literature to date.

Race/ethnicity and age were the only significant predictors of telehealth usability, across a wide range of patient- and visit-level variables. Caregivers of 18- to 21-year-old patients reported significantly lower interaction quality and satisfaction and future use compared with caregivers of infants younger than 1 year; caregivers of non-Hispanic Asian patients reported lower ease of use, interface quality, interaction quality, satisfaction, and future use compared with White; and caregivers of non-Hispanic Black patients reported significantly higher satisfaction and future use compared with White. These results provide insight into the capability of large healthcare systems to implement new models of care when prompted by crisis and the patient/family experience of receiving such care in that context.

The COVID-19 pandemic ushered in an era of challenges that warranted rapid and widespread implementation of telemedicine to provide ongoing care, as described in our study. The United States Department of Health and Human Services issued guidance allowing for broad access to telehealth services, without the geographic or location restrictions that previously applied. ¹⁸ In addition, New York State prohibited insurance companies from requiring payment for telehealth services or from denying services that would otherwise be covered due to their being provided via telehealth. ¹⁹ These structural changes allowed rapid and widespread implementation of telemedicine to be feasible in our center, as it did across the country. A number of these structural changes have now been rolled back, or will be by the end of 2024, ¹⁸ and additional studies will need to evaluate implementation and health outcomes after the tightening of these restrictions. At the same time, many traditional health care institutions are focusing on digital transformation at a system level to deliver care at the right place and time for each individual patient, creating more opportunities for the expansion of telemedicine services. ²⁰

Nationwide, telemedicine utilization has decreased from that seen in the Spring 2020 but still remains higher than prepandemic levels, and the practice of telemedicine will likely continue. ¹⁰ Our findings build upon much of the literature describing pediatric telemedicine utilization both before and after the onset of the COVID-19 pandemic, in which telemedicine has a role in care of both acute and chronic conditions for a wide breadth of pediatric conditions across primary and subspecialty care. ^10,21 Our data suggest that telemedicine was an appropriate platform for care delivery in most cases, with less than 0.2% of visits including a recommendation that the patient be seen for immediate in-person evaluation.

The largely positive experience with telemedicine in our sample is similar to that described in other studies of the telemedicine experience during COVID-19. ^{14,22 –24} Our results build on these qualitative and subspecialty-specific studies by quantifying the patient experience across general and subspecialty pediatric clinical care, with no identifiable differences in patient experience noted between the two. In addition, our response rate of 17% is even higher than the 5–14% response rates reported in similar studies, ^14,23,24 further strengthening the findings.

Race-based differences in patient perspectives of traditional (in-person) pediatric ambulatory care have been well documented. One large study of pediatric primary care showed that Asians rated their experience as significantly worse than non-Hispanic Whites, even after adjustment for language, access, and other sociodemographic variables. ²⁵ Findings from a large qualitative study of pediatric caregivers highlight experiences of judgment and disrespect that undermine trust in care, particularly among publicly insured Asian families. ²⁶ Our findings reveal similar trends in patient-reported experiences of telemedicine at the onset of the pandemic, with caregivers of Asian patients reporting significantly lower scores across most usability domains. Surveys conducted more recently in adult patient populations show consistent findings. ²⁷ Thus, our findings of lower telehealth usability among Asian families compared with White families may reflect negative experiences with ambulatory medical care in general, in addition to negative experiences with telehealth specifically.

In contrast, caregivers of non-Hispanic Black patients in our sample reported significantly higher satisfaction with telemedicine than those of White patients. It has been repeatedly documented that Black Americans have had disproportionately less access to telemedicine visits that include a video component (rather than audio only) compared with non-Hispanic White patients. ^6,10 Our study only included video visits. Thus, our finding may reflect a relative improvement from baseline experience or expectations, in which access is limited, rather than an absolute difference in telemedicine experience between Black and White patients. Additional studies that examine both access and experience are necessary to further inform programs and policies seeking to advance equity.

Our findings should be interpreted in the context of several limitations. Apart from the patient-reported outcomes, our data came from clinical documentation and administrative records, which were sometimes incomplete and contained varying levels of detail. Our survey was in English only; therefore, our null findings related to preferred language and telehealth usability are based on data from families who would be least likely to experience language-related telehealth barriers. We used age and race/ethnicity of the patient to identify sociodemographic differences in telemedicine usability but did not obtain data on age and race/ethnicity from the responding caregiver, which may significantly impact responses. Our sample size of survey respondents was large enough to detect race-based differences, but a much larger sample would likely be needed to examine differences of intersectional identities beyond what can be accomplished by controlling for these variables statistically.

Conclusions

Telemedicine was successfully implemented across a wide variety of specialties, visit types, and patient demographics in the largest pediatric ambulatory care network in New York State at the onset of the COVID-19 pandemic. Telehealth usability was significantly lower for caregivers of Asian patients compared with White patients, across multiple domains, and for caregivers of older patients compared with the youngest infants. Further research is necessary to identify modifiable drivers of the patient and family experience, particularly in non-Hispanic Asian communities.