Documented Reasons of Cancellation and Rescheduling of Telehealth Appointments During the Pandemic

Abstract

Background and Objective:

The COVID-19 pandemic increased the use of telehealth around the world. The aim is to minimize health care service disruption as well as reducing COVID-19 exposure. However, one of the major operational concerns is cancellations and rescheduling (C/Rs). C/Rs may create additional burden and cost to the patient, provider, and the health system. Our aim is to understand the reasons for C/Rs of the telehealth session after the scheduled start time.

Materials and Methods:

We reviewed electronic health records (EHRs) to identify the C/R reasons for behavioral health and speech language pathology departments. Documented C/Rs in the medical charts were identified from EHR by using a keyword-based and Natural Language Processing (NLP)-supported EHR search engine. From the search results, we randomly selected 200 notes and conducted a thematic analysis.

Results:

We identified four themes explaining C/R reasons. Most frequent theme was “technicality” (47, 36%), followed by “engagement” (34, 25%), “scheduling” (31, 24%), and “unspecified” (20, 15%). The findings showed that technical reasons are the leading cause of C/Rs, constituting 36% of the cases (95% confidence interval [CI]: 29–43%). Notably, “engagement” constituted a sizeable 25% (95% CI: 19–31%) of C/Rs, as a result of the inability to engage a patient to complete the telehealth session.

Conclusions:

The study shows that engagement is one of the new challenges to the pediatric telehealth visits. Future studies of new engagement models are needed for the success of telehealth. Our findings will help fill the literature gaps and may help with enhancing the digital experience for both caregivers and providers, reducing wasted time and resources due to preventable C/Rs, improving clinical operation efficiency, and treatment adherence.

Introduction

The COVID-19 pandemic accelerated the implementation of telehealth across the United States and around the world. The rapid rollout of telehealth was key to minimize health care service disruption while reducing COVID-19 exposure at health care facilities.^1
–3 Clinical service lines that do not heavily rely on in-person examinations but more on cognitive therapy practice as well as treatments requiring caregiver-as-a-coach, such as behavioral health and speech language pathology, had experienced much higher conversion rate to the new modality of telehealth.

As we recognize the tremendous benefits of telehealth, there have also been concerns on the challenges this new modality of health care delivery might have imposed on patient experience and equity, such as technology access, digital literacy, and attitude shift toward telehealth.^4,5 Current literature suggests that technology access might be one of the major barriers for the adoption of telehealth^5,6 as well as medical issues and personal preferences.⁷ However, there is a gap in the literature about identification of barriers in telehealth for the pediatric patients during the pandemic, especially for the service lines with higher telehealth demand and volume.

One of the major outcomes of telehealth barriers is cancellations and rescheduling (C/Rs), which creates additional burden and cost to patient, provider, and the health system.⁷ In this article, our aim is to understand the reasons of C/R of the telehealth session after the scheduled start time. This type of “last-minute” C/Rs could be the most impactful for multiple reasons. First, the time slot could no longer be reallocated for another visit leading to lost revenue. For example, the annual financial impact to a clinical therapy department of 50 providers can reach $2.0 million, assuming a reimbursable rate of $150 per hour and a C/R rate of 14%, based on the numbers of a Midwest pediatric hospital. Second, patients and/or providers might be frustrated and stressed for not being able to complete a visit when they had already committed time for it in addition to future scheduling and time allocation burden, leading to poor satisfaction with clinical experience. Therefore, understanding the reasons of C/Rs could potentially help to improve patient experience, engagement, and health system readiness, as well as to reduce health care service access inequity.

To timely understand the reasons behind last-minute telehealth C/Rs in a pediatric setting during a time frame in COVID-19 pandemic for two highly telehealth adaptable services, behavioral health, and speech language pathology, we conducted chart reviews of clinical notes and analyzed using thematic analysis to identify the C/R reasons. The study period was selected from the beginning of April to the end of May 2020, which was the period that the hospital had high volume of telehealth visits (∼44,500 telehealth visits per month at the hospital, and 40,000 telehealth visits per month at behavioral health and speech language pathology departments) and anticipated to be the time period to have more C/R encounters. We hope this preliminary study can guide quality improvement initiatives and the design of future studies.

Materials and Methods

In this study, we reviewed the medical records to develop an understanding about the C/R reasons. Unlike common data collection methods (surveys and interviews) to obtain a patient's experience and perspective, we focused on the available information to retrospectively analyze and synthesize the data that were documented in the clinical system by providers. Free-text and open narrative style notes and documentations provided rich contexts around the C/Rs. Regarding the data access and collection period, use of medical records enabled timely mining of existing data and feedback to the operational team during the pandemic. This study has been assessed by the institutional review board and exempt from ethical board approval.

Data Collection

Documented C/Rs in the medical charts were identified from medical records by using a keyword-based and Natural Language Processing (NLP)-supported EHR search engine.⁸ We initiated the search with the keywords of “cancellation” and “rescheduling” separately. Then, we formed an expanded query string including the alternative forms of the keywords (e.g., abbreviations, misspellings, and synonyms) suggested by the search engine to conduct a comprehensive search for C/Rs (Table 1). The keywords were combined with a videoconferencing software, Zoom, to capture only telehealth visits.

Table 1.

Keywords Used to Identify Cancellations and Rescheduling in Clinical Notes

CANCELLATION-RELATED WORDS	RESCHEDULING-RELATED WORDS
cancel	reschedule
cancelling	rescheduled
canceled rescheduled	resheduled
cancelled rescheduled	re-scheduling
cancel re-schedule	reschedule-
canceling	re-schedule
cancel/reschedule	reshedule
cancel	re-scheduled
cancelling rescheduling	r/s
cancelling	reschdule
cxl	reshcedule
canceled	rescheduel
	rescedule
	rescheduling
	rechedule
	reschedules
	rescheudle
	reschd
	rebook

The inclusion criteria were (1) the telehealth visit is scheduled between April 1 and May 29, 2020, (2) the telehealth visit is reported in progress notes, (3) the specialty of the telehealth visit is behavioral health or speech language pathology, and (4) the patient notes have C/Rs documentation associated with the visit (excluding the notes mentioning prior or future appointment C/Rs). The selected time frame reflects the early peak demand on telehealth at the hospital. Therefore, the time frame was selected to capture more encounters. We limited our search to progress notes, which is the category for almost all the telehealth visit notes, to efficiently identify the notes containing telehealth C/Rs. In addition, we focused on behavioral health and speech language pathology departments, which had the highest volume of telehealth visits. To limit repetitive review of the same patients, we captured unique patient notes that report the first note having C/Rs from each patient. In line with our study aim, we also limited our analysis to the notes that captured C/R of the visits directly associated with the note, excluding notes reporting prior or later events.

Analysis

We conducted a thematic analysis⁹ using predefined codes to guide the analysis.¹⁰ A predefined codebook was generated based on prior literature and operational experience from our clinical team to code and categorize the reasons for C/R the telehealth visits.¹¹ The first 30 notes were reviewed and coded by two researchers independently and their scores were compared. The inter-rater reliability score was “almost perfect” among raters (Cohen's k = 0.859, p < 0.001, 95% confidence interval [CI] [0.77–0.95]).¹² The disagreements were minor and resolved by reaching the consensus. The rest of the coding was completed by one of the researchers. During the coding process, additional codes were identified. All authors discussed and reached consensus to include, exclude, or merge the codes based on their level of relevance to this study and frequency of occurrences. Eventually, codes were reviewed and grouped under potential themes, and the thematic map was created. Coding and analysis of the data were completed using Microsoft Office 365 software. Finally, we calculated the 95% CI for the proportion of each theme category, that is, the range of values within which we are 95% certain that the true proportion lies.

Results

For the study time frame, 9,546 unique patients were identified to having telehealth visits for behavioral health and speech language pathology. From these 9,546 unique patients, 607 clinical notes met our keyword criteria. Out of 607, we selected 200 clinical notes by random sampling. Microsoft Excel software was used to list and randomized clinical notes (using RAND() formula). Then, we manually reviewed and removed the notes with false-positive C/Rs.

After notes with false-positive identification of C/Rs were removed, there were 122 clinical notes containing documented C/Rs, representing 9,546 unique patients of total population. False positives were the notes mentioning future or past C/Rs or have the query keyword as part of a template. Figure 1 shows the demographic characteristics of age group, gender, race, and ethnicity of this sampled C/Rs cohort and the telehealth visit population (unique patients), including 95% CIs of the various proportions. Most of the patients were male (57%), school age child (40%) or adolescent (39%), White (45%), and non-Hispanic or Latino (90%). In total, 7,356 (77%) of the total cohort and 93 (76%) of the C/Rs cohort had telehealth visits for behavioral health, whereas 2,430 (25%) of the total cohort and 29 (24%) of the C/Rs cohort had telehealth visits for speech language pathology. The C/R demographic proportions were not found significantly different from the total telehealth population within 95% CIs, except for “White” and “Black or African American” races (highlighted with red box in Fig. 1).

Fig. 1.

Demographic distribution for C/R cohort and total telehealth visit population (by unique patients) with 95% confidence interval proportions (right side figure). C/R, cancellations and rescheduling.

The themes and subthemes for the reasons of C/Rs are summarized in Table 2. The most frequent theme was “technicality” (47, 36%), followed by “engagement” (34, 25%), “scheduling” (31, 24%), and “unspecified” (20, 15%).

Table 2.

Reasons of Cancellations and Rescheduling

THEMES	n (%)	95% CI (LOWER-BOUND TO UPPER-BOUND)	SUBTHEMES	CODE FREQUENCY, n (%)
Technicality	47 (36)	29–43%	Technical difficulties	35 (25.4)
			Digital literacy	7 (5.1)
			Spatial availability	4 (2.9)
			Digital availability	3 (2.2)
			Finding internet access	2 (1.4)
			Clinician's technical difficulty	2 (1.4)
Engagement	34 (25)	19–31%	Patient's state during the visit	29 (21)
			Need for translation services	2 (1.4)
			Personal preference	2 (1.4)
			Concerns about privacy	1 (0.7)
Scheduling	31 (24)	18–30%	Conflicts	25 (18.1)
			Natural or environmental events	4 (2.9)
			COVID-19–related impacts	2 (1.4)
Unspecified	20 (15)	7–20%	No reason specified	20 (14.5)
Total				138 (100)

N = 122 patients.

CI, confidence interval.

Technicality theme represents the patient/caregivers who were having problems with accessing the service due to technical problems. The problems include issues with logging into account (technical difficulty), lack of knowledge about technology (digital literacy), place to maintain the visit (spatial availability), unable to find a digital device available to connect (digital availability), and issues with no or poor internet connection (finding internet access). In few cases, clinicians also had technical issues with connection. Technical difficulties were highly occurring subtheme for technicality (35, 25.4%).

“Parent had difficulty connecting to Zoom; on the call for 15 minute and stated having internet connection difficulties and wanted to call later to re-schedule today's evaluation.” [Patient (pt).14. Behavioral health]

“Clinician has been in contact with dad in order to attempt to help him with technical difficulties so that client can have a telehealth session. Clinician attempted to walk through the steps with dad as well as call [patient portal] support. Dad was still unable to figure out how to ZOOM.” [pt.119. Behavioral health]

“After waiting for the client to enter the Zoom appointment, called caregiver. Began completing the BH-ROS for about 10 minutes when the client's phone died. The client will need to re-schedule to complete the Diagnostic Assessment.” [pt.134. Behavioral health]

Engagement theme represents patients who were not in a state to engage during telehealth visit due to sleeping, having personal issues, or physically and emotionally not available. In addition, some patient/caregivers needed translation services to communicate that was not available at the time of visit. A few patient/caregivers indicated having concerns or problems with telehealth as well as preference for in-person visits. Patients' state to engage was a highly occurring subtheme for engagement (29, 21%).

“Client was accompanied by her mother for ST telehealth session via Zoom. Client was asleep at onset of session. Mom reported client had a rough night and did not sleep much. Client remained asleep despite mom's attempts at waking her up before the session. SLP and mom agreed to cancel today's session” [pt.28. Speech pathology]

“During beginning of Zoom call, client stated she did not want to have session. Clinician attempted to talk with client about why she did not want to have session and attempted to de-escalate client.” [pt.2. Speech pathology]

“MOC explained that client was using the tablet and was able to attend session at [time]. When rejoining the zoom link, the MOC explained that it had been a rough night of client not sleeping and asked to reschedule.” [pt.7. Behavioral health]

Scheduling theme represents patient/caregivers who had other scheduled or unscheduled events, or their work was ongoing or forgot the appointment (conflicts). In addition, some of the sessions were cancelled due to natural events such as tornado. Few notes were identified that are explicitly associated with COVID-19–related impact on C/Rs. Conflicts were highly occurring subtheme for scheduling (25, 18.1%).

“Session did not take place as [patient] reported she is at work. Scheduled next session for…” [pt.8. Behavioral health]

“FOC still logged into video chat at the start of the appointment. He requested to reschedule telling the writer that they had something going on, but it was not related to the client.” [pt.13. Behavioral health]

“Patient checked in 16 minutes late as mother forgot his appointment. Parent and SLP rescheduled appointment” [pt.81. Speech pathology]

“A tornado warning was issued for both the client and clinician after client checked into appointment. Once interpreter services connected to the appointment at approximately [time], the clinician ensured the family's safety and rescheduled the appointment.” [pt.128. Behavioral health]

Unspecified theme represents the patients who cancelled or rescheduled the session, but the reason was not recorded in the clinical notes.

“Family missed zoom session. Clinician contacted caregiver via the phone and left VM discussing no-show and requesting a phone call back to re-schedule appt “[pt.10. Behavioral health]

Discussion

Principal Findings

The findings showed that technical reasons are the leading cause of C/Rs, constituting 35% of the cases (95% CI: 29–43%). Observed technical reasons, such as log-in problems as part of technical difficulties, internet access issues, digital availability, and digital literacy were also observed in other telehealth responses in the literature.^13,14 Some of the technical issues could potentially be mitigated if a test session was a default step before the appointment. At the time of this study, our EHR system did not accommodate telehealth test session scheduling due to rapid roll out and time limitations given the sudden increased demand. Instead, clinical staff schedules a call ahead of time to prepare families for the process of telehealth visits before their first visit. This informal instructional session might have helped reducing potential C/Rs due to technical reasons. Future implementation of a robotic test session to confirm the quality of the video connection can be a scalable way to provide the assistance as well as reducing the burden to clinical staff.

Another theme for C/Rs is scheduling, which made up 24% (95% CI: 18–30%] of the reasons. Conflicts with the appointment were frequently mentioned, including other scheduled events, unscheduled events, and forgetting the appointment. COVID-19–related reasons and natural events were also reasons for scheduling issues but occurring less. Using technology to send out reminders to battle the potential scheduling conflicts and forgetfulness has been proven to be helpful in care delivery,^15,16 and also implemented for telehealth visits.¹⁷ Recently proposed 3-3-3 approach of sending reminders at three different time points: 3 weeks, 3 days, and 3 h ahead might do even more to improve appointment confirmation rate.¹⁸ We expect a similar strategy to be successful for telehealth visits, given the practices and regulations.¹⁹

Engagement was one of the most striking themes for C/R reasons. Unlike the technical difficulties and scheduling conflicts that are commonly occurring in virtual visits, engagement posits a new phenomenon regarding patient–provider engagement in virtual environment. In this study, there was a noticeable C/Rs proportion of 25% (95% CI: 19–31%) that were attributed to the inability to engage a patient to complete the telehealth session.

Finally, the analysis of demographic comparisons of the sampled C/R group versus the overall telehealth population indicated statistically significant underrepresentation of Whites in C/Rs and overrepresentation of Black in C/Rs. This may be related to the racial disparities and social determinants that may pose a barrier to accessing broadband internet, end user devices including laptops, phones, and the ability to pay for data plans.²⁰

Engagement During the Telehealth Visits

Patient engagement during clinical encounters is often dependent of a variety of patient-specific and structural factors related to the clinical visit. However, in the context of telehealth visits, unique issues related to parental understanding of expectations of preparedness and child's comfort in their natural environment may play a role. The convenience related to telehealth visits may be considered as a supplement to in-person visits and misconstrued as inferior or less important that in-person visit. For example, inability to get a child out of bed or “having a late night and the child was still sleeping” was cited in the clinical notes. This could result in a few conclusions, such as the activities of daily living that get a child ready to come to an in-person visit also result in improving their ability to engage in a session or possibility that reduction in formality of a telehealth visit results in less need to prepare their child for sessions. As an example, having a child get out of bed, get dressed, and take a car ride to their in-person session may improve the child's ability to attend to the therapy session in comparison with a telehealth session where these same activities of daily living did not occur.

The engagement problems may also reflect inadequate health literacy considerations when communicating about the intent of telehealth visits and the expectations related to patient participation during these visits. For example, clinically we noticed that several parents were unclear about the need for child to be present during the telehealth visit as they believed telehealth visit was like a phone call for additional information in between visits. In contrast, youth may be much more in control of their home environment providing them with leverage to express their resistance to participate in a clinical visit (almost half of the patients who are not in state of visit were under 12 years old). Although it is unclear whether the absence during visit was related to significant changes in routine and school closures secondary to COVID-19 instead of deliberate lack of engagement.

Simmons-Mackie and Kovarsky²¹ explained that “a successful clinical interaction is crucial to the effective enactment of therapy.” They described the critical interaction as engagement and defined it as a level of interpersonal involvement displayed by participants in social situations or interactive activities.

The level of engagement is also critical for speech language pathology clinical services as the basis of language acquisition requires a desire for or need for use of language to get personal needs met. Speech language pathologists, as with many therapeutic specialists, are trained experts in using activities and actions in a therapy session to obtain engagement from the patient. Once engagement is obtained, then progress on therapeutic goals is attempted. As the results of this study, patient engagement within telehealth was an unexpected barrier and frequent cancellation reason. Even though we were not able to find evidence on “why,” literature suggests that technology may create a “bystander” feeling for children with language disorders, or a feeling like active participation in communication is too difficult.²² In the future, concerns for engagement through telehealth could be further exacerbated in children with social language disorders, where the language goals include the pragmatic components of language development, such as eye contact, body language, and subtle social cues that could be difficult to interpret through videoconferencing.

In contrast, for the specialties that do not need physical manipulation or surgical intervention, telehealth could be comparable and sometimes preferable to in-person visits. For example, the evidence base for feasibility, clinical utility, patient satisfaction, and clinical outcomes for telehealth in pediatric behavioral health is well established.²³ However, there are specific adaptations and related competencies or best practices for telehealth encounters. It is valuable to train providers in telehealth best practices to ensure clinically meaningful visits with authentic patient experience. Owing to rapid implementation and prioritizing the continuity of care, the training for clinical workforce in specific specialty skills was limited. This may be leading to the perception of lower quality of engagement than in-person visits.

Future Directions of Telehealth Engagement With Pediatric Patients

This study presented unique aspects for telehealth visit C/Rs that might be different from in-person visits. Although the technical reasons are unique to telehealth and may receive more attention during implementation and evaluation,¹³ this study shows that engagement is one of the new challenges to the pediatric telehealth visits. Regarding the behavioral health and speech language pathology discussions before, providers and decision makers should strategize the methods to improve patient and caregiver readiness for telehealth visits and methods to improve engagement. Currently, health care teams create the scheduling documents to prepare children and families for each visit and to address common technology troubleshooting solutions.

However, there is a need for changing the goals of a therapy. Child engagement was often shifted during telehealth, focus on caregiver engagement and coaching where clinicians are training family members to be the facilitators of therapeutic intervention. Anecdotal evidence showed that clinicians are embracing the parent-focused model wherein parents were taught therapeutic interventions through telehealth and asked to perform those interventions on their children throughout the week as a beneficial treatment option. Such new methods suggest providers to closely follow the telehealth visitations and norms established between patient and provider.²⁴ In addition, improvement in clinical workflow^1,25 and telehealth design^4,26 are suggested steps that may further improve the engagement as well as reducing risks of technical and scheduling problems in telehealth visits. Finally, further research is necessary to understand and intervene the underrepresentation of White race in C/Rs and overrepresentation of African American or Black race in C/Rs.

Limitations

Our study was limited in terms of the sample size, specialty, and geographical area. Therefore, the findings may not be representative for other specialties or regions. Telehealth visits were limited to only video visits through Zoom, excluding telephone visits. Majority of the coding was completed with one coder even though the initial peer coding showed high alignment among coders. The medical records were only reflecting observations based on the provider's notes, without patient or caregiver insights. In addition, health institutions do not require providers to document the reasons of C/Rs, which may introduce inconsistency in the documentation practice. In addition, we focused on unique patients to reduce repetitive results from the same patients, therefore, we did not capture the frequency of C/R visits for each patient or the changes over the time during the study period. Since the study reflects the early periods of telehealth visitations after the pandemic, themes may not be reflective of the population currently (e.g., increased familiarity to the technology and less technical problems).

A few future research directions can address the limitation of this study, to generalize the findings, and to expand the scope in terms of overcoming the physician documentation bias by surveying patients and families directly and ensuring the generalizability of the findings by including multiple hospitals nationally and internationally.^3,27 Furthermore, the association and comparison between the reasons of C/Rs with other factors, such as new patients versus follow-up patients, telehealth versus in person, specialty (e.g., behavioral health vs. speech language pathology), C/R frequency analysis, and number of previous no shows, and in the context of telehealth versus in-person visits regarding the impact of the health equity and disparity in the reasons of C/Rs are suggested future studies.⁵

Conclusions

This study provided insight about telehealth C/R reasons that were captured in the medical records. Our findings will help fill the literature gaps and may help with enhancing the digital experience for both caregivers and providers, improvement of the financial bottom line of clinical operations, and reduction of wasted time and resource.

Footnotes

Authors' Contributions

S.L. conceived the presented idea. E.S., Y.H., and S.L. outlined and drafted the article. E.S. and D.L. conducted coding and thematic analysis. Y.H. conducted the quantitative analysis. Collectively, E.S., Y.H., D.L., U.R., L.P., and S.L. contributed, critically reviewed, and revised the article. The final version of the article is approved by all the authors.

Disclaimer

The statements in this publication are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.

Acknowledgments

The authors thank Dan Digby and Rajesh Ganta for their support in data pull and presentation.

Disclosure Statement

No competing financial interests exist.

Funding Information

Research reported in this publication was partially funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-2017C1-6413).

References

Wosik

, Fudim

, Cameron

, et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Informatics Assoc, 2020; 27:957–962.

Hollander

, Carr

. Virtually perfect? Telemedicine for Covid-19. N Engl J Med, 2020; 382:1679–1681.

Fisk

, Livingstone

, Pit

. Telehealth in the context of COVID-19: Changing perspectives in Australia, the United Kingdom, and the United States. J Med Internet Res, 2020; 22:e19264.

Liu

, Huang

, Baldacchino

, et al. Telehealth for “non-critical” patients with chronic diseases during the pandemic. J Med Internet Res, 2020; 22:e19493.

Roberts

, Mehrotra

. Assessment of disparities in digital access among Medicare beneficiaries and implications for telemedicine. JAMA Intern Med, 2020; 180:1386–1389.

Scott Kruse

, Karem

, Shifflett

, Vegi

, Ravi

, Brooks

. Evaluating barriers to adopting telemedicine worldwide: A systematic review. J Telemed Telecare, 2018; 24:4–12.

Kemp

, Williams

, Sharma

, et al. Barriers associated with failed completion of an acute care general surgery telehealth clinic visit. Surgery, 2020; 168:851–858.

Moosavinasab

, Sezgin

, Huang

, Sun

, Hoffman

, Lin

DeepSuggest: Using neural networks to suggest related keywords for a comprehensive search of clinical notes. 2020; DOI: 10.13140/RG.2.2.26374.11840.

Braun

, Clarke

. Using thematic analysis in psychology. Qual Res Psychol, 2006; 3:77–101.

10.

Cassell

, Symon

. Essential guide to qualitative methods in organizational research. London, UK: Sage, 2004.

11.

Almathami

HKY

, Win

, Vlahu-Gjorgievska

. Barriers and facilitators that influence telemedicine-based, real-time, online consultation at patients' homes: Systematic literature review. J Med Internet Res, 2020; 22:e16407.

12.

McHugh

ML.

Interrater reliability: The kappa statistic. Biochem Medica, 2012; 22:276–282.

13.

Jnr

BA.

Use of telemedicine and virtual care for remote treatment in response to COVID-19 pandemic. J Med Syst, 2020; 44:1–9.

14.

Triana

, Gusdorf

, Shah

, Horst

. Technology literacy as a barrier to telehealth during COVID-19. Telemed J E Health, 2020; 26:1118–1119.

15.

Arora

, Burner

, Terp

, et al. Improving attendance at post-emergency department follow-up via automated text message appointment reminders: A randomized controlled trial. Acad Emerg Med, 2015; 22:31–37.

16.

Robotham

, Satkunanathan

, Reynolds

, Stahl

, Wykes

. Using digital notifications to improve attendance in clinic: Systematic review and meta-analysis. BMJ Open, 2016; 6:e012116.

17.

Soegaard Ballester

, Scott

, Owei

, Neylan

, Hanson

, Morris

. Patient preference for time-saving telehealth postoperative visits after routine surgery in an urban setting. Surg (United States), 2018; 163:672–679.

18.

Higgins

How the 3-3-3 approach revolutionizes appointment confirmation rates. Available at https://www.healthitoutcomes.com/doc/how-the-approach-revolutionizes-appointment-confirmation-rates-0001 (last accessed September 29, 2020 ).

19.

Berg

, Picoraro

, Miller

, et al. COVID-19—A guide to rapid implementation of telehealth services: A playbook for the pediatric gastroenterologist. J Pediatr Gastroenterol Nutr, 2020; 70:734.

20.

Anderson

, Kumar

Lower-income Americans still lag in tech adoption | Pew Research Center. Pew Research Center. Available at https://www.pewresearch.org/fact-tank/2019/05/07/digital-divide-persists-even-as-lower-income-americans-make-gains-in-tech-adoption/ (last accessed October 14, 2020 ).

21.

Simmons-Mackie

, Kovarsky

. Engagement in clinical interaction: An introduction. Semin Speech Lang, 2009; 30:5–10.

22.

Orsmond

, Shattuck

, Cooper

, Sterzing

, Anderson

. Social participation among young adults with an autism spectrum disorder. J Autism Dev Disord, 2013; 43:2710–2719.

23.

Clinical update: Telepsychiatry with children and adolescents. J Am Acad Child Adolesc Psychiatry 2017;56:875–893.

24.

Gordon

, Solanki

, Bokhour

, Gopal

. “I'm not feeling like I'm part of the conversation” patients' perspectives on communicating in clinical video telehealth visits. J Gen Intern Med, 2020; 35:1751–1758.

25.

Patel

, Cobb

, Wright

, et al. Rapid development of telehealth capabilities within pediatric patient portal infrastructure for COVID-19 care: Barriers, solutions, results. J Am Med Inform Assoc, 2020; 27:1116–1120.

26.

Childs

, Unger

, Li

. Rapid design and deployment of intensive outpatient group-based psychiatric care using telehealth during COVID-19. J Am Med Inform Assoc, 2020; 27:1420–1424.

27.

Smith

, Thomas

, Snoswell

, et al. Telehealth for global emergencies: Implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare, 2020; 26:309–313.