Self-Triage Use,Subsequent Healthcare Utilization,and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Abstract

Background

Self-triage is becoming more widespread, but little is known about the people who are using online self-triage tools and their outcomes. For self-triage researchers, there are significant barriers to capturing subsequent healthcare outcomes. Our integrated healthcare system was able to capture subsequent healthcare utilization of individuals who used self-triage integrated with self-scheduling of provider visits.

Methods

We retrospectively examined healthcare utilization and diagnoses after patients had used self-triage and self-scheduling for ear or hearing symptoms. Outcomes and counts of office visits, telemedicine interactions, emergency department visits, and hospitalizations were captured. Diagnosis codes associated with subsequent provider visits were dichotomously categorized as being associated with ear or hearing concerns or not. Nonvisit care encounters of patient-initiated messages, nurse triage calls, and clinical communications were also captured.

Results

For 2168 self-triage uses, we were able to capture subsequent healthcare encounters within 7 days of the self-triage for 80.5% (1745/2168). In subsequent 1092 office visits with diagnoses, 83.1% (891/1092) of the uses were associated with relevant ear, nose and throat diagnoses. Only 0.24% (4/1662) of patients with captured outcomes were associated with a hospitalization within 7 days. Self-triage resulted in a self-scheduled office visit in 7.2% (126/1745). Office visits resulting from a self-scheduled visit had significantly fewer combined non-visit care encounters per office visit (fewer combined nurse triage calls, patient messages, and clinical communication messages) than office visits that were not self-scheduled (−0.51; 95% CI, −0.72 to −0.29; P < .0001).

Conclusion

In an appropriate healthcare setting, self-triage outcomes can be captured in a high percentage of uses to examine for safety, patient adherence to recommendations, and efficiency of self-triage. With the ear or hearing self-triage, most uses had subsequent visit diagnoses relevant to ear or hearing, so most patients appeared to be selecting the appropriate self-triage pathway for their symptoms.

Keywords

self-triage self-scheduling primary care nonvisit care access to care outpatient visits triage patient portal practice management

Introduction

There are many web-based tools to help patients with questions about their symptoms. Some tools have a focus on providing a differential diagnosis. Symptom checkers such as WebMD® give users a listing of possible diagnoses for their symptoms and then give recommendations based on the possible diagnoses.¹ Other symptom checkers perform a triage function, giving online recommendations for further care and evaluation without delivering a list of diagnoses. For example, the United Kingdom (UK) has National Health Services (NHS) 111, an online triage system that can direct patients to get different levels of care based on their symptoms. NHS 111 specifically states “111 online will not give you a diagnosis, but we will direct you to the best place to get help for your symptoms.”² The Netherlands has an app “should I see a doctor” that asks triage questions and gives recommendations such as “contact the clinic” or gives self-care advice.³ During the COVID pandemic, a number of self-triage tools were developed to direct patients to proper areas of care before scheduling a COVID test.^4,5

Online symptom checkers have major barriers to gathering outcomes data. Many symptom checkers are used anonymously and are not integrated with full-service healthcare institutions, so outcomes cannot be captured. Self-triage tools that are associated with healthcare institutions and collect patient identifiers may not have the necessary ecosystem to gather outcomes data. For example, online patients told to get urgent care for their symptoms might go to an out-of-network emergency department (ED) or urgent care center. Without complete interoperability of electronic health records (EHRs), there is no straightforward way to know what happened after self-triage.

Mayo Clinic developed a menu of symptoms for self-triage, including ear or hearing concerns. In this study, we examined patient demographics and diagnoses associated with its use. We also examined healthcare utilization within 7 days of the self-triage, including face-to-face visits in the office, ED visits, hospitalizations, and nonvisit care (NVC) with telephone triage, patient messages, and other patient communication encounters.

Our aim in this manuscript is to describe both process and clinical outcomes following self-triage for a specific symptom complex, ear or hearing. Aligned with a managerial epidemiology emphasis, we present multiple outcomes that span interests across separate stakeholder groups, including clinicians, management, and healthcare process and quality specialists.

Methods

Overview

This was a retrospective study examining several different outcomes following self-triage. Our research questions addressed multiple issues including process performance, patient safety, and data adequacy. In Table 1, we summarize the major research questions addressed in this study.

Table 1.

Major Research Questions With Data and Method Summary.

Research question	Data and method summary
Is self-triage use appropriate to the ear/hearing algorithm? (Do subsequent visits have a high proportion of ear or hearing diagnoses?)	Proportion of subsequent visits where provider diagnoses are ear or hearing diagnosis codes.
Can self-triage subsequent encounters be captured in adequate counts and proportions?	Counts/proportion of self-triage use with a subsequent encounter captured within 7 days of self-triage.
Are subsequent deaths attributable to self-triage?	Manual review of subsequent deaths following triage.
How serious are the conditions associated with ear/hearing self-triage use?	Counts/proportions of hospital admissions and observations within 7 days following self-triage. Emergency Severity Index for ED subsequent visits.
What is the frequency of visit care categories following self-triage?	Counts/proportions of different categories of subsequent visit access (office visits, ED visits, telemedicine visits).
What is the frequency of nonvisit care following self-triage?	Counts/proportions of different categories of nonvisit care access (patient messages, nurse triage, communication encounters).
What is the time course of patient visit and nonvisit encounters following self-triage?	Counts of visit and nonvisit care categories by time after self-triage.
Does self-scheduling affect subsequent nonvisit care?	Subsequent nonvisit care counts per use compared between those with self-scheduled visits versus not self-scheduled.
Are Home Care recommendations associated with adverse outcomes or unintended office or ED visits?	Manual review of outcomes of home care for adverse events. Counts/proportions of home care recommendations with subsequent ED and office visits.
Does self-triage subsequent healthcare utilization match the self-triage recommendation?	Counts/proportions of users whose subsequent encounters include the self-triage recommended healthcare (nurse triage call, schedule an appointment, stay at home).

Abbreviation: ED, emergency department.

Setting

This study took place at Mayo Clinic Rochester (MCR) and Mayo Clinic Health Care Systems (MCHS). Mayo Clinic is a multispecialty medical practice with multiple locations in the US and internationally. Self-triage of symptoms has been part of Mayo Clinic Patient Online Services since 2020. Patients who have a Mayo Clinic primary care physician and who are registered in Mayo Clinic Patient Online Services (the patient portal) can access symptom self-triage. Our study only included patients who had primary care providers in MCR and MCHS.

Symptom Self-Triage Process

Mayo Clinic symptom self-triage is available both online and via mobile app through Patient Online Services (patient portal). When patients select a symptom, an embedded software algorithm presents them with questions designed to help determine the symptom urgency and type of care needed. Based on their answers the algorithm generates a recommendation for further care. Where appropriate, questions are accompanied by illustrations and photos. The questions and recommendations in the software are authored and reviewed by physicians and providers of Mayo Clinic. Outcomes such as those reported here are reviewed and used to adjust the algorithms.

Ear or Hearing Concerns Process Description

“Ear or Hearing Concerns” was one of several symptom choices that could be self-triaged at the time of the study. A few page views of the ear or hearing self-triage can be seen in supplemental files.

Recommendations for care in the ear or hearing concern self-triage are: “Schedule a Provider Visit,” “Call Nurse Triage,” or “Home Self-Care” with instructions for home care if no provider visit was deemed necessary. It should be noted that instructions for Home Self-Care did not include recommendations for home nurse visits or other provider care delivered to the home. For those with “Call Nurse Triage,” a telephone number is given which provides telephonic advice from Mayo Clinic nurses 24/7. The “Schedule a Provider Visit” gave the patient an option to self-schedule their provider visit. The questions, responses, and recommendations of the algorithmic self-triage are documented in the medical record. During the study, patients were limited to only one use of a specific symptom for self-triage within the same calendar day.

Data Collection

Data collection occurred from December 18, 2020, through June 22, 2022. Interactions with the healthcare system are described as “encounters” in Epic^™, the EHR used at Mayo Clinic. We captured the initial patient self-triage “encounter” and additional Epic encounters for the subsequent 7 days following self-triage. The 7-day time frame was chosen based on previous triage research.⁶ Subsequent encounter data contained the type of encounter, such as office visit, telemedicine visit, ED visit, hospital admission, triage nurse call, patient message (portal message), or clinical communication. With encounters that contained a provider visit such as an office visit, we collected the primary diagnosis code (ie, first listed) for that visit. We did not review additional diagnosis codes beyond the first listed code.

Diagnosis codes associated with the visit encounters were in the International Classification of Disease 10 (ICD 10) coding system and were collapsed into 12 categories that corresponded to ear or hearing diagnoses: (1) otitis externa, (2) nonsuppurative otitis media, (3) suppurative otitis media, (4) cerumen impaction, (5) hearing loss, (6) tinnitus and other, (7) otalgia, (8) eustachian tube disorder, (9) tympanic membrane disorder, (10) dizziness, (11) vertigo, and (12) mastoid related. These 12 categories were then collapsed into what we called ear/hearing. To have an even more comprehensive list of otorhinolaryngology-related diagnoses, we added 4 additional diagnoses with potential otorhinolaryngology (ear, nose and throat [ENT]) implications. These additional more comprehensive ENT diagnostic categories were (13) upper respiratory illness, sinusitis, pharyngitis, laryngitis, (14) cough or fever, (15) exposure to COVID, and (16) COVID-19. Thus, the ENT diagnoses comprised 16 diagnosis categories, the original 12 ear/hearing diagnoses described above, and the 4 more general diagnoses including COVID. The complete codebook for these categories is included in the supplemental files.

We used the Emergency Severity Index (ESI) as a measure of the severity of patients who visited the ED within 7 days of self-triage use. The ESI stratifies patients into 5 groups from 1 (most urgent) to 5 (least urgent). The US Agency for Healthcare Research and Quality explains the coding on its website and the ESI has been studied for reliability and validity.^7,8

To evaluate the effect of self-scheduling on subsequent NVC, we totaled counts of nurse triage calls, patient-initiated messages, and clinical communication encounters for an NVC measure (see Figure 1).

Figure 1.

Counts of self-triage use with subsequent visit and nonvisit encounter counts.

Statistics and Ethics

Stata 17.0 was used for statistical analysis. This study met the institutional review board criteria for exemption (IRB-2020-006809).

Results

User Demographics

There were 2168 uses of self-triage by 2044 users. There were no subsequent encounters in 7 days from the time of the self-triage for 423 (19.5%) of the uses in 382 (18.7%) unique users, leaving 1745 (80.5%) uses and 1662 (81.3%) users having encounters within 7 days of the self-triage that we used for outcomes analysis. Of the 1745 uses, 749 (43%) were on weekends or between 7 pm and 7 am.

There were significant differences in age categories between those with and without subsequent encounters. We found that the 18- to 34-year category was the main driver of the age difference as shown in Table 2. There was no significant difference in sex, race, and ethnicity between those who had subsequently captured encounters in their health record and those who did not. Table 2 shows the demographics of the users, comparing those with and without captured subsequent encounters up to 7 days after self-triage.

Table 2.

Comparison of Patients With and Without Captured Encounters for 7 Days Following Self-Triage.

Demographic	All self-triage ear or hearing users, N = 2044 n(%)	Self-triage users with captured subsequent encounters N = 1662 n (%)	Self-triage users with NO captured subsequent encounters N = 382 n (%)	P value for demographic difference between users with and without subsequent captured encounters*
Age (years)				<.001
0-17	501 (24.5)	415 (25.0)	86 (22.5)
18-34	579 (28.3)	437 (26.3)	142 (37.2)
35-49	465 (22.8)	385 (23.2)	80 (20.9)
50-64	334 (16.3)	289 (17.4)	45 (11.8)
65-79	165 (8.1)	136 (8.2)	29 (7.6)
80+	0 (0)	0 (0)	0 (0)
Sex				.47
Female	1413 (69.1)	1143 (68.8)	270 (70.7)
Race				.51
White	1935 (94.7)	1567 (94.3)	368 (96.3)
Black	25 (1.22)	22 (1.32)	3 (0.8)
Asian	28 (1.37)	23(1.4)	5 (1.3)
Other	41 (2.01)	37 (2.2)	4 (1.1)
Unknown/Not disclosed	15 (0.7)	13 (0.8)	2 (0.5)
Ethnicity				.55
Hispanic	67 (3.3)	58 (3.5)	9 (2.4)
Not Hispanic	1954 (95.6)	1586 (95.4)	368 (96.3)
Undisclosed/Unknown	23 (1.1)	18 (1.1)	5 (1.3)

*H^o : demographic values are equal.

Self-Triage Recommendations for Those With and Without Captured Subsequent Encounters

There were 3 mutually exclusive recommendations from self-triage: home care, call nurse triage, and schedule a visit. The percent of home care recommendations was 2.8% (12/423) for the group without subsequent encounters compared to 2.5% (44/1745) of those who had subsequent encounters within 7 days. Thus, a recommendation to stay at home would have only accounted for a tiny fraction of those who had no subsequent encounters. Overall, there were no statistically significant differences in self-triage recommendations between those who had additional captured data and those without (P = .86). The percentage of recommendations to schedule a visit was 51.4% (897/1745) and 50.1% (212/423), respectively, for those with and without subsequent encounters, and for nurse triage was 46.1% (804/1745) and 47.0% (199/423).

Subsequent Healthcare Utilization

There were 7773 subsequent encounters categorized into 55 different encounter types by our Epic EHR. Although there is patient and clinic time and effort involved in many of these encounter types, we focused on the encounters that were more directly involved in patient care and took more provider time. Captured encounter types such as ancillary orders, clinical support, documentation, immunization, pharmacy visit, history (administrative changes to record), and medication refills showed that the patient was active in our healthcare system but did not involve much provider effort or a provider-coded diagnosis. Office visits, ED visits, telemedicine visits, and hospitalizations were encounter types that generally resulted in a coded diagnosis and involved provider time. Nonvisit care (patient-initiated messages, nurse triage calls, and clinical communication encounters) was also part of this study because significant provider effort was involved. However, the NVC encounters did not typically have an associated diagnosis code.

For the 7773 subsequent encounters within 7 days, 50% occurred within 22.5 h of the self-triage and 81% occurred less than 4 days (96 h) after self-triage. Figure 2 shows the frequency of office visits by how long the face-to-face visit occurred after self-triage. The total counts are 70 (6.4%) fewer than 1092 shown in Figure 1 because some of the scheduling that occurred during the 7-day follow-up, especially those for ENT specialists and audiologists, had to be matched with visit openings that were more than 7 days out. Since the visits were scheduled in the 7-day period, we had diagnosis data on those 70 visits that were pertinent to this study even though the actual office visit itself did not occur within 7 days. Figures 2 and 3 show the visit care (office and ED visits) counts by time after self-triage.

Figure 2.

Counts of face-to-face office visits by hours after self-triage.

Figure 3.

Counts of emergency department visits by hours after self-triage.

Figures 4, 5, and 6 show the NVC (nurse triage calls, patient messages, and clinical communication encounters) counts by hours after self-triage.

Figure 4.

Counts of nurse triage calls by hours after self-triage.

Figure 5.

Counts of patient message threads by hours after self-triage. Counts include only the initial patient message in a thread, not complete back-and-forth message counts within the message thread.

Figure 6.

Counts of clinical communication encounters by hours after self-triage. Counts include only initial communication in a potential back-and-forth communication thread.

Diagnoses Following Self-Triage

Table 3 shows the capture of primary diagnosis codes from office visits, ED visits, and telemedicine visits. Over 81% of self-triage encounters with a subsequent office visit had an ENT-related primary diagnosis and 83% of unique patients with subsequent office visits had an ENT-related primary diagnosis.

Table 3.

Subsequent Encounter Types Within 7 Days Following Ear/Hearing Self-Triage and Percent Unique Encounters and Unique Patients With Diagnosis of ENT (all 16 Coding Categories) or Ear/Hearing Diagnoses (12 Coding Category Subset).

Encounter type	Office visits	ED visits	Telemedicine visits	All encounters
Unique encounters	1092	129	51	7773
Encounters where diagnosis codes available	1092	125	51	2491^a
Percent unique encounters with primary diagnosis available (n)	100 (1092)	96.9 (125)	100 (51)	32.0 (2491)
Percent unique encounters with at least one of 16 ENT diagnoses (n)	81.6 (891)	58.9 (76)	58.8 (30)	20.0 (1554)
Percent unique encounters with at least one of 12 ear/hearing diagnoses (n)	70.2 (767)	38.0 (49)	19.6 (10)	12.3 (958)
Unique patients	915	116	49	1604
Percent of patients with at least 1 of 16 ENT diagnoses (n)	83.1 (760)	62.9 (73)	59.2 (29)	69.7 (1118)
Percent of patients with at least 1 of 12 ear/hearing diagnoses (n)	71.0 (650)	41.4 (48)	20.4 (10)	45.9 (736)

Abbreviations: ENT, ear, nose and throat; ED, emergency department.

There were many encounters with diagnosis codes that were not associated with provider visits. Those encounters were often lab tests requiring a specific diagnosis, such as COVID swabs prior to being seen face-to-face.

Hospitalizations and Death Following Self-Triage

There were 4 hospitalizations from 4 unique patients accounting for 0.24% (4/1662) of the self-triage uses. Three were hospitalized with pneumonia, with hospitalizations occurring 10, 24, and 150 h after self-triage, A fourth patient was hospitalized 13 h after self-triage and had a pharyngeal abscess. Hospital observation admissions occurred in 3 (0.18%) unique patients at 5, 11, and 54 h from self-triage; diagnoses were vertigo, mastoiditis, and chest pain, respectively. No patient admitted to the hospital for an observational or inpatient admission had a home care recommendation from self-triage.

We searched the EHR of each self-triage user for subsequent death. Only 3 deaths were discovered, all greater than 250 days from the date of self-triage. Review of these deaths did not uncover any connection between the cause of death and their self-triage of ear or hearing concerns.

Nurse Triage Following Self-Triage

Nurse triage was recommended in 1003 of 2168 self-triage uses (46.3%). Nurse triage encounters occurred within 7 days in 608 uses of self-triage, but only 36.6% (367/1003) of those in which nurse triage was recommended. In 39.6% (241/608) of subsequent nurse triage encounters, there was not an associated self-triage recommendation to call nurse triage. However, a recommendation for self-scheduling could have indirectly generated a nurse triage call. Patients were directed to call nurse triage after a recommendation to schedule an appointment if a self-scheduled slot was not available within 24 h. We could not capture how often that occurred.

Nurses in 3.9% (24/608) of the calls gave an ED recommendation for 1.3% (22/1662) of patients for whom we had outcomes data. Only 9 ED visits of the 129 (7.0%) were associated with these nurse ED triage recommendations. Thus, adherence to nurse triage ED recommendation for those who had subsequent encounters was 37.5% (9/24).

Home Care Outcomes

Of the 56 home care recommendations given (44 that had subsequent captured encounters), there was only one ED visit that occurred 6 h and 40 min after self-triage with an ED diagnosis of allergic rhinitis. There was no nurse triage associated with this visit and a CT head was done during the ED visit before dismissal and no further encounters occurred for the rest of the 7 days.

Overall, the 56 home care recommendations were associated with 22 office face-to-face visits for 20 unique patients, 8 nurse triage calls (5 patients), 2 telemedicine visits (2 patients), and 18 patient message threads (13 patients).

Emergency Department Subsequent Encounters

There were 129 hospital ED encounters captured for 116 unique patients. By design, no patient was directly referred to the ED through self-triage. Patients who had symptoms that were assessed in self-triage to have features that required more urgent care were recommended to call a nurse triage telephone number that was available 24/7 for MCR and MCHS patients. As noted above only 9 of the ED visits (7%) were associated with a triage nurse recommendation to seek emergent care.

We captured the Emergency Severity Index (ESI) in 127 of the 129 total ED encounters. The mode of the 127 ESI captured ED encounters was 4 (n = 48, 45.7%). There were no ED encounters that were given an ESI of 1; only 3 encounters had an ESI of 2 (2.4%), with diagnoses of anxiety, dizziness, and calcaneal fracture. Lesser urgency ESIs of 3, 4, and 5 accounted for 97.6% (124/127). We also did not find major patient misclassifications of a symptom such as an ED visit for myocardial infarction or abdominal catastrophe shortly after an ear/hearing self-triage.

Four ED encounters did not have an associated diagnosis. Chart review showed they left without complete evaluation, but 2 of the 4 did have swabs for streptococcus and COVID before leaving.

Self-Scheduled Visits and Appointment Recommendation Adherence

We had outcomes data on 897 self-triage recommendations to schedule an appointment (self-scheduling was an option for all 897). Of these, 14% (126/897) created and completed a self-scheduled visit. Completed self-scheduled appointments occurred in 7.2% (126/1745) of the total self-triage uses that had subsequent 7-day encounters. Self-scheduled office visits compared to those not self-scheduled had significantly fewer NVC encounters (total count of nurse triage, patient messages, and clinical communications). There were 106 self-scheduled uses from self-triage resulting in a single office visit with a mean of 0.42 NVC encounters per visit (SD 0.72; CI 95, 0.29-0.56). There were 727 self-triage uses not self-scheduled resulting in a single office visit that had a mean of 0.93 NVC encounters (ST 1.1; CI 95, 0.85-1.0). The decrease in NVC was −0.51 encounters per use for the self-scheduled (95% CI, −0.72 to −0.29; P < .0001). This was a percent decrease of 54% NVC encounters compared to those visits not self-scheduled. Further analysis showed that the decrease in NVC in the self-scheduled group was mostly (90%) due to fewer nurse triage calls.

When scheduling an appointment was recommended, overall adherence to complete an office visit was 62% (555/897). Completed self-scheduled visits accounted for 23% (126/555) of those complying with the recommendation to schedule.

Patient-Initiated Messages and Communication Encounters After Self-Triage

Patient-initiated messages through the patient portal followed self-triage in 27.7% (483/1745) of the uses. Figure 5 shows that patient-initiated messages were mostly within 24 h after self-triage. Of the 483, there were 473 that the patients chose to categorize as a medical advice request and 10 were selected by the patients online to be a patient schedule request. Content review by string searches of messages showed that 51% (255/483) had content related to ear or hearing, 31% (150/483) contained content about “antibiotics” or “infection” and 15% contained content about “COVID.”

Clinical communication encounters within our MCR and MCHS primary care practices are generally related to appointment scheduling requests or patient requests that may have come from patients via telephone. However, for the patients using the ear/hearing self-triage, many of the clinical communications were from appointment schedulers and nurses arranging COVID swabs. During the pandemic, there was a low threshold for obtaining COVID swabs prior to office visits.

Discussion

In over 80% of the completed self-triage encounters, we were able to capture subsequent encounters. The large percentage of the subsequent ear or hearing diagnoses demonstrated that patients were appropriately selecting the correct symptom path and engaged in seeking healthcare for a specific concern. The ear/hearing self-triage with self-scheduling gave patients a 24/7 mobile and online ability to get recommendations for care, and to schedule care in one encounter. Self-scheduling an office visit after self-triage was associated with significantly less NVC utilization when compared with subsequent office visits not self-scheduled (−0.51 NVC encounters per scheduled office visit; 95% CI, −0.72 to −0.29; P < .0001).

Practice Implications

We found 14% of patients self-scheduled and completed an office visit when offered a self-scheduled appointment. Other studies of self-scheduling show varied uptake of self-scheduled appointments. A general online appointment process for scheduling was used by 11% of patients in primary healthcare practice in Australia.⁹ At Mayo Clinic, USA, 5% of well-child visits and 21% of mammograms were self-scheduled.^10,11 At the peak of COVID, up to 44% of COVID-19 PCR testing was self-scheduled.⁵

Self-scheduling has been successful in other appointment contexts, including well-child visits, screening mammograms, and lab specimen visits (nasal swab specimen collection for COVID-19 PCR tests).^5,10,11 Self-scheduling associated with self-triage shows some promise to help the practice decrease some NVC associated with scheduling. Our current study showed that self-scheduled office visits with self-triage had significantly fewer NVC encounters than those office visits associated with self-triage that did not self-schedule. This result is consistent with findings from self-triage and self-scheduling of COVID PCR testing which showed savings of thousands of hours of nurse triage time and staff scheduler time.^5,12

It was out of the scope of this study to fully explore NVC that was associated with self-triage. However, our limited content review of over 400 patient-initiated messages showed that almost a third of them had content containing the terms “antibiotics” and “infection.” Subsequent review of these messages showed that many were from patients requesting antibiotics for a self-diagnosed “infection.” Patient messages now represent a significant and increasing part of NVC in primary care.¹³ These patient messages soliciting antibiotics for self-diagnosed ear conditions underscore the need for studies like this to examine both visit and NVC outcomes as we strive to get a more comprehensive view of how patients interact with the healthcare system and the impact of our interventions.

Only 36% called nurse triage when instructed by self-triage to do so. Adherence was better at 62% with the self-triage recommendation to schedule a visit. A web-based triage system in the Netherlands found 57% adherence based on a follow-up questionnaire of only 35 respondents.¹⁴ NHS 111 online triage users reported 67.5% adherence to advice given.¹⁵ Further study will be needed to determine how adherence data along with diagnosis outcomes and disease severity could help guide modifications to self-triage algorithms.

Patient Implications

Even though nurse telephone triage was available 24/7 during the study, 43% of self-triage uses were outside extended weekdays hours of 7 am to 7 pm. Self-triage perhaps met the needs of some patients who preferred an afterhours online “Check Symptoms” over a telephonic interaction. Self-triage adds to the growing list of options for patients to get advice about their symptoms and to help them determine whether they need urgent care.

It was outside the scope of this study to assess the accuracy of patients’ decisions versus accuracy of the self-triage. However, the outcomes diagnoses supported that patients were accessing the appropriate triage path.

Overall safety appeared good. Hospital admissions and observations accounted for just 0.24% (4/1662) and 0.18% (3/1662), respectively. There were no self-triage attributable deaths and most of the ED visits had an ESI that was low acuity. We did not identify major patient safety concerns.

Implications for Future Self-Triage Research and Development

This study shows the need for additional broad-based outcomes research for self-triage. Figure 1 shows the complex outcomes pathways that patients can pursue after self-triage, including both visit and NVC. Future research will need to examine the impact of self-triage in the increasingly complex healthcare ecosystem that includes multiple ways to access medical care. We found that 7.4% (129/1745) of uses were associated with an ED visit for subsequent care, despite none being directly advised to go to the ED by self-triage. Low hospitalization rates and lesser urgency ESIs suggest that many could have been seen in the office. Unfortunately, we were unable to determine whether there were access issues that prevented timely scheduling of office visits, leaving patients with only an ED option for care. A priority for future research is to capture real-time access data that the patient sees at the time of self-scheduling a visit. Other immediate access modalities such as on-demand video or phone visits should be explored to see whether those would be alternative ways to deliver more immediate recommendations for care.

Our analysis of subsequent ED visits, manual review of hospitalizations, and subsequent deaths suggested that users did not need to schedule visit appointments within one day for our “schedule a provider visit” recommendation. These results helped our development team to recommend changing the self-scheduling window from 1 day to 3 days. We are assessing the results of this change using the methodology shown by this study.

The ability to capture outcomes in a high percentage of self-triage uses is critically important for the future development of self-triage algorithms. As mentioned in the introduction, there are significant barriers to obtaining outcomes data for self-triage. In fact, much of the current published research on self-triage avoids difficult-to-obtain outcomes assessments. Instead, researchers have created vignettes that purport to assess the accuracy of self-triage software algorithms.^16,21 Vignettes are designed to recreate how a patient might interact with an online symptom checker. For example, to test whether a patient would get an appropriate recommendation for symptoms of appendicitis, a vignette is created to mimic appendicitis symptoms, and someone enters those symptoms into the self-triage symptom checker. If the anticipated outcome is achieved, the symptom checker is “validated” as accurate. Wallace et al have recently published a systematic review of the literature on assessing the accuracy of self-triage with a conclusion that large-scale primary studies using real patient data are needed to study this area further.²² Our study shows that with the appropriate healthcare system and infrastructure, patient outcomes can be obtained in a high percentage of uses.

Limitations

Our study methodology utilized primary diagnosis codes (first listed) for subsequent in-person encounters after self-triage. Ear or hearing diagnoses could have been listed #2, #3, #4, or lower in the sequence of visit-related diagnoses. Thus, it is likely we underestimated ear and hearing diagnoses following self-triage.

This was a retrospective study, and we have no matched comparison group. COVID-19 was active during the entirety of the study and undoubtedly had some impact on patient behavior regarding self-triage recommendations.

Patients highly value timely access to care as well as care continuity in decision-making about appointments.^23,24 We did not have real-time data on the appointment slots that patients were being offered for self-scheduling. Some of the subsequent ED visits and nurse triage calls could have been from a lack of ability to self-schedule appointments.

Over the course of the study, there were some repeat users (2168 uses from 2044 users). These represent a group that adds complexity to the results. Future work will need to examine those that use self-triage multiple times and analyze their impact on subsequent healthcare utilization.

We were able to report on relevant ear or hearing visits even after the 7-day capture limit because specialty ENT and audiology appointments more than 7 days from self-triage were booked within the 7-day time limit. This allowed us to report on outcomes of ENT and audiology specialty visits that did not occur within the 7-day limit. If self-triage users waited longer than 7 days to book a subsequent visit, then those visits were not captured in this study. The histograms in “Results” showed a rapid decline of encounters over 7 days for both visit and nonvisit outcomes, so the 7-day limit is likely capturing a large piece of subsequent utilization.

Supplemental Material

sj-jpg-1-hme-10.1177_23333928231168121 - Supplemental material for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Supplemental material, sj-jpg-1-hme-10.1177_23333928231168121 for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms by Frederick North, Teresa B Jensen, Robert J Stroebel, Elissa M Nelson, Brenda J Johnson, Matthew C Thompson, Jennifer L Pecina and Brian A Crum in Health Services Research and Managerial Epidemiology

Supplemental Material

sj-jpg-2-hme-10.1177_23333928231168121 - Supplemental material for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Supplemental material, sj-jpg-2-hme-10.1177_23333928231168121 for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms by Frederick North, Teresa B Jensen, Robert J Stroebel, Elissa M Nelson, Brenda J Johnson, Matthew C Thompson, Jennifer L Pecina and Brian A Crum in Health Services Research and Managerial Epidemiology

Supplemental Material

sj-jpg-3-hme-10.1177_23333928231168121 - Supplemental material for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Supplemental material, sj-jpg-3-hme-10.1177_23333928231168121 for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms by Frederick North, Teresa B Jensen, Robert J Stroebel, Elissa M Nelson, Brenda J Johnson, Matthew C Thompson, Jennifer L Pecina and Brian A Crum in Health Services Research and Managerial Epidemiology

Supplemental Material

sj-jpg-4-hme-10.1177_23333928231168121 - Supplemental material for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Supplemental material, sj-jpg-4-hme-10.1177_23333928231168121 for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms by Frederick North, Teresa B Jensen, Robert J Stroebel, Elissa M Nelson, Brenda J Johnson, Matthew C Thompson, Jennifer L Pecina and Brian A Crum in Health Services Research and Managerial Epidemiology

Supplemental Material

sj-xlsx-5-hme-10.1177_23333928231168121 - Supplemental material for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms

Supplemental material, sj-xlsx-5-hme-10.1177_23333928231168121 for Self-Triage Use, Subsequent Healthcare Utilization, and Diagnoses: A Retrospective Study of Process and Clinical Outcomes Following Self-Triage and Self-Scheduling for Ear or Hearing Symptoms by Frederick North, Teresa B Jensen, Robert J Stroebel, Elissa M Nelson, Brenda J Johnson, Matthew C Thompson, Jennifer L Pecina and Brian A Crum in Health Services Research and Managerial Epidemiology

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Frederick North

Elissa M Nelson

Brenda J Johnson

Matthew C Thompson

Jennifer L Pecina

Brian A Crum

Supplemental material

Supplemental material for this article is available online.

References

WebMD Symptom Checker. 2022, Accessed November 7, 2022. https://symptoms.webmd.com/

NHS 111 online. 2022, Accessed November 7, 2022. https://111.nhs.uk/

Verzantvoort

NCM

Teunis

Verheij

TJM

, et al.

Self-triage for acute primary care via a smartphone application: practical, safe and efficient?

PLoS One. 2018;13:e0199284. 2018/06/27. doi:10.1371/journal.pone.0199284

Judson

Odisho

Neinstein

, et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc. 2020. 2020/04/09. doi:10.1093/jamia/ocaa051

North

Nelson

Majerus

, et al. Self-scheduling process efficiency and utilization of online self-scheduling of lab tests: a retrospective analysis of self-scheduled appointments for COVID testing. Health Serv Res Manag Epidemiol. 2022;9:23333928221125034. doi:10.1177/23333928221125034

North

Varkey

. How serious are the symptoms of callers to a telephone triage call centre? J Telemed Telecare. 2010;16:383‐388. 2010/08/18. doi:10.1258/jtt.2010.091016

Implementation Handbook Emergency Severity Index (2020, accessed December 11, 2022), https://www.ahrq.gov/patient-safety/settings/emergency-dept/esi.html https://www.ena.org/docs/default-source/education-document-library/esi-implementation-handbook-2020.pdf?sfvrsn=fdc327df_2

Tanabe

Gimbel

Yarnold

, et al. Reliability and validity of scores on the emergency severity index version 3. Acad Emerg Med. 2004;11:59‐65. 2004/01/08. doi:10.1197/j.aem.2003.06.013

Zhang

Yan

. Patients’ adoption of the e-appointment scheduling service: a case study in primary healthcare. Stud Health Technol Inform. 2014;204:176‐181. doi:10.3233/978-1-61499-427-5-176

10.

North

Nelson

Buss

, et al. The effect of automated mammogram orders paired with electronic invitations to self-schedule on mammogram scheduling outcomes: observational cohort comparison. JMIR Med Inform. 2021;9:e27072. 2021/12/09. doi:10.2196/27072

11.

North

Nelson

Majerus

, et al. Impact of web-based self-scheduling on finalization of well-child appointments in a primary care setting: Retrospective comparison study. JMIR Med Inform. 2021;9:e23450. doi:10.2196/23450

12.

Judson

Pierce

Tutman

, et al. Utilization patterns and efficiency gains from use of a fully EHR-integrated COVID-19 self-triage and self-scheduling tool: a retrospective analysis. J Am Med Inform Assoc. 2022;29:2066‐2074. doi:10.1093/jamia/ocac161

13.

North

Luhman

Mallmann

, et al.

A retrospective analysis of provider-to-patient secure messages: how much are they increasing, who is doing the work, and is the work happening after hours?

JMIR Med Inform 2020. 2020/07/17;8:e16521. doi:10.2196/16521

14.

Nijland

Cranen

Boer

, et al. Patient use and compliance with medical advice delivered by a web-based triage system in primary care. J Telemed Telecare. 16:8‐11. doi:10.1258/jtt.2009.001004

15.

Turner

Knowles

Simpson

, et al. Impact of NHS 111 Online on the NHS 111 telephone service and urgent care system: a mixed-methods study (2021). https://www.ncbi.nlm.nih.gov/books/NBK575180/

16.

Gilbert

Mehl

Baluch

, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open. 2020;10:e040269‐e040269. doi:10.1136/bmjopen-2020-040269

17.

Hill

Sim

Mills

. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. Med J Aust. 2020;212:514‐519. 20200511. doi:10.5694/mja2.50600

18.

Schmieding

Kopka

Schmidt

, et al. Triage accuracy of symptom checker apps: 5-year follow-up evaluation. J Med Internet Res. 2022;24:e31810. doi:10.2196/31810

19.

Semigran

Linder

Gidengil

, et al. Evaluation of symptom checkers for self diagnosis and triage: audit study. Br Med J. 2015;351:h3480‐h3480. doi:10.1136/bmj.h3480

20.

Shen

Nguyen

Gregor

, et al. Accuracy of a popular online symptom checker for ophthalmic diagnoses. JAMA Ophthalmol. 2019;137:690‐692. doi:10.1001/jamaophthalmol.2019.0571

21.

Yoshida

Thomas Clark

. Accuracy of online symptom checkers for diagnosis of orofacial pain and oral medicine disease. J Prosthodont Res. 2021;65:186‐190. 2020/09/18. doi:10.2186/jpr.JPOR_2019_499

22.

Wallace

Chan

Chidambaram

, et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Digit Med. 2022;5:118. 20220817. doi:10.1038/s41746-022-00667-w

23.

Oliver

Deal

Howard

, et al. Patient trade-offs between continuity and access in primary care interprofessional teaching clinics in Canada: a cross-sectional survey using discrete choice experiment. BMJ Open. 2019;9:e023578. 2019/03/25. doi:10.1136/bmjopen-2018-023578

24.

Rubin

Bate

George

, et al. Preferences for access to the GP: a discrete choice experiment. Br J Gen Pract. 2006;56(531):743‐748. 2006/09/30.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.08 MB

0.05 MB

0.06 MB

0.13 MB

0.02 MB