How serious are the symptoms of callers to a telephone triage call centre?

Abstract

Adverse events such as deaths following telephone triage calls are rare, suggesting that the process is basically safe. However, if calls tend to concern mostly minor illnesses, then adverse events following calls would be uncommon even if the triage process itself was flawed. We investigated hospitalization rates following triage calls and compared them to hospitalization following two other types of medical access, emergency department (ED) visits and office visits. For the Ask Mayo Clinic telephone triage centre, hospitalization rates for adult calls concerning chest pain and abdominal pain were each 13%. Based on national survey data, hospitalization for adult ED visits concerning the same symptoms were 33% (chest pain) and 19% (abdominal pain). Office visits had hospitalization rates significantly lower than triage calls in all age groups, while ED visits had higher hospitalization rates than triage calls in all age groups. There are both qualitative and quantitative similarities between triage calls and ED visits and, using hospitalization as an indicator, some subgroups of triage calls are nearly as serious as ED visits.

Introduction

Telephone triage is widely used around the world and some call centres may handle millions of calls each year. Many call centres, such as NHS Direct in the UK and Ask Mayo Clinic in the USA, use computerized decision support to help make the correct recommendation.^1,2

Despite the widespread use of telephone triage, there remain concerns about its safety. Several studies have shown that triage call centres can vary significantly in their advice to the same call; these studies used specially trained callers who could recite the same ‘standardized’ call to different call centres.³ The trained callers played out a specific symptom in staged calls by giving scripted answers to questions they were likely to be asked. One recent study from the Netherlands showed that in these standardized calls only 21% of the obligatory questions needed to make appropriate recommendations were asked.⁴ It was also noted that urgency was underestimated in 41% of the calls. The Dutch study tested call centres without computerized decision support, but even those using computerized decision support showed large differences in recommended urgency depending on the software used.⁵

Although triage systems have been shown to have significant inconsistencies in the questions asked and the advice given during standardized calls, adverse events as a result of triage appear to be uncommon. In a study of just under 33,000 calls to a US paediatric call centre, Kempe et al. found no deaths within one week following the calls.⁶ In a randomized controlled trial involving 93,000 patients in the UK using death as an outcome, there was no difference between patients who had triage and those with no call triage.⁷ The apparent discrepancy between the inconsistency of the triage process and the lack of evidence of harm was recently addressed in an editorial in the BMJ.⁸ In that editorial, Car et al. stated that ‘the most credible explanation for this discrepancy is that because serious adverse outcomes are relatively rare, most of the studies have lacked the power to detect harm to patients’.

We have examined this question by investigating the seriousness of symptoms in triage calls. We hypothesized that an explanation for the lack of adverse effects following triage may be because the calls were mainly about benign conditions that would have a benign course. If the majority of calls were about minor problems, this would explain why adverse effects were so rare and why even inconsistent triage might have little effect on adverse events like death. To assess the severity of the symptoms involved with the calls, we compared hospitalization following triage calls to hospitalization following other types of medical access, including emergency department (ED) visits and outpatient visits.

Methods

Ask Mayo Clinic (AMC) is a telephone triage call centre used by callers in the US. It is available by subscription only. The subscribers are mainly large corporations who purchase access to AMC as a benefit for their employees. Currently over two million people in the US have access to the AMC call centre. The call centre employs about 40 registered nurses who use proprietary computerized decision support software, ExpertRN, to help triage calls. The software contains 39 algorithms and 79 guidelines which help the nurses to ask appropriate questions and make recommendations based on the answers.

Data sources

We compared data from AMC calls to national data available in the National Hospital Ambulatory Medical Care Survey (NHAMCS) and the National Ambulatory Medical Care Survey (NAMCS) from the US Centers for Disease Control (CDC). For the AMC data we used all completed symptom assessment calls in 2006 and 2007 where adequate identifying characteristics were available to search for insurance claims (n = 90,126). From those calls, 20,230 (22%) had private insurance information that we could examine for claims information. We included all 20,230 calls in the analysis. Of these calls, 12,709 (63%) had insurance claims made within seven days of the call.

For the CDC comparison data, ED visit data were obtained from the NHAMCS and data concerning outpatient office visits were obtained from the NAMCS. These datasets and an explanation of their content were downloaded via the Internet.⁹ To match the year of the calls we used the NHAMCS and NAMCS datasets from 2006. National data from the 2007 NHAMCS and NAMCS was not available at the time of analysis.

For AMC callers, hospitalization was inferred if a claim had a place of service code which indicated a hospital inpatient. By using claims data we were able to identify hospitalizations in callers wherever the hospitalization took place, including other hospitals than the Mayo Clinic. Only hospitalizations occurring on the same calendar date as the call, the next day, or call date plus two days were included in the analysis. This ensured that hospitalizations occurring more than 72 hours after a call were excluded from the analysis. For the NHAMCS and NAMCS national data we used a dichotomous variable, ADMITHOS, which indicated whether or not the patient was admitted to the hospital from the ED or from the outpatient clinic. For the NHAMCS and NAMCS data, elapsed time between ED or office visit and hospitalization was not available. However, hospitalization was categorized as a visit disposition in both the NHAMCS and NAMCS datasets, so hospital admission would have occurred soon after the visit, probably within hours.

In the ExpertRN decision support system, algorithms and guidelines are named according to the symptoms they assess. For example there are algorithms for chest pain, abdominal pain and breathing problems. We assigned the algorithm name or guideline used during the call as the reason for the call. The NHAMCS and NAMCS comparison data had a primary reason for visit data field (RFV13D) which allowed us to obtain a symptom by symptom comparison between our call data and the national data. For example, there is an abdominal pain algorithm in ExpertRN which we matched with the unique reason for visit code 1545 (abdominal pain) in the CDC datasets. There was not always complete correspondence of the AMC reason for call categories with the national dataset reason for visit categories.

Statistics

A standard package (SAS 9.1.3, SAS Institute, Cary, NC) was used for the estimates from the national data sources. We used the CDC-recommended code in SAS for obtaining estimates from the NHAMCS and NAMCS data.¹⁰ Odds ratios were calculated using logistic regression and two way tables in SAS.

Table 1

Hospitalization rates following Ask Mayo Clinic triage calls, ED visits and outpatient visits

	Hospitalization (%) [95% CI]			Odds ratio [95% CI]
Age group (years)	AMC triage calls	NAMCS office visits	NHAMCS ED visits	Hospitalization from ED visit versus AMC triage	Hospitalization from office visits versus AMC triage
0–2	1.8 [1.5–2.2]	0.30 [0–0.61]	6.3 [5.0–7.6]	3.6 [2.8–4.6]	0.16 [0.06–0.39]
3–17	1.6 [1.3–2.0]	0.09 [0–0.19]	4.1 [3.5–4.8]	2.7 [2.1–3.5]	0.05 [0.02–0.17]
18–39	2.5 [2.1–2.9]	0.27 [0.09–0.46]	6.9 [6.0–7.8]	2.9 [2.4–3.5]	0.11 [0.06–0.18]
40–64	4.2 [3.6–4.9]	0.30 [0.11–0.49]	16 [15–18]	4.5 [3.8–5.3]	0.07 [0.05–0.10]
65 and over	9.6 [7.7–12]	0.34 [0.2–0.47]	35 [33–38]	5.1 [4.0–5.9]	0.03 [0.02–0.05]

The study was approved by the appropriate ethics committee.

Results

For the AMC calls, hospitalization occurred most often within two days of a call (Figure 1). We confined our analysis only to those 547 cases (3%) hospitalized within 72 hours of the triage call. Hospitalization rates for AMC triage calls varied by age, with a low of 2% for ages 3–17 years to a high of 10% for age 65 years and above (Table 1). For ED visits from the 2006 NHAMCS data, the hospitalization rate went from a low of 4% for ages 3–17 years to 35% for those 65 years and over. Hospitalization rates associated with office visits from the 2006 NAMCS data were 0.3% for all age groups except the 3–17 years group, where it was 0.1% (Table 1). For both the NHAMCS ED visits and AMC calls, the hospitalizations followed similar age trends while hospitalization associated with office visits from the NAMCS data remained relatively constant (Figure 2).

Figure 1

Hospital admissions in the days following an AMC call (day 0 represents hospitalization on the same day as the triage call)

Figure 2

Proportion (%) of AMC calls, proportion of ED visits and proportion of outpatient office visits resulting in hospitalization

Frequencies of hospitalization were greater for NHAMCS ED visits than for symptom assessment calls to AMC in every age group (Table 1). However, hospitalizations were much more common with symptom assessment calls to AMC than with NAMCS office visits. Odds ratios comparing hospitalizations with these different forms of medical access reveal that the odds of being hospitalized were about 3–5 times greater for those attending an ED than for callers to the AMC telephone triage call centre.

For the AMC data, six call symptom categories accounted for 63% of the hospital admissions. Abdominal pain calls accounted for the most hospitalizations (146, 27%). There were 58 admissions from telephone complaints of chest pain (11%). This was followed by hospitalizations related to unwell infant calls (51, 10%), calls about vomiting and nausea (46, 9%), breathing problems (22, 4%) and skin problems (20, 4%).

The national data showed that the reason for an office visit is frequently not because of a specific symptom. The top eight reasons for adult office visits in the national data were progress visits (7%), general medical examinations (5%), skin problems (4%), postoperative checks (3%), medication checks (3%) prenatal examinations (3%), gynaecological examinations (2%) and test results (2%). Unlike the national office visits, the national ED visits were generally for specific symptoms. The top eight national reasons for adult visits to the ED were all related to symptoms and were: abdominal pain (7%), chest pain (6%), back symptoms (3%), headache (3%), shortness of breath (3%), vomiting or nausea (3%), non-localized pain (2%) and skin concerns (2%). The top adult reasons for symptom calls to AMC are shown in Table 2.

Table 2

Comparison of frequency rank and frequency percent of the top 8 reasons for Ask Mayo Clinic call with NHAMCS ED visit, and NAMCS office visit (ages 18 years and over) by same symptom

Reason for AMC call	Number of	Frequency rank of call or visit			Frequency of calls or visits (%) [95% CI]
(age 18 years and over)	AMC calls	AMC calls	ED visits	Office visits	AMC calls	ED visits	Office visits
Abdominal pain	932	1	1	12	9.1 [8.6–9.7]	7.4 [7.0–7.8]	1.4 [1.1–1.6]
Skin problems	916	2	8	3	9.0 [8.4–9.5]	2.2 [2.1–2.4]	4.0 [3.4–4.6]
Cold symptoms	866	3	45	23	8.5 [7.9–9.0]	0.6 [0.5–0.7]	0.98 [0.82–1.1]
Eye or vision problems	801	4	15	8	7.8 [7.3–8.4]	1.5 [1.3–1.6]	2.1 [1.8–2.5]
Diarrhoea	520	5	49	86	5.1 [4.7–5.5]	0.6 [0.5–0.7]	0.28 [0.19–0.34]
Vomiting or nausea	504	6	6	89	4.9 [4.5–5.4]	2.9 [2.6–3.1]	0.26 [0.18–0.34]
Sore throat	457	7	14	29	4.5 [4.1–4.9]	1.5 [1.4–1.7]	0.85 [0.72–0.98]
Chest pain or discomfort	449	8	2	22	4.4 [4.0–4.8]	6.1 [5.7–6.5]	1.0 [0.77–1.2]

For the 20,230 calls with insurance data available, the age group 18–39 years had the highest proportion of calls (28%), followed by newborn to 3 years of age (25%), 3–17 (24%), 40–64 (19%), and 65 years and older (4%). Sixty percent of all calls to AMC concerned females.

The frequencies of hospitalization were greater for ED visits based on the NHAMCS data than for symptom assessment calls to AMC in every age group (Table 1). However, hospitalizations were much more common with symptom assessment calls to AMC than with office visits based on the NAMCS data. Odds ratios comparing hospitalizations with these different forms of medical access reveal that the chance of being hospitalized are about 3–5 times greater for those attending an ED than for callers to the telephone triage call centre.

Hospitalization following calls usually occurred quickly. Of those hospitalized within one week of a call, 46% occurred within 24 hours and 77% occurred within 48 hours (Figure 1). The odds ratio of hospitalization from the ED compared to hospitalization following a triage call showed some increase with age as well. Those aged 65 years and older were 5 times more likely to have problems requiring hospital admission when presenting to the ED compared to a telephone call. Children aged 3–17 years were only 2.7 times more likely to be hospitalized when they presented to an ED compared to an AMC call. Symptom calls in the 65 years and older age group had hospitalization rates close to 10%, which exceeded hospitalization rates for ED visits for those below age 40 years (Table 1).

When analyzed by symptom type, there was significant variation in how closely the calls matched the hospitalization rate of ED visits. For example, calls about adult abdominal pain had a rate of hospitalization similar to that of an ED visit. The odds ratio of ED hospitalization for adult abdominal pain compared to triage call hospitalization was 1.6 (Table 3). On the other hand there were symptoms where hospitalization was much less likely to occur following a call than for an ED visit. For the symptom of diarrhoea in an adult, the odds ratio was 19 for hospitalization following an ED visit compared to a triage call. The confidence intervals on some of the odds ratios were wide but for chest pain and abdominal pain the upper limit was 3.5. For the two most common sources of hospitalization (chest and abdominal pain), visits to the ED were only a few times more likely to result in hospitalization than a call to AMC.

Table 3

Comparison of hospitalization frequencies and odds ratios of NHAMCS ED visits to Ask Mayo Clinic visits by specific symptoms

	Hospitalization frequency (%) [95% CI]		Odds ratio of hospitalization ED to AMC [95% CI]
Call symptom	AMC	ED visits	Unadjusted	Adjusted*
Chest pain (adult age 18 years and over)	13 [10–16]	33 [30–37]	3.5 [2.6–4.6]	2.5 [1.8–3.5]
Abdominal pain (age 3–7 years)	6 [4–9]	8 [4–11]	1.3 [.7–2.4]	1.4 [.66–2.8]
Abdominal pain (adult)	13 [11–16]	19 [17–21]	1.6 [1.2–1.9]	1.7 [1.3–2.1]
Symptom call age less than 3 years	1.8 [1.5–2.2]	6.3 [5–7.6]	3.6 [2.8–4.6]	4.2 [3.1–5.8]
Skin concern (age 3–17 years)	0.88 [0.4–1.9]	1.1 [0.62–2.3]	1.3 [.31–5.1]	1.3 [.14 –11]
Skin (adult)	0.98 [0.5–1.8]	6 [4.1–8]	6.5 [3.1–14]	6.7 [2.9–16]
Diarrhoea (adult)	1.5 [0.78–3]	27 [19–35]	23 [10–51]	19 [8–48]
Breathing concern (age 3–17 years)	6.1 [2.4–14.6]	10 [5–15]	1.7 [.56–5.4]	1.6 [0.4–6.4]
Breathing concern (adult)	12 [7–18]	37 [33–41]	4.5 [2.6–7.8]	2.6 [1.4–4.7]
Pregnancy concern	5.4 [3.1–9.3]	16 [11–22]	3.2 [1.7–5.9]	2.8 [1.3–6.4]
Vomiting and nausea (age 3–17 years)	2.4 [1.2–4.6]	7.1 [4–10]	3.2 [1.3–7.5]	3.8 [1.4–11]
Vomiting and nausea (adult)	6.3 [4.5–8.8]	21 [18–25]	4.0 [2.7–5.9]	3.6 [2.3–5.7]

*Adjusted for age, sex and private insurance

The proportion of calls about adult females was significantly different from ED and office visits. Females accounted for 72% (95% CI: 71 to 73) of all adult calls but only 61% (95% CI: 60 to 62) of adult office visits and 56% (95% CI: 55 to 56) of adult ED visits. Hospitalization odds ratios by symptom were adjusted for sex to account for these differences between populations (Table 3). The paediatric calls (ages 0–17 years) were similar in their gender mix compared to the paediatric office and ED visits. Females accounted for 48% (95% CI: 47 to 49) of all paediatric calls, 49% (95% CI: 47 to 50) of paediatric office visits and 48% (95% CI: 47 to 49) of paediatric ED visits.

Discussion

To our knowledge, this is the first study to compare severity of telephone triage calls versus other types of medical access. Our study indicates that triage centre calls are more likely to end in hospitalization than an office visit, but less likely to end in hospitalization than an ED visit. In fact, a visit to a typical US ED is 3–5 times more likely to result in hospitalization than an AMC call. In comparison, the odds of hospitalization following a call to our triage centre were up to 20 times greater than that of a typical office visit.

Of those who had a searchable claims record, triage calls resulted in hospitalization in 3% of cases within 72 hours of the call. This information is important to both triage centre training and employers who subscribe to such centres. The odds of hospital admissions increased with increasing age, and this should be taken into account during triage training and algorithm development.

Qualitatively, AMC calls had more similarities to ED visits than outpatient visits. The top eight reasons for both ED visits and AMC calls were because of symptoms, but only one of the top eight reasons for an adult office visit was about symptoms. Of the top eight reasons for an AMC call, four were also in the top eight reasons for an ED visit, but there were only two shared reasons for visits among the top eight reasons for office visits and AMC visits (Table 2). If ED visits are reasonably similar to calls then algorithm designers can use information from the ED datasets as a starting point for algorithm design. In addition, a national ED dataset may also benchmark how sensitive the telephone triage algorithms need to be. For example, a symptom resulting in 30% subsequent hospitalizations on ED visits would require more urgent endpoints than a symptom resulting in 3% hospitalizations.

One of the strengths of the present study is that we used hospitalization data to judge the severity of the problem addressed. Hospitalization is a widely accepted outcome measure for judging the severity of a medical problem and, in the case of AMC, hospitalization is not likely to be influenced by the triage recommendation. Physicians making admitting decisions about patients did not have AMC records to influence them. One problem in using hospitalization data is that hospitalization itself can be an adverse event if a delay in diagnosis from triage results in hospitalization. To detect this requires a medical record review which we did not do but it appears to happen very uncommonly.¹¹ As new quantitative measures are developed to assess the comparative quality and safety of triage, we will need quantitative outcomes such as subsequent hospitalization to compare the underlying populations in terms of pre-call risk. Comparison of adverse events associated with triage can then be adjusted for the underlying pre-call severity of illness. At present, we know that the calls and the underlying severity of illness for the AMC call centre were more similar to ED visits than to office visits. In future, we would like to be able to compare severity of calls with other call centres so that we can develop a better basis for comparing risks of triage with different systems.

Comparison of calls where insurance data was available to those without insurance data revealed some significant differences and may limit the ability to generalize our findings to all AMC calls. Women were the subject of 60% of calls where insurance data was obtainable but accounted for 63% of the calls where insurance data was not obtainable (P <0.0001). The age distributions were also significantly different between the calls where insurance information was and was not available. Adults over age 17 years were the subject of 52% of the calls where insurance data was available but accounted for 64% where insurance data could not be obtained (P < 0.0001). Another limitation was our inability to adjust the hospitalizations for all potentially important demographic factors. Although we were able to adjust odds ratios for age, sex and private insurance (Table 3), our call and insurance data lacked the race, ethnic and socioeconomic information required for a more comprehensive comparison between ED and office visits.

It has been stated that triage systems are safe.^6,12 However, the appearance of safety could simply be the result of an underlying calling population with low risk for adverse events.⁸ Under such circumstances Car et al. argued that sample sizes may not have been sufficiently large to show an effect. Our results show that some subgroups of calls have hospitalization rates sometimes similar to that of ED visits. However, this alone does not refute Car et al.'s argument. In fact, we did find evidence that sample sizes need to be quite high in order to make inferences about death associated with triage. For example, using the national ED visit data for deaths following hospitalization (where discharge status is death) we found death to be a quite rare event even for some populations using the ED. For the 17,390 paediatric ED visits sampled in the 2005 and 2006 NHAMCS dataset, there were only two deaths identified following the 770 hospital admissions, i.e. an estimated death rate per hospitalized child of 0.26% (95% CI: 0.0 to 0.63). Adjusting for the AMC paediatric hospitalization rate of 1.7%, it would take 45,000 AMC paediatric calls to generate a similar 770 hospital admissions. Since the confidence level of the ED data contains 0, even this number of calls with no deaths occurring would not distinguish it from the ED visits at the P < 0.05 level. So comparing deaths with this group of patients is problematic given the large numbers needed. Death rates for adults are significantly higher for the national ED data so comparisons using death as an outcome in adults may be less daunting. The estimated death rate per hospitalized adult for abdominal pain ED visits is 1.8% (95% CI: 0.8 to 2.8) and for chest pain it is 1.1% (95% CI: 0.5 to 1.7). Adjusting for hospitalization we would need a sample of about 1600 adult abdominal pain calls or 2700 adult chest pain calls without a single death to demonstrate that deaths associated with calls are fewer than deaths with ED visits. The total number of adult calls needed to get those values are 18,000 for abdominal pain and 61,000 for chest pain. This comparison to ED visits (where hospitalization is more common) suggests that comparison of AMC to other triage centres would need even greater sample sizes to make conclusions about significant differences in adverse outcomes such as death.

Compared to patient office visits, calls to a triage centre are more likely to result in hospitalization, but are less likely than ED visits. For some subgroups of triage calls, hospitalization is nearly as frequent as following an ED visit. More research is necessary to determine how the severity of illness associated with the call varies by call centre and call type. Adverse events associated with triage should be considered in the context of how serious the symptoms are underlying the call, perhaps using subsequent hospitalization to adjust for differences in caller populations. Despite our results showing that some subgroups of calls have significant rates of hospitalization, sample sizes necessary to distinguish between call centres on adverse events such as death may be in the tens of thousands.

References

North

, Varkey

A retrospective study of adult telephone triage calls in a US call centre. J Telemed Telecare 2009;15:165–70

O'Cathain

, Sampson

, Munro

, Thomas

, Nicholl

. Nurses' views of using computerized decision support software in NHS Direct. J Adv Nurs 2004;45:280–6

Giesen

, Ferwerda

, Tijssen

Safety of telephone triage in general practitioner cooperatives: do triage nurses correctly estimate urgency?

Qual Saf Health Care 2007;16:181–4

Derkx

, Rethans

, Muijtjens

, Quality of clinical aspects of call handling at Dutch out of hours centres: cross sectional national study. BMJ 2008;337:a1264

O'Cathain

, Webber

, Nicholl

, Munro

, Knowles

. NHS Direct: consistency of triage outcomes. Emerg Med J 2003;20:289–92

Kempe

, Bunik

, Ellis

How safe is triage by an after-hours telephone call center?

Pediatrics 2006;118:457–63

Lattimer

, George

, Thompson

, Safety and effectiveness of nurse telephone consultation in out of hours primary care: randomised controlled trial. The South Wiltshire Out of Hours Project (SWOOP) Group. BMJ 1998;317:1054–9

Car

, Koshy

, Bell

, Sheikh

. Telephone triage in out of hours call centres. BMJ 2008;337:a1167

10.

11.

Kempe

, Luberti

, Belman

, Outcomes associated with pediatric after-hours care by call centers: a multicenter study. Ambul Pediatr 2003;3:211–7

12.

Bunn

, Byrne

, Kendall

Telephone consultation and triage: effects on health care use and patient satisfaction. Cochrane Database Syst Rev 2004;CD004180