A cross-sectional study of the quality of telephone triage in a primary care out-of-hours service

Abstract

We evaluated the quality of telephone triage and the appropriateness of the decisions resulting from it at a primary care out-of-hours service. Four simulated clinical cases were used in the Incognito Standardized Patient method: an adult with nosebleed, an adult with fever, a child with fever and a child with vomiting. There was a set of obligatory questions for each case, translated from those used in a previous study. Quality was assessed by the proportion of questions asked by the call-handlers during telephone triage versus those that should have been asked. A total of 22 out-of-hours doctors were involved in the study, working in two different locations in the Verona city area. Over a 4-month period each of the four simulated clinical cases was used five times in calls to the two centres involved. The proportion of obligatory questions asked compared to those expected to be asked was 27–36%. On three occasions out of the 40 simulations, all of which were considered to be manageable by telephone, the patient was advised to go to an outpatient clinic for a face-to-face evaluation. The average duration of the calls was 3 min 47 s. The quality of telephone triage in the regions studied was low and provided considerable room for improvement. This is relevant to patient safety and risk management of the service.

Introduction

Telephone triage is the central part of a primary care out-of-hours service as it is the phase in which patient information is collected to determine the urgency of a medical problem and to decide on the care needed by the patient. There have been various studies of telephone triage in terms of security,^1–3 differences in performance between different health care personnel,⁴ effectiveness in reducing visits to general practitioners (GPs) and hospital referrals,^5–8 patient satisfaction^9,10 and the propensity of patients to follow the advice given to them.^11,12

Telephone triage should meet one of the criteria of quality which is to minimize the clinical risks that it entails. Thus it is important for the health care personnel involved in a primary care out-of-hours service to receive appropriate training and to have access to modern support tools. The quality of telephone triage and the appropriateness of the decisions made are complex, and depend on the communication skills, medical knowledge and experience of call-handlers in assessing the clinical information collected through telephone triage.^1,12,13 Recent studies have investigated the quality of telephone triage in primary care out-of-hours services in the Netherlands^1,14,15 and the UK,¹⁶ but there have not been any studies in Italy.

The aim of the present paper was to evaluate the validity of telephone triage by comparing the questions asked by call-handlers during telephone triage with those that should have been asked, and examining their decisions about therapy.¹

Methods

We evaluated the quality of telephone triage and the appropriateness of the decisions resulting from it, at two call centres in Verona serving Verona Province. In this service, the call-handlers were doctors. A total of 22 out-of-hours doctors were involved in the study, working in two different locations. Before the study began, the doctors were given an information letter about the study. All the doctors gave their consent to participate. The study was approved by the appropriate ethics committee and by the local public company providing out-of-hours care services (ULSS 20).

Clinical cases

Four standardized clinical cases were used: an adult with nosebleed, an adult with fever, a child with fever and a child with vomiting. There was a set of obligatory questions for each case, translated from those used in a previous study.¹ In the Italian translation, it was necessary to change the number of questions in order to express the same content (Table 1). We also used the expected decisions in terms of how the clinical cases should have been managed following the teleconsultation conducted by the primary care out-of-hours centre, proposed in a previous study.¹ The previous study used a focus group of GPs with experience in telephone triage to define the lists of questions for each case study, based on national protocols. Our clinical cases were chosen so that only advice via telephone was required and the patient (or parent) was advised to call back only if necessary. Different advice given by telephone triage call-handlers, such as advice to see their GP the next day, to contact the local health care centre to see a doctor or require a doctor’s visit at home, or obtain a hospital referral, was considered incorrect.

Table 1.

Obligatory questions to be asked in each case.

Question
Adult with nosebleed
1. Duration of nosebleed?
2. How many nosebleeds have you had in the past 48 hours?
3. How much blood has been lost?
4. Have you vomited any blood?
5. Has there been a blow to the head?
6. Are you feeling faint?
7. Might you have a foreign object in the nose?
8. Do you have bruises anywhere?
9. Do you have a bleeding disorder?
10. Do you take anticoagulant treatment?
Adult with fever
1. Did the fever start with shaking/rigour?
2. How high is the temperature?
3. Do you have a stiff neck?
4. Do you have any difficulties with breathing?
5. Is there any pain when passing urine?
6. How much fluid has been taken in the past 12 hours?
7. When did you last pass urine?
8. Any major long term health problem?
9. Did you recently travel abroad?
Child with vomiting
1. Can you describe the child’s behaviour now?
2. How often did the child vomit in the past six hours?
3. How much did the child vomit?
4. Did the child vomit blood?
5. Did the child drink in the past six hours?
6. When did the child pass urine the last time?
7. Did the child complain about pain when passing urine?
8. Does the child have a rash anywhere?
9. Does the child have fever?
10. Does the child complain about a headache?
11. Can the child touch forehead on knees(or kiss knees)?
12. Does the child complain about abdominal pain?
13. Does the child complain about photophobia?
14. Did the child have a head injury recently?
15. Did the child eat the wrong food?
Child with fever
1. Can you describe the child’s behaviour now?
2. How high is the temperature of the child?
3. Has the child had a fit?
4. Does the child have pain anywhere?
5. Has the child got, or has he or she had, a headache?
6. Can the child touch forehead on knees (or kiss knees)?
7. Does the child seem breathless or is there indrawing of the chest/tummy?
8. How much fluid has been taken in the past 12 hours?
9. When did the child last pass urine?
10. Does the child have a rash?
11. Did the child recently travel abroad?

Handling of clinical cases

The four cases were handled by primary care out-of-hours telephone triage staff in a four month study period beginning in June 2011. The cases were randomly proposed using the Incognito Standardized Patient (ISP) method.^14,17–20 One of the researchers would telephone incognito one of the out-of-hours services in the Verona city area and give information concerning an illness without revealing that the case was simulated for the purposes of study and was not real.¹

The clinical cases did not require the call-handler to advise a home visit or to call the emergency ambulance service. If the person answering the calls had given such advice, callers would have revealed their true identity immediately and the nature of the study. Over the 4-month period each of the four simulated clinical cases was used five times in calls to the two centres involved, i.e. there was a total of 40 calls. No sample size calculation was conducted before the study began.

Because of their monthly shifts, each telephone triage call-handler received about 1 call per month. The calls were made from 21:30 to 23:30 for night shifts and from 11:00 to 19:00 for day shifts, thus avoiding calling at times particularly inconvenient for the telephone triage staff (for example, late at night and at the times of shift change).

The researchers involved in the role of the Incognito Standardized Patient were three men and three women, aged 24–42 years. They received structured instructions, and were supervised during their performance in order to reduce heterogeneity in the presentation of the cases.

To reduce the chance of repeating a clinical case to the same doctor, we took into account the shifts of the telephone triage participants. When making the calls we switched between simulated patients of both sexes, changing the patient names given to the doctor for every telephone call. We also used as many different telephones as possible, because the number was visible to the doctor and was stored in an archive of previous calls.

The standardized clinical cases were administered with the support of a written framework where the ISP could take notes and flag the questions asked. All the calls were recorded and listened to again if necessary.

Outcome measures

For each call made we noted: (1) the proportion of compulsory questions the call-handler asked compared to the questions on the standard reference list; (2) the proportion of correct decisions made by the call-handler; and (3) the length of the call itself. The duration of the simulated call was measured from the moment the conversation began until the parties hung up.

We could not link the out-of-hours doctors' demographic data to performance data because ISPs were blind with regard to doctors' identity. Also, out-of-hours doctors do not usually introduce themselves at the beginning of telephone consultations and asking for the doctor's identity could suggest to the call-handler that the caller was not a real patient.

Results

An analysis of the percentage of obligatory questions asked compared to those expected to be asked revealed that it was consistently below 100% for each of the four clinical cases. In descending order, 36% of the questions that should have been asked in the clinical case of a child with vomiting, 32% in the case of a child with fever, 28% in the case of the adult with fever and 27% of the adult with nosebleed (Tables 2 –5).

Table 2.

Adult with nosebleed. Number of doctors (of 10) who asked the obligatory questions.

	Number
How many episodes in the last few hours?	6
How long ago did nosebleed end?	5
Amount of blood lost?	5
Any recent concussion?	4
Did or do you feel any object in your nose or nasal passages?	4
Medicine recently taken or currently being taken?	2
General feeling of weakness?	1
Any bruises on the skin?	0
Any problems concerning coagulation?	0
Any blood in vomit?	0

Table 3.

Adult with fever. Number of doctors (of 10) who asked the obligatory questions.

	Number
Any flu symptoms?	8
Duration of febrile state?	7
Highest temperature measured?	6
Any nausea, vomiting or diarrhoea?	4
Chills?	1
Any recent trips abroad?	1
Any problems urinating?	1
Any blotches on skin?	0
Any recent illnesses?	0
Any neck stiffness or neck pain?	0

Table 4.

Child with vomiting. Number of doctors (of 10) who asked the obligatory questions.

	Number
General conditions of the child?	9
Any stomach ache?	9
How many times has the child vomited?	8
Running a temperature?	5
Headache?	5
How much did the child vomit?	4
Any possibility of having eaten foods gone bad?	4
Has the child ingested liquids in the last 6 hrs?	4
Any recent head injuries?	3
Any back stiffness?	2
Is the child sensitive to light?	0
Any blotches on the skin?	0
Is urination painful?	0
Any blood visible in vomit?	0
Last time child urinated?	0

Table 5.

Child with fever. Number of doctors (of 10) who asked the obligatory questions.

	Number
Any flu symptoms?	8
How high is the temperature?	8
General conditions of child?	7
How long has fever lasted?	7
Any particular pains?	5
Headache?	4
Any back stiffness or pain?	2
How much liquid consumed in past few hours?	2
Any nausea/vomiting/diarrhoea?	2
Any blotches on skin?	1
Any recent trips abroad?	1
Stomach ache?	0
Has the child had a recent injury?	0
Last time child urinated?	0
How is child’s breathing?	0

There were a few questions that were important in determining the clinical severity of the patients' symptoms, such as for the adult with fever, eight doctors asked about flu symptoms but none asked about neck pain. The doctors tended to ask a few questions, probably because the symptoms described in the simulated cases were very common and in most situations they were unlikely to mask serious diseases. Both in the adult (Tables 2 and 3) and the child patients (Tables 4 and 5), no attention was given to symptoms or conditions which might signal a more severe disease, such as asking about travel abroad to exclude any tropical disease.

In analysing the management decisions we found, as expected, that no doctor advised calling the emergency service or requested a home visit to the patient, in any of the simulations. In only three clinical cases out of the 40 simulated (2 adults with fever and 1 adult with epistaxis), all of which were considered to be manageable by telephone, the patient was advised to go to an outpatient clinic for a face-to-face evaluation. In the remaining 37 cases, the call handler opted for telephone advice aimed at self-management of the patients (Table 6). The average duration of the 40 simulated calls was 3 min 47 s.

Table 6.

Management decisions concerning the 40 simulated cases.

	Number
Patients instructed to treat themselves	21
Patients instructed to treat themselves and call back if necessary	16
Patients advised to go to an outpatient clinic	3
Patient advised to call a doctor for a home visit	0
Patient advised to go to hospital	0
Total	40

At the end of the study we asked the doctors involved if they had suspected that the calls received were fake calls; no doctor gave a positive response.

Discussion

The results of the study showed that there were significant deficiencies in the telephone triage provided in the out-of-hours primary care service and, therefore, ample room for improvement. Indeed the study showed that the quality of telephone triage, in terms of mandatory questions asked during the call, was well below the desired standard, although the ability to assess the case, the clinical management decisions made and the treatment advised was almost always appropriate (it should be kept in mind, however, that the cases were not especially severe). The deficiencies found during triage might have been due to the call-handlers’ inexperience or lack of training, but were probably the result of fatigue resulting from night shifts, repetitiveness and the difficulty of dealing with the callers’ emotions and expectations together with the problem of making a remote evaluation of the seriousness and urgency of the case.

In comparing the results of the previous study conducted in the Netherlands and those of our study, involving the same simulated clinical cases, we found 70% versus 36% of mandatory questions asked in the case of a child with vomiting, 47% versus 32% in the case of the child with fever, 36% versus 28% in the case of the adult with fever, and 54% compared to 27% of the adult with nosebleed, respectively. A service in which only one-third of the questions necessary to make a correct decision is actually asked represents a high medical risk and could become a medicolegal burden on patients and doctors. Telephone triage is a delicate phase of a telephone consultation during which there is the risk that insufficient information is given by the caller resulting in an underestimation of the case, with potentially serious consequences for the patient, or an overestimation of the case with advice for inappropriate treatment that could also be economically disadvantageous. In other words, the quality of the telephone triage service and the risks that teleconsultations and telediagnoses entail are closely related.

The present study had several limitations: (1) the small size due to limited resources; (2) the use of clinical cases of mild severity, because the call-handler might have decided that a face-to-face consultation was required; (3) the complexity of the ISP method; (4) the lack of a comprehensive statistical analysis; (5) the limited generalizability; (6) the use of standardized clinical cases built by expert consensus, but not directly supported by evidence-based guidelines (because no evidence based clinical guideline provides questions to be administered in a telephone triage setting). Nonetheless, the study demonstrated that the model is feasible, and enabled us to assess to what extent and in what areas the quality of telephone triage concerning out-of-hours primary care requires improvement.

Several suggestions for future research emerged from the study. We chose not to call at times particularly inconvenient for the out-of-hours staff (e.g. late night and shift change). However, these times could be particularly important, as these are times when mistakes might be more likely to occur. Another question that emerges from the study regards the strategies that should be adopted in order to improve the quality of telephone triage. These include the effect of specific training, in particular training on management techniques and structuring telephone consultations,¹⁵ and the effectiveness, efficiency and security that computerized support could offer the telephone triage doctors in decision making. Previous studies have shown that decision-making tools are safe and can reduce the clinical risks associated with unassisted telephone triage call handlers, as well as being cost-effective and possibly reducing the number of unnecessary visits to emergency room services.^21,22

The results of the present study show that the quality of telephone triage in the regions studied was low and that there was room for improvement. The doctors answering the telephone asked a very low proportion of obligatory questions and this could affect patient safety and risk management for the service itself. Research is required to identify ways of improving the situation.

Footnotes

Acknowledgements

We are grateful to Hay Derkx (Maastricht University) for his collaboration.

References

Derkx

Rethans

Muijtjens

. Quality of clinical aspects of call handling at Dutch out of hours centres: cross sectional national study. BMJ 2008; 337: a1264–a1264.

Huibers

Smits

Renaud

Giesen

Wensing

. Safety of telephone triage in out-of-hours care: a systematic review. Scand J Prim Health Care 2011; 29: 198–209.

Altmeyer-Ennis

Littler

. The effective management of out of hours treatment in primary care using telephone clinical assessment software (TeleAssess). J R Army Med Corps 2010; 156: 212–3.

Murdoch

Barnes

Pooler

Lattimer

Fletcher

Campbell

. Question design in nurse-led and GP-led telephone triage for same-day appointment requests: a comparative investigation. BMJ Open 2014; 4: e004515–e004515.

Dunt

Wilson

Day

Kelaher

Gurrin

. Impact of telephone triage on emergency after hours GP Medicare usage: a time-series analysis. Aust New Zealand Health Policy 2007; 4: 21–21.

Raftery

. Nurse telephone consultation for out of hours primary care can save money through reduced ER admission, surgery attendance and GP home visits. Evidence-based Healthcare 2000; 4: 61–161.

Forrest

. Primary care in the United States: primary care gatekeeping and referrals: effective filter or failed experiment? BMJ 2003; 326: 692–5.

Fry

. Barriers and facilitators for successful after hours care model implementation: reducing ED utilisation. Australas Emerg Nurs J 2009; 12: 137–144.

Bunn

Byrne

Kendall

. Telephone consultation and triage: effects on health care use and patient satisfaction. Cochrane Database Syst Rev 2004; 4: CD004180–CD004180.

10.

Lee

Baraff

Wall

Guzy

Johnson

Woo

. Parental compliance with after hours telephone triage advice: nurse advice service versus on-call pediatricians. Clin Pediatr (Phila) 2003; 42: 613–9.

11.

Williams

Warren

McKim

Janzen

. Caller self-care decisions following teletriage advice. J Clin Nurs 2012; 21: 1041–50.

12.

Lee

Guzy

Johnson

Woo

Baraff

. Caller satisfaction with after-hours telephone advice: nurse advice service versus on-call pediatricians. Pediatrics 2002; 110: 865–72.

13.

Derkx

Rethans

Muijtjens

. ‘Quod scripsi, scripsi.' The quality of the report of telephone consultations at Dutch out-of-hours centres. Qual Saf Health Care 2010; 19: e1–e1.

14.

Derkx

Rethans

Maiburg

. Quality of communication during telephone triage at Dutch out-of-hours centres. Patient Educ Couns 2009; 74: 174–8.

15.

Derkx

Rethans

Knottnerus

Ram

. Assessing communication skills of clinical call handlers working at an out-of-hours centre: development of the RICE rating scale. Br J Gen Pract 2007; 57: 383–7.

16.

Campbell

Britten

Green

. The effectiveness and cost-effectiveness of telephone triage of patients requesting same day consultations in general practice: study protocol for a cluster randomised controlled trial comparing nurse-led and GP-led management systems (ESTEEM). Trials 2013; 14: 4–4.

17.

Rethans

Gorter

Bokken

Morrison

. Unannounced standardised patients in real practice: a systematic literature review. Med Educ 2007; 41: 537–49.

18.

Van der Vleuten

CPM

Swanson

. Assessment of clinical skills with standardized patients: state of the art. Teach Learn Med 1990; 2: 58–76.

19.

Rethans

Norcini

Barón-Maldonado

. The relationship between competence and performance: implications for assessing practice performance. Med Educ 2002; 36: 901–9.

20.

Moriarty

McLeod

Dowell

. Mystery shopping in health service evaluation. Br J Gen Pract 2003; 53: 942–6.

21.

Meer

Gwerder

Duembgen

Zumbrunnen

Zimmermann

. Is computer-assisted telephone triage safe? A prospective surveillance study in walk-in patients with non-life-threatening medical conditions. Emerg Med J 2012; 29: 124–8.

22.

Dale

Higgins

Williams

. Computer assisted assessment and advice for “non-serious” 999 ambulance service callers: the potential impact on ambulance despatch. Emerg Med J 2003; 20: 178–83.