What makes a high-quality electronic consultation (eConsult)? A nominal group study

Introduction

Coordinated patient care relies on effective communication between specialists and primary care providers (PCPs).¹ Higher PCP-reported levels of communication with specialists are associated with favourable patient outcomes such as lower hospitalization rates.² In contrast, poorly written communication may compromise patient safety and delay diagnoses while leading to inadequate follow-up and redundancies in resource utilization.³ Unfortunately, both PCPs and specialists are often dissatisfied with written communication received from one another and have different expectations as to what should be included in consultation letters.^1,4,5

If the goal is to improve communication, then an evaluation of the written correspondence with feedback to specialists would be a first step. To our knowledge, three tools have been developed to assess consultation letters. One tool is the Sheffield Assessment Instrument, a 20-point checklist with a global rating scale;⁶ applying the tool to paediatric consultation letters led to improved quality scores over a three-month period.⁷ Similarly, specialists using another tool consisting of a nine-point rating scale for peer assessment of internal medicine consultation letters reported positive changes to their letter-writing which were sustained six months after using the scale.⁸ Finally, Sewell et al.⁹ applied the 15-point Quality of Consultation Assessment Tool (QCAT)¹⁰ to derive a composite quality score to rate the assessment, plan and communication domains for ambulatory gastroenterology consultation letters. After consultants attended a QCAT-informed intervention to improve written consultancy skills, their consultation letter quality scores improved when compared to pre-intervention.⁹

In addition to challenges in specialist-to-PCP communication, increasing wait times to gain access to specialists remains a barrier for effective ambulatory care.¹¹ As a result, there are ongoing innovations to improve PCP access to specialist care. One example is the use of electronic consultations (eConsults) which have been described as ‘asynchronous, consultative, provider-to-provider communications within a shared electronic health record (EHR) or web-based platform.’¹² eConsults enable PCPs to submit patient-specific clinical inquiries to specialists who provide direct advice to the PCP without needing to meet the patient face-to-face in most cases.¹³ Upon receiving a PCP inquiry, eConsult specialists may (a) provide advice without requesting a face-to-face (F2F) consultation with the patient, (b) request that the PCP refer the patient for F2F consultation while providing advice that can be implemented before the F2F visit, or (c) ask for additional information from the PCP. Upon closing the eConsult, the PCP receives a summarised log detail of the eConsult that can be incorporated in the patient’s file for future reference. They also fill out a mandatory close-out survey to assess overall satisfaction and educational value of the eConsult while documenting its outcome, for example, whether the eConsult confirmed or provided a new course of action for the PCP, and if the PCP originally contemplated a F2F consultation. PCPs report mainly positive experiences with eConsult, citing timely advice received, enhanced communication between providers, and the educational benefits of eConsult, for example, advice received for a specific patient can be applied to future patient encounters.^12–15 Similarly, specialists report that eConsult can improve specialist-to-PCP communication while facilitating targeted education delivered to the PCP.^16,17

While health-care providers describe improved communication using eConsults, specific components of eConsult advice that distinguish between high and low quality communication remain unclear given the differences between F2F and eConsult correspondence. A F2F consultation letter follows an in-person clinical encounter between specialist and patient after a referral request by a health-care provider. The letter acts as documentation of information gathered by the specialist including the patient’s history of presenting illness, past medical history, social history, physical examination and relevant investigations as guided by clinical questions posed by the referring health-care provider (often the PCP). It concludes with a management plan which stipulates who is most responsible for providing ongoing care: the specialist with continued follow-up, shared between specialist and referring provider, or solely the referring provider without specialist follow-up. In contrast, since an eConsult does not involve a specialist-patient encounter, the PCP is responsible for all data-gathering and determining what data to share with the specialist. Thus, specialist-to-PCP communication via an eConsult focuses primarily on advice to the PCP, who in turn persists as the most responsible provider for carrying out specialist advice.²³

Although tools exist for rating written consultation letters, these do not necessarily capture elements that are unique to eConsults.^7,8,10 Key elements of a high-quality eConsult response must be identified before developing an assessment tool that can provide eConsult specialists direct feedback on their written correspondence. As worldwide implementation of eConsult grows with time – some institutions report three-fold increases in eConsult utilization over three years – such feedback is required to assure and sustain high-quality communication via eConsults.^17–19

The purpose of this study was to generate a list of key items that providers identify as important for high-quality specialist correspondence through eConsults.

Methods

We used the nominal group technique (NGT) to generate and prioritise items for a list of features representing high-quality eConsult advice. NGT is a formal consensus group method widely used in health-care research to synthesise expert opinions and enhance decision-making.²⁰ We selected NGT for this study for its ability to generate a large number of items while its F2F component allows for debate and clarification.²¹ Published recommendations on conducting consensus group methodology were followed to ensure rigour and reproducibility.²²

Nominal group session

A skilled moderator (SHM) with expertise in consensus methods led the session in October 2017. To begin, the lead author (CT) summarised the project to ensure that participants clearly understood the study rationale. After the moderator explained NGT procedures, participants were asked: ‘What items should be included in a tool to measure the quality of an eConsult?’ Each participant had 15 min to privately write down items. Using a flip chart, a research assistant transcribed one item from each participant in a round-robin format until no items remained. The moderator facilitated group discussion of each item; similar items were combined where appropriate.

The lead author then presented a summarised literature review on existing assessment tools for consultation letters (Table 1). This review was conducted to identify studies examining the use of educational instruments to improve specialist-to-PCP written communication. A search strategy (Supplemental material Appendix A) was applied to MEDLINE and EMBASE databases. Articles were included if they reported an instrument assessing the quality of correspondence sent from specialists to PCPs, and its impact on future specialist-to-PCP communication. Reference lists of included articles were reviewed for relevant articles not covered by the search strategy. Conference abstracts were excluded. Three studies met the inclusion criteria (Supplemental material Appendix B).^7,8,10 A similar search incorporating eConsult-related keywords and applied to EMBASE only (Supplemental material Appendix C) did not reveal articles meeting the inclusion criteria.

OPEN IN VIEWER

Table 1.

Summary of literature review of existing assessment tools for consultation letters. After the round-robin stage (see text), nominal group technique (NGT) participants were presented with abbreviated versions of the Sheffield Assessment Instrument for Letters (SAIL) by Fox et al.,⁷ the consultation letter rating scale by Keely et al.⁸ and the Quality of Consultation Assessment Tool (QCAT) by Tuot et al.¹⁰

Item	SAIL7	Keely8	QCAT10	Generated by NGT participants prior to literature review
Referring PCP question(s) addressed	X		X	X
Differential diagnosis			X
Clear management plan	X	X	X	X
Rationale for:
Investigations	X		X	X
Treatment (±evidence-based)			X	X
Anticipatory guidance			X
Specifies need for follow-up	X			X
Written communication
Brevity		X
Clarity	X	X		X
Organization		X
Educational value		X	X	X
Global rating	X	X	X

PCP: primary care provider.

We presented the summarised literature review after the round robin to avoid biasing participants during initial item generation. A final review of items allowed participants to add, combine or remove items as guided by group discussion. The research assistant compiled and organised all items onto paper documents that each participant used to anonymously rank each item on a nine-point scale: 1–3 not essential, 4–6 neutral, 7–9 essential to include.

Rankings were tabulated and presented for group discussion. We determined a priori that an item would be included if ≥70% of participants ranked the item as 7–9 with no more than two participants rankings as 1–3. Likewise, if an item received ≥70% of participant rankings as 1–3 with fewer than two rankings 7–9, it would be excluded. All other scenarios would not meet criteria for consensus and thus would require additional discussion prior to re-ranking. We anticipated three rounds of ranking. If more than 20 items remained after the second round, the third round would be used to prioritise a top 10 list of remaining items.

Results

Three-hundred and three PCPs met inclusion criteria. Twelve PCPs agreed to participate, nine confirmed, and eight attended the three-hour NGT session. Eight specialists were contacted; three confirmed and attended the session. Among PCPs, there were six family physicians (five in physician group practices, one solo practice) and two nurse practitioners (both from community health centres) serving six distinct neighbourhoods within Ottawa, Canada. Specialists were econsultants for dermatology (solo practice), haematology (community group practice), and paediatric orthopaedics (academic centre).

Figure 1 summarises the NGT item generation process. Round-robin discussion produced an initial list of 49 items (items listed in Supplemental material Appendix D). After presenting the summarised literature review (Table 1), an additional item that was not generated from round-robin discussion (anticipatory guidance) was added after discussion among NGT participants; this was done independently of the study investigators. Further moderator-guided group discussion allowed items to be combined or removed where redundant, resulting in a list of 29 items for first iteration of ranking.

OPEN IN VIEWER

Figure 1.

Flow diagram of item generation using nominal group technique.

Table 2 displays results of the initial ranking. Using consensus criteria determined a priori, 18 items were included, 11 items were listed for re-rank and discussion and no items were excluded. Among 319 total rankings, 225 were 7–9 (71%), 79 were 4–6 (25%), and 15 (4.7%) were 1–3; 10 of these ‘exclude’ ratings were submitted by two specialist raters.

OPEN IN VIEWER

Table 2.

Results after first round of item ranking and subsequent group discussion.

After private ranking							After group discussion
#	Item description	Mean a	1–3	4–6	7–9	Resultb	Resultc
1	Specific management details, e.g. work-up, red flags	8.73	0	0	11	Include	Combine with #16
2	Clarity	8.55	0	0	11	Include	Keep as is
3	Current: up-to-date, meets standard of care	8.45	0	1	10	Include	Keep as is
4	Relevant	8.36	0	1	10	Include	Combine with #9 and #11
5	Respectful: professional, kind tone	8.27	0	2	9	Include	Keep as is
6	Anticipatory guidance	8.18	0	0	11	Include	Keep as is
7	Specific treatment details: e.g. medication dosage, titration, monitoring, cost	8.09	0	1	10	Include	Keep as is
8	Actionable advice: realistic and doable	8.09	0	0	11	Include	Combine with #15
9	Complete, fully-read, all points addressed	7.91	0	1	10	Include	Combine with #4 and #11
10	When to refer/is an in-person consultation required	7.91	1	0	10	Include	Keep as is
11	Patient-specific, personalised advice	7.73	0	1	10	Include	Combine with #4 and #9
12	Medications that are readily available	7.55	0	2	9	Include	Addressed by #7 and #8
13	Openness to further communication, dialogue	7.45	0	0	11	Include	Keep as is
14	Educational: interaction is a learning experience	7.36	0	3	8	Include	Keep as is
15	Reference to local resources	7.09	0	2	9	Include	Combine with #8
16	Additional investigations required	7.00	0	3	8	Include	Combine with #1
17	Timeliness	6.91	1	2	8	Include	Keep as is
18	Advice for immediate action and information for later	6.45	1	2	8	Include	Combine with #24
19	Multiple management options provided	6.91	0	4	7	Re-rank	Remove
20	Grammar, spelling	6.55	0	6	5	Re-rank	Remove
21	Feedback given on quality of images provided by PCP	6.27	1	3	7	Re-rank	Remove
22	Response reflects quality of data provided by PCP	6.27	0	6	5	Re-rank	Combine with #25
23	Patient-shareable language	6.09	1	7	3	Re-rank	Remove
24	Additional materials provided	6.00	1	6	4	Re-rank	Combine with #18
25	Specialist seeks out clarification from PCP	5.82	1	6	4	Re-rank	Combine with #22
26	Acknowledgment of a difficult case	5.82	1	5	5	Re-rank	Remove
27	Patient satisfaction	5.55	2	5	4	Re-rank	Remove
28	Consistency between specialists	5.36	3	4	4	Re-rank	Remove
29	Answer changes PCP practice	5.27	2	6	3	Re-rank	Remove

PCP: primary care provider.

^aMean scores of 11 rankings for each item on a scale from 1–9; ^baccording to a priori criteria (see text), items were either included for final list, set aside for re-ranking or excluded entirely; ^cin lieu of a second round of ranking, items were further refined via group discussion.

The initial intent after the first round was to discuss only items flagged for re-ranking. However, it became apparent that further refinement and clarification was required as some of the 11 re-rank items appeared to overlap with the 18 include items. Also, a second round of ranking was unlikely to exclude further items as less than 5% of total ratings by participants were 1–3. Although our intention was to strictly follow the NGT process as determined a priori to maximise methodological rigour, modifications were made following the first round of ranking considering the above results.

Instead of a second round of ranking, each of the 29 items was discussed and either left alone, combined or removed as felt appropriate by the group as shown in Table 2. Among the 18 items initially meeting consensus for inclusion, one was removed – ‘Available medication’ – as the idea was represented by the existing items ‘Medication’ and ‘Realistic’, for example, specialist advice that is actionable or doable for PCPs. Of the 11 items requiring re-ranking, three were combined with existing items and eight were removed as on further discussion they were deemed not relevant. For example, items that were quickly dismissed as irrelevant to the purpose of the tool included: grammar/spelling, quality of images, language that could be shared with patient, and patient satisfaction. Others were described as not core to the purpose: acknowledgement of a difficult case and whether the answer changed PCP practice. During this process, the moderator was careful to seek out individuals who had initially proposed the item to ensure that agreement was unanimous and all views were represented.

The result was a revised list of 14 items (Table 3); it is worth noting that any items that were ultimately removed outright following group discussion did not meet a priori criteria for final inclusion following the first round. In order to prioritise items, participants were asked to select their top 10 items, with the most important listed as 10 and the least as one. Ratings from one participant were incomplete and thus excluded.

OPEN IN VIEWER

Table 3.

Prioritization of items generated by nominal group technique. Participants were given the list of 14 items below and asked to rank 10 items from 1–10, where 10 was the item they wanted most included. Ratings from one participant were incomplete and thus excluded.

Ranking	Description	Mean score
1	Specific management details, e.g. work-up, red flags	8.2
2	All concerns fully addressed with relevant, patient-specific and personalised advice	8.1
3	Current: up-to-date, meets standard of care	8.1
4	Specific treatment details: medication dosage, titration, monitoring, cost, availability	6.1
5	Actionable advice (realistic and doable) with references to local resources	5.2
6	Clarity	4.3
7	Anticipatory guidance	3.7
8	Respectful: professional, kind tone	2.5
9	Timeliness	2.5
10	When to refer: is an in-person consultation required	2.4
11	Educational: interaction is a learning experience	1.4
12	Separate advice for immediate action and additional material for future reference	1.3
13	Openness to further communication, dialogue	0.9
14	Specialist seeks clarification from PCP if inadequate data provided	0.3

PCP: primary care provider.

Discussion

Our NGT approach to identifying key components of high-quality specialist advice generated features pertinent to eConsults. While many items had overlapping features with those previously described by rating scales assessing conventional F2F consultation letters, other items were more applicable to the eConsult setting. These findings reflect inherent similarities and differences in specialist-to-PCP communication done via F2F consultation letters versus eConsults; whereas in both cases the specialist provides patient-specific advice via written correspondence, the PCP is nearly always the most responsible provider for implementing advice if received through an eConsult.²³

The highest-ranked NGT-generated item called for specific management details. Although some F2F consultation letter rating scales include ‘management’ as a heading, no additional guidance is given other than ‘clear plan of investigation’.^7,8 Often following F2F visits, it is the specialist rather than the PCP who is most responsible for carrying out recommended advice, for example, ordering investigations and prescribing medication. Thus, if the specialist wishes to defer responsibility to the PCP, then this should be made explicit in an F2F letter. In contrast, since the PCP remains solely responsible for carrying out specialist advice via eConsult, it makes sense to encourage eConsult specialists to provide a step-by-step approach for patient workup while identifying red flags, that is, specific adverse outcomes to take action on if they occur. The next highest-ranked item was to address all PCP concerns using advice individualised to the patient in question. While rating scales for F2F consultation letters encourage all key issues to be identified and addressed, none included patient-specific advice as a quality item.^7,8,10 As NGT participants voiced displeasure at receiving ‘copy and paste’ responses, eConsult specialists can be encouraged to cater their responses based on all available patient information, particularly if more than one management option is available. While some clinical questions are frequently asked via eConsult, PCPs want more than a generic approach to a common clinical scenario – the nuances of a particular case and how they specifically relate to each individual patient must be acknowledged.

The highest-ranked item after final top 10 ranking that was generated by NGT discussion and not previously described by F2F consultation letter rating scales was actionable or doable advice. While traditional F2F consultation letters do not always indicate whether specific management or treatment has been actioned by the specialist, in an eConsult it is made explicit that the PCP remains the most responsible for discussing, implementing and following-up on specialist recommendations.²⁴ Since the specialist does not take direct responsibility of the patient through eConsult, it seems natural that PCPs would prefer advice that they can implement on their own using readily available resources. For example, while it is reasonable for specialists to recommend common laboratory or imaging studies, it is not practical to ask PCPs to carry out procedures requiring specialist expertise (e.g. intra-articular platelet-rich plasma injections for musculoskeletal injuries). Also, specialists should not expect that all PCPs can independently organise parental therapeutics (e.g. intravenous iron for iron deficiency anaemia or intravenous bisphosphonates for osteoporosis).

Another item generated via NGT as more applicable to eConsults over F2F correspondence is ‘when to refer/is an in-person consultation required’. While two-thirds of eConsult cases can be resolved without a F2F specialist visit and PCPs who use eConsults have lower referral rates compared to those that do not, specialists are not explicitly instructed to try and avoid eConsult conversion to F2F referral.^25,26 Based on NGT discussion, specialists should be encouraged to outline specific scenarios when the eConsult should be converted to a F2F referral where applicable. While NGT participants did not prioritise this particular item as highly as others, this item remains applicable to eConsults; PCPs originally contemplate a F2F referral request in over 60% of cases submitted to our eConsult service, whereas the PCP overtly asks the eConsult specialist if a F2F clinic visit is needed in 18% of initial PCP inquiries.²⁷

A third item generated by NGT not previously found in published F2F consultation letter-rating scales was using respectful language which promotes collegiality and open dialogue. eConsult exchanges have been identified as a way to improve collegiality between primary and specialist providers.^28,29 They provide the option for both PCPs and specialists to ask additional questions to each other through sustained asynchronous discussion, although in our experience this is only used in 10% of cases. Following top 10 ranking, this item did not place as highly as others, reflecting how back-and-forth communication occurs uncommonly and how PCPs favour other aspects when reporting satisfaction with eConsult.²⁹

In contrast, items found in traditional F2F consultation letter rating scales that were not generated by NGT discussion included items pertaining to syntax and concise use of language: grammar, spelling, brevity, summary of impression and plan, and overall organization of the eConsult advice. While PCPs may favour succinct consultation notes following F2F visits, NGT participants preferred that specialists adopt a ‘don’t hold anything back’ approach when providing eConsult advice. This way, a PCP can appreciate the specialist’s thought and decision-making processes. The more advice given, the more opportunities for the PCP to understand the rationale behind the specialist’s advice while acquiring new knowledge to apply to future patients. With the organization of the eConsult itself not prioritised by NGT participants, specialists can focus more on providing specific advice to the PCP while spending less time on technical details such as text formatting and summarising details already available to the PCP such as patient medical history, family history and social history; these details are often not considered ‘value-added service’ to either PCPs or their patients.³⁰

Next steps

The key eConsult reply components generated through this robust strategy will now be used to create resources that can be used for training specialists providing eConsults and creating an instrument that may be useful for evaluating quality of responses. After establishing feasibility, such an instrument can undergo a larger scale validation study. For example, additional PCPs and specialists can be recruited to rate eConsults in real-time (e.g. while they are being responded to and read), along with a formal assessment of inter-rater variability and generalisability using the instrument. The quality of specialist response can be explored as a potential explanatory variable for eConsult outcomes such as PCP satisfaction, change in PCP practice behaviour, and referral outcomes such as avoided face-to-face referrals.

Supplemental Material