Abstract
Background:
The COVID-19 pandemic has exacerbated wait times for pediatric specialty care. Transformative technologies such as electronic referral (eReferral—automation of patient information) and electronic consultations (eConsult—asynchronous request for specialized advice by primary care providers) have the potential to increase timely access to specialist care. The objective of this study was to present an overview of the current state and characteristics of referrals directed to a pediatric ambulatory medical surgery center, with an emphasis on the innovative use of an eConsult system and to indicate key considerations for system improvement.
Methods:
This cross-sectional study was conducted at a specialized pediatric acute care hospital in Ottawa, Ontario. Secondary data were obtained over a 2-year period during the COVID-19 pandemic (2019–2022). To gain insights and identify areas of improvement related to the factors pertaining to referrals and eConsults at the process and system levels, quality improvement (QI) methodologies were employed. Descriptive statistics provide a summary of the trends and characteristics of referrals and the utilization of eConsult.
Results:
Among the 113,790 referrals received, 31,430 were denied. Most common reasons for referral denial were other/null (e.g., unspecified) (29.3%), inappropriate referrals (12.6%), and duplicate referrals (12.4%). Four clinics (e.g., endocrinology, cardiology, neurology, and neurosurgery) reported a total of 277 eConsults, with endocrinology accounting for 95.0% of all eConsults. QI findings revealed the need for standardized workflows among specialties and ensuring that eConsult options are accessible and integrated within the electronic medical record (EMR).
Conclusions
: Refining the pediatric referral management process and optimizing eConsult through existing clinical systems have the potential to improve the timeliness and quality of specialty care. The results inform future research initiatives targeting improved access to pediatric specialty care and serve as a benchmark for hospitals utilizing EMRs and eConsult.
Introduction
The provision of timely care is paramount in the pediatric context. Accumulating evidence suggests that early identification and management of medical conditions can impact children and youth's quality of life and health trajectories. 1 –3 However, delayed access to specialty care can lead to adverse complications, including developmental delays, psychosocial implications, and increased costs for families. 2,4 By promptly treating pediatric conditions, long-term negative impacts for patients and their families may be preventable. 4,5
Recent electronic health innovations have exemplified promising alternatives for improving referral efficiency and increasing access to care, such as electronic referrals (eReferral) and electronic consultation (eConsult). 6 –8 eReferral represents the reporting of patient appointment details and consultation information between providers through automated systems. 6 eConsult is an asynchronous electronic tool that allows primary care providers (PCPs) to communicate with specialists regarding patients' care needs. 8 eConsult utilization has been suggested to increase referral efficiency by reducing the rate of inappropriate referrals when integrated into existing referral systems. 9,10
In the context of Canada, long wait times are frequent sources of frustration among patients and clinicians. 8,11 The COVID-19 pandemic has further magnified patient waiting times, causing significant delays in specialty services. 12,13 During the peak waves of the pandemic, many hospitals in Canada had to postpone nonessential surgeries, resulting in a substantial backlog of services. 14,15 Such cancellation policies have also impacted pediatric health care, preventing many children and youth from receiving specialty services. 13,16 –18 Reports have suggested that children and youth across Ontario are waiting longer than adults for medically necessary care, 19 further highlighting the importance of prioritizing pediatric care within the Canadian referral system.
Currently, knowledge gaps exist for exploring health care utilization patterns by pediatric referrals and eConsult processes within electronic medical record (EMR) systems. To address this gap, we conducted a study that examined and analyzed an eConsult system used at the Children's Hospital of Eastern Ontario (CHEO), one of the few standalone pediatric hospitals in Canada. Specifically, we (1) provide an overview of the volume and characteristics of referrals directed to the Ambulatory Medical Surgery (MedSurg) clinics to determine their capacity and role in improving access to pediatric services; (2) describe the broader contextual challenges with the current referral system to inform similar efforts in other hospitals; and (3) recommend avenues for referral and eConsult system improvement using quality improvement (QI) methodologies that can act as a benchmark for future initiatives.
Methods
STUDY SETTING
CHEO is one of only two specialized acute care pediatric hospitals in the province of Ontario. It is located in the capital Ottawa and serves a catchment of over 500,000 children and youth per year, aged 0–17 years, from the Champlain region, Western Quebec, Regions of Nunavut, and Northern Ontario. 20 Employing over 3,000 staff and health care professionals, CHEO provides a broad range of pediatric inpatient and outpatient specialist services across 8 departments and 38 subspecialties. 20
OVERVIEW OF THE TECHNOLOGY/SYSTEM
CHEO receives referrals in two routes, including conventional referral methods (e.g., PCP submits request via fax/mail) or through an integrated EMR referral platform (e.g., PCP submits request via EpicCare Link). Once the referral is received, all referral management and processing are supported by the Epic EMR system (Epic Systems, Verona, WI). EpicCare Link is a HIPPA-compliant extension of Epic that enables PCPs to electronically refer patients to CHEO and grants selected access to patient information. 21,22 As a referral platform, EpicCare Link aims to remove manual redundancies in the referral process while also improving care continuity by increasing digital communication between providers. 23 It is important to note that the uptake of EpicCare Link for referral management at CHEO is currently limited. This could be attributed to the need for providers to log into another platform in addition to their local EMR.
Regardless of the route, referrals are triaged and deemed to be authorized, denied, incomplete, or redirected based on the workflow of the respective clinic (Fig. 1). Authorized referrals are appropriate requests that can proceed to be scheduled as an appointment, based on an assigned priority (1: seen within 1 week; 2: seen within 30 days; 3: seen within 90 days; 4: seen within 6 months). Denied referrals are primarily requests that are duplicates or those deemed unsuitable to the respective clinic. Referrals that are missing key patient information are deemed to be incomplete referrals. Finally, redirected referrals occur when a new department is selected during triage.

Hospital referral and eConsult process. Overview of the referral and eConsult process at CHEO. Describes the general procedure of patients being referred to CHEO specialists, and the steps taken after the referral is received. CHEO, Children's Hospital of Eastern Ontario; PCPs, primary care providers.
Specialists can specify the reason for referral denial in the EMR. However, during this process, specialists may choose to electronically request additional information based on the information provided or respond with specialized advice and guidance for the requesting PCP through the EMR. In this context, specialists are performing eConsults; however, it is EMR-initiated and occurs outside the regional/provincial eConsult platforms existing in Ontario. In addition, specialists at CHEO can receive internal eConsult requests from colleagues via electronic platforms (e.g., email, Epic communication). After providing consultations through secure electronic servers (e.g., EMRs), clinicians can bill for their services through the Ontario Health Insurance Plan Schedule of Benefits. Two fee codes exist for eConsults: one for clinicians requesting an eConsult (K738) and one for clinicians providing an eConsult (K739). 24
STUDY DESIGN
Using a cross-sectional design involving secondary data analysis, we examined the current state and characteristics of referrals and the utilization of eConsult. A QI lens was also applied to gain a larger understanding of the referral process and system-level considerations through stakeholder engagement and process map development.
DATA SOURCES
All referrals and eConsult billing codes directed to CHEO's MedSurg clinics, which frequently receive the majority of referrals, were identified using the hospital's EMR system between June 1, 2019, and June 30, 2022. The volume of referrals received and denied per specialty clinic, the reason for referral denial, the rate of scheduling appointments, and the volume of eConsults provided by billing codes (K738/K739) were generated from the EMR.
Various key stakeholders were engaged to gain insight into the referral process and the acceptability of EMR-generated eConsults and to identify areas for improvement. These stakeholders included clinicians in ambulatory care, Epic analysts, a committee focused on standardizing referral workflows, and QI advisors. Process mapping was used to identify and analyze the steps in the current referral workflow to inform opportunities for eConsult integration.
DATA ANALYSIS
Descriptive statistics (frequencies and percentages) were reported for categorical variables. Continuous variables not following a normal distribution were reported using medians and interquartile ranges (IQR). Since the rate of scheduling appointments demonstrated wide variation (0–1,176 days), all extreme outliers were removed (i.e., values more than three times the IQR). All data management and statistical analysis were performed using IBM SPSS Statistics (v28.0.1.1). Given the QI nature of this study, no ethics board review was required.
Results
REFERRAL VOLUMES AND PATTERNS
Of the 32 specialty clinics encompassing CHEO's MedSurg Center, 127,259 referrals were identified. Due to the scope of this study, our analyses focused on high-volume clinics, defined as the clinics receiving the top 2% of referrals. This filtering resulted in analyzing the referral characteristics of 17 clinics, in which 113,790 referrals were received and 31,430 referrals were denied. Figure 2 presents the distribution of the total referrals and denied referrals for each of the specialties considered.

Total volume of referrals and denied referrals in CHEO MedSurg clinics (June 1, 2019–June 30, 2022). Depicts the volume of referrals received and denied referrals in the respective MedSurg clinics analyzed in this study. Use of secondary data analysis conducted on EMR-generated data between June 1, 2019, and June 30, 2022. Descriptive statistics were utilized to offer a comprehensive overview of the existing status. EMR, electronic medical record.
The orthopedics clinic received the largest volume of referrals (n = 20,023; 17.6%), followed by the ear, nose, and throat (ENT) clinic (n = 9,089; 8.0%) and the ophthalmology clinic (n = 8,727; 7.7%). Cardiology reported the largest volume of denied referrals (n = 3,231; 10.3%), followed by ENT (n = 2,516; 8.0%) and ophthalmology (n = 3,097; 9.9%) clinics. Surprisingly, 47% of all referrals received by the sleep laboratory clinic were denied, representing the largest ratio of referrals denied compared with referrals received.
AUTHORIZED REFERRALS AND APPOINTMENT SCHEDULING RATE
The 17 specialty clinics had 68,158 referrals with valid closed referral status, indicating that an appointment to see a CHEO specialist had been scheduled. However, when assessing the scheduling rate of referrals, several outliers existed in the data. After removing values three times the IQR (3.6%), the rate of scheduling appointments was analyzed in 65,728 referrals. Figure 3 shows the time gap between dates when referrals were received and when the patient appointment was booked using the 1 to 4 referral priority scale.

Scheduling rate using CHEO referral priority ranking (1–4) in MedSurg clinics (June 1, 2019, to June 30, 2022). Using the 1 to 4 referral priority scale (1: seen within 1 week; 2: seen within 30 days; 3: seen within 90 days; 4: seen within 6 months) to assess appointment urgency, figure shows the respective days in which the referral was received and the subsequent date the referral had been booked an appointment. Use of secondary data analysis conducted on EMR-generated data between June 1, 2019, and June 30, 2022. All extreme outliers, defined as values three times greater than the interquartile range, were excluded from the analysis.
Within the 17 analyzed MedSurg clinics, 21.0% of referrals were scheduled appointments in the priority 1 time frame (<7 days), 27.3% within priority 2 (8–30 days), 22.2% within priority 3 (31–90 days), and 17.1% within priority 4 (91–182 days). A total of 12.4% of referrals were scheduled beyond the priority scale, indicating a scheduling rate longer than 6 months.
As a proxy measure for patient wait time, we reported the median and IQR for the number of days from referrals received to patients being scheduled for an appointment (Fig. 4). The shortest median wait times were reported in plastic surgery, orthopedics, and cardiology (6, 10, and 21 days, respectively), whereas the longest median wait times were reported in the sleep laboratory clinic, neurosurgery, and respirology (99, 104, and 105 days, respectively), thus indicating limited capacity and/or inability to timely accommodate additional patients.

Median days to appointment scheduled in CHEO MedSurg clinics (June 1, 2019 to June 30, 2022). Utilizing medians and interquartile ranges, figure demonstrates a proxy measure for patient waiting time. This includes the number of days from the referral being received to the patient's appointment being scheduled. Use of secondary data analysis conducted on EMR-generated data between June 1, 2019, and June 30, 2022. All extreme outliers, defined as values three times greater than the interquartile range, were excluded from the analysis. Use of secondary data analysis conducted on EMR-generated data.
REASONS FOR DENIED REFERRALS
The reasons for denied referrals were not clearly presented in the EMR system as the coding approach was not specific. Table 1 shows the breakdown of the reasons for denied referrals as reported in the EMR. In 29.3% of referrals, the denial criterion was classified as other/null, whereas 12.6% of referrals were denied based on the patient not meeting the clinical criteria, and 12.4% of referrals were indicated to be duplicates, reflecting inefficiencies in the process.
Reasons for Denied Referrals in the Children's Hospital of Eastern Ontario Ambulatory MedSurg Clinics (June 1, 2019, to June 30, 2022)
Post hoc analyses on the volumes and reasons for denied referrals per MedSurg clinic (Supplementary Table S1) identified that in two clinics receiving the largest volume of referrals (e.g., orthopedics and ophthalmology), the other/null criterion accounted for 24.2% and 27.5% of denied referrals, respectively. However, the third clinic that received the highest volume of referrals (e.g., ENT) most frequently denied referrals by forwarding the request to physicians or services in the community (n = 691, 27.5%). This suggests that many of the referrals received were not treated, thus implying either inefficiency in the referral process or missed potential capacity to care for patients.
Surprisingly, 18.1% of denied referrals in the ophthalmology clinic, 20.9% in the pediatric surgery clinic, and 29.7% in the plastic surgery clinic were indicated as duplicate referrals, which present an opportunity for establishing a filtering layer that captures duplicate referrals before being submitted in the EMR. The patient not meeting the clinical criteria was reported as the reason for denial in 21.1% of dentistry referrals, 40.4% of the gastroenterology, hepatology, and nutrition (Gastro/Hepatology/Nutr) referrals, and 43.5% of endocrinology referrals. However, the clinical criteria were not specified or explained. In the clinic reporting the highest volume of denied referrals (e.g., cardiology), 43.5% of the denied referrals were forwarded to physicians or services in the community.
CHEO ECONSULT UTILIZATION
In total, 332 fee codes compensating specialists for eConsult services from 16 different departments at CHEO were documented in the EMR system between June 1, 2019, and July 30, 2022. However, only four of these specialties aligned with the MedSurg clinics analyzed in this report (e.g., endocrinology, cardiology, neurology, and neurosurgery), accounting for 268 (83.2%) of the EMR-initiated eConsult billing codes (Fig. 5).

Ambulatory MedSurg eConsult utilization (CHEO MedSurg) between (June 1, 2019–July 30, 2022). Depicts the volume of eConsults requested and provided in the respective MedSurg clinics analyzed in this study. Use of secondary data analysis conducted on EMR-generated data between June 1, 2019, and June 30, 2022. Descriptive statistics used to summarize data.
The clinic most frequently utilizing eConsult billing codes was endocrinology (n = 254, 94.8%). Endocrinology reported 83.3% of all eConsult requests (code K738) and 98.5% of all eConsults provided (code K739). This particular specialty can serve as a benchmark to understand the critical success factors associated with the high percentage of eConsults provided.
STAKEHOLDER ENGAGEMENT
The preliminary stakeholder analysis revealed that a process for providing eConsults within Epic at CHEO was established by one of the endocrinologists and that their peers were empowered to follow this process when reviewing referrals that could be eConsults. The application of eConsults within referral workflows was presented to the Department of Pediatrics Division Chief Meeting, where it was determined that the initiative may enhance the quality and safety of care and that eConsults provided are not being captured with the K739 billing code. Epic analysts revealed that it is possible to create an eConsult-related option for referral denial; however, at present, there is no existing mechanism within the EMR system to transfer or convert existing referrals into patient encounters.
Discussion
Our results identified pediatric specialty clinics receiving the highest volume of referrals, system-level factors with denied referrals, and the scheduling rate of referrals. We also demonstrated that the clinics are currently billing for EMR-initiated eConsults at CHEO.
Our findings suggest that of all referrals received by the top percent of CHEO MedSurg clinics, 48.4% of referrals were denied. The primary reasons for denied referrals at CHEO are as follows: other/null (e.g., indicating no specific reason for referral denial), the patient not meeting the clinical criteria (e.g., based on clinic derivatives), and duplicate referrals (e.g., multiple referral requests). Studies have suggested that EMRs can be useful tools to improve care coordination and information management; however, the use of this technology may impact the quality of data that is documented. 25 Our study showed that the criterion for denied referrals was unified across clinics, demonstrating an absence of standardized information quality and continuity throughout the EMR system. In addition, we speculate that denied referrals being forwarded to physicians or services in the community may be a result of disconnected referral coordination practices, the complexity of referred patients, and patient waitlists exceeding hospital capacity.
We also found that there are high volumes of duplicate referrals in the current system. This is an important finding for understanding referral management processes, as evidence has demonstrated that duplicate referrals are a common source of health system inefficiency. 26 Furthermore, the results showed that large variations exist in the rate at which referrals were scheduled appointments. Considering that an electronically integrated referral and consultation system may reduce the volume of denied referrals by allowing for nonurgent clinical questions to be addressed electronically, 27 these results apply to future scale-up initiatives intended to reduce patient waiting time.
The majority of the eConsult billing codes were performed in the endocrinology clinic, which compares well to what has been reported for the adult population. 28 –32 Yet, our findings suggest that the current rate of eConsult use at the hospital is below its potential capacity compared with what has been reported for eConsult utilization among adults. This may be associated with several factors, such as a lack of awareness of the ability to bill for EMR-initiated consultations or challenges of integrating eConsult into everyday clinical practice. In addition, this may be attributed to the distinct challenges and complexities that must be considered when delivering technology innovations in the context of children, youth, and their families. 33 We expect that transforming the EMR system referral workflow to include a direct option for specialists to provide an eConsult may give patients broader access to specialized care and further support the requests PCPs in the community.
In our post hoc analysis, we created and examined process maps of the eConsult process in Epic to highlight the areas for improvement. This allowed for the detection of redundancies and facilitated collaboration among the project team members to design a future state process map that incorporates EMR-initiated eConsults. Figure 6 represents the final process map developed and validated by stakeholders within ambulatory care, including medical staff, patient services staff, and management. This process map acts as a benchmark for other hospitals using the Epic EMR system and/or eConsult for their patients.

Recommended future state workflow for eConsults in epic. Shows an ideal state eConsult workflow within the Epic EMR system that was developed and validated by stakeholder input. Final process map developed and validated by stakeholders within ambulatory care, including medical staff, patient services staff, and management. This process map acts as a benchmark for other hospitals using the Epic EMR system and/or eConsult for their patients. Three areas for improvement. *Option does not currently exist. Recommend to Epic analyst to include. **Accessible in chart review.
Three areas for improvement in eConsult integration into the referral workflow using Epic have been identified via process mapping: (1) updating referral rejection list options to include eConsult, (2) adding a quick link to the documentation button, and (3) creating a standard workflow that can be utilized by all providers across the various specialties. The next steps should include a communication plan to inform stakeholders of changes to eConsult integration into the referral. The communication plan is an important step in the QI process, as it will help ensure a smooth transition to the updated workflow and that all stakeholders are aware of how to utilize the new processes. 34
It is important to recognize some limitations associated with this research. We demonstrate referral characteristics and eConsult utilization within a single pediatric hospital, therefore the findings are unlikely to represent a wide range of health care settings and limit its generalizability. We also acknowledge the inherent complexities existing within the pediatric system, 35 which may restrict the applicability of our findings. Nevertheless, the findings present a first step toward understanding the referral and eConsult process and can inform future eConsult efforts and projects implemented by hospitals in general and in the context of pediatric care. Furthermore, the data utilized in this study were extracted retrospectively from the EMR system, which may have resulted in inaccurate or missing records, and the denied referral data captures referral characteristics available through the EMR system.
Thus, our results do not account for contextual variations that may occur in denied referrals, as we did not investigate the precise workflow of each clinic. Further exploration into common referral trends and barriers to accessing care specific to each clinic (e.g., patient demographics and clinical practices) may help to support future developments. It is important to note that the scheduling rate was used as a proxy measure for patient wait time. Therefore, our results do not identify when the patient was seen by a specialist. In line with the Quadruple Aim of health care, 36 obtaining the perspectives of patients, specialists, and PCPs on integrating electronic environments makes an interesting proposition for further study.
Conclusions
The results of this study represent a first step to understanding the process of eConsult and identifying areas for improving efficiencies and enhancing system capacity. Given the ongoing health policy discussions on increasing access to pediatric specialty care through electronic systems, we identified key areas in the referral management process that can be improved, while leveraging existing EMR systems. Refining the way referrals and specialty consultations are performed may help ensure that children, youth, and their families receive timely access to specialty care.
Footnotes
Acknowledgments
We thank the CHEO Information Services team for generating the EMR reports. We also thank the stakeholders who enriched our research by sharing their expertise on the referral and eConsult process with us.
Authors' Contributions
B.O.: Conception, design/methodology, analysis, development and final approval of the article. K.M.: Conception, design/methodology, analysis, and final approval of the article. J.M.: Design/methodology, analysis, development and final approval of the article. M.J.: Conception, design/methodology, analysis, and final approval of the article.
Disclosure Statement
No competing financial interests exist.
Funding Information
No funding was received for this article.
Supplementary Material
Supplementary Table S1
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
