Abstract
Background
Overnight radiology resident discrepancies have been described in multiple studies; however, study of resident discrepancies specific to pediatric radiology is limited.
Purpose
To examine radiology resident discrepancies as they pertain to a large pediatric hospital system.
Material and Methods
A total of 21,560 preliminary reports issued by 39 residents over a one-year period were scored as agreement, minor discrepancy, or major discrepancy by faculty members using a modification of the 2009 RADPEER scoring system. Residents were trainees of three different diagnostic radiology programs: large university-based, medium-sized community-based, or small community-based. Discrepancy rates were evaluated based on resident postgraduate year, program, and imaging modality. The effect of a general pediatric radiology report versus pediatric neuroradiology report of a CT scan was also tested. CT was the only modality in which there were comparable numbers of studies scored by both general pediatric radiologists and neuroradiologists.
Results
The rate of major resident to faculty assessment discrepancies was 1.01%, and the rate of minor resident to faculty assessment discrepancies was 4.47%. Major discrepancy rates by postgraduate years 3-5 were 1.08%, 0.75%, and 1.59%, respectively. Major discrepancy rates were highest for MR (11.22%), followed by CT (1.82%), radiographs (0.91%), and ultrasound (0.56%). There was no significant difference in discrepancy rate between residency programs and general pediatric radiology report of a CT versus pediatric neuroradiology report of a CT.
Conclusion
Radiology discrepancy rates for residents issuing preliminary reports at a large children’s hospital system are similar to those reported for adult procedures.
Introduction
Generating preliminary reports during on-call responsibilities serves as a common method to enhance training at many academic medical centers. Radiology services at large pediatric hospitals have evolved in recent years, such that many of these centers now have overnight coverage by attending faculty who can provide final reports. A few major children’s hospitals still utilize radiology residents to provide preliminary interpretations, but this is becoming less common as imaging volumes increase and children’s hospitals request final interpretations at increasing rates (1).
Accurate resident preliminary interpretations are important to providing high quality while ensuring patient safety, but residency programs differ regarding exposure to pediatric radiology on call. Accuracy rates of residents have been described in multiple publications (2–7), but most of these publications view resident accuracy from a global lens without regard to pediatric radiology specifically. One study that focused on supervised interpretations of adult radiology studies by Weinberg et al. (3) found that out of a total of 135,902 resident scores, the major discrepancy rate of resident to faculty assessment was 1.7%. The study also demonstrated that differences in discrepancy rates were not significant between the postgraduate years (PGY-3 = 1.8%, PGY-4 = 1.7%, PGY-5 = 1.5%) or between imaging modalities. The imaging modality with the highest major discrepancy rates was magnetic resonance imaging (MRI) (3.7%), followed by computed tomography (CT) (2.4%), radiographs (1.4%), and ultrasound (0.6%) (3). Additional adult radiology studies published evaluating the accuracy of preliminary interpretations of head CT scans by on-call radiology residents found that the rate of significant errors by the on-call residents was low (4,8,9). Two studies demonstrated a significant discrepancy rate of 2% and 2.5%, and another study demonstrated a similar finding of 1.7% (4,8,9). A study that examined discrepancy rates on pediatric neuroimaging studies demonstrated a major discrepancy rate of 0.17% on pediatric neuroimaging studies and found that most discrepancies occurred on CT examinations (10).
Another variable to consider is the discrepancy rate of pediatric body studies compared to pediatric neuroimaging studies when the attending radiologists for these two study types are mutually exclusive as is common in large academic radiology departments. Interpreting neuroimaging studies is common for radiology residents at large pediatric medical centers, but it brings its own challenges as imaging parameters such as field of view, slice thickness, and radiation dosage may differ in children.
Children’s hospitals may benefit from data regarding resident preliminary report accuracy of pediatric radiology procedures. Likewise, residency programs interested in maintaining independent resident call duties at children’s hospitals are served by these data as well.
The aim of the present study was to examine radiology resident discrepancy rates as they pertain to a single large pediatric hospital system to better inform stakeholders regarding staffing decisions in the future. Comparisons are made on a modality-specific basis, between resident levels of training, and between residents from different programs working at the same children’s hospital system. Additionally, a comparison is made between general pediatric (body) radiology reports and pediatric neuroradiology reports of CT studies.
Material and Methods
The present study took place at a large children’s hospital system in which residents independently interpreted exams and issued preliminary reports at a 496-bed facility from 22:00 to 08:00. Residents also remotely interpreted exams performed at a 72-bed sister children’s hospital during these hours. Diagnostic radiology residents were members of one of three programs: a large university-based program with approximately 13 residents per class; a medium-sized community-based program with 6–7 residents per class (community program 1); and a small-sized community-based program with 4–5 residents per class (community program 2). All residents served suburban and urban populations at their home program location and at the large pediatric hospital in which the study took place. The small community-based program residents were only at the PGY-4 or PGY-5 level when working at the site studied here. Residents of the other two programs were deemed eligible for overnight call at the institution studied here as of the beginning of PGY-3.
A total of 21,560 radiology procedures were reported by 39 residents and scored by 22 faculty members over the course of 12 consecutive months. All exams were performed on patients aged 1 day to 18 years. All resident preliminary interpretations issued on radiographs, ultrasounds, CT scans, and MRI scans were retrospectively reviewed over the 12-month period. Each exam was scored by either a faculty general pediatric radiologist or a faculty pediatric neuroradiologist on the morning after the preliminary report was made. In every case, the faculty radiologists made their decision within the clinical context provided by encounter notes in the electronic medical record at the time of interpretation. Reports were scored using the Weinberg et al. (3) modification of the 2009 RADPEER method (11) as agreement (rating of 1 or 2), minor discrepancy (judged by the attending radiologist not to alter clinical management, scored as 2T), and major discrepancy (management-altering, scored as 3 or 4). Discrepancy rates were evaluated based on PGY level of the resident, imaging modality, and residency program of the resident. The modalities tested were X-rays/radiography, ultrasound, or CT for general pediatric radiology, and CT or MRI for pediatric neuroradiology.
The discrepancy rate for residents issuing preliminary reports of general pediatric radiology CT studies was compared to the discrepancy rate for pediatric neuroradiology CT studies. The rating pediatric neuroradiologists were faculty radiologists who scored resident preliminary reports involving head or spine CT submitted overnight. All CTs of other body parts were scored by general pediatric radiologists. The MRI modality was not compared between the two different report types, body and neuroimaging studies, because of the small sample size of body MRI scans, as hospital orders of emergent body MRI scans on the pediatric population studied overnight were uncommon. MRI performed overnight was almost always a neurological case that was scored by a neuroradiologist. Likewise, X-rays and ultrasounds are not interpreted by neuroradiologists at this institution. Differences between groups were analyzed using a chi-square test with a significance level of 0.05.
Results
Out of 21,560 radiology procedures, the study found that the rate of major resident to faculty assessment discrepancies was 1.01% and the rate of minor resident to faculty assessment discrepancies was 4.47%. The discrepancy rate by year of training is shown in Table 1. Major discrepancy rates by year of training for PGYs 3-5 were 1.08%, 0.75%, and 1.59%, respectively, showing no correlation in progression of training. The difference is statistically significant between PGY-3 and PGY-4 residents (P = 0.03), and additionally, between PGY-4 and PGY-5 residents (P = 0.03), but not between PGY-3 and PGY-5 residents (P = 0.23).
Resident preliminary report scores and discrepancy rates in percentage by year of training.
Values are given as n (%).
*Definition of scoring scale: agreement (rating of 1 or 2), minor discrepancy (rating of 2T), and major discrepancy (rating of 3 or 4). The difference in major discrepancy rates is statistically significant between PGY-3 and PGY-4 residents (P = 0.03), and additionally, between PGY-4 and PGY-5 residents (P = 0.03), but not between PGY-3 and PGY-5 residents (P = 0.23).
PGY, postgraduate year.
The discrepancy rate by modalities is shown in Table 2. Major discrepancy rates were highest for MRI (11.22%), followed by CT (1.82%), radiographs (0.91%), and ultrasound (0.56%). The difference in discrepancy between modalities are all statistically significant (P < 0.02).
Resident preliminary report scores and discrepancy rate in percentage by modality.
Values are given as n (%).
*Definition of scoring scale: agreement (rating of 1 or 2), minor discrepancy (rating of 2T), and major discrepancy (rating of 3 or 4). The difference in major discrepancy rates between modalities are all statistically significant (P < 0.02).
CT, computed tomography; MRI, magnetic resonance imaging; US, ultrasound.
The discrepancy rates by residency program and by general pediatric radiology CT versus pediatric neuroradiology CT is shown in Tables 3 and 4. The difference between major discrepancy rates by residency program (university-based versus community-based versus another community-based program) was not significant (P > 0.07). The large university-based, medium-sized community-based, and small community-based programs had major discrepancy rates of 1.06%, 0.85%, and 1.40%, respectively. The difference between major discrepancy rates by general pediatric radiology CT and pediatric neuroradiology CT was also not significant (P = 0.71). The major discrepancy rate of general pediatric radiology CT scans was 1.94% and major discrepancy rate of pediatric neuroradiology CT scans was 1.75%.
Resident preliminary report scores and discrepancy rates in percentage by type of program.
Values are given as n (%).
*Definition of scoring scale: agreement (rating of 1 or 2), minor discrepancy (rating of 2T), and major discrepancy (rating of 3 or 4). The difference between major discrepancy rates by residency program (university-based vs. the larger community-based vs. the smaller community-based program) was not significant (P > 0.07).
Resident preliminary report scores and discrepancy rates in percentage by type of report, general pediatric radiology CT vs. pediatric neuroradiology CT.
Values are given as n (%).
*Definition of scoring scale: agreement (rating of 1 or 2), minor discrepancy (rating of 2T), and major discrepancy (rating of 3 or 4). The difference between major discrepancy rates by general pediatric radiology CT and pediatric neuroradiology CT was not significant (P = 0.71).
A separate treatment of the types of studies that resulted in scores of 3 or 4 was not formally attempted, though there was a predominance of unrecognized extremity abnormalities, likely a result of lesser familiarity with the skeletally immature patient.
Discussion
The results depicted here show a similar resident major discrepancy rate compared to previously published rates for residents which were in the range of 0.33%–1.9% (2). The overall discrepancy rate for the university-based residency program affiliated with this children’s hospital was previously reported as 1.7% (3), so the present study demonstrates a decreased rate for the children’s hospital affiliated with the program. A meta-analysis analyzing 388,123 adult CT procedures demonstrated a total discrepancy rate of 7.7% and a major discrepancy rate of 2.4% (12).
Surprisingly, major discrepancy rates by year of training showed no effect due to progression of training, which differs with other reports that showed improvement of discrepancy rates with increasing resident experience (3,13). The PGY-4 residents in the study had the lowest discrepancy rate of 0.75%, which demonstrated a significant difference when compared to PGY-3 and PGY-5 residents. The PGY-5 residents had the highest discrepancy rate of 1.59%. The low discrepancy rates and lack of correlation between the years of training suggest that there is little added value to a requirement that a resident be at a specific PGY level to provide preliminary interpretations at a children’s hospital similar to the conclusions of other reports (3,13).
The study also found that major discrepancy rates were highest for MRI, followed by CT, radiographs, and ultrasound, which coincides with the report by Weinberg et al. (3). All modalities demonstrated significant differences between one another in terms of major discrepancy rates. This suggests that each modality has its own unique challenges and requires a different skillset, with MRI having the most challenges.
There was no difference between resident major discrepancy rates by type of report, general pediatric radiology CT and pediatric neuroradiology CT, which coincides with similar reports (8–10). This suggests that the skillsets needed to interpret these two types of CT scans do not differ.
To our knowledge, this is the first study to investigate the performance of residents from different programs working at a common pediatric hospital. Interestingly, there was no difference demonstrated between the three residency programs. There was no effect from residents having been trained at a large-sized university-based program compared to those trained at medium and small-sized, community-based programs.
Providing 24-hour radiology attending coverage can add considerable costs to the pediatric hospital, particularly in facilities that do not maintain enough volume to justify this attending radiologist presence. This may even be more relevant in children’s hospitals where emergent CT is less common (14). Furthermore, radiology residents and faculty have expressed concerns that the loss of the autonomous call experience may negatively impact resident education (15). An additional benefit of utilizing resident-derived preliminary reports is the value of a “double read” by two physicians that would not be inherent if a single radiologist interpreted all exams. This added value of having a radiology resident generate an independent interpretation has not been studied.
The present study has some limitations. Just as in most retrospective studies of resident discrepancy rates, no comparison is made to an assessment of discrepancy rates that may occur with independent attending radiologist interpretations. Attempts were not made to ascertain discrepancy rates on a more granular level than modality (e.g. extremity radiographs vs. chest radiographs). These results may not be generalizable to all radiology residency programs.
In conclusion, radiology discrepancy rates for residents issuing preliminary reports at a large children’s hospital are similar to those reported for adult radiology procedures. Children’s hospitals may be justified in maintaining an independent radiology resident presence overnight rather than requiring attending radiologist coverage.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
