Clinical evaluation of AI software for rib fracture detection and its impact on junior radiologist performance

Abstract

Background

The detection of rib fractures (RFs) on computed tomography (CT) images is time-consuming and susceptible to missed diagnosis. An automated artificial intelligence (AI) detection system may be helpful to improve the diagnostic efficiency for junior radiologists.

Purpose

To compare the diagnostic performance of junior radiologists with and without AI software for RF detection on chest CT images.

Materials and methods

Six junior radiologists from three institutions interpreted 393 CT images of patients with acute chest trauma, with and without AI software. The CT images were randomly split into two sets at each institution, with each set assigned to a different radiologist First, the detection of all fractures (AFs), including displaced fractures (DFs), non-displaced fractures and buckle fractures, was analyzed. Next, the DFs were selected for analysis. The sensitivity and specificity of the radiologist-only and radiologist-AI groups at the patient level were set as primary endpoints, and secondary endpoints were at the rib and lesion level.

Results

Regarding AFs, the sensitivity difference between the radiologist-AI group and the radiologist-only group were significant at different levels (patient-level: 26.20%; rib-level: 22.18%; lesion-level: 23.74%; P < 0.001). Regarding DFs, the sensitivity difference was 16.67%, 14.19%, and 16.16% at the patient, rib, and lesion levels, respectively (P < 0.001). No significant difference was found in the specificity between the two groups for AFs and DFs at the patient and rib levels (P > 0.05).

Conclusion

AI software improved the sensitivity of RF detection on CT images for junior radiologists and reduced the reading time by approximately 1 min per patient without decreasing the specificity.

Keywords

Rib fracture multidetector computed tomography artificial intelligence thoracic injuries

Introduction

Rib fractures (RFs) occur in 10% of all traumatic events (1) and are frequently identified after thoracic injuries (2). Computed tomography (CT) represents the most accurate imaging modality to diagnose RFs and is routinely used in the clinical setting (3,4). However, detecting rib fractures on hundreds of CT images is time-consuming and susceptible to missed diagnosis in clinical practice because of various human and environmental factors (5,6). Urbaneja et al. (7) reported that the sensitivity of RF detection by radiologists was 68.4%–79.1% using conventional CT reformats. In addition, subtle RFs can be visually inconspicuous and appear similar to non-fracture abnormalities such as osteoporosis or respiratory motion artifacts on CT images (8). Thus, an automated artificial intelligence (AI) detection system may be instrumental in helping radiologists interpret CT images, increasing the diagnostic efficiency of RF, and consequently improving patient outcomes (9).

The aim of the present study was to assess the diagnostic accuracy of junior radiologists to identify RFs in CT chest images with and without the assistance of AI software. The software was integrated into the clinical workflow and used in the concurrent reader mode.

Material and Methods

Introduction of institution codes

Three institutions participated in this multicenter study and were coded as Institution 01, Institution 02, and Institution 03, respectively. Each institution was equipped with the same AI workstation, and all the radiologists received the same training to familiarize themselves with the operation procedures before the trial began. There were 15 cases outside the research data used for the training process. In the reading session with AI assistance, all RFs automatically detected by AI were shown to the readers for review. The readers were notified that AI findings may or may not be accurate, and they need to make the final diagnosis at their discretion.

Patient enrollment

Approval from the local institutional board was obtained at each clinical institution and patient informed consent was waived.

Acute trauma patients were consecutively and prospectively enrolled from Institution 02 and Institution 03 from August 2019 to September 2019 using the same inclusion/exclusion criteria. The inclusion criteria were as follows: (i) aged ≥18 years; (ii) patients admitted to the emergency department with acute trauma histories, such as traffic accidents or violence, and clinicians recommended for CT examination after routine management triage; and (iii) qualified chest CT scan with complete image data. Cases were excluded from the study for any of the following reasons: (i) poor image quality, such as poor positioning or respiratory artifacts; (ii) pathological RFs or other non- RF lesions; (iii) chest wall deformities; or (iv) other factors that made the data unsuitable for the software evaluation, i.e. incomplete images of the scan series or incorrect DICOM format that cannot be imported by the software.

All the chest CT scans were acquired from five multidetector CT scanners at Institution 02 and Institution 03: Brilliance 64 (Philips Healthcare), Brilliance iCT 256 (Philips Healthcare), Lightspeed 16 (GE Healthcare), NeuViz 128 (NMS healthcare), and Aquilion ONE 640 (TOSHIBA healthcare). The scanning parameters are shown in Supplementary Material 1 (Table S1).

Lesion definition

The clinical diagnostic criteria are as follows: (i) history of injuries, such as blunt thoracic trauma and athletic activities; (ii) clinical manifestations range from mild discomfort to life-threatening conditions, such as pneumothorax, splenic laceration, and pneumonia; (iii) image evidence of RF. The RFs in this study included displaced fractures (DFs), non-displaced fractures, and buckle fractures. DFs are characterized by obvious lucency associated with distraction or displacement of fracture fragment. Non-displaced fractures are defined by the disruption of the cortex with minimal to no displacement. Buckle fractures are considered to be an irregularity of one or both cortical margins without observable fracture lucency (Fig. 1) (10).

Fig. 1.

Examples of different patterns of rib fractures. (a) Displaced fracture on the right 11th rib. Obvious fracture lucency associated with disrupted cortical continuity and a substantial abnormality in alignment (arrow) were detected. (b) Non-displaced fracture on the right seventh rib. Disrupted cortical continuity with no displacement (arrow) was detected. (c) Buckle fracture on the right sixth rib. Irregular cortical margins and no fracture lucency (arrow) was detected.

First, all the detected fractures (termed “all fractures” [AF]), including DFs, non-displaced fractures, and buckle fractures, were analyzed in this study to evaluate the detection accuracy with and without AI assistance. Furthermore, given that serious trauma complications are more likely associated with displaced rib fractures (11), we also performed specific analysis of DFs only in the study.

Reference standard

The reference standard for CT image interpretation was established by three senior radiologist experts (all with >15 years of reading experience) at Institution 01. A consensus review was performed by all three experts to obtain the final RF evaluation result (Supplementary Material 2: Fig. S1). Each fracture position was marked by manually drawing a tight bounding box encompassing the fracture site, and the types of the fracture patterns including DFs, non-displaced fractures, and buckle fractures were also recorded.

AI software description

The AI software applied in our research was trained with separate datasets collected from different medical institutions based on deep convolutional neural networks to automatically detect RFs. The model architecture, named RB-V-Net, is shown in Fig. 2. The network consists of a contracting path and an expansive path, to make use of both global image context and local image details. Each layer along the pathways comprises a number of residual blocks with skip connections (R-modules) to alleviate the problem of vanishing gradients, or bottleneck residual blocks (B-modules) to reduce the number of parameters and complexity of the network.

Fig. 2.

Structure of RB-V-Net. The network consists of a contracting path and an expansive path. Each layer along the pathways comprises a number of residual blocks with skip connections (R-modules) to alleviate the problem of vanishing gradients, or bottleneck residual blocks (B-modules) to reduce the number of parameters and complexity of the network.

The network was trained with cropped image patches, with patch size of 96 × 96 × 96 voxels, and voxel spacing of 0.6 × 0.6 × 0.6 mm to improve the model capability of detecting more subtle fractures. The network parameters were initialized using the Kaiming initialization method. To improve the generalization performance of the model, data augmentation was employed in the training including random elastic deformation (0∼2 mm), anisotropic random scaling (0.9–1.1), and small random angle rotation (−5° to + 5°) around a random axis. Focal loss (α = 0.75 and γ = 2) as the target loss function was minimized with Adam optimizer (initial learning rate = 10⁻⁴, β1 = 0.9, and β2 = 0.999). The optimal training epoch of the model was selected based on the minimum loss of the validation data.

Image interpretation

The CT images were interpreted by six junior radiologists (all with about two years of trauma reading experience), two from each of the three institutions. All the CT images were randomly divided into two sets of images at each institution, with each set assigned to one of the two radiologists. In a routine manner, the readers went through all cases over two sessions using a commercial medical image workstation (Carestream Health, Toronto, Canada) in axial, coronal, and sagittal views. The multiplanar reconstruction (MPR), curved planar reformation (CPR), and volume rendering (VR) views were used to confirm the fracture whenever needed. The CT images with RFs automatically marked by AI were included in the reading session with AI assistance. For each radiologist, each set was randomly divided into two parts (Part A and Part B), and the radiologist read the same CT scan twice, with and without AI software (uAI-BoneCare) assistance, with a one-month washout period. Each radiologist read part A of the assigned CT scans without AI for the first time and with AI for the second time after one month, whereas part B of the data was examined with AI for the first time and without AI for the second time to make fair comparisons between two different readings. When assisted with AI, the radiologist showed all RF candidates automatically detected by AI models for immediate assessment immediately after each CT scan was loaded (Fig. 3).

Fig. 3.

Flow diagram of the patient selection and randomization for reading. The assigned CT data were randomly divided into two parts, Part A and Part B. AI , artificial intelligence; CT, computed tomography R, radiologist.

Considering that a patient can have multiple RFs, the interpretation results of radiologists were compared with the reference standard and classified as one of the four categories at the patient level (Fig. 4). Given that each rib can also present multiple fracture sites, similar rationale can be extended to define evaluation metrics at the rib level (Fig. 4). As previously mentioned, tight bounding boxes were marked around each fracture site found for both radiologist readings and reference standard. Only when the intersection over union (IoU) between two bounding boxes is >0, the finding of the radiologist readings was considered to be a true positive. The consistency between the radiologist readings and reference standard at both the patient level and rib level is shown in Fig. 4.

Fig. 4.

Evaluations of the interpretation results by comparing the interpretations of the reference standard and radiologist at the patient, rib, and lesion levels.

Statistical analysis

Principle of statistical analysis

The software used for statistical analysis was SAS (version 9.4, SAS Institute, Cary, NC, USA). The mean, standard deviation (SD), minimum, and maximum age are presented. The comparison of age between the patients with and without RF was performed by group t-tests, and the chi-square test was used for the comparison of sex. The 95% confidence interval (CI) of differences in sensitivity and specificity was based on the Newcombe–Wilson score method at the patient and rib levels. Comparison of the detection rate (sensitivity) of RF at the lesion level was based on the binomial test. The significance level was α = 0.025 (unilateral) and the degree of assurance was 80%. Logistic regression analysis was used to compare the RF detection performance across three institutions and six individual radiologists. The average reading time for RF detection in the radiologist-AI group and the radiologist-only group was compared by t-test The sample size calculation is described in Supplemental Material 3.

Primary endpoint

The sensitivity of RF detection at the patient level was tested using a superiority trial. If the lower limit of the 95% CI of the sensitivity difference between two groups was >0, we considered that the RF detection sensitivity of the radiologist-AI group was superior to that of the radiologist-only group.

The specificity of RF detection at the patient level was tested using a non-inferiority trial. If the lower limit of the 95% CI of the specificity difference between two groups was greater than −10%, we considered that the RF detection specificity of the radiologist-AI group was non-inferior to that of the radiologist-only group.

Secondary endpoint

The sensitivity and specificity at the rib level, and the area under the curve (AUC) at both the patient and rib levels were statistically compared. At the lesion level, the sensitivity was compared between the radiologist-AI group and the radiologist-only group. Logistic regression was used to compare the diagnostic accuracy between institutions and radiologists. The average reading time of the radiologist-AI group and the radiologist-only group were recorded and compared statistically.

Results

Baseline characteristics

In this study, 260 patients were enrolled in Institution 02 and 140 patients were enrolled in Institution 03. Among them, five patients in Institution 02 dropped out because of a lack of a complete history of chest trauma, and two patients in Institution 03 were excluded due to incomplete imaging data. Finally, the CT images of 393 patients were anonymized and distributed to the three institutions, and 1179 (393 cases × 3 institutions) CT reading results (each with and without the assistance of AI) were obtained and analyzed (Fig. 3).

A total of 393 patients (245 men, 148 women; age range = 18–89 years; mean age = 45.6 ± 14.5 years) were included in the present study. In total, 629 fractures in 187 patients (displaced fractures = 196, non-displaced fractures = 108, buckle fractures = 325) were detected during reference standard reading; among them, 90 patients had DFs (Table 1).

Table 1.

Baseline demographic characteristics.

Index	AFs*					DFs^†
Index	RF	Non-RF	Total	Statistics	P value	RF	Non-RF	Total	Statistics	P value
Number	187	206	393	–	–	90	303	393	–	–
Age (years)	49.3 ± 13.7 (20–81)	42.2 ± 14.5 (18–89)		5.02	0.001	52.6 ± 12.9 (23–81)	43.5 ± 14.3 (18–89)		5.45	0.001
Sex
Male	130 (69.52)	115 (55.83)	245	7.83	0.005	63 (70.00)	182 (60.07)	245	2.92	0.088
Female	57 (30.48)	91 (44.17)	148			27 (30.00)	121 (39.93)	148

Values are given as n (%) or mean ± SD (range). T-test was used for statistical analysis of age; chi-square test was used to analyze sex and injury history.

*Includes all the detected DFs, non-displaced fractures, and buckle fractures.

^†

Focuses only on the detection of DFs.

AFs, all fractures; DF, displaced fracture; RF, rib fracture.

Primary performance metrics

Among the 1179 CT readings, 561 readings were detected with 1887 lesions of AF in 1641 ribs; 618 readings and 26,655 ribs were found to be without fractures. With regard to DFs, 270 readings were detected with 588 lesions of DF in 270 ribs; 909 readings and 27,744 ribs were found to be without DFs.

Regarding the patient-level analysis, a significant difference was found between the sensitivity of the radiologist-AI group and radiologist-only group (AFs: 89.66% vs. 63.46%; P < 0.001; DFs: 94.44% vs. 77.78%; P < 0.001). Regarding AFs, the sensitivity difference between the two groups was 26.20% with the 95% CI lower limit at 21.42; for DFs, the difference was 16.67% with 95% CI lower limit at 10.98. It indicated that the sensitivity of RF detection by radiologist-AI was superior to that of radiologist-only for both AFs and DFs (Table 2, Figs. 5–7).

Fig. 5.

Axial chest CT images of an example of displaced DF detection in a 40-year-old man with a history of acute chest trauma. (a) Reference standard interpretation of DF (white arrow); (b) radiologist-only interpretation of DF, radiologist 01 marked the DF with a green box on the axial chest images; (c) radiologist-AI interpretation of DF, the AI software marked the DF with a red box and radiologist 01 agreed with the automated DF detection (true-positive finding). AI, artificial intelligence; CT, computed tomography; DF, displaced fracture.

Fig. 6.

Axial chest CT images of an example of RF detection in a 55-year-old man with a history of acute chest trauma. (a) Reference standard interpretation of a DF (white arrow); (b) radiologist-only interpretation of RF, radiologist 06 marked the DF with a green box (true-positive finding), and an extra fracture (false-positive finding, red arrow); (c) radiologist-AI interpretation of RF, the DF was marked by the radiologist-AI reading (true-positive findings). AI, artificial intelligence; CT, computed tomography; DF, displaced fracture; RF, rib fracture.

Fig. 7.

Axial chest CT images of an example of subtle RF detection in a 65-year-old man with a history of acute chest trauma. (a) Reference standard interpretation of the subtle RF (white arrow); (b) radiologist-only interpretation of RF, radiologist 02 marked the subtle RF with a green box (true-positive finding); (c) radiologist-AI interpretation of RF, the subtle RF were missed by radiologist-AI reading (false-negative finding). AI, artificial intelligence; CT, computed tomography; DF, displaced fracture; RF, rib fracture.

Table 2.

Sensitivity and specificity of RF detection at the patient level.

Index	AFs*				DFs^†
Index	Radiologist-AI group	Radiologist-only group	Difference^‡	P value	Radiologist-AI group	Radiologist-only group	Difference^‡	P value
Sensitivity	503/561(89.66)	356/561(63.46)	(26.20)[21.42–30.85]	0.001	255/270 (94.44)	210/270 (77.78)	(16.67)[10.98–22.42]	0.001
Specificity	550/618(89.00)	577/618(93.37)	(−4.37)[−7.57 to −1.21]	0.125	904/909 (99.45)	906/909 (99.67)	(−0.22)[−0.98 to 0.49]	0.925
AUC	0.89	0.78	0.11[0.09–0.13]	0.001	0.97	0.89	0.08[0.06–0.11]	0.001

Values are given as proportions, percentages in parentheses, and 95% confidence intervals in brackets unless otherwise indicated.

*Includes all the detected DFs, non-displaced fractures, and buckle fractures.

^†

Focuses only on the detection of DFs.

^‡

The difference indicates the values of the radiologist-AI group – the values of the radiologist-only group.

AFs, all fractures; AI, artificial intelligence; AUC, area under the curve; DF, displaced fracture; RF, rib fracture.

No significant difference was found between the specificity of the radiologist-AI group and radiologist-only group (AFs: 89.00% vs. 93.37%; P = 0.125; DFs: 99.45% vs. 99.67%; P = 0.925). Regarding AFs, the specificity difference between the two groups was −4.37%, with the 95% CI lower limit at −7.57; for DFs, the difference was −0.22% with the 95% CI lower limit at −0.98. It indicated that the specificity of RF detection by the radiologist-AI group was not inferior to that of the radiologist-only group (Table 2).

Secondary performance metrics

Regarding the rib-level analysis, the sensitivity of the radiologist-AI group proved to be significantly higher for the detection of both AFs and DFs (AFs: 83.36% vs. 61.18%; P < 0.001; DFs: 94.75% vs. 80.25%; P < 0.001). Regarding AFs, the sensitivity difference between the two groups was 22.18%, with the 95% CI lower limit at 19.19; for DFs, the difference was 14.49% with the 95% CI lower limit at 10.69. It indicated that the sensitivity of RF detection by radiologist-AI was superior to that of the radiologist-only group at the rib level (Table 3).

Table 3.

Sensitivity and specificity of RF detection at the rib level.

Index	AFs*				DFs^†
Index	Radiologist-AI group	Radiologist-only group	Difference^‡	P value	Radiologist-AI group	Radiologist-only group	Difference^‡	P value
Sensitivity	1368/1641 (83.36)	1004/1641 (61.18)	(22.18)[19.19–25.12]	0.001	523/552(94.75)	443/552 (80.25)	(14.49)[10.69–18.35]	0.001
Specificity	26,431/26,655 (99.16)	26,513/26,655 (99.47)	(−0.31)[−0.45 to −0.17]	0.478	27,725/27,744 (99.93)	27,728/27,744 (99.94)	(−0.01)[−0.06 to 0.03]	0.980

Values are given as n (%) and 95% confidence intervals in brackets unless otherwise indicated.

*Includes all the detected DFs, non-displaced fractures, and buckle fractures.

^†

Focuses only on the detection of DFs.

^‡

The difference indicates the values of the radiologist-AI group – the values of the radiologist-only group.

AFs, all fractures; AI, artificial intelligence; AUC, area under the curve; DF, displaced fracture; RF, rib fracture.

No significant difference was found between the specificity of the radiologist-AI group and radiologist-only group (AFs: 99.16% vs.99.47%; P = 0.478; DFs: 99.93% vs. 99.94%; P = 0.980). Regarding AFs, the specificity difference between the two groups was −0.31% with the 95% CI lower limit at −0.45; for DFs, the difference was −0.01%, with the 95% CI lower limit at −0.06. It indicated that the specificity of RF detection by the radiologist-AI group was not inferior to that of the radiologist-only at rib level (Table 3). At the lesion level, the sensitivity of the radiologist-AI group was significantly higher than that of radiologist-only group (AFs: 82.19% vs. 58.45%; P < 0.001; DFs: 94.56% vs. 78.40%; P < 0.001) (Table 4).The AUC score of RF detection was significantly different between the radiologist-AI group and radiologist-only group (AFs: 0.89 vs. 0.78; P < 0.001; DFs: 0.97 vs. 0.89; P < 0.001) at the patient level (Table 2). The AUC score of RF detection and the diagnostic accuracy for each specific rib between two groups are shown in Supplemental Materials 4 (AFs: Table S2; DFs: Table S3).

Table 4.

Detection rate of RFs at the lesion level.

Index	AFs*				DFs^†
Index	Radiologist-AI group	Radiologist-only group	Statistics	P value	Radiologist-AI group	Radiologist-only group	Statistics	P value
Sensitivity(TPR)	1551/1887(82.19)	1103/1887(58.45)	−15.963	0.001	556/588 (94.56)	461/588 (78.40)	−8.102	0.001
1-Sensitivity(FNR)	336/1887(17.81)	784/1887(41.55)			127/588(21.60)	32/588(5.44)

Values are given as n (%).

*Includes all the detected DFs, non-displaced fractures, and buckle fractures.

^†

Focuses only on the detection of DFs.

AFs, all fractures; AI, artificial intelligence; DF, displaced fracture; FNR, false-negative rate; RF, rib fracture; TPR, true-positive rate.

Considering the intra- and inter-rater variability due to different radiologists and institutions, the RF detection performance comparison between the radiologist-AI group and radiologist-only group was performed across three institutions and six individual radiologists. According to the logistic regression analysis results of RF detection accuracy, no significant difference was found in the diagnostic accuracy across three institutions (AFs: Institution 01 vs. Institution 02 vs. Institution 03, P = 0.736; DFs: P = 0.431) or interaction between institutions and groups (AFs: P = 0.689; DFs: P = 0.470). A significant difference was found among six radiologists regarding the accuracy of RF detection (P < 0.001), implying that the reading experience and skills of different radiologists could affect their reading accuracy (Table 5).

Table 5.

Logistic regression analysis of the RF detection accuracy.

Factors	AFs*		DFs^†
Factors	Wald value	P value	Wald value	P value
Groups	5.708	0.017	0.064	0.800
Radiologists	51.199	0.000	34.796	0.001
Institutions	0.114	0.736	0.619	0.431
The interaction of institutions and groups	0.160	0.689	0.525	0.470

Values are given as Wald values unless otherwise indicated.

*Includes all the detected DFs, non-displaced fractures, and buckle fractures.

^†

Focuses only on the detection of DFs.

AFs, all fractures; DF, displaced fracture; RF, rib fracture.

The average reading time for RF detection in the radiologist-AI group was 128.43 ± 76.61 s, which was significantly shorter than that in the radiologist-only group (179.54 ± 101.61 s; P < 0.001), with a reduction of approximately 1 min per case on average.

Discussion

In this multicenter, diagnostic study, we applied AI software to assist with RF detection and assessed the diagnostic significance of fractures identified by radiologist-AI readings compared with that identified by radiologist-only readings. Our results showed that it substantially improved the accuracy of RF detection for junior readers to detect RFs with the assistance of AI software on chest CT images. For AFs that contained multiple patterns of RFs (DFs, non-displaced fractures, and buckle fractures), AI software improved the RF detection sensitivity by 26.20% (95% CI = 21.42–30.85), and the AUC score increased by 11% (95% CI = 9–13) at the patient level. For DFs, the sensitivity improved from 77.78% to 94.44%, and the AUC score increased from 0.89 to 0.97.

The detection of RFs remains a challenging diagnostic task, particularly for inexperienced readers (10,12) because each rib is complex in shape with a long, curved shaft across numerous axial slices. In addition, fracture patterns can be highly variable and inconspicuous such as buckle or hairline fractures. It has been shown that readers are particularly prone to overlook subtle and mild fractures in asymptomatic patients (13). In the present study, the overall sensitivity of AF detection in the radiologist-only group was 58.45% (1103/1887) at the lesion level, which was substantially lower than that of DF-only detection (58.45% vs. 78.40%; P < 0.001). Moreover, both sensitivities were considerably increased with the assistance of AI software (AFs: 58.45% vs.82.19%, DFs: 78.40% vs. 94.56%; P < 0.001), which suggests the diagnostic accuracy improvement by the AI software regardless of the types of the RF patterns.

AI software was developed as a computer-aided diagnostic tool to increase the RF detection performance in many areas (14 –16). Lindsey et al. (15) developed a deep learning model to detect fractures on radiographs with a diagnostic accuracy similar to that of senior subspecialized orthopedic surgeons. Nevertheless, the inherent low sensitivity and limitation on the complication evaluation of radiographs may reduce the clinical value of the model. Zhou et al. (17) used a combined model of two convolutional neural networks (CNNs) to detect and classify RFs from CT images, and they showed an improved efficiency by the proposed CNNs with comparable diagnostic efficiency to that of expert radiologists at the lesion level. Weikert et al. (16) trained a deep learning-based algorithm for automated detection of acute and chronic RFs on whole-body trauma CT, which may be used as a screening tool to avoid false-negative radiology reports in clinical application. However, the retrospective analysis was performed on the data acquired from only one vendor and one center, and the reference standard was defined by the written CT reports.

In this research, the validation methodology of the AI software included the consistency evaluation between the radiologist reading and reference standard reading, and the effectiveness evaluation of the radiologist-only group and radiologist-AI group. During the image interpretation, a bounding box was drawn at the site of fracture on all the slices of the CT images on the reference standard reading and radiologists reading. The IoU between the two bounding boxes was used to evaluate the consistency of the reading. The detection effectiveness of the radiologist-only group and radiologist-AI group were evaluated at the patient, rib, and lesion levels, as shown in Fig. 4.

One advantage of our research is that it had a multicenter design, and consecutive cases were collected in a real clinical setting. More importantly, we analyzed the diagnostic performance of RF detection by comparing radiologist-only and radiologist-AI readings for AFs and DFs at three levels (patient, rib, and lesion levels). Our work shows that RF detection with the assistance of the AI software in clinical practice could improve the diagnostic performance of junior readers on CT images, possibly leading to earlier therapeutic intervention and better patient prognoses. The reporting time of RF detection was reduced by AI software (approximately 1 min), making its routine use as a concurrent reader effective in terms of reading time.

To reduce bias in this study, we employed the following approaches in our clinical trial. First, all the researchers in different clinical institutions received the same training before the trial, including the operation procedures and recording methods. The reading criteria were standardized to minimize inconsistency caused by the subjective tendency of the researchers. Second, a one-month washout period was set between the two rounds of readings to decrease the bias favorable to the second round of readings introduced by the memory. Furthermore, all the CT scans assigned to the same radiologist were randomly divided into two halves in opposite reading order. The first half of the data was interpreted by the radiologist without AI in the first round, while the second half of the data was examined using AI in the first round. To further reduce the memory influence, the reading order of the dataset was randomly reshuffled before the second round of readings. Finally, all the images were reviewed by the same radiologist twice, with and without AI, to reduce the institutional differences and inter-observer variability impact on the group comparison.

The present study had several limitations. First, although the reference standard was established based on three senior radiologist experts, occult fractures that are not readily visible because of imaging limitations might be missed in the reference standard. Follow-up chest CT images scanned a few weeks after the trauma injury may help locate those occult fractures by the features like sclerosis or callus formation. However, in our study, the patients were prospectively enrolled and retrospectively analyzed, and a follow-up CT scan was not ethically allowed, particularly for subtle fractures. Second, the number of radiologists recruited in the study was limited given that the trial results substantially relied on the experience of the participating readers. A larger pool of emergency medicine clinicians with diverse levels of experience is required for future studies. Third, our results showed that the reading time of RF detection was shorter when AI was used. While in the clinical setting, a complete chest CT reading includes much more evaluation including the whole area (e.g. lesions in the chest wall, lung, pleural space, mediastinum, and spinal areas). Therefore, the real clinical reading time requires a more comprehensive validation. Last but not least, the benefit of AI software also requires further investigation when radiologists adopt more advanced curved planar reformation visualization techniques to observe and diagnose RFs, helping radiologists to improve their RF detection rate compared with reading standard multiplanar reformations (5,6).

In conclusion, in this multicenter, diagnostic trial, we showed that AI software improved the sensitivity of RF detection on CT images for junior radiologists and reduced the reading time by approximately 1 min per patient without decreasing the specificity.

Supplemental Material

sj-docx-1-acr-10.1177_02841851211043839 - Supplemental material for Clinical evaluation of AI software for rib fracture detection and its impact on junior radiologist performance

Supplemental material, sj-docx-1-acr-10.1177_02841851211043839 for Clinical evaluation of AI software for rib fracture detection and its impact on junior radiologist performance by Xiang Liu, Dijia Wu, Huihui Xie, Yufeng Xu, Lin Liu, Xiaofeng Tao and Xiaoying Wang in Acta Radiologica

Footnotes

Acknowledgments

The authors gratefully acknowledge the technical support of Xin Yue and Suisui Zhang from Beijing Smart Tree Medical Technology Co. Ltd. The authors also gratefully acknowledge the six junior radiologists including Jiejin Yang and Mingming Ma from Peking University First Hospital, Xiaoqing Sun and Erjuan Wang from the China-Japan Union Hospital of Jilin University, and Jiliang Ren and Qi Fan from the Shanghai Ninth People's Hospital for reading the CT images. They also acknowledge Jia He and Yongqing Gao from Tongji University School of Medicine for statistical support.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Xiaoying Wang

Supplemental material

Supplemental material for this article is available online.

References

Sirmali

Türüt

Topçu

, et al. A comprehensive analysis of traumatic rib fractures: morbidity, mortality and management. Eur J Cardiothorac Surg 2003;24:133–138.

Senekjian

Nirula

. Rib fracture fixation. Crit Care Clin 2017;33:153–165.

Traub

Stevenson

McEvoy

, et al. The use of chest computed tomography versus chest X-ray in patients with major blunt trauma. Injury 2007;38:43–47.

Livingston

Shogan

John

, et al. CT Diagnosis of Rib fractures and the prediction of acute respiratory failure. J Trauma 2008;64:905–911.

Dankerl

Seuss

Ellmann

, et al. Evaluation of Rib fractures on a single-in-plane image reformation of the Rib cage in CT examinations. Acad Radiol 2017;24:153–159.

Ringl

Lazar

Töpker

, et al. The ribs unfolded – a CT visualization algorithm for fast detection of rib fractures: effect on sensitivity and specificity in trauma patients. Eur Radiol 2015;25:1865–1874.

Urbaneja

De Verbizier

Formery

, et al. Automatic rib cage unfolding with CT cylindrical projection reformat in polytraumatized patients for rib fracture detection and characterization: feasibility and clinical application. Eur J Radiol 2019;110:121–127.

Martin

Eltorai

Dunn

, et al. Clinical management of rib fractures and methods for prevention of pulmonary complications: a review. Injury 2019;50:1159–1165.

Yang

Gao

Liu

, et al. Performance and reading time of automated breast US with or without computer-aided detection. Radiology 2019;292:540–549.

10.

Talbot

Gange

Jr Chaturvedi

, et al. Traumatic rib injury: patterns, imaging pitfalls, complications, and treatment. Radiographics 2017;37:628–651.

11.

Park

Ryu

Cho

, et al. A treatment case of delayed aortic injury: the patient with posterior rib fracture. Korean J Thorac Cardiovasc Surg 2014;47:406–408.

12.

Mauf

Held

Gascho

, et al. Flat chest projection in the detection and visualization of rib fractures: a cross-sectional study comparing curved and multiplanar reformation of computed tomography images in different reader groups. Forensic Sci Int 2019;303:109942.

13.

Okamoto

Ueda

Ichinose

. Cardiac injury due to the rapid progress of the dislocation of rib fractures: a rare case that required urgent open reduction and internal rib fixation. Respir Med Case Rep 2019;27:100840.

14.

Zhang

Jia

, et al. Improving rib fracture detection accuracy and reading efficiency with deep learning-based detection software: a clinical evaluation. Br J Radiol 2021;94:20200870.

15.

Lindsey

Daluiski

Chopra

, et al. Deep neural network improves fracture detection by clinicians. Proc Natl Acad Sci USA 2018;115:11591–11596.

16.

Weikert

Noordtzij

Bremerich

, et al. Assessment of a deep learning algorithm for the detection of rib fractures on whole-body trauma computed tomography. Korean J Radiol 2020;21:891–899.

17.

Zhou

Wang

Tang

, et al. Automatic detection and classification of rib fractures on thoracic CT using convolutional neural network: accuracy and feasibility. Korean J Radiol 2020;21:869–879.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB