A Signal-detection Experiment Measuring the Effect of Computer-aided Detection on Radiologists' Performance

Abstract

Purpose. To evaluate how the specificity and sensitivity of computer-aided detection (CADe) algorithm outputs affected radiologists' diagnostic performances, the authors studied the effects of 25 simulated CADe algorithms with various sensitivities and specificities (from 60% to 100%). Methods. Six novice radiologists read 200 images that were produced by computer and randomly displayed on CRT, and their detection performances were evaluated with receiver operating characteristic analysis. Results. There were significant differences in performance among CADe types (p < 0.001). Discussion and Conclusion. The overall accuracy of CADe outputs is the most significant factor affecting radiologists' performances in detection and interpretation of images. There is an approximately linear relationship between the sensitivity (specificity) of the CADe output and the reader's sensitivity (specificity), and the slope of reader sensitivity (specificity) as a linear function of CADe sensitivity (specificity) can be considered to be a positive number less than unity. Key words: computers, diagnostic aid; probabilistic judgments; receiver operating characteristic curve (ROC); radiologist decision making; diagnostic concordance. (Med Decis Making 2000;20:343-351)

Get full access to this article

View all access options for this article.

References

Giger ML , Doi K , MacMahon H , Metz CE , Yin F-F

Pulmonary nodules. computer-aided detection in digital chest images

Radiographics. 1990;10:41-51.

Chan HP , Doi K , Vyborny CJ , et al. Improvement in radiologists' detection of clustered microcalcifications on mammograms The potential of computer-aided diagnosis Invest Radiol 1990;25:1102-10.

Matsumoto T , Yoshimura H , Doi K , et al. Image feature analysis of false-positive diagnoses produced by automated detection of lung nodules. Invest Radiol. 1992;27:587-97

Yoshimura H , Giger ML , Doi K , MacMahon H , Montner SM Computerized scheme for the detection of pulmonary nodules. A nonlinear filtering technique Invest Radiol 1992 ;27: 124-9

Vyborny CJ , Giger ML. Computer vision and artificial intelligence in mammography. AJR 1994;162:699-708.

Kobayashi T , Xu X-W , MacMahon H , Metz CE , Doi K. Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs. Radiology. 1996;199:843-8

Metz CE , Herman BA , Shen JH. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med 1998;17.1033-53.

Dorfman DD , Berbaum KS , Metz CE. Receiver operating characteristic rating analysis Generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992;27:723-31.

Roe CA , Metz CE Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol. 1997 ;4:298-303.

10.

Efron B , Tibshirani RJ. The jackknife In An Introduction to the Bootstrap New York: Chapman and Hall, 1993.