Development and evaluation of a computer-based decision support system for diffuse lung diseases at high-resolution computed tomography

Abstract

Background

High-resolution computed tomography (HRCT) is essential in narrowing the possible differential diagnoses of diffuse and interstitial lung diseases.

Purpose

To investigate the value of a novel computer-based decision support system (CDSS) for facilitating diagnosis of diffuse lung diseases at HRCT.

Material and Methods

A CDSS was developed that includes about 100 different illustrations of the most common HRCT signs and patterns and describes the corresponding pathologies in detail. The logical set-up of the software facilitates a structured evaluation. By selecting one or more CT patterns, the program generates a ranked list of the most likely differential diagnoses. Three independent and blinded radiology residents initially evaluated 40 cases with different lung diseases alone; after at least 12 weeks, observers re-evaluated all cases using the CDSS.

Results

In 40 patients, a total of 113 HRCT patterns were evaluated. The percentage of correctly classified patterns was higher with CDSS (96.8%) compared to assessment without CDSS (90.3%; P < 0.01). Moreover, the percentage of correct diagnosis (81.7% vs. 64.2%) and differential diagnoses (89.2% vs. 38.3%) were superior with CDSS compared to evaluation without CDSS (both P < 0.01).

Conclusion

Addition of a CDSS using a structured approach providing explanations of typical HRCT patterns and graphical illustrations significantly improved the performance of trainees in characterizing and correctly identifying diffuse lung diseases.

Keywords

Interstitial lung disease diffuse lung disease computed tomography decision support system

Introduction

Pulmonary diseases are a leading cause of death and disability worldwide. Many disease entities affecting the lung have similar clinical presentation, like dyspnea and cough, making it difficult for clinicians to characterize, diagnose, and treat (1). On this account, chest radiography and computed tomography (CT) are crucial modalities for first-line assessment of such patients. Furthermore, high-resolution computed tomography (HRCT) is essential in narrowing the possible differential diagnoses and enables correct diagnosis of various pulmonary diseases, especially of diffuse and interstitial lung diseases (ILDs) (2,3). Diffuse lung diseases are a group of pulmonary disorders which affect the lung parenchyma, and each have separate and often unique HRCT patterns (3). These disorders can appear solely in the lung, or as a significant component of a multisystem disorder (4 –6).

Computer-based systems are increasingly being used as an aid for detection and interpretation of diseases, particularly in radiology (7,8). Computer systems that assist radiologists in the interpretation of images can be separated into computer-aided detection (CAD) systems and computer-based decision support systems (CDSS) (9). In addition, machine leaning techniques are becoming more and more important for CAD systems, especially for pattern recognition and lesion characterization as these algorithms can learn from data (10). Although multiple prior studies have described computer-aided detection and diagnosis methods for lung nodules, only a few studies have evaluated computer-based systems for other lung abnormalities (9,11 –16). Moreover, experience with CDSS at HRCT is scarce, and there has been limited evaluation of their diagnostic performance (17 –19).

Because of the large amount of various HRCT patterns and different pulmonary diseases, the correct diagnosis of diffuse lung diseases can be challenging. We designed a new software that assists in the evaluation of HRCT using a structured approach with descriptions of typical HRCT patterns and graphic illustrations. Thus, the aim of the present study was to investigate whether this CDSS can improve diagnostic performance in characterizing and identifying diffuse lung diseases at HRCT.

Material and Methods

Software development

Our CDSS was coded in C++ and runs on the operating system Microsoft Windows (Microsoft, Redmond, WA, USA). The computer program includes 10 different sections based on the four most basic patterns of HRCT providing explanations and graphical illustrations: reticular opacities; lung nodules; increased lung density; and decreased lung density (20). In addition, the software contains pathologic HRCT findings of the pleura and chest wall. The first section focuses on linear and reticular opacities, intra- and interlobular septal thickening, as well as interstitial fibrosis and honeycomb pattern (20 –25). Two further sections include the basic pattern lung nodules and are further classified in smaller lung nodules (section 2) (Fig. 1) and pulmonary masses (section 3). Every radiographic pattern can be selected separately and is described in more detail under the corresponding sketch (Fig. 2). The chapter on increased lung density is subdivided into consolidation (section 4) and ground-glass attenuation (section 5). Furthermore, decreased lung density is divided into emphysema (section 6) (Fig. 3), cystic pattern (section 7), and cavitary lesions (section 8). Different types of bronchiectasis are specified in section 9. The last section (section 10) covers pleural and chest wall lesions, e.g. pleural masses and calcifications. Each part focuses on the associated HRCT patterns, on average 9–10 per section. A navigation bar at the top of each window allows to switch between the different topics.

Fig. 1.

Section of the software displaying descriptions of small pulmonary nodules. Small nodules can be classified according to their distribution on high-resolution computed tomography as centrilobular, perilymphatic, or random. Moreover, centrilobular nodules can appear as tree-in-bud pattern or with ground-glass attenuation. A navigation bar at the top of the window allows to switch between the different topics.

Fig. 2.

Same section as in Fig. 1 after centrilobular distribution has been selected. By selecting the pattern, a more detailed description appears under the picture. After activating the checkbox, the algorithm takes the pattern into account to find the correct diagnosis. The right button at the bottom of this page opens the diagnostic evaluation and displays a ranked list of appropriate differential diagnoses.

Fig. 3.

Section of the software displaying different types of emphysema. Pulmonary emphysema can be classified into centrilobular, panlobular, and paraseptal emphysema. The high-resolution computed tomography pattern illustrated here shows a centrilobular emphysema.

Overall, the CDSS includes 97 different radiographic patterns, and more than 190 different diseases of the thorax are implemented in the program. This computer program represents a separate program which is not built into the picture archiving and communication system (PACS). To obtain the correct diagnosis, the software uses an algorithm based on a scoring system. This method considers frequent diseases as well as rare diseases following the classification of the current literature, recent data, and expert knowledge (6,20,22,26). Rare and frequent diseases for each HRCT pattern are assigned to a certain point score according to their likelihood. By choosing two or more CT patterns, the program summates the corresponding scores. Commonly described combinations of HRCT patterns result in extra scores, whereas exclusion criteria of certain diseases are assigned to negative scores. Each of the 190 different diffuse lung diseases included was implemented manually into the program. As such a rating system for these large number of diffuse lung diseases does not yet exist, we used expert knowledge to adapt the scores of some diseases and differential diagnoses in the validation phase. After selecting the occurring HRCT patterns, the program produces a ranked list of possible differential diagnoses for each case (Fig. 4). The CDSS is programmed to show only the 20 most likely differential diagnoses sorted by their score and the diagnosis on top of the list is considered as the main diagnosis. The CDSS is currently not commercially available.

Fig. 4.

This page shows the output of the computer-based decision support system. After selecting the occurring high-resolution computed tomography patterns (in this case centrilobular nodules), the program produces a ranked list of possible differential diagnoses.

Study population

The institutional review board approved this retrospective study with a waiver of informed consent. Our institutional CT database was retrospectively searched to identify patients with different ILDs. Thereafter, a group of four radiologists (with 8–30 years of experience in HRCT) selected suitable cases to include rare as well as more common diffuse lung diseases. Further selection criteria for diffuse lung diseases included the following that every single lung disease may only occur once, and clinical data of all study patients were accessible in the local electronic medical records system.

The final study population consisted of 40 patients (mean age = 59.2 ± 14.9 years, age range = 21–82 years), including 24 men (mean age = 60.3 ± 15.5 years, age range = 32–82 years) and 16 women (mean age = 57.6 ± 11.8 years, age range = 21–79 years). All included patients had undergone a clinically indicated CT examination.

CT image acquisition

CT examinations were acquired on a 192-slice dual-source CT system (SOMATOM Force, Siemens Healthcare, Forchheim, Germany), on a 128-slice dual-source CT system (SOMATOM Definition Flash, Siemens), or on a 128-slice single-source CT unit (SOMATOM Definition AS, Siemens). Automatic exposure control was used in all groups (CAREdose 4D, Siemens). All datasets were acquired in craniocaudal direction in deep inspiratory breath-hold and in the single-energy mode.

Detector collimation of 192 × 0.6 mm and 128 × 0.6 was achieved with the dual-source CT systems, respectively, and the collimation of the single-source CT unit was 128 × 0.6 mm. The tube voltage was set at 120 kV, the reference tube current-time product was 150 mAs, and the pitch was 1.0 for all CT systems. All series were reconstructed as axial and coronal slices, with a section thickness of 1.0 mm and an increment of 0.5 mm.

Standard of reference

The study population consisted of 40 patients with known diffuse lung diseases and corresponding HRCT findings. A subset of 28 patients underwent additional lung biopsy. The other 12 patients had clinically proven diagnosis of diffuse lung diseases under consideration of all clinical data (e.g. imaging data, medical history, blood analysis, pulmonary function test, response to treatment, outcome). The reference standard of the included diffuse lung diseases was made by a multidisciplinary team and was secured at the beginning of the study. In addition, two radiologists with more than 15 years of HRCT experience and one pulmonologist with more than 10 years in clinical practice defined the number of occurring HRCT patterns and differential diagnoses. In case of discrepant ratings, a consensus reading was performed between the expert reviewers. Table 1 shows all included diffuse lung diseases with the associated number of CT patterns and differential diagnoses.

Table 1.

List of cases included in the study.

	Disease	No. of CT patterns	Differential diagnoses
1	ABPA	3	3
2	Acute hypersensitivity pneumonitis	2	5
3	Alpha-1 antitrypsin deficiency	1	2
4	ARDS	3	4
5	Asbetosis	5	4
6	Aspergillosis	3	3
7	Atypical pneumonia	2	4
8	Bronchiolitis	2	2
9	Candida pneumonia	2	3
10	Chronic hypersensitivity pneumonitis	3	3
11	Cystic fibrosis	3	3
12	DIP	3	7
13	Emphysema with bronchitis	2	2
14	Eosinophilic pneumonia	4	5
15	GPA	4	5
16	GvHD	3	3
17	LCH	3	4
18	LIP	3	6
19	Lobar pneumonia	2	3
20	Lymphangioleiomyomatosis	2	3
21	Mesothelioma	3	3
22	Miliary tuberculosis	2	2
23	NSCLC	2	3
24	NSIP	4	5
25	NTM	2	2
26	Obliterative bronchiolitis	2	5
27	OP	3	5
28	Pulmonary alveolar proteinosis	2	6
29	Pulmonary edema	2	4
30	Pulmonary hemorrhage	3	2
31	Radiation-induced lung injury	3	3
32	RB	3	3
33	Round atelectasis	2	3
34	Sarcoidosis	3	6
35	Scleroderma	5	5
36	Septic embolism	3	2
37	Silicosis	3	3
38	SLE	3	3
39	Tuberculosis	3	3
40	UIP	5	5
	Total	113	147

Data are number of associated CT patterns and differential diagnoses defined by the group of experts.

ABPA, allergic bronchopulmonary aspergillosis; ARDS, acute respiratory distress syndrome; CF, cystic fibrosis; CT, computed tomography; DIP, desquamative interstitial pneumonia; GvHD, graft-versus-host disease; LCH, Langerhans cell histiocytosis; LIP, lymphocytic interstitial pneumonia; GPA, granulomatosis with polyangiitis; NSCLC, non-small-cell lung carcinoma; NSIP, non-specific interstitial pneumonia; NTM, pulmonary non-tuberculous mycobacterial infection; OP, organizing pneumonia; RB, respiratory bronchiolitis; SLE, systemic lupus erythematosus; UIP, usual interstitial pneumonia.

Performance analysis

Three radiology residents with 4–6 years of experience in CT imaging independently performed the performance analysis in individual sessions. Window settings were arranged in the standard lung tissue window (level = –500 HU, width = 2000 HU) by default, but observers could adjust these values to improve visualization. During the first reading session, the observers received a questionnaire to describe the visible HRCT patterns of the lung parenchyma and the pleura to provide the most likely diagnosis and differential diagnoses to the best of their knowledge. This questionnaire included a list of all CT pattern and differential diagnosis that were implemented into the program. The second reading session involved evaluation of the same HRCT datasets using CDSS. The observers marked the most similar HRCT pattern in the software at their own discretion. Thereafter, the CDSS evaluated their findings and generated a ranked list of possible differential diagnoses according to their likelihood. For each case, readout sessions were separated by at least 12 weeks to avoid potential recall bias.

After the CDSS had generated a list of possible diagnoses for each case, the group of experienced physicians who had determined the reference standard reviewed the diagnoses to determine whether the program had suggested any additional differential diagnoses that were appropriate or whether any cases should be eliminated.

Statistical analysis

Statistical analysis was performed with commercially available software packages (IBM SPSS Statistics, version 24, IBM, Armonk, NY, USA and MedCalc Statistical Software version 18, MedCalc Software bvba, Ostend, Belgium). Proportions of the numbers of correctly recognized HRCT patterns, correct diagnoses, and appropriate differential diagnoses were calculated for each observer, respectively, and the results were averaged. Furthermore, a score for not appropriated CT patterns and differential diagnoses was calculated that reflects the average number of additional CT pattern and differential diagnoses suggested by the computer and the reviewers. The overall difference between the performance analyses was tested for statistical significance using the Wilcoxon signed-rank test. An alpha level of less than 0.05 was considered to indicate a significant difference.

Inter- and intrarater agreement were analyzed by means of intraclass correlation coefficients (ICC) analysis with 95% confidence intervals (CI). The ICC value was interpreted in the following way (27): <0.40 = poor agreement; 0.40–0.59 = fair agreement; 0.60–0.74 = good agreement; and 0.75–1.0 = excellent agreement.

Results

Pattern recognition

Table 2 shows the results of the performance analysis. In 40 patients, a total of 113 HRCT patterns were evaluated. The reviewers correctly detected on average 106.3 HRCT patterns when using CDSS, whereas 97.0 patterns were detected by the radiologists alone during the first reading session without using the software. Subsequently, the percentage of correctly classified HRCT patterns was significantly higher (P < 0.01) for evaluation with software support (96.8%), compared to image evaluation without CDSS (90.3%). Results of the subgroup analysis in the group of patients with additional lung biopsy or with clinically proven diagnosis are presented in Table 3.

Table 2.

Results of the performance analysis.

Parameter	Performance radiologists (%)	Performance CDSS (%)	P value	Interrater reliability	Intrarater reliability
CT pattern	90.3	96.8	<0.01	0.72 (0.59–0.83)	0.76 (0.67–0.82)
Correct diagnosis	64.2	81.7	<0.01	0.40 (0.21–0.59)	0.17 (–0.02 to 0.34)
Differential diagnoses	38.3	89.2	<0.01	0.90 (0.84–0.94)	0.48 (0.33–0.60)
Additional diagnoses	0.2 per case	3.5 per case	–	–	–

Values are the mean based on the results of the two observers. Values in brackets are 95% confidence intervals.

CDSS, computer-based decision support system; CT, computed tomography.

Table 3.

Results of the subgroup analysis.

Parameter	Performance radiologists (%)	Performance CDSS (%)	P value	Performance radiologists (%)	Performance CDSS (%)	P value
	Patients with lung biopsy			Patients with clinically proven diagnosis
CT pattern	89.6	97.4	<0.01	92.1	95.4	<0.19
Correct diagnosis	60.7	75.0	<0.01	72.2	97.2	<0.06
Differential diagnoses	36.6	85.5	<0.01	42.5	97.9	<0.01

Values are the mean based on the results of the two observers. Values in brackets are 95% confidence intervals.

CDSS, computer-based decision support system; CT, computed tomography.

Inter-observer agreement for the detection of HRCT patterns was good between both reading sessions (ICC = 0.72, 95% CI = 0.59–0.83). Furthermore, intrarater reliability showed excellent agreement (ICC =0.76, 95% CI = 0.67–0.82).

Correct diagnosis

The radiologists correctly identified the diagnosis in 64.2% of the cases without CDSS and in 81.7% of the cases with the addition of CDSS. Therefore, performance indices regarding confirmation of the correct diagnosis were substantially higher when using CDSS compared to the evaluation without CDSS (P < 0.01) (Fig. 5).

Fig. 5.

A 73-year-old male patient with pulmonary hemorrhage of the right upper lobe. This case includes the computed tomography patterns reticular opacities, intra- and interlobular septal thickening, and ground-glass attenuation. The diagnostic performance for pattern recognition and identifying the correct diagnosis of this case was higher in the evaluation with CDSS compared to the evaluation without software support. The differential diagnoses provided by the CDSS were atypical pneumonia and interstitial edema. CDSS, computer-based decision support system.

Statistical analysis showed fair interrater (ICC = 0.40, 95% CI = 0.21–0.59) and a poor intrareader agreement (ICC = 0.17, 95% CI = –0.02 to 0.34).

Differential diagnoses

In 12 instances, the program proposed differential diagnoses that the physicians had not originally listed but found reasonable to consider as appropriate in retrospect. After this final validation stage, 147 appropriate differential diagnoses remained for the 40 cases. Application of the CDSS resulted in 130.0 appropriate differential diagnoses (3.3 per case), whereas the radiologists only noted 56.3 differential diagnoses (1.4 per case) without using CDSS. Consequently, statistical analysis showed superior results when using CDSS regarding the percentage of correct differential diagnoses (89.2%) compared to image evaluation without software support (38.3%; P < 0.01). However, the software also listed 3.5 additional differential diagnoses per case which were not defined as accurate by the group of experts. By comparison, reviewers only mentioned 0.2 additional diagnoses per case without using the software. The mean duration to enter the information and receive the results of the CDSS was 84 s (range = 32–165 s).

Interrater agreement analysis showed an excellent agreement (ICC = 0.90, 95% CI = 0.84–0.94), whereas intrarater analysis showed a fair agreement (ICC = 0.48; 95% CI = 0.33–0.60).

Discussion

The aim of the present study was to investigate the impact of a novel CDSS for detecting and diagnosing diffuse lung diseases as seen on HRCT. We found that the performance for the identification of the correct diagnosis and appropriate differential diagnoses could be improved using our CDSS compared to HRCT evaluation without. More specifically, addition of CDSS resulted in a higher number of approximately 17% correct diagnoses and 51% differential diagnoses compared to the evaluation of the radiologists alone. Furthermore, with software support, the radiologists were able to detect more HRCT patterns compared to the evaluation without CDSS. However, the program presented a relatively long list of differential diagnoses for each case that in some cases also included some diagnoses which a knowledgeable radiologist or pulmonologist would not consider as appropriate. In future versions of the CDSS, we aim to reduce the number of differential diagnosis as a higher number of differential diagnosis could be a confounding factor in the daily clinical routine. On the other hand, the program suggested some diagnoses that the experts later agreed were adequate to include in the list of appropriate differential diagnosis. Moreover, the program has additional functions that we did not evaluate within the context of this study. These functions include displaying the HRCT signs and patterns, describing the HRCT patterns in detail, and guiding the user through a structured HRCT evaluation. Additionally, the CDSS provides scores that indicate the relative likelihood of each diagnosis. In our opinion, the CDSS might be most powerful for radiology trainees, not only to diagnose the correct lung disease, but also to learn the different HRCT patterns and differential diagnoses.

While several prior studies have evaluated the effects of computer-based diagnosis systems for lung abnormalities, there has been limited evaluation of their performance. Sluimer et al. (17,18) presented an automated method for textural analysis of complete HRCT image slices that can be used to indicate abnormal areas of the lung tissue. In their study, the computer program was able to identify abnormal lung tissue with an accuracy comparable to that of radiologists (17). In a recent study performed by Jun et al. (16) a computer system was evaluated that differentiates between usual interstitial pneumonia (UIP) and non-specific interstitial pneumonia (NSIP). The results of this study indicated that the computer program can be used to provide an initial screening method for UIP and NSIP.

The present study has some limitations. First, we performed a retrospective analysis of HRCT examinations and used all available clinical and imaging data to establish a reference standard. A prospective large-scale study should be performed to validate our initial experience. Second, the study population of 40 patients is rather small to validate the software for other diffuse lung diseases and bias our study results. However, the main intention of the present study was to introduce a new CDSS for diffuse lung disease and to evaluate this software in a first clinical test. Another limitation is that diagnostic performance is conditional on reader experience and thus our results depend on the average experience of the three reviewers. Additionally, no attending radiologists took part in the validation study of the software. However, all our readers had reasonable training and experience of at least four years. Third, we allowed for a time interval of 12 weeks between the evaluation of the radiologists alone and with software support. Therefore, a certain learning effect in the evaluation with CDSS can be expected. On the other hand, including different cases for the evaluation of the software system would limit the comparability of the results. Fourth, a specific evaluation of the differential diagnoses according to their likelihood was not performed. Fifth, one radiologist who took part in the consensus reading was also directly involved in the software development. This might be a bias in the performance analysis. Lastly, the patient population in this study contained only a specific number of patients with different diffuse lung diseases as we included each disease only once. Therefore, the results of this study cannot be directly translated to patients with other lung diseases. However, due to the similar algorithms for detection and identifying of HRCT patterns, we expect results to be comparable.

In conclusion, the results of our study demonstrate that our novel CDSS providing descriptions of HRCT signs and graphical illustrations can significantly improve the performance of trainees in detecting and correctly identifying diffuse lung diseases. Application of our CDSS routinely in the evaluation of HRCT may be helpful for observers with limited experience as well as for assessment of cases with multiple HRCT patterns.

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: SSM received speakers’ fees from Siemens Healthcare. JLW received speakers’ fees from GE Healthcare and Siemens Healthcare.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Simon S Martin

References

Jawad

Chung

Lynch

, et al. Radiological approach to interstitial lung disease: a guide for the nonradiologist. Clin Chest Med 2012; 33:11–26.

Wells

AU.

High-resolution computed tomography in the diagnosis of diffuse lung disease: a clinical perspective. Semin Respir Crit Care Med 2003; 24:347–356.

Horst

Gholipour

Nair

, et al. Differential diagnoses of fibrosing lung diseases. Br J Radiol Open 2019; 1:20190009.

Kim

Lee

Johkoh

, et al. Interstitial lung diseases associated with collagen vascular diseases: radiologic and histopathologic findings. Radiographics 2002; 22:151–165.

Tazelaar

Viggiano

Pickersgill

, et al. Interstitial lung disease in polymyositis and dermatomyositis. Clinical features and prognosis as correlated with histologic findings. Am Rev Respir Dis 1990; 141:727–733.

Raghu

Remy-Jardin

Myers

, et al. Diagnosis of Idiopathic Pulmonary Fibrosis. An Official ATS/ERS/JRS/ALAT Clinical Practice Guideline. Am J Respir Crit Care Med 2018; 198:e44–e68.

Huang

Park

Yan

, et al. Added value of computer-aided CT image features for early lung cancer diagnosis with small pulmonary nodules: a matched case-control study. Radiology 2018; 286:286–295.

Liang

Tang

, et al. Low-dose CT screening for lung cancer: computer-aided detection of missed lung cancers. Radiology 2016; 281:279–288.

Firmino

Morais

Mendoca

, et al. Computer-aided detection system for lung cancer in computed tomography scans: review and future prospects. Biomed Eng Online 2014; 13:41.

10.

Suzuki

Machine learning in computer-aided diagnosis of the thorax and colon in CT: A survey. IEICE Trans Inf Syst 2013;E96-D:772–783.

11.

Goo

JM.

A computer-aided diagnosis for evaluating lung nodules on chest CT: the current status and perspective. Korean J Radiol 2011; 12:145–155.

12.

Gurung

Maataoui

Khan

, et al. Automated detection of lung nodules in multidetector CT: influence of different reconstruction protocols on performance of a software prototype. Rofo 2006; 178:71–77.

13.

Madero Orozco

Vergara Villegas

Cruz

Sanchez VG

, et al. Automated system for lung nodules classification based on wavelet feature descriptor and support vector machine. Biomed Eng Online 2015; 14:9.

14.

Chen

Suzuki

MacMahon

Development and evaluation of a computer-aided diagnostic scheme for lung nodule detection in chest radiographs by means of two-stage nodule enhancement with support vector classification. Med Phys 2011; 38:1844–1858.

15.

Way

Sahiner

Chan

, et al. Computer-aided diagnosis of pulmonary nodules on CT scans: improvement of classification performance with nodule surface features. Med Phys 2009; 36:3086–3098.

16.

Jun

Park

Seo

, et al. Development of a computer-aided differential diagnosis system to distinguish between usual interstitial pneumonia and non-specific interstitial pneumonia using texture- and shape-based hierarchical classifiers on HRCT images. J Digit Imaging 2018; 31:235–244.

17.

Sluimer

Prokop

Hartmann

, et al. Automated classification of hyperlucency, fibrosis, ground glass, solid, and focal lesions in high-resolution CT of the lung. Med Phys 2006; 33:2610–2620.

18.

Sluimer

van Waes

Viergever

, et al. Computer-aided diagnosis in high resolution CT of the lungs. Med Phys 2003; 30:3081–3090.

19.

Salisbury

Gross

Chughtai

, et al. Development and validation of a radiological diagnosis model for hypersensitivity pneumonitis. Eur Respir J 2018; 52:1800443.

20.

Elicker

Pereira

Webb

, et al. High-resolution computed tomography patterns of diffuse interstitial lung disease with clinical and pathological correlation. J Bras Pneumol 2008; 34:715–744.

21.

Antoniou

Margaritopoulos

Tomassetti

, et al. Interstitial lung disease. Eur Respir Rev 2014; 23:40–54.

22.

Collins

CT signs and patterns of lung disease. Radiol Clin North Am 2001; 39:1115–1135.

23.

Devaraj

Imaging: how to recognise idiopathic pulmonary fibrosis. Eur Respir Rev 2014; 23:215–219.

24.

Hering

Tuengerthal

Kraus

[ Standardized CT/HRCT-classification of the German Federal Republic for work and environmental related thoracic diseases]. Radiologe 2004; 44:500–511.

25.

Jacobi

Thalhammer

Grundmuster im CT der Lunge und ihre Differenzialdiagnose. Radiologie up2date 2006; 6:311–334.

26.

Gotway

Reddy

Webb

, et al. High-resolution CT of the lung: patterns of disease and differential diagnoses. Radiol Clin North Am 2005; 43:513–542, viii.

27.

Cicchetti

Sparrow

SA.

Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 1981; 86:127–137.