Diagnosis of the coronavirus disease 2019 with chest computed tomography: A retrospective inter-observer agreement study between radiologists and clinicians

Abstract

Background:

Effective triage and early detection are very important for the control and treatment of coronavirus disease 2019. For this purpose, reverse transcription polymerase chain reaction and chest computed tomography are used in emergency departments.

Objective:

The aim of the study was to examine the diagnostic performance of computed tomography and to compare the inter-observer agreement among radiologists and between clinicians, in a coronavirus disease 2019 pneumonia high-prevalence area.

Methods:

After exclusions, 534 patients were retrospectively included in this study. Reverse transcription polymerase chain reaction was considered as gold standard for diagnosis. All computed tomography images were independently reviewed by two radiologists who were blinded to reverse transcription polymerase chain reaction results and other clinical information. Each computed tomography scan was scored in four categories as typical, intermediate, atypical, and negative, regarding coronavirus disease 2019 pneumonia according to Radiological Society of North America guideline. As for the evaluation of the diagnostic performance, typical and intermediate appearances were accepted as positive for coronavirus disease 2019. In addition, the computed tomography scans were scored by two clinicians as coronavirus disease 2019 positive and negative.

Results:

The study group included 534 patients after the exclusion criteria. As a result of the repeated reverse transcription polymerase chain reaction tests, 396 (74%) patients were diagnosed with coronavirus disease 2019, 138 (26%) patients had a negative result and were evaluated as a control group. When the reverse transcription polymerase chain reaction results were referenced as the gold standard; the accuracy rates of radiologists and clinicians (R1, R2, C1, and C2) in the diagnosis of coronavirus disease 2019 were 78%, 79%, 73%, and 71%, their sensitivity rates were 83%, 83%, 74%, and 75%, and the negative predictive values were 57%, 58%, 49%, and 46%, respectively. Inter-observer agreements among the reviewers ranged from good to excellent.

Conclusions:

Radiological Society of North America guideline related to coronavirus disease 2019 has excellent inter-observer agreement among chest radiologists. In this study, radiologists and clinicians have presented similar and good diagnostic performances in the evaluation of coronavirus disease 2019–suspected patients with chest computed tomography in high-epidemic area.

Keywords

Coronavirus disease 2019 reverse transcriptase polymerase chain reaction severe acute respiratory syndrome coronavirus 2 computed tomography

Introduction

The coronavirus disease 2019 (COVID-19) broke out in the Hubei area in China, during December 2019 and caused pneumonia by mostly affecting lower respiratory tracts. The novel virus is severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and despite great efforts to prevent, its spread around the world continues.

Due to high number of asymptomatic patients and rapid infection potential and the absence of a recent effective medical treatment or a vaccine, the early diagnosis of infected individuals and isolation from the community have great significance. Despite its relatively low sensitivity, the real-time reverse transcription polymerase chain reaction (RT-PCR) is accepted as the gold standard in the diagnosis of COVID-19.^1–5 However, the sensitivity reported in clinical trials can vary between 42% and 83% according to the prevalence of the disease, the duration between the beginning of the symptoms and applying to a healthcare organization, the viral load in the sample, the place of the sample collection, problems in terms of transportation, and the quality of the kits used.^2,5–8 In addition, due to the fact that the RT-PCR results take time, kit supplies are being limited and not easily accessible especially in underdeveloped or developing countries, additional scanning and diagnostic tests are required. Chest computed tomography (CT) is not recommended to be used as COVID-19 scanning and diagnostic test by many radiology associations and communities, but it is commonly used with this purpose in our country and in the world.

Numerous articles examining the diagnostic performance of a chest CT in COVID-19 pneumonia are available in the literature.^9–14 However, there are limited articles examining the inter-observer agreement among the evaluators on this matter^10,11 and articles about the comparison of radiologists and clinicians are not present.

The aim of this study is to examine the diagnostic performance of a chest CT examination and compare the consistency of radiologists among themselves and between clinicians, in a COVID-19 pneumonia high-prevalence area (in the peak of the disease), considering the COVID-19 standardized diagnostic criteria recommended by the Radiological Society of North America (RSNA).

Methods

This retrospective study was approved by our hospital ethics committee. The need for signed informed consent was waived.

Patient population

Our hospital is a training and research hospital and one of the referral centers for pandemic in Istanbul, Turkey. In this study, we included patients who presented to COVID-19 clinic of our hospital between 4 April and 11 April 2020. In our study, a total of 714 were examined, who had a RT-PCR test due to high suspicion of COVID-19 during these dates (symptomatic or asymptomatic cases that had contact history). Patients who did not have chest CT in picture archiving and communication system (PACS) (103 patients) and patients with time difference between CT scan and sample taking for RT-PCR exceeded 3 days (72 patients) were excluded from the study. After exclusions, the final cohort consisted of 534 patients, 230 (43%) of them were female, and 304 (57%) of them were male, with average age of 48 years (standard deviation (SD) ± 17). RT-PCR was considered as gold standard for diagnosis, and any positive result from the first or repetitive RT-PCR results was considered the definitive diagnosis for COVID-19.

Imaging protocol and techniques

The chest CT scans in this study were obtained using the standard dose protocol of our hospital with a 128-slice multi-detector CT scanner (Optima; General Electric Healthcare, Wisconsin, USA). All CT scans were performed during a single breath-hold without contrast administration. The imaging parameters used were as follows: tube voltage: 120 kV p, tube current (regulated by automatic dose modulation): 80–200 mA s, slice thickness: 5 mm, matrix: 512 × 512, and field of view: 350 mm× 350 mm. The scans were retrospectively reconstructed in the sagittal and coronal planes (1.25 mm thickness, 0.625 mm spacing).

Imaging analysis

All CT images were reviewed by two radiologists who were blinded to RT-PCR results and other clinical information, with 9 and 12 years of experience in interpreting chest CT imaging (F.C.—radiologist 1 (R1) and O.G.—radiologist 2 (R2), respectively), on a PACS imaging workstation (Infinitt PACS; Infinitt Healthcare, Seoul, Korea). For patients who had more than one CT examination in the system, the first CT was taken into consideration. The radiologists evaluated the CT scans independently from each other based on the RSNA statement on reporting chest CT findings related to COVID-19.¹⁵ Each CT scan was scored in four categories as typical, intermediate, atypical, and negative regarding COVID-19 pneumonia according to this guideline. As for the evaluation of the diagnostic performance, typical and intermediate appearances were accepted as positive for COVID-19 and cases that were atypical and did not display pneumonia were accepted as negative for COVID-19. In addition, the CT scans were independently scored as COVID-19 positive and negative in the PACS imaging workstation by two clinicians who were blinded to the RT-PCR results and other clinical information (one pulmonologist A.M.D., clinician 1 (C1), the other being an emergency medicine specialist M.C., clinician 2 (C2), with 6 and 15 years of experience, respectively), in light of the CT findings identified in the literature.^16–18

Statistical analysis

IBM SPSS 22.0 was used for statistical analysis. Continuous variables were expressed as mean ± SD and compared through the analysis of variances or independent sample t-tests. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of chest CT were calculated using RT-PCR results as the gold standard. A 95% confidence interval was provided by the Wilson score method. Inter-observer agreement was evaluated using kappa statistics (slight agreement: 0.01 ⩽ κ ⩽ 0.20, fair agreement: 0.21 ⩽ κ ⩽ 0.40, moderate agreement: 0.41 ⩽ κ ⩽ 0.60, good agreement: 0.61 ⩽ κ ⩽ 0.80, and excellent agreement: 0.81 ⩽ κ ⩽ 1.00). Qualitative data were compared using a chi-square test. p < 0.05 was considered statistically significant.

Results

The study group included 534 patients in total after the exclusion criteria, consisting of 304 (57%) males and 230 (43%) females, and the average age being 48 (±17) years. As a result of the repeated RT-PCR tests in these 534 patients, 396 (74%) patients were diagnosed with certain COVID-19, 138 (26%) patients had a negative result and were evaluated as a control group. RT-PCR was repeated for 89 of the 138 patients who had a negative result, due to the inconsistency between the patient clinic, radiology, and tests results. The test was not repeated with 49 of the 138 patients who had a negative test results, since COVID-19 pneumonia was not considered both clinically and radiologically. In the RT-PCR negative patient subgroup, an average of two tests ranging from 1 to 7 was performed. In the cases, the time difference between the chest CT and RT-PCR examinations varied from 0 to 3 days and was 1.1 days on average. Considering the 534 patients in total that were evaluated by radiologists, the number of patients that had typical COVID-19 CT findings for R1 and R2 were 261 (48.9%) and 265 (49.6%), respectively. The number of patients for intermediate, atypical findings, and negative for pneumonia for each radiologist were 120(22.5%)–110(20.6%), 25(4.7%)–29(5.4%), and 128(24%)–130(24.3%), respectively. The distribution for RT-PCR positive and negative patient subgroups of the same values are specified in Table 1.

Table 1.

Correlation between chest CT categories of radiologists and RT-PCR.

CT categories for COVID-19 appearance	Positive RT-PCR	Negative RT-PCR	Total
R1 typical	233 (58.8)	28 (20.3)	261 (48.9)
R1 intermediate	97 (24.5)	23 (16.7)	120 (22.5)
R1 atypical	10 (2.5)	15 (10.9)	25 (4.7)
R1 negative	56 (14.1)	72 (52.2)	128 (24)
R2 typical	244 (61.6)	21 (15.2)	265 (49.6)
R2 intermediate	86 (21.7)	24 (17.4)	110 (20.6)
R2 atypical	11 (2.8)	18 (13)	29 (5.4)
R2 negative	55 (13.9)	75 (54.3)	130 (24.3)

R: radiologist; COVID-19: coronavirus disease 2019; RT-PCR: reverse transcriptase polymerase chain reaction; CT: computed tomography.

Values are expressed as n (%).

When the RT-PCR results were referenced as the gold standard, during the peak period of the pandemic in a high-prevalence area, the sensitivity, specificity, PPV, NPV, and accuracy values of the chest CT in the diagnosis of COVID-19 for each radiologist and clinician are given in Table 2. In Table 3, inter-observer agreement among the radiologists and among clinicians for diagnosing of COVID-19 pneumonia in the total cohort, RT-PCR positive, and RT-PCR negative patient subgroups was examined.

Table 2.

The diagnostic performance of radiologists and clinicians with chest CT for COVID-19 pneumonia.

Observers	Results (n)				Diagnostic performance (%)					p value
Observers	TP	TN	FP	FN	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	Accuracy (95% CI)	p value
R1	330	87	51	66	83.33 (81.57–88.77)	63.04 (53.75–70.52)	86.61 (82.98–89.97)	56.86 (51.28–67.89)	78.08 (74.65–82.96)	<0.001
R2	330	93	45	66	83.33 (75.85–83.95)	67.39 (47.85–65.00)	88.0 (80.09–87.72)	58.49 (41.59–57.77)	79.21 (69.88–78.99)	<0.001
C1	293	99	39	103	73.98 (69.43–78.30)	71.73 (62.83–78.58)	88.25 (84.13–91.39)	49 (41.64–55.89)	73.41 (69.62–77.35)	<0.001
C2	297	83	55	99	75.0 (70.49–79.25)	60.14 (50.78–67.78)	84.37 (80.00–87.88)	45.60 (37.86–52.84)	71.16 (66.34–75.98)	<0.001

R: radiologist; C: clinician; COVID-19: coronavirus disease 2019; CI: confidence interval; PPV: positive predictive value; NPV: negative predictive value; TP: true positive; TN: true negative; FP: false positive; FN: false negative.

Table 3.

Inter-observer agreement between radiologists and clinicians.

Patients groups	Agreement (%)	κ	SE	p value	Interpretation
Total cohort
All observers		0.784		<0.001	Good
R1 versus R2	96.3	0.909	0.02	<0.001	Excellent
R1 versus C1	88.6	0.745	0.03	<0.001	Good
R1 versus C2	89.7	0.762	0.03	<0.001	Good
R2 versus C1	88.6	0.747	0.03	<0.001	Good
R2 versus C2	89.3	0.755	0.03	<0.001	Good
C1 versus C2	88.3	0.756	0.29	<0.001	Good
RT-PCR positive subgroup
All observers		0.745		<0.001	Good
R1 versus R2	97	0.891	0.031	<0.001	Excellent
R1 versus C1	89.1	0.681	0.044	<0.001	Good
R1 versus C2	90.7	0.720	0.042	<0.001	Good
R2 versus C1	89.6	0.696	0.043	<0.001	Good
R2 versus C2	90.2	0.705	0.043	<0.001	Good
C1 versus C2	89.9	0.734	0.039	<0.001	Good
RT-PCR negative subgroup
All observers		0.736		<0.001	Good
R1 versus R2	94.2	0.872	0.044	<0.001	Excellent
R1 versus C1	87	0.706	0.063	<0.001	Good
R1 versus C2	87	0.725	0.060	<0.001	Good
R2 versus C1	85.5	0.659	0.070	<0.001	Good
R2 versus C2	87	0.719	0.061	<0.001	Good
C1 versus C2	85.5	0.682	0.063	<0.001	Good

R: radiologist; C: clinician; RT-PCR: reverse transcriptase polymerase chain reaction; κ: kappa value; SE: standard error.

Discussion

As many countries have started to alleviate and even lift the restrictions about the pandemic, correspondingly, the number of patients has started to increase. Therefore, radiologists and even other clinicians, especially emergency medicine specialists, should be alert to the CT findings of COVID-19 pneumonia.

Effective triage and early detection are very important for the control and treatment of COVID-19 disease. The chest CT continues to be increasingly used in patient management, due to the RT-PCR test kits being limited especially in underdeveloped or developing countries, the relatively low sensitivity of the test, and the late test results.

In our study, when we performed the classification according to the RSNA statement on reporting chest CT findings related to COVID-19,¹⁵ in the total cohort half of the patients displayed typical appearances for COVID-19 (48.9% for R1 and 49.6% for R2) (Figures 1 and 2). In the study conducted by Byrne et al.¹⁹ with three independent radiologists, similar results were obtained (46.2% for R1, 52.8% for R2, and 45.9% for R3). In the RT-PCR positive patient group of our study, most of the patients displayed typical appearances of COVID-19 (58.8% for R1 and 61.6% for R2). In the study conducted by Miranda Magalhães Santos et al.,¹⁰ the patient rate who displayed typical appearance in the total cohort was found to be 41.3%, and this rate was found as 83.3% for the RT-PCR positive patient subgroup. This difference between the values in the classification made according to the same criteria may be caused by the difference in prevalence. In our study, when we accept the typical and intermediate appearance as positive for COVID-19, the sensitivity, specificity, PPV, NPV, and accuracy values in the chest CT for R1 and R2 were found as 83.3%–83.3%, 63%–67.4%, 86.6%–88%, 56.9%–58.5%, and 78.1%–79.2% for each radiologist, respectively. In the study conducted by Miranda Magalhães Santos et al.,¹⁰ the same values were found as 83%, 97%, 97%, 86%, and 91%, respectively. In our study, we think that our low specificity and NPV values might be due to the difference in the patient population, disease prevalence, and the fact that Miranda Magalhães Santos et al.¹⁰ accepted the intermediate appearance as being negative for COVID-19 as well. In our study, the intermediate appearance was found in the RT-PCR positive subgroup at high rates (24.5% for R1 and 21.7% for R2), and we think that this appearance should also be accepted as COVID-19 positive in high-prevalence areas. Thus, the specificity will decrease, while the sensitivity increases, and more patients will be isolated until their RT-PCR results come out. Also, classification of patients according to the degree of suspicion according to their CT findings and hospitalization of patients at different floors until the result of RT-PCR may provide additional benefits in terms of isolation. In our study, atypical appearance was found in small rates in the RT-PCR positive group (2.5% for R1 and 2.8% for R2) and is consistent with the values found in the literature.^16,20–22 In the RT-PCR positive patient subgroup of our study, pneumonia was not observed in the chest CT at a considerably important rate (14.1% for R1 and 13.9% for R2). During the first few days after the disease symptoms begin, CT can also be negative just like RT-PCR. This decreases the sensitivity of the CT. Observing pneumonia findings in the chest CT peaks 6–13 days after the symptoms begin.^23–25 Therefore, the chest CT, especially during the first days when the symptoms begin, should not be used for COVID-19 diagnostic exclusion.

Figure 1.

A 47-year-old man with pneumonia caused by coronavirus disease 2019 (COVID-19) confirmed by RT-PCR. (a and b) Axial unenhanced chest CT images at different levels show mainly peripherally located multiple round ground-glass opacities.

Figure 2.

A 72-year-old man with pneumonia caused by coronavirus disease 2019 (COVID-19) confirmed by RT-PCR. (a and b) Axial unenhanced chest CT images at different levels show mainly peripherally located multiple patchy ground-glass opacities.

To our knowledge, no study comparing the diagnostic performance of radiologists and clinicians has been found. In our study, the accuracy rates of radiologists and clinicians (R1, R2, C1, and C2) in the diagnosis of COVID-19 were 78%, 79%, 73%, and 71%, their sensitivity rates were 83%, 83%, 74%, and 75%, and the NPV were 57%, 58%, 49%, and 46%, respectively. The overall accuracy, sensitivity, and NPV of the radiologists were slightly higher than the clinicians. However, if clinicians had evaluated chest CT scan together with clinical history, examination, and laboratory findings, their diagnostic performance would have increased. The specificity and PPV values of the evaluators were similar (Table 2).

Our results show that there is an overall excellent inter-observer agreement for total cohort and RT-PCR positive and negative subgroups according to the RSNA expert consensus statement reporting guidelines among radiologists in suspected COVID-19 cases. In the study conducted by Byrne et al.¹⁹ with three independent radiologists, similar results were obtained. In their study, categorizing their patients’ CT scans according to the RSNA expert consensus statement reporting guidelines as typical, intermediate, atypical, and negative, they looked at the inter-observer agreement between radiologists in these subgroups. According to the results of their study, there was excellent agreement among radiologists in terms of typical, atypical, and negative CT appearances regarding COVID-19 and good agreement in intermediate appearance.

This multidisciplinary, inter-observer study showed good to excellent inter-observer agreement between all reviewers (between radiologists, radiologist and clinicians, and between clinicians) (Table 3). Inter-observer agreement among the radiologists was excellent. Inter-observer agreement on the diagnosing of COVID-19 pneumonia is slightly greater among radiologists than clinicians. A good inter-observer agreement means better patient selection for isolation and treatment, resulting in better outcomes.

Limitations

There were some limitations to our study. First, it was a retrospective and designed as single-centered. Second, the study was conducted in an epidemic area during the peak period of the disease, with a specific patient group that consisted of 74% COVID-19 positive patients. Consequently, the diagnostic performance of the CT examination was affected by the prevalence of the disease. Apart from the pandemic period, a decrease might be seen in these diagnostic values of chest CT. Third, the RT-PCR examination that was reported to have false negative rates was considered as the gold standard.^1–6 In order to compensate for this potential false negative of the initial RT-PCR, multiple tests were applied in cases where clinical and radiological suspicions persist, especially in RT-PCR negative cases. In addition, it is mentioned in the literature that there may be false positive RT-PCR results due to technical artifacts and technical cross-contamination at any point along the sample chain of testing.²⁶

Conclusion

We have shown that the RSNA expert consensus statement reporting guidelines related to COVID-19 has excellent inter-observer agreement among chest radiologists. COVID-19 pneumonia in the chest CT shows typical and intermediate findings in 83.3% of patients, and this high rate provides easily differentiable diagnosis for radiologists and clinicians. In a high-epidemic area, while the sensitivity and accuracy rates of radiologists were slightly high, radiologists and clinicians have similar diagnostic performance in the evaluation of COVID-19 pneumonia with chest CT. In conclusion, radiologists and clinicians have presented similar and good diagnostic performances in the evaluation of COVID-19 suspected patients with chest CT in high-epidemic area.

Footnotes

Author contributions

F.C. and O.G. contributed to the literature search; F.C., O.G., E.S.A., M.F.K., and S.N.D. contributed to the study design and drafting the article; F.C., O.G., and E.S.A. contributed to the data collection; F.C. and M.F.K. were involved in gaining ethical approval, statistical advice, and statistical data analysis; F.C., O.G., E.S.A., and M.F.K. contributed to the data interpretation; F.C. and S.N.D. contributed to the supervision and quality control; F.C. wrote the first draft of the article. All authors reviewed and edited the article and approved the final version of the article.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Informed consent

The data were collected retrospectively from Hospital Automation System and Archives, after obtaining ethical approval. The need for signed informed consent was waived due to retrospective design of the study.

Ethical approval

The study was approved by the Institutional Ethics Committee of Gaziosmanpasa Training and Research Hospital, and the ethical approvals were obtained prior to initiation of the research work.

Human rights

No human rights were violated during this study and the study was in accordance with Declaration of Helsinki.

ORCID iDs

Ferhat Cengel

Mustafa Calik

Mustafa Asim Demirkol

References

Guan

, et al. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 2020; 382(18): 1708–1720.

Corman

Landt

Kaiser

, et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Euro Surveill 2020; 25(3): 2000045.

Zhang

Wang

Deng

, et al. Recent advances in the detection of respiratory virus infection in humans. J Med Virol 2020; 92(4): 408–417.

Gezer

Ergan

Barış

, et al. COVID-19 S: a new proposal for diagnosis and structured reporting of COVID-19 on computed tomography imaging. Diagn Interv Radiol 2020; 26(4): 315–322.

Yao

, et al. Stability issues of RT-PCR testing of SARS-CoV-2 for hospitalized patients clinically diagnosed with COVID-19. J Med Virol 2020; 92(7): 903–908.

Yang

Hou

, et al. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology 2020; 296(2): E32–E40.

Wang

Gao

, et al. Detection of SARS-CoV-2 in different types of clinical specimens. J Am Med Assoc 2020; 323(18): 1843–1844.

Zou

Ruan

Huang

, et al. SARS-CoV-2 viral load in upper respiratory specimens of infected patients. N Engl J Med 2020; 382(12): 1177–1179.

Yang

Hou

, et al. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology 2020; 2020: 200642.

10.

Miranda Magalhães Santos

Paula Alves Fonseca

Pinheiro Zarattini Anastacio

, et al. Initial results of the use of a standardized diagnostic criteria for chest computed tomography findings in coronavirus disease 2019. J Comput Assist Tomogr 2020; 44(5): 647–651.

11.

Luo

, et al. Diagnostic performance between CT and initial real-time RT-PCR for clinically suspected 2019 coronavirus disease (COVID-19) patients outside Wuhan, China. Respir Med 2020; 168: 105980.

12.

Xie

Zhong

Zhao

, et al. Chest CT for typical coronavirus disease 2019 (COVID-19) pneumonia: relationship to negative RT-PCR testing. Radiology 2020; 296(2): E41–E45.

13.

Long

Shen

, et al. Diagnosis of the coronavirus disease (COVID-19): rRT-PCR or CT? Eur J Radiol 2020; 126: 108961.

14.

Himoto

Sakata

Kirita

, et al. Diagnostic performance of chest CT to differentiate COVID-19 pneumonia in non-high-epidemic area in Japan. Jpn J Radiol 2020; 38(5): 400–406.

15.

Simpson

Kay

Abbara

, et al. Radiological society of North America expert consensus statement on reporting chest CT findings related to COVID-19. Endorsed by the society of thoracic radiology, the American College of Radiology, and RSNA. J Thorac Imaging 2020; 35(4): 219–227.

16.

Chung

Bernheim

Mei

, et al. CT imaging features of 2019 novel coronavirus (2019-NCoV). Radiology 2020; 295(1): 202–207.

17.

Cheng

Cao

, et al. Clinical features and chest CT manifestations of coronavirus disease 2019 (COVID-19) in a single-center study in Shanghai, China. AJR Am J Roentgenol 2020; 215(1): 121–126.

18.

Shi

Han

Jiang

, et al. Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study. Lancet Infect Dis 2020; 20(4): 425–434.

19.

Byrne

O’Neill

Müller

, et al. RSNA expert consensus statement on reporting chest CT findings related to COVID-19: interobserver agreement between chest radiologists. Can Assoc Radiol J. Epub ahead of print 2 July 2020. DOI: 10.1177/0846537120938328.

20.

Salehi

Abedi

Balakrishnan

, et al. Coronavirus disease 2019 (COVID-19): a systematic review of imaging findings in 919 patients. AJR Am J Roentgenol 2020; 215(1): 87–93.

21.

Xia

Coronavirus disease 2019 (COVID-19): role of chest CT in diagnosis and management. AJR Am J Roentgenol 2020; 214(6): 1280–1286.

22.

Miao

Jin

Miao

, et al. Early chest computed tomography to diagnose COVID-19 from suspected patients: a multicenter retrospective study. Am J Emerg Med. Epub ahead of print 19 April 2020. DOI: 10.1016/j.ajem.2020.04.051.

23.

Bernheim

Mei

Huang

, et al. Chest CT findings in coronavirus disease 2019 (COVID-19): relationship to duration of infection. Radiology 2020; 295(3): 685–691.

24.

Pan

Sun

, et al. Time course of lung changes at chest CT during recovery from coronavirus disease 2019 (COVID-19). Radiology 2020; 295(3): 715–721.

25.

Song

Shi

Shan

, et al. Emerging 2019 novel coronavirus (2019-NCoV) pneumonia. Radiology 2020; 295(1): 210–217.

26.

Katz

Civantos

Sargi

, et al. False-positive reverse transcriptase polymerase chain reaction screening for SARS-CoV-2 in the setting of urgent head and neck surgery and otolaryngologic emergencies during the pandemic: clinical implications. Head Neck 2020; 42(7): 1621–1628.