Identification of COVID-19 samples from chest X-Ray images using deep learning: A comparison of transfer learning approaches

2.1 Relevant investigations about COVID-19 in the medical domain

Fang et al. proposes a study to examine the sensitivity of chest CT image and viral nucleic acid detection technique using real-time polymerase chain reaction (RT-PCR) for the detection of COVID-19 cases [28]. Fifty-one subjects who had a travel or residency history in Wuhan within 14 days and had a fever or acute respiratory symptoms, with an average age of 45 years, are taken part of this study. Thirty-six patients diagnosed positive for COVID-19 in the 1st RT-PCR test. Twelve of them get positive results on the 2nd RT-PCR tests (1 to 2 days). Two patients get positive by 3rd RT-PCR tests (2-5 days) and one patient by four tests (7 days). However, 98% of the 51 patients shown evidence of viral pneumonia on the 1st day. Therefore, the finding of this study is that the sensitivity of chest CT is higher than RT-PCR (98% vs. 71%, respectively).

Bernheim et al. analyzed chest CTs of 121 symptomatic positive COVID-19 patients and finds the relationship of the outcome of chest CTs in time between symptom onset of the initial CT scan (i.e., early, 0 to 2 days (36 patients), intermediate 3 to 5 days (36 patients) and late 6 to 12 days (25 patients)) [29]. The observation of this investigation is that 28% of the early patients (10/36), 76% of intermediate patients (25/33), and 88% of late patients (22/25) show bilateral lung disease.

Tao et al. examines 1014 patients in Wuhan from 6th January to 6th February and observes that chest CT is more sensitive for the diagnosis of COVID-19 over RT-PCR [11]. Out of 1014 patients, 601 patients test positive in RT-PCR, while 888 of them tests positive in chest CT. They also analyze that the mean interval time is 5.1 ± 1.5 days from initial negative to positive and 6.9 ± 2.3 days from positive to negative in RT-PCR.

In order to find the role of chest CT and CXR in the management of COVID-19, Rubin et al. in [30] comprise a multidisciplinary panel of radiologists and pulmonologists from 10 different countries with experience of managing COVID-19 patients. Fourteen key questions and corresponds to 11 decision points are presented on three aspects, such as varying risk factors, community condition, and resource constraints. The results are analyzed and found that there are five primary and three additional recommendations for the medical practitioners to use of CXR and CT in the management of COVID-19.

2.2 Related DL works in CAD of COVID-19

Farooq et al. recommend a deep learning-based approach, namely COVID-ResNet, to differentiate COVID-19 cases form other types of pneumonia cases in [31]. The author also builds an open-access dataset for CXR image research. A pre-trained ResNet is used, which is followed by three steps of fine-tunes work. In the three fine-tuned steps, input images are resized continuously form 128 × 128 × 3, 224 × 224 × 3 to 229 × 229 × 3. Among 6009 images they utilize in the proposed algorithm, 68 of them are COVID-19 cases. They obtained an overall accuracy of 96.23% for four-class classification.

In [32], Afshar et al. suggests a binary classification of the CXR images based on the Capsule Network framework known as COVID-CAPS to identify COVID-19 cases. 90% of the total dataset is employed for training, and 10% is for validation. The advanced framework reaches an accuracy of 95.7%, and the pre-trained framework achieves an accuracy of 98.3%.

Wang et al. introduce COVID-Net [33], which is a deep CNN based approach to detect COVID-19 cases form CXR images. The network is trained on publicly available 13,800 CXR images. Among them, 183 are COVID cases. The model achieves an overall accuracy of 92.6% in a three-class classification.

A new deep learning-based framework, namely COVIDX-Net, is suggested by Hemdan et al. [34] to diagnose COVID-19 from CXR images automatically. The network is composed of VGG19, DenseNet121, ResNetV2, InceptionV3, InceptionResNetV2, Xception and MobileNetV2. A total of 50 CXR images is considered for testing the representation of the proposed algorithm; among them, 25 are positive COVID-19 images. 80% of the images are used for training, and rests are used for testing. All images are resized to 224 × 224 pixels. VGG19 and DenseNet201 achieve an accuracy of 90%.

Shety et al. [35] suggests a deep learning-based framework for the detection of COVID-19 from CXR images. The proposed methodology utilizes different CNN models to extract features and fed into SVM for classification. Twenty-five normal and 25 COVID-19 images are used to evaluate the performance. 60%, 20%, and 20% of images are used for training, validation, and testing the model. ResNet50 achieves the highest classification accuracy of 95.38%.

Maghdid et al. proposes an AI tools that can help radiologists to diagnose COVID-19 cases from X-rays and CT scan images quickly [36]. Modified pre-trained AlexNet is used as the backbone of the network. 170 CXR and 361 CT images are utilized in this model. 50% of the overall dataset is granted for training, and 50% is for validation. Modified CNN achieves an accuracy of 94.1%, and a pre-trained network obtains 98% accuracy in binary classification.

Apostolopoulos et al. [37], employ state of the art CNN model called MobileNet from 3905 X-Ray images, correspond to six different classes. Moreover, 455 COVID-19 CXR images are also incorporated. The images are resized to 200 × 200 pixels, and the augmentation task is performed in the preprocessing step. They achieved an accuracy of 99.18% in binary classification tasks and 87.66% for the seven class classification tasks.

Loey et al. suggests Generative Adversarial Network (GAN) with deep transfer learning for the detection of COVID-19 from CXR images in [38]. Since there is a shortage of COVID-19 CXR images; therefore, to generate more images, they employ GAN. A total number of 307 images are collected in four different classes, such as COVID-19, normal, bacterial pneumonia, and viral pneumonia. In the training and validation step with GAN, 90% of the dataset is utilized, while 10% is kept for testing. Pretrained AlexNet, GoogleNet, and Resnet18 are used as transfer learning. In the proposed model they achieved an accuracy of 80.6%, 85.3% and 100% on four class (GoogleNet), three class (AlexNet) and two class (GoogleNet) classification problem respectively.

As discussed above, promising COVID-19 identification results have been obtained using CXR with transfer learning and various deep learning models, mostly CNN, have been utilized. However, the relative performance results of various CNN transfer learning are unclear. This paper focuses on the performance comparison of CNN models (15 total) for CXR COVID-19 identification using transfer learning.

2.3 Related deep transfer learning works in CAD fields

Zhang et al. proposes a deep transfer learning-based classification approach for the classification of the cervical cell into normal and abnormal [39]. One private dataset called HEMLBC and publicly available Herlev dataset are considered to evaluate the performance of the proposed method. In the preprocessing work, they perform the patch extraction and image augmentation operations. They achieve an accuracy of 98.3% and 98.6% on the Herlev and HEMLBC dataset individually.

Chen et al. introduces a deep transfer learning-based framework based on the Inception-V3 network for the classification of cervical immunohistochemistry images in [40]. The data augmentation task is performed in the preprocessing step. They obtain an average accuracy of 77.3%.

Song et al. carries out a transfer learning-based CNN that is pre-trained on the ImageNet dataset for the classification of breast histopathology images [41]. First, the images are represented in the Fisher Vector (FV) encoding of local features. Then, an adaptation layer is designed for fine-tuning. Finally, an overall accuracy of 87% obtained with 30% testing data.

Hall et al. [42] explore that chest X-ray images are beneficial for the diagnosis of COVID-19 viral infection. They utilize pre-trained ResNet50 and tuned on 102 COVID-19 cases and 102 other pneumonia cases and achieves an overall accuracy of 90.7%. Moreover, in another experiment, they combine pre-trained VGG16, ResNet50, and custom made CNN train and test on 33 unseen COVID-19 CXR and 208 pneumonia cases and obtained an overall accuracy of 94.4%.

In [43], Zhang et al. proposes a deep learning model to identify COVID-19 CXR images from standard CXR images. Pretrained ResNet is used as a backbone of the network. 50% of 100 COVID-19 and 1430 Normal CXR images are considered for training, and rest are regarded for testing. This algorithm can detect 96% of COVID-19 cases and 70.65% non-COVID-19 cases.

Abbas et al. suggests that their previously developed CNN, namely Decompose, Transfer and Compose network (DeTraC-Net), utilize pre-trained ResNet-50 as transfer learning to classify COVID-19 chest X-ray images from normal and severe acute respiratory syndrome cases [44]. 50 standard, 105 COVID-19, and 11 SARS CXR images are applied to evaluate the performance of their proposed method and achieve an accuracy of 95.12%.

Narin et al. [45] proposes a CNN based approach to detect coronavirus pneumonia among patients using CXR. Here, three pre-trained network models are utilized, namely, ResNet50, InceptionV3, and Inception-ResNetV2. 50 COVID-19 CXR and 50 normal CXR are used to check the performance of the proposed model. 80% of the dataset employed for training and 20% is for testing. They observed that ResNet50 gives the highest classification accuracy of 98% in binary classification, among other models.

3.1 Dataset

3.1.1 Dataset organization

To test the effectiveness of our proposed method, we have utilized CXR images from COVID-19 patients, regular pneumonia patients, and healthy patients. We have collected the images from two publicly available databases. Dr. Joseph Cohan created a publicly accessible CXR and CT image database in the GitHub repository for positive COVID-19 patients, which also includes severe acute respiratory syndrome (SARS), Middle East respiratory syndrome (MARS) and acute respiratory distress syndrome (ARDS) [46]. This repository also incorporated publicly available medical image repositories, such as the Radiological Society of North America (RSNA), and the Italian Society of Medical and Interventional Radiology (SIRM). The database is regularly updating with new cases. For the benefit of our experiment, we have only considered the CXR images. Moreover, the healthy and pneumonia CXR images are collected from the Kaggle repository called “Chest X-Ray Images (Pneumonia)” [47]. An example of the collected dataset is presented in Fig. 2

OPEN IN VIEWER

Fig.2

An example of Normal, Pneumonia, and COVID-19 CXR images.

3.2 State of the art deep learning models

In this subsection, we have provided a brief overview of the state of the art deep learning models that are used in this experiment and their significant points in a tabular form. Table 2 exhibits all-important attributes of a deep neural network, such as the number of parameters, total depth of the network (including activation layer, batch normalization and so on), top 5 accuracies on ImageNet dataset, and significant remarks.

Nowadays, a prevalent deep learning method in a computer vision system is transfer learning (TL). TL helps us to build an accurate model without starting the learning process from scratch instead start from patterns that have been learned on solving different problem [56, 57]. Therefore, it saves time and shows excellent results on a small dataset. The transfer learning process can be comprised of two steps. Selecting a pre-trained DL model is the first step of transfer learning. There is a wide range of pre-trained models available in the literature that is trained on a large scale of benchmark dataset to solve a similar problem that we intend to solve. For example, in Keras lot of pre-trained models are available such as VGG, Inception, ResNet. So we only need to choose the suitable one for our task. After that, we need to fine-tune the model based on the size of our dataset and the similarity of our dataset with the pre-trained model. For instance, it is necessary to train the entire model, if we have a large dataset but different from the pre-trained model dataset. Nevertheless, we need to train a few layers and freeze the other layers if we have a small dataset that is different from the pre-trained model dataset.

Overfitting is a prevalent issue in machine learning models. It appears when a model memorizes the training dataset without learning important features, trends, or boundaries. Then it can not perform well on unseen data. The clear sign of a machine learning overfitting is if the error on the validation set is much higher than the error on the training set. Alternatively, if the model’s accuracy is very high concerning the training data and drops significantly with new data, it is suspected to have an overfitting problem. Figure 3 illustrates an overfitting problem during training when no data augmentation is performed. There are several techniques to reduce the overfitting problem. First of all, increasing the number of training datasets can help to overcome this problem. Secondly, data augmentation, like rotating the image, zooming in or out, horizontal or vertical flip operations, is beneficial in this regard. Thirdly, dropout, L1, and L2 regularization are also some popular options. Finally, a simple model structure consisting of fewer layers and neurons can reduce the overfitting problem.

OPEN IN VIEWER

Fig.3

An example of an overfitted accuracy/loss curve while training DenseNet201 without data augmentation.

In this study, we have applied VGG series, Xception, ResNetV1 and ResNetV2 series, Inception series, DenseNet series, and MobileNet networks in the transfer learning process, where the weights are pre-trained on ImageNet dataset. In the transfer learning section in Fig. 1-(c), we first import the pre-trained model from Keras, where we freeze the convolutional base and fine-tune the top layers. The convolutional base usually extracts the features. After the convolutional base, we place the flatten layer, which transforms the two-dimensional feature matrix into a vector. After the flatten operation, the fully connected layer (FCL) with the ReLU activation function is used, and the number of neurons is fixed to 1024, 512, respectively. Finally, we fed the output to the softmax activation layer for the final classification. For all these networks, the input size is (224 × 224 × 3), initial learning rate 0.001, the batch size is 32, the number of epochs is 50, and the ’rmsprop’ optimizer is employed. We have adopted "ReduceLROnPlateau" as a Keras callback function to reduce the learning rate when the result stops improving. The learning rate will gradually reduce by a factor of 0.3. This function also helps the network to reduce the overfitting problem. Fig. 1-(c) shows some representative feature maps extracted from the 1st convolution layer of the VGG network. It demonstrates that the TL technique is capable of extracting some necessary information from the images.

4.1 Experimental Setup

Google Colaboratory (also known as Colab) is a free cloud service based on Jupyter Notebooks for machine learning education and research [58, 59]. In Jupyter Notebook, python 2 and 3 are preconfigured with essential machine learning libraries, such as Tensorflow, Matplotlib, and Keras, that require no installation on a local machine to start. The runtime is fully functional for deep learning applications with GPU support using NVIDIA Tesla K80. Moreover, the code is stored in Google drive. In this experiment, we have used Google Colab Notebooks for training and testing our model.

4.2 Evaluation Method

To select a suitable evaluation metric is essential to overcome the bias among the differentiation of algorithms. For the classification standard, precision, recall, F1 score, and accuracy are the most prevalent measures [60, 61]. Precision is the number of correctly identified samples among all the identified samples. The recall is the number of correctly identified samples from all the positive representations. F1 score is the harmonic average of precision and recall. Accuracy is the proportion of correctly classified samples from the total number of samples [62]. Table 3 exhibits the mathematical expression for precision, recall, F1 score, and accuracy. In Table 3, the number of accurately labeled positive samples is TP (True Positive), the correctly detected negative sample is TN (True Negative), the number of negative examples classified as positive is FP (False positive), and the number of positive specimens predicted as unfavorable is FN (False Negative). For multiclass classification finding the values of TP, TN, FP, FN is a bit different than the binary classification problem since there are no positive or negative classes. In a confusion matrix(assuming x axis is the predicted class label and y axis is the true label), the total number of TP for a class is when the predicted class match with the target class. FN for a class is the sum of the values in a corresponding row, excluding the TP value. For a class, FP is the sum of the values in a corresponding column, excluding the TP value. TN for a particular class will be the sum of all columns and rows, excluding that class’s column and row.

To evaluate the performance of the recommended method, we have calculated the precision, recall, F1 score, and accuracy of each model.

Table 4 illustrates the performance metrics of different deep TL models for the identification of COVID-19 cases from healthy and pneumonia cases on the testing dataset. The accuracy of testing data demonstrates the accuracy of any deep learning models.

It can be observed from Table 4 that the highest precision and recall value for distinguishing COVID-19 cases is 0.950, reached by VGG19, followed by VGG16 with a value of 0.923 and 0.900, respectively. The InceptionResNetV2 attains the lowest precision and recall value of 0.652 and 0.375, individually. To detect the normal cases, VGG16 and VGG19 obtain the most considerable precision and recall value of 0.855 and 0.960, sequentially. On the other hand, MobileNetV1 delivers the lowest precision rate of 0.541. Additionally, ResNet50 and ResNet101 produce the lowest recall value of 0.660. For identifying the pneumonia cases, VGG19 and VGG16 achieve the most considerable precision and recall value of 0.975 and 0.820, sequentially, whereas InceptionResNetV2 and MobileNetV1 show the least performance.

Among all the tested models, MobileNetV1, InceptionResNetV2, ResNet50, ResNet101 and ResNet152 provide the lowest classification accuracy of around 60%, while the VGG19 and VGG16 produce the highest accuracy of 89.3% and 88.57%, individually. ResNet101V2 and InceptionV3 also show considerable performance and contribute an accuracy of 84.28% and 82.14%, respectively. ResNetV2, ResNet152V2, DenseNet121, DenseNet169, DenseNet201 and XceptionNet also recorded reasonable accuracy of around 75%.

Figure 4 depicts the resulted confusion matrices on unseen test dataset to show the classification performance of each TL model more intuitively. As we observe from Table 4 that VGG19 and VGG16 attain the highest classification accuracy. Therefore, if we look at their confusion matrix, it is perceived that VGG19 can accurately recognize 38 images in COVID class, though two images are labeled as Normal class. Besides, In the normal image class, 48 images are identified precisely; but, one image is listed in each COVID and Pneumonia class. Lastly, in the Pneumonia class, 39 copies are distinguished correctly, yet 10 images are predicted as normal and one as COVID. In the VGG16 model, it is observed that 36 images are correctly categorized as normal but four images as pneumonia. The model also classifies 47 samples accurately, although two as COVID and one as pneumonia. Finally, 41 images are listed correctly as pneumonia; however, eight as normal and one as COVID. Furthermore, if we observe at the confusion matrix of the least performing algorithms such as MobileNetV1 and InceptionResNetV2, it is apparent that both of the networks are able to detect 33 and 15 copies in COVID class correctly, however, six and four copies as Normal, 1 and 21 samples as Pneumonia, respectively. Additionally, both models can classify 40 and 38 standard samples correctly, though 3 and 5 samples are detected as COVID, and seven samples of both models are identified as Pneumonia. Lastly, MobileNetV1 can identify only 13 images correctly as Pneumonia, yet 9 samples as COVID and 28 as Normal. InceptionResNetV2 can predict 34 samples correctly as Pneumonia, including 3 samples as COVID and 13 samples as Normal. Therefore, it is obvious to implement the VGG19 and VGG16 models for the identification of the patient’s health status using chest x-ray images.

OPEN IN VIEWER

Fig.4

The resulted confusion matrix of different TL models. (a) is the confusion matrix of VGG16 generated on test dataset. Similarly, (b) is generated on VGG19, (c) is on ResNet50, (d) is on ResNet50V2, (e) is on ResNet101, (f) is on ResNet101V2, (g) is on ResNet152, (h) is on ResNet152V2 (i) is on DenseNet121, (j) is on DenseNet169, (k) is on DenseNet201, (l) is on MobileNetV1, (m) is on XceptionNet, (n) is on InceptionV3, (o) is on InceptionResNetV2.

OPEN IN VIEWER

Fig.5

Accuracy and loss curves of selected TL models. Here, (a) exhibits the training and validation accuracy and loss curves of VGG16 model. Similarly, (b) presents the accuracy and loss curves of VGG19 model, (c) is generated on ResNet101V2, (d) is on DenseNet121, (e) is on MobileNetV1, (f) is on XceptionNet, (g) is on InceptionResNetV2, (h) is on InceptionV3.

We have selected a few training and loss curves per CNN model series to provide a glimpse of how the networks perform concerning to epoch in Figure 5. It is perceived from the figure that the VGG series has the most stable curves and delivers the highest training and validation accuracy. If we study the accuracy and loss curves of VGG16 and VGG19, it is discerned that both of the operations develop a similar trend. The accuracy curves of train data for both the network is increasing quickly to around 82% form epoch 0 to 10 and converges to a value of over 88% after epoch 50. Similarly, the accuracy curve of validation data reaches to around 88%. For the loss curve of training and validation data, the loss swiftly decreases to approximately 1% within the first five epochs and stays stable until 50 periods. However, if we observe the training and loss curves of the worst accomplishing networks such as MobileNetv1 and InceptionResnetV2, it is regarded that the training and validation accuracy is fluctuating considerably and produce training accuracy of about 61% and 68%, respectively, after 50 epochs. Moreover, it is discerned that there is no significant gap between the training and validation curve. Hence we can not say the model is overfitting or underfitting indeed at some epoch the model converges, where both training and validation accuracies are almost equal. In this experiment, we employ the data augmentation technique on the training set, making the training set challenging to predict and make validation set easy to predict. Therefore, at some point, the validation accuracy is slightly higher. It is also recognized that for XceptionNet, the validation accuracy is slightly higher than the training accuracy is because the Xception model holds dropout configuration, where the training accuracy states with the dropout, but validation accuracy is without dropout.

As we have already mentioned, in this experiment, Google Colaboratory is utilized, which provides the Tesla K80 GPU platform. It is observed from Table 5 that the VGG series has average training time of around 24 seconds per epoch. ResNet50 series demands approximately 27 seconds per period for training, while ResNet101, ResNet101V2, ResNet152V2 needs about 33 seconds per epoch. MobileNet, DenseNet series, and XceptionNet have an average training time roughly 29 seconds per epoch. InceptionV3 and InceptionResNetV2 have an average training time of almost 29 seconds per epoch. ResNet152 takes the longest time to train per epoch, which is around 39s. Though the networks demand quiet time for training, the testing time is less, which is around 5 seconds.

Recently, Deep learning technique in the field of machine learning has been fascinating researchers significantly. The improvement of its utilization presents an opportunity to analyze medical images and solve critical tasks. Identification and classification of COVID-19 cases from Normal and Pneumonia cases using CXR images can help to isolate the infectious subjects, which is a significant step to fight against the virus. In this study we have trained and tested 15 different deep CNN models in a transfer learning process and their comparative evaluation is presented.

There are some essential factors for an individual model that influence their performance, such as imaging modality, image content, image quantity, distribution of the dataset, the structure of the model, model complexity, loss function, optimizer, number of epochs and so on. The accuracy records in Table 4 illustrate that VGG nets perform relatively well compared to ResNets, InceptionNets, DenseNets, MobileNet, and XceptionNet. Therefore, it is evident that a shallow network performs well than very deep networks in this type of image dataset. It also observed that the number of network parameters for VGGNets is considerably higher than other networks (Table 2). There are few observations on that. If we analyze the architecture of VGG, it is observed that the network architecture is very simple and straightforward. VGG uses a very small receptive field (3 filters), followed by the ReLU activation function. Small filters help to have more weight layers, which helps to improve performance.

If we perceive the ResNets behavior from Table 4, the performance of the ResNetV1 decreases from 64.3% to 62.85% as the network depth increases. As we know, with the progress of network depth, the vanishing gradients and degradation problem become prominent. On the other hand, ResNetV2 provides a better result compare to ResNetV1. The optimal accuracy of 84.28% is achieved by ResNet101V2, whereas ResNet152V2 achieves 77.14% accuracy. So it is remarked that network depth can improve the performance but at a certain label. The improved version of ResNet implies Batch normalization and ReLU activation before the convolution operation, which can be the reason behind its improved performance.

In contrast, the DensNet series provide good performance with the increase of the network layer. It can be because DenseNet concatenates the output from the previous layers, whereas ResNet does the additive operation among the previous layer with the future layer. It is also recognized that the lower version of the Inception network (InceptionV3) performs better on our dataset, compared to InceptionResNetV2. ResNet and InceptionNet demonstrate that extremely deep networks are not suitable for our dataset. Table 5 displays that VGGNets are computationally less expensive than other types of networks. Therefore it is easier to train.