An adaptive weighted ensemble learning network for diabetic retinopathy classification

Abstract

Diabetic retinopathy (DR) is one of the leading causes of blindness. However, because the data distribution of classes is not always balanced, it is challenging for automated early DR detection using deep learning techniques. In this paper, we propose an adaptive weighted ensemble learning method for DR detection based on optical coherence tomography (OCT) images. Specifically, we develop an ensemble learning model based on three advanced deep learning models for higher performance. To better utilize the cues implied in these base models, a novel decision fusion scheme is proposed based on the Bayesian theory in terms of the key evaluation indicators, to dynamically adjust the weighting distribution of base models to alleviate the negative effects potentially caused by the problem of unbalanced data size. Extensive experiments are performed on two public datasets to verify the effectiveness of the proposed method. A quadratic weighted kappa of 0.8487 and an accuracy of 0.9343 on the DRAC2022 dataset, and a quadratic weighted kappa of 0.9007 and an accuracy of 0.8956 on the APTOS2019 dataset are obtained, respectively. The results demonstrate that our method has the ability to enhance the ovearall performance of DR detection on OCT images.

Keywords

Diabetic retinopathy ensemble learning decision fusion

1 Introduction

Diabetes affects 200 million people worldwide, including 20 million in the United States alone. Diabetic retinopathy (DR) disease, a specific microvascular complication of diabetes, is the leading cause of blindness in the US working-age population [1, 2]. The prevalence of DR increases with the duration of diabetes and almost all diabetic patients have some retinopathy after 20 years [3]. Studies have shown that the use of optical coherence tomography (OCT) is helpful in measuring the thickness of DR areas and in detecting and quantifying the extent of edema. However, OCT false positives are often subclinical diabetic macular edema (DME) cases that cannot be detected clinically. Therefore, the risk of disease progression is still increased [4 –9].

Deep learning methods have shown great success in almost all machine learning related areas, especially the convolutional neural networks have recently been widely used to diagnose DR [10 –13] by analyzing fundus images [14, 15]. For instance, a deep-learning and a texture feature extraction model are usually combined, or a deep-learning based diagnosis model are trained, to serve as the feature representation of diabetic retinal images so that the diagnoistic decisions are made [16 –18].

Although there is a range of research efforts devoting to improve the overall performance of diabetic retinal diagnostic models, these methods tend to predict classes with a large number of samples, and they can hardly work satisfactorily for that of the relatively small proportion of samples. For DR detection, the number of samples of normal eyes in the dataset far exceeds the number of samples with diabetic retinopathy. This makes it easier for the deep learning models to learn the features of normal examples during the training, while the feature representation for diabetic retinopathy is relatively insufficient. The classification model tends to predict the normal class due to the dominance of normal samples. However, the classes corresponding to the fundus images with lesions cannot be trained well due to the data imbalance problem. Thus, the imbalanced data may cause performance degradation for the classification. Besides, the commonly used classifiers and stable models are sensitive to noise and outliers, which may lead to poor classification results. Moreover, due to the weak generalization ability of the classifier, the classification result may be sensitive to the disturbance and change of the input data, thus affecting the robustness of the model. Ensemble learning can effectively deal with the aforementioned problems. By combining the prediction results of multiple base classifiers, the overall recognition and generalization ability could be enhanced, meanwhile avoiding the overfitting problem. More importantly, Quadratic Weighted Kappa (QWK) score is proved to be a very important indicator for all stages of DR disease. The major causes of poor QWK scores are imbalanced distribution of classes or poor quality of images [19].

Inspired by this, in the present work, we propose an adaptive weighted ensemble model to facilitate the DR diagnosis based on OCT images. Specifically, we build an ensemble learning model to consist three base models (i.e. Swin Transformer, FocalNet and ResNet) for better performance. To better utilize the cues provided by each base model, we propose a novel adaptive weighted fusion strategy based on the Bayesian theory. With this strategy, especially in the case of unbalanced samples, the weight of the model with high QWK will be set a relative high value corresponding to the minority category of class. By giving higher weights to the less class learners, more attention will be paid to the predictive ability of minority classes. Thus, it effectively tackles the problem of imbalanced distribution of classes.

Our major contributions can therefore be summarized as follows: i) We propose an adaptive weighted ensemble model for the task of DR detection, to reduce the risk of overfitting and promote the performance of the model on unseen data. ii) We put forward a novel decision fusion method in terms of Bayesian theory, which can effectively solve the problem of unbalanced sample size. iii) experimental results on public datasets demonstrate the superiority of our ensemble learning model and the proposed fusion strategy.

2 Related work

Deep learning plays a vital role in various computer vision tasks, and it has been widely used in the field of medical imaging for diagnosis. In Adriman’s study [20], deep learning and texture feature extraction methods were combined to classify DR. The feature extraction process in this method is performed in the convolutional layer and the pooling layer, which is not sensitive to the size of the input data. Gayathri et al. [21] put forward to utilize multi-channel convolutional neural network and conventional machine learning classifier for the task of DR classification. Gangwar et al. [22] developed an Inception-ResNet-v2 model based on transfer learning for automatic DR detection. Kassani et al. [23] presented a modified Xception architecture for the diagnosis of DR disease. These methods are not sensitive to the problem of unbalanced data distribution. The success of ensemble learning in the recognition task inspired Oh et al. [24] to employ ensemble learning and the active sample selection to solve the imbalanced biomedical data classification problem and improve the prediction and generalization ability of the model. The active sample selection means building a classifier by starting from a small balanced subset of training data and training the classifier iteratively by adding informative examples into the current training set. It solves the imbalanced data problem by iteratively sampling the useful training examples from the entire training data. The active sample selection process is an iterative procedure that includes a training phase and an example selection phase. In the training stage, the model is trained using only the examples in the training set. In the example selection phase, informative examples in the rest of the set are chosen to the training set in terms of the predefined measure of usefulness of the examples with respect to the current model. However, the proposed active sample selection method is easily affected by the sample selection strategy, and thus leading to information bias or negligence of important samples.

In this paper, we propose an adaptive weighted ensemble model based on three base models for better performance in the task of DR diagnosis. For this purpose, a novel decision fuse strategy is further introduced to dynamically adjust the weight distribution of these base models to alleviate the negative effects potentially caused by the problem of unbalanced sample size. Specifically, the weight of each base model in decision-making fusion is calculated based on the Bayesian theory in terms of the key evaluation indicators after each iteration. Thus, it is able to cooperate and reinforce each other to improve the robustness and generalization ability of the overall model, and the introduction of adaptive weights supplements the classification performance of the class of small sample size.

3 Proposed method

The overall architecture of our proposed adaptive weighted ensemble model is shown in Fig. 1. It is primarily composed of three parts: data preprocessing, model training, and decision fusion. Data preprocessing and enhancement are conducted in the first part. The outputs of this part are fed into the model training module to train and iterate through three base models (i.e. Swin Transformer, FocalNet and ResNet50). To achieve a better performance, we employ these base models to extract the discriminative features for all three categories of the data. The Swin Transformer improves its ability to process global information by introducing shifted windows. The FocalNet uses hierarchical context aggregation to support multi-scale image classification, making the model more robust to images of different scales and improve classification performance. The ResNet50 solves the problem of gradient explosion in neural network training. To better utilize the cues contained in each base model, we record the loss, accuracy, and quadratic weighted Kappa for each base model during each verification round. In terms of these indicators, we calculate and dynamically update the weights of each model based on our proposed adaptive weighted fusion strategy. The results of these three models are concatenated and fused in the decision fusion part, aiming at enhancing the classification performance, free from imbalanced distribution of classes.

Fig. 1

The architecture of the proposed adaptive weighted ensemble learning model.

3.1 Data preprocessing

For the raw data, we randomly flip the image horizontally and vertically by using image processing functions (e.g. RandomRotation, RandomHorizontalFlip and RandomVerticalFlip). This augmentation operation could provide mirror and symmetric variations, increase the diversity of data, and help the model better understand images from different perspectives and orientations. We also employ AutoAugment for data augmentation. Specifically, we add random rotations at arbitrary angles between –15° and +15°. We assign a 60% horizontal flip probability and a 40% vertical flip probability to the data of each epoch. Both of them aim to increase the randomness of the data, and thus improve the generalization and robustness ability of the model. The data enhancement and normalization are further executed to facilitate the subsequent feature extraction. Figure 2 shows three representative examples of image preprocessing.

Fig. 2

Representative examples of image preprocessing. From left to right columns, the images are (a) raw data, (b) random rotated and flipped images, and (c) enhanced images, respectively. Non-DR denotes non-diabetic retinopathy, NPDR means non-proliferative diabetic retinopathy, and PDR represents proliferative diabetic retinopathy.

3.2 Model training

For better performance, three base models (SwinTransformer, FocalNet and ResNet50) are assembled in the ensemble learning model to accomplish the task of DR detection. Swin Transformer provides a better global view by continuously dividing the image into smaller patches, enabling the patches to interact with each other. Compared with traditional convolutional neural networks, Swin Transformers are more accurate when dealing with large-scale objects, which refers to that it is good at extracting global semantic information. By introducing shifted windows, the Swin Transformers pay more attention to the relationship between different areas in the lesion image when learning data features. In DR classification, this means that the model can more effectively capture the distribution of widely distributed normal blood vessels and diseased areas in the fundus images, allowing the network to more comprehensively understand the contextual information of the entire retina. Config.1 illustrates the detailed weight configuration information of Swin Transformer. FocalNet uses hierarchical context aggregation to support multi-scale image classification, which can make the model more robust to images of different scales and improve classification performance. Config.2 represents the configurations of Focal Net. The residual block in ResNet50 has a clear information flow, making the decision-making process easy to explain and understand. The detailed architectures of ResNet50 is given in Table 1. By leveraging each model’s strengths, the overall performance could be enhanced significantly in comparison with single model. The pre-trained weights on ImageNet are imported into the Swin Transformer and FocalNet. The ResNet50 encapsulated in torchvision are employed. We reconstruct the connection layers relevant to our classification task.

Config 1. Configurations of Swin transformer
NAME: swin_tiny_patch4_window7_224
TYPE: swin
DROP_PATH_RATE: 0.2
SWIN:
EMBED_DIM: 96
DEPTHS: [2 , 2]
NUM_HEADS: [3 , 24]
WINDOW_SIZE: 7

Config 2. Configurations of FocalNet
NAME: focalnet_tiny_srf
TYPE: focalnet_tiny_srf
DROP_PATH_RATE: 0.2
FOCAL:
EMBED_DIM: 96
DEPTHS: [2 , 2]
FOCAL_LEVELS: [2 , 2]
FOCAL_WINDOWS: [3 , 3]

Table 1
Structure of ResNet

layer name output size 50-layer

conv1 112× 112 7× 7, 64, stride 2

conv2_X 56× 56 $[\begin{matrix} 1 \times 1, 64 \\ 3 \times 3, 64 \\ 1 \times 1, 256 \end{matrix}]$ × 3

conv3_X 28×28 $[\begin{matrix} 1 \times 1, 128 \\ 3 \times 3, 128 \\ 1 \times 1, 512 \end{matrix}]$ × 4

conv4_X 14×14 $[\begin{matrix} 1 \times 1, 256 \\ 3 \times 3, 256 \\ 1 \times 1, 1024 \end{matrix}]$ × 6

conv5_X 7× 7 $[\begin{matrix} 1 \times 1, 512 \\ 3 \times 3, 512 \\ 1 \times 1, 2048 \end{matrix}]$ × 3

1×1 average pool, 3-d fc, softmax

layer name	output size	50-layer
conv1	112× 112	7× 7, 64, stride 2
conv2_X	56× 56	$[\begin{matrix} 1 \times 1, 64 \\ 3 \times 3, 64 \\ 1 \times 1, 256 \end{matrix}]$ × 3
conv3_X	28×28	$[\begin{matrix} 1 \times 1, 128 \\ 3 \times 3, 128 \\ 1 \times 1, 512 \end{matrix}]$ × 4
conv4_X	14×14	$[\begin{matrix} 1 \times 1, 256 \\ 3 \times 3, 256 \\ 1 \times 1, 1024 \end{matrix}]$ × 6
conv5_X	7× 7	$[\begin{matrix} 1 \times 1, 512 \\ 3 \times 3, 512 \\ 1 \times 1, 2048 \end{matrix}]$ × 3
	1×1	average pool, 3-d fc, softmax

During the training stage, in order to push the model learn and fit the data better, the SoftTargetCrossEntropy is utilized. Meanwhile, Apex’s automatic mixed precision (AMP) is introduced to speed up the model training without loss of accuracy. Mixup is further employed to increase the diversity of the data.

We calculate the loss, accuracy, and quadratic weighted Kappa for each sub-model during each verification round. The custom AverageMeter class is used to record and manage these indicators. In terms of these indicators, we calculate and dynamically update the weights of each model based on our proposed fusion strategy. Details of the proposed adaptive weighted decision fuse strategy will be explained and discussed in the following subsection.

3.3 Decision fusion

To better accomplish the decision fusion task, we develop a novel dynamic weight updating method based on the Bayesian theory. Regarding the evaluation criteria collected after each verification, the key evaluation criteria employed in this work include the QWK, accuracy, and loss values. The loss value of each model is as the prior probability of the fusion strategy, denoted as ω _loss,i. We normalize it backward to obtain $L_{i}^{'}$ , meaning the inverse proportion of the loss value in each model. We have $L_{i}^{'} = 1 - \frac{ω_{loss, i}}{\sum_{i = 1}^{3} ω_{loss, i}},$ (3-1) where i represents the i^th (i = 1, 2, 3) base model.

To prevent $L_{i}^{'}$ fluctuating wildly, the coefficients α₁ and β₁ are introduced as below, $L_{i} = α_{1} + β_{1} \times \frac{L_{i}^{'}}{\sum_{i = 1}^{3} L_{i}^{'}},$ (3-2) where α₁ and β₁ are regularization factors. They can drive the data be mapped in a smaller range, avoiding the overfitting issue.

Further, the normalized QWK score of overall model under the current loss ratio is denoted as K_i. According to the total probability formula, the proportion distribution of the QWK of each model can be obtained: $Q = \sum_{i = 1}^{3} (K_{i} \times L_{i}) .$ (3.3) Thus, the proportion of overall model in terms of the Bayesian weight can be expressed as: $P_{i} = \frac{L_{i} \times K_{i}}{Q} .$ (3-4)

Let the accuracy of each model be denoted as A_i. It is used as the initial weight for each model and is multiplied with P_i element by element to obtain the updated weight array . is further added with P_i to get the weight to prevent the individual model from overfitting. ${W^{″}}_{i} = P_{i} \times A_{i},$ (3-5) ${W^{'}}_{i} = P_{i} + W^{″} .$ (3-6) Analogously, we use the coefficient α₂ and β₂ to ensure the stability of the overall model, formulated as: $W_{i} = α_{2} + β_{2} \times \frac{{W^{'}}_{i}}{\sum_{i = 1}^{3} {W^{'}}_{i}}$ (3-7)

Compared to directly using P_i as the weight of each model, the updated weight W_i is adjusted by using the accuracy of each model in the validation as a factor to tune each weight individually. This personalized weight adjustment is able to make the model better adapt to the requirements of specific tasks and promote the flexibility and generalization ability of the model.

4 Experiments and results

4.1 Dataset

In the present work, we conduct experiments on two retinal image datasets to verify the effectiveness of our method. The Diabetic Retinopathy Analysis Challenge 2022 (DRAC2022) [25] dataset is from Shanghai Sixth People’s Hospital and SVision Imaging, Ltd, containing a total of 611 images. It is the first version related to MICCAI 2022. It consists of 329 Non-DR, 212 NPDR, 70 PDR, and 386 unlabeled testing data for DR classification task. The NPDR usually manifests as microaneurysm, vein beading and retinal microvascular abnormalities. The damage to the blood vessel walls can lead to blood vessel leakage, causing retinal tissue edema. The Non-DR shows no abnormality. The vitreous body inside the eyeball should be transparent, and the blood vessels should show a regular and clear branching pattern. The PDR is characterized by neurovascular complications, such as vitreous hemorrhage, intravitreal neovascularization, and retinal traction detachment, which can cause blindness. The major difference between PDR and NPDR is that PDR is featured with new and abnormal blood vessels, while the main manifestation of NPDR is microvascular disease. More importantly, with this dataset, different algorithms can test their performance and make fair comparisons with other algorithms. We believe this dataset is an important milestone for automatic image quality assessment, lesion segmentation, and DR grading.

Another dataset used in this work was provided by the DR Detection Challenge (APTOS 2019) [26] sponsored by the Asia-Pacific Tele-Ophthalmology Society in 2019. This dataset contains a series of fundus images for classifying and grading DR. Each image has been marked by a clinical physician and divided into five grades according to severity: "0" means no lesion (No_DR), "1" means mild non-proliferative diabetic retinopathy (Mild), "2" means moderate non-proliferative diabetic retinopathy (Moderate), "3" indicates severe non-proliferative diabetic retinopathy (Severe), and "4" indicates proliferative diabetic retinopathy (Proliferative). The goal of this dataset is to classify fundus images through machine learning and deep learning algorithms to help physicians in the early diagnosis and treatment of DR disease. The image sizes of the experimental datasets of both DRAC2022 and APTOS 2019 are all 224×224 pixels.

4.2 Experiment setup

During the training stage, we update all weights with a batch-size of 16 in each iteration. The training epoch, learning rate and num_workers are 80, 10^-4 and 4, respectively. The ImageNet1K weights officially provided by focal-net and swin-transformer are used as the initial weight of the model. The training data is divided into two subsets, the training set and the validation set. As suggested by previous works [27, 28], 80% of the data are used for training and 20% for validation.

Regarding the experiments, we first carry out an ablation experiment to compare the performance between the proposed ensemble learning model and single deep learning models. To further verify the effectiveness of our proposed adaptive weighted fusion strategy, we perform comparative experiments with our method and the commonly used decision fusion approaches. Additionally, we also compare our experimental results with the previous related research work for the task of diagnosis of DR disease.

We employ four widely used metrics to assess the model performance: accuracy (ACC), precision, Quadratic Weighted Kappa score (Wk), and AUC value. The formulas of these metrics are as follows. Accuracy is defined as the ratio of correctly predicted labels to the total number of labels. $Accuracy = \frac{TP + TN}{TP + FN + FP + TN},$ (4-1) where TP (true positive) is the number of correctly classified diabetic retinopathy samples, FP (false positive) is the number of non-diabetic retinopathy samples misdiagnosed as positive, TN (true negative) is the number of correctly classified non-diabetic retinopathy samples, and FN (false negative) is the number of diabetic retinopathy samples misdiagnosed as negative ones. $Precision = \frac{TP}{TP + FP} .$ (4-2)

Precision means the correct proportion of positive example recognition. In other words, it denotes the ratio of actually positive samples out of all the positive predictions. High Precision is crucial for the task of reducing misdiagnosis in DR classification, as it reduces the risk of incorrectly predicting healthy samples. $Wk = 1 - \frac{\sum_{ij} w_{i, j} O_{i, j}}{\sum_{i, j} w_{i, j} E_{i, j}} .$ (4-3)

Among them, w_i,j is the weight matrix calculated based on the difference between actual label i and the predicted label j. O_i,j represents the number of the i^th category that is classified to be the j^th category. E is calculated as the outer product between the actual histogram vector of outcomes and the predicted histogram vector. Wk is a ratio between –1 and 1 that measures the consistency between model predictions and actual labels. The closer Wk is to 1, the better the model’s predictions is.

AUC (area under the ROC curve) represents the degree of separability. It is an important evaluation metrics for checking the classification model’s performance. It tells how much the model is capable of distinguishing between classes. The higher the AUC value, the better the model is at distinguishing between patients with the DR disease and no disease.

4.3 Experimental results

4.3.1 Comparison with single models

On the DRAC2022 dataset, we submit the results of the testing data with our proposed ensemble model to the official site for verification. A Wk value of 0.8487 and an AUC value of 0.9002 are obtained. Meanwhile, the accuracy and precision of the proposed method are 0.9343 and 0.9214, respectively. We compare the performance of the proposed ensemble learning model and the corresponding single models. Additionally, another two state-of-art object detection models are also developed for comparison. The performance of different models are shown in Table 2. One can see, among these four indicators, our method achieves the best performance in terms of accuracy, Wk and Precision, and a comparable good AUC value. We also list several representative predictive instances of DR classification on the DRAC2022 dataset (see Table 4). Those results demonstrate that ensemble learning method has advantages over the single deep learning methods.

Table 2
Performance of different models on the DRAC2022 dataset

Model ACC Wk AUC Precision

FocalNet 0.8903 0.8063 0.8903 0.8714

Swin transformer 0.8889 0.8108 0.9033 0.9017

ResNet 0.6653 0.6023 0.7109 0.8357

ShffleNetV2 0.8857 0.6106 0.5052 0.8857

GoogleNet 0.8285 0.7498 0.5170 0.8286

Our method 0.9343 0.8487 0.9002 0.9214

Model	ACC	Wk	AUC	Precision
FocalNet	0.8903	0.8063	0.8903	0.8714
Swin transformer	0.8889	0.8108	0.9033	0.9017
ResNet	0.6653	0.6023	0.7109	0.8357
ShffleNetV2	0.8857	0.6106	0.5052	0.8857
GoogleNet	0.8285	0.7498	0.5170	0.8286
Our method	0.9343	0.8487	0.9002	0.9214

Table 3

Performance of different models on the APTOS 2019 dataset

Model	ACC	Wk	AUC	Precision
FocalNet	0.8956	0.8851	0.8992	0.8361
Swin transformer	0.8826	0.8826	0.8929	0.8347
ResNet	0.8573	0.8498	0.6725	0.7958
ShffleNetV2	0.8828	0.8771	0.6865	0.7784
GoogleNet	0.8644	0.8461	0.6045	0.7527
Our method	0.8956	0.9007	0.8942	0.8458

Table 4

Prediction results of different models on DRAC2022 dataset

Image	Label	Our Method	FocalNet	Swin transformer	ResNet	GoogleNet	ShffleNet
	PDR	PDR	PDR	PDR	Non_DR	NPDR	NPDR
	Non_DR	Non_DR	Non_DR	NPDR	Non_DR	NPDR	NPDR
	Non_DR	Non_DR	NPDR	Non_DR	Non_DR	NPDR	NPDR
	PDR	PDR	PDR	PDR	Non_DR	NPDR	Non_DR
	NPDR	NPDR	NPDR	NPDR	Non_DR	PDR	Non_DR
	NPDR	NPDR	NPDR	Non_DR	Non_DR	NPDR	Non_DR
	PDR	PDR	PDR	PDR	Non_DR	NPDR	Non_DR
	NPDR	NPDR	NPDR	NPDR	Non_DR	PDR	NPDR

To further evaluate the proposed method, we also test it on the APTOS 2019 dataset. Table 3 shows the experimental results of different models. Remarkable quantitative results are achieved by our method, with an accuracy of 0.8956, a QWK of 0.9007, a AUC value of 0.8942, and a precision of 0.8458. Representative instances of DR classification on the APTOS 2019 dataset are shown in Table 5. The performance difference between our method and other methods further illustrates the effect of our proposed ensemble learning approach. The ensemble learning is helpful in enhancing the performance of DR classification.

Table 5

Prediction results of different models on the APTOS2019 dataset

Image	Label	Our Method	FocalNet	Swin transformer	ResNet	GoogleNet	ShffleNet
	Moderate	Moderate	Severe	Moderate	Moderate	Severe	PDR
	Moderate	Moderate	PDR	Moderate	Moderate	Severe	Mild
	Moderate	Moderate	Moderate	Mild	Mild	Severe	Mild
	Moderate	Moderate	Moderate	Mild	Moderate	Severe	Mild
	No_DR	No_DR	No_DR	No_DR	Mild	PDR	Mild
	No_DR	No_DR	No_DR	No_DR	Mild	Severe	Mild
	PDR	PDR	Severe	PDR	PDR	Severe	PDR
	PDR	PDR	PDR	Mild	Mild	Severe	PDR

4.3.2 Comparison with different fusion schemes

We have verified the effect of ensemble learning on the performance of the classification model. In this section, we compare our method with other decision fusion strategies, including simple average, weighted average, and Bayesian average. Simple average refers to that the outputs of the base models are directly summed, and then divided by the number of base models to produce the final output, that is allocating equal weight for each model. Weighted average allows to assign weighted or proportioned weight, that is, each model can be assigned a unique weight. Bayesian average takes uncertainty into account. It uses Bayesian statistical analysis to combine information from different sources, resulting in a more accurate decision result. They are seperately applied to both the DRAC2022 and APTOS 2019 datasets. The comparison results are recorded in Tables 6 and 7. Examples of the corresponding DR classification results are illustrated in Tables 8 and 9, respectively. It can be observed that the proposed adaptive weighted fusion strategy outperforms the conventional fusion schemes.

Table 6
Performance of different decision fusion methods on the DRAC2022 dataset

Decision Fusion ACC Wk AUC

Simple average 0.8492 0.8168 0.8857

Bayesian average 0.8842 0.8179 0.8972

Weighted average 0.8857 0.8269 0.9000

Our method 0.9343 0.8487 0.9214

Decision Fusion	ACC	Wk	AUC
Simple average	0.8492	0.8168	0.8857
Bayesian average	0.8842	0.8179	0.8972
Weighted average	0.8857	0.8269	0.9000
Our method	0.9343	0.8487	0.9214

Table 7

Performance of different decision fusion methods on the APTOS 2019 dataset

Decision Fusion	ACC	Wk	AUC
Simple average	0.8430	0.8900	0.8431
Bayesian average	0.8438	0.8910	0.8417
Weighted average	0.8430	0.8920	0.8431
Our method	0.8956	0.9007	0.8458

Table 8

Prediction results of different decision fusion on the DRAC2022 dataset

Image	Label	Our Method	Weighted Average	Bayesian average	Simple average
	PDR	PDR	NPDR	NPDR	NPDR
	Non_DR	Non_DR	NPDR	Non_DR	NPDR
	Non_DR	Non_DR	NPDR	NPDR	NPDR
	NPDR	NPDR	PDR	PDR	PDR
	Non_DR	Non_DR	NPDR	NPDR	NPDR

Table 9

Prediction results of different decision fusion on the APTOS2019 dataset

Image	Label	Our Method	Weighted Average	Bayesian average	Simple average
	Mild	Mild	Moderate	Moderate	Moderate
	Moderate	Moderate	Mild	Mild	Mild
	No_DR	No_DR	Mild	Mild	Mild
	No_DR	No_DR	Mild	No_DR	Mild
	PDR	PDR	Mild	Mild	Mild

4.3.3 Comparison with related works

Most of the previous research efforts have used deep learning techniques to detect the presence or absence of DR disease. Nevertheless, it it still a challenging problem to exactly detect the DR disease. It is necessary to make a relative comparison on related works using the same dataset. Gangwar et al. [22] utilized the transfer learning on pre-trained Inception-ResNet-v2, and added customized CNN layer blocks on top of Inception-ResNet-v2 to build a hybrid model. The accuracy rate obtained on the APTOS 2019 is 82.18%. Kassani et al. [23] developed a modified Xception architecture as the feature extractor, combining a multi-layer perceptron for DR severity classification, achieving a classification accuracy of 83.09% on the APTOS 2019 dataset. However, these works ignore the fact that a single model is sensitive to noise and unstable, which might lead to weak generalization ability. Thus, in our work, we employ an ensemble learning based on multiple models to reduce the variance and the generalization error. Most importantly, we propose an adaptive weighted decision fusion strategy to comprehensively excavate the cues implicit in different classification models, to maximize the prediction effect. As a result, better classification accuracy of 89.56% is obtained with our proposed approach. More specific details can be found in Table 10. Since no relevant articles were reported for the DRAC2022 dataset, we did not perform comparative analysis on this dataset.

Table 10
Performance of different decision fusion methods on the APTOS 2019 dataset

Model Total Data Training Data Validation Data Testing Data ACC

Inception-ResNet-v2 [21] 3,662 2,851 - 811 82.12

Modified Xception [22] 3,662 2,651 622 343 83.09

Our method 3,662 2,746 559 357 89.56

Model	Total Data	Training Data	Validation Data	Testing Data	ACC
Inception-ResNet-v2 [21]	3,662	2,851	-	811	82.12
Modified Xception [22]	3,662	2,651	622	343	83.09
Our method	3,662	2,746	559	357	89.56

4.4 Discussion

4.4.1 Data distribution

The data distribution is not always balanced in the real world, in which case it is difficult to achieve the best diagnostic results for the minority category of class. The distribution of the classes regarding these two datasets leveraged in the experiment is illustrated in Fig. 3. We can find that the quantity of samples with lesions and the quantity of normal samples are extremely unbalanced. Without artificially increasing the data size, during the training and validation stages, the weight of each model is adjusted dynamically according to the feedback of the key indicators. By setting more reasonable weights, the performance of the overall model is promoted, suggesting the proposed fusion scheme is helpful to alleviate the problem of imbalanced distribution of classes. Our proposed adaptive weighted ensemble model is able to better utilize the cues contained in each base models, identify intrinsic features in DR OCT images, and focus on them especially for the class of small sample size, to facilitate the diagnosis of the lesion.

Fig. 3

Data distributions of the DRAC and APTOS datasets.

4.4.2 Result analysis

Comparing the performance of different models, it indicates that the proposed method is effective and feasible in the task of DR classification. The usage of ensemble learning is superior to that of a single deep learning model. The intuition behind is that, each single model has its own strengths, they can capture different aspects and patterns in the data, and the same mistakes normally hardly occur between different models on the testing dataset. Therefore, by integrating the predictions of multiple models, the ensemble model makes the classification results less sensitive to the details of training, which can reduce bias and variance, and promote the overall performance and robustness. Notably, when the dataset is unbalanced, if the fuse method or weight distribution adopted is unreasonable, the ensemble performance may be not as good as the result by single model. When simple average is utilized as the fusion method, equal weights are allocated for each base model. However, the emphasis of these adopted three models are different. For example, Swin Transformers are capable of strong feature representation for both the local and the global features between different categories because of its strengths. The role of this model is undermined with “equal weight”. In this case, the weight distribution is deemed to unreasonable. To promote the overall recognition and generalization ability of the classification model, we adopt an adaptive weight allocation strategy to dynamically adjust the weights according to the performance of each base model. Models with better performance can be assigned with higher weights, thereby increasing their influence in the ensemble model. (c.f. Tables 2 vs. 6 and Tables 3 vs. 7).

Moreover, during the training stage, we collect the key indicators, in terms of which the weights of each model can be dynamically determined based on the Bayesian theory. Compared with other decision fusion strategies, the proposed fusion strategy can better identify the classification-oriented information contained in the models because of the optimal weight distribution. Better performance, reliability, and generalization capabilities can be guaranteed with the proposed approach. Overall, the proposed model outperforms existing methods for DR classification task.

Additionally, the proposed ensemble model makes the best of these three base models’ strengths to excavate the comprehensive and complementary information of the entire retina for higher classification performance. This could also be the key for other underlying models to enhance the overall recognition and generalization ability. The present work provides a general ensemble learning framework and decision fusion scheme for medical image classification tasks. Moreover, the models with unique features in other scenarios are encouraged to introduce into the ensemble learning framework to improve its performance. The proposed method has a potential to be extended to more pattern recognition fields where different models need to jointly understand category-related features beyond that of diabetic retinopathy classification.

5 Conclusions

In this article, we propose an adaptive weighted ensemble learning network for DR detection. It is capable of automatically and dynamically assign the weight distribution of each base model based on the Bayesian theory in terms of the key assessment criteria. The network could make greatest use of the clues provided by these multiple models through our proposed decision fusion strategy to achieve better performance even in the case with unbalanced distribution of classes. The experimental results on two datasets suggest that the proposed method outperforms the existing approaches for DR detection. We believe that the performance could be further enhanced with both the improvement of image quality and more advanced deep ensemble learning techniques.

Acknowledgements

This research was funded by the National Natural Science Foundation of China (Grant Nos. 61902282, 62071330 and 62201385).

Data availability statement

Copies of the DRAC2022 dataset and the APTOS 2019 dataset can be obtained free of charge from https://drac22.grand-challenge.org and https://www.kaggle.com/datasets/mariaherrerot/aptos2019, respectively

References

Lili Yuen , Pouya Saeedi , Musarrat Riaz , Suvi Karuranga , Hema Divakar , Naomi Levitt , Xilin Yang , David Simmons , Projections of the prevalence of hyperglycaemia in pregnancy in and beyond: Results from the international diabetes federation diabetes atlas, 9th edition, Diabetes Research and Clinical Practice 157 (2019), 107841.

Centers for Disease Control, Prevention et al., National diabetes fact sheet: national estimates and general information on diabetes and prediabetes in the united states, Atlanta, GA: US department of health and human services, centers for disease control and prevention, 201(1) (2011), 2568–2569.

Klein

, Klein

, Moss

et al. The wisconsic epidemiologic study of diabetic retinopathy: Xvii, The 14-year incidence and progression of diabetic retinopathy and associated risk factors in type 1 diabetes, Ophthalmology 105(10) (1998), 1801–1815.

Yue Pan , Limao Zhang , Xianguo Wu , Miroslaw Skibniewski

, Multi-classifier information fusion in risk analysis, Information Fusion 60 (2020), 121–136.

Richard Spaide

, James Fujimoto

, Nadia Waheed

, Srinivas Sadda

, Giovanni Staurenghi , Optical coherence tomography angiography, Progress in Retinal and Eye Research 64 (2018), 1–55.

Karen Schaal

, Marion Munk

, Iris Wyssmueller , Lieselotte Berger

, Martin Zinkernagel

, Sebastian Wolf , Vascular abnormalities in diabetic retinopathy assessed with swept-source optical coherence tomography angiography widefield imaging, Retina 39(1) (2019), 79–87.

Paulo Stanga

, Alessandro Papayannis , Emmanouil Tsamis , Francesco Stringa , Tim Cole , Yvonne D’Souza , Assad Jalil , New findings in diabetic maculopathy and proliferative disease by swept-source optical coherence tomography angiography, OCT Angiography in Retinal and Macular Diseases 56 (2016), 113–121.

Zhang

, Rezaei

K.A.

, Saraf

S.S.

, Chu

, Wang

R.K.

, Ultra-wide optical coherence tomography angiography in diabetic retinopathy quant, Imaging Med Surg 8(8) (2018), 743.

Jonathan Russell

, Yingying Shi, , John Hinkle

, Nathan Scott

, Kenneth Fan

, Cancan Lyu , Giovanni Gregori , Philip Rosenfeld

, Longitudinal widefield swept-source oct angiography of neovascularization in proliferative diabetic retinopathy after panretinal photocoagulation, Ophthalmology Retina 3(4) (2019), 350–361.

10.

Ramzi Adriman , Kahlil Muchtar , Novi Maulina , Performance evaluation of binary classification of diabetic retinopathy through deep learning techniques using texture feature, Procedia Computer Science 179 (2021), 88–94.

11.

Shuying Liu , Weihong Deng Very deep convolutional neural network based image classification using small training sample size, In 2015 3rd IAPR Asian conference on pattern recognition (ACPR), pages 730–734. IEEE, 2015.

12.

Alex Krizhevsky , Ilya Sutskever , Geoffrey Hinton

, Imagenet classification with deep convolutional neural networks, Communications of the ACM 60(6) (2017), 84–90.

13.

Leilei Jin , Hong Liang Deep learning for underwater image recognition in small sample size situations, In OCEANS 2017-Aberdeen, pages 1–4. IEEE, 2017.

14.

Geert Litjens , Thijs Kooi , Babak Ehteshami Bejnordi , Arnaud Arindra Adiyoso Setio , Francesco Ciompi , Mohsen Ghafoorian , Jeroen Awm Van Der Laak , Bram Van Ginneken , Clara Sànchez

, A survey on deep learning in medical image analysis, Medical Image Analysis 42 (2017), 60–88.

15.

Keiron O’Shea , Ryan Nash An introduction to convolutional neural networks, arXiv preprint arXiv:1511.08458, 2015.

16.

Gayathri

, Varun Gopi

, Palanisamy

, Diabetic retinopathy classification based on multipath cnn and machine learning classifiers, Physical and Engineering Sciences in Medicine 44(3) (2021), 639–653.

17.

Ling Dai , Liang Wu , Huating Li , Chun Cai , Qiang Wu , Hongyu Kong , Ruhan Liu , Xiangning Wang , Xuhong Hou , Yuexing Liu et al. A deep learning system for detecting diabetic retinopathy across the disease spectrum, Nature Communications 12(1) (2021), 3242.

18.

Yue Pan , Limao Zhang , Xianguo Wu , Miroslaw Skibniewski

, Multi-classifier information fusion in risk analysis, Information Fusion 60 (2020), 121–136.

19.

Lincoln Chivinge , Leslie Kudzai Nyandoro , Kudakwashe Zvarevashe Quadratic weighted kappa score exploration in diabetic retinopathy severity classification using efficientnet, In 2022 1st Zimbabwe Conference of Information and Communication Technologies (ZCICT), pages 1–9. IEEE, 2022.

20.

21.

Gayathri

, Varun Gopi

, Palanisamy

, Diabetic retinopathy classification based on multipath cnn and machine learning classifiers, Physical and Engineering Sciences in Medicine 44(3) (2021), 639–653.

22.

Akhilesh Kumar Gangwar , Vadlamani Ravi Diabetic retinopathy detection using transfer learning and deep learning, In Evolution in Computational Intelligence: Frontiers in Intelligent Computing: Theory and Applications (FICTA 2020), Volume 1, pages 679–689. Springer, 2021.

23.

Sara Hosseinzadeh Kassani , Peyman Hosseinzadeh Kassani , Reza Khazaeinezhad , Michal Wesolowski

, Kevin Schneider

, Ralph Deters Diabetic retinopathy classification using a modified xception architecture, In 2019 IEEE international symposium on signal processing and information technology (ISSPIT), pages 1–6. IEEE, 2019.

24.

Sangyoon Oh , Min Su Lee , Byoung-Tak Zhang , Ensemble learning with active example selection for imbalanced biomedical data classification, IEEE/ACM Transactions on Computational Biology and Bioinformatics 8(2) (2010), 316–325.

25.

Qinqin Zhang , Kasra Rezaei

, Steven Saraf

, Zhongdi Chu , Fupeng Wang , Ruikang Wang

, Ultra-wide optical coherence tomography angiography in diabetic retinopathy, Quantitative Imaging in Medicine and Surgery 8(8) (2018), 743.

26.

Sohier Dane Karthik Maggie, Aptos 2019 blindness detection, 2019.

27.

Shenming Hu , Xinze Luan , Hong Wu , Xiaoting Wang , Chunhong Yan , Jingying Wang , Guantong Liu , Wei He , Accv: automatic classification algorithm of cataract video based on deep learning, BioMedical Engineering OnLine 20 (2021), 1–17.

28.

Harrison Bai

, Robin Wang , Zeng Xiong , Ben Hsieh , Ken Chang , Kasey Halsey , Thi My Linh Tran , Ji Whae Choi , Dong-Cui Wang , Lin-Bo Shi et al. Artificial intelligence augmentation of radiologist performance in distinguishing covid-19 from pneumonia of other origin at chest ct, Radiology 296(3), E156–E165, 2020.