Transfer learning model for false positive reduction in lymph node detection via sparse coding and deep learning

Abstract

Transfer learning technique is popularly employed for a lot of medical image classification tasks. Here based on convolutional neural network (CNN) and sparse coding process, we present a new deep transfer learning architecture for false positive reduction in lymph node detection task. We first convert the linear combination of the deep transferred features to the pre-trained filter banks. Next, a new point-wise filter based CNN branch is introduced to automatically integrate transfer features for the false and positive image classification purpose. To lower the scale of the proposed architecture, we bring sparse coding process to the fixed transferred convolution filter banks. On this basis, a two-stage training strategy with grouped sparse connection is presented to train the model efficiently. The model validity is tested on lymph node dataset for false positive reduction and our approach indicates encouraging performances compared to prior approaches. Our method reaches sensitivities of 71% /85% at 3 FP/vol. and 82% /91% at 6 FP/vol. in abdomen and mediastinum respectively, which compare competitively to previous approaches.

Keywords

Deep learning lymph node false positive reduction transfer learning method sparse coding algorithm

1 Introduction

Remarkable progress has been made in the fields such as cloud computing [9], data mining [49] and 5 G [50]. And significant development has also been achieved in computer vision analysis, thanks to the usability of wide-scale labeled databases and convolutional neural network (CNN). Therefore, many researchers use deep machine learning methods for medical image processing [5 , 47] and security [52, 53]. Whereas, different with traditional algorithms relied on hand-designed features [31], training CNN from scratch is impracticable because of the lack of labeled data for many medical image processing tasks. Moreover, annotating medical images needs professional medical field knowledge, which is tiresome and expensive. The size of biomedical image datasets is small in contrast to natural image datasets, such as IMAGENET [21].

To address this problem, transfer learning (TL) method is triumphantly employed in deep learning domain for medical image processing [30] where a network pre-trained on a source dataset is partly beneficial for target domain. Concretely, transfer learning includes two strategies: feature extracting and fine tuning. The former directly views the “off-the-shelf” features [34] derived from a pre-trained deep neural network (DNN) as a feature vector. Then train a new model from scratch for the target task. While fine tuning technique takes advantage of the pre-trained DNN to initialize the training parameters of the target model, and fine-tuning it with a smaller learning rate [27]. The issue of how to exploit the “off-the-shelf” features more efficiently for target tasks brings about lots of feature selection methods [25], e.g. wrapper, embedded-based and filter approaches, etc. The fine-tuning method, due to the database bias between source and target domains, unfortunately needs a large number of labeled training samples on a new task, and many digital medical image processing scenarios are not available [45]. Therefore, plentiful novel methods have been introduced, like domain adaptation techniques [20, 26], ensemble based approach [10] and multitask learning [7].

In practice, multisource transfer learning method has been used for a lot of computer-aided diagnosis fields [10 , 36]. Transferring knowledge from multiple source datasets to medical imaging analysis tasks could be more beneficial than barely from unrelated natural images. Moreover, the over-fitting problem of the target network can be eased by multisource transfer learning method based on the varying modality databases, which is very applicable for small-scale biomedical imaging databases. Nevertheless, the manner that acquiring the maximum benefit from the factory of the multisource transferred features becomes a tricky issue for the multisource transfer learning technique. While there are many feature fusion or selection techniques [25, 27] knowledge ensemble mechanisms [12] in computer-aided diagnosis field, they tend to the hand-craft methods. One is unavoidable to confront such a problem that which layers of the transferred DNNs are frozen or whether fixed or fine-tuning the pre-trained parameters for target task. It is usually addressed through an amount of well-designed experiments in practice. At the same time, the validity of transfer learning is also affected by the specific CNN frameworks (VGG, AlexNet etc.) trained on the source or target databases.

In this paper, we propose a new transfer learning method via CNN and sparse coding algorithm. We first convert the combination of the deep transferred features to the pre-trained filter banks. Next, a new point-wise convolution based CNN branch is introduced to automatically integrate transfer features for the false and positive image classification purpose. To lower the scale of the proposed architecture, we bring sparse coding process to the fixed transferred convolution filter banks. A two-stage training procedure with grouped sparse connection is also introduced to train the model efficiently. In other words, the proposed approach could be viewed as a “meta-learning” method, where our model can learn to learn from the fixed pre-trained parameters. The model validity is tested on lymph node (LN) database for false positive reduction and the proposed model indicates encouraging performances compared to prior approaches.

We give the primary contributions of this paper: 1) through 1×1 convolution, a new CNN based framework is introduced for multisource transfer learning. 2) We utilize sparse coding algorithm on the multisource transferred filters to decrease the model complexity and ease over-fitting phenomenon. 3) To improve our model feasibility, we devise a 2-phase training scheme via dropout and grouped sparse connection.

Rest of the part of the paper is organized as follows: first we briefly review the related literatures about deep transfer learning methods for medical image classification in Section 2. Section 3 illustrates the new designed framework. Section 4 introduces experiments and corresponding discussions on LN false positive reduction task. We give the conclusion in Section 5.

2 Related work

2.1 Convolutional neural network and transfer learning in medical image classification

CNN and transfer learning reveal superior performance in many medical classification tasks with different kinds of image modalities. In [44], three critical items, CNN frameworks, database characteristics and transfer learning, for computer-aided detection (CADe) are investigated for two clinical problems: thoraco-abdominal lymph node detection and interstitial lung disease classification. In [32], a two-layer neural network equipped with the features transferred from one pre-trained convolutional neural network is proposed to better classify the Human Epithelial-2 cell images. Beyond that, A small-scaled annotated dental image database is applied to three independent convolutional neural networks for diagnosis classification [33]. Reference [22] utilizes pre-trained Inception V3 model, and fine-tunes it on the annotated endoscopy images to detect the gastrointestinal bleeding. Relied on VGG-Net based transfer learning technique and a novel FCNet, the liver fibrosis in ultrasonic images are efficiently categorized [28]. In order to deal with a low quality and small-scale training database for skin lesion classification, the pre-trained AlexNet model is modified and fine-tuning process is also introduced to classify dermal images [16]. Moreover, multisource transfer learning technique is also utilized in [26] for lung pattern analysis. The literatures mentioned above demonstrate the effectiveness of CNN and transfer learning in medical image analysis field.

2.2 Parameter predicting in convolutional neural network

As above described, the point-wise convolution is used to acquire or predict the adaptive transfer knowledge (i.e. CNN filters in this paper) for target domain. Some researchers introduce several parameter predicting methods in deep learning manner. In [15], considering the parameterization redundancy existed in convolutional neural network, the authors employ a fraction of weight values (5%) to precisely predict the remaining parameters for each pre-trained convolution kernel. Based on the smoothness characteristic of the pre-trained weights, kernel-based method is used to obtain the factorized weight matrices to form a pre-trained parameter space. In addition, a learnet model is introduced in [4] to acquire deep model weights for one shot learning. Different with the previous method, it can non-linearly predicts the entire weight values for each layer by means of a given exemplar image. Lately, a lookup-based convolutional neural network is introduced in [3]. It encodes convolutions by several lookups to a dictionary which can be adaptively learned to structure the weight space of convolutional neural networks. Through training parameter dictionary and a small set of linear combinations jointly, the approach in [3] is able to accelerate the processes of model training and inference obviously.

3 Methods

3.1 The point-wise convolution based transfer learning

As previously described, we use the point-wise convolutional layer to straightly get the linear combination of transferred filters to generalize the model for the target domain. Figure 1 shows the whole framework.

Fig. 1

The whole framework of the proposed method (a) schematic diagram (b) point-wise convolutional layer based structure.

Figure 1(a) gives a plain schematic of the proposed approach and Fig. 1(b) shows the concrete view in convolutional (Conv) layer configuration. The transferred feature layer is consisting of the fixed pre-trained convolutional kernels of a certain deep model for the source databases.

Then we utilize the point-wise convolutional operation to linearly combine the multisource transferred features. Moreover, we also introduce the activation functions for point-wise convolutional layer, which is to increase the nonlinearity to the transferred features. We stack Fig. 1(b) structure layer by layer and obtain the fundamental framework of the proposed approach.

Unfortunately, directly performing this primary configuration is impracticable, and we must further improve its efficiency. Figure 2 demonstrates the elaborate structure corresponding to Fig. 1(b), where a certain input and output channels are considered respectively.

Fig. 2

The convolution process of transferred feature layer and 1×1 Conv layer for one single input and output channels.

Concretely, provided there are n input pre-trained convolutional filters ${f_{i}}_{i = 1}^{n}$ in the l - th convolutional layer. According to forward propagation process for the single input I_j, the convolutional output features of layer l is calculated by $F M^{(l)} = {g (I_{j} * f_{i})}_{i = 1}^{n}$ where g (·) denotes the activation function. Then point-wise convolution operation is implemented on FM^(l) for k - th output feature map (denoted as $a_{I_{j} \to k}^{(l)}$ ), which is corresponding to the input I_j. The process can be represented by Equation (1): $a_{I_{j} \to k}^{(l)} = g (\sum_{i = 1}^{n} w_{i} F M_{i}) = g (\sum_{i = 1}^{n} w_{i} g (I_{j} * f_{i}))$ (1) where w_i (i = 1, ⋯ , n) represents the trainable parameter of point-wise convolutional layer.

Based on Equation (1), we can deduce that the bottleneck of above framework is the undesirable numerous pre-trained filters, i.e. ${f_{i}}_{i = 1}^{n}$ , which leads to an impractical number of convolutional operations. Specifically, we set m input feature maps, k output feature maps and n transferred filters in this layer, there will be mn + mnk times convolutional operation. In practice, the quantity of pre-trained filters n is larger than m or k (e.g. k ≈ m = 256, n = 256 × 6).

To solve this issue, firstly the mean value of the output feature map, presented in Equation (1), can be estimated based on the approximate Equation (2) which is first introduced in [19]: $E [g (M * W)] \approx g (E [M * W])$ (2) where M is the input feature maps; W denotes convolutional filters; g (·) represents activation function. Applying Equation (2) to Equation (1), we get $\begin{matrix} E [a_{I_{j} \to k}^{(l)}] \approx g (E [\sum_{i = 1}^{n} w_{i} g (I_{j} * f_{i})]) \\ = g (E [g (\sum_{i = 1}^{n} {\tilde{w}}_{i} (I_{j} * f_{i}))]) \\ \approx g (g (E [\sum_{i = 1}^{n} I_{j} * {\tilde{w}}_{i} f_{i}])) \\ \approx g (E [I_{j} * \sum_{i = 1}^{n} {\tilde{w}}_{i} f_{i}]) \end{matrix}$ (3)

In Equation (3), one can certainly put the w_i into g (·) to turn into ${\tilde{w}}_{i}$ . We argue that this will not introduce any deviation for the final classification loss objective function, as the w_i and ${\tilde{w}}_{i}$ are all trainable parameters. For instance, with the generally employed activation function: ReLU (i.e. g (x) = x, if x > 0 or 0 otherwise), it is reasonable to make the w_i and ${\tilde{w}}_{i}$ be each other’s negatives to keep the same linear combination of the input transferred feature maps in the current layer.

In Equation (3), we approximately transform our raw point-wise convolutional layer based transfer learning structure, illustrated in Fig. 1(b), to a traditional convolution layer, excepting that its totally trainable convolutional kernels are replaced by a trainable linear combination of the fixed pre-trained filters. Based on this assumption, the model complexity and computation burden can be largely reduced by decreasing the intrinsic redundancy existing in the pre-trained filter space. In the next section, we will use the sparse coding algorithm to achieve this purpose.

3.2 The sparse coding algorithm on pre-trained filters

The former literatures [3, 15] reveal that a fair amount of redundant information exists in the parameter space of some deep learning based networks. Based on this fact, we can improve the feasibility of our model. At the same time, researchers have proposed many related approaches to cope with this redundancy, for example, kernel-based dictionary [15], autoencoder [1], factorized parameter matrix based model [4] and structured sparsity learning [13, 48].

Here, the sparse coding algorithm [37] on transferred filters is utilized to better cooperate with the 1×1 Conv layer. Specifically, we first extract the over-complete basis (i.e. ${A_{j}}_{j = 1}^{c}$ as a sparse dictionary) from pre-trained filter banks. Then substitute them for the transferred feature layer (i.e. ${f_{i}}_{i = 1}^{n}$ in Equation (3)). Consequently, based on the over-complete basis, the trainable linear combination in Equation (3) will generate the adaptive features for target tasks. Meanwhile, the number of over-complete basis c is much less than the transferred features n, which is also beneficial for model efficiency. In view of this, our method falls into “meta-learning” domain (i.e. we learn the sparse characteristics in the pre-trained parameter space from source domain in order to learn more adaptive features for the target tasks). As for the sparsity of the final learned features, we discuss it in the next section.

The objective function of sparse coding algorithm is shown below $J (A, s) = {∥ As - f ∥}^{2} + λ \sqrt{s^{2} + ɛ} + γ {∥ A ∥}^{2}$ (4) where A is the over-complete basis. s is a set of sparse features to represent the pre-trained convolutional filters, i.e. f. Moreover, in order to realize the L1 norm regularization in practice, we use $\sqrt{x^{2} + ɛ}$ in place of |x|, where ɛ is a “smoothing parameter”.

3.3 Features sparsity network based on grouped sparse connection

Rely on the over-complete basis extracted from transferred filters, the complexity of the 1×1 Conv layer is reduced dramatically. The total number of convolution operation in section 3.1 decreases to mc + mck where c is the number of over-complete basis. However, notice the sparsity among the weights on the basis (i.e. s in Equation (4) or ${\tilde{w}}_{i}$ in Equation (3)), the final convolution operation number can be reduced further. Corresponding to the 1×1 Conv layer, there are many redundant connections in every single input and output channel. Therefore, based on several methods about sparse convolution [13, 24], we devise a 2-phase training strategy in this paper.

Specifically, suppose every shared over-complete basis in l - th layer corresponds to a “meta-feature” for a certain input channel. In the beginning, we don’t know which one is necessary or redundant for a specific output feature channel. Similar to [3, 15], the convolution can be executed before the linear combination of over-complete basis for efficiency. The corresponding formula is given as $g (\sum_{i = 1}^{c} w_{i} (I_{j} * A_{i}))$ . Then the sparse weights can be learned with 1×1 Conv layer. If we keep tracing on all these weights throughout model training phase, there will be more supererogatory computation than traditional CNN mode (i.e. c times more) due to the additional transferred feature layer. At the same time, the feature sparsity described above is underutilized. Therefore, a more sophisticated training approach is needed here.

To this end, based on Dropout [19] and DropConnect [46] techniques, we introduce a new grouped sparse connection for the point-wise convolutional layer. Notice that the point-wise convolutional layer can also be viewed as fully connected (FC) layer with shared parameters for each spatial location in feature map domain. Figure 3 is a simple schematic comparison between Dropout, DropConnect and our grouped sparse connection network. Figure 3(a) depicts that Dropout network simply sets the output propagating to the next layer to 0 at random. DropConnect randomly lets the weights of neurons be 0 at current layer. Whereas our grouped sparse connection network can be viewed as a compromise version, which zero weights are randomly inter-grouped. In other words, the number of random zero weights connected with the same neuron in the next layer is fixed. For example, each group in Fig. 3(c) (denoted as V_i·) corresponds to a certain input channel and each V_ij represents its “meta-feature”. It means that we set the same sparse connection rate for all output channels. For clarity, the zero-masks of the three networks are shown in Fig. 4, where the white and grey blocks denote zero or non-zero elements respectively.

Fig. 3

The comparison between Dropout, DropConnect and our grouped sparse connection (the dotted lines represent zero outputs in (a) or zero weights in (b) and (c)).

Fig. 4

The zero-masks of Dropout, DropConnect and our grouped sparse connection network (two groups with spare connection rate, 0.6 in (c)).

Next, to implement our sparse connection network in practice, we adopt a 2-phase training mode. In the first phase, zero-masks are generated randomly for every mini-batch. Therefore, the over-complete bases are randomly selected to assemble the corresponding transferred features. That is, we maintain a parameter list for every 1×1 Conv layer to update the corresponding parameters in back propagation process. After certain iterations, we put a hard sparse constraint to the parameters in the second training phase. The first p maximum values in parameter lists are located and their corresponding masks are set to one for every input channel in 1×1 Conv layer. Then in the following iterations, the masks are frozen. We argue that the first p maximum parameter values correspond to the p most important features for a given input channel, and the others should be forced to zero to keep the learned features sparsity. Based on the proposed 2-phase training strategy, our network becomes more feasible and efficient. As for the related hyper-parameters, they are determined in section 4.

At last, similar with [19], the validation results in the first training phase can be acquired using the following equation: $\begin{matrix} E_{M} [g (M ⊙ W) v] \approx g (E_{M} [(M ⊙ W) v]) \\ = g ((1 - p) Wv) \end{matrix}$ (5) where M is the mask template; W is the parameter of 1×1 Conv layer; v is the input channel; p is the sparse rate in here; and · represents Hadamard product, i.e. element-wise product of two vectors.

3.4 The whole framework for false positive reduction in lymph node detection

Compared with the primitive Conv layer module in Fig. 1, the detail of the new improved CNN architecture in one layer is illustrated in Fig. 5.

Fig. 5

The architecture of proposed 1×1 Conv based transfer learning network.

We display the effectiveness of the proposed method in the application of false positive reduction task for lymph node. A concrete CNN model is designed based on the practical CT LN dataset scale (595 abdominal LNs from 86 patients and 388 mediastinal LNs from 90 patients). Specifically, we design the network architecture with five convolutional layers, two fully connected layers, and a binary classification softmax layer. Because the ROI is relatively small within every CT scan domain, the 3×3 convolution filters are used to extract the corresponding local spatial features [2] and pooling layers are abandoned excluding the last global average pooling layer [23]. Stacking the proposed 1×1 Conv based network block shown in Fig. 5 layer by layer, the overall structure of our CNN model is given by Fig. 6, where the point-wise convolutional layers contain the whole learnable weights of the network.

Fig. 6

The overall architecture of the proposed CNN model.

4 Results and discussion

In this section, we implement our model on the publicly available lymph node CT datasets for false positive reduction task [40, 42]. The datasets contain 90 patients with 389 mediastinal LNs, and 86 patients with 595 abdominal LNs. In section 4.1, we firstly present the multisource and target databases used in this paper. Following that is the experimental protocol in section 4.2. The strategy of hyper-parameter selection and the corresponding discussion are displayed in section 4.3. Lastly, section 4.4 gives the comparison between the proposed model and other representative methods.

4.1 The source and target databases

In this paper, we employ multiple source datasets to make the filter bank dictionary in sparse coding algorithm more generalized and adaptable for the target task. Here we use six source datasets, including one natural images database (IMAGENET [21]), four texture databases (ALOT [6], DTD [11], KTH-TIPS [14] and Flickr Material Dataset [43]) and one medical image dataset (Diabetic Retinopathy [5]). Considering training our model on the source databases separately is inefficient even impractical, we instead directly borrow the transferred filters from the “off-the-shelf” pre-trained VGG-M model [8] which is introduced by the former literatures [27, 35].

Next, we present the CT lymph node databases for target task. Firstly, clinicians labeled 388 mediastinal lymph nodes as positive class in CT images of 90 cases and 595 abdominal lymph nodes in 86 cases. Secondly, we utilize the negative lymph nodes candidate sets detected by two previous CADe algorithms [40] as the false positive samples for mediastinal and abdominal lymph nodes in this paper. Through the two detection algorithms, 94% -97% sensitivity at 25-35 FP/vol can be achieved. The whole LN databases are listed in Table 1.

Table 1
Lymph node detection datasets

Dataset Patients LNs TP FP

Mediastinal 90 388 960 3208

Abdominal 86 595 1005 3484

Dataset	Patients	LNs	TP	FP
Mediastinal	90	388	960	3208
Abdominal	86	595	1005	3484

With this CT lymph node dataset, an image classification task is implemented for false positive reduction task, where the true LNs are labeled as positives, the negatives are negatives.

4.2 Experimental setup

According to [40], All false-positive samples (>15 mm away from true LN locations), i.e. 3208 false-positive detections in the mediastinum and 3484 in the abdomen, are utilized as negative LN candidate data for training the model. All patients are randomly divided into five subsets at the patient level to execute a 5-folded cross-validation. We let the ratio of positive and negative training patches to be 1, i.e. a balanced training set is guaranteed. Here, we simply utilize single input channel for gray image patches, which can be readily generalized to multiple. All 32×32 pixels image patches are centered at the LN candidate coordinates. In addition, a training data augmentation method is used. Every training image patch is rotated at there degrees: 90, 180, 270 to corresponding 3 consecutive axial slices (up to 2 mm). Hence, we obtain the final training samples of which number is 12 times larger than the original. The soft-tissue window level is [–100, 200 HU] [40].

In practice, according to the fundamental architecture depicted in Fig. 6, we design five models with different scales, i.e. [8, 16, 16, 32, 32, 64, 128], [16, 16, 32, 32, 64, 128, 256], [32, 32, 64, 64, 128, 256, 512], [32, 64, 64, 128, 256, 256, 512], [32, 64, 128, 256, 512, 512, 1024], where each figure denotes the channel or node number in convolutional or fully connected layers successively. The classification results of each model are illustrated in Section 4.2. Model performance is evaluated by Free-response Receiver Operating Characteristic (FROC) and the area-under-the-FROC-curve (AUC). In model reference phase, similar with [40], we directly average the probability values {P₁ (x) , ⋯ , P₅ (x)} generated from the above five model configurations to gain the integral classification result for one input patch x using the following equation. $p (x | {P_{1} (x), . . ., P_{5} (x)}) = \frac{1}{5} \sum_{i = 1}^{5} P (x)$ (6)

We use Theano and MATLAB to deploy our model and execute the sparse coding algorithm respectively. The framework is implemented on Nvidia Tesla K40 m GPU. The model is trained by minimizing the classification cross entropy loss function with Stochastic Gradient Descent. The mini-batch size is 50. Initial learning rate is 0.01, reduced by a factor of 10 after every 25 epochs, and we train the model 200 epochs. Here ReLU is employed as the activation function and the trainable parameters are initialized by msra [18]. And about 8-10 hours are taken to train one network.

4.3 Hyper-parameter selection and model analysis

In order to optimally determine the hyper-parameters of the proposed model, the network configuration, [32, 32, 64, 64, 128, 256, 512] is utilized as a baseline model to implement hyper-parameter selection procedure for mediastinal LN detection task. Then successively apply the selected optimal hyper-parameters to the other four networks.

The primary hyper-parameters contain the number of over-complete basis c, the spare rate p, the number of iterations N of the first training phase and the number of source domain datasets etc.

Firstly, the model performance with different values of the spare rate p is presented. To this end, the other hyper-parameters, i.e. c = 18, N = 5000, are fixed and successively set p∈ { 1, 0.7, 0.5, 0.3, 0.15 }. Based on 5-folded cross-validation, the average FROCs of mediastinal lymph nodes classification task are illustrated in Fig. 7.

Fig. 7

The classification FROCs with different values of spare rate p.

In Fig. 7, the behavior of the spare rate p is like the dropout ratio in Dropout technique. When p ≈ 0.5, the variety of random combinations of neurons is maximum and the corresponding model can generalize well to new target domain. Whereas, we argue that the optimal p is also influenced by the number of over-complete basis c. When increase c, the feature sparsity, i.e. the number of zero parameters, will be degraded and lead to more trainable parameters. In the next experiment, however, we will demonstrate that the range of c is suggested to be small. Hence, we could set the p ≈ 0.5 in the following experiments.

In the following, we set different values of c, the number of over-complete basis. The corresponding FROCs are shown in Fig. 8. The four model performances are visually close, which indicates that the sufficient transferred features could be extracted with c = 16 for the target task here and increasing its value will only bring more complexity and computation to our method. This conclusion can also be applied to the number of iterations N for the first training phase of which FROCs are not shown here for brevity. According to the experimental results with different values of N, the optimal performance can be obtained as N approximates to 5000 (i.e. about 2500 times update for each trainable variable). And increasing N will not bring more optimal results to our model. So, we set c = 16, p = 0.5, N = 5000 in the remaining experiments.

Fig. 8

The FROCs with different numbers of over-complete basis c.

Next, to verify the necessity and effectiveness of introducing multisource transfer datasets to our model, we investigate four scenarios: transfer learning based on the aforementioned six source datasets, texture images, natural images (ImageNet), and random transferred features (Gaussian random dictionaries) respectively. The model performances in Fig. 9 demonstrate that the network which is equipped with the transferred knowledge derived from the six source datasets gives the best classification result, as expected. After that is the network transferred from the texture image domain. The remaining scenarios are the two networks with random dictionaries and transferred features pre-trained on ImageNet successively. It is worth mentioning that the network transferred from ImageNet dataset performs the worst. It is contrary to the observation in [27] where the transferred knowledge from the large-but-unrelated dataset provides greater benefits to target task. We argue that one of the most probable reasons is the difference in transfer learning methods. Our model is equipped with the transferred filters layer by layer, which is similar to the approach used in [51]. The deeper layer features pre-trained from the uncorrelated source datasets are more difficult to generalize to the target task than from the source domains with less dataset biases (e.g. texture image dataset here). In practice, this is also verified by the reality that encouraging performances have been obtained by transferring knowledge from the texture image databases for biomedical image processing [10 , 36]. Moreover, compared with transferring from the natural images, the Gaussian random dictionaries show better classification result, possibly because we normalize the dictionaries to [-1, 1]. In general, random convolutional kernel can be viewed as an edge detector in shallow layers. Whereas for deep level filters, we infer that the point-wise convolutional parameters after the randomly initialized dictionaries are much easier to train than the dictionaries transferred from natural image dataset with dataset biases. It seemingly coincides with the negative transfer phenomenon [38] where the model that utilizes transferred knowledge performs worse than the original one. In addition, we list the classification results, (i.e. AUC, accuracy (Acc.), sensitivity (Sen.) and specificity (Spe.) along with the corresponding 95% confidence interval (CI)) of the 5-folded cross-validation with different transfer source datasets in Table 2. The AUC p-value of Student’s t test between the six source domains and texture images is 0.046, which is lower than the significance level α = 0.05. It further demonstrates that the new introduced six source domain transfer knowledge can indeed improve the model classification performance.

Table 2

The five-fold classification results of the proposed model with different transfer source datasets

Six source domains	AUC	Acc.	Sen.	CI (95%)	Spe.	CI (95%)
1	0.907	0.82	0.82	0.67-0.91	0.80	0.66-0.90
2	0.911	0.83	0.72	0.57-0.83	0.94	0.83-0.98
3	0.908	0.82	0.93	0.82-0.97	0.71	0.56-0.82
4	0.913	0.83	0.88	0.75-0.95	0.79	0.65-0.88
5	0.912	0.83	0.76	0.61-0.86	0.88	0.75-0.95
Texture images
1	0.904	0.81	0.81	0.67-0.90	0.82	0.69-0.91
2	0.907	0.82	0.82	0.67-0.91	0.80	0.66-0.89
3	0.906	0.82	0.87	0.75-0.95	0.76	0.62-0.87
4	0.909	0.83	0.81	0.67-0.90	0.84	0.72-0.93
5	0.908	0.82	0.93	0.82-0.98	0.71	0.56-0.82
Random bases
1	0.889	0.80	0.89	0.77-0.95	0.69	0.54-0.81
2	0.892	0.81	0.87	0.75-0.94	0.78	0.64-0.88
3	0.890	0.80	0.86	0.72-0.93	0.80	0.66-0.89
4	0.893	0.81	0.82	0.68-0.90	0.80	0.66-0.89
5	0.893	0.81	0.84	0.70-0.92	0.78	0.64-0.88
Natural images
1	0.880	0.78	0.67	0.52-0.79	0.88	0.75-0.95
2	0.881	0.78	0.73	0.58-0.84	0.80	0.66-0.89
3	0.881	0.78	0.81	0.67-0.90	0.73	0.58-0.84
4	0.883	0.79	0.78	0.63-0.87	0.75	0.60-0.85
5	0.882	0.79	0.73	0.58-0.84	0.81	0.67-0.90

Fig. 9

The FROCs with different source datasets.

In conclusion, through above selected optimal hyper-parameters, five networks with different configurations are implemented separately for false positive LN reduction task. The corresponding FROCs and AUCs of mediastinal and abdominal lymph node classification are illustrated in Fig. 10(a) and (b) respectively. Note that due to the 5-folded cross-validation strategy at patient level, the range of false positives per patient in the average FROC curves for mediastinal and abdominal lymph nodes is given in 0–30. In Fig. 10(a) and (b), the sensitivities are about 75% /85% at 3 FP/vol. and 86% /91% at 6 FP/vol. in abdomen and mediastinum respectively, which compare competitively to previous approaches. The detailed comparison is presented in the next section.

Fig. 10

The FROCs of the five models with different configurations, (a) mediastinal lymph node classification (b) abdominal lymph node classification.

In the model inference phase, we present several abdominal LN patches in the test set in Fig. 11, which are correctly classified by our model with averaged confidence scores p > 0.95 derived from Equation (6). The top row corresponds to true positive abdominal LNs, and the bottom cases are the true negatives. Intuitively, our model can effectively recognize the prominent samples with higher confidence scores.

Fig. 11

The correctly classified abdominal lymph nodes with higher confidence. Top row: positives; bottom row: negatives.

We illustrate four subsets of trained convolutional 3×3 filters randomly extracted from the second Conv layer of model 3 in Fig. 12. The top two rows are trained for mediastinal lymph node classification and the bottom two are for abdomen lymph node classification. Visually, the various appearances indicate the generalization ability of the proposed transfer learning mechanism for different target datasets.

Fig. 12

Four trained filter subsets randomly extracted from the second convolutional layer of model 3.

4.4 Comparison with previous methods

In this section, relied on the competitive classification results depicted in Section 4.3, we compare the proposed approach with the others. The classification results are straightly cited from corresponding work. The results comparison is given in Table 3 where AUC and TPR/3FP are utilized as performance and the figures denoted in bold correspond the best results. The networks, CifarNet, AlexNet-*, and GoogLeNet-* are borrowed from [44], and the “*-H” denotes high resolution input with 256×256 pixels, “-L” represents low resolution input (64×64), “*-RI” is training from scratch, “*-TL” is transfer learning from ImageNet.

Table 3
Comparison of our method with other previous work on lymph node detection.

Region Mediastinum Abdomen

Method AUC TPR/3FP AUC TPR/3FP

[17] – 0.53 – –

[42] –0.63 – 0.70

[41] –0.78 – 0.78

[39] 0.92 0.70 0.94 0.83

[40] 0.92 0.70 0.94 0.83

CifarNet 0.91 0.70 0.81 0.44

AlexNet-ImNet 0.89 0.63 0.80 0.41

AlexNet-RI-H 0.94 0.79 0.92 0.67

AlexNet-TL-H 0.94 0.81 0.92 0.69

GoogLeNet-RI-H 0.85 0.61 0.80 0.48

GoogLeNet-TL-H 0.94 0.81 0.92 0.70

AlexNet-RI-L 0.94 0.77 0.88 0.61

GoogLeNet-RI-L 0.95 0.85 0.91 0.69

Ours 0.95 0.85 0.92 0.75

Region	Mediastinum	Abdomen
[17]	–	0.53	–	–
[42]	–0.63	–	0.70
[41]	–0.78	–	0.78
[39]	0.92	0.70	0.94	0.83
[40]	0.92	0.70	0.94	0.83
CifarNet	0.91	0.70	0.81	0.44
AlexNet-ImNet	0.89	0.63	0.80	0.41
AlexNet-RI-H	0.94	0.79	0.92	0.67
AlexNet-TL-H	0.94	0.81	0.92	0.69
GoogLeNet-RI-H	0.85	0.61	0.80	0.48
GoogLeNet-TL-H	0.94	0.81	0.92	0.70
AlexNet-RI-L	0.94	0.77	0.88	0.61
GoogLeNet-RI-L	0.95	0.85	0.91	0.69
Ours	0.95	0.85	0.92	0.75

Due to different methods have different training and dataset preparation processes respectively, it is hard to simply compare our model performance with others. We instead discuss the mechanisms that our model is different with the others. Firstly, the approaches in [17 , 42] are part of traditional machine learning methods where only low-level features are employed. Secondly, among the deep learning based approaches, the CifarNet is underperforming, because of the large model scale with low image sizes. And the GoogLeNet-RI-H is not immune from over-fitting phenomenon. In other words, lymph node training images are not sufficient to train the complex network with 22 layers. It is worth mentioning that the approaches that are equipped with transfer learning mechanism (denoted as “*-TL”) generally show better results than those training from scratch (denoted as “*-RI”).

Compared with random initialization, transfer learning indeed provides a more beneficial proposal for initializing the CNN parameters. Meanwhile, relied on the new introduced multisource transfer learning scheme, encouraging performances can also be acquired by our method on mediastinum lymph nodes false-positive reduction problem. For abdomen lymph nodes, a novel 2.5D method with a new data augmentation strategy is presented in [40], which gives the best classification results for abdomen lymph nodes. Literature [44] points out that the ideal classification results shown in [39] can be attributed to a specific dataset preparation protocol. That is a more enterprising extraction of random views in a large range of fields-of-view to obtain additional spatial contextual information. Admittedly, the strategy depicted in [39, 40] paves one of the future ways to improve the proposed framework performance.

5 Conclusions

In this paper, we introduce a new multisource transfer learning based framework for CT lymph node false positive reduction via sparse coding algorithm. Deep transfer learning with CNN framework reveals enough practicability for the target task in here. More importantly, the 1×1 convolutional layer can be utilized not only to combine features from different channels as many former literatures verified, but also to successfully transfer knowledge from multisource domains. In the future, we continue to further investigate the intrinsic characteristics of transferred features to improve the generalizability and efficiency of the proposed model for more medical image processing problems.

Footnotes

Acknowledgments

This work is supported by the National Natural Science Foundation of China under Grant No. 61976126 and the Natural Science Foundation of Shandong Province under Grant No. ZR2019MF003.

References

Akilan

, Wu

Q.M.J.

, Yang

and Safaei

, Fusion of transfer learning features and its application in image classification, IEEE 30th Canadian Conference on Electrical and Computer Engineering (2017), 1–5.

Anthimopoulos

, Christodoulidis

, Ebner

, Christe

and Mougiakakou

, Lung pattern classification for interstitial lung diseases using a deep convolutional neural network, IEEE Transactions on Medical Imaging 35(5) (2016), 1207–1216.

Bagherinezhad

, Rastegari

and Farhadi

, LCNN: Lookup-based convolutional neural network, IEEE Conference on Computer Vision and Pattern Recognition (2017), 7120–7129.

Bertinetto

, Henriques

J.F.

, Valmadre

, Torr

P.H.S.

and Vedaldi

, Learning feed-forward one-shot learners, Advances in Neural Information Processing Systems (2016), 523–531.

Bhuvaneswari

and Ganesh Vaidyanathan

, Classification and grading of diabetic retinopathy images using mixture of ensemble classifiers, Journal of Intelligent & Fuzzy Systems (2021), 1–13, doi: 10.3233/JIFS-211364.

Burghouts

G.J.

and Geusebroek

J.M.

, Material-specific adaptation of color invariant features, Pattern Recognition Letters 30(3) (2009), 306–313.

Caruana

, Multitask learning, Machine Learning 28(1) (1997), 41–75.

Chatfield

, Simonyan

, Vedaldi

and Zisserman

, Return of the devil in the details: Delving deep into convolutional nets (2014), arXiv preprint arXiv:1405.353.

Chen

C.M.

, Huang

, Wang

K.H.

, Kumari

and Wu

M.E.

, A secure authenticated and key exchange scheme for fog computing, Enterprise Information Systems (2020), 1–16, doi: 10.1080/17517575.2020.1712746.

10.

Christodoulidis

, Anthimopoulos

, Ebner

, Christe

and Mougiakakou

, Multisource transfer learning with convolu-tional neural networks for lung pattern analysis, IEEE Journal of Biomedical and Health Informatics 21(1) (2016), 76–84.

11.

Cimpoi

, Maji

, Kokkinos

, Mohamed

and Vedaldi

, Describing textures in the wild, IEEE Conference on Computer Vision and Pattern Recognition (2014), 3606–3613.

12.

Colbaugh

, Glass

and Gallegos

, Ensemble transfer learning for Alzheimer’s disease diagnosis, The 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2017), 3102–3105.

13.

Culurciello

, Jin

, Dundar

and Bates

, An analysis of the connections between layers of deep neural networks (2013), arXiv preprint arXiv:1306.0152.

14.

Dana

K.J.

, Ginneken

B.V.

, Nayar

S.K.

and Koenderink

J.J.

, Reflectance and texture of real-world surfaces, Acm Transactions on Graphics 18(1) (1999), 1–34.

15.

Denil

, Shakibi

, Dinh

, Ranzato

and Freitas

N.D.

, Predicting parameters in deep learning, Advances in Neural Information Processing Systems (2013), 2148–2156.

16.

Elmahdy

M.S.

, Abdeldayem

S.S.

and Yassine

I.A.

, Low quality dermal image classification using transfer learning, IEEE EMBS International Conference on Biomedical & Health Informatics (2017), 373–376.

17.

Feulner

, Zhou

S.K.

, Hammon

, Hornegger

and Comaniciu

, Lymph node detection and segmentation in chest CT data using discriminative learning and a spatial prior, Medical Image Analysis 17(2) (2013), 254–270.

18.

, Zhang

, Ren

and Sun

, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, IEEE International Conference on Computer Vision (2015), 1026–1034.

19.

Hinton

G.E.

, Srivastava

, Krizhevsky

, Sutskever

and Salakhutdinov

R.R.

, Improving neural networks by preventing co-adaptation of feature detectors, Computer Science 3(4) (2012), 212–223.

20.

Hoffman

, Tzeng

, Donahue

, Jia

, Saenko

and Darrell

, One-shot adaptation of supervised deep convolutional models, arXiv preprint arXiv:1312.6204 (2013).

21.

Jia

, Wei

, Socher

, Li

L.J.

, Kai

and Li

F.F.

, Imagenet: A large-scale hierarchical image database, In 2009 IEEE Conference on Computer Vision and Pattern Recognition IEEE (2009), 248–255.

22.

, Zhang

, Liu

and Xie

, Exploring transfer learning for gastrointestinal bleeding detection on small-size imbalanced endoscopy images, 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2017), 1994–1997.

23.

Lin

, Chen

and Yan

, Network in network (2013), arXiv preprint arXiv:1312.4400.

24.

Liu

, Min

, Foroosh

, Tappen

and Penksy

, Sparse convolutional neural networks, IEEE Conference on Computer Vision and Pattern Recognition (2015), 806–814.

25.

Liu

, Xie

, Zhang

, Yu

, Niu

and Sun

, Feature selection and thyroid nodule classification using transfer learning, In 2017 IEEE 14th International Symposium on Biomedical Imaging (2017), 1096–1099.

26.

Long

, Cao

, Wang

and Jordan

M.I.

, Learning transferable features with deep adaptation networks, arXiv preprint arXiv:1502.02791 (2015).

27.

Menegola

, Fornaciali

, Pires

, Bittencourt

F.V.

, Avila

and Valle

, Knowledge transfer for melanoma screening with deep learning, In 2017 IEEE 14th International Symposium on Biomedical Imaging (2017), 297–300.

28.

Meng

, Zhang

, Cao

, Zhang

and Hu

, Liver fibrosis classification based on transfer learning and FCNet for ultrasound images, IEEE Access 5 (2017), 5804–5810.

29.

and Singla

, Discrete-MultiResUNet: Segmentation and feature extraction model for knee MR images, Journal of Intelligent & Fuzzy Systems (2021), 1–11, doi: 10.3233/jifs-211459.

30.

Pan

S.J.

and Yang

, A survey on transfer learning, IEEE Transactions on Knowledge and Data Engineering 22(10) (2010), 1345–1359.

31.

Peng

and Ma

, A level set method to image segmentation based on local direction gradient, KSII Transactions on Internet and Information Systems 12(4) (2018), 1760–1778.

32.

Phan

H.T.H.

, Kumar

, Kim

and Feng

, Transfer learning of a convolutional neural network for HEp-2 cell image classification, IEEE 13th International Symposium on Biomedical Imaging (2016), 1208–1211.

33.

Prajapati

S.A.

, Nagaraj

and Mitra

, Classification of dental diseases using CNN and transfer learning, 5th International Symposium on Computational and Business Intelligence (2017), 70–74.

34.

Razavian

A.S.

, Azizpour

, Sullivan

and Carlsson

, CNN features off-the-shelf: an astounding baseline for recognition, IEEE Conference on Computer Vision and Pattern Recognition Workshops (2014), 806–813.

35.

Ribeiro

, Häfner

, Wimmer

, Tamaki

and Uhl

, Exploring texture transfer learning for colonic polyp classification via convolutional neural networks, IEEE 14th International Symposium on Biomedical Imaging (2017), 1044–1048.

36.

Ribeiro

and Uhl

, Exploring texture transfer learning via convolutional neural networks for iris super resolution, International Conference of the Biometrics Special Interest Group 1–5 (2017), doi: 10.23919/BIOSIG.2017.8053505.

37.

Rigamonti

, Sironi

, Lepetit

and Fua

, Learning separable filters, IEEE Conference on Computer Vision and Pattern Recognition (2013), 2754–2761.

38.

Rosenstein

M.T.

, Marx

, Kaelbling

L.P.

and Dietterich

T.G.

, To transfer or not to transfer, NIPS 2005 Workshop on Transfer Learning 898 (2005), 1–4.

39.

Roth

H.R.

, Lu

, Liu

, Yao

, Seff

, Cherry

, Kim

and Summers

R.M.

, Improving computer-aided detection using convolutional neural networks and random view aggregation, IEEE Transactions on Medical Imaging 35(5) (2015), 1170–1181.

40.

Roth

H.R.

, Lu

, Seff

, Cherry

K.M.

, Hoffman

, Wang

, Liu

, Turkbey

and Summers

R.M.

, A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations, International Conference on Medical Image Computing and Computer-Assisted Intervention (2014), 520–527.

41.

Seff

, Lu

, Barbu

, Roth

and Summers

R.M.

, Leveraging mid-level semantic boundary cues for automated lymph node detection, International Conference on Medical Image Computing and Computer-Assisted Intervention (2015), 53–61.

42.

Seff

, Lu

, Cherry

K.M.

, Roth

H.R.

and Summers

R.M.

, 2D view aggregation for lymph node detection using a shallow hi-erarchy of linear classifiers, International Conference on Medical Image Computing and Computer- Assisted Intervention (2014), 544–552.

43.

Sharan

, Rosenholtz

and Adelson

, Material perception: What can you see in a brief glance? Journal of Vision 9(8) (2009), 784–784.

44.

Shin

H.C.

, Roth

H.R.

, Gao

, Le

, Xu

, Nogues

, Yao

, Mollura

and Summers

R.M.

, Deep convolutional neural networks forcomputer-aided detection: CNN architectures, dataset characteristicsand transfer learning, IEEE Transactions on Medical Imaging 35(5) (2016), 1285–1298.

45.

Tzeng

, Hoffman

, Darrell

and Saenko

, Simultaneous deep transfer across domains and tasks, IEEE International Conference on Computer Vision (2015), 4068–4076.

46.

Wan

, Zeiler

, Zhang

, Cun

Y.L.

and Fergus

, Regularization of neural networks using DropConnect, International Conference on Machine Learning (2013), 1058–1066.

47.

Wang

, Peng

, Wang

and Wang

, Skin lesion image segmentation based on adversarial networks, KSII Transactions on Internet and Information Systems 12(6) (2018), 2826–2840.

48.

Wen

, Wu

, Wang

, Chen

and Li

, Learning structured sparsity in deep neural networks, Advances in Neural Information Processing System (2016), 2074–2082.

49.

J.M.-T.

, Lin

J.C.-W.

and Tamrakar

, High-utility itemset mining with effective pruning strategies, , Article No, ACM Transactions on Knowledge Discovery from Data 13(6) (2019), Article No. 58.

50.

T.-Y.

, Lee

, Obaidat

M.S.

, Kumari

and Chen

C.-M.

, An authenticated key exchange protocol for multi-server architecture in 5G networks, IEEE Access 8 (2020), 28096–28018.

51.

Yosinski

, Clune

, Bengio

and Lipson

, How transferable are features in deep neural networks?, Advances in Neural Information Processing Systems (2014), 3320–3328.

52.

T.-Y.

, Lee

Y.-Q.

, Chen

C.-M.

, Tian

and Al-Nabhan

N.A.

, An enhanced pairing-based authentication scheme for smart grid communications, Journal of Ambient Intelligence and Humanized Computing (2021), https://doi.org/10.1007/s12652-020-02740-2.

53.

T.-Y.

, Yang

, Lee

, Chen

C.-M.

, Pan

J.-S.

and Lslam

S.K.H.

, Improved ECC-based three-factor multiserver authentication scheme, , Security and Communication Networks 2021 (2021), 6627956.

Transfer learning model for false positive reduction in lymph node detection via sparse coding and deep learning

Abstract

Keywords

1 Introduction

2 Related work

2.1 Convolutional neural network and transfer learning in medical image classification

2.2 Parameter predicting in convolutional neural network

3 Methods

3.1 The point-wise convolution based transfer learning

4.1 The source and target databases

Table 1 Lymph node detection datasets Dataset Patients LNs TP FP Mediastinal 90 388 960 3208 Abdominal 86 595 1005 3484

Footnotes

Acknowledgments

References

Table 1
Lymph node detection datasets

Dataset Patients LNs TP FP

Mediastinal 90 388 960 3208

Abdominal 86 595 1005 3484