Predicting dose-volume histogram of organ-at-risk using spatial geometric-encoding network for esophageal treatment planning

Abstract

Dose-volume histogram (DVH) is an important tool to evaluate the radiation treatment plan quality, which could be predicted based on the distance-volume spatial relationship between planning target volumes (PTV) and organs-at-risks (OARs). However, the prediction accuracy is still limited due to the complicated calculation process and the omission of detailed spatial geometric features. In this paper, we propose a spatial geometric-encoding network (SGEN) to incorporate 3D spatial information with an efficient 2D convolutional neural networks (CNN) for accurate prediction of DVH for esophageal radiation treatments. 3D computed tomography (CT) scans, 3D PTV scans and 3D distance images are used as the multi-view input of the proposed model. The dilation convolution based Multi-scale concurrent Spatial and Channel Squeeze & Excitation (msc-SE) structure in the proposed model not only can maintain comprehensive spatial information with less computation cost, but also can extract the features of organs at different scales effectively. Five-fold cross-validation on 200 intensity-modulated radiation therapy (IMRT) esophageal radiation treatment plans were used in this paper. The mean absolute error (MAE) of DVH focusing on the left lung can achieve $2.73 \pm 2.36$ , while the MAE was $7.73 \pm 3.81$ using traditional machine learning prediction model. In addition, extensive ablation studies have been conducted and the quantitative results demonstrate the effectiveness of different components in the proposed method.

Keywords

Dose volume histogram deep learning convolutional neural network feature learning esophageal treatment planning

1. Introduction

High quality treatment plans are crucial for esophageal radiation therapy which aims to maximize radiation dosage at the PTV while minimizing the dosage at OARs [3,5,25]. DVH is a histogram relating radiation dose to tissue volume in radiation therapy planning [4], which is the most commonly used plan evaluation tool to compare doses from different treatment plans. Since planners need to rely on their previous experience, which makes the planning process subjective, iterative, and susceptible to variation between different planners. Apparently, accurate and automatic DVH prediction will contribute to obtain the high quality treatment plans efficiently.

In the past decades, DVH prediction has attracted the considerable attention of the researchers from both academia and industry. There is a complex spatial geometry between PTV and OARs. The challenge of DVH prediction is how to model these spatial geometry and handle the complex relationship between geometric and dosimetric information [17].

A common approach in DVH prediction to represent the geometric information between PTV and OARs is using manually defined feature descriptor. The most popular descriptors are distance-to-target histogram (DTH) [33,36] and overlap volume histogram (OVH) [30,31]. The DTH encodes the spatial relationship between the OARs and the PTV. DTH is equivalent to OVH when the Euclidean form of the distance function is used. Though DTH/OVH based methods have made certain progress, one concern regarding the DTH and OVH is that they pay less attention to the geometry shape of the organs and PTV. These geometry shapes were proved to be one of the important factors affecting the dose coverage [34]. However, it is very difficult to represent geometric features accurately by manual definition.

Recently, several studies explored dosimetric features as a new avenue for DVH prediction. Ma et al. [23,24] used the original dosimetric parameters as dosimetric representation and incorporated it with geometric feature. Cao et al. [1] proposed individual dose-volume histograms (IDVH) to represent dosimetric information, which is extracted from individual conformal beams in different directions. Similar to geometric representation, designing an effective dosimetric descriptor manually is also difficult.

On the basis of feature representation, many machine learning based algorithms are utilized to learn the mapping from the feature to DVH. E.g., the correlation between DTH and DVH had been previously modeled by principal component analysis (PCA) and support vector regression (SVR) [36]. Zhang et al. [35] proposed an ensemble method that combines the strengths of various linear regression models, including stepwise, lasso, elastic net, and ridge regression. Fan et al. [6] took usage of kernel density estimation algorithm. Inspired by the great success of deep neural networks in representation learning [8,9], there are a few deep learning-based approach have been proposed for DVH prediction. For instance, Jiao et al. [16] utilized generalized regression neural network [28], Cao et al. [1] adopted a gated recurrent unit-based recurrent neural network, and Ma et al. [24] used deep convolutional neural network. Although these deep learning based methods have achieved good results, the deep model is only used to complete the mapping from pre-defined features to DVH.

Moreover, there are several studies effort to compute 2D DVH by predicting the 3D radiotherapy dose distributions. Specifically, the inputs of the model are contoured CT images while the outputs are 3D radiotherapy dose distributions, then the 2D DVH is computed based on the mapping relationship between 2D and 3D dose distributions. E.g., Gronberg et al. [11] presented a 3D densely connected U-Net with dilated convolutions to model the input contoured CT images. Kearney et al. [18] introduced a generative adversarial networks (GANs) to model multimodal inputs (CT, PTV, and OARs). Although this kind of method is effective, 3D dose distributions annotation is very difficult. To the best of our knowledge, there are no deep learning methods to learn the optimal feature representation for 2D DVH prediction from the original image data directly.

To deal with the above issues, we propose a novel Spatial Geometric-Encoding Network (SGEN), which aims to incorporate 3D spatial information with 2D CNN, and predict DVH for esophageal radiation treatments effectively and accurately in a unified framework. To achieve this goal, three different types of image data are used as inputs to the model, i.e., 3D CT scans, 3D PTV scans and 3D distance images. The above three kinds of image data can complement each other in terms of describing the geometric relationship between PTV and OARs. Meanwhile, to avoid the high computational cost by using 3D CNNs with 3D inputs, the dilation convolution based multi-scale concurrent spatial and channel squeeze & excitation (msc-SE) structure is integrated into our network structure, which can speed up the training while enhancing the ability of multi-scale spatial geometric information representation. In addition, the proposed method has been compared with conventional DVH prediction methods in terms of MAE of DVH for different OARs based on 200 IMRT esophageal treatment plans.

The main contribution of this work can be summarized as follows:

A spatial geometric-encoding network based on CNN is proposed to predict DVH of OARs for esophageal radiation treatments. It can effectively learn the mapping from the multi-view data to DVH in an end-to-end manner.

The dilation convolution based multi-scale concurrent spatial and channel squeeze & excitation (msc-SE) structure is proposed which not only can maintain comprehensive spatial information with less computation cost, but also can extract the features of organs at different scales effectively.

Extensive experiments on esophageal cancer benchmark dataset have been conducted. The results demonstrate the effectiveness of the proposed method.

We note that Liu et al. [22] also applied the CNN for DVH prediction via handle the contours of PTV and OARs directly in an end-to-end way. The main differences between our method and [22] are three-fold. First, the task is different. [22] focused on nasopharyngeal cancer while our method is applied to esophageal cancer. Second, the input is different. We consider three types of images (3D CT scans, 3D PTV scans and 3D distance images) as input data simultaneously, while [22] only model the organ contours image. Finally, the structure of the CNN is different. [22] proposed a general connected residual deconvolution network while the dilation convolution based multi-scale concurrent spatial and channel squeeze & excitation (msc-SE) structure is adopted in our backbone, which makes the proposed network has better ability of extracting spatial geometric features.

The remainder of this paper is organized as follows. We present the details of the proposed methods in Section 2. Section 3 provides the experimental results and analysis. Finally, we conclude the paper in Section 4.

2. The proposed method

In this section, we first introduce the input and output of the proposed method. Then, we present model details. At last, we provide more implementation details of the proposed method.

Fig. 1.

Examples of input data: (a) computed tomography, (b) binarized PTV scan, (c) heat map of distance image.

2.1. Input and output data

To provide the detailed spatial distance features of each OARs to PTV, we calculated distance images from the binarized PTV scans, which present the shortest distance from each voxel to the boundary of PTV. As shown in Fig. 1, in order to make full use of the complementary characteristics of different types of data, CT, PTV scans and distance images are considered as input of the proposed SGEN. The size of each image is $N \times W \times H$ , where N is the number of slices. The output of the SGEN is the prediction DVH vector.

2.2. Architecture of the model

2.2.1. Overview

The general framework of the proposed SGEN is shown in Fig. 2. In order to obtain more comprehensive geometric feature representation, 3D CT scans, 3D PTV scans and 3D distance images are inputed into 2D CNN based feature extractor. To ensure that the output features of different 2D CNNs are in a common space, for each input, the structures of the 2D CNN are same but the parameters are different. Features of different inputs are concatenated to obtain high-level representation. To avoid overfitting, an autoencoder is introduced to compress the original DVH feature vector. Then, a fully connected layer is utilized to predict the compressed DVH feature vector from the high-level representation, not original DVH feature vector. Finally, a decoder of the pre-trained autoencoder is used to recover the original DVH feature vector from the compressed DVH feature vector. We can see that it includes two sub-network: 2D CNN based feature extractor and autoencoder, details are in below.

Fig. 2.

The general framework of the proposed SGEN method. 3D CT scans, 3D PTV scans and 3D distance images are inputed into 2D CNN based feature extractor. For each input, the structures of the 2D CNN are same but the parameters are different. Features of different inputs are concatenated to obtain high-level representation. Then, a fully connected layer is utilized to predict the compressed DVH feature vector. Finally, a decoder of a pre-trained autoencoder is used to recover the original DVH feature vector from the compressed DVH feature vector.

2.2.2. Feature extractor

The backbone of the proposed 2D CNN based feature extractor is a 4 layer DenseNet [14] like network. As illustrated in Fig. 2, it includes a dense block, a transition block and a msc-SE module for each layer.

Dense block As shown in Fig. 3, each dense block includes two $3 \times 3$ convolutions followed by a bottleneck layer. The parameter of padding is set to 1 in all convolutional layers to ensure that the feature map generated by each convolutional layer in the dense block has the same size. Consequently, the output of $l^{t h}$ layer in dense block is denoted by: $\begin{matrix} (1) & x_{l} = H_{l} ([x_{0}, x_{1}, \dots x_{l - 1}]), l = 1, 2, 3, 4 \end{matrix}$ where $[x_{0}, x_{1}, \dots x_{l - 1}]$ refers to the concatenation of the feature maps produced in $0, \dots, l - 1$ . $H_{l} ()$ refers to a non-linear transformation, which composed of batch normalization (BN) [15], followed by a parametric rectified linear unit (PReLU) [12] and a $3 \times 3$ convolution layer. Besides, the growth rate is set to the default value of 12, which represents the number of feature maps output by each convolution layer in dense block.

Transition block The transition block includes a $1 \times 1$ convolution layer and a $2 \times 2$ average pooling layer. The compression factor θ is set to 0.5 in order to reduce the number of feature-maps at transition block.

Fig. 3.

Overview of a branch of the dense block.

Fig. 4.

Illustration of the proposed msc-SE module. It contains s-SE (spatial squeezing-and-excitation), c-SE (channel squeeze-and-excitation) and dilation convolution based multi-scale feature fusion blocks.

msc-SE module The biggest difference between our feature extractor and original DenseNet is that a msc-SE module is proposed to get more representative feature. In medical image processing with input in 3D format, 3D CNN module is often used to extract image features. However, the training of 3D CNN network is difficult due to the large computation cost required. Fortunately, instead of directly implementing a 3D CNN module, spatial and channel squeeze-and-excitation (sc-SE) module [27] is developed which is able to acquire intrinsic spatial geometric information with less computation cost. Moreover, different organs have different scales, an excellent feature representation should be scale-invariant. To fuse both spatial-wise and channel-wise information within local receptive fields, and handle different organ scales, msc-SE (dilation convolution based multi-scale concurrent spatial and channel squeeze & excitation) module was proposed and applied to each layer. As shown in Fig. 4, the proposed msc-SE has three components: s-SE, c-SE and multi-scale fusion blocks.

Spatial squeezing-and-excitation (s-SE) block was originally proposed for 2D image segmentation task [7]. s-SE block focus on concentrate on the important spatial locations and ignore the irrelevant ones. As illustrated in Fig. 4, in s-SE block, a series of feature maps generated after dense and transition blocks can be converted to $U = [u_{1}, u_{2}, \dots u_{N}]$ , where $u_{k} \in H \times W$ represents ith channels with height H and width W. Thus, the size of U is $N \times H \times W$ . It is fed into a $1 \times 1$ convolutional layer with an activation operation (Relu) to get a spatial attention map s with size of $H \times W$ . Then, we again apply a element-Wise matrix multiplication between all feature channels in U and s. Finally, the output of s-SE is defined as below: $\begin{matrix} (2) & U_{s - SE} = [u_{1} \cdot s, u_{2} \cdot s, \dots u_{N} \cdot s] \end{matrix}$

Channel squeezing-and-excitation (c-SE) block was originally proposed for 2D image classification task [13]. c-SE block focus on class inter-dependencies via emphasizing important channels while neglecting less important ones. For input feature $U = [u_{1}, u_{2}, \dots u_{N}] \in N \times H \times W$ , c-SE is performed by a global average pooling layer, producing vector $z \in R^{1 \times 1 \times N}$ with its kth element: $\begin{matrix} (3) & z_{k} = \frac{1}{H \times W} \sum_{i}^{H} \sum_{j}^{W} u_{k} (i, j) \end{matrix}$ the vector $Z = [z_{1}, z_{2}, \dots z_{N}]$ represents the global spatial information of the feature maps, then the vector $\hat{Z} = {\hat{z}}_{1}, {\hat{z}}_{2}, \dots {\hat{z}}_{N}$ is obtained through a fully connected layer, an activation layer (Relu) and another fully connected layer. The output of the c-SE block is expressed as: $\begin{matrix} (4) & U_{c - SE} = [σ ({\hat{z}}_{1}) u_{1}, σ ({\hat{z}}_{2}) u_{2}, \dots σ ({\hat{z}}_{N}) u_{N}] \end{matrix}$ where $σ ()$ represents the output of the sigmoid layer and it indicates the importance of the kth channel. As the training process iterates, the network gradually learns important channels and ignores less important ones.

The $U_{sc - SE}$ is a combination of the above two SE blocks, which produces the maximum output of the two blocks. This recalibration encourages the network to learn more meaningful feature maps, that are relevant both spatially and channel-wise.

On the basis of $U_{sc - SE}$ , we propose a multi-scale fusion block to get scale-invariant feature $U_{msc - SE}$ . Dilated convolution [32] is a common strategy to enlarge the features’ receptive field in object detection. Inspired by it but different from other multi-branch strategies [20,21], we stack two successive dilated residual blocks with different dilation rates in the $3 \times 3$ convolution layers to generate output features with multiple receptive fields, covering all organs’ scales.

2.2.3. Autoencoder

DVH is a two-dimensional plot which computes the volume fraction of an OAR in different distances from PTVs. In our work, 100 points is sampled evenly from each cumulative DVH and each point includes a volume fraction value and a dose value. a 100-dimensional volume feature vector $V_{i} = [v_{1}, v_{2}, \dots, v_{100}]$ and a 100-dimensional dose vector $D_{i} = [d_{1}, d_{2}, \dots, d_{100}]$ is generated by the 100 points. Where i represents different OARs. However, 100-dimensional dose feature vector costs more computational resource and the loss function of label value with high dimension could hinder the training process of the network. Therefore, feature dimension reduction is applied here to DVH feature vectors.

Autoencoder performs better for nonlinear dimensionality reduction [29]. The autoencoder in our work is shown in Fig. 5, the encoder and decoder are structurally symmetric, it includes an encoder composed of seven encoding layers and a decoder composed of seven decoding layers. By pre-training each layer, autoencoder can finally reduce a 100-dimensional DVH feature vector into 5-dimensional one (experimentally determined).

Fig. 5.

Overview of the proposed autoencoder structure. The blue box represents the 100-dimensional feature vector as the original input, the coding layer is represented by a gray box, and the decoding layer is represented by a black box. The number below the box represents the dimension of the feature vector in each layer, and the orange box represents a 100-dimensional DVH feature vector reconstructed by the decoder.

2.3. Implementation details

Fig. 6.

Flowchart of the implementation.

The flowchart of the implementation is shown in Fig. 6. In training stage, we first utilize DVH vectors to train the autoencoder, which aims to obtain DVH encoder and decoder. Then, the encoder is used to compute the compressed DVH vector corresponding to the original DVH vectors. Finally, 3D CT scans, 3D PTV scans and 3D distance images are deemed as data, their corresponding compressed DVH vectors are deemed as label to train the DVH prediction model. The whole SGEN model includes both DVH autoencoder and prediction models. In test stage, 3D CT scans, 3D PTV scans and 3D distance images are fed into the pre-trained DVH prediction model firstly to predict the compressed DVH vector. Then, this compressed DVH vector is fed into the pre-trained DVH decode model to obtain the final predicted DVH vector. The MAE is calculated based on the predicted DVH vector and their corresponding groundtruth DVH vector.

3. Experiments

To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on our construct dataset. In the experiments, we firstly compare the proposed SGEN method with three typical baselines to evaluate its peroformance. Then, we provide further analysis of the SGEN method. It includes the convergency investigation and the impact of different components in our framework. It is worth noting that, the purpose of this section is to demonstrate the effectiveness of the proposed SGEN rather than achieve state-of-the-art results by all means.

3.1. Dataset and settings

200 esophageal IMRT treatment plans were collected and used for the evaluation. All IMRT plans were clinically approved for treatment of esophageal radiotherapy. All plans used the same setting (6 MV photon beam with gantry angles $135^{\circ}$ , $165^{\circ}$ , $195^{\circ}$ , $0^{\circ}$ , $30^{\circ}$ , $225^{\circ}$ , and $330^{\circ}$ ). 160 of them are used for training and the rest is for testing.

The entire network is trained on a Nvidia V100 GPU in PyTorch. The initial learning rate is set to 0.001 and Adam [19] is utilized as optimizer. Training is deemed as finish when the error of training set does not decrease for 10 epochs.

3.2. Evaluation metric

According to the ICRU 83 report [10], these indexes include the average dose, fractional lung volumes (V5, V10, V20, V30), fractional heart volumes (V30, V35), maximal dose of spinal cord (D2), etc. The five-fold cross-validation method was used for performance evaluation (200 data were divided into 5 groups. In each experiment, one group was taken as test data, while the other four groups were deemed as training data. A total of 5 experiments were carried out. The mean value and standard deviation were calculated as the final experimental results). The MAE of the endpoint sampled from DVHs was used to evaluate the accuracy of the prediction models: $\begin{matrix} (5) & MAE = \frac{1}{40} \sum_{i = 1}^{40} | V_{i, g t} - V_{i, pred} | \end{matrix}$ where $V_{i, g t}$ is the groundtruth value of DVH, $V_{i, pred}$ is the DVH predicted by our method.

3.3. Comparison with baselines

To verify the effectiveness of our proposed method, we compare the proposed method with three typical baselines in the experiments. The first one is traditional machine learning based method PCA+SVR [36], The second one is deep learning based method 2D-Unet [26], that is 2D-Unet is utilized as backbone in our SGEN. The last one is 3D-Unet [2]. The performance comparisons are reported in Table 1 and Fig. 7, the parameters of three backbone networks used in our DVH prediction framework are reported in Table 2. From Fig. 7, Table 1 and Table 2 we can conclude that (1) deep learning based method is far better than traditional method, (2) the proposed network structure is effective.

Table 1
Performance comparison of the proposed SGEN with baselines and w or w/o msc-SE structure in terms of MAE scores on the constructed benchmark dataset. The highest score is shown in boldface

OARs Methods Dose Endpoint error

V5 V10 V20 V30 V35 D2

Left lung PCA+SVR [36] 7.73 ± 3.81% 7.34 ± 5.27% 6.21 ± 4.69% 7.06 ± 5.91% - -

2D-Unet [26] 4.01 ± 3.43% 4.67 ± 2.71% 6.01 ± 3.19% 6.29 ± 1.90% - -

3D-Unet [2] 3.12 ± 2.13% 2.94 ± 2.63% 3.52 ± 2.61% 3.29 ± 1.90% - -

Ours w/o msc-SE 3.35 ± 2.57% 3.01 ± 2.85% 5.57 ± 3.06% 4.86 ± 2.70% - -

Ours 2.73 $\pm$ 2.36% 2.78 $\pm$ 2.16% 3.41 $\pm$ 2.63% 2.57 $\pm$ 2.81% - -

Right lung PCA+SVR [36] 5.14 ± 5.91% 7.21 ± 6.27% 6.91 ± 5.05% 7.29 ± 4.48% - -

2D-Unet [26] 3.93 ± 5.35% 4.63 ± 4.61% 5.31 ± 3.97% 5.05 ± 3.81% - -

3D-Unet [2] 3.12 ± 1.99% 2.98 $\pm$ 2.36% 3.61 ± 2.54% 3.09 ± 1.69% - -

Ours w/o msc-SE 3.54 ± 2.64% 3.89 ± 3.01% 4.54 ± 2.95% 4.88 ± 2.46% - -

Ours 3.03 $\pm$ 2.27% 3.16 ± 2.12% 2.71 $\pm$ 1.82% 2.82 $\pm$ 2.37% - -

Heart PCA+SVR [36] - - - 8.97 ± 4.87% 9.32 ± 5.71% -

2D-Unet [26] - - - 6.48 ± 4.95% 6.44 ± 5.06% -

3D-Unet [2] - - - 3.16 ± 2.72% 3.75 ± 3.12% -

Ours w/o msc-SE - - - 5.43 ± 4.17% 5.71 ± 3.59% -

Ours - - - 2.43 $\pm$ 2.12% 3.05 $\pm$ 2.96% -

Cord PCA+SVR [36] - - - - - 9.93 ± 7.83%

2D-Unet [26] - - - - - 6.85 ± 5.83%

3D-Unet [2] - - - - - 4.31 ± 3.76%

Ours w/o msc-SE - - - - - 5.94 ± 3.48%

Ours - - - - - 2.92 $\pm$ 3.01%

OARs	Methods	Dose Endpoint error
Left lung	PCA+SVR [36]	7.73 ± 3.81%	7.34 ± 5.27%	6.21 ± 4.69%	7.06 ± 5.91%	-	-
2D-Unet [26]	4.01 ± 3.43%	4.67 ± 2.71%	6.01 ± 3.19%	6.29 ± 1.90%	-	-
3D-Unet [2]	3.12 ± 2.13%	2.94 ± 2.63%	3.52 ± 2.61%	3.29 ± 1.90%	-	-
Ours w/o msc-SE	3.35 ± 2.57%	3.01 ± 2.85%	5.57 ± 3.06%	4.86 ± 2.70%	-	-
Ours	2.73 $\pm$ 2.36%	2.78 $\pm$ 2.16%	3.41 $\pm$ 2.63%	2.57 $\pm$ 2.81%	-	-
Right lung	PCA+SVR [36]	5.14 ± 5.91%	7.21 ± 6.27%	6.91 ± 5.05%	7.29 ± 4.48%	-	-
2D-Unet [26]	3.93 ± 5.35%	4.63 ± 4.61%	5.31 ± 3.97%	5.05 ± 3.81%	-	-
3D-Unet [2]	3.12 ± 1.99%	2.98 $\pm$ 2.36%	3.61 ± 2.54%	3.09 ± 1.69%	-	-
Ours w/o msc-SE	3.54 ± 2.64%	3.89 ± 3.01%	4.54 ± 2.95%	4.88 ± 2.46%	-	-
Ours	3.03 $\pm$ 2.27%	3.16 ± 2.12%	2.71 $\pm$ 1.82%	2.82 $\pm$ 2.37%	-	-
Heart	PCA+SVR [36]	-	-	-	8.97 ± 4.87%	9.32 ± 5.71%	-
2D-Unet [26]	-	-	-	6.48 ± 4.95%	6.44 ± 5.06%	-
3D-Unet [2]	-	-	-	3.16 ± 2.72%	3.75 ± 3.12%	-
Ours w/o msc-SE	-	-	-	5.43 ± 4.17%	5.71 ± 3.59%	-
Ours	-	-	-	2.43 $\pm$ 2.12%	3.05 $\pm$ 2.96%	-
Cord	PCA+SVR [36]	-	-	-	-	-	9.93 ± 7.83%
2D-Unet [26]	-	-	-	-	-	6.85 ± 5.83%
3D-Unet [2]	-	-	-	-	-	4.31 ± 3.76%
Ours w/o msc-SE	-	-	-	-	-	5.94 ± 3.48%
Ours	-	-	-	-	-	2.92 $\pm$ 3.01%

3.4. Convergency

Figure 8 shows the training loss curves of the proposed SGEN versus the different number of training epochs. From the results, we can see that loss value decreases almost monotonously and smoothly. It becomes stable after 1000 epochs, which illustrates that the proposed method can be trained by the general gradient descent method efficiently.

3.5. Impact of different components

3.5.1. Impact of msc-SE structure

Compared with the general DenseNet, the biggest difference of our network is the msc-SE structure. To evaluate its impact, we compare it with the feature extractor without msc-SE module. Results are also reported in Table 1 and Fig. 7. We can observe that the predicted DVH of the network with msc-SE module is closer to the clinically approved DVH. We think reasons are two folds: (1) sc-SE block is able to fuse both spatial-wise and channel-wise information within pixel-wise context. (2) The proposed stacked dilation based multi-scale feature fusion block can handle organs at different scales.

3.5.2. Impact of multi-view inputs

The input of the proposed SGEN combines three terms, which aims to handle complementary information. To investigate the impact of these terms on the performance of the proposed method, we developed and evaluated six variations of SGEN: SGEN without 3D CT scans and 3D distance images (SGEN1), SGEN without 3D PTV scans and 3D distance images (SGEN2), SGEN without 3D CT scans and 3D PTV images (SGEN3), SGEN without 3D distance images (SGEN4), SGEN without 3D CT scans (SGEN5), SGEN without 3D PTV scans (SGEN6). The optimisation procedure of these six cases is similar to the proposed SGEN.

Table 3 shows the performance comparisons of SGEN and its six variations on the constructed benchmark dataset. From the results, we can see that the full SGEN performs best, which indicates that all of the three input terms in our framework contribute to the final DVH prediction accuracy. We can also see that SGEN4, SGEN5, SGEN6 outperform SGEN1, SGEN2 and SGEN3 with a large margin, which demonstrate that information in CT scans, PTV scans and distance images are complementary. Based on the above analysis, we find that modeling 3D CT scans, 3D PTV scans and 3D distance images simultaneously is a valuable strategy for DVH prediction.

Fig. 7.

Comparison of clinical DVHs and predicted DVHs of (a) left lung, (b) right lung, (c) heart, and (d) spine cord by different models. The solid black line represents the DVH calculated from the clinically approved plan while the dashed line in different colors are calculated from different prediction models.

3.6. Limitations of the SGEN

Beam angle information is not modeled in the proposed SGEN. In IMRT, the number of radiation fields and gantry angle also affect the dose distribution of radiotherapy plan. For example, two organs at risk have the same spatial distance from the target area, but they have different dose distributions, the reason is that one organ may be in the beam and the other out of the beam. This means that the beam angle plays an important role in the modeling of dose distance relationship. In the future, we will explore how to use the beam angle information to achieve more accurate DVH prediction.

Table 2
The parameters of three backbone networks used in our DVH prediction framework

Backbone Params (Million)

DenseNet-53 12.4

3D-Unet 29.3

SGEN 9.7

Backbone	Params (Million)
DenseNet-53	12.4
3D-Unet	29.3
SGEN	9.7

Fig. 8.

The training loss curves of SGEN versus the different number of training epochs on the constructed dataset.

Table 3

Performance comparison of the proposed SGEN and its six variations in terms of MAE scores on the constructed benchmark dataset. The highest score is shown in boldface

OARs	Methods	Dose Endpoint error

		V5	V10	V20	V30	V35	D2
Left lung	SGEN1	7.54 ± 7.89%	9.54 ± 8.41%	11.31 ± 6.16%	9.95 ± 6.84%	-	-
	SGEN2	5.41 ± 7.48%	7.64 ± 8.26%	8.81 ± 6.41%	9.53 ± 7.42%	-	-
	SGEN3	5.74 ± 5.17%	6.13 ± 6.74%	5.82 ± 4.56%	8.99 ± 4.47%	-	-
	SGEN4	3.35 ± 2.57%	3.01 ± 2.85%	5.57 ± 3.06%	4.86 ± 2.70%	-	-
	SGEN5	5.17 ± 4.74%	4.53 ± 4.67%	5.73 ± 4.15%	8.79 ± 5.33%	-	-
	SGEN6	2.99 ± 3.64%	3.48 ± 3.78%	5.31 ± 3.63%	3.81 ± 3.32%	-	-
	Full SGEN	2.73 $\pm$ 2.36%	2.78 $\pm$ 2.16%	3.41 $\pm$ 2.63%	2.57 $\pm$ 2.81%	-	-
Right lung	SGEN1	8.31 ± 7.72%	9.32 ± 8.15%	10.41 ± 7.84%	9.43 ± 5.48%	-	-
	SGEN2	6.12 ± 5.45%	7.38 ± 4.16%	7.78 ± 8.43%	6.72 ± 5.23%	-	-
	SGEN3	6.01 ± 5.31%	6.83 ± 4.50%	7.42 ± 6.31%	6.61 ± 5.04%	-	-
	SGEN4	4.86 ± 4.64%	5.43 ± 4.73%	6.43 ± 4.34%	4.75 ± 3.64%	-	-
	SGEN5	5.59 ± 5.02%	6.52 ± 4.38%	7.03 ± 5.73%	5.75 ± 4.88%	-	-
	SGEN6	3.33 ± 3.25%	3.41 ± 3.08%	3.95 ± 2.69%	3.19 ± 3.64%	-	-
	Full SGEN	3.03 $\pm$ 2.27%	3.16 $\pm$ 2.12%	2.71 $\pm$ 1.82%	2.82 $\pm$ 2.37%	-	-
Heart	SGEN1	-	-	-	12.31 ± 7.36%	10.54 ± 7.92%	-
	SGEN2	-	-	-	9.43 ± 6.89%	9.72 ± 8.12%	-
	SGEN3	-	-	-	9.77 ± 6.38%	9.95 ± 7.90%	-
	SGEN4	-	-	-	6.24 ± 4.47%	6.98 ± 5.25%	-
	SGEN5	-	-	-	8.99 ± 5.82%	9.72 ± 8.32%	-
	SGEN6	-	-	-	4.58 ± 3.42%	5.21 ± 3.73%	-
	Full SGEN	-	-	-	2.43 $\pm$ 2.12%	3.05 $\pm$ 2.96%	-
Cord	SGEN1	-	-	-	-	-	12.62 ± 7.72%
	SGEN2	-	-	-	-	-	9.62 ± 5.98%
	SGEN3	-	-	-	-	-	11.47 ± 6.44%
	SGEN4	-	-	-	-	-	6.42 ± 4.86%
	SGEN5	-	-	-	-	-	9.03 ± 6.75%
	SGEN6	-	-	-	-	-	4.06 ± 4.17%
	Full SGEN	-	-	-	-	-	2.92 $\pm$ 3.01%

4. Conclusion

In this paper, we proposed a new approach (SGEN) for DVH prediction of OARs. Our proposed method can handle 3D CT scans, 3D PTV scans and 3D distance images directly and simultaneously. Meanwhile, a dilation convolution based multi-scale concurrent spatial and channel squeeze & excitation (msc-SE) structure and an autoencoder structure are presented, which make our method can extract the spatial geometric features of PTV and OARs quickly and accurately. Extensive experimental results on the constructed esophageal benchmark dataset and the comprehensive analysis have demonstrated the effectiveness of the proposed DVH prediction strategy, and it could lead to superior performance compared with typical baselines.

Footnotes

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (No. 61902104, 11975312), the Anhui Provincial Natural Science Foundation (No. 2008085QF295, 1908085J25, 1808085MF209) and the University Natural Science Research Project of Anhui Province (No. KJ2020A0651).

Conflict of interest

The authors have no conflict of interest to report.

References

Cao ,

Zhuang ,

Chen and

Liu , Application of dose-volume histogram prediction in biologically related models for nasopharyngeal carcinomas treatment planning, Radiation Oncology 15(1) (2020), 1–9. doi:10.1186/s13014-019-1449-z.

Ö.

Çiçek ,

Abdulkadir ,

S.S.

Lienkamp ,

Brox and

Ronneberger , 3D U-Net: Learning dense volumetric segmentation from sparse annotation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 424–432.

I.J.

Das ,

C.-W.

Cheng ,

K.L.

Chopra ,

R.K.

Mitra ,

S.P.

Srivastava and

Glatstein , Intensity-modulated radiation therapy dose prescription, recording, and delivery: Patterns of variability among institutions and treatment planning systems, Journal of the National Cancer Institute 100(5) (2008), 300–307. doi:10.1093/jnci/djn020.

Drzymala ,

Mohan ,

Brewster ,

Chu ,

Goitein ,

Harms and

Urie , Dose-volume histograms, International Journal of Radiation Oncology* Biology* Physics 21(1) (1991), 71–78. doi:10.1016/0360-3016(91)90168-4.

P.C.

Enzinger and

R.J.

Mayer , Esophageal cancer, New England Journal of Medicine 349(23) (2003), 2241–2252. doi:10.1056/NEJMra035010.

Fan ,

Wang ,

Zhang and

Hu , Iterative dataset optimization in automated planning: Implementation for breast and rectal cancer radiotherapy, Medical physics 44(6) (2017), 2515–2531. doi:10.1002/mp.12232.

Fu ,

Liu ,

Tian ,

Li ,

Bao ,

Fang and

Lu , Dual attention network for scene segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3146–3154.

Gao ,

Zhang and

Xu , I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 8303–8311.

Gao ,

Zhang and

Xu , Learning to model relationships for zero-shot video classification, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.

10.

Grégoire and

Mackie , State of the art on dose prescription, reporting and recording in intensity-modulated radiation therapy (ICRU report no. 83), Cancer/Radiothérapie 15(6–7) (2011), 555–559.

11.

M.P.

Gronberg ,

S.S.

Gay ,

T.J.

Netherton ,

D.J.

Rhee ,

L.E.

Court and

C.E.

Cardenas , Technical note: Dose prediction for head and neck radiotherapy using a three-dimensional dense dilated U-net architecture, Medical Physics 6 (2021), 14827.

12.

He ,

Zhang ,

Ren and

Sun , Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1026–1034.

13.

Hu ,

Shen and

Sun , Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.

14.

Huang ,

Liu ,

Van Der Maaten and

K.Q.

Weinberger , Densely connected convolutional networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4700–4708.

15.

Ioffe and

Szegedy , Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International Conference on Machine Learning, PMLR, 2015, pp. 448–456.

16.

S.X.

Jiao ,

M.L.

Wang ,

L.X.

Chen and

X.-W.

Liu , Evaluation of dose-volume histogram prediction for organ-at risk and planning target volume based on machine learning, Scientific reports 11(1) (2021), 1–9. doi:10.1038/s41598-020-79139-8.

17.

I.J.

Kalet and

Paluszynski , Knowledge-based computer systems for radiotherapy planning, American Journal of Clinical Oncology 13(4) (1990), 344–351. doi:10.1097/00000421-199008000-00015.

18.

Kearney ,

J.W.

Chan ,

Wang ,

Perry ,

Descovich ,

Morin ,

S.S.

Yom and

T.D.

Solberg , DoseGAN: A generative adversarial network for synthetic dose prediction using attention-gated discrimination and generation, Scientific reports 10(1) (2020), 1–8.

19.

D.P.

Kingma and

Ba , Adam: A method for stochastic optimization, 2014, arXiv preprint 1412.6980.

20.

Li ,

Chen ,

Wang and

Zhang , Scale-aware trident networks for object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 6054–6063.

21.

Li ,

Zhang and

Chen , Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1091–1100.

22.

Liu ,

Chen ,

Men ,

Yi and

Dai , A deep learning model to predict dose–volume histograms of organs at risk in radiotherapy treatment plans, Medical Physics 47(11) (2020), 5467–5481. doi:10.1002/mp.14394.

23.

Ma ,

Kovalchuk ,

M.K.

Buyyounouski ,

Xing and

Yang , Dosimetric features-driven machine learning model for DVH prediction in VMAT treatment planning, Medical physics 46(2) (2019), 857–867. doi:10.1002/mp.13334.

24.

Ma ,

Kovalchuk ,

M.K.

Buyyounouski ,

Xing and

Yang , Incorporating dosimetric features into the prediction of 3D VMAT dose distributions using deep convolutional neural network, Physics in Medicine & Biology 64(12) (2019), 125017. doi:10.1088/1361-6560/ab2146.

25.

B.E.

Nelms ,

Robinson ,

Markham ,

Velasco ,

Boyd ,

Narayan ,

Wheeler and

M.L.

Sobczak , Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems, Practical radiation oncology 2(4) (2012), 296–305. doi:10.1016/j.prro.2011.11.012.

26.

Ronneberger ,

Fischer and

Brox , U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 234–241.

27.

A.G.

Roy ,

Navab and

Wachinger , Recalibrating fully convolutional networks with spatial and channel “squeeze and excitation” blocks, IEEE transactions on medical imaging 38(2) (2018), 540–549. doi:10.1109/TMI.2018.2867261.

28.

D.F.

Specht

et al., A general regression neural network, IEEE transactions on neural networks 2(6) (1991), 568–576. doi:10.1109/72.97934.

29.

Vincent ,

Larochelle ,

Bengio and

P.-A.

Manzagol , Extracting and composing robust features with denoising autoencoders, in: Proceedings of the 25th International Conference on Machine Learning, 2008, pp. 1096–1103. doi:10.1145/1390156.1390294.

30.

Wu ,

Ricchetti ,

Sanguineti ,

Kazhdan ,

Simari ,

Chuang ,

Taylor ,

Jacques and

McNutt , Patient geometry-driven information retrieval for IMRT treatment plan quality control, Medical physics 36(12) (2009), 5497–5505. doi:10.1118/1.3253464.

31.

Wu ,

Ricchetti ,

Sanguineti ,

Kazhdan ,

Simari ,

Jacques ,

Taylor and

McNutt , Data-driven approach to generating achievable dose-volume histogram objectives in intensity-modulated radiotherapy planning, International Journal of Radiation Oncology* Biology* Physics 79(4) (2011), 1241–1247. doi:10.1016/j.ijrobp.2010.05.026.

32.

Yu and

Koltun , Multi-scale context aggregation by dilated convolutions, 2015, arXiv preprint 1511.07122.

33.

Yuan ,

Ge ,

W.R.

Lee ,

F.F.

Yin ,

J.P.

Kirkpatrick and

Q.J.

Wu , Quantitative analysis of the factors which affect the interpatient organ-at-risk dose sparing variation in IMRT plans, Medical physics 39(11) (2012), 6868–6878. doi:10.1118/1.4757927.

34.

Yuan ,

Ge ,

Li ,

Zhu ,

Yin and

Wu , TH-E-BRB-05: Modeling the correlation between OAR dose sparing and patient’s anatomy in head and neck IMRT, Medical Physics 38(6) (2011), 3867–3867. doi:10.1118/1.3613562.

35.

Zhang ,

Q.J.

Wu ,

Xie ,

Sheng ,

F.-F.

Yin and

Ge , An ensemble approach to knowledge-based intensity-modulated radiation therapy planning, Frontiers in oncology 8 (2018), 57. doi:10.3389/fonc.2018.00057.

36.

Zhu ,

Ge ,

Li ,

Thongphiew ,

F.-F.

Yin and

Q.J.

Wu , A planning quality evaluation tool for prostate adaptive IMRT based on machine learning, Medical physics 38(2) (2011), 719–726. doi:10.1118/1.3539749.

Predicting dose-volume histogram of organ-at-risk using spatial geometric-encoding network for esophageal treatment planning

Abstract

Keywords

1. Introduction

2. The proposed method

2.2. Architecture of the model

2.2.1. Overview

3.1. Dataset and settings

3.2. Evaluation metric

3.3. Comparison with baselines

3.5. Impact of different components

3.5.1. Impact of msc-SE structure

3.5.2. Impact of multi-view inputs

Table 2 The parameters of three backbone networks used in our DVH prediction framework Backbone Params (Million) DenseNet-53 12.4 3D-Unet 29.3 SGEN 9.7

Footnotes

Acknowledgements

Conflict of interest

References

Table 2
The parameters of three backbone networks used in our DVH prediction framework

Backbone Params (Million)

DenseNet-53 12.4

3D-Unet 29.3

SGEN 9.7