CNN-transformer hybrid network for automated segmentation of multiple defects in concrete bridges

Abstract

Regular inspection of damaged areas of concrete bridges is critical to the safe service of bridges. However, the complexity of bridge environments and multiple defect types pose challenges to deep learning-based inspection methods. To address these issues, this study proposes a novel segmentation model for multiple defects in concrete bridges, named ConcreteBridgeSeg-Net (CBS-Net). Firstly, it employs a convolutional neural network (CNN) to extract local features, followed by a novel Transformer integrated with a Multi-head Transposed Attention (MDTA) mechanism for global feature extraction. Subsequently, an auxiliary feature extraction branch parallel to Transformer was designed to enhance local feature extraction, and a channel feature fusion module (CFFM) was employed to fuse the feature information from both. Lastly, Efficient Channel Attention (ECA) is embedded within skip connections to minimize interference from redundant information. The experimental results show that the mean intersection over union (mIou), mean pixel accuracy (mPA), and mean precision of CBS-Net in the test set are 78.51%, 88.79%, and 86.76%, respectively, and the accuracy metrics are higher than those of other state-of-the-art segmentation models. Notably, CBS-Net achieves an inference speed of 38.72 frames per second (FPS) with an input image size of 512 × 512, demonstrating a fast segmentation rate while maintaining segmentation accuracy. Furthermore, it exhibits excellent generalization performance on public datasets, thereby confirming its applicability to real-world bridge inspection scenarios.

Keywords

bridge detection deep learning concrete multiple defect transformer semantic segmentation

Introduction

Bridges play a critical role in transportation networks. Over time, concrete bridges undergo various forms of deterioration such as cracking, spalling, and exposed rebar due to factors like environmental corrosion, material aging, and heavy loads (Deng et al., 2019). Failure to promptly address these issues can compromise the stability and safety of the bridge structure. Therefore, regular inspection of the concrete structures’ surfaces and prompt maintenance of damaged areas are essential.

The traditional method of detecting surface defects on bridges relies heavily on manual inspection. Inspectors determine the type and severity of defects through visual examination and manual marking. However, manual inspections are labor-intensive, time-consuming, and prone to inaccuracies. Consequently, the transport industry has an urgent need for more automated methods of bridge surface defect detection.

Deep learning is an artificial intelligence technique based on multi-layered neural networks. It can automatically extract key features from massive data and has been used to address challenges in structural health monitoring and defect detection (Cha et al., 2024). In recent years, researchers in the field of civil engineering have utilized deep learning methods to automatically detect surface defects in infrastructure, including concrete defect image classification (Cha et al., 2017), defect detection (Cha et al., 2018), and crack segmentation (Kang et al., 2020). Among them, deep learning-based concrete defect segmentation can achieve pixel-level segmentation results, which are the basis for quantitatively assessing the severity of defects and are therefore more suitable for concrete damage detection. Convolutional neural network (CNN), a well-established architecture in deep learning, excels at extracting image features and demonstrates high computational efficiency in processing image data (Krizhevsky et al., 2017). Therefore, semantic segmentation models employing CNN as the fundamental architecture, such as Fully Convolutional Networks (FCN) (Shelhamer et al., 2016), U-Net (Ronneberger et al., 2015), LinkNet (Chaurasia and Culurciello, 2017), SegNet (Badrinarayanan et al., 2017), etc., have been extensively utilized for surface defect segmentation in concrete structures, including cracking and spalling of building structures (Kumar et al., 2021), tunnel lining cracks (Lee et al., 2020), and bridge structure cracks (Wang et al., 2020a). However, these semantic segmentation models are not specifically designed for concrete damage segmentation and still need to be improved in identifying complex concrete defects. Several researchers have proposed methods to improve the feature extraction capability of CNNs (Yang et al., 2023). For instance, Deng et al. (2020) enhanced segmentation accuracy by integrating spatial pyramid pooling modules to expand the receptive field of the feature map based on LinkNet network. Karaaslan et al. (2021) employed attention-guided techniques to enhance SegNet performance in segmenting defects in concrete infrastructure. (Zoubir et al., 2024) introduced a convolutional attention module to augment crack feature extraction based on U-Net. Due to the receptive field shrinking to a localized area after multiple down-sampling in CNNs, they excel at capturing local features but encounter difficulty in precisely capturing long-range dependencies.

In order to address the challenge of long-range dependency modeling, Transformer networks based on self-attention mechanisms have become a research focus (Han et al., 2022). These networks improve the understanding of contextual information by capturing dependencies between different positions in the input sequence. Some scholars have applied the Transformer architecture to crack segmentation in concrete structures. For example, Wang and Su (2022) proposed the SegCrack model, which uses a hierarchical Transformer structure as the encoder to achieve pixel-level segmentation of complex cracks. Similarly, Li et al. (2024a) introduced Segformer, a Transformer-based model for concrete crack detection in diverse scenarios. This model demonstrated superior segmentation performance compared to CNN-based approaches. Although Transformer-based models have achieved significant breakthroughs, Transformers exhibit high computational complexity in shallow networks and demand substantial computational resources. Furthermore, they require large amounts of training data to achieve satisfactory segmentation accuracy. Consequently, several studies have proposed combining CNN and Transformer to enhance model performance by leveraging their respective advantages. Ali and Cha (2022) proposed the IDSNet network, which combines a CNN with a self-attention module for segmentation of internal damage in concrete; similarly, Kang and Cha (2022) proposed the STRNet lightweight network, which improves crack recognition accuracy by adding a multi-head self-attention module to the decoder. Experiments show that the mIoU of these models is more than 90% and the FPS at 1280 × 800 resolution is more than 45. Zhou et al. (2023) incorporated Swin-Transformer (Liu et al., 2021), which is based on hierarchical structure, local self-attention, and a window shift mechanism to capture multi-scale features and reduce computation, into DeepLabv3 + to construct a joint backbone network for extracting local and global information, significantly improving the crack segmentation accuracy. Li et al. (2024b) proposed a hybrid U-shaped CrackTrNet model for concrete dam crack detection by embedding a standard Transformer into the CNN layer, demonstrating superior segmentation accuracy over existing state-of-the-art networks.

Current hybrid networks integrating CNN-Transformer architectures have achieved notable advancements in concrete defect detection. However, these models are primarily designed for crack segmentation, with their Transformer modules relying on local windows or standard self-attention to capture spatial global dependencies. Such approaches, constrained by fixed window sizes, impede the effective capture of long-range information, limiting their performance in segmenting complex concrete structures with multiple defects. Additionally, the complexity and multi-scale nature of concrete bridge damage backgrounds pose challenges for the precise segmentation of multiple defects. To address these challenges, this study proposes a multi-defect segmentation model for concrete bridges, termed Concrete Bridges Segmentation Network (CBS-Net). The key contributions are as follows:

(1) A transformer module integrating the Multi-Dconv Head Transposed Attention (MDTA) mechanism is proposed. It generates self-attention through cross-channel information and can extract contextual information efficiently, which is superior to traditional self-attention mechanisms.

(2) The designed auxiliary feature extraction branch and channel feature fusion module effectively integrate detailed and global semantic information through deep fusion, enhancing the model’s segmentation accuracy for multi-scale defects such as cracks and spalling.

(3) An Efficient Channel Attention (ECA) mechanism is embedded in the decoder’s skip connections to better convey information about the characteristics of concrete defects, and to reduce interference from complex environmental backgrounds.

(4) The proposed CBS-Net can accurately segment various types of defects, with precision metrics outperforming other mainstream segmentation models, making it suitable for detecting multiple concrete defects.

The paper’s structure is outlined as follows: Section Network structure provides an overview of the proposed model’s structure and details of each module, while Section Dataset and experimental configuration covers the dataset, evaluation metrics, and experimental parameters. Section Experimental results and analysis presents the experimental results and analysis, discussing the effectiveness of each module and comparing the models. Finally, Section Conclusion encapsulates the main conclusions.

Network structure

The overall framework of CBS-Net, as shown in Figure 1, consists of encoder and decoder structures. A hybrid architecture of CNN and Transformer is used as the encoder. CNN is adept at swiftly extracting primary features, while the Transformer focuses on capturing global feature correlations to efficiently extract the features of the concrete defect images. To prevent loss of detailed features during global feature extraction by the Transformer, introducing an auxiliary feature extraction branch (AFEB) to run in parallel with the Transformer, aiming to enhance local feature extraction following CNN feature extraction. Subsequently, a channel feature fusion module (CFFM) integrates the feature information from the Transformer and AFEB. The decoder is employed to restore information lost during feature extraction primarily through up-sampling modules to recover the original size of the feature map and via skip connections for concatenating features of different scales. Furthermore, an ECA module is introduced in jump connections to enhance feature transfer capability.

Figure 1.

Overall structure of CBS-Net.

CNN encoder

The CNN encoder utilized in this study employs the first four stages of the Resnet50 network (He et al., 2016) to capture local features and edge information. As shown in Figure 2, Stage 1 represents preprocessing, consisting of 7 × 7 convolutions, stride 2 convolution kernels, and max pooling convolution. For the input image (H × W × 3), the image is down-sampled twice: first, by a 7 × 7 convolution with a stride of 2, and second, by max pooling. The number of output channels is 64, and the resulting H × W size of the image is 1/4 of the original image.

Figure 2.

CNN network structure.

The latter three stages primarily consist of multiple bottleneck units, which comprise two 1 × 1 convolutions, one 3 × 3 convolution, and shortcut branches. As depicted in Figure 2, the Bottleneck adjusts the channel number through a 1 × 1 convolution in the shortcut branch and then adds it to the feature map of the main path. Bottleneck1 represents a convolutional residual module that reduces the size of the feature map by half through a 1 × 1 convolutional kernel with a stride of 2 in the shortcut branch and adjusts the number of channels for the next layer’s residual structure. Bottleneck2 represents a standard residual module where the input feature layer is directly added to the output of the main path via a shortcut branch without altering the size of the feature map. This construction method can mitigate gradient vanishing and information loss in deep neural networks.

Transformer encoder

The Transformer block is shown in Figure 3 and mainly consists of a self-attention mechanism, feedforward network (FFN) layer, and layer normalization (LN). The self-attention mechanism is utilized to capture the correlation of diverse location information and acquire global features. The FFN serves for nonlinear transformations, facilitating the model to learn more abstract features and further enhancing its fitting capability. Additionally, the Layer-Normalization contributes to improving the stability and convergence speed of the model.

Figure 3.

Transformer structure.

The computational overhead of Transformers primarily arises from the self-attention mechanism, where the computational burden of the self-attention weight matrix grows exponentially with increasing image resolution. Existing methods solve this problem mainly by dividing the image into multiple patches. However, this approach is limited by the size of the patches and cannot adequately capture long-range information interactions. Therefore, this study employs the multi-Dconv head transposed attention (MDTA) (Zamir et al., 2022) to compute self-attention across channels for generating global feature information.

The structure of MDTA is depicted in Figure 4. Given the input tensor x ∈ R^{H × W × C}, after layer normalization processing, the context feature information across channels is first aggregated using a 1 × 1 convolution, followed by coding of spatial context information of channels using a 3 × 3 depth convolution to obtain linear projections of query (Q), key (K), and value (V). Subsequently, the dot-product of Q and transposed K generates an attention map of size R^C×C, reducing computational complexity compared to the traditional self-attention map $R^{{(H \times W)}^{2}}$ . The softmax function is then applied to determine the importance of each query vector, yielding the attention score map A. Finally, the score map A is multiplied by value to generate the final output. The calculation methods for Q, K , V and MDTA are as follows:

Q = W_{d}^{Q} W_{P}^{Q} X, K = W_{d}^{K} W_{P}^{K} X, V = W_{d}^{V} W_{P}^{V} X

(1)

Attention (X) = V \cdot Softmax (\frac{K^{T} \cdot Q}{α})

(2)

where: X represents the input tensor, W_p represents 1 × 1 point by point convolution, and W_d represents 3 × 3 depth convolution. K ∈ R^{HW × C}, Q ∈ R^{HW × C}, V ∈ R^{HW × C}. α is a learnable scaling parameter, which is used to control the dot product size of K^T and Q.

Figure 4.

Illustration of MDTA.

The self-attention computation incorporates a multi-head mechanism, enabling the simultaneous and independent computation of each attention head. This approach assigns the features of different channels to separate self-attention heads, with each attention head being weighted by a parameter matrix to yield varying attention outputs. The results of the parallel computation of multiple attention heads are combined through the Concat function to get the final output. The calculation method is shown below:

h e a d_{i} = Attention (Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V})

(3)

MDTA (X) = Concat (h e a d_{1}, \cdot \cdot \cdot, h e a d_{h}) W^{o}

(4)

Where:

Q W_{i}^{Q}

K W_{i}^{K}

and

V W_{i}^{V}

denote the attentional weights in the i-th attentional head, respectively. Concat denotes the splicing operation. W^o denotes the weight matrix used for linear transformation.

Auxiliary feature extraction branch

The Transformer encoder primarily extracts global features. However, considering the relatively small size of cracks and exposed rebar defects in concrete, an auxiliary branch is added to extract subtle features and texture information. In this auxiliary feature extraction branch, the Convolutional Modulation Block (Hou et al., 2024) is utilized to extract local features. This module effectively interacts with local features while maintaining computational efficiency, thereby improving the recognition accuracy of subtle defects.

The convolutional modulation block is illustrated in Figure 5. Given an input feature tensor x ∈ R^{H × W × C}, it is initially divided into two branches x₁ and x₂. Branch x₁ performs depth-wise convolution with an 11 × 11 kernel to generate a local feature map A₁, while branch x₂ generates a feature map V₁ through a linear transformation between channels using 1 × 1 convolution. Subsequently, the feature map A₁ and the feature map V₁ are multiplied element by element (Hadamard Product), where the output of A₁ is used as weights to modulate the feature V₁. Finally, a 1 × 1 convolution is applied to map the generated weight information back to the original feature dimensions, thereby obtaining the final output of the module.

Figure 5.

Convolutional modulation block structure.

Channel feature fusion module

Transformers and auxiliary branches are structures with different characteristics. Therefore, the channel feature fusion module (CFFM) is employed in this study to effectively integrate their respective features. This module aims to minimize redundant and irrelevant channel information, and thereby facilitating more precise segmentation results through enhanced interaction between local and global feature information.

The specific process is illustrated in Figure 6. Firstly, the feature maps of Transformer and AFEB are concatenated (Cat) in the channel dimension. Subsequently, the number of channels is adjusted using 1 × 1 convolution to halve the channels of the model. The SE (squeeze-and-excitation) module (Hu et al., 2018) is then incorporated to facilitate a deep fusion of local and global features. The SE module employs global average pooling to compress the input feature map into a 1 × 1 × C feature vector. After traversing two fully connected layers, the Sigmoid function computes the probability distribution of the vector for obtaining channel weights. Finally, these channel weight values are multiplied by the matrix corresponding to each original feature map channel to yield a feature map with channel attention.

Figure 6.

Channel feature fusion module.

Decoder

The decoder is utilized for reinstating contextual information and primarily comprises an up-sampling block along with skip connections. The up-sampling block doubles the size of the input image by bilinear interpolation. Skip connections facilitate the fusion of output concrete defect features at various scales.

The transmission process of skip connection paths for features of different scales is relatively lengthy, which hinders the effective conveyance of detailed features such as small cracks and rebars to the decoder, thereby impacting the accuracy of concrete defect image segmentation. To tackle this issue, we embed the Efficient Channel Attention (ECA) (Wang et al., 2020b) mechanism, which prioritizes essential detail features by assigning weights to the feature maps. High weights are allocated to emphasize significant information, while low weights are utilized to filter out irrelevant details.

The structure of ECA is depicted in Figure 7. Firstly, perform Global Average Pooling (GAP) on the input feature map to obtain weights W₁ of size 1 × 1 × C, achieving the fusion of global feature information. Then, use adaptive one-dimensional convolution with a kernel size of k to obtain channel weights W₂. Subsequently, calculate the probability distribution of channel weights W₂ through the Sigmoid activation function to obtain attention weights W₃. Finally, multiply the attention weights W₃ by the input feature map to obtain the weighted feature map.

Figure 7.

ECA module.

Dataset and experimental configuration

Dataset

Owing to the absence of publicly accessible semantic segmentation datasets for multiple surface defects of concrete bridges. This paper collects images of surface defects on concrete bridges by photographing 17 concrete bridges. During the shooting process, a consistent device (with a camera resolution of 12 megapixels) was used for all images. The shooting angle was kept as parallel to the defect as possible. There are no fixed lighting settings, including variations in sunny and cloudy conditions, so the images captured closely resemble actual engineering scenes. Following meticulous manual screening, 1365 high-resolution images capturing clear and comprehensive representations of three prevalent types of concrete defects—spalling, exposed rebar, and cracks—were selected. All images were resized to 512 × 512 × 3.

The robustness and generalization capability of the trained model are influenced by both the quantity and quality of the dataset. Therefore, we used data enhancement methods to expand the original concrete defect images. Specific methods include: (1) Adding noise: Gaussian noise was added pixel-by-pixel, with the noise mean fixed at 0 and the standard deviation randomly selected between 0.01 and 0.05 to simulate different degrees of noise interference. (2) Brightness change: The original image was mixed with an all-black image using linear interpolation, where the mixing ratio is randomly selected between 0.35 and 0.9 to dynamically adjust brightness to 35–90% of the original. (3) Rotational change: The image was rotated by 90°, 180°, and 270° respectively to simulate different shooting angles. A total of 2050 concrete defect images were obtained by randomly combining the above methods (at least two enhancement methods were applied to each image). From this expanded dataset, 80%, 10%, and 10% of the images were randomly selected for inclusion in the training set, validation set, and test set respectively. Table 1 provides details on the number and size of images for different damage types.

Table 1.

Number of concrete defect data sets.

Damage type	Resolution	Total number	Training set	Validation set	Test set
Crack	512 × 512	490	392	49	49
Rebar	512 × 512	550	440	55	55
Spalling	512 × 512	350	280	35	35
Multiple defects	512 × 512	660	528	66	66

The Labelme software is utilized for annotating concrete defects. In the labeling process, the concrete defect image is first enlarged to the maximum extent and then labeled along the defect contour using the polygon labeling tool to form a label file, which contains the defect location, label name, and pixel value. The image of concrete structure defects and the image annotation process are depicted in Figure 8.

Figure 8.

Concrete defect image and annotation method. (a) Concrete structure defect image. (b) Image labeling process.

Experimental environment and parameter configuration

The semantic segmentation model algorithm uses Python 3.8 as the programming language, with Pytorch 1.13 as the deep learning framework, and CUDA version 11.7. The computer configuration includes 32 GB of RAM, an Intel-i5-13400 central processing unit (CPU), and an NVIDIA GTX 4070 graphics processing unit (GPU).

The model’s training period was set to 120 epochs, and using the Adam algorithm as the optimizer, with a momentum parameter of 0.9 and employing the cosine function for learning rate decay. The initial learning rate was set to 0.001, and each batch trains with 6 images. To ensure reproducible training results and avoid randomness in experimental outcomes, the order of data loading was fixed by setting a specific random seed.

Evaluation metrics

The commonly used evaluation metrics for semantic segmentation models include Class Pixel Accuracy (CPA), mean Pixel Accuracy (mPA), Class Precision (CPre), mean Precision (mPre), Intersection over Union (IoU), and mean Intersection over Union (mIoU). The definition of these indices is associated to the confusion matrix, as illustrated in Table 2. The values in the columns of the confusion matrix represent the predicted number of pixels for category i, while those in the rows represent the actual number of pixels for category i. The diagonal line C_i,i represents the accurately predicted number of pixels for category i.

Table 2.

Confusion matrix.

Confusion matrix		Predicted						Sum of the rows
Confusion matrix		Class 1	Class 2	…	Class i	…	Class k	Sum of the rows
Ture	Class 1	C _1,1	C _1,2	…	C _1,i	…	C _1,k	$\sum_{j = 1}^{K} C_{1}$
	Class 2	C _2,1	C _2,2	…	C _2,i	…	C _2,k	$\sum_{j = 1}^{K} C_{2, j}$
	…	…	…	…	…	…	…
	Class i	C _i,1	C _i,2	…	C _i,i	…	C _i,k	$\sum_{j = 1}^{K} C_{i, j}$
	…	…	…	…	…	…	…
	Class k	C _k,1	C _k,2	…	C _k,i	…	C _k,k	$\sum_{j = 1}^{K} C_{k, j}$
Sum of the columns		$\sum_{j = 1}^{K} C_{j, 1}$	$\sum_{j = 1}^{K} C_{j, 2}$		$\sum_{j = 1}^{K} C_{j, i}$		$\sum_{j = 1}^{K} C_{j, k}$

CPAi is the ratio of the number of pixels correctly predicted as category i to the total number of pixels predicted as category i (the sum of category i columns in Table 2). The calculation formula is as follows:

C P A_{i} = \frac{{C_{i}}_{,}_{i}}{\sum_{j = 1}^{K} {C_{j}}_{,}_{i}}

(5)

mPA is the average of the sum of the CPA values across K categories. The calculation formula is as follows:

m P A = \frac{1}{K} \sum_{i = 1}^{K} C P A_{i}

(6)

CPre_i represents the number of predicted pixels that correctly belong to class i and the number of pixels that actually belong to class i. The calculation formula is as follows:

C P r e_{i} = \frac{{C_{i}}_{,}_{i}}{\sum_{j = 1}^{K} {C_{i}}_{,}_{j}}

(7)

mPre represents the summed average of CPre values of K categories. The calculation formula is as follows:

m P r e = \frac{1}{K} \sum_{i = 1}^{K} \Pr e_{i}

(8)

The meaning of IoU is that the ratio of the intersection and union between the prediction and truth of a certain category. For category i, the intersection of prediction and reality is the number of pixels correctly predicted belonging to category i, and the union of prediction and reality is the sum of the number of pixels predicted belonging to category i (summation of class i columns in Table 2) and the number of pixels belonging to class i (summation of class i rows in Table 2) minus the number of pixels correctly predicted belonging to class i. The calculation formula is as follows:

I o U_{i} = \frac{C_{i, i}}{\sum_{j = 1}^{K} C_{j, i} + \sum_{j = 1}^{K} C_{i, j} - C_{i, i}}

(9)

mIoU is the average of the sum of the IoU values across K categories. The calculation formula is as follows:

m I o U = \frac{1}{K} \sum_{i = 1}^{K} I o U_{i}

(10)

Frames per second (FPS) are selected to evaluate the real-time performance of the model, where FPS is calculated as follows:

F P S = \frac{M}{T}

(11)

where: T is the time required for detection and M is the number of images detected.

Loss function

During the process of training a neural network, the loss function is employed to calculate the discrepancy between the predicted and actual values. Utilizing this calculated discrepancy, the model iteratively optimizes its training parameters to minimize the disparity between predicted and actual values.

The size of concrete defects varies, and the pixel count of images of cracks and exposed rebar defects is relatively low. To address this issue of pixel imbalance in such images, in this paper, cross-entropy (CE) loss (Zhang and Sabuncu, 2018), Focal loss (Lin et al., 2017) and Dice loss (Sudre et al., 2017) were set respectively for comparative experiments.

The Focal loss function introduces weight coefficients α and γ to the CE loss function in order to adjust the weights of difficult and easily classified samples. This reduces the impact of easy-classified samples on overall losses, allowing the model to focus more on difficult classified samples. The Dice loss function addresses pixel ratio imbalances by disregarding a large number of background pixels when calculating ratio of the intersection and union. The formulas for CE loss, Focal loss, and Dice loss are as follows:

L_{C E} = - \frac{1}{N} \sum_{i = 1}^{N} x_{i} (1 - p_{i}) \log (p_{i})

(12)

L_{F o c a l} = - \frac{1}{N} \sum_{i = 1}^{N} x_{i} α {(1 - p_{i})}^{γ} \log (p_{i})

(13)

L_{D i c e} = 1 - \frac{2 \sum_{i = 1}^{N} x_{i} p_{i} + ε}{\sum_{i = 1}^{N} x_{i} + \sum_{i = 1}^{N} p_{i} + ε}

(14)

where: N represents the total number of pixel points, x_i denotes the labeled value of the i-th pixel point, p_i indicates the predicted value of the i-th pixel point. α and γ are adjustment coefficients with specific values (α = 0.25 and γ = 2). ε is a smoothing coefficient to prevent the denominator from taking the value of zero, and ε is set to 0.001.

Experimental results and analysis

The performance of the proposed CBS-Net model is evaluated through experiments in this section. To ensure a fair comparison, all experiments are conducted under the same experimental environment and dataset, with performance evaluation carried out using the test set. Unless specified otherwise, consistent parameters are employed across all experiments. The experiments primarily encompass: (1) To determine the most suitable loss function for concrete structure multi-defect scenarios, comparative experiments were conducted using different loss functions. (2) Conducting ablation experiments to assess the efficacy of each module within CBS-Net. (3) Comparing CBS-Net with current state-of-the-art models to determine its superior performance. (4) Validating the generalization performance of the proposed CBS-Net on public datasets.

Loss function comparison results

Table 3 shows the comparison results of recognition accuracy evaluation metrics for three different loss functions of CBS-Net configurations. As shown in Table 3, CBS-Net has the highest recognition accuracy when using the Dice loss function, with mIoU of 78.51%, mPA of 88.79%, and mPre of 86.76%, surpassing those obtained using the CE and Focal loss functions. This indicates that Dice loss can effectively address the issue of pixel imbalance in the multi-defect segmentation scenario of concrete structures.

Table 3.

Comparison results of three different loss functions for CBS-Net configuration.

Loss function	IoU (%)			CPA (%)			mIoU (%)	mPA (%)	mPre (%)
Loss function	R	S	C	R	S	C	mIoU (%)	mPA (%)	mPre (%)
CE_loss	63.95	87.37	65.53	77.59	92.78	77.39	72.28	82.59	84.47
Focal_loss	67.73	88.26	66.43	81.66	93.73	79.51	74.17	84.96	85.43
Dice_loss	72.86	89.10	73.59	84.39	94.65	87.34	78.51	88.79	86.76

Notes: S denotes spalling, R denotes exposed rebar, and C denotes crack.

Ablation experiment

To further validate the efficacy of each module within the proposed CBS-Net architecture, comprehensive ablation experiments were conducted using CNN as the baseline model. The Transformer Encoder, Aux-branch, and ECA modules were individually integrated into the baseline model, resulting in models denoted as CNN + Trans, CNN + Trans + Aux-branch, and CNN + Trans + Aux-branch + ECA (CBS-Net) respectively. In addition, the Transformer encoder for standard self-attention is added to the baseline model for comparing the computational efficiency of MDTA and standard self-attention. The Dice loss function was utilized during training while maintaining consistent experimental parameters. The evaluation metrics for each model are presented in Table 4 with segmentation results depicted in Figure 9.

Table 4.

Ablation experiment results of different components.

Nets	Transformer	Aux-branch	ECA	mIou (%)	mPre (%)	FPS (f/s)
CNN				73.71	82.68	43.67
CNN + standard transformer layer				75.08	85.63	31.28
CNN + trans	√			76.67	85.97	40.71
CNN + aux-branch		√		75.35	84.76	41.23
CNN + ECA			√	74.64	84.03	42.59
CNN + trans + aux-branch	√	√		77.92	86.34	39.52
CNN + trans + aux-branch + ECA (CBS-Net)	√	√	√	78.51	86.76	38.72

Figure 9.

Example of segmentation in ablation experiment.

Table 4 demonstrates that the mIoU of CNN + Trans has increased by 2.96% and the mPre has increased by 3.29% compared to CNN. As shown in Figure 9, when utilizing only the CNN encoder for segmentation of concrete bridges with multiple defects, issues such as partial spalling and discontinuous crack segmentation arise. The incorporation of Transformer into a serial encoder structure improves the problem of discontinuous cracks, indicating that connecting CNN and Transformer in series can effectively extract global information of concrete bridges defects. Furthermore, CNN + Trans + Aux-branch exhibits a further improvement in mIoU by 1.25% compared to CNN + Trans. While segmenting more global defects, the model with an auxiliary branch can further segment local concrete spalling, indicating that the auxiliary feature extraction branch is adept at capturing local features, and the channel feature fusion module effectively integrates the auxiliary branch with Transformer, thereby enhancing the segmentation accuracy of concrete defects. Additionally, after adding the ECA module to CNN + Trans + Aux-branch, the mIoU increases by 0.59%, and the mPre increases by 0.42%, enhancing the model’s focus on the characteristics of concrete defects.

The addition of these modules improves segmentation accuracy, but inevitably reduces the image segmentation speed of the model. Therefore, the impact of each module on image segmentation speed was tested in ablation experiments. The inference speed of each model is shown in Table 4. Under the same experimental conditions, the inference speed of the network with MDTA reaches 40.71 FPS, representing a relative improvement of 30.15% compared to the 31.28 FPS achieved by the network with standard self-attention. This indicates that MDTA effectively reduces computational complexity while maintaining model performance. The FPS (frames per second) of CBS-Net is 38.72, although its inference speed is slightly reduced compared to the baseline CNN model. Given the essential requirement for real-time segmentation, which is typically considered to be more than 30 FPS, the proposed network is capable of meeting the demands for real-time detection of concrete defect images.

Model comparison

To verify the superior performance of the constructed CBS-Net, this study compares it with currently advanced segmentation models (such as PSPNet (Zhao et al., 2017), DeepLabv3+ (Chen et al., 2018), Res-Unet (Xiao et al., 2018), HRNet (Wang et al., 2020c), Segformer (Xie et al., 2021), UTCD-Net (Zhang et al., 2023), SDDNet (Choi and Cha, 2019)). To ensure the fairness of the comparative experiments, all semantic segmentation models used the Dice loss function and employed the concrete structure multi-defect dataset established in this paper. Comparative experiments were conducted in the same experimental environment, where the training set was used to train model parameters, and the test set was used to evaluate various performance metrics of the models.

Figure 10(a) illustrates the loss function curves of the different models during the training phase, while Figure 10(b) depicts the validation error curves. It is evident from Figure 10 that after 100 training iterations, both the training and validation losses for each model stabilize without significant fluctuations, indicating convergence. Notably, CBS-Net exhibits superior learning ability as evidenced by its lower loss and validation error values compared to several comparative models.

Figure 10.

Error curve. (a) Training loss curve (b) Verification error curve.

Comparison of recognition accuracy

The comparison results of accuracy metrics for different models are presented in Table 5. Among them, CBS-Net demonstrates superior overall accuracy compared to other comparative models, achieving mIoU, mPA, and mPre values of 78.51%, 88.79%, and 86.76% respectively. In comparison to HRNet based on CNN networks, it outperforms by 4.24%, 2.77%, and 3.78%; while compared to UTCD-Net based on CNN-Transformer hybrid architecture, it surpasses by 3.46%, 2.18%, and 2.52%. Furthermore, in the test results for spalling defects, CBS-Net achieved an IoU of 89.1% and a CPA of 94.65%, which was the highest among the comparative models. In the test results for small-scale defects such as cracks and exposed rebar, CBS-Net also demonstrated the highest accuracy among the comparative models. CBS-Net exceled in segmenting both larger areas and minute defects, indicating that it effectively extracts feature information across different scales.

Table 5.

Comparison results of accuracy evaluation metrics for different models.

Models	IoU (%)			CPA (%)			mIoU (%)	mPA (%)	mPre (%)
Models	S	R	C	S	R	C	mIoU (%)	mPA (%)	mPre (%)
SDDNet	84.76	64.45	67.43	92.44	79.19	85.31	72.21	85.65	79.65
PSPNet	86.43	66.51	53.45	93.00	79.80	76.86	68.79	83.22	78.71
DeeplabV3+	84.40	64.61	63.27	91.21	80.76	83.21	70.76	85.06	79.08
ResU-Net	84.58	65.78	71.15	92.58	81.31	85.22	73.83	86.37	82.42
HRNet	84.77	69.51	68.55	92.04	82.87	83.16	74.27	86.02	82.98
Segformer	85.65	69.58	68.63	93.39	82.45	84.09	74.62	86.64	83.38
UTCD-Net	84.87	69.76	70.52	92.93	81.42	85.49	75.05	86.61	84.24
CBS-Net (ours)	89.10	72.86	73.59	94.65	84.39	87.34	78.51	88.79	86.76

Notes: S denotes spalling, R denotes exposed rebar, and C denotes crack.

By comparing the semantic segmentation images output by the models, one can intuitively evaluate the model’s segmentation performance. Figure 11 illustrates the segmentation results of models. As shown in Figure 11, all models accurately segment the contours of spalling defects in cases with no obvious background interference and a single damaged area. In terms of exposed rebars, CBS-Net excels at even smaller exposed rebar segmentation. When it comes to crack segmentation and dealing with subtle net-like cracks, all models display varying degrees of missed detections due to the difficulty in identifying such fine cracks. Among these models, CBS-Net segmentation results closely align with labeled images and demonstrate better continuity indicating that CBS-Net is more effective in extracting global crack characteristics.

Figure 11.

The segmentation results of different models.

As illustrated in Figure 12, when tested for the segmentation of multiple defects, other comparative models demonstrate notable omissions or erroneous segmentations. Specifically, in Figure 12(a), DeepLabv3 + incorrectly recognizes a portion of exposed rebar as spalling; in Figure 12(b), ResU-Net fails to recognize small-scale spalling; in Figure 12(c), PSPNet exhibits several instances of discontinuous crack segmentation; and in Figure 12(d), the HRNet missed the segmentation of exposed rebar. Additionally, while Segformer is capable of segmenting a wider range of overall defects, its precision in segmenting local, detailed defects is limited. In contrast to the other comparative models, CBS-Net does not exhibit significant missed detections or false detections. It is not only capable of accurately segmenting overall defects but also precisely identifies local, minute defects, indicating that CBS-Net is more suitable for the segmentation task of concrete structure multi-defect.

Figure 12.

Comparison of multiple defect segmentation results.

Efficiency comparison

To comprehensively evaluate the model complexity and segmentation accuracy, we compared the five models with high accuracy in Section Model comparison based on the number of model parameters and the speed of inference. Table 6 shows the number of parameters and inference speed for the different models.

Table 6.

Comparison of efficiency indicators of different models.

Models	Params (M)	FPS (f/s)	mIoU (%)
ResU-Net	43.93	36.89	73.83
HRNet	29.53	15.36	74.27
Segformer	47.23	19.89	74.62
UTCD-Net	46.92	19.23	75.05
CBS-Net (ours)	28.63	38.72	78.51

Notes: The image size in the efficiency test was 512 × 512 pixels and a total of 100 images were tested.

As shown in Table 6, CBS-Net has 28.63 M parameters and an FPS of 38.72 f/s. Compared to other models, CBS-Net has fewer parameters and the fastest inference speed. This indicates that the combined architecture of Transformer and CNN guarantees recognition accuracy. Moreover, the MDTA self-attention mechanism within the Transformer structure effectively reduces computational complexity. In summary, CBS-Net exhibits superior comprehensive performance.

Public dataset testing

To assess the generalization capability of the proposed CBS-Net in practical applications, this study will evaluate the trained model using datasets from diverse scenarios. We chose the three models with better performance in Section Model comparison for comparative experiments on the ConcreteCrack (Özgenel, 2019) and Concrete Bridge Defects (He et al., 2024) public dataset. The ConcreteCrack dataset contains 458 high-resolution (4320 × 3024) concrete crack images with pixel-level annotations, and 10% of the images were randomly selected for testing. The concrete bridge defects dataset contains 241 concrete spalling images and 286 exposed rebar images with an image resolution of 256 × 256, randomly selecting 10% of the images for cross-validation. The cross-validation results in the public dataset are shown in Table 7. The test results show that CBS-Net has higher accuracy with mIoU and mPA of 73.87% and 83.61%, respectively. In addition, as shown in the segmentation results of Figure 13, CBS-Net can more accurately segment different classes of concrete defects, demonstrating that the proposed model exhibits excellent generalization performance.

Table 7.

Public dataset validation results.

Models	CPA(%)			IoU (%)			mPA (%)	mIoU (%)
Models	Crack	Spalling	Rebar	Crack	Spalling	Rebar	mPA (%)	mIoU (%)
HRNet	80.41	84.69	67.56	67.15	74.39	60.48	77.55	67.34
Segformer	81.75	87.76	68.32	67.99	76.43	61.87	79.28	68.76
UTCD-Net	82.21	91.84	69.23	69.68	79.97	62.61	81.09	70.75
CBS-Net (ours)	87.93	92.91	69.99	74.62	82.51	64.49	83.61	73.87

Figure 13.

Comparison of segmentation results on public datasets.

Conclusion

This paper proposes a multi-defect segmentation model for concrete bridges, namely CBS-Net. By conducting ablation experiments on CBS-Net, comparing it with currently advanced models, and testing it on public datasets, the main conclusions are as follows:

(1) The ablation experiments demonstrate that incorporating a Transformer encoder into the CNN network enhances the recognition capability of global features, and adding an auxiliary feature extraction branch effectively improves the precision of local detail segmentation. Furthermore, introducing the ECA attention mechanism through skip connections reduces the interference from background information, thus further enhancing the recognition accuracy.

(2) The recognition accuracy indicators of CBS-Net for multiple concrete defects, including mIoU, mPA, and mPre, have achieved 78.51%, 88.79%, and 86.76% respectively. These accuracy indicators surpass those of other currently advanced semantic segmentation models. Furthermore, the segmentation results of concrete structure defect images illustrate that CBS-Net excels at accurately segmenting mixed and minute defects, indicating its suitability for segmenting multiple defects in concrete bridges.

(3) Segmentation speed tests show that CBS-Net achieves an FPS of 38.72 f/s, indicating good detection speed while ensuring segmentation accuracy.

(4) When tested on public datasets, CBS-Net exhibits excellent accuracy indicators, demonstrating its outstanding generalization performance.

There are some limitations to this paper. Firstly, the training data cover only a limited range of defect types, which may impact the segmentation accuracy of the model when applied to specific defects. Secondly, Secondly, model performance was not validated on mobile devices. Future work will concentrate on two main aspects: firstly, expanding the diversity of the dataset to improve the model’s adaptability to more complex defects; secondly, optimizing the model’s structure by employing pruning techniques to reduce computational complexity while maintaining accuracy, thereby enabling easier deployment on mobile inspection devices.

Footnotes

Acknowledgments

The author gratefully acknowledges the financial support of the National Natural Science Foundation of China (Grant No. 51708188) for this study. The author sincerely appreciates the editors and anonymous reviewers for their valuable suggestions in further improving this manuscript.

Author contributions

Zihang Yu: Writing – original draft, Methodology, Investigation, Validation. Caiping Huang: Writing – review & editing, Funding acquisition, Validation. Hui Li: Visualization, Data curation.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: this work was supported by the National Natural Science Foundation of China (Grant No. 51708188) for this study.

ORCID iDs

Zihang Yu

Caiping Huang

Hui Li

Data Availability Statement

Data will be made available on request.*

References

Ali

Cha

(2022) Attention-based generative adversarial network with internal damage segmentation using thermography. Automation in Construction 141: 104412.

Badrinarayanan

Kendall

Cipolla

(2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(12): 2481–2495.

Cha

Choi

Büyüköztürk

(2017) Deep learning-based crack damage detection using convolutional neural networks. Computer-Aided Civil and Infrastructure Engineering 32(5): 361–378.

Cha

Choi

Suh

, et al. (2018) Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering 33(9): 731–747.

Cha

Ali

Lewis

, et al. (2024) Deep learning-based structural health monitoring. Automation in Construction 161: 105328.

Chaurasia

Culurciello

(2017) Linknet: exploiting encoder representations for efficient semantic segmentation. In 2017 IEEE visual communications and image processing (VCIP), St. Petersburg, FL, USA, 10–13 December 2017, pp. 1–4.

Chen

Zhu

Papandreou

, et al. (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018, 801–818.

Choi

Cha

(2019) SDDNet: real-time crack segmentation. IEEE Transactions on Industrial Electronics 67(9): 8016–8025.

Deng

Yan

Nie

(2019) A simple corrosion fatigue design method for bridges considering the coupled corrosion-overloading effect. Engineering Structures 178: 309–317.

10.

Deng

Mou

Kashiwa

, et al. (2020) Vision based pixel-level bridge structural damage detection using a link ASPP network. Automation in Construction 110: 102973.

11.

Han

Wang

Chen

, et al. (2022) A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(1): 87–110.

12.

Zhang

Ren

, et al. (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA, 27-30 June 2016, 770–778.

13.

Deng

(2024) A novel MO-YOLOv4 for segmentation of multi-class bridge damages. Advanced Engineering Informatics 62: 102586.

14.

Hou

Cheng

, et al. (2024) Conv2former: a simple transformer-style convnet for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 46: 8274–8283.

15.

Shen

Sun

(2018) Squeeze-and-excitation networks. In: IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA, 18–23 June 2018, pp. 7132–7141.

16.

Kang

Cha

(2022) Efficient attention-based deep encoder and decoder for automatic crack segmentation. Structural Health Monitoring 21(5): 2190–2205.

17.

Kang

Benipal

Gopal

, et al. (2020) Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning. Automation in Construction 118: 103291.

18.

Karaaslan

Bagci

Catbas

(2021) Attention-guided analysis of infrastructure damage with semi-supervised deep learning. Automation in Construction 125: 103634.

19.

Krizhevsky

Sutskever

Hinton

(2017) ImageNet classification with deep convolutional neural networks. Communications of the ACM 60(6): 84–90.

20.

Kumar

Sharma

Kota

(2021) Automatic multiclass instance segmentation of concrete damage using deep learning model. IEEE Access 9: 90330–90345.

21.

Lee

Hwang

Choi

, et al. (2020) Estimation of crack width based on shape-sensitive kernels and semantic segmentation. Structural Control and Health Monitoring 27(4): e2504.

22.

Zhang

Zhu

, et al. (2024a) Automatic crack detection on concrete and asphalt surfaces using semantic segmentation network with hierarchical Transformer. Engineering Structures 307: 117903.

23.

Yuan

Ren

, et al. (2024b) CNN-Transformer hybrid network for concrete dam crack patrol inspection. Automation in Construction 163: 105440.

24.

Lin

Goyal

Girshick

, et al. (2017) Focal loss for dense object detection. In: IEEE international conference on computer vision (ICCV), Venice, Italy, 22–29 October 2017, 2999–3007.

25.

Liu

Lin

Cao

, et al. (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: IEEE/CVF international conference on computer vision (ICCV), Montreal, QC, Canada, 10–17 October 2021, pp. 9992–10002.

26.

Özgenel

ÇF

(2019) Concrete Crack Segmentation Dataset. London: Mendeley Data, V1.

27.

Ronneberger

Fischer

Brox

(2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, 5–9 October 2015, Springer International Publishing, 234–241.

28.

Shelhamer

Long

Darrell

(2016) Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(4): 640–651.

29.

Sudre

Vercauteren

, et al. (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, Québec City, QC, Canada, 14 September 2017, Cham, Springer International Publishing, 240–248.

30.

Wang

(2022) Automatic concrete crack segmentation model based on transformer. Automation in Construction 139: 104275.

31.

Wang

Zhang

Pan

, et al. (2020a) An automated inspection method for the steel box girder bottom of long-span bridges based on deep learning. IEEE Access 8: 94010–94023.

32.

Wang

Zhu

, et al. (2020b) ECA-Net: efficient channel attention for deep convolutional neural networks. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 3–19 June 2020, 11531–11539.

33.

Wang

Sun

Cheng

, et al. (2020c) Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(10): 3349–3364.

34.

Xiao

Lian

Luo

, et al. (2018) Weighted res-unet for high-quality retina vessel segmentation. In: 2018 9th international conference on information technology in medicine and education (ITME), Hangzhou, China, 19–21 October 2018, 327–331.

35.

Xie

Wang

, et al. (2021) SegFormer: simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems 34: 12077–12090.

36.

Yang

Huang

Kong

, et al. (2023) A deep segmentation network for crack detection with progressive and hierarchical context fusion. Journal of Building Engineering 75: 106886.

37.

Zamir

Arora

Khan

, et al. (2022) restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, New Orleans, LA, USA, 18–24 June 2022, 5718–5729.

38.

Zhang

Sabuncu

(2018) Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montréal, Canada, 32: 8792–8802.

39.

Zhang

Shao

Wang

(2023) Unifying transformer and convolution for dam crack detection. Automation in Construction 147: 104712.

40.

Zhao

Shi

, et al. (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017, 2881–2890.

41.

Zhou

Zhang

Gong

(2023) Hybrid semantic segmentation for tunnel lining cracks based on Swin Transformer and convolutional neural network. Computer-Aided Civil and Infrastructure Engineering 38(17): 2491–2510.

42.

Zoubir

Rguig

El Aroussi

, et al. (2024) Pixel-level concrete bridge crack detection using convolutional neural networks, gabor filters, and attention mechanisms. Engineering Structures 314: 118343.