A deep neural network-based vehicle re-identification method for bridge load monitoring

Abstract

The accurate tracking of vehicle loads is essential for the condition assessment of bridge structures. In recent years, a computer vision method that is based on multiple sources of data from monitoring cameras and weight-in-motion (WIM) systems has become a promising strategy in bridge vehicle load identification for structural health monitoring (SHM) and has attracted increasing attention. The implementation of vehicle re-identification, namely, the identification of the same vehicle from images that were captured at different locations or time instants, is the key topic of this study. In this study, a vehicle re-identification method that is based on HardNet, a deep convolutional neural network (CNN) specialized in picking up local image features, is proposed. First, we obtain the vehicle point feature positions in the image through feature detection. Then, the HardNet is employed to encode the point feature image patches into deep learning feature descriptors. Re-identification of the target vehicle is achieved by matching the encoded descriptors between two images, which are robust toward scaling, rotation, and other types of noises. A comparison study of the proposed method with three published vehicle re-identification methods is performed using vehicle image data from a real bridge, and the superior performance of our proposed method is demonstrated.

Keywords

vehicle re-identification feature descriptor computer vision deep learning bridge load monitoring

Introduction

The primary function of bridge structures is to carry vehicle loads (e.g., heavy trucks and cars); therefore, vehicle loads are the primary loads of bridges. The determination of vehicle loads is one of the research foundations of time-varying condition assessment, remaining life prediction, and ultimate bearing capacity calculation of bridge structures (Chen et al., 2019; Lan et al., 2011). The installed weight-in-motion (WIM) system can provide vehicle weight information as vehicles pass a measurement site, while a monitoring camera system installed along the bridge records the real-time position information of the vehicles. The integration of the above two aspects of information is a promising strategy for bridge vehicle load identification in structural health monitoring (SHM) and has attracted increasing attention in recent years. Vehicle re-identification is the key step of this technique. The objective of vehicle re-identification is to quickly recognize and track the position of target vehicles from images, which were captured at various time instants or by different cameras, by establishing vehicle matching relationships between images. In the SHM community, Chen et al. (2016) were the first to combine monitoring cameras and WIM systems for the estimation of bridge vehicle loads, in which grayscale features, edge features, and image centroids were employed to identify the same vehicle in images that were captured by different monitoring cameras. However, these features are sensitive to illumination changes; hence, the recognition accuracy for various scenarios cannot be guaranteed. Dan et al. (2019) assumed that all cameras on a bridge are aligned over time and that the fields of view (FOVs) of adjacent cameras overlap, and they employed a target handoff algorithm under the overlapped field of view to perform vehicle re-identification, which is only applicable for these special cases. Zhou et al. (2020) presented a method that identifies only nine vehicle types (e.g., sedan car, mini bus, and large truck) on a bridge and set a representative load value for each type of vehicle, which rendered the bridge vehicle load identification relatively inaccurate. Therefore, the exploration of highly accurate vehicle re-identification methods is important for the identification of the vehicle loads of bridges.

Recently, SHM based on computer vision has attracted extensive attention in civil engineering, such as visual inspection and damage detection (Gao and Mosalam, 2018; Morgenthal and Hallermann, 2014), crack width measurement (Coca et al., 2020; Yang et al., 2019), and dynamic displacement measurement (Dong and Catbas, 2019; Dong et al., 2020). The vehicle re-identification based on computer vision in the large-scale traffic monitoring scene can be regarded as a problem of the nearest neighbor search for repetitive image retrieval (Mei et al., 2014), is of substantial significance in intelligent traffic (Zhang et al., 2011), urban computing (Zheng et al., 2014), and road travel time estimation (Kwong et al., 2009). The image-based vehicle re-identification methods can be divided into two main categories: metric learning and feature representation-based methods (Zhu et al., 2019).

Metric learning (Xing et al., 2002), which is also known as distance metric learning or similarity learning, aims at learning a metric space in which the samples of the same class and of different classes are close to and far away from each other, respectively. It is a commonly applied method in computer vision tasks such as image retrieval (Lee et al., 2008), pedestrian re-identification (Yu et al., 2017), and face recognition (Nguyen and Bai, 2010). Guo et al. (Guo et al., 2018) proposed a coarse-to-fine ranking loss for metric learning under which images of the same vehicle as close as possible, which was used to discriminate between images of different vehicles and of vehicles from different vehicle models. To solve the problems of intra-class differences and inter-class similarities, Bai et al. (2018) designed a group-sensitive triplet embedding method that uses an end-to-end approach for metric learning. The method can focus on the details of a vehicle and can achieve satisfactory re-identification performance, but the training of the model is unstable, and the convergence speed is slow.

For the feature representation-based methods, the employed features can be divided into handcrafted features and deep learning features according to the strategy of feature acquisition. Handcrafted features are extracted according to manual design rules; therefore, they are interpretable. Lowe (2004) proposed the scale invariant feature transformation (SIFT) method, which is robust against image rotation, scaling, brightness changes, viewing angle changes, and affine transformation, but it is time consuming. The method performs feature detection in the difference of Gaussian scale-space pyramid to obtain the position and scale of a point feature. Then, the gradient information of the 16 × 16 neighborhood that is centered on the point feature is transformed into a 128-dimensional descriptor for feature matching. Leutenegger et al. (2011) developed the binary robust invariant scalable keypoints (BRISK) method, which can perform as well as SIFT but has faster processing speed. This method employs the adaptive and generic accelerated segment test (AGAST) algorithm to quickly locate point features in the scale-space pyramid. Then, the sampling points are obtained by sampling concentric circles in the point feature neighborhood, the gray values are compared between the sampling points, and finally, a cascaded binary bit string is obtained for describing each point feature. However, the performance of the handcrafted feature-based re-identification method depends on the prior knowledge and parameter adjustment of the handcrafted feature designer. Moreover, the generalization capability of the model is poor. Liu et al. (2016a) proposed the bag of words with scale-invariant feature transform (BOW-SIFT) and bag of words model with color name features (BOW-CN) methods, which are based on texture and color features, respectively. Chen et al. (2009) presented a vehicle retrieval system that is based on vehicle color and multi-instance learning in a complex background, which is robust against car orientation and color changes. Brown (2010) obtained color features by using weighted and quantization interval color histograms for the retrieval of vehicles. Zapletal and Herout (2016) employed the color histogram and the histogram of oriented gradients (HOG) features with linear regression to perform vehicle re-identification.

Deep neural networks, especially convolutional neural networks (CNNs), have injected new vitality into the development of feature representation-based vehicle re-identification methods. The image features that are extracted by deep neural networks such as GoogLeNet, AlexNet, and VGGNet (He et al., 2016; Krizhevsky et al., 2012; Simonyan and Zisserman, 2014) have been introduced into vehicle re-identification in recent years. Liu et al. (2016a) proposed a fusion of attributes and color features (FACT) method, which combines BOW-SIFT texture features, BOW-CN color features, and GoogLeNet deep learning features with multimodal features to analyze image similarity for vehicle re-identification. Subsequently, Liu et al. (2016b) added license plate verification information and spatiotemporal correlation information to the FACT method and proposed a progressive vehicle re-identification (PROVID) approach based on a deep neural network. Tang et al. (2017) proposed a multimodal learning architecture in which LBP and BOW-CN are used directly as the input of the neural network. After learning by a multilayer perceptron and CNN, they are fused to produce a robust and recognizable feature representation for vehicle re-identification. The main drawback of the deep learning features used in the current vehicle re-identification methods is that they are often obtained by training the classification task with the entire vehicle image as input; hence, these methods cannot fully extract the local information, which is important for distinguishing vehicles. In addition, compared to handcrafted features such as SIFT, these deep learning features are more sensitive to image rotation and scaling.

In recent years, the study of deep learning-based local image feature descriptors, such as HardNet (Mishchuk et al., 2017), has provided a powerful tool for feature-based computer vision research. In contrast to deep learning features mentioned above, HardNet feature possesses satisfactory scale and rotation invariance properties, which is realized by model training using image patches instead of the entire images; thereby, the local information of images can be fully extracted. As a CNN-based local image feature descriptor, HardNet has demonstrated excellent feature description capabilities in a wide range of tasks, such as wide baseline stereo, patch verification, and instance retrieval benchmarks (Mishchuk et al., 2017).

In this study, a HardNet deep learning descriptors-based vehicle re-identification method is proposed, which is highly accurate and robust for re-identifying individual vehicles. Compared with the other existing methods, it can re-identify individual vehicles rather than simple vehicle types, and possesses less restriction on the camera placements along the bridge. First, it obtains the point feature positions in a vehicle image through a feature detection method. Then, the HardNet network is employed to encode the point feature image patches into robust deep learning feature descriptors. Finally, the matching relationship of point features between two images is established through the descriptors, and the target vehicle is re-identified.

This article is organized as follows. In the section Vehicle Re-identification Method Based on HardNet Features, the proposed vehicle re-identification method, which is based on the HardNet local feature descriptor, is presented. Then, the vehicle image data from a real bridge are used to evaluate the performance of the proposed method, and a comparison study with two vehicle re-identification methods that are based on handcrafted features is performed in the section Validation Test. Finally, the conclusions of this study are presented in the section Conclusions.

Vehicle re-identification method based on HardNet features

Point features are prominent points in an image that do not easily change under influencing factors such as light, affine transformation, and noise. Point features include corner points, edge points, bright spots in dark areas, and dark points in bright areas. When object occlusion occurs or an image has readily observable scale and slight changes, image matching can be still achieved effectively by employing these point features. In the proposed vehicle re-identification method, image matching is the core step, which is used to establish a vehicle point feature matching between two images.

The point feature-based image matching procedure can be divided into three steps: feature detection, feature description, and feature matching. In feature detection, the point features in an image, which are to be matched with those in other images, are identified. Then, in feature description, the detected point features and their surrounding areas (small image patches) are encoded into a compact and stable descriptor. Finally, feature matching is performed to search for possible matching pairs of point feature descriptors in two images.

To realize satisfactory matching performance, the descriptors of different point features should be sufficiently distinguishable. Meanwhile, the descriptors should be robust against angle changes, scale changes, and brightness changes. Traditional handcrafted feature descriptors are typically designed artificially for low-level features (such as pixel intensity gradients) through biological knowledge or professional experience, which are easily affected by image transformations (e.g., scale changes, brightness changes, and viewing angle changes); thus, these descriptors inevitably suffer from information loss. In this study, a HardNet descriptor is introduced, which is a data-driven deep learning local feature descriptor that can fully extract high-level local features. These high-level features are robust against image changes and, hence, can produce excellent description performance.

Feature detection

For point feature detection, to ensure the scale invariance of image point features, we construct an image scale-space pyramid with inspiration from the BRISK algorithm (Leutenegger et al., 2011). The image scale-space pyramid consists of n octave layers $c_{i}$ and n intra-octave layers $d_{i}$ with multiple scales, i = {0, 1…, n−1}, typically n = 4, where each intra-octave layer $d_{i}$ lies between octave layers $c_{i}$ and $c_{i + 1}$ . In this scale-space pyramid, each octave layer $c_{i}$ is obtained by down-sampling image $c_{i −1}$ by a factor of 2, while each intra-octave layer $d_{i}$ is obtained by down-sampling $d_{i −1}$ by a factor of 2, except for the first intra-octave layer $d_{0}$ , which is obtained by down-sampling the original image $c_{0}$ by a factor of 1.5, as illustrated in Figure 1 (Leutenegger et al., 2011).

Figure 1.

Scale-space pyramid (the squares denote pixels; the number of pixels on the side of octave c_i is twice that of c_i+1, and the same holds for intra-octaves d_i and d_i+1).

Therefore, the scale t for layers $c_{i}$ and $d_{i}$ is expressed as

t (c_{i}) = 2^{i} t (d_{i}) = 2^{i} \cdot 1.5

(1)

After the image scale-space pyramid was constructed, the candidate point feature regions and the salient areas of each layer should be detected. First, the pixel to be detected is selected as the center pixel, and the detection radius is set to be 3. Then, all 16 pixels are selected, including the circle, as illustrated in Figure 2 (Rosten and Drummond, 2006). The AGAST algorithm (Mair et al., 2010), which can quickly detect point features in various images and adapt to environmental changes, is employed to detect whether the gray values of N (typically N = 9) consecutive pixels among the 16 pixels are greater than

I_{p} + s

or less than

I_{p} - s

, where

I_{p}

and s are the gray value of the pixel to be detected and a selected threshold value, respectively. If the above condition is satisfied, the center pixel will be darker or brighter than its neighborhood, which, therefore, can be considered as a candidate point feature.

Figure 2.

Feature detection in an image patch.

In the next step, to obtain the most stable point feature in a region, non-maxima suppression (Leutenegger et al., 2011) is performed on all candidate point features (pixels) to remove redundant features which are not brightest or not darkest in their neighborhoods. As a result, the function value V of each candidate point feature, which represents image saliency, will be calculated

V = \max (\sum_{x \in S_{bright}} | I_{x} - I_{p} | - s, \sum_{x \in S_{dark}} | I_{p} - I_{x} | - s)

(2)

where

S_{bright} = {x | I_{x} \geq I_{p} + s}

S_{dark} = {x | I_{x} \leq I_{p} - s}

, and

I_{x}

represent the gray value of pixel

x

on the circumference of the circle. If the value V of the point feature is the largest in the neighborhood, it will be kept; otherwise, this point feature will be removed. This process is repeated for all candidate point features, and the final retained point features are the most salient (the darkest or brightest) one in the region.

Based on this procedure, we can obtain the point feature positions in each layer of the scale-space pyramid. Since the saliency of the image is continuous not only on the image but also in the scale dimension, the subpixel and continuous scale refinement are performed on each detected position. First, the V values of 3 × 3 patches that surround these point features are fitted with 2D quadratic functions to find the subpixel maximum V values. Next, these values are used to fit a 1D parabola (see Figure 1) to yield the final value estimate, which is the maximum value of the 1D parabola, and its corresponding scale is the determined scale. Finally, the image coordinates are re-interpolated between the patches in the layers next to the determined scale, and the precise coordinates of the image point features are obtained, as presented on the right side of Figure 1. Next, the HardNet descriptor will be introduced for feature description.

Feature description based on the HardNet deep convolutional neural network

The goal of HardNet (Mishchuk et al., 2017) is to learn an excellent feature descriptor, which can fully describe the information of local areas of an image. This is achieved by training the network with a specially selected set of descriptor triplets that contains matching descriptors which are originated from the same point feature area and the closest non-matching descriptor which is originated from another. And the loss function of HardNet aims at maximizing the distance between the closest positive (matching descriptors) and closest negative samples (non-matching descriptors) in a batch.

The structure of HardNet is illustrated in Figure 3. It consists of six consecutive convolutional layers with 3 × 3 convolution kernels and one convolutional layer with 8×8 convolution kernels. Except for the last layer, all convolutional layers are padded with zeroes to preserve the spatial size information. Meanwhile, a batch normalization layer and an ReLU non-linear layer are added after each convolutional layer. It is found that the pooling layer will reduce the performance of the descriptor; hence, no pooling layer is present in the HardNet network structure, and regularization with a dropout rate of 0.3 is applied before the last convolutional layer. In our method, the input to the network is $32 \times 32$ pixels grayscale patches that have been normalized by subtracting the mean and dividing by the standard deviation of the intensity per patch. The output of the network is a 128-dimensional feature descriptor that has been normalized by the L₂-norm.

Figure 3.

HardNet deep convolutional neural network structure.

For the training of HardNet, the data of small image patches are employed, each of which contains a point feature. The loss function is defined as

Loss = \frac{1}{L} \sum_{i = 1 \dots L} \max {0, 1 + d (a_{i}, p_{i}) - \min [d (a_{i}, p_{j_{\min}}), d (a_{k_{\min}}, p_{i})]}

(3)

where

a_{i} \in ℝ^{N}

and

p_{i} \in ℝ^{N}

represent descriptors of the ith pair of matching local image patches

A_{i}

and

P_{i}

, respectively, from two images, and

d (•)

is the Euclidean distance between two descriptors

d (a_{i}, p_{i}) = {‖ a_{i} - p_{i} ‖}_{2} = \sqrt{\sum_{n = 1}^{N} {(a_{i, n} - p_{i, n})}^{2}}

(4)

According to the loss function in equation (3), corresponding to each pair of matching local image patches, a descriptor triplet

(a_{i}, p_{i}, p_{j_{\min}})

(a_{i}, p_{i}, p_{k_{\min}})

is generated. The sampling process for obtaining this descriptor triplet is illustrated in Figure 4. First, a batch of L pairs of matching local image patches

{A_{i}, P_{i}}_{i = 1 \dots L}

is input into the network to produce feature descriptors

{a_{i}}_{i = 1 \dots L}

and

{p_{i}}_{i = 1 \dots L}

, which correspond to

{A_{i}}_{i = 1 \dots L}

and

{P_{i}}_{i = 1 \dots L}

, respectively. The Euclidean distance calculation is performed between each pair of feature descriptors, and an

L \times L

distance matrix is obtained. Then, the elements of the distance matrix are searched to find the closest non-matching descriptors (shown in red)

p_{j_{\min}}

and

a_{k_{\min}}

to the two elements of each pair of descriptors

(a_{i}, p_{i})

(shown in green), where

j_{min} = argmi n_{j = 1 \dots L, j \neq i} d (a_{i}, p_{j})

and

k_{min} = {argmin}_{k = 1 \dots L, k \neq i} d (a_{k}, p_{i})

Figure 4.

Sampling process for obtaining a descriptor triplet in the loss function.

Finally, L descriptor triplets $(a_{i}, p_{i}, p_{j_{\min}})$ or $(a_{i}, p_{i}, p_{k_{\min}})$ are fed into the loss function in equation (3). By using the loss function above, the training of the HardNet network can achieve the goal that the descriptors of a set of positive samples, originating from the matching point feature pairs, are as close as possible in terms of minimizing $d (a_{i}, p_{i})$ , while the descriptors of a set of negative samples (i.e., originating from non-matching point features) are sufficiently distinct in terms of maximizing $\min [d (a_{i}, p_{j_{\min}}), d (a_{k_{\min}}, p_{i})]$ .

In the illustrative application in the next section, the training of the HardNet model is performed on the combination of the UBC Phototour (Brown and Lowe, 2007) dataset and the HPatches (Balntas et al., 2017) dataset, which can ensure the richness and diversity of the training data. The stochastic gradient descent optimization algorithm is employed for network optimization. The basic learning rate, momentum, and weight decay are set to 10, 0.9, and 0.0001, respectively, and the learning rate decays linearly. The network weights are initialized orthogonally, with a gain of 0.6 and a deviation of 0.01. All hyper-parameters above are determined according to the default settings in (Mishchuk et al., 2017).

After the training of the HardNet network model, point feature description is realized by the input and output relationship. Based on the HardNet descriptor, point feature matching is investigated in the next section.

Feature matching based on the HardNet descriptor

To match HardNet descriptors in two images, a two-step feature matching procedure is employed in this study. The brute force matching method and SIFT matching criteria (Lowe, 2004) are first combined to find possible matching relationships. Then, considering that there may be a false match in the matching relationships, the random sample consensus (RANSAC) (Fischler and Bolles, 1981) algorithm is utilized to remove those outliers.

In the first step, brute force matching is performed: a HardNet point feature descriptor $a_{1}$ is selected in the first image, and the Euclidian distance is calculated using equation (4) between $a_{1}$ and HardNet descriptor ${b_{i}}$ of all detected point features in the next image. Then, the first two descriptors $b_{1}$ and $b_{2}$ with the smallest distance from $a_{1}$ are retained. Next, the SIFT matching criterion is applied to filter some false positive matches

I f \frac{d (a_{1}, b_{1})}{d (a_{1}, b_{2})} < r, a c c e p t (a_{1}, b_{1}), e l s e d i s c a r d

where r is a predefined ratio threshold. The value of r should not be excessively large, since it may produce false negatives; however, a very small value of the ratio threshold r may lead to filtering out correctly matching point feature pairs. We will select the r value as 0.65 in our illustrative example later. The procedure will be repeated for all descriptors in the first image to produce all matching relationships between the two images.

To further increase the matching accuracy, the RANSAC (Fischler and Bolles, 1981) algorithm, which is an outlier removing algorithm for geometric transformation and estimation of multiple view relationship between corresponding point features, is employed to remove mismatched relationships by learning a feature matching model between images.

In the RANSAC algorithm, as the definition of a matching model M, the homography matrix H of a group of point feature pairs is inferred from four non-collinear point feature pairs that are randomly selected from the matching pair set. The matrix H describes the projection mapping relationship between the two images

s_{c} [\begin{array}{l} x^{'} \\ y^{'} \\ 1 \end{array}] = H [\begin{array}{l} x \\ y \\ 1 \end{array}] = [\begin{array}{l} h_{11} h_{12} h_{13} \\ h_{21} h_{22} h_{23} \\ h_{31} h_{32} h_{33} \end{array}] [\begin{array}{l} x \\ y \\ 1 \end{array}]

(5)

where

{[x, y, 1]}^{T}

and

{[x^{'}, y^{'}, 1]}^{T}

are the pixel coordinates of the point features on the original and target images, respectively, in the homogeneous coordinate system and s_c is the scale parameter. After the establishment of the matching model M, the reprojection error (Andrew, 2001) can be estimated for each point feature pair, which is formulated as

Re = \sum_{i = 1}^{np} [{({x^{'}}_{i} - \frac{h_{11} x_{i} + h_{12} y_{i} + h_{13}}{h_{31} x_{i} + h_{32} y_{i} + h_{33}})}^{2} + {({y^{'}}_{i} - \frac{h_{21} x_{i} + h_{22} y_{i} + h_{23}}{h_{31} x_{i} + h_{32} y_{i} + h_{33}})}^{2}]

(6)

where np is the number of point feature pairs. The reprojection error describes how well the pixel coordinates of the target image fit the model output of those of the original image. If there are mismatched point feature pairs, the reprojection errors for these point feature pairs should become significantly large, and these error outliers can be employed to eliminate mismatched point pairs.

Vehicle re-identification

The HardNet feature representation-based image matching method establishes vehicle point feature matching relationships between two vehicle images. In the proposed method, the number of matching point features is the basis for judging whether the vehicle to be identified is coincided with the target vehicle. However, difficulties can be encountered in determining a gap between vehicles that effectively should be considered matching vehicles and other vehicles. Corresponding to each target vehicle, a sequence of vehicles is arranged in descending order of the numbers T of the point feature matching pairs with the target vehicle. The numbers of matching point feature pairs are examined (in descending order), and the threshold T_n is selected as the sudden decrease point in T. If the number of matching point feature pairs satisfies T > T_n, the vehicle to be re-identified will be considered to coincide with the target vehicle; otherwise, it is considered an interfering vehicle. The steps for the proposed vehicle re-identification method are summarized as follows, and a flowchart of the image matching, which is the key component of the proposed vehicle re-identification method, is presented in Figure 5.

Construct the image scale-space pyramid and detect point features in vehicle images by running the AGAST algorithm.

Input the point feature image patches into the trained HardNet deep neural network to obtain HardNet point feature descriptors.

Use the brute force matching and SIFT matching criteria to initially establish a point feature matching between images. Then, remove the remaining mismatched point feature pairs by running the RANSAC algorithm.

Judge whether the vehicle to be identified coincides with the target vehicle according to whether the number of matching point feature pairs exceeds the selected threshold T_n.

Figure 5.

The flowchart of the image matching based on HardNet network model.

When the monitoring information of WIM at one cross-section of a bridge is available, it is tractable to identify the spatial information of vehicle loads on this bridge for each time instance by combining the WIM data and the video images from cameras along the bridge. Once a vehicle is detected in the region of the WIM location, vehicle detection and segmentation methods can be employed to obtain this vehicle image from the camera video at the location of the WIM, and the associated vehicle weight, which is measured by the WIM, can also be extracted. Then, for a time instant, using the video images that have been captured by various cameras along the bridge, all vehicles can be re-identified and located by running our proposed vehicle re-identification method. In combination with the weight information of each re-identified vehicle, the spatial information of vehicle loads on bridges for this time instant can finally be obtained. The flowchart of bridge vehicle load identification using the proposed vehicle re-identification method is shown in Figure 6.

Figure 6.

The flowchart of bridge vehicle load identification.

Validation test

To evaluate the performance of the proposed method, monitoring image data of vehicles on the Dayang Port Bridge are used. The Dayang Port Bridge is an extra-large bridge in Qidong City, China. It is 817.2 m long and 33.5 m wide and has six lanes in both directions. Since heavy trucks are the main concern for examining the loading conditions and service behaviors of bridges, only vehicles of this type are considered in this illustrative application.

In Figure 7, the images of a target vehicle and three vehicle candidates for re-identification are presented. To match the point features with a focus on the target vehicle, the vehicle detection and segmentation process is conducted on the original image, as shown in Figure 8, which can be simply realized using vehicle segmentation methods (Ren et al., 2016; Zhang et al., 2019).

Figure 7.

Images of the target vehicle and a set of vehicles to be re-identified.

Figure 8.

Detection and segmentation of the target vehicle.

Our proposed image matching method, which is based on the HardNet point feature descriptor, is applied to establish the matching relationships between the target vehicle and each vehicle to be identified, and the results are presented in Figure 9. The lines in the figures represent the matching relationships that have been established between the two images, and the ends of the lines are the corresponding two point features. To show the matching effect more clearly, the images are displayed in grayscale. According to Figure 9, Vehicle 3 has significantly more matching relationship lines (194) with the target vehicles compared with the other two vehicles (0, 3). Therefore, Vehicle 3 can be identified as the same vehicle as the target vehicle, which is consistent with the ground truth information.

Figure 9.

Matching relationships between the target vehicle and all candidate vehicles to be identified.

To evaluate the performance of our proposed method (called the HardNet method), a comparison study with two published point feature-based vehicle re-identification methods that are based on the SIFT feature (Lowe, 2004) (called the SIFT method) and based on the BRISK feature (Leutenegger et al., 2011) (called the BRISK method) is also conducted, as shown in Figures 10–13. Various scenarios are investigated, which include matching of a vehicle front (Figure 10) and rear (Figure 11) during the daytime and matching of a vehicle front (Figure 12) and rear (Figure 13) during the night. Compared with the two published methods, significantly more point feature matching pairs are established by our HardNet method for each case. The BRISK method only produces less than half the numbers of point feature matching pairs as our method (the numbers are 56, 3, 31, and 72 compared with 226, 48, 77, and 183 for the four cases that correspond to Figures 10–13, respectively), while the numbers of point feature matching pairs that are identified by the SIFT method are even smaller (only 9, 3, 9, and 0 for the four cases). This implies that our HardNet method can establish vehicle point feature matching relationships stably and effectively in various scenarios with brightness, viewing angle, and scale changes; therefore, it can substantially outperform the two traditional methods in vehicle re-identification.

Figure 10.

Illustrative example of the matching performance comparison of the three methods using images of vehicle fronts that were captured during the daytime.

Figure 11.

Illustrative example of the matching performance comparison of the three methods using images of vehicle rears that were captured during the daytime.

Figure 12.

Illustrative example of the matching performance comparison of the three methods using images of vehicle fronts that were captured during the night.

Figure 13.

Illustrative example of the matching performance comparison of the three methods using images of vehicle rears that were captured during the night.

To evaluate the accuracy of vehicle re-identification, the indices of mean average precision (mAP) and rank-k (Khan and Ullah, 2019) are introduced. Corresponding to each target vehicle, a sequence of vehicle candidate images for re-identification is arranged in descending order of the number of point feature matching pairs with the target vehicle. For a target vehicle, the average precision is defined as

AP = \frac{\sum_{l = 1}^{h} S (l) \times c (l)}{M_{c}}

(7)

where h and M_c are the total number of vehicle images to be identified and the number of correctly matching images in the sequence, respectively; S(l) is the proportion of correctly matching images among the first l images; and c(l) is defined as follows

c (l) = {\begin{cases} 1,If the l th image is a correctly matching image \\ 0,otherwise \end{cases}

(8)

The average of the AP values for all target vehicles is denoted as mAP

m A P = \frac{\sum_{u = 1}^{V} A P (u)}{V}

(9)

where V is the total number of target vehicles.

Based on the same vehicle image sequence, another index for a target vehicle is defined as

r a n k ‐ k = {\begin{cases} 1,If at least one of the first k images is a correctly matching image \\ 0,otherwise \end{cases}

(10)

where the k values are selected artificially. If there are multiple target vehicles, rank-k refers to the average value over all target vehicles (Khan and Ullah, 2019).

Based on the two indices, the three point feature-based methods above and Chen’s method (Chen et al., 2016), which is based on hybrids of grayscale intensity features, edge features, and image centroids, are applied to the 579 sets of image data (180 target vehicles are considered), and the results in terms of the two indices are tabulated in Table 1. The mAP values of SIFT, BRISK, and Chen’s methods are 32.28%, 77.49%, and 67.54%, respectively, while the index value increases to 93.79% for our proposed HardNet method. Our method also produces substantially larger rank-k values than the other two regardless of whether k is selected as 1, 5, or 10. Our method can produce a rank-k value of 98.89% when k = 1, which indicates that the vehicle candidates that have the most point feature matching pairs with the target vehicle coincide with each other well. However, this is not exactly the case for the other three methods, especially for the SIFT method. Therefore, the learning of the HardNet point feature descriptor leads to satisfactory results, and vehicle re-identification through point feature matching becomes much more reliable. Therefore, the HardNet method realizes the best performance among all methods in terms of the average precision and matching reliability of the results. However, it is worthwhile noting that as the feature matching procedure is time consuming, the computational efficiency of the proposed method is relatively low, and the method cannot be appropriately applied for real-time vehicle re-identification.

Table 1.

Evaluation results of the three methods on real vehicle image data.

Method	Descriptor type	mAP, %	Rank-1, %	Rank-5, %	Rank-10, %
SIFT	Handcrafted descriptor	32.28	46.67	65.00	71.11
BRISK	Handcrafted descriptor	77.49	88.89	96.11	96.11
Chen’s	–	67.54	86.11	89.44	90.00
HardNet	Deep learning descriptor	93.79	98.89	100	100

Conclusions

Vehicle loads are the primary loads of bridge structures, and the identification of vehicle loads is essential for time-varying condition assessment, remaining life prediction, and ultimate bearing capacity calculation of bridges. Aiming at the realization of vision-based vehicle recognition for bridge vehicle load identification together with measurement information of WIM systems, a vehicle re-identification method that is based on monitoring image data and deep learning is proposed in this study. The robust HardNet deep CNN local image descriptor is introduced for establishing the point feature matching relationships between the target vehicle and the vehicle to be identified. Three steps are performed: feature detection, feature description, and feature matching. The performance of the method is evaluated by applying it to vehicle image data sets from a real bridge. It is demonstrated that our proposed method can re-identify vehicles with high accuracy under various scenarios with brightness, viewing angle, and scale changes. Compared with three published methods, our proposed method can identify significantly more matching point features and, thus, produce superior vehicle re-identification results.

When the monitoring information from a weight-in-motion system is available, we can identify the spatial information of vehicle loads on the bridge for each time instant by employing the vehicle re-identification method. In future studies, it would be useful to explore the practicality of the method when the image quality is poor (e.g., when the vehicles are photographed using low resolution camera or in foggy and hazy weather).

Footnotes

Acknowledgments

This research is financially supported by the Open Funding of State Key Laboratory of Safety and Health for In-Service Long Span Bridges (Grant No. 1800008) and the National Natural Science Foundation of China (Grant No. 51778192).

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Li Hui

Yong Huang

References

Andrew

(2001) Multiple view geometry in computer vision. Kybernetes 30(9/10): 1333–1341.

Bai

Lou

Gao

, et al. (2018) Group-sensitive triplet embedding for vehicle reidentification. IEEE Transactions on Multimedia 20(9): 2385–2399.

Balntas

Lenc

Vedaldi

, et al. (2017) HPatches: A benchmark and evaluation of handcrafted and learned local descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii: IEEE, 5173–5182.

Brown

(2010) Example-based color vehicle retrieval for surveillance. In: 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance. Las Vegas, NV: IEEE, 91–96.

Brown

Lowe

(2007) Automatic panoramic image stitching using invariant features. International Journal of Computer Vision 74(1): 59–73.

Chen

Hsieh

, et al. (2009) Vehicle retrieval using eigen color and multiple instance learning. In: 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing. Kyoto, Japan: IEEE, 657–660.

Chen

Bao

, et al. (2016) Identification of spatio-temporal distribution of vehicle loads on long-span bridges using computer vision technology. Structural Control and Health Monitoring 23(3): 517–534.

Chen

Bao

Chen

, et al. (2019) Modelling the spatial distribution of heavy vehicle loads on long-span bridges based on undirected graphical model. Structure and Infrastructure Engineering 15(11): 1485–1499.

Coca

G-L

Romanescu

Ş-C

Botez

Ş-M

, et al. (2020) Crack detection system in AWS cloud using convolutional neural networks. Procedia Computer Science 176: 400–409.

10.

Dan

Yan

(2019) Identification of moving loads based on the information fusion of weigh-in-motion system and multiple camera machine vision. Measurement 144: 155–166.

11.

Dong

C-Z

Catbas

(2019) A non-target structural displacement measurement method using advanced feature matching strategy. Advances in Structural Engineering 22(16): 3461–3472.

12.

Dong

C-Z

Celik

Catbas

, et al. (2020) Structural displacement monitoring using deep learning-based full field optical flow methods. Structure and Infrastructure Engineering 16(1): 51–71.

13.

Fischler

Bolles

(1981) Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6): 381–395.

14.

Gao

Mosalam

(2018) Deep transfer learning for image-based structural damage recognition. Computer-Aided Civil and Infrastructure Engineering 33(9): 748–768.

15.

Guo

Zhao

Liu

, et al. (2018) Learning coarse-to-fine structured feature embedding for vehicle re-identification. Proceedings of the AAAI Conference on Artificial Intelligence 32(1): 6853–6860.

16.

Zhang

Ren

, et al. (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE, 770–778.

17.

Khan

Ullah

(2019) A survey of advances in vision-based vehicle re-identification. Computer Vision and Image Understanding 182: 50–63.

18.

Krizhevsky

Sutskever

Hinton

(2012) Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25: 1097–1105.

19.

Kwong

Kavaler

Rajagopal

, et al. (2009) Arterial travel time estimation based on vehicle re-identification using wireless magnetic sensors. Transportation Research Part C: Emerging Technologies 17(6): 586–606.

20.

Lan

(2011) Traffic load modelling based on structural health monitoring data. Structure and Infrastructure Engineering 7(5): 379–386.

21.

Lee

Jin

Jain

(2008) Rank-based distance metric learning: An application to image retrieval. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, Alaska: IEEE, 1–8.

22.

Leutenegger

Chli

Siegwart

(2011) BRISK: Binary robust invariant scalable keypoints. In: 2011 International Conference on Computer Vision. Barcelona, Spain: IEEE, 2548–2555.

23.

Liu

, et al. (2016a) Large-scale vehicle re-identification in urban surveillance videos. In: 2016 IEEE International Conference on Multimedia and Expo (ICME). Seattle, WA: IEEE, 1–6.

24.

Liu

Mei

, et al. (2016b) A deep learning-based approach to progressive vehicle re-identification for urban surveillance. In: European Conference on Computer Vision. Cham: Springer, 869–884.

25.

Lowe

(2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2): 91–110.

26.

Mair

Hager

Burschka

, et al. (2010) Adaptive and generic corner detection based on the accelerated segment test. In: European Conference on Computer Vision. Berlin, Heidelberg: Springer, 183–196.

27.

Mei

Rui

, et al. (2014) Multimedia search reranking: A literature survey. ACM Computing Surveys 46(3): 1–38.

28.

Mishchuk

Mishkin

Radenovic

, et al. (2017) Working hard to know your neighbor’s margins: Local descriptor learning loss. In: Neural Information Processing Systems. Long Beach, CA: NIPS, 4826–4837.

29.

Morgenthal

Hallermann

(2014) Quality assessment of unmanned aerial vehicle (UAV) based visual inspection of structures. Advances in Structural Engineering 17(3): 289–302.

30.

Nguyen

Bai

(2010) Cosine similarity metric learning for face verification. In: Asian Conference on Computer Vision. Berlin, Heidelberg: Springer, 709–720.

31.

Ren

Girshick

, et al. (2016) Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6): 1137–1149.

32.

Rosten

Drummond

(2006) Machine learning for high-speed corner detection. In: European Conference on Computer Vision. Berlin, Heidelberg: Springer, 430–443.

33.

Simonyan

Zisserman

(2014) Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556.

34.

Tang

Jin

, et al. (2017) Multi-modal metric learning for vehicle re-identification in traffic surveillance environment. In: 2017 IEEE International Conference on Image Processing (ICIP). Beijing, China: IEEE, 2254–2258.

35.

Xing

Jordan

Russell

et al. (2002) Distance metric learning with application to clustering with side-information. Advances in Neural Information Processing Systems 15: 521–528.

36.

Yang

(2019) Rapid detection of building cracks based on image processing technology with double square artificial marks. Advances in Structural Engineering 22(5): 1186–1193.

37.

Zheng

(2017) Cross-view asymmetric metric learning for unsupervised person re-identification. In: Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 994–1002.

38.

Zapletal

Herout

(2016) Vehicle re-identification for automatic video traffic surveillance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Las Vegas, NV: IEEE, 25–31.

39.

Zhang

Wang

F-Y

Wang

, et al. (2011) Data-driven intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems 12(4): 1624–1639.

40.

Zhang

Zhou

Zhang

(2019) A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision. Computer-Aided Civil and Infrastructure Engineering 34(6): 471–487.

41.

Zheng

Capra

Wolfson

, et al. (2014) Urban computing: Concepts, methodologies, and applications. ACM Transactions on Intelligent Systems and Technology 5(3): 1–55.

42.

Zhou

Pei

, et al. (2020) Vehicle weight identification system for spatiotemporal load distribution on bridges based on non-contact machine vision technology and deep learning algorithms. Measurement 159: 107801.

43.

Zhu

Zeng

Huang

, et al. (2019) Vehicle re-identification using quadruple directional deep learning features. IEEE Transactions on Intelligent Transportation Systems 21(1): 410–420.