Bridge inspection is an important step in preserving and rehabilitating transportation infrastructure for extending their service lives. The advancement of mobile robotic technology allows the rapid collection of a large amount of inspection video data. However, the data are mainly the images of complex scenes, wherein a bridge of various structural elements mix with a cluttered background. Assisting bridge inspectors in extracting structural elements of bridges from the big complex video data, and sorting them out by classes, will prepare inspectors for the element-wise inspection to determine the condition of bridges. This article is motivated to develop an assistive intelligence model for segmenting multiclass bridge elements from the inspection videos captured by an aerial inspection platform. With a small initial training dataset labeled by inspectors, a Mask Region-based Convolutional Neural Network pre-trained on a large public dataset was transferred to the new task of multiclass bridge element segmentation. Besides, the temporal coherence analysis attempts to recover false negatives and identify the weakness that the neural network can learn to improve. Furthermore, a semi-supervised self-training method was developed to engage experienced inspectors in refining the network iteratively. Quantitative and qualitative results from evaluating the developed deep neural network demonstrate that the proposed method can utilize a small amount of time and guidance from experienced inspectors (3.58 h for labeling 66 images) to build the network of excellent performance (91.8% precision, 93.6% recall, and 92.7% f1-score). Importantly, the article illustrates an approach to leveraging the domain knowledge and experiences of bridge professionals into computational intelligence models to efficiently adapt the models to varied bridges in the National Bridge Inventory.
XuYBaoYChenJ, et al. Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images. Struct Health Monit2019; 18(3): 653–674.
12.
HoskereVNarazakiYHoangTA, et al. Madnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. J Civil Struct Health Monit2020; 10: 757–773.
13.
HuFZhaoJHuangY, et al. Structure-aware 3D reconstruction for cable-stayed bridges: a learning-based method. Comput Aided Civil Infrastruct Eng2021; 36(1): 89–108.
14.
DongCZCatbasFN.A review of computer vision–based structural health monitoring at local and global levels. Struct Health Monit2020; 20: 692–743.
15.
BaoYTangZLiH, et al. Computer vision and deep learning–based data anomaly detection method for structural health monitoring. Struct Health Monit2019; 18(2): 401–421.
16.
DengJLuYLeeVCS. Imaging-based crack detection on concrete surfaces using you only look once network. Struct Health Monit2020; 20: 484–499.
17.
NarazakiYHoskereVHoangTA, et al. Automated vision-based bridge component extraction using multiscale convolutional neural networks, 2018, https://arxiv.org/abs/1805.06042
18.
NarazakiYHoskereVHoangTA, et al. Vision-based automated bridge component recognition with high-level scene consistency. Comput Aided Civil Infrastruct Eng2020; 35(5): 465–482.
19.
YeumCMChoiJDykeSJ.Automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure. Struct Health Monit2019; 18(3): 675–689.
20.
KangKLiHYanJ, et al. T-CNN: tubelets with convolutional neural networks for object detection from videos. IEEE Trans Circuit Syst Video Technol2018; 28(10): 2896–2907.
21.
ZhuXXiongYDaiJ, et al. Deep feature flow for video recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, 21–26 July 2017, pp. 2349–2358. New York: IEEE.
22.
ZhuXWangYDaiJ, et al. Flow-guided feature aggregation for video object detection. In: Proceedings of the IEEE international conference on computer vision, Venice, 22–29 October 2017, pp. 408–417. New York: IEEE.
LinTYMaireMBelongieS, et al. Microsoft COCO: common objects in context. In: European conference on computer vision, Zurich, 6–12 September 2014, pp. 740–755. New York: Springer.
25.
GirshickRDonahueJDarrellT, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, 23–28 June 2014, pp. 580–587. New York: IEEE.
26.
UijlingsJRVanDeSandeKEGeversT, et al. Selective search for object recognition. Int J Comput Vis2013; 104(2): 154–171.
27.
GirshickR. Fast R-CNN. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015, pp. 1440–1448. New York: IEEE.
28.
HeKGkioxariGDollárP, et al. Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, Venice, 22–29 October 2017, pp. 2961–2969. New York: IEEE.
GaoYMosalamKM.Deep transfer learning for imagebased structural damage recognition. Comput Aided Civil Infrastruct Eng2018; 33(9): 748–768.
31.
ZhangCChangCC. Surface damage detection for concrete bridges using single-stage convolutional neural networks. In: Health Monitoring of Structural and Biological Systems XIII, Volume 10972, Denver, CO, 4–7 March 2019. New York: SPIE.
32.
GopalakrishnanKGholamiHVidyadharanA, et al. Crack damage detection in unmanned aerial vehicle images of civil infrastructure using pre-trained deep learning model. Int J Traffic Transp Eng2018; 8: 1–14.
33.
PapandreouGChenLCMurphyKP, et al. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015, pp. 1742–1750. New York: IEEE.
34.
MittalSTatarchenkoMBroxT.Semi-supervised semantic segmentation with high-and low-level consistency. IEEE Trans Pattern Anal Mach Intel2019; 43: 1369–1379.
35.
TrigueroIGarcíaSHerreraF.Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowl Inform Syst2015; 42(2): 245–284.
36.
HungWCTsaiYHLiouYT, et al. Adversarial learning for semi-supervised semantic segmentation, 2018, https://arxiv.org/abs/1802.07934
37.
MisraIShrivastavaAHebertM. Watch and learn: Semi-supervised learning for object detectors from video. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, 7–12 June 2015, pp. 3593–3602. New York: IEEE.
TianYChengGGelernterJ, et al. Joint temporal context exploitation and active learning for video segmentation. Pattern Recogn2020; 100: 107158.
40.
BeluchWHGeneweinüTNrnbergerA, et al. The power of ensembles for active learning in image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, 18–23 June 2018, pp. 9368–9377. New York: IEEE.
41.
SenerOSavareseS. Active learning for convolutional neural networks: a core-set approach. In: International Conference on Learning Representations, https://openreview.net/forum?id=H1aIuk-RW
GalYIslamRGhahramaniZ.Deep bayesian active learning with image data. In: Proceedings of the 34th international conference on machine learning, Sydney, NSW, Australia, 6–11 August 2017, pp. 1183–1192. New York: ACM.
44.
YangLZhangYChenJ, et al. Suggestive annotation: a deep active learning framework for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, Quebec City, QC, Canada, 10–14 September 2017, pp. 399–407. New York: Springer.
45.
SiddiquiYValentinJNießnerM. ViewAL: active learning with viewpoint entropy for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Seattle, WA, 13–19 June 2020, pp. 9433–9443.
46.
GoodfellowIBengioYCourvilleA.Deep learning. Massachusetts, MA: MIT Press, 2016.
47.
HeKZhangXRenS, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, 27–30 June 2016, pp. 770–778. New York: IEEE.
48.
GhenG.Bridge inspection robot deployment systems (BIRDS). Technical report, Department of Transportation, 2020, https://rip.trb.org/view/1596887
49.
DuttaAZissermanA. The VIA annotation software for images, audio and video. In: Proceedings of the 27th ACM international conference on multimedia MM ’19, Nice, 21–25 October 2019, pp. 2276–2279. New York: ACM.
