The advent of artificial intelligence and deep learning has provided sophisticated functionality for sensor fusion and object detection and classification which have accelerated the development of highly automated and autonomous ships as well as decision support systems for maritime navigation. It is, however, challenging to assess how the implementation of these systems affects the safety of ship operation. We propose to utilize marine training simulators to conduct controlled, repeated experiments allowing us to compare and assess how functionality for autonomous navigation and decision support affects navigation performance and safety. However, although marine training simulators are realistic to human navigators, it cannot be assumed that the simulators are sufficiently realistic for testing the object detection and classification functionality, and hence this functionality cannot be directly implemented in the simulators. We propose to overcome this challenge by utilizing Cycle-Consistent Adversarial Networks (Cycle-GANs) to transform the simulator data before object detection and classification is performed. Once object detection and classification are completed, the result is transferred back to the simulator environment. Based on this result, decision support functionality with realistic accuracy and robustness can be presented and autonomous ships can make decisions and navigate in the simulator environment.
Maritime safety committee (MSC) 99/wp.9 annex 1. Regulatory scoping exercise for the use of maritime autonomous surface ships (MASS). Report of the working group. Technical report, International Maritime Organization (IMO), UK, 2018.
2.
BrandsæterASmefjellGvan de MerweK, et al. Assuring safe implementation of decision support functionality based on data-driven methods for ship navigation. In: Proceedings of the 30th European safety and reliability conference and the 15th probabilistic safety assessment and management conference, Venice, Italy, 1–5 November 2020. Singapore: Research Publishing.
3.
Accident Investigation Board Norway. Part one report on the collision on 8 November 2018 between the frigate HNoMS Helge Ingstad and the oil tanker Sola TS outside the sture terminal in the hjeltefjord in hordaland county. Technical report, Accident Investigation Board Norway, Defence Accident Investigation Board Norway, Norway, 2019.
MislevyRJ. Evidence-centered design for simulation-based assessment. Mil Med2013; 178(10): 107–114.
7.
ZghyerROstnesR. Opportunities and challenges in using ship-bridge simulators in maritime research. In: Proceedings of Ergoship 2019, Haugesund, Western Norway University of Applied Sciences (HVL), 2019.
8.
SchönerHP. “How good is good enough?” in autonomous driving. In: LangheimJ (ed.) Electronic components and systems for automotive applications. Cham: Springer, 2019, pp.119–142.
PoratheT., et al. Human-centred design in the maritime domain. In: DS 85-1: Proceedings of NordDesign 2016, Trondheim, Norway, 10–12 August 2016, vol. 1, pp.175–184.
12.
BekkeheienLMW. Synthesizing photo-realistic images from a marine simulator via generative adversarial networks. Master’s Thesis, Norwegian University of Science and Technology, Norway, 2020.
13.
KoopmanPFerrellUFratrikF, et al. A safety standard approach for fully autonomous vehicles. In: 2019 international conference on computer safety, reliability, and security, Turku, Finland, 10–13 September, pp.326–332. Springer.
14.
SalayRQueirozRCzarneckiK. An analysis of ISO 26262: Using machine learning safely in automotive software. arXiv preprint arXiv:170902435, 2017.
15.
KoopmanPWagnerM. Challenges in autonomous vehicle testing and validation. SAE Int J Transp Saf2016; 4(1): 15–24.
16.
KoopmanPWagnerM. Toward a framework for highly automated vehicle safety validation. SAE technical paper 2018-01-1071, 2018.
17.
WoodMRobbelPMaassM, et al. Safety first for automated driving. Technical report, Aptiv Services US, LLC; AUDI AG; Bayrische Motoren Werke AG; Beijing Baidu Netcom Science Technology Co., Ltd; Continental Teves AG & Co oHG; Daimler AG; FCA US, LLC; HERE Global B.V.;Infineon Technologies AG; Intel; Volkswagen AG., 2019.
18.
BrandsæterAKnutsenKE. Towards a framework for assurance of autonomous navigation systems in the maritime industry. In: HaugenSBarrosAvan GulijkC, et al. (eds) Safety and reliability–safe societies in a changing world. London: CRC Press, 2018, pp.449–457.
19.
BrinkmannMBödeELammA, et al. Learning from automotive: Testing maritime assistance systems up to autonomous vessels. In: OCEANS 2017-Aberdeen, Aberdeen, 19–22 June 2017, pp.1–8. IEEE.
ISO 26262:2018. Road vehicles – functional safety. Standard, International Organization for Standardization, Geneva, CH, 2018.
25.
ISO/PAS 21448:2019(E). Road vehicles—safety of the intended functionality. Standard, International Organization for Standardization, Geneva, CH, 2019.
26.
ANSI/UL 4600. ANSI/UL 4600 standard for safety for the evaluation of autonomous products. Standard, Underwriters Laboratories Inc., 2020.
27.
BishopPBloomfieldR. A methodology for safety case development. Saf Reliab2000; 20(1): 34–42.
28.
RingbomH. Regulating autonomous ships—concepts, challenges and precedents. Ocean Dev Int Law2019; 50(2–3): 141–169.
29.
EndsleyMR. Design and evaluation for situation awareness enhancement. Proc Hum Factors Soc Annu Meet1988; 32(2): 97–101.
30.
EndsleyMR. Situation awareness global assessment technique (SAGAT). In: Proceedings of the IEEE 1988 national aerospace and electronics conference, Dayton, OH, 23–27 May 1988, pp.789–795. IEEE.
31.
EndsleyMR. Direct measurement of situation awareness: validity and use of SAGAT. In: EndsleyMRGarlandDJ (eds) Situation awareness analysis and measurement. Lawrence Erlbaum Associates, 2000, pp.147–173.
32.
ØvergårdKINazirSSolbergAS. Towards automated performance assessment for maritime navigation. TransNav Int J Mar Navig Saf Sea Transp2017; 11(2): 229–234.
33.
KobayashiH. Use of simulators in assessment, learning and teaching of mariners. WMU J Marit Aff2005; 4(1): 57–75.
34.
AdadiABerradaM. Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access2018; 6: 52138–52160.
35.
BrandsæterAGladIK. Explainable artificial intelligence: how subsets of the training data affect a prediction. arXiv preprint arXiv:201203625, 2020.
36.
IsolaPZhuJYZhouT, et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, 21–26 July 2017, pp.1125–1134. IEEE.
37.
ZhuJYParkTIsolaP, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, Venice, Italy, 22–29 October 2017, pp.2223–2232. IEEE.
38.
KangEKooHJYangDH, et al. Cycle-consistent adversarial denoising network for multiphase coronary ct angiography. Med Phys2019; 46(2): 550–562.
39.
SandfortVYanKPickhardtPJ, et al. Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks. Sci Rep2019; 9(1): 1–9.
40.
RenCXZiemannATheilerJ, et al. Cycle-consistent adversarial networks for realistic pervasive change generation in remote sensing imagery. In: IEEE Southwest symposium on image analysis and interpretation (SSIAI), Albuquerque, NM, 29–31March2020, pp.42–45. IEEE.
41.
HoffmanJTzengEParkT, et al. Cycada: cycle-consistent adversarial domain adaptation. In: Proceedings of the 35th international conference on machine learning, Stockholm, Sweden, 10–15 July 2018, pp.1989–1998. PMLR.
42.
de CurtóJDuvallR. Cycle-consistent generative adversarial networks for neural style transfer using data from chang’e-4. arXiv preprint arXiv:201111627, 2020.
43.
GoodfellowIPouget-AbadieJMirzaM, et al. Generative adversarial nets. In: GhahramaniZWellingMCortesC, et al. (eds) Advances in neural information processing systems. Berlin: Springer, 2014, pp.2672–2680.
44.
CurtóJDZarzaICDe La TorreF, et al. High-resolution deep convolutional generative adversarial networks. arXiv preprint arXiv:171106491, 2017.
45.
AntoniouAStorkeyAEdwardsH. Data augmentation generative adversarial networks. arXiv preprint arXiv:171104340, 2017.
46.
ZhuJYKrähenbühlPShechtmanE, et al. Generative visual manipulation on the natural image manifold. In: European conference on computer vision, Amsterdam, The Netherlands, 8–16 October 2016, pp.597–613. Cham: Springer.
47.
WangXGuptaA. Generative image modeling using style and structure adversarial networks. In: European conference on computer vision, Amsterdam, The Netherlands, 8–16 October 2016, pp.318–335. Cham: Springer.
48.
RadfordAMetzLChintalaS. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:151106434, 2015.
49.
OdenaAOlahCShlensJ. Conditional image synthesis with auxiliary classifier gans. In: Proceedings of the 34th international conference on machine learning, International Convention Centre, Sydney, 6–11 August 2017, pp.2642–2651. PMLR.
50.
LiuMYBreuelTKautzJ. Unsupervised image-toimage translation networks. In: Advances in neural information processing systems, Long Beach, CA, December 2017, pp.700–708. Red Hook, NY: Curran Associates Inc.
51.
HuangXLiuMYBelongieS, et al. Multimodal unsupervised image-to-image translation. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018, pp.172–189. Cham: Springer.
52.
LeeHYTsengHYHuangJB, et al. Diverse image-to-image translation via disentangled representations. In: Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018, pp.35–51.
FuHGongMWangC, et al. Geometry-consistent generative adversarial networks for one-sided unsupervised domain mapping. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, 15–20 June 2019, pp.2427–2436. IEEE.
55.
TripathySKannalaJRahtuE. Learning image-to-image translation using paired and unpaired training samples. In: Asian conference on computer vision, Perth, WA, 2–6 December 2018, pp.51–66. Cham: Springer.
