A fast recognition algorithm of ship hatch in bulk cargo terminal based on point cloud contour extraction

Abstract

Port loading automation systems can improve the efficiency of cargo transfer, save port operation time and create greater economic benefits. The recognition of ship hatch is the basis and premise of building an automatic loading system, it is a major time cost in the loading system meanwhile. How to identify the hatch quickly and accurately is an important problem that needs to be solved urgently under the actual production needs of ports. In order to save the time of ship hatch recognition, this paper proposes a fast hatch recognition algorithm based on point cloud contour extraction. The ship point cloud model generated by lidar scanning is preprocessed to remove the noise and isolated points in the model. Projecting the preprocessed point cloud on the XOY plane, converting the three-dimensional point cloud into a two-dimensional image, extracting the outline further getting the point cloud pixels with linear features of the two-dimensional image by $α - shape$ algorithm. Projecting the feature point cloud to the X-axis to classify the hatches. According to the center point of each class of point cloud, searching for the nearest neighbor hatch edge feature points to calculate hatch coordinates in the world frame. An experimental study was carried out on the scan data of actual docked ships at Guoneng Tianjin Port. The results show that the algorithm can realize the hatch recognition quickly which the speed is increased by 424% compared with the previous algorithm and the identification accuracy meets the actual production needs of the port.

Keywords

port loading automation ship hatch recognition plane projection point cloud contour extraction

Introduction

Ocean shipping plays a pivotal role in cargo transportation with low cost and huge throughput when the transoceanic transportation of huge volumes of cargo can only be done by it especially. Port is the hub of marine transportation which realizes the conversion of land and sea. It is the eternal pursuit of the port to continuously improve its production efficiency and create higher economic benefits. Guoneng Tianjin Port Corporation, which production and operation field is coal, has a series of complex transportation processes from coal entering the port to leaving the port, of which the most important and complex is the loading link. It needs to improve the efficiency of loading predominantly to increase the production capacity of the port which directly limits the throughput level of the port’s cargo. Due to the complex operating environment of bulk cargo ports and the various types of industrial equipment involved, most of the industrial equipment in bulk ports is still manually operated, which greatly limits the efficiency of cargo transfer. It is an effective way to improve production efficiency and create greater economic benefits to establish the automatic loading system^1–5 and carry out loading operation continuously and stably. Ship hatch identification is the first link of loading operation in the automatic ship loading system which is the foundation and prerequisite. The hatch recognition program calculates the hatch coordinates and feeds them back to the ship loader motion control system for coal loading operations in real-time. The speed of recognition directly restricts the efficiency of coal loading and affects the coal output of the port. It is imperative to establish a set of fast and accurate hatch identification procedures which is a requirement of actual port production.

Lidar is an important sensing device in the automation system of ship loading^6–8 as it can provide stable ship scanning data and is not affected by time, light, and weather conditions which meets the needs of port operations. Compared with single-line lidar, multi-line lidar data is more abundant and more robust. It can not only ensure the stability of point cloud stitching but also provide a larger viewing angle in a static state, which is convenient for subsequent real-time hatch identification. There has been some certain research foundation for the identification of ship hatches through the processing of lidar data.^9–11

Mi⁹ proposed a fast ship identification algorithm based on the ship point cloud generated by line lidar scanning. By projecting the ship point cloud into the XOY plane, the point cloud pixels are counted along the X-axis and the Y-axis separately for the hatch recognition. But its research object is limited to the point cloud of whole ship scanning, which is not a common process in actual production. Taking Guoneng Tianjin Port as an example, it takes more than 10 min to perform a full ship scan, which affects the output of thousands of $tons$ of coal, and the entire port will affect the production of tens of millions of $tons$ of coal throughout the whole year. Secondly, when the ship loader is stationary or the moving range is small, it is impossible to identify the hatch based on the single-line lidar point cloud used in this paper, and the real-time performance is out of the question. Furthermore, the volume of the point cloud of single-line lidar is much smaller than surface lidar, and the number of lidars used in the paper is less, which can get a higher recognition speed. When the area lidar point cloud is used as the research object for hatch recognition, the recognition speed is difficult to achieve. Finally, when the lidar is unable to scan the whole ship, the algorithm is easy to fail when dealing with the defect point cloud.

Xi-Liang et al.¹⁰ is also processing single-line lidar data. The point cloud is projected onto the XOY plane and transformed into a two-dimensional image. A variable size rectangular box is used as a seed searching in the two-dimensional image and extracting rectangular features. This method is suitable for a more regular point cloud, and the determination of search starting point will have greater randomness for the irregular point cloud. At the same time, when using the matrix operator to find the starting point, it needs to traverse the whole point cloud data while the whole hatch identification process takes too much time when the volume of the point cloud is large.

Miao¹¹ transforms the real-time point cloud in the operation area into a two-dimensional image, uses the $canny$ operator for edge extraction, and then fits the hatch edge line to achieve high-precision hatch recognition. The algorithm has good stability and excellent hatch recognition accuracy. In most cases, the hatch recognition error is not more than 0.3 $m$ . However, this algorithm is based on processing a single-frame point cloud. For the whole ship point cloud model, it needs to be recognized from ship beginning to ship end which will take a lot of time.

These algorithms all process the ship’s point cloud or the ship’s point cloud projected on a two-dimensional plane. When the merging point cloud model obtained is huge, it greatly limits the speed of hatch recognition. For this reason, this article considers that avoiding the overall point cloud, but choosing to process the point cloud outline, can greatly reduce the volume of the point cloud and further improve the speed of hatch recognition.

In order to ensure the port production efficiency and improve the ship automation system, this paper proposes a fast identification algorithm of ship hatch. $α - shape$ algorithm is used to process the point cloud projected on the XOY plane to get the outline of the point cloud in the plane. Extracting straight-line feature in the contour point cloud, and then it is projected on the X-axis to cluster the hatch point cloud. According to the cluster center, the search area is designed to determine the hatch feature points of each cluster, and the hatch edge recognition is realized. The algorithm can not only recognize the hatch of the whole ship point cloud, but also the real-time point cloud. At the same time, when the hatch point cloud is defective, the algorithm has a certain recognition effect and fault tolerance. The algorithm can recognize the hatch quickly and the accuracy which meets the requirements of the port automatic loading system.

Point cloud modeling

Before the ship hatch recognition, the lidar’s three-dimensional scan model of the ship must be obtained firstly. The experimental site of this article is the Tianjin Port Terminal, and the research objects are mainly but not limited to the Tianjin Port Shenhua series ships. Figure 1 shows the real 3D model of the Tianjin Port Wharf, in which the yellow mechanism is the model of the ship loader, and the ship model is one of the Shenhua series ships of Tianjin Port. As shown in Figure 1, the world coordinate system OXYZ is established, and the coordinate origin O is established at the starting point of the port berth. Taking the center of the ship loader guide rail as the X-axis, and facing the left side of the ship loader as the positive direction. The Y-axis and the X-axis are in the same plane of the port shore which the direction perpendicular to the X-axis pointing to the ship is the positive direction. And the Z-axis is perpendicular to X-Axis and Y-axis, in the form of a $right - handed$ coordinate system with the positive direction pointing to the sky. The real-time position of the ship loader is collected by the encoder, including the position of the ship loader on the X-axis, the pitch angle of the boom, the telescopic distance of the telescopic boom, and the swing angle of the slide.

Figure 1.

3D model of Tianjin Port Wharf.

Six LIVOX horizon lidars are fixed on the ship loader as the splicing of ship point clouds with the specific location as shown in Figure 1. Each lidar scans the ship’s area in real-time in the form of a fixed angle of view. Through calibrating the attitude matrix of the lidar in advance, the transformation matrix from lidar coordinates to the world coordinate system is obtained in combination with the position information of the ship loader measured by the encoder. Under normal working conditions, the ship loader is moving left or right with a constant speed of 1 $m / s$ . When the ship loader moves from the bow to the stern, the real-time point cloud data of Livox and the real-time encoder data of the ship loader are merged to generate a three-dimensional point cloud model of the entire ship. The splicing process of the point cloud is shown in formula 1 where $i$ represents the point cloud data of the $i$ -th frame, the callback frequency of the lidar is set to 1 $hz$ , that is, one frame is called back per second. $P_{i}$ is the point cloud where six lidars’ data in the $i$ -th frame are fused into the ship loader coordinate system. $P_{accu}$ is cumulative splicing data of point cloud in the world coordinate system, $n$ means that the lidar stops merging at the $n$ -th frame. $R_{encode r_{i}}$ is the transformation matrix from the ship loader coordinate system to the world coordinate system corresponding to the point cloud in the $i$ -th frame which value is determined by the real-time encoder data, see formula 2. In formula 2, $x_{encode r_{i}}$ is the position of the ship loader on the X axis, $y_{encode r_{i}}$ , $z_{encode r_{i}}$ is the position of the ship loader on the Y axis and the Z axis. There are two constant numbers normally. When the ship loader completes the movement from the bow to the stern, the total splicing point cloud of the ship is shown in Figure 2 which is the merging result of the Shenhua 512 ship at Tianjin Port Wharf where the green part is the lidar scan data. It can be seen from Figure 2 that the three-splicing model of the whole ship point cloud is favorable, and the merging effect directly affects the accuracy of hatch recognition. In the previous experimental test, the splicing accuracy of the point cloud was within 10 $cm$ , which fully meets the accuracy of hatch recognition.

P_{accu} = \sum_{i = 1}^{n} R_{encode r_{i}} P_{i}

(1)

R_{encode r_{i}} = (\begin{matrix} 1 & 0 & 0 & x_{encode r_{i}} \\ 0 & 1 & 0 & y_{encode r_{i}} \\ 0 & 0 & 1 & z_{encode r_{i}} \\ 0 & 0 & 0 & 1 \end{matrix})

(2)

Figure 2.

Lidar scanning model of Shenhua 512.

Recognition algorithm

Preprocessing of splicing point cloud

The point cloud model obtained is mixed with a lot of noise and isolated points after completing the original lidar point cloud splicing. It is a normal phenomenon that the occurrence of noise and isolated points for the number of pixels is large as the laser lidar used is of surface laser data. In order to ensure the accuracy of hatch recognition, the first step is to filter noise and outliers. Traversing each pixel $p_{i}$ in the point cloud model through the method of variance test specifically, and calculating the pixel mean $\bar{p}$ and variance $va r_{p}$ of $K$ pixels in its local neighborhood by formula 3 and formula 4. Adding the pixel to the filtering point cloud $P_{inliler}$ when the neighborhood variance is within the threshold range $va r_{threshold}$ and removing the pixels that exceed the threshold $va r_{threshold}$ , as shown in formula 5. After preprocessing the point cloud model, the noise and isolated points in the model can be removed, and the main body of the ship can be highlighted to facilitate the identification of the hatch moreover.

\bar{p} = \sum_{i = 1}^{K} p_{i}

(3)

{var}_{p} = \frac{sqrt ((p_{i} - \bar{p})^{2})}{k}

(4)

P_{inlier} = \sum p_{i}, {var}_{p} < {var}_{threshold}

(5)

Contour extraction

Projecting the preprocessed point cloud model onto the XOY plane, as shown in formula 6. Among them, $p_{i} . z$ represents the height value of the $i$ -th point cloud in the world coordinate system, and $p_{i} . i$ represents the light intensity of the $i$ -th point cloud. This projection does not reduce the pixels that the point cloud pixels still include all the information of the hatch after the projection. The purpose of the projection is to extract the contour of the point cloud in the two-dimensional plane. In the specific operation, the height value of each point is assigned to the intensity variable for temporary storage, so that the height can be restored later, and then the height value of each point cloud is directly set to 0. Since the actual port ship’s pitch angle attitude is not absolutely 0 $degree$ , there will be some calculation deviations by directly setting the altitude to 0. However, the length of the ship is generally more than 100 $m$ , and the pitch angle of the ship is generally not large. The deviation value is relatively small for the specific hatch point cloud. It has little effect on the hatch recognition result even if ignoring it.

Using the $α - shape$ ¹² algorithm to extract the outline of the point cloud in the two-dimensional plane. The principle of the $α - shape$ algorithm is shown in Figure 3. For any two-dimensional pixel $p (x, y)$ of the point cloud, making a circle with it as the center and 2 $α$ as the radius, and searching for the neighborhood contained in the range recorded as point set $Q$ .

Figure 3.

Schematic diagram of $α - shape$ principle.

$q_{i}$ represents any element in $Q$ , $q_{i} . x$ represents the X coordinate of $q_{i}$ in the world coordinate system, and similarly, $q_{i} . y$ represents the Y coordinate of $q_{i}$ in the world coordinate system. $p . x$ represents the point $p (x, y)$ in the X coordinate in the world coordinate system, $p . y$ represents the Y coordinate of the point $p (x, y)$ in the world coordinate system. By analogy, $p 1 . x$ , $p 1 . y$ , $p 2 . x$ , $p 2 . y$ respectively represent the coordinates of the center of the circle passing two points in the world coordinate system. Each pixel $q_{i}$ in $Q$ is traversed, and the center coordinates $p$ 1 and $p$ 2 through two points $p$ and $q_{i}$ with a radius of $α$ are calculated by formula 7 where the value of $H$ is shown in formula 8. For all pixels in $Q$ except the $q_{i}$ point, calculating the distances to $p$ 1 and $p$ 2. When the distances from all elements to $p$ 1 or to $p$ 2 are greater than $α$ , it means that $p (x, y)$ is a boundary point. Traversing all the pixels in $Q$ when there is an element meeting the boundary conditions, $p (x, y)$ is the boundary point, and then jumping out of the loop. When there are no elements with boundary conditions in $Q$ , it is proved that $p (x, y)$ is not a boundary point.

In the specific operation of the ship point cloud model shown in Figure 2, the $α$ value is set to 0.3 after parameter adjustment. After extracting the contour of the point cloud in the two-dimensional plane by $α - shape$ , the obtained contour point cloud in the two-dimensional plane is shown in Figure 4. As can be seen from Figure 4, the green part is the projection of the ship’s point cloud on the XOY plane, and the red part is the point cloud contour extracted by the $α - shape$ algorithm. The entire point cloud contour is relatively accurate and complete.

{\begin{matrix} p_{i} . i = p_{i} . z \\ p_{i} . z = 0 \end{matrix}

(6)

\begin{matrix} p_{1} . x = p . x + \frac{1}{2} (q_{i} . x - p . x) - H \times (q_{i} . y - p . y) \\ p_{1} . y = p . y + \frac{1}{2} (q_{i} . y - p . y) - H \times (p . x - q_{i} . x) \\ p_{2} . x = p . x + \frac{1}{2} (q_{i} . x - p . x) + H \times (q_{i} . y - p . y) \\ p_{2} . y = p . y + \frac{1}{2} (q_{i} . y - p . y) + H \times (p . x - q_{i} . x) \end{matrix}

(7)

\begin{matrix} H = \sqrt{\frac{α^{2}}{S^{2}} - \frac{1}{4}} \\ S^{2} = {(q_{i} . x - p . x)}^{2} + {(q_{i} . y - p . y)}^{2} \end{matrix}

(8)

Figure 4.

Contour extraction result of point cloud in XOY plane by $α - shape$ algorithm.

Line feature extraction

After obtaining the contour point cloud in the two-dimensional plane, the linear feature extraction is performed on it. The bow and stern cables are fixed to ensure that the hull is basically parallel to the wharf sideline. For each pixel in the contour point cloud, setting a circular with a radius of $β$ around it, calculating the variances of the pixels in the local neighborhood along the X-axis and the Y-axis respectively as formula 9 and formula 10. $\bar{p} . x$ represents the average of the X-axis coordinates in the neighborhood of point $p$ , $\bar{p} . y$ represents the average of the Y-axis coordinates in the neighborhood of point $p$ , ${var}_{p . x}$ represents the variance of the X-axis coordinates in the neighborhood of point $p$ , ${var}_{p . y}$ represents the variance of the Y-axis coordinate in the neighborhood of the $p$ point. $t$ represents the number of points in the local neighborhood. Comparing with the variance threshold $va r_{threshold_line}$ using the smaller value of variances of the pixels along the X-axis and the Y-axis. When the neighborhood variance of the pixel is less than the threshold, the pixel is considered to be a point on a straight line and added to the $P_{line}$ linear filtering point cloud as formula 11. It will be removed if pixels do not meet the neighborhood threshold. Performing linear feature extraction on the contour point cloud in Figure 4, after parameter adjustment, the value of $β$ is set to 1, the value of $va r_{threshold_line}$ is set to 0.2, and the final filtering result is shown in Figure 5. The red part of Figure 5 is the contour point cloud after the linear feature extraction, mostly in the form of a linear point cloud gathered around the hatch, which verifies the effect of the linear feature extraction.

\begin{matrix} \bar{p} . x = \frac{\sum_{i = 1}^{t} p_{i} . x}{t} \\ {var}_{p . x} = \sqrt{\frac{\sum_{i = 1}^{t} {(p_{i} . x - \bar{p} . x)}^{2}}{t}} \end{matrix}

(9)

\begin{matrix} \bar{p} . y = \frac{\sum_{i = 1}^{t} p_{i} . y}{t} \\ {var}_{p . y} = \sqrt{\frac{\sum_{i = 1}^{t} {(p_{i} . y - \bar{p} . y)}^{2}}{t}} \end{matrix}

(10)

P_{line} = \sum p_{i}, min ({var}_{p . x}, {var}_{p . y}) < {var}_{threshold_line}

(11)

Figure 5.

Contour point cloud after linear feature extraction.

Projection along the X-axis

The contour point cloud after linear feature extraction is projected onto the X-axis for clustering processing. It can be seen from Figure 5 that the contour point cloud is mostly the horizontal point cloud and the vertical point cloud surrounding the hatch after the linear feature extraction. By projecting to the X-axis, the point cloud around the same hatch can be gathered together and use clustering algorithms to easily separate them. In the specific operation, the Y-axis coordinate of each point is assigned to the intensity variable for temporary storage, so that the Y-axis coordinate can be restored later, and the Y-axis coordinate of each point cloud is directly set to 0 as formula 12. Ships need to be fixed with cables at the bow and stern when berthing, and it cannot be ensured that they are completely parallel to the edge of the wharf with little error of the ship’s front and back. The method of directly setting the Y-axis coordinate of each point cloud to 0 has little effect on the result of hatch recognition.

Using the $Euclidean$ distance segmentation algorithm to cluster the point cloud projected on the X-axis, as shown in formula 13, $p_{i}$ , $q_{i}$ are any two points in the point cloud set, $dL$ represents the $Euclidean$ distance, and the distance threshold is set to 1, the minimum number of classification point is 100, and the maximum number of classification point is 1000. The final result after projection along the X-axis is shown in Figure 6. Using the $Euclidean$ distance segmentation algorithm, it is easy to separate the point cloud projected on the X-axis, and the final classification number is equal to the number of ship’s hatches.

{\begin{matrix} p_{i} . i = p_{i} . y \\ p_{i} . y = 0 \end{matrix}

(12)

dL = \sqrt{\sum_{i = 1}^{n} {(p_{i} - q_{i})}^{2}}

(13)

Figure 6.

Projection result of contour point cloud on X axis.

Feature point search

When the point cloud clustering is completed, the center point $p_{cente r_{i}}$ of each category is calculated respectively, and the search areas $Δ y$ and $Δ x$ along the X-axis and along the Y-axis are set around the center point. After parameter adjustment, the value of $Δ y$ and $Δ x$ is taken as 3 specifically. Calculating the point cloud pixel closest to the center point in the search area by formula 14 and formula 15, which is the feature point of the edge of the hatch, and generating the coordinates of the hatch according to the horizontal and vertical coordinates of the feature point as formula 16.

After obtaining the feature points of the hatch, searching for points satisfied formula 16 in the initial three-dimensional point cloud model of the ship, setting the search neighborhood with the radius of $gamma$ with the specific value of 0.5. The pixel height value of the point cloud in the neighborhood is averaged to get the final hatch height. The identification of the hatch is completed.

The complete algorithm of hatch identification is summarized in Table 1.

Table 1.

Steps of hatch recognition algorithm.

Step	Method
Step1	Pre-processing of the original stitching point cloud, variance filtering to remove discrete noise
Step2	Extracting the point cloud contour on the XOY plane by $α - shape$ algorithm
Step3	Performing linear feature extraction on the contour point cloud
Step4	Performing cluster process on the contour point cloud after linear feature extraction
Step5	Searching for feature points on the edge of the hatch

\begin{matrix} p_{hatc h_{i}_left} . x = \min (p_{cente r_{i}} . x - p_{i} . x) \\ p_{cente r_{i}} . y - Δ y < p_{i} . y < p_{cente r_{i}} . y + Δ y \\ p_{hatc h_{i}_right} . x = \min (p_{i} . x - p_{cente r_{i}} . x) \\ p_{cente r_{i}} . y - Δ y < p_{i} . y < p_{cente r_{i}} . y + Δ y \end{matrix}

(14)

\begin{matrix} p_{hatc h_{i}_bottom} . y = \min (p_{cente r_{i}} . y - p_{i} . y) \\ p_{cente r_{i}} . x - Δ x < p_{i} . x < p_{cente r_{i}} . x + Δ x \\ p_{hatc h_{i}_top} . y = \min (p_{i} . y - p_{cente r_{i}} . y) \\ p_{cente r_{i}} . x - Δ x < p_{i} . x < p_{cente r_{i}} . x + Δ x \end{matrix}

(15)

\begin{matrix} (p_{hatc h_{i}_left} . x, p_{hatc h_{i}_bottom} . y) \\ (p_{hatc h_{i}_left} . x, p_{hatc h_{i}_top} . y) \\ (p_{hatc h_{i}_right} . x, p_{hatc h_{i}_bottom} . y) \\ (p_{hatc h_{i}_right} . x, p_{hatc h_{i}_top} . y) \end{matrix}

(16)

Experiment

Experiment 1

T600 made by TUWEI is used as the computing platform to receive and process lidar’s point cloud data which is based on NVIDIA Jetson Xavier core board with the CPU of 8-core Carmel arm structure and 16GB memory. The hatch recognition algorithm proposed in Table 1 was used to identify the three-dimensional point cloud model of the ship in Figure 2, and the final results are shown in Figures 7 and 8. Figure 7 is the perspective of the recognition results from the top of the Z-axis, and Figure 8 is the recognition result is the view of the bottom of the Z-axis. It can be seen that the proposed algorithm realizes each hatch recognition of the ship correctly, and the recognition accuracy is fine. In addition, there is a missing corner in the point cloud of the leftmost hatch, and the algorithm still makes a correct identification, indicating that the algorithm has a certain degree of robustness.

Figure 7.

Hatch recognition result of the algorithm in this paper viewed from the top.

Figure 8.

Hatch recognition result of the algorithm in this paper viewed from the bottom.

The specific hatch identification result and algorithm time-consuming are shown in Tables 2 and 3. The actual ship hatch size data is obtained from the Shenhua 512 ship chart, and the algorithm in this paper is compared with the Miao¹¹ algorithm. It can be seen from Table 2 that the algorithm in this paper has the largest error in the recognition of cabin $No .$ 3 with a difference of 2.15 $m$ in length and 2 $m$ in width. Due to the selection of feature points in the hatch edge during the algorithm processing, this paper selects the point closest to the hatch clustering center, and comprehensively considers the anti-collision detection requirements of the ship loader when the hatch is distributed, and reserves a certain safety margin from the hatch. Combined with the experimental results in Figures 7 and 8, the hatch recognition results of the algorithm in this paper are all within the actual hatch range which is in line with the actual production needs of the port.

Table 2.

Hatch recognition results of Shenhua 512.

Hatch number	1	2	3	4	5
Actual length of hatch ( $m$ )	17.67	20.09	20.32	20.24	20.23
Length of the method in this paper ( $m$ )	16.3	18.33	18.17	18.38	18.53
Length of Miao¹¹-1 method ( $m$ )	18.43/17.47	20.0	19.7	19.89	19.95
Length of Miao¹¹-2 method ( $m$ )	18.26	19.88	0	0	19.81
Length of Miao¹¹-3 method ( $m$ )	0	0	0	0	0
Actual width of hatch ( $m$ )	16.1	16.1	16.1	16.1	16.1
Width of the method in this paper ( $m$ )	14.5	14.3	14.1	14.2	14.2
Width of Miao¹¹-1 method ( $m$ )	15.98/11.83	16.05	15.98	16.06	16.0
Width of Miao¹¹-2 method ( $m$ )	15.75	16.17	0	0	15.93
Width of Miao¹¹-3 method ( $m$ )	0	0	0	0	0

Table 3.

Shenhua 512 ship hatch identification time-consuming.

Algorithm	Method in this paper	Miao¹¹-1	Miao¹¹-2	Miao¹¹-3
Time-consuming ( $s$ )	15.302	80.219	403.862	1029.13

As shown in Table 2, the Miao¹¹ algorithm has taken several different sets of original point cloud down-sampling parameter values for comparison. For the same ship point cloud model, the VoxelGrid¹³ filter is used for down-sampling filtering with the grid size parameter value of 0.2, the result of down-sampling the original point cloud by twice the voxel is recorded as Miao¹¹-1, the result after the original point cloud is down-sampled by the voxel is recorded as Miao¹¹-2, and the result of the original point cloud without down-sampling is recorded as Miao¹¹-3. It can be seen from Table 2 that after down-sampling the original point cloud data by 2 times the voxel as Miao¹¹-1, the recognition accuracy reached the highest. The hatch recognition error is basically within 0.3 $m$ , and the most accurate cabin $No .$ 2 recognition with length difference of 0.09 $m$ and width difference of 0.05 $m$ . Accurate hatch recognition results enable the ship loader to make more effective planning of the loading path. However, due to the lack of some point clouds at the edge of cabin $No .$ 1, there is a set of more results for the identification of the hatch of cabin $No .$ 1, and the recognition error is larger. The length of the first group is 0.76 $m$ longer than the true value which increases the risk of the machine colliding with the bulkhead during the cloth distribution resulting in an accident. Another set of data has a difference of 4.27 $m$ in width which severely restricts the route planning of the ship loader.

In addition, it can be seen from Table 3 that compared with the method of Miao,¹¹ the algorithm speed of recognition in this paper has been significantly improved in terms of the algorithm time-consuming. Even though the algorithm in this article directly performs hatch recognition on the original stitching point cloud without down-sampling, the final recognition time is 15.302 $s$ , and the fastest recognition speed of Miao¹¹ is to down-sampling the original point cloud by 2 times as Miao¹¹-1, which the recognition time is 80.219 $s$ . The recognition time after down-sampling as Miao¹¹-2 is 403.862 $s$ , and it takes 1029.13 $s$ to directly recognize the original point cloud as Miao¹¹-3. For in the process of hatch recognition, the algorithm in this paper is to process all the point cloud at one time and perform contour extraction, clustering, and classification; while the method of Miao¹¹ is to process a segment of the point cloud. In the case of the point cloud model of the whole ship, it is necessary to scan and identify sequentially from the bow to the stern which is large time-consuming. In the actual production process of the port, time means economic benefits. This article increases the speed of point cloud recognition for a whole ship by 424% and reduces the time by more than 1 min. For the entire port, it may increase 10 million tons of output throughout the year.

Experiment 2

As shown in Figure 9, the second experiment is the hatch recognition result of the point cloud generated by the lidar splicing when the ship loader is stationary. As can be seen from Figure 9, in this experiment, the algorithm in this paper still has a good recognition effect for local point cloud with a total of 891 contour point cloud extracted pixels. The algorithm takes 3.39 $s$ which fully meets the real-time requirements for the hatch identification of the ship loader on moving. Under normal working conditions, the lateral movement speed of the ship loader is set to 1 $m / s$ . The length of ships in Tianjin Port is generally more than 100 $m$ with 5 or 7 cabins. When the ship loader is moving, it will take at least 10 s on each hatch, which fully satisfies the algorithm in this paper for dynamic identification. It can further improve the efficiency of automatic ship loading.

Figure 9.

Hatch recognition result under static view.

Conclusion and future work

This paper proposes a fast recognition algorithm for ship hatch edges based on point cloud contour extraction. The contour is extracted by projecting the ship point cloud model onto a two-dimensional plane, and further according to the hatch features, the contour point cloud is extracted in a horizontal and vertical straight-line feature, and the point cloud is projected to X-axis for clustering to realize different cabins separation. Searching for the nearest neighbor feature points of each hatch cluster center while considering the actual production anti-collision requirements of the port, and finally realize the identification of ship hatch. According to the actual docking ship Shenhua 512 three-dimensional point cloud model data of Guoneng Tianjin Port, the experimental results show that the algorithm proposed in this paper can quickly identify the ship’s hatch, and has good reliability and robustness. Compared with the previous algorithm, the speed of the algorithm in this article is several times faster. For the actual production anti-collision requirements of the port, the identification result fully meets the production needs, and there is an opportunity to merge point cloud while performing the hatch recognition, which further improves the efficiency of automated port shipment.

In order to further improve the algorithm in this paper and make it more stable in the actual production environment of the port, there are still some problems to be further studied and improved.

Improving the preprocessing effect of the scanning point cloud. Now, after the preprocessing algorithm, this paper mainly uses the variance filtering of the point cloud to remove the noise and outliers in the overall point cloud and highlight the main body of the ship point cloud. Later, it can consider adopting more extensive filtering methods, such as light intensity filtering, threshold filtering of the number of the point cloud in the neighborhood, to further highlight the main point cloud and eliminate point cloud noise in the hatch to improve the stability of the algorithm.

Improving the accuracy of hatch recognition. When searching for the feature points of the hatch, this paper selects the feature points closest to the center point considering the anti-collision requirements of the ship loader in actual production. Due to the selection of the nearest point, the recognition result is relatively conservative, which restricts the path planning of the ship loader and is not conducive to better deployment of cloth work. Later, the method of point cloud fitting on the side of the cabin can be used to further improve the recognition result.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported partially by the State Key Laboratory of Robotics and System (Grant No. SKLRS-2018-KF-12), the Heilongjiang Natural Science Foundation (Grant No. LH2019F020), the National Natural Science Foundation of China (Grant No. 62073101), and the Self-Planned Task of State Key Laboratory of Robotics and Systems of the Harbin Institute of Technology (Grant No. SKLRS202007B).

ORCID iDs

Yuan Li

Zhan Li

References

Sangeerth

Lakshmy

KV.

Blockchain based smart contracts in automation of shipping ports. In: 2021 6th international conference on inventive computation technologies ICICT, 2021, pp.1248–1253. New York: IEEE.

Muhammad

Kumar

Cianca

, et al. Improving port operations through the application of robotics and automation within the framework of shipping 4.0. In: 2018 21st international symposium on wireless personal multimedia communications WPMC, 2018, pp.387–392. New York: IEEE.

Zhang

Liu

, et al. Optimizing the stowage planning and block relocation problem in inland container shipping. IEEE Access 2020; 8: 207499–207514.

Qifan

Research and development of port bulk cargo automatic handling equipment. China Mech Eng 2008; 19(23): 2797–2803.

Lee

Deep learning–assisted real-time container corner casting recognition. Int J Distrib Sens Netw 2019; 15(1): 1–11.

Chen

Zhao

Automatic docking recognition and location algorithm of port oil loading arm based on 3D laser point cloud. In: 2020 IEEE international conference on mechatronics and automation ICMA, 2020, pp.615–620. New York: IEEE.

Celic

Cuculic

Valcic

Remote sensing for ship emissions monitoring in Adriatic ports: an approach. In: Proceedings ELMAR − 2012, 2012, pp.263–266. New York: IEEE.

Jia

Guoxing

Dou

Intelligent security protection and auxiliary operation system of container yard based on lidar. Port Sci Technol 2020; 4(06): 1–5.

Shen

, et al. Ship identification algorithm based on 3D point cloud for automated ship loaders. J Coast Res 2015; 73: 28–34.

10.

Xi-Liang

Shen

, et al. Algorithmic study on bulk cargo carrier detection and positioning based on 3D laser vision. Chin J Constr Mach 2014; 12: 555559.

11.

Miao

, et al. A novel algorithm of ship structure modeling and target identification based on point cloud for automation in bulk cargo terminals. Meas Control 2021; 54: 155–163.

12.

Teichmann

Capps

Surface reconstruction with anisotropic density-scaled alpha shapes. In: Proceedings visualization, 1998, pp.67–72. New York: IEEE.

13.

Shi

Zhu

, et al. RGB-D semantic segmentation and label-oriented voxelgrid fusion for actuate 3D semantic mapping. In: IEEE transactions on circuits and systems for video technology, 2021. New York: IEEE.