Multi-stage deep learning perception system for mobile robots

Abstract

This paper presents a novel multi-stage perception system for collision avoidance in mobile robots. In the here considered scenario, a mobile robot stands in a workspace with a set of potential targets to reach or interact with. When a human partner appears gesturing to the target, the robot must plan a collision-free trajectory to reach the goal. To solve this problem, a full-perception system composed of consecutive convolutional neural networks in parallel and processing stages is proposed for generating a collision-free trajectory according to the desired goal. This system is evaluated at each step in real environments and through several performance tests, proving to be a robust and fast system suitable for real-time applications.

Keywords

Deep learning machine learning obstacle detection computer vision path planning

1. Introduction

In recent years, the field of applied robotics has expanded from factories to other increasingly complex environments, where tasks are less repetitive and require cooperation between humans and machines. Motion planning, which has become a crucial subject in this area, aims to enable robots to automatically compute their movements from different high-level task descriptions acquired through sensors [1].

Since the environment may change at any time, detection algorithms must be robust and fast enough to detect any change involving a modification to previously planned movements. Obstacle detection, a critical stage in navigation tasks, aims to estimate the position and size of every object in the environment that could present a collision risk. In practice, this information is acquired using sensors such as scanners, radars, sonars, or stereo vision cameras [2].

1.1 Related works

Deep learning or deep neural networks represent an alternative approach to traditional computer vision techniques. Due to an increase in available storage and computational capabilities, these have won numerous pattern recognition and machine learning contests in recent years [3]. Thanks to its generalization capacity, which enables models to develop new tasks based on previously learned information, deep networks have been applied to diverse perception applications, such as road segmentation in autonomous driving [4], obstacle detection [2, 5], and indoor mobile robot navigation [6, 7].

In [7], a perception system for pedestrian detection and evasion in corridors was developed, which employs a hybrid detector by mixing traditional techniques with Convolutional Neural Networks (CNN), allowing to get a real-time performance of 10 frames per second (fps). In [4], a Faster Region-based CNN (R-CNN) model was implemented in a GPU with post-filtering to detect and consider objects that are common in road-navigation tasks; this model can run at 10 fps using an Nvidia GeForce 980.

Authors in [8] developed a CNN model based on AlexNet [9] by applying several segmentation algorithms combined with the Pascal dataset [10]. In [11], a perception system to detect and plan trajectories along fire areas was developed; by employing a YOLO [12] based neural network trained to locate and classify fire areas (as safe or as hazardous), while a modified version of A ${}^{*}$ algorithm [13] was employed to generate a fire hazard free trajectory. Other works have been focused on treating some common training-related problems like overfitting, with techniques such as Dropout [14], batch normalization [15], and dataset augmentation [16].

Human-robot interaction has been another topic of interest for many researchers in the field of robotics; understanding how human gestures are related to speech content can be applied to improving human-machine communication. In [17], an architecture that relates phrases and gestures through body expressions was proposed in order to evaluate different language models using semantics. In [18] a methodology for co-verbal gestures recognition employing text and the automatic improvement of execution time in collaboration with humans was developed [19]. Another topic of interest for achieving better human-machine interactions is eye-tracking for controlling gaze-based human-machine interfaces through CNN’s [20, 21]. Likewise, other efforts have focused on improving the perception of artificial eyes, such as the optical flow calculation for multi-aperture sensors [22], and dense depth map estimation from unfocused images [23].

1.2 Proposed perception system

Although many works have been developed on the subject of mobile robot perception, most have focused on solving specific problems, e.g., detecting specific objects such as obstacles or targets or generating evasion trajectories with prior knowledge of the environment, which in most cases requires sending information to a computer with higher computational capacity in order to achieve a consistent processing speed. Thus, the development of fast generic embedded systems remains a challenge at present, where the generalization of tasks is also a key point.

This work proposes a novel perception system for mobile robots. This system is composed of three main steps: First, a detection system comprising two neural networks in parallel is employed to detect a set of objects that may be of interest during interaction with the environment (a set of potential targets to reach and a set of potential obstacles to avoid), while a third network is employed to identify gestures from human partners in order to define a single goal to reach when several are available. Second, stereo vision and filtering algorithms are used to recover the 3D position of every detection. Finally, all the information is applied to generate a collision-free trajectory from the robot’s initial position to the desired target (by interpreting the human gestures). All steps were implemented in real-time using a single Jetson TX1 developer board equipped with an Intel Movidius computer stick. Compared with other works, the main contributions of this paper are the following:

A full vision-based perception system is proposed, with different stages for detection, classification, 3D position estimation, and collision-free trajectory generation.

The detection system employs RGB images to develop all detections (obstacles and targets), which in turn allows directly using the depth of those pixels containing an object of interest, instead of processing the pixels from the depth map, which would increase processing speed for real-time applications.

The system can recognize a set of gestures from human partners commonly used to indicate a desired direction or target. This information is then employed with support algorithms to choose a target to reach when several are available.

Two new labeled datasets for supervised learning have been developed: the first includes indoor images for obstacle detection tasks, and the second indoor and outdoor images for human hand-gesture recognition. Each includes thousands of real images taken in different environments.

Stages were designed to be as fast and lightweight as possible for its use as an embedded system, capable of running on a Jetson TX1 board with an average processing speed of 15 fps.

Another advantage remains in the capacity of the networks to recognize the object classes, allowing to modify the interaction of the robot depending on the object type, in this case, to consider human navigation orders. Also, the presented obstacle detection network allows detecting obstacles at floor level by employing RGB images directly, which information enables a richer set of features compared with other system types (like radars or laser-based systems), by allowing to extract additional information from the pixels that correspond to an obstacle.

Figure 1.

Perception problem scheme. Guided by pedestrian hand signals, a mobile robot must plan a trajectory to a specific target while avoiding collisions with the objects in its path.

The rest of the paper is organized as follows: Section 2 describes the problem definition covered in this work, while Section 3 describes each stage of the detection system, composed by parallel neural networks. Section 4 covers the spatial reconstruction and trajectory generation modules, while Section 5 discusses experimental results and analysis of the whole perception system. Finally, Section 6 illustrates the conclusions and future work of the present paper.

2. Problem definition

This paper considers the following situation: A mobile robot is standing in an environment waiting for a human command to reach a specific target. When the order is given, and the target is detected, the robot must plan its trajectory while avoiding collisions with the other objects in its path (Fig. 1).

Figure 2.

Proposed perception system composed of 6 main blocks. 1: Input block, 2: Preprocessing step, 3: Parallel neural network system, 4: Spatial reconstruction, 5: Decision scheme, and 6: Trajectory generation. The size and dimensions of the processed data are shown at the bottom of each step.

Figure 3.

Preprocessing step. Input RGB process scheme.

Within the scope of this work, the following restrictions are considered:

A depth map of the robot’s vision is computed at all times using an efficient stereo system [24].

For terrain robots, it is assumed that obstacles are at floor level; objects above this area are taken as if they were on the floor, which is a common situation for indoor environments.

It is considered that the position of obstacles and targets can present slight variations between detection frames; therefore, tracking and filtering algorithms were designed under this limitation.

The above restrictions allow a robot to navigate securely through an indoor setting, where corridors, walls, pedestrians, and the majority of objects usually are touching the floor.

2.1 System definition

Initially, a single YOLO-based [25] network was proposed to unify all the detection tasks; in practice, however, it had difficulties generalizing hand signal recognition across other specific classes. Moreover, at the first step, an obstacle detection task may not require the class of each detected object, as any class represents a collision risk; the most important information to know is the position and dimension of each obstacle. By considering the previous information, the problem was divided into separate tasks and models; the proposed perception system (summarized in Fig. 2) is composed of 6 main blocks:

Input Block: an RGB image and a depth map are acquired from the environment employing a stereo camera.

Preprocessing Step: the input RGB image is processed in parallel in two ways according to the input specifications of subsequent modules.

Detection System: the processed images are introduced through the detection system, which consists of three neural networks in parallel;

The first generates an array with the position of the nearest obstacle for every 20 pixels on the image, while the second generates bounding boxes with the position of potential targets.

As a second stage, if a pedestrian is detected, a third neural network is employed to determine if a hand signal (left, stay, or right) is given by each of the pedestrians.

Spatial Reconstruction: using camera calibration parameters and the information from the depth map, the spatial position of all objects is calculated, employing several filtering and tracking algorithms to deal with noise and other detection problems.

Decision Scheme: if multiple hand signals are given by human partners in a multiple target scheme, a simple voting scheme is used by considering each partner’s location to determine a single target.

Trajectory Generation: using the RRT algorithm [26], the information gathered in the previous stages is employed to generate a 2-D evasion trajectory from the camera origin position. This trajectory is preserved until the selected goal moves a $Z_{t}$ distance from its previous position, or another object is selected as the goal, or when any object implies a collision with the current trajectory.

2.2 Preprocessing step

Through a ZED stereo camera, a 3 $\times$ 480 $\times$ 680 RGB image $I$ and a 480 $\times$ 640 depth map $D_{m}$ are acquired from the environment. Then, the input RGB image is reshaped according to the specification of each neural network on the detection block (Fig. 3):

For obstacle detection, the input image is divided into 32 columns of width $w=$ 20 pixels and trimmed horizontally above the $v=$ 200 pixel limit (floor level), then halved to give a final 32 $\times$ 3 $\times$ 100 $\times$ 10 array of stripes or “stixels” $I_{s}$ .

For object detection according to YOLO [12], the same input image is resized to a standard 1 $\times$ 3 $\times$ 448 $\times$ 448 array $I_{r}$ .

3. Detection system

To define the detection system, it is necessary first to establish the differences between the object types involved while executing a movement through the environment:

Figure 4.

StixelNet, a modified neural network structure. F indicates the number of filters, h and w the size of the kernel on convolutional and pooling layers, N the number of neurons in fully-connected layers, and BN the position of batch normalization layers.

Figure 5.

StixelNet, output layer behavior. The network is trained to predict the nearest obstacle position for every 20 pixels of the image.

For obstacles, it is of primary importance to ensure that its position does not present a risk of collision during the execution of a movement; regardless of kind or class, any object is a potential obstacle if the robot can collide with it.

For other objects that require some level of interaction, the class is an important fact as the interaction may vary based on the target type in terms of approaching distance, the robot’s final pose, or other practical and security considerations.

For hand signal recognition, a prior search for pedestrians may enable fast detection by not processing irrelevant pixels; by focusing the search on the surrounding area where pedestrians have previously been detected (where the configuration of the arms can be directly detected). Thus, this task can be added as a second processing step once a pedestrian is detected in the environment.

By taking the previously mentioned considerations, the general detection task was divided into three sub-tasks; obstacle, object, and hand signal recognition. In this work, a detection system composed of three parallel neural networks was employed to deal with each detection task. Each network will be discussed in detail in the following subsections, including their structures, processing schemes, training and implementation processes.

3.1 Obstacle detection problem

3.1.1 Neural network structure

The obstacle detection problem may be described as follows: given an input image $I$ , the neural network must give the pixel position $B_{o}(u,v)$ of all objects that could present a collision risk while executing a movement across the environment. To address this problem, the StixelNet neural network structure proposed in [2] for road segmentation is used. This network has 5 layers; the first two convolutional and the last three fully connected. ReLU non-linearity is applied at the output of all layers except the last, which has a Softmax function; for the convolutional layers, a Max-pooling layer follows the ReLU operation as well. To find the best model, several architectures were tested by varying the hyperparameters of the original model. At the end of the process, the architecture presented in Fig. 4 was selected:

Figure 6.

StixelNet on some validation images, obstacle detection (left) and probability matrix (right). A darker color in the matrix represents a higher probability $P(v)$ that an obstacle appears in row $v$ .

Figure 7.

Tiny Yolo performance of some validation images.

A smaller input image was considered, consisting only of the region bounded by the horizontal pixels $[h_{\min},h]$ , in order to focus the network on the area where an obstacle may appear (floor level), avoiding processing pixels with no relevant information, and consequently increasing processing speed.

The size and stride of all filters were modified to adjust the network to the new work resolution (100 $\times$ 10 pixels).

In order to improve the training process, batch normalization [15] and dropout layers [14] were added to prevent overfitting during training.

3.1.2 Forward processing

This network receives a single 3 $\times$ 100 $\times$ 10 RGB image vertical stripe or stixel $I_{s}$ ; its objective is to find the vertical pixel location $v$ of the average bottom point of the closest obstacle presented in $I_{s}$ , where if 0 is the top row of the image and $h$ the bottom row, $v$ is considered to be a positive integer in the relevant domain $[h_{\min},h]$ – the minimum possible row $h_{\min}$ in which an obstacle can appear given a roughly known camera position (Fig. 5). Thus, each neuron $k$ in the output layer $B_{s}$ of size $N$ represents the probability $P(v)$ that the average bottom point of the closest obstacle in $I_{s}$ is in row $v$ , whose image position is determined by the following equation:

$\displaystyle v=\frac{h-h_{\min}}{N}k+h_{\min},u=ws-\frac{w}{2}.$ (1)

For this work, an obstacle detection zone with limits between $h_{\min}=$ 280 and $h=$ 480 divided into $N=$ 50 sections is considered.

3.1.3 Dataset building process

In order to build the dataset for the network, a ZED stereo camera was used to take pairs of images and depth maps. During the data acquisition process, two different scenes were considered: First, images were taken while passing through hallways and interiors, where people, walls, and different types of obstacles were presented. Second, in a fixed place, the position of different types of obstacles was varied in the range $[h_{\min},h_{\max}]$ . For each image, the depth map and manual segmentations were used to detect the nearest obstacle at every 20 horizontal pixels in the image.

After the process, 100 images distributed into 3,200 stixels were collected. By taking advantage of both the obstacle detection zone $h\in[h_{\min},h_{\max}]$ (which is a fraction of the original image) and the detection scheme (classification in 50 zones), a data augmentation process was performed; in this process, the detection zone of each stixel was displaced until it reached the top of the image, allowing multiple samples to be generated from a single labeled stixel.

By carrying out the data augmentation process, 115,500 stixels were generated; 104,000 were used for the training set and 11,500 for the validation step. The neural network was designed and trained using Caffe [27], through 20,000 iterations with a batch size of 64 stixels and a learning rate of 5.25 $\times$ 10 ${}^{-5}$ , using Adam [28] and PL-Loss function [2]. Once trained, the neural network can detect one obstacle per every 20 pixels of the original image (Fig. 6).

3.2 Target detection

You Only Look Once (YOLO) [12] is a state-of-the-art convolutional neural network that employs bounding boxes to achieve object detection and classification, its most recent version (YOLOV3) was released in 2019. Although there are multiple sets of pre-trained models like Tiny-Yolo for 80 classes, they require a considerable amount of memory to run at an acceptable speed. In practice, due to hardware limitations with the other system modules, it was necessary to retrain a smaller model with a fewer set of classes. In this work, the set of classes c $=$ {person, bottle, chair, table, screen, laptop, computer mouse, dining table, and cup} were considered in order to develop a new smaller model.

3.2.1 Neural network structure

In this stage, a short version of Tiny YOLOV3 [12] was employed. This architecture has 13 convolutional layers, with a Max-pool layer placed at the output of each convolution terminating in a set of standard YOLO detection layers which generates a $N\times N\times[3\ast(4+1+\textit{classes})]$ tensor at the output, which codifies predictions for 4 bounding boxes, 1 objectness, and a vector with scores for the 9 classes.

Figure 8.

SignalNet, network architecture. F indicates the number of filters, h, and w the size of the kernel on convolutional and pooling layers, N the number of neurons in fully-connected layers, and BN the position of batch normalization layers.

3.2.2 Dataset building process

A set of images taken from the MS-COCO [29], according to the set of 9 classes, were considered to train and test the reduced model: a total of 118,287 images were taken for training and 19,358 for testing. Using the darknet framework [12], the neural network was trained through 500,200 iterations with a batch size of 64 and a learning rate of 2 $\times$ 10 ${}^{-4}$ . At the end of the process, the network was able to perform object detection with new samples at different scales (Fig. 7).

3.3 Hand signal detection

3.3.1 Neural network structure

The hand signal detection problem may be described as follows: If a person is detected, the neural network must determine if any of the available hand signals (go left, stay, or go right) are indicated. By taking a speed performance criterion into account, this neural network was designed to be as small and fast as possible without a (significant) loss of generalization capacity, after attempting multiple classification-based configurations, the structure presented in Fig. 8 was used, which consists of two convolutional layers followed by a fully-connected layer with three neurons at the output, which classifies the input bounding box areas from previous steps in each one of the hand-signal gestures.

Figure 9.

SignalNet, output layer behavior.

Figure 10.

SignalNet performance on some validation images. The probability output vector is presented below each image (for right, stop and left hand-signal classes).

3.3.2 Forward processing

This network receives a resized 3 $\times$ 100 $\times$ 100 RGB image according to the bounding box area detected by the previous step, each neuron on the output layer is trained to detect if one of the available signals is indicated by a pedestrian (Fig. 9).

3.3.3 Dataset building process

The dataset for this network was built using 2,700 480 $\times$ 640 RGB acquired images of random people through indoor and outdoor scenarios, indicating each available hand signal (go left, stay or go right). The dataset was then duplicated, considering data augmentation techniques such as image mirroring (which in turn enables to produce additional labels for a single image; a mirrored left-sign image may serve as a right-sign image, and vice-versa). Each image in the dataset was processed by the reduced YOLO model of the detection system to get the bounding box coordinates of the pedestrians in each image, which were then employed with different crop rules to extract the bounding box area and build the final dataset, by considering only the upper half of the bounding box area (above the person’s waist, which is a suitable region for correctly detecting a hand signal), from 1.0 to 1.5 of the total box height. At the end of the process, a full dataset of 24,000 images was generated, distributed into 21,500 for training, and 2,500 for testing.

The neural network was designed and trained using Caffe [27], through 10,000 iterations with a batch size of 128 images and a learning rate of 2.5 $\times$ 10 ${}^{-4}$ , using Adam [28] as the learning law. Once trained, the neural network learns to efficiently predict the three types of signals through new sample images (Fig. 10).

4. Post-processing step

4.1 Spatial reconstruction

Each neural network output $B_{o}(u,v)$ and $B_{t}(u,v)$ is used with the camera intrinsic matrix $K$ and depth map $D_{m}$ to obtain the spatial coordinates $P_{oi}(x_{c},y_{c})$ , $P_{ti}(x_{c},y_{c})$ of each detected object $i$ via the transformation [30]:

$\displaystyle P_{oi}=z_{c}K^{-1}B_{oi}.$ (2)

where $z_{c}$ is obtained directly from depth map according to the median pixel neighborhood of each detection region (a 20 pixels square around each pixel center for stixels and bounding boxes).

$P_{o}(k)$ , $P_{t}(k)$ , $B_{t}(k)$ , $F_{o}(k-1)$ , $F_{t}(k-1)$ , $b_{\textit{thresh}}$ , $e_{\textit{thresh}}$ $F_{t}(k)$ , $F_{o}(k)$ Get an incoming set of obstacles and targets $O(k)=\{P_{t},P_{o}\}$ ( $F_{t}$ is empty) Assign a filter $\tilde{P}_{ti}(k)$ to every $P_{ti}(k)\in P_{t}$ Initialize $F_{t}(k)=\tilde{P_{t}}(k)$ Update $\tilde{P}_{ti}(k)$ with constant speed Calculate cost table: $C_{t}=||\tilde{P}_{t}(k),P_{t}(k)||^{2}_{2}$ Find correspondences between frames: $P_{tj}(k)=\textit{Hungarian}(C_{t},\tilde{P}_{ti}(k))$ (each pair of $\tilde{P}_{ti}(k)$ , $P_{tj}(k)$ ) $d=\textit{IOU}(\tilde{P}_{ti}(k),P_{tj}(k))$ ( $d<b_{\textit{thresh}}$ & $C_{t}(i,j)<e_{\textit{thresh}}$ ) Update position of $\tilde{P_{ti}}(k)$ Create a new entry $\tilde{P}_{tj}(k)$ on $F_{t}(k)$ ( $F_{o}$ is empty) Assign a filter $\tilde{P}_{oi}(k)$ to each $P_{oi}(k)\in P_{o}$ Initialize $F_{o}(k)=\tilde{P_{o}}(k)$ Generate $\tilde{P}_{oi}(k)$ with constant speedCalculate cost table: $C_{o}=||\tilde{P}_{o}(k-1),P_{o}(k)||^{2}_{2}$ Find correspondences between frames: $P_{oj}(k)=\textit{Hungarian}(C_{o},\tilde{P}_{oi}(k))$ (each pair of $\tilde{P}_{oi}(k)$ , $P_{oj}(k)$ ) ( $C_{o}(i,j)<e_{\textit{thresh}}$ ) Update position of $\tilde{P}_{oi}(k)$ Create a new entry $\tilde{P}_{oj}(k)$ on $F_{o}(k)$ $F_{t}(k)$ , $F_{o}(k)$ Object tracking algorithm

4.2 Position tracking

Once relevant objects are detected by the previous steps, a filtering and tracking scheme is proposed to deal with the noise related to position estimation (due to incorrect depth calculation by the hardware) as well as multiple detections of a single object (due to movements across the enviroment or to multiple bounding boxes on the same object). In this scheme, for each pair of detection arrays $P_{o}(k)$ and $P_{t}(k)$ , the following tracking and filtering algorithm is employed (Algorithm 1):

During the first detection, a constant velocity Kalman filter [31] with states $\tilde{P}_{i}=\{x,y,\dot{x},\dot{y}\}$ is assigned to each detected object $P_{i}$ , which is then stored in a global dictionary $F=\{F_{o},F_{t}\}$ .

When a new frame arrives, the position of all $\tilde{P}_{i}$ filters in the global dictionary is updated, by employing a constant velocity.

Euclidean distance, as well as the Intersection Over Union (IOU) measurements, are employed with the Hungarian algorithm to establish correspondences between objects stored in the global dictionary and the incoming frame.

For an object with correspondence in a previous frame, its Kalman filter position $\tilde{P}_{i}$ is updated, while for those detected with no correspondences, a new Kalman filter is created.

If a previously detected object does not receive a correspondence during $T$ frames, then it is deleted from the dictionary.

In this work, by considering a general processing speed of 15 fps, a rate of $T_{\textit{targets}}=$ 60, and $T_{\textit{obstacles}}=$ 15 was employed to store the information from detected bounding boxes and obstacles respectively.

4.3 Decision scheme

Given a set $P_{t}$ of 2D positions of detected persons (labeled with one of the three available hand signals) and other potential targets, it is necessary to define a single target to reach according to the hand signals provided. By considering the selected target $T_{g}$ as that which carries out the majority of hand signal directions, a simple voting scheme is employed to determine which target to reach, while the rest of the objects stored in $P_{t}$ are considered obstacles during the trajectory generation step.

4.4 Trajectory generation

Once a single target is defined, it is necessary to define a trajectory $L$ from the origin that allows reaching point $T_{g}$ with no collisions with the points stored in $P_{o}$ and $P_{t}$ by a security radius $Q$ . For this work, the RRT algorithm proposed in [26] is employed; two branches, $V_{0}$ and $V_{t}$ at the origin and the target position $T_{g}$ , respectively, both grow iteratively in random directions through collision-free links until a connection can be achieved. Once a collision-free trajectory is calculated through the first selected target $T_{g}(0)$ , it is preserved until any of the following conditions met:

A new position of the target $T_{g}(k)$ implies a displacement $||T_{g}(0),T_{g}(k)||^{2}_{2}>T_{z}$ outside the previous end-point path calculation, where $T_{z}\in\mathbb{R}$ is a threshold measurement for ensuring a security zone through the target.

A new detected position $P_{t}(k)$ or $P_{o}(k)$ compromises a collision for the current path.

By taking the previous considerations, a collision-free trajectory is continuously recalculated once the first target is detected by adapting it to the changes in the environment.

Once a set of points for a collision-free trajectory is calculated, it is necessary designing a controller that enables a mobile robot to follow the desired path. At present several techniques have been applied for that purpose, such as PID (Proportional-Integral-Derivative), feedback linearization, sliding mode, or fuzzy logic controllers.

In [32], an uncoupled linear PID controller for a differential wheeled robot is proposed, by proving the system stability employing the Lyapunov theory, the proposed control is able to track different proposed trajectories. Authors in [33] proposed a robust tracking control for a differential-drive wheeled robot with nonholonomic constraints by employing the feedback linearization technique. In [34], a sliding mode control derivated from a Lyapunov function and feedback linearization is presented, which presents robustness against matched perturbations. Other works that employ a combination of several control laws can be consulted in [35, 36, 37].

On the other hand, there are some cases where the trajectory points are not smooth enough for a desired mobile robot, where a trajectory planning technique is required. To see some works related to this issue, consult [38].

5. Experimental results

In this section several performance tests were carried out in order to evaluate each perception system module:

Detection accuracy of the three parallel neural networks of the detection system.

3D position estimation by employing detection modules with depth map.

A full system evaluation through different real scenarios.

5.1 Obstacle detection

Stixelnet was evaluated through a digital validation dataset with the aim of measuring its performance across the whole detection zone $h\in[h_{\min},h_{\max}]$ . This dataset is composed of images with different types and colors of obstacles distributed equitably in the detection zone. For each image $i$ with an obstacle in position $v_{d}(i)$ , the neural network was employed to generate a detection $v(i)$ , in order to compare and evaluate both values according to the following scheme:

Only those detections that meet the condition $v_{d}(i)-\epsilon\leqslant v(i)\leqslant v_{d}(i)+\epsilon$ were taken as correct, where $\epsilon=4$ pixels is a tolerance measure, in accordance with contiguous detection regions (which has a minimum separation of 4 pixels).

For each detection, the average median quadratic error $e(v,v_{d})$ between real and detected positions was calculated.

The validation set was composed of 5,000 images with 10 different obstacles in 10 scenarios across the 50 regions of the detection zone $[h_{\min},h_{\max}]$ ; these results are presented in Table 1. A median squared error of 41.51 $px^{2}$ was obtained across the entire set, while an average performance of 85% was obtained for the entire detection zone according to the aforementioned tolerance. As can be seen, the neural network can generalize in all zones except the farthest (near 280 pixels).

Table 1
StixelNet, detection zone evaluation

StixelNet, detection zone evaluation
Pixel	M.S.E	A.P	Pixel	M.S.E	A.P
478	9.32	0.69	378	1.84	0.95
474	6.08	0.81	374	2.2	0.94
470	3.28	0.86	370	1.76	0.94
466	5.92	0.81	366	1.24	0.97
462	4.88	0.82	362	2.04	0.94
458	4.12	0.86	358	1.32	0.96
454	4.56	0.83	354	2.12	0.94
450	4.4	0.84	350	1.28	0.97
446	3.84	0.87	346	1.44	0.96
442	4.04	0.86	342	1.76	0.94
438	2.8	0.89	338	1.84	0.95
434	4.76	0.83	334	1.8	0.95
430	4.44	0.84	330	3.8	0.87
426	4.28	0.87	326	1.84	0.91
422	3.0	0.87	322	1.72	0.91
418	3.32	0.87	318	1.0	0.95
414	2.56	0.91	314	0.96	0.97
410	8.56	0.83	310	0.24	0.99
406	3.0	0.88	306	0.32	1.0
402	4.72	0.83	302	0.92	0.98
398	4.04	0.85	298	3.28	0.97
394	6.92	0.8	294	4.36	0.93
390	5.44	0.81	290	10.8	0.61
386	3.52	0.92	286	20.48	0.06
382	2.2	0.92	282	29.28	0.05

M.E: Mean Error, A.P: Average Performance.

This may be an effect of the data augmentation process, which leaves this area with fewer examples than the others (which are more likely to benefit from the augmentation process). Nevertheless, since this is the farthest zone, it may not be a problem when executing a movement, as the potential obstacles in that zone will register in closer zones as the robot approaches the target while moving through the calculated trajectory.

Figure 11.

Position estimation for several obstacles samples employing the modules of the perception system. Top: Horizontal distance estimation, Bottom: Depth estimation.

5.2 Object detection

To evaluate the performance of the network, the validation dataset from MS-COCO (composed of 3,360 images) was used under the mean Average Precision (mAP) metric. For every image, the network was used to estimate the position and class of all objects; this measurement was then compared to the labels to calculate a performance estimation under the following criteria:

$\displaystyle\textit{Precision}=\frac{TP}{TP+FP},$

(3) $\displaystyle\textit{Recall}=\frac{TP}{TP+FN}.$

where $T P$ , $F P$ , and $F N$ indicate the number of true positive, false positive, and false negatives samples, respectively, by considering an Intersection over Union (IoU) index of 0.5. These results are presented in Table 2, which shows a general performance of mAP $=$ 35.9, consistent with the results presented in the original publication [12] for 80 classes with a deeper network (53 layers).

Table 2
YOLO evaluation on dataset

YOLO evaluation results

Class mAP Class mAP Class mAP

Person 51.2 Chair 20.8 Laptop 53.6

Bottle 21.3 Table 24.9 Mouse 48.7

Cup 25.1 TV 58.4 Keyboard 41.8

mAP: Mean Average Precision.

5.3 Hand signal detection

The generated validation dataset (composed by 2500 samples) was employed to evaluate the performance of the network; by employing the same scheme as in previous evaluations, a detection is considered correct only if the predicted class is the same as the true label. After evaluations, the network achieved a general performance of 90%, 98%, and 90% for left, stop, and right classes, respectively (Table 3).

Table 3
SignalNet evaluation results through validation dataset

SignalNet evaluation results
Signal	A.P	Signal	A.P	Signal	A.P
Left	0.90	Stop	0.98	Right	0.90

A.P: Average Performance.

5.4 3D reconstruction

The obstacle detection and 3D reconstruction modules were employed to estimate the position of 132 manually measured objects. In this scheme, the median square error $e=e(P_{r},P_{d})$ between real ( $P_{r}$ ) and estimated ( $P_{d}$ ) positions was calculated according to the following considerations:

To avoid false positives, only those detections inside a threshold $P(k)>$ 0.30 were considered.

The camera position $q_{0}(x_{0},y_{0})$ was considered as the origin of the workspace, whereby all the measurements are referred to this point.

To get obtain the real 3D position of each object, manually measured distances were taken from the workspace origin (camera).

For objects with a stixel in more than one stripe, the average position was employed.

After the evaluations, a median quadratic error of $e=$ 0.097 m ${}^{2}$ for both dimensions (Fig. 11) was achieved.

5.5 Real-time implementation

The whole perception system was implemented on an Nvidia Jetson-TX1 developer board [39] equipped with an Intel Movidius compute stick [40], through individual nodes using the ROS framework in C $++$ with Caffe, OpenCV, and Darknet libraries. For the specific case of YOLO, the original implementation of [25] was modified to include 3D position calculation and object tracking modules, while a ZED stereo camera was used as a depth and image sensor for the whole system. After implementation, the system can generate up to 32 spatial detections $P_{o}(x_{c},z_{c})$ for obstacles on the floor, multiple detections $P_{t}(x_{c},z_{c})$ for targets and gestures, and a set of points $E_{p}(x,y)$ to define a collision-free trajectory from the origin to the selected target. This implementation has an average processing speed of 15 fps (Fig. 12).

Figure 12.

Perception system, full implementation diagram. The average processing speed of each stage is presented.

5.6 Full system evaluation

Several experiments were carried out to evaluate the whole perception system, varying the position of different types of obstacles, people, and objects according to the following considerations:

First, the system was employed to evaluate different scenarios with measured static pedestrians and objects, in order to evaluate the different models working together while having a measured position of each estimation.

Second, the system was employed to track the position of pedestrians moving across the workspace, while the other modules are working to generate a collision-free trajectory according to the conditions mentioned in Section 4.4.

As mentioned above, the camera frame was considered as the origin, whereby all the measurements are referred to this point.

It is considered the situation to calculate a trajectory from the initial robot position (camera frame) to the desired target, while objects on the workspace are changing its position.

5.6.1 Static objects

Given the above considerations, the perception system was used to generate a trajectory from different representative obstruction problems, according to the following scheme:

Fixed scenarios with pedestrians, objects, and different hand signal configurations indicating a single target to follow were considered.

Each scenario was evaluated using the whole perception system, generating an estimated 2D position for each detected object, including a hand signal label for pedestrians, which was then employed to choose a single target to reach.

A collision-free trajectory was generated for each case, considering a safety zone of $Q=$ 0.2 m around each detected obstacle.

These results are presented in Fig. 13, which shows that the system can perform the different perception tasks by detecting all obstacles on the floor, all people in the environment, and all hand signals given by each of the latter. Also, the collision-free trajectory generated in each experiment allows reaching the desired target while avoiding collisions with all objects along the path. Figure 13 also shows that the farther an object was during the experiments, the more difficult it was to determine its exact 3D position (by estimating a slightly different position). On the other hand, as can be seen, due to the detection method used, which involved dividing the image into 32 columns, each stixel tends to cover a greater horizontal distance as the depth increases (from the bottom to the top of the image), while the method also implies that obstacles fully occluded by other objects (in vertical position) are not considered during the path planning process (case 2 in Fig. 13). Nevertheless, both cases can be solved using the replanning scheme; by taking new frames when the mobile robot is moving across the path, new views of the obstruction problem will be available, enabling the detection of new obstacles that were occluded in previous frames, as well as of objects registering in closer zones that were previously farther zones.

Figure 13.

Experimental results of real-time implementation by considering static objects. OpenCV interface (top), ROS RViz interface (middle), and representative problem (bottom). The above detections (obstacles, targets, and hand signals) are represented with different marker types in each interface. As can be seen, the perception system is able to recognize all objects, as well as the hand signals given by the human partners. Also, the generated trajectory allows reaching the target without colliding with the real obstacles in each scenario (by the considered security zone Q).

Figure 14.

Experimental results of real-time implementation by considering moving objects. Tracking: The system is intended to track a single defined target while moving across the environment. Signal detection: The system is intended to recognize and track a target defined by the hand-signals. In both cases, a collision-free trajectory is calculated each time that any of the conditions mentioned in Section 4.4 are met.

5.6.2 Dynamic objects

As a second evaluation step, a new set of scenarios with moving pedestrians was considered to evaluate the tracking and replanning algorithms, according to the following scheme:

First, a scenario with three moving pedestrians was considered. The system is intended to track each one while defining a collision-free trajectory to a single defined one.

Second, two moving pedestrians indicating hand-signals were considered. In this scheme, the system is intended to recognize, track, and calculate a collision-free trajectory to the goal defined by the hand-signals.

Once the first trajectory is generated, it is preserved until the conditions presented in Section 4.4 are met, or a new target is defined by the hand-signals given. A target security zone of $T_{z}=$ 0.6 m was considered, as well as a threshold measurement $b_{\textit{thresh}}=$ 0.2 and $d=$ 0.2 m for bounding boxes and obstacles respectively.

If a target zone with center at the goal position cannot be defined due to collisions with other obstacles, a new center is calculated by selecting a collision-free random point near the target (inside a distance $d<T_{z}$ from the target). This consideration ensures faster convergence as well as a smoother tracking of the goal.

These results are presented in Fig. 13. For the first case, the system is able to track the selected pedestrian and generating a collision-free trajectory to reaching it every time that one of the previously mentioned conditions is met. On the other hand, for the second experiment, the hand-sign detection network is able to recognize the gestures given by the pedestrians, allowing to change the target each time that is required, while the position of each pedestrian is correctly employed to set the target to reach. As can be seen, the different modules are able to work together, where the tracking and filtering algorithms allow tracking the position changes of the objects involved in the workspace.

For additional multimedia resources about these experiments or access to the datasets employed to train the networks, visit: drive.google.com/drive/folders/1X-BePp-GPb4Nhf48Co_NBOMTEwGMuwQR.

6. Conclusions

In this paper, a perception system for collision avoidance in mobile robots was proposed. Using a set of three neural networks in parallel, this system is capable of detecting objects on the floor that could be obstacles while executing a movement, as well as a set of objects that could be targets. This obstacle detection scheme is similar to the human ability to perceive depth without the need to establish correspondences between two points of view, allowing to detect obstacles by employing monocular RGB images directly. The system is also able to recognize human hand signals, which in turn may help to define a single target to reach when several are available. By taking advantage of modern hardware devices, the system can generate a 2D evasion trajectory from the robot’s position to a specifically desired target. To measure its performance, the system was evaluated in each stage through several performance tests according to each detection task, proving to be a robust and fast system suitable for real-time implementations.

As future work, we will consider different approaches for dynamic obstacles and targets based on the objects type, the recognition of a wide variety of human gestures, and new filtering and replanning schemes for considering new obstruction problems and situations with a robot in motion.

Footnotes

Acknowledgments

The authors would like to thank CONACYT project A1-S-10412 Percepción, Aprendizaje y Control de Robot Humanoide and CINVESTAV-IPN for the economic and technological support for the realization of this work.

References

Choset

. Principles of robot motion: theory, algorithms, and implementation. 1st ed. Cambridge, MA: MIT press; 2005.

Levi

Garnett

, et al. StixelNet: A Deep Convolutional Network for Obstacle Detection and Road Segmentation In: British Machine Vision Conference; 2015. pp. 109.1–109.12.

Schmidhuber

. Deep learning in neural networks: An overview. Neural Networks. 2015 Jan; 61(1): 85–117.

Prabhakar

Kailath

, et al. Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. In: IEEE Region 10 Symposium. IEEE; 2017. pp. 1–6.

Szegedy

Toshev

, et al. Deep neural networks for object detection. In: Advances in Neural Information Processing Systems; 2013. pp. 2553–2561.

Chen

, et al. Door recognition and deep learning algorithm for visual based robot navigation. In: IEEE International Conference on Robotics and Biomimetics. IEEE; 2014. pp. 1793–1798.

Ribeiro

Mateus

, et al. A real-time Deep Learning pedestrian detector for robot navigation. In: IEEE International Conference on Autonomous Robot Systems and Competitions. IEEE; 2017. pp. 165–171.

, et al. Detection and discrimination of obstacles to vehicle environment under convolutional neural networks. In: Chinese Control And Decision Conference. IEEE; 2018. pp. 337–341.

Krizhevsky

Sutskever

, et al. Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems; 2012. pp. 1097–1105.

10.

Everingham

Eslami

, et al. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision. 2015 Jan; 111(1): 98–136.

11.

Hua Yu

Heo

, et al. Fire Area Detection based on Convolutional Neural Network and Improved A

{}^{*}

Path Planning. In: International Conference on Information and Communication Technology Robotics. IEEE; 2018. pp. 1–4.

12.

Redmon

Farhadi

. YOLOv3: An Incremental Improvement. arXiv:180402767. 2018.

13.

Hart

Nilsson

, et al. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics. 1968 July; 4(2): 100–107.

14.

Srivastava

Hinton

, et al. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research. 2014 Jun; 15(1): 1929–1958.

15.

Ioffe

Szegedy

. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv: 150203167. 2015.

16.

Mikołajczyk

Grochowski

. Data augmentation for improving deep learning in image classification problem. In: International Interdisciplinary PhD Workshop. IEEE; 2018. pp. 117–122.

17.

Almagro

Fresno

, et al. Speech gestural interpretation by applying word representations in robotics. Integrated Computer-Aided Engineering. 2019 Dec; 26(1): 97–109.

18.

Luo

Okita

. Synchronized gesture and speech production for humanoid robots. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE; 2010. pp. 4617–4624.

19.

Pellegrinelli

Pedrocchi

. Estimation of robot execution time for close proximity human-robot collaboration. Integrated Computer-Aided Engineering. 2018 Dec; 25(1): 81–96.

20.

Vera

Pardo

, et al. DeepEye: Deep convolutional network for pupil detection in real environments. Integrated Computer-Aided Engineering. 2019 Dec; 26(1): 85–95.

21.

Eivazi

Santini

, et al. Improving real-time CNN-based pupil detection through domain-specific data augmentation. In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications; 2019. pp. 1–6.

22.

Zhang

, et al. A multi-aperture optical flow estimation method for an artificial compound eye. Integrated Computer-Aided Engineering. 2019 Feb; 26(2): 139–157.

23.

Zhang

, et al. Geometry based three-dimensional image processing method for electronic cluster eye. Integrated Computer-Aided Engineering. 2018 May; 25(3): 213–228.

24.

ZED stereo camera, SN 4226 (Stereo Labs. San Francisco, California).

25.

Bjelonic

. YOLO ROS: Real-Time Object Detection for ROS; 2018. Accessed on: Mar. 15, 2019. [Online]. Available: https://github.com/leggedrobotics/darknet_ros. leggedrobotics.

26.

Kuffner

LaValle

. RRT-connect: An efficient approach to single-query path planning. In: Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings. IEEE; Vol. 2. 2000. pp. 995–1001.

27.

Jia

Shelhamer

, et al. Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia. ACM; 2014. pp. 675–678.

28.

Kingma

. Adam: A method for stochastic optimization. arXiv:14126980. 2014.

29.

Lin

Maire

, et al. Microsoft coco: Common objects in context. In: European Conference on Computer Vision. Springer; 2014. pp. 740–755.

30.

Davies

. Computer and machine vision: theory, algorithms, practicalities. 4th ed. Egham, Surrey: Academic Press; 2012.

31.

Saho

. Kalman Filter for Moving Object Tracking: Performance Analysis and Filter Design. Kalman Filters-Theory for Advanced Applications. 2017.

32.

Carmona

Sung

, et al. Stable PID Control for Mobile Robots. In: International Conference on Control, Automation, Robotics and Vision; 2018. pp. 1891–1896.

33.

Chwa

. Tracking control of differential-drive wheeled mobile robots using a backstepping-like feedback linearization. IEEE Transactions on Systems, Man, and Cybernetics – Part A: Systems and Humans. 2010 Aug; 40(6): 1285–1295.

34.

Yang

Kim

. Sliding mode control for trajectory tracking of nonholonomic wheeled mobile robots. IEEE Transactions on Robotics and Automation. 1999 Jun; 15(3): 578–587.

35.

Hou

Zou

, et al. Adaptive Control of an Electrically Driven Nonholonomic Mobile Robot via Backstepping and Fuzzy Approach. IEEE Transactions on Control Systems Technology. 2009 Jul; 17(4): 803–815.

36.

Yang

Wang

, et al. Trajectory tracking for a wheeled mobile robot with an omnidirectional wheel on uneven ground. IET Control Theory & Applications. 2020 Jun; 14(7): 921–929.

37.

Huang

Hung

, et al. Smooth switching robust adaptive control for omnidirectional mobile robots. IEEE Transactions on Control Systems Technology. 2015 Jan; 23(5): 1986–1993.

38.

Cheon

Kim

. Online bidirectional trajectory planning for mobile robots in state-time space. IEEE Transactions on Industrial Electronics. 2019 Aug; 66(6): 4555–4565.

39.

Jetson TX1 developer board, SN 03221160052931 (NVIDIA Corporation. Santa Clara, California).

40.

Intel movidius computer stick, SN 03221160052931 (Intel Corporation. Santa Clara, California).s

YOLO evaluation results
Class	mAP	Class	mAP	Class	mAP
Person	51.2	Chair	20.8	Laptop	53.6
Bottle	21.3	Table	24.9	Mouse	48.7
Cup	25.1	TV	58.4	Keyboard	41.8

Multi-stage deep learning perception system for mobile robots

Abstract

Keywords

1. Introduction

1.1 Related works

1.2 Proposed perception system

2.2 Preprocessing step

3. Detection system

3.1.1 Neural network structure

3.2 Target detection

3.2.1 Neural network structure

3.3 Hand signal detection

3.3.1 Neural network structure

3.3.3 Dataset building process

4. Post-processing step

4.1 Spatial reconstruction

4.3 Decision scheme

4.4 Trajectory generation

5. Experimental results

5.1 Obstacle detection

Table 1 StixelNet, detection zone evaluation

Table 3 SignalNet evaluation results through validation dataset

5.5 Real-time implementation

5.6.1 Static objects

6. Conclusions

Footnotes

Acknowledgments

References

Table 1
StixelNet, detection zone evaluation

Table 3
SignalNet evaluation results through validation dataset