RGB-D,Laser and Thermal Sensor Fusion for People following in a Mobile Robot

Abstract

Detecting and tracking people is a key capability for robots that operate in populated environments. In this paper, we used a multiple sensor fusion approach that combines three kinds of sensors in order to detect people using RGB-D vision, lasers and a thermal sensor mounted on a mobile platform. The Kinect sensor offers a rich data set at a significantly low cost, however, there are some limitations to its use in a mobile platform, mainly that the Kinect algorithms for people detection rely on images captured by a static camera. To cope with these limitations, this work is based on the combination of the Kinect and a Hokuyo laser and a thermopile array sensor. A real-time particle filter system merges the information provided by the sensors and calculates the position of the target, using probabilistic leg and thermal patterns, image features and optical flow to this end. Experimental results carried out with a mobile platform in a Science museum have shown that the combination of different sensory cues increases the reliability of the people following system.

Keywords

Sensor Fusion People Tracking Mobile Robot

1. Introduction

Developing robots able to collaborate with humans in real-world domains such as material transport, rescue, health care, manufacturing, etc. poses numerous advantages. A requirement to achieve this objective is to develop the robot ability to accurately and robustly detect and track human partners in order to generate the proper behaviour. This research deals with the issues related to developing a reliable “person following” behaviour that allows the robot to accompany a human. Once the target person is detected, the robot attempts to drive directly toward the person's location. To this aim our approach is to endow the robot with multimodal perception that enables it to fuse information from multiple sensors, assimilate these multimodal data in real time and then respond at the timescale of the interaction.

The primary requirement of this research has been to investigate the realization of a human tracking system based on low-cost sensing devices. Recently, research on sensing components and software led by Microsoft has provided useful results for extracting the human pose and kinematics [1]. The Kinect motion sensor device offers visual and depth data at a significantly low cost. While the Kinect is a great innovation for robotics, it has some limitations. First, the depth map is only valid for objects that are further than 80cm away from the sensing device. A recent study [2] about the resolution of the Kinect proves that for mapping applications the object must be in the range of 1-3 m in other to reduce the effect of noise and low resolution. Second, the Kinect uses an IR projector with an IR camera, which means that sunlight could have a negative effect, taking into account that the sun emits in the IR spectrum. Third, the Kinect relies on algorithms for detection of human activities captured by a static camera. In mobile robot applications the sensor configuration is embedded into the robot, which is usually moving. As a consequence, the robot is expected to deal with environments that are highly dynamic, cluttered and frequently subject to illumination changes.

To cope with this, our work is based on the hypothesis that the combination of multiple sensors, a Kinect, a thermopile array sensor (Heimann HTPA thermal sensor) and a Hokuyo Laser can significantly improve the robustness of human detection. Thermal vision helps to overcome some of the problems related to colour vision sensors, since humans have a distinctive thermal profile compared to non-living objects (therefore human pictures are not considered as positive) and there are no major differences in appearance between different people in a thermal image. Another advantage is that the sensor data does not depend on light conditions so people can also be detected in complete darkness. As a drawback, some phantom detections near heat sources such as industrial machines or radiators may appear. Therefore, it is a promising research direction to combine the advantages of different sensing sources because each modality has complementary benefits and drawbacks as it has been shown in other works, [3], [4], [5], [6] and [7].

We have experimented in a Science museum with different elements exposed and people moving around and strong illumination changes due to weather conditions.

The rest of the paper is organized as follows: In Section II related work in the area of human tracking is presented. We concentrate mainly on work done using multiple sensor fusion for people tracking. Section III describes the proposed approach and Section IV the experimental setup. Section V shows experimental results and Section VI shows conclusions and future work.

2. Related work

People detection and tracking systems have been studied extensively due to the increasing demand of advanced robots that must integrate natural Human-Robot Interaction capabilities in order to perform some specific tasks for humans or in collaboration with them. As a complete review on people detection is beyond the scope of this work (an extensive work can be found in [8] and [9]) we focus on most related work.

To our knowledge, two approaches are commonly used for detecting people using a mobile robot. The first includes vision-based techniques and the second approach combines vision with other modalities, normally range sensors, such as laser scanners or sonar like in [10], [11]. Recent computer vision literature is rich in people detection approaches in colour images. Most approaches focus on a particular feature: the face [12]; [13], the head, [14], the upper body or the torso, [15], the entire body, [16], just the legs, [17], or multimodal approaches that integrate motion information [3]. All methods for detecting and tracking people in colour images on a moving platform face similar problems and their performance depends heavily on the current light conditions, viewing angle, distance to persons and variability of appearance of people in the image.

Apart from cameras, the most common devices used for people tracking are laser sensors. One of the most popular approaches in this context is to extract the legs' position by detecting moving blobs that appear as local minima in the range image. [18] presents a system for detecting legs and following a person with only laser readings. A probabilistic model of a leg shape is implemented, along with a Kalman filter for robust tracking. [19] addresses the problem of detecting people using multiple layers of 2D laser range scans. Other implementations such as [20] also use a combination of face and laser-based leg detection.

Most existing combined vision-thermal-based methods, [4], [5], [6], [7], concern non-mobile applications in video monitoring applications and especially for pedestrian detection where the pose of the camera is fixed. Another approach, [21], shows the advantages of using thermal images for face detection. They suggest that the fusion of both visible and thermal-based face recognition methodologies yields better overall performance.

However, to the author's knowledge, there is little published work on using thermal sensor information to detect humans using mobile robots. The main reason for the limited number of applications using thermal vision so far is probably the relatively high price of this kind of sensor. [22] shows the use of thermal sensors and grey scale images to detect people in a mobile robot. A drawback of most of these approaches is the sequential integration of sensory cues. People are detected by thermal information only and are subsequently verified by visual or auditory cues.

3. Proposed approach

We propose a multimodal approach, which is characterized by parallel processing and filtering of sensory cues, as is shown in Figure 1. Since our algorithm is intended to run in a robot, our implementation is based on the ROS system [23].

Figure 1.

Approach combining three input clues from RGB-D sensor, laser and thermal sensor using a particle filter approach.

The People tracking system is based on a Hokuyo laser sensor, a HTPA thermal sensor developed by Heimann and a Kinect sensor, mounted on a RMP Segway mobile platform, which are shown in Figure 2. Each sensor provides input to three detection units: Leg detection, Vest Detection and Thermal Detection explained later in this section. The Detection units feed target candidates to the Particle filter where the information is fused. The Best candidate from all the particles is then selected to set the position to be reached by the robot in order to follow the target person.

The HTPA allows the measurement of the temperature distribution of the environment, where very high resolutions are not necessary, such as person detection, surveillance of temperature critical surfaces, hotspot or fire detection, energy management and security applications. The thermopile array can detect infrared radiation; we convert this information into an image where each pixel corresponds to a temperature value. The sensor only offers a 32×31 image that allows a rough resolution of the environment temperature. The benefits of this technology are very small power consumption, as well as the high sensitivity of the system.

Kinect provides depth data that we transform into depth images, it uses near infrared light to illuminate the subject and the sensor chip measures the disparity between the information received by the two IR sensors. It provides a 640×480 distance (depth) map in real time (30fps). In addition to the depth sensor, the Kinect also provides a traditional 640×480 RGB image.

The scanning laser range finder chosen for the leg and obstacles detection tasks is a Hokuyo UTM-30LX. This laser provides a measuring area of 270 angular degrees, from 0.1 to 30m in depth and an angular resolution of 0.25 (1080 readings per scan).

Figure 2.

The used robotic platform: a Segway RMP 200 provided with the Kinect, a Hokuyo laser and the thermal sensor

2.1 Leg detection

The proposed Leg Detection makes use of the probabilistic leg pattern presented in previous works [18]. The proposed system has implemented a leg model as a sequence of max >min > max > min > max given the laser readings (as in [3]), see Figure 3. Based on the five points shown in the figure (P_a, P_b, P_c, P_d and P_e) different measures are defined, such as the distance between the legs and the distance between the legs and the background.

Figure 3.

Leg pattern based on the work of Belloto et al. [3].

2.2 Vest detection

As stated above, RGB-D images are used to track the target and add information to the particle filter. The position of the target will be estimated by Vest Detection and this information will be provided by means of the distance from the robot's centre, the angle from the robot's front direction and its height. Taking into account the proposed particle filter's structure, only the information on the angle and distance will be used in further steps of the proposed approach. The next paragraphs will give details about Vest Detection.

The vest detection method is intended to be used with a person that is wearing an emergency vest. The foreseen applications for the people following behaviour implemented by the robot are mainly related with robots supporting emergency personnel, in rescue activities for instance. In this method an RGB filter is applied in order to focus the attention towards the emergency vest's yellow colour, obtaining a binary image where the white pixels correspond to the target colour. After that, the binary image is filtered by erasing first the very small white areas and then applying morphological dilation and erosion operations (the result is shown in Figure 4 (a)). If there is more than one white region in the resulting image, the one with the bigger area is selected. The chosen area is used to extract some image features (corners with big Eigenvalues) to be tracked in the received images (see Figure 4 (b)). The optical flow is calculated for the detected corners using the Lucas-Kanade [24] method. In each frame, after calculating the optical flow, the centroid of the corners will be extracted, this being the target's estimated position.

Figure 4.

Binary image original with features Emergency vest detection

To increase the reliability of the tracking avoiding errors, especially when the target turns or changes its perspective, the image features are recalculated each time the distance between them changes or some points disappear.

2.3 Thermal detection

People present a thermal profile different from their surrounding environment. The temperature detected in the pixel corresponding to a person is usually around 37 degrees Celsius, with a slight tendency to be slightly lower, due to the presence of hair or clothes over the skin.

We implement a procedure that, given a thermal image, computes a vector of 32 floating point numbers. The nth element of this vector corresponds to the estimation of the probability a person is in column n (a column of 31 elements).

This computation is performed in three steps:

–

First of all, a likelihood of corresponding to a person is assigned to every thermopile pixel. This likelihood is computed under the assumption that the temperature of a person is normally distributed with mean μ and standard deviation σ. To compute the mean and the standard deviation we have taken several thermal images of people standing in front of the sensor and the pixels corresponding to those images have been taken into account to compute the parameters. Several values have been tested and values of mean = 36 andstandard deviation = 2 have been used with good experimental results.

–

Then smoothing of the likelihood matrix is performed by convolution with a Gaussian kernel with a width of five pixels [25].

–

And finally the maximum of column n is assigned to the nth element of the aforementioned likelihood vector.

2.4 Fusion using particle filter

Once the Leg Detection, Vest Detection and Thermal Detection are defined, it is necessary to merge and fuse this information to perform complete person tracking. To this end a Sequential Importance Resampling (SIR) [26] particle filter is proposed, as it fits in this kind of nonlinear and non-Gaussian problem.

Focusing on the posed problem, the state in time t will be defined as a pair of x and y positions, X(t)=[x(t), y(t)]. As it is not possible to model the target's movement pattern a priori, the state transition is defined as

X (t) = X (t - 1) + V (t - 1)

(1)

where X(t-1) is the previous state vector and V(t-1) is the process noise.

The observation, on the other hand, is defined by the information provided by Leg Detection, Vest Detection and Thermal Detection. So based on those three information sources each particle will be defined by a leg likelihood, a vest likelihood and a thermal likelihood.

Finally the tracking procedure of the particle filter is done as shown in Figure 5.

Figure 5.

People following algorithm using particle filter.

2.4.1 Leg Likelihood

As discussed above, the Leg Detection based on previous work [3] will analyse each laser reading set and will provide an array of possible leg angles and depths with a likelihood above a given threshold. To calculate the leg likelihood for each state or particle the nearest possible leg is detected from the information provided by the Leg Detection Unit. Based on this leg position the leg likelihood is calculated as

d i s t = \sqrt{r_{l a s e r}^{2} + r_{t}^{2} - {2 r}_{l a s e r} r_{t} c o s (θ_{l a s e r} - θ_{t})}

(2)

P_{l e g} (X_{t} | Z_{t}) = Φ (r_{l a s e r}) e^{- d i s t}

(3)

where dist is the radial distance between the particle and the legs detected by the laser scan and P_leg is calculated as the multiplication of the likelihood of the leg pattern described in Section 2.1 and an exponential function, returning smaller values as the distance increases.

2.4.2 Vest Likelihood

The information provided by the Vest Detection is also used to compute the camera likelihood. This likelihood is calculated using the angle estimated by means of the image features and optical flow. The probability is calculated as:

d i s t = {θ_{v e s t} - θ}_{t} ¶

(4)

P_{v e s t} (X_{t} | Z_{t}) = e^{- d i s t}

(5)

where dist is the distance between a particle's angle and the detected feature's central position and P_vest is calculated as an exponential function.

2.4.3 Thermal Likelihood

The thermal likelihood is calculated using the information provided by Thermal detection. To that end, for each particle the nearest value of the evidence array that exceeds a certain threshold is detected and, based on this angle, the likelihood is calculated as

d i s t = {θ_{t h e r m} - θ}_{t} ¶

(6)

P_{t h e r m} (X_{t} | Z_{t}) = Φ (r_{t h e r m} {) e}^{- d i s t}

(7)

where dist is the distance between the particle and the person detected by the thermal sensor and P_therm is calculated as the multiplication of the likelihood of the thermal profile of the person and an exponential function.

2.4.4 Final Likelihood

Once the likelihoods are calculated, the final likelihood of a state given an observation is calculated as

P (X_{t}| Z_{t}) = {α P}_{v e s t} (X_{t}| Z_{t}) + {β P}_{l a s e r} (X_{t}| Z_{t}) + γ P_{t h e r m} (X_{t} | Z_{t})

(8)

where P_vest(X_t|Z_t)), P_laser(X_t|Z_t) and P_therm(X_t|Z_t) are the likelihoods calculated from the information provided by Leg Detection, Vest Detection and Thermal Detection respectively and α, β and γ are coefficients to weight the three likelihoods. The addition of these coefficients allows more importance to be given to one of the information sources if desired. During the evaluation process a set of values are tested in order to obtain a good performance.

3. Experimental evaluation

In this section, we discuss the experimental evaluation of our approach. This section describes the tests that have been carried out in order to assess the performance of the different feature detectors considered, with regard to the task described in the previous sections.

We quantitatively evaluate our algorithm using data collected with a mobile robot moving in a Science museum.

During the experiments, the robot was remotely controlled. We asked different people to walk naturally in front of the robot. We collected two sets of data (320×240 RGB-D image+284 laser+32×31 thermal) at a frequency of 4Hz in different areas of the museum. The fist dataset contains 2862 data and the second dataset 2300 images.

Figure 6 shows some images taken by the robot. As it can be seen, lighting conditions affect to the image treatment, as there are crystal corridors in the museum (Figure 6 (a) and Figure 6 (b)). In addition there are some aesthetic elements that are detected as people, such as the big figure in the door in Figure 6 (c).

Figure 6.

Eureka! Science Museum

For all the datasets, we hand-annotated the position of the people to be tracked, selecting the centre of the vest as a target point. From the depth images we consider a bound of 20 pixels in order to extract the distance from the robot. The annotation is provided for four images per second. The picture in the right corner of Figure 6 (d) shows a people following sequence where the annotated position is represented.

In order to establish the relative position between the target person and the robot a three-dimensional cylindrical coordinate system is defined, taking into account the posed problem and the available sensors as shown. Each target's position will be defined with three values (ρ,φ,z) where ρ represents the Euclidean distance from the robot's centre, φ represents the angle from the robot's front direction and z is the height as shown in Figure 7. Taking into account that the robot's movement type is restricted to a plane, it was decided to leave out the z-axis in this step of the experiment as it does not add any information to the tracking process.

Figure 7.

Robot coordinate system

Figure 8 represent the range of data in the angle and depth used in the evaluation process.

Figure 8.

Annotated ground truth data for the two datasets used in the experiments (angle and depth position of the target person). The target persons are in an angle range of 50-320 pixels and a depth range of 100-200cm.

4. Experimental results

The performance of the people tracking system is evaluated in terms of error in the estimation, in distance and in angle, during person following tasks. Table 1 and Table 2 show a summary of the mean and standard deviation of errors obtained by the different tracking systems and the combination of them. The evaluation has been performed using the detectors individually: only Legdetection, only Vest detection, only Thermal detection and in combination of two giving the same weight to both detectors (vest and thermal, leg and thermal, leg and vest) and in combination of three with different weights. It should be noted that the fusion of the three sensor sources improves in both angle estimation and distance estimation; the results obtained by each estimator individually or in combination of two can be seen in Figure 9 and Figure 10. The Vest detector provides the best estimations individually as well as in combination with other sensors. For this reason in the final combination the Vest detector has a higher weight than the other sensors.

Figure 9.

Mean errors in angle estimation for the different combinations used in the two datasets.

Figure 10.

Mean errors in depth estimation for the different combinations used in the two datasets.

Table 1.

Results in term of error in angle (degrees). L refers to leg detection, V vest detection and T thermal detection. The number before the letter refers to the weight given to each detector.

	Dataset 1		Dataset 2
Sensor	Mean	Standard deviation	Mean	Standard deviation
L	66,17	53,37	100,02	56,19
V	30,68	28,26	29,77	22,024
T	69,35	32,11	85,69	32,11
0.5V0.5T	69,43	31,65	30,20	23,36
0.5L0.5T	56,26	49,18	104,91	54,79
0.5L0.5V	28,42	25,58	29,71	25,17
0.3L0.3V0.3T	33,29	32,75	30,49	24,50
0.15L0.7V0.15T	17,44	22,54	14,91	14,51

Table 2.

Results in term of error in depth (m). L refers to leg detection, V vest detection and T thermal detection. The number before the letter refers to the weight given to each detector. The thermopile sensor is not used to estimate the position in depth of the tracked person.

	Dataset 1		Dataset 2
Sensor	Mean	Standard deviation	Mean	Standard deviation
L	0,51	0,45	0,94	0,43
V	0,27	0,33	0,39	0,19
0.5V0.5T	0,62	0,25	0,39	0,22
0.5L0.5T	0,50	0,45	0,97	0,46
0.5L0.5V	1,87	0,28	0,39	0,20
0.3L0.3V0.3T	0,25	0,29	0,38	0,21
0.15L0.7V0.15T	0,10	0,31	0,40	0,22

Laser and thermal detection individually provide the worst results, as well as the combination of both. It seems that a thermal sensor in combination with the rest of the sensors deteriorates the estimations and in the case of the laser and vest combination the results are better than laser vest and thermal with the same weight. However, the final combination considering the thermal information with a weight of 0.15 improves the vest and laser combination.

For the results achieved in the working range (1 to 2m distance to the target) with the final combination of sensors it is clear that the best behaviour in general is obtained by the particle filter combining all the sensory cues.

5. Conclusions

In this paper, we have introduced a multimodal approach to detect and track people in indoor spaces from a mobile platform. This approach has been designed to manage three kinds of input images, colour, depth and temperature to detect people. As shown in the experimental evaluation, using complementary sensors and fusing their results using particle filtering results in robust and accurate person detection.

In the near future we aim:

–

To develop improved detectors combining/fusing visual cues using particle filter strategies, including face recognition and motion information, in order to track people gestures.

–

To improve the algorithm's parameter adjustment without hand tuning, using machine learning approaches.

–

To integrate with robot navigation planning ability to explicitly consider humans in the loop during robot movement.

–

To extend to other scenarios. This is a first implementation of a system that can be extended toward an outdoors scenario.

Footnotes

6. Acknowledgements

The work described in this paper was partially conducted within the ktBOT project and funded by KUTXA Obra Social.

References

Shotton

Fitzgibbon

Cook

Sharp

Finocchio

Moore

Kipma

and Blake

. Realtime human pose recognition in parts from a single depth image. Computer Vision and Pattern Recognition, 2011.

Khoshelham

and Elberink

S.O.

. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, 12 (1):1437–1454, 2012.

Bellotto

and Hu

. A bank of unscented Kalman filters for multimodal human perception with mobile service robots. International Journal of Social Robotics, 2 (2):121–136, 2010.

St-Laurent

Pravost

and Maldague

. Thermal imaging for enhanced foreground-background segmentation. The 8^th Quantitative Infrared Thermography (QIRT) conference, Padova, Italy, June 28 – 30, 2006.

Aliakbarpour

Hofmann

Kaiser

and Rigoll

. Fusion of multimodal sensors in a voxel occupancy grid for tracking and behaviour analysis. Proc. 12th Intern. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Delft, The Netherlands, April 13 − 15, 2011.

Johnson

M. J.

and Bajcsy

. Integration of thermal and visible imagery for robust foreground detection in tele-immersive spaces. Information Fusion, 2008 11th International Conference on, 2008.

Zin

Takahashi

and Hama

. Fusion of infrared and visible images for robust person detection. Image Fusion, 2011. InTech, Available from: http://www.intechopen.com/articles/show/title/fusion-of-infraredand-visible-images-for-robust-persondetection.

Schiele

. Visual People Detection - Different Models, Comparison and Discussion. IEEE International Conference on Robotics and Automation (ICRA), May, 2009.

Cielniak

. People Tracking by Mobile Robots using Thermal and Colour Vision. PhD thesis, Department of Technology Orebro University, 2007.

10.

Guan

L.Y.

S.S.

and Loh

A. P.

. Robust human detection and identification by using stereo and thermal images in human-robot interaction. International Journal of Information Acquisition, 4 (2):122, 2007.

11.

Kalyan

Balasuriya

Ura

and Wijesoma

. Sonar and vision based navigation schemes for autonomous underwater vehicles. International Conference on Automation, Robotics and Computer Vision, I:437–442, 2004.

12.

Hjelmas

and Low

B. K.

. Face detection: A survey. Computer Vision and Image Understanding, 83 (3):236–274, 2001.

13.

Yang

M.H.

Kriegman

and Ahuja

. Detecting faces in images: A survey. Transactions on Pattern Analysis and Machine Intelligence, 24 (1):3458, 2002.

14.

Murphy-Chutorian

and Trivedi

M. M.

. Head pose estimation in computer vision: A survey. IEEE Transcations on Pattern Analysis and Machine Intelligence, 31 (4):607–626, 2009.

15.

Kruppa

Santana

M. Castrillón

and Schiele

. Fast and robust face finding via local context. Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS), pages 157–164, 2003.

16.

Dalal

and Triggs

. Histograms of oriented gradients for human detection. International Conference on Computer Vision & Pattern Recognition CVPR 2005 2:886–893, 2005.

17.

Papageorgiou

and Poggio

. A trainable system for object detection. International Journal of Computer Vision, 38 (1):15–33, 2000.

18.

Martinez-Otzeta

J. M.

Ibarguren

Ansuategui

and Susperregi

. Laser based people following behaviour in an emergency environment. Proceedings of the 2nd International Conference on Intelligent Robotics and Applications, ICIRA '09, pages 3342, Berlin, Heidelberg, 2009. Springer-Verlag. ISBN 978-3-642-10816-7, 2009.

19.

Mozos

O. Martinez

Kurazume

and Hasegawa

. Multipart people detection using 2D range data. International Journal of Social Robotics, 2 (1):3140, 2010. ISSN 1875-4791.

20.

Bellotto

and Hu

. Multisensor data fusion for joint people tracking and identification with a service robot. 2007 IEEE International Conference on Robotics and Biomimetics ROBIO: 1494–1499, 2007.

21.

Gundimada

Asari

V. K.

and Gudur

. Face recognition in multi-sensor images based on a novel modular feature selection technique. Inf. Fusion, 11:124–132, 2010.

22.

Treptow

Cielniak

and Duckett

. Active people recognition using thermal and grey images on a mobile security robot. Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2005), Edmonton, Canada, August 2-6 2005.

23.

Quigley

Gerkey

Conley

Faust

Foote

Leibs

Berger

Wheeler

and Ng

. ROS: an open-source Robot Operating System. ICRA Workshop on Open Source Software, 2009.

24.

Bruce

L. D.

and Kanade

An iterative image registration technique with an application to stereo vision. Proceedings of the 7th International Joint Conference on Artificial Intelligence, 2:674–679, 1981.

25.

Ozkok-Ataser

Kovacs

Ciobanu

Dancea

and Vrtovec

. 2D edge detection using cost minimization. http://ssip2003.info.uvt.ro/projects/teamE/abstract.htm

26.

Arulampalam

Maskell

Gordon

N. J.

Clap

, A Tutorial on Particle Filters for On-line Nonlinear/Non-Gaussian Bayesian Tracking, IEEE Transactions of Signal Processing, 50 (2):174–188, 2002.