Distributed Intelligent Assistance Robotic System with Sensor Networks Based on Robot Technology Middleware

Abstract

This paper proposes a distributed intelligent assistant robotic system in order to improve the quality of the elderly people's life in the population aging society. The system is composed of embedded distributed sensor networks and multirobot intelligent platform. In the proposed system, we use SP²ATM (simultaneous path planning and topological mapping) with RBPF (Rao-Blackwellized particle filter) for path planning and localization of mobile robots. In order to perform service tasks, an accurate and reliable 3D environment map is reconstructed by using an effective 3D reconstruction technique from multiview stereo. Besides, the system provides user human tracking services based on multifeature mean shift under the double-layer locating mechanism. To improve the feasibility and reliability of the system, this proposed system is developed based on distributed control technology RTM (robot technology middleware). This paper presents the architecture of the proposed system and the experimental results verify effectiveness of the approaches.

1. Introduction

Assistance robotic system is particularly important for elderly people, with the increasingly serious aging problem. In recent years a lot of research results have been achieved in this field. Chung et al. [1] developed an intelligent service robot to help humans transport the object which they want to move. Rampinelli et al. [2] designed a system capable of obtaining the pose of robotic wheelchairs by the cameras in fixed positions to support the aged and disabled. Both Song et al. [3] and Tseng et al. [4] proposed a surveillance robot to monitor the home environment for security. van Osch et al. [5] developed a teleoperated robot which was controlled by a human operator to perform tasks normally done by humans in an environment. Alvarez-Santos et al. [6] presented a tour-guide robotic system that is able to communicate with users by recognizing hand gestures and providing voice feedback. Lu et al. [7] built an intelligent home space for service robot based on multipattern information model and wireless sensor networks. In order to simplify the design process of assistance robotic system and increase reusability of robotic modules, distributed technology was introduced by [8–10].

Although service robot has been widely researched, some key technologies such as path planning, 3D reconstruction, and human tracking are bottlenecks for developing intelligent assistant robotic system. To address this problem, researchers have done many works. Oriolo [11] adopted SRT (sensor-based random tree) exploration method and randomly selected from the boundary the next exploration target. However, the positioning error restricted mobile robot to realize autonomous exploration in the case of high position accuracy. There exist many approaches to 3D reconstruction from pairs of images [12, 13]; these approaches were typically too computationally intense for real-time applications. In order to track targeted person, many researchers have accomplished some achievements. Cipolla and Masanobu [14] primarily considered foreground segmentation based on disparity image. Other approaches [15, 16] assumed that face detection could be used in the following system. However, the real-time tracking performance of these algorithms needs to be improved. The algorithm [17] perfectly eliminates the background interference to target model and track, but the tracking based on color cannot accurately locate target in the case that there are similar color objects in environment.

In this paper, we focus on modular design theory and the key technologies for assistance robotic system. We developed a distributed intelligent assistant robotic system based on RTM. The indoor assistant robotic system is composed of embedded distributed sensor networks and multirobots platform. Distributed sensor networks, which transplant open RTM-aist on the embedded platform under Cortex-A8 framework, are adopted to perceive indoor environment. And robot can accurately deliver the objects which the user wants to the place where the aged or disabled is by robot path planning and positioning systems. We developed SP²ATM algorithm with RBPF to realize path planning in an unknown environment. This algorithm effectively improves the location precision of mobile robots and lowers computation complex to $O (N \cdot \log 2 M)$ . In order to perceive home environment, we propose an effective 3D reconstruction approach, which embeds optical flow-scene flow framework into stereo vision-based 3D scene reconstruction for an environment-learning assistance mobile robot. Compared with the other methods, the proposed approach has the advantage of being robust to illumination change and the accuracy of the deformed surface of the environment can be improved. Besides, aiming at realizing a human following task in the system, a human tracking method based on multifeature mean shift under the double-layer locating mechanism is proposed to solve the problem of distinguishing target, occlusion, and quick turning. In order to enable the system to be extended and integrated easier for a new system or new applications, distributed control technology RTM is introduced into the developed assistance robotic system. RTM enables assistance system to be language independence and operating system independence. RTM aims to modularize robotic functional elements as RTC (RT component). RTCs of the system can be implemented by different programming languages, run in different operating system, or connected in different networks to interoperate. This paper details the architecture of the proposed system and gives some experimental results to verify the effectiveness of the approaches.

The remainder of this paper is organized as follows: Section 2 describes the overview of system. Section 3 presents the developed key technologies for the assistant robotic system. Section 4 details the implementation of RTC. The experimental results are given in Section 5 and Section 6 concludes the paper.

2. System Overview

The proposed system is shown in Figure 1. Compared with the main methods for robot to get environment information by equipped camera and many expensive sensors on the classical robotic platform, the approach of distributed sensor networks proposed in this paper has many advantages such as parallelism, flexibility, and fault tolerance, which can share the information from sensors, overcome the uncertainty of the sensors, and improve the efficiency of performing complex tasks. The robotic system is implemented by Open RTM-aist based on distributed technology. Embedded cameras in distributed sensor networks (implemented in Cortex-A8) are installed in the rooms and corridors to monitor the indoor environment. The robot (xPartner robot AS-R) which is mounted with URG scanning LRF (laser range finder) and Flea2 digital camera uses the improved SP²ATM with RBPF-SLAM algorithm to help user transmit object. Meanwhile, the 3D map can be reconstructed by the stereo vision system which is installed on xPartner robot, when the xPartner robot automatically moves in the room. In addition, aiming at realizing a human following task in a cluttered environment, the robot (an American Mobile Robot Inc. Pioneer 3-DX) which is mounted with a stereo camera and a RFID uses multifeatures and various kinds of sensory information to provide user human tracking service.

Figure 1

System structure.

3. Key Technologies

Three key technologies (path planning, 3D reconstruction, and human tracking) are developed in the distributed intelligent assistant robotic system. The system can not only provide object delivery service based on SP²ATM with RBPF-SLAM algorithm but also provide users with monitor tracking service by multifeature mean-shift algorithm. Meanwhile, the position of the robots can be displayed on the 3D reconstruction environment maps in real time.

3.1. Path Planning

Path planning is the key technology to provide object delivery service. Under the grid-mixed topology map model, SP²ATM algorithm is easily implemented to path planning and does not need more storage space. In addition, this algorithm can effectively reduce the space complexity and computational complexity of the large-scale environment grid map. Besides, based on the structure of HTM (hierarchical topology map), node update rules provide a global optimal path planning strategy to improve exploration efficiency further. On this basis, RBPF-SLAM algorithm based on the grid map is introduced into the robot autonomous exploration mission. The node is constantly updated by using particle filter during the position estimation of the mobile robot. This algorithm effectively improves the location precision of the topological node, thus preventing robot operation failure caused by accumulated position error.

The improved SP²ATM with RBPF-SLAM algorithm is mainly adopted in the function module to achieve object delivery tasks. In this part, SP²ATM algorithm [19, 20] is used for environment exploration and the RBPF-SLAM is adopted for robot localization. AST [21] (admissible space tree) is a set of nodes and graphs demonstration, which is addressed to describe the admissible space in the environment and facilitate map building for mobile robot. After that, HTM [21] is maintained with two-layered structure which describes, respectively, the possible nodes generated by AST and an undirected graph of topology nodes.

Sometimes, some nodes which are closed to the established topology or possible nodes may not belong to a former topology's AST. In order to avoid revisiting and guarantee the global convergence of instant goal node, node updating rules [21] are used to append, rebind, or get rid of those nodes.

When the robot fails to find any possible nodes in current AST, it is necessary to backtrack to a nearest topology node which is determined by Bellman-Ford algorithm adopted in [21].

In order to improve the location precision of the topological node, RBPF-SLAM is used for path planning. The complexity of the SLAM problem arises from the very high-dimensional state space which consists of the variables describing the environment state and the robot pose. PF (particle filter) is an effective Monte-Carlo approach for SLAM. Murphy presented a RBPF-based mobile robot SLAM. The general idea of this approach is the decoupling of map estimation and robot pose determination based upon the global map estimation. Compared with EKF, RBPF requires less demanding; its computation complex is reduced to $O (N \cdot \log 2 M)$ . In our scheme, for saving memory, each particle carries a sample pose of robot individually and all particles share a common environment map.

The SP²ATM with RBPF-SLAM algorithm is detailed in Algorithm 1.

Algorithm 1: SP²ATM with RBPF-SLAM.

(1) Collect Laser data;

(2) Robot Pose Prediction;

(3) Particle sampling according to Proposal distribution;

(4) Scan matching for each particle;

(5) Update of particle weight;

(6) Compute effective particle number;

(7) If need resampling

(8) if need resampling

(9) Particle resampling

(10) End;

(11) Posterior estimation

(12) Get AST and section;

(13) Obtain possible nodes; When the AST is built, the furthest node is chosen as the current node ${P_{j}}^{c}$ , $P^{c} = {{P_{j}}^{c}}$

(14) Build HTM; $T_{i} = {A, P}$ The current node $T_{i}$ the current robot pose information and the AST information; Where, P is the current robot pose;

(15) Update the nodes;

(16) Get topological nodes;

(17) Map building;

END

As Shown in Figure 2, the service robot can generate topological maps to help users deliver objects using Algorithm 1.

Figure 2

Path planning and topological mapping.

3.2. Scene Flow-Based Environment 3D Reconstruction

When the robot automatically moves in the room, users can check the current robot position on the 3D reconstruction environment map. An effective reconstruction approach, which embeds optical flow-scene flow framework into stereo vision-based 3D scene reconstruction for an environment-learning mobile robot, is proposed. This part is inspired by the idea of casting the surface adjustment problem as the nonrigid motion recovery issue. As optical flow vector field estimation outperforms traditional dense disparity for its inherent advantage of being robust to illumination change and being optimized and smoothed in global sense, the deformed surface is improved in quality dramatically. Block diagram of reconstruction scheme is shown in Figure 3.

Figure 3

Block diagram of reconstruction scheme.

For the proposed 3D reconstruction technology in this paper, SIFT (scale-invariant feature transform) corner detector is employed for interesting feature points extraction.

The multiview camera setup parameters, including focal, principal point, and baseline length, must be carefully calibrated based on the digitized points derived by triangulation measurement.

To improve position accuracy of the calculated point set in practical implementation, bundle adjustment technique is employed to refine the scattered points reconstruction to produce jointly optimal 3D structure and viewing parameter (camera pose and/or calibration) estimates concurrently [22], since both of which are corrupted by noise.

Now, building a raw deformable surface mesh based on the acquired discrete points efficiently is considered. For an implicit surface building mission, energy minimization can be solved using RBF (radial basis function), a well-established mathematical tool for solving scattered data interpolation problems [23], which accounts for its popularity in surface recovery applications.

Implicit surface tessellation is needed for surface digital presentation in computer and for human visualization, and usually implicit surface is converted into triangulated models. During this process, two steps are mainly involved. Firstly, 3D space is divided into adjacent cubic cells whose corners are evaluated in the implicit surface function: negative values are considered inside the surface, positive values outside. Then, within each cell, the intersections of cell edges with the implicit surface are connected to form one or more polygons.

The adjustment problem is cast as a nonrigid motion recovery issue. For this purpose, the reference frame image is back-projected onto the base mesh model surface and the textured model is projected into the comparison camera frame to synthesize an image prediction in that camera [24]. The synthesized image is in fact a feedback to the grabbed comparison image frame, depending upon the assumption that the reconstructed polygonal mesh surface is the true description of real scene structure.

The main objective of optical flow calculation is to derive a flow field which estimates the motion of pixels in two consecutive image frames [25] and, in turn, to estimate the relative motion between the camera and the objects in the scene. Consider

\begin{matrix} \min E = \min (\int_{Ω}^{} | \nabla u | + λ | I_{1} (x + u) - I_{0} (x) | d Ω) . \end{matrix}

(1)

As demonstrated in Figure 4, the point $X_{j}$ denotes the projected point on primitive mesh model along a reference camera ray and $u_{j}^{i}$ is the back projected image point on the ith comparison camera corresponding to intersection $X_{j}$ , while $u_{j}^{' i}$ denotes real observation. The raw estimation is refreshed into an updated position:

\begin{matrix} X_{j}^{'} = X_{j} + {\bar{e}}_{j} . \end{matrix}

(2)

Figure 4

Surface adjustment based on scene flow.

The direction and the corresponding unit vector ${\bar{r}}_{j}$ along the ray specified by the projection center of reference camera and the current image point is easy to be determined; therefore,

\begin{matrix} {\bar{e}}_{j} = {\bar{r}}_{j} \cdot ρ_{j}, \end{matrix}

(3)

where $ρ_{j}$ signifies magnitude of this vector. Equation (3) multiplies by jacobian $J_{X_{j}}^{i}$ matrix and yields

\begin{matrix} Δ {\vec{u}}_{j}^{i} = J_{X_{j}}^{i} \cdot {\vec{r}}_{j} \cdot ρ_{j} = ξ_{j}^{i} \cdot ρ_{j}, \end{matrix}

(4)

where $ξ_{j}^{i}$ is the 2 × 1 inner product between each Jacobian matrix with its associated reference ray vector ${Δ \vec{u}}_{j}^{i}$ the estimated optic flow vector in the ith image frame, which is determined by the above stated TV-L1 method [25]:

\begin{matrix} Δ {\vec{u}}_{j}^{i} = u_{j}^{' i} - u_{j}^{i} . \end{matrix}

(5)

So, the adjusted raw polygonal mesh based on scene flow is applied to obtain accurate 3D reconstruction model of real environment to enable users to understand the current position of robot.

3.3. Person Tracking

The assistance robot can also provide the human tracking service. A method which fuses multifeatures and couples various kinds of sensory information is proposed in this paper. Because the RF signals have an excellent range in indoor environments and the tag ID can distinguish other passersby and followers, the RFID detection system is embedded in our system (Figure 5). In order to deal with the unstable signals of radio frequency [26], the approach applies the Bayesian rule to determine the location of the object.

Figure 5

Color-texture used for tracking.

In order to improve the precision of localization, stereo cameras are adopted to track the target by multifeature. The image preprocessing of stereo vision benefits from the coarse location estimated by RFID. Then, prelocation based on head-shoulder is utilized to determine the candidate targets.

However, the single feature cannot reliably distinguish target between other people. Therefore, a multifeature mean shift to track the target further is utilized as shown below.

In the mean-shift tracking algorithm, the desired object is firstly selected with the position given by prelocation based on head-shoulder. The pixel locations inside the rectangle are denoted by ${x_{i}}$ , $i = 1,2, 3, \dots, n$ . The selected region is considered the object model, where its color histogram is calculated by

\begin{matrix} {\hat{q}}_{u} = C_{q_{u}} \sum_{i = 1}^{n} k_{new} ({∥ \frac{x_{i} - x_{0}}{h} ∥}^{2}) δ [b (x_{i}) - u], \\ k_{new} ({∥ \frac{x_{i} - x_{0}}{h} ∥}^{2}) = k ({∥ \frac{x_{i} - x_{0}}{h} ∥}^{2}) \times B_{hs} (x_{i}), \\ B_{hs} (x_{i}) = {\begin{cases} 1, & if x_{i} \in headshoulder \\ 0, & if x_{i} \notin headshoulder . \end{cases} \end{matrix}

(6)

Here, $B_{hs} (x_{i})$ is the binary image of the silhouette. $k_{new} ({∥ (x_{i} - x_{0}) / h ∥}^{2})$ is the improved convex kernel function based on the silhouette of target, which is utilized to overcome color or texture clutter in background, various illuminations, and other related limitations. As a consequence, the value of new kernel in the background is 0. While in the foreground, the value of new kernel is the same with Epanechnikov function. $b (x_{i})$ is the bin number $(1,2, \dots, m_{c})$ associated with the color at the pixel of normalized location $x_{i}$ , $δ (x)$ is the delta function, and $C_{q_{u}}$ is the constant normalization.

The pixel locations of the candidate area, centered at y in the current frame, are represented by ${x_{i}}$ , $i = 1,2, \dots, n_{h}$ . The probability of the color u in the candidate model is given by

\begin{matrix} {\hat{p}}_{u} (y) = C_{h_{u}} \sum_{i = 1}^{n_{h}} k_{new} ({∥ \frac{y - x_{i}}{h} ∥}^{2}) δ [b (x_{i}^{*}) - u], \end{matrix}

(7)

where $C_{h_{u}}$ is the constant normalization.

Local binary pattern (local binary patterns, LBP) is an effective texture description operator, and we implement LBP to describe the texture of person's clothes. Similarly, the texture histogram of the target is defined as

\begin{matrix} {\hat{q}}_{v} = C_{q_{v}} \sum_{i = 1}^{n} k_{new} ({∥ \frac{x_{i} - x_{0}}{h} ∥}^{2}) δ [t (x_{i}) - v], \end{matrix}

(8)

where $C_{q_{v}}$ is the constant normalization and $t (x_{i})$ is the bin number $(1,2, \dots, m_{t})$ .

The probability of the texture v in the candidate model is given by

\begin{matrix} {\hat{p}}_{v} (y) = C_{h_{v}} \sum_{i = 1}^{n_{h}} k_{new} ({∥ \frac{y - x_{i}}{h} ∥}^{2}) δ [t (x_{i}^{*}) - v], \end{matrix}

(9)

where $C_{H_{v}}$ is the constant normalization.

In this system, both color and texture features are integrated into the mean-shift tracking framework. In order to combine the target color and texture information, a color-texture similarity function based on Bhattacharyya coefficient is introduced to seek the position of maximum likeness between the object and candidate models:

\begin{matrix} \hat{ρ} (y) = {\hat{ρ}}_{u} {\hat{ρ}}_{v} = \sum_{u = 1}^{m_{c}} \sqrt{{\hat{p}}_{u} (y) {\hat{q}}_{u}} \sum_{v = 1}^{m_{t}} \sqrt{p_{v} (y) q_{v}}, \end{matrix}

(10)

where $\hat{ρ} (y)$ is color-texture similarity function, ${\hat{ρ}}_{u}$ is color Bhattacharyya coefficient, and ${\hat{ρ}}_{v}$ is texture Bhattacharyya coefficient. The goal in each frame is to estimate the object that maximizes (10). It should be noted that $y_{0}$ is the estimated object location in the previous frame. The Bhattacharyya coefficient is approximated in the current frame by its first-order Taylor expansion

\begin{array}{l} \hat{ρ} (y) \approx \frac{C_{h}}{2} \sum_{i = 1}^{n_{h}} w_{i} ({∥ \frac{y - x_{i}}{h} ∥}^{2}), \end{array}

(11)

\begin{array}{l} w_{i} = \sum_{u = 1}^{m_{c}} \sqrt{{\hat{p}}_{u} (y) {\hat{q}}_{u}} \sum_{v = 1}^{m_{t}} \sqrt{\frac{{\hat{q}}_{v}}{{\hat{p}}_{v} (y_{0})}} δ (t (x_{i}) - v) \\ + \sum_{v = 1}^{m_{t}} \sqrt{{\hat{p}}_{v} (y) q_{v}} \sum_{u = 1}^{m_{c}} \sqrt{\frac{{\hat{q}}_{u}}{{\hat{p}}_{u} (y_{0})}} δ (c (x_{i}) - u), \end{array}

(12)

where $w_{i}$ is weight for (11).

The search for the new object location of maximum likeness $y_{1}$ in the current frame can be achieved by using the gradient based mean-shift optimization. During this procedure, the target position is iteratively moved from the current position ${\hat{y}}_{0}$ to the new position $y_{1}$ according to

\begin{matrix} y_{1} = \frac{\sum_{i = 1}^{n_{h}} x_{i} w_{i} g ({∥ ({\hat{y}}_{0} - x_{i}) / h ∥}^{2})}{\sum_{i = 1}^{n_{h}} w_{i} g ({∥ ({\hat{y}}_{0} - x_{i}) / h ∥}^{2})}, \end{matrix}

(13)

where $g (x) = - k_{new}^{'} (x)$ .

4. Implementation of RTCs

The RT system is constructed by connecting the ports of multiple components as an aggregation of each RTC function. The OpenRTM-aist is an open source implementation of the RT-Middleware specification based on CORBA technology. It is developed by the National Institute of Advanced Industrial Science and Technology Intelligent Systems Research Institute Task Intelligence Research Group.

In this paper, to improve the flexibility, reliability, and real time performance, distributed intelligent assistant robotic system is designed by using RT middleware. Underlying functional components of the system are partitioned according to the following fundamental components.

To obtain module partition projects with different granularities, fuzzy tree graph clustering algorithm and AHP (analytical hierarchy process) are utilized to analyze correlation between modules and classify samples from the view of function and structure. Furthermore, the system is evaluated from four aspects such as integration complexity, module cohesion, and coupling between modules and system load balancing degree. These four aspects provide reasonable and effective solutions for robot module partition granularity problem.

According to the requirements of the system to provide the services, the partition functional components are divided into LRF (laser range finder) data acquisition, vision image obtaining, mobile robot control, feature extraction, map generation based on LRF, and so forth. The components structure chart of this system for indoor assistant robots is shown in Figure 6.

Figure 6

Modular system structure.

4.1. Architecture of RTC

The RT system is constructed by connecting the ports of multiple components as an aggregation of each RTC function. RTC is made up of six elements. The architecture of RTC is shown in Figure 7. The component profile shows some information of the RTC, such as the component object. Activity, which is the core logic of a component, must be implemented separately in each component. And it describes the state machine and core logic which is executed as processing (action) at the time of each state or the transition among states. Execution context executes the activities according to the state.

Figure 7

RT component architecture.

4.2. RTCs Implementation

All functions proposed in this paper, which can be operated with each other to implement complex function, are packaged into RTCs. There are four classes of function modules implemented in this system, SP²ATM with RBPF-SLAM modules, person tracking modules, 3D reconstruction modules, and distributed sensor modules. We will take SP²ATM with RBPF-SLAM modules as example to describe function modules implementation. The implemented system is detailed as below.

(A) LRF Data Getting Component. Getting raw data from LRF and control function classes includes

Outport<TimedLongSeq>: output data from LRF.

(B) Vision Image Acquiring Component. Getting image information from camera and control function classes includes

Outport<TimedOcterSeq>: output image from camera.

(C) Mobile Robot Control Component. Control function classes include

(i)

Inport<Vector2D>: control command for mobile robot;

(ii)

Serverport (provided): provide the pose information of mobile robot.

(D) Node Finding Component. Finding out the possible nodes includes

(i)

Inport<TimedLongSeq>: get LRF data;

(ii)

Outport<TimedDoubleSeq>: output the possible nodes and LRF data.

(E) Update Nodes Component. Updating the set of possible nodes includes

(i)

Inport<TimedDoubleSeq>: get the possible nodes and LRF data;

(ii)

Outport<TimedDoubleSeq>: output the possible nodes, topology nodes, and LRF data;

(iii)

Outport<TimedVectorSeq>: output the control command of mobile robot.

When a system is designed with above RTCs, users just need to know that the laser data can be acquired from InPort and the local and global maps can be attained from OutPort without understanding the details of the encapsulated SP²ATM.

5. Experimental Results

In this part, to verify the effectiveness of this proposed intelligent assistant robotic system based on distributed technology RTM, the experiments are performed in the laboratory. All RTCs of services are distributed in different locations and make up the intelligent assistant robotic system. During the experiment process, laser data reading module of SP²ATM is operated on the robot platform, and possible nodes searching module is operated in the user's computer. The interoperation between these modules is implemented by GIOP (general inter-ORB protocol) under RTM framework.

5.1. Hardware Platform

The hardware setup mainly consists of two mobile robots. The robot for performing person following task is composed of an American Mobile Robot Inc. Pioneer 3-DX mounted with a stereo camera and RFID. Two FLEA2 FL2-08S2M cameras produced by point grey company are fixed nearly parallel in optical axis direction and look forward to constitute a stereo camera. Two color images and a depth image can be captured from the stereo camera and fed to a computer by an IEEE 1394B cable. Their focal length is about 3.5 mm. The baseline is 140 mm and the horizontal FOV (field of view) is about 45°. Resolution of both the images is 800 × 600. The RFID system is composed of a RFID tag, two antennas, and a card reader. The ID tag is attached to a given person. The antennas can still detect the given person. Then, the card reader receives the detecting data and transmits it to the computer for data processing. The horizontal field of view is 60°, which is wider than that of the stereo camera.

The other robot, for helping user transmit object, is the xPartner robot AS-R with URG scanning laser range finder and Flea2 digital camera. The URG scanning LRF offers both serial (RS-232) and USB interfaces to provide laser scans. The field of view for this detector is 240 degrees and the angular resolution is 0.36 degrees. Distances are reported from 20 mm to 4 meters, and the resolution is 1 mm. Besides, Flea2 digital camera is used as vision system with 3.5 mm length and 70° field of view. Digital images are acquired into laptop computer through 1394 bus.

5.2. Results

Some experiments were carried out. Figures 8 and 9 show the results of different mean-shift algorithms. As shown in Figure 8(a), the mean shift based on texture cannot precisely track given person due to similar texture. Although the algorithm [17] perfectly eliminates the background variation by the modified kernel function, the tracking algorithm based on color cannot accurately locate target because of the similar background color. In comparison with the above mean-shift algorithms, the proposed method and algorithm [18] are more accurate, which is attributed to combining the color and texture. However, the iteration times of the proposed approach is less than the algorithm [18] due to the improved convex kernel function as shown in Figure 10, prelocation based on head-shoulder and RFID, which increasingly improve the real-time performance. Figures 11(a)–11(h) show the accurate results of human tracking under disturbed situations. Figures 11(a)–11(d) show one person passed by between robot and the user during the process of the straight line tracking, and in Figures 11(e)–11(h) another person passed by during the tracking process of left turn. The passer disturbed the robot tracking process, but the proposed system can overcome the effects caused by interference.

Figure 8

A comparison of mean-shift tracking methods.

Figure 9

Different mean-shift algorithm tracking accuracy.

Figure 10

Different algorithms compared to the number of iterations.

Figure 11

Distributed indoor assistant robotic system results.

The robot autonomous path planning results for object delivery are shown in Figures 11(i)–11(n). In the experiment, the robot started from the room which user A stayed in and passed corridor to the room where user B stayed. The movement process is shown in Figures 11(j) and 11(k). Then, the robot obtained a book from user B and came back with the same path to user A, as shown in Figures 11(k)–11(n). During the experiment process, users can not only observe the robot position on 3D reconstruction environment maps but also observe each room and corridor situations by distributed sensors. RTM framework enables assistance system to be effectively independent of operating systems and programming languages, increases the reusable value of function modules, and simplifies the development processes. The experiment results verify the effectiveness of three key technologies of this proposed system, and distributed control technology makes the system structure flexible.

6. Conclusion

In this paper, we have developed a distributed intelligent assistance robotic system and proposed three key technologies for providing some basic services to support the aged or disabled. In the developed system, the SP²ATM algorithm was improved with RBPF-SLAM to realize the accurate location of mobile robots. On this basis, the 3D reconstruction approach, which has the advantage of being robust to illumination change and the high accuracy of the deformed surface compared with the other methods, was implemented to reconstruct home environment. Besides, human tracking accuracy was improved by fusing multifeatures and coupling various kinds of sensory information. In addition, all developed technologies were integrated by using distributed control technology RTM; therefore, the flexibility and scalability of the system were improved. The experiments verified the effectiveness of the proposed intelligent assistant robotic system and advancement of the three key technologies. For the future work, we will focus on extension and optimization of system function.

Footnotes

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The research work is supported by the Key Program of Beijing Natural Science Foundation (KZ201110005004) and the National Natural Science Foundation (61175087 and 61105033).

References

Chung

H.-Y.

Hou

C.-C.

Chen

Y.-S.

Chao

C.-L.

An intelligent service robot for transporting object

Proceedings of the IEEE 22nd International Symposium on Industrial Electronics (ISIE ′13)

May 2013

Taipei, Taiwan

1 6

2-s2.0-84881653288

10.1109/ISIE.2013.6563645

Rampinelli

MacHado

V. P.

Vassallo

R. F.

Neto

A. F.

Bastos

T. F.

Pizarro

Use of computer vision for localization of a robotic wheelchair in an intelligent space

Proceedings of the 4th ISSNIP-IEEE Biosignals and Biorobotics Conference (BRC ′13)

February 2013

Rio de Janerio, Brazil

1 6

2-s2.0-84876762436

10.1109/BRC.2013.6487537

Song

Yin

Zhou

Cheng

A surveillance robot with hopping capabilities for home security

IEEE Transactions on Consumer Electronics 2009 55 4 2034 2039

2-s2.0-75449091384

10.1109/TCE.2009.5373766

Tseng

C.-C.

Lin

C.-L.

Shih

B.-Y.

Chen

C.-Y.

SIP-enabled surveillance patrol robot

Robotics and Computer-Integrated Manufacturing 2013 29 2 394 399

2-s2.0-84868535937

10.1016/j.rcim.2012.09.009

van Osch

Bera

van Hee

Koks

Zeegers

Tele-operated service robots: ROSE

Automation in Construction 2014 39 152 160

2-s2.0-84881321689

10.1016/j.autcon.2013.06.009

Alvarez-Santos

Iglesias

Pardo

X. M.

Regueiro

C. V.

Canedo-Rodriguez

Gesture-based interaction with voice feedback for a tour-guide robot

Journal of Visual Communication and Image Representation 2014 25 2 499 509

2-s2.0-84876398912

10.1016/j.jvcir.2013.03.017

Tian

Zhou

Xue

Song

Building an intelligent home space for service robot based on multi-pattern information model and wireless sensor networks

Intelligent Control and Automation 2012 3 1 90 97

10.4236/ica.2012.31011

Jia

Lin

Wang

Takase

Network distributed multi-functional robotic system supporting the elderly and disabled people

Journal of Intelligent and Robotic Systems: Theory and Applications 2006 45 1 53 76

2-s2.0-33744936785

10.1007/s10846-005-9028-y

Elkady

Joy

Sobh

Valavanis

A structured approach for modular design in robotics and automation environments

Journal of Intelligent & Robotic Systems 2013 72 1 5 19

2-s2.0-84872060028

10.1007/s10846-012-9798-y

10.

Rashid

Broxvall

Saffiotti

A middleware to integrate robots, simple devices and everyday objects into an ambient ecology

Pervasive and Mobile Computing 2012 8 4 522 541

2-s2.0-84864283516

10.1016/j.pmcj.2012.06.003

11.

Oriolo

Vendittelli

Freda

Troso

The SRT method: randomized strategies for exploration

Proceedings of the IEEE International Conference on Robotics and Automation

May 2004

New Orleans, La, USA

4688 4694

2-s2.0-3042638929

12.

Klein

Murray

Parallel tracking and mapping for small AR workspaces

Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR ′07)

November 2007

Nara, Japan

225 234

2-s2.0-50649108974

10.1109/ISMAR.2007.4538852

13.

Smith

Reid

Davison

A. J.

Real-time monocular SLAM with straight lines

Proceedings of the British Machine Vision Association

2006

17 26

14.

Cipolla

Masanobu

Stereoscopic tracking of bodies in motion

Image and Vision Computing 1990 8 1 85 90

2-s2.0-0025388192

15.

Bellotto

Vision and laser data fusion for tracking people with a mobile robot

Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO ′06)

December 2006

Kunming, China

7 12

2-s2.0-37049014034

10.1109/ROBIO.2006.340251

16.

Huang

Lao

High-performance rotation invariant multiview face detection

IEEE Transactions on Pattern Analysis and Machine Intelligence 2007 29 4 671 686

2-s2.0-33947536269

10.1109/TPAMI.2007.1011

17.

Mazinan

A. H.

Amir-Latifi

Applying mean shift, motion information and Kalman filtering approaches to object tracking

ISA Transactions 2012 51 3 485 497

2-s2.0-84859530388

10.1016/j.isatra.2012.02.002

18.

Dai

Y.-M.

Wei

Lin

Y.-N.

An improved Mean-shift tracking algorithm based on color and texture feature

Journal of Zhejiang University (Engineering Science) 2012 46 2 212 217

2-s2.0-84859401280

10.3785/j.issn.1008-973X.2012.02.005

19.

S. S.

Zhang

Abraham

A. T.

Rebsamen

Simultaneous path planning and topological mapping (SP2ATM) for environment exploration and goal oriented navigation

Robotics and Autonomous Systems 2011 59 3-4 228 242

2-s2.0-79952041795

10.1016/j.robot.2010.12.003

20.

Abraham

A. T.

S. S.

Tao

P. Y.

A topological approach of path planning for autonomous robot navigation in dynamic environments

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ′09)

October 2009

4907 4912

2-s2.0-76249118167

10.1109/IROS.2009.5354771

21.

Huang

W. H.

Beevers

K. R.

Topological map merging

International Journal of Robotics Research 2005 24 8 601 613

2-s2.0-23844432672

10.1177/0278364905056348

22.

Triggs

Mclauchlan

Hartley

Fitzgibbon

Bundle adjustment—a modern synthesis

Vision Algorithms: Theory and Practice 2000 1883

Berlin, Germany

Springer

298 372 Lecture Notes in Computer Science

10.1007/3-540-44480-7_21

23.

Jin

Sun

Peng

Subdivision interpolating implicit surfaces

Computers and Graphics 2003 27 5 763 772

2-s2.0-0142124837

10.1016/S0097-8493(03)00149-3

24.

Newcombe

R. A.

Davison

A. J.

Live dense reconstruction with a single moving camera

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR ′10)

June 2010

San Francisco, Calif, USA

1498 1505

2-s2.0-77956000176

10.1109/CVPR.2010.5539794

25.

Zach

Pock

Bischof

A duality based approach for real-time TV-L1 optical flow

Proceedings of the 29th DAGM conference on Pattern recognition

2007

Heidelberg, Germany

214 223

26.

Jia

Sheng

Shang

Takase

Robot localization in indoor environments using radio frequency identification technology and stereo vision

Advanced Robotics 2008 22 2-3 279 297

2-s2.0-40549105456

10.1163/156855308X292583

Distributed Intelligent Assistance Robotic System with Sensor Networks Based on Robot Technology Middleware

Abstract

1. Introduction

2. System Overview

3. Key Technologies

3.1. Path Planning

Algorithm 1: SP2ATM with RBPF-SLAM.

3.2. Scene Flow-Based Environment 3D Reconstruction

3.3. Person Tracking

4. Implementation of RTCs

4.1. Architecture of RTC

4.2. RTCs Implementation

5. Experimental Results

5.1. Hardware Platform

5.2. Results

6. Conclusion

Footnotes

Conflict of Interests

Acknowledgments

References

Algorithm 1: SP²ATM with RBPF-SLAM.