Visual Recognition System for Cleaning Tasks by Humanoid Robots

2.1. Overview of Cleaning Task

In this study, “cleaning” is defined as a task that puts the table back into a clean or tidy state, as initially memorized by the robot. We define rubbish as any object on the target table that was not previously memorized by the robot (unknown objects). In contrast, every memorized object on the tabletop is not considered rubbish (we call such objects goods) and should be maintained in the same position as it was before cleaning. Here, upon task completion, the objects that are classified as rubbish are cleaned, while those classified as goods are left on the tabletop in their initial state.

According to the graspability of the objects existing on the table, the objects are classified as “graspable” (i.e., objects that can be grasped by the robot) and “ungraspable” (i.e., objects that cannot be grasped by the robot). Furthermore, ungraspable objects can be categorized as “small” and “big.” Small objects are objects of low height and are difficult to segregate with only plane detection. Normally, small objects are light and fine, such as grills and sugar. Most small objects on the tabletop can be thought of as rubbish; given their light weight, these objects can be removed using tools such as a vacuum cleaner. By contrast, big objects have heights such that they can be segregated by only plane detection, but are too heavy to be grasped by the robot, such as an electric kettle or coffee maker. In a normal situation, big objects tend to be categorized as goods. However, there are some situations in which small objects fall under the goods category, e.g., necklaces and documents, or big objects fall under the rubbish category, such as big boxes. This situation is out of the scope of our study. However, in our opinion, this problem can be solved by asking for human assistance.

For graspable objects, graspable rubbish can be moved to the dustbin, while graspable goods can be moved to suitable locations. An overview of the cleaning task is shown in Fig. 1.

OPEN IN VIEWER

Figure 1.

Overview of cleaning task. The top panel illustrates the learning phase, i.e., learning the table's tidy state, which is the target of the cleaning task, while the bottom panel shows the cleaning task including tabletop recognition and object manipulation (e.g., object grasping and vacuuming ungraspable objects).

It can be seen from Fig. 1 that the cleaning task involves two phases: learning and task execution. In the learning phase, table properties such as table colour, material and location are learned by the robot. Moreover, the locations of goods on the tabletop are memorized as the initial or tidy state of the table. Once the tidy state is learned, the task-execution phase can be executed at any time. The task-execution phase includes tabletop recognition, that is, to detect and recognize the objects on the table. The graspable objects are grasped and placed at suitable positions, while the ungraspable rubbish is cleaned using a vacuum cleaner.

The basic elements required to accomplish the cleaning task are navigation, object manipulation and perception. Robot navigation is achieved using a laser range finder (LRF) for performing simultaneous localization and mapping (SLAM). The 3D information captured by the visual sensor [1] is used and rapidly exploring random tree (RRT)-based [12] path planning is employed for object manipulation. Robot perception is achieved by incorporating multiple cues such as colour, texture, 3D point-clouds and NIR reflection intensity for constructing a visual recognition system comprising object detection, object recognition and material recognition. The details of robot perception are discussed in section 3.

2.2. Robot Platform

The robot platform used in this paper is shown in Fig. 2. The robot is equipped with a visual sensor (for details see section 3.1), arms capable of six degrees of freedom (DOF), and 6-DOF hands. Moreover, omnidirectional wheels and a laser range finder (LRF) enable the robot to move freely within a room. To realize the previously described cleaning task, the robot should be equipped with a handy vacuum cleaner for cleaning the ungraspable rubbish. A handy vacuum cleaner is installed on the robot, with its hose set on the right hand. Thus, graspable rubbish, such as cans, can be put into the dustbin given that the left hand can grasp objects. Moreover, an XBee wireless radio module is installed on the vacuum cleaner to enable the robot to switch the vacuum cleaner on for cleaning ungraspable rubbish and switch it off automatically when done.

OPEN IN VIEWER

Figure 2.

Robot equipped with 3D visual sensor and handy vacuum cleaner.

3.1. Visual Sensor

In this study, we use a 3D visual sensor [1]. This sensor consists of one TOF and two CCD cameras, as shown in Fig. 2. Here, colour information and 3D point clouds can be acquired in real time by calibrating the TOF and the two CCD cameras. Moreover, NIR reflection intensities can be acquired from the TOF camera because this sensor uses intensity values for calculating the confidence values of measured depth information. Therefore, colour, texture, 3D point clouds and NIR reflection intensities, which are captured by the sensor, are used for realizing object detection, object recognition and material recognition.

3.2. Multiple-Cue-based 3D Object Recognition

Multiple cues acquired from the visual sensor are used to construct the object recognition system shown in Fig. 3. The proposed system is divided into the learning and recognition phases. Object extraction, which is the first step in both phases, is built based on motion-attention and plane detection. The motion-attention-based method uses a motion detector for extracting an initial object region; the object region is then refined using the colour and depth information of the initial region. For more details on this method, please refer to [13]. For detecting objects on a table, the plane detection [14] technique is beneficial. The 3D randomized Hough transform (3DRHT) is used for fast and accurate plane detection. In the learning phase, an object database consisting of multiple feature vectors in various views is generated. Here, scale-, translation- and rotation-invariant properties are desirable. A histogram-based feature is employed in the proposed method because it can realize the rotation-and translation-invariant properties. Moreover, we can use 3D information to normalize scale and achieve the scale-invariant property. Because it is difficult to achieve the view-invariant property, we realized the same by matching all features obtained from various viewpoints. Next, we explain the features employed in this study.

OPEN IN VIEWER

Figure 3.

Object recognition system.

3.2.2. Object Recognition

In the recognition phase, the target object is compared to the object database. First, the Bhattacharyya distance of each feature D_f(h_k^f,h_in^f) is calculated as follows:

D f ( h k f , h in f ) = 1 − ∑ i = 1 N f h k f ( i ) × h in f ( i ) .

(1)

Here, f, h_k^f, h_in^f and N_f respectively represent the feature type f ∈ {colour, texture, shape} reference histogram of feature f that belongs to class k in the object database, histogram of feature f for a given target object, and size of those histograms. The likelihood of the target object for feature f is defined as follows:

P ( h in f | k ) ∝ exp { − D f ( h k f , h in f ) 2 σ f 2 } ,

(2)

where σ_f² represents a variance of distances for feature f, which is calculated by cross validation over the object database (see section 3.2.3).

Assuming that the features are independent, the integrated likelihood P(h_in|k) is defined as follows:

P ( h in | k ) = ∏ f { P ( h in f | k ) } w in f ∝ exp { − ∑ f w in f D f ( h k f , h in f ) 2 σ f 2 } ,

(3)

where w_in ^f is the weight of feature f, which is determined adaptively considering the environment and similarity among objects in the database (see section 3.2.4). w_in ^f is normalized as Σ_fw_in ^f = 1. Finally, the result is determined from k-NN among the integrated distances:

d ( k ) = ∑ f w in f D f ( h k f , h in f ) 2 σ f 2 .

(4)

k-NN is easy to implement in our approach because we use adaptive weights, which change according to input data.

3.2.3. Database Cross-Validation

The parameter σ_f² in (2), which expresses the variance of distances, can be calculated by cross-validation over the object database. Consider that each object has V feature vectors for each feature f. Let the reference histogram of feature f that belongs to the class k for a given view v be h_kv^f. The parameter σ_f² can then be calculated as follows:

σ f 2 = 1 K − 1 ∑ k = 1 K ( min ( D k f ) − μ f ) 2 ,

(5)

where, min(D_k^f) indicates the minimum distance in D_k^f, which is defined as follows:

D k f = { D f ( h k v f , h k v ′ f ) | 1 ≤ v ≤ V ,1 ≤ v ′ ≤ V , v ′ ≠ v } .

(6)

Here, µ_f is the mean of all min(D_k^f) and K denotes the number of objects in the database.

3.2.4. Adaptive Weights

The weights of each feature affect overall performance in multiple-features-based recognition. The weights can be calculated in a straightforward manner considering the environment and similarities (colour, texture, shape, etc.) among objects in the database. For example, in a dark environment, shape information, which is calculated based on 3D point clouds, is the most reliable type of information because it is robust to changes in illumination. However, objects with similar shape cannot be distinguished using only shape information. In this case, colour or texture information offers better reliability than shape information. Therefore, we determine the weights using the following policies.

For a given feature, if the minimum distance between the target object and database is large, the feature will not afford effective recognition. This will decrease the integrated likelihood of the recognition results and lead to false recognition. Therefore, weight should be set inverse to the minimum distance.

Database complexity, i.e., the similarity of a given feature among different objects, is also responsible for false recognition. To avoid this, the minimum distance from different objects can be used as a parameter for controlling the weights.

Based on these policies, the weights are calculated as follows:

w in f = γ f ∑ f γ f , γ f = min ( D ^ in f ) min ( D in f )

(7)

where,

D in f = { D f ( h in f , h k v f ) | 1 ≤ k ≤ K ,1 ≤ v ≤ V } ,

(8)

D ^ in f = { D f ( h in f , h k v f ) | 1 ≤ k ≤ K , k ≠ k m i n f ,1 ≤ v ≤ V } ,

(9)

k_min^f represents the index of the minimum distance and is defined as k m i n f = a r g m i n k D i n f .

3.3. NIR-based Material Recognition

In this section, we discuss the features used for material recognition. Material information represented by the reflectivity coefficient is an important part of the features because this information is different for each material. Additionally, distance and incident angle should be considered for robust recognition.

3.3.1. Radiance Model for TOF Camera

NIR reflection intensities obtained using the TOF camera are not affected by lighting conditions [20–22]. However, these intensities vary widely according to the integration time δt, distance to the object's surface d, and incident angle θ. To understand the nature of NIR reflection intensity, we consider the radiance model here.

Assume a Lambertian object surface. Generally, reflection from a Lambertian surface is diffuse. Then, a simple radiance model, shown in Fig. 4, is used for representing NIR reflection intensity. According to this model, the intensity of a certain wavelength (λ = 850nm for the TOF camera) can be written as follows:

L Q ( λ ) = ∫ I e ( λ , d ) K d ( λ ) c o s θ d t = I e ( λ , d , δ t ) K d ( λ ) c o s θ

(10)

OPEN IN VIEWER

Figure 4.

Simple radiance model for TOF camera.

where L_Q, I_e, K_d, θ, d and δt represent, respectively, the reflection intensity, illuminant (NIR-light emitted from the TOF camera), reflectivity coefficient, incident angle relative to illuminant direction (centre of the TOF camera), surface-illuminant distance, and integration time. The normal vector ñ and incident angle θ can be obtained from the following equations:

n ^ = ∑ k = 1 N w k n ^ k ∑ k = 1 N w k , c o s θ = 〈 p → , n ^ 〉 ∥ p → ∥ ∥ n ^ ∥ ,

(11)

where p → is a position vector directed from the centre of the TOF camera O to point P on the reflective surface. w_k denotes the weight of the neighbouring normal vector. The weight is set to 1 if all 3D points of the neighbouring triangle mesh exist. If not, it is set to 0. Therefore, we represent the material information as m(L_Q, δt, d, θ). All of these variables can be determined using the data captured by the TOF camera and used to realize the material recognition system shown in Fig. 5.

OPEN IN VIEWER

Figure 5.

Proposed material recognition system.

3.3.2. Classification Using SVM Classifiers

As mentioned above, the feature vector m(L_Q, δt, d, θ) captures the reflectivity coefficient of the object's material. This information should be determined in various combinations to ensure robust material recognition. However, incorporating all elements into one classifier may result in significantly increased learning time and memory consumption. Here, the material map, which consists of multiple SVM classifiers, is proposed for overcoming this issue. SVM is used because it yields good classification performance. Normally, SVM is used for binary classification. However, it can be expanded for multi-class classification by considering all combinations of binary classifiers in the multi-class problem. In this study, we use LIBSVM for SVM implementation; for further information on SVM, please refer to [23]. In this study, two-dimensional data, which consist of NIR reflection intensity L_Q and the incident angle θ of each pixel in the object region, are used for classifier training.

For generating a material map, a set of training data consisting of NIR and depth images is collected in the learning phase. Now, let d^min, d^max, δt^min, δt^max, respectively, be the minimum and maximum distances d and integration times δt in the collected training data. Furthermore, let T and D be the grid numbers of δt and d, respectively. Here, we select the pixels of interest x m = { x 1 m , x 2 m , … , x N m m } ( d m i n ≤ d ( x m ) ≤ d m a x , δ t m i n ≤ δ t ( x m ) ≤ δ t m a x ) for each material m ∈ {1,2, …,M} from the collected training images. The notation d(*) and δt(*) represent, respectively, the distance and the integration time of pixel *, while Nm expresses the number of pixels that belong to material m. Then, material information m(L_Q, δt, d, θ) of x^m is determined. To build the material map (d, δt), shown in Fig. 5, the pixels in x^m are classified according to each gridded distance and integration time ( n d m , n t m ) . For a given pixel x i m ∈ x m , n d m and n t m can be calculated as follows:

n d m = ⌊ d ( x i m ) − d m i n ( d m a x − d m i n ) D ⌋ ,

(12)

n t m = ⌊ δ t ( x i m ) − δ t m i n ( δ t m a x − δ t m i n ) T ⌋ ,

(13)

so that each grid ( n d m , n t m ) has x m d t = { x 1 m , x 2 m , … , x N m d t m } pixels. Here, mdt denotes pixels with distance d and integration time t, which belong to the material m, while Nmdt denotes the number of such pixels. The final step in the learning phase is to train x^mdt (m ∈ {1,2,… M}) using the multi-class SVM classifier as a model on grid ( n d m , n t m ) .

In the recognition phase, the target object is extracted using plane detection [14], and material information is calculated. The material map can be used if δt and d are known. Here, the δt can be set by adjusting a parameter of the TOF camera, while d can be measured using the TOF camera. Once d and δt of a given pixel are known, (12) and (13) are used to determine the grid ( n d m , n t m ) and the SVM model of that grid is used to estimate the material of the given pixel.

3.3.3. Object-based Smoothing

The above method gives results on a pixel-basis; consequently, the results are inconsistent. Therefore, these are smoothed to obtain the final result. We consider object-based smoothing here.

Let M be the number of materials to be classified. V i m represents the number of votes for a material m inside the i-th object region D_i. Each vote v i m ∈ { 1,2, … , M } is determined as a label predicted from each selected SVM model, as mentioned in the previous section. The probability of material m at pixel (u, v) inside the object region P(u, v|m, i) is defined as follows:

P ( u , v | m , i ) = V i m ∑ k = 1 M V i m .

(14)

This probability represents the confidence of material recognition at pixel (u, v), and the final result is determined from the maximum number of votes. For object-based smoothing, the final voting process is performed among the pixels inside the object region to determine the target object's material. The final decision regarding an object's material m̂_i is arrived at as follows:

m ^ i = argmax m ∑ ( u , v ) ∈ D i P ( u , v | m , i ) .

(15)

Because the proposed system is implemented in a real robot, it is possible for the robot to change pose and/or distance to the target object if the confidence of a given recognition result is low.

3.4. GMM-based Ungraspable Object Detection

When a robot cleans objects of adequate height, such as plastic bottles and cans, on a table, the plane detection method described in [14] can be applied for detecting objects on the table, recognizing them using the proposed recognition system, and then grasping them. However, small objects such as paper waste are difficult to detect using plane detection.

In this study, the table surface colour is learned in the learning phase, and colour differences between the table and other objects are used for object detection. More specifically, for robustness to changes in illumination, the table colour (i.e., hue and saturation in the HSV colour space) is represented using a Gaussian mixture model (GMM); this model is then used in the detection phase for discriminating the table from other objects. The GMM is employed because it has good colour-segmentation performance. However, object detection will fail if the colour of the table and objects is identical when using colour as the only discriminating factor. Therefore, NIR reflection intensity, which includes material information, is an important object-detection cue. Here, incident angle θ in the material information can be adjusted, similar to that in the learning phase; therefore, it can be ignored. Therefore, the feature vector I(H, S, L_Q), which consists of the table's colour (H and S) and NIR reflection intensity L_Q, is modelled using the GMM to achieve more robust small object detection. An overview of the system is shown in Fig. 6.

OPEN IN VIEWER

Figure 6.

Overview of GMM-based ungraspable object detection.

Our task in the learning phase is to maximize the likelihood of all pixels that belong to the table. Let I_i be the feature vector of a pixel i on the table, and K be the number of Gaussian components in the GMM that has parameters Θ = (π₁, µ₁, Σ₁,…, π_K, µ_K, Σ _K ). Then, the likelihood of the feature vector I_i can be calculated as follows:

P ( I i | Θ ) = ∑ k = 1 K π k N ( I i | μ k , Σ k ) ,

(16)

where, Nscr;(*|*) represents a Gaussian distribution. The parameter Θ is estimated using the EM algorithm. In this study, we conducted preliminary experiments to find the best number of Gaussian components for the GMM model and determined that K = 3 yields the best result. In future, the Dirichlet process mixture (DPM) [24, 25] can be used for determining the number of mixture models.

In the recognition phase, the likelihood of each pixel I_i on the table is calculated using (16) and binarized as follows for generating a binarized map.

b ( I i ) = { 0 ( P ( I i | Θ ) ≥ P t ) 1 ( P ( I i | Θ ) < P t )

(17)

Here, P^t is an empirically defined threshold. Thereafter adjunctions are applied to the generated binarized map for refining it. Finally, connected-component labelling is employed, and the area of region R^ℓ of each candidate S(R^ℓ) (ℓ ∈ {1,2,…, L}) is calculated as follows:

S ( R ℓ ) = ∑ i ∈ R ℓ b ( I i ) ,

(18)

where L represents the number of labels. Candidate regions are filtered, and only the region R^ℓ ∈ {S^min ≤ S(R^ℓ) ≤ S^max} is selected as a detected object. Here, S^min and S^max, respectively denote the minimum and maximum area of region allowed; these values are determined empirically.

Given that these small objects are ungraspable by the robot, a handy vacuum cleaner is used to clean each selected region R*.

5.1. Evaluation of 3D Object Recognition

An experiment is carried out in the living room shown in Fig. 8 using 50 objects, as shown in Fig. 9. A user shows each object from various angles to the robot, and the object database, which contains the feature vectors of 50 frames per object, is generated in the learning phase. In the recognition phase, recognition is performed at five different locations under different illumination conditions, as shown in Fig. 10. Here, 50 image frames (10 frames per location) of each object are recognized, while the learning phase is completed at location 1. We use six different feature combinations, i.e., colour; texture; shape; integration of colour and texture; integration of colour, texture, an d shape without any weights; and integration of colour, texture, and shape with adaptive weights. The standard SIFT [19] and Colour-SIFT [27] approaches were applied for reference purposes. However, these approaches are based on keypoint matching.

OPEN IN VIEWER

Figure 8.

Experimental environment: (a)living room, and (b)two-dimensional map of room.

OPEN IN VIEWER

Figure 9.

Objects used for evaluation of 3D object recognition and their respective categories.

OPEN IN VIEWER

Figure 10.

Varying illumination conditions. A fluorescent lamp is used at Locations 1, 4 and 5, while an incandescent lamp is used at Locations 2 and 3. The learning phase was conducted at Location 1.

For comparing the recognition results with regard to object properties, five categories, as shown in Fig. 9, were selected. Category 1 comprises objects having well-defined colours with different shapes and sizes. Category 2 comprises objects of the same colour and similar shapes. Category 3 comprises boxes of various sizes and textures. Category 4 contains cans of the same shape and size. Category 5 contains white dishes of different shapes and sizes.

The recognition rates of each feature for all objects in all locations are listed in Table 1. Moreover, Fig. 11 shows the recognition results for each object category (i.e., recognition results by category for all locations), while the results for each location (i.e., recognition results by location for all objects) are shown in Fig. 12.

OPEN IN VIEWER

Figure 11.

Recognition results by category.

OPEN IN VIEWER

Figure 12.

Recognition results by location.

OPEN IN VIEWER

Table 1.

3D object recognition results

Feature	Recognition rate [%]
Colour information	60.2
SIFT [19]	47.1
Texture information	66.8
Shape information	67.0
Colour-SIFT [27]	62.7
Colour and texture information	76.0
Integration of colour, texture and shape information without weighting	89.1
Integration of colour, texture and shape information with adaptive weights	91.5

5.1.3. Recognition Using Integrated Information

It can be inferred from Table 1 that the recognition results can be improved considerably by integrating colour, texture and shape information. Especially when adaptive weights are employed, the deviation in recognition for each category or location can be suppressed, thus resulting in the best recognition rates compared with other methods.

The adaptive weights have a considerable bearing on the results of categories 2 and 5, as shown in Fig. 11. This is because the objects in these two categories have fewer textures, colours and shapes that can act as recognition clues. However, the colour information becomes unstable as the illumination conditions change. In this case, the weight assigned to colour information is adaptively decreased in the proposed method. In contrast, recognition of objects belonging to categories 1 and 3 is not as difficult because these objects offer an adequate number of recognition clues. Therefore, the method using adaptive weights offers no significant improvement in the recognition of objects in these categories over the method using fixed weights.

Fig. 13 shows examples of adaptive weights by category and location. It can be seen that the weight of shape information at location 5 is high compared to those of the other information types. This is because, at this location, the reliability of colour or texture information is low owing to dark lighting conditions. By contrast, because object learning was conducted at location 1, the weights at that location are varied according to the object properties. For example, the weight of shape information for category 4 (i.e., the category with objects of similar size and shape) is the least. In contrast, shape information has high weight for category 5, because colour or texture information is unreliable for this category.

OPEN IN VIEWER

Figure 13.

Examples of adaptive weights. Each histogram depicts the weight of each feature. Variation of weights across the five locations is illustrated from left to right in a category, while variation across the five categories is illustrated from top to bottom in a location.

Changes in viewpoints of an object seemed to be the main source of false recognition in this experiment. It is impossible for the robot to observe an object from equally spaced viewpoints because a human user shows the object to the robot. Such biased observation samples are responsible for deviations in the voting process. In future, this can be avoided through autonomous learning by the robot.

5.2. Evaluation of Material Recognition

In order to evaluate the proposed material recognition algorithm, we conducted an experiment using 60 objects; these can be classified as woods, fabrics, papers, plastics, cans and ceramics, as shown in Fig. 14.

OPEN IN VIEWER

Figure 14.

Objects used for material recognition.

The recognition task was performed from a distance of 60-80cm according to the leave-one-out method. The confusion matrix of the average recognition rate is shown in Fig. 15, and the mean of the recognition rates was 70.2%. False recognition mainly occurred with woods and ceramics. High variance in NIR reflection intensities with woods and ceramics led to such false recognition.

OPEN IN VIEWER

Figure 15.

Confusion matrix of material recognition with six categories. The right side depicts the corresponding average recognition rate of each colour in the confusion matrix.

In contrast, the recognition rates of burnable (i.e., woods, fabrics and papers) and unburnable (i.e., plastics, cans and ceramics) objects was 97.7%. Because material recognition is performed on separate burnable and unburnable rubbish, this result qualifies the method for cleaning.

Examples of material recognition in a real scenario are shown in Fig. 16. We can see that paper packs, cans and plastic cups were detected from the scenes, and the materials of which these objects were made were recognized correctly. In addition, the confidence of the recognition results for each class is shown.

OPEN IN VIEWER

Figure 16.

Examples of material recognition in real scenarios: (a) input colour image (1024 × 768), (b) NIR image (176 × 144), (c) segmented image (176 × 144), (d) probability map of cans (176 × 144), (e) probability map of plastics (176 × 144), and (f) probability map of paper packs (176 × 144). In the segmented image, the white colour represents the table, and the black colour indicates the low accuracy of depth pixels.

5.3. Evaluation of Ungraspable Object Detection

In order to validate the proposed ungraspable object detection scheme, the experiment was conducted using 15 small objects including paper waste and powders (sugar, milk, coffee, etc.) shown in Fig. 17, which are put on four different tables. Here, the two types of scenes, clean and messy, were considered for each table, and small object detection was conducted for a total of 12 scenes and 116 objects. An example of scenes is shown in Fig. 18. The PASCAL Visual Object Challenge (VOC) evaluation metric [28] was used to calculate detection rate. A candidate detection was considered correct if the size of the intersection of the predicted bounding box and the ground truth bounding box was more than half the size of their union. Only one of multiple successful detections for the same ground truth was considered correct, the rest were deemed as false positives. The result of small object detection was 79.3%. Because the NIR reflection intensity is included in the object detection system, the effect of changes in illumination can be reduced. However, the NIR reflection intensities of a few objects, such as coffee, were close to those of the table. In this case, if the colour of the table were different (e.g., white table), such objects could be detected using colour information. Overall, this result is good considering the main purpose of ungraspable object detection is to detect the approximate location for the vacuum cleaner.

OPEN IN VIEWER

Figure 17.

Ungraspable objects.

OPEN IN VIEWER

Figure 18.

Recognition results on four tables. For each table, three images are presented: a colour image of resolution 1024 × 768 (top), depth image of resolution 176 × 144 (middle), and NIR image of resolution 176 × 144 (bottom). Detected frames: ungraspable objects (red); identified objects: by object recognition (blue), by material recognition (green).

5.4. Evaluation of Cleaning Tasks

The proposed cleaning task was then implemented on a real robot platform. Visual recognition, which was conducted on the tabletops, is shown in Fig. 18. Recognition results of graspable objects that were to be disposed (rubbish, indicated by green frames), put on another table for tidying the target table (goods, indicated by blue frames), and the objects required to be cleaned away using the vacuum cleaner (ungraspable rubbish, indicated by red frames), are shown.

A quantitative evaluation of the cleaning task is difficult to do because the definition of “similar” as the initial state is difficult to measure. However, the proposed method can be evaluated on qualitative criteria such as “How close are the results to the initial state?”.

An action plan is generated based on the visual recognition results for performing the cleaning task. Figure 19 shows how the robot actually performed the cleaning task based on the flowchart shown in Fig. 7 (see section 4). It can be seen from Fig. 19 that the robot, which memorized the initial tabletop state, navigates to the table and recognizes it. Thereafter, the robot searched for graspable rubbish and disposed of the same into the dustbin. After all graspable rubbish was eliminated, the robot grasped the goods and put them on another table except the last goods item. The last goods item was grasped in the left hand, and ungraspable rubbish was cleaned by the vacuum cleaner in the robot's right hand. After vacuuming the ungraspable rubbish, the robot navigated to the other table where it had placed the goods to bring them back to their original locations.

OPEN IN VIEWER

Figure 19.

Example of cleaning task. The task is performed according to the planning shown in Fig. 7. The main parts of the robot's action are navigation, recognition and object manipulation, including vacuuming ungraspable rubbish.

To evaluate the proposed method, the cleaning task was conducted over 10 trials; seven trials succeeded, while three failed. In the failed trials, the robot either missed when throwing rubbish into the dustbin or placing the goods on another table. In these cases, the target table was cleaned, but, as the whole, the task was not performed as desired. Among the successful trials, five trials were considered qualitatively clean, while in the other two trials, ungraspable rubbish was left on the table. Examples of several conditions after the completion of cleaning are shown in Fig. 20. Overall, the robot's navigation errors during the task were responsible for the lack of accuracy in placing the goods in their initial positions and failure to position the vacuum cleaner at the desired location for cleaning the ungraspable rubbish.

OPEN IN VIEWER

Figure 20.

Examples of conditions when the cleaning task is completed. The top left of the figure shows the initial (tidy) condition, while the right panel shows the untidy condition. The bottom figures illustrate a few examples of executed cleaning tasks, with results changing worse to better.