Recent progress on tactile object recognition

Introduction

At present, many robots are equipped with visual sensors. However, the visual sensing is subject to many restrictions in practical applications, such as lighting conditions and occlusions. Tactile sensing is another sensing modality which is widely used in robotics. Different from visual sensing, tactile sensors are capable of perceiving some physical properties (e.g. softness/hardness, texture, temperature) of an object directly. Humans also utilize touch sensory receptors as an important means to perceive and interact with the environment. Early in the 1980s, experiments with local anesthesia indicated the importance of humans’ tactile sensing for a stable grasp. ¹ It was found that the applied grip force was critically balanced to optimize the motor behavior, so that slipping between the skin and the gripped object did not occur and the grip force did not reach exceedingly high values. Therefore, introducing tactile perception to the robot not only simulates human’s perception and cognitive mechanisms to a certain extent but also satisfies the strong demand for practical robotic applications.

With the development of modern sensors, control, and artificial intelligence technology, extensive research has been conducted on tactile sensors, grasp stability analysis, and object recognition based on tactile sensing. One early review ² elaborated the state of the art in tactile sensing and its likely research motivations. It also pointed out the increasing emphasize and understanding on tactile sensing in the task of dexterous manipulation. Broadly speaking, tactile properties refer to any properties measured through contact, including pressure, force, temperature, and so on, while a narrow definition of tactile sensing requires that there must be force measurements involved.

Research on robotic tactile perception has expanded in recent years. Journals such as IEEE Transactions on Robotics, IEEE Sensor Journal, and Robotics and Autonomous Systems, all published reviews on robotic tactile perception. ^3,4,5 In addition, IEEE Transactions on Robotics released a special issue on the theme of Robotic Sense of Touch in 2011. ⁶ Furthermore, the journal IEEE Transactions on Haptics was officially inaugurated in 2008.

Tactile sensors are essential to tactile perception. According to operation principles, tactile sensors can be classified into several types, such as piezoresistive, capacitive, piezoelectric, quantum tunnel effect, optical, barometric, structure-borne sound, and so on. Kappassov et al. ⁵ compared the advantages and disadvantages of 28 kinds of tactile sensors in detail, while Denei et al. ⁷ developed a multimodal tactile sensor that contained a capacitive sensor array, audio measurement, and proximity perception for recognizing different materials. Although many researchers still use self-developed tactile sensors, commercial tactile sensors play an increasingly important role in robotics. The most representative examples are BioTac (http://www.syntouchllc.com/Products/BioTac/), PPS (http://www.pressureprofile.com/), Weiss (http://www.weiss-robotics.de/en.html), and Tekscan (http://www.tekscan.com/).

Tactile perception technology has been widely applied to robotics. For example, Bekiroglu et al. ⁸ used a three-fingered dexterous hand to catch objects of different shapes. A tactile model built from the hidden Markov model was used to judge object-grasping stability. Additionally, tactile perception technology has also been applied to sliding detection, ⁹ object localization, ^10,11 tactile servo, ¹² and 3-D modeling. ¹³

Object recognition has always been a key problem in robotics and also important for environment perception. Therefore, how to make use of tactile information and realize object recognition arouse wide attention from researchers, and tactile object recognition has become a major research direction. However, these studies lack a unified theoretical framework. Tactile sensor data acquisition mechanisms differ from each other, and different grasping movements in tactile information acquisition also affect data characteristics. Therefore, no mature method for data collection and classification is established. To help address this problem, Kappassov et al. ⁵ divided tactile object recognition research into three major categories: tactile object identification, texture recognition, and contact pattern recognition. In this article, we adopt different guidelines based on tactile exploratory procedure (EP) which can be used to differentiate the shape of rigid objects, the texture of material surfaces, and the deformation of soft objects. Typical examples of these objects are metal parts, cloth, and fruits, respectively. Figure 1 shows representative objects extracted from research papers illustrating these three kinds of tasks. For rigid object recognition, shape information is primarily used. Objects for texture recognition generally have standard and regular shapes, so it is important to investigate their surface properties for recognition. In contrast, deformable objects are primarily recognized by their stiffness and damping characteristics.

OPEN IN VIEWER

Figure 1.

Three kinds of representative objects which are usually used for tactile recognition research. Left: rigid objects used in the study by Luo et al. ¹⁴ ; middle: textured material used in the study by Sinapov et al. ¹⁵ ; right: deformable objects used in the study by Drimus et al. ¹⁶ The figures in left and middle panels have been reproduced with permission from IEEE. The figure in right panel has been reproduced with permission from Elsevier.

In this article, we divide the current studies on the tactile object recognition into three subcategories and present detailed discussions on them. In addition, we also discuss some advanced topics such as visual–tactile fusion, EP, and data sets. Please note that in the study by Dahiya et al., ³ the tactile sensing was classified with respect to some criterions such as the sensing principle, location of the sensor, the task to be done, and so on, while our work focuses on extrinsic tactile sensing according to the above classification.

The rest of this article is organized as follows: In “Exploratory procedure” section, we analyze the EP for tactile information acquisition. In “Tactile perception for shape,” “Tactile perception for texture,” and “Tactile perception for deformable objects” sections, we discuss how to use the tactile sensor to perceive the shape, stiffness, and texture. “Visual–tactile fusion for object recognition” section is about visual–tactile fusion. “Data sets” section gives the information about some data sets. The final section gives the conclusion.

Exploratory procedure

Since tactile signals can only be obtained by contacting and touching, the EP for collecting the tactile data is very important. Generally speaking, recognizing shape requires to contact and grasp the object; recognizing materials requires to slide on the surface of the robot; and recognizing deformability requires to squeeze the object. However, there does not exist a unified strategy for the EP. According to the findings of experimental psychology, ¹⁷ human needs six kinds of actions to explore objects, which were as follows:

Press the object to determine the stiffness of an object.

The above actions can be imitated by the robotic manipulator to perform the tactile exploration. In the existing work, the main EPs can be roughly classified as the following categories:

Passive mode: In this case, the robotic manipulator is fixed, and the human operator hands over the object to the manipulator. The robotic hand mounted on the manipulator blindly grasps the object and performs actions such as press, squeeze, and so on. For example, in the study by Chitta et al., ¹⁸ the human operator grasps the bottle and put it in the space between the two-finger hand, and then, a press action is performed to obtain tactile sequences. Such a passive mode is very popular in the recognition tasks for textured materials. Because the diversity of the materials, various EPs are combined together to obtain sufficient tactile information. Both the work by Sinapov et al. ¹⁵ and Chu et al. ¹⁹ take five different EPs for such task.

From the above discussions, we find that different EPs should be performed to recognize the object’s shape, stiffness, texture, and so on. It is difficult, if not impossible, to recognize all properties of an object using one single tactile EP. Therefore, we naturally classify the existing work on tactile object recognition according to action of tactile EP. In the following three sections, we will discuss how to use the tactile sensor to perceive the object’s shape, texture, and deformability.

Tactile perception for shape

Shape classification of rigid objects using tactile signals has a long history. For example, Allen and Roberts ²⁵ used touch point cloud data to fit the quadratic surface model and further realized the tactile shape recognition. Russell and Wijaya ²⁶ described the process of object recognition using a mobile robot equipped with eight tentacle tactile sensors. While moving, a tentacle tactile sensor cannot only scan and instantly record the shape of objects but also recognize, catch, and move the object according to internal programs. After analyzing the tactile information of grasped objects, the least square fitting method and circle fitting method were used to classify and recognize objects. Conversely, Jin et al. ²⁷ proposed the Gaussian process classification method for sparse tactile point cloud. Generally, a single grasp can only perceive the shape of the part that is in contact with the tactile sensor, so it can only handle the situation where objects are with simple shapes. Recently, Liu et al. ²⁸ described the local shape of objects by extracting the covariance matrix of the tactile array signal. Most of the existing work can only deal with simple shaped objects.

When the tactile data are regarded as a low-resolution gray image, image processing and computer vision technologies can be used to extract features from the tactile array signal and classify different shapes. As aforementioned, tactile sensors can only perceive limited parts of objects, so the information from different parts of the object is expected to be obtained through multiple grasps. A bag-of-words (BoW) model can then be constructed for tactile signals. Based on the BoW feature, a naive Bayes classifier can be trained to classify the objects. However, this method requires a manipulator to grasp an object by many times to collect tactile information from different parts. Furthermore, the direct use of tactile image data introduces a lot of noises. To avoid these problems, Pezzementi et al. ²² used the SIFT feature to deal with the tactile image and relaxed restrictions on the position and orientation of objects. Ho et al. ²⁹ applied this method to the task of sliding detection. Tactile sensors collect tactile signals through direct contact. However, since the position of objects may change during the contact, directly applying feature extraction methods in image processing may cause some problems. To address this, Luo et al. ¹⁴ proposed Tactile-SIFT, a new descriptor suitable for tactile image processing, and it was used to construct a BoW model. Figure 2 lists some of the aforementioned experimental objects for tactile shape recognition. These objects usually have obvious shape features, and some are even specially designed for algorithm verification (please see the first row of the left column in Figure 2). It is important to note that object recognition does not only rely on tactile sensors. In reality, visual sensors have already been widely used in this task, and therefore, how to combine the visual and tactile sensing to accurately recognize different objects is worth for further research.

OPEN IN VIEWER

Figure 2.

The representative objects used for shape-based tactile object recognition. Those figures have been reproduced with permission from IEEE. ^21,22,14

Tactile perception for texture

The texture information reflects the distribution characteristics of microstructures on the surface of an object. Classification for the object’s material can be determined by surface texture features. Tactile sensors can perceive a lot of material information that visual sensors have difficulty to perceive. There is an extensive of research in tactile object recognition. To classify the material texture usually requires actions like scraping, sliding, and rubbing to obtain vibration signals. Therefore, it is necessary to consider characteristics of the time series. The most intuitive solution is to employ the signal processing methods. For example, Sinapov et al. ¹⁵ recognized 20 different textures using five scraping actions, Heyneman and Cutkosky ³⁰ recognized eight different textured discs, Romano and Kuchenbecker ³¹ recognized 15 kinds of surface materials, and Jamali and Sammut ³² identified eight surface textures by simulating human behaviors through multiple contacts and implementing a majority voting. Conversely, Dallaire et al. ³³ directly used the slope and peak of the tactile time series to develop a perception model based on the nonparametric Bayesian method and used it to recognize 28 different discs. Additionally, Chitta et al. ¹⁸ learned 25 haptic adjectives for textured materials. All of these works extracted features using the discrete Fourier transform or other frequency domain analysis tools. From another viewpoint, Strese et al. ³⁴ pointed out the similarity between tactile and speech signals, and resorted to the Mel-frequency cepstral coefficients, which are commonly used in audio processing to extract features. Common methods for classification include the nearest-neighbor classifier, support vector machine (SVM), ^15,35 and Gaussian process. ³⁴ One example of using the SVM classifier is seen in the study by Chathuranga et al. ³⁶ In this study, magnetic flux obtained three-dimension tactile values and treated the covariance matrix of these values as features to classify eight texture materials by SVM. More recently, Baishya and Bauml ³⁷ developed deep learning method for robust material classification with tactile skins. Figure 3 shows representative examples of experimental objects for texture recognition, and Figure 4 lists the corresponding operation scenes. Until now, most of the existing work focuses on regular textured objects, and the complicated natural textured objects receive very little attentions.

OPEN IN VIEWER

Figure 3.

Representative textured objects used for texture-based tactile recognition. Left: the 20 material surfaces used in the study by Sinapov et al. ¹⁵ ; middle: the 8 material surfaces used in the study by Jamali and Sammut ³² ; right: the 43 material surfaces used in the study by Strese et al. ³⁴ Those figures have been reproduced with permission from IEEE.

OPEN IN VIEWER

Figure 4.

The manipulation scenarios. Left: scratching trajectories performed by the robot on the plastic kitchen roll. Middle: the robot drags the finger across each material at a constant speed. Right: hardware setup used for the texture recording in the study by Strese et al. ³⁴ Those figures have been reproduced with permission from IEEE. ^15,32,34

Texture materials usually belong to surface characteristics. Therefore, it can also be perceived by visual sensors. Compared to visual sensors, tactile sensors obtain finer texture characteristics and also yield additional information through vibration and sliding. However, most current texture recognition task is limited to simple object shapes. For objects with complex shapes, further work is required to investigate how to design reasonable contact actions and use tactile information for analysis.

Tactile perception for deformable objects

Compared to tactile recognition based on shape and texture, object recognition for deformable objects pays more attention to surface characteristics and the internal state of the object, which cannot be directly obtained by normal visual sensors. Because of the complexity of object deformation, it is difficult to describe the deformation mechanism with a unified model.

Based on the changes of one-dimension force on fingertip, Chitta et al. ¹⁸ proposed a simple and useful method for tactile feature extraction and designed a classification algorithm to classify empty and full bottles. This resulted in a method that could perceive the internal state of an object by touch. For the 5-finger iCub robotic hand, Soh and Demiris ²⁰ used the Gaussian process to extract features from tactile sequences and proposed an incremental object recognition method. It was used to identify soft toys, hard books, and bottles and cans containing different amounts of liquid. Drimus et al. ¹⁶ described a self-designed tactile sensor with 64 tactile sensing modules arranged in an 8 × 8 grid. This sensor was attached to the Schunk2 and Schunk3 fingers, which were used to grasp various objects to build a tactile data set. After the data collection process, distance was measured using dynamic time warping (DTW) and the nearest neighborhood classifier was used to classify objects.

The aforementioned works primarily use hands with two, three, or five fingers, and each finger is able to collect data. However, most of these works only concatenate data from different fingers through simple splicing or counting statistics, which pay more attention to individual fingers rather than the relationship between fingers. In recent years, sparse coding and dictionary learning methods have been successfully applied to fields such as signal processing and pattern recognition, which have attracted the attention from many researchers. Classifiers based on sparse coding has become an important method for dealing with images, text, and videos, but it is rarely implemented in tactile object recognition. This is because the tactile sequence does not satisfy the linear reconstruction assumption and conventional linear sparse coding cannot be applied. To solve this problem, Nguyes et al. ³⁸ introduced kernels to sparse coding. This method not only reduced reconstruction error but also made dealing with tactile time sequences in non-Euclidean space more convenient. On the other hand, in practical applications, complex signals imply some potential structure information in addition to sparsity. The question of how to make full use of this information has increased the academic attention on structured sparse coding. Recently, Liu et al. ³⁹ proposed joint sparse coding to model the relationship between fingers. This method effectively improved object recognition accuracy and also provided a new unified processing approach for multifinger tactile sequences. In the study by Liu et al., ^40,41 the dictionary learning method was further introduced to improve the performance.

Figures 5 and 6 provide literature examples of experimental objects and their corresponding operation scenarios for the identification of tactile deformation. Objects used in these studies are common household objects such as fruits or containers like cans and bottles. Furthermore, the operation patterns of these objects differ from each other. All these factors make the object recognition very challenging. To address this, Madry et al. ⁴² proposed a spatiotemporal unsupervised feature learning method that made full use of the correlation information from tactile sequences both in time and space domains, and the method was validated on different data sets. ^20,16 Another study ²⁴ considered the deformation of food and designed operations with knives and forks to obtain the tactile information. After training, 12 different foods can be successfully recognized. On the other hand, Liarokapis et al. ⁴³ used a soft manipulator and simplified the grasp process. A random forest classifier was designed based on deformation and rigidity. Xu et al. ²³ integrated force, oscillation, and temperature signals from BioTac sensor to deal with both the texture and deformation recognition.

OPEN IN VIEWER

Figure 5.

Representative objects used for stiffness-based tactile recognition. The second figure has been reproduced with permission from Elsevier. Other figures have been reproduced with permission from IEEE. ^18,16,20,39

OPEN IN VIEWER

Figure 6.

The corresponding manipulation operation. The second figure has been reproduced with permission from Elsevier. Other figures have been reproduced with permission from IEEE. ^18,16,20,39,

Most of the aforementioned works used tactile sensors integrated on fingers and grasped objects with the precise grasp. However, many manipulators also have tactile sensors on their palm, which inspire some scholars to grasp objects with a power grasp. For example, Navarro et al. ⁴⁴ and Schmitz et al. ⁴⁵ used this grasp to recognize multiple objects based on a single layer neural network and deep learning technology. Ma et al. ⁴⁶ also used this power grasp strategy to classify 16 different objects. Figures 7 and 8 list the experimental objects and grasp sceneries used in these works. A future trend is to combine the two grasp methods together to obtain better performances.

OPEN IN VIEWER

Figure 7.

Representative objects used for the power grasp strategy. The figures in left and Middle panels have been reproduced with permission from IEEE. The figure in right panels has been reproduced with permission from Springer. ^44,45,46

OPEN IN VIEWER

Figure 8.

The manipulation scenes. The figures in left and middle panels have been reproduced with permission from IEEE. The figure in right panels has been reproduced with permission from Springer. ^44,45,46

Visual–tactile fusion for object recognition

To achieve fine operation, robots are usually equipped with a variety of sensors. If all sensors only perceive and understand the surrounding environment with independent methods for different modalities, the intrinsic relationship between different modalities is cutoff, which severely reduces the intelligence of the perception action. Therefore, in order to determine the object’s location and attribute information, we must study the multimodal (e.g. visual and tactile modalities) fusion theory and methods. The problem of visual–tactile fusion for perception has attracted the attention of many robotic researchers. The internationally renowned academic conference Robotics Science and Systems 2015 specifically organized a workshop named Learning for visual and Tactile Interaction.

Visual and tactile modalities are quite different from each other. Firstly, the format, frequency, and range of object information are different. The tactile sensing obtains the information about objects when it continuously touch the object, while the visual modality can simultaneously obtain multiple different features of an object at a distance. Furthermore, some features can only be obtained by one single perceptual mode. For example, the color of an object can only be obtained visually, while the texture, hardness, and temperature of a surface can be more naturally obtained through the tactile sensing. Asynchronous information obtained from two modalities and different perception ranges brings a great challenge to multimodal fusion. There are various methods for visual and tactile fusion. Son et al. ⁴⁷ reported the experimental result about using both visual and tactile information for manipulator grasping and pointed out that with the introduction of tactile information, more precise result can be obtained. Boshra and Zhang ⁴⁸ realized localization of polyhedron by integrating visual and tactile information.

The application of visual–tactile fusion is very extensive. Ilonen et al. ⁴⁹ used the visual images and tactile signal for 3-D reconstruction. In the study by Bjorkman et al., ⁵⁰ the visual image was first used for rough modeling, and then the tactile signals were used for fine-tuning. The grasp stability learning using visual–tactile fusion were investigated by Bekiroglu et al. ⁵¹ The work on dense mapping was recently developed by Bhattacharjee et al., ⁵² which assumed that the visually similar scene should have similar tactile characteristics. In the study by Prats et al., ⁵³ the author developed a system which integrates the visual, tactile, and force to perform the task of complicated opening doors. Luo et al. ⁵⁴ investigated the correspondence between visual and tactile features.

As to the visual–tactile object recognition, the existing work are still limited. Woods and Newell ⁵⁵ presented a detailed explanation on this problem. An early work was developed by Kroemer et al., ⁵⁶ which focused on utilizing visual information to help tactile feature extraction. In that work, the author discovered a very important problem which indicated that the visual data and the tactile data are weakly paired. In addition, they just used the tactile information for classification, while visual information is just used for training. Very recently, Gao et al. ⁵⁷ and Zheng et al. ⁵⁸ utilized deep learning method for the joint learning of visual and tactile information. All of the above work deals only with the textured material. For deformable objects, Guler et al. ⁵⁹ presented an interesting work which tried to tell the internal state of a container. The author used a fixed Kinect camera to capture the object in order to detect the deformability, and then the tactile signals were introduced to analyze the internal state. However, this method requires the 3-D model of the object and therefore it is difficult to be extended to unfamiliar objects. In Figure 9, we list some representative data collection scenes or the algorithm architectures, which are adopted in the corresponding references.

OPEN IN VIEWER

Figure 9.

Some experimental scenes of algorithm architectures of the visual–tactile fusion work. Those figures have been reproduced with permission from IEEE. ^56,57,59

As to the multimodality fusion, the joint sparse coding method has become a very effective strategy. ⁶⁰ This method benefits from the sparse coding technique and is able to characterize the intrinsic relationship among different modalities by requiring the coding vectors to share the common sparsity pattern. Since the coding vectors are not required to be completely equal, the difference among them can also be effectively preserved. Such coding strategies has already been successfully used in face recognition and identity validation. ⁶¹ Very recently, Liu et al. ⁶² attempted to develop joint sparse coding method for visual–tactile fusion and solved the intrinsic weakly pairing problem by relaxing the joint coding requirements. In the left panel of Figure 10, we list some representative experimental objects. In the right panel of Figure 10, we can clearly see the complementary roles of the two modality. In fact, we can see two bottle, one is full and the other is empty. Their visual appearances are very similar while their tactile characteristics differ significantly. Liu et al. ^62,63 provided a number of such examples to illustrate motivation of visual fusion for object recognition. Currently, there still exists great challenges due to the gap between different perception modalities.

OPEN IN VIEWER

Figure 10.

Left: representative objects used for visual–tactile fusion recognition. Right: two objects which exhibits similar appearances and different tactile signals. ⁶² Those figures have been reproduced with permission from IEEE.

Data sets

Currently, there are a lot of visual object recognition data sets which are developed as benchmark. Some of them could be used for robotic application, such as the 300 objects data sets developed by Lai et al. ⁶⁴ As a comparison, the construction work of tactile object data set is still in a very preliminary stage. Most of the existing work uses their own developed small-scale data sets and isn’t released for public usage, which prevents the comparison under same standards. Here, we summarize some representative data sets as below:

Conclusions

In summary, tactile information and visual–tactile fusion information are very important in the process of robot precise operation and are also the focus of research in the field of robot perception. However, the research work in this area is faced with many challenges at present. The specific summary is as follows:

Although object recognition based on tactile information has made great progress and many research results are achieved on the recognition of shape, texture, and deformation, there is still a long way to practical applications. In the meantime, the recognition of deformable objects has become a main direction of tactile object recognition because it combines the surface characteristics and the internal state of the object. Machine learning has become the mainstream method for tactile object recognition. At present, the reported methods include nearest-neighbor classification, SVM, decision tree, hidden Markov model, Gaussian process, and Bayesian learning. Application of deep learning in tactile signal processing is also started. However, it is still a challenging problem of how to achieve a more efficient and accurate classification by effectively extracting the tactile features of ordinary objects.

Overall, visual–tactile fusion and other multimodal perception technology provide a more extensive and comprehensive perception information for robots to understand and interact with the environment. It is of great significance to improve robots’ operation intelligence level as well. More research is expected to conduct in the aspects of signal processing, machine learning, and robotic manipulation.