Embodied tactile perception and learning

2.1 Finger tactile sensing

Tactile sensors are widely used in robotic fingers, which can be divided into fingertips and other parts of the finger. A tactile sensor can be integrated at the tip of the robot's finger (i.e., end of the finger) and used to recognize an object’s attributes (e.g., texture, hardness, and shape) by performing different exploratory operations, including sliding, squeezing, pushing, and tapping. Fig. 2 shows some representative work. Winstone et al. [20] integrated a TACTIP tactile sensor that mimics the structure of a human finger on the ELU‐2 Elumation robotic hand, to analyze the texture characteristics of objects. SynTouch Inc. developed a multi‐modal fingertip tactile sensor, called BioTac, which has been integrated into robots (e.g., Shadow, Barret, and Allegro) for tasks such as material classification, object recognition, and slip detection [21]. Schmitz et al. [22] designed a flexible printed circuit board integrated with a capacitive tactile sensing array and developed a fingertip tactile sensor for iCub robots. Koiva et al. [23] designed a rigid three‐dimensional (3D) resistive tactile sensor array using the laser technology, and mounted it on the fingertips of the Shadow robot. Meanwhile, Jara et al. [24] described PPS RoboTouch (Pressure Profile Systems, Inc.) capacitive tactile array installed on four fingertips of the Allegro robotic hand to collect the grasping force for a dexterous manipulation. A fingertip optical tactile sensor, called GelSight, can measure pressure, torque, shear force, and slipping, and is used to estimate object hardness and recognize the object material [25]. The leading author’s group recently installed optical tactile sensors on the fingertips to realize object material recognition and slip estimation [13].

OPEN IN VIEWER

Fig. 2.

Representative work on fingertip tactile sensing. (A) Reproduced with permission from Ref. [20], ©Springer Nature, 2013. (B) Reproduced with permission from Ref. [21], ©IEEE, 2013. (C) Reproduced with permission from Ref., ©IEEE, 2010. (D) Reproduced with permission from Ref. [25], ©IEEE, 2019.

Besides fingertips, tactile sensors are also widely used in other parts of the finger. Fig. 3 shows some representative work. Schmidt et al. [26] proposed the integration of a PPS capacitive tactile array into the gripper of the PR2 robot to explore the characteristics of the object surface to achieve human–robot interaction. To mimic the function of human fingers, Jamali and Sammut [27] used randomly distributed strain gauges and polyvinylidene fluoride (PVDF) films to develop mechanical fingers for identifying different surface textures. Yousef et al. [28] integrated an optical tactile sensor array with 41 contact points into a two‐finger robotic hand, to collect tactile information. Meanwhile, Teshigawara et al. [29] proposed the integration of highly sensitive tactile sensors into a lightweight multifinger robotic hand for slip detection. Heyneman et al. [30] designed a capacitive tactile sensing array and mounted it on Robotiq's grippers to identify different types of slip. Drimus et al. [31] designed an 8 × 8 tactile array based on a piezoresistive conductive rubber and mounted it on a three‐finger Schunk SDH robotic hand to classify deformable objects. In addition, Odhner et al. [32] integrated a barometer‐based tactile sensor array into a finger of an iRobot–Harvard–Yale robotic hand. Suárez‐Ruiz et al. [33] developed a CoRo Lab tactile sensor with a 4 × 7 resolution. They used it to detect the pressure, contact position, and vibration of Robotiq’s fingers. Tenzer et al. [34] proposed the integration of a microelectro mechanical system (MEMS) barometer into the fingers of a RightHand Robotics ReFlex TakkTile hand to feel the force distribution. Moreover, Jamone et al. [35] proposed a magnetic tactile sensor and installed it on a robotic hand for obstacle avoidance and object recognition. Based on the magnetic technology, Paulino et al. [36] proposed a three‐axis tactile sensor and installed it on the fingers of a Vizzy robot to detect normal and shear forces. Funabashi et al. [37] proposed the mounting of uSkin tactile sensors on the fingers of Allegro robots for object recognition. Wilson et al. [38] also indicated that integrating multiple GelSight optical tactile sensors in the fingers of a robotic hand can improve grasping efficiency.

OPEN IN VIEWER

Fig. 3.

Representative work on finger tactile sensing. (A) Reproduced with permission from Ref. [36], ©IEEE, 2017. (B) Reproduced with permission from Ref. [37], ©IEEE, 2018.

Robotic finger tactile sensing is widely used in many scenarios. Many tactile sensors have been equipped on commercial robotic hands (e.g., Shadow, Barret, Schunk SDH, and Robotiq) and robot platforms (e.g., UR, PR2, and iCub) for practical applications (e.g., environmental detection and object attribute recognition).

2.2 Palm tactile sensing

Palm tactile sensing plays an important role in applications such as dexterous manipulation (e.g., grasping and kneading), power grasp, and human–robot interaction (e.g., handshake and flicking). Fig. 4 shows some representative work. Capacitive tactile sensing arrays based on PPS have been installed in the palms of some commercial robotic hands. For example, a capacitive tactile sensor array with 96 contact points was mounted on a Barret Hand, 24 of which were distributed on the palm [39]. The Robonaut robot uses data gloves to detect the force distribution at 19 contact points. Tomo et al. [40] integrated a magnetic silicone tactile detection unit on the palm of a robotic hand. Wang et al. [41] placed four contacting points on the palm of the designed service robot. Pastor et al. [42] integrated a high‐resolution tactile array containing 1400 tactile sensing units on the palm of a robotic hand, which collaboratively worked with the under‐actuated fingers. The palm has a limited contact with the object during the operation; hence, it is greatly affected by the hand’s operating configuration. A palm usually needs to work together with the finger part to implement a task such as object material classification.

OPEN IN VIEWER

Fig. 4.

Representative work on palm tactile sensing. (A) Reproduced with permission from Ref. [39], ©IEEE, 2016. (B) Reproduced with permission from Ref. [41], ©IEEE, 2019. (C) Reproduced with permission from Ref. [42], ©the authors, 2019.

2.3 Arm tactile sensing

During a robotic manipulation, the robotic hand and arm must expand the workspace and increase the dexterity of the operation. Many researchers currently focus on hand tactile sensing, ignoring the importance of arm tactile sensing. Arm tactile sensing plays an important role in human–robot interaction, safe operation, and environment detection. Fig. 5 shows some representative work.

OPEN IN VIEWER

Fig. 5.

Representative work on arm tactile sensing. (A) Reproduced with permission from Ref. [43] , ©IEEE, 2015. (B) Reproduced with permission from Ref. [44], ©IEEE, 2017. (C) Reproduced with permission from Ref. [45], ©IEEE, 2018.

Accordingly, the Institute of Cognitive Systems at the Technical University of Munich designed a hexagonal tactile module to detect temperature, vibration, and light touch to simulate a multi‐modal tactile perception of the human skin. The tactile module was made into a patch and installed on the manipulator from KUKA Robotics Corp to improve its control performance [46]. Bhattacharjee et al. [43] used a robotic arm covered with capacitive tactile sensing arrays to identify a cluttered environment. Albini et al. [44] tiled capacitive tactile sensors on a robotic arm to distinguish various human–robot interactions, such as movement and rotation. Vergara et al. [45] integrated an electronic skin containing 373 tactile sensing units into a UR manipulator to ensure safety when the workspace was shared with other people. Meanwhile, Leboutet et al. [47] used tactile sensing information on an arm to implement compliant operation control. A research team from the Harbin Institute of Technology recently studied a method of deploying tactile sensors on the arms of nursing robots to achieve a safe operation [48]. Research on arm tactile sensing is presently in the beginning stage. Many researchers have adopted a "sleeve"‐style installation for integration into existing robotic arms, which is difficult to adapt to the large extension movement of a robotic arm.

2.4 Torso tactile sensing

In robots, the torso refers to body parts such as the chest, abdomen, and back. Besides the fingers, palms, and arms, torsos also play an important role in the humanoid robot and human–robot interaction. Fig. 6 shows some representative work.

OPEN IN VIEWER

Fig. 6.

Representative work on torso tactile sensing. (A) Reproduced with permission from Ref. [49], ©Taylor & Francis, 2015. (B) Reproduced with permission from Ref. [50], ©IEEE, 2019.

In reality, torso tactile sensing is also likely applicable to the hands, arms, legs, and other parts of the body. Argall et al. [51] summarized the applications of robotic torso tactile sensing in early human–robot interaction systems. In recent years, research on torso tactile sensing has attracted much attention with the continuous development of the large‐scale electronic skin technology. Metta et al. [52] used a capacitive tactile sensing unit to cover the torso of an iCub robot for cognitive ability research. Harada et al. [53] equipped a fingerprint‐like three‐axis tactile force sensor on a robot's body for tactile and slip detection. Mittendorfer et al. [49] imitated the human skin and developed a multi‐modal tactile sensor array, called CellulARSkin, to detect information such as proximity, vibration, temperature, and pressure. A tactile demonstration guides the movement of the entire body to grasp unknown objects. Kaboli et al. [54] further applied CellulARSkin to the torso of a small humanoid robot, NAO. Büscher et al. [55] designed a flexible, stretchable tactile sensor that can seamlessly cover a robot's body for daily human–robot interactions. Ku et al. [50] equipped a humanoid robot with a distributed tactile sensing array to identify 18 social touch patterns (e.g., hit, tap, and pat). Our research team recently installed a tactile array on the back of a service robot to recognize the tactile emotion during the human–robot interaction [56].

Distributed sensing in large‐scale, high‐resolution electronic skin and high‐speed signal interconnection are presently the main bottlenecks for equipping robotic torsos with tactile sensors [57].

2.5 Leg–foot tactile sensing

The research on leg–foot tactile sensing currently focuses on the feet, because they are the areas with the richest contact with the ground. The foot's tactile sensing helps a robot to sense the ground characteristics, adjust gait, improve walking efficiency, and ensure walking stability. Fig. 7 shows some representative work. Shill et al. [58] and Bednarek et al. [59] studied biped and multi‐pedal robots, respectively, in terrain classification, to measure physical interaction information and realize a terrain classification using the tactile sensors integrated into the feet. Wu et al. [60] used a flexible capacitive sensor array on robotic feet to measure the tactile force distribution with the ground and adaptively adjust the gait. Guadarrama et al. [61] studied tactile feedback control for a biped robot when it adapts to an unknown terrain. A research team at Massachusetts Institute of Technology (MIT) realized autonomous high‐speed bounding of Cheetah robots on a complex terrain using tactile intelligence [62]. Meanwhile, the research team from the Harbin Institute of Technology recently studied a terrain classification method using tactile sensors mounted on the bottom of the wheeled robot [63]. A research team in Beijing Nano‐Institute, Chinese Academy of Sciences, designed a smart insole with a novel air pressure‐driven structure to accurately monitor and distinguish various gait patterns [64].

2.6 Advanced tactile sensing for joints

Besides the fingers, palms, arms, torsos, legs, and feet, tactile sensing can also be deployed on tentacles and shoulders in a real robotic system. The current research mainly aims at specific parts of the robot. The resolution is not very high, although a significant progress has been made to develop the robotic tactile sensing technology. A large‐scale distributed tactile sensing system forms the basis for robots to realize embodied perception and learning, while a large gap still exists between the characteristics of the existing array‐type tactile sensors and the sensing ability of the human skin. Although a breakthrough has recently been made in the structure and materials of the electronic skin, achieving high flexibility and elasticity is still challenging. Large‐area tactile sensing arrays generally have poor scalability and are not easy to cut and splice. Manufacturing high‐sensitivity tactile sensing arrays is complicated and expensive, and therefore, are difficult to mass‐produce. Considering these challenges, deploying large-area tactile sensors on any robotic surface has not been realized yet, thereby restricting the development of robotic embodied intelligence.

OPEN IN VIEWER

Fig. 7.

Representative work on torso tactile sensing. (A) Reproduced with permission from Ref. [61], ©the authors, 2019. (B) Reproduced with permission from Ref. [59], ©IEEE, 2019.

The deforming and stretching characteristics of tactile sensors must be considered because of robot kinematics. Traditional tactile sensors are difficult to implement because of wiring constraints. Some tactile sensors are based on an optical measurement and are limited by the imaging distance and field of view. Recently, tomography‐based tactile sensors have been developed [65 –67], which, in principle, can overcome the limitations of wiring, and have received great attention in the field of flexible stretchable electronic skin. Electrical impedance tomography (EIT) is an imaging technique that uses electrodes to measure the voltage or current signals from the boundaries of the electrical conductors and reconstructs the internal conductive distribution. It can be used to implement robotic tactile sensing if a flexible stretchable material is used to develop the electrodes and attached to the robot surface. Tomography‐based tactile sensors are expected to provide a new solution for the development of robotic embodied tactile sensing.

In Ref. [68, 69], rubber materials were used as electrical conductors and EIT was implemented to reconstruct the resistivity distribution. The stretchable electronic skin was designed in Ref. [70, 71], which can adapt to the elbow of a robot with 19 electrodes and was designed into a thin, flexible, and stretchable artificial skin that can be applied on the fore arm and upper arm of the robot to obtain information (e.g., contact position, contact duration, and contact intensity). Furthermore, Pugach et al. [72] and Park et al. [73] used neural networks to implement image reconstruction. Fig. 8 shows some representative work on tactile sensing based on EIT.

OPEN IN VIEWER

Fig. 8.

Representative work on tactile sensing based on the EIT. (A) Reproduced with permission from Ref. [68], ©IEEE, 2007. (B) Reproduced with permission from Ref. [65], ©the authors, 2016.

Compared to traditional tactile sensors, EIT has advantages of low cost, low power consumption, no internal wiring, flexibility, scalability, and repeatability. It has become an important tool for the electronic skin because of its outstanding performance in designing a deformable, soft, flexible, and stretchable multipoint tactile sensor and detecting irregular surfaces and narrow areas. However, EIT has some shortcomings. Besides its higher requirements for electrical conductors, its temporal and spatial resolutions are low. Its current applications on robots are mainly limited to pose detection, security protection, and human–robot interaction. Tactile sensors based on EIT are generally more suitable for a spatial resolution of 10–40 mm and a temporal resolution within 45 Hz. This is obviously not enough for robotic tasks, such as object recognition and robotic grasping [74].

Electrical capacitance tomography (ECT) is another imaging method for visualizing permittivity distribution in the interior of a dielectric object by measuring the capacitance at the boundary, and has been widely used in industrial applications. Muhlbacher‐Karrer et al. [75] described an application of ECT in conductor detection. While two soft‐field imaging modalities (i.e., ECT and EIT) have some similarities, ECT can provide a quantitative measurement. However, note that EIT can provide a qualitative measurement only. For the electronic skin, ECT can realize non‐contact perception and object internal imaging. While non‐contact perception can provide predictive information for a safe operation, internal imaging is beneficial for the tactile perception of complex movements, such as power grasp, embracing, and chest/back contact. In addition, using a capacitor array on a robot enables the performance of imaging diagnosis on key parts of a robot, which is significant in maintaining the robot itself. However, there have been few reports on the application of ECT in the fields of robotic tactile sensing and electronic skin.

3.1 Tactile feature learning and classification

The pattern recognition methods of robotic tactile perception have been described in detail in [18, 19]. Accordingly, many features have been proposed for tactile perception, including time domain, frequency domain, geometric, and statistical features. Some previous studies have been conducted to convert tactile signals into low‐resolution images. Image features, such as image moments and scale‐invariant feature transform, are then extracted. Many classifiers, including nearest neighbors, naive Bayes, decision trees, support vector machines, and artificial neural networks (ANNs) [76], are applied to tasks of object recognition, slip detection, and grasp quality evaluation. However, most of these classifiers use off‐the‐shelf feature extractors to process tactile signals. They have neither fully explored the inherent characteristics of the tactile signals nor considered the practical application of tactile perception.

Inspired by the sparse coding idea, dictionary‐based representation learning has been recently used to learn middle‐level features through low‐level tactile features. A new classifier has been designed based on structured sparse coding [77], which can effectively describe the dynamic characteristics of tactile signals and is suitable for a few samples of tactile data. Feature coding and classifiers can be adjusted to correspond to specific perception tasks. The method has been applied to tasks such as heterogeneous finger information fusion, tactile adjective attribute identification, and tactile object classification, thereby laying a foundation for the feature and classifier joint design.

With the development of deep learning and its successful applications in the fields of image, video, and speech, some researchers have begun applying deep neural networks to tactile information processing. Madry et al. [78] used unsupervised learning methods to learn the spatio‐temporal features of tactile sequences and obtained the best results for multiple tactile object recognition datasets. Afterward, deep learning methods quickly attracted researchers in the field of tactile perception. To solve the material recognition problem, a tactile sensor with a 4 × 4 resolution was used to collect a 2‐second sequence. A 1500‐dimensional sample was then obtained after 750 Hz sampling. The 1500 × 4 × 4 spatio‐temporal signal was converted into a 1500 × 16 two‐dimensional signal to achieve classification using a conventional neural network (CNN) [79]. Ji et al. [80] proposed the usage of weights of a trained sparse auto‐encoder to initialize those of a CNN for tactile signals, to improve performance and convergence. Zheng et al. [81] further studied full convolutional network learning. A hybrid deep learning structure with a CNN and a recurrent neural network were designed to process sequential tactile information online and solve the tactile emotion recognition problem in the human–robot interaction process [82]. Sohn et al. [83] applied deep learning to large‐scale electronic skin tactile perception. By using various contact forces, the obtained tactile spatio‐temporal sequence information was integrated and a 3D CNN was designed to realize object recognition [42]. CNNs are mostly designed for images; thus, good results can be expected on tactile sensors based on optical measurement. Yuan et al. [84] used a CNN to represent the tactile image obtained by the GelSight sensor. A recursive neural network was then employed to model the object deformation over time. Subsequently, a shape‐independent hardness estimation method based on deep learning was designed. Polic et al. [85] developed a convolutional self‐encoder for feature extraction in optical tactile sensors. Considering the lack of labeled samples, Erickson et al. [86] proposed a method based on semi‐supervised learning for object recognition based on generative adversarial networks. This method enables a robot to estimate the texture of everyday objects with approximately 90% accuracy when 92% of the training data are not labeled. A zero‐shot learning method was recently proposed for robotic tactile fabric recognition [87]. The University of Science and Technology of China used an optical equipment to obtain high‐resolution tactile information and a 50‐layer residual network ResNet to process the diffraction patterns, to improve the sensor performance [88].

Tactile data are more difficult to collect and label than visual and auditory modalities; thus, the performance of the current tactile deep learning is not significant. At present, most of the deep neural networks for tactile information processing are borrowed from the visual field and cannot specifically capture the tactile information characteristics. The shortcomings of deep learning in terms of long time and high energy consumption also restrict its further application in robotic tactile perception. In addition, relying on several manually labeled samples for offline learning, deep learning approaches can hardly be applied to robotic tactile sensing tasks, including online motion, exploration, and contact.

3.2 Active tactile perception

Although progress has been made in tactile feature learning and classification in recent years, most of the studies still collect data based on predefined rules and use pattern recognition methods to process them. Tactile perception can only provide local information on environment characteristics; thus, efficiently achieving a comprehensive perception of the environment is difficult. One of the important characteristics of robotic embodied perception is that it is an active process.

Active tactile perceptions attracted attention in the 1980s [89] for enabling robots to control the position and movement of tactile sensing systems through sensory exploration. The autonomous perception ability of a robot is enhanced with the optimized sensory information obtained from the environment and the reduced number of explorations. Seminara et al. [90] introduced the progress of active tactile perception from the perspective of autonomous learning and safe interaction. The current robotic active tactile perception focuses on two main aspects: shape reconstruction，and recognition and classification.

For shape reconstruction, Strub et al. [91] designed an active exploration strategy by considering the errors between object shape representation and tactile feature localization, simultaneously, to automatically realize the shape representation of objects. Abraham et al. [92] verified that low‐resolution binary contact sensors can be used to achieve shape estimation through effective active exploration. Jamali et al. [93] used Gaussian process classification to effectively sample an object’s surface and proposed an effective active learning strategy for its tactile exploration. Sommer et al. [94] designed an active exploration strategy for multifinger robots to adjust the number of contact points in real time, such that a robot can autonomously adapt to the object shape. Driess et al. [95] improved the traditional discrete point query method in the Gaussian process method through smooth path optimization, to avoid inefficient touch and retraction motion. A method for the fast shape estimation of unknown objects was proposed by introducing tactile uncertainty estimation into the exploration time constraints [96]. Ottenhaus et al. [97] maximized the estimated information gain in information theory to minimize the expected cost of exploration actions. They realized an active reconstruction of the object shapes from the sparse tactile data obtained from the robot fingers. However, limited by the accuracy of tactile sensors and exploration actions, the high‐precision and fast shape reconstruction of complex unknown objects through efficient active tactile exploration remains unsolved.

In recognition and classification, Saal et al. [98] resorted to information theory to analyze the dynamic parameters of bottles filled with different liquids, to estimate uncertainty. They then chose the next best action to quickly learn the dynamic parameters of objects through active perception. Zhang et al. [99] used the Monte Carlo tree search to optimize the grasping motion of a robot and realize active object shape recognition. Lepora et al. [100] established an active Bayesian tactile perception method to analyze the estimated uncertainty of each step of the action. The robotic operation was optimized to obtain more information in the next action. This method was applied on the tactile sensors integrated into the fingertips and tentacles [101]. Martinez‐Hernandez et al. [102, 103] further used multifinger coordination to realize active tactile object shape recognition. Tanaka et al. [104] transformed the active tactile object recognition problem into an optimal control problem. Active tactile exploration was achieved by optimizing the informativeness of the exploration and the compliance of action. It was then combined with the Gaussian process for object shape recognition. With a multivariate Bayesian classifier, Sun et al. [105] proposed a recognition method that fuses tactile information, such as friction and surface roughness, and local geometric features through active exploration. Xu et al. [106] realized the fusion of surface hardness and geometry information using active exploration with a tactile stylus. Kaboli et al. [107] recently designed an active tactile sensing method by modeling an uncertain workspace, so that a robot can actively explore the unknown workspace.

The abovementioned results showed that most of the current application scenarios of active tactile perception are relatively simple. The tactile exploration process is too long to be applied to an open environment. In addition, most of the tactile exploration strategies are based on the uncertainty measurement of the state estimation and are not closely integrated with the perception task. Moreover, most of the current active explorations are limited to robotic hands and arms, and movements from the other parts of the robot are rare.

3.3 Multimodal fusion perception

Although tactile signals play an important role in robotic embodied perception, they have limitations. Tactile perception is usually limited to the local area of contact, and the perception process is limited by the contact process. Many real environments and object properties may not be suitable for measuring signals only by tactile modality. In addition, the service time of tactile sensing is affected by frequent contact with the environment. Real robotic systems are usually equipped with other sensors, such as visual auditory. Visual auditory and tactile modalities complement each other in both perceptual range and characteristics. Therefore, multimodal sensors must be comprehensively leveraged to achieve fusion perception for robotic embodied perception and learning.

Visual and tactile modalities exhibit a big difference. Their format, frequency, and range of information vary greatly. Visual modality is generally more suitable for processing features such as color and shape, while tactile modality is more suitable for processing temperature, hardness, and other characteristics. Both visual and tactile modalities can be used to process surface material. The former generally deals with coarser materials, while the latter deals with finer materials. However, tactile modality can usually obtain information on objects with robot contact, and requires a long exploration process. Meanwhile, visual modality can be used to obtain considerable information on objects in the field of view. The asynchrony and difference in the sensing range of the two modalities bring great challenges to information fusion. Previous studies on visual–tactile fusion mainly focused on 3D object reconstruction [43, 108]. Accordingly, visual–tactile object recognition has attracted attention in recent years. Kroemer et al. [109] studied the use of visually assisted tactile feature extraction and texture classification, indicating that a key problem in visual–tactile fusion is the difficult pairing of visual images with the corresponding tactile data. To solve this problem, a visual–tactile object recognition method was established using weakly paired sparse coding and dictionary learning techniques [110, 111].

Recently, deep learning has been applied to solve visual–tactile fusion problems. For example, Gao et al. [112] used CNNs to study the joint learning method of visual images and tactile data. Zheng et al. [81] improved the network structure. Kerzel et al. [113] proposed a multichannel deep neural network structure and performed a spectrum analysis on the vibration and texture data collected during the exploration, by classifying 32 daily life materials. Yuan et al. [114] investigated the method of fusing an optical image and a tactile image collected by GelSight. Deep adversarial learning was used to solve a weakly paired problem between visual and tactile modalities [115, 116].

When a robot interacts with the environment, sound signals can be easily obtained besides tactile signals. Sound signals have the advantage of being superior to visual or tactile modalities in identifying certain internal states or materials. Similar to visual–tactile fusion, visual‐sound‐tactile modal information presents a more obvious weakly paired characteristic. Strese et al. [117] built a multimodal dataset that contained more than 100 types of surface materials. The problem in visual–sound–tactile heterogeneous fusion material recognition was further addressed using the sparse coding framework [118, 119].

Recently, researchers have focused on realizing multimodal robotic active perception by using visual–sound–tactile modal information. Yang et al. [120] combined passive vision with active touch to provide visual priori for tactile perception. Yuan et al. [121] proposed the use of vision for a robot to find the most suitable grasping point, for obtaining high‐quality tactile signals. However, in these processes, the order of sensor usage is manually preset. The problem of comprehensively scheduling sensors of different modalities to achieve efficient sensing based on task requirements is still challenging. Ferreira et al. [122] used the hierarchical Bayesian framework to analyze the emergence of robotic multimodal active perception. Taniguchi et al. [123] introduced information gain to calculate the uncertainty of different modalities under this framework, which was used to realize the automatic scheduling of visual, auditory, and tactile modalities. However, this study was restricted to very limited environments. An active visual–sound–tactile multimodal scheduling method was recently established by introducing adversarial reinforcement learning. The efficient perception of materials using seven types of modalities was achieved [124].