Shared visuo-tactile interactive perception for robust object pose estimation

2.1. Interactive perception

Interactive perception or perceptive manipulation is any kind of purposeful manipulation actions performed to simplify or enhance the perception of the environment (Bohg et al., 2017). Interactive perception techniques rely upon effective scene understanding in order to plan and execute manipulative actions. The scene understanding iteratively improves upon performing manipulation actions. In unstructured cluttered scenarios, the target object may have multiple other objects overlapping on it in random configurations. A typical choice for scene understanding in computer vision is scene graph which is a data structure that describes objects in a scene and the relationships between these objects (Johnson et al., 2015). Support graphs, a type of scene graph, have been introduced to describe the support relations between objects in the scene through geometric reasoning (Kartmann et al., 2018; Mojtahedzadeh et al., 2015; Schwarz et al., 2018). Sui et al. (2017) presented an axiomatic scene estimation method to describe the relationship between objects and object poses as a scene graph for manipulation. Mitash et al. (2019) developed a Monte Carlo Tree Search-based technique for scene understanding leveraging physics-priors of objects in clutter for pose estimation. Zhang et al. (2021) tackled the issue of inferring object relationships through a neural network performing classification task on all possible pairwise permutations between the objects in the scene.

Scene understanding is followed by planning manipulation actions in clutter which is a challenging task and has received immense research interest. Typically, research works on grasping in clutter falls into model-based approaches or data-driven methods. Analytical or model-based approaches have been deeply studied in Bicchi and Kumar (2000). Berenson and Srinivasa (2008) devised an optimisation method for generating the gripper pose in clutter given the knowledge of the object shape to ensure force closure. Some works such as Moll et al. (2017) and Dogar et al. (2012) rely upon physics simulators to model the robot-object and object-object interactions while performing manipulation actions in clutter. More recent studies involving data-driven grasping have leveraged deep neural networks (DNNs) to achieve significant results as detailed in Bohg et al. (2013). Several works leveraging data-driven methods have focused towards grasping a priori known objects (Detry et al., 2011; Goldfeder et al., 2007; Miller et al., 2003; Przybylski et al., 2011) or familiar objects with known object class by matching with a known database of grasping information or ranking based on prior grasping experience (Detry et al., 2012; Mahler et al., 2016; Patten et al., 2020). For grasping of unknown objects, prior works rely upon global shape or features from sensory data and a set of heuristics (Bohg et al., 2013; Morrison et al., 2020; Schaub and Schöttl, 2020; Schmidt et al., 2018). For instance, Morrison et al. (2020) developed an object-independent grasp synthesis method from depth images using their generative grasping convolutional neural network (GG-CNN). Tactile sensors have also been used for manipulation of deformable objects (Kaboli et al., 2016). However, most of the prior works focused on singulated objects (single objects in the scene) or structured clutter wherein there are multiple objects in the scene but they are spread apart with minimal contact between them. However, in dense unstructured clutter there are multiple objects which are densely packed over one another in randomised configurations. In such scenarios, relying upon only grasping actions can severely handicap the robot as some objects may be very hard to grasp due to the surrounding clutter. Some recent works have leveraged both prehensile manipulation such as grasping and non-prehensile manipulation such as pushing actions (Danielczuk et al., 2019; Dutta et al., 2023; Murali et al., 2022b; Zeng et al., 2018). If the objective is to grasp and retrieve an object, then pushing is often used to singulate the object away from clutter aiding subsequent grasp action (Grimm et al., 2021). However in mechanical search the goal is to retrieve a target object, both grasp and pushing actions are used in conjunction in order to declutter the workspace around the target object (Danielczuk et al., 2019; Murali et al., 2022b). The work by Danielczuk et al. (2019) focused on retrieving a target object from clutter and they used a heuristic that removes the largest object first. Furthermore, the target object in their work is known a priori. Their framework is based on deep reinforcement learning to learn the synergy between grasping and pushing and is more data-hungry and compute-intensive.

In comparison with the prior work reviewed above, we focus on estimation of the pose of an unknown target object in dense unstructured clutter. We present a declutter scene graph-based approach that directly encodes the relationship between objects in the scene as well as the type of action to perform (grasp/push) to declutter the objects. Our formulation ensures that only necessary minimal actions are performed such that the target object is not occluded for pose estimation. The actions are also chosen automatically based on the grasp affordance of the object.

2.2. Object reconstruction

Object reconstruction follows the process of capturing the shape and appearance of a 3D object by moving a suitable sensor around the object. Typically, object reconstruction approaches can be classified into statistical model-based and deep learning-based generative models (Phang et al., 2021). Some statistical methods involve capturing point clouds from various viewpoints and aggregating into a common coordinate frame using point cloud registration (Delmerico et al., 2018). Xie et al. (2021) designed a generative model based on PointNet (Qi et al., 2017) capable of performing reconstruction and interpolation. We focus on active object reconstruction techniques wherein the sensing locations are chosen autonomously by robots to improve performance efficiency and avoid an exhaustive search. In this regard, Delmerico et al. (2018) compared various next-best-view (NBV) strategies for an object in an uncluttered scene in simulation demonstrating the usability of information-theoretic criteria for efficient reconstruction. Similarly, Bissmarck et al. (2015) devised an efficient volumetric NBV algorithm exploiting frontier voxels and spatial hierarchy. While these works are limited to one camera, more recently, multiple cameras have been used for reconstruction by exploiting the joint uncertainty in the modelling (Lauri et al., 2020). Along similar lines, Cui et al. (2019) devised a multi-sensor strategy for next best view calculation with a laser range sensor and RGB-D camera based on occupied voxel metrics.

Typical vision sensors, in particular depth sensors are sensitive to transparent and specular objects producing erroneous or missing regions in the measurements. To overcome this limitation, recent works have leveraged transfer learning techniques from the RGB modality which are comparatively less sensitive to transparency/specularity into the depth modality and utilized off-the-shelf depth-based grasping methods for manipulating such objects (Weng et al., 2020). Similarly, Sajjan et al. (2020) developed a technique to reconstruct transparent objects wherein convolutional neural networks are used to infer the normals, contours and semantic segmentation from RGB images which are used to refine the depth estimate in order to recover the shape of transparent objects. Along similar lines, Zhang et al. (2022) developed a transformer-based architecture for depth completion when provided with instance-based segmentation and RGB images for reconstruction of transparent objects. A detailed review on robotic perception for transparent objects is found in Jiang et al. (2023). As majority of the vision-based techniques for reconstruction of transparent objects depend on the availability of high fidelity RGB image priors, it cannot be always ensured in unstructured environments wherein the environmental lighting conditions can vary. Low and bright lighting conditions cause erroneous results with transparent objects (Sajjan et al., 2020). Moreover, such transparent and specular objects when present in cluttered environments are also challenging for vision-based sensing due to shadows and occlusions.

On the contrary, tactile sensors are more robust to the ambient lighting conditions, and are relatively more robust to the transparency or specularity of the objects compared to visual sensors. Thus, tactile sensors have been used independently or in conjunction with vision sensors for robotic object perception (Dutta et al., 2024; Kaboli et al., 2015, 2018; Murali et al., 2022c). Gaussian processes implicit surfaces (GPIS) have been used for shape reconstruction from both vision and tactile inputs (Rustler et al., 2022; Suresh et al., 2022). However, GPIS approaches are known to be computationally expensive (Schulz et al., 2018). Similarly, Wang et al. (2018) designed a framework for generating 3D shape of objects from a single visual image using learnt shape priors which is refined using tactile sensing. They also performed uncertainty based next-best-touch (NBT) computation while keeping the camera static.

Previous works used touch to refine the shape estimated prior by vision in a two-step process. In comparison, in this work we present a novel approach for shared active object reconstruction using a joint information gain metric for sensor selection criteria and next-best-view (NBV) or next-best-touch (NBT) execution. The robots equipped with vision and tactile sensors coordinate autonomously to reconstruct the object with minimal actions for the objective of object pose estimation and avoid overlapping regions of data collection.

2.3. Object pose estimation

Object pose estimation is a broad field of research in computer vision and robotics with approaches broadly categorised depending on using 2D data (RGB images) or 3D data (RGB-D images or point clouds) as input (He et al., 2020). Here, we review point cloud based approaches for object pose estimation as they are relevant to our work. Typically, with point cloud-based approaches, a point cloud that corresponds to the object CAD model is registered or matched with the sensor acquired point cloud from the scene and the output of the registration process yields the 6 DoF pose of the object. Iterative closest point (ICP) and its variants are popular point-cloud based approaches for pose estimation (Pomerleau et al., 2013). They fall into the category of simultaneous correspondence estimation and pose estimation methods wherein there is an iterative alternation between estimation of the closest point in the target point set and minimisation of the distance between the corresponding points (Besl and McKay, 1992). However, such methods are local approaches and require good initialisation as they tend to converge to local minima. In contrast, other approaches rely upon finding dense point-to-point corresponding using feature extraction and then optimise for 6D pose (Gentner et al., 2023; Huang et al., 2021b; Rusu et al., 2009; Yang et al., 2020). Choukroun et al. (2006) devised a Bayesian filtering approach for only the rotation estimation (so-called Wahba’s problem) between two coordinate systems. Whereas in this work, we tackle the problem of full SE (3) pose and scale estimation by converting the non-linear problem of pose estimation into decoupled rotation and translation estimation by exploiting the geometry of the measured point clouds. Recently, deep learning based approaches have been used to learn robust features for generating correspondences followed by an optimization such as RANSAC (Deng et al., 2018; Zeng et al., 2017). Deep learning approaches have also been used to regress the pose directly using an end-to-end approach. This is done by learning to regress the pose parameters directly from the features of the input point clouds (Huang et al., 2021a; Pais et al., 2020; Yang et al., 2019).

Contrary to instance-based methods, recent works have addressed the pose estimation of unknown objects from known object categories without any prior instance-specific CAD models available which is known as category-level object pose estimation. Wang et al. (2019) introduced the problem of category-level pose estimation and presented the Normalised Object Coordinate Space (NOCS) that produces a shared canonical representation for all object instances in each category. The predicted NOCS map is used to extract the pose and shape of objects with the observed depth map. Lee et al. (2021) extended the NOCS map with a CNN-based category level pose estimation with RGB images with little or no depth information. Similarly, other works have used variational auto-encoders (VAE) for generating the canonical 3D point clouds and the pose is regressed using another deep neural network (Chen et al., 2020). Some works explicitly model the intra-class shape variations using deformation from pre-learned shape priors (Tian et al., 2020). In addition to pose estimation, Deng et al. (2022) combined their category-level auto-encoder with a particle filter framework for tracking of unknown objects in an iterative manner. The method relies upon accurate depth estimation and semantic segmentation as input. Similarly, Wen and Bekris (2021) performed 6D pose tracking for unknown objects using learnt networks for segmentation and keypoint extraction and pose graph optimisation for pose tracking. As the accuracy of category-level estimation is far from satisfactory in comparison to instance-level methods, some works perform iterative point cloud pose refinement after finding the categorical shape prior (Liu et al., 2022b).

3.1. Problem formulation and framework

The objective is to accurately identify the rotation R ∈ SO (3), position t ∈ R 3 and scale S ∈ R 3 of an unknown target object in dense clutter by sharing the scene perception between visual and tactile sensing. The target object belongs to one of the N c O known categories of the objects and can be opaque or transparent. However, no knowledge of the object model is assumed a priori. Firstly, in order to identify the individual rotations and positions of objects, the robots autonomously and deterministically declutter the scene through interactive perception (Figure 2(a)). Secondly, vision and tactile sensing are used by the robots to extract a shared scene representation and explore the unknown object for reconstruction (Figure 2(b)). Finally, the reconstructed object model is used for pose estimation using our S-TIQF algorithm with sensor acquired point cloud (Figure 2(c)). Furthermore, if there is a discrepancy between the visual and tactile point cloud data, it is typically due to incorrect hand-eye calibration and a novel in-situ visuo-tactile hand-eye calibration solution employing again our S-TIQF algorithm is also presented (Figure 2bi). The notations used in this article have been detailed in Table 1.OPEN IN VIEWER

OPEN IN VIEWER

Table 1.

List of Notations.

List of Notations
R	Unknown rotation matrix ∈ SO(3)
t	Unknown translation ∈ R 3
S	Unknown scale ∈ R 3
G	Scene graph with vertices V & edges E
O T	Target object
μ q	Grasp threshold
μ o	Overlap threshold
d ij	Minimum distance between contours i, j
q k	Grasp quality measure
A H B	Homogeneous transform transforming a point from frame B to frame A, ∈ R 4 × 4
A	Set of possible actions
G res	Occupancy grid with resolution {res}
z	Sensor measurement
D(a)	Energy cost of action a
P	Point Cloud
R = {r1, r2…r n }	Set of rays for ray-casting
p(b)	Probability of input b
q ˜	Quaternion
ρ	Uncertainty in point correspondences
K t	Kalman gain at time t
∑	Covariance matrix
ζ	Simulated annealing cooling rate
N C o	Object Category

3.2. Visuo-tactile-based interactive scene decluttering

Before being able to estimate the pose of individual objects using our proposed novel S-TIQF algorithm, we may need to declutter a possibly cluttered scene. As objects may be present in random configurations in the scene, a method and formalism are necessary to encode the spatial and support relationships between the objects. We encode such relationships in the form of a scene graph termed declutter graph and has been presented in our previous work (Murali et al., 2022b). We briefly describe it in this section for the sake of completeness and readability. The decluttering process is shown in Figure 2(a).

The declutter graph is a directed graph G = ( V , E ) where the vertices of the graph V represent the objects in the scene and the edges E from v i ∈ V to v j ∈ V encodes explicitly the actions needed for decluttering or singulating object v _j from object v _i . Implicitly, the edges represent the spatial and support relationships between the objects in V . The root node of the graph represents the target unknown object O _T which we seek to identify the shape and pose. The declutter graph is constructed based on outputs from a semantic segmentation network and grasp affordance network as shown in Figure 2(a). An RGB image and depth image are taken as inputs for the semantic segmentation network and grasp affordance network, respectively. We use an off-the-shelf semantic segmentation network (Chen et al., 2017) and grasp affordance network (Morrison et al., 2020). We fine-tuned the pre-trained models with our real-world object datasets in clutter and their respective segmentation masks. The output of the semantic segmentation network M s e g provides the various objects present in the scene O k ∈ O . We assume that there is exactly one instance of the target object in the scene and the clutter objects belong to other categories dissimilar to the target object for the sake of decluttering. The edges of the graph are extracted through the overlap or proximity metric from M s e g defined as follows.

Definition 1

Overlap Metric: Two objects representing the vertices of the graph v _i , v _j constitute an edge e i j ∈ E if the overlap measure is greater than the threshold μ _o . The overlap measure is defined as the Intersection over Union (IoU) value, that is, I o U i j = ( C i ∩ C j ) / ( C i ∪ C j ) where C defines all points in the minimum area bounding box of the respective object masks.

Definition 2

Proximity Metric: Two objects representing vertices of the graph v _i , v _j constitute an edge e i j ∈ E if the proximity measure is less than the threshold μ _d . The proximity measure d _ij is defined as the shortest distance between the two contours of the object masks.

Thus, an edge e i j ∈ E is given by

e i j = I o U i j ( I o U i j > μ o ) 1 / d i j ( d i j < μ d ) ∧ ( I o U i j ≤ μ o ) 0 otherwise

(1)

Each edge e ∈ E also has an action attribute attached with them. The grasp affordance network provides a grasp action a ^grasp and a grasp quality measure q _k as output which defines the edge attribute. For an edge e _ij directed from v _i to v _j , the edge e _ij is attributed with a grasp or a push action a ^push for object v _j based on a grasp threshold μ _q as:

a k = a k g r a s p q k ≥ μ q a k p u s h q k < μ q

(2)

3.3. Shared visuo-tactile-based active object reconstruction

As the shape of the target object is unknown, reconstruction of the object is necessary for pose estimation and other possible downstream tasks such as precise manipulation. Our framework autonomously chooses (a) which sensor to use, (b) where to perform sensing and (c) how much of the object information is necessary for the chosen objective of pose estimation. In a single sensor scenario, the next best action selection problem seeks to find the optimal next sensory action to perform based on current knowledge of the environment in order to maximise the information gain that is calculated through an objective function. In a multi-agent and multi-sensor scenario, there is additionally the sensor selection problem which seeks to find the optimal sensor to employ given the current knowledge of the environment and incentivises the coordination between the agents as well as reducing the redundant data collection. In our case, the two robots equipped with a visual RGB-D sensor and tactile sensor array, respectively, as shown in Figure 1 are tasked to reconstruct the object in a coordinated and time-efficient manner.

3.3.1. Vision and tactile action sampling

For the Next-Best-View (NBV) and Next-Best-Touch (NBT) selection, we perform Monte-Carlo sampling of the visual and tactile actions, respectively, around the target object. The centroid o _centroid of the target object is extracted from the semantic segmentation mask.

For NBV sampling, N _nbv viewpoints are sampled on the hemisphere space centred on o _centroid of the target object. The radius of the hemisphere is determined empirically considering the maximum reach of the robot and any possible vertical offset between o _centroid and the robot base frame. The Panda robot has a maximum kinematic reach of 855 mm and we set the radius to a nominal value of 550 mm to avoid singularities at the kinematic extremity. A viewpoint a v ∈ A v is defined by the position p v ∈ R 3 and orientation R ^v ∈ SO(3) of the camera frame expressed in the world coordinate frame ^W H _C . The constraint on a ^v is that the camera orientation must be towards the object of interest that is, the Z-axis of the camera frame which points outward from the camera needs to pass through the centroid of the target object. The position p ^v is randomly sampled as a point on the hemisphere (Marsaglia, 1972). The points which are sampled outside the kinematic limits of the robot are discarded. The rotation matrix R ^v is calculated through the angle-axis formulation { e ^ , θ } as follows:

h ^ = p v − o c e n t r o i d ‖ p v − o c e n t r o i d ‖ ,

(3)

θ = cos − 1 ( h ^ ⋅ Z ^ ) , e ^ = h ^ × Z ^ ‖ h ^ × Z ^ ‖ .

(4)

where Z ^ = { 0,0,1 } the Z-axis of the world coordinate frame W. The rotation matrix R ^v is calculated from the angle-axis formulation using the Rodrigues’ formula.

For the NBT sampling, we define the tactile action as a t ∈ A t by the position p t ∈ R 3 and direction d ^ . The positions are randomly sampled as points on each face of the oriented bounding box of the object except the bottom face where the object rests on the table. The bounding box is determined from the 2D semantic segmentation mask of the target object and a predefined height. The direction d ^ of each p ^t is calculated as the normal perpendicular to the face of the bounding box. The NBV and NBT sampling are graphically shown in Figure 3.OPEN IN VIEWER

Figure 3.

Next best view (NBV) and next best touch (NBT) action selection.

3.3.2. Active sensor selection and next best action selection

At each iteration, the next best action a* is selected from the set A = A v ⋃ A t using the joint information gain approach. We discretise the space around the target object into a 3D voxelised probabilistic occupancy grid G with resolution G _res . Each grid cell g _i ∈ G is represented by a Bernoulli random variable X _g which represents the probability if the grid cell is occupied X _g = 1 or unoccupied X _g = 0. We assume these Bernoulli random variables are independent which allows the calculation of the probabilities of the occupancy grid p(g _i ). The confidence of the reconstruction can be calculated as the uncertainty of the grid through the Shannon Entropy as:

H ( G ) = − ∑ g i ∈ G p ( g i ) l o g ( p ( g i ) ) + ( 1 − p ( g i ) ) l o g ( 1 − p ( g i ) )

(5)

Given a point cloud captured by the camera or tactile sensor, the occupancy grid is updated with probabilities using the respective sensor models (Hornung et al., 2013). We define a virtual sensor measurement model for visual and tactile sensors respectively for the NBV and NBT calculations. The visual sensor generates the point cloud using a time-of-flight (ToF) sensor. The virtual vision sensor model is defined by a set of beam measurements R v = r 1 , r 2 , r 3 , … r n v where r n v refers to the maximum number of rays. These rays are rotated such that they span the field-of-view of the sensor. Similarly, we define a virtual tactile sensor which casts a set of rays R _t = r₁, r₂, r₃, …r _taxel where r _taxel refers to the number of taxels in the sensor array. Raytracing is used to update the grid cells, wherein the grid cells where the ray terminates is updated as hits and remaining grid cells are updated as misses. Given the observed grid cell g and the measurement from sensor observation z, the log-odds is updated as L(g|z) = L(g) + l (z) wherein L(g) = log p(g)/1 − p(g) and

l ( z ) = l o g p h 1 − p h z = ^ hit l o g p m 1 − p m z = ^ miss

(6)

where p _h and p _m are the probabilities of hit and miss which are user-defined values set to 0.7 and 0.4, respectively, as in Hornung et al. (2013). The posterior probability p(g|z) can be computed by inverting L(g|z).

For a single sensor case, the expected information gain by taking an action a k ∈ A and corresponding expected measurement z ^ t is given by the Kullback–Leibler (KL) divergence between the posterior entropy after integrating the expected measurements and the prior entropy:

E [ I ( p ( g i | a t , z ^ t ) ) ] = H ( p ( g i ) ) − H ( p ( g i | a t , z ^ t ) )

(7)

Hence, the action a* that maximises the information gain can be selected as the next best action:

a * = arg max a k ∈ A ( E [ I ( p ( g i | a k , z ^ k ) ) ] ) .

(8)

A näive way of extending to multiple sensors is to compute eq. (8) for each sensor. However, this would result in collection of redundant data due to sensor data overlap. Furthermore, there would be no incentive for coordination between the robots as well as leveraging the vision and tactile sensors with complementary properties. Hence, we present a joint sensor selection and action selection method for vision and tactile sensors. We can utilise the same occupancy grid formulation for integrating the sensor information from vision and tactile sensors. For each cell of the occupancy grid, the probabilistic evidence from each sensor needs to be updated. We can define an energy cost D (a _t ) that encodes the time taken to perform the robot action. In general, performing visual actions is faster than performing tactile actions with the robot. Hence, we set D ( a t v ) < D ( a t t ) for all cases unless the target object is transparent. In case of objects that are transparent, the visual sensor produces erroneous data and we set D ( a t t ) < D ( a t v ) , thus preferring the tactile actions in such cases. Our method for detection of transparent objects is described in following subsection. Hence the optimization for the sensor selection and next best action is performed by:

a * = arg max a k ∈ A E [ I ( p ( g i | a k , z ^ k ) ) ] D ( a k ) .

(9)

3.3.4. Category-level object shape reconstruction

In order to recognize the shape of category-level objects, we present a self-supervised learning approach with an auto-encoder network that aims to reconstruct the original point cloud when provided a subsampled point cloud. The network is trained on only synthetic object models belonging to the same category but not identical as the real-world objects. We generate a dataset D of synthetic point clouds from synthetic CAD models available in the ShapeNet repository (Chang et al., 2015). The trained network is directly used with visual and tactile point clouds from real-world objects. This avoids expensive real-world data collection and annotation process. The network consists of a feature-extraction encoder and upsampling decoder unit as shown in Figure 5. The input point clouds P i n are randomly sampled with different sampling factors to produce point clouds with point numbers between 60 and 1024. The output reconstructed point P o u t from the network is fixed to 2048 points. This is done to emulate tactile-only and visuo-tactile point clouds which are sparse and dense, respectively. The low density of the input point clouds provides a challenge for shape reconstruction as other simpler techniques such as interpolation cannot be used.OPEN IN VIEWER

Figure 5.

Architecture for the reconstruction network.

3.3.4.1. Feature-extraction encoder architecture

The encoder creates a high dimensional feature vector from a possibly sub-sampled point cloud as its input. This feature vector encodes the input point cloud’s overall geometric shape data. For the encoder, we employ a modified PointNet architecture (Qi et al., 2017). The encoder network produces a feature vector of 1024 dimension by selecting the informative and distinctive parts of the point cloud. The encoder consists of [1 × 1] convolutions with output channels size (64, 64, 128, 1024) with the first convolutional layer with kernel size [1 × 3] to encode the input point cloud of N × 3 dimension. The convolution layers are aggregated by a max-pooling layer. Furthermore, we add a self-attention layer (Zhang et al., 2019) whose outputs are aggregated with the max-pooled features to provide the global feature vector. The self-attention mechanism allows the model to weigh the importance of each point with respect to the other points in the input point cloud.

3.3.4.2. Upsampling decoder architecture

The upsampling decoder enlarges the feature vector input to generate a more detailed output point cloud P o u t . The decoder is made up of a fully connected layer with an output dimension of 1024, as well as five deconvolutional or transpose convolutional layers. These layers have specific kernel sizes and output channels, as depicted in Figure 5. This decoder produces a point cloud output with 2048 points, which is dense enough to effectively reconstruct the original point cloud.

3.3.4.3. Loss function

We utilize the Chamfer distance (Borgefors, 1986) as the loss function which ensures the reconstructed point cloud follows the 3D shape of the ground truth point cloud. Given the input point cloud to our network prior to subsampling P i n and the reconstructed output point cloud P o u t , the loss is defined as:

L C D ( P i n , P o u t ) = 1 | P i n | ∑ p 1 ∈ P i n min p 2 ∈ P o u t ‖ p 1 − p 2 ‖ 2 + 1 | P o u t | ∑ p 2 ∈ P o u t min p 1 ∈ P i n ‖ p 2 − p 1 ‖ 2 ,

(10)

where |•| refers to the cardinality (number of points) of the point cloud and ‖•‖₂ refers to the L2 norm. The loss L C D represents the average distance between the pair-wise closest points in the two point clouds. Hence, the overall loss function is provided as L r e c = α L C D wherein α is the weight which is set empirically during training.

3.4. Visuo-tactile-based robust pose estimation

3.4.1. Stochastic translation-invariant quaternion filter (S-TIQF)

Upon reconstructing the object point cloud, we are able to perform pose estimation which is to compute the unknown scale S ∈ R 3 , rotation R ∈ SO(3), and translation t ∈ R 3 . Given the two point sets ( S and O ) with point-to-point correspondences, the point cloud registration problem is formulated as:

s i = S . ( R o i ) + t i = 1 , … N ,

(11)

where s i ∈ R 3 are points in the point cloud S and o i ∈ R 3 are the corresponding points belonging to the point cloud O and. represents the element-wise product. Typically, for object pose estimation, the point cloud S is derived from the scene with sensor measurements and the point cloud O is derived from the object model. In our case, we use the reconstructed object model from the Section 3.3.4 as O .

3.4.1.1. Scale estimation

The reconstructed object point cloud O is uniformly scaled within a [0,1]³ cube. To find the absolute scale S, we compute the ratio of the axis aligned bounding box (AABB) of the scene S and object O point clouds, that is, if {(x _min , x _max ), (y _min , y _max ), (z _min , z _max )} represents the AABB for a point cloud, then:

S = | x m a x − x m i n | S | x m a x − x m i n | O , | y m a x − y m i n | S | y m a x − y m i n | O , | z m a x − z m i n | S | z m a x − z m i n | O

(12)

Subsequently, we can scale the object point cloud O by S and we are left with the estimation of R and t. Eq. (11) simplifies to s i = R o i ′ + t where o i ′ = S o i , for i = 1, 2, …N.

3.4.1.2. Stochastic initial alignment

We presented our Translation-Invariant Quaternion Filter (TIQF) in our prior work (Murali et al., 2021) which is a Bayesian filtering approach for point cloud registration applicable for dense visual and sparse tactile point clouds. However, TIQF is sensitive to initialisation conditions. Figure 6 shows an example error surface for point cloud registration with TIQF using the Stanford Bunny dataset (Levoy et al., 2005). It is obtained by varying the initial position about one axis in the range of [−5.0, 5.0] and initial orientation about one axis in the range of [−π, π]. The error is calculated as the root mean squared error of the distance metric between corresponding points. We note from Figure 6 that the error surface contains multiple local minima in which the optimization can be trapped depending upon the initial conditions. We solve this problem with a stochastic initialization method for TIQF that is robust against local optima termed StochasTIQF (S-TIQF).OPEN IN VIEWER

Figure 6.

Error surface calculated as the distance between corresponding points of two clouds upon performing TIQF with initialisation parameters (translation and rotation) varied (best viewed on-screen and in colour).

The stochastic initial alignment is performed through Simulated Annealing (Bertsimas and Tsitsiklis, 1993). Simulated Annealing (SA) is a well-known stochastic probabilistic method for approximating the global optima for a given function f (b). In SA, a temperature variable is used to guide the exploration. An initial temperature t = t₀ is chosen and the temperature is reduced in each iteration according to the geometric cooling rate, that is, t′ = tζ where ζ is the cooling rate. This is termed as the annealing schedule. At t = t₀, an initial state b = b₀ is chosen at random and the cost is computed using the cost function c₀ ← f (b₀). At every iteration, a random state in the neighbourhood of the current state is chosen and the difference in cost Δc is calculated. The probability of accepting the new state is provided by the following condition:

p ( b ′ ) = 1 Δ c ≤ 0 e − Δ c t Δ c > 0

(13)

The new state b′ is accepted if p(b′) > random (0, 1). The process is repeated until a pre-defined temperature threshold is reached t < t _min or for a fixed number of iterations. Random restarts are also used wherein t is set to t₀ when t ≤ t _min . In our experiments, random restarts are performed 10 times. In order to use simulated annealing with TIQF, a cost function for SA needs to be designed that upon finding the solution provides a good initialization for TIQF to extract the rotation and translation estimates. The cost is defined as the root mean squared error of the nearest neighbour point-to-point distances for each state b = {R, t}. The nearest neighbourhood correspondence assignment allows fast computation of the costs and thereby allowing larger iterations of SA. The state with minimal cost naturally minimizes the distance between the two point clouds. Hence for two point sets S and O , the cost is defined as follows:

f ( b ) = 1 | O | ∑ ‖ min ( S − ( R S A O + t S A ) ) ‖ 2

(14)

The temperature variable allows exploration in the initial phase thereby escaping the local minima and gradually converges to an optimal solution. The estimated rotation R _SA and translation t _SA are used for TIQF to find the accurate solution.

3.4.1.3. Correspondence estimation

A crucial factor in point cloud registration from eq. (11) is the knowledge of point correspondences in the two point sets. In realistic scenarios, the point correspondences are not known a priori. On the one hand, simultaneous pose and correspondence estimation methods such as ICP and its variants rely upon nearest neighbour search for extracting point correspondences while iteratively improving the pose in successive steps (Besl and McKay, 1992). On the other hand, correspondence based methods extract point correspondences through feature matching and may employ rejection techniques to remove outlier correspondences prior to performing registration. In the case of visual and tactile point clouds, there are further challenges: (a) the point density difference between visual and tactile point clouds and (b) visual point clouds can be captured in one shot whereas tactile point cloud is aggregated through sequential tactile actions. Due to the point sparsity, typical feature-based correspondence matching algorithms are not accurate as they depend on local surface information. Similarly, nearest neighbour search as used in ICP is not robust to outliers and can get stuck in local minima.

We use the mutual nearest neighbours or Best-Buddies Pairs (BBP) (Oron et al., 2017) to estimate the point correspondences. It has been shown in Oron et al. (2017) that the BBP measure is robust to outliers and differences in point density but in the context of template matching in the image domain. The point p i ∈ P and q j ∈ Q are Best Buddy Pairs (BBP) if p _i is the nearest neighbour of q _j in point cloud Q and q _j is the nearest neighbour of p _i in point cloud P . Mathematically, it can be written as:

b b p ( p , q , P , Q ) = 1 { N N ( p i , Q ) = q j } ∧ { N N ( q i , Q ) = p j } 0 otherwise

(15)

where N N ( p i , Q ) = arg min q ∈ Q d ( p i , q ) and d(p _i , q) is a distance measure. Typically, the nearest neighbours can be calculated based on Euclidean distances as d ( p i , q ) = ‖ q − ( R ^ p i + t ^ ) ‖ where R ^ and t ^ are the current rotation and translation estimates respectively. Furthermore, we also include the normals in case there are multiple candidates for nearest neighbours. The surface normal for each point in the dense visual point cloud is estimated as the normal of the plane tangent at the local neighbourhood of the point and for sparse tactile point cloud, the surface normals are directly measured from the tactile sensors as our sensors provide three-axis force measurements. The corresponding points p _i and q _j must also have their associated normals n p i and n q j oriented approximately in similar directions, that is, the NN (p _i , q _j ) = 1 if arccos ( n q j . R ^ n p j ) is less than an user-defined threshold.

3.4.1.4. Rotation estimation

The estimation of rotation and translation is decoupled and performed in consecutive steps. The decoupling is done by computing the relative vectors between pairs of corresponding points as s _ji = s _j − s _i and o j i ′ = o j ′ − o i ′ . Eq. (11) is simplified as:

s j − s i = ( R o j ′ + t ) − ( R o i ′ + t ) ,

(16)

s j i = R o j i ′ .

(17)

Eq. (17) is independent of translation t, hence these measurements are termed as translation-invariant measurements. Next, we cast the rotation estimation problem into the Bayesian estimation framework. We denote the rotation estimate R ^ in its quaternion form as the state x which needs to be estimated by measurements z obtained via actions a upto time t. As the pose of the target object remains unchanged, it is regarded as a static Bayesian network. Hence, the state estimate is provided by a recursive Bayes filter as:

p ( x | z 1 : t , a 1 : t ) = η p ( z t | x , a t ) p ( x | z 1 : t − 1 , a 1 : t − 1 ) ,

(18)

where η is a normalization constant.

We estimate the current belief p(x|z_1:t, a_1:t) through a Kalman filter. Herein, we demonstrate the formal derivation of the linear state and measurement model and the associated noise models. Eq. (17) can be reformulated using quaternions algebra as:

s ˜ j i = x ⊙ o ˜ j i ⊙ x * ,

(19)

where ⊙ is the quaternion product, x* is the conjugate of x, s ˜ j i = { 0 , s j i } and o ˜ j i = { 0 , o j i } . As x is an unit quaternion, and using the fact that x* ⊙ x = x ⊙ x* = 1 to get:

s ˜ j i ⊙ x − x ⊙ o ˜ j i = 0 .

(20)

Definition 3

The quaternion multiplication can be reformulated in matrix form as:

q ˜ 1 ⊙ q ˜ 2 = q 0 1 − q 1 T q 1 q 1 × + q 0 1 I 3 q 2 ,

(21)

= q 0 2 − q 2 T q 2 − q 2 × + q 0 2 I 3 q 1 ,

(22)

where [v]^× is the skew-symmetric matrix formed from the vector v.

Hence, we can reformulate eq. (20) as:

0 − s j i T s j i s j i × x − 0 − o j i T o j i − o j i × x = 0

(23)

0 − ( s j i − o j i ) T ( s j i − o j i ) ( s j + s i + o j + o i ) × 4 × 4 x = 0 .

(24)

Eq. (24) is of the form Hx = 0 where H is the pseudo-measurement matrix such that

H = 0 − ( s j i − o j i ) T ( s j i − o j i ) ( s j + s i + o j + o i ) × ∈ R 4 × 4

(25)

The pseudo-measurement matrix H depends only on the measurements points s _ji and the model points o _ji . Furthermore, it can be inferred that, in the no noise case, the true state x must lie in the nullspace of H.

Eq. (24) can be reformulated as a pseudo-measurement model as:

H x = z h ,

(26)

and enforcing the pseudo-measurements z ^h = 0. For each time-step t, we obtain H _t based on newly obtained measurement points ( s ji ) t and transformed model points ( o ji ) t . If z _t represents the measurements vector, and v _t represents the measurement noise at time t, then:

z t = z t h + v t .

(27)

Using eq. (26), we can reformulate eq. (27) as follows:

H t x t = z h + v t .

(28)

0 = H t x t − v t .

(29)

Eq. (29) represents a linear equation in the state x _t with the state-dependent noise term v _t . The measurement noise v _t is considered as zero mean noise with covariance P t v . The exact expressions for the state-dependent measurement noise are defined as follows. Let P t x represent the covariance matrix of state x _t and P t v the covariance matrix of the measurement noise term v. We adopt the equation for the covariance of the measurement noise similar to Choukroun et al. (2006) which is motivated from a fundamental proposition from Stochastic Filtering Theory (Jazwinski, 1970) [Chap 3, Pg. 90] as:

P t v = 1 4 ρ t r ( x ^ t − 1 x ^ t − 1 T + P t − 1 x ) I 4 − ( x ^ t − 1 x ^ t − 1 T + P t − 1 x ) ,

(30)

where ρ is a constant which corresponds to the uncertainty of the correspondence measurements, tr refers to trace and x ^ t refers to the mean of x _t at time t. The process model is given as x _t = x_t−1 as it represents the time-invariant rotation estimate of the object.

Hence, we can define the Kalman filter equations as follows:

x t = x t − 1 − K t H t x t − 1

(31)

P t x = I − K t H t P t − 1 x

(32)

K t = P t − 1 x H t T H t P t − 1 x H t T + P t v − 1 ,

(33)

where x_t−1 is the state estimate at t − 1, K _t is the Kalman gain and P t − 1 x is the covariance matrix of the state at t − 1.

We must note that the Kalman filter does not preserve the constraints on the state-variables such as the unit-norm property of the quaternion. Hence, a common technique is to normalise the state and the associated covariance matrix after each update:

x t = x t ‖ x t ‖ 2 , P t x = P t x ‖ x t ‖ 2 2 .

(34)

The rotation estimate x (quaternion) can be converted to the rotation matrix form R ∈ SO (3) and inserted into eq. (11).

3.4.1.5. Translation estimation

Once the rotation estimate R is found, the translation estimate t is computed in closed form:

t = 1 N ∑ i = 0 N ( s i − R o i ′ ) .

(35)

Thus, with each iteration of the S-TIQF we obtain a new rotation and translation estimate that is used to transform the model. The transformed model is used to recompute correspondences and repeat the TIQF update steps. We calculate the change in homogeneous transformation between iterations Δ_S−TIQF < ξ ^conv , that is, if the difference in the output pose is less than a specified threshold which in our experiments is 0.1 mm and 0.1°, respectively, and/or maximum number of iterations in order to check for convergence (max_it_S−TIQF = 100). The pseudo-code of the S-TIQF algorithm is shown in Algorithm 2.

3.5. Visuo-tactile hand-eye calibration

As described in our framework in Figure 2bi, if there is a discrepancy between visual and tactile point clouds for a static object, it is typically due to incorrect hand-eye calibration (c.f., Figure 18(a)). Conventionally, the hand-eye calibration is performed using a specialized target such as a calibration grid as shown in Figure 7(a). However, this process is time-consuming as it adds additional overhead such as specialized targets and calibration procedures. Furthermore, the grid-based calibration technique may contain residual errors as it is dependent on the chosen robot end-effector poses, lighting conditions, and sensor noise. Recent works have introduced deep-learning based markerless hand-eye calibration methods using segmentation and differentiable rendering techniques to regress the camera-to-robot pose based on input images of the robot and associated joint kinematics (Labbé et al., 2021; Lu et al., 2023). While the disadvantages of solely relying upon visual images such as occlusions, challenging backgrounds for segmentation, and lighting conditions still apply, there is an additional overhead of training requirement for various types and kinematic configurations of the robots. These methods are typically used to compute the camera-to-robot (eye-to-hand) transform and they need to be extended for eye-in-hand cases wherein the camera is attached to the end-effector of the robot as in our scenario. In this work, we relax the need for a specific calibration artifact or target and demonstrate how to perform hand-eye calibration using any known object present in the workspace of the robot, hence termed as in situ calibration. Furthermore, by combining visual and tactile perception, we effectively provide grounding to the estimation and correct the visual estimate with sparse tactile data for improving hand-eye calibration.OPEN IN VIEWER

Consider the two-manipulator system shown in Figure 7(b). We cast the hand-eye calibration problem of finding H C E P as a point cloud registration problem given any arbitrary object. The camera frame C can be expressed in the world coordinate frame W as follows:

H C W = H B P W H E P B P H C E P ,

(36)

H C E P = H E P W − 1 H C W ,

(37)

Let’s denote H ^ C W as the estimated ^W H _C . We can estimate ^W H _C through point-set registration that minimises the following cost function:

f ( H ^ C W , P i v C , P i t W ) = 1 N ∑ i = 1 N ‖ H ^ C W P i v C − P i t W ‖ 2 ,

(38)

P i t W = H B u W H T B u P t T ,

(39)

As the tactile data is high fidelity, we aim to register the dense visual point cloud ^C P ^v to the sparse tactile point cloud ^W P ^t using our S-TIQF algorithm as detailed in Section 3.4.1. Note that any point cloud registration method can be used but as we demonstrate in Section 4, state-of-the-art point cloud registration methods perform poorly in dense-sparse registration whereas our S-TIQF approach shows high accuracy even with low number of points. The S-TIQF algorithm produces the homogeneous transform H ^ C W as output. Plugging the H ^ C W value into eq. (37), we can obtain the required hand-eye calibration matrix H C E P .

4.1. Experimental setup

The experimental setup shown in Figure 1 consists of a Universal Robots UR5 robot with a tactile sensorised Robotiq 2F140 Gripper and Franka Emika Panda robot with the standard Panda Gripper. The tactile sensor array of the two-finger gripper is acquired from XELA robotics^© and Contactile^©. The Contactile sensors embedded on one finger on the outer and inner-side comprises of 3 × 3 tactile array. The XELA sensors embedded on the other finger comprises of 6 × 4 array on the outer side and 4 × 4 array on the inner side of the finger. The fingertip of the finger sensorised with the XELA sensors also has 6 × 1 array to touch objects from the top. Each taxel of both types of sensor arrays provides three-axis force measurements. This configuration allows the robot to acquire tactile data while touching with the outer side and from the fingertip. We intentionally used two different types of tactile sensors that are based on different operating principles in order to show that our framework is agnostic to tactile sensing technology. The normalised force values of the tactile sensors are measured and contact is established when the force exceeds the baseline threshold f _ts ≥ τ _f where τ _f = 1.1. The contact points P o b s t expressed in the common world frame W are added to the tactile point cloud P ^t after every action. An Azure Kinect DK^©RGB-D camera is rigidly attached to the Panda Gripper with a custom-designed flange which provides the vision point cloud P ^v . Hand-eye calibration is performed to find the transformation between the Panda Gripper and the camera frame and consequently transformed into the common world coordinate frame. All operations involving point clouds uses the Point Cloud Library (PCL) (Rusu and Cousins, 2011), occupancy grid computations uses OctoMap library (Hornung et al., 2013), and the overall setup uses a ROS-based framework. All robot experiments are run on a workstation using Ubuntu 18.04 with Intel^©Xeon Gold 5222 CPU. Our reconstruction network is implemented using the TensorFlow framework and training/inference is performed on Nvidia Quadro RTX 4000 GPU. The maximum allowed speeds for the UR5 and Panda were 75 mm/s and 100 mm/s, respectively, due to safety constraints. Network implementation details: we used the ADAM optimiser, learning rate set to 10⁻⁴, momentum 0.9 and batch size 8. All layers of the encoder-decoder uses batch normalisation and the decay rate initialized at 0.5 and gradually increased to 0.99 with decay step size 2 × 10⁵. During training with our synthetic dataset D , random voxel-grid subsampling is done to have input point clouds with point size between 60 and 1024 in order to account for sparse tactile point clouds and dense visual point clouds. The hyperparameter α for the reconstruction loss is set to 100. The ground-truth point cloud and mesh of all objects were obtained using a scanning device. The output point cloud size for the network is set to 2048 points.

4.2. Active visuo-tactile-based target object reconstruction

As the target objects list contains both transparent and opaque objects, our framework automatically prefers tactile exploration for transparent objects and visuo-tactile exploration for opaque objects using the joint criteria defined in eq. (9). An exemplary case that demonstrates the benefit of our method is shown in Figure 8(c). The ketchup bottle has parts of opaque and reflective regions. The vision point cloud shown in red captures the overall shape but contains some missing points in the reflective region (highlighted in the green box). Due to our information gain method for object exploration, tactile acquisitions are performed only in the regions where the visual points are missing or there is uncertainty due to noisy data (around the edges). Similarly for transparent object shown in Figure 8(d), the visual data is completely missing (shown in red points) and the tactile data captures the overall shape of the object (shown in blue points).

For reconstruction evaluation, we use the Chamfer distance (CD) metric defined in eq. (10) with the ground truth point clouds shown in Figure 10. The chamfer distance (CD) is defined as the sum of the average nearest-neighbour distance between one point cloud and the other and vice versa. Lower CD value denotes higher reconstruction precision. For the ideal case where the reconstructed point cloud exactly matches the ground truth, CD ≈ 0.0 m. Qualitatively, through empirical analysis, we found that CD < 0.01 m denotes accurate reconstruction, 0.01 m < CD < 0.1 m denotes good reconstruction while CD > 0.2 m implies poor reconstruction of the point cloud. The qualitative results for the reconstruction with vision and tactile data with our network are shown in Figure 9. Our method, with the help of the learned model over the category-level synthetic objects, is able to reconstruct the object even with sparse input point clouds. We performed five repeated experimental trials for each target object and with each exploration strategy: active, random and uniform, resulting in 210 total trails (14 objects, 3 strategies, 5 repetitions). We note that for opaque objects, the shared visual and tactile data result in a higher accuracy of reconstruction (CD < 2 cm) as seen in Figure 10(a). The visual point clouds for opaque objects capture all the sides of the objects due to the active visual exploration. On average, combining with tactile data improves visual reconstruction accuracy by 17%. For opaque objects, the tactile reconstruction accuracy is relatively worse due to the fact that incomplete tactile point clouds are collected as the robot only explores regions unseen by the camera.OPEN IN VIEWER

OPEN IN VIEWER

Figure 10.

Quantitative reconstruction results showing the chamfer distance (CD) metric of the reconstructed point cloud compared with the ground-truth point cloud for (a) opaque objects and (b) transparent objects. The bar graph represents the average values and the error bars represent the standard deviation.

For transparent objects, even a sparse input point cloud provides acceptable reconstruction accuracies as seen in Figure 10(b). In this case, since no visual point cloud is available, the robot explores the object with only tactile sensing in an information-gain seeking strategy. All objects are accurately reconstructed (CD < 2 cm) by the network except the wineglass2. The reason for the lower accuracy for wineglass2 is due to its peculiar shape, and it is out of distribution to the training dataset.

Furthermore, we performed a comparison study of our active exploration strategy with baseline random and uniform strategies for both vision and tactile modality. The baseline strategies for tactile exploration are defined as follows: the bounding box on the target object is discretised into a 3D grid with each grid cell of size 3 cm × 3 cm which corresponds to the size of the sensor patch. The robot is moved to touch the grid cell closest to its base frame and sequentially touches each cell in a uniform manner. In contrast, the random strategy involves choosing the next possible grid cell in a randomised manner. In a similar manner, we define random and uniform exploration strategies for the vision modality: viewpoints on the hemisphere sphere are sampled uniformly in the same way as described in Section 3.3 and the robot starts from one extreme possible position and sequentially moves to the next viewpoint in a uniform manner. The random strategy chooses one among the possible sampled viewpoints at random. As we want to compare the sample efficiency of the actions and to have an unbiased comparison with each strategy, we limit to 20 actions for tactile-only actions for transparent objects and to 3 visual actions and 5 tactile actions for the remaining opaque objects. The results for reconstruction of opaque objects with visuo-tactile sensing is shown in Figure 11(a) and transparent objects with tactile sensing in Figure 11(b). As the exploration is performed for the objective of object reconstruction, CD is used as a metric for comparison. For both transparent and opaque objects, increasing the number of exploratory actions reduces the CD value. For transparent objects that rely upon only the sense of touch, active exploration converges to an accuracy of CD ≈ 2 cm within 20 touches. Random exploration converges to CD ≈ 5 cm in 20 touches, but with higher variance due to the stochasticity of exploratory actions. Uniform exploration has the least accuracy due to the fixed nature of exploration, which often collects redundant data, leading to long data collection times. In contrast for opaque objects, random and active strategies perform similarly on average (CD ≈ 1.5 cm) and subsequently active tactile strategy slightly improves the reconstruction accuracy (CD ≈ 1.1 cm). Negligible reduction in CD value is reported with random and uniform tactile actions following visual perception for opaque objects.OPEN IN VIEWER

4.3. Category-level visuo-tactile-based pose estimation

In order to benchmark our stochastic TIQF (S-TIQF) and previously proposed TIQF methods, firstly we perform instance-level pose estimation where the object model point cloud is obtained from the ground truth mesh on the Stanford Scanning Repository benchmark. The following state-of-the-art methods are used for comparison: Iterative Closest Point (ICP) (Besl and McKay, 1992), Sparse iterative closest point (S-ICP) (Bouaziz et al., 2013), Random sample consensus (RANSAC) (Fischler and Bolles, 1981), Truncated least squares Estimation And SEmidefinite Relaxation (TEASER++) (Yang et al., 2020) and PREDATOR (Huang et al., 2021a). We compare with both local registration methods such as ICP and S-ICP, global optimization methods such as RANSAC and TEASER++ and learning methods such as PREDATOR. We chose these popular baselines are they are often used in the literature for the point cloud registration task. Furthermore, some of these baselines such as ICP and RANSAC are also used to perform the final registration task with learning-based methods where the features are learnt using a neural network. We also compare with the learning-based method termed PREDATOR (Huang et al., 2021a) which learns to predict the registration of point clouds with low overlap between each other as is the case with visuo-tactile point clouds. We used the pretrained model of PREDATOR (available with their open-source implementation ¹ ) and set the hyper-parameters as suggested in the paper with the exception of first_subsampling_dl = 0.01, dgcnn_k = 5 for sparse point clouds. Secondly, to demonstrate the flexibility of our method, we use the PhoCal dataset (Wang et al., 2022) to perform a feasibility study for category-level pose estimation using the NOCS-based (Wang et al., 2019) framework. For real robot experiments, we use the reconstructed point cloud of the objects from our reconstruction network as the object point cloud and the acquired vision and/or tactile point cloud as the scene point cloud for category-level pose estimation.

4.3.1. Benchmark experiments

4.3.1.1. Stanford scanning repository benchmark

In order to benchmark our methods against the state-of-the-art, we use the standard point cloud registration benchmark from the Stanford Scanning repository (Levoy et al., 2005). We used six CAD models from the dataset namely bunny, dragon, happy Buddha, Lucy, statue and armadillo (Levoy et al., 2005). In order to have an unbiased comparison of pose estimation, we used the model point cloud derived from the CAD mesh in the dataset. This is done because errors in shape reconstruction can propagate and influence pose estimation. Each model point cloud is sampled uniformly from the CAD mesh to have 1024 points. The scene point cloud is sampled randomly from the CAD mesh and the point numbers are set to 20, 40, 80 and 120 points. The varying degree of sparsity can test the robustness of our approach against state-of-the-art methods. The model and scene point clouds are normalized and scaled to lie within a [−1, 1]³ m cube. In order to evaluate the sensitivity of our method against local optima, the initial pose for the model point cloud is randomly chosen from a position range of [−5.0, 5.0] m and rotation from [−180°, 180°] for each experimental trial. The correspondence estimation for ICP and S-ICP is based on nearest neighbourhood search whereas RANSAC and TEASER++ are based of Fast Point Feature Histograms (FPFH) descriptors (Rusu et al., 2009). For each selected model from the Stanford scanning repository, the experiment is repeated five times with the initial pose randomly varied for each trial. The errors are measured as using the Average Distance of model points with Indistinguishable views metric (ADI) which is insensitive to object symmetries (Hinterstoisser et al., 2013). The ADI metric is measured as:

e r r a d i = 1 | O | ∑ p 1 ∈ O min p 2 ∈ O ‖ ( R g t p 1 + t g t ) − ( R e s t p 2 + t e s t ) ‖ ,

(40)

where (R _gt , t _gt ) and (R _est , t _est ) refers to ground-truth and estimated rotation and translation, respectively, O refers to the object model point cloud and the points p 1 ∈ O and p 2 ∈ O belong to the object point cloud and denote the closest corresponding points when O is transformed by {R _gt , t _gt } and {R _est , t _est }, respectively.

Qualitative results with the Stanford Bunny model are shown in Figure 12. Quantitative results evaluated with all the models selected from the Stanford scanning repository are provided in Figure 13. It can be seen that for all levels of point sparsity (20–120 points), our S-TIQF outperforms baselines (p < .001 for Welch’s t-test in all cases except for scene cloud with 40 points with TIQF where p < .01). Interestingly, S-TIQF also outperforms TIQF method and this is due to the stochastic initial alignment used in S-TIQF. For instance, in the case of the scene point cloud with 20 points, S-TIQF outperforms the closest baseline S-ICP by 45% on average and 38% for scene point cloud with 120 points. The results corroborate the known weaknesses of correspondence-based techniques such as RANSAC and TEASER++ as they rely upon features for estimation correspondences. These correspondences remain fixed throughout the pose estimation process. Due to point sparsity, feature extraction methods such as FPFH fail to generate valid correspondences. Similarly, our S-TIQF and TIQF outperforms the learning-based method PREDATOR by more than 50% on average for all levels of point sparsity (20–120). The point sparsity and absence of neighbourhood points is challenging for the graph neural network in PREDATOR to extract good features for the overlapping regions. Furthermore, simultaneous pose and correspondence methods such as ICP and TIQF perform relatively well on sparse data but rely on good initialization. Our S-TIQF approach removes the need for good initialization through the stochastic search for initial alignment. Furthermore, to demonstrate that the effectiveness of our approach, we performed an ablation study shown in the Appendix A.3 where we provide the output from the stochastic alignment (SA) module to the baseline methods ICP and S-ICP. We show that S-TIQF still outperforms the modified baseline algorithms SA + ICP and SA + S-ICP by at least 20% in terms of ADI error (cf., Table 3).OPEN IN VIEWER

Figure 12.

Qualitative results on the Stanford bunny dataset: The grey mesh represents the model at ground truth for reference, the blue sparse point cloud represents the scene point cloud and the red dense point cloud represents the transformed model point cloud after performing point cloud registration.

OPEN IN VIEWER

Figure 13.

Pose error calculated as ADI error for models from the Stanford scanning repository. The object point cloud consisting of 1024 points is sampled from the models while the scene point cloud is randomly sampled from the model and consists of (a) 20, (b) 40, (c) 80 and (d) 120 points, respectively. p values calculated by Welch’s t-test shown as ∗. The bar plot represents the average and the error bars represents the standard deviation.

4.3.1.2. PhoCal dataset benchmark

We conducted a feasibility study with the PhoCal dataset (Wang et al., 2022) to demonstrate category-level pose estimation with our method. In contrast to our reconstruction network which is trained on point clouds of synthetic objects belonging to the same category as the real-world objects, the Normalized Object Coordinate Space (NOCS)-based framework (Wang et al., 2019) can also be used to generate the point cloud of the objects. Given RGB inputs, the NOCS network learns a NOCS map which is a shared canonical space of objects within a category. The NOCS map can be combined with the depth map to lift from 2D image to 3D point cloud space. This is used as the object point cloud and the point cloud from the depth map is considered as the scene point cloud for point cloud registration. Furthermore, point cloud registration methods such as Umeyama algorithm (Umeyama, 1991) are used to perform pose estimation with the NOCS-based framework (Wang et al., 2019). The PhoCal dataset (Wang et al., 2022) contains the RGB, depth and learnt NOCS maps of real world objects belonging to different categories particularly for photometrically challenging objects. In order to perform accurate 6D pose annotation, the authors of the PhoCal dataset (Wang et al., 2022) used a tool-tip on a robotic manipulator to manually touch the object at various locations that are sparsely distributed on the object. We use these touch points as the tactile point cloud. The learnt NOCS maps are used to generate the object model point cloud and the depth map provides the visual point cloud. We compare our approach with methodology introduced in Wang et al. (2019) for pose estimation. Figure 14 shows an example from the PhoCal dataset demonstrating the rendered NOCS map (Figure 14(b)) and the reconstructed models (Figure 14(d)). We note that the reconstructed point clouds are partial (see bottle and fork in Figure 14(d)) as only the visible portions of the scene are used to generate the NOCS map and provides further challenges for pose estimation. Figure 15 shows the comparison results of S-TIQF method against the Umeyama approach (Umeyama, 1991) used in (Wang et al., 2019). We demonstrate that our approach outperforms the baseline method for tactile point clouds by approximately 35% median ADI error and about 20% ADI error when applied to dense visual point clouds (p < .001). We also evaluated the scale estimation approach we use for visual and tactile sensing based point clouds. The scale error is calculated as:

e r r s c a l e = ( s x g t − s x e s t ) 2 + ( s y g t − s y e s t ) 2 + ( s z g t − s z e s t ) 2

(41)

wherein, S e s t = { s x e s t , s y e s t , s z e s t } represents the estimated scale and the S g t = { s x g t , s y g t , s z g t } denotes the ground-truth value. Lower error denotes better estimation. The scale error for tactile based estimation is 0.017 ± 0.0035 and for vision based estimation is 0.03 ± 0.0049. On average, the scale error is 43% lower for tactile-based perception compared to the visual perception. As the dataset consists of transparent and specular objects, the visual sensing method provides incomplete point clouds with a large number of missing or erroneous points whereas the tactile sensing approach provides sparse but approximately uniformly distributed point clouds which can be seen from Figure 14(c). Furthermore, tactile sensing is insensitive to the transparency or specularity properties of the objects and provides high fidelity point measurement.OPEN IN VIEWER

Figure 14.

Qualitative results using the PhoCal dataset: (a) RGB input, (b) rendered NOCS map, (c) visual and tactile point cloud, and (d) reconstructed model point clouds from NOCS maps in (b).

OPEN IN VIEWER

Figure 15.

Comparison of our method against the NOCS (Umeyama method) (Wang et al., 2019) performed as a feasibility study with the PhoCal dataset. p values calculated by Welch’s t-test shown as ∗.

4.3.2. Robotic experiments

In order to validate our method in real world settings, we carried out extensive experiments using the robotic setup shown in Figure 1 and everyday objects shown in Figure 8. Similar to the benchmark experiments, our S-TIQF and TIQF methods are compared against the same baseline methods. The model point cloud is derived from the reconstructed point cloud from the reconstruction network. The scene point cloud comprises of vision and/or tactile data. For each target object, the experiment is repeated five times by randomising the cluttered scene for each iteration. Similar to the previous experiments, the initial pose is sampled randomly from [−5.0, 5.0] m and [−180°, 180°] for each trial and the same initial pose is provided to all comparison methods. The quantitative results for transparent objects are shown in Figure 16(a) and opaque objects in Figure 16(b). The results with transparent objects are similar to the benchmark experiments due to the nature of sparse tactile point clouds and S-TIQF outperforms the baseline approaches. For instance, S-TIQF outperforms the next best baseline method S-ICP by nearly 40% on average for sparse tactile point clouds (p < .001). In comparison, it can be seen from Figure 16(b) that for dense visual point clouds, nearly all the methods perform equally well and our S-TIQF method compares favourably with the state-of-the-art (p < .001). Our method achieves an average ADI error of 2.1 cm whereas TEASER++ achieves an error of 2.5 cm for dense visuo-tactile point clouds (p = .04034). The learning-based approach PREDATOR also performs on-par with other baselines for the dense visuo-tactile point clouds with average ADI error of 3.1 cm. However, the performance of PREDATOR with sparse tactile point clouds for transparent objects is much worse, with the average accuracy of S-TIQF nearly 65% better than PREDATOR. The PREDATOR method assumes sufficiently dense local point features even if there is minimal overlap for the overlap attention module which is not the case for sparse tactile point clouds and results in lower performance. In fact, it can be seen that the combined visuo-tactile point clouds results in better accuracy than visual or tactile point clouds alone, demonstrating the importance of shared perception. For instance, the accuracy improves by ∼ 35 % for S-TIQF using the combined visuo-tactile point cloud instead of either vision or tactile point clouds alone. In this case, it should be noted that the higher levels of inaccuracies with tactile data are a result of the active object exploration strategy wherein tactile data are only collected in locations of higher uncertainty and inaccessible locations to visual data. Hence, it can be concluded that our S-TIQF method provides highly accurate pose estimation for both sparse tactile and dense visual data. The object-wise pose estimation results with S-TIQF is shown in Figure 17 and detailed in the Discussion section (Section 4.5).OPEN IN VIEWER

Figure 16.

Average pose error for real-world objects for (a) opaque objects with visuo-tactile perception and (b) transparent objects with tactile perception. p values calculated by Welch’s t-test shown as ∗.

OPEN IN VIEWER

Figure 17.

Object-wise pose estimation results with S-TIQF. The figure shows a standard box-and-whisker plot with the box extended between the upper and lower quartile and line showing the median. The whiskers show the range of the data.

4.4. Visuo-tactile hand-eye calibration

This section provides comparative studies performed for hand-eye calibration of our in situ approach and standard methods using calibration grid with the algorithm originally presented by Tsai (Tsai and Lenz, 1989). For the calibration grid method, the grid was fixed at a suitable distance from the camera such that it is clearly within the field of view of the camera. Ten different viewpoints were chosen manually ensuring that the different end-effector rotations were incorporated. The experiment was repeated five times. Our in situ visuo-tactile calibration approach does not require a specialized grid. Any object in the workspace of the robot can be used as long as an accurate point cloud corresponding to the object is available. The object must be immobilised and multiple visual pointclouds are captured from different viewpoints. With an incorrect hand-eye calibration, the point clouds from different views would not overlap accurately and result in the scenario shown in Figure 18(a). Furthermore, when tactile data are collected from the same object, resulting in the tactile point cloud, an incorrect hand-eye calibration can be described in the scenario shown in Figure 18(b). The tactile sensors are rigidly attached to the end-effector and the robot kinematics are accurate enough to provide a grounding of the object pose. Using the calibration grid method, an acceptable accuracy can be achieved, but residual errors would still be present. The qualitative results are shown in Figure 18(c). Using our in situ approach with the S-TIQF method, a highly accurate solution can be obtained (Figure 18(d)). Quantitative results are shown in Figure 18(e). Our approach achieves < 1 cm error in position and < 5° error in rotation.OPEN IN VIEWER

4.4.1. Robot kinematic accuracy benchmark for calibration

Our in situ visuo-tactile-based hand-eye calibration method depends on the accuracy of the kinematic calibration (especially for the robot with tactile sensing). Although this assumption is commonly used in the case of hand-eye calibration (Sun and Hollerbach, 2008), we benchmarked the kinematic accuracy of both robots using an external sensor system to evaluate the effect on hand-eye calibration. We used the OptiTrack motion capture system (NaturalPoint, Inc.) to track a specially designed coordinate marker frame with embedded markers that is attached to the gripper fingers, as shown in Figure 19(a). The motion capture system has an average accuracy of 0.1 mm. The UR5 robot, which is sensorized with tactile sensors, has a pose repeatability of 0.1 mm and is kinematically calibrated by the manufacturer. ² The Franka Emika robot which is sensorised with the camera also has a pose repeatability of 0.1 mm obtained from the datasheet of the manufacturer. Both robot’s end effector were moved in arbitrary trajectories covering all 6 DoF by the human user with manual hand guiding and the pose of the end-effector was extracted from the kinematic model and using the motion capture system, respectively. The static offset which arises due to pose difference between the robot’s end-effector frame and the designed marker frame which is attached to the end-effector that needs to be compensated. The end-effector poses expressed in the world-coordinate frame are extracted from the robot kinematic model and the poses of the marker frame expressed in the world coordinate frame extracted from the Optitrack system are used while the robot is stationary and the static offset is measured as the averaged RMSE of the two poses. We compare the accuracy between these poses in Figure 19(b) and (c) where the plots of the end-effector trajectory are shown with the kinematic pose calculation in blue and the motion capture calculated pose in red. The numerical results are shown in Table 2. We recognize the intrinsic uncertainties inherent in our comparative analysis: sporadically, the human operator might occlude the markers from the field of view of certain OptiTrack cameras during manual guidance, despite the strategic deployment of six OptiTrack cameras surrounding the workspace. Hence, we measure the median error and median absolute deviation to disregard spurious outlier points. We note that the kinematic accuracy of the UR5 robot with our benchmarking 0.303 ± 1.82 mm, which is crucial for the calculation of the tactile point clouds. We note that this accuracy for the tactile measurements is within the tolerance bounds. The kinematic discrepancies observed in the Panda robot are more pronounced (±4 mm median absolute deviation); however, their impact on the hand-eye calibration process remains minimal, as the tactile point cloud serves as the reference for the registration of the corresponding visual point cloud. Our method works identically for the case where the camera is kept static and visual point cloud of the object is registered to the tactile point cloud.OPEN IN VIEWER

Figure 19.

(a) Benchmarking the robot kinematics with high-precision marker-based motion capture system (OptiTrack) for (b) UR5 robot and (c) Franka Panda robot. The robot poses are shown in blue and the pose calculated by the motion capture system in red.

OPEN IN VIEWER

Table 2.

Numerical results from kinematic calibration benchmark showing the median error and median absolute deviation.

	UR5	Panda
Calibration error (mm)	0.303 ± 1.82	0.432 ± 3.342

4.5. Discussion