Contact State Estimation by Vision-Based Tactile Sensors for Dexterous Manipulation with Robot Hands Based on Shape-Sensing

Abstract

We propose a new method for estimating the contact state of objects with varying shapes on a vision-based fluid-type tactile sensor, which touch pad is an elastic transparent membrane of silicon rubber with dotted pattern printed on its inner side. The membrane is filled with translucent red colored water. The proposed method leads to better understanding of the object's shape and movement, and can be applied for accomplishing reliable and dexterous handling tasks by robot hands.

Keywords

Sensors Object Location Estimation Contact Region Estimation Dexterous Handling Robotics

1. Introduction

Quality of tactile sensors is a key factor that determines the efficiency of the robot in the completion of tasks that require dexterity of robot hands and precise handling of objects [1]. Various tactile sensing methods have been introduced based on resistive, capacitive, piezoelectric, ultrasonic, electromagnetic sensing elements [2]–[7]. In order to achieve dexterous handling of robots, it is required to obtain not only contact forces [8][9] but also information on slippage [10]–[17] on the sensor surface.

Shape-sensing methods for objects were also developed in [18]–[22]. Tactile information about the object shape allows the robot control system to choose grasping strategy when robot's fingers conform to the shape of the grasped object. With the shape-sensing that enables one to detect deformation of the object in contact, robots can dexterously grasp soft and fragile objects.

Although the sensors in the literature provide information about contact force, slippage, shape of the sensor surface and so on, they don't give information about the contact region on the sensor surface. The contact region allows us to estimate the object shape in an accurate manner when combined with the shape information. In order to avoid the risk that the sensor surface tears or the grasped object collapses, and also to enhance the grasp task stability with a sufficiently large contact area, we need to evaluate the contact pressure based on the contact region.

On the other hand, precise manipulation tasks that involve grasping and rolling of an object require information about the exact location of the object in contact. A sensor surface with an elastic sensor body is easily deformed by applied forces. This adds uncertainty when estimating the location of the object, and if the sensor is applied to a robot hand, it will bring about large manipulation error. Y. Yamada et al. developed a robot skin that uses tiny reflector chips arranged in a matrix form on the elastic sensor surface to provide information about the exact location of an object [23]. The proposed sensing method in [23] gives information about the position and orientation of the object. However, the sensor does not give enough information about the orientation of objects with uniform shapes such as spheres or cylinders.

Our group proposed a vision-based tactile sensor that consists of an elastic body-type touch pad, a CCD camera, and LED light [16][24]-[26], that simultaneously acquires the contact force/moment as well as how much an object is slipping. Robot hands achieve more reliable and dexterous handling with the use of not only the contact force and slippage, but also shape-sensing information. A fluid-type tactile sensor was also developed for shape-sensing of the touch pad [27]. The fluid-type sensor [27] consists of en elastic membrane of silicon rubber filled with translucent red water.

In this paper, we propose a new method for estimating the contact region and location of various-shaped objects on a vision-based fluid-type tactile sensor. The previously developed sensing method [27] can be applied in conjunction with the proposed method. We focus on the curvature radius of the elastic membrane to estimate the contact region, considering the balance between the tensional forces of the membrane and the inner pressure. Position of the object is obtained by using the contacting dots, whose displacement corresponds to the translational displacement of the object. The rotation angles of the object are calculated from the rotation matrix of the basis vectors that were constructed on the contact surface. Since the contact region is changed for a rolling object, we should identify the location of the dots in a new contact region at each sampling time. The rotation angles are calculated by accumulating changes of the rotation angles at each sampling time. The usefulness of the proposed method is confirmed through the experimental results. The field of the application of this sensor is not only anthropomorphic robotic hands with five fingers but also industrial robotic hands and robotic hands with simple structures different from human hands.

2. Vision-based Tactile Sensor

In this study, we used the fluid-type tactile sensor and the sensing methods developed in our previous work [27]. Fig. 1 shows the configuration of the vision-based tactile sensor. It consists of a CCD camera, LED lights, a transparent acrylic plate, and a touch pad shaped as an elastic membrane of silicon rubber. Translucent red colored water fills the membrane. For detecting the state of contact, we use a CCD camera with dimensions 8 × 8 × 40 mm and LED lights with dimensions 60 × 60 × 60 mm. The touch pad has a hemispherical shape. Its curvature radius is 20 mm and the height is 13 mm. The touch pad has a two-layered structure. The outer membrane is black, while the inner membrane is white, as shown in Fig. 2. Elastic membranes of silicon rubber with thickness of 0.5 mm are used for each layer. A dotted pattern is printed on the inner membrane. The dots have diameter of 0.7 mm and the distance between two neighbor dots in the array is 1.5 mm. The LED source uses a polarized light filter to eliminate the light reflected from the transparent acrylic plate and to produce a uniform light beam directed in parallel to the image capturing direction. The CCD camera takes images with a sampling frequency of 30 Hz of the inner spherical surface of the touch pad through the acrylic plate. Fig. 3 shows the captured images. Each image frame has 640 pixels × 482 effective pixels. Fig. 3(a) shows the captured image when the sensor is not in contact with the object. Fig. 3(b) shows the image when the sensor is in contact with an object.

Figure 1.

Configuration of vision-based tactile sensor.

Figure 2.

Configuration of proposed touch pad.

Figure 3.

Captured images; (a) non-contact state. (b) contact state.

3. Estimation of Tactile Information

In the proposed method in this chapter, we use the shape sensing method of [27] to acquire the shape of the membrane. However, we can apply the proposed method in conjunction with different sensors and shape-sensing methods, which include stereo vision systems and so on. Therefore, the proposed method represents a significant advance when compared to the previous work of [27].

3.1 Theory of Contact Region Estimation

When the touch pad is in contact with an object, we consider a small surface element of the elastic membrane in Fig. 4. The shape of each surface element may be kept either as a result of contact with the object or in situations when the tensional force of the (non-contact) surface element and the inner pressure of the touch pad are balanced. That's why it is difficult to conclude from the shape of the touch pad whether each surface element is in contact with the object or not.

Figure 4.

Tensional force - inner pressure relation on a convex membrane; (a) surface element with center point P and s axis, (b) surface element with positive R(θ) and R(θ⁺π/2) (Type A), (c) surface element with negative R(θ) and R(θ⁺π/2) (Type B). (d) surface element with negative R(θ) and positive R(θ⁺π/2) (Type C).

In this study, we assume that the contact surface of an object consists of flat and convex surfaces (Assumption A), the inner pressure of the membrane is larger than the outer pressure (Assumption B), the bending moment and thickness of the membrane can be ignored (Assumption C) and the contact surface of the object is large enough compared to a single surface element of the membrane (Assumption D). Since a concave object consisting of flat and convex surfaces or an object with non-contact concave surfaces can be properly handled with the proposed method, Assumption A is not strict. In general, Assumption B is satisfied. We use an elastic membrane of silicon rubber with a thickness of 1.0 mm which is soft and thin enough to satisfy Assumption C. By using a small surface element of the membrane, Assumption D can be easily satisfied by using a high-resolution camera while handling ordinary objects. Moreover, we can detect whether the contact occurs or not and estimate the shape of the contact object roughly even if Assumption D is not satisfied. Therefore, these assumptions are very practical. Under these assumptions, the convex surface element of the membrane is considered in the non-contact region while the concave/flat surface element of the membrane is considered in the contact region.

We estimate the contact region by analysis of the curvature radius of the elastic membrane. As a result of the tensional force on the membrane and inner pressure of the filled fluid, the elastic membrane bulges outside of the touch pad from the contact region. The effect of the tensional force and inner pressure on the surface element is illustrated in Fig. 4, where the point P is the center point of the surface element in Fig. 4(a). When we consider a certain axis s on the membrane (fixed to the sensor) as shown in Fig. 4(a), the inner pressure p is described in the form of the Young–Laplace equation as follows

\frac{σ (θ)}{R (θ)} + \frac{σ (θ + π / 2)}{R (θ + π / 2)} = p (0 \leq θ < \frac{π}{2} [rad])

(1)

Here, σ(θ) is the tensional force per unit length of the membrane, θ is the angle between the direction of σ(θ) and s axis as shown in Fig. 4(a). |R(θ)| is defined as the absolute value of the curvature radius of the surface element. The sign of R(θ) is determined depending on the convexity of the surface element in (1). The sign of R(θ) is defined as plus when the surface element is convex in the direction of σ(θ), while the sign of R(θ) is defined as minus when the surface element is concave. Since σ(θ) in all directions as well as p must both be positive, any flat or concave surface element of the membrane does not satisfy (1) and thus must be subjected to forces other than the inner pressure of the filled fluid. Such a surface element belongs to the contact region. By assuming that the contact surface of the object is convex, any convex surface element of the membrane is surely in the non-contact region. Therefore, the sufficient condition for the point P to belong to the contact region is that the surface element is flat or concave as follows:

\frac{1}{R (θ)} \leq 0 (0 \leq θ < π [rad])

(2)

Here, the surface element is generally classified into three types of surface elements: a convex surface element (Type A), concave/flat surface element (Type B) and surface element (Type C) which is concave/flat but locally convex as shown in Fig. 4(b), (c) and (d), respectively. Condition (2) regards Type A as a surface element in the contact region, and Type B and Type C as a surface element in the non-contact region.

3.2 Calculation of Curvature Radiuses of Surface Elements

Next, we calculate the curvature radiuses in various directions of each surface element of the membrane from the shape of the touch pad. Let us define x, y and z axes in Fig. 5, where x and y axes are parallel to the acrylic plate and z axis is vertical to x and y axes. We also define u axis on the x−y plane and v axis parallel to z axis as shown in Fig. 5. The angle θ between u axis and x axis ranges from 0 to π rad. The z value of the touch pad at (x, y) is defined as l(x, y). The curvature radius of the membrane at (x, y, l(x, y)) is measured in the u−v plane that is perpendicular to the x−y plane. Here, we assume that the shape of a small surface element around (x, y, l(x, y)) is spherical in an arbitrary u−v plane. The curvature radius can be defined by three points P₁, P₂, and P₃ that lie on them. The reference points have x-y-z coordinates: P₁(x-rcosθ, y-rsinθ, l(x-rcosθ, y-rsinθ)), P₁(x, y, l(x, y)) and P₃(x+rcosθ, y+rsinθ, l(x+rcosθ, y+rsinθ)). From their coordinates we define the following equations that describe the membrane circular surface element,

Figure 5.

The curvature radius in the u-v plane.

{(x - c_{x θ})}^{2} + {(y - c_{y θ})}^{2} + {(z - c_{z θ})}^{2} = R {(x, y, θ)}^{2}

(3)

\frac{y - c_{y θ}}{x - c_{x θ}} = \tan θ

(4)

R (x, y, θ) = {\begin{cases} | R (x, y, θ) | (l (x, y) \geq c_{z θ}) \\ - | R (x, y, θ) | (l (x, y) < c_{z θ}) \end{cases}

(5)

where point C(c_xθ, c_yθ, c_zθ) is the center of the spherical surface element and R(x, y, θ) is the curvature radius in the u-v plane in Fig. 5.

3.3 Consideration of Error of Curvature Radiuses

No matter how accurate the shape sensing method is, the obtained shape of the membrane should contain some estimation error due to disturbances such as sensor noise. This error fluctuates the curvature radiuses calculated in Section 3.2. If the surface element is nearly flat but concave, it may have positive R(x, y, θ). Therefore, in this study the condition (2) for the point (x, y) to be in the contact region is modified as follows.

\frac{1}{R (x, y, θ)} < δ (0 \leq θ < π [rad])

(6)

In order to determine the threshold value δ, we use the average values E_R(x, y, θ) and the standard deviations σ_R(x, y, θ) of each 1/R(x, y, θ) when the touch pad is not in contact with the object. We can discriminate in an accurate manner if a point (x, y) is in the non-contact region by setting δ as follows,

δ = \min {E_{R} (x, y, θ) - 2 σ_{R} (x, y, θ) : (x, y) ∊ A, 0 \leq θ < π [rad]}

(7)

where A is the set of x-y coordinates in the touch pad.

Here, a smaller standard deviation of the left-hand side 1/R(x, y, θ) is desirable for the accurate estimation of the contact region. Taking the average of 1/R(x, y, θ) with respect to various θ at the point of (x, y) on the condition (6) yields the following:

f (x, y) = \frac{1}{n} \sum_{i = 1}^{n} \frac{1}{R (x, y, π \frac{i}{n})} < δ

(8)

where

δ = \min {E_{f} (x, y, θ) - 2 σ_{f} (x, y, θ) : (x, y) ∊ A}

(9)

Here, E_f(x, y, θ) and σ_f(x, y, θ) are the average values of each f(x, y) and the standard deviations of each f(x, y) when the touch pad doesn't contact with the object. The value of S calculated with (9) is smaller than δ calculated with the condition (7) because σ_R(x, y, θ) in (8) is smaller than σ_f(x, y, θ) in (6). By (8), we can judge the contact more robustly than (6). In this study, we assume that the contact surface of the object is convex. Although we regard the surface element (Type C) of Fig. 4(d) as being in the non-contact region, it may satisfy (8) when R(θ) is a small negative number. Therefore, when R(x, y, θ) is negative, we set R(x, y, θ) to −∞, which yields 1/R(x, y, θ) = 0.

3.4 Estimation of Object Location

Using this method, we calculate the location of objects that are not slipping on the sensor surface. In order to estimate the location of the object, we focus on the positions of the dots printed on the inner membrane. The three-dimensional positions of the dots are obtained by using the geometrical relationship presented in Fig. 6. That leads to the following equation:

Figure 6.

Geometrical relationship between the three-dimensional dot position and the one in the captured image.

p_{i, j} = [\begin{array}{l} x_{i, j} \\ y_{i, j} \\ z_{i, j} \end{array}] = [\begin{array}{l} ({x^{'}}_{i, j} - x_{c e n t}) (1 + l \sin φ_{2} / h \tan φ_{1}) + x_{c e n t} \\ ({y^{'}}_{i, j} - y_{c e n t}) (1 + l \sin φ_{2} / h \tan φ_{1}) + y_{c e n t} \\ l (x_{i, j}, y_{i, j}) \end{array}]

(10)

where p _i,j is the position vector of the dot i, j: that is, i-th dot from the left and j-th dot from the top. (x_i,j, y_i,j and z_i,j) is the Cartesian position of p _i,j. l(x_i,j, y_i,j) is the depth of the pad at point (x_i,j, y_i,j) and h is the distance between the camera and the acrylic plate. x_cent and y_cent are x-y values of the CCD camera, x'_i,j and y'_i,j are the positions on the acrylic plate which the LED lights reflects off the membrane at (x_i,j, y_i,j, z_i,j) and passes through on, as shown in Fig. 6. From the geometrical relationship, the values of the dot in the captured image and h, we can calculate x'_i,j, y'_i,j and ϕ₂. We obtain ϕ₁ from ϕ₂ and the relative refractive index between the air and the water.

Here, we define the contact reference dot p _I,J as the dot with the biggest displacement from the initial position, that is, the position when the touch pad doesn't contact with an object. The contact reference dot should be always in contact with object. When the object doesn't slip, the displacement of the object is equal to that of the contact reference dot. Here, the contact surface of the object must be large enough compared to the contact reference dot.

Next, we obtain a relative orientation of an object right after it starts to contact with the sensor. It should be noted that the rotation angle of the object is equal to the rotation angle of the surface element around the contact reference dot if the object doesn't slip. In order to obtain the rotation angle of the contact reference dot, we calculate the rotation matrix R of the surface element around the contact reference dot. Here, we assume that a set of dots which consists of the contact reference dot and the eight dots adjacent to the contact reference dot are in continuous contact with the object. Thus, the contact surface of the object must be large enough compared to the set of dots. When the object does not slip, the relative positions and the relative orientations of these eight points to the contact reference dot stay at the same ones. In order to calculate the rotation angles of the contact reference dot, we define three basis vectors n ₁, n ₂ and n ₃ around the contact reference dot on the following way.

n_{1} = \sum_{k = - 1}^{1} (p_{I + 1, J + k} - p_{I - 1, J + k}) / | \sum_{k = - 1}^{1} (p_{I + 1, J + k} - p_{I - 1, J + k}) |

(11)

n_{2} = \sum_{k = - 1}^{1} (p_{I + k, J + 1} - p_{I + k, J - 1}) / | \sum_{k = - 1}^{1} (p_{I + k, J + 1} - p_{I + k, J - 1}) |

(12)

n_{3} = n_{1} \times n_{2}

(13)

The relation between n ₁, n ₂ and n ₃ in the k-th image and l-th image is represented as follows.

[\begin{matrix} n_{1}^{l} \\ n_{2}^{l} \\ n_{3}^{l} \end{matrix}] = {}^{k}R^{l} [\begin{matrix} n_{1}^{k} \\ n_{2}^{k} \\ n_{3}^{k} \end{matrix}]

(14)

In the above equation we use l and k to note sampling indexes of the image. ^k R ^l is the matrix that represents the rotation between the k-th and l-th images. It is obtained by multiplying both sides of (14) by [ n ^{k
₁}, n ^{k
₂}, n ^{k
₃}]^−T. We calculate the rotation angles around x, y and z-direction ^kθ^l_x, ^kθ^l_y and ^kθ^l_z between the k-th and l-th images by solving the following equation,

{}^{k}R^{l} = {}^{k}R_{z}^{l} {}^{k}R_{y}^{l} {}^{k}R_{x}^{l}

(15)

where ^k R ^l_x, ^k R ^l_y and ^k R ^l_z are the rotation matrixes around x, y and z-direction, respectively. Note that the orientation of the object is not equal to ^kθ^l_x, ^kθ^l_y and ^kθ^l_z when the contact reference dot is changed in some situations such a rolling of the object. The rotation angles of the object ^mθⁿ_objx, ^mθⁿ_objy, and ^mθⁿ_objz, can be calculated as the sums of the values of ^kθ^k+1_x, ^kθ^k+1_y and ^kθ^k+1_z from each image captured by the CCD camera.

{}^{m}θ {_{o b j}^{n}}_{x} = \sum_{k = m}^{n - 1} {}^{k}θ_{x}^{k + 1}, {}^{m}θ {_{o b j}^{n}}_{y} = \sum_{k = m}^{n - 1} {}^{k}θ_{y}^{k + 1}, {}^{m}θ {_{o b j}^{n}}_{z} = \sum_{k = m}^{n - 1} {}^{k}θ_{z}^{k + 1}

(16)

By using ^mθⁿ_objx, ^mθⁿ_objy and ^mθⁿ_objz and the contact region which is estimated in Section 3.1, 3.2 and 3.3, we can determine whether the object is rolling or rotating. Note again that the rotation angle of the object is equal to the rotation angle of the surface element around the contact reference dot if the object doesn't slip. Therefore, the object should be rotating but not rolling when the value of ^mθⁿ_objx, ^mθⁿ_objy or ^mθⁿ_objz is not nearly 0 and the contact region isn't moving. When the value of ^mθⁿ_objx, ^mθⁿ_objy or ^mθⁿ_objz is not nearly 0 and the contact region is moving, the object should be rolling.

In this study, the proposed method can estimate the position and rotation angle of the object only when the slippage is not occurred between the object and the set of nine dots. However, the previous work of [17], [25], [26] can estimate the partial slippage of the object on the sensor surface and control the grip force to prevent the slippage by analyzing the dot displacement. With this method, we can accurately estimate the object location without the slippage of the object occurs. Furthermore, when the slippage of the object occurs and re-contacts, we can restart to estimate the object location after the slippage.

4. Experimental Results

4.1 Estimation of Contact Region

Fig. 7 shows experimental results of the estimation of the contact region when the normal force is applied in z-direction to the sensor which is in contact with the object. In order to determine parameter δ, we obtained the average values E_f(x, y) and the standard deviations σ_f(x, y) of each f(x, y), where y=18.5 mm and x was changed at intervals of 0.77 mm. E_f(x, y) and σ_f(x, y) were calculated by using 200 captured images when the touch pad did not contact. δ was set to 0.002 mm⁻¹ which is the minimum value of E_f(x, y)-2σ_f(x, y) as in the condition (9). n was set to 18, which was considered from the preliminary experiment to be enough to decrease the directional dependence and the deviation of the curvature radiuses. For the surface element with a small r, we can calculate the curvature radiuses with a small surface element. The surface element size has trade-off between the size of σ_f(x, y) and the estimation accuracy of r for a sharp curve. In this experiment, r was determined as 1.92 mm, which was obtained experimentally. We estimated the contact region based on the condition (8) of the points, which were at intervals of 0.38 mm in x and y directions, on the membrane. When a point satisfies (8), we regard the area within 0.5 mm from that point as in the contact region.

In Fig. 7, gray area is in the calculated contact region. The contact regions of the various shaped objects were obtained by the proposed method, where the maximum estimation error was about 3mm. The estimation error of the contact region in Fig. 7 is mostly due to the estimation error of the curvature radiuses by variation of the shape-sensing method of [27]. Although the accuracy for the boundary detection may have errors up to 3mm, we can detect angles of sharp edges down to 20 degree as shown in Fig. 7(f). In fact, the bending moment of the membrane influences the shape of the surface element of the membrane and thus, there are the transitions between convex and concave surface elements of the membrane around the boundary. Moreover, a larger size of the surface element is more influenced by the transitions. However, the contact region can be better estimated when we decrease the size of the surface element without the deterioration in detection accuracy by using a high resolution camera.

Figure 7.

Estimation results of the contact region: (a) is square shape. (b) is ring shape. (c) is the shape consisting of right angles. (d), (e) and (f) are angular shapes with the sharp angle of 65, 50, 20 degree, respectively.

4.2 Estimation of Object Location

In this section, we estimated the object location by using the method of Section 3.4. For comparison, we measured the actual position and angle of the objects from the side view captured by the side camera as shown in Fig. 8, and observed that the slippage between the object and the sensor was not detected. We did an experiment when an object was in contact with the sensor and moved within the x-z plane. The proposed method obtained the position of the object in reference to the position when the object started to contact with the sensor. Fig. 9 shows the experiment result. The upper graph shows the position of the object in the x direction. The lower graph shows the position of the object in z direction at the corresponding sampling time of the captured image. The results show that the object position is successfully estimated. Maximum errors do not exceed 0.3 mm in x-direction and 0.3 mm in z-direction. These errors occurred because of the variation of the shape estimation of [27].

Figure 8.

Side view of the touch pad.

Figure 9.

Estimation results of the object position.

During the test for estimation of the object orientation the object moves in z-direction and rotates around the x-axis. The proposed method obtained the rotation angle of the object in reference to the angle when the object started to contact with the sensor. Experimental results are shown in Fig. 10. The touch pad is in contact with the flat surface in Fig. 10(a) and with the edge of 90 degree in Fig. 10(b). The object orientation was estimated by using the rotation matrix of the three basis vectors as defined in Section 3.4. The maximum error is caused by the variation of the shape estimation and does not exceed 3 deg. Therefore, the proposed method can be applied to calculate the position and the orientation of various-shaped objects with not only flat surface but also edge.

Figure 10.

Estimation results of the object orientation: (a) contact with the flat surface. (b) contact with the edge of 90 degree.

In Fig. 11 we provide the estimation result of the object orientation when the object rolls and contacts at the same time with the touch pad. In this experiment the contact reference dot frequently changes. Despite of that, the sensor accurately estimates the rotation angle of the object by accumulating changes of the rotation angle of the contact reference dot. This shows that the orientation estimation method can be applied also in tasks when the contact region is changing. The size of the set of nine dots that was used to calculate the rotation angles is about 3 × 3 mm and is considered small enough for the sensor to contact with various objects.

Figure 11.

Estimation results of the object orientation when object rolls.

4.3 Simultaneous Acquisition of Object Location and Contact Region

In this experiment, we obtain the information of the contact region and the object location simultaneously by applying the proposed method, where the slippage between the object and the sensor was not detected. Fig. 12 shows the object orientation in upper figure and y_I,J in lower figure when the cylindrical object contacts and rolls at the same time. In Fig. 13, we provide the calculated contact region with respect to the sampling indexes. Gray area in Fig. 13 is in the calculated contact region. The contact region and the object orientation were obtained simultaneously and the contact region and the y value of the contact reference dot y_I,J are changing with the rotation angle of the object. The simultaneous acquisition of the tactile information can give the state of the object in detail.

Figure 12.

Obtained object orientation for the rolling object on the sensor.

Figure 13.

Obtained contact region for rolling object on the sensor.

4.4 Fusion of Contact Region and Other Tactile Information

The contact region is crucial for the dexterous handling or stability of precise manipulation tasks and also for sensing of the other tactile information such as the contact pressure, slippage, object's shape and so on. For example, the method [27] proposed by our group and others in [18]–[22] provide the shape of the sensor surface but they cannot give the contact region between the sensor and the object. Since the fit of the shapes between the sensor and the object is occurred only in the contact region, the accurate estimation of the object's shape requires not only the sensor's shape but also the contact region.

The proposed contact region estimation method can be combined with the shape-sensing method of [27] for better estimation of the object. Fig. 14 shows the shape of the touch pad calculated by our method [27] when the touch pad was in contact with two different objects, where the shape is shown on the x-z plane with y=18.5 mm. The object of Fig. 14(a) has triangular shape in the x-z plane and 30 mm deep in the y-direction and the object of Fig. 14(b) has 1 mm wide in the x-direction, 15 mm high in the z-direction and 150 mm deep in the y-direction. Although objects with different shapes contacted to the sensor, the calculated shape were similar between two results in Fig. 14. It is difficult to estimate the object shape from the shape of the touch pad. We show in Fig. 15 the contact region obtained by the proposed method, and gray area in Fig. 14 and Fig. 15 is in the calculated contact region. From the range of the contact region, we can confirm that the triangular shaped object contacts in Fig. 14(a) and the thin object contacts in Fig. 14(b) and the contact of the thin object is unstable. This shows that the estimation of curvature radius on the contact region is effectively applied for shape sensing and may help to support the stable grasping task.

Figure 14.

Estimated shape when the touch pad is in contact with two different objects; (a) is a triangular shaped object. (b) is a thin object.

Figure 15.

Estimated contact region when the touch pad is in contact with two different objects; (a) is a triangular shaped object. (b) is a thin object.

5. Conclusion

In this paper, we have proposed a new method to estimate the contact region of the sensor, and the location of the object by using the vision-based tactile sensor. We focused on the curvature radius of the elastic membrane to estimate the contact region, considering the balance between the tensional forces of the membrane and the inner pressure. In order to decrease the estimation error, we modified the theoretical condition for the point to belong to the contact region and used the average of the curvature radiuses in different directions. The position of the object was obtained by using the dot positions. The translational displacement of the object corresponded to the displacement of the contact reference dot. The rotation angles of the object were calculated from the rotation matrix of the three basis vectors by accumulating changes of the rotation angles of the rotation matrix. We provide some experimental results that confirm the practicality of the method. Future work includes the implementation of the developed sensor to a robot hand.

Footnotes

6.

The authors would like to thank Mr. K. Yamamoto for his help to prepare the experimental setup, and Dr. Dimitar Stefanov, principal lecturer of Coventry University, UK for his valuable comments on the description of this paper.

References

Dahiya

R.S.

Metta

Valle

Sandini

“Tactile Sensing—From Humans to Humanoids”

IEEE Trans. on Robotics, Vol. 26, No. 1, pp. 1–20, Feb, 2010.

Lee

M. H.

and Nicholls

H. R.

, “Tactile sensing for mechatronics-a state of the art survey” Mechatronics, 9, pp. 1–31, Jan, 1999.

Noda

Hoshino

Matsumoto

and Shimoyama

, “A shear stress sensor for tactile sensing with the piezoresistive cantilever standing in elastic material” Sensors and Actuators A: Physical, Vol. 127, Issue. 2, pp. 295–301, Mar, 2006.

Yang

Chen

and Motojima

, “Tactile sensing properties of protein-like single-helix carbon microcoils” Carbon, Vol. 44, Issue. 15, pp. 3352–3355, Dec, 2006.

Takao

Sawada

and Ishida

, “Monolithic Silicon Smart Tactile Image Sensor With Integrated Strain Sensor Array on Pneumatically Swollen Single-Diaphragm Structure” IEEE Trans. Electron Devices, Vol. 53, No. 5, pp. 1250–1259, May, 2006.

Mei

W. J.

Chen

and Chan

M. H.

, “An integrated MEMS three-dimensional tactile sensor with large force range” Sensors and Actuators A: Physical, Vol. 80, Issue. 2, pp. 155–162, Mar, 2000.

Wen

and Fang

, “Tuning the sensing range and sensitivity of three axes tactile sensors using the polymer composite membrane” Sensors and Actuators A: Physical, Vol. 145–146, pp. 14–22, Jul-Aug, 2008.

Kamiyama

Vlack

Mizota

Kajimoto

Kawakami

and Tachi

, “Vision-Based Sensor for Real-Time Measuring of Surface Traction Fields” IEEE Computer Graphics and Applications, Vol.25, No.1, pp. 68–75, Jan-Feb, 2005.

Ohka

, “Optical Three-axis Tactile Sensor” Mobile Robots: Perception & Navigation, pp. 111–136, 2007.

10.

Yoshikai

Tajima

Kagami

Shinoda

Inaba

and Inoue

, “Slip detecting by tactile sensor using Acoustic Resonant Tensor Cell and its application for grasping” Journal of the Robotics Society of Japan, Vol. 20, No. 8, pp.868–875, 2002.

11.

Tremblay

M. R.

and Cutkosky

M. R.

, “Estimating Friction Using Incipient Slip Sensing During a Manipulation Task” 1993 IEEE International Conference on Robotics and Automation, Vol. 1, pp. 429–434, Apr, May, 1993.

12.

Koda

and Maeno

, “Grasping Force Control in Master-Slave System with Partial Slip Sensor” 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4641–4646, Oct 2006.

13.

Canepa

Petrigliano

Campanella

, and Rossi

D. D.

, “Detection of Incipient Object Slippage by Skin-Like Sensing and Neural Network Processing” IEEE Trans. Systems, Man, and Cybernetics, Part B: Cybernetics, Vol. 28, No. 3, pp.348–356, Jun, 1998.

14.

Teshigawara

Shimizu

Tadakuma

Aiguo

Shimojo

and Ishikawa

, “High sensitivity slip sensor using pressure conductive rubber” Proceedings of 2009 IEEE Conference on Sensors, pp. 988–991, Oct, 2009.

15.

Ikeda

Kurita

Ueda

Matsumoto

and Ogasawara

, “Grip Force Control for an Elastic Finger using Vision-based Incipient Slip Feedback” 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 1, Issue. 28, pp. 810–815, Oct, 2004.

16.

Obinata

Ashis

Watanabe

and Moriyama

, “Vision Based Tactile Sensor Using Transparent Elastic Fingertip for Dexterous Handling” Mobile Robots: Perception & Navigation, pp. 137–148, 2007.

17.

Watanabe

and Obinata

, “Grip Force Control Using Vision-Based Tactile Sensor for Dexterous Handling” European Robotics Symposium, Vol.44, pp. 113–122, Mar, 2008.

18.

Saga

Kajimoto

and Tachi

, “High-resolution tactile sensor using the deformation of a reflection image” Sensor Review, Vol. 27, Issue. 1, pp. 35–42, 2007.

19.

Saga

Kuroki

Kawakami

and Tachi

, “Fibratus tactile sensor using reflection on an optical lever” 34th Int. Conf. On Computer Graphics and Interactive Techniques (SIGGRAPH2007), 2007.

20.

Nagata

Ooki

, and Kakikura

, “Feature Detection with an Image Based Compliant Tactile Sensor” 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 2, pp. 838–843, Oct. 1999.

21.

Ferrier

N. J.

, and Brockett

R. W.

“Reconstructing the Shape of a Deformable Membrane from Image Data”

The International Journal of Robotics Research, Vol. 19, No. 9, pp. 795–816, 2000.

22.

Johnson

M. K.

and Adelson

E. H.

, “Retrographic sensing for the measurement of surface texture and shape” 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1070–1077, Jun, 2009.

23.

Yamada

Iwanaga

Fukunaga

Fujimoto

Ohta

Morizono

, and Umetani

, “Soft Viscoelastic Robot Skin Capable of Accurately Sensing Contact Location of Objects” Proc. of IEEE/RSJ/SICE Int. Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 105–110, Aug, 1999.

24.

Obinata

Kurashima

, and Moriyama

, “Vision-based tactile sensor using transparent elastic fingertip for dexterous handling,” In Proceedings of 36th International Symposium on Robotics, TU41(CDROM), 2005.

25.

Ito

Kim

and Obinata

, “Slippage Degree Estimation by Using Vision-Based Tactile Sensor for Dexterous Handling” In Proceedings of 9th IFAC Symposium on Robot Control (SYROCO2009), 2009.

26.

Ito

Kim

and Obinata

, “Slippage Degree Estimation for Dexterous Handling of Vision-Based Tactile Sensor” Proceedings of 2009 IEEE Conference on Sensors, pp. 449–452, Oct, 2009.

27.

Ito

Kim

Nagai

and Obinata

, “Shape Sensing by Vision-based Tactile Sensor for Dexterous Handling of Robot Hands” In Proceedings of 2010 IEEE Conference on Automation Science and Engineering, Aug, 2010 (to be published).