Visual and Haptic Integration in the Estimation of Softness of Deformable Objects

Abstract

Softness perception intrinsically relies on haptic information. However, through everyday experiences we learn correspondences between felt softness and the visual effects of exploratory movements that are executed to feel softness. Here, we studied how visual and haptic information is integrated to assess the softness of deformable objects. Participants discriminated between the softness of two softer or two harder objects using only-visual, only-haptic or both visual and haptic information. We assessed the reliabilities of the softness judgments using the method of constant stimuli. In visuo-haptic trials, discrepancies between the two senses' information allowed us to measure the contribution of the individual senses to the judgments. Visual information (finger movement and object deformation) was simulated using computer graphics; input in visual trials was taken from previous visuo-haptic trials. Participants were able to infer softness from vision alone, and vision considerably contributed to bisensory judgments (∼35%). The visual contribution was higher than predicted from models of optimal integration (senses are weighted according to their reliabilities). Bisensory judgments were less reliable than predicted from optimal integration. We conclude that the visuo-haptic integration of softness information is biased toward vision, rather than being optimal, and might even be guided by a fixed weighting scheme.

Keywords

integration softness visuo-haptic multisensory material perception

References

Alais

Burr

(2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14(3), 257–262. doi:10.1016/j.cub.2004.01.029

Backus

B. T.

Banks

M. S.

(1999). Estimator reliability and distance scaling in stereoscopic slant perception. Perception, 28(2), 217–242. doi:10.1068/p2753

Tiest

W. M. Bergmann

Kappers

A. M.

(2006). Analysis of haptic perception of materials by multidimensional scaling and physical measurements of roughness and compressibility. Acta Psychologica, 121(1), 1–20. doi:10.1016/j.actpsy.2005.04.005

Tiest

W. M. Bergmann

Kappers

A. M.

(2008). Kinaesthetic and cutaneous contributions to the perception of compressibility. In Ferre

(Ed.), Haptics: Perception, devices and scenarios (pp. 255–264). Berlin: Springer

Calvert

G. A.

Spence

Stein

B. E.

(Eds.). (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press

DiFranco

Beauregard

Srinivasan

M. A.

(1997). The effect of auditory cues on the haptic perception of stiffness in virtual environments. Proceedings of the ASME Dynamic Systems and Control Division, 61, 17–22

Drewing

Ernst

M. O.

(2006). Integration of force and position cues for shape perception through active touch. Brain Research, 1078, 92–100. doi:10.1016/j.brainres.2005.12.026

Drewing

Ramisch

Bayer

(2009). Haptic, visual and visuo-haptic softness judgments for objects with deformable surfaces. In EuroHaptics Conference, 2009 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics 2009. Third Joint, Salt Lake City, UT (pp. 640–645). IEEE

Drewing

Wiecki

T. V.

Ernst

M. O.

(2008). Material properties determine how force and position signals combine in haptic shape perception. Acta Psychologica, 128(2), 264–273. doi:10.1016/j.actpsy.2008.02.002

10.

Ernst

M. O.

Banks

M. S.

(2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429–433. doi:10.1038/415429a

11.

Ernst

M. O.

Bulthoff

H. H.

(2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162–169. doi:10.1016/j.tics.2004.02.002

12.

Ernst

M. O.

(2007). Learning to integrate arbitrary signals from vision and touch. Journal of Vision, 7(5). doi:10.1167/7.5.7

13.

Gepshtein

Burge

Ernst

M. O.

Banks

M. S.

(2005). The combination of vision and touch depends on spatial proximity. Journal of Vision, 5(11), 1013–1023. doi:10.1167/5.11.7

14.

Helbig

H. B.

Ernst

M. O.

(2007). Optimal integration of shape information from vision and touch. Experimental Brain Research, 179(4), 595–606. doi:10.1007/s00221-006-0814-y

15.

Helbig

H. B.

Ernst

M. O.

(2008). Visual-haptic cue weighting is independent of modality-specific attention. Journal of Vision 8(1):21, 1–16. doi:10.1167/8.1.21

16.

Hillis

J. M.

Watt

S. J.

Landy

M. S.

Banks

M. O.

(2004). Slant from texture and disparity cues: Optimal cue combination. Journal of Vision, 4(12), 967–992. doi:10.1167/4.12.1

17.

Hollins

Bensmaia

Karlof

Young

(2000). Individual differences in perceptual space for tactile textures: Evidence from multidimensional scaling. Perception and Psychophysics, 62(8), 1534–1544. doi:10.3758/BF03212154

18.

Jacobs

R. A.

(2002). What determines visual cue reliability? Trends in Cognitive Sciences, 6(8), 345–350. doi:10.1016/S1364-6613(02)01948-4

19.

Kuschel

Di Luca

Buss

Klatzky

R. L.

(2010). Combination and integration in the perception of visual-haptic compliance information. IEEE Transactions on Haptics, 3(4), 234–244. doi:10.1109/TOH.2010.9

20.

Landy

M. S.

Maloney

L. T.

Johnston

E. B.

Young

(1995). Measurement and modeling of depth cue combination: In defense of weak fusion. Vision Research, 35(3), 389–412

21.

Mascaro

S. A.

Asada

H. H.

(2001). Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction. IEEE Transactions on Robotics and Automation, 17(5), 698–708

22.

Nardini

Jones

Bedford

Braddick

(2008). Development of cue integration in human navigation. Current Biology, 18(9), 689–693. doi:10.1016/j.cub.2008.04.021

23.

Oruc

Maloney

L. T.

Landy

M. S.

(2003). Weighted linear cue combination with possibly correlated error. Vision Research, 43(23), 2451–2468

24.

Perotti

V. J.

Todd

J. T.

Lappin

J. S.

Phillips

(1998). The perception of surface curvature from optical motion. Perception and Psychophysics, 60(3), 377–388

25.

Rosas

Wagemans

Ernst

M. O.

Wichmann

F. A.

(2005). Texture and haptic cues in slant discrimination: Reliability-based cue weighting without statistically optimal cue combination. Journal of the Optical Society of America a-Optics Image Science and Vision, 22(5), 801–809

26.

Serwe

Drewing

Trommershäuser

(2009). Combination of noisy directional visual and proprioceptive information. Journal of Vision, 9(5), 28 21–14. doi:10.1167/9.5.28

27.

Srinivasan

M. A.

Lamotte

R. H.

(1995). Tactual discrimination of softness. Journal of Neurophysiology, 73(1), 88–101

28.

Srinivasan

M. A.

Beauregard

G. L.

Brock

D. L.

(1996). The impact of visual information on the haptic perception of stiffness in virtual environments. Proceedings of the ASME Dynamic Systems and Control Division, 58, 555–559

29.

Stein

B. E.

Meredith

M. A.

(1993). The merging of the senses. Nature, 364(6438), 588–588

30.

Stevens

S. S.

(1956). The direct estimation of sensory magnitudes – loudness. American Journal of Psychology, 69(1), 1–25. doi:10.2307/1418112

31.

van Beers

R. J.

Sittig

A. C.

Gon

J. J.

(1996). How humans combine simultaneous proprioceptive and visual position information. Experimental Brain Research, 111(2), 253–261. doi:10.1007/BF00227302

32.

van Beers

R. J.

Sittig

A. C.

Gon

J. J.

(1999). Integration of proprioceptive and visual position-information: An experimentally supported model. Journal of Neurophysiology, 81(3), 1355–1364

33.

Wichmann

F. A.

Hill

N. J.

(2001). The psychometric function: I. Fitting, sampling, and goodness of fit. Perception and Psychophysics, 63(8), 1293–1313. doi:10.3758/BF03194544

34.

W. C.

Basdogan

Srinivasan

M. A.

(1999). Visual, haptic, and bimodal perception of size and stiffness in virtual environments. Proceedings of the ASME Dynamic Systems and Control Division, 67, 19–26

35.

Young

M. J.

Landy

M. S.

Maloney

L. T.

(1993). A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Research, 33(18), 2685–2696