Objective: The aim of this study was to assess how background visual motion and the relative movement of sound affect a head-mounted display (HMD) wearer’s performance at a task requiring integration of auditory and visual information. Background: HMD users are often mobile. A commercially available speaker in a fixed location delivers auditory information affordably to the HMD user. However, previous research has shown that mobile HMD users perform poorly at tasks that require integration of visual and auditory information when sound comes from a free-field speaker. The specific cause of the poor task performance is unknown. Method: Participants counted audiovisual events that required integration of sounds delivered via a free-field speaker and vision on an HMD. Participants completed the task while either walking around a room, sitting in the room, or sitting inside a mobile room that allowed separate manipulation of background visual motion and speaker motion. Results: Participants’ accuracy at counting target audiovisual events was worse when participants were walking than when sitting at a desk, p = .032. Compared with when they were sitting at a desk, participants’ accuracy at counting target audiovisual events showed a trend to be worse when they experienced a combination of background visual motion and the relative movement of sound, p = .058. Conclusion: Multisensory integration performance is least effective when HMD users experience a combination of background visual motion and relative movement of sound. Eye reflexes may play an important role. Application: Results apply to situations in which HMD wearers are mobile when receiving multimodal information, as in health care and military contexts.
Behar, I., Wiley, R.W., Levine, R.R., Rash, C.E., & Walsh, D.J. ( 1990). Visual survey of Apache aviators (VISAA) (No. ADA230201). Fort Rucker, AL: U.S. Army Aeromedical Research Laboratory.
2.
Bertelson, P., & de Gelder, B. ( 2004). The psychology of multimodal perception. In C. Spence & J. Driver (Eds.), Crossmodal space and crossmodal attention (pp. 141-177). New York, NY: Oxford University Press.
3.
Colonius, H., & Arndt, P. ( 2001). A two-stage model for visual-auditory interaction in saccadic latencies. Perception & Psychophysics, 63, 126-147.
4.
Eriksson, L., Berglund, A., Willen, B., Svensson, J., Petterstedt, M., Carlander, O., . . . Allerbo, G. ( 2008). On visual, vibrotactile, and 3D audio directional cues for dismounted soldier way-point naviation. In Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting (pp. 1282-1286). Santa Monica, CA: Human Factors and Ergonomics Society.
5.
Groh, J.M., & Werner-Reiss, U. (2002). Visual and auditory integration . In V. S. Ramachandran (Ed.), Encyclopaedia of the human brain (Vol. 4, pp. 739-752). San Diego, CA: Academic Press.
6.
Hutton, S.B., & Tegally, D. ( 2005). The effects of dividing attention on smooth pursuit eye tracking. Experimental Brain Research, 163, 306-313.
7.
Jack, C.E., & Thurlow, W.R. ( 1973). Effects of degree of visual association and angle of displacement on the "ventriloquism" effect. Perceptual and Motor Skills, 37, 967-979.
8.
Lajoie, Y., Teasdale, N., Bard, C., & Fleury, M. ( 1993). Attentional demands for static and dynamic equilibrium . Experimental Brain Research, 97, 139-144.
9.
Laramee, R.S., & Ware, C. ( 2002). Rivalry and interference with a head-mounted display . ACM Transactions on Computer-Human Interaction, 9, 238-251.
10.
Leigh, R., & Zee, D. ( 1999). The neurology of eye movements (3rd ed.). New York, NY: Oxford University Press.
11.
Liu, D., Jenkins, S., & Sanderson, P.M. (2009). Patient monitoring with head-mounted displays. Current Opinion in Anaesthesiology , 22, 796-803.
12.
Liu, D., Jenkins, S., Sanderson, P., Watson, M., Leane, T., Kruys, A., & Russell, W.J. ( 2009). Monitoring with head-mounted displays: Performance and safety in a full-scale simulator and part-task trainer. Anesthesia & Analgesia, 109, 1135-1146.
13.
Paige, G.D., Telford, L., Seidman, S.H., & Barnes, G.R. ( 1998). Human vestibuloocular reflex and its interactions with vision and fixation distance during linear and angular head movement. Journal of Neurophysiology, 80, 2391-2404.
14.
Pheasant, S., & Haslegrave, C.M. (2006). Bodyspace: Anthropometry, ergonomics and the design of work (3rd ed.). London, England: Taylor and Francis.
15.
Riecke, B.E., Väljamäe, A., & Schulte-Pelkum, J. (2009). Moving sounds enhance the visually-induced self-motion illustion (circular vection) in virtual reality . ACM Transactions on Applied Perception, 6(2), 7:1-7:27.
16.
Sanderson, P.M., Watson, M.O., Russell, W.J., Jenkins, S., Liu, D., Green, N., . . . Krupenia, S. ( 2008). Advanced auditory displays and head-mounted displays: Advantages and disadvantages for monitoring by the distracted anesthesiologist . Anesthesia & Analgesia, 106, 1787-1797.
17.
Saville, D.J. ( 1990). Multiple comparison procedures: The practical solution . American Statistician, 44, 174-180.
18.
Schweigart, G., Mergner, T., & Barnes, G. ( 1999). Eye movements during combined pursuit, optokinetic and vestibular stimulation in macaque monkey. Experimental Brain Research, 127, 54-66.
19.
Sharpe, J.A., & Johnston, J.L. (1993). The vestibulo-ocular reflex: Clinical, anatomic, and physiologic correlates. In J. A. Sharpe & H. O. Barber (Eds.), The vestibulo-ocular reflex and vertigo (pp. 15-39). New York, NY : Raven Press.
20.
Slutsky, D.A., & Recanzone, G.H. (2001). Temporal and spatial dependency of the ventriloquism effect. Neuroreport, 12, 7-10.
21.
Sparrow, W.A., Bradshaw, E.J., Lamoureux, E., & Tirosh, O. ( 2002). Ageing effects on the attention demands of walking. Human Movement Science, 21, 961-972.
22.
Teramoto, W., Watanabe, H., & Umemura, H. ( 2008). Change of temporal-order judgment of sounds during long-lasting exposure to large-field visual motion. Perception, 37, 1649-1666.
23.
Teramoto, W., Watanabe, H., & Umemura, H., Matsuoka, K., & Kita, S. ( 2004). Judgment biases of temporal order during apparent self-motion . IEICE Transactions on Information and Systems, E87D, 1466-1476.
24.
Thompson, M.B., & Sanderson, P.M. (2008). Multisensory integration with a head-mounted display: Sound delivery and self-motion. Human Factors, 50, 789-800.
25.
Thompson, M.B., Tear, M.J., & Sanderson, P.M. (2010). Multisensory integration with a head-mounted display: Role of response modality. Human Factors, 52, 92-104.
26.
Thurlow, W.R., & Jack, C.E. ( 1973). A study of certain determinants of the ventriloquism effect. Perceptual and Motor Skills, 36, 1171-1184.
27.
Väljamäe, A. ( 2009). Auditorily-induced illusory self-motion: A review. Brain Research Reviews, 61, 240-255.
28.
Veltman, J.A., Oving, A.B., & Bronkhorst, A.W. (2004). 3-D audio in the fighter cockpit improves task performance. International Journal of Aviation Psychology, 14, 239-256.
29.
Wickens, C.D. ( 2002). Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3, 159-177.
30.
Williams, I., Mulhall, L., Mattingley, J., Lueck, C., & Abel, L. ( 2006). Optokinetic nystagmus as an assessment of visual attention to divided stimuli. Journal of Clinical Neuroscience, 13, 828-833.
31.
Zampini, M., Guest, S., Shore, D.I., & Spence, C. ( 2005). Audiovisual simultaneity judgments. Perception and Psychophysics, 67, 531-544.
32.
Zotkin, D.N., Duraiswami, R., Grassi, E., & Gumerov, N.A. ( 2006). Fast head-related transfer function measurement via reciprocity . Acoustical Society of America, 120, 2202-2215.
