Gaze transfer in remote cooperation: Is it always helpful to see what your partner is attending to?

Abstract

Establishing common ground in remote cooperation is challenging because nonverbal means of ambiguity resolution are limited. In such settings, information about a partner's gaze can support cooperative performance, but it is not yet clear whether and to what extent the abundance of information reflected in gaze comes at a cost. Specifically, in tasks that mainly rely on spatial referencing, gaze transfer might be distracting and leave the partner uncertain about the meaning of the gaze cursor. To examine this question, we let pairs of participants perform a joint puzzle task. One partner knew the solution and instructed the other partner's actions by (1) gaze, (2) speech, (3) gaze and speech, or (4) mouse and speech. Based on these instructions, the acting partner moved the pieces under conditions of high or low autonomy. Performance was better when using either gaze or mouse transfer compared to speech alone. However, in contrast to the mouse, gaze transfer induced uncertainty, evidenced in delayed responses to the cursor. Also, participants tried to resolve ambiguities by engaging in more verbal effort, formulating more explicit object descriptions and fewer deictic references. Thus, gaze transfer seems to increase uncertainty and ambiguity, thereby complicating grounding in this spatial referencing task. The results highlight the importance of closely examining task characteristics when considering gaze transfer as a means of support.

Keywords

Gaze transfer Cooperation Pointing Ambiguity Resolution Grounding

Get full access to this article

View all access options for this article.

References

Almor

(1999). Noun-phrase anaphora and focus: The informational load hypothesis. Psychological Review 106, 748–765.

Ballard

D.H.

Hayhoe

M.M.

Pelz

J.B.

(1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience 7, 66–80.

Brennan

S.E.

Chen

Dickinson

C.A.

Neider

M.B.

Zelinsky

G.J.

(2008). Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition 106, 1465–1477.

Bruner

(1981). The pragmatics of aquisition. The child's construction of language Deutsch

New York, NY: Academic Press 42–64.

Carletta

Hill

R.L.

Nicol

Taylor

de Ruiter

J.P.

Bard

E.G.

(2010). Eyetracking for two-person tasks with manipulation of a virtual world. Behavior Research Methods 42, 254–265.

Chafe

W.L.

(1976). Givenness, contrastiveness, definiteness, subjects, topics and point of view. Subject and topic Li

C.N.

New York, NY: Academic Press 25–55.

Clark

H.H.

(1996). Using language Cambridge, UK: Cambridge University Press.

Clark

H.H.

Brennan

S.E.

(1991). Grounding in communication. Perspectives on socially shared cognition Resnick

L.B.

Levine

J.M.

Teasley

S.D.

Washington, DC: APA Books 127–149.

Doherty-Sneddon

Anderson

O'Malley

Langton

Garrod

Bruce

(1997). Face-to-face and video mediated communication: A comparison of dialogue structure and task performance. Journal of Experimental Psychology: Applied 3, 105–125.

10.

Ferreira

V.S.

Slevc

L.R.

Rogers

E.S.

(2005). How do speakers avoid ambiguous linguistic expressions?. Cognition 96, 263–284.

11.

Fukumura

van Gompel

R.P. G.

Pickering

M.J.

(2010). The use of visual context during the production of referring expressions. Quarterly Journal of Experimental Psychology 63, 1700–1715.

12.

Griffin

Z.M.

(2004). Why look? Reasons for eye movements related to language production. The integration of language, vision, and action: Eye movements and the visual world Henderson

J.M.

Ferreira

New York, NY: Taylor and Francis 213–247.

13.

Griffin

Z.M.

Bock

(2000). What the eyes say about speaking. Psychological Science 11, 274–279.

14.

Hanna

J.E.

Brennan

S.E.

(2007). Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language 57, 596–615.

15.

Hansen

J.P.

Andersen

A.W.

Roed

(1995). Eye-gaze control of multimedia systems. Advances in human factors/ergonomics: Symbiosis of human and artifact – Future computing and design for human–computer interaction, Proceedings of the Sixth International Conference on Human–Computer Interaction Anzai

Ogawa

Mori

Amsterdam: Elsevier Science B.V 20, 37–42.

16.

Helmert

J.R.

Pannasch

Velichkovsky

B.M.

(2008). Influences of dwell time and cursor control on the performance in gaze driven typing. Journal of Eye Movement Research 2, 1–8.

17.

Jacob

R.J. K.

(1991). The use of eye movements in human–computer interaction techniques: What you look at is what you get. ACM Transactions on Information Systems (TOIS) 9, 152–169.

18.

Kaur

Tremaine

Huang Wilder

Gacovski

Flippo

(2003). Where is “it”? Event synchronization in gaze–speech input systems Paper presented at the 5th International Conference on Multimodal Interfaces New York, NY.

19.

Kibrik

A.A.

Dobrov

G.B.

Loukachevitch

N.V.

Zalmanov

D.A.

(2010). Referential choice as a probabilistic multi-factorial process Paper presented at the Fourth International Conference on Cognitive Science Tomsk, Russia.

20.

Koons

D.B.

Sparrell

C.J.

Thorisson

K.R.

(1993). Integrating simultaneous input from speech, gaze, and hand gestures. Intelligent multimedia interfaces Maybury

Menlo Park, CA: MIT Press 257–276.

21.

Land

M.F.

Mennie

Rusted

(1999). The roles of vision and eye movements in the control of activities of daily living. Perception 28, 1311–1328.

22.

Litchfield

Ball

L.J.

Donovan

Manning

D.J.

Crawford

(2008). Learning from others: Effects of viewing another person's eye movements while searching for chest nodules. Proceedings of SPIE Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment Sahiner

Manning

D.J.

San Diego, CA.

23.

Majaranta

Räihä

K.-J.

(2002). Twenty years of eye typing: Systems and design issues Paper presented at the Eye Tracking Research & Applications Symposium, ETRA '02 New York, NY.

24.

Marshall

C.R.

Novick

D.G.

(1995). Conversational effectiveness in multimedia communications. Information Technology & People 8, 54–79.

25.

Mehta

Sadasivan

Greenstein

Gramopadhye

A.K.

Duchowski

A.T.

(2005). Evaluating different display techniques for communicating search strategy training in a collaborative virtual aircraft inspection environment Paper presented at the Human Factors and Ergonomics Society Annual Meeting Orlando, FL.

26.

Monk

A.F.

Gale

(2002). A look is worth a thousand words: Full gaze awareness in video-mediated conversation. Discourse Processes 33, 257–278.

27.

Mulckhuyse

van Zoest

Theeuwes

(2008). Capture of the eyes by relevant and irrelevant onsets. Experimental Brain Research 186, 225–235.

28.

Müller

Helmert

J.R.

Pannasch

Velichkovsky

B.M.

(2011). Following closely? The effects of viewing conditions on gaze versus mouse transfer in remote cooperation Paper presented at the European Conference on Computer-Supported Cooperative Work, ECSCW '11 Aarhus, Denmark.

29.

Neider

M.B.

Chen

Dickinson

Brennan

S.E.

Zelinsky

G.J.

(2010). Coordinating spatial referencing using shared gaze. Psychonomic Bulletin & Review 17, 718–724.

30.

Qvarfordt

Beymer

Zhai

(2005). RealTourist – A study of augmenting human–human and human–computer dialogue with eye–gaze overlay Paper presented at the Human–Computer Interaction Conference, INTERACT '05 Rome, Italy.

31.

Richardson

D.C.

Dale

Kirkham

N.Z.

(2007). The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue. Psychological Science 18, 407–413.

32.

Starker

Bolt

R.A.

(1990). A gaze-responsive self-disclosing display Paper presented at the Conference on Human Factors in Computing Systems, SIGCHI '90 New York, NY.

33.

Stein

Brennan

S.E.

(2004). Another person's eye gaze as a cue in solving programming problems Paper presented at the 6th International Conference on Multimodal Interfaces State College, PA.

34.

Velichkovsky

B.M.

(1995). Communicating attention: Gaze position transfer in cooperative problem solving. Pragmatics and Cognition 3, 199–222.

35.

Velichkovsky

B.M.

(2002). Heterarchy of cognition: The depths and the highs of a framework for memory research. Memory 10, 5/6 405–419.

36.

Velichkovsky

B.M.

Sprenger

Unema

P.J. A.

(1997). Towards gaze-mediated interaction: Collecting solutions of the “Midas touch problem” Paper presented at the Human–Computer Interaction Conference, INTERACT '97 London, UK.

37.

Vertegaal

Velichkovsky

B.M.

Van der Veer

G.C.

(1997). Catching the eye: Management of joint attention in cooperative work. ACM SIGCHI Bulletin 29, 87–92.

38.

Vesterby

Voss

J.C.

Hansen

J.P.

Glenstrup

A.J.

Hansen

D.W.

Rudolph

(2005). Gaze-guided viewing of interactive movies. Digital Creativity 16, 193–204.

39.

Whittaker

(1995). Rethinking video as a technology for interpersonal communications: Theory and design implications. International Journal of Human–Computer Studies 42, 501–529.