Abstract
Recent findings from social attention research suggest direct engagement with others is a necessary condition for the social cognitive development of both autistic children and their typically developing peers. These findings come from studies that have used eye-tracking technology and paradigms for measuring social attention in naturalistic, real-time settings. Social interactions allow two social partners to coordinate their attention in order to understand each other. Using a framework for studying social cognitive development and social attention in the context of social interactions, this article proposes the use of eye-tracking paradigms for social attention research and presents an intervention called second-person interactions (SPI). The article provides a review of eye-tracking research and interventions for social attention and describes the methods for assessing social attention using the SPI intervention.
Keywords
Introduction
Used to study eye movements and gaze behavior in multiple research disciplines, eye tracking is becoming a ubiquitous technology and expected to become more widespread in the future (Holmqvist et al., 2022). Despite its widespread use, eye tracking is relatively new in the education field (Jarodzka et al., 2021). As an emerging technology, eye tracking has the potential to significantly impact how educators assess complex phenomena, including human interaction and the subtle yet dynamic manifestations of interrelated behaviors that change across time and contexts (Bacos, 2020a).
The study of social attention, psychological processes that enable human social interaction, is also growing. Social attention has been discussed in the literature since the 1970s (Chance & Larsen, 1976), and more recently, because of advances in eye tracking, social attention research has seen tremendous growth in the past 20 years. While much of the growth has been in the developmental, neuroscience, and clinical literature (Salley & Colombo, 2016), the use of eye tracking for education research (Jarodzka et al., 2017) and teaching and learning is also expanding (Jarodzka et al., 2021).
Eye tracking has shown promise in intervention research targeting social attention skills for autistic individuals (Wang et al., 2020). Autism spectrum disorder (ASD) is diagnosed when both “persistent deficits in social communication and social interaction” and “restricted, repetitive patterns of behavior, interests, or activities” are present in early development (American Psychiatric Association, 2022, Autism Spectrum Disorder section). The diagnostic criteria for ASD includes “deficits in nonverbal communicative behaviors used for social interaction” such as “abnormalities in eye contact” (American Psychiatric Association, 2022, Autism Spectrum Disorder section). A diagnosis of ASD is a participant inclusion criterion for many of the studies discussed in this paper. When referring to the target population in these studies, some authors use people-first language (e.g., children with autism). However, considering recent survey findings that indicate a preference for identity-first language (e.g., autistic children) (Taboas et al., 2023) and evidence that suggests people-first language may be more stigmatizing (Gernsbacher, 2017), the authors of this paper have chosen to use identity-first language.
Eye tracking offers several affordances useful to assessing social attention in contexts that are especially challenging for autistic individuals (Frazier et al., 2017). For example, eye tracking affords the ability to capture dynamic gaze data during social interactions that may be used to determine how social attention behaviors are related to real-life social behaviors (van Rijn et al., 2019). In addition to capturing dynamic measures in real time, eye tracking has significant potential for enhancing interventions for social interaction (Vernetti et al., 2018).
Social interaction is a key context for studying social attention using eye tracking. Social attention has several different conceptualizations. In the social attention literature, the construct is generally categorized into three functions: (a) social behavior, (b) social motivation, and (c) social visual attention (Salley & Colombo, 2016). Joint attention, the social behavior function, serves the role of communicating and coordinating attention (Mundy et al., 2000). Social motivation is the tendency to visually orient toward social information (Chevallier et al., 2012). While social visual attention is orientation toward social versus nonsocial information in the context of other social partners or agents (Chita-Tegmark, 2016). In the context of social interactions, social attention has been conceptualized as dynamic and bidirectional processes for coordinating shared attention behaviors and social understanding (Hoehl & Bertenthal, 2021). This conceptualization of social attention comes from the second-person framework.
Theoretical Framework
The second-person framework is a theoretical approach from the social neuroscience field (Schilbach et al., 2013) that has advanced in recent years to examine how both social cognitive and behavioral mechanisms, including social attention behaviors, are fundamentally different during social interaction compared to social observation (Redcay & Schilbach, 2019). The second-person framework emphasizes that directly engaging in social interactions with others is fundamental to how social cognition develops (Schilbach et al., 2013). That is, social cognition arises primarily from second-person perspectives gained from interacting with others (i.e., second-person interactions) instead of third-person observations of others (see Figure 1). Measuring Social Attention from Second-Person and Third-Person Perspectives. Note. The top row illustrates a second-person perspective of social attention. The 3-panel format affords a temporal view of the social interaction, however, subtle changes in social attention are not depicted. The bottom row illustrates a third-person perspective.
A third-person perspective is used to measure an individual’s responses when they are observing others rather than interacting with them (Redcay & Saxe, 2013). While third-person approaches are still needed for social cognitive research, they are insufficient. The detached third-person perspective fails to recognize the reciprocal attributions social partners make from a second-person perspective to interpret, predict, and influence each other’s behavior while the interaction is unfolding (Hoehl & Bertenthal, 2021). Therefore, a second-person approach (i.e., measuring social attention behavior during social interactions) is needed to study how social cognitive processes emerge and integrate with social attention.
Eye-tracking methods using a second-person approach afford a dynamic view of social attention processes, capturing how they emerge in real time during social interactions (Schilbach et al., 2013). When designed to measure social attention during social interaction, eye-tracking paradigms can be used to study the bidirectional nature of social attention processes that arise from the interaction (Hoehl & Bertenthal, 2021; Mundy & Bullen, 2022). These bidirectional processes may be understood using the dual function of gaze framework. This framework conceptualizes social attention as processes for both channeling information and signaling communication to others (Argyle & Cook, 1976). That is, individuals can alternate between selectively attending to social information (i.e., channeling) or communicating social information (i.e., signaling) using social attention behaviors (i.e., joint attention).
Social attention processes may also be understood as bidirectional in their relation to social cognitive processes (see Figure 2). Alternating between channeling and signaling may involve bottom-up and top-down processing systems (Klein et al., 2009). People form impressions of others using bottom-up processes related to social attention (i.e., reflexive orienting to a socially salient stimulus) and top-down process related to social cognition (i.e., action- and goal-oriented processes) (Tatler et al., 2017). Alternating between the use of these two systems during social interactions suggests a bidirectional relation between social attention and social cognition in social contexts (Mundy & Bullen, 2022). The systems of bottom-up and top-down processing may explain the atypical patterns of gaze behavior measured in eye-tracking studies of autistic individuals (Falck-Ytter et al., 2022). Specifically, the decrease in gaze seen in autistic individuals in response to social stimuli in the context of social interactions may be the result of challenges associated with integrating perception (i.e., bottom-up) and action-based (i.e., top-down) processes (Bolis & Schilbach, 2018). However, more research is needed to understand how these two systems emerge during social interactions and why autistic individuals have difficulties with social attention in interactive contexts. Bidirectional Relation Between Social Attention and Social Cognition. Note. Social partners alternate between bottom-up processes related to social attention (i.e., attending to a stimulus) and top-down processes related to social cognition (i.e., action- and goal-oriented processes). These bidirectional processes mediate the social interaction during which the social partners coordinate their attention to promote engagement and social understanding. While this figure affords a visual depiction of the bidirectional nature of social attention, a limitation is the figure does not show the temporal changes as the interaction is unfolding.
Eye-Tracking Research
Early methods of recording eye movements were elaborate and difficult to setup. New technologies developed in the 1970s simplified the setup for expanding research opportunities. As a result eye-tracking studies of social cognition began in the 1990s (Vecera & Johnson, 1995) and have since expanded into studies of social attention in autistic and typically developing individuals (Frazier et al., 2017; Puce & Bertenthal, 2015).
Eye-Tracking Research on Social Attention
Eye-tracking technology has enabled the design of innovative paradigms for researching social attention and has shown potential for analyzing individual differences in how and why social attention develops (Hoehl & Bertenthal, 2021). Growing evidence from eye-tracking studies over the last two decades supports a link between real-world social difficulties seen in autistic individuals and atypical gaze patterns detected using eye-tracking measures such as reduced gaze toward human interaction stimuli (Chita-Tegmark, 2016; Frazier et al., 2017; Papagiannopoulou et al., 2014; Wall et al., 2023). An important affordance of eye tracking is that it can detect subtle differences in social attention in autistic individuals compared to typically developing individuals. For instance, a constellation of eye-tracking measures can be used to analyze nuanced social attention differences (e.g., variability in gaze behavior associated with social communication impairments) that may contribute to a better understanding of how they impact social functioning in autistic individuals (Nayar et al., 2022).
Eye-Tracking Measures of Social Attention
Eye-Tracking Measures of Social Attention.
To investigate how and why social attention develops, research is beginning to shift to new paradigms. A second-person approach to social attention research may offer new insights on measures of social attention that help explain how and why it changes across individuals and contexts, and over time (Hoehl & Bertenthal, 2021). Given the complexity of investigating social attention processes, multiple measures are needed. Multimodal research may offer new avenues for investigating the interrelated functions of social attention (i.e., social behavior, social motivation, and social visual attention). Though measured individually, the functions should be measured together to understand the interactive nature of social attention development (Mundy & Bullen, 2022). Rather than rely on eye tracking alone, a multimodal setup can integrate eye tracking with other modalities for conducting research. For example, iMotions, is an eye-tracking platform that can integrate several biosensors (e.g., facial expression analysis and EEG) into a single computer (Pedersen, 2021). For social attention, the use of eye tracking may be combined with video recording to capture verbal responses and gestures that may provide additional information and validity to the research.
Methodological Challenges
An important methodological challenge to address when using eye tracking for research is to design paradigms for measuring social attention in the context of social interaction. In addition to addressing the ecological validity of the eye-tracking paradigm used, participant characteristics (e.g., physiology of the eye and age) should also be considered in relation to the quality of eye-tracking data collected. For example, eye-tracking data from Asian participants tend to be worse quality because of the narrower eyelid aperture found in that population (Blignaut & Wium, 2014). Another consideration is how to address data loss. For example, children are more likely to look away from the monitor, which may count as data loss, however it is recommended that those periods of time looking away should be excluded (Hessels et al., 2015).
Depending on the research question and how the constructs under investigation are defined and operationalized, the use of different experimental designs and types of eye trackers should be considered (Holmqvist et al., 2011, 2022). Remote eye trackers, also referred to as screen-based or stationary eye trackers, are a type of video-based eye tracker that can view the participant’s eyes and keep track of their eye movements from a distance (Holmqvist et al., 2011). Often placed on a table in front of the study participants while stimuli are presented on a monitor, remote eye trackers generally do not require contact with participants. Remote eye trackers are easy to setup and frequently used because they can measure where the participant looks while enabling free head movement (Niehorster et al., 2018). Head-mounted eye trackers may offer an alternative, more ecologically valid approach to record gaze behaviors of participants in real-life settings (Franchak et al., 2011; Noris et al., 2012). Head-mounted systems are eye trackers mounted on top of a helmet, cap, head-band, or a pair of glasses (Holmqvist et al., 2011). Compared to other eye trackers, they allow maximum mobility. Compared to remote eye trackers, head-mounted eye trackers have lower resolution, a term that refers to the precision in detecting the smallest movement of the eyes (Holmqvist et al., 2011).
Given the complexity of the processes involved in attending to a stimulus, researchers can gain better insights from controlled experimental settings provided by remote eye tracking systems compared to the less controlled and lower-resolution measures provided by head-mounted eye trackers (Hoehl & Bertenthal, 2021). From more controlled experimental settings, researchers can gain a more precise view of how and in what moments an individual attends to a social partner’s actions and gaze direction.
Eye-Tracking Paradigm for Simulated Social Interactions
One way to address limitations of remote eye trackers is to design stimuli from a first-person perspective to simulate social interactions (Hoehl & Bertenthal, 2021). Video stimuli from a first-person perspective (e.g., recordings of an actor talking directly to the camera) may be used to elicit the participant’s social attention behaviors from a second-person perspective (e.g., the participant responding to gaze cues directed by the actor in the video). This method addresses the limitations of remote eye tracking by providing a naturalistic context through which dynamic processes of social attention can be recorded and analyzed.
Eye-Tracking Paradigm for Social Attention Research and Interventions
The eye-tracking paradigm for social attention research and interventions proposed in this article is based on recommended methods of applying a second-person approach to study social interactive behaviors from the neuroscience (Redcay & Schilbach, 2019) and developmental psychology literature (Boyer et al., 2020; Hoehl & Bertenthal, 2021). As an approach to social attention intervention from a second-person perspective, this article proposes an eye-tracking paradigm that uses video stimuli of avatars and eye-movement modeling. Components of the proposed paradigm are based on methods recommended for a social attention intervention that uses eye tracking, eye-movement modeling, and simulated dyadic social interactions (Bacos, 2020b). The avatars in the proposed paradigm serve as social partners that elicit social attention behaviors of participants by using child-directed speech and joint attention initiations from a first-person perspective to simulate social interactions. Eye-movement modeling is added to the video stimuli to assess and guide participants’ social attention in response to simulated interactions with the avatars. The use of this paradigm not only assesses social attention behaviors from a second-person perspective but allows for comparisons between the gaze behavior of autistic children and their typically developing peers through visual analysis of their gaze patterns. Avatars as social partners and eye-movement modeling as social attention guides are described next as second-person approaches to eye-tracking paradigms for social attention interventions.
Avatars as Social Partners
The use of avatars in virtual environments is a powerful method for simulating social interaction (Bombari et al., 2015). An avatar is a computer-generated digital representation of a real person (i.e., human-avatar) or a synthetic one (i.e., agent-avatar) (Blascovich et al., 2002). Virtual humans are avatars designed to resemble humans in appearance and behavior (Bombari et al., 2015). Given that one’s social attention behavior depends on the actions of the other social partner (Boyer et al., 2020), to simulate social interactions an avatar needs to display human movements and social cues (e.g., head and eye movements) that can elicit social attention behaviors. This article uses the term avatar to mean a virtual human with some aspects controlled by humans (i.e., the avatar’s head and eye movements and facial expressions) and others controlled by the computer (i.e., the avatar movements are prerecorded as videos and played by the computer in a programmed sequence).
There are several benefits to using animated avatars to create the video stimuli. A recent study found that autistic individuals were more likely to engage in interactions with live animation avatars than with humans (Kellems et al., 2022). This finding suggests interaction with avatars compared to humans promotes higher levels of engagement for autistic individuals.
Standardization is another benefit of avatars as it allows for the central components of the video stimuli (i.e., eye movements and joint attention cues) to be controlled across intervention sessions (Bombari et al., 2015). Differences in participant behaviors observed across sessions cannot be attributed to a change in the actor’s style of nonverbal behaviors (e.g., speed and direction of shifting gaze) because it is replicated with fidelity across videos. What may vary is the appearance of the avatars and the images cued for joint attention. The use of various avatars may be necessary to control for testing effects and to maintain the participant’s interest.
The use of avatars in the proposed eye-tracking paradigm for social attention intervention aligns with recommendations for applying a second-person approach in at least two ways. First, the use of avatars for simulating social interaction provide control over a participant’s social partners by standardizing them for consistency in terms of their behavior, appearance, and the environmental characteristics in which they are presented (Bombari et al., 2015). Such manipulations of social partners and environmental stimuli would be impossible in real-world settings. Second, the use of avatars and virtual environments address the constraints of observing behaviors during social interactions as they occur naturally in real life (Bombari et al., 2015). Rather than wait for social situations to present themselves, simulations of social interactions can be created to elicit the behaviors targeted for intervention.
Eye-Movement Modeling as Social Attention Guides
Eye-movement modeling added to the video stimuli of avatars used to simulate social interactions may be used to guide participants’ social attention behaviors (Bacos, 2020b). This approach to modeling behavior is based on eye-movement modeling examples (EMME) from the learning sciences (van Gog et al., 2009) and multimedia learning literature (Mayer, 2010). EMMEs are a novel form of video modeling that uses eye tracking to improve performance on a variety of tasks (e.g., medical diagnosis tasks; Brams et al., 2021) and promote learning in several academic domains, including STEM fields (e.g., biology; Krebs et al., 2021). Similar to video modeling, an evidence-based practice (Qi et al., 2018) based on social learning theory (Bandura, 1977), EMMEs support observational learning using video recordings of a model demonstrating how to perform a task. The novel element in EMMEs is the use of eye tracking to add a graphic or type of eye-movement display (e.g., dot, circle, or spotlight) to the video to signal the model’s eye movements when performing the task (Jarodzka et al., 2012, 2013). This element allows researchers to study how EMMEs can both guide learners while viewing complex stimuli and provide insight into the use and regulation of cognitive processes during multimedia learning (Jarodzka et al., 2013; van Gog et al., 2009).
Based on characteristics of EMMEs, a method for enhancing social attention through eye-movement modeling was presented as a potential intervention approach (Bacos, 2020b). The eye-movement modeling method was described as an eye-tracking paradigm designed to assess and guide the social attention of autistic children and typically developing children when engaged in simulated social interactions.
An advantage of the eye-movement modeling method for guiding social attention is that it can serve as a visual support that explicitly cues the participant to attend to relevant stimuli in real time as the social interaction is happening. A visual support is a visual display (e.g., a circle overlaying an image on a screen) that supports the learner in engaging in a target behavior (e.g., looking at the relevant stimuli) without the need for additional prompts (Hume et al., 2021). Visual supports for autistic children are particularly important (Hume et al., 2014). For example, they may help autistic children by making concepts difficult to understand more concrete. The eye-movement modeling method using visual supports to guide social attention is presented as a second-person approach to the proposed eye-tracking paradigm for the social attention intervention described in the subsequent sections.
Second-Person Interactions (SPI) Intervention
The following intervention, second-person interactions (SPI), is presented as an exemplar of the proposed eye-tracking paradigm described in the previous section. Using a second-person approach, the aim of the intervention is to assess and guide the participant’s social attention in the context of simulated social interactions. The SPI intervention is designed to provide interactive contexts in which the social attention of autistic children can be measured using eye tracking and guided using a typically developing peer’s modeled eye movements as a visual support and a method for visual analysis. To create interactive contexts, the intervention uses videos of avatars to elicit the participant’s social attention. Eye-tracking equipment and software are used to record and analyze the social attention measures defined in the intervention.
The methods for developing and implementing the SPI intervention described in this section are based on the recommendations from the literature on second-person approaches to eye-tracking paradigms (Bacos et al., 2023; Boyer et al., 2020; Hoehl & Bertenthal, 2021; Redcay & Schilbach, 2019) and the eye-movement modeling method discussed previously in this article (Bacos, 2020b).
To support the feasibility of implementing the intervention to practice settings, the methods proposed for the SPI intervention are based on the use of low-cost eye-tracking equipment and analysis procedures. While this limits the level of analysis recommended for research, the methods are sufficient for assessing the target behaviors identified for the intervention.
Target Population
The SPI intervention is designed for autistic children and their typically developing peers aged 3–8 years. Participants may benefit from eye-movement modeling guides generated by their typically developing peers to improve their social interactions with each other. Autistic children who meet the following criteria may benefit from the intervention: • received a formal diagnosis of ASD and receives special education services through an individualized education program (IEP) eligible under the category of autism • has no significant impairments in vision and hearing • has an individualized education program which identifies goals for improving social behavior skills • based on classroom observations and reports from parents and teachers, (a) exhibits atypical gaze patterns, (b) has difficulty reading social cues (e.g., does not notice when classmate is excited or upset), and/or (c) has poor joint attention skills (e.g., does not follow the eye gaze of others or look at objects when others are looking at them) • has visual discrimination and language skills (i.e., can discriminate between images and select one when prompted) • has the ability to stay seated and watch a video for at least 10 minutes
Intervention Setting
The intervention may be conducted in a variety of settings, including the classroom and a clinic setting. The intervention can be set up in a small room equipped with a small table, chair, and equipment to record the target behaviors of the child. Any distracting stimuli (e.g., learning materials such as books or toys) should be removed from the area where the child will be seated so that only the necessary equipment and furniture will be visible to the child.
Materials
Equipment and Software
The following equipment and software are recommended for running the intervention: (a) a desktop or laptop computer capable of running the (b) Tobii Eye Tracker 4C, (c) the Tobii Eye Tracking Core Software, (d) Tobii Ghost software, (e) Open Broadcaster Software, and (f) VLC. These software applications are described in the next sections. To capture additional data, a webcam may also be set up to record participant responses to the video stimuli.
Tobii Eye Tracker 4C
The Tobii Eye Tracker 4C (Tobii, 2017) is a low-cost screen-based eye tracker (i.e., also referred to as a remote eye tracker) that operates at a distance of 20–37 inches at a frequency of 90 Hz, which means it tracks where the user is looking 90 times per second. In addition to tracking gaze, the 4C also tracks head movements. To use the tracker for research, a Tobii Pro Upgrade Key can be installed to provide extended access to eye-tracking data, including gaze data for the left and right eye (i.e., X and Y coordinates of the gaze point).
Tobii Eye Tracking Core Software (Version 2.16)
The Tobii Eye Tracker 4C requires installation of the Tobii Eye Tracking Core Software (Tobii, 2018). The software is needed to run the calibration process each time the eye tracker is used.
Tobii Ghost (Version 1.14.1)
Formerly named Streaming Gaze Overlay, Tobii Ghost (Tobii, 2023) is an application that integrates with the Tobii Eye Tracker 4C and Open Broadcaster Software to generate an overlay (i.e., graphic representing the user’s tracked eye movements). The overlay can be set up to appear as a visible layer on top of any media (e.g., video) that is running on the user’s computer.
OBS Studio (Version 29.0.2)
OBS Studio (also known as Open Broadcaster Software (OBS), 2023) is free and open-source software for video recording and live streaming. It provides high-performance real-time video and audio capturing and mixing.
VLC (Version 3.0.8)
VLC (VideoLAN, 2023) is a free and open-source cross-platform multimedia player. It can be used to create a playlist of videos for the participants to watch as part of the intervention. It is also useful for video analysis as it allows playing videos frame by frame using hotkeys (e.g., pressing “e” on the keyboard will jump to the next frame). Hotkeys and the length of jumps forward and backward in the video can be customized.
Video Stimuli
Videos can be used to simulate social interactions and provide a context for assessing social attention behaviors as they unfold in real time (Boyer et al., 2020). The stimuli for the SPI intervention are created using animated avatars whose human-controlled movements and speech are prerecorded as videos. As previously discussed, an avatar is a computer-generated digital representation of a real person (i.e., human-avatar) or a synthetic one (i.e., agent-avatar) (Blascovich et al., 2002). The avatars generated for this intervention are controlled by both humans (i.e., head and eye movements and facial expressions) and the computer (i.e., prerecorded videos of the avatar’s movements are run on the computer in a programmed sequence).
A commercially available option for generating avatars is Apple’s Messages app. Using the app on an iPhone or iPad Pro, users can generate an animated avatar (i.e., Memoji). The appearance of the avatar can be customized by selecting options for different skin tones, hairstyles, eyes, and other physical attributes. The movements of the avatar are controlled by the user through a feature of the app that uses the iPhone or iPad Pro’s cameras to mirror the eye and head movements as well as the facial expressions of the user. The user’s animated avatar and voice can be recorded as a video using the Messages app.
Avatars for Simulated Social Interactions
Different avatars can be generated to create videos that simulate social interactions between the child participant and the avatar as the social partner. The avatar can elicit social attention behaviors (i.e., joint attention) from the participant through child-directed speech (e.g., telling a story) and gaze cueing, which is looking at an object to communicate their interest in it (Frischen et al., 2007). Directing their gaze toward an object in the scene signals to the participant to look where the avatar is looking. Clip art images representing objects of interest (e.g., image of an apple) can be added to the scene to allow opportunities for the avatar to talk about the object as part of a story and produce gaze cues for the participant to follow. Video recordings of the animated avatar looking in a specific direction can be generated. Then basic video editing and presentation software can be used to insert the avatar recordings in front of a background displaying the selected images. These scenes can be presented as short segments or combined into a longer video. Multimedia players such as VLC (VideoLAN, 2023) can be used to create video playlists that run the videos in a specific sequence.
Eye-Movement Modeling Guides
While video stimuli simulating social interactions can be designed to elicit social attention behaviors in a real-time context, a visual support (i.e., eye-movement modeling) can be added to these videos to guide the participant (Bacos, 2020b). Using Tobii Ghost and OBS Studio (see Equipment and Software), eye-movement modeling (EMM) as a social attention guide can be created by recording a model’s eye movements while watching the videos. Tobii Ghost generates the EMM guide based on the viewer’s eye movements and OBS Studio records a video output that can be reviewed for later analysis. Typically developing individuals with whom the participant has opportunities to interact with (e.g., peer or teacher) are recommended to model the social attention behaviors in response to the video stimuli. Before capturing their eye movements to produce the EMM guides, models should be instructed to give their full attention to the avatar in the video (i.e., do not avert gaze outside the social stimuli or the video display) and follow all social cues (e.g., follow the avatar’s joint attention cues).
Measures
Social Attention Behavior
The primary target behaviors of the SPI intervention are the participant’s social attention behaviors (i.e., joint attention) during interactions with the avatar in the video. Two measures of joint attention are taken: (a) responding to joint attention (RJA) and (b) initiating joint attention (IJA). The RJA measure is the participant’s response to the avatar’s joint attention cues recorded as the frequency of looking at the object of interest (i.e., object at which the avatar is looking). The IJA measure is the participant’s initiations of gaze cues when verbally prompted by the avatar’s question (e.g., What did I forget?). During the scene when the avatar faces forward, waiting for a response, the IJA measure is recorded as the participant’s frequency of transitions from looking at the avatar’s face and the previously cued object of interest.
Social Motivation
A secondary target behavior is the participant’s social motivation. The social motivation measure is the participant’s response to social versus nonsocial stimuli presented in the videos. The measure is recorded as the frequency of looking at social stimuli (i.e., avatar’s face and objects cued via joint attention) versus nonsocial stimuli (i.e., all other areas in the scene).
Social Visual Attention
The SPI intervention is designed to give the participant the ability to see gaze responses to the video stimuli modeled by another person using an eye-movement modeling (EMM) guide. To test the visual salience of the EMM guide, the intervention tracks the participant’s visual attention to the guide. Thus, a tertiary target behavior is recorded as the frequency of looking at the EMM guide.
Procedures
Defining Areas of Interest to Measure Social Attention
An area of interest (AOI) is “a segment of a stimulus space (often defined by screen pixel boundaries in monitor-based eye tracking)” that identifies a meaningful area of the stimulus (Holmqvist et al., 2022, p. 368). Using an AOI approach, social attention measures can be analyzed by calculating the frequency and duration the participant looks at the defined AOI (e.g., the avatar’s face). Fixation refers to a period of time when the participant’s gaze remained at the same location with respect to the gaze target (Jovancevic-Misic & Hayhoe, 2009). A measure of individual fixations provides insight into the bottom-up (e.g., selectivity based on stimulus salience) and top-down (e.g., action-oriented goals) processes that modulate the number and duration of fixations (Tatler et al., 2017). These processes involve encoding the stimulus information and deciding both where one looks and when one moves one’s eyes (Tatler et al., 2017).
Eye-Tracking Measures for Areas of Interest.
Note. The measures of social attention constructs are calculated by the defined areas of interest (AOI).
Data Collection
The same method used to create the eye-movement modeling (EMM) guides is used to collect participant gaze data and conduct a visual analysis of target behaviors. Tobii Ghost and OBS studio (see Equipment and Software) are used to create the dynamic visual overlay representing the participant’s eye movements (i.e., participant EMM) in response to the video stimuli. Using this method, the participant’s gaze data are collected in response to the videos with the EMM guide. For example, in Figure 3(a), the blue bubble in the upper left corner, located over the object of interest (i.e., clipart image), represents the visual support (i.e., EMM guide). The red bubble closely following the EMM guide represents the participant’s EMM (i.e., participant’s gaze location). In Figure 3(b), the bubble in the upper left corner represents the participant’s EMM in response to the avatar’s question. No EMM guide appears during segments of the video when IJA is assessed because the participant’s independent initiated gaze (i.e., without the guide) is required for the IJA measure. For baseline measures of social attention, the participant’s gaze data can be collected in response to the videos without the EMM guide. Joint Attention Behaviors Measured in the Intervention. (a) responding to joint attention (RJA). (b) initiating joint attention (IJA). Note. The objects of interest (i.e., clipart images) appearing in the four corners of each scene were redacted due to copyright restrictions.
Code Sheet
The following code sheet is provided as an example of how to code and record the social attention measures defined for the intervention (see Figure 4). The gaze behaviors are coded across time (i.e., in seconds) for the participant and the model in relation to the joint attention cues in the video (i.e., avatar’s gaze location). Code Sheet for Data Collection and Analysis. Note. Gaze locations for the participant (i.e., participant EMM) are coded with the following labels: (a) face (i.e., avatar’s face), (b) object RJA (i.e., responding to joint attention), (c) IJAs: face-object (i.e., initiating joint attention), (d) NS (i.e., nonsocial gaze regions), and (e) gaze at EMM per second (i.e., participant’s gaze location with respect to the EMM guide).
Data Sample
Data Sample From the SPI Intervention.
Note. The AOI time is the total time range in seconds defined for the AOI. The rate is calculated by dividing the frequency (i.e., number of social attention measures counted) by the AOI time.
Implementation Procedures
Recording procedures
The intervention uses the following recording procedures: (a) automated recording through the eye-tracking system, (b) screencast recording through OBS Studio, and (c) participant video recording through a webcam.
Setup
The recommended setup for the intervention (see Figure 5) includes one desktop computer, a two-monitor display set to extended view (i.e., video stimuli presented to the participant on the primary screen and screencast monitoring viewable on the secondary screen), an eye tracker, and software (see Materials). Although not necessary, a webcam may also be used to capture a video recording of the participant while watching the screen. A privacy screen may be placed around the primary screen to shield the participant’s view of the secondary screen and any distracting stimuli. To stabilize the gaze of the participant, a chair with a backrest is recommended to secure the participant between the chair and the table. Eye gaze data (i.e., screencasts of the gaze overlay generated via Tobii Ghost) using the Tobii Eye Tracker 4C are recorded using OBS Studio. Intervention Setup.
Baseline procedures
For baseline measures, versions of the videos without the eye-movement modeling (EMM) guide are presented to participants. Recording the participants’ gaze behavior without the EMM guide allows for assessment of their independent responses. Each session starts with the eye tracker’s calibration process. Then the recording equipment is turned on before playing the video for the participant.
Intervention procedures
All the intervention procedures are the same as the baseline procedures except the participants are presented versions of the videos with the EMM guide.
The steps for implementing the baseline and intervention sessions are as follows: 1. Set up the study equipment and software (see Materials) as shown in Figure 5. 2. On the primary monitor for the participant: (a) open the Tobii Eye Tracking Core Software, (b) open Tobii Ghost and select the settings for recording the gaze overlay, and (c) open VLC to select the video(s) for the session. 3. On the secondary monitor for the researcher, open OBS Studio and select the capture settings: (a) screencast of the Tobii Ghost layer, (b) screencast of the videos selected in VLC, (c) webcam of the participant facing the monitor, and (d) audio from the video. 4. Invite the participant to get seated in front of the primary monitor display. 5. Explain to the participant what they will be doing during the session (i.e., sit still in front of a monitor and watch a video while an eye tracker records what they see). 6. On the primary monitor for the study participant, start the calibration procedure using the Tobii Eye Tracking Core Software. This procedure takes approximately 2 minutes. 7. After completing the calibration, on the primary screen: (a) open the selected video in VLC and set the video to full screen and (b) start recording in OBS Studio. 8. Instruct the participant to sit still and look at the screen and begin playing the video selected in VLC. 9. On the secondary screen, monitor the recording of the screencast, Tobii Ghost overlay, and webcam feed. 10. At the end of the video, stop recording in OBS Studio and ensure the video file has been saved to the computer.
Analysis
The data in the SPI intervention are analyzed manually to assess the participant’s social attention during the simulated social interactions. Manual analysis of data is used for the SPI intervention for several reasons. First, the target behaviors recorded during the intervention can be analyzed visually using the gaze overlay video recordings (i.e., generated by Tobii Ghost and OBS Studio), which can be played frame by frame using a media player (e.g., VLC) to tally the frequency of social attention measures observed. Second, creating algorithms or purchasing software for automated coding of gaze data can be costly and requires specialized skills. In addition, when analyzed manually, the data codes can be edited easily whenever changes are made to the video stimuli. Finally, manual analysis allows for direct monitoring of the quality of the data and assessment of the participant’s responses to the video stimuli, and it allows better monitoring of the participant’s performance and engagement (Holmqvist et al., 2022).
Evaluation
Multiple instruments may be used to evaluate the intervention. For example, the Social Responsiveness Scale—Second Edition (SRS-2; Constantino & Gruber, 2012) may be used to measure pre- and post-intervention differences in behaviors related to social attention. The SRS-2 is a 65-item rating scale (α = .95) that measures social impairment associated with ASD and quantifies its severity. The Behavior Intervention Rating Scale (BIRS; Elliott & Treuting, 1991) may be used to measure the effectiveness of the intervention. The BIRS is a 24-item instrument (α = .97) designed to measure perceptions of treatment acceptability and effectiveness of interventions.
Social validity
In addition, surveys can be designed to gather information from teachers and parents of the child about the social validity of the intervention. A social validity survey for child participants also may be used to collect information about the child’s perceptions about the intervention. In terms of generalization of social attention skills to real-world settings, research suggests social attention skills taught using eye-tracking interventions can generalize to real-world settings. For example, autistic children who received training on a desktop remote eye tracker were able to generalize social attention behavior to real-world classroom interactions (McParland et al., 2021). Whereas the training was effectively implemented using a remote eye tracker, a head-mounted eye tracker was needed to determine generalization. To generalize the SPI intervention, additional research will need to examine not only social attention behaviors in response to stimuli on a screen, but also social attention in the context of real-world interactions with a responsive social partner. In addition to considering the use of head-mounted eye trackers, researchers may consider other methods such as in vivo assessments (e.g., behavioral observations of live interaction) to determine the generalizability of the intervention.
Discussion
Implications for Research and Practice
Some of the challenges and limitations of applying a second-person perspective to eye-tracking research and interventions practice include the need for more ecologically valid settings that increase the likelihood of generalization to real-world environments. One approach that was recently recommended in the literature was to create simulated interactions by presenting videos of actors from a first-person perspective (Boyer et al., 2020). The eye-tracking paradigm proposed in this article uses this approach to simulate social interactions by eliciting social attention behaviors that can be recorded to measure the participant’s response and engagement. In addition, eye-movement modeling was recommended to guide participants’ social attention and to conduct visual analyses of the gaze behaviors of autistic children compared to the gaze behaviors of typically developing peers.
Eye tracking can aid autistic children in several ways. First, it provides a means to capture social attention behaviors that would not be possible through behavioral observation alone. The data collected from eye trackers can help educators who work with autistic children assess where and when support may be needed during social interactions. Second, eye tracking can be used to generate eye-movement models that can guide autistic children as the interaction is unfolding and allow opportunities for repeated practice in a controlled setting. Finally, eye tracking can be combined with other sensors (e.g., galvanic skin response) to capture multimodal data, which collectively, may improve understanding of how social attention manifests differently across individuals and contexts.
Limitations and Future Directions
Some of the limitations of the methods proposed in this article are related to the equipment used in the study. The use of a remote eye tracker, for example, restricts participant movement. While participants can move their head, their body is required to remain stationary because of the limited tracking range the eye tracker can measure the participant’s gaze data in response to the stimuli presented (Holmqvist et al., 2022). Another limitation of using a remote eye tracker is that some gaze behaviors recorded in lab settings such as responses to a specific social cue may not generalize to real-world, real-time social interactions (Franchak & Yu, 2022).
Other limitations include the time-consuming process of designing the video stimuli and manually coding the data that can be analyzed from the participants’ interactions with the intervention. Some of this may be automated using software, however, that would limit opportunities to redesign the stimuli as it may be more difficult to edit once embedded in a computer program. A key factor that has been shown to significantly impact social attention in autistic individuals is the type of stimuli (Frazier et al., 2017). For example, stimuli depicting social interactions seem to be more challenging than other stimuli, including images of people and other complex and dynamic visual scenes. Paradigms that enable iterative design cycles for creating and evaluating stimuli may allow for more innovation in researching and designing interventions for social attention.
The second-person perspective is a new conceptualization that still needs further research. Although recent findings have found support for this view, more evidence is needed to determine whether social cognition and the social attention behaviors that enable social interactions and social understanding emerge primarily from interactions rather than third-person observations. Despite the challenges and limitations of this approach, paradigms following the second-person perspective offer exciting new avenues for research and practice.
Conclusion
This article provided a review of the current literature on eye-tracking research to understand social attention and inform the design of interventions for autistic individuals. Several theoretical frameworks were discussed including the second-person perspective, a new conceptualization of social cognition that has been used to design research based on the view of social attention as dynamic and bidirectional processes that are best observed in the context of social interactions (Hoehl & Bertenthal, 2021). A second-person view emphasizes the need to design research that focuses on the child-peer, child-teacher, or child-caregiver dyad, rather than third-person perspectives based on observation alone (Schilbach et al., 2013). An interactionist perspective will inspire the creation of innovative paradigms for eye-tracking research on social attention as well as interventions that can assess the dynamic and complex nature of social attention behaviors unfolding in real-time settings and during development.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
