Quantifying Visual Attention of Teams During Workload Transitions Using AOI-Based Cross-Recurrence Metrics

Abstract

Cross-recurrence quantification analysis (CRQA) metrics may offer a means to provide information about the quality of collaboration in real-time. The goal of the present work is to use Area of Interest (AOI) based CRQA metrics to analyze the eye-tracking data of 10 pairs who participated in a shared unmanned aerial vehicle (UAV) command and control task. We are interested in how teams respond to workload transitions and how it affects AOI-based CRQA metrics. The results showed that as workload increased, team members spent a longer time on the same task which may indicate that they are coordinating together on a task, or they are not adapting and getting “trapped” in certain tasks. The findings suggest that CRQA AOI-based metrics are sensitive to workload changes and validate these metrics in unraveling the visual puzzle of how workload impacts scanpath patterns which contribute to quantifying the adaptation process of pairs over time. This also has the potential to inform the design of real-time technology in the future.

Keywords

Team Coordination Cross-Recurrence Quantification Analysis Eye Tracking Workload Transitions

Introduction

Data-rich domains such as aviation (Helmreich, 1997), military (Alonso et al., 2006), and healthcare (Despins, 2009) rely on multiple operators to coordinate together and accomplish a shared goal. With technology and automation becoming increasingly complex, systems and organizations are requiring teammates to complete more tasks that rarely stay at one constant level of cognitive workload. Rather, these environments require operators to manage shifts between low and high levels of workload.

There is a need to analyze and account for workload transitions when studying team performance in complex domains (Atweh et al., 2022). To do so, researchers need to find quantitative measures that can provide insights on how teammates collaborate in real time. In recent years, researchers have been using eye tracking technology, an infrared-based technique that provides a trace of people’s eye movements (Lin et al., 2004) to study individual and team responses to workload and stress. Specifically, eye tracking provides output in terms of fixations and saccades. Fixations are spatially stable gaze points during which time visual processing takes place (Poole & Ball, 2006) while saccades are the rapid eye movements in between fixations, during which time no visual processing occurs (Yarbus, 1967). Tracking a pair’s eye movements simultaneously—i.e., dual eye tracking—has been explored to study joint attention in collaborative learning situations (Villamor & Rodrigo, 2022).

Studies that use eye tracking to study pair’s performance and attention allocation often use gaze coupling/overlap which refers to moments when teammates are looking at the same Area of Interest (AOI). Previous work has shown that the coupling of gaze between collaborating partners may improve the quality of interaction and comprehension (Richardson & Dale, 2005), but this is not always the case (Villamor & Rodrigo, 2018). To date, the focus has been on the percentage of cross-recurrent fixations and similarities between the teammates’ trajectories. While these analyses are needed, there is a need to also explore the percentage of identical scanpath segments over time and the average duration the teammates are in synch, especially within the context of workload transitions (El Iskandarani et al., 2023).

Analyzing AOIs in unmanned aerial vehicle (UAV) tasks is important because it can help to improve the effectiveness and efficiency of military operations. UAVs have become increasingly important in military operations because they can be used for a wide range of tasks, such as reconnaissance, surveillance, target acquisition, and weapon delivery. AOI analysis can provide valuable insights into the performance of the UAV system and its ability to meet the overall team’s goal. Moreover, complex domains, such as military operations, can gain a better understanding of the operational environment and identify potential threats or opportunities. In addition, focusing on AOI based analyses can help identify specific tasks where operators are not noticing. This information can then be used to optimize resource allocation and ensure that resources are being used efficiently and effectively (Dindar et al., 2022). Cross-recurrence quantification analysis (CRQA) AOI-centric metrics offer a means to accomplish this goal of supporting operators in real time.

Related Work

Cross-Recurrence Analysis

Cross-recurrence quantification analysis or CRQA is an extension of Recurrence Quantification Analysis (RQA) (Marwan & Kurths, 2002) that is used to quantify how frequently two systems exhibit similar patterns of change or movement in time. CRQA is a useful statistical tool for dynamic systems as it is used to find relation or interrelation between time series and quantify how the similarity between them unfolded over time. It takes two different time series of the same information as input and tests between all points of the first trajectory with all points of the second trajectory forming a cross-recurrence plot (CRP). R_i,j is the recurrence plot as defined by Eq. 1 below (Marwan et al., 2007):

R_{i j} = Θ (\in_{i} - ‖ \vec{x_{i}} - \vec{x j} ‖)

(1)

where x_i and x_j are the phase space trajectories of time series i and time series j respectively. Θ(x) is the Heaviside function and ∈ is the threshold. The states of a natural or engineering dynamic system usually change over time. The state of a system x can be described by its d state variables,

x₁(t), x₂(t), . . ., x_d(t). The vector $\vec{x (t)}$ in a d-dimensional space is called phase space. The system’s evolving state over time traces a path, which is called the phase space trajectory of the system.

CRPs can be used for the study of differences between two processes or for the alignment and search for the matching sequences of the two data series even when the cross correlation fails or if the system is dynamic over time. It has been proven that recurrence is a fundamental property of dynamic systems, which means that after some time the system will reach the state that is arbitrarily close to the former states and pass through a similar evolution. CRPs permit visualization and quantification of these recurrent state patterns. Within the context of collaboration, CRPs have been proposed and used as a general method to unveil the coordination and interlocking of two people (Hajari et al., 2016). Moreover, it has been used to analyze this coordination in the context of eye tracking as well by analyzing CRPs generated from comparing gaze patterns of individuals to determine how closely two collaborators follow each other. It can be used to measure how much and when two subjects look at the same spot (Nüssli, 2011).

CRQA defines several measures that can be assessed along the diagonal and vertical dimensions of the recurrence plot. For the diagonal dimension, we have recurrence rate, determinism, average and longest diagonal length, and entropy. For the vertical dimension, we have: laminarity (LAM) and trapping time (TT) (Marwan & Kurths, 2002; Table 1). We classified LAM and TT as AOI-centric CRQA metrics as they are largely determined by where the person is looking (i.e., AOI). Table 1 provides a description of these metrics.

Table 1.

Description of AOI-based CRQA Metrics.

CRQA Metric	Description
Laminarity (LAM)	Refers to the percentage of recurrence points forming vertical lines which denotes the percentage of time pairs stay in the same regions.
Trapping Time (TT)	Represents the average time two trajectories stay in the same region. TT is an indication of the prolonged duration where the pairs tend to focus on certain regions of the screen.

To calculate LAM and TT, first a vertical line (with v the length of the vertical line) marks a time interval in which a state does not change or changes very slowly: $\vec{x_{i}}$ ≈ $\vec{x_{j}}$ , $\vec{x_{i}}$ ≈ $\vec{x_{i}}$ ₊₁, . . ., $\vec{x_{i}}$ ≈ ${\vec{x}}_{j + v v - 1}$

The total number of vertical lines P(v) of the length v in the plot is then given by Eq. 2 below where N is the number of points on the phase space trajectory (Marwan et al., 2007):

p (v) = \sum_{i, j = 1}^{N} (1 - R_{i, j}) (1 - R_{i, j + v}) Π_{k = 0}^{v - 1} R_{i, j + k}

(2)

LAM is the ratio between the recurrence points forming the vertical structures and the entire set of recurrence points. he computation of LAM is realized for those v that exceed a minimal length $v_{m i n}$ in order to decrease the influence of the tangential motion. which can be computed using Eq. 3 below:

L A M = \frac{\sum_{v = v_{m i n}}^{N} v P (v)}{\sum_{v = 1}^{N} v P (v)}

(3)

TT is the average length of vertical structures, and its computation also requires the consideration of a minimal length $v_{m i n}$ , as in the case of LAM. TT estimates the mean time that the system will abide at a specific state or how long the state will be trapped and is given by Eq. 4 below:

T T = \frac{\sum_{v = v_{m i n}}^{N} v P (v)}{\sum_{v = v_{m i n}}^{N} P (v)}

(4)

Application of CRQA and Analysis of Team Coordination

Richardson and Dale (2005) first used CRP to analyze gaze similarity two people. They studied the relationship between a speaker and a listener based on their eye movements and found that the coupling between a speaker’s and a listener’s eye movements was an indicator of listener engagement. Later, CRQA was used to quantify team collaboration (Pietinen et al., 2010). It was found that a high rate of overlapping fixation could possibly be a sign of efficient collaboration but could also could inform of problems in comprehension (Zheng et al., 2016). Another study used gaze cross-recurrence analysis to measure the coupling of the programmers’ focus of attention. Their findings also showed that pairs who used text selection to perform collaborative references have high levels of gaze cross-recurrence.

More recent studies started focusing on using CRQA in the analysis of environmental factors that affect team performance such as prior knowledge (Villamor & Rodrigo, 2018), speech and communication strategies (Russell et al., 2012), and leadership techniques (Dindar et al., 2022). However, no research has yet to explore how workload transitions and data overload affect teams in data-driven domains. Thus, our work aims to use the novel analysis of CRQA to understand how teammates adapt to changes in workload over time using AOI-centric metrics. The goal of the present work is to apply CRQA to eye-tracking data of pairs of operators in the context of command and control of unmanned aerial vehicles (UAVs) while they are subject to workload transitions. Ideally, we can start to quantify the adaptation process teams go through in response to changes in workload. We expect that both LAM and TT would increase as workload increases (Villamor & Rodrigo, 2018; Zheng et al., 2016).

Methodology

Participants

Ten pairs of undergraduate students (20 students total) at the University of Virginia were recruited for the study (M = 21.3 years, SE = 0.24 years). Each pair consisted of one male and one female who did not previously know each other. The experiment lasted from 75-90 minutes and participants were compensated $10/hour for their time. This study was approved by the University of Virginia’s Institutional Review Board (IRB-SBS #3480).

Experimental Setup

The design of the simulation was based on the ‘Vigilant Spirit Control Station’ the Air Force uses to develop interfaces to control multiple UAVs (Feitshans et al., 2008; Figure 1). Pairs were collocated, but each participant viewed separate monitors and used separate mice to input responses. The simulation was networked so participants could see inputs from their partner in real-time (e.g., when Participant 1 responded to a chat message, Participant 2 could see his/her response in realtime).

Figure 1.

The experimental setup with the testbed shown on two networked computers.

Two desktop-mounted FOVIO eye trackers with a sampling rate of 60 Hz were used to collect point of gaze data. The average degree of error for this eye tracker is 0.78° (SD = 0.59°; Eyetracking, 2011).

UAV Tasks and Point Values

Each pair was responsible for completing a primary task and three secondary tasks—i.e., four tasks total—for up to 16 UAVs (Figure 1). Although all tasks were the pair’s responsibility, only one participant from each pair had to complete each task. The primary task was the target detection task where pairs monitored UAV video feeds and indicated whether a target—

i.e., a semi-transparent cube—was present. The secondary tasks included a rerouting task (avoiding the no-fly zone), fuel leak task (maintaining UAV health), and chat message task (responding to chat messages). These tasks and their structure emulate the multitasking, dynamic, and interdependent environment of a UAV command and control.

Table 2 shows the point value associated with each task. Points were assigned to emphasize the priority of the primary task (i.e., target detection) as well as to convey the severity of incorrectly or not attending to a task (e.g., UAV flies through no-fly-zone). Also, we informed pairs that the highest scoring pair would earn an additional $10 to incentivize performance. Response times for each task for each pair were recorded as well.

Table 2.

Point System for UAV Simulation.

Response	Points Per Response
Correctly recognizing a target	+100
Correctly recognizing a non-target	+50
Completing any secondary task (i.e., reroute, fuel leak, and chat message)	+30
An incorrect or no response to any task (e.g., false positive or no response to the target detection task, UAV flies through a no-fly-zone, or a UAV health is not maintained)	-100

Workload Conditions

Workload was manipulated by varying the number of active UAVs for the primary target detection task. There were two workload conditions: low and high. For the low workload condition, 3-5 UAVs were active at all times and for the high workload condition 13-16 UAVs were active at all times. Pairs always completed the low workload condition before the high workload condition and each condition was 15 minutes long.

Experimental Procedure

Participants of each pair read and signed the consent form and were then briefed about the study’s goals and task expectations. Participants then independently completed a fiveminute training session. By the end of the training session, participants had to demonstrate they could achieve 70% accuracy for all tasks. We then informed the pairs about how the simulation was networked and provided them three minutes to introduce themselves to one another and discuss anything they deemed necessary. There were no restrictions on how the participants could interact during these three minutes and they could choose to coordinate strategies before the experimental portion of the study. Afterwards, the participants completed the low workload condition, were provided a short break, and then completed the high workload condition. Participants could communicate verbally with each other during the experimental portions of the study. The same tasks appeared at both stations, but a participant could not see the cursor movements of their teammate. At the conclusion of the study, participants were compensated for their time.

Data Analysis

After we gathered the eye tracking data from the FOVIO eye tracker, we filtered the datasets and removed invalid entries. The data loss across all participants and trials was on average 11.9% (SD = 11.2%). We detected fixations and saccades using the code developed by the Riggs Lab. This code is used to analyze eye tracking data collected from experimental studies with participants and it serves two main purposes: (1) filtering the eye tracking dataset and (2) detecting fixations and saccades based on Nyström and Holmqvist’s (2010) velocity-based and data-driven adaptive algorithm. The code, implemented in Python, first takes the raw eye tracking files as input, and filters out empty or invalid recordings. Then, it passes the data through a Butterworth smoothing filter and calculates the angular velocities in preparation for the data-driven iterative algorithm, which keeps iterating until the absolute difference between the newly calculated velocity threshold and the previous one converges to less than 1°.

The data that we gathered from the FOVIO eye tracker is 2D, which are the x- and y-coordinates of the eye tracking fixation point. However, to better understand the problem and to be able use the CRP package, we projected the data from 2D into a 1D space. Because the x- and y- coordinates are not necessarily correlated, applying Principal Component Analysis (PCA) to reduce dimensionality is not feasible. Moreover, any sort of projection to reduce dimensionality will introduce some error into the system. Therefore, the only two possible distance metrics are either the Euclidean or Manhattan distance to calculate the distance (d) between eye’s fixation point and the origin. Based on the literature, the Euclidean distance is widely used in this case and is shown to effectively project the data based on Eq. 5 below (Crowley, 2008; Shockley, 2004):

E u c l i d e a n d i s t a n c e (d) = \sqrt{x^{2} + y^{2}}

(5)

We proceeded to get the cross-recurrence plots for each pair and used Marwan et al.’s (2007) CRP MATLAB toolbox for that end and computed the CRQA metrics.

Results

The mean of total points scored in the low workload condition was 24,942 (SD = 2,003) and for high workload it was 63,991 (SD = 7,772). The mean response time in the low workload condition was 2.13 (SD = 0.194) and for high workload it was 3.11 (SD = 0.22). Welch paired t-tests revealed significant differences in total points (t(9) = −19.24, p < .001) and response time (t(9) = −16.51, p < .001) means between low and high workload.

The mean LAM value for the low workload condition was 0.903 (SD = 0.059) and for the high workload it was 0.932 (SD = 0.032) as shown in Figure 2. The mean TT value for the low workload condition was 9.154 seconds (SD = 1.398) and for the high condition it was 10.492 (SD = 2.192) as shown in Figure 3.

Figure 2.

Mean values of the Laminarity (LAM) CRQA Metric. The error bars represent the standard error of the mean values.

Figure 3.

Mean values of the Trapping Time (TT) CRQA Metric. The error bars represent the standard error of the mean values and * denotes significant differences.

To determine whether there is a difference in the CRQA results between low and high workload scenarios, paired t-tests were performed for both metrics. There was a statistically significant difference between the TT means of the low and high workload ((9) = −2.39, p = 0.041), but not between the LAM means for low and high workload (t(9) = −2.043, p = 0.071).

Discussion & Conclusion

The goal of this work sought to understand whether and to what extent AOI-based CRQA metrics in terms of recurrent fixations, scanpath similarities, and durations of agreement of pairs working on a command and control task change when workload changed. The results here show that TT is significantly higher when workload increased. A high trapping time is an indicator that team members are spending a longer duration on a particular AOI before transitioning to another AOI. This meant that pairs spent more time on the same AOIs in high workload scenarios. Therefore, they were collaborating on the same task together; however, we do not know based on TT whether this improved performance or not. For example, a high TT may indicate that team members are spending a longer time discussing a particular issue with regards to a certain task or it could indicate they are getting “trapped” in a certain task which could limit the team’s ability to adapt to changing circumstances (i.e., workload transitions) or to effectively collaborate.

Based on the TT results, it seems that team members need to spend an extended period of time working together to ideally develop a strategy to account for increase in workload. The findings suggest that TT is sensitive to workload changes and can quantify how teams adapt to workload changes. This validates CRQA under AOI-based metrics such as TT and highlights the need for research in order to inform the design of displays by integrating methods where teammates can know where their teammates are looking such as gaze sharing (Atweh et al., 2023; Siirtola et al., 2019). For example, D’Angelo and Begel (2017) developed a system where a pair of programmers were shown what the other was looking at while they worked, and they found providing this shared gaze information aids in coordination and effective communication. Moreover, Akkil et al. (2016) developed a shared gaze interface called GazeTorch which facilitated the collaboration in physical tasks. Several other studies found that shared gaze improved performance and remote collaboration in several domains (Lee et al., 2017). Consequently, future investigations could explore the extent to which attentional focus on the primary task may inadvertently lead to reduced attentional resources allocated to secondary tasks. This exploration could help inform the development of support systems and training interventions that ensure operators maintain situational awareness across multiple tasks, mitigating the risk of overlooking critical information.

Future work should also examine the effect of workload transitions in complex and data-driven environments with a larger sample size. Because we had a limited sample size, we were unable to perform AOI-centric CRQA analysis on the best and worst performing pairs to determine if CRQA metrics are predictors of performance. With a larger sample size, we may find that there is a significant difference between workload levels for LAM as it neared significance with this initial analysis. If significant, the LAM analysis would provide insights on where the participants were scanning on the display, but also how are they scanning by pinpointing the rate of transitions to and from the same and different AOIs.

Overall, this work shows that, as workload increases, pairs tend to struggle more and spend more time on certain AOIs. Therefore, our work highlights a need to further study teams in high workload environments aiming to better understand and support team coordination. The findings also provide support for design solutions that encourage teammates to scan a display in a similar or identical fashion between AOIs. Future research needs to further explore how to effectively use this information and consider other potential environmental features, e.g., the impact of seeing a partner’s gaze in real-time. Furthermore, this work also provides more future work in terms of team dynamics and integration of experts with novices by showing novices the scanning approach of expert teammates as the workload increases. The findings support the potential of technology to rely on these metrics to inform and improve collaboration. Nevertheless, our work highlights the value of real-time measures in data-driven and multitasking domains to better understand differences in collaboration success which in turn can inform technology to effectively assist operators with changing workloads in real-time and hopefully gets us one step closer towards quantifying collaboration in complex domains.

Footnotes

ORCID iD

Jad A. Atweh

References

Atweh

J. A.

Hazimeh

Riggs

S. L.

(2023). Can Real-Time Gaze Sharing Help Team Collaboration? A Preliminary Examination of its Effectiveness with Pairs. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 0(0). https://doi.org/10.1177/21695067231193659

Akkil

James

J. M.

Isokoski

Kangas

(2016). GazeTorch: Enabling gaze awareness in collaborative physical tasks. Conference on Human Factors in Computing Systems - Proceedings, 07-12.

Atweh

J. A.

Moacdieh

N. M.

Riggs

S. L.

(2022). Identifying Individual-, Team-, and Organizational-Level Factors that Affect Team Performance in Complex Domains Based on Recent Literature. Proc. of the Human Factors and Ergonomics Society Annual Meeting, 66(1), 1795–1799. https://doi.org/10.1177/1071181322661213

D’Angelo

Begel

(2017). Improving communication between pair programmers using shared gaze awareness. Conference on Human Factors in Computing Systems - Proceedings, 2017-January.

Dindar

Jarvela

Ahola

Huang

Zhao

(2022). Leaders and Followers Identified by Emotional Mimicry During Collaborative Learning: A Facial Expression Recognition Study on Emotional Valence IEEE Transactions on Affective Computing, 13(3), 1390–

El Iskandarani

Atweh

J. A.

McGarry

S. P. D.

Riggs

S. L.

Moacdieh

N. M.

(2023). Does It MultiMatch? What Scanpath Comparison Tells us About Task Performance in Teams. Journal of Cognitive Engineering and Decision Making. https://doi.org/10.1177/15553434231171484

Hajari

Cheng

Zheng

Basu

(2016). Determining team Cognition from Delay Analysis using cross recurrence plots. 38th Annu Inter. Conf. of the IEEE Eng. in Medicine and Biology Society.

Lee

Kim

Lee

Dey

Piumsomboon

Norman

Billinghurst

(2017). Improving collaboration in augmented video conference using mutually shared gaze. International Conference on Art. Reality and Telexistence and Eurographics Symposium on Virtual Environments.

Marwan

Carmen Romano

Thiel

Kurths

(2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438(5–6).

10.

Marwan

Kurths

(2002). Nonlinear analysis of bivariate data with cross recurrence plots. Physics Letters A, 302(5–6), 299–307.

11.

Pietinen

Bednarik

Tukiainen

(2010). Shared visual attention in collaborative programming. Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering – CHASE

12.

Poole

Ball

L. J.

(2006). Eye Tracking in HCI and Usability Research. In Encyclopedia of Human Computer Interaction (pp. 211–219).

13.

Richardson

D. C.

Dale

(2005). Looking To Understand: The Coupling Between Speakers’ and Listeners’ Eye Movements and Its Relationship to Discourse Comprehension. Cognitive Science, 29(6).

14.

Russell

S. M.

Funke

G. J.

Knott

B. A.

Strang

A. J.

(2012). Recurrence Quantification Analysis Used to Assess Team Communication in Simulated Air Battle Management. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 468–472.

15.

Shockley

K. D.

(2004). Cross Recurrence Quantification of Interpersonal Postural Activity.

16.

Siirtola

Špakov

Istance

Räihä

K. J.

(2019). Shared Gaze in Collaborative Visual Search. International Journal of HumanComputer Interaction, 35(18), 1693–1705.

17.

Villamor

M. M.

Rodrigo

M. M. T.

(2022). Predicting Pair Success in a Pair Programming Eye Tracking Experiment Using Cross-Recurrence Quantification Analysis. APSIPA Transactions on Signal and Information Processing, 11(1). https://doi.org/10.1561/116.00000031

18.

Villamor

Rodrigo

M. M.

(2018). Predicting successful collaboration in a pair programming eye tracking experiment. UMAP 2018 - Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization, 263–268. https://doi.org/10.1145/3213586.3225234