Eye Tracking-Based Adaptive Displays: A Review of the Recent Literature

Keywords and Selection Criteria

Our search keyword (KW) sets consisted of the following:

Each search string was of the structure KW1 AND KW2 AND KW3, with all combinations from within each set used. Google Scholar was used as the search engine into which all of the combinations of search keywords were entered. The resulting studies were then subjected to the following five search criteria. To be considered in the review, studies should:

A total of eight papers were first obtained using the keyword search and selection criteria. Snowballing was then adopted using those eight papers. Snowballing refers to checking the references of each paper (backward snowballing), as well as the papers that have cited each one (forward snowballing; Wohlin, 2014). The same search criteria were then applied to the papers that resulted from backward and forward snowballing. The final count was a total of 18 papers (5 from journals, and 13 from conference proceedings). Figure 1 shows the breakdown of papers by year.

OPEN IN VIEWER

Figure 1.

Number of papers broken down by year.

RQ1: How are eye tracking-based adaptive displays used?

Eye tracking-based adaptive displays have been used in different domains, but the most prominent one in our review was reading applications (broadly defined as the use of any display with text, graphs, and data; 7 papers), followed by maps and aerial images (4 papers) and gaming applications (4 papers as well). The other domains that were represented, with one paper each, were learning (specifically, massive open online courses, or MOOCs), watching videos, and medicine (specifically, dentistry). The vast majority of papers used desktop-mounted eye trackers (13 papers), whereas one used a head-mounted eye tracker. The other papers did not specify the type of eye tracker. The most commonly used eye tracker sampling frequency was 60 Hz (7 papers), with other reported frequencies being 300 Hz (2 papers), and one each reporting 30, 90, and 120 Hz. In general, eye tracking has been used to three broad and non-mutually exclusive ends: 1) to highlight information that is currently on the screen, 2) to update or alter the display in some form, or 3) to record the location of gaze.

RQ2: What eye tracking metrics have been used?

By far the most used metric was simply the location of gaze, where researchers used the x and y positions of fixations or gazepoints, and not any higher order metrics, to trigger adaptations (Bektaş et al., 2015, 2019; Gobel et al., 2018; Ho et al., 2018; Jo et al., 2015; Kim et al., 2020; Kurzhals et al., 2020; Lankes et al., 2019, 2020; Sharma et al., 2016; Tateosian et al., 2017; Vidal et al., 2015). The total dwell time in an AOI has also been used to good extent (Cheng et al., 2018; Kim et al., 2020; Toreini et al., 2022). Other types of metrics have been used more sparingly. Lallé et al. (2019) used the number of fixations on an AOI rather than the dwell time. Cheng et al. (2018) and Spiel et al. (2019) used the transitions between AOI, which is the rate at which a user switches between fixating one AOI and another. A larger number of transitions is assumed to indicate more user uncertainty. Finally, the gaze angle, or the angle between the previous and current gaze point on one hand and the previous gaze point and the target AOI on the other, was also used by Castner et al. (2020). An angle less than 10 degrees would indicate movement towards the target.

For most of these metrics, there was no window of time reported during which time the metrics were calculated. Instead, it was considered that the point of gaze, for example, was calculated in near real time (e.g., Bektas et al., 2015; Kurzhals et al., 2020; Tateosian et al., 2017). In one case, a window of time was considered: a one-minute window was used to calculate the number of fixations to an AOI (Sharma et al., 2016).

RQ3: How has the adaptation been triggered?

Across our review, we found that the adaptations were triggered either in real time, based on the location of gaze, or based on some threshold. The former accounts for most cases, with studies using the location of fixations or gaze points within a certain AOI to trigger an adaptation (Bektas et al., 2015, 2019; Gobel et al., 2018; Ho et al., 2018; Jo et al., 2015; Kurzhals et al., 2020; Lankes, 2020; Lankes et al., 2019; Tateosian et al., 2017; Vidal et al., 2015). Alternatively, an appropriate threshold can be selected for some metric. For example, a threshold for dwell time can be considered, and when the time goes above the threshold, the adaptation is triggered. In one case, 600 ms was considered as the threshold by Kim et al. (2020). Another threshold that has been reported was 8 seconds (Toreini et al., 2018). In one case, the exact threshold dwell time was not reported (Cheng et al., 2018).

The number of fixations has also been compared to a selected threshold; this has typically been determined using a combination of previous studies and further pilot studies that have established the average or expected number of fixations on an AOI or target. Lallé et al. (2019) triggered the adaptation when the number of fixations was above 40% of the average number expected on a sentence. In another similar case, in the context of learning in MOOCS, a score was given for each gaze point based on whether participants were looking right where they should (maximum score of 255) or not at all (score of 0; Sharma et al., 2016). The adaptation was then triggered if the score was below the threshold calculated from previous participants whose performance was between the 33^rd and 66^th percentiles. In some cases, it was not clear exactly how the adaptation was triggered (Castner et al., 2020; Spiel et al., 2019; Toreini et al., 2022).

What can eye tracking-based adaptive displays provide?

This review revealed that eye tracking-based adaptive displays have been relatively more prominent in certain domains (namely, as support for reading, maps, and gaming). The applications were largely successful, with many studies citing improved performance through adaptation and increased satisfaction of users. The majority of the adaptive displays created did not use a very large eye tracker sampling rate (mainly 60 Hz). While that is promising in terms of what can be done with eye trackers, it also provides more questions about the ceiling of eye tracking-based adaptive displays and whether we have not yet come close to exploring all that might be done.

At the same time, it may be that we have reached that ceiling and without data from other sensors/tools (e.g., face recognition, EEG, heart rate, etc.), eye tracking-based adaptive displays are very limited. Is it possible to envision everyday eye tracking-based adaptive displays outside of virtual reality, gaze-based input, or supporting users with special needs? Across a span of seven years, there were less than 20 papers on the subject and in many cases the studies did not provide enough detail or thorough testing to be able to judge the applications. It would be interesting to see more use of eye tracking-based adaptive displays for interruption management, for example. There is also no clear use of different sensory channels or modalities, which could make use of Multiple Resource Theory (Wickens, 2008) to provide additional data to users. In other words, it could be that some auditory notification or information could be triggered instead of any additional visual information. The education domain is another area that could benefit from more adaptive displays that support online learning, potentially together with interruption management. Other specific applications that were not evident in our review but that could benefit from further research include display clutter management, detection of distraction or mind-wandering, and vigilance decrement detection.

What are the best metrics to use?

The review revealed that most adaptive displays rely on one or two metrics, with most relying simply on the location of gaze. Would more metrics be able to reveal more about a person’s attention or would they not be useful in the context of an adaptive display? Is there a limitation on processing power that would create a lag and prevent the adaptation from being triggered quickly enough? This could be a problem for higher-order eye tracking metrics (Sharafi et al., 2020), where these might not be calculated quickly enough to be of use. Other metrics may be redundant and highly correlated, or may require too many fixations to be calculated sufficiently quickly to be useful.

To further explore this last issue, a moving window of time may be needed during which time the metrics are calculated. The exact duration of this window will likely depend on the context and the task, with more high-level or complex metrics requiring longer windows. But what combination of metrics works best for different tasks or contexts? Would a window of one minute as suggested by Sharma et al. (2016) be useful in other contexts? The focus would have to be on finding ways to shorten the window as much as possible for different tasks.

How should the adaptation be triggered?

The review revealed that none of the adaptive displays in recent years have relied on prediction algorithms such as regression. Instead, either the location of gaze alone or a threshold on dwell time or fixation number has been used. Could there be any merit in using more advanced machine learning models? This would go together with using additional eye tracking metrics, in which case these would form the input to any algorithm that then triggers the adaptation. Would this provide any added value? One thing to consider is the added time for this calculation to be done, which would need to be factored into the moving window. Moacdieh and Sarter (2017) used logistic regression models to predict when users are affected by clutter using three eye tracking metrics but did not close the loop and create any display adaptations. Could it be that the same result could have been obtained using a simpler threshold approach? This is something that would need to be explored further.