Abstract
Objective:
The objective was to use eye tracking to trace the underlying changes in attention allocation associated with the performance effects of clutter, stress, and task difficulty in visual search and noticing tasks.
Background:
Clutter can degrade performance in complex domains, yet more needs to be known about the associated changes in attention allocation, particularly in the presence of stress and for different tasks. Frequently used and relatively simple eye tracking metrics do not effectively capture the various effects of clutter, which is critical for comprehensively analyzing clutter and developing targeted, real-time countermeasures.
Method:
Electronic medical records (EMRs) were chosen as the application domain for this research. Clutter, stress, and task difficulty were manipulated, and physicians’ performance on search and noticing tasks was recorded. Several eye tracking metrics were used to trace attention allocation throughout those tasks, and subjective data were gathered via a debriefing questionnaire.
Results:
Clutter degraded performance in terms of response time and noticing accuracy. These decrements were largely accentuated by high stress and task difficulty. Eye tracking revealed the underlying attentional mechanisms, and several display-independent metrics were shown to be significant indicators of the effects of clutter.
Conclusion:
Eye tracking provides a promising means to understand in detail (offline) and prevent (in real time) major performance breakdowns due to clutter.
Application:
Display designers need to be aware of the risks of clutter in EMRs and other complex displays and can use the identified eye tracking metrics to evaluate and/or adjust their display.
Introduction
Display clutter represents a problem for operators in many data-rich, safety-critical domains such as aviation (Kim & Kaber, 2009), security (Zhu & Sun, 2012), and medicine (Singh, Spitzmueller, Petersen, Sawhney, & Sittig, 2013). It leads to significant attention and performance decrements, including degraded object recognition (Bravo & Farid, 2006; Yeh, Merlo, Wickens, & Brandenburg, 2003), delayed visual search (Beck, Trenchard, van Lamsweerde, Goldstein, & Lohrenz, 2012; Henderson, Chanceaux, & Smith, 2009; Moacdieh & Sarter, 2012; Neider & Zelinsky, 2011), increased memory load (Westerbeek & Maes, 2011), and confusion (Alexander, Stelzer, Kim, & Kaber, 2008; Lohrenz, Trafton, Beck, & Gendron, 2009).
Despite the agreement that “clutter” is a concern, there is no consensus on how to define, measure, or overcome this multifaceted problem. Based on a recent review of the literature, we refer to clutter as the presence of performance and attentional decrements that result from the interaction between high density of irrelevant data and poor display organization (Moacdieh & Sarter, 2014; Rosenholtz, Li, & Nakano, 2007). Some factors, such as high workload and inexperience, have been shown to exacerbate the effects of clutter (Naylor, 2010). However, the impact of other factors like task difficulty or stress (i.e., the interaction between physical and/or psychological demands, people’s perception of how well they can cope, and their perception of how important these demands are; Staal, 2004) remains less clear. The combination of clutter and stress may present a significant risk in domains where the ability to quickly and accurately extract important information is critical for safety and efficiency. This research therefore aimed to assess the performance effects and identify the underlying attentional mechanisms resulting from clutter, stress, and task difficulty in the context of search and noticing tasks. We focused on these tasks given that they are most commonly associated with and affected by clutter (e.g., Rosenholtz et al., 2007).
Background
Research on display clutter typically adopts image processing, performance evaluation, and/or subjective evaluation as assessment techniques. Image processing provides a mathematically defined, display-based measure of clutter (e.g., Camp, Moyer, & Moore, 2010; Rosenholtz et al., 2007). From a human factors perspective, such a measure is of limited use as it does not reflect how factors like stress interact with design to shape performance and attention. Performance evaluation can fill that gap (e.g., Chen & Barnes, 2012; Wickens, Nunes, Alexander, & Steelman, 2005). However, frequently used measures such as response time and accuracy assess performance outcome only; they do not trace the underlying attentional mechanisms associated with clutter. Similarly, subjective evaluation using questionnaires and interviews (e.g., Kaufmann & Kaber, 2010) fails to trace the effects of clutter on attention management and cannot be used in real time. Capturing the attentional mechanisms underlying the performance costs of clutter is important since (a) it can suggest targeted countermeasures and thus inform the redesign of displays and (b) it helps detect the effects of clutter early on so that the display can be adjusted in real time, forming the basis for an adaptive display.
One means of tracing attentional allocation is using eye tracking (see Figure 1; for more details, see McCarley & Kramer, 2006; Poole & Ball, 2005). Several eye tracking metrics have been shown to be valid indicators of the effects of stress on attention allocation (e.g., Di Nocera, Camilli, & Terenzi, 2007). Eye tracking has also been used to study clutter (e.g., Beck et al., 2012; Beck, Lohrenz, & Trafton, 2010; Grahame, Laberge, & Scialfa, 2004; Neider & Zelinsky, 2011; Zhu & Sun, 2012). But relatively few and simple eye tracking metrics, such as fixation number and mean fixation duration, tend to be used, which may not provide the detail necessary to capture the various aspects of clutter. In addition, many of the metrics used to date rely on areas of interest (AOIs), experimenter-defined areas of the display on which analysis takes place (e.g., Beck et al., 2010; Hegarty, De Leeuw, & Bonura, 2008; Josephson & Holmes, 2006). However, AOI-dependent metrics require high eye tracker accuracy to be reliable (Goldberg & Helfman, 2011) and cannot be easily generalized across displays. Finally, the metrics that best reflect the effects of clutter early in the search process have yet to be identified, although these would be invaluable for adapting displays prior to significant performance decrements.
Fixations are usually depicted as a circle whose diameter is proportional to fixation duration. Saccades are represented as lines between two successive fixations, whereas areas of interest are typically drawn as rectangles. All of the fixations and saccades together create the scanpath.
To understand the effects of clutter on attention, one needs to examine how clutter affects the location/spread, directness, and duration of eye movements (Moacdieh & Sarter, 2014; see Table 1). Location metrics depend only on fixation coordinates; they show whether clutter causes a dispersion of eye movements across the display, thus preventing the user from focusing on important information. Directness measures differ in that the sequence of fixations is taken into account; these measures can indicate whether clutter made search less ordered or systematic. Finally, duration measures indicate how long a person looked at a particular area and relate clutter primarily to the difficulty extracting information from the display. Increased spread, less directness, and increased duration of fixations are the potential clutter-induced attentional changes that we focus on in this research.
Overview of the Eye Tracking Metrics Used in This Study
Grid size width was chosen to be around half the width of the medical history column (see later description of the display), and the length was equivalent to three rows.
Research Goals
The objectives of our research were to (a) determine the effects of display clutter, stress, and task difficulty on search and noticing performance and (b) examine the attentional mechanisms underlying these performance costs using AOI-independent eye tracking metrics. We defined visual search as the process of actively looking for and locating a given target whose location is unknown (Treisman, 1991), whereas noticing referred to “the conscious registration of some event” (Schmidt, 1995, p. 29), in our case important or critical events, without being given a specific target. The scope of this study was limited to analyzing the effects of the excess data aspect of clutter, the most prominent aspect of the phenomenon in the literature to date. Our expectations were that (a) clutter would lead to worse visual search and noticing performance, and that this would be accentuated by high stress and task difficulty, and (b) AOI-independent eye tracking metrics would help explain this degradation in performance by revealing changes in attention allocation.
The application domain for this research was electronic medical records (EMRs) used in the emergency department (ED) of a large research hospital. EMRs are digital versions of a patient’s medical health records that are used by physicians to diagnose and treat patients (King, Patel, & Furukawa, 2012). Studies have shown that EMR clutter, and specifically data overload, leads to increased search time for information (Duftschmid et al., 2013), decreased search accuracy (Zeng, Cimino, & Zou, 2002), and more time spent using the EMR overall (Murphy, Reis, Sittig, & Singh, 2012). However, the attentional mechanisms underlying these performance costs remain unclear, as does the interaction between clutter and stress, which, in the ED, is created by a high volume of patients and the serious and time-critical procedures involved (Trzeciak & Rivers, 2003).
Methods
Participants
The participants in this experiment were 15 medical practitioners (7 female and 8 male). They included 11 residents, 1 staff physician, and 3 physician assistants who were employed in the ED of a large research hospital. Their average age was 30.1 years (SD = 3.3), and their average years of experience in this ED was 4.3 years (SD = 4.0). All participants had been trained to use the EMR currently employed at the hospital. Their average experience using the EMR was 2.0 years (SD = 0.75). All participants gave informed consent and received compensation for their time. This study was approved by the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board.
Simulated EMR
We replicated the medical history page of the EMR currently being used by all research participants. The page was created using Windows Presentation Foundation with XAML and C#, and was populated with fabricated but realistic patient data. Following discussions and observations with ED physicians, we identified this page as being one of the pages most affected by data overload. It displays the patient’s medical, surgical, social, and family history (see Figure 2; copyright restrictions prevent us from displaying the actual page). To the right of the medical and surgical history columns, there is a column displaying the date of each condition or procedure. Similarly, the social history column has another column to the right that provides brief details about each item (e.g., how many packs of cigarettes a day if the person is a smoker). The family history section also consists of two columns, one showing a particular medical condition and the other indicating the family member who suffered from it. Within each of the sections, the data in each column are not ordered; rather, the order of presentation depends on when it was entered by medical providers. In this experiment, the page was nonscrollable, and participants could not use the links to navigate to other pages. All the information needed for the experiment was present on the page.
Outline of the page used in the experiment (right), including an example of the medical history entries in a low or high clutter condition (left; the number next to each entry is an identifier for each disease/complaint).
Experiment Setup
The EMR was displayed on a 19-inch monitor with a resolution of 1,280 × 1,024 pixels. An Applied Science Laboratories D-6 desktop-mounted eye tracker was used (sampling rate: 60 Hz; accuracy: less than 1° visual angle; precision: 0.5° visual angle). The eye tracker was placed directly in front of the computer monitor such that the aperture was around 2 inches from the screen. The participants were seated at a distance of around 24 inches from the eye tracker, making for a visual angle of around 60° with the screen in the horizontal and vertical directions. Calibration took place at the start of the experiment using a 9-point grid. The duration of the calibration procedure varied across participants but generally took around 5 minutes.
Experiment Design
The three independent variables were clutter (low, high), stress (low, high), and type of search task (simple, difficult). These were varied within participants, leading to a 2 × 2 × 2 full-factorial design. The combination of these three variables translated to eight versions of the EMR page, each containing data for a different fictitious patient scenario. For each patient scenario, there was a corresponding (a) summary of the patient’s symptoms, similar to what a physician would receive in a triage note, (b) search target, that is, information that a physician might look for in a similar situation, (c) noticing target, that is, information that a physician needed to make the correct diagnosis, and (d) suggested diagnosis. Each patient scenario corresponded to one experiment trial, and participants were exposed to the same one trial per condition.
The suggested diagnosis was presented to participants as the initial assessment by a medical student. Participants were told to first answer the search question (i.e., find the search target) and then use the rest of their time to assess the likelihood of the diagnosis based on the available EMR data. Note that they were not requested to make a final diagnosis about the patient, as such a task would entail examining more than just one EMR page. Participants provided all their answers verbally. Collaborating physicians in the ED confirmed that the patient scenarios were realistic and mimicked some of the tasks they perform using their EMR. Table 2 shows all the scenario details, including the text that was presented to physicians and the search and noticing targets. The scenarios are equivalent apart from the manipulated variables and involve searching for a target in one of the eight columns related to medical, surgical, social, or family history.
Text of the Experiment Trials Shown to Participants and the Targets That They Had to Search for or Detect
Note. High stress scenarios are in italics; search target shown in bold and underlined here.
The three experimental variables were manipulated as follows:
Clutter
Clutter was operationally defined as the number of text entries (medical/surgical condition, social history entry, or family history entry) in the EMR display. High clutter was created by adding information that was irrelevant to the current condition of the patient. For example, for a patient presenting with abdominal pain, irrelevant information could include entries about skin conditions or ear disorders. The low versus high clutter tasks were equivalent in that (a) there was the same amount of relevant information (i.e., the same number of relevant entries), (b) the eccentricity of the search and noticing targets was held to within less than a fifth of a page of difference (always in the same position horizontally and never more than 300 pixels apart vertically in a 1,680 × 1,050 pixel image), and (c) the two questions were comparable (e.g., they both required search for an entry in the medical history section). There was an average of 15 (SD = 2.3) entries in low clutter and an average of 41 (SD = 2.8) entries in high clutter.
Stress
Stress was induced using a combination of techniques. First, the high stress scenarios involved life-threatening patient conditions, as opposed to less serious conditions for low stress tasks. Second, a time limit of 20 seconds was imposed during the high stress scenarios. This limit was determined empirically, following pilot tests with physicians. Third, participants were told that the person with the best performance in these high stress tasks (in terms of search time and accuracy) would be paid an extra $50, in addition to the compensation for participating in the experiment. Fourth, participants were instructed to be as fast as possible, whereas in the low stress condition, participants were asked only to be both fast and accurate. Finally, the high stress scenarios were accompanied by ventilator and heart rate monitor alerts, sounds that physicians would typically associate with critical conditions.
Task
Both simple and difficult search tasks were used. Simple search tasks involved finding only one given piece of information in the display (e.g., “Does the patient have a history of gastritis?”), whereas difficult tasks involved finding several instances of a given item and performing some comparison or judgment (e.g., “When was the patient’s most recent aortic root repair?”).
The dependent measures consisted of performance, eye tracking, and subjective data. The performance measures were search time, or the time to find the search target (excluding cases where the answer was incorrect, a response was not provided, or the trial timed out), screen time (the total amount of time spent scanning the display, including both search and noticing tasks), search accuracy (whether participants correctly identified the search target), and noticing accuracy (whether participants detected the important information in each EMR page). For both types of accuracy, an error was defined as any case where the participant either did not respond or provided the incorrect answer. With regard to eye tracking, all of the measures in Table 1 were calculated. Finally, subjective measures included clutter ratings and workload assessments using the NASA Task Load Index (TLX).
Experiment Procedure
First, participants were given a training session that included a sample task scenario. This was done to familiarize them with the different tasks and required responses. Participants were informed that this study was intended to evaluate the design of EMRs. Next, the eye tracker was calibrated, after which the experiment began. For each trial, the screen first displayed an introduction page that had a description of the patient’s symptoms, the initial diagnosis made by a medical student, the specific question related to the proposed diagnosis (see Table 2), and whether participants had 20 seconds or unlimited time for that task. The presentation order of the variables was counterbalanced by dividing participants into two groups, each of which did the experiment in a fixed random order. The order of the second group was the inverse of the first. When participants were ready, they pressed a button to replace the question screen with the corresponding EMR page. Participants then searched through that page to find the answer to the given question. They were instructed that the search task was their priority, and they needed to provide the answer verbally first and also think aloud throughout.
After participants had provided their answer to the search task, they continued scanning the display to look for other case-relevant information for as long as they needed (limited to 20 seconds in the high stress conditions). The EMR page was then replaced by a question page that asked participants to explain whether and why they thought the initial diagnosis was likely or not. They completed all trials with no breaks in between, with the eight pages taking around 10 minutes. After completing all trials, participants filled out a debriefing questionnaire in which they entered clutter ratings for sample images and NASA-TLX ratings for the low and high stress conditions.
Results
Unless otherwise specified, results were analyzed using a 2 × 2 × 2 repeated-measures analysis of variance (ANOVA), with Bonferroni corrections applied for multiple statistical tests. Significance was set at p < .05, and partial eta-squared (η p 2) was used as a measure of effect size. The ANOVA results are reported for statistically significant results only, with some descriptive values highlighting notable trends. We refer to the different conditions using L/H for low/high stress and S/D for simple or difficult tasks. Error bars on graphs indicate the standard error of the mean (SEM).
Performance Results
Search time
We considered only correct answers, all of which were given within less than the 20-second time limit imposed in the high stress situation. Clutter caused a slight overall increase in search time from 5.85 seconds (SEM = 1.56) in the low clutter condition to 7.78 seconds (SEM = 2.07) in the high clutter condition, with a significant simple simple main effect of clutter in the high stress and difficult task condition only, F(1, 10) = 9.095, p = .013 (Figure 3b). In this case, four participants’ data were discarded due to incorrect answers.
Response time in seconds in (a) low stress and (b) high stress.
Screen time
We considered only low stress conditions (i.e., cases with no time limit) for screen time to avoid including the cutoff time of 20 seconds in the results. One participant’s screen time could not be calculated due to a technical error during the experiment. Clutter significantly increased the amount of time spent extracting information from the display, from an overall average of 20.05 seconds (SEM = 5.35) in low clutter to 26.38 seconds (SEM = 7.05) in high clutter, main effect of clutter: F(1, 14) = 11.789, p = .004, η p 2 = .331. There was also a simple simple main effect of clutter in the simple task and low stress condition, F(1, 14) = 11.841, p = .016, η p 2 = .458, with high clutter resulting in longer screen time (see Figure 4).
Screen time in seconds for the low stress conditions.
Search and noticing accuracy
To determine the effects of clutter, stress, and task difficulty on the likelihood of missing important information, we performed a mixed-model binary logistic regression for each page with clutter, stress, and task as fixed effects and participants as the random effects. For the search tasks, clutter did not result in a significant increase in the number of misses (2 and 6 misses in the low and high clutter condition, respectively, from a total of 60 summed across all 15 participants in each condition). For the noticing tasks, clutter resulted in a significant increase in the number of misses in all four conditions (see Figure 5). Regression analysis showed that clutter (odds ratio [OR] = 5.25, 95% confidence interval [CI] = 2.153–12.856, p < .001), stress (OR = 3.83, 95% CI = 1.584–9.263, p = .003), and task difficulty (OR = 3.83, 95% CI = 1.584–9.263, p = .003) were all significant predictors of misses.
Number of noticing misses in (a) low stress and (b) high stress (summed across all participants).
Eye Tracking Results
One participant’s data had to be discarded due to poor eye tracking discrimination, resulting in a total of 14 participants for the eye tracking analysis. After eliminating the periods corresponding to eye blinks, the raw eye tracking data (gaze points) were separated into fixations (minimum duration for fixations was 100 ms) and saccades (gaze points had to be within a radius of 2° visual angle; Goldberg & Kotval, 1999). We first examined the search eye tracking metrics (i.e., from start until search target detection) to determine the underlying cause of degraded search performance. We then calculated the overall eye tracking metrics (i.e., throughout the entire time participants spent looking at the screen) to examine the reasons behind poor noticing performance. Finally, we calculated early eye tracking metrics using data from only the first 4 seconds of search to identify metrics that can capture the effects of clutter early on. We chose this time window as it was shorter than almost all the response times but still long enough to calculate the metrics. Only 9 out of 111 (around 8%) of responses were made in less than 4 seconds. In these cases, the actual response time was used instead of 4 seconds.
Search eye tracking metrics
Results showed a main effect of clutter on four location metrics: number of fixations, convex area, spatial density, and NNI (see Table 3). For the directness metrics, there was a significant main effect of clutter on scanpath length per second and mean saccade length. The duration metric was not significantly affected by clutter.
Mean Values of the Search Metrics in the Different Clutter Conditions
Also see Figure 6.
Note that in three cases the convex hull area could not be calculated because there were not sufficient fixation points to create the triangulation matrix.
Some of the main effects need to be interpreted with caution since there were also significant interaction effects for a number of metrics (see Figure 6, left, and Table 3). First, there were significant clutter × task interactions for the number of fixations (Figure 6a, left), spatial density (Figure 6b, left), and scanpath length (Figure 6c, left) and simple main effects of clutter in the difficult conditions, number of fixations: F(1, 13) = 8.75, p = .011, η p 2 = .402; spatial density: F(1, 13) = 8.807, p = .011, η p 2 = .393; scanpath length: F(1, 13) = 5.359, p = .038, η p 2 = .292. As can be seen in Figures 6a, 6b, and 6c (right), the increase in the metrics in the difficult condition can be largely attributed to the increase in the high stress and difficult task condition.
Significant interaction effects for the (a) number of fixations, (b) spatial density, and (c) scanpath length (left) and most noticeable increase in the high stress, difficult task (HD) conditions (right). LS = low stress, simple task; LD = low stress, difficult task; HS = high stress, simple task.
There was also a significant clutter × stress interaction effect for mean saccade length (see Figure 7 and Table 3). However, in this case, clutter caused a significant decrease in mean saccade length in the low stress condition, simple main effect of clutter in low stress: F(1, 13) = 10.451, p = .007, η p 2 = .446.
Significant interaction effects for the mean saccade length.
Overall eye tracking metrics
For the location metrics, there was a significant main effect of clutter such that the number of fixations, convex hull area, spatial density, and NNI were higher in high clutter (see Table 4). For the directness metrics, only the mean saccade length significantly decreased in high clutter. Finally, the duration metric was not significantly affected by clutter. In this overall phase, we did not analyze the clutter × stress and clutter × task interaction effects given the unequal amount of time that participants could spend on the screen in the low and high stress cases.
Mean Values of the Overall Eye Tracking Metrics in the Different Clutter Conditions
Early eye tracking metrics
Finally, we calculated once again the metrics that showed a significant main or interaction effect for clutter in the search or overall phase, but this time over a period of 4 seconds. None of the metrics showed significant main effects of clutter; however, convex hull area, NNI, scanpath length per second, and mean saccade length showed a similar trend to what we observed during the search and overall phases (see Table 5).
Mean Values of the Early Eye Tracking Measures in the Different Clutter Conditions
Significant interaction effects were found only for the mean saccade length. There were both significant clutter × stress interactions and clutter × task interactions (see Figure 8 and Table 5). Clutter significantly decreased the mean saccade length overall in both the simple task, simple main effect of clutter: F(1, 13) = 5.564, p = .07, η p 2 = .3 (Figure 8a) and low stress, simple main effect of clutter: F(1, 13) = 7.243, p = .036, η p 2 = .358 (Figure 8b) conditions, as well as the low stress and simple task condition in particular, simple simple main effect of clutter: F(1, 13) = 12.405, p = .016, η p 2 = .488 (Figure 8c). The opposite trend was seen in the case of high stress and difficult tasks (Figure 8d).
Early values of mean saccade length: (a) clutter × task interaction, (b) clutter × stress interaction, and (c) values in the low and high stress conditions.
Subjective Results
Participants were asked to rate the amount of clutter for one of the low clutter and one of the high clutter EMR pages on a 10-point scale, with 10 referring to maximum clutter. The ratings were analyzed using a Wilcoxon test for ordinal data. High clutter resulted in a significant increase in ratings from a median of 3 (SD = 1.6) in low clutter to 8 (SD = 2.3) in high clutter (median increase = 3; z = 3.316, p = .001).
Participants also rated their perceived mental workload in the low stress and high stress conditions using NASA-TLX scales (scale: 0 to 20 [largest effect]). These rankings were also analyzed using a Wilcoxon test. As can be seen in Table 6, there were significant effects of stress on all six scales, with high stress notably resulting in increased impressions of temporal demand and frustration.
Results of the NASA Task Load Index (NASA-TLX) Ratings Along the Different Dimensions
Discussion and Conclusion
Our goals in this study were twofold. First, we wanted to assess the effects of clutter, in combination with stress and task difficulty, on visual search and noticing performance. The subjective data confirm that we were able to manipulate clutter and stress, as intended. Results showed that, with high clutter, screen time increased significantly; also, participants missed more critical information. These noticing misses occurred even without a time constraint.
As expected, high stress and task difficulty exacerbated the performance effects of clutter. Clutter significantly affected search time only in the high stress and difficult task condition, and search time tended to increase overall with high stress, despite the presence of a time limit and the instruction to search as quickly as possible. This result supports previous studies that have highlighted the detrimental effects of stress and time pressure on performance (e.g., Quigley, Nelson, Carriere, Smilek, & Purdon, 2012). Search accuracy, on the other hand, was not significantly affected by clutter or stress. One possible explanation for this is that the search tasks used in this study were not sufficiently difficult. In terms of the speed–accuracy trade-off, participants seemed to compromise speed in favor of accuracy.
Our second goal was to explain the observed performance effects using AOI-independent eye tracking metrics and to determine which metrics best reflect the effects of clutter on attention allocation. Both during search and overall, performance effects were best reflected in location metrics (specifically, the total number of fixations, convex hull area, spatial density, and NNI). This finding suggests that participants were distracted by unimportant data. The resulting large spread of fixations likely resulted in the increased response time and number of misses, consistent with results from previous studies (Moacdieh, Prinet, & Sarter, 2013; Moacdieh & Sarter, 2012). High clutter also affected directness measures. Specifically, mean saccade length and scanpath length per second decreased significantly. Thus, although fixations overall were spread widely across the screen, successive fixations were not very far apart. It seems that participants adopted a slower and possibly more orderly search to compensate for high clutter. Moreover, the duration metric was not significantly affected by clutter. This suggests that extracting and understanding information was not a problem for participants. We also examined the eye tracking metrics over the first 4 seconds of each trial. There were no significant effects of clutter in this case, but convex hull area, NNI, scanpath length per second, and mean saccade length revealed similar trends to those observed in the search and overall phases.
Finally, the performance decrements resulting from high stress and task difficulty were also reflected in the eye tracking metrics. The number of fixations, spatial density, and scanpath length were all significantly affected by clutter in the difficult task conditions. The effect of clutter also appeared to be most noticeable in the high stress and complex task condition. Thus, any algorithms looking to identify the effects of clutter for a judgment and multiple comparison task, especially under high stress, should consider using these metrics. On the other hand, mean saccade length would appear to be less useful in the case of high stress and difficult tasks as the effect of clutter was significant only in low stress and simple task conditions.
The practical implications of this study include the use of eye tracking metrics to create intelligent, adaptive displays where adjustments to information presentation are triggered in real time. The fact that the early eye tracking metrics revealed a similar trend suggests that adaptations may be triggered early on and before significant performance effects occur, although more research is needed on this topic. The metrics can also be used offline to detect and analyze the effects of clutter on the spread, directness, and duration of search, providing display designers and researchers with a framework for the comprehensive evaluation of clutter.
The main limitation of this experiment is the relatively small number of participants. Combined with the Bonferroni corrections, this meant that the differences between several mean values narrowly failed to reach significance. Also, asking participants to think aloud may have been disruptive, although physicians may often talk to other medical personnel while using their EMR.
Key Points
Electronic medical record clutter degrades response time and noticing accuracy, particularly in conditions of high stress and task difficulty.
Eye tracking metrics suggest that the performance decrements are due to an increased spread of eye fixations and a slower and more methodical search.
A number of eye tracking metrics reflect the effects of clutter on the search process and can help inform the design of displays, as well as form the basis for early detection and adjustment of displays in real time.
In particular, all of the metrics identified are AOI-independent, meaning that they are not limited to particular displays or domains.
Footnotes
Acknowledgements
We would like to thank Travis Ganje, Benjamin Bassin, and David Somand for their helpful insights and feedback. We would also like to thank the medical personnel who participated in the experiment.
Nadine Moacdieh is a PhD candidate in the Department of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She earned her MSE in industrial and operations engineering from the University of Michigan, Ann Arbor in 2012.
Nadine Sarter is a professor in the Department of Industrial and Operations Engineering, Center for Ergonomics, at the University of Michigan. She earned her PhD in industrial and systems engineering from Ohio State University in 1994.