Abstract
Objective
The present study examines the cognitive effects of placing icons in unexpected spatial locations within websites.
Background
Prior research has revealed evidence for cognitive conflict when web icons occur in unexpected locations (e.g., cart, top left), generally consistent with a dynamical systems models. Here, we compare the relative strength of evidence for both dual and dynamical systems models.
Methods
Participants clicked on icons located in either expected (e.g., cart, top right) or unexpected (e.g., cart, top left) locations while mouse trajectories were continuously recorded. Trajectories were classified according to prototypes associated with each cognitive model. The dynamical systems model predicts curved trajectories, while the dual-systems model predicts straight and change of mind trajectories.
Results
Trajectory classification revealed that curved trajectories increased (+11%), while straight and change of mind trajectories decreased (−12%) when target icons occurred in unexpected locations (p < .001).
Conclusion
Rather than employing a single cognitive strategy, users shift from a primarily dual-systems to dynamical systems strategy when icons occur in unexpected locations.
Application
Potential applications of this work include the assessment of cognitive impacts such as mental workload and cognitive conflict during real-time interaction with websites and other screen-based interfaces, personalization and adaptive interfaces based on an individual’s cognitive strategy, and data-driven A/B testing of alternative interface designs.
Keywords
INTRODUCTION
Many human-computer interaction (HCI) tasks require searching for visual targets such as icons. Designers often place these targets in consistent locations—for example, shopping carts usually appear at the top right of ecommerce websites. Consequently, users expect that targets will appear in these “typical” locations—an effect known as “location typicality” (Roth et al., 2013). However, designers sometimes move icons from expected locations to alternative locations based on user research, visual constraints, or specific business objectives. How do these decisions impact cognition and performance? Improving our understanding of the cognitive strategies underlying everyday HCI tasks can help designers create more effective and personalized interfaces. Since researchers proposed that mouse movements could be used to improve interfaces by helping to deliver personalized content (Mueller & Lockerd, 2001), there is a need for empirical work examining quantitative features of mouse movements during everyday HCI tasks (Katerina & Nicolaos, 2018).
Ericson et al. (2021) used mouse-tracking to record continuous mouse trajectories while participants searched for icons occurring in either expected (e.g., cart, top right) or unexpected (e.g., cart, top left) locations. Their results revealed “cognitive conflict” (Kieslich & Hilbig, 2014; Rand et al., 2012; Weis & Wiese, 2017) when icons occurred in unexpected locations, and mouse trajectories were generally consistent with a dynamical systems strategy (Beer, 2000; Dale et al., 2007; Kelso, 1995; Spivey & Dale, 2006; van Gelder, 1998; Warren, 2006) in which users continuously adjust their mouse movements as the task unfolds. Our analysis of Ericson et al.'s (2021) data more explicitly compares evidence for two competing models of the underlying cognitive dynamics: (1) the dual-systems model (Evans, 2008; Gideon & Yaacov, 2009; Hehman et al., 2015; Kahneman, 2011; Strack & Deutsch, 2014), and (2) the dynamical systems model (Beer, 2000; Dale et al., 2007; Kelso, 1995; Spivey & Dale, 2006; van Gelder, 1998; Warren, 2006).
This article is structured as follows. First, we review dual-systems and dynamical systems models of human cognition, and the predictions associated with each model. We then describe the methods and dataset, present a new analysis of Ericson et al.’s (2021) data, and interpret the findings with respect to each cognitive model.
Two Models of Human Cognition
Dual-systems models posit that decision making is influenced by two cognitive systems (Hehman et al., 2015; Kahneman, 2011; Strack & Deutsch, 2014): System I is fast, intuitive, and emotional; in contrast, System II is slow, deliberative, and rational. Dual-systems models generally imply that decisions are consciously reached prior to motor output. For example, before clicking on a target icon in an interface, one decides which target to seek, searches for it, finds it, and then executes the necessary motor movements to click on it. “Discrete changes of mind” are associated with this model (Kieslich et al., 2020); for example, if the target occurs in an unexpected location, one moves toward the remembered target location, and then instantaneously executes a corrective movement toward the actual target location. In essence, dual-systems models posit a discontinuous rather than continuous model of cognition.
In contrast, cognitive science increasingly conceptualizes human cognition in terms of dynamical systems (Beer, 2000; Dale et al., 2007; Kelso, 1995; Spivey & Dale, 2006; van Gelder, 1998; Warren, 2006). Dynamical models of decision-making posit that decision-making and motor movements are mutually influential and co-evolve gradually over time, and that conscious and unconscious processes compete continuously to influence choices between partially activated response alternatives as tasks unfold (Spivey & Dale, 2006). Prior research has found increasing support for dynamical systems models of human behavior and cognition across a range of tasks and phenomena in human psychology (Freeman, 2018; Guastello & Liebovitch, 2009; Hehman et al., 2015; Spivey & Dale, 2006; Warren, 2006) and ergonomics (Guastello, 2017).
Mouse-Tracking Research in Human-Computer Interaction
Mouse-tracking (Freeman, 2018; Grimes & Valacich, 2015; Jenkins. et al., 2019; Jenkins et al., 2015; Kieslich et al., 2019a; Spivey & Dale, 2006) is increasingly used to examine spatiotemporal dynamics of essential HCI tasks (Gideon et al., 2009; Grimes et al., 2013; Hehman et al., 2015; Hibbeln et al., 2014; Jenkins. et al., 2019). Building upon Fitts’ law (Fitts, 1954; Fitts & Peterson, 1964), mouse trajectory analysis can support inferences regarding users’ behavioral intentions (Katerina & Nicolaos, 2018). Prior research has shown that mouse movements can be used to assess personality (Khan et al., 2008), mood (Zimmermann et al., 2003), self-efficacy (Dijkstra, 2013), and other psychological constructs including emotional valence and arousal (Grimes et al., 2013). User experience measures can also be derived from mouse tracking data, including whether an experience is pleasant (Navalpakkam & Churchill, 2012). Generally, dual-systems and dynamical systems models of cognition provide clear predictions regarding mouse trajectories toward competing response alternatives (Kieslich et al., 2020; Stillman et al., 2018). The dynamical systems model generally predicts “curved” or “curved change of mind” trajectories (Figure 1: curved, cCoM); in contrast, the dual-systems model generally predicts straight trajectories or trajectories with discrete “jumps” (Figure 1: straight, dCoM, dCom2). Bimodal response distributions for “curvature indices,” such as area under curve (AUC), are also characteristic of dual-processing cognitive strategies in mouse-tracking studies (Freeman & Dale, 2013; Stillman et al., 2018). Prototype trajectories. Prototype trajectories: cCoM = continuous change of mind; dCoM = discrete change of mind; dCoM2 = double change of mind. Used with permission of Springer Nature BV, from Kieslich et al. (2020). Design factors in mouse-tracking: What makes a difference? Behavior Research Methods, 52, 317–341; permission conveyed through Copyright Clearance Center, Inc.
The Present Study
We used mouse-tracking to examine evidence for two models of human cognition when website elements occur in unexpected locations. Ericson et al. (2021) placed web objects (e.g., cart) in expected (top right) and unexpected locations (top left) within modified images of real websites, and participants searched for target objects while mouse movements were continuously recorded. They found evidence for cognitive conflict when targets appeared in unexpected locations, and the results were generally consistent with a dynamical systems strategy. The present study extends their work through a trajectory prototype classification analysis (Kieslich et al., 2020; Wulff et al., 2019) that explicitly assesses the strength of evidence for dual-systems and dynamical systems models. Trajectories were classified based on similarity to prototype trajectories predicted by each model (Figure 1). In the present study, the dynamical systems model predicts that trajectories will exhibit smooth, continuous attraction toward competing response options (expected vs. unexpected locations) because both options are continuously and simultaneously evaluated (Spivey & Dale, 2006). Dynamical systems models generally predict that dynamic competition among response alternatives is resolved gradually during the task, and reflected in a graded range of continuously curved or “curved change of mind” (cCoM) trajectories and unimodal distributions of curvature indices. Area under the curve (AUC) served as the curvature index used in the present study (Figure 2b). The dual-systems model predicts that icons (e.g., cart) in unexpected locations (e.g., top left) will yield a mixture of trajectories that indicate either (a) a clear preference for a single target location as evidenced by trajectories directed straight toward the target (when System I and System II agree), or (b) discrete (dCoM) or double change of mind (dCoM2) trajectories as evidenced by an initial movement (guided by System I) toward the non-chosen target followed by an instantaneous corrective movement (guided by System 2) toward the chosen target (Kieslich et al., 2020; Wulff et al., 2019). Displays and assessment of area under curve (AUC). (a) Wireframe depicting the layout of one website. Actual displays were modified, high-fidelity images of real websites. For some icons, the expected location was top left (e.g., menu); for others, the expected location was top right (e.g., cart). A target appeared in only one of these locations on any given trial. (b) Visual depiction of how area under curve (AUC) between the ideal and actual mouse trajectories was determined in the present study.
METHODS
Here, we summarize Ericson et al.’s (2021) methods and dataset, focusing on aspects of their experimental design most relevant to our analysis.
Participants
101 college students (59F, 42M; MAGE = 19.8 years, SDAGE = 1.9 years) participated in the experiment and were randomly assigned to either group A or group B. This sample size was chosen to ensure adequate power (1 – β > 0.95) to detect medium effects (w = 0.5) (Erdfelder et al., 1996, 2009). Participants received a $10 Amazon eGift card. This research complied with the American Psychological Association Code of Ethics, tenets of the Declaration of Helsinki, and was approved by Bentley University’s Institutional Review Board (IRB). Informed consent was obtained from each participant.
Equipment
Displays were presented using OpenSesame [v 3.2.4; Python v. 2.7.13] (Mathôt et al., 2012) via a monitor (Hewlett Packard Z23i IPS; 1024 × 768 pixels) and desktop computer (64-bit Intel® CoreTM i5-4690S CPU @ 3.20 GHz with 8 GB of RAM, Windows 10). Participants used a wired mouse with sensitivity settings halfway between default “fast” and “slow.” Mouse trajectories (100 Hz sampling rate) were collected using Moustrap (Kieslich et al., 2016; Kieslich et al., 2019b; Kieslich & Henninger, 2017) for OpenSesame.
Displays
Sixteen images of real websites were modified using Adobe Photoshop and presented in .png format (800 × 600 pixels) within a 1024 × 768 pixel region (Figure 2a). Icons from Google’s Material Design (Silva, n.d.) were modified to match the aesthetics of each website.
Design
The experiment employed a between-subjects design. Participants were randomly assigned to one of two experimental groups (A, B). Control targets (Settings, Search, Privacy Policy, Log Out, Logo, Language, Help, Feedback, Careers, Apps) appeared in expected locations (e.g., Settings, top right). Probe targets (Cart, Menu, Account, Upload, Save, Home) appeared in unexpected locations (e.g., Cart, top left) (Figure 2). To avoid priming or learning effects that might reveal the manipulation, expected locations were based on empirical location typicality data (Ericson et al., 2021), and each group (A, B) encountered 3 of 6 probes (Group A: Cart, Menu, and Account; Group B: Upload, Save, and Home) in unexpected locations. Thus, each participant completed 16 trials in total, of which 3 were designated probe-unexpected trials, and 3 were effectively additional control trials (probe-expected trials). Control trial results are plotted in Figure 3, and probe trial results in Figure 4. Mouse trajectory classification results: Control trials. Top row: Trajectory prototypes: Middle rows: Classification of actual trajectories into trajectory prototype categories, expressed as percentages of the total number of trajectories for each experimental group (A, B). Bottom row: Bar graph of prototype classification results. Mouse trajectory classification: Probe trials. Top row: Trajectory prototypes: Middle rows: Classification of actual probe trial trajectories into trajectory prototype categories, expressed as percentages of the total number of trajectories for each experimental group (A, B). Bottom row: Bar graph of prototype classification results.
Procedure
After providing informed consent, training trials exposed participants to the visual search task instructions (find and click on the target as quickly and accurately as possible) using neutral stimuli (circles, squares, etc.). During the test phase, participants completed the visual search task (16 trials total). On each trial, instructions with the name of the target (e.g., cart) (5 secs) were followed by a fixation dot (1 sec). The mouse pointer was then automatically centered on the screen, and the website image was revealed. The participant then searched for and clicked on the target (maximum 15 secs). This “static start procedure” (i.e., mouse movements began after the website stimulus appeared) (Dale et al., 2007; Kieslich & Hilbig, 2014; Koop & Johnson, 2011; Mattek et al., 2016) was selected because “dynamic start procedures” (i.e., the initial mouse movement triggers the appearance of the website) risk revealing the manipulation (Scherbaum & Kieslich, 2018). After completing the exit survey, participants were debriefed and issued the incentive. On average, the entire experiment lasted 22 minutes (SD = 12 mins).
Data Analysis
Data were analyzed using R Studio (Team, 2016), and trajectories were analyzed using Mousetrap (Kieslich et al., n.d., 2016, 2019b; Kieslich & Henninger, 2017). Prior to classification, trajectories were time-normalized and remapped so that endpoints shared the same direction across trajectories. Analysis of both control and probe targets was conducted between groups.
Trajectory Classification
Trajectories were classified into prototypes (Figure 1) by clustering raw trajectories using k-means (Hartigan & Wang, 1979) to compute a dissimilarity matrix for all trajectory pairs.
Bimodality
The dual-systems model predicts more bimodal trajectories (Freeman & Dale, 2013; Stillman et al., 2018). Bimodality of area under the curve (AUC) was assessed using Mousetrap (Kieslich et al., n.d., 2016; Kieslich et al., 2019b; Kieslich & Henninger, 2017), which computes a bimodality coefficient (BC) (Pfister et al., 2013) and corresponding significance level (p-value) using Hartigan’s dip test (Hartigan & Hartigan, 1985). Hartigan’s dip test examines an alternative hypothesis (HA) of non-unimodality (bimodality or multimodality) against a null hypothesis (H0) of unimodality (1 mode) (Hartigan & Hartigan, 1985). The conventional cutoff for non-unimodality is BC >.555, and p < .05 (Freeman & Ambady, 2010; Kieslich et al., 2019b; Pfister et al., 2013).
Manipulation Checks
Manipulation check questions included in the exit survey revealed that 10/101 (10%) of participants reported explicitly noticing that some targets occurred in unexpected locations.
RESULTS
Trajectory Classification
Control Trials
Control trial results appear in Figure 3. Control trials were designed to compare inter-group performance for targets located only in expected (E) locations; inter-group differences in baseline task performance could explain performance differences on probe trials. A Pearson’s chi-squared test failed to reveal a significance between experimental groups (A, B), χ2 (4) = 0.69, p = .95, and the percentage of trajectories classified into each prototype differed by no more than 1% between groups A and B. Thus, control trial trajectories were similarly categorized into prototypes across the two experimental groups, suggesting that any performance differences on probe trials are due to encountering targets in unexpected locations. On average across groups A and B, 77.5% of control trial trajectories matched prototypes predicted by the dual-systems model (straight, dCoM, dCoM2; Group A, 77%; Group B, 78%), while 22% of control trial trajectories matched prototypes predicted by the dynamical systems model (curved, cCoM; Group A, 23%; Group B, 21%). Thus, even when targets occurred in expected locations, trajectories indicated a mix of discrete and continuous changes of mind, though there is no a priori reason to expect changes of mind on control trials.
Probe Trials
Probe trial results are summarized in Figure 4. The dynamical systems model predicts more curved and curved change of mind (cCoM) trajectories. A shift in the trajectories associated with either the dynamical systems or dual-systems hypotheses on probe trials would indicate that targets in unexpected (U) locations impact users underlying cognitive strategies. A chi-square test revealed a significant difference in trajectories to probe targets in unexpected locations, χ2 (4) = 44.9, p < .001; the relative percentage of trajectories predicted by the dual-systems model decreased (straight, dCoM, dCoM2; Δ = −12% total), while the relative percentage of trajectories predicted by the dynamical systems model increased (curved, cCoM; Δ = +11% total). This suggests that placing targets in unexpected locations led participants to shift from a discrete (dual-systems) strategy to a more continuous (dynamical systems) cognitive strategy.
Bimodality of Responses
Control Trials
Bimodality Results
Note: Bimodality coefficients (BC) and significance (p) values for individual targets in each of the two target sets (A, B) and for control/expected (E) and probe/unexpected (U) trials.
Probe Trials
BCs for AUC on probe trials were small (all BCs ≤ .27) and failed to reach significance (all ps > .05) for both target locations (expected, unexpected). Exploratory data analysis revealed a non-significant trend toward multimodality on probe trials: BCs were higher when targets were located in unexpected (BC = .27) than expected (BC = .05) locations. Kernel density estimates for control and probe (expected, unexpected) trials are plotted in Figure 5. However, non-significant BCs (>.555) are generally inconsistent with dual-systems models of mouse-tracking tasks (e.g., as defined by Freeman & Dale, 2013). Density plot: Area under the curve (AUC). Lines: Kernel density estimates for Area Under the Curve (AUC) values grouped by trial type.
Individual Probe Targets
Bimodality coefficients for individual probe targets (Table 1) did not reach significance (all ps > .05) despite a non-significant shift trend toward unimodality for 2 of the 3 items in Set A (cart, account), and a non-significant shift from unimodality to bimodality for the remaining target (cart).
In contrast, BCs exhibited were more mixed for Set B. The BC for upload was higher when it occurred in an expected location (BC = .46) than in an unexpected location (BC = .19), tentatively suggesting unimodality consistent with the dynamical systems model. BCs for home were similar regardless of target location (expected, BC = .43; unexpected, BC = .42). The BC for save (BC = .65) was the only BC higher than the bimodality threshold (>.555), and this occurred when save was placed in an unexpected location; however, Hartigan’s dip test for the BC did not reach significance. This latter result for save tentatively suggests that responses may have become more bimodal for save when it was in an unexpected location, potentially consistent with the dual-systems model. In sum, the bimodality results did not distinguish between the two cognitive models when considering individual probe targets.
DISCUSSION
As expected, placing icons in expected locations (control trials) revealed similar trajectories across the two experimental groups. On average, 77.5% of trajectories to targets in expected locations resembled prototypes predicted by the dual-systems model, while 22% resembled trajectory prototypes predicted by the dynamical systems model. Even when targets occurred in expected locations, trajectories indicated both discrete (dual systems) and continuous (dynamical systems) changes of mind, suggesting that participants were sometimes uncertain about the locations of some targets when they occurred in expected (E) locations. Overall, these results suggest that designers should keep in mind that placing icons in expected locations may not necessarily lead to straightforwardly predictable patterns of behavior.
In contrast, when targets appeared in unexpected locations (probe trials), the percentage of trajectories predicted by the dual-systems model decreased (−12%), while the relative percentage of trajectories predicted by the dynamical systems model increased (+11%). Contrary to dual-systems model predictions that generally apply to mouse-tracking research (e.g., as defined by Freeman & Dale, 2013), we found that comparing the bimodality of a key curvature measure (area under the curve, AUC) for targets occurring in expected vs. unexpected locations failed to reveal any significant effects for either control trials or probe trials (all ps > .05, all BCs <.555). Although our task did not include two simultaneously visible response alternatives as in other mouse tracking studies (e.g., Spivey & Dale, 2006), we found that change of mind trajectories accounted for between 15–16% of control trial and 12–23% of probe trial trajectories. This implies that the non-visible response alternative(s) were activated during the task for at least some participants (Kieslich et al., 2020).
Taken together, these results suggest that placing icons in unexpected locations led participants to shift from reliance on discrete (dual-systems) strategies to more continuous (dynamical systems) strategies. That is, rather than executing a ballistic movement toward the expected target location—which could indicate that the task is easy (Thibbotuwawa et al., 2012) or that the target has been found—participants instead execute a more continuous and curved movement. In turn, this suggests a shift to a more continuous cognitive strategy that makes use of online perceptual information to continually correct movements as new information is obtained over time. In sum, probe trial results support increased reliance on dynamical systems rather than dual-systems cognitive and behavioral strategies when target icons occur in unexpected locations.
Our results suggest an analogy between navigation in the physical world and website navigation. For example, the “behavioral dynamics” approach (Warren, 2006) to human behavior posits that behavior can be accounted for by modeling animals and environments as informationally and mechanically coupled dynamical systems (Kugler et al., 1980; Yates & Iberall, 1973). In this model and in the language of dynamical systems, solutions to tasks correspond to attractors, avoided states correspond to repellers, and behavioral transitions correspond to bifurcations (Warren, 2006). For example, in locomotion tasks where participants must walk across a virtual room from a starting location to a goal identified by a target pole, the goal pole acts like an attractor, an obstacle pole acts like a repeller, and an agent’s choice of specific routes to the goal pole correspond to bifurcations in the system’s dynamics (Fajen & Warren, 2003). Similar reasoning has been applied to mouse movements (Freeman, 2018; Freeman & Dale, 2013; Spivey & Dale, 2006; Zgonnikov et al., 2017). Applying this dynamical perspective to our findings suggests that the expected target location acts as an attractor that “pulls” on a participant’s decision process, yielding more curved mouse trajectories. Rather than prompting a discretely predictive (“either or”) search process characterized by explicitly planned trajectories, realizing that a target is not in the expected location prompts increased reliance on a continuous search strategy that leverages online perceptual information as the task unfolds. In other words, when a web target (e.g., cart) is an unexpected location (e.g., top left), we shift from a strategy of explicitly recalling and ballistically moving toward the most likely alternative target location to a strategy of smoothly adjusting our mouse trajectory as we continuously search for and update our estimate of where the target is actually located.
There are numerous situations in human-computer interaction in which speed and reaction time are critical. This is especially true for expert domains such finance, military, aviation, and healthcare. Understanding and identifying the cognitive strategies people use during a task can help designers create more responsive and personalized systems. Although the present study focused on the design of commerce websites, targets often need to be moved to unexpected locations in a wide variety of applications and other screen-based interfaces. As a result, the present study has broad implications for user experience design. For example, our results suggest that aggregating mouse movements from many individuals to identify shifts from dual systems to dynamical systems strategies could help user researchers and designers identify confusing design elements. In addition, experts often execute routinized sequences of actions in interfaces. For example, when interacting with an electronic medical record, a physician might regularly click on a series of interface elements (A, B, C) when selecting an appropriate medication for a patient’s medical condition. If the user’s cognitive strategy shifts prior to completing the final action (C), this may indicate that the physician is uncertain about the most appropriate medication to select, or how to select it. Because the physician may not be explicitly aware of this brief moment of hesitation, the interface could alert them to the possibility that they might be uncertain about the best medication to choose. Such a notification might enable them to make a more explicit and appropriate decision about a patient’s treatment plan.
Limitations and Future Directions
Here, we examine several limitations of our study. Because our study used a static starting procedure (see Methods) (Dale et al., 2007; Kieslich & Hilbig, 2014; Koop & Johnson, 2011; Mattek et al., 2016; Scherbaum & Kieslich, 2018), future research should compare the impact of both static and dynamic starting procedures when targets occur in unexpected locations. This could also help inform models of human motor control that apply in three dimensions (Jax et al., 2003) and virtual environments (Deng et al., 2019). Because our study focused on website use, future work should also a broader range of screen-based interfaces. For example, many healthcare organizations are evaluating electronic medical records (EHR) workflows to identify potential improvements (Vankipuram et al., 2019). Understanding the cognitive strategies that patients and healthcare professionals employ while interacting with medical interfaces has the potential to reduce critical errors that can have adverse consequences for patients. Assessing performance against these cognitive models could also contribute to research that has revealed age-related differences in mouse cursor control (Smith et al., 1999). Because we did not compare mouse tracking to eye tracking, future studies should also gather eye-tracking data to examine the relationship between mouse and eye movements when targets occur in unexpected locations. Combining mouse-tracking data with electroencephalography (EEG) data could contribute to research examining the relationship between mental workload and brain potentials during mouse use (Nittono et al., 2003). Because our study was not designed to assess the emotional impacts of websites and icons, it is possible that our data might reflect additional cognitive processes besides cognitive conflict, such as emotional valence and arousal (Grimes et al., 2013), or integral or incidental affective states (Blanchette & Richards, 2010); future work should therefore gather a broader range of psychophysiological and affective measures to parse out the influence of affect-related variables. Finally, to limit learning effects and avoid revealing the experimental manipulation, we employed a between-subjects design, and each participant completed a single trial for each icon. Future work should therefore examine individual differences in performance by obtaining estimates of within-subjects variability for individual targets.
CONCLUSIONS
Placing website elements in unexpected locations resulted in a shift from straight and change of mind trajectories to more curved trajectories (p < .001). Rather than employing a single cognitive strategy, users shifted from a primarily dual-systems to dynamical systems strategy when target icons occurred in unexpected locations. Realizing that the target was “out of place” prompted greater reliance on a continuous search strategy that leverages online perceptual information, rather than a discretely predictive (“either or”) search strategy and explicitly planned trajectories. This sheds light on the nature of underlying cognitive dynamics of interacting with websites, and contributes new evidence that mouse trajectories can serve as a powerful diagnostic tool for understanding where people expect interface elements to appear. Our results can inform algorithms and machine learning techniques that use mouse movements to infer user intentions and provide real-time assistance during interactions with screen-based interfaces. Our results could also inform automated A/B testing algorithms by helping to identify confusing tasks and features. Trajectory classification could help implicitly assess content novelty by quantifying shifts from dual to dynamical systems strategies. Our study also contributes data that may be useful to designers who develop personalization algorithms and adaptive interfaces that respond to long-term changes in users’ cognitive strategies over time.
Key Points
Participants searched for and clicked on targets in expected (e.g., cart, top right) or unexpected (e.g., cart, top left) locations while mouse trajectories were recorded. Results revealed an increase in trajectories associated with the dynamical systems model and decrease in trajectories associated with the dual systems model when targets occurred in unexpected locations. Users may shift from discretely predictive strategies to continuously adaptive strategies when targets occur in unexpected locations within a screen-based interface. Potential applications include the assessment of mental workload, interface personalization, and automated A/B testing.
