Abstract
To find a target during visual search, observers often need to make multiple eye movements, which results in a scan path. It is an open question whether the saccade destinations in scan paths are planned ahead. In the two experiments reported here, we investigated this question by focusing on the observer’s ability to deviate from potentially planned paths. In the first experiment, the stimulus configuration could change during the initial saccade. We found that the observer’s ability to deviate from potentially planned paths crucially depended on whether altered configurations could be processed with sufficient rapidity. In a follow-up experiment, we asked whether planned paths can include more than two saccade destinations. Investigating the influence of potentially planned paths on a secondary task demonstrated that planned paths can include at least three saccade destinations. Together, these experiments provide the first evidence of scan-path planning in visual search.
The decrease in visual resolution toward the retinal periphery often requires observers to make eye movements to find an object. Typically, several eye movements are required, and the resulting sequence is referred to as a scan path. It is tempting to assume that each eye movement is based on a decision made during the directly preceding fixation, which would imply that a scan path is the result of a sequence of individual decisions. However, it is also possible that the scan path, or at least part of it, is planned ahead. That is, a number of upcoming saccade destinations may be decided on at once.
Outside of research on saccades, it has long been established that many motor actions are planned as sequences, for example in speech and typing (Sternberg, Monsell, Knoll, & Wright, 1978) and in finger tapping (Rosenbaum, Inhoff, & Gordon, 1984). Nevertheless, saccades are often studied as isolated reflexive responses. The reflexive nature of saccades is evident from studies demonstrating that, regardless of target properties, initial saccades are often biased toward conspicuous locations (e.g., Van Zoest & Donk, 2004) and visual onsets (e.g., Theeuwes, Kramer, Hahn, & Irwin, 1998). This bias is also reflected in many models of oculomotor selection, such as the influential salience model (Itti & Koch, 2000), that use stimulus contrasts to predict fixation locations. On the contrary, vast amounts of evidence show that the observer’s goal is crucial in ocolumotor selection. Some recent studies have even suggested that the predictive power of saliency models might be confounded because selection is biased toward objects, which are inherently in locations with contrasts (e.g., Henderson, Brockmole, Castelhano, & Mack, 2007; Stirk & Underwood, 2007).
Despite the fact that Zingale and Kowler (1987) demonstrated the ability to prepare sequences of saccades, evidence of scan-path planning in search is limited. Search tasks have been used to study the influence of visual input in the execution of saccade pairs (e.g., Araujo, Kowler, & Pavel, 2001; Caspi, Beutter, & Eckstein, 2004; McPeek, Skavenski, & Nakayama, 2000), and these studies have demonstrated that second saccade destinations can be based on visual input presented during the first fixation. However, in these studies, peripheral input could reveal the target location (or a location having a higher likelihood of holding the target). Hence, when the initial saccade aimed for a distractor, peripheral input during the initial fixation may have provided an incentive to initiate programming of a second saccade to override the erroneous first one. This second saccade was typically preceded by a short intersaccade interval. In many search tasks in daily life, however, distinguishing between target and distractors requires foveation of objects—that is, orienting the eye so that the object falls on the part of the retina (the fovea) that yields the greatest resolution. Whether an object is the target can be determined only after a saccade has been executed. In contrast to tasks in which the target can be detected peripherally, the resulting scan paths of such exploratory saccades vary greatly in length. Whether scan paths are planned ahead in such saccadic search tasks remains unanswered.
There may be a lack of evidence for scan-path planning in the literature because scan paths can best be viewed as a series of individual decisions, but it could also be due to the ability to deviate from planned paths. That is, if a planned scan path can still be adapted during its execution, evidence for planning is difficult to uncover. Hence, any plausible planning model should also take into account the ability to deviate from planned paths. As both the processing of visual input and the preparation of a new saccade require time, we expect that this ability depends on whether new input can be processed with sufficient rapidity.
In the present experiments, we investigated whether scan paths are planned ahead in search. Experiment 1 focused on the second saccade and how the required processing time affects the ability to deviate from planned paths. Finding clear influences of planning, we extended the question in Experiment 2 to whether planned paths can consist of more than two saccades.
Experiment 1
Experiment 1 included two primary conditions of interest: In the no-switch condition, the stimulus configuration remained static throughout the trial. In the switch condition, the target could switch position with another stimulus element during execution of the first saccade. If the second saccade destination was planned ahead, the second saccade should aim for the old position of the target. If the saccade was not planned ahead, the previous stimulus configuration should not affect future saccade destinations.
Processing of visual input and saccade preparation both require time. Whether an upcoming saccade can deviate from a planned path may depend on whether stimulus adaptations can be processed quickly enough and, thus, be used to program the upcoming saccade. Luminance contrast can be used to manipulate processing time of visual input (e.g., Jakobsson & Johansson, 1992). Therefore, two contrast conditions were included: In the low-contrast condition, more time was required to distinguish target from distractors than in the high-contrast condition. If processing speed of stimulus elements determines whether an upcoming saccade can be programmed from scratch, observers should adapt to configuration changes more frequently in the high-contrast than in the low-contrast condition.
Method
Observers
Five observers (age range = 22–30 years) who worked or studied at Utrecht University participated voluntarily in the experiment. Four of the observers were naive to the purpose of the experiment, and the fifth observer was author J. P. De Vries.
Stimuli
Three disks (1.6° wide) were placed at the corners of an imaginary equilateral triangle on a gray background (42 cd/m2) at equal distance (11.8°) from fixation (Fig. 1a). Two of the disks were white (84 cd/m2), and the third was a darker shade. One of the white disks was the target and was distinguished from the other white disk (the distractor) by the addition of a small horizontal line. The distractor thus required foveation to be distinguished from the target. The dark disk (the nontarget) also contained a horizontal line, but it could be distinguished from the target peripherally because it was darker than the white disks. Depending on the contrast condition, the nontarget was either black (~1 cd/m2; high-contrast condition) or gray (51 cd/m2; low-contrast condition). Luminance intensities were chosen on the basis of a pilot study. The horizontal line distinguishing target from distractor was added only once the initial saccade was detected; this was done to ensure that the target could never be distinguished peripherally.
Example of a trial sequence in the low-contrast condition of Experiment 1. At the onset of each trial (a), the display contained three disks—a white target, a white distractor, and a dark nontarget with a horizontal line—presented at equal distances from a central fixation cross. The background was gray during the experiment (though it is shown here in white for purposes of illustration). Observers had been instructed to search for the target on each trial. A horizontal line appeared inside the target disk after the initial saccade was detected. On 50% of the trials (b), the positions of the three disks remained the same throughout the trial (no-switch condition). On the remaining 50% of trials (c), the target and nontarget would swap places if the initial saccade was made toward the distractor (switch condition).
Procedure
The observer’s task was to find the target as quickly as possible. Trials were initiated by pressing the space bar. After a stimulus-onset asynchrony (SOA) of random duration (range = 250–750 ms), the display appeared. Observers indicated that they had found the target by pressing the “0” key on a numeric keypad.
Except for the addition of the horizontal line in the target after the first saccade, display configurations in the no-switch condition remained static throughout the trial. In the switch condition, the target and nontarget location would swap positions, but only when the initial saccade was made toward the distractor. The trial sequence for the two conditions is depicted in Figure 1. There were 150 trials each for the high-contrast and low-contrast condition, which were randomly mixed.
Apparatus and eye movement analysis
Stimuli were programmed in MATLAB (The MathWorks, Natick, MA), generated on an Apple Macintosh G5, and displayed on a LaCie (Tigard, OR) 22-in. CRT monitor at a resolution of 1,024 × 768 pixels (120 Hz refresh rate). Eye movements were recorded using an EyeLink II eye tracker (SR Research, Kanata, Ontario, Canada; 500 Hz). The observer’s head was supported by a chin rest at a viewing distance of 64 cm. Observers viewed displays binocularly, but we recorded eye movements from the left eye only. See the Supplemental Material available online for saccade-detection parameters.
Analysis and results
Data from the switch condition were used to analyze whether saccades were planned ahead (results from the no-switch condition are presented in the Supplemental Material). Figure 2 shows the distributions of saccades toward the new and the old target location as a function of the intersaccade interval in the two contrast conditions. In the low-contrast condition, nearly all second saccades aimed for the old target position (i.e., its position during the initial fixation). This suggests that the second saccade is indeed planned ahead. In the high-contrast condition, the second saccade aimed for the new target position more frequently than in the low-contrast condition (paired t test, p < .05). Because more time is required to process low-contrast objects than high-contrast objects, these findings suggest that processing time is a crucial factor in determining the ability to deviate from a planned path. Moreover, Figure 2 shows that in the high-contrast condition, the underlying distributions of saccades toward the old and the new target location were not only different but also hardly overlapped: Saccades toward the new target location were preceded by longer intersaccade intervals than saccades toward the old target location. This suggests either that longer intersaccade latencies allowed for saccade preparation toward the new location or that on early detection of the configurational change, the response was delayed to allow for preparation of a saccade based on new input.
Results from Experiment 1: histogram showing the distribution of second saccade destinations on switch trials as a function of the preceding intersaccade interval (binned in 20-ms intervals). Results are shown separately for the low- and high-contrast conditions. The data were collapsed across observers.
Experiment 2
Having found that saccade destinations are planned ahead, we next investigated whether planned paths can include more than two saccade destinations. Thus, in Experiment 2, we focused on the next extension of the scan path, the third saccade.
Rationale
When a path is planned, saccade preparation can commence earlier. Hence, observers should be able to traverse a planned path faster than deviating from it. On the basis of this rationale, we created a novel paradigm that enabled us to compare the time required to traverse a potentially planned path with the time required to deviate from that path. To make this comparison, we needed to be able to predict the potentially planned path and force observers to follow it.
Predicting the path
To predict the path observers would follow, we needed to determine what elements observers would select and in what order they would select them. We addressed the former concern by accompanying the white target with two white distractors and three black nontargets (see Fig. 3a). A pilot study demonstrated that black nontargets were rarely fixated, so we knew that the targets and distractors would be selected rather than the nontargets. A vertical line appeared inside the target and a horizontal line appeared inside each distractor so target and distractors could be distinguished. To ensure target and distractor could not be distinguished peripherally, the target appeared as a distractor (i.e., it contained a horizontal line) until the initiation of the initial saccade.
Standard stimulus display, example of a cue trial, and types of cuing used in Experiment 2. The standard stimulus display (a) consisted of two white distractors (disks with horizontal lines), one white target (disk with a vertical line), and three black nontargets arranged in three columns, each of which contained one black and one white element (the target could appear in any column). The fixation point always appeared at the left of the stimulus elements to facilitate left-to-right scanning. Trials on which observers had to find the target were randomly intermixed with cue trials (b), in which all disks in the stimulus display were covered with dark gray masks as soon as the first saccade was detected. As the first saccade approached the left-most column, the mask covering the location previously occupied by the white disk in the middle column turned red (represented here by a dotted circle). As the eyes reached this first cue, it turned gray again, and one of the disks in the right-most column was cued. The predicted path (c; indicated by the dotted line in these examples) was through the white disks from left to right. In on-path cue trials (d), the first and second cues each appeared at the locations previously occupied by white disks. In off-path cue trials (e), the first cue appeared at the location previously occupied by the white disk, but the second cue appeared at the location previously occupied by the black nontarget.
To predict the exact scan path that observers would select, we also needed to know the order in which the elements in the display would be selected. Therefore, we organized the elements in three vertical columns of two elements each. Each column contained both a black nontarget and either a white target or a distractor. A fixation dot was placed to the left of the columns so that the most attractive procedure would be to scan the white elements from left to right (Fig. 3c).
Forcing the path
To compare eye movements traversing the potentially planned path with eye movements deviating from this path, we created cue trials (Fig. 3b), which were randomly intermixed in this experiment with search trials (on which all disks remained visible throughout the trial, as in the previous experiment). In cue trials, dark-gray masks covered all elements on detection of the initial saccade, and observers had to switch from searching for the target to following a red cue. When the first saccade approached the location previously occupied by the leftmost white element, a red cue appeared on the potentially planned path in the middle column (the location previously occupied by a white element). As the second saccade approached this cue, the disk reverted to a dark gray mask, and a disk in the right-most column turned red. This disk was either in the previous location of the white element (on-path condition; Fig. 3d) or in the nontarget location (off-path condition; Fig. 3e).
Distractor inhibition
When traversing the path takes longer in the off-path condition than in the on-path condition, an alternative explanation (compared with path planning) could be that distractor inhibition causes increased latencies toward cues on locations previously occupied by nontargets (e.g., Müller, von Mühlenen, & Geyer, 2007). To be able to examine this hypothesis, we split the off-path condition into two subconditions: the distractor condition and the empty condition. In the distractor condition (Fig. 4a), the right-most column contained both a white element (target or distractor) and a black nontarget, whereas in the empty condition (Fig. 4b), the right-most column contained only a white element (the other space was left empty). Other than this difference, trials in the search, on-path cue, and off-path cue conditions proceeded as described. Regardless of cue trial type, the trial was completed when the cue in the right-most column was fixated.
Examples of trials in (a) the distractor condition and (b) the empty condition in Experiment 2. At the start of all trials in the distractor condition, the right-most column contained both a white disk and a black disk; in the empty condition, the right-most column contained only the white disk (with the other location left empty). In search trials, all disks remained visible throughout the trial. In cue trials, all disks were covered by gray masks after the start of the first saccade. In on-path cue trials, the final cue appeared at the location of the white disk in both the distractor and empty conditions; in off-path cue trials, the final cue appeared either at the location of the black disk (distractor condition) or in the empty location (empty condition). The dotted arrows show the predicted scan paths, and the dotted circles show which disks were cued in each type of trial.
Method
Observers
Eight observers who worked or studied at Utrecht University participated in Experiment 2. Six of the observers were naive to the purpose of the experiment, and the other two observers were authors J. P. De Vries and I. T. C. Hooge.
Procedure
The apparatus and SOAs were the same as in Experiment 1. Likewise, observers were instructed to search for the target and press “0” when they identified it, and they initiated each trial by pressing the space bar. Stimulus properties were similar to those of Experiment 1; however, disks were slightly larger (1.8º). Also, to test our hypothesis, we used a more complex display than in Experiment 1, and we varied the procedure by including cue trials as outlined above (further details can be found in the Supplemental Material).
Analysis and results
Data from the cue trials were used to evaluate whether the third saccade was planned ahead (search-trial data are reported in the Supplemental Material). The cue latencies for the second cue (appearing prior to the third saccade) are presented in Figure 5a. Cue latencies were calculated by subtracting the onset time of the cue from the moment the eye landed on the respective cue. As expected, observers were faster to complete potentially planned paths than to complete unplanned paths (25 ms faster in the on-path condition compared with the off-path condition; paired t test, p < .05).
Results from cue trials in Experiment 2: mean cue latencies for the second cue (preceding the third saccade) as a function of path condition. Cue latencies were calculated by subtracting the onset time of the cue from the moment the eye landed on the cue. Results are shown in (a) for on- and off-path cue trials in the distractor and empty conditions combined. Results are also broken down separately for on- and off-path cue trials in (b) the distractor condition only, in which the second cue appeared on a location previously occupied by a nontarget, and (c) the empty condition only, in which the second cue appeared on a previously empty location. Error bars show standard errors of the mean.
To verify that distractor inhibition cannot explain the current results, we also examined the two subconditions: the distractor condition and the empty condition. For both trial types, cue latencies were longer in the off-path condition than in the on-path condition (distractor condition: 31 ms, p < .05, Fig. 5b; empty condition: 35 ms, p < .05, Fig. 5c). Although these results are in line with scan-path planning, they cannot be attributed to distractor inhibition.
General Discussion
In two experiments, we studied the extent of scan-path planning in visual search. The results of Experiment 1 suggest that scan paths are planned ahead and that the ability to deviate from a previously planned path depends on whether new visual input can be processed sufficiently quickly. Experiment 2 elaborated on these findings and demonstrated that planned paths may entail up to three saccades. These findings provide the first concrete evidence of scan-path planning in search. Moreover, they add to the evidence that oculomotor selection is less reflexive than often portrayed; despite the use of a highly conspicuous cue, observers were slower to follow the cue when it moved off an already planned path.
We can only speculate on the mechanisms behind this planning. Even though the vast amount of literature on parallel processing (e.g., Becker & Jürgens, 1979; Ray, Schall, & Murthy, 2004) makes it tempting to assume that parallel processing of saccades underlies the current results, more abstract versions of planning are also possible. For instance, prioritizing locations in memory could also underlie path planning without requiring advance preparation of saccades. This could be implemented by having a queue of upcoming saccade destinations stored in memory that is automatically accessed for the preparation of upcoming saccades. As such, resources concerned with target detection could be focused primarily on the currently fixated area. At the same time, if new input does reveal new potential targets, this more adaptive version of path planning allows for new locations to be incorporated in the planned path.
It should be noted that the use of these simple structured displays leaves open the question whether the current findings extend to search in daily life. However, as scan paths were planned ahead in our experiments, it is likely that this ability does not merely exist for search in simple displays. Moreover, even in complex noisy surroundings, the observer’s performance approximates that of an ideal observer using peripheral input to select locations that maximize the probability of locating the target (Najemnik & Geisler, 2005). Comparing simulated peripheral input with real-life targets has also been demonstrated to be a powerful tool in modeling search data (e.g., Zelinsky, 2008).
These above findings on selection in search appear contradictory to many studies on single saccades, in which typically goal-driven selection is limited. For instance, our previous findings suggest that at the moment of selection, some potential targets have not been processed yet (De Vries, Hooge, Wiering, & Verstraten, 2011). Also, it has been shown that during a substantial period before saccade initiation (saccadic dead time), peripheral input is irrelevant in determining the upcoming saccade destination (e.g., Ludwig, Mildinhall, & Gilchrist, 2007). Nevertheless, as detection thresholds for peripheral features are time dependent (Geisler & Chou, 1995), the chance that a target is detected (and distractors rejected as potential targets) increases during the dead time. Through planning, this information can benefit the selection of future saccade destinations. Thus, planning is potentially essential for efficient search. Further, whereas the order of visiting objects might not seem important in laboratory settings when the head is fixed toward a small display, scenes in daily life span wider scopes and generally also require head movements. Thus, path planning can have considerable benefits.
Footnotes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
