Abstract
Representations in visual working memory (VWM) influence attention and gaze control in complex tasks, such as visual search, that require top-down selection to resolve stimulus competition. VWM and visual attention clearly interact, but the mechanism of that interaction is not well understood. In the research reported here, we demonstrated that in the absence of stimulus competition or goal-level biases, VWM representations of object features influence the spatiotemporal dynamics of extremely simple eye movements. The influence of VWM therefore extends into the most basic operations of the oculomotor system.
Visual attention requires a means to specify and keep active the properties of the to-be-attended item in order to guide attention to locations in the visual field that contain task-relevant features (Bundesen, 1990; Desimone & Duncan, 1995; Duncan & Humphreys, 1989; Wolfe, 1994). Several theories rely on the assumption that visual working memory (VWM) is the substrate of this feature template (Bundesen, Habekost, & Kyllingsbaek, 2005; Desimone & Duncan, 1995). Indeed, VWM has been found to modulate orienting when multiple objects compete for selection, such that covert and overt attention are biased toward items that match VWM content (Han & Kim, 2009; Hollingworth & Luck, 2009; Mannan, Kennard, Potter, Pan, & Soto, 2010; Olivers, 2009; Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005; Soto, Humphreys, & Heinke, 2006). However, these biases could operate relatively late in processing and at a fairly high level of the visual system, where stimulus competition is maximal (Kastner & Ungerleider, 2000). In the experiments reported here, we sought to determine whether VWM can also modulate rapid, reflexive eye movements in the absence of stimulus competition.
Participants in these experiments executed a saccade to an abrupt-onset target whose color matched or did not match a concurrent VWM representation. In most trials, the target was the only stimulus in the display. Current theory and evidence diverge on whether VWM content should influence such simple orienting saccades. VWM involves sustained activation of feature-specific neural populations from the primary visual cortex through inferotemporal cortex (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009; Yoon, Curtis, & D’Esposito, 2006). Projections from visual cortex to the superior colliculus (SC; Fries, 1984) and frontal eye fields (FEF; Stanton, Bruce, & Goldberg, 1995) might allow VWM representations to influence even the most elementary oculomotor operations. However, current models of simple orienting saccades emphasize stimulus-driven processing, positing minimal influence of representations in working memory (Ludwig & Gilchrist, 2002; Mulckhuyse, van Zoest, & Theeuwes, 2008; van Zoest, Donk, & Theeuwes, 2004). For example, van Zoest et al. (2004) argued that saccades generated in less than approximately 200 ms are driven solely by the physical properties of the stimulus. Existing evidence on VWM-modulated orienting does not challenge this claim directly: Saccade latencies to memory-matching objects have been observed to be well over 200 ms, even when the matching object is an abrupt onset within the search array (Mannan et al., 2010).
In addition, current neurocomputational theories of oculomotor control hold that spatial information is the sole source of bias in saccade target selection (Kopecz & Schöner, 1995; Marino, Trappenberg, Dorris, & Munoz, 2012; Trappenberg, Dorris, Munoz, & Klein, 2001; Wilimzig, Schneider, & Schöner, 2006) and therefore do not predict that feature memory will influence orienting. Finally, stimulus competition plays a central role in theories of VWM-based selection (e.g., Desimone & Duncan, 1995), and it is not known if VWM will influence orienting when competition is eliminated. An effect of feature memory on rapid, reflexive saccades in the absence of stimulus competition and top-down selective demands would demonstrate that the influence of VWM extends to the most fundamental operations of the oculomotor system.
Experiment 1
In each trial of Experiment 1, participants maintained a color in VWM as they executed a saccade to a target disk (Fig. 1). First, a color square was presented, to be held in memory for a discrimination test at the end of the trial (color memory task; for the results of the color memory task, see Memory Results in the Supplemental Material available online). Next, participants completed the saccade task. A colored target disk appeared to the left or right of fixation, and participants oriented their gaze to this disk as quickly as possible. The color of the target disk either matched or mismatched the color in memory. The remembered color was irrelevant to the saccade task. Participants simply directed their gaze to the only visible object. This design eliminated goal-level biases in orienting. The use of single, isolated targets allowed us to examine the effect of VWM on saccades with latencies near the limit of human capabilities.
Illustration of the trial sequence in Experiment 1. Participants began by fixating a central cross (not displayed). A colored memory square was presented for 300 ms, followed by central fixation for 700 ms. A single, colored target object (here, represented as a black disk) was then presented to the left or right of fixation, and participants executed a saccade to that object as quickly as possible. After the target was fixated, there was a 200-ms delay before the presentation of the memory-test stimuli. Participants selected the color alternative that matched the color of the original memory square. The eye icons show horizontal fixation position throughout the trial. The illustrations at the bottom show examples of saccade-task stimuli in target-only trials and distractor-present trials (in which a single distractor appeared simultaneously with the target, either above or below fixation). In the target-only trials, the color of the saccade target could either match or mismatch the color of the memory square. In the distractor-present trials, the target and distractor were always different colors. Either the target matched the memory color, the distractor matched the memory color, or neither matched the memory color.
On a subset of trials, a distractor object was included in the target display (distractor-present trials). This allowed us to examine the effect of VWM match in a competitive context (Walker, Kentridge, & Findlay, 1995). The distractor disk appeared simultaneously with the target, either above or below central fixation (Fig. 1). Participants were instructed to ignore the distractor and to fixate the target, which appeared on the horizontal midline. The target, distractor, or neither of the two objects matched the color held in VWM.
Method
Participants
Participants were 18 to 30 years old and had uncorrected 20/20 vision. Twelve participants completed this experiment.
Apparatus and stimuli
Stimuli were displayed on a 17-in. CRT monitor (120-Hz refresh rate) at a viewing distance of 70 cm. The right eye was monitored by an SR Research EyeLink 1000 eye tracker (SR Research Ltd., Kanata, Ontario, Canada). Stimuli appeared on a gray background with a white central fixation cross (0.3°).
The central memory square subtended 1.6° × 1.6° visual angle. The color category of the square was chosen randomly from a set of three (red, green, and blue). Within the selected category, the specific color value was selected randomly from a set of four similar colors (details about the color values we used can be found in Supplementary Methods in the Supplemental Material). The memory-test display contained two squares: one in the memory color and one in a foil color selected randomly from the remaining three colors in the same category. The two squares appeared 2.5° to the left and to the right of center, with left/right assignment determined randomly. Using a within-category discrimination task minimized verbal encoding.
The saccade display contained only the target on 75% of trials (target-only trials) and the target plus a distractor on 25% of trials (distractor-present trials). The target (1° diameter) appeared on the horizontal midline. Its left/right location was selected randomly, and its eccentricity was selected randomly within a range (4.61°–7.06°). The target’s color was drawn either from the color category of the memory square (target match) or randomly from one of the two remaining color categories (no match). In target-match trials, the target’s color was either an exact match for the to-be-remembered color or an inexact match. On inexact-match trials, the color of the saccade target was later used as the foil color in the memory-test display. Using the target’s color as the foil in these trials made it counterproductive for participants to intentionally attend to the color of the target for the purpose of aiding memory for the original color square.
On distractor-present trials, the target was accompanied by a 0.66°-diameter distractor disk placed 1.3° above or below the fixation cross (its location was randomly determined). In the target-match condition, the target matched the color category of the memory square (exactly or inexactly), and the distractor did not. In the distractor-match condition, the distractor matched the color category of the memory square (exactly or inexactly), and the target did not. In the no-match condition, neither the target nor the distractor matched the color category of the memory square.
For both target-only trials and distractor-present trials, the saccade-latency and landing-error results were identical for exact and inexact color matches, so all eye movement analyses were collapsed across this variable.
Procedure
Each trial began with central fixation. After a delay of 400 ms, the color memory square appeared for 300 ms, followed by a blank screen (fixation cross only) for a delay of 700 ms (long enough to eliminate sensory persistence of the memory-square stimulus; Irwin & Yeomans, 1986). Then, the saccade display appeared. Participants were instructed to execute a saccade to the target as rapidly as possible and to avoid fixating the distractor if one appeared. Targets always appeared on the horizontal midline, so there was never a reason to make a vertical eye movement.
After a fixation was detected in the target region, the target display remained visible for 200 ms before the test display was presented. Participants then pressed one of two buttons to indicate which of the two test squares matched the color of the memory square.
After 18 trials of practice, participants completed 384 experimental trials. There were 288 target-only trials (144 no match, 144 target match) and 96 distractor-present trials (32 no match, 32 distractor match, 32 target match), randomly intermixed.
Data analysis
A combined velocity (> 30°/s) and acceleration (> 8,000°/s2) threshold was used to detect saccades. Trials were eliminated if the eyes were more than 1° from central fixation when the target stimulus appeared (8.0% of trials across all experiments) or if saccade latency was greater than 500 ms or less than 60 ms (4.4% of trials across all experiments).
Results
Target-only trials
VWM content influenced both the spatial and the temporal properties of saccade execution (for more detailed descriptions of the saccade results for all five experiments, see Eye Movement Results in the Supplemental Material). First, we assessed horizontal landing error (the absolute distance between the saccade landing position and the center of the target; see the left graph in Fig. 2a). Saccades landed closer to the target when the target matched the color in VWM (M = 0.76°) than when it did not (M = 0.87°), t(11) = 2.78, p = .02, with saccades to nonmatching targets tending to undershoot the target location to a greater extent than saccades to matching targets (see the right graph in Fig. 2a). Equivalent results were obtained when using the signed error instead of absolute error.
Distributions of horizontal landing-position error and eccentricity for target-only trials in (a) Experiment 1 and (b) Experiment 2 and distributions of horizontal landing-position eccentricity for distractor-present trials in (c) Experiment 1 and (d) Experiment 2. Dashed vertical lines show the range in which the target could appear, and solid vertical lines show the position of the distractor.
In addition, saccades to targets that matched the memory color in VWM had shorter latencies than saccades to nonmatching targets (M = 148 ms and M = 159 ms, respectively), t(11) = 7.22, p < .001.
Distractor-present trials
Figure 2c illustrates the landing-position eccentricity for the distractor-present trials. Forty-seven percent of saccades landed closer to the distractor than to the target when the distractor matched the memory color (11.5% landed within 1.5° of the distractor’s center). Only 15.2% of saccades landed closer to the distractor than to the target when neither object matched the memory color (2.0% landed within 1.5° of the distractor’s center). When the target matched the memory color, only 6.0% of saccades landed closer to the distractor than to the target (0.5% landed within 1.5° of the distractor’s center). All pairwise contrasts were reliable, ps < .05. Saccade latency exhibited a complementary pattern of results: Conditions with higher accuracies had shorter latencies (see Experiment 1 in the Supplemental Material).
Experiment 2
To generate even faster saccades, we repeated the procedure used in Experiment 1 but removed the fixation cross 100 ms before the onset of the saccade target. Participants were 18 to 30 years old and had uncorrected 20/20 vision. Twelve different participants completed this experiment.
As in Experiment 1, for target-only trials, saccade landing error was lower for target-match trials (M = 0.86°) than for no-match trials (M = 0.90°; Fig. 2b, left panel; see Fig. 2b, right panel, for landing-position eccentricity), but this difference did not reach significance, t(11) = 1.21, p = .25. Also as in Experiment 1, saccade latency was shorter for targets that matched the color in VWM (M = 125 ms) than for nonmatching targets (M = 136 ms), t(11) = 2.83, p = .02. Thus, VWM influenced oculomotor processes occurring within 150 ms of target onset. The results from distractor-present trials replicated those from Experiment 1 (Fig. 2d; for detailed results, see Experiment 2 in the Supplemental Material).
Experiments 3 Through 5
We conducted three control experiments (for detailed methods and results, see Experiment 3, Experiment 4, and Experiment 5 in the Supplemental Material).
Experiment 3: incidental memory for color
Participants were 18 to 30 years old and had uncorrected 20/20 vision. Twelve different participants completed this experiment. To rule out the possibility of a strategic bias to orient gaze toward a matching color, we made color an incidental feature of the to-be-remembered object. Participants remembered the orientation of a colored, star-shaped object (see Fig. S1 in the Supplemental Material). In the test display, they discriminated the original star from a foil differing by 10° of orientation. Because color had no relationship with the requirements of the memory task (orientation discrimination), there was no incentive for participants to attend to any particular color in the saccade display.
For target-only trials, the match between the color of the to-be-remembered object and the color of the saccade target had a reliable effect on the latency of the orienting saccade but no reliable effect on landing error. Results for the distractor-present trials replicated the results of Experiment 1 in full.
Experiment 4: controlling for priming
Participants were 18 to 30 years old and had uncorrected 20/20 vision. Twelve different participants completed this experiment. To rule out passive priming as an explanation for the effects observed in Experiments 1 through 3 (Bichot & Schall, 2002; McPeek & Keller, 2001), we followed the same procedure as in Experiment 1 but told participants that the color square at the beginning of each trial simply indicated that the trial was about to begin and did not administer a memory test at the end of each trial. Thus, participants viewed the color square but had no demand to remember it. All effects of color match on saccade landing error and latency were eliminated. Thus, the results of Experiments 1 through 3 were not due to priming.
Experiment 5: eliminating a demand to process distractors
Participants were 18 to 30 years old and had uncorrected 20/20 vision. Fifteen different participants completed this experiment. In Experiment 5, distractor-present trials were not included; all trials were target-only trials. This eliminated any experiment-level demand to discriminate the target from a distractor. In addition, to reduce the predictability of target location, we expanded the range of target eccentricity (to between 3.03° and 7.06°). Participants completed 400 trials: 200 match trials and 200 no-match trials. In all other respects, the method was the same as in Experiment 1. The effects of color match on saccade landing error and latency observed in Experiment 1 were replicated in full.
Discussion
These results demonstrate that even the most rapid and simple saccade targeting processes depend on a close integration of object information maintained in VWM and sensory input. Modulation of eye movements by VWM is not limited to situations in which stimulus competition must be resolved or to longer-latency saccades.
Four considerations point to a visual-sensory locus of the interaction between VWM and orienting observed in the experiments reported here. First, other sources of saccade modulation were not involved: Goal-level biases were controlled, as was spatial attention prior to target onset (target position was unpredictable). Second, an effect of VWM on saccades generated in less than 150 ms is consistent with the modulation of initial sensory input to oculomotor regions. Third, visual neurons in SC and FEF are not themselves color selective; color-memory effects are therefore likely to be generated in color- sensitive sensory systems that project to SC and FEF (White, Boehnke, Marino, Itti, & Munoz, 2009). Finally, evidence of feature-specific VWM activation in early visual cortex (Harrison & Tong, 2009; Serences et al., 2009) renders initial sensory modulation plausible. Facilitated sensory processing of memory-matching targets would then increase the speed and accuracy of the computation of the endpoint of the saccade. This same mechanism would increase the relative salience of an object in a competitive context, leading to the observed advantage for memory-matching targets and distractors in the distractor-present trials. These processes are analogous to feature-based attention mechanisms that modulate sensory processing at both attended and unattended locations (Zhang & Luck, 2009); our results show that feature-based modulation of sensory processing can be an automatic consequence of VWM maintenance.
The results contrast with earlier evidence that rapidly generated saccades are driven solely by the physical properties of the stimulus (Ludwig & Gilchrist, 2002; Mulckhuyse et al., 2008; van Zoest et al., 2004). They indicate that the initial perceptual “salience” of a stimulus (i.e., the efficiency of the sensory response or the strength of the sensory signal) is a joint property of the physical attributes of the stimulus and the match between those attributes and the content of VWM. Other authors have opposed this type of explanation, arguing instead that initial perceptual salience is equivalent for stimuli that match and stimuli that do not match a target template, but subsequent inhibition of a distractor is impaired when it shares features of the target (Mulckhuyse, Van der Stigchel, & Theeuwes, 2009). In this view, template properties do not influence initial salience; they only guide the later resolution of competition. Such an account cannot apply to the results reported here, however, because in the main condition, only a target was present: There was no competition and no demand for inhibition. Thus, the data reported here provide strong evidence that the initial perceptual salience of a stimulus is influenced by the strategic maintenance of object features.
Do these data reflect an early effect of top-down, goal-level factors on orienting? VWM content was orthogonal to task goals, and thus the effects observed cannot be described as goal directed; rather, there appears to be an obligatory interaction between the content of VWM and target processing. However, the dissociation between VWM content and the features of the target in our experiments is somewhat artificial. Under normative conditions, VWM would represent the target template, and the same mechanism observed here would facilitate rapid orienting to a target. Thus, our results are consistent with theories of selection emphasizing that attentional orienting—including extremely rapid orienting—is contingent on a representation of target features (Folk, Remington, & Johnston, 1992).
Footnotes
Declaration of Conflicting Interests
The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.
Funding
The experiments were supported by National Institutes of Health Grant R01EY017356.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
