Working Memory-Driven Attention in Real-World Search

Abstract

People’s attention is well attracted to stimuli matching their working memory. This memory-driven attentional capture has been demonstrated in simplified and controlled laboratory settings. The present study investigated whether working memory contents capture attention in a setting that closely resembles real-world environment. In the experiment, participants performed a task of searching for a target object in real-world indoor scenes, while maintaining a visual object in working memory. To create a setting similar to real-world environment, images taken from IKEA®’s online catalogue were used. The results showed that participants’ attention was biased toward a working memory-matching object, interfering with the target search. This was so even when participants did not expect that a memory-matching stimulus would appear in the search array. These results suggest that working memory can bias attention in complex, natural environment and this memory-driven attentional capture in real-world setting takes place in an automatic manner.

Keywords

working memory attention real-world search visual search

A stimulus matching the contents maintained in working memory powerfully captures attention (Downing, 2000; Han & Kim, 2009; Soto, Heinke, Humphreys, & Blanco, 2005). This working memory-driven attentional capture has been predominantly investigated by dual-task paradigms. In such a paradigm, participants are required to maintain a colored object in working memory. During the working memory maintenance interval, they are also required to search for a specific target among distractors. Importantly, each search item is presented in a colored place holder, which can have the memory-matching color. When the search target is located in the place holder with the memory-matching color (Valid trial), search performance (reaction time [RT] or accuracy) is enhanced, compared to when the memory-matching place holder contains a distractor (Invalid trial) or when the memory-matching color is absent in the search display (Neutral trial). This finding suggests that attention is biased toward memory-matching stimuli. Notably, it has been harshly debated whether this working memory-driven attentional bias is because stimuli matching the contents of working memory automatically capture attention (Olivers, Meijer, & Theeuwes, 2006; Soto et al., 2005; Soto, Humphreys, & Heinke, 2006) or because participants strategically bias attention toward the memory matching stimulus to refresh their memory (Carlisle & Woodman, 2011; Woodman & Luck, 2007). Despite this, it is commonly agreed that a working memory-matching stimulus is given priority for attentional selection.

While it is commonly agreed that a working memory-matching stimulus is given priority for attentional selection, an important issue that remains to be clarified is whether working memory-driven attention is also observed in naturalistic and real-life settings. Nearly all previous studies of working memory-driven attention used highly controlled and much simplified stimuli, such as simple geometric shapes with saturated colors. The use of these controlled stimuli is beneficial to unravel the fundamental, elementary nature of human cognition. However, recently, Peelen and Kastner (2014) emphasized that vision researchers should also pay attention to the development and use of experimental settings and stimuli that closely mimic real-world environment. In daily life, we rarely search for things like colored Landolt-Cs or tilted lines. Instead, a plate on the table (Ehinger, Allen, & Wolfe, 2016) or a weapon in a luggage bag (Biggs, Cain, Clark, Darling, & Mitroff, 2013) is often the target stimulus in real-world search. Hence, in the present study, we investigated whether working memory-driven attentional capture, which has mainly been observed in simplified and highly controlled laboratory settings, would also be found in real-world visual search.

Creating a visual search task that simulates real-world search is quite challenging. Many images in preexisting database are not readily usable for creating a real-world search task for the following reasons. First, for one to design a search experiment, a common search target should be included in each search display. Second, to investigate working memory-driven attention in real-world search, a dual-task paradigm consisting of a working memory task and a visual search task should be employed. To create such a paradigm, one need images of scenes to be searched and images of individual objects appearing in the visual search display as memory stimuli.

To overcome these limitations and investigate working memory-driven attentional capture in real-world search, we used images of IKEA® catalogue. This catalogue provides many kinds of indoor scene images containing real-world objects, whose images are also separately available. Taking advantage of this resource, we tested whether the maintenance of a visual object in working memory also evokes attentional bias toward memory-matching stimuli in real-world search.

Experiment 1

Participants

Sixty-seven (age: 19–25 years; 37 males) undergraduate students participated for course credit. The experimental protocol was approved by the Chungnam National University institutional review board. Informed consent was obtained from each participant. To determine an appropriate sample size, we ran a pilot test, which yielded the effect size of .489 (Cohen’s d). Based upon this effect size, it was estimated that a sample of 60 should be enough to detect a significant effect with the probability of .96. Hence, the current sample size of 67 should be sufficient.

Stimuli and Apparatus

The experiment was designed and run with Psychopy (Peirce, 2007). Stimuli were displayed on a 20-in. LCD monitor with a gray background. To create a setting similar to natural, real-world environment, we used images taken from IKEA®’s online catalogue. This catalogue includes images of real-world, indoor scenes such as living room, dining room, kitchen, and kids room. To create a real-world search task, we presented the image of a clock or a calendar (about 1.7 × 1.7 of visual angle), serving as the search target, in these indoor scene images (29° × 18°). Notably, the images of individual objects included in each scene were also separately available, as they were commercial products for sale. These individual object images served as memory samples (13° × 13°). There were two different types of trials. In the memory-matching distractor trials, a memory sample was presented in the visual search display, whereas in the memory-nonmatching distractor trials, no memory object appeared in the search display. Importantly, participants were not informed about the presence of memory-matching stimulus in the search display.

Design and Procedure

In each trial, participants performed a dual task, consisting of a working memory task and a visual search task (see Figure 1). At the beginning of each trial, an object image was presented for 5,000 ms. Participants were instructed to encode and maintain this memory sample in working memory. The presentation of the memory sample was followed by the 500-ms presentation of a mask. Then, a real-world visual search task was presented. In the visual search task, participants searched for a calendar or a clock and indicated whether there was a calendar or a clock via button presses.

Figure 1.

Depiction of a trial. In the actual experiment, the images from IKEA^® catalogue were used.

Importantly, there were two different types of trials: memory-matching and memory-nonmatching distractor trials. In the memory-matching distractor trials, a memory-matching object was presented as a distractor in the visual search display. The size these memory-matching distractors ranged from 2° × 0.6° to 6° × 4° of visual angle. By contrast, in the memory-nonmtaching distractor trials, participants memorized the object that would not be presented at the visual search array, such that no memory-matching stimulus was present in the search display. Participants were not informed of the presence of the memory-matching distractor trials. The search display remained until responses. Immediately following search responses, another real object was presented for memory test. Participants indicated whether the test object matches or does not match the memory sample.

The experiment included a total of 20 trials, consisting of 8 memory-matching distractor trials and 12 memory-nonmatching distractor trials. The initial four trials were all memory-nonmatching distractor trials. From the fifth trial, the memory-matching and memory-nonmatching distractor trials were randomly intermixed. We intended to include relatively a few number of trials in the experiment to prevent that participants, who might notice the presence of memory-matching stimuli in the search display, suspect about the relationship between the working memory task and the search task, thereby affecting their search strategy.

Results and Discussion

Data from the initial four trials, which were memory-nonmatching distractor trials, were excluded from the analysis as these served as practice. These trials were also to prevent that participants expect the appearance of memory-matching stimuli in the search display. Including these trials did not change the results.

To examine whether memory-matching stimuli capture attention during search, visual search RT data from trials with correct search and memory responses were subject to a repeated measures one-way ANOVA with memory-distractor match (memory-nonmatching vs. memory-matching) as a factor. This analysis revealed that the main effect of memory-distractor match was significant, F(1, 66) = 7.944, p = .006, $η_{p}^{2}$ = .107. Specifically, the mean RT for the memory-matching distractor trials was significantly longer than for the memory-nonmatching distractor trials (see Figure 2(a)). Mean accuracies of the search task and memory test were 94% and 90%, respectively (see Figure 2(b) and (c)). These did not differ across the types of trials, p’s > .30.

Figure 2.

Results of Experiment 1. Reaction time results (a). Accuracy of visual search task (b). Accuracy of memory task (c). Error bars represent standard errors of the mean. **p <.01.

These results suggest that the presence of the memory-matching stimulus in the search display captures attention, interfering with the search process. While this is consistent with a large body of previous studies (Downing, 2000; Han & Kim, 2009; Soto et al., 2005; Woodman & Luck, 2007), what is novel here is that working memory-driven attentional capture was found in complex, real-world environment.

Having shown working memory-driven attentional capture in real-world visual search, we examined whether such attentional capture is influenced by participants’ expectation regarding the presence of the memory-matching stimulus in the visual search display. Specifically, participants were not informed that some trials include a memory-matching stimulus in the search display. However, as the experiment proceeds, it is likely that participants notice the presence of memory-matching distractor trials. Knowing that memory-matching stimuli can be present in the search display might incentivize participants strategically attend to memory-matching stimuli during search, influencing the capture effect (Woodman & Luck, 2007). To investigate this issue, we ran an additional RT analysis, similar with a previous study (Asplund, Todd, Snyder, Gilbert, & Marois, 2010). As noted earlier, in the experimental session, there were eight memory-matching and eight memory-nonmatching distractor trials randomly intermixed, preceded by four practice trials, all of which were memory-nonmatching distractor trials. The 16 trials after the practice were divided into 4 epochs, each of which includes 2 memory-matching and 2 memory-nonmatching distractor trials. For example, the first epoch (Epoch 1) included the first and second memory-matching and memory-nonmatching distractor trials (Figure 3).

Figure 3.

RT results for each epoch. Error bars represent standard errors of the mean. *p <.05, Bonferroni corrected.

To examine whether the memory-driven attentional capture effect changes over time, we ran a two-way repeated measures ANOVA with memory-distractor match (memory-nonmatching vs. memory-matching) and epoch (first, second, third, and fourth) as factors. The results showed that the main effect of distractor type was significant, F(1, 66) = 9.582, p = .002, $η_{p}^{2}$ = .126. The main effect of epoch was not significant, F(3, 198) = 0.136, p > .9, $η_{p}^{2}$ = .002. The interaction between distractor type and epoch was not significant, either, F(3, 198) = 0.708, p > .70, $η_{p}^{2}$ = .006.

Even though the interaction was not significant, we applied separate pairwise t tests to each epoch for an exploratory purpose. As such, statistical thresholds for each t test were adjusted by Bonferroni correction. These t tests revealed that at the first epoch, the mean RT for the memory-matching distractor trials was significantly longer than for the memory-nonmatching distractor trials, t(66) = 2.58, p = .048, d = 0.315, Bonferroni corrected. At the other epochs, there were numerical differences across the trial types, which were not significant, p > .07.

These results suggest that working memory-driven attentional capture takes place without participants’ expectation regarding the presence of the memory-matching stimulus in visual search. Given that there was no interaction between epoch and memory-distractor match, we cannot draw strong conclusion about how expectation modulates the capture effect. However, it is clear that at the very early phase of the experiment, working memory-matching distractor powerfully captured attention. This suggests that the observed memory-driven attentional capture is involuntary and automatic.

Related to the aforementioned point, significant effect by the memory-matching distractor was found only at the first epoch but not at other epochs. We presume that this is because the current search task is highly attention-demanding, which usually dissipates the capture effect. In previous studies (Han & Kim, 2009; Woodman & Luck, 2007), when attentional demand for visual search was high, the memory-driven attentional capture effect was not well detected, compared to when the search demand was moderate. The current search, employing highly complex objects in real-world environment, might have recruited a high level of attentional control, necessitating serial allocation of spatial attention (Woodman & Luck, 2003); search RT was longer than those from previous studies.

Notably, even though the demand of attentional control for the present search task was high, the appearance of unexpected, memory-matching stimulus captured attention, interfering with the search process. As the experiment proceeds, participants might be familiarized with the presence of memory-matching distractors. In this case, the attentional control for search is fully implemented to handle the memory-matching distractors. Again, as the interaction between epoch and memory-distractor match was not significant, a strong conclusion cannot be drawn here. Further studies will be fruitful to elucidate this issue.

Experiment 2

This experiment aims at testing whether the result of Experiment 1 truly reflect working memory-driven attentional effect rather than priming evoked by the repeated presentation of the objects in the memory and search displays. Following an anonymous reviewer’s suggestion, in the present experiment, objects stimuli were sequentially presented as memory stimuli. Then, a symbolic cue denoted which object should be encoded and maintained in working memory. In the following visual display, a maintained object or a discarded object was included as a distractor. We predict that only a distractor matching the maintained object, but not a distractor matching the discarded object, captures attention and interferes with the search process.

Participants

Twenty (age: 19–25 years; 37 males) undergraduate students participated for monetary compensation. To determine the sample size, we considered a previous study using a similar paradigm, in which two memory stimuli and a post cue denoting which stimulus should be maintained in working memory (van Moorselaar, Theeuwes, & Olivers, 2014) were utilized. In Experiment 4 of this study, data from 18 participants were analyzed. Hence, we expected that a sample of 20 should be sufficient.

Stimuli and Apparatus

All stimuli and apparatus were identical to those of Experiment 1, except for the following. In the present experiment, two objects were sequentially presented. Then, a symbolic cue was presented to indicate what object should be memorized. Arabic numbers “1” and “2” were used as symbolic cues. The height of each symbolic cue was 1° of visual angle.

Design and Procedure

In this experiment, at the beginning of each trial, two objects were sequentially presented, with each stimulus being presented for 1,000 ms. Then, a symbolic cue (“1” or “2”), indicating which of the two should be maintained in working memory, was presented, followed by the presentation of a visual search display. The visual search array could contain the maintained object or the discarded object (Figure 4). All other details were identical to those of Experiment 1.

Figure 4.

An example depiction of a trial. In the actual experiment, the images from IKEA® catalogue were used.

Results and Discussion

The data analysis was done in a similar manner with Experiment 1. As shown in Figure 5, when the memorized object was presented as a distractor, search response was significantly slower than when the discarded object was present in the search display (M = 2,030 ms vs. M = 1,794 ms), F(1,19) = 4.797, p = .0412, $η_{p}^{2}$ = 0.202. Search accuracy and memory accuracy did not differ between different types of trials. This result suggests that the delayed search response by the presence of memory-matching distractor was not due to simple priming but due to memory-driven attentional capture. This result is also consistent with the claim that only an object actively maintained in working memory captures attention, but a discarded object or an accessory item in working memory does not (Olivers, Peters, Houtkamp, & Roelfsema, 2011).

Figure 5.

Results of Experiment 2. Reaction time results of visual search task (a). Accuracy of visual search task (b). Accuracy of memory task (c). Error bars represent standard errors of the mean. *p <.05.

General Discussion

The present study investigated working memory-driven attentional capture in natural, real-world visual search. In the process of searching for a target in real-world environment, a stimulus matching working memory contents captured attention, interfering with the target search. While this is a consistent result with a large body of studies regarding working memory-driven attention (Dowd & Mitroff, 2013; Dowd, Pearson, & Egner, 2017; Downing, 2000; Han & Kim, 2009; Soto et al., 2005; Woodman & Luck, 2007), what is novel of the present study is that similar working memory-driven attentional capture takes place in a setting that closely mimic natural, real-world environment.

Another notable feature of the present study is that a relatively few number of trials (20 trials) were included in the experiment and participants were not informed of the presence of memory-matching stimuli in visual search display. This was to prevent that any demand characteristics or expectation regarding the memory-matching stimuli affect the search process. The finding that significant capture was found in this short experiment points to the robustness of working memory-driven attentional capture in real world visual search.

To conclude, the present study demonstrates that that working memory contents capture attention in real-world environment. Similarly with the cases of simplified laboratory settings, this memory-driven attentional capture in real-world visual search seems to take place in an automatic and involuntary manner.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Research Foundation grant funded by the Korean Government (NRF-2016S1A5A2A02925551) and Chungnam National University Research Fund.

References

Asplund

C. L.

Todd

J. J.

Snyder

A. P.

Gilbert

C. M.

Marois

(2010). Surprise-induced blindness: A stimulus-driven attentional limit to conscious perception. Journal of Experimental Psychology: Human Perception and Performance, 36, 1372–1381. doi:10.1037/a0020551

Biggs

A. T.

Cain

M. S.

Clark

Darling

E. F.

Mitroff

S. R.

(2013). Assessing visual search performance differences between Transportation Security Administration Officers and nonprofessional visual searchers. Visual Cognition, 21, 330–352. doi:10.1080/13506285.2013.790329

Carlisle

N. B.

Woodman

G. F.

(2011). Automatic and strategic effects in the guidance of attention by working memory representations. Acta Psychologica, 137, 217–225. doi:10.1016/j.actpsy.2010.06.012

Dowd

E. W.

Mitroff

S. R.

(2013). Attentional guidance by working memory overrides salience cues in visual search. Journal of Experimental Psychology: Human Perception and Performance, 39, 1786–1796. doi:10.1037/a0032548

Dowd

E. W.

Pearson

J. M.

Egner

(2017). Decoding working memory content from attentional biases. Psychonomic Bulletin & Review, 24(4), 1252–1260. doi:10.3758/s13423-016-1204-5

Downing

P. E.

(2000). Interactions between visual working memory and selective attention. Psychological Science, 11, 467–473. doi: 10.1111/1467-9280.00290

Ehinger

K. A.

Allen

Wolfe

J. M.

(2016). Change blindness for cast shadows in natural scenes: Even informative shadow changes are missed. Attention, Perception, & Psychophysics, 78, 978–987. doi:10.3758/s13414-015-1054-7

Han

S. W.

Kim

M. S.

(2009). Do the contents of working memory capture attention? Yes, but cognitive control matters. Journal of Experimental Psychology: Human Perception and Performance, 35, 1292–1302. doi:10.1037/a0016452

Olivers

C. N.

Meijer

Theeuwes

(2006). Feature-based memory-driven attentional capture: Visual working memory content affects visual attention. Journal of Experimental Psychology: Human Perception and Performance, 32, 1243–1265. doi:10.1037/0096-1523.32.5.1243

10.

Olivers

C. N.

Peters

Houtkamp

Roelfsema

P. R.

(2011). Different states in visual working memory: When it guides attention and when it does not. Trends in Cognitive Sciences, 15, 327–334. doi:10.1016/j.tics.2011.05.004

11.

Peelen

M. V.

Kastner

(2014). Attention in the real world: Toward understanding its neural basis. Trends in Cognitive Sciences, 18, 242–250. doi:10.1016/j.tics.2014.02.004

12.

Peirce

J. W.

(2007). PsychoPy–Psychophysics software in Python. Journal of Neuroscience Methods, 162, 8–13. doi:10.1016/j.jneumeth.2006.11.017

13.

Soto

Heinke

Humphreys

G. W.

Blanco

M. J.

(2005). Early, involuntary top-down guidance of attention from working memory. Journal of Experimental Psychology: Human Perception and Performance, 31, 248–261. doi:10.1037/0096-1523.31.2.248

14.

Soto

Humphreys

G. W.

Heinke

(2006). Working memory can guide pop-out search. Vision Research, 46, 1010–1018.

15.

van Moorselaar

Theeuwes

Olivers

C. N. L.

(2014). In competition for the attentional template: Can multiple items within visual working memory guide attention? Journal of Experimental Psychology: Human Perception and Performance, 40, 1450–1464.

16.

Woodman

G. F.

Luck

S. J.

(2003). Serial deployment of attention during visual search. Journal of Experimental Psychology: Human Perception and Performance, 29), 121–138. doi:10.1037/0096-1523.29.1.121

17.

Woodman

G. F.

Luck

S. J.

(2007). Do the contents of visual working memory automatically influence attentional selection during visual search? Journal of Experimental Psychology: Human Perception and Performance, 33, 363–377. doi:10.1037/0096-1523.33.2.363