Building Bridges: Visual Search Meets Action Control via Inter-Trial Sequence Effects

Inter-trial Priming Effects in Visual Search

Our information processing capacity is limited, and our motor behavior can only be directed to a limited subset of stimuli. Selective attention refers to the collection of mechanisms that allow us to prioritize some stimuli in our environment over others. In the lab, mechanisms of attentional selection are often studied using visual-search tasks.

In a typical experiment, participants are instructed to search for a target and to respond to some aspect of that target, on each trial. For instance, the target might be defined as the uniquely colored object (or singleton) among homogeneous nontargets (e.g., the only red diamond among green diamonds or vice-versa, Bravo & Nakayama, 1992) and participants are instructed to make some discrimination with regard to this target (e.g., determine whether the left or right corner of the target diamond is chipped). All stimuli, the target and distractors alike, are equally likely to have each of the possible reported features, so that participants cannot find the target based on its reported feature. In addition, the values of the reported feature (chipped corner on the left or right) are chosen so that it is necessary to focus attention on the target in order to identify the reported feature and select the correct response. The measure of interest is how long it takes for participants to respond (reaction times) and how often they are correct (accuracy) as a function of a variety of independent variables thought to affect search difficulty.

Many effects on search time have traditionally been interpreted in terms of how efficient attention is allocated toward the target, either in terms of how fast distractors are rejected (e.g., Duncan & Humphreys, 1989) or, from another theoretical perspective, in terms of how strongly attention is guided toward the target (Wolfe, 2021). Specifically, much research has focused on the interplay between aspects of the current stimulus display (stimulus-driven guidance) and the intentions of the observer (goal-directed guidance). However, other studies showed that past experience (also referred to as selection history), which cannot be readily assimilated to either stimulus-driven or goal-directed guidance, also determines search performance. In particular, unpredictable repetitions from previous trials were shown to speed search considerably.

Inter-trial Effects in Action Control

The ability to act according to an instructed action goal and the cognitive consequences thereof are typically analyzed in experimental paradigms that each focus on a specific aspect of an action. This research tradition employs a variety of quite distinct paradigms. We therefore first briefly describe the paradigms for which inter-trial effects were observed and how they relate to the control of action.

Task Switching

The task-switching paradigm was invented to study cognitive control/flexibility (needed to perform goaldirected behavior) in dynamic and changing environments with different action demands. In order to analyze task-switching costs typically two different tasks are presented in alternation (see Kiesel et al., 2010; Koch, Poljac, et al., 2018 for reviews). Task-switching costs refer to the performance decrement on trial n when the task changes from trial n – 1 relative to when it repeats. Some researchers have argued that while performing one task, participants inhibit the other task (inhibition account, Koch et al., 2010; Mayr & Keele, 2000; Schuch & Koch, 2003). Others have suggested that when the task changes, participants have to re-configure the task set to a specific stimulus-response (S-R) mapping, a process that takes time (re-configuration account, Rogers & Monsell, 1995; for a similar idea in visual search, see the Dimension Weighting Account introduced above). Yet others have linked switching costs to task cues that trigger an episodic-retrieval process (Koch & Allport, 2006). The retrieval includes features of the previous task, the previous response, and/or the previous action goal (Arrington et al., 2007; Logan & Bundesen, 2003). In the case of a task switch, the retrieved features do not match the features of the current trial, leading to a performance cost. Further evidence that task switching costs are related to processes of binding and retrieval is provided by the ubiquitous finding that task-switch costs depend on whether the response switches or repeats from trial n – 1 to trial n (e.g., Altmann, 2011; Hübner & Druey, 2006; Koch, Frings, & Schuch, 2018; Mayr & Kliegl, 2003; Rogers & Monsell, 1995).

Negative Priming

The negative-priming paradigm was invented to investigate inhibitory control of distracting stimuli that is exerted while selectively responding to a target (Tipper, 1985; Tipper et al., 1994; for a recent review Frings et al., 2015). In this task, a target and a distractor are presented on each trial. When the prime distractor from trial n – 1 serves as the probe target on trial n, performance costs are observed relative to a neutral condition, where the probe target stimulus was absent on trial n – 1. Two main accounts have been suggested to explain negative-priming effects. Inhibition models (Houghton & Tipper, 1994) propose that the perceptual representation of the prime distractor is inhibited on trial n – 1 and that such inhibition persists into trial n, impairing probe processing in the case of a distractor-to-target repetition. By contrast, episodic-retrieval theories (Neill, 1997) posit that during probe processing, the last episode containing the probe target is automatically retrieved and when the retrieved information is incompatible with the current information (in the case of a distractor-to-target repetition) interference emerges. More recent accounts explain negative priming in terms of event-file binding in the prime trial and event-file retrieval in the probe trial (Rothermund et al., 2005; see also Mayr & Buchner, 2006). So, while the retrieval process seems comparable to what Neill (19997) suggested, the idea here is that the whole prime episode, including the response, is retrieved.

Stimulus-Response-Binding Tasks

The S1R1-S2R2 paradigm (Hommel, 1998) and the distractor-response-binding task (Frings et al., 2007) were invented to investigate the impact of feature/stimulus repetitions on action control. In the S1R1-S2R2 task, an arbitrary response (R1) is executed while a stimulus (S1) is displayed. In the next display, a second stimulus (S2) is presented, which the participant has to process, because it determines the second response (R2). If aspects of the stimulus and/or the response repeat between S1R1 and S2R2, facilitation (in the case of complete repetitions) or interference (in the case of partial repetitions) is observed relative to complete changes. In the distractor-response-binding task, a distractor appears simultaneously with the target on each trial. The distractor either repeats or changes between two consecutive displays, and so does the response to the target. Distractor-repetition effects are modulated by whether or not the response repeats between prime and probe. The prevailing explanation for the pattern of results in both tasks is that features of the stimuli (S1 or the distractor) are integrated with response codes on trial n – 1 and that stimulus repetition on trial n retrieves the previous response, which facilitates responding when it matches the currently required response (on trial n; complete repetition) and interferes with it in case of a mismatch (partial repetition).

Gratton Effect

The Gratton effect (for a review see, e.g., Verguts & Notebaert, 2008) refers to the sequential modulation of congruency effects, that is, the finding that the congruency effect in trial n is typically smaller when trial n – 1 is incongruent than when it is congruent. Research on the Gratton effect is particularly concerned with cognitive control and more specifically with how conflicting information is handled. Ignoring incongruent information on trial n seems to be easier if trial n – 1 was also incongruent. The conflict-monitoring approach (Botvinick et al., 2001) suggests that detecting a conflict (as in an incongruent trial) increases cognitive control, leading to less interference when conflict is encountered again in the subsequent trial. However, others have observed that this pattern is particularly strong when the stimulus repeats from trial n – 1 to trial n, which opens the door to an alternative explanation in terms of binding and retrieval (e.g., Davelaar & Stevens, 2009; Mayr et al., 2003).

Action-Planning Tasks

Humans can prepare themselves for acting. The key question in action-planning studies is how the mental preparation of an action in a prime trial impacts the initiation of a related action in a probe trial. In a typical action-planning task, the prime response (on trial n – 1) is not executed, then the probe response (on trial n) is executed, and only thereafter the prime response is executed. This sequential prime-probe procedure has proven to be an ideal tool to reveal the underlying structure of action plans (Kunde et al., 2002; Rosenbaum & Kornblum, 1982; Stoet & Hommel, 1999). Specifically, feature binding during action planning on trial n – 1 can hinder or facilitate the initiation of feature-overlapping actions on trial n, though the specific conditions and reasons for such costs and benefits have remained elusive so far. Explanations in terms of feature binding and retrieval extend previous theorizing in terms of the code-occupation hypothesis (which assumes that features of the planned response are bound and less accessible for the prime response; Fournier et al., 2015).

The Binding and Retrieval in Action Control (BRAC) Framework

As is clear from the foregoing review, most action-control tasks share a prime-probe structure (or trial n – 1 to trial n structure) ² and for each, an episodic-retrieval account has been invoked as an alternative to other paradigm-specific accounts. Recently, the Binding and Retrieval in Action Control (BRAC) framework has been suggested (Frings et al., 2020; Frings et al., 2015) as a common account for the vast corpus of findings on paradigm-specific action-control processes.

BRAC assumes that the different aspects of the stimulus configuration (stimulus, context, and cue), of the response (response goal, decision, and effector), and of the effect (sensory and affective) are integrated into a short-term memory entry that is labeled “event-file” in line with the Theory of Event Coding (TEC; Hommel, 2004). In addition, BRAC assumes that upon repetition of any feature of an event file, the event file containing this feature is retrieved and thereby can modulate current actions (i.e., the building of the current event file).

Binding and retrieval are treated as separable processes in the BRAC framework and BRAC describes action control in terms of dynamic event-file management (integration/binding of features into event files and the retrieval of event files). In each of the abovementioned action-control tasks, event-file binding happens during trial n – 1 (i.e., the prime trial) and if a feature of the event repeats on trial n (i.e., the probe trial), retrieval starts. The typical so-called “S-R binding effects” or “action effects” that emerge in these tasks are according to BRAC always a compound effect of a binding proper plus retrieval.

In addition, BRAC assumes that both top-down and bottom-up control exert their influences separately on binding and retrieval. For instance, top-down control mechanisms can modulate the binding process (e.g., features receiving much attention might be more likely to become integrated into event files) and/or the retrieval process (e.g., features that are ignored might be less effective retrieval cues). Likewise, they can influence binding and retrieval by fostering different levels of semantic representation (e.g., task rules, framing, mind sets, speed/accuracy tradeoffs, and instruction-based effects). In addition, some findings suggest “bottom-up control” of binding and retrieval, that is, modulations due to stimulus contingencies (Giesen & Rothermund, 2015), emotional stimuli (Waszak & Pholulamdeth, 2009), and perceptual configurations (as a result of Gestalt mechanisms; Laub et al., 2018; Frings & Rothermund, 2017) reflect influences on binding and retrieval processes by variables not originating in the observer. Location also influences binding and retrieval as repetitions versus changes of the locations of the stimuli to which one responds influence whether features and responses are bound in a binary fashion (when location repeats) or whether all stimulus features are integrated into a single object and this feature compound is bound to the response (when location changes; Singh & Frings, 2020).

Taken together, BRAC can potentially integrate paradigm-specific findings on sequential-action effects in terms of dynamic event-file management (Beste et al., 2023). We now go a step further and link the sequential effects in visual search and BRAC-centered sequential effects in action control in the next part of our article.

Theoretical Commonalities and Future Directions

Visual-search and action-control theorizing is typically based on classic sequential information processing models, in that it assumes a sequential processing stream broadly consisting of a selection, identification and response stages (see Figure 2). Finer-grained subdivisions of the processing stream have been suggested (e.g., Eimer, 2014; Pashler, 1998; Zehetleitner et al., 2012). For instance, Zehetleitner et al. (2012) also posited three sequential stages and divided each into two substages: the search stage includes attentional selection, followed by identification of the target-defining feature; the discrimination stage includes identification of the response-defining feature followed by response selection; finally, the response stage includes response planning followed by response execution.OPEN IN VIEWER

Figure 2.

Processing stages and focus of visual-search and action-control studies. These processing stages are compatible with sequential information-processing models suggested by other authors (e.g., Eimer, 2014; Pashler, 1998; Zehetleitner et al., 2012).

While it is unlikely that these stages proceed in a strictly serial order, the standard tasks used in visual search and action control can be well described by such sequential models. An important observation for the present purposes, however, is that the two research traditions have focused on different stages. Visual-search research is mainly concerned with the selection stage and only marginally with the identification stage, while action-control research is mainly concerned with the identification and response stages. However, as explained in the introduction, since the real-life behavior that each field aims at understanding, ultimately, is search for action, it is clear that all three stages need to be integrated. Such integration becomes even more critical if one acknowledges that not only do early stages influence processing in late stages but also late stages of previous search episodes can influence processing in early stages.

We therefore suggest that visual-search studies should integrate findings on response selection and planning from action-control studies. In particular, insights on the role of retrieval, which is a core explanatory construct in the action-control literature, should motivate further research drawing on the findings and methods of the action-control literature. Indeed, while all three processing stages are typically passed through when a new visual scene is encountered, retrieval from previous event files may provide shortcuts or modulate how each of these processes unfolds, in ways that are often not intended by the researchers (e.g., Henson et al., 2014). Likewise, we suggest that action-control research should integrate findings and theorizing from visual-search studies on selection. In particular, it is noteworthy that this research has typically used displays that are so simple (sometimes consisting of just a single letter) that attentional selection as typically examined via visual-search paradigms plays no significant role for task performance—a state of affairs that has led action-control researchers to neglect the selection stage. It is thus imperative to determine whether the conclusions drawn from the extant action-control literature generalize to more complex displays.

In the next sections, we review the few visual-search studies that have addressed the role of episodic retrieval in inter-trial priming as well as the few action-control studies that have employed more complex displays in sequential effects and point to the issues most in need of future research.

The Role of Episodic Retrieval for Inter-trial Priming Effects in Visual Search

While research on inter-trial priming in visual search has mainly focused on its attentional component—thereby giving the primary impetus for the notion that not only stimulus salience and goals but also selection history, guide attention (e.g., Luck et al., 2020)—it has largely neglected the role of episodic retrieval. Retrieval accounts have typically been lumped together into the camp that challenges weighting accounts of inter-trial priming in visual search, despite substantial dissimilarities between them. As a result, little effort has been expended to test the different episodic-retrieval accounts against each other. In addition, retrieval accounts that emphasize the stage of motor-response selection have been discussed separately for feature, location-, and dimension-repetition effects, with no overarching model accounting for the findings from these three lines of research.

What insights has this research yielded so far? Evidence for the idea that inter-trial priming reflects retrieval of prioritization rules is scarce: the idea entirely relies on the finding that in a conjunction-search task where, on each trial, a cue indicated the target-defining features of the upcoming target, target-feature repetitions sped search but did not reduce search slopes (Hillstrom, 2000)—a finding that is open to many alternative accounts (e.g., Amunts et al., 2014; Ramgir & Lamy, 2021; Wolfe et al., 2022).

Likewise, the claim that inter-trial priming speeds the decision of whether a candidate target is, in fact, the target, also relies on the findings from a single study (Huang et al., 2004). The authors found that the benefit of repeating the target-defining feature was strongly modulated by repetition of its reported feature as well as by repetition of an irrelevant feature of the target. They concluded that selection of the target is slowed when repetition from the previous trial is partial relative to a complete repetition or change (nicely fitting the standard explanation from action-control approaches). However, they did not report whether the three-way interaction between the three types of repetition was significant, as their account would predict: indeed, such an interaction should emerge if any partial repetition slows selection relative to a complete change. In addition, it was not possible to determine whether repetition of the reported feature or of the motor response drove the interaction with the target-defining feature, because these were confounded in that study.

The latter limitation also applies to most of the studies held to support the idea that episodic retrieval affects motor responses in visual search. The main piece of evidence in these studies is that the benefit of repeating the target-defining feature on successive trials (e.g., Lamy et al., 2010; Lamy et al., 2011; Lamy & Kristjánsson, 2013; Yashar et al., 2013), target location (e.g., Gokce et al., 2014; Hilchey et al., 2018), or target-defining dimension (e.g., Töllner et al., 2008) is typically much larger when the reported feature repeats than when it changes. Yet, only two studies addressed the distinction between repetition of the motor response and repetition of the reported feature.

Töllner et al. (2008) used event-related brain potentials (ERPs) and examined whether the interaction between dimension repetition and repetition of the reported feature (or motor response) modulated the lateralized readiness potential (LRP). They found the interactive pattern only in the latency of the stimulus-locked LRP but not in the latency of the response-locked LRP. As stimulus-locked LRP differences may reflect differences pertaining to any processing stage(s) occurring prior to response planning, these findings constrain but do not pin down the exact stage at which the interaction emerged.

Yashar and Lamy (2011) adopted a different approach and associated two rather than just one reported feature to each of two alternative motor responses (i.e., one button press, if either the left or right corner of the target diamond was chipped, and another button press if either the upper or lower corner of the target diamond was chipped). This design allowed them to test the effects of motor-response and reported-feature repetitions independently. The authors found that only repetition of the motor response interacted with repetition of the target-defining feature, thereby supporting the idea that episodic retrieval affects the selection and/or execution of motor responses during visual search.

Current research on the role of episodic retrieval in inter-trial priming effects during visual search suffers from three main limitations. First, unlike in the action-control literature, only a minority of studies examined interactions between different types of repetitions, whereas examining these interactions is the gold-standard tool to study episodic retrieval in the action-control literature. For example, Kristjánsson (2006) conducted a systematic investigation of the effects of repeating target-defining, reported and irrelevant features on three different feature dimensions but did not report the interactions between these effects. In a later study, Kristjánsson (2009) reported the interactions between relevant and irrelevant features but did not examine whether these were modulated by response repetitions—which is crucial to link attentional selection and motor control processes. More generally, in its current state, the visual-search literature does not provide a detailed answer as to whether each feature of the target object retrieves the response associated with this object independently, or whether the conjoined target features have a unique contribution in retrieving this response.

Second, as we showed in the previous section, very few studies disentangled stimulus-related repetitions (i.e., repetition of the target’s reported feature) from action-related repetitions (i.e., motor-response repetition), a distinction that is fundamental for bridging between episodic-retrieval accounts in the visual-search and action-control literatures.

Finally, very few studies have exploited known properties of episodic retrieval processes to investigate the role of these processes in visual search. One notable exception is a series of experiments conducted by Thomson and Milliken (2011, 2012, 2013). These authors reasoned that if inter-trial priming of the target- and distractor-defining features (aka, PoP, Maljkovic & Nakayama, 1994) reflects episodic-retrieval processes, these effects should be sensitive to contextual effects. Accordingly, they explored whether repeating versus changing the task context or increasing the discriminability of a search trial by endowing it with rare target and distractor colors, with a rare location, or with a rare stimulus configuration would influence the effect. These authors reported mixed findings. On the one hand, task changes reduced inter-trial priming and the effect lasted considerably longer for rare than for frequent-color trials. On the other hand, there was no effect of rare versus frequent locations or configurations. Crucially, however, the authors did not examine interactions involving motor responses. Relatedly, Lamy et al. (2011) showed that response retrieval can rely on more global aspects of the search episode: they found that response repetition sped search when both the target and distractor colors repeated and slowed search when these colors swapped but had no effect on performance when either the target’s or the distractors’ color changed to a new color.

Taken together, there is to date little evidence that contextual information unrelated to the target’s or distractors’ features can trigger episodic retrieval of a similar previous trial, in tasks that involve search—unlike in the action-control literature.

The Role of Display Complexity and Salience in Action Control

While the previous section focused on the role of retrieval in visual search, discussed through the lens of action-control studies, we now focus on the selection stage (that has been neglected in action control research) against the background of visual-search theorizing. BRAC (Frings, et al., 2020) emphasizes that integration and retrieval are two independent processes that can be separately modulated by top-down and bottom-up influences. Different modulators can increase or weaken integration/retrieval strength (each separately), and thereby increase/reduce S-R binding effects.

Hommel (2004) had initially suggested that salience can modulate feature integration and that a more salient stimulus should therefore become more strongly integrated with a response. However, this author later developed a different view on integration and binding processes, according to which everything is unconditionally integrated without any effects of bottom-up factors such as salience (Hommel, 2005). Thus, the currently prevailing assumption is that all stimuli during the prime event are equally integrated with the response into an event file. Despite the strong implications of the role of bottom-up influences for action-control research, the empirical evidence on this issue is surprisingly scarce and inconsistent.

On the one hand, indirect support for the claim that bottom-up influences do not affect binding processes comes from recent studies that analyzed the influence of perceptual modulators of binding other than salience, namely, figure-ground segmentation (Schmalbrock & Frings, 2022) and grouping (Schmalbrock et al., 2022). In a nutshell, these studies showed that grouping and figure-ground segmentation modulate retrieval but do not affect binding/integration.

On the other hand, some recent studies explored salience effects on action control and reported findings that are at odds with the idea of unconditional binding. Complementing previous studies on the prioritization of action-control mechanisms by top-down modulators through feature weights (Memelink & Hommel, 2013), Schmalbrock et al. (2021) was the first study to highlight the role of the bottom-up stimulus salience. They explored how salience affects S-R integration in a variant of Hommel’s (1998) S1R1-S2R2-paradigm. They presented the relevant prime letter alongside seven filler letters. The relevant letter had either the same color as the filler letters (non-salient prime letter) or a different color (salient prime letter). S-R binding effects were found only for salient prime letters. These results were interpreted as evidence that salience modulates the integration stage—since the salience manipulation was only applied to the prime, the results could not reflect modulation of the retrieval stage. Relatedly, Qiu et al. (2022) showed that salient contexts (an accompanying tone with varying loudness or emotional content) boosted the integration of distractors and responses in a forced-four-choice auditory negative-priming task (Mayr & Buchner, 2006).

As is clear from the foregoing review, the role of salience in action control is still poorly understood. Because research on action control has focused on the demands on response selection, in most experimental tasks in this field displays consisted of very few stimuli, thereby effectively not taxing mechanisms responsible for stimulus selection (see Figure 2). In contrast, in visual-search studies, displays typically include many stimuli and the processes involved in stimulus selection therefore play a prominent role. Relying on the observation that their earlier finding of a salience effect on integration (Schmalbrock et al., 2021) was observed with prime displays containing an unusual large number of stimuli (eight), Schmalbrock et al. (2023) surmised that the effect of salience on S-R binding effects in action-control might be contingent on display complexity. To test this hypothesis, they contrasted prime displays with few versus many stimuli. They found that salience indeed impacted upon feature binding only in displays with many stimuli. These findings suggest that the effect of salience on feature integration and binding in previous action-control studies has likely been largely underestimated, because these studies used very few stimuli and therefore posed low demands on the selection stage.

For experiments in the laboratory that focus on specific aspects of response selection, the fact that stimulus selection has been neglected does not pose a problem. However, most theoretical frameworks in the action-control literature (such as TEC, Hommel, 1998, 2004 or BRAC, Frings et al., 2020) are meant to describe actions not only in the lab but also in the real world. If action-control theories claim this scope, they must acknowledge the impact of selection mechanisms and their determinants (such as salience) in perceptually more complex scenarios.

Concluding Remarks

In their daily lives, humans select sensory input, coordinate it with current goals, and then select responses to interact with the environment. In cognitive-science research, these different processes have been largely insulated from each other, and specific research traditions, investigating the details of these stages, have developed separately. These research strands, which were originally motivated by the overarching question of how humans interact with their environment, have produced specific paradigms and/or phenomena and specific explanatory frameworks/theories tailored to these. While this approach has produced many intriguing insights into the mechanisms involved in selecting or responding, it has also led to considerable fragmentation in cognitive-science research, thereby impeding scientific progress (Balietti et al., 2015). There is thus an urgent need to increase efforts in theory-building (Eronen & Bringmann, 2021) and to better re-link psychological research fields that examine the very same concepts from different perspectives. We here suggest such a link between visual search and action control via striking structural similarities in task design. We argue that the commonalities between these lines of research should be emphasized and that findings and theories from one field hold a strong—yet hitherto unharnessed—potential to boost progress in the other.