Building Bridges: Visual Search Meets Action Control via Inter-Trial Sequence Effects

Inter-trial Priming Effects in Visual Search

Our information processing capacity is limited, and our motor behavior can only be directed to a limited subset of stimuli. Selective attention refers to the collection of mechanisms that allow us to prioritize some stimuli in our environment over others. In the lab, mechanisms of attentional selection are often studied using visual-search tasks.

In a typical experiment, participants are instructed to search for a target and to respond to some aspect of that target, on each trial. For instance, the target might be defined as the uniquely colored object (or singleton) among homogeneous nontargets (e.g., the only red diamond among green diamonds or vice-versa, Bravo & Nakayama, 1992) and participants are instructed to make some discrimination with regard to this target (e.g., determine whether the left or right corner of the target diamond is chipped). All stimuli, the target and distractors alike, are equally likely to have each of the possible reported features, so that participants cannot find the target based on its reported feature. In addition, the values of the reported feature (chipped corner on the left or right) are chosen so that it is necessary to focus attention on the target in order to identify the reported feature and select the correct response. The measure of interest is how long it takes for participants to respond (reaction times) and how often they are correct (accuracy) as a function of a variety of independent variables thought to affect search difficulty.

Many effects on search time have traditionally been interpreted in terms of how efficient attention is allocated toward the target, either in terms of how fast distractors are rejected (e.g., Duncan & Humphreys, 1989) or, from another theoretical perspective, in terms of how strongly attention is guided toward the target (Wolfe, 2021). Specifically, much research has focused on the interplay between aspects of the current stimulus display (stimulus-driven guidance) and the intentions of the observer (goal-directed guidance). However, other studies showed that past experience (also referred to as selection history), which cannot be readily assimilated to either stimulus-driven or goal-directed guidance, also determines search performance. In particular, unpredictable repetitions from previous trials were shown to speed search considerably.

Inter-trial Effects in Action Control

The ability to act according to an instructed action goal and the cognitive consequences thereof are typically analyzed in experimental paradigms that each focus on a specific aspect of an action. This research tradition employs a variety of quite distinct paradigms. We therefore first briefly describe the paradigms for which inter-trial effects were observed and how they relate to the control of action.

The Binding and Retrieval in Action Control (BRAC) Framework

As is clear from the foregoing review, most action-control tasks share a prime-probe structure (or trial n – 1 to trial n structure) ² and for each, an episodic-retrieval account has been invoked as an alternative to other paradigm-specific accounts. Recently, the Binding and Retrieval in Action Control (BRAC) framework has been suggested (Frings et al., 2020; Frings et al., 2015) as a common account for the vast corpus of findings on paradigm-specific action-control processes.

BRAC assumes that the different aspects of the stimulus configuration (stimulus, context, and cue), of the response (response goal, decision, and effector), and of the effect (sensory and affective) are integrated into a short-term memory entry that is labeled “event-file” in line with the Theory of Event Coding (TEC; Hommel, 2004). In addition, BRAC assumes that upon repetition of any feature of an event file, the event file containing this feature is retrieved and thereby can modulate current actions (i.e., the building of the current event file).

Binding and retrieval are treated as separable processes in the BRAC framework and BRAC describes action control in terms of dynamic event-file management (integration/binding of features into event files and the retrieval of event files). In each of the abovementioned action-control tasks, event-file binding happens during trial n – 1 (i.e., the prime trial) and if a feature of the event repeats on trial n (i.e., the probe trial), retrieval starts. The typical so-called “S-R binding effects” or “action effects” that emerge in these tasks are according to BRAC always a compound effect of a binding proper plus retrieval.

In addition, BRAC assumes that both top-down and bottom-up control exert their influences separately on binding and retrieval. For instance, top-down control mechanisms can modulate the binding process (e.g., features receiving much attention might be more likely to become integrated into event files) and/or the retrieval process (e.g., features that are ignored might be less effective retrieval cues). Likewise, they can influence binding and retrieval by fostering different levels of semantic representation (e.g., task rules, framing, mind sets, speed/accuracy tradeoffs, and instruction-based effects). In addition, some findings suggest “bottom-up control” of binding and retrieval, that is, modulations due to stimulus contingencies (Giesen & Rothermund, 2015), emotional stimuli (Waszak & Pholulamdeth, 2009), and perceptual configurations (as a result of Gestalt mechanisms; Laub et al., 2018; Frings & Rothermund, 2017) reflect influences on binding and retrieval processes by variables not originating in the observer. Location also influences binding and retrieval as repetitions versus changes of the locations of the stimuli to which one responds influence whether features and responses are bound in a binary fashion (when location repeats) or whether all stimulus features are integrated into a single object and this feature compound is bound to the response (when location changes; Singh & Frings, 2020).

Taken together, BRAC can potentially integrate paradigm-specific findings on sequential-action effects in terms of dynamic event-file management (Beste et al., 2023). We now go a step further and link the sequential effects in visual search and BRAC-centered sequential effects in action control in the next part of our article.

The Role of Episodic Retrieval for Inter-trial Priming Effects in Visual Search

While research on inter-trial priming in visual search has mainly focused on its attentional component—thereby giving the primary impetus for the notion that not only stimulus salience and goals but also selection history, guide attention (e.g., Luck et al., 2020)—it has largely neglected the role of episodic retrieval. Retrieval accounts have typically been lumped together into the camp that challenges weighting accounts of inter-trial priming in visual search, despite substantial dissimilarities between them. As a result, little effort has been expended to test the different episodic-retrieval accounts against each other. In addition, retrieval accounts that emphasize the stage of motor-response selection have been discussed separately for feature, location-, and dimension-repetition effects, with no overarching model accounting for the findings from these three lines of research.

What insights has this research yielded so far? Evidence for the idea that inter-trial priming reflects retrieval of prioritization rules is scarce: the idea entirely relies on the finding that in a conjunction-search task where, on each trial, a cue indicated the target-defining features of the upcoming target, target-feature repetitions sped search but did not reduce search slopes (Hillstrom, 2000)—a finding that is open to many alternative accounts (e.g., Amunts et al., 2014; Ramgir & Lamy, 2021; Wolfe et al., 2022).

Likewise, the claim that inter-trial priming speeds the decision of whether a candidate target is, in fact, the target, also relies on the findings from a single study (Huang et al., 2004). The authors found that the benefit of repeating the target-defining feature was strongly modulated by repetition of its reported feature as well as by repetition of an irrelevant feature of the target. They concluded that selection of the target is slowed when repetition from the previous trial is partial relative to a complete repetition or change (nicely fitting the standard explanation from action-control approaches). However, they did not report whether the three-way interaction between the three types of repetition was significant, as their account would predict: indeed, such an interaction should emerge if any partial repetition slows selection relative to a complete change. In addition, it was not possible to determine whether repetition of the reported feature or of the motor response drove the interaction with the target-defining feature, because these were confounded in that study.

The latter limitation also applies to most of the studies held to support the idea that episodic retrieval affects motor responses in visual search. The main piece of evidence in these studies is that the benefit of repeating the target-defining feature on successive trials (e.g., Lamy et al., 2010; Lamy et al., 2011; Lamy & Kristjánsson, 2013; Yashar et al., 2013), target location (e.g., Gokce et al., 2014; Hilchey et al., 2018), or target-defining dimension (e.g., Töllner et al., 2008) is typically much larger when the reported feature repeats than when it changes. Yet, only two studies addressed the distinction between repetition of the motor response and repetition of the reported feature.

Töllner et al. (2008) used event-related brain potentials (ERPs) and examined whether the interaction between dimension repetition and repetition of the reported feature (or motor response) modulated the lateralized readiness potential (LRP). They found the interactive pattern only in the latency of the stimulus-locked LRP but not in the latency of the response-locked LRP. As stimulus-locked LRP differences may reflect differences pertaining to any processing stage(s) occurring prior to response planning, these findings constrain but do not pin down the exact stage at which the interaction emerged.

Yashar and Lamy (2011) adopted a different approach and associated two rather than just one reported feature to each of two alternative motor responses (i.e., one button press, if either the left or right corner of the target diamond was chipped, and another button press if either the upper or lower corner of the target diamond was chipped). This design allowed them to test the effects of motor-response and reported-feature repetitions independently. The authors found that only repetition of the motor response interacted with repetition of the target-defining feature, thereby supporting the idea that episodic retrieval affects the selection and/or execution of motor responses during visual search.

Current research on the role of episodic retrieval in inter-trial priming effects during visual search suffers from three main limitations. First, unlike in the action-control literature, only a minority of studies examined interactions between different types of repetitions, whereas examining these interactions is the gold-standard tool to study episodic retrieval in the action-control literature. For example, Kristjánsson (2006) conducted a systematic investigation of the effects of repeating target-defining, reported and irrelevant features on three different feature dimensions but did not report the interactions between these effects. In a later study, Kristjánsson (2009) reported the interactions between relevant and irrelevant features but did not examine whether these were modulated by response repetitions—which is crucial to link attentional selection and motor control processes. More generally, in its current state, the visual-search literature does not provide a detailed answer as to whether each feature of the target object retrieves the response associated with this object independently, or whether the conjoined target features have a unique contribution in retrieving this response.

Second, as we showed in the previous section, very few studies disentangled stimulus-related repetitions (i.e., repetition of the target’s reported feature) from action-related repetitions (i.e., motor-response repetition), a distinction that is fundamental for bridging between episodic-retrieval accounts in the visual-search and action-control literatures.

Finally, very few studies have exploited known properties of episodic retrieval processes to investigate the role of these processes in visual search. One notable exception is a series of experiments conducted by Thomson and Milliken (2011, 2012, 2013). These authors reasoned that if inter-trial priming of the target- and distractor-defining features (aka, PoP, Maljkovic & Nakayama, 1994) reflects episodic-retrieval processes, these effects should be sensitive to contextual effects. Accordingly, they explored whether repeating versus changing the task context or increasing the discriminability of a search trial by endowing it with rare target and distractor colors, with a rare location, or with a rare stimulus configuration would influence the effect. These authors reported mixed findings. On the one hand, task changes reduced inter-trial priming and the effect lasted considerably longer for rare than for frequent-color trials. On the other hand, there was no effect of rare versus frequent locations or configurations. Crucially, however, the authors did not examine interactions involving motor responses. Relatedly, Lamy et al. (2011) showed that response retrieval can rely on more global aspects of the search episode: they found that response repetition sped search when both the target and distractor colors repeated and slowed search when these colors swapped but had no effect on performance when either the target’s or the distractors’ color changed to a new color.

Taken together, there is to date little evidence that contextual information unrelated to the target’s or distractors’ features can trigger episodic retrieval of a similar previous trial, in tasks that involve search—unlike in the action-control literature.

The Role of Display Complexity and Salience in Action Control

While the previous section focused on the role of retrieval in visual search, discussed through the lens of action-control studies, we now focus on the selection stage (that has been neglected in action control research) against the background of visual-search theorizing. BRAC (Frings, et al., 2020) emphasizes that integration and retrieval are two independent processes that can be separately modulated by top-down and bottom-up influences. Different modulators can increase or weaken integration/retrieval strength (each separately), and thereby increase/reduce S-R binding effects.

Hommel (2004) had initially suggested that salience can modulate feature integration and that a more salient stimulus should therefore become more strongly integrated with a response. However, this author later developed a different view on integration and binding processes, according to which everything is unconditionally integrated without any effects of bottom-up factors such as salience (Hommel, 2005). Thus, the currently prevailing assumption is that all stimuli during the prime event are equally integrated with the response into an event file. Despite the strong implications of the role of bottom-up influences for action-control research, the empirical evidence on this issue is surprisingly scarce and inconsistent.

On the one hand, indirect support for the claim that bottom-up influences do not affect binding processes comes from recent studies that analyzed the influence of perceptual modulators of binding other than salience, namely, figure-ground segmentation (Schmalbrock & Frings, 2022) and grouping (Schmalbrock et al., 2022). In a nutshell, these studies showed that grouping and figure-ground segmentation modulate retrieval but do not affect binding/integration.

On the other hand, some recent studies explored salience effects on action control and reported findings that are at odds with the idea of unconditional binding. Complementing previous studies on the prioritization of action-control mechanisms by top-down modulators through feature weights (Memelink & Hommel, 2013), Schmalbrock et al. (2021) was the first study to highlight the role of the bottom-up stimulus salience. They explored how salience affects S-R integration in a variant of Hommel’s (1998) S1R1-S2R2-paradigm. They presented the relevant prime letter alongside seven filler letters. The relevant letter had either the same color as the filler letters (non-salient prime letter) or a different color (salient prime letter). S-R binding effects were found only for salient prime letters. These results were interpreted as evidence that salience modulates the integration stage—since the salience manipulation was only applied to the prime, the results could not reflect modulation of the retrieval stage. Relatedly, Qiu et al. (2022) showed that salient contexts (an accompanying tone with varying loudness or emotional content) boosted the integration of distractors and responses in a forced-four-choice auditory negative-priming task (Mayr & Buchner, 2006).

As is clear from the foregoing review, the role of salience in action control is still poorly understood. Because research on action control has focused on the demands on response selection, in most experimental tasks in this field displays consisted of very few stimuli, thereby effectively not taxing mechanisms responsible for stimulus selection (see Figure 2). In contrast, in visual-search studies, displays typically include many stimuli and the processes involved in stimulus selection therefore play a prominent role. Relying on the observation that their earlier finding of a salience effect on integration (Schmalbrock et al., 2021) was observed with prime displays containing an unusual large number of stimuli (eight), Schmalbrock et al. (2023) surmised that the effect of salience on S-R binding effects in action-control might be contingent on display complexity. To test this hypothesis, they contrasted prime displays with few versus many stimuli. They found that salience indeed impacted upon feature binding only in displays with many stimuli. These findings suggest that the effect of salience on feature integration and binding in previous action-control studies has likely been largely underestimated, because these studies used very few stimuli and therefore posed low demands on the selection stage.

For experiments in the laboratory that focus on specific aspects of response selection, the fact that stimulus selection has been neglected does not pose a problem. However, most theoretical frameworks in the action-control literature (such as TEC, Hommel, 1998, 2004 or BRAC, Frings et al., 2020) are meant to describe actions not only in the lab but also in the real world. If action-control theories claim this scope, they must acknowledge the impact of selection mechanisms and their determinants (such as salience) in perceptually more complex scenarios.