No Subliminal Memory for Spaced Repeated Images in Rapid-Serial-Visual-Presentation Streams

Abstract

Keywords

attention consciousness short term memory open materials

Recently in Psychological Science, Thunell and Thorpe (2019) demonstrated a repetition effect in rapid-serial-visual-presentation (RSVP) streams of images presented at 15 Hz. They included one-distractor and two-distractor conditions determined by the number of intervening distractors between repetitions. They observed a striking increase in detection of repetitions as the number of presentations increased (see Fig. 1b, dashed red and solid purple curves, original figure in inset).

Fig. 1.

Theory and reanalysis of Thunell and Thorpe data. (a) Three theories of how the brain responds to repeated presentations. The stimulus sequence is shown in black as three presentations of the same stimulus. The awareness transient reflects the conscious experience of the presented stimulus. Three accumulation regimes are shown. Case 1: Evidence accumulates across presentations, each of which yields, if only brief, a conscious percept. Case 2: Evidence accumulates without conscious percepts. Case 3: Evidence dissipates between presentations, none of which generate a conscious percept. (b) Plot from Thunell and Thorpe: original in inset, one-distractor (solid purple curve), two-distractor (dashed red curve), with error regions. Main plot: probability of first seeing as repetition analysis illustrated with added annotations: vertical lines (solid and dotted) with (double) arrowheads, and three solid horizontal lines. (c) Results of probability of first seeing as a repetition analysis of data in (b), with colors and labeling indicating corresponding points. Right side of (c): results of model comparison on probability of seeing as repetition for first time. Models are performing better if their corresponding circles are further to the left. Generalizable (out-of-sample) comparisons are between open circles, which are statistically reliable if confidence interval of second-best model does not intersect vertical line.

This finding seems to suggest a durability of memory representations for these fleeting items, which need to span presentations for a stimulus to be detected as repeating. They also apparently suggest that memory traces accumulate, with the strength of the representation of the repeating stimulus increasing across repetitions, making it progressively easier to detect (for alternative accumulation regimes, see Fig. 1a).

Thunell and Thorpe highlighted the importance of the brain’s detection of repetitions for its ability to learn from the statistical regularities in the environment. That is, if a stimulus repeats, then the brain learns to efficiently detect it. The repetition effect that Thunell and Thorpe observed could underlie this learning.

The fact that these repetition effects were observed when stimuli were presented rapidly, and indeed in some conditions exceptionally rapidly (e.g., at 120 Hz), seems to suggest that the processes that underlie detection of repetitions, and by extension also statistical learning, operate unconsciously. This is the question that we consider, with “access awareness” our focus, because reportability is taken as indicating conscious perception.

One would assume that there is at least some degree of durability of representations for consciously experienced stimuli presented in RSVP, as, for example, suggested by recall performance in RSVP whole-report experiments (Nieuwenstein & Potter, 2006). However, do Thunell and Thorpe’s findings definitely imply that there is also durability for presentations that do not reach awareness, which would correspond to Case 2 in Figure 1a? For example, the following, no-subliminal-accumulation scenario could explain Thunell and Thorpe’s repetition effect:

Repetition-independent (first) breakthrough: A prespecified stimulus can be effectively searched for in RSVP (Bowman et al., 2013; Potter, 1976). However, no such task set is imposed in a pure repetition task, where any stimulus could be the repeating item. Additionally, it could be that below-threshold registration of a stimulus that is neither prespecified nor familiar dissipates rapidly, preventing unconscious detection of repetitions (see Bowman et al., 2014, for evidence of this). Indeed, a stimulus may be propelled for the first time into awareness merely because it is poorly masked by other stimuli and not because it has been frequently repeated.

Subliminal search, with supraliminal task set: However, when a stimulus has broken into awareness, it then creates a conscious memory trace, providing a template to search for in the remainder of the RSVP stream.

If there were indeed no subliminal detection of repetition, one could still observe a repetition effect in RSVP. This is because the probability that a stimulus had previously broken through (for the first time) into awareness would increase with the number of repetitions, and the repeating item would continue to be seen with high probability in the ensuing search, with a supraliminal task set.

However, the no-subliminal-accumulation scenario makes a key prediction: The probability, on a particular presentation, of consciously perceiving a stimulus for the first time should not change with the number of presentations. That is, if we focus on the trials for which the repeating target was not observed on any of the previous $i - 1$ repetitions, the probability of consciously seeing the target on presentation i (relative to this set of not-previously-seen trials) should be the same, whatever the i.

We cannot directly observe this probability of first seeing; however, we can calculate the probability of first seeing as a repetition from Thunell and Thorpe’s (2019) data, which they have kindly made openly accessible.

Method

Thunell and Thorpe experiment

Thunell and Thorpe presented long sequences of images at RSVP rates (key condition: 15 Hz), which included subsequences containing repetitions of images. Two conditions were obtained dependent on the number of intervening distractors presented between repetitions, giving the one-distractor and two-distractor conditions. Each repetition subsequence contained a number of presentations of the repeating item, which varied from two to 10. Participants pressed a button when they saw a repetition. However, importantly, each repetition subsequence ran to completion whether a button was pressed or not. Thus, if a subsequence of N repetitions was presented, even if the repetition was seen earlier than the Nth repetition, that detection would count to the N presentations condition. This naturally leads to a cumulative interpretation of the experiment, whereby the N repetitions condition can be considered the same as the N – 1 condition up to the N – 1st presentation. This cumulative interpretation will be key to our reanalysis of Thunell and Thorpe’s data.

Figure 1b shows their main finding (original in inset), in which the probability of detecting the repetition increases with the number of presentations in the subsequence. This seems to suggest that brain representations accumulate from presentation to presentation, which, in turn, suggests that memory traces must last from one presentation to the next.

Key concepts

We first define key concepts underlying our analyses. We give informal and formal definitions alongside each other. Further details can be found in the work by Avilés et al. (2020), particularly the appendices:

Assume a sequence of N presentations of a repeating stimulus. We call this repeating stimulus the target, even though participants do not know its identity. Each presentation is labeled with a number, from 1 to N.

Define $S e e_R e p e a t i n g (j)$ to be true if the target is seen as repeating on the j th presentation.

Define the probability first seen as repetition on presentation i, denoted as p_i, as

the conditional probability that the target is seen on presentation i, given that it has not been seen as repeating on any previous presentation, which can be formalized as

\begin{array}{l} p_{i} = p (S e e_R e p e a t i n g (i) \\ | \forall j \in ℕ (1 \leq j < i) . \neg S e e_R e p e a t i n g (j)) . \end{array}

The key property that we are interested in is invariance to number of repetitions, which holds if

the probability that the target is first seen as a repetition is the same for all presentations, from the second onward, which can be formalized as

\forall i, j \in ℕ (2 \leq i, j \leq N) . p_{i} = p_{j} .

If the probability of first seeing as a repetition (p_i) is indeed invariant across presentations, then there is no evidence accumulation across repetitions before the target is consciously perceived (see Fig. 1a, Case 3). Alternatively, if this probability increases, then there is unconscious accumulation, and memory traces can last across repetitions (see Fig. 1a, Case 2).

This intuition was confirmed in stochastic simulations by Avilés et al. (2020). Specifically, if the probability of first seeing the target was constant across repetitions and, when seen, its probability of continuing to be seen was 1, then a flat probability of first seeing as a repetition curve was generated. In contrast, if the probability of first seeing the target increased across repetitions, as it would if evidence accumulated, then an increasing probability of first seeing as a repetition curve was generated (see Avilés et al., 2020, Fig. B1 in the appendix).

Reanalysis

The key step to assessing the probability of first seeing as a repetition in Thunell and Thorpe’s data is to determine for any number of presentations, i, the number of times that the repetition would have already been seen within i - 1 presentations. As already discussed, a natural way to do this is to consider the cumulative probability; indeed, Bowman and Avilés (2022) mathematically verified that this cumulative approach is consistent with the probability of first seeing as a repetition intuition. The cumulative approach proceeds as follows:

We are interested in the cumulative probability that the target has been consciously perceived at least once as a repetition by presentation i. We denote this as $c p (i)$ . For example, in Figure 1b (main panel), $c p (3)$ is data point b, which also corresponds to the sum of two vertical, solid, double-arrowed lines: green and blue.

The second concept is the proportion of trials for which the target has not been consciously perceived as a repetition by presentation i. This is denoted as $n p (i)$ and defined as

n p (i) = 1 - c p (i) .

For example, in Figure 1b, $n p (2)$ is the blue, vertical, dotted, double-arrowed line.

“Repetitions detected by i th presentation” minus “repetitions detected by $(i - 1)$ th presentation” gives the repetitions that one would expect to have been detected for the first time on the ith presentation. This corresponds to $c p (i) - c p (i - 1)$ . For example, in Figure 1b, $c p (3) - c p (2)$ is given by the blue, vertical, solid, double-arrowed line.

Using these concepts, we can now give a definition of the probability of first seeing as a repetition, denoted as p_i above, as follows:

p_{(i + 1)} = \frac{c p (i + 1) - c p (i)}{n p (i)} .

For example, in Figure 1b, p3 = (cp(3) – cp(2)) / np(2) corresponds to a fraction formed from vertical double-arrowed lines (i.e., dark blue solid divided by dark blue dotted).

Again, the key property that underlies our claims is invariance to number of repetitions, that is, $\forall i, j \in ℕ (2 \leq i, j \leq N) . p_{i} = p_{j}$ .

This then enables us to calculate what we are interested in from Thunell and Thorpe’s data, giving us the results shown on the left in Figure 1c. We present data points associated only with the rising arm of the curve from Thunell and Thorpe because the data saturate at a lower ceiling than 100%.

Model comparison

In Figure 1c, there are two plausible models: (a) subliminal accumulation (the alternative hypothesis; see Fig. 1a, Case 2), in which the curve increases with the number of presentations (i.e., slope is positive), and (b) subliminal dissipation (the null hypothesis; see Fig. 1a, Case 3), in which the curve is horizontal (i.e., slope is zero). We operationalized the alternative hypothesis as a full Bayesian regression model, which incorporates positive or zero slopes, and the null hypothesis as a Bayesian regression model with just a constant term. For the model comparisons, the posterior distributions were sampled using the No U-Turn Monte Carlo Markov Chain Sampler (Hoffman & Gelman, 2014), implemented in the Python package PyMC3 (Salvatier et al., 2016).

We used leave-one-out cross-validation (Vehtari et al., 2017) to determine the best-fitting model (see right side of Fig. 1c), and Bayes factors (BFs) were obtained using Savage-Dickey (Wetzels et al., 2009).

Results

For the model comparison, our results focus on the out-of-sample deviances, which are presented as open circles in Figure 1c (right side). For the one-distractor condition, the full model (intercept + number of presentations, i.e., zero or positive slopes) performed better than the null model (intercept, i.e., zero slope), although the difference in deviance (0.29, SE = 2.37) was inconclusive, as was the BF (BF₁₀ = 1.83). For the two-distractor condition, results were conclusive; the null model had the lowest deviance (best fit; difference in deviance = 1.96, SE = 1.13), and the null model’s deviance did not fall within the confidence interval of the full model’s deviance. Consistent with this, the BF for the two-distractor condition indicated moderate evidence for the null (BF₀₁ = 4.01).

With respect to this null finding, a possibility that we cannot completely exclude is that there is unconscious evidence accumulation, but its occurrence in the data is counteracted by another effect that reduces across presentations. A candidate for this reducing effect would be variability in the ease with which different images can be perceived in RSVP, with easy repeating images perceived with high probabilities of detection on earlier presentations and difficult repeating images perceived with lower probabilities of detection on later presentations. However, these two effects would have to perfectly counteract each other in order to generate a completely horizontal probability of first seeing as a repetition curve, which may be considered unlikely. Investigation of this alternative explanation awaits further research.

Additionally, Thunell and Thorpe observed above-chance recognition (in a final memory recognition test) for repeating images that were not detected as repeating. However, it could be that this residual memory is exclusively due to images that are repeated but consciously perceived only once. As a result, these images would not be seen as repeating but would be recognizable. Additionally, the increase in memory through repetitions could be due to recency effects and/or an increasing probability of perceiving a stimulus (although not seeing it as repeating) with the number of presentations.

Discussion

Thus, our findings suggest that, at least in the two-distractor condition, first breakthrough into awareness may be totally uninfluenced by the number of previous presentations. This suggests that the registration of a stimulus in RSVP dissipates rapidly if that registration does not itself generate a breakthrough into awareness. Given the suggested link between performance on the repetition task and statistical learning, such dissipation indicates that stimuli presented among competitors (as is the case in RSVP) that are not consciously perceived need to be repeated very rapidly for learning to occur. When there is a stimulus onset asynchrony of more than perhaps 220 ms between repetitions, representations do not accumulate unconsciously and thus do not seed learning. Additionally, a stimulus onset asynchrony of 220 ms is small; indeed, one might argue that such brief intervals between presentations are extremely rare in the real world.

The essence of the repetition task is that a stimulus becomes salient only when it is repeated; for example, it is not salient on its first presentation, but a trace of that first presentation needs to endure so that it can be observed as salient on future presentations. Thus, a precondition for the brain being able to perform the repetition task is that it exhibits what could be called incidental durability (i.e., builds resilient memories for stimuli that are currently not salient). The finding of a flat curve for the probability of first seeing as a repetition for the two-distractor condition suggests that incidental durability may be highly limited for subliminal representations. This in turn may suggest that the representation of episodic information, which is fundamentally incidental in nature, is available only consciously. This idea resonates with theories of conscious experience proposed by us, the tokenized percept hypothesis (Avilés et al., 2020), and Kanwisher (2001).

Footnotes

Acknowledgements

We thank the editor and three reviewers for very valuable insights that have greatly strengthened this commentary.

Transparency

Action Editor: Patricia J. Bauer

Editor: Patricia J. Bauer

Author Contributions

H. Bowman and A. Avilés developed the study concept. H. Bowman wrote the formal analysis. A. Avilés analysed the data supervised by H. Bowman. H. Bowman wrote the original draft, and H. Bowman and A. Avilés wrote the review and edited the text. Both authors approved the final version of the manuscript for submission.

References

Avilés

Bowman

Wyble

(2020). On the limits of evidence accumulation of the preconscious percept. Cognition, 195, Article 104080. https://doi.org/10.1016/j.cognition.2019.104080

Bowman

Avilés

(2022). Formal treatment of evidence accumulation. https://www.cs.kent.ac.uk/people/staff/hb5/formal%20treatment%20for%20evidence%20accum.pdf

Bowman

Filetti

Alsufyani

Janssen

(2014). Countering countermeasures: Detecting identity lies by detecting conscious breakthrough. PLOS ONE, 9(3), Article e90595. https://doi.org/10.1371/journal.pone.0090595

Bowman

Filetti

Janssen

Alsufyani

Wyble

(2013). Subliminal salience search illustrated: EEG identity and deception detection on the fringe of awareness. PLOS ONE, 8(1), Article e54258. https://doi.org/10.1371/journal.pone.0054258

Hoffman

M. D.

Gelman

(2014). The No-U-Turn Sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1), 1593–1623.

Kanwisher

(2001). Neural events and perceptual awareness. Cognition, 79(1–2), 89–113.

Nieuwenstein

M. R.

Potter

M. C.

(2006). Temporal limits of selection and memory encoding: A comparison of whole versus partial report in rapid serial visual presentation. Psychological Science, 17(6), 471–475.

Potter

M. C.

(1976). Short-term conceptual memory for pictures. Journal of Experimental Psychology: Human Learning and Memory, 2(5), 509–522.

Salvatier

Wiecki

T. V.

Fonnesbeck

(2016). Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2, Article e55. https://doi.org/10.7717/peerj-cs.55

10.

Thunell

Thorpe

S. J.

(2019). Memory for repeated images in rapid-serial-visual-presentation streams of thousands of images. Psychological Science, 30(7), 989–1000.

11.

Vehtari

Gelman

Gabry

(2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432.

12.

Wetzels

Raaijmakers

J. G.

Jakab

Wagenmakers

E. J.

(2009). How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin & Review, 16(4), 752–760.