How Visual Short-Term Memory Maintenance Modulates Subsequent Visual Aftereffects

Abstract

Prolonged viewing of a visual stimulus can result in sensory adaptation, giving rise to perceptual phenomena such as the tilt aftereffect (TAE). However, it is not known if short-term memory maintenance induces such effects. We examined how visual short-term memory (VSTM) maintenance modulates the strength of the TAE induced by subsequent visual adaptation. We reasoned that if VSTM maintenance induces aftereffects on subsequent encoding of visual information, then it should either enhance or reduce the TAE induced by a subsequent visual adapter, depending on the congruency of the memory cue and the adapter. Our results were consistent with this hypothesis and thus indicate that the effects of VSTM maintenance can outlast the maintenance period.

Keywords

adaptation tilt aftereffect short-term memory visual perception

Prolonged visual stimulation can change the visual system’s sensitivity to external input, giving rise to perceptual phenomena such as afterimages and aftereffects. For example, in the tilt aftereffect (TAE), the viewing of an oriented stimulus causes shifts in subsequently perceived orientations (Gibson & Radner, 1937; Hofmann & Bielschowsky, 1909; Kohn, 2007). Adaptation paradigms are important tools for psychologists because the resulting aftereffects can reveal the underlying properties of perceptual systems; for this reason, adaptation is often referred to as the psychologist’s microelectrode (Frisby, 1979).

Although aftereffects based on prolonged visual stimulation have been examined extensively, it is not known whether active short-term memory maintenance induces such effects. Adaptation results from prolonged neuronal responding to a sensory stimulus, and electrophysiological evidence indicates that visual short-term memory (VSTM) is associated with neuronal activity in the visual cortex during the maintenance period (e.g., Pasternak & Greenlee, 2005). Thus, it may be that VSTM maintenance can induce aftereffects. In the experiments reported here, we investigated this issue by examining how holding a stimulus in short-term memory modulates the impact of subsequent visual adaptation that takes place after memory maintenance has ended. We accomplished this by using a sequential adaptation paradigm in which the first half of each trial involved VSTM maintenance and the second half involved visual adaptation. We hypothesized that if VSTM maintenance induces effects similar to those induced by visual adaptation, then its effects should summate with those of a subsequently presented visual adapter, just as the effects of two sequential visual stimuli summate (Greenlee & Magnussen, 1988). Specifically, the TAE should be enhanced when a memory cue and subsequent adapter are of similar orientation, and reduced when the two differ (Greenlee & Magnussen, 1988). In order to dissociate any adaptation effect induced by VSTM maintenance from effects induced by purely visual processes or a general increase in cognitive load induced by memory engagement, we included a control condition in which participants passively viewed the memory cue and also controlled for the engagement of VSTM.

Method

Participants

Twenty-two observers (5 females, 17 males), who were naive to the aims of the study, participated in two experiments (11 in each experiment). All observers provided informed consent and were given a monetary reward for their participation.

Stimuli

The stimuli and task were controlled by E-Prime 2.0 (Psychology Software Tools, Inc., Pittsburgh, PA; http://www.pstnet.com/eprime.cfm). All stimuli except for control-task memory cues were sinusoidal luminance-modulated gratings (with a diameter of 6° of visual angle; generated with MATLAB, The MathWorks, Natick, MA), presented foveally on a gray background from a viewing distance of 57 cm. The spatial frequency of the gratings was 1.44 cycles/°. The memory cues, distractors, adapters, and memory probes had a Michelson contrast of 0.9; the contrast of the TAE probes was 0.5 (cf. Knapen, Rolfs, Wexler, & Cavanagh, 2010). In Experiment 2, the control-task memory cues were blue or red squares that were either small (diameter = 4.5° of visual angle) or large (diameter = 6.5° of visual angle). The stimuli were presented on a 15-in. screen with a resolution of 1024 × 768 pixels.

Procedure

We used a sequential adaptation paradigm in which the first half of each trial involved VSTM maintenance and the latter half involved visual adaptation (see Fig. 1 for timelines of trials in the two experiments). Each trial in the VSTM condition of Experiment 1 began with a black fixation cross appearing in the middle of the screen, followed by a blank screen and the memory cue (a grating tilted ±20° or ±40° from the vertical), which participants were instructed to hold in memory. This memory cue was followed by a maintenance period during which a distractor (a horizontal grating) was presented, 1 s after offset of the cue. The appearance of a fixation cross signaled the end of the maintenance period; participants were instructed to stop holding the cue in memory when the cross appeared.

Fig. 1.

Timelines of experimental trials in the two experiments. In the visual short-term memory (VSTM) condition of Experiment 1 (a), observers were instructed to maintain a memory cue in VSTM. Following the end of the memory-maintenance period, an adapter was presented. Finally, observers indicated the direction in which a probe stimulus was tilted. In the control condition of this experiment, observers were presented with the same sequence of stimuli but were not asked to hold the memory cue in memory. In Experiment 2 (b), a fixation cross at the start of each trial was followed by a blue or red square (control-task memory cue), and then the memory cue. In the VSTM condition, observers were asked to hold the memory cue in memory and to passively view the control-task memory cue; in the control condition, they were asked to hold the control-task memory cue in memory and to passively view the memory cue. TAE = tilt aftereffect.

The next display showed a visual adapter with an orientation of ±20° from vertical. Thus, the orientation difference between the memory cue and the visual adapter was 0°, +20°, −40°, or −60°. Positive value indicates that the sign of tilt was the same (i.e., both the memory cue and the adapter were tilted to the left or to the right); negative values indicate that the signs were opposite. For example, an orientation difference of +20° indicates that the orientations of the memory cue and adapter were +40° and +20°, respectively (i.e., both tilted to the right), or −40° and −20°, respectively (i.e., both tilted to the left). An orientation difference of −40° indicates that one of these stimuli had an orientation of +20° and the other had an orientation of −20°. An orientation difference of −60° indicates that the orientations of the memory cue and adapter were −40° and +20° or +40° and −20°, respectively. There was no combination for which the orientation difference was −20°, +40°, or +60°. After visual adaptation, a TAE probe (a grating tilted −1°, 0°, or +1° from the vertical; duration = 40 ms) was presented, and observers were asked to report the direction of the perceived tilt.

To ensure that subjects were engaged in memory maintenance, we randomly included memory catch trials. On these trials, the maintenance period was followed by a memory probe (grating tilted ±10° relative to the memory cue), and participants were asked to indicate the direction of the tilt relative to the memory cue.

To dissociate effects induced by VSTM maintenance from effects induced by passive viewing of the stimuli, in both experiments we included a control condition in which participants were presented with the same sequence of stimuli as in the VSTM condition but were not asked to remember the memory cue. In addition, we designed Experiment 2 to control for memory load by asking participants in the control condition to hold in memory stimulus features that were irrelevant for the TAE (Fig. 1b). Specifically, the trials in this experiment were the same as in Experiment 1 except that the memory cue was preceded by a fixation cross and then the control-task memory cue. (In addition, some of the display durations were changed; see Fig. 1.) In the VSTM condition, participants were instructed to passively view the control-task memory cue and to hold the memory cue in memory. In the control condition, their task was instead to hold the colored memory-task cue in memory and to passively view the tilted memory cue. On catch trials in the control condition, to ensure that participants held the control-task memory cue in memory, we asked them to indicate whether the stimulus was blue and small, blue and large, red and small, or red and large. Catch trials in the VSTM condition were as described for Experiment 1.

Experiment 1 was run in two sessions, each session containing four blocks (two VSTM and two control blocks). Each VSTM block contained 48 trials (32 TAE trials, with 8 trials per orientation difference, and 16 memory catch trials); each control block contained 32 trials. Experiment 2 was carried out in one session consisting of six blocks (three VSTM blocks and three control blocks, each of which consisted of 48 trials, as in the VSTM blocks in Experiment 1). The order of VSTM and control blocks was counterbalanced across participants. Each experiment was preceded by a practice block in which observers were acquainted with the VSTM task.

Results

Data analysis

The magnitude of the TAE in each memory condition was calculated by averaging responses across the three TAE probes (i.e., tilt of −1°, 0°, and +1° from the vertical). As observers were asked to press “1” for perceived leftward tilt and “2” for perceived rightward tilt, a mean value of 1 would indicate a maximal leftward bias, and a mean value of 2 a maximal rightward bias. To combine trials with leftward-tilting (i.e., –20°) and rightward-tilting (i.e., +20°) adapters, we calculated an overall TAE measure by subtracting the mean response for the rightward-adapter trials from the mean response for the leftward-adapter trials. Thus, values above 0 indicate the presence of a TAE (i.e., a tendency to report that the probe’s orientation was the opposite of the adapter’s orientation), and the maximum value possible was 1 (a TAE of 1 would result from a maximal rightward bias for the leftward-adapter trials and a maximal leftward bias for the rightward-adapter trials). Graphs displaying the mean of participants’ responses in each memory condition as a function of the TAE probe’s orientation are in the Supplemental Material available online.

Experiment 1

Figure 2a shows the magnitude of the TAE induced by the visual adaptation as a function of the orientation difference between the memory cue and the visual adapter, separately for the VSTM and control conditions. Two participants were removed from the analysis because of low performance on memory catch trials (accuracy of 55% and 63%, respectively). A high level of performance on these trials was important because it indicated that participants were engaging VSTM consistently throughout the task. The mean performance on memory catch trials was 84% (SD = 6%). We conducted a repeated measures analysis of variance (ANOVA) with factors of memory condition (VSTM, control), congruency of the memory cue and adapter (congruent direction of tilt, incongruent direction of tilt), and memory-cue orientation (±20°, ±40°). This analysis revealed a significant main effect of congruency, F(1, 8) = 10.26, p = .013, _p² = .562, and an interaction between memory condition and congruency, F(1, 8) = 18.97, p = .002; _p² = .703. No other main effects or interactions were significant. The interaction indicates that the TAE was modulated differently by the memory demand depending on whether the memory cue and adapter were congruent or incongruent. A pairwise comparison showed a nonsignificant trend for the TAE to be higher in the VSTM than in the control condition when the memory cue and the adapter were congruent, t(8) = 1.64, p = .14; when they were incongruent, there was a trend in the opposite direction, t(8) = 1.55, p = .17.

Fig. 2.

The mean magnitude of the tilt aftereffect (TAE) in the visual short-term memory (VSTM) and control conditions of (a) Experiment 1 and (b) Experiment 2 as a function of the orientation difference between the memory cue and the visual adapter. The error bars indicate ±1 SEM.

Experiment 2

Figure 2b shows the magnitude of the TAE induced by the visual adaptation as a function of the orientation difference between the memory cue and the visual adapter, separately for the VSTM and control conditions. We conducted a repeated measures ANOVA with factors of memory condition (VSTM, control), congruency of the memory cue and adapter (congruent, incongruent), and memory-cue orientation (±20°, ±40°). This analysis revealed a significant main effect of congruency, F(1, 10) = 12.62, p = .005, _p² = .558, and an interaction between memory condition and congruency, F(1, 10) = 5.41, p = .042, _p² = .351. Thus, memory demand modulated the TAE, and the direction of this modulation depended on congruency between the memory cue and the visual adapter. No other main effect or interaction was significant. A pairwise comparison showed that the TAE was higher in the VSTM than in the control condition when the memory cue and the adapter were congruent, t(10) = 2.22, p = .05; when they were incongruent, there was a trend in the opposite direction, t(10) = 1.69, p = .12. The mean performance on memory catch trials was 87% (SD = 6%) in the VSTM condition and 91% (SD = 5%) in the control condition.

Combined analysis of Experiments 1 and 2

To investigate the source of the interaction between memory condition and congruency in Experiments 1 and 2 (i.e., whether the results reflected facilitation in the congruent condition or inhibition in the incongruent condition, or both), we combined the two experiments in a single analysis in order to increase statistical power. We conducted a repeated measures ANOVA with within-subjects factors of memory condition (VSTM, control), congruency of the memory cue and adapter (congruent, incongruent), and memory-cue orientation (±20°, ±40°) to determine if there were any differences between the results of the two experiments. Experiment (Experiment 1, Experiment 2) was entered as a between-subjects factor. This analysis revealed a significant main effect of congruency, F(1, 18) = 22.37, p = .001, _p² = .554, and an interaction between memory condition and congruency, F(1, 18) = 10.57, p = .004, _p² = .370. No other main effects or interactions were significant. No differences were found between the experiments, F(1, 18) = 0.025, p = .875, _p² = .001. A pairwise comparison showed that the TAE was higher in the VSTM than in the control condition when the memory cue and the adapter were congruent, t(19) = 2.81, p = .011; when they were incongruent, the TAE was significantly lower in the VSTM than in the control condition, t(19) = 2.14, p = .045.

In summary, the combined analysis of Experiments 1 and 2 indicates that VSTM maintenance significantly increased the strength of the TAE induced by a subsequent visual adapter when the memory cue was congruent with the adapter. When the memory cue and the adapter were incongruent, the TAE was reduced by VSTM maintenance.

Discussion

Our results show that VSTM maintenance can modulate the TAE induced by subsequent visual adaptation. Specifically, an enhancement of the TAE was found when the orientations of the memory cue and the visual adapter were congruent; a reduction of the TAE was observed when they were incongruent. These results are similar to previous results for sequential adaptation using two visual adapters; adaptation was facilitated when the two adapters were of similar orientation and was reduced when they differed in orientation by 30° to 60° (Greenlee & Magnussen, 1988). These facilitatory and inhibitory effects are likely to reflect summation within and competition between orientation channels, respectively (e.g., Blakemore & Campbell, 1969; Campbell & Maffei, 1971; Tolhurst & Thompson, 1975). The similarity of our results with these previous results for sequential adaptation to two visual stimuli indicates that VSTM maintenance induces adaptation-like effects.

A key aspect of our results is that a modulation of the TAE was found when we compared the VSTM condition with a condition in which the memory cue was presented without the requirement to hold it in memory. Thus, the observed effects could not have been fully driven by passive memory traces induced by the memory cue. Furthermore, in Experiment 2, we controlled for memory load by asking participants to perform a visual memory task that did not involve the maintenance of orientation information (control condition) and thus was unlikely to affect the mechanisms responsible for the TAE. The interaction between memory condition and congruency of the memory cue and adapter in this experiment indicates that the observed effects in the VSTM condition were not simply due to a higher cognitive load distracting participants from the visual adapter.

Given that VSTM maintenance and visual adaptation were not concurrent, our results are unlikely to reflect attentional competition between working memory content and the visual adapter. Furthermore, as VSTM maintenance modulated the TAE differentially as a function of congruency, the observed effects cannot be explained as due to an increase in cognitive load impairing visual adaptation, a possibility also ruled out by the control condition of Experiment 2. It is also unlikely that a cost in switching attention from VSTM content to the visual adapter influenced our results, as the end of the maintenance period was signaled by the appearance of a fixation cross prior to the adapter’s onset, and participants therefore had sufficient time to shift their attention to the visual adapter. Finally, the fact that these effects survived the presentation of a distractor (which would have disrupted any passive memory trace), coupled with a high level of performance on catch trials, demonstrates that active VSTM maintenance was involved.

In summary, our results demonstrate that the effects of VSTM maintenance can outlast the maintenance period: VSTM maintenance modulated the magnitude of an aftereffect induced by a subsequently presented adapter. This indicates that the prolonged neuronal activity induced by VSTM maintenance can modulate the efficacy of subsequent visual processing, as is also the case with prolonged visually induced neuronal activity.

Footnotes

Declaration of Conflicting Interests

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

Funding

J. S. is supported by the Academy of Finland (137485) and the Emil Aaltonen Foundation.

Supplemental Material

Additional supporting information may be found at

References

Blakemore

Campbell

F. W.

(1969). On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images. Journal of Physiology, 203, 237–260.

Campbell

F. W.

Maffei

(1971). The tilt after-effect: A fresh look. Vision Research, 11, 833–840.

Frisby

(1979). Seeing: Illusion, brain and mind. Oxford, England: Oxford University Press.

Gibson

J. J.

Radner

(1937). Adaptation, after-effect and contrast in the perception of tilted lines. I. Quantitative studies. Journal of Experimental Psychology, 20, 453–467.

Greenlee

M. W.

Magnussen

(1988). Interactions among spatial frequency and orientation channels adapted concurrently. Vision Research, 28, 1303–1310.

Hofmann

F. B.

Bielschowsky

(1909). Ueber die Einstellung der scheinbaren Horizontalen und Vertikalen bei Betrachtung [On the adjustment of the perceived horizontal and vertical]. Pflügers Archiv, 126, 453–475.

Knapen

Rolfs

Wexler

Cavanagh

(2010). The reference frame of the tilt aftereffect. Journal of Vision, 10(1), Article 8. Retrieved from http://www.journalofvision.org/content/10/1/8.full.pdf+html

Kohn

(2007). Visual adaptation: Physiology, mechanisms, and functional benefits. Journal of Neurophysiology, 97, 3155–3164.

Pasternak

Greenlee

M. W.

(2005). Working memory in primate sensory systems. Nature Reviews Neuroscience, 6, 97–107.

10.

Tolhurst

Thompson

P. G.

(1975). Orientation illusions and after-effects: Inhibition between channels. Vision Research, 15, 967–972.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.09 MB