Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception

Abstract

Can what we imagine in our minds change how we perceive the world in the future? A continuous process of multisensory integration and recalibration is responsible for maintaining a correspondence between the senses (e.g., vision, touch, audition) and, ultimately, a stable and coherent perception of our environment. This process depends on the plasticity of our sensory systems. The so-called ventriloquism aftereffect—a shift in the perceived localization of sounds presented alone after repeated exposure to spatially mismatched auditory and visual stimuli—is a clear example of this type of plasticity in the audiovisual domain. In a series of six studies with 24 participants each, we investigated an imagery-induced ventriloquism aftereffect in which imagining a visual stimulus elicits the same frequency-specific auditory aftereffect as actually seeing one. These results demonstrate that mental imagery can recalibrate the senses and induce the same cross-modal sensory plasticity as real sensory stimuli.

Keywords

cross-modal plasticity auditory perception mental imagery multisensory integration open data

Humans have the fascinating ability to mentally simulate sensory experiences at will, that is, to elicit mental imagery. Mental imagery enables problem solving, adaptive behavior, and creativity; and the pictures, sounds, and tactile perceptions evoked in our mind’s eye are an integral part of our conscious experiences and of what it means to be human. Mental imagery has also been found to enhance ongoing perception. For instance, imaging a visual stimulus at a specific location facilitates the detection of an actual visual stimulus at the same location (Farah, 1985; Ishai & Sagi, 1994); and mental imagery in one sensory modality can influence perception of stimuli in other sensory modalities (Berger & Ehrsson, 2013, 2014, 2017; Lacey, Flueckiger, Stilla, Lava, & Sathian, 2010). Perceptually based theories of mental imagery (Kosslyn, 1994; Kosslyn, Ball, & Reiser, 1978; Kosslyn, Ganis, & Thompson, 2001) suggest that mental imagery utilizes some of the same cognitive processes (Farah, 1985; Schlegel et al., 2013) and brain mechanisms (Berger & Ehrsson, 2014; Ehrsson, Geyer, & Naito, 2003; Kosslyn et al., 2001) as actual perception. Thus, in many ways, mental imagery is similar to veridical perception (albeit less vivid); however, although it is clear that mental imagery can influence ongoing veridical perception, whether it can induce plasticity in our cortical sensory systems across sensory modalities to change future perception remains unknown.

One of the most basic forms of sensory plasticity is the cross-modal plasticity that underlies the continuous recalibration of the senses (Bavelier & Neville, 2002; Shimojo & Shams, 2001). Coherent perception of the world around us depends on the efficient integration of information from the different senses (e.g., vision, audition, and touch; Stein, Stanford, & Rowland, 2014; Wallace et al., 2004), and a prerequisite for successful multisensory integration is that the different senses are spatially and temporally aligned (i.e., well calibrated). This ensures that the sounds and visual impressions from a single audiovisual event are perceived as occurring at the same time and as originating from the same place; for example, hearing a bark and seeing a dog through a window are bound in space and time rather than perceived as separate perceptual events. However, active recalibration processes are needed to maintain spatiotemporal correspondence between the senses despite differences in the spatial and temporal precision of sensory signals, various forms of environmental noise, and differences in the growth and aging of our sensory organs and pathways.

One example of this type of active recalibration process can be observed in the ventriloquist illusion (Alais & Burr, 2004; Howard & Templeton, 1966), in which the perceived location of a sound is shifted toward the location of a simultaneously presented but spatially disparate visual stimulus. Importantly, repeated exposure to this illusion leads to the ventriloquism aftereffect, in which the mislocalization of auditory stimuli toward the no longer present visual stimulus persists even when the auditory stimuli are presented alone afterward (Frissen, Vroomen, & de Gelder, 2012; Kopco, Lin, Shinn-Cunningham, & Groh, 2009; Radeau & Bertelson, 1977). This spatial recalibration of unisensory auditory perception in the ventriloquism aftereffect is demonstrative of visually induced plasticity of the auditory system (Recanzone, 1998; Woods & Recanzone, 2004). This type of multisensory recalibration occurs between each of the senses (Bavelier & Neville, 2002; Harris, 1963; Stein & Stanford, 2008) and represents a basic form of sensory plasticity. Thus, in order to examine whether mental imagery can lead to cross-modal cortical plasticity, we examined whether imagined visual stimuli could lead to a visual-imagery-induced ventriloquism aftereffect in a series of experiments using a paradigm that brings together mental imagery and the ventriloquist illusion (Berger & Ehrsson, 2013, 2014).

In our first experiment, we made use of a two-alternative forced-choice task in which we presented the participants with white noise bursts from five different locations (0°, ±8°, ±16°) in random order from behind a screen that was acoustically transparent but visually opaque. We instructed them to determine whether the sounds came from the left or the right of the central fixation point. These perceptual judgments were preceded by 30-s adaptation periods in which the participants repeatedly imagined a white disk flashing directly in front of them at a frequency of 1 Hz while a white noise auditory stimulus was presented simultaneously but in one of three different spatial locations (0°, −8°, or +8°; see Fig. 1 ). We hypothesized that if visual imagery can induce a ventriloquism aftereffect, then we should observe a significant shift in the location where the participant can no longer distinguish between a sound that comes from the left and a sound that comes from the right—that is, the point of subjective equality (PSE) for sound localization—following leftward compared with rightward adaptation. Specifically, we predicted that repeated exposure to an auditory stimulus presented to the right of the imagined visual stimulus (i.e., leftward adaptation) would increase the probability of a leftward response for auditory stimuli presented alone from different locations along the azimuth, and therefore, a shift of the PSE for those auditory stimuli to the right. Conversely, we predicted that repeated exposure to an auditory stimulus presented to the left of the imagined visual stimulus (i.e., rightward adaptation) would increase the probability of a rightward response for auditory stimuli presented alone from different locations along the azimuth, and therefore, a shift of the PSE for those auditory stimuli to the left. An additional experiment (Experiment 2) was conducted using real visual stimuli for comparison with the results from imagined visual stimuli.

Fig. 1.

Schematic overview of the sequence of events within an adaptation block. Each adaptation block involved five repetitions of the sequence in the top part of the figure (150 exposure trials; 100 localization trials) with only one of the adaptation conditions (i.e., leftward, same location, or rightward) repeated during the adaptation phases of that block. The order of the adaptation blocks was counterbalanced across participants. The cue phase of the countdown reminded the participants of the frequency, location, and content of the stimulus they were to imagine seeing at the end of the countdown and for the duration of the adaptation phase. The bottom panel outlines the spatial relationship between imagined visual and real auditory stimuli during the adaptation phase of each block. Stimuli and angles are for explanatory purposes and are not drawn to scale.

We also designed additional experiments (Experiments 3–6) to test whether the imagery-induced ventriloquism aftereffect depends on the sound frequencies. Previous studies have demonstrated that the ventriloquism aftereffect does not transfer across disparate auditory frequencies (Recanzone, 1998; Woods & Recanzone, 2004). This is believed to be because early auditory areas are tonotopically organized; that is, different patches of cortex process sounds of different frequencies in a systematic spatial organization (Saenz & Langers, 2014; Shrem & Deouell, 2014). Consequently, the plastic changes in the spatial receptive fields of individual neurons in the early auditory cortex should be specific to the frequency band that was recalibrated during adaptation (Recanzone, 1998; Woods & Recanzone, 2004). Thus, we hypothesized that if the imagery-induced ventriloquism aftereffect relies on the same cortical plasticity in the early auditory cortices as the classic ventriloquism aftereffect, then the imagery-induced aftereffect should be elicited only when the adaptation stimulus and test stimulus contain similar frequencies and should be eliminated when they contain different frequencies.

Method

Participants

One hundred forty-four participants in total participated in Experiments 1 through 6. Sets of 24 participants participated in Experiment 1 (mean age = 24.16 years, SD = 3.59; 16 females), Experiment 2 (mean age = 24.29 years, SD = 5.5; 12 females), Experiment 3 (mean age = 27 years, SD = 6.7; 15 females), Experiment 4 (mean age = 24 years, SD = 3.87; 20 females), Experiment 5 (mean age = 30.12 years, SD = 7.74; 13 females), and Experiment 6 (mean age = 25.89 years, SD = 4.75; 17 females). All participants were recruited from the student population in the Stockholm area, were healthy, reported no history of psychiatric illness or neurologic disorder, and reported no impairments of hearing or vision (or had corrected-to-normal vision). A sample size of 22 participants was predetermined to yield a power of 80% and to detect an effect of comparable size to previous studies using similar independent and dependent measures (Berger & Ehrsson, 2013, 2014). Therefore, data collection was stopped once the appropriate number of participants was reached and the experiment was fully counterbalanced. All participants provided written informed consent before the start of the experiment, and the Regional Ethical Review board of Stockholm approved the experiments.

Experiments 1 and 2 materials and procedures

The auditory stimuli consisted of a white noise burst lasting 50 ms, with rise and fall times of 5 ms. Auditory stimuli were presented from five self-amplified monitors arranged in a semicircular array (0°, ±8°, and ±16°) on a shelf behind a black, acoustically transparent (but visually opaque) fabric. All visual stimuli were presented directly in front of the participant 0° from an EP1080 digital-light-processing overhead projector (Optoma, Fremont, CA; 3,600 lumens). The presentation of the auditory and visual stimuli was controlled using PsychoPy (Peirce, 2007) software on a 24-in. (60.96 cm) iMac computer. Eye position was monitored during the experiments with an eye-tracking system (see Eye Gaze Tracking and Analysis and Figs. S1–S3 in the Supplemental Material available online).

After the general experimental procedures were described to the participants, they sat with their head in a chin rest at a height of 115 cm, 113 cm from the acoustically transparent black screen that the visual stimuli were projected onto and 122 cm from the auditory monitors. The participants were informed that they would hear sounds come from different locations behind the screen in front of them, and they would have to determine whether they heard the sound come from the left or the right of the fixation cross in the center of the screen. To familiarize the participants with the task and to be sure they were able to perform the auditory localization task, we had participants perform a practice session of 50 localization trials (10 per location). Following the practice session, the participants completed the experiment. The adaptation phases were split into three blocks. The order of the blocks was counterbalanced across participants. Each adaptation block consisted of a countdown from 5, just above the fixation point. A white disk flashed (100 ms) on the screen on the fixation point at the same time as 3 through 1 appeared in the countdown. In Experiment 1, the participants were instructed that they should imagine seeing this white disk as vividly as possible at the same frequency and location as the disk presented at 3 through 1 in the countdown. Thus, the countdown was followed by a 30-s exposure phase with auditory stimuli presented at the same time as the imagined visual stimulus. In Experiment 2, the visual stimulus was actually presented, rather than imagined during the exposure periods, and participants were instructed to maintain their fixation (i.e., to look where the white disk was flashing). Auditory stimuli were presented either 8° to the left, 8° to the right, or in the same location as the real or imagined visual stimulus, in the right adaptation, left adaptation, or same-location-adaptation blocks, respectively. A red fixation-cross appeared at the end of 30 s. In Experiment 1, the red fixation-cross indicated that the participants should stop imagining the visual stimulus, and in both Experiments 1 and 2, it cued the participants to prepare to identify the location of the sounds. Following the red fixation-cross, a white fixation-cross appeared, and there were 10 test trials (2 per auditory location), in which participants indicated whether they heard the sound come from the left or the right of the fixation-cross. The adaptation and test phases were repeated five times per block. Importantly, the fact that the predicted effects here are in the opposite direction of the adapting auditory stimuli (i.e., a sound to the left of the centrally imagined visual stimulus led to a rightward bias in auditory localization) obviates the concern that these results reflect implicit or explicit response strategies or could be explained by differences in executive attention rather than a genuine cross-modally induced remapping of acoustic space perception. Additionally, eye-tracking data were collected to confirm that the participants maintained fixation at the center of the screen during the adaptation phases of the experiments (see Figs. S1–S3).

Experiments 3 and 4 materials and procedures

A 4 kHz sine-wave auditory stimulus lasting 50 ms was used during the exposure phases of Experiments 3 and 4. The auditory stimulus used during the test phases, however, was the same white noise auditory stimulus used in the exposure and test phases of Experiments 1 and 2. All visual stimuli were the same as in Experiments 1 and 2. The procedures of Experiments 3 and 4 were identical to those of Experiments 1 and 2, respectively.

Experiments 5 and 6 materials and procedures

A 4 kHz sine-wave auditory stimulus lasting 50 ms was used during the exposure phases and the test phases of Experiments 5 and 6. All visual stimuli were the same as in the experiments above. The procedures of Experiments 5 and 6 were identical to those of Experiments 1 and 2, respectively.

Statistics

For all experiments, a logistic regression was fitted to the localization data of each participant for each adaptation condition. From these fits, the PSE was calculated as the spatial location for which the probability of making a rightward response was 50% for each participant. Repeated measures analyses of variance (ANOVAs) were conducted for each experiment to assess whether there was a significant effect of adaptation condition (i.e., leftward, rightward, and same-location adaptation) on the participants’ PSEs. Planned comparisons between PSEs from the leftward- and rightward-adaptation conditions were conducted to assess the presence or absence of a ventriloquism aftereffect. Complementary planned comparisons between the leftward- and same-location-adaptation conditions, and between the rightward- and same-location-adaptation conditions, were then conducted to assess the directionality of the shift in the PSEs from the perceived center. All pairwise comparisons between adaptation conditions were Bonferroni-corrected for multiple comparisons.

Results

Experiment 1

A repeated measures ANOVA revealed a significant shift in the participants’ PSEs across the three adaptation conditions, F(2, 46) = 15.29, p < .001, η_G² = .17. A planned comparison between the participants’ PSEs following leftward adaptation and rightward adaptation was subsequently conducted to examine whether mental imagery of a visual stimulus led to an imagery-induced ventriloquism aftereffect; that is, whether repeatedly imagining a visual stimulus at the center of the screen while auditory stimuli were presented at the same time but 8° to the left (i.e., rightward adaptation) led to a leftward shift in the participants’ PSEs compared with repeatedly imagining a visual stimulus at the center of the screen while auditory stimuli were presented at the same time but 8° to the right (i.e., leftward adaptation). This analysis revealed a significant difference between the participants’ PSEs following leftward and rightward adaptation, t(23) = 5.15, p < .001, d = 1.05, 95% CI = [0.194, 0.452] (see Fig. 2a ). Further analyses confirmed that this imagery-induced aftereffect was the result of a significant shift of the participants’ PSEs to the left following rightward adaptation, t(23) = −2.75, p = .03, d = 0.56, 95% CI = [−0.224, −0.032], and to the right following leftward adaptation, t(23) = 2.98, p = .018, d = 0.61, 95% CI = [0.060, 0.330], compared with when the sound was presented in the same location as the imagined visual stimulus (0°; see the Supplemental Material for additional analyses—all raw data and scripts used for statistical analyses are publicly available at https://osf.io/6htsx/).

Fig. 2.

Visual-imagery and real-visual-stimulus-induced ventriloquism aftereffects. The plots show logistic regression curves fitted to the group data from the visual-imagery-adaptation conditions in Experiment 1 (a) and real-visual-stimulus-adaptation conditions in Experiment 2 (b). The dotted lines denote the points of subjective equality (PSEs) for each condition. The bar graphs display the mean PSEs as a function of each adaptation condition for each experiment. Error bars represent ±1 SE.

Experiment 2

Consistent with Experiment 1, a repeated measures ANOVA revealed a significant shift in the participants’ PSEs across the three adaptation conditions, F(2, 46) = 26.76, p < .001, η_G² = .25. Moreover, the results from this experiment confirmed that real visual stimuli presented during the ventriloquism adaptation periods also led to a significant ventriloquism aftereffect, t(23) = 6.66, p < .001, d = 0.37, 95% CI = [0.262, 0.498] (see Fig. 2b ; see the Supplemental Material for additional analyses). We then directly compared the strength of the ventriloquism aftereffect from real visual stimuli in Experiment 2 and the imagery-induced ventriloquism aftereffect from imagined visual stimuli in Experiment 1. The strength of the ventriloquism aftereffect was calculated for each participant as the average of the distance between the leftward- and same-location-adaptation PSEs and the positively scored distance between the rightward and same-location-adaptation PSEs, that is,

\begin{array}{l} aftereffect strength = (leftward-adaptation \\ PSE - same-location adaptation PSE) + \\ ((rightward-adaptation PSE - same-location \\ adaptation PSE) \times - 1) / 2) . \end{array}

Remarkably, there was no significant difference between the strength of the aftereffect for real visual stimuli and the strength of the aftereffect for imagined visual stimuli, t(46) = 0.67, p = .504, d = 0.19, 95% CI = [–0.056, 0.113]. Taken together, Experiments 1 and 2 revealed that visual imagery led to a ventriloquism aftereffect, and one that was comparable in strength to the aftereffect produced by real visual stimuli.

Experiment 3

Experiment 3 examined whether the imagery-induced ventriloquism aftereffect on the white noise auditory stimuli used in the experiments above would be eliminated when the adapted auditory stimulus was a 4 kHz tone instead of a white noise stimulus. In line with this prediction, no significant shifts in the participants’ PSEs between imagery adaptation conditions was observed when the auditory stimuli in the adaptation (i.e., 4 kHz tones) and test (i.e., white noise bursts) phases were different, F(2, 46) = .15, p = .86, η_G² = .002. Moreover, we found that the imagined visual stimulus had no measurable ventriloquism aftereffect, t(23) = .21, p = 1.00, d = 0.04, 95% CI = [−0.115, 0.142], on white noise auditory stimuli when the adapted auditory stimulus was a 4 kHz tone (see Fig. 3a ). We also directly compared the strength of the imagery-induced aftereffect from this experiment (Experiment 3), when the adapted auditory stimulus and the test stimulus were different, with the corresponding effect in Experiment 1, when these stimuli were identical (white noise stimuli). In support of our hypothesis, we found that the strength of the aftereffect was significantly stronger when the adapted auditory stimulus and the test stimulus were the same (Experiment 1) than when they were different (Experiment 3), t(46) = 3.5, p = .001, d = 1.01, 95% CI = [0.161, 0.244].

Fig. 3.

Group localization data from a white noise stimulus following adaptation to a 4 kHz tone. The plots show logistic regression curves fitted to the group white-noise localization data from the visual-imagery-adaptation conditions in Experiment 3 (a) and real-visual-stimulus-adaptation conditions in Experiment 4 (b). The dotted lines denote the points of subjective equality (PSEs) for each condition. The bar graphs display the mean PSEs as a function of each adaptation condition for each experiment. Error bars represent ±1 SE.

Experiment 4

Experiment 4 was conducted using real visual stimuli for comparison with Experiment 3. Although a repeated measures ANOVA revealed a significant difference in the participants’ PSEs following the different adaptation conditions in this experiment, F(2, 46) = 4.09, p = .023, η_G² = .028, a pairwise comparison between leftward and rightward-adaptation conditions revealed that there was no significant ventriloquism aftereffect on white noise following adaptation to a real visual stimulus and a 4 kHz tone, t(23) = 1.55, p = .402, d = 0.32, 95% CI = [−0.024, 0.172] (see Fig. 3b and the Supplemental Material for additional analyses).

Experiment 5

Experiment 5 was conducted to test the prediction that an imagery-induced ventriloquism aftereffect should be present when the test and adapted auditory stimuli were both 4 kHz tones. Confirming this prediction, together with the results of the preceding two experiments, would conclusively demonstrate that the aftereffect depends on the congruency of the sound frequencies of the two stimuli. The results from this experiment showed a significant shift in the participants’ PSEs for the different adaptation conditions when the test auditory stimuli and the adapted auditory stimuli were both 4 kHz tones, F(2, 46) = 9.8, p < .001, η_G² = .097. Moreover, as predicted, a significant imagery-induced ventriloquism aftereffect was present when the test auditory and the adapted auditory stimuli were both 4 kHz tones, t(23) = 3.91, p < .001, d = 0.80, 95% CI = [0.198, 0.642] (see Fig. 4a and the Supplemental Material for additional analyses).

Fig. 4.

Ventriloquism aftereffects for 4 kHz tones. The plots show logistic regression curves fitted to the group 4 kHz tone-localization data following adaptation to a 4kHz tone from the visual-imagery-adaptation conditions in Experiment 5 (a) and real-visual-stimulus-adaptation conditions in Experiment 6 (b). The dotted lines denote the points of subjective equality (PSEs) for each condition. The bar graphs display the mean PSEs as a function of each adaptation condition for each experiment. Error bars represent ±1 SE.

Experiment 6

In this experiment, a repeated measures ANOVA revealed that there was a significant shift in the participants’ PSEs for a 4 kHz tone following the adaptation with a 4 kHz tone and real visual stimuli for the different adaptation conditions, F(2, 46) = 13.13, p < .001, η_G² = .152. Consistent with Experiment 5, a planned comparison revealed that there was a significant ventriloquism aftereffect, t(23) = 4.37, p < .001, d = 0.89, 95% CI = [0.252, 0.706], in Experiment 6 (see Fig. 4b). In summary, the results from Experiments 1, 3, and 5 demonstrate that the imagery-induced aftereffect occurs only when the adapted auditory and test auditory stimuli are of a similar frequency. These results provide compelling behavioral evidence that the effect is driven by plasticity of tonotopically organized auditory representations, likely in the early auditory cortex (for additional analyses supporting this conclusion, see the Supplemental Material).

Discussion

The results show that mental imagery can induce central plasticity and recalibrate spatial representations of the senses. Therefore, imagined sensations not only have the capacity to influence ongoing perception in the same (Craver-Lemley & Reeves, 1987; Farah, 1989; Kosslyn et al., 2001) or different sensory modalities (Berger & Ehrsson, 2013, 2014, 2017; Lacey et al., 2010) but can also effectively reshape future perception of real stimuli through neural plasticity mechanisms across sensory modalities. The finding that a ventriloquism aftereffect can be triggered by visual mental imagery shows that mental imagery is capable of inducing a very basic form of cross-modal cortical plasticity. The fact that this effect depended on the specific frequency of auditory stimuli that were paired with the imagined visual stimuli during the ventriloquism phase suggests that both the classic ventriloquism aftereffect and the imagery-induced ventriloquism aftereffect rely on the plasticity of the tonotopically organized auditory cortex.

This plasticity-inducing potential of mental imagery is consistent with the well-known efficiency of mental imagery training strategies in sports psychology and mobility rehabilitation after a stroke (de Vries & Mulder, 2007; Driskell, Copper, & Moran, 1994) and indicates that these positive behavioral effects from mental imagery training strategies may at least partially come from plasticity in central sensory representations (Striem-Amit, Cohen, Dehaene, & Amedi, 2012). The imagery-induced plasticity demonstrated here is also consistent with neuroimaging and neurostimulation studies that have shown changes in cortical activation or cortical excitability as a result of mental imagery (Cicinelli et al., 2006; Page, Szaflarski, Eliassen, Pan, & Cramer, 2009).

From a broader perspective, our findings suggest that the basic plasticity that maintains the alignment of the senses for multisensory integration and perception is influenced by the contents of our mind. The present results also demonstrate that the contents of our mental images can integrate with signals from real incoming sensory signals to change our perception of external events. The ventriloquist illusion depends on the integration of visual and auditory signals and the formation of a coherent multisensory representation of a single external audiovisual event. Importantly, the ventriloquism aftereffect occurs only after a period of repeatedly experiencing the ventriloquist illusion, which is based on audiovisual integration. Hence, the observed visual-imagery-induced aftereffects also provide conclusive evidence that the imagery-induced ventriloquist illusion involves the same multisensory integration mechanisms as the classic illusion involving real visual stimuli.

Three technical issues regarding the present work should briefly be discussed. First, white noise was chosen for the main auditory stimuli for our experiments due to its high perceptual reliability for auditory source localization (Stevens & Newman, 1936). For this reason, unlike previous studies (Lewald, 2002; Recanzone, 1998), the present experiments have demonstrated a lack of transference of the ventriloquism aftereffect (from both real and imagined visual stimuli) between one frequency and white noise rather than between two disparate frequencies. However, given that we found transference of ventriloquism adaptation between a tone of a single frequency (i.e., a 4 kHz tone) to a tone of the same frequency (Experiments 5–6) but a lack of transference between that same tone and auditory stimuli comprised of the full frequency spectrum (i.e., white noise; Experiments 3–4), we expected the same lack of transference to be observed between two tones of disparate frequencies. Second, although the results of the imagery experiments (Experiments 1, 3, and 5) and their consistency with the real visual stimulus versions of the experiments (Experiments 2, 4, and 6) presented here provide strong indirect evidence that the participants imagined the visual stimuli when instructed, this cannot not be directly verified. Future work may utilize neuroimaging techniques in order to independently verify that participants follow instructions and imagine the visual stimuli as instructed. Third, regarding the generalizability of our results, we expected our results to generalize to any population of healthy participants with normal (or corrected-to-normal) vision and hearing, given the consistent pattern of results across several experiments for both real and imagined visual stimuli across a large participant pool. In light of previous work we have conducted on the integration of real and imagined sensory stimuli for different combinations of sensory stimuli (see Berger & Ehrsson, 2013, 2014, 2017), we also anticipated that these cross-modal adaptation effects from imagined stimuli would generalize to other perceptual paradigms in which mental imagery in one modality is known to change ongoing perception in another modality and cross-modal adaptation is expected to follow.

Collectively, the present results provide strong support for perception-based theories of mental imagery (Kosslyn, 1994; Kosslyn et al., 2001) by demonstrating that mental imagery can engage the same multisensory integration, multisensory recalibration, and cortical plasticity mechanisms as veridical perception. In sum, these findings have important implications for our understanding of cortical plasticity, mental imagery, and human perception.

Supplemental Material

Supplemental Material, BergerOpenPracticesDisclosure – Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception

Supplemental Material, BergerOpenPracticesDisclosure for Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception by Christopher C. Berger and H. Henrik Ehrsson in Psychological Science

Supplemental Material

Supplemental Material, BergerSupplementalMaterial – Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception

Supplemental Material, BergerSupplementalMaterial for Mental Imagery Induces Cross-Modal Sensory Plasticity and Changes Future Auditory Perception by Christopher C. Berger and H. Henrik Ehrsson in Psychological Science

Footnotes

Action Editor

Ralph Adolphs served as action editor for this article.

Author Contributions

C. C. Berger and H. H. Ehrsson designed the study. C. C. Berger conducted the experiments and analyzed the data. C. C. Berger drafted the manuscript and H. H. Ehrsson provided critical revisions. Both authors approved the final version of the manuscript for submission.

Declaration of Conflicting Interests

The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article.

Funding

This research was supported by Riksbankens Jubileumsfond, the James S. McDonnell Foundation, the Swedish Research Council, and Torsten Söderbergs Stiftelse. C. C. Berger was supported by a grant from the Swedish Research Council (2017-00276).

ORCID iD

Christopher C. Berger

Supplemental Material

Additional supporting information can be found at

Open Practices

All data have been made publicly available via the Open Science Framework and can be accessed at https://osf.io/6htsx. The materials have not been made publicly available, and the analyses were not preregistered. The complete Open Practices Disclosure for this article can be found at https://journals-sagepub-com.web.bisu.edu.cn/doi/suppl/10.1177/0956797617748959. This article has received the badge for Open Data. More information about the Open Practices badges can be found at .

References

Alais

Burr

(2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257–262.

Bavelier

Neville

H. J.

(2002). Cross-modal plasticity: Where and how? Nature Reviews. Neuroscience, 3, 443–452.

Berger

C. C.

Ehrsson

H. H.

(2013). Mental imagery changes multisensory perception. Current Biology, 23, 1367–1372.

Berger

C. C.

Ehrsson

H. H.

(2014). The fusion of mental imagery and sensation in the temporal association cortex. Journal of Neuroscience, 34, 13684–13692.

Berger

C. C.

Ehrsson

H. H.

(2017). The content of imagined sounds changes visual motion perception in the cross-bounce illusion. Scientific Reports, 7, Article 40123. doi:10.1038/srep40123.

Cicinelli

Marconi

Zaccagnini

Pasqualetti

Filippi

M. M.

Rossini

P. M.

(2006). Imagery-induced cortical excitability changes in stroke: A transcranial magnetic stimulation study. Cerebral Cortex, 16, 247–253.

Craver-Lemley

Reeves

(1987). Visual imagery selectively reduces vernier acuity. Perception, 16, 599–614.

de Vries

Mulder

. (2007). Motor imagery and stroke rehabilitation: A critical discussion. Journal of Rehabilitation Medicine, 39, 5–13.

Driskell

J. E.

Copper

Moran

(1994). Does mental practice enhance performance? Journal of Applied Psychology, 79, 481–492.

10.

Ehrsson

H. H.

Geyer

Naito

(2003). Imagery of voluntary movement of fingers, toes, and tongue activates corresponding body-part-specific motor representations. Journal of Neurophysiology, 90, 3304–3316.

11.

Farah

M. J.

(1985). Psychophysical evidence for a shared representational medium for mental images and percepts. Journal of Experimental Psychology: General, 114, 91–103.

12.

Farah

M. J.

(1989). Mechanisms of imagery-perception interaction. Journal of Experimental Psychology: Human Perception and Performance, 15, 203–211.

13.

Frissen

Vroomen

de Gelder

(2012). The aftereffects of ventriloquism: The time course of the visual recalibration of auditory localization. Seeing and Perceiving, 25, 1–14.

14.

Harris

C. S.

(1963). Adaptation to displaced vision: Visual, motor, or proprioceptive change? Science, 140, 812–813.

15.

Howard

I. P.

Templeton

W. B.

(1966). Human spatial orientation. London, England: Wiley.

16.

Ishai

Sagi

(1994). Visual imagery facilitates visual perception: Psychophysical evidence. Journal of Cognitive Neuroscience, 9, 476–489.

17.

Kopco

Lin

I.-F.

Shinn-Cunningham

B. G.

Groh

J. M.

(2009). Reference frame of the ventriloquism aftereffect. Journal of Neuroscience, 29, 13809–13814.

18.

Kosslyn

S. M.

(1994). Image and brain: The resolution of the imagery debate. Cambridge, MA: MIT Press.

19.

Kosslyn

S. M.

Ball

T. M.

Reiser

B. J.

(1978). Visual images preserve metric spatial information: Evidence from studies of image scanning. Journal of Experimental Psychology: Human Perception and Performance, 4, 47–60.

20.

Kosslyn

S. M.

Ganis

Thompson

W. L.

(2001). Neural foundations of imagery. Nature Reviews Neuroscience, 2, 635–642.

21.

Lacey

Flueckiger

Stilla

Lava

Sathian

(2010). Object familiarity modulates the relationship between visual object imagery and haptic shape perception. NeuroImage, 49, 1977–1990.

22.

Lewald

(2002). Rapid adaptation to auditory-visual spatial disparity. Learning & Memory, 9, 268–278.

23.

Page

S. J.

Szaflarski

J. P.

Eliassen

J. C.

Pan

Cramer

S. C.

(2009). Cortical plasticity following motor skill learning during mental practice in stroke. Neurorehabilitation and Neural Repair, 23, 382–388.

24.

Peirce

J. W.

(2007). PsychoPy–Psychophysics software in Python. Journal of Neuroscience Methods, 162, 8–13.

25.

Radeau

Bertelson

(1977). Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception & Psychophysics, 22, 137–146.

26.

Recanzone

G. H.

(1998). Rapidly induced auditory plasticity: The ventriloquism aftereffect. Proceedings of the National Academy of Sciences, USA, 95, 869–875.

27.

Saenz

Langers

D. R. M.

(2014). Tonotopic mapping of human auditory cortex. Hearing Research, 307, 42–52.

28.

Schlegel

Kohler

P. J.

Fogelson

S. V.

Alexander

Konuthula

Tse

P. U.

(2013). Network structure and dynamics of the mental workspace. Proceedings of the National Academy of Sciences, USA, 2013, 1–6.

29.

Shimojo

Shams

(2001). Sensory modalities are not separate modalities: Plasticity and interactions. Current Opinion in Neurobiology, 11, 505–509.

30.

Shrem

Deouell

L. Y.

(2014). Frequency-dependent auditory space representation in the human planum temporale. Frontiers in Human Neuroscience, 8, Article 524. doi:10.3389/fnhum.2014.00524

31.

Stein

B. E.

Stanford

T. R.

(2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255–266.

32.

Stein

B. E.

Stanford

T. R.

Rowland

B. A.

(2014). Development of multisensory integration from the perspective of the individual neuron. Nature Reviews Neuroscience, 15, 520–535.

33.

Stevens

S. S.

Newman

E. B.

(1936). The localization of actual sources of sound. The American Journal of Psychology, 48, 297–306.

34.

Striem-Amit

Cohen

Dehaene

Amedi

(2012). Reading with sounds: Sensory substitution selectively activates the visual word form area in the blind. Neuron, 76, 640–652.

35.

Wallace

M. T.

Roberson

G. E.

Hairston

W. D.

Stein

B. E.

Vaughan

J. W.

Schirillo

J. A.

(2004). Unifying multisensory signals across time and space. Experimental Brain Research, 158, 252–258.

36.

Woods

T. M.

Recanzone

G. H.

(2004). Visually induced plasticity of auditory spatial perception in macaques. Current Biology, 14, 1559–1564.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.09 MB

0.54 MB