Modal-based attention modulates the redundant-signals effect: Role of unimodal target probability

Abstract

Multisensory integration includes two behavioral manifestations: the modality dominance effect and the redundant-signals effect (RSE). RSE is a multisensory improvement effect in which individuals respond more quickly and accurately to bimodal audiovisual (AV) targets than to unimodal auditory (A) or visual (V) targets. Previous studies have confirmed that RSE is the product of modality interactions between different modalities. The goal of this study was to systematically investigate the effects of the modality dominance manipulated by modal-based attention and unimodal target probability on RSE. The results showed that when paying attention to both the A and V modalities (Exp. 1), RSE was not significantly different between unimodal target probabilities. When selectively paying attention to the A modality (Exp. 2A), RSE was also not significantly different between unimodal target probabilities. However, when selectively paying attention to the V modality (Exp. 2B), the magnitude of RSE showed a significant decreasing trend with the increasing probability of V targets. Our study is the first to reveal that the unimodal target probability significantly modulates RSE in visual selective attention, and this modulatory effect of the unimodal target probability on RSE is opposite to the modulatory effect on the modality dominance effect.

Keywords

redundant-signals effect (RSE)modal-based attention unimodal target probability modality dominance effect

We live in a world surrounded by rich information derived from many sensory modalities (i.e., vision, audition, and touch). Different types of modality information usually interact and converge into a robust, coherent, meaningful representation known as multisensory integration (MSI) (Ernst & Bülthoff, 2004; Lewkowicz & Ghazanfar, 2009; Talsma et al., 2010). MSI includes different behavioral manifestations. One manifestation is the redundant-signals effect (RSE) generated by the coactivation facilitation of different modalities under modality interaction, in which individuals respond faster and more accurately to simultaneous bimodal audiovisual (AV) targets than to unimodal auditory (A) or visual (V) targets (RT_unimodal – RT_bimodal) (Hershenson, 1962; Kinchla, 1974). Another behavioral manifestation is the modality dominance effect, indicating that one sensory modality becomes dominant in the modality interaction and then is preferentially processed during the multisensory perception (Fang et al., 2020; Murray et al., 2018), that is, the Colavita effect (Colavita, 1974; Colavita et al., 2013) or sound-induced flash illusions (Shams et al., 2000, 2002).

According to the literature, modal-based attention can modulate the multisensory RSE (Harrar et al., 2014; Yang et al., 2016). Modal-based divided attention and modal-based selective attention may influence RSE through different mechanisms (Tang et al., 2016). In the modal-based divided attention task, simultaneous attention to the A and V modalities will induce coactivation (Miller, 1982, 1991). The activations induced by both stimuli add up, and the summed activation enables faster response initiation of bimodal stimuli, which greatly promotes RSE performance (Blurton et al., 2014; Gondan & Minakata, 2016; Miller, 2004). Previous studies have demonstrated that RSE is robust and more pronounced in modal-based divided attention tasks (Mishra & Gazzaley, 2012; Talsma et al., 2007; Yang et al., 2016). The difference is that a prior entry study showed that paying attention to a specific modality can speed up information processing (Vibell et al., 2007). Modal-based selective attention may lead to modality dominance when exclusively focusing on one modality and ignoring the other. The information enhances in the attended modality takes precedence and subsequently suppresses information processing in the unattended modality, making RSE difficult (Guerreiro et al., 2014; Johnson & Zatorre, 2006; Lakatos et al., 2009; Spence et al., 2001). Thus, in some modal-based selective attention tasks, the RSE disappears completely (Mozolic et al., 2008; Talsma et al., 2007; Wu et al., 2012). However, preferential attention to one modality can occasionally spread across both space and modality to another synchronously presented but task-irrelevant modality, constituting a multisensory object bestowed with sensory enhanced processing (Busse et al., 2005; Zimmer et al., 2010). Thus, RSE can be observed even if only one modality is attended (Kaya & Kafaligonul, 2021; Van Der Burg et al., 2011; Wang et al., 2022). These inconsistent results indicate that the modality dominance of modal-based attention modulation may affect RSE.

Another method of manipulating the modality dominance involves changing the unimodal target probability (Egeth & Sager, 1977; Spence, 2009; Zampini et al., 2005). Previous studies have revealed that the magnitude of the modality dominance effect varies with the A and V unimodal target probability. For example, Sinnett et al. (2007) investigated incorrect responses to AV targets by changing the A and V unimodal target probability from 40% V:40% A to 60% V:20% A in the Colavita task. They found that the Colavita effect (visual dominance effect), in which participants made more incorrect V responses than incorrect A responses to AV targets, increased along with the increasing target probability of the V modality. Similarly, Zhang et al. (2018) modulated the A and V unimodal target probability from 1:3 to 3:1 in the sound-induced flash illusion task. The researchers observed that the double flash illusion (auditory dominance effect) increased along with the increasing target probability of the A modality. The above studies indicate that the modality dominance for unimodal target probability manipulation can effectively modulate the magnitude of the modality dominance effect. Specifically, the greater the target probability of the dominant modality, the greater the corresponding modality dominance effect.

An interesting question is whether the modality dominance manipulated by unimodal target probability can affect the RSE. Some indirect evidence hints at the possibility. The target probability effect (TPe, RT_{low probability} − RT_{high probability}) reveals significant differences in reaction time (RT) between unimodal targets with different probabilities in the location detection task (Laberge & Tweedy, 1964; Miller & Pachella, 1973). The response to the high probability modality is typically faster and earlier than the response to the low probability modality (Hon et al., 2013; Lucci et al., 2016). Thus, the unimodal target probability may also lead to modality dominance through prior entry (Vibell et al., 2007). More importantly, Van Der Stoep et al. (2017) proposed that the RT differences in unimodal processing indicates that the modality of prior entry becomes the dominant modality driving the response, and the contribution of the nondominant modality of later entry driving the response is weakened. The unequal contributions of dominant and nondominant modalities weaken the information integration between different modalities. Conversely, when none of the modalities dominate perception under the same unimodal target probability, the equal contribution between different modalities often leads to the largest benefits of RSE (Ernst & Banks, 2002; Leone & McCourt, 2013; Otto et al., 2013). As noted in location detection tasks, the modality dominance manipulated by unimodal target probability may also modulate the magnitude of RSE.

Taken together, the modal-based attention and the unimodal target probability can both induce modality dominance. However, it is unclear how these two methods will affect the RSE. The purpose of this study was to systematically investigate the effects of the modality dominance manipulated by modal-based attention and unimodal target probability on RSE in the location detection task. Three unimodal target probabilities were assessed: 10% V:40% A, 25% V:25% A, and 40% V:10% A. Modal-based attention includes modal-based divided attention (A and V) and modal-based selective attention (A or V). We conducted a total of two experiments. In Experiment 1, the location detection task of three unimodal probabilities was performed using modal-based divided attention. In Experiment 2, the location detection task of three unimodal probabilities was completed using modal-based selective attention. Experiments 2A and 2B were location detection tasks of three unimodal probabilities based on auditory selective attention and visual selective attention, respectively. We calculated the multisensory response enhancement (MRE) under each condition to measure the RSE. By comparing MRE performance in different conditions, we uncovered the effects of modality dominance on the RSE.

Experiment 1

Method

Experiment 1 aimed to investigate the effect of the modality dominance manipulated by modal-based divided attention and unimodal target probability on RSE. The participants were asked to pay attention to both the A and V modalities and perform location detection responses for A, V, and AV targets at different unimodal target probability conditions.

Participants

We calculated the sample size for a 3 × 3 two-way repeated-measures analysis of variance (ANOVA) using GPower 3.1.7 software (Faul et al., 2009; Van Der Stoep et al., 2017). The results indicated that to detect an effect size of η_p² = 0.23 (α = 0.05; 1–β = 0.80), a sample of at least 15 participants was needed. Thirty-three individuals participated in the experiment. We excluded one participant because the correct response at one of the levels was less than 80%, so that the final sample consisted of 32 participants (24 females, mean age: 21.84 years, range: 18–29 years), all of whom were right-handed. All participants had normal vision and hearing and no neurological or psychiatric disorders. All participants provided written informed consent before participating. The experiment was approved by the Ethics Committee of Liaoning Normal University and was performed in accordance with the principles expressed in the Declaration of Helsinki.

Apparatus, Stimuli, and Design

We performed the experiment using a Windows Power computer equipped with E-prime (version 1.0, Psychology Software Tools, Inc.) experimental software, and we gathered responses using a Microsoft keyboard. The E-prime software controlled the presentation of all stimuli. The fixation stimulus was a 0.05° × 0.05° white cross (Red, green, blue [RGB]: 255, 255, 255, respectively; 155.2 cd/m²) in the center of the screen. The unimodal V target was a 4° × 4° white horizontal square wave grating. The target was randomly presented at 6° below and 12° to the left and right of the central fixation. All visual stimuli were displayed on a black background on a 19-inch computer monitor (75 Hz, 1,024 × 768) located 60 cm from the participants in a dark, quiet room. The unimodal A target was a 1,000 Hz sinusoidal tone with an amplitude of 60 dB (as measured with an audiometer from the location of the participant; 10 ms rise and fall of the signal). All auditory stimuli were presented via two loudspeakers placed at both the left rear and the right rear of the monitor. The bimodal AV target consisted of the combined presentation of the unimodal A and V targets and was always spatially and temporally aligned.

The experiment contained 9 blocks of 1,440 testing trials. Each unimodal target probability consists of three blocks with a total of 480 trials: 16 V trials (8 left, 8 right), 64 A trials (32 left, 32 right), and 80 AV trials (40 left, 40 right) per block in the 10% V:40% A condition; 40 V trials (20 left, 20 right), 40 A trials (20 left, 20 right), and 80 AV trials (40 left, 40 right) per block in the 25% V:25% A condition; and 64 V trials (32 left, 32 right), 16 A trials (8 left, 8 right), and 80 AV trials (40 left, 40 right) per block in the 40% V:10% A condition. Within each block, the A, AV, and V targets were presented in random order. Similarly, the order in which the unimodal target probability was presented was balanced across participants. Before each unimodal target probability testing, the participant needed to complete a practice block with corresponding probability. A total of 40 trials were performed per practice block. Formal testing did not start until the participants understood the experimental requirements. This design enables them to have a correct rate of at least 80%.

Procedure and Task

As shown in Figure 1, each trial started with the presentation of a central fixation cross for a random duration of 850−1,350 ms. Next, the A, V, or AV targets appeared with equal probability toward the left or right for 100 ms. Following the presentation of the target, there was a response interval of 1,000 ms. Participants were instructed to keep focusing on the fixation cross throughout the experiment and press a button as quickly as possible when the A, V, or AV targets were presented. Participants were told to press the “F” key (left hand) if the targets were on the left and to press the “J” key (right hand) if the targets were on the right. After the end of a trial, the next trial began automatically. The total duration of the experimental task was approximately 45 min, including time for a break between the blocks. After each block was completed, participants were given feedback about their performance.

Figure 1.
Illustration of targets and procedure. The size and location of the targets are shown in the left panel, and the sequence of events and the duration (starting from the top) of AV target is illustrated in the right panel.

Data Analysis

We excluded error trials and trials with RT less than 100 ms or greater than 1,000 ms. Approximately 2.3% of the overall data points were excluded from further analysis. We computed the accuracy (ACC), and median RT of each participant in each condition for the data analysis. For the ACC and RT data, we conducted a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 3 (target modality: A, V, and AV) repeated-measures ANOVA. By comparing the modality RT differences in each unimodal target probability, we observed the RSE performance and the target probability effect. We used the Greenhouse–Geisser epsilon correction to correct for nonsphericity. We performed Bonferroni corrections for the post-hoc comparisons.

In addition to analyzing the ACC and median RT, we used MRE to measure the RSE. The RSE-box (Otto, 2019) for MATLAB (version 2018b) was used to investigate the amount of MRE (also called the redundancy gain), namely, the speedup in the bimodal condition compared to the fastest unimodal condition. We calculated MRE based on the cumulative distribution functions (CDF) of the A, V, and AV targets. First, we removed RT outliers at the individual participant and condition levels using the absolute deviation around the mean as the criterion (Leys et al., 2013) via the outCorrect function. We then downsampled the RT in each condition to the number of RT in the condition with the least number of trials left after outlier correction (i.e., the minimum of the number of trials in the A, V, and AV conditions; using the sampleDown function). We employed the getGain function to calculate the area between the bimodal AV CDF and the fastest of the two unimodal CDFs, that is, Grice's bound (Grice et al., 1984). Grice's bound is equal to the maximum probability (p) of the unimodal A and unimodal V RT (t), with t ranging from 0 to 1,000 ms, and can be calculated using Formula (1). The area between Grice's bound and the AV CDF (in gray) indicates the amount of MRE. Next, we computed the amount of MRE in each unimodal target probability using the respective unimodal and bimodal conditions (Van Der Stoep et al., 2021). We utilized one-sample t tests to determine whether MRE existed in each unimodal target probability. We employed one-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA to test for MRE differences between unimodal target probabilities. We computed the effect size of Cohen's dz for the one-sample t tests. We calculated the effect size of partial eta-squared (η_p²) for the repeated-measures ANOVA.
$P (R T_{G r i c e} < t) = m a x [P (R T_{A} < t), P (R T_{V} < t)]$
(1)
Finally, to test whether the RSE was related to the differences in RT between A and V targets (Figure 2), we also conducted Pearson correlation analyses between the differences in RT between A and V targets and MRE under modal-based divided attention. We performed all statistical analyses using SPSS version 17.0 software. We drew all figures using Adobe illustrator CS4 software (14.0.0).

Figure 2.
RT summary of Experiment 1. The average of median RT for the A, AV, and V targets and the significant RT differences between modalities in each unimodal target probability condition of Experiment 1 (paying attention to A and V). The error bars represent the standard errors. All significant differences are indicated with an asterisk ( p < .01, p < .001, respectively).

Results

Accuracy (ACC)

Experiment 1 yielded a higher correct detection rate for the A, V, and AV targets with an overall ACC of 98%. We entered the ACC into a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 3 (target modality: A, V, AV) repeated-measures ANOVA. The results showed a significant main effect for the target modality, F(2, 62) = 6.433, p = .003, η_p² = 0.172, but not for the unimodal target probability, F(1.509, 46.764) = 0.334, p = .657, η_p² = 0.011. No significant interaction was noted between these two factors, F(2.658, 82.402) = 1.752, p = .169, η_p² = 0.054.

Reaction Time (RT)

We first determined whether participants responded faster to the AV targets than to the A and V targets. Figure 2 shows the median RT of the A, AV, and V targets for each target probability condition in the modal-based divided attention task. We performed a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 3 (target modality: A, V, AV) repeated-measures ANOVA on the RT. The main effect of the target modality was significant, F(1.476, 45.766) = 111.115, p < .001, η_p² = 0.782. In addition, the interaction between the unimodal target probability and the target modality was significant, F(2.508, 77.745) = 31.073, p < .001, η_p² = 0.501. Simple effect analysis indicated that the A RT were significantly faster than the V RT (p = .002), and both the A RT and the V RT were slower than the AV RT (all p < .001) in the 10% V:40% A condition. The A RT and the V RT were slower than the AV RT (all p < .001) in the 25% V:25% A condition. The V RT were significantly faster than the A RT, and both the A RT and the V RT were slower than the AV RT (all p < .001) in the 40% V:10% A condition.

Multisensory Response Enhancement

To determine the amount of speed-up in the AV condition relative to the fastest A or V condition, we analyzed the amount of MRE (the gray area in Figure 3(a) to (c)) for each unimodal target probability condition. One-sample t tests provided very strong evidence of MRE being significantly greater than zero in the 10% V:40% A (M = 33 ms, SE = 2, t(31) = 13.674, p < .001, Cohen's dz = 2.417), 25% V:25% A (M = 31 ms, SE = 2, t(31) = 16.292, p < .001, Cohen's dz = 2.880), and 40% V: 10% A (M = 30 ms, SE = 2, t(31) = 15.940, p < .001, Cohen's dz = 2.818) conditions. One-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA employed for MRE showed that the main effect of the unimodal target probability was not significant, F(1.581, 49.004) = 0.561, p = .534, η_p² = 0.018, and no significant MRE difference was noted between unimodal target probabilities (Figure 3(d)).

Figure 3.
Multisensory RSE summary of Experiment 1 (paying attention to A and V). (a) The MRE area (gray shading) between the AV CDF and Grice's bound in the 10% V:40% A condition. (b) The MRE area between the AV CDF and Grice's bound in the 25% V:25% A condition. (c) The MRE area between the AV CDF and Grice's bound in the 40% V:10% A condition. (d) Comparison of MRE amounts under 10% V:40% A, 25% V:25% A, and 40% V:10% A conditions. Error bars represent the standard errors.

To establish whether RSE was related to the differences in RT between A and V targets, we used the Pearson correlation for the MRE and the differences in RT between A and V targets. However, no significant correlation was noted between the significant differences in RT between A and V targets and the magnitude of MRE in the 10% V:40% A (Pearson r = .069, p = .706) and 40% V:10% A (Pearson r = − .171, p = .350) conditions.

The RT analysis in Experiment 1 revealed a significant redundancy effect given that AV RT were significantly faster than A RT and V RT under all unimodal target probabilities. More importantly, the RT analysis in Experiment 1 revealed that unimodal target probability modulated differences in RT between A and V targets. Specifically, when the unimodal target probability is different (target probability effect, 10% V:40% A and 40% V:10% A conditions), the RT of unimodal A and V are significantly different. When the unimodal target probability is the same (25% V:25% A condition), the RT of unimodal A and V exhibit no significant difference. However, the MRE analysis in Experiment 1 showed that MRE was very robust, and had no significant differences were noted regardless of whether the RT between unimodal A and V targets were the same or different. In other words, the unimodal RT differences due to the unimodal target probabilities do not appear to be associated with MRE. Correlation analysis again confirmed the relationship between unimodal RT differences and MRE. Although the unimodal target probability can lead to different unimodal RT differences (significant or insignificant), no significant correlation was noted between the significant differences in RT between A and V targets and MRE in the 10% V:40% A and 40% V:10% A conditions. In summary, the results under modal-based divided attention indicate that unimodal RT differences caused by the unimodal target probability do not alter the MRE. Therefore, the modality dominance induced by modal-based divided attention and unimodal target probability cannot effectively regulate RSE.

Experiment 2

The results of Experiment 1 show that the modality dominance of modal-based divided attention and unimodal target probability manipulation had no significant effect on RSE. The aim of Experiment 2 was to investigate the effect of the modality dominance manipulated by modal-based selective attention and unimodal target probability on RSE. The participants were asked to selectively pay attention to the A or V modalities and perform location detection responses to A and AV targets, or V and AV targets at different unimodal target probability conditions.

Method

Participants

We calculated the sample size for a 3 × 2 two-way repeated-measures ANOVA using GPower 3.1.7 software (Faul et al., 2009; Van Der Stoep et al., 2017). The results indicated that to detect an effect size of η_p² = 0.23 (α = 0.05; 1–β = 0.80), a sample of at least 21 participants was needed. Thirty-four new individuals participated in Experiment 2. We screened the participants according to the same criteria as in Experiment 1. We excluded two participants, and the final dataset comprised 32 participants (23 females, mean age: 21.47 years, range: 18–30 years). Thirty-two participants were right-handed and had normal or corrected-to-normal vision and normal hearing. The experimental protocol was approved by the Ethics Committee of Liaoning Normal University and was conducted in accordance with the principles expressed in the Declaration of Helsinki.

Apparatus, Stimuli, and Design

The apparatus, stimuli, and design for Experiment 2 were identical to those in Experiment 1.

Procedure and Task

The procedure for Experiment 2 was identical to that noted for in Experiment 1 with the exception of the use of modal-based selective attention tasks. Each participant was required to complete two subexperiments: Experiment 2A and Experiment 2B. Experiment 2A involved the auditory selective attention task, and participants were asked to ignore the V targets and respond to the A and AV targets. Experiment 2B involved the visual selective attention task, and participants were asked to ignore the A targets and respond to the V and AV targets. The participants needed to press the “F” key in response to the left targets and the “J” key in response to the right targets as quickly and accurately as possible. Each subexperiment of the modal-based selective attention task lasted 45 min, and the participants completed experiments 2A and 2B with an interval of greater than half a day. The order of the modal-based selective attention and the unimodal target probability were balanced across participants.

Data Analysis

The data exclusion criteria were identical to those in Experiment 1. In the auditory selective attention task (Exp. 2A), approximately 1.3% of the overall data points were excluded from further analysis. We conducted a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: A, AV) repeated-measures ANOVA for the ACC of the A and AV targets, and the error rate of the V targets was subjected to a one-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA. In the visual selective attention task (Exp. 2B), approximately 1.0% of the overall data points were excluded from further analysis. We conducted a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: V, AV) repeated-measures ANOVA for the ACC of the V and AV targets. The error rate of the A targets was subjected to a one-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA.

It should be noted that the measurement of RSE in Experiment 2 was different from that in Experiment 1 due to the lack of RT in the unattended targets. According to the study of Lunn et al. (2019), bimodal RT are significantly faster than unimodal RT, which is a manifestation of RSE. We first revealed the magnitude of RSE by examining the difference between the bimodal RT and the unimodal RT. In the auditory selective attention task (Exp. 2A), the RT were subjected to a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: A, AV) repeated-measures ANOVA. In the visual selective attention task (Exp. 2B), the RT were subjected to a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: V, AV) repeated-measures ANOVA. In addition, in the study of Van Der Stoep et al. (2021), when a modality has an absolute response dominance over another modality (e.g., the visual CDF, Figure 2(b)), the CDF of the dominance modality is Grice's bound. The MRE can also be measured by directly calculating the area (e.g., the gray area between the AV CDF and the V CDF) between the bimodal CDF and dominance unimodal CDF and then revealing the magnitude of RSE. For example, MRE in auditory selection attention should be measured by calculating the area between AV CDF and A CDF in each unimodal target probability. Finally, we used a one-sample t test to examine the significance of MRE. We employed one-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA to further compare the RT differences (A minus AV, or V minus AV) between unimodal target probabilities and the MRE between unimodal target probabilities.

Results

Accuracy (ACC)

The auditory selective attention task (Exp. 2A) yielded a higher correct detection rate for the A and AV targets with an overall ACC of 98%. The 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: A, AV) repeated-measures ANOVA on ACC showed a significant main effect for the target modality, F(1, 31) = 10.046, p = .003, η_p² = 0.245, but not for the unimodal target probability, F(1.448, 44.894) = 0.766, p = .432, η_p² = 0.024. The interaction between the target modality and the unimodal target probability was not significant, F(1.685, 52.224) = 1.985, p = .154, η_p² = 0.060. In addition, we also observed fewer incorrect detections of the V target with an overall error rate of 2.4%. One-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA was employed to assess the error rate and showed that the main effect of the unimodal target probability was significant, F(1.298, 40.224) = 13.864, p < .001, η_p² = 0.309. Post-hoc tests showed that the error rate in the 10% V:40% A condition was significantly greater than error rate in the 25% V:25% A (p = .005) and 40% V:10% A (p = .001) conditions.

The visual selective attention task (Exp. 2B) yielded a higher correct detection for the V and AV targets with an overall ACC of 99%. The 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: V, AV) repeated-measures ANOVA on ACC indicated a significant main effect for the target modality, F(1, 31) = 10.565, p = .003, η_p² = 0.254, but not for the unimodal target probability, F(1.424, 45.540) = 3.142, p = .069, η_p² = 0.092. The interaction between the target modality and the unimodal target probability was not significant, F(2, 62) = 2.013, p = .142, η_p² = 0.061. In addition, we also observed less error detection for the A target with an overall error rate of 1.7%. One-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA was employed to assess error rate and showed that the main effect of the unimodal target probability was significant, F(1.208, 37.459) = 16.183, p < .001, η_p² = 0.343. Post-hoc tests showed that the error rate in the 40% V:10% A condition was significantly greater than the error rate in the 10% V:40% A (p < .001) and 25% V:25% A (p = .003) conditions.

Reaction Time (RT)

Figure 4(a) shows the median RT to the A and AV targets for each unimodal target probability condition in the auditory selective attention task (Exp. 2A). We performed a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: A, AV) repeated-measures ANOVA on the RT. Both the main effect of the unimodal target probability, F(2, 62) = 39.307, p < .001, η_p² = 0.559, and the main effect of the target modality were significant, F(1, 31) = 40.245, p < .001, η_p² = 0.565. Although the interaction between the unimodal target probability and the target modality was not significant, F(1.465, 45.408) = 0.688, p = .464, η_p² = 0.022, paired samples t tests revealed that the AV RT were significantly faster than the A RT in the 10% V:40% A, 25% V:25% A, and 40% V:10% A conditions (all p < .001). We also performed a one-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA on the RT differences of A minus AV. The main effect of the unimodal target probability was not significant, F(1.465, 45.408) = 0.688, p = .464, η_p*² = 0.022, and no significant RT differences of A minus AV was noted between unimodal target probabilities.

Figure 4.
RT summary of Experiment 2. (a) The average of the median RT for the A and AV targets and the significant RT differences between modalities in each unimodal target probability condition of Experiment 2A (paying attention to A). (b) The average of the median RT for the V and AV targets and the significant RT differences between modalities in each unimodal target probability condition of Experiment 2B (paying attention to V). The error bars represent the standard errors. All significant differences are indicated with an asterisk (*** p < .001).

Figure 4(b) outlines the median RT of the V and AV targets for each unimodal target probability condition in the visual selective attention task (Exp. 2B). We entered the RT into a 3 (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) × 2 (target modality: V and AV) repeated-measures ANOVA. The main effect of the unimodal target probability, F(2, 62) = 69.959, p < .001, η_p² = 0.693, and the main effect of the target modality were significant, F(1, 31) = 126.819, p < .001, η_p² = 0.804. In addition, the interaction between the unimodal target probability and the target modality was also significant, F(2, 62) = 9.874, p < .001, η_p² = 0.242, and simple effect analysis demonstrated that the AV RT were significantly faster than the V RT in the 10% V:40% A, 25% V:25% A, and 40% V:10% A conditions (all p < .001). We also performed a one-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA on the RT differences of V minus AV. The main effect of the unimodal target probability was significant, F(2, 62) = 9.874, p < .001, η_p² = 0.242. Post-hoc tests revealed that the RT differences of V minus AV in the 10% V:40% A (p = .036) condition was significantly larger than the RT differences of V minus AV in the 25% V:25% A (p = .006) and 40% V:10% A (p = .003) conditions.

Multisensory Response Enhancement

We analyzed the amount of MRE (the gray area in Figure 5(a) to (c)) for each unimodal target probability condition in the auditory selective attention task (Exp. 2A). One-sample t tests provide strong evidence of the MRE being significantly greater than zero in the 10% V:40% A (M = 29 ms, SE = 5, t(31) = 5.982, p < .001, Cohen's dz = 1.057), 25% V:25% A (M = 33 ms, SE = 6, t(31) = 5.488, p < .001, Cohen's dz = 0.970), and 40% V:10% A (M = 39 ms, SE = 8, t(31) = 4.867, p < .001, Cohen's dz = 0.860) conditions. One-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA employed for MRE showed that the main effect of the unimodal target probability was not significant, F(1.547, 47.953) = 1.922, p = .166, η_p² = 0.058. No significant difference in MRE was noted under 10% V:40% A, 25% V:25% A, 40% V:10% A conditions (Figure 5(d)).

Figure 5.
Multisensory RSE summary of Experiment 2A (paying attention to A). (a) The MRE area (gray shading) between the AV CDF and the A CDF in the 10% V:40% A condition. (b) The MRE area between the AV CDF and the A CDF in the 25% V:25% A condition. (c) The MRE area between the AV CDF and the A CDF in the 40% V:10% A condition. (d) Comparison of MRE amounts under 10% V:40% A, 25% V:25% A, and 40% V:10% A conditions. Error bars represent the standard errors.

Similarly, we also analyzed the amount of MRE (the gray area in Figure 6(a) to (c)) for each unimodal target probability condition in the visual selective attention task (Exp. 2B). One-sample t tests indicated significant MRE in the 10% V:40% A (M = 40 ms, SE = 4, t(31) = 9.704, p < .001, Cohen's dz = 1.715), 25% V:25% A (M = 29 ms, SE = 3, t(31) = 9.091, p < .001, Cohen's dz = 1.607), and 40% V:10% A (M = 25 ms, SE = 3, t(31) = 9.908, p < .001, Cohen's dz = 1.751) conditions. One-way (unimodal target probability: 10% V:40% A, 25% V:25% A, 40% V:10% A) repeated-measures ANOVA employed for MRE showed that the main effect of the unimodal target probability was significant, F(2, 62) = 15.880, p < .001, η_p² = 0.339. Post-hoc tests revealed that MRE in the 10% V:40% A (p = .036) condition was significantly greater than the MRE in the 25%V:25%A (p = .001) and 40% V:10% A (p < .001) conditions (Figure 6(d)).

Figure 6.
Multisensory RSE summary of Experiment 2B (paying attention to V). (a) The MRE area (gray shading) between the AV CDF and the V CDF in the 10% V:40% A condition. (b) The MRE area between the AV CDF and the V CDF in the 25% V:25% A condition. (c) The MRE area between the AV CDF and the V CDF in the 40% V:10% A condition. (d) Comparison of MRE amounts under 10% V:40% A, 25% V:25% A, and 40% V:10% A conditions. The error bars represent the standard errors. All significant differences are indicated with an asterisk ( p < .01, * p < .001, respectively).

The RT analysis in Experiment 2 also revealed a significant redundancy effect given that AV RT were significantly faster than either A RT or V RT in all unimodal target probability conditions. The RT results in Experiment 2 show some differences from the RT results in Experiment 1. Experiment 2A revealed that the RT differences of A minus AV did not change with the unimodal target probability, whereas Experiment 2B revealed that the RT differences of V minus AV significantly decreased with the increasing probability of the V target. Similarly, the MRE results in Experiment 2 also show some difference from those in Experiment 1. Specifically, Experiment 2A detected significant MRE values for all unimodal target probabilities in auditory selective attention. Unfortunately, no significant difference in the magnitude of MRE was noted between unimodal target probabilities. Of note, despite the lack of a significant difference, MRE shows a decreasing trend as the probability of A target increases. Experiment 2B found significant MRE at all unimodal target probabilities in visual selective attention. However, a significant difference in the magnitude of the MRE was noted between unimodal target probabilities, and the MRE decreased with the increasing probability of V target. The comparison analysis found that although the pattern of the MRE changes in these two selective tasks was similar, the unimodal target probability caused the MRE to significantly change exclusively in visual selective attention. In short, these results indicate that the modality dominance induced by unimodal target probability and modal-based selective attention occasionally exhibit a modulating effect on RSE.

Discussion

Using the location detection task, we explored the effect of the modality dominance manipulated by modal-based attention and unimodal target probability on RSE. When paying attention to both the A and V modalities (Exp. 1, Figure 3(d)), we found that the RSE was robust and that it does not vary with the unimodal target probability. When paying attention to a single modality, the RSE in auditory selective attention (Exp. 2A, Figure 5(d)) also did not change significantly with the unimodal target probabilities. However, the RSE in visual selective attention (Exp. 2B, Figure 6(d)) tended to decrease significantly with the increasing probability of V targets. The modality dominance of unimodal target probability manipulation modulated RSE more significantly in visual selective attention than in auditory selective attention.

When paying attention to both the A and V modalities, the unimodal target probability modulated the RT differences between the A and V modalities but not the MRE. These results seem inconsistent with previous studies. Van Der Stoep et al. (2017) found that the RT differences between A and V modalities with the same unimodal target probability were significantly negatively correlated with MRE. The RT differences between the A and V modalities was smaller, whereas the amount of rMRE was larger. Our study shows that the different unimodal target probabilities (10% V:40% A and 40% V:10% A) can also lead to significant RT differences between the A and V modalities; however, the RT differences were not significantly correlated with MRE. In addition, no significant differences in MRE were noted among unimodal target probabilities. More importantly, although the RT differences between the A and V modalities differed between unimodal target probabilities, the MRE did not differ significantly between unimodal target probabilities. In other words, the RT differences between the A and V modalities mediated by the unimodal target probability did not effectively influence the RSE in this study.

As Van Der Stoep et al. (2017) suggested, the RT differences in the same A and V unimodal target probability may reflect differences in signal strength rather than differences in sensory processing times. Different signal strengths may lead to the response of one modality dominating the response of the other modality, thus influencing integration processing (Tang et al., 2019). Therefore, the magnitude of MRE may be regulated by the modality signal strengths differences. Here, the signal strength helps to explain our current findings. When attention was paid to both the A and V modalities (Exp. 1, Figure 2), no significant RT differences was noted between the A and V modalities at the same unimodal target probability, indicating that the A and V signal strengths were the same in our study. Although different unimodal target probabilities can induce modality dominance and lead to significant RT differences between the A and V modalities, the A and V signal strengths did not change. Thus, MRE did not change significantly with the unimodal target probabilities in modal-based divided attention. In other words, the modality dominance of modal-based divided attention and unimodal target probability manipulation can induce significant RSE, but the unimodal target probability in modal-based divided attention cannot effectively modulate the magnitude of RSE.

Surprisingly, the RSE is consistently significant when focusing selectively on one modality. MRE in auditory selective attention did not differ significantly in unimodal target probability (Exp. 2A, Figure 5(d)), whereas MRE in visual selective attention decreased significantly with the increasing probability of V target (Exp. 2B, Figure 6(d)). The spread of attention across modalities may play a critical role in this perceptual performance enhancement (Zimmer et al., 2010). Previous research has shown that attention to one sensory modality can spread to encompass simultaneous signals from another modality, even when they are task-irrelevant and from a different location (Busse et al., 2005; Fiebelkorn et al., 2010; Kaya & Kafaligonul, 2021). Our study always presented the visual and auditory components of the bimodal targets simultaneously at the same location, which greatly promoted the spread of attention across modalities and weakened the modality dominance induced by modal-based selective attention enhancement and suppression process (Wahn & König, 2017). Thus, focusing on one modality does not always eliminate RSE. Interestingly, we also found that the unimodal target probabilities in visual selective attention significantly modulated RSE. Selective attention makes modalities generate attention demands, and RSE will be affected by these attention demands (Pollmann & Zaidel, 1999). Studies have shown that the unimodal target probability can change attention demands (Hon et al., 2016; Hon & Tan, 2013), which may explain why MRE varies with unimodal target probabilities in modal-based selective attention. Overall, the modality dominance with modal-based selective attention and unimodal target probability manipulation can induce significant RSE. The unimodal target probability in modal-based selective attention can occasionally significantly modulate RSE.

The significant modulation of RSE by the unimodal target probability is mainly reflected in visual selective attention. The MRE decreases with the increase of the modality probability with attention dominance, which is contrary to the trend observed wherein the modality dominance effect increases along with the increase in the probability of dominance modality (Egeth & Sager, 1977; Sinnett et al., 2007; Zhang et al., 2018). The reason for this opposite trend is that modality dominance has different effects on RSE and modality dominance effects. Modality dominance is not conducive to RSE but helps to induce the modality dominance effect. Previous studies have shown that attention-related mechanisms play a crucial role in object-binding processes (Robertson, 2003), and focal spatial selection results in the preferential processing of an object (Duncan, 2006). Modal-based selective attention may also induce modality dominance processing. In this study, visual selective attention manipulated the dominance of the V modality. Increasing the probability of V targets will deepen the modality dominance between the A and V modalities, thus leading to a gradual decrease in RSE. The “modality dominated assumption” (Welch & Warren, 1980) also supports the role of modal-based selective attention. Many factors affect modality dominance, such as the reliability of modality information (Ernst & Bülthoff, 2004; Fetsch et al., 2012; Helbig & Ernst, 2008), the sensitivity of modality to task (Cai & Connell, 2015; Navarra et al., 2010), and the availability of modality attention (Fang et al., 2020; Poole et al., 2021). However, modal-based attention plays a key role in modulating the modality dominance effect. In the current study, the modulation of RSE by the unimodal target probability observed in visual selective attention is likely influenced by the role of modality dominance.

Of note, the influence of the unimodal target probability on RSE also showed some differences between the auditory and visual selective attention. The MRE in visual selective attention was more sensitive to the changes in unimodal target probability than in auditory selective attention. First, the spread of attention across modalities provided evidence that the transfer of selective attention from visual-to-auditory features operates in a fundamentally different manner than the transfer from auditory-to-visual features. Visual representations have a greater influence on their auditory counterparts than vice versa, and the asymmetric modality dominance caused by modal-based selective attention may lead to different effects of unimodal target probability on RSE (Molholm et al., 2007). In addition, the role of modality attention may be task-dependent (Wahn & König 2015a, 2015b, 2016, 2017). Once the attentional selection has occurred, performance on tasks similar in nature to selected stimuli generally improves (Alais et al., 2006). Thus, visually selective attention promotes performance on visual-related tasks, whereas auditory selective attention enhances performance on auditory-related tasks. Spatial tasks are typically visual dominance tasks (Diaconescu et al., 2013; Odegaard et al., 2016; Robinson et al., 2016), and time-related tasks are usually auditory dominance tasks (Chen & Vroomen, 2013; Morein-Zamir et al., 2003; Talsma et al., 2010). Participants in this study performed left-right key responses depending on the target location. High spatial sensitivity determines that the current location detection task is a visual dominance task. Unsurprisingly, the RSE under this task is sensitive to visual selective attention. However, future research needs to further test this hypothesis in auditory dominance tasks.

Summary

In conclusion, this study aimed to explore the effect of modality dominance manipulated by modal-based attention and unimodal target probability on RSE. The results demonstrate that the modality dominance of modal-based attention and unimodal target probability manipulation can induce significant RSE but do not always significantly modulate RSE. Specifically, when paying attention to both the A and V modalities, the unimodal target probability has no modulating effect on RSE. Similarly, when selectively paying attention to the A modality, the unimodal target probability also has no modulating effect on RSE. However, when selectively paying attention to the V modality, RSE is significantly modulated by the unimodal target probability. RSE decreases with the increasing probability of V targets. This study is the first to demonstrate that the modality dominance of unimodal target probability manipulation can modulate RSE in visual selective attention. However, the modulating effect of the unimodal target probability on RSE is opposite to the modulating effect of the unimodal target probability on the modality dominance effect.

Footnotes

Author Contribution(s)

Xueli Wang: Writing – original draft.

Yingnan Wu: Formal analysis.

Zepeng Xing: Software.

Xinzhong Cui: Methodology.

Min Gao: Conceptualization; Supervision.

Xiaoyu Tang: Writing – review & editing.

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Youth Project of Humanities and Social Sciences Financed by Ministry of Education of China (22YJC190020, X.T.), the Natural Science Foundation of Liaoning Province of china (2022-MS-312, X.T.) and the Natural Science Basic Scientific Research Project of Educational Department of Liaoning Province of china (LJKZ0987, X.T. and LJKQZ2021091, M.G.).

ORCID iD

Xiaoyu Tang

References

Alais

Morrone

Burr

. (2006). Separate attentional resources for vision and audition. Proceedings of the Royal Society B: Biological Sciences, 273(1592), 1339–1345. https://doi.org/10.1098/rspb.2005.3420

Blurton

S. P.

Greenlee

M. W.

Gondan

. (2014). Multisensory processing of redundant information in go/no-go and choice responses. Attention, Perception, & Psychophysics, 76(4), 1212–1233. https://doi.org/10.3758/s13414-014-0644-0

Busse

Roberts

K. C.

Crist

R. E.

Weissman

D. H.

Woldorff

M. G

. (2005). The spreadof attention across modalities and space in a multisensory object. Proceedings of the National Academy of Sciences, 102(51), 18751–18756. https://doi.org/10.1073/pnas.0507704102

Cai

Z. G.

Connell

. (2015). Space-time interdependence: Evidence against asymmetric mapping between time and space. Cognition, 136, 268–281. https://doi.org/10.1016/j.cognition.2014.11.039

Chen

Vroomen

. (2013). Intersensory binding across space and time: A tutorial review. Attention, Perception, & Psychophysics, 75(5), 790–811. https://doi.org/10.3758/s13414-013-0475-4

Colavita

F. B

. (1974). Human sensory dominance. Perception & Psychophysics, 16(2), 409–412. https://doi.org/10.3758/BF03203962

Colavita

Tomko

Weisberg

. (2013). Visual prepotency and eye orientation. Bulletin of the Psychonomic Society, 8(1), 25–26. https://doi.org/10.3758/BF03337062

Diaconescu

A. O.

Hasher

McIntosh

A. R

. (2013). Visual dominance and multisensory integration changes with age. NeuroImage, 65, 152–166. https://doi.org/10.1016/j.neuroimage.2012.09.057

Duncan

. (2006). EPS Mid-Career Award 2004: Brain mechanisms of attention. Quarterly Journal of Experimental Psychology, 59(1), 2–27. https://doi.org/10.1080/17470210500260

10.

Egeth

Sager

. (1977). On the locus of visual dominance. Attention, Perception, & Psychophysics, 22(1), 77–86. https://doi.org/10.3758/BF03206083

11.

Ernst

M. O.

Banks

M. S

. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429–433. https://doi.org/10.1038/415429a

12.

Ernst

M. O.

Bülthoff

H. H

. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8(4), 162–169. https://doi.org/10.1016/j.tics.2004.02.002

13.

Fang

Tao

Chen

. (2020). Top-down attention modulates the direction and magnitude of sensory dominance. Experimental Brain Research, 238(3), 587–600. https://doi.org/10.1007/s00221-020-05737-7

14.

Faul

Erdfelder

Buchner

Lang

A.-G

. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149

15.

Fetsch

C. R.

Pouget

DeAngelis

G. C.

Angelaki

D. E

. (2012). Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neuroscience, 15(1), 146–154. https://doi.org/10.1038/nn.2983

16.

Fiebelkorn

I. C.

Foxe

J. J.

Molholm

. (2010). Dual mechanisms for the cross-sensory spread of attention: How much do learned associations matter? Cerebral Cortex, 20(1), 109–120. https://doi.org/10.1093/cercor/bhp083

17.

Gondan

Minakata

. (2016). A tutorial on testing the race model inequality. Attention, Perception, & Psychophysics, 78(3), 723–735. https://doi.org/10.3758/s13414-015-1018-y

18.

Grice

G. R.

Canham

Gwynne

J. W

. (1984). Absence of a redundant-signals effect in a reaction time task with divided attention. Perception & Psychophysics, 36(6), 565–570. https://doi.org/10.3758/BF03207517

19.

Guerreiro

M. J. S.

Anguera

J. A.

Mishra

Van Gerven

P. W. M.

Gazzaley

. (2014). Age-equivalent top–down modulation during cross-modal selective attention. Journal of Cognitive Neuroscience, 26(12), 2827–2839. https://doi.org/10.1162/jocn_a_00685

20.

Harrar

Tammam

Pérez-Bellido

Pitt

Stein

Spence

. (2014). Multisensory integration and attention in developmental dyslexia. Current Biology, 24(5), 531–535. https://doi.org/10.1016/j.cub.2014.01.029

21.

Helbig

H. B.

Ernst

M. O

. (2008). Visual-haptic cue weighting is independent of modality-specific attention. Journal of Vision, 8(1), 21. https://doi.org/10.1167/8.1.21

22.

Hershenson

. (1962). Reaction time as a measure of intersensory facilitation. Journal of Experimental Psychology, 63(3), 289–293. https://doi.org/10.1037/h0039516

23.

Hon

Chan

. (2016). Rare targets are less susceptible to attention capture once detection has begun. Psychonomic Bulletin & Review, 23(2), 445–450. https://doi.org/10.3758/s13423-015-0921-5

24.

Hon

Tan

C.-H

. (2013). Why rare targets are slow: Evidence that the target probability effect has an attentional locus. Attention, Perception, & Psychophysics, 75(3), 388–393. https://doi.org/10.3758/s13414-013-0434-0

25.

Hon

., Yap

M. J.

Jabar

S. B.

(2013). The trajectory of the target probability effect. Attention, Perception, & Psychophysics, 75(4), 661–666. https://doi.org/10.3758/s13414-013-0429-x

26.

Johnson

J. A.

Zatorre

R. J

. (2006). Neural substrates for dividing and focusing attention between simultaneous auditory and visual events. NeuroImage, 31(4), 1673–1681. https://doi.org/10.1016/j.neuroimage.2006.02.026

27.

Kaya

Kafaligonul

. (2021). Audiovisual interactions in speeded discrimination of a visual event. Psychophysiology, 58(4), e13777. https://doi.org/10.1111/psyp.13777

28.

Kinchla

R. A

. (1974). Detecting target elements in multielement arrays: A confusability model. Perception & Psychophysics, 15(1), 149–158. https://doi.org/10.3758/BF03205843

29.

Laberge

Tweedy

J. R

. (1964). Presentation probability and choice time. Journal of Experimental Psychology, 68(5), 477–481. https://doi.org/10.1037/h0048691

30.

Lakatos

O’Connell

M. N.

Barczak

Mills

Javitt

Schroeder

. (2009). The leading sense: Supramodal control of neurophysiological context by attention. Neuron, 64(3), 419–430. https://doi.org/10.1016/j.neuron.2009.10.014

31.

Leone

L. M.

McCourt

M. E

. (2013). The roles of physical and physiological simultaneity in audiovisual multisensory facilitation. i-Perception, 4(4), 213–228. https://doi.org/10.1068/i0532

32.

Lewkowicz

D. J.

Ghazanfar

A. A

. (2009). The emergence of multisensory systems through perceptual narrowing. Trends in Cognitive Sciences, 13(11), 470–478. https://doi.org/10.1016/j.tics.2009.08.004

33.

Leys

Ley

Klein

Bernard

Licata

. (2013). Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4), 764–766. https://doi.org/10.1016/j.jesp.2013.03.013

34.

Lucci

Berchicci

Perri

R. L.

Spinelli

Di Russo

. (2016). Effect of target probability on pre-stimulus brain activity. Neuroscience, 322(13), 121–128. https://doi.org/10.1016/j.neuroscience.2016.02.029

35.

Lunn

Sjoblom

Ward

Soto-Faraco

Forster

. (2019). Multisensory enhancement of attention depends on whether you are already paying attention. Cognition, 187, 38–49. https://doi.org/10.1016/j.cognition.2019.02.008

36.

Miller

. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14(2), 247–279. https://doi.org/10.1016/0010-0285(82)90010-X

37.

Miller

. (1991). Channel interaction and the redundant-targets effect in bimodal divided attention. Journal of Experimental Psychology: Human Perception and Performance, 17(1), 160–169.https://doi.org/10.1037/0096-1523.17.1.160

38.

Miller

. (2004). Exaggerated redundancy gain in the split brain: A hemispheric coactivation account. Cognitive Psychology, 49(2), 118–154. https://doi.org/10.1016/j.cogpsych.2003.12.003

39.

Miller

J. O.

Pachella

R. G

. (1973). Locus of the stimulus probability effect. Journal of Experimental Psychology, 101(2), 227–231. https://doi.org/10.1037/h0035214

40.

Mishra

Gazzaley

. (2012). Attention distributed across sensory modalities enhances perceptual performance. Journal of Neuroscience, 32(35), 12294–12302. https://doi.org/10.1523/JNEUROSCI.0867-12.2012

41.

Molholm

Martinez

Shpaner

Foxe

J. J

. (2007). Object-based attention is multisensory: Co-activation of an object’s representations in ignored sensory modalities. European Journal of Neuroscience, 26(2), 499–509. https://doi.org/10.1111/j.1460-9568.2007.05668.x

42.

Morein-Zamir

Soto-Faraco

Kingstone

. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17(1), 154–163. https://doi.org/10.1016/S0926-6410(03)00089-2

43.

Mozolic

J. L.

Hugenschmidt

C. E.

Peiffer

A. M.

Laurienti

P. J

. (2008). Modality-specific selective attention attenuates multisensory integration. Experimental Brain Research, 184(1), 39–52. https://doi.org/10.1007/s00221-007-1080-3

44.

Murray

M. M.

Eardley

A. F.

Edginton

Oyekan

Smyth

Matusz

P. J

. (2018). Sensory dominance and multisensory integration as screening tools in aging. Scientific Reports, 8(1), 8901. https://doi.org/10.1038/s41598-018-27288-2

45.

Navarra

Alsius

Soto-Faraco

Spence

. (2010). Assessing the role of attention in the audiovisual integration of speech. Information Fusion, 11(1), 4–11. https://doi.org/10.1016/j.inffus.2009.04.001

46.

Odegaard

Wozny

D. R.

Shams

. (2016). The effects of selective and divided attention on sensory precision and integration. Neuroscience Letters, 614, 24–28. https://doi.org/10.1016/j.neulet.2015.12.039

47.

Otto

T. U

. (2019). RSE-box: An analysis and modelling package to study response times to multiple signals. The Quantitative Methods for Psychology, 15(2), 112–133. https://doi.org/10.20982/tqmp.15.2.p112

48.

Otto

T. U.

Dassy

Mamassian

. (2013). Principles of multisensory behavior. Journal of Neuroscience, 33(17), 7463–7474. https://doi.org/10.1523/JNEUROSCI.4678-12.2013

49.

Pollmann

Zaidel

. (1999). Redundancy gains for visual search after complete commissurotomy. Neuropsychology, 13(2), 246–258. https://doi.org/10.1037/0894-4105.13.2.246

50.

Poole

Miles

Gowen

Poliakoff

. (2021). Shifting attention between modalities: Revisiting the modality-shift effect in autism. Attention, Perception, & Psychophysics, 83(6), 2498–2509. https://doi.org/10.3758/s13414-021-02302-4

51.

Robertson

L. C

. (2003). Binding, spatial attention and perceptual awareness. Nature Reviews Neuroscience, 4(2), 93–102. https://doi.org/10.1038/nrn1030

52.

Robinson

C. W.

Chandra

Sinnett

. (2016). Existence of competing modality dominances. Attention, Perception, & Psychophysics, 78(4), 1104–1114. https://doi.org/10.3758/s13414-016-1061-3

53.

Shams

Kamitani

Shimojo

. (2000). What you see is what you hear. Nature, 408(6814), 788. https://doi.org/10.1038/35048669

54.

Shams

Kamitani

Shimojo

. (2002). Visual illusion induced by sound. Cognitive Brain Research, 14(1), 147–152. https://doi.org/10.1016/S0926-6410(02)00069-1

55.

Sinnett

Spence

Soto-Faraco

. (2007). Visual dominance and attention: The Colavita effect revisited. Perception & Psychophysics, 69(5), 673–686. https://doi.org/10.3758/BF03193770

56.

Spence

. (2009). Explaining the Colavita visual dominance effect. Progress in Brain Research, 176, 245–258. https://doi.org/10.1016/S0079-6123(09)17615-X

57.

Spence

Nicholls

Driver

. (2001). The cost of expecting events in the wrong sensory modality. Perception & Psychophysics, 63(2), 330–336. https://doi.org/10.3758/BF03194473

58.

Talsma

Doty

T. J.

Woldorff

M. G

. (2007). Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cerebral Cortex, 17(3), 679–690. https://doi.org/10.1093/cercor/bhk016

59.

Talsma

Senkowski

Soto-Faraco

Woldorff

. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Sciences, 14(9), 400–410. https://doi.org/10.1016/j.tics.2010.06.008

60.

Tang

Gao

Yang

Ren

Zhang

. (2019). Bimodal-divided attention attenuates visually induced inhibition of return with audiovisual targets. Experimental Brain Research, 237(4), 1093–1107. https://doi.org/10.1007/s00221-019-05488-0

61.

Tang

Shen

. (2016). The interactions of multisensory integration with endogenous and exogenous attention. Neuroscience & Biobehavioral Reviews, 61, 208–224. https://doi.org/10.1016/j.neubiorev.2015.11.002

62.

Van Der Burg

Talsma

Olivers

C. N. L.

Hickey

Theeuwes

. (2011). Early multisensory interactions affect the competition among multiple visual objects. NeuroImage, 55(3), 1208–1218. https://doi.org/10.1016/j.neuroimage.2010.12.068

63.

Van Der Stoep

Van Der Smagt

M. J.

Notaro

Spock

Naber

. (2021). The additive nature of the human multisensory evoked pupil response. Scientific Reports, 11(1), 1–12. https://doi.org/10.1038/s41598-020-80286-1

64.

Van Der Stoep

Van Der Stigchel

Nijboer

T. C. W.

Spence

. (2017). Visually induced inhibition of return affects the integration of auditory and visual information. Perception, 46(1), 6–17. https://doi.org/10.1177/0301006616661934

65.

Vibell

Klinge

Zampini

Spence

Nobre

A. C

. (2007). Temporal order is coded temporally in the brain: Early event-related potential latency shifts underlying prior entry in a cross-modal temporal order judgment task. Journal of Cognitive Neuroscience, 19(1), 109–120. https://doi.org/10.1162/jocn.2007.19.1.109

66.

Wahn

König

. (2015a). Audition and vision share spatial attentional resources, yet attentional load does not disrupt audiovisual integration. Frontiers in Psychology, 6, 1084. https://doi.org/10.3389/fpsyg.2015.01084

67.

Wahn

König

. (2015b). Vision and haptics share spatial attentional resources and visuotactile integration is not affected by high attentional load. Multisensory Research, 28(3-4), 371–392. https://doi.org/10.1163/22134808-00002482

68.

Wahn

König

. (2016). Attentional resource allocation in visuotactile processing depends on the task, but optimal visuotactile integration does not depend on attentional resources. Frontiers in Integrative Neuroscience, 10, 13. https://doi.org/10.3389/fnint.2016.00013

69.

Wahn

König

. (2017). Is attentional resource allocation across sensory modalities task-dependent? Advances in Cognitive Psychology, 13(1), 83–96. https://doi.org/10.5709/acp-0209-2

70.

Wang

Qian

Zhao

Tang

Zhang

. (2022). Modal-based attention modulates attentional blink. Attention, Perception, & Psychophysics, 84(2), 372–382. https://doi.org/10.3758/s13414-021-02413-y

71.

Welch

R. B.

Warren

D. H

. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88(3), 638–667. https://doi.org/10.1037/0033-2909.88.3.638

72.

Yang

Nakamura

Shen

Ohta

Abe

(2012). Delayed audiovisual integration of patients with mild cognitive impairment and Alzheimer’s disease compared with normal aged controls. Journal of Alzheimer’s Disease, 32(2), 317–328. https://doi.org/10.3233/JAD-2012-111070

73.

Yang

Ren

Yang

D. O.

Yuan

(2016). The influence of selective and divided attention on audiovisual integration in children. Perception, 45(5), 515–526. https://doi.org/10.1177/0301006616629025

74.

Zampini

Shore

Spence

. (2005). Audiovisual prior entry. Neuroscience Letters, 381(3), 217–222. https://doi.org/10.1016/j.neulet.2005.01.085

75.

Zhang

Tang

Ning

Wang

. (2018). The effects of modal-based endogenous attention on sound-induced flash illusion. Acta Psychologica Sinica, 50(11), 1212–1221. https://doi.org/10.3724/SP.J.1041.2018.01212

76.

Zimmer

Itthipanyanan

Grent-’t-Jong

Woldorff

M. G

. (2010). The electrophysiological time course of the interaction of stimulus conflict and the multisensory spread of attention. European Journal of Neuroscience, 31(10), 1744–1754. https://doi.org/10.1111/j.1460-9568.2010.07229.x