Abstract
Subsecond temporal processing is crucial for activities requiring precise timing. Here, we investigated perceptual learning of crossmodal (auditory–visual or visual–auditory) temporal interval discrimination (TID) and its impacts on unimodal (visual or auditory) TID performance. The research purpose was to test whether learning is based on a more abstract and conceptual representation of subsecond time, which would predict crossmodal to unimodal learning transfer. The experiments revealed that learning to discriminate a 200-ms crossmodal temporal interval, defined by a pair of visual and auditory stimuli, significantly reduced crossmodal TID thresholds. Moreover, the crossmodal TID training also minimized unimodal TID thresholds with a pair of visual or auditory stimuli at the same interval, even if crossmodal TID thresholds are multiple times higher than unimodal TID thresholds. Subsequent training on unimodal TID failed to reduce unimodal TID thresholds further. These results indicate that learning of high-threshold crossmodal TID tasks can benefit low-threshold unimodal temporal processing, which may be achieved through training-induced improvement of a conceptual representation of subsecond time in the brain.
Subsecond temporal processing plays a crucial role in understanding dynamic events like speech and music. Numerous studies have demonstrated that training can improve performance in subsecond temporal tasks, such as temporal interval discrimination (TID), which requires the judgment of whether a test interval is longer (or shorter) than a standard interval (Bueti & Buonomano, 2014; Wright et al., 1997).
TID learning has been shown to be modality-specific, in that learning only transfers from audition to vision, but not vice versa (Bratzke et al., 2012; McGovern et al., 2016). This unidirectional crossmodal transfer of TID learning is interpreted as a result of dominant auditory temporal processing that also takes over time coding in other senses due to its high precision (Bratzke et al., 2012; Guttman et al., 2005; Kanai et al., 2011; McGovern et al., 2016). TID learning is also specific to the trained interval (Karmarkar & Buonomano, 2003; Meegan et al., 2000; Nagarajan et al., 1998). For instance, learning a 100-ms interval between two sounds may not transfer to a 50-ms or 200-ms interval. These results have been interpreted as challenges against a dedicated centralized clock that would function as a pacemaker and accumulator to track time (Creelman, 1962; Treisman, 1963), as the improved machinery of a centralized clock after training would predict modality- and interval-unspecific TID learning. Rather, the results are more consistent with the involvement of more peripheral timing mechanisms distributed over sensory modalities and brain areas (Ivry & Schlerf, 2008; Paton & Buonomano, 2018).
In two previous studies, we reported that TID learning can completely transfer across visual and auditory modalities in both directions (Xiong et al., 2022), as well as from one interval to another interval (Guan et al., 2024), through double training. Double training is a method we originally developed in visual perceptual learning research (Xiao et al., 2008; Xiong et al., 2016; Zhang et al., 2010). It involves a primary training task whose learning is typically specific to the trained condition. However, the learning transfer can be achieved when a secondary training task is practiced, through which the participants receive exposure to the transfer stimulus condition of the primary training task. For example, visual TID learning at a 100-ms interval, which is normally highly modality specific, can transfer completely to an auditory TID at the same 100-ms interval if an additional orthogonal task, such as tone frequency discrimination, is practiced at a 100-ms interval (Xiong et al., 2022). Similarly, auditory TID learning at a 100-ms interval becomes transferable to a 200-ms interval if tone frequency discrimination at 200 ms is also practiced (Guan et al., 2024). These results suggest that TID learning is likely the learning of an abstract and conceptual representation of subsecond time that is invariant to sensory modality and temporal interval (Guan et al., 2024; Xiong et al., 2022). This proposal is in alignment with our previous conclusions that visual, auditory, and tactile perceptual learning is mainly learning of sensory concepts (Hu et al., 2021; Wang et al., 2016; Xie & Yu, 2019; Xiong et al., 2019).
A temporal interval can be defined not only by unimodal cues but also by crossmodal ones, such as an interval comprising a visual stimulus followed by an auditory one or vice versa (V–A and A–V intervals, Figure 1). It is unclear whether and how crossmodal TID learning affects unimodal temporal judgment. Any putative distributed timing mechanisms detecting a crossmodal interval have to integrate inputs from two modalities, so that they are likely different from those responding to unimodal temporal intervals in certain aspects. If learning is based on putative crossmodal timing mechanisms, as crossmodal TID thresholds are higher than unimodal TID thresholds (Grondin & Rousseau, 1991; also see Figures 2 and 3), little learning transfer would be predicted from crossmodal timing mechanisms to unimodal ones due to the precision difference, similar to the reasoning that high-threshold visual TID learning cannot transfer to low-threshold auditory TID. However, if learning is based on an abstract representation of subsecond time, as we proposed previously, crossmodal TID learning should be transferable to unimodal TID. In this study, we tested these predictions in two experiments by examining the transfer of crossmodal TID learning to unimodal TID tasks.
Illustrations of crossmodal and unimodal TID trials and experimental designs. (a) Crossmodal TID training and learning transfer to visual TID. The auditory stimulus was a 15-ms tone pips, and the visual stimulus was a 15-ms Gabor patch. The standard interval was 200 ms long, and the comparison interval was 200 + Δt ms. In a given trial, the standard and comparison intervals were presented in random order with a 900 ms time gap. (b) Crossmodal TID training and learning transfer to auditory TID. (c) An illustration of experimental sessions.
Transfer of crossmodal auditory–visual or visual–auditory TID learning to unimodal visual TID. Upper panels: The average A–V and V–A TID learning curve and the average V–V TID thresholds before and after crossmodal TID training and with extra direct training (left), and a summary of individual and average percentage improvements (right). Lower panels: Individual data. Participants S1-1–S1-6 were involved in A–V TID training, and S1-7–S1-15 were involved in V–A TID training.
Transfer of crossmodal auditory–visual TID learning to unimodal auditory TID. Upper panels: The average A–V TID learning curve (left), the average A–A TID thresholds before and after A–V TID learning and with extra direct A–A TID training (middle), and a summary of individual and average percentage improvements (right). Lower panels: Individual data.
Methods
Participants and Apparatus
Thirty-one college students, including 9 males and 22 females, with an average age of 21.9 years, participated in this study. These participants had normal or corrected-to-normal vision and normal hearing, with pure-tone thresholds of 20 dB hearing level or less across 0.5–6 kHz. They had no prior exposure to visual psychophysical or psychoacoustic experiments and were unaware of the study's purpose. Prior to data collection, all participants provided informed consent. The research protocol received approval from the Peking University Institutional Review Board and was conducted in compliance with the Code of Ethics outlined by the World Medical Association's Declaration of Helsinki.
The experiments took place in a soundproof booth, and the stimuli were generated using a Matlab-based software Psychtoolbox-3 (Pelli, 1997). The participants were presented with auditory stimuli through Sennheiser HD-499 headphones, while visual stimuli were displayed on a 24.5-in. Acer XN253Q LCD monitor. The monitor had a resolution of 1,920 pixels × 1,080 pixels and a refresh rate of 240 Hz. The monitor was calibrated using an 8-bit look-up table, resulting in a mean luminance of 43.5 cd/m2. A chin-and-head rest was used to support the participant's head.
Stimuli and Procedures
The visual TID stimuli consisted of two 15-ms Gabor gratings separated by a 200-ms interval (V–V in Figure 1a). Each Gabor grating had a fixed orientation of 0°, a spatial frequency of 1 cycle/deg, and a contrast level of 100%. The auditory TID stimuli comprised two tone pips, each lasting for 15 ms, with a standard temporal interval of 200 ms between them (A–A in Figure 1b). The tones had a smooth 5-ms cosine ramp at both ends and were consistently set at a frequency of 1 kHz and a sound pressure level (SPL) of 86 dB. To create the crossmodal stimuli, a visual stimulus (the same 15-ms Gabor grating) was followed by an auditory stimulus (the same 15-ms tone pip) or vice versa (V–A or A–V in Figure 1). These crossmodal stimuli were also separated by a 200-ms interval. The length of the interval was the duration between the offset of the first stimulus and the onset of the second stimulus. The choice of a standard temporal interval of 200 ms was based on pilot tests, which revealed that participants were unable to accurately perform crossmodal TID at a 100-ms interval.
Data were collected using 2AFC trials throughout the experiments. Each forced-choice trial commenced with a visual fixation point at the center of the computer screen for 300 ms. Subsequently, two pairs of stimuli were displayed in a random order, with a 900-ms time gap between them. One of the pairs represented a standard interval of 200 ms, while the other pair represented a comparison interval of 200 ms + Δt (where Δt represented a variable duration). Participants were instructed to indicate whether the first or second pair of stimuli had a longer interval by pressing the corresponding left or right arrow key on a computer keyboard. Following each response, the screen would display a happy or sad cartoon face, feedbacking whether the response was correct or incorrect. A blank screen would then appear for a random period ranging from 500 to 1000 ms before the next trial started.
The TID thresholds were measured with a staircase procedure. In each staircase, the initial difference (Δt) was 100 ms for unimodal TID trials or 160 ms for crossmodal TID trials, which was updated following a three-down one-up rule for a converging accuracy of 79.4%. The step size was 0.05 log units. Each staircase ended after 60 trials. The threshold was calculated as the mean of the last six reversals or all reversals if fewer than six reversals were observed. The participants practiced 20 trials before starting the experiment formally.
Two experiments were conducted to investigate the crossmodal to unimodal transfer of TID learning. Experiment I assessed the transfer of crossmodal TID learning to visual TID with the temporal interval fixed at 200 ms (Figure 2), while Experiment II assessed the transfer of crossmodal TID learning to auditory TID, also at a 200 ms interval (Figure 3). Each experiment spanned 10 daily sessions (Figure 1c), with a maximum of 2 days between each session. An experiment consisted of a pretraining session on the first session (S1) and a posttraining session on the seventh session (S7), each containing 10 staircases, half for the crossmodal condition and half for the unimodal condition. The crossmodal and unimodal thresholds were measured in alternating staircases in an A–B–B–A… order for half the participants and an B–A–B–A… order for the other half. Between the pre- and post-training sessions, the participants would practice a crossmodal TID task for five sessions (S2–S6). In addition, the participants would practice the unimodal TID task for three extended training sessions (S8–S10). A training session lasted approximately 50 min and contained eight staircases.
Sample Size
The decision regarding the sample sizes was based on a previous TID learning study by Wright et al. (1997) that used similar stimuli (their Figure 4, 100 ms–1 kHz condition). For power analysis, we used the G*Power software. In our study, the examination of learning and transfer involved comparing pre- and post-training thresholds. The sample size for each group was thus determined using the t-tests family for the difference between two dependent measures (matched pairs). To attain 80% power at a significance level of p = .05 and consider a comparable effect size of Cohen's d = 1.34 as reported by Wright et al., a minimum sample size of seven participants was deemed necessary. The actual sample size for each experimental condition exceeded 7, which were 11 and 9 for crossmodal V–A and A–V training in Experiment I, respectively, and 11 for crossmodal A–V training in Experiment II.
Data Analysis
Improvement following the initial crossmodal TID training was assessed as the percentage change in threshold: (thresholdS1 − thresholdS7) / thresholdS1 × 100%, and improvement subsequent to the later extra unimodal training was calculated as the percentage change in threshold: (thresholdS7 − thresholdS10) / thresholdS7 × 100%. The learning and transfer effects were analyzed using the ‘lm’ function in R. A general linear model (GLM) was employed on each experimental dataset. This model combined three one-sample t-tests into one, thereby reducing Type-I errors associated with multiple comparisons. The model compared the percent improvement to 0.
Being primarily interested in the transfer of crossmodal TID learning to unimodal TID, we excluded two participants in Experiment I who did not show any crossmodal TID learning. In addition, a previous study reported that about one fourth of participants failed to display visual or auditory TID learning at a reference interval of 200 ms (Bueti et al., 2012). Therefore, we expected that in our study some participants might not display much unimodal TID improvement after crossmodal TID training and ensued unimodal TID training. Similar to the measures taken by Bueti et al. (2012), we excluded three participants in Experiment I and one participant in Experiment II who showed less than 5% unimodal TID threshold reduction at the end of the 10-session crossmodal and unimodal TID training. Data from the remaining 15 participants in Experiment I and 10 participants in Experiment II were analyzed.
Results
Experiment I: Crossmodal to Visual Transfer of TID Learning
In the first experiment, one group of participants (N = 20) practiced the crossmodal TID task with the temporal interval defined by either an auditory signal followed by a visual signal (A–V TID, N = 9) or a visual signal followed by an auditory signal (V–A TID, N = 11), and the learning transfer to the same 200-ms temporal interval defined by two visual signals (V–V TID) was examined. Five participants’ data were excluded from data analysis, either because the A–V/V–A TID performance failed to improve after training (two participants) or because the V–V TID performance improved by <5% after crossmodal TID training and ensued V–V TID training (see Methods). Moreover, the remaining 15 participants’ data in the A–V or V–A training conditions were analyzed together because the transfer results were similar (see below).
A GLM model analysis revealed significant improvements of crossmodal and unimodal V–V TID thresholds (F3, 42 = 26.496, p < .001, η2 = 0.654). More specifically, training reduced crossmodal TID thresholds by 40.26% ± 4.09% (t14 = 8.300, p = .000, Cohen's d = 4.436), from 68.69 ± 6.44 ms to 41.27 ± 5.00 ms (mean ± 1 standard error). The learning also transferred to untrained V–V TID significantly, reducing V–V TID thresholds by 15.45% ± 4.33% (t14 = 3.185, p = .003, Cohen's d = 1.702), from 30.21 ± 3.71 ms to 24.16 ± 2.27 ms (Figure 2). The participants were then asked to practice the V–V TID task for three additional sessions, which, however, did not lead to further significant reduction of V–V TID thresholds (improvements = 3.29% ± 5.93%; t14 = 0.678, p = .501, Cohen's d = 0.363), which included a 6.83% ± 12.67% reduction (t5 = 0.740, p = .471) with the A–V training subgroup and a 0.93% ± 5.82% reduction (t8 = 0.189, p = .852) with the V–A training subgroup. These results thus indicate a full transfer of crossmodal (A–V/V–A) TID learning to visual TID.
Experiment II: Crossmodal to Auditory Transfer of TID Learning
In the second experiment, a new group of participants (N = 11) practiced a crossmodal TID task. The crossmodal temporal interval was defined by an auditory signal followed by a visual signal (A–V TID), but not the other way around as two crossmodal interval conditions in Experiment I produced similar transfer results. This time the transfer of crossmodal TID learning to the same 200-ms temporal interval defined by two auditory signals (A–A TID) was examined. One participant's data were excluded from data analysis because the A–A TID performance was improved by <5% after the crossmodal TID training and ensued A–A TID training (see Methods).
A GLM model analysis suggested significant improvements of crossmodal and unimodal A–A TID thresholds (F3, 27 = 16.096, p < .001, η2 = 0.641). Training reduced A–V TID thresholds by 36.68% ± 5.44% (t9 = 5.536, p < .001, Cohen's d = 3.690), from 63.91 ± 6.63 ms to 39.21 ± 3.74 ms. The learning also transferred to the A–A TID thresholds significantly, reducing A–A TID thresholds by 27.83% ± 4.13% (t9 = 4.200, p < .001, Cohen's d = 2.800), from 12.87 ± 1.23 ms to 9.14 ± 0.93 ms (Figure 3). The participants were then asked to practice the A–A TID task for three additional sessions, which, like the V–V TID group, did not lead to further reduction of TID thresholds (improvements = 0.50% ± 9.22%; t9 = 0.077, p = .939, Cohen's d = 0.051). These results thus indicate a full transfer of crossmodal (A–V) TID learning to auditory TID.
Discussion
The experiments demonstrated that crossmodal TID learning can transfer to enhance unimodal visual and auditory TID performance. Furthermore, this learning transfer eliminates the need for additional unimodal TID training as the latter no longer affects unimodal TID thresholds. These findings suggest that crossmodal TID learning has fully transferred to unimodal TID tasks, maximizing the latter's performance.
In terms of thresholds, crossmodal A–V/V–A TID is about 2.5 times as high as the unimodal V–V TID (Figure 2) and 5 times as high as A–A TID (Figure 3). Even the posttraining crossmodal TID thresholds remain substantially higher than the pretraining thresholds of auditory and visual TID tasks. Therefore, learning cannot be directly mapped from crossmodal TID to unimodal TID, which excludes the possibility that the learning transfer is directly based on putative distributed timing mechanisms handling crossmodal time information. For the same reason of precision differences, enhanced attention or memory of temporal intervals resulting from crossmodal TID training cannot account for full crossmodal to unimodal learning transfer either. Furthermore, the transfer of learning cannot be attributed to the improvement of generalized decision-making strategies, as TID learning is task-specific and does not transfer to unrelated tasks like tone frequency discrimination (Xiong et al., 2022). Instead, it can be inferred that certain more fundamental knowledge of temporal information, irrespective of whether it is crossmodal or unimodal, is improved through the crossmodal TID training. This conclusion is consistent with, and thus offers further support to our proposition that TID training improves an abstract and conceptual representation of subsecond time (Guan et al., 2024; Xiong et al., 2022).
It is important to recognize that a conceptual representation of subsecond does not necessarily equate to a dedicated centralized clock. While a dedicated centralized clock measures exact time, a conceptual representation of time is incapable of doing so since the time information is abstracted. Therefore, time needs to be measured by specific timing mechanisms, which could involve a dedicated centralized clock, distributed timing mechanisms, or a combination of both (Wiener et al., 2011). Furthermore, our transfer results do not necessarily support the existence of a dedicated centralized clock. Training of a high-threshold temporal task, such as crossmodal TID, would not refine the machinery of the hypothesized centralized clock that is supposedly capable of handling more precise unimodal timing tasks, which is inconsistent with results from the current study.
Nonetheless, regardless of the debate on dedicated or distributed timing mechanisms, the accumulating evidence from our previous TID learning transfer studies (Guan et al., 2024; Xiong et al., 2022) and the current one strongly supports that subsecond time information is represented at an abstract and conceptual level. Moreover, training enhances the precision of this conceptual representation, so that TID learning can transfer across sensory modalities, specific temporal intervals, and from crossmodal to unimodal.
Footnotes
Acknowledgements
This research was supported by a National Science and Technology Innovation 2030 (STI2030) Major Project grant (2022ZD0204600), China.
Author Contribution(s)
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Science and Technology of the People's Republic of China (grant number 2022ZD0204600).
