The effect of harmonization on cortical magnetic responses evoked by music of rapidly changing tonalities

Methods

Stimuli

Auditory stimuli of piano tones were generated with Overture 4.0 (GenieSoft, Inc.) and edited with E-Prime 1.1 (Psychology Software Tools, Inc.). To explore the effects of harmonization on modulation perception, we used three harmonized melodies and three unharmonized melodies as stimuli. The unharmonized melodies were generated by removing the harmony from the harmonized melodies. All melodies were generated through sequential modulations of the single motive “do-mi-fa-fa” in a movable-do solmization. This motive was four beats (one bar) in length, with one note on each beat. Each beat lasted 0.8 s.

The stimuli were characterized by remote modulations that occurred regularly; three flats were added to the key signature for every four beats or one bar. Figure 1 shows the musical score of a harmonized melody used as stimuli. The motive “do-mi-fa-fa” appeared in the soprano voice of this four-part harmony. The progression I-vi-IV-ii^ø₇ is the generic unit of sequential modulations. These modulations are chromatic modulations since the submediant was lowered chromatically in the progression of IV-ii^ø₇, as highlighted by the grey rectangles in Figure 1. The half-diminished seventh ii^ø₇ in a major key is also the seventh chord on the leading tone vii^ø₇ of its relative minor key, which is resolved to the tonic chord in the major mode. The chord progression vii^ø₇-I established a new tonality. It should be noted that a rigid definition of modulation refers to a firmly established change of tonality. The rapidly changing tonalities in our stimuli were not firmly established and the chord progression vii^ø₇-I should be regarded as a means of tonicization. To address the phenomenon of tonality change in a controlled and feasible manner, we followed previous studies (e.g., Koelsch et al., 2003; Tillmann, Janata, & Bharucha, 2003) to distill this concept to its raw ingredients and used simplified stimuli. In this paper, the terms sequential modulation and chromatic modulation were used to specify how the tonalities were changing irrespective of the firm establishment of tonalities.

OPEN IN VIEWER

Figure 1.

Musical score of a harmonized stimulus with chromatic modulations. All melodies used as stimuli were generated through sequential modulations of the single motive “do-mi-fa-fa”, which appeared in the soprano voice of this four-part harmony. Three flats were added to the key signature for every four beats or one bar. The harmonic progression I-vi-IV-ii^ø₇ is the generic unit of sequential modulations. The submediant was lowered chromatically in the progression of IV-ii^ø₇ (highlighted by gray rectangles).

In the example shown in Figure 1, the motive “do-mi-fa-fa” started from C major, passing through E♭ major, G♭ major, A major, and back to C major to start another cycle of modulations. In addition to the harmonized melody presented in Figure 1, there was one harmonized melody passing through C# major, E major, G major, and B♭ major, and one harmonized melody passing through D major, F major, A♭ major, B major. Continuously ascending in minor thirds would eventually result in extremely high pitches that are not musically meaningful. If the pitch ascended to higher than C7, the motive “do-mi-fa-fa” was lowered by two octaves and then ascended again. This was done to avoid extremely high pitches in the stimuli. ERFs associated with unexpected large downward melodic leaps were discarded. Both the harmonized melodies and the unharmonized melodies underwent sequential modulations every four beats.

There were six runs in the MEG scan session, with one modulating melody used in one run. A modulating melody was 64 bars in length and thus underwent 64 modulations. The intensities of the stimuli were 60–70 dB SPL (sound-pressure level). Participants were instructed to covertly sing along with the modulating melody with the use of octave generalization; they freely transposed the melody by octaves for covert singing.

Procedures

Prior to the MEG scan the participants performed two tests to ensure that they had a good capacity for relative pitch. In the first test they were asked to sing an unfamiliar melody (B♭ major, 4 bars) in movable-do solmization after hearing it. If they could sing fluently, they were judged as possessors of relative pitch and could take the next test. All participants passed this singing test. In the second test the participants were asked to sing along with the harmonized modulating melody shown in Figure 1 in a movable-do solmization. This test was performed to judge their ability of tonal schema shifts. Two participants did not pass this singing test and were excluded from the MEG experiment.

The MEG experiment was performed in a dimly lit magnetically shielded room (MSR, Yokogawa Electric Corporation). Prior to MEG data acquisition, each participant’s head shape was digitized, and head position indicator coils were used to localize the position of the participant’s head inside the MEG helmet. A localizer scan was performed to verify the presence of identifiable MEG responses to 1 kHz and 250 Hz pure tones and to determine adequate head positioning inside the machine. Air-conducted sounds were delivered through silicon tubes.

The procedure of the MEG experiment is schematically illustrated in Figure 2. There were six runs with the unharmonized melody and the harmonized melody carried out alternately. The duration of one run was 0.8 s × 4 beats × 64 bars = 204.8 s. A 1-minute break was inserted between the runs. The total duration of the MEG experiment was approximately 26 minutes. Throughout the MEG scan, participants were instructed to listen to the stimuli and to covertly sing along with the well-learned melodies in movable-do solmization.

OPEN IN VIEWER

Figure 2.

Schematic illustration of experimental procedure. Abbreviations: UM, unharmonized melody; HM, harmonized melody.

Results

In response to modulating melodies, the auditory evoked N1m and P2m were evident in all participants. Figure 3 shows the grand-average waveforms recorded from all MEG channels and the topographies of N1m and P2m elicited by unharmonized tone “do” (the first beat of a modulating melody). These topographies suggest strong MEG signals over the temporoparietal regions, which were ROIs for signal analysis.

OPEN IN VIEWER

Figure 3.

The grand-average waveforms and topographies of the auditory evoked magnetic field elicited by unharmonized tone “do” (the first beat of a modulating melody). The bottom panel shows the superimposed waveforms recorded from all MEG channels, with the N1m and P2m components identified. The strongest positive peak and negative peak of N1m were recorded in the ROIs. The top panel shows the topographies of these two components. Abbreviations: L, left; R, right.

A three-way ANOVA was conducted on the factors of harmonization (texture), beat, and brain hemisphericity. The results for N1m revealed that the main effect of harmonization was significant, F(1, 13) = 7.42, p = .017. The interaction between harmony and beat was significant, F(3, 13) = 5.79, p = .01. The analysis of the simple main effect of beat was significant for HM, F(3, 13) =10.44, p < .001, but not for UM, F(3, 13) = .56, p = .65. The results also revealed that there were significant simple main effects of harmonization for the first beat, F(1, 13) = 10.65, p = .001, and the fourth beat, F(1, 13) = 22.26, p < .001. Figure 4 shows that harmonization reduced N1m for the first beat and the fourth beat. This finding suggests that the difficulty of covert singing the pitch names along with the modulating melodies was reduced by supporting harmony. Figure 4 also shows that the subdominant chord on the third beat elicited strongest N1m for HM.

OPEN IN VIEWER

Figure 4.

Amplitude of the N1m component at the left and right temporoparietal regions. Abbreviations: L, left; R, right; UM, unharmonized melody; HM, harmonized melody. Error bars represent standard error of means (SEM).

The results for P2m revealed significant main effects of harmony, F(1, 13) = 5.06, p = .043, beat, F(3, 13) = 11.51, p < .001, and hemisphericity, F(1, 13) = 7.19, p = .019. Figure 5 shows that harmonization enhanced P2m and the amplitude of this response was larger for the right temporoparietal region compared to the left temporoparietal region. P2m enhancement for HM suggests that harmonization facilitates pitch categorization.

OPEN IN VIEWER

Figure 5.

Amplitude of the P2m component at the left and right temporoparietal regions. Abbreviations: L, left; R, right; UM, unharmonized melody; HM: harmonized melody. Error bars represent standard error of means (SEM).

Discussion

In an attempt to investigate the influence of harmonization on the tonal processing of chromatic modulation, we recorded auditory-evoked magnetic fields in participants while they were covertly singing the pitch names along with the modulating melodies. There were two versions of the stimuli: the harmonized version and the unharmonized version. The stimuli were characterized by chromatic modulations that occurred regularly. We found that harmonization significantly reduced the N1m amplitude for the first and the fourth beats, on which the new tonality was established. Based on previous studies, this effect may be due to the directed attention for achieving the task of singing along with the unharmonized modulating melodies. For the harmonized melodies, the subdominant chord on the third beat elicited stronger N1m relative to other beats. Moreover, harmonization led to a significant enhancement of P2m, which may correlate with the ease of pitch categorization. For the harmonized version, P2m reached its minimum amplitude on the fourth beat, where a chromatic half-diminished seventh chord may deteriorate a quick processing of pitch categorization.

The N1/N1m component of the auditory-evoked field has been shown to exhibit sensitivity to a variety of acoustic attributes of sounds, such as pitch (Soeta & Nakagawa, 2008) and spectral attributes of vowels (Makela, Alku, & Tiitinen, 2003). The N1/N1m generators are located within or near the border of Heschl’s gyrus in the auditory cortex (Kuriki, Kanda, & Hirata, 2006; Okamoto, Stracke, Bermudez, & Pantev, 2011; Pantev et al., 1995). Over the past few decades, converging evidence has indicated that attentional effort affects the auditory-evoked N1m. Woldorff et al. (1993) recorded the neuromagnetic fields while participants listened selectively to sequences of rapidly presented tones in one ear and ignored tones of a different pitch in the opposite ear. Compared to unattended tones, attended tones evoked larger N1m. Ahveninen et al. (2011) further tested the modulatory effect of attention using noise-masked stimuli. It was found that noise masking reduced the N1m amplitude in response to unattended tones, and selective attention canceled out this attenuating effect.

The finding of N1m enhancement for the unharmonized modulating melodies may be interpreted in terms of increased demand of attention. In the absence of supporting harmony, the task of singing the pitch names along with the modulating melody was likely driven by top-down control and more attention-demanding. On the first and the fourth beats of our stimuli, the unharmonized melody elicited significantly stronger N1m compared to its harmonized version. As the rules for pitch-to-pitch-name associative transformation were changing on these two beats, the wrong rules may compete with the correct ones. The enhancement effect of N1m may reflect the auditory attention for suppressing the wrong rules. When the task of singing along with modulating melodies was achieved in the absence of harmony, the retrieved memory of irrelevant pairs of pitch and pitch name may have an effect similar to noise masking. This view is in line with the “early selection” theories, which suggest that auditory attention is explained by enhancement of relevant and suppression of irrelevant inputs (Woldorff et al., 1993). This view is also in line with the notion of tonal priming; compared to the unharmonized stimuli, the harmonized stimuli are more efficient in priming relevant tonalities and suppress irrelevant tonalities (Bharucha & Stoeckig, 1987; Tillmann et al., 2000; Tillmann, Janata, & Bharucha, 2003; Tillmann, Janata, Birk, et al., 2003).

Another noteworthy finding is that N1m amplitude varied significantly across different beats for harmonized modulating melodies. The subdominant chord on the third beat induced the strongest N1m, whereas the half-diminished seventh chord on the fourth beat induced the weakest N1m. Participants covertly sang the same pitch name “fa” for these two chords, and attention effort alone does not seem to explain the significant decrease of N1m amplitude in this chord progression. Please note that the subdominant chord on the third beat is identical to the tonic chord on the first beat except for their context and, to a lesser extent, the spacing of voices. The enhanced N1m response to the subdominant chord might be attributed to the harmonic context of chromatic modulations. A candidate explanation is that the emotional significance of different chords caused amplitude variation of N1m. Using emotionally conditioned auditory stimuli, Brockelmann and colleagues found increased N1m amplitude for emotion-associated stimuli relative to neutral stimuli (Brockelmann et al., 2013; Brockelmann et al., 2011). In Western music, tonal modulations are usually associated with characteristic emotional responses. For example, Ludwig van Beethoven (1770–1827) employed the more remote median and submediant keys as substitutes for dominants to create a dissonance of greater power and excitement (Rosen, 1972). In our view, the subdominant chord on the third beat of the modulating melody is a preparation for chromatic modulation and might be of high emotional salience. During repeated exposure to sequential modulations, participants may anticipate hearing the “beautiful” chromatic inflection of the submediant after this subdominant chord. As in some masterpieces of the Classical era, this chromatic inflection always occurs in the inner voice (Aldwell & Schachter, 2003). Therefore, its processing may be particularly attention-demanding. We thus speculate that N1m amplitude reached its maximum for the subdominant chord as the result of attentional bias to the anticipated chromatic inflection. Although plausible, this speculation awaits future research, and additional studies are needed to elaborate the issue of emotional salience for the melodic factors of chord progression.

While the amplitude N1/N1m tends to positively correlate with the level of selective or emotional attention, the amplitude of P2/P2m tends to negatively correlate with attention level (Michie, Solowij, Crawford, & Glue, 1993). It has been suggested that the enhancement effect of P2m may reflect inhibition of attention to repeated stimuli that have little significance (Sheehan, McArthur, & Bishop, 2005). In an MEG study on the effect of training and habituation on neural responses, Ross and Tremblay (2009) observed a decrease for N1m amplitude and an increase of P2m amplitude for the same repeated speech-sound stimuli. The amplitude of P2/P2m has been related to the processes of stimulus categorization and object identification. When the difficulty of auditory discrimination tasks increased, the amplitude of P2 decreased (Cranford et al., 2004). Moreover, Tong et al. (2009) reported that the P2 amplitude was enhanced by training, and this enhancement was strongly associated with the reaction time for discrimination. Recently, P2m sources were localized in the ventral auditory pathway for object identification (Ross et al., 2013). With respect to music perception, humans tend to categorize musical pitches into discrete entities and to label each categorized pitch a name. We propose that pitch categorization may have impacts on P2m amplitude.

Because the amplitude of P2/P2m is sensitive to stimulus experience and training, a few experiments were dedicated to the analysis of the effect of musical expertise on neuroplasticity. For example, musicians exhibited a P2 enhancement effect for the sound of their own instrument but not for pure tones (Shahin, Roberts, Pantev, Trainor, & Ross, 2005). In the present study, participants were able to sing along with modulating melodies and should be regarded as musically trained. We found significant enhancement of P2m amplitude by harmonization. This finding is in accord with the results by Kuriki et al. (2006), in which the P2m was significantly stronger for chords than single tones. Harmonization alone cannot explain the P2 enhancement effect in our study, because the P2m amplitude varied significantly across different beats. For the harmonized modulating melody, P2m was weakest on the fourth beat, where the tonal ambiguity of a half-diminished seventh chord may deteriorate pitch categorization. In our view, harmonization enhanced P2m because it facilitated pitch categorization through tonal cues. The effects of harmony and expertise on the neural activity in the auditory cortices were also reported by Brattico et al. (2009), who found that when two nonprototypical chords (a dissonant chord and a mistuned chord) appeared in a context of major chords, the change-related magnetic mismatch response was stronger in musicians than in nonmusicians. In addition, the inferior parietal lobules were recruited in musicians but not in nonmusicians during harmonic processing (Schmithorst & Holland, 2003). Our finding that the temporoparietal regions showed enhanced P2m for harmonized versus unharmonized melodies is in accordance with these previous studies.

A limitation of this initial MEG study on chromatic modulation was that the four chords used as stimuli were not matched in acoustic and harmonic respects. The chords on the first, second, and third beats were major and minor triads, whereas the modulating chord on the fourth beat was a seventh chord. A partial solution to this problem for future studies on remote modulations is to use progressions of seventh chords as in jazz music. Furthermore, the highly simplified stimuli could have established various keys, and it would be interesting to conduct a similar study in future with less tonally ambiguous stimuli. Another limitation was the gender bias in our participants (13 females and 4 males). Further research should examine whether male and female listeners differ in perceiving key changes.

Conclusions

In musical practice, remote modulations are sometimes accomplished by chromatic modulations. Our study provides a first attempt to delineate the neural correlates of the influence of harmonization on perceiving chromatic modulations. Harmonization reduces the N1m amplitude and enhances the P2m amplitude. These two components may reflect the level of top-down control and the ease of pitch categorization, respectively. These findings lend support to the notion that neural correlates of a shifting tonal schema are not restricted to the frontal cortices, but also observable in the auditory cortices. It remains to be investigated how the frontal cortices and the auditory cortices interact during musical modulations. Moreover, we found N1m amplitude changes around the chromatic inflection that characterizes the tonality change. Further investigations are needed to evaluate the cognitive-emotional effects of the melodic relations between diatonic and non-diatonic chords.