Musical expertise,musical style,and visual attention

Musicians’ visual expertise

Wong and Gauthier (2012) reported that crowding effects from irrelevant distractors for musical notation stimuli were smaller for expert musicians than for comparison groups. Importantly, for other visual stimuli, the crowding effects for musicians and non-musicians were similar. Waters, Underwood, and Findlay (1997) tested a comparison task of simultaneously presented stimuli (82 bars with 5 notes) where observers had to judge whether the two were the same or different. They found that experienced musicians were able to perform the comparisons with fewer and shorter fixations between the two stimuli. Waters et al. (1997) speculated that experienced musicians might have higher visual pattern recognition skills for these stimuli. Consistent with this, there is indeed a considerable amount of evidence showing that experienced musicians can recognize musical notation stimuli as meaningful chunks (Bean, 1938; Halpern & Bower, 1982; Salis, 1980; Sloboda, 1978; Soares, 2015) rather than as individual notes (Fasanaro, Spitaleri, Valiani, & Grossi, 1990; Wolf, 1976).

Other evidence suggests that expectations about likely continuations of musical phrases, stemming from learning of predictable patterns or logical continuations in music, can lead to errors in music reading. As an example, Sloboda (1976) reported that skilled pianists often make the so-called proofreader error, where they “correct” musical notation such that it becomes more consistent with what is likely within a particular context based on their prior expectancies. They may misread a discordant note as a harmonious one as in the so-called “Goldovsky experiment” where only an inexperienced music reader noticed a printing error of a single note in Brahms’ Capriccio (op. 76, no. 2) where a natural G was written within a C-sharp major triad instead of G sharp. Experienced music readers tended not to notice musical notation errors that went against expectations. As Sloboda (1978, p. 12) puts it,

good sight reading is based [. . . on determining the] probable continuations within an idiom. For conventional tonal music this implies a knowledge (implicit or explicit) of harmonic or rhythmic rules and an ability to translate this knowledge into the movements necessary to produce the appropriate sounds.

Since certain music combinations are more likely than others, Pinkerton (1956) suggested that musicians learn the transitional probabilities between notes or musical phrases. This seems analogous to well-known findings of visual statistical learning where observers come, over time, to implicitly expect a stimulus that has a higher probability of following a particular stimulus than others (Fiser & Aslin, 2001; Sigurdardottir et al., 2017; Turk-Browne, Jungé, & Scholl, 2005).

Waters, Townsend, and Underwood (1998) speculated that sight reading of written music relied on perceptual skills such as pattern recognition, prediction, and the ability to generate auditory representations. Among their key findings from tests on expert pianists was that perceptual recognition from briefly presented musical patterns was strongly affected by observers’ sight-reading skills and that processing groups of notes involves more than efficiency at processing single notes, suggesting that efficient grouping of notes is a key skill for reading music. Relatedly, they found that skilled sight readers of musical notation were able to encode and use larger chunks when comparing two musical notation patterns than less skilled readers. This suggests that there may be differences in the processing levels where this information is represented, reminiscent of the differences in category levels discussed above.

Working memory may also play an important role in expertise as experienced musicians may have more efficient working memory than novices. Hansen, Wallentin, and Vuust (2012) found that experienced musicians had better verbal working memory (measured with the traditional forward digit span task) than novices but found no significant differences in visual-spatial memory. Their findings were further supported by a meta-analysis conducted on 29 studies by Talamini, Altoè, Carretti, and Grassi (2017) who reported that the verbal working memory of experienced musicians was on average moderately better than of novices. But again, no differences in visual-spatial working memory were found.

In brain imaging experiments, a general finding regarding expertise is that the brain shows considerable reorganization abilities following training (see, for example, Kristjánsson et al., 2016). Findings of neural mechanisms in relation to music expertise show that there is great flexibility and competition between functions for space within the brain and that music expertise affects several neuronal networks (Jäncke, 2009), even changing the function of visual occipitotemporal areas (Mongelli et al., 2017). Music training may displace word-selective areas (Mongelli et al., 2017), affect connectivity, and can cause expansion of areas selective for certain stimulus types. Generally, expertise seems to entail the emergence of both novel specialization and competition between classes of stimuli (de Beeck & Baker, 2010; Kristjánsson et al., 2016; Sigurdardottir et al., 2015).

The network involved in music reading seems extensive (Wong & Gauthier, 2010), including the visual cortex (Mongelli et al., 2017) and the fusiform gyrus (Muayqil, Davies-Thompson, & Barton, 2015). Overall, these findings agree with findings on expertise of various sorts, revealing that the functional architecture in the brain is strongly modulated by musical expertise. Despite all this, it is not yet quite clear how attentional function is affected by musical expertise. Our aim here is to study this using the well-known change blindness paradigm.

Change blindness

The literature on visual attention clearly shows that human observers are quite insensitive to large changes in their visual environment, especially when attending to something else in the visual scene, even when their gaze is centered on the changing location (Kristjánsson, 2006a; Most, Scholl, Clifford, & Simons, 2005). A large amount of salient visual information is not noticed if it is not explicitly attended. Apparently, we process much less information from a scene than we think. But do such attention effects differ between novices and those with visual experience in a particular domain, in our case written music?

Some initial change blindness studies involved tests where changes are introduced while observers make saccades. Saccades are very fast eye movements from one point to another (see, for example, Leigh & Zee, 2006). McConkie and Zola (1979) showed that if observers make an eye movement to a different location in a complex visual scene, they are surprisingly insensitive to changes made to the scene during the eye movement. Similarly, Grimes (1996) showed that when observers viewed natural scenes and were instructed to remember details for a subsequent test, they often missed large changes across saccades. If a change happens while the eye is not moving, it may induce transient visual signals that cue its location. But during a saccade, the visual perception of the environment is blocked, eliminating these clues to the changes. This is why it is important to present a blank screen between the original scene and the changed scene when studying change blindness. Neisser and Becklen (1975) showed that observers instructed to count passes between basketball players missed surprisingly salient events (such as a man changing into a woman). The so-called change blindness paradigm that we use was introduced by Rensink et al. (1997). They found that it takes observers a surprisingly long time to notice dramatic changes between two otherwise identical pictures of a visual scene if a picture of a changed scene is alternatively presented with the original picture at a rapid rate or if a visual event (such as a blank field or mudsplashes) appears between the two views while the changes take place (O’Regan, Rensink & Clark, 1999). Importantly, this was not confined to paradigms involving saccades.

But what happens if the scene involves something that we are highly familiar with—for example, discernible makes and models of cars or different types of birds for car and bird experts, respectively? Werner and Thies (2000) tested experts in American football on change blindness, using images relevant to their field of expertise. They found that expertise increased change detection rates for images related to their expertise domain. Reingold, Charness, Pomplun, and Stampe (2001) tested expert chess players, intermediate chess players, and novices, finding that experts were faster than the two other groups at noticing changes between two images of a given chess position if the pieces were arranged in meaningful arrangements that could occur in a chess game. However, if the pieces were randomly distributed with complete disregard for chess rules, the experts performed no better than novices on the task. The visual span of the experts was also larger than that of intermediate players and novices when they viewed realistic chess positions but not random layouts. Moreover, the experts made fewer eye movements when scanning the real chess positions and were more likely to fixate between the pieces rather than directly on individual pieces—suggesting that they picked up arrangement patterns of the chess pieces that are dependent on context rather than the absolute position of the individual pieces (Reingold et al., 2001). Notably, this is reminiscent of the findings of eye fixation patterns of music experts discussed above (Waters et al., 1997; for review, see Madell & Hebert, 2008). Furthermore, a recent unpublished change detection study by Kleinsmith and Sheridan (2019) revealed that when music scores were rotated by 90°, the trained musicians’ performance advantage disappeared. Finally, it is interesting to note that change detection for melodies delivered auditorily has been found to rely on schematic processing and that trained musicians relied more on the schematic gist of the melody than non-experts (Agres, 2019).

Change blindness is well suited to research into the interplay between attention and memory, more specifically visual working memory (VWM; Kristjánsson, 2006a; Simons & Rensink, 2005). It is also an efficient tool for measuring VWM capacity (Kristjánsson, 2006b; Luck & Vogel, 1997; see Simons, 2000). As explained above, in the change blindness paradigm, a figure is shown, followed by a blank screen and a second, slightly changed figure. This process is repeated several times until the observer either detects the change or the allowed time limit is reached. This may enable assessment of how working memory performance is affected by experience. Might observers experienced with reading music, for example, be at an advantage if they easily organize the written music into chunks?

Current aims

There is no consensus on how to define musical expertise. In the literature, the focus has principally been on performance (see, for example, Ericsson et al., 1993). However, reaching the level of expert performer inevitably requires mastery over a wide array of different skills (Mishra, 2019). Our aim was to focus on music reading. No instrumental playing took place in our experiment, so it did not involve what is normally called sight reading, that is, the reading of previously unseen material with simultaneous performance (also called prima vista). Rather, our design involved silent reading, a topic which has not been widely investigated in the literature (Penttinen, 2013, p. 11).

The reading of music is a highly trained skill, not dissimilar to chess in that the arrangement of visual items has a particular meaning for those with training, whereas to novices they seem rather meaningless. But music reading can also involve patterns that are unconventional although they still correspond to music. Contemporary classical music, for instance, may have highly unusual and complex structure, differing significantly from the classical repertoire. In some cases, contemporary music carries no patterns at all, or they are very difficult to define (Aiello, 2000; Williamon, 1999). In our study, the stimuli were music staves containing examples of traditional and contemporary music, and a third category with random notes. The criteria we used were that the traditional material should be in the key of C-major and demonstrating simple, regular rhythmic patterns with low diversity of note durations, whereas the contemporary examples would be atonal, from the chromatic scale, and demonstrating higher rhythmic complexity and diversity in note durations. Finally, the random stimuli did not correspond to any rational method of composition and are best described as musical nonsense. We used the change blindness flicker paradigm to compare the performance of experienced musicians and novices in detecting small changes made to examples of musical notation corresponding to the above criteria.

We expected experienced musicians to detect changes faster than the novices and that the difference between the groups would be larger for traditional than contemporary music, while change detection would be similar for random “music.” This would be in line with findings on other domains of expertise discussed above, but the novelty of our study consists in adding music to the domains already studied. This is part of an endeavor to understand the performance of experienced music readers on various cognitive and perceptual tasks, and how generalizable their expertise is. Is it truly task-bound, or does it also apply to other situations, including random notes that do not carry musical meaning?

Method

Stimuli

The stimuli were monophonic, written in the treble clef (G-clef) in three different musical styles: (1) traditional: in the key of C-major, no accidentals, regular rhythms in 4/4; (2) contemporary: atonal, using the chromatic scale, irregular rhythms, with or without bar marks and/or time signatures; and (3) random (nonsense). The stimuli were produced by manually playing short note sequences on the MIDI keyboard controller into CALMUS, with the software taking over the composition process, but always according to instructions from the first author. To allow for meaningful comparison between styles, for the traditional and the contemporary stimuli, three comparable patterns for input cells were used: (1) ascending scales (for the traditional: diatonic; for the contemporary: chromatic); (2) descending intervals (for the traditional: minor thirds; for the contemporary: differing descending intervals); and (3) repeated notes. For the random stimuli, the first author (who is not a keyboard player) hit the keys with eyes closed in an unmusical fashion as possible (see Supplemental Appendix B for examples of stimuli). The software then secured the randomness of the output. Scores, approximately 12 bars of length, were generated for each input, from which three examples of varying lengths for each musical style were selected: (a) one bar; (b) one line (containing three to four bars, or equivalent); and (c) two lines (containing six to eight bars, or equivalent; Figure 1).

OPEN IN VIEWER

Figure 1.

Examples of Stimuli and the Experimental Procedure.

Complexity levels were controlled and matched across conditions, using three criteria: (1) total visual information load (i.e., the sum of visual elements contained in each stimulus); (2) diversity of the visual elements (durations, types of accidentals, types of rests, etc.); and (3) musical, or semantic, complexity (tonality, predictability in compositional development, pitch range, shape of melody, etc.). For example, the traditional and the contemporary stimuli contained on average the same number of notes. However, the contemporary examples had various accidentals and much more variety in rhythmical notation (rests, dotted notes, triplets, etc.). Their musical complexity derives partly from the latter, as well as the fact that being atonal, using the chromatic scale, there is a larger number of notes to draw from. Thus, the contemporary material is less predictable and offers less expectation about likely continuations (see above). It may therefore be harder to build “chunks.” Finally, the random stimuli are the most varied, ranging from very simple materials to highly complex ones (see Supplemental information for the whole stimulus set).

The total number of music examples was 27, and the total number of stimuli produced was therefore 54, one “original” and one with a change. Changes were introduced in only one place for each example, using the notational software Sibelius. In all cases, the pitch of a note was either moved up or down by a semitone. This was done to reduce the number of variables involved in the design. The locations of the stimuli were adjusted so that their horizontal and vertical center was always at the center of the screen.

Results

A three-way repeated-measures ANOVA (rmANOVA) with country (Iceland/England), group (experienced musicians/novices), and music types (contemporary/traditional/random) revealed no significant differences between those who participated in Iceland and England (all ps for main effects and interactions >.11), so we combined the data from the two testing sites.

Before analyses, we removed trials where DTs deviated more than 3 SDs from each participant’s mean within each condition and where DTs were shorter than 75 ms. The total number of removed trials, including those with no or incorrect responses, was 111 or 11.1% of the data. After removing outliers and trials with no responses, the average DT was 12.8 s (SD = 11.0 s). The average correct detection rate was 87.8%. When participants correctly detected the changes, they needed 11.4 repetitions (SD = 9.3 repetitions) on average to detect the changes. Overall, the experienced musicians needed 10.6 s to detect the changes, whereas the novices needed 16.0 s. This difference was significant, F(1, 32) = 14.0, p < .001. The experienced musicians correctly detected the changes on 90.3% of the trials, but the novices correctly detected the changes on 80.7% of the trials and this difference was significant, F(1, 32) = 4.8, p = .036. Figure 2 shows DTs and accuracy for the different groups as a function of music type.

OPEN IN VIEWER

Figure 2.

Detection Times.

Change detection by music type (accuracy)

The experienced musicians correctly detected changes in the traditional music on 90.0% (SD = 30.1%) of trials, but the novices correctly detected changes on 87.4% (SD = 33.3%) of trials. The difference between the groups was small (2.4) and not significant, adjusted p = .990. For contemporary music, the experienced musicians correctly detected changes on 90.2% (SD = 29.8%) of trials, whereas the novices correctly detected changes on 70.8% (SD = 45.6%) of trials. The difference (19.4) was significant, adjusted p = .003. The average correct detection of changes in the random “music” condition was 91.0% for experienced musicians (SD = 28.7%) but 85.5% for novices (SD = 35.4%). As for traditional music, the difference (5.5) was not significant, adjusted p = .879. The main effects of group and music type were significant, F(1, 32) = 4.82, p = .036, and, F(2, 64) = 5.70, p = .005, respectively. However, after adjusting the p-values, no significant effects of music types remained (all ps > .13) Furthermore, the interaction between the factors was significant, F(2, 64) = 7.36, p = .001 (see Figure 2(b)). We should note that ceiling effects may have occurred (see Figure 4) because the 60 s that observers had to finish the task was seemingly a rather long time (the grand average DT was 12.8 s), and shorter times might have revealed larger differences especially considering the large differences in DTs (the overall median correct detection rate was 100%). It is nevertheless important to highlight that the detection rates for contemporary music differed by 20 percentage points between the experienced musicians and novices.

Figures 3 and 4 then show box plots for the experienced musicians versus novices for DTs and correct detection, respectively. The distributions of DTs were generally narrower for experienced musicians than for novices, with the largest difference for contemporary music. Median DTs were also lower for experienced musicians than for novices. The effect of expertise (training) for the contemporary music is particularly notable, both for accuracy and for the DT distribution width.

OPEN IN VIEWER

Figure 3.

Distribution of Individuals’ Average Detection Times as a Function of Group and Music Type. The diamonds represent the grand mean within each group and each condition.

OPEN IN VIEWER

Figure 4.

Distributions of Individuals’ Average Accuracy as a Function of Group (Experienced Musicians/Novices) and Music Type.

Discussion

Experienced musicians were significantly faster at finding the changes in contemporary and traditional music scores, whereas there was only a nonsignificant numerical difference for random scores. Overall, this is in accordance with our hypothesis, as well as results from other expertise domains such as chess (Reingold et al., 2001) and American football (Werner & Thies, 2000). The difference between experienced musicians and novices was particularly notable for contemporary music (see Figures 2 to 4). Consistent with this, the correct detection rate was only significantly higher for experienced musicians than for novices in the contemporary condition. In the other two conditions, the correct detection ratio was numerically higher for experienced musicians than for novices, but this was not statistically significant. There is reason to believe that the DTs involve a better measure than the correct detection rates because the maximum time for detection was 60 s, but the average DT was 12.8 s (SD = 11.0 s), potentially causing ceiling effects. Furthermore, the median detection percentage was 100% for both groups and all musical conditions, although means were lower.

The patterns in Figure 2 are interesting because the performance of the experienced musicians is quite stable for both DTs (panel a) and accuracy (panel b), whereas there are relatively large variations for the novices (see also Figures 3 and 4). From this—and the fact that experienced musicians detected the changes more quickly—we can conclude that experienced musicians are better at detecting changes in written music than novices. Interestingly, the largest group difference involved contemporary music. This suggests that the general public has considerable experience with visually presented traditional music but that experienced musicians are highly efficient at encoding music, even when it is contemporary and complex—in other words, a clear effect of visual expertise.

General discussion

Experienced musicians were better than novices at detecting changes in musical notation. Importantly, this applied especially to stimuli containing actual music because the difference was not significant for random stimuli. We suggest that experienced musicians can, overall, process visually presented music stimuli more quickly even when the task involves detecting a simple visual change that does not strongly alter the musical content (such as moving a single pitch up or down a semitone). Our results are similar to what has been found for chess experts (Reingold et al., 2001; Waters et al., 1997), but we are the first to show such effects for musicians’ visual expertise. An important analogy in our experiment is that when the musical stimuli are random, the advantage of the experienced musicians is strongly diminished, becoming nonsignificant, just as when the individual chess pieces are randomly distributed so that no meaningful pattern exists, the chess players’ advantage disappears. The hints of a difference between experienced musicians and novices in our random condition might reflect a crucial difference between the domains in that even when random, the stimuli can be read as some form of meaningful music. While the random arrangement of pieces on a chessboard can violate the basic rules, such as when a white pawn is found in Row 1 (an impossible chess position), it may be more difficult to devise a truly random stimulus that experienced musicians will not treat as some form of music. Nevertheless, in our study, they performed far more similarly to the novices for those stimuli than actual music stimuli.

Why do our experienced musicians perform better? Might their grouping and chunking mechanisms allow them to quickly organize the musical notation stimuli perceptually? This has been the favored explanation for the results of chess players. Note-for-note musical reading is almost certainly impossible for efficient music reading, which requires chunking similar to when we read whole words rather than single letters. This would allow grouping, especially of regular and familiar note patterns. Since the contemporary material is more complex, and therefore more difficult to group (or chunk), we expected to see the largest differences between experienced musicians and novices for the traditional stimuli.

Expertise with particular tasks or stimuli can affect how observers process and remember expertise-related stimuli (as in the use of categories, discussed in the Introduction). The exceptional cognitive abilities of experts on various performance domains have been demonstrated repeatedly (see, for example, Ericsson & Kintsch, 1995; Ericsson, Patel, & Kintsch, 2000). Expert chess players can, for instance, remember chess positions in far more detail than novices. De Groot’s (1946/1978) landmark experiment showed chess masters’ amazing ability to accurately recall positions on chess boards after only a few seconds of exposure. However, Chase and Simon (1973a, 1973b), demonstrated that this was only true if the chess positions corresponded to actual chess positions, not if the chess pieces were in nonsensical arrangements. This is consistent with results from the change blindness studies discussed above. These results have been replicated and confirmed across other domains such as electronic circuitry (Egan & Schwartz, 1979), computer programming (Ehrlich & Soloway, 1984), and music (Sloboda, 1976). Chase and Simon (1973a, 1973b) concluded that since chess masters were no better than novices in recalling random positions on chess boards, their performance advantage relied on their ability to organize material into meaningful chunks encoded at early perceptual stages. Their skill was then explained by the way they use these organized perceptual chunks to access chess board structures stored in long-term memory, and when combined with advanced problem-solving skills, they could quickly search for, and find, the best moves (see also Newell & Simon, 1972).

Consistent with this, performance on traditional music—where meaningful chunks are presumably most likely to appear in musical scores—was better here for experienced musicians than for novices, and the difference was larger than for the random stimuli. But our results deviate from this because the largest differences were found for the less conventional and notably more complex contemporary music. This was unexpected and suggests that any notes on a musical stave may be grouped into “meaningful” music, and also that musical notation is probably somewhat familiar to at least some within our novice group. Hence, we posit that expertise seems to help with the more unfamiliar and complex patterns, which in our case nevertheless obeyed fundamental rules of music notation. Furthermore, research on working memory has shown that experts are even able to chunk items in random layouts more efficiently than novices (Sala & Gobet, 2017). However, since no direct measures of working memory were used in our study, we cannot address this issue. But in light of the literature (Hansen et al., 2012; Sala & Gobet, 2017; Talamini et al., 2017), we speculate that better working memory for the experienced musicians, may have contributed to their performance.

Furthermore, given the differences in complexity between musical styles, we expected shorter DTs within groups for the traditional music than the contemporary examples. Here, however, our results for experienced musicians were also unexpected because they took numerically less time to detect changes for the more complex material. This suggests that there is ground for studying effects of complexity of musical material per se on attentional functioning and VWM in more detail. We will address this in future experiments.

Note also that eye movement patterns of expert chess players differ from those of novices when they view actual chess positions (Reingold et al., 2001; Waters et al., 1997). It will therefore be interesting to measure differences in eye movement patterns between music experts and novices when searching for changes in the staves. Experiments intended to answer this question are currently under way in our lab.

It is important to keep in mind that to minimize the number of variables, our stimuli deliberately contained only changes of one sort (either up or down by a semitone). Changing other elements of the musical notation might have brought other results. Also, although there were notable differences between styles, more exaggerated differences might have increased performance differences between groups (e.g., more irregular rhythmic and melodic patterns for the contemporary and random stimuli and simpler traditional examples).

Conclusions and implications

We believe that studying musicians’ visual performance for musical notation is of importance for the domain, given that classical musicians’ first contact with the music they have to study, rehearse, memorize, and perform is almost without exception through musical notation. How they navigate visually through this material, what draws their attention, and how this feeds into their VWM processes are key factors.

If searching through materials of differing complexity as our participants do here can be seen as a problem-solving task, our results on attentional function and VWM may help design future experiments aimed at casting a light on expertise and working memory capacity.

The change blindness flicker paradigm has been shown to be an effective tool to study these topics. We used this paradigm to investigate how expertise and the style of visually presented musical material (traditional/contemporary/random) affect the performance of experienced musicians compared with novices. Our experienced musicians generally outperformed the novices, supporting earlier findings suggesting that how we apply attention across the visual field relies strongly on observers’ experiences within that context. However, for the random conditions, the difference between the two groups was nonsignificant, in line with existing research on expertise, such as in chess. Interestingly, the largest difference between experienced musicians and novices was seen for the most complex examples (i.e., contemporary). Investigating the effect of both visual and semantic complexity on attention and memory for music will be of interest in future studies.

Supplemental Material