Abstract
Learning to write music in the staff notation used in Western classical music is part of a musician’s training. However, writing music by hand is rarely taught formally, and many musicians are not aware of the characteristics of their musical handwriting. As with any symbolic expression, musical handwriting is related to the underlying cognition of the musical structures being depicted. Trained musicians read, think, and play music with high-level structures in mind. It seems natural that they would also write music by hand with these structures in mind. Moreover, improving our understanding of handwriting may help to improve both optical music recognition and music notation and composition interfaces. We investigated associations between music training and experience, and the way people write music by hand. We made video recordings of participants’ hands while they were copying or freely writing music, and analysed the sequence in which they wrote the elements contained in the musical score. The results confirmed experienced musicians wrote faster than beginners, were more likely to write chords from bottom to top, and they tended to write the note heads first, in a flowing fashion, and only afterwards use stems and beams to emphasize grouping, and add expressive markings.
Writing text and transcribing music into notation are very personal ways to record content using symbols that can later be read, understood, and, in the case of music, played. Learning to write music in the staff notation used in Western classical music is traditionally part of a musician’s training. However, there are few texts formalizing how musicians should actually write music by hand—such as the books by Archibald Jacob (1947), Clinton Roemer (1974), and the Norton Manual of Music by George Heussenstamm (1987)—and most musicians are not aware of them. Most of the people who write music by hand, even professional musicians, are not professional music copyists. Here we explore the idea that musical handwriting is related not only to a particular graphical style and the writer’s motor ability, but also to the writer’s music education and mental representations of musical structures. To our knowledge, there has been no study to date of the relationship between music handwriting and the perception of musical structures.
Analysis of handwritten music
Analyzing handwritten music is one of the goals of optical music recognition (OMR; Calvo-Zaragoza et al., 2020). Tasks involving handwritten music in OMR research include writer identification (Gordo et al., 2013); staff removal (Dalitz et al., 2008), a priority for OMR (Fornés & Sánchez, 2014; Rebelo et al., 2012); and the comparison of several scores recently proposed by Riba et al. (2017). Studies of handwritten music are now based on large corpora such as the CVC-MUSCIMA database (Fornés et al., 2012), which consists of 20 pages of music reproduced in handwriting by 50 different people, 1 and the HOMUS dataset, comprising 32 musical symbols written by 100 different musicians (Calvo-Zaragoza & Oncina, 2014). 2
Structure and music perception
Several studies have focused on the roles of musical elements in reading, perceiving, and remembering music, such as melody coherence (Halpern & Bower, 1982), patterns (Waters et al., 1997), phrase units (Sloboda, 1977), and tonal structures (Krumhansl, 1991). The eye movements of skilled musicians while sight reading were investigated by Sloboda (1984), and Goolsby (1994) showed that trained musicians look ahead in the score and then back again to the location of the music they are playing. They use shorter and more efficient fixations on patterns of several notes at a time (see review by Lehmann & McArthur, 2002). Lehmann and Ericsson (1996) found that efficient sight readers could infer missing notes and correct errors in the score. Furthermore, Drai-Zerbib et al. (2011) showed trained musicians look back at the score less often and for shorter durations, and may perceive and retrieve music information regardless of the modality (visual or auditory) in which it was perceived.
How does music training influence music handwriting?
Our point of departure is that trained musicians write music in the same way they read, think, and model music, possibly using grouping structures such as the ones proposed by Deutsch (1982). Janzen et al. (2014) studied the effect of music training on continuous, timed movements while performing. Writing music is likely to involve similar thought processes and even body movements; some musicians sing or hum, or imagine themselves to be playing. George (2003) studied the recognition of musical symbols on the basis of handwritten input, but, to our knowledge, no-one has studied the dynamics of handwriting music. We define dynamics in this context as the movements of the hand and arm that determine how musical symbols are written, in which order, and at what speed.
Only a few musicians write music regularly and, in this age of digital tools, rarely, if ever, by hand. One could thus wonder whether it is worth studying music handwriting. However, many musicians learned to write music by hand early in their training, and the influence of that learning may play a role in their notational choices throughout their entire musical life. Indeed, musicians may naturally tend to want to reproduce some of their handwriting habits when they use notation software to write music. Moreover, studying the natural (or musical) way of writing music could improve OMR. Calvo-Zaragoza et al. (2020), for example, consider OMR “in terms of inverting [the] process” in which music is laid down as a “structured assembly of notes (. . .) embodied in a medium such as paper” (p. 5). In this process, the dynamics of music handwriting influence the static appearance of the completed score. The order in which elements are notated may affect their position, spacing, orientation, and even shape.
The study of music handwriting could inform the design of music notation and composition interfaces for both personal computers and mobile devices. Recent commercial applications such as NotateMe and StaffPad claim to provide an easy way of handwriting music using a touchscreen interface. However, it seems few experienced musicians use these applications or, indeed, those that have been developed specifically for research projects, and in fact no participant in our study reported the use of touchscreen notation software. Despite recent improvements, the programs available often lack the flexibility and efficiency of pen on paper. Even if they make use of novel gestures (such as swiping the touchscreen of a mobile device), it would benefit their design if their developers had a better understanding of the habits of those who write music by hand.
In the present study, we looked at the influence of music training on simple tasks related to handwritten Western music notation. Handwriting includes many components such as hand movements, variations in the shape of musical symbols, choice of orientation, and positioning of symbols. We focused on the sequence in which the symbols were written, as this can be observed in video recordings and transcribed unambiguously using a coding system of mutually exclusive categories. It is a simple variable to investigate empirically but is also potentially useful for shedding light on musicians’ cognition of musical structures while writing.
Method
Design
We examined associations between music training and the dynamics of writing music by hand, operationalized as the sequence. Specifically, we examined the effects of one independent variable (music education) on several dependent variables describing the order in which the elements of a musical score are written: bar lines, note heads, accidentals, stems, beams, expressive markings, and dynamic markings. We also compared the time taken by beginners and experienced musicians to carry out a series of notation tasks.
Participants
The study was carried out in the Hauts-de-France region, at the Regional Conservatory of Music (Conservatoire à Rayonnement Régional) in Amiens and in La Plaine Images in Tourcoing, a non-music education workplace. Participants were recruited via advertisements in these places. No compensation for their participation was offered.
A total of 24 individuals gave their written informed consent to take part in the experiment. Their participation was voluntary. Research was conducted in accordance with the Singapore statement on research integrity, but no further ethical approval was sought. Six participants were teachers at the conservatory, the others were with or without a musical background from the conservatory or elsewhere. We video recorded each participant as they were carrying out the notation tasks detailed below. The participants used the same pen (a STABILO Point 88, 0.4 mm line width with red ink) and the same type of music paper (A4, landscape orientation, four staves). Two participants were excluded because they were left handed, and a third was excluded because the video recording was inadequate for analysis.
Table 1 presents the sample, which included 21 right-handed participants (16 males, five females, with an average age of 30.6 years, SD = 11.1). Eight were categorized as beginners, reported no music training (including experience), or a maximum of 3 years’ training (average age of 27.6 years, SD = 4.6). The other 13 participants were categorized as experienced musicians, and reported a minimum of 8 years of training (average age of 32.5 years, SD = 13.3). No participants reported having between 4 and 7 years of music training. Five of the 13 experienced musicians described themselves as professional musicians. No beginners were used to writing music by hand and none wrote music using notation software, but most of the experienced musicians reported sometimes or regularly writing music, and roughly half also used notation software.
Data on participants, including self-reported habits on notation tasks. These data were collected before the experiment. Participants may have reported playing none, one, or several instruments. Question marks indicate missing data.
Procedure: Tasks
The participants carried out five tasks in which it would be possible to observe the order in which they notated particular elements. The first four tasks involved copying short passages of tonal music, shown in Figure 1. In each case, the participant was instructed to “copy this passage in such a way that it can be read by another musician (Merci de recopier cet extrait comme si vous prépariez la partition pour qu’elle soit lue par un autre musicien).”
Passages of music to be copied in each task: (1) Frère Jacques (chorus); (2) J’ai du bon tabac (chorus); (3) passage inspired by Pachelbel’s Canon with added expressive and dynamic markings; (4) a tonally plausible sequence of chords.
The short passages to be copied in Tasks 1 and 2 were simple melodies with key signatures, time signatures, bar lines, and note heads, with upwards and downwards stems on the left-hand and right-hand side of the note head as appropriate. Task 1 was intended to orient the participant and was not recorded as part of the data collection. The recording of Tasks 2, 3, and 4 were intended to test the order in which participants notated each element. The passage to be copied in Task 3 was a melody with expressive and dynamic markings. The passage to be copied in Task 4 consisted only of chords and a single accidental. Task 5 was optional; participants were invited to carry out free writing by notating a short piece of music, either from memory or improvised for the purpose of the study.
Procedure: Video recording, processing, and coding into sequences of symbols
The video recordings captured the hands of the participants from two points of view, the front and the side, with two Sony HDR-PJ420VE recorders. We thus had 21 sets of recordings of Tasks 2, 3, and 4 carried out by the right-handed participants. A further 13 recordings were obtained from the participants who carried out the optional Task 5. The researchers observed the participants while they were carrying out the tasks and made notes representing their subjective impressions, but these are not reported in this article.
The 34 videos were then coded using sequences of symbols describing the sequence in which the participant wrote each element (see Figure 2). The coding system was as follows: clef (c), sharp (#), flat (b), time signature (T), bar line (measure) (M), final bar (F), crotchet (quarter note) note head (n), minim (whole note) note head (O), upward stem (i), downward stem (!), beam (_), semiquaver (sixteenth note) beams (=), slurs (ˆ), dynamic markings (p, <, >), and rest (s). The positions of the notes of the chords in Task 4 are numbered from 1 (lowest) to 3 or 4 (highest). The boxed symbols show the location on the video recording. These sequences of symbols are available, as open-source data, at http://www.algomus.fr/handwriting.
Still images captured from the video recording. Top left, Participant 6. Top right, Participant 21. Bottom, duration of each task and sequence of symbols representing its performance. Note that Participant 21 did not include dynamic markings in Task 3 and wrote an improvisation in Task 5.
Each type of musical symbol was coded, using the mutually exclusive categories shown above, so the sequence could be analyzed using text processing. Only the musical symbols were coded, not participants’ hand movements. The total time taken by a participant to perform each task was also calculated to evaluate the ease with which they performed them (see Figure 3).
Duration of each task. The p values reported are computed according to a two-sample t-test.
Results
To reduce the potential for bias, the sequences of symbols were analyzed by a researcher blind to participant group (i.e., beginners vs. experienced musicians). As the sample size was small, Table 2 presents the results of Fisher’s exact and Fisher-Freeman-Halton tests comparing the two groups.
Results of Tasks 2, 3, and 4. The p values reported are computed according to Fisher’s exact tests and Fisher-Freeman-Halton tests.
Speed of Notation
Figure 3 shows that the experienced musicians carried out the tasks more than twice as fast, on average, than the beginners (two-sample t-tests, p =.022, .005, and .005 on Tasks 2, 3, and 4).
Clef, key signature, time signature, bar lines
Both groups of participants started by writing the clef, sometimes followed by the key signature. All but one wrote the bar lines as soon as they had written all the note heads in the bar, or shortly thereafter. There was one outlier (Participant 15, experienced musician) who wrote all the bar lines before writing any notes.
Chords
There was a significant association between group membership and the sequence in which the note heads of chords were written (p < .001). All but two of the experienced musicians wrote the note heads of each chord from bottom to top, according to the way they may have been taught in harmony lessons to understand chords from the bass note upwards, whereas all the beginners wrote the chords from top to bottom, as is usual when writing text.
Accidental
The only accidental occurring in the tasks was a sharp sign in Task 4. There was a significant association between group membership and the sequence in which that accidental was written (p < .002). All the beginners copied the score from left to right, as they were accustomed to writing text from left to right, and wrote the sharp sign first, before writing the note heads of the chord. All but three of the experienced musicians added it after they had written the note heads of the chord, one of them not until they had written all the chords in the bar. As an accidental modifies the pitch of the related note head, this may be considered similar to the addition of a diacritical mark such as an accent above or below a letter.
Note heads, stems, and beams
There was no significant association between group membership and the direction in which participants wrote stems (upwards or downwards, p = .20). None of the participants wrote all the stems upwards; 12 wrote them downwards, and nine wrote them in both directions. The direction of writing is likely to be affected, however, by the position of the stem in relation to the note head and whether it is isolated or one of a group of notes. Indeed, 12 participants joined the first and last notes of a group without lifting the pen from the paper, in one or more cases, using a U- or inverted U-shaped gesture, as shown in Figures 4(b) and 5(c).
Handwriting behavior of three experienced musicians: (a) note heads prior to addition of beams (Participant 11, saxophone player); (b) adding beam using inverted U-shape gesture (Participant 9, multi-instrumentalist); (c) adding slur, having written note heads, stems, and beams (Participant 1, saxophone and guitar player).
Examples of sequences in which groups of four quavers (eighth notes) were written. The last three of the nine elements to be written are shown in grey. (a) δ = 1, (b) δ = 2.5, and (c) δ = 2.5 (inverted U shape).
The way notes are grouped under the same beam also influences the sequence in which note heads and stems are written. We calculated δ for each grouping: the average number of free note heads written before the stems were added. Three examples are presented in Figure 5. There was a significant association between group membership and this number (p = .047). All but one of those who consistently added stems to note heads one at a time, resulting in δ = 1 or close to 1 (see Figure 5(a)), were beginners, whereas experienced musicians tended to add the beams once they had written the note heads (see Figure 5(b)) and sometimes before adding some stems, for example when using an inverted U-shaped gesture (see Figure 5(c)). Two participants, both experienced musicians, wrote the stems long after the note heads, producing δ > 6 on at least one task (see Figure 4(a)).
The passage to be copied in Task 3 included, in the first bar, eight semiquavers (sixteenth notes). After an octave jump, there is an ascending scale with seven of these notes, slurred. Nine participants, all experienced musicians, wrote them as a continuous group of seven or eight, three of these participants grouped them into two groups of four, whereas all the beginners grouped them by beat or randomly. The only person who systematically wrote a stem after each note head in the case of sixteenth notes was Participant 17, a very experienced musician who was one of the fastest writers (he took only 63 seconds to carry out Task 3, whereas the average time for other experienced musicians was 113 seconds).
Expressive and dynamic markings
A score without expressive and dynamic markings is still readable, so musicians may choose to add them after the other elements. There were significant associations between group membership and sequences in which the staccato (p = .022) and dynamic (p = .044) markings were written. Most beginners added staccato markings to the quavers one at a time, whereas all but one of the experienced musicians added them having written all four quavers. All but one of the beginners and less than half of the experienced musicians added the dynamics and slurs as soon as they had written the notes above them, whereas the remainder of the experienced musicians added them to the otherwise complete score (see Figure 4(c)).
Free writing
The 13 participants who carried out the optional Task 5 were all experienced musicians who wrote out improvised or well-known melodies, either pop or classical. Only one participant used two staves. Here the behaviors of experienced musicians confirmed the previous findings, in that they tended to write chords from the bottom to the top, and stems after note heads, beat by beat, measure by measure, or having written all the note heads. In that task, only five participants used accidentals, and four of these five experienced musicians also wrote accidentals as soon as they had written the note to which it applied. Unusual behaviors were demonstrated by four participants: Participant 20 wrote the time signature and bar lines last; Participant 2 wrote the key signature last; Participants 8 and 11 wrote note heads representing crotchets (quarter notes) and quavers in the form of unfilled circles, and filled them in only when they had added the stems and beams.
Discussion
The observation of different sequencing when writing symbols by hand is not restricted to music. Some writers in English cursive postpone writing elements such as the top stroke of the lowercase t to the end of the syllable or a word. Similar decisions can be made when writing in other languages, such as Arabic and Chinese, by hand, as certain rules must be followed but some flexibility is permitted. Choices can also be made when practicing calligraphy, the fine art of writing, enabling the dynamics of forming letters to be varied.
Even monophonic music scores contain several layers of information, such as pitches, rhythms, the way they are to be articulated, and dynamics. These layers are encoded using a large set of symbols, allowing considerable freedom as to how they are written by hand. Non-musicians and beginners do not understand what they are writing, or their understanding is imperfect. When they copy music, they are likely to follow the common Western practice of writing from left to right and from top to bottom. Trained musicians read, think, and play music keeping its high-level structures in mind, even if these structures may differ from one musician to another. It makes sense that they would also write music by hand with these structures in mind, with a deeper understanding of how its components fit together.
This study showed significant associations between training and handwriting habits in that experienced musicians were more likely than beginners to write chords from bottom to top and add some elements later than others, such as writing the note heads first, in a flowing fashion, and only afterwards using stems and beams to emphasize grouping, and add expressive markings. Some of these observations are explained by traditional ways of practicing and teaching music such as modelling chords by describing the relationships between its tones and the bass of the chord. Even when they are not in root position, chords are almost always discussed and sung from bottom to top. The authors of the textbooks on writing music by hand do not, however, agree. Whereas Heussenstamm (1987) makes no recommendation, both Jacob (1947) and Roemer (1974) suggest writing chords according to the direction of the stem, and thus possibly from top to bottom. It would be possible to test experienced musicians’ practice in future by presenting tasks including chords with stems.
The differences observed between experienced musicians’ behaviors when writing stems and beams may relate to efficiency, their habits, or their mental representations. Beams are especially challenging in music typesetting but are handled in textbooks together with topics such as stem length and stem direction that we did not address in this study. Jacob (1947) recommends writing note heads first, then stems, and finally beams. For groups of notes with a complex, not-balanced, contour, Heussenstamm (1987) recommends writing note heads first, then the outer stems, the beam, and finally the inner stems.
The passages of music used in this study were tonally coherent and predictable for experienced classical musicians. As such musicians may correct mistakes, or deviations from what they expect, when sight-reading music (Sloboda, 1976), it could be tested whether they would correct such deviations when copying passages of music. Further studies could also analyze how long it takes participants to write each element of the score, present tasks including more elements, compare left-handed with right-handed participants, and investigate associations between the instruments played by participants, on the assumption that this affects musicians’ mental representations for music and music handwriting. They could also analyze potential differences between handwriting when transcribing audio input into musical notation and composing.
Being aware of how people write music by hand could have consequences for OMR studies. Current challenges in OMR relate both to detecting individual elements and ordering them appropriately (Pacha et al., 2019). Different layers of symbols deriving from the dynamics of handwriting can be identified even in static, scanned images, and their elements can be used in writer identification or other OMR tasks. Finally, further studies could combine handwriting analysis and studies involving the participation of users of new interfaces for notating music to improve them.
Footnotes
Acknowledgements
We thank all the participants who took part in the study. We thank the editor-in-chief and the anonymous reviewers for their insightful comments.
Author contributions
AB, FG, and MG conceived the study. AB performed the experiments and captured the video recordings. AB and FL transcribed the video recordings. MG and FL analyzed the data and drafted the manuscript. All authors approved the final version of the manuscript.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was partially funded by the French CPER MAuVE (ERDF, Région Hauts-de-France) and by a grant from the French Research Agency (ANR-11-EQPX-0023 IRDIVE).
