A case study of diverse multimodal influences on music improvisation using visual methodology

Introduction

This example of music redesign into the mode of speech was part of a larger study of cases coded as transformational and transmodal redesign of music improvisation of 5-year-olds in the classroom with peers, and at home. Coded as transmodal redesign in the home, it demonstrated transduction or fresh realisation of familiarised concepts of music by semiotic import of prior exploratory moments of music invention.

Sebastian had recently relocated from Ethiopia to inner-urban Australia, and was fluent in both English and Amharic. His mother worked in the film industry, possessing diverse talents incorporating dance, acting and vocal skills. She taught him traditional Amharic songs that incorporated stamping on the beat and a full circle jump in the air. During visits to their home, it was not unusual to see Sebastian dancing, singing and playing instruments in quick succession, often simultaneously. At times his mother prompted a response by posing a question or making an observation, scaffolding his music invention by focusing attention on one aspect of his activity or singing to a recording of Michael Jackson’s “Smooth Criminal” and clicking the beat with her fingers as he danced. She also built his song repertoire through song sharing of nursery rhymes or songs that Sebastian had learned at school, and taught him dance moves to Michael Jackson songs. She spent time as a volunteer assistant in the classroom, supporting his transition from home to school. In this way she built a reality for relating to the world and making transitions from interactions with his mother to those with his peers and teachers, as he accessed semiotic import of composing resources.

Home was the primary place for Sebastian’s music as habitus. As new arrivals to Australia, his mother had developed his capacity as musician and performer. He watched her teaching Ethiopian dance classes in the community. She also drew on her work in the film industry. She expanded his vocabulary using verbal linguistic mode by requesting he find another way to sing or describe events: “use different words to describe the piano”; “what sort of animals are in the song?”; “What did you like about our trip to the city last week?”. His expressive language skills and sense of self-efficacy were remarkable for his age. Consequently he displayed purpose, method, character and strength in his music improvisation, and his affective responses in social interaction.

Captured by his mother’s video recording, Sebastian interacted with her while he sang and played the piano in their garage, a familiar practice. His mother had, in earlier sessions, suggested familiar rhymes and pop songs for him to sing, and he did so, accompanying his singing with clustered notes played on the keyboard. However, his music improvisation was different, as Sebastian led the music, exploring his own music vocally as well as in accompaniment. It demonstrated furthering of understanding gained through a chain of prior repeated experiences of transformational redesign.

Methodology

This study was part of a larger embedded case study where video data of thematically coded “music play” events were selected for analysis according to two broad categories: children’s transformational redesign and transmodal redesign. These occurred during music composing events observed over time at home and at school. Children, whose parents gave informed consent, were selected for their interest and willingness to participate in classroom music events. They were from diverse cultural backgrounds, in both rural and urban schools. Parents took videos of the children in music games, with instruments or singing at home, either solo or in spontaneous interactions with family members. Some classroom friends were recruited for the small group music interactions at school, their parents also giving informed consent. This particular case study was set in the inner urban home of Sebastian.

Sebastian sat at a piano in the garage with his mother beside him. She would encourage him to sing and play an improvised piece of music, asking: “How about the Michael Jackson song when he was little: A-B-C?” She then modelled the singing using a tunefully modulated and projected voice revealing a slight American accent, inviting Sebastian to follow her in the same manner. She encouraged him by saying: “Sing it to me: use your real voice.”

The overall study differed from that of a normal case study in that the researcher worked within a social semiotic framework, assuming that children’s modal perceptions were always ordered, and the selections of modes were intentional and purposeful (Mavers, 2009). In analysing this music improvisation, a visual transcript, unique to this study, was made to show how the interaction of modes occurred as the music unfolded, as well as the order of effectiveness from higher to lower for carrying the meaning.

Music modes were represented within the music score. Other modes were set out beneath, as a score. Their order varied according to the meaning being conveyed. Therefore tools (modal resources) were flexible. Sebastian was seen to shift this preference for modes a number of times, depending on how he wished to emphasise meaning visually, aurally or spatially (in close proxemics to his mother). When all modes occurred in simultaneity in the last two bars, he had established new meaning in linguistic form. This inventive music practice and redesign was to be regarded as complex, diverse and dynamic, changing within the local culture. Therefore the case study could not be fully encapsulated, enclosed and categorised, or it would lose its dynamism. Understanding was to be ephemeral and provisional, based on theme of transmodal redesign. Video data of children’s music invention posed difficulties for analysis, for the nonverbal nature of music, its personal and cultural variations, and its temporal instantiation in time, made it problematic for data collection and analysis that has in the past privileged spoken or written modes (Flewitt, 2006, p. 27). In addition, the world does not always afford researchers with evidence they seek (Erickson, 1986).

Disconfirming data or alternative explanations must be considered, and the more convincing interpretations decided upon due to the weight of evidence and simultaneity of modes (Bezemer, 2008; Erickson, 2004; Mavers, 2011). The researcher collected, coded and analysed all data to provide consistency of interpretation. Multimodal analysis was checked with two co-raters, experts in the field. One was the researcher’s supervisor and the other a leader in social semiotics at University College London. Visual methodology provided the researcher with a convincing case for interpreting and analysing nonverbal data such as modes of music, gesture and proxemics (relations in the physical space). By foregrounding “the visual” as both a methodological and an analytical tool, the researcher can enrich participant interactions and responses, shaping the types of data that are produced.

Multimodal transcription is not new (Erickson, 1986; Ochs, 1979). The body of literature on transcription is substantial, but it mostly privileges linguistics. Studies applying a social semiotic framework and multimodal analysis to the investigation of early composing processes are still few in number and are selective, partial, representational and interpretive (Mavers, 2011; Ranker, 2009). This study of Sebastian’s improvisational processes in music used multiple methods of data collection, transcription and analysis. It provided in-depth and detailed embodied re-presentation of aspects or facets of his music invention. In so doing, it did not privilege linguistics but required focusing a lens on all modes as aspects of the whole, conceding that the researcher cannot see the whole in its entirety. In microanalysis, smaller units provide “significant opportunities for analysis, enhancing the insights into the single case” (Yin, 2009, p. 46). Choices of representation were made to “shape the account of social interaction in significant ways” (Bezemer & Mavers, 2011, p. 203).

Multimodal redesign: The space of music dialogue

Alexander (2008), Burnard (2012) and Green (2011) have noted that intuitive prior knowledge is passed on in music invention as children enact agency and identity, building on cultural ideas and framings. These observations were the basis of the researcher in this study formulating a conceptual framework as a lens for investigating young children’s contextualised music invention. A systematic approach to analysis of such diverse music was thought to assist in determining how transmodal and transformational redesign in music invention could be realised, and how this would contribute to learning. It was possible to see patterns emerging: to perceive a new way of understanding how transformational and transmodal redesign were realised. In all cases of transmodal redesign examined in the larger study, the co-occurrence of music modes (elements of music) selected in simultaneity with other modes, resolved dissonant (disjunctive) elements, to find a solution through music invention.

The space of music dialogue (Figure 1) is essentially a space of multimodal redesign, revealing how students in improvisation orchestrate elements of music as central modes within a space of other possible modes. Moments of music dialogue made during music improvisation are opportunities for learning, some initiated by actions of a teacher or parent, or with students. These modes included gesture, spoken and sung voice (verbal and audio linguistic modes), mimesic or technological mode and audio modes (instruments, and elements of music: pitch, meter, tempo, rhythm, dynamics and timbre; Figure 1, the space of modal designs – the “cornea” of the eye). As students express verbalised ideas in musical form through song, instrument or both, reinterpreting a recorded piece through movement, or orchestrating a group performance on percussion, they are seen to synthesise one or more musical ideas or elements of music, and extend these through multimodal combinations. This is identified as the use of music dialogue for redesign.

OPEN IN VIEWER

Figure 1.

The space of music dialogue.

Complex interactions of modal redesign in music are represented in the central circles in the diagram (Figure 1). In viewing music invention, the outer circle (the “iris”) represents ways by which all students use transformational redesign during interaction to enhance learning. Transmodal redesign is represented in the innermost circle (the “pupil”), where ways of knowing that promote conceptual understanding during transfer of meaning across modes, are observed in only some recorded interactions. Students who have a more secure understanding of music modes, the elements of music, will have experienced repeated transformational redesign in a chain of semiosis over time. The space of music dialogue (Figure 1) represents ways students realise conceptual knowledge by redesign in music invention. Examination of aspects of redesign is assisted by a framework with which to examine music events.

Sebastian’s garage rap

In previous events, Sebastian was outstanding in his ability to reorder or reconfigure modes to feature selected elements of music. He responded by interpreting these using selections of notes and playing phrases with a variety of dynamics and tone colour, sometimes repeating the rhythm to listen to the timbre of the piano. In this case study, Sebastian used the upright piano at home to rehearse, refine and fix previous experiences of music into speech rhyme. In previous music events at home, Sebastian also looked to his mother for cues as to how he might continue.

Description

In this case study, Sebastian used transmodal redesign to shift a rendition of songs, some of which he learned at school (There Was An Old Woman; 1-2-3-4-5, Once I Caught A Fish Alive; Hello Everybody) and some of which his mother taught him (Michael Jackson’s “Smooth Criminal” and “A-B-C”) from the mode of music to a principal mode of speech. He had previously made transformations in the mode of music as he spoke these while making percussive sounds on the beat, clustered notes played over the entire register of the piano. He made sounds similar to Stockhausen’s triadic sounds and the 12-tone row. This was an immediate impression on first hearing the music, and later reinforced when in analysis the clustered tri-tone was repeated in the manuscript. His playing was rhythmically complex and structured but fluid, redesigning subtle nuances of sounds. Then, playing a very soft, slow and expressive segment he explored the lyrical potential of the piano, note by note swaying his body and head in circular movements, tilting his face and gazing upwards to listen to the sounds. Sebastian’s fingers curled over the notes, as he rotated his hands in a relaxed manner. Sometimes he used voice and piano, and jumped into the air using gestural mode of whole body movement to transition from a song to another (an Ethiopian tradition, as discovered in an interview with his mother).

Sebastian’s mother was partial to the quieter, more expressive section of playing (Figure 2, bars 1–4 and bars 35–42), adding “I love it!” In these segments Sebastian was more reflective, displaying his affective responses. His mother later shared with the researcher her thoughts on his playing: “when he plays like this I can see his soul.” The rapport between mother and son was important for building focus, attention, social skills such as sharing and cooperation through music. Involvement of his mother made redesign more complete. She supported his playing by asking questions about his favourite music, or singing the first line of a Michael Jackson song.

OPEN IN VIEWER

Figure 2.

Music score transcript and modal configuration showing music to speech rhyme: Sebastian’s garage rap.

Analysis

The transcript in Figure 2 demonstrated the interaction of music (audio) and other modes, the modal configuration being aligned to the piano and voice. This showed when music mode had principal focus (the start, when Sebastian’s mother requested he play a song) and at the end when the mode of speech (linguistics) was the dominant. Throughout this invention, the gestural mode of movement was also powerful and became the principal mode when a shift of meaning occurred in the last two bars (Figure 2). Proximity to his mother was also influential when Sebastian needed support in how to continue. However, he quickly forgot her presence as he weaved words together with the underlying support of the piano music. His meaning making was important as a way of consolidating his musical identity and also his ability to draw on appropriate modes in redesign of meaning across domains. This assisted him in making an effective and fluent transition from home to school.

Modal density (modes from higher to lower order) is represented in Figure 3.

OPEN IN VIEWER

Figure 3.

Modal density of Sebastian’s home music event: Garage rap.

The order of modes in Figure 3 was representational of this composing event, characteristic of a transmodal moment in the space of a familiar cultural context, the home. The space was the foundation for possible shifts in meaning while moving from the audio mode of music (singing and playing the piano to Michael Jackson pop songs) to a mode of speech rhymes. The rhymes began as words of a traditional English rhyme found in a familiar literacy text used in school: “There Was An Old Woman” and integrated fragments from “1-2-3-4-5, Once I Caught A Fish Alive!” However, in this fresh invention or playful experimentation, Sebastian pinned down prior experiences expressed in the principal mode of music by articulating his conceptual understanding in the principal mode of speech/verbal linguistics. As seen in Figure 2, Sebastian included some words from the traditional Amharic language of his Ethiopian origin while inventing a new speech rhyme through transmodal redesign. He foregrounded “Then he sit me te-ri did a” as an expression that made his father very present, meaning he sat with him to play the piano.

Sebastian began:

My lov-ly piano is ve-ry a use-ful. (spoken in meter, adding piano clusters)

My use-ful pi – ana! Pioano is ve-ry good.

It’s ve-ry shiny and ve-ry dee- clean.

Very, very good to play on.

Ther e – was and old wo-man

Who swallowed – a piano!

The piano was so lo – swallowed

And I had no piano!

I had no piano - I had no piano,

No more pianos Daddy could play!

I need to play – ready to play!

1-2-3-4-5 Then he sit me te-ri did a

There was an old wo-maannn! (Mum: make up a song about Mummy).

My mummy is ve-ry zihg- ny and I love to play with her (speaks strong metrical rhyme)

I love to eat with her

Play at the park with the netball hoooops.

And I love to play with the netball hoops.

And then: (Sebastian began a “rap” dance, sliding to sides, playing over all the keys).

Original familiar nursery rhymes were transformed through redesign in the mode of speech as Sebastian grappled with cognitive dissonance, synthesising ideas to solve the problem of creating new songs/rhymes (Custodero, 2006). He built on the underlying rhythmic motif and ideas in the original text with the disjunctive (conflicting or nonsense) idea of a “swallowed” piano (Figure 2, Bars 16, 18 and 20). Here Sebastian added percussive piano clusters, shifting the mood, presenting a new idea that resembled “opera” combining movement, gesture, voice and music in a multimodal ensemble of meaning. The modes he drew upon to make meaning occurred in simultaneity at this point in his music invention, indicating the realisation of transmodal redesign (Figure 2). He had not only transformed previous musical knowledge and experiences by reordering these modes. He had knowingly and purposefully pinned down the meaning during a moment of folding of previous experiences through transmodal redesign. Sebastian did this as he used syncretism by drawing on cultural practices related to his previously acquired music repertoire in the composing process. He also selected features of music and repeated, extended or expanded these.

Sebastian extended ideas within the lyrics, thereby shifting his meaning making. He achieved this by playing with rhythmic elements, repetition and language devices. The text became not just an account, a narrative, but a more condensed form, a poem or rhyme using meter, extension of phrases (I had no piano, I had no piano – no more pianos daddy could play) or syllabic repetition (I need to play – ready to play). He used the device of expansion of an idea using the word “play” and what that meant in terms of playing a piano and later playing in the park with the netball hoops. The rhythmic elements of the rhyme were representations of time that organised and underpinned Sebastian’s invention and word play, closely allied to spatial representations. He invested in the element of musical duration, just as a musical geometer would use numbers to divide space. All these elements demonstrated Sebastian’s cognitive effort.

For much of the music event Sebastian improvised freely with fast, syncopated and complex rhythmic phrases, using the entire register of the piano in loud and percussive effect, and with diminuendo to the end of the passage. He stretched both hands out to either end of the keyboard to encompass its length. His arms flew up and down energetically with hands raised to shoulder level then returning to the keys. Shoulders were raised then relaxed in response to variations in dynamics and tone. The whole body movement of the rap and the accompanying speech rhyme were the dominant modes for most of this invention in verbal linguistic mode (Figure 2). When Sebastian realised the climax in the intensity of the music’s sound, he used all modes in simultaneity (Figure 2, Bars 39–46) and, in the final three bars, he leaped from one end of the piano to the other in wide strides to emphasise the joyousness, the affective response, connected to the game with his mother throwing the ball into the netball hoop.

This seven-bar climax to his garage rap demonstrated the shift in meaning associated with drawing on and combining all modes in simultaneity to realise the music mode as a new dominant mode of speech rhyme. His gaze at his reflection in the piano (Figure 2, Bar 10) influenced the lyrics of his speech rhyme about his “shiny” piano. The proximity of his mother and the emotional ties to her may have been influential in his shift from the use of nursery songs and extensions of these, to a verbal realisation of meaning as he wove his spoken rhyme about his mother. All modes interacted in relationship to each other in complex ways to shape the meaning made by Sebastian’s garage rap.

Discussion

Sometimes there were repeated elements of music, such as accent (bars 6 and 7; 18 and 21) or dynamics (Figure 2, bar 34 and bar 43) or rhythmic motifs such as crotchets followed by a minim (Figure 2, bars 35–40) repeated for emphasis and to reinforce meaning. Sebastian’s inventions in playing and singing were musical in the sense of both Bach’s (1723) two part inventions and the tri-tone in Messiaen’s (1929) Préludes for piano. The tri-tone, used so often in Messiaen’s works, was frequently selected (preferenced) by Sebastian to add colour (Figure 2, Bars 1–7; 14 and 15; 17 and 18; 35, 43).

Sebastian’s instinctive use of more than one mode for communication in music invention has been seen in this study to reveal a depth of thinking that transcends communication, being a form of resistance (Deleuze & Guattari, 2011) and, indeed, of conceptual understanding. This was particularly observed when there existed conflict or tension as he moved meaning across from “No more pianos” to the lyrics about his dad and then his mum. He obtained a resolution in the unfolding of the music as modes occurred in simultaneity. The seven-bar climax to his garage rap demonstrated the shift in meaning where all modes occurred in simultaneity, including gaze at his reflection in the piano, realising the music mode redesigned as a new dominant mode of speech rhyme (Figure 2, Bar 10). The scope of his constructs was enlarged, contributing to the robustness of understandings. A summary of modal redesign is shown in Table 1.

OPEN IN VIEWER

Table 1.

Summary of transmodal redesign from music to speech rhyme: Sebastian’s garage rap.

Transmodal redesign from music to speech rhyme: Sebastian’s garage rap	ModesVerbal linguisticsRhymeRhythmHarmonyRests – silenceGestureGaze	Role of modes	Objects,materialsavailable	Objects,materialsused	Comments
Content Sebastian, by transmodal redesign, shifted an operatic rendition of songs from school (There Was An Old Woman; 1-2-3-4-5, Once I Caught A Fish Alive; Hello Everybody) and from home with his mother (Michael Jackson’s “Smooth Criminal” and “A-B-C”) from the mode of music to a principal mode of speech. He invented a new rhyme about his piano, then playing netball with his mummy. Context Inner city urban home	Gestural mode Whole body, Torso, Head, Shoulders, Finger independence Visual mode Gaze Audio mode Piano affordances Rhythm Silence Phrasing Inflexion Accent Spatial mode Proxemics Verbal linguistics Rhyme Repetition “Very … Swallowed No … Play” Humour Nonsense	Salience Shift of the meaning was through the text, the lyrics. This was achieved using rhythmic devices of language. The text became a condensed form, a rhyme with meter, extension of phrases (I had no piano, no piano, no more pianos daddy could play) and syllabic repetition (I need to play – ready to play). Semiotic weighting 1. Verbal 2. Music 3. Gesture 4. Proxemic Relations between modes Simultaneity of modes Shifts of modes	Piano	Piano	Shifts of understanding were made as he used voice (verbal linguistics) and piano (audio), and jumped into the air excitedly using the mode of whole body movement (gesture) to transition from a song to a new idea, using multiple modes in simultaneity. Sebastian grappled with cognitive dissonance, resistance then synthesised ideas to solve the problem of creating new songs/rhymes. Prior learning with mother in home assisted his knowledge of elements of music. The proximity of his mother helped shift from the mode of song to spoken rhyme.

Sebastian used the interaction and dialogue of modes and prior cultural influences to create a multimodal ensemble of meaning making, These operated as a vocabulary that expanded possibilities for expressing the inexpressible, for making dialogue with the music and with his mother, and for re-presenting his prior knowledge, in order to strengthen his musical identity (Pahl & Roswell, 2006). Sebastian’s stable beat while working in complex rhythmic patterns helped extend previous ideas. He experimented, transforming his musical ideas using modes of body movement, head tilted to one side, shoulders raised and lowered, gaze directed up, and audio mode of piano, “reaching” for a new way of supporting the meaning of the lyrics. Incorporated resources of Ethiopian song and dance, pop music, timbres of instruments and his mother’s songs, purposefully selected loud or soft, right or left hand or both together.

Sebastian’s garage rap, while spoken, incorporated many resources for music invention: Ethiopian song and dance rhythms, music technology, instruments and his mother’s songs. He example of music invention. It was an opportunity for Sebastian to draw on prior experiences of was purposeful in making selections of loud and soft, right and left hand or both together while playing the piano, redesigning them to support the meaning in the lyrics. The rap was a masterful songs, nursery rhymes, dance rhythms and voice to create a spontaneous linguistic text. Expansion of ideas worked into the driving metrical text created new meanings, and new understandings of the structures and elements of music and language.

Conclusion

This research study featured interconnectivity of perspectives in a multimodal ensemble “suggestive of discrete parts brought together as a synthesised whole, where modes, like melodies played on different instruments, are interrelated in complex ways” (MODE, 2015). Over time, Sebastian made interpretations across principal modes, from music to language. He developed conceptual understanding as he “took on” various identities, situated creativities or cultural framings (Burnard, 2012) and chained them into coherent lines of thinking and enquiry (Alexander, 2008). His conceptual understanding and cultural awareness was expanded and enriched. Sebastian’s transmodal redesign was masterful. Language (verbal linguistics) was only one of many modes operating in a complex network of meaning making, facilitating communication and problem solving. Music modes, the elements of music, were essential in contributing to further understanding. Change led to new meaning.

The significance of this music event for learning was the way Sebastian realised transmodal redesign, making a shift in musical understanding through interaction. Shifting meaning through music dialogue from the mode of music to that of speech (verbal linguistics), Sebastian solved problems such as effective and apt use of resources during interaction to communicate (Figure 2). Communication involved modes of gesture, proxemics between mother and child, and audio mode: listening and responding with piano and rhyme. Resources of tempo, dynamics and phrasing (music modes) were pivotal throughout the composing event, assisting the shift in conceptual understanding. Modal resources, through chains of semiosis, also reinforced Sebastian’s identity.

In multimodal creative practices viewed in the space of music dialogue, educators can determine how resources are selected for meaning making; how new conceptual knowledge (elements of music, ideas and events) is being created; and how diverse modes or media are featured. Together these strands influence and enrich pedagogic choices in the delivery of music curricula, as in other essential learning areas. As Burnard (2012, p. 238) concluded, “there is a need to claim a place for musical creativities as a critically important area of musical knowledge, musical learning, and pedagogical knowledge of a different type”.