Grains of Sound: Visual and Sonic Textures in Sand or Peter and the Wolf

Grains of sand

Whether due to critics’ tendencies to canonize male experimental artists over women, or to the limited circulation of her films, Leaf’s work is too infrequently discussed in proportion to the impact of her material-based experiments on the history of animation. After Sand, which earned her an invitation to work for the National Film Board of Canada in 1972, Leaf created several sand films, the Academy Award-nominated film The Street (1976, painted on glass), and the ambitious Two Sisters (1990, scratched on 70mm film). Despite receiving grand prizes at international festivals, her work has attracted much less scholarly attention than that of fellow NFB filmmaker Norman McLaren. Yet, her innovations similarly have transformed animation, guided by a long-standing commitment to find the right medium to portray the images in her imagination. She drew frequently from the age of 3 or 4, her mother Barbara Leaf (quoted in Leaf et al., 1992: 2) recalls, and ‘never wanted instruction – she wanted to discover what she could do with any medium’. From age 11 or 12, Leaf painted landscapes with oil paints; on a college trip to Mexico, she painted her way around the country on bus trips instead of taking a Spanish course. While at Radcliffe College in the late 1960s, Leaf took Harvard art classes and found animation almost by accident in her final year.

When Leaf’s painting teacher, Albert Alcalay, recommended that she take Derek Lamb’s animation class, the college senior was timid about the new medium at first. But that changed after Lamb asked students to bring in objects to animate on a light box. Leaf brought sand from the Gloucester, Massachusetts beach in front of her family’s cottage. As she created sand silhouettes, Lamb celebrated her quick command of a novel animation technique; he wrote later: ‘From the first few motion tests it was obvious that you’d found your medium’ (cited in Leaf et al., 1992: 4). Lamb recalled a class visit from Norman McLaren, who created shapes at Leaf’s station ‘like a kid in a sand box’. Like Lamb, McLaren was startled at the innovation, exclaiming, ‘“In all my years of animation I’ve never thought of this!”’ (cited in Leaf et al., 1992: 4).

Leaf was not the first to animate with sand, however. The Swiss husband and wife duo Ernest ‘Nag’ and Gisèle Ansorge animated black quartz sand in their short film The Ravens (aka The Crows, 1967). The Ansorges found sand while trying to depict blood moving inside the body for a film about heart problems. As Paul Wells (2018: 38) explains, sand allowed them to surpass restrictions of line in hand-drawn animation to create ‘mutability, flux, and, ironically, the immateriality of the spirit, and forces beyond the concrete environment’. But Leaf’s practice brought the medium to emotional and aesthetic maturity. As Midhat ‘Ajan’ Ajanovic (2002) states, ‘she was the first one to achieve visual composition of such an unimagined lyric quality by using that method’. Leaf developed lyrical qualities in sand animation through a meticulous attention to detail.

Using tools such as forks and combs, as well as her own fingers, Leaf formed her sand figures on a light box underneath a mounted camera. Photos reveal Leaf’s thorough absorption with – and even within – the materials of production: she leans over a scene, face bent in concentration, lit up from above and below. With each slight adjustment to the shapes or movements formed in sand, she photographs two frames, adjusts them, takes two more frames, and so on. In 1976, Ronald H Blumer compared her process to weaving in three dimensions. Just like a weaver synthesizes shapes into one cloth, Leaf smoothly dissolves between characters and scenes instead of cutting shots. She privileges the metamorphoses of sand – an aggregate material with many tiny grains that flock and disperse. As she touches and is touched by this nebulous material, her figures possess what Blumer (1976: 22) calls a ‘great plastic consistency . . . moving as if powered by the gentle fingers of the wind’. Leaf bends to sand’s granular agency: she shapes the wolf into a cloud of sand that Peter blows away at the end of the film, letting sand remake her figures. The playfulness of this image inspires childlike delight, but to fully accept Blumer’s metaphor would be to naturalize Leaf’s talent. Indeed, in order to achieve the coveted ‘effortless’ motion of the wind in the trees, she drew every pose – paralleling at an artisanal level the painstaking labor of in-betweeners at animation studios.

Leaf performed the roles that several individuals would in a studio: she drew key frames and in-betweens, and cleaned up stray sand grains. As Hannah Frank (2019: 6) explains, in-betweeners and clean-up artists – who were often women – were considered noncreative laborers despite their training and high-stakes work. Whereas Frank analyzes overlooked details in still frames as photographic records of animators’ production, I study moments when grains of sand and grains of sound spontaneously collide, since there are no photographic records of Leaf’s production process. ³ Leaf also dealt with errors differently; in cel animation, animators can discard and redo individual frames if they make a mistake. Leaf could only redo entire sequences, since she could not recreate the exact placement of sand grains in the previous frame (Blumer, 1976: 22). She carefully displaced grains from frame to frame to retain each object’s form but also to show its next movement. Displacement is the message of her medium: Leaf’s granular transformations efface her labor and emphasize sand as collaborative agent. Her films are archives of the impressions of her hands and those of sand’s volatile unfoldings. Before she displaces each grain, she must sense sand’s tendency to disperse and act accordingly.

An appreciation of Leaf’s intensive process can be gained by a close study of Sand’s opening. Before the main titles, Peter cartwheels into the frame (see Figure 1). As he spins in, his outstretched legs and arms are richly defined. To add to these difficult details, Peter’s cartwheel is mimicked by a bird, whose beak and wings almost merge with Peter’s hands. Beyond the mirrored motion, though, the bird’s limbs also resemble Peter’s claw-like hands. One can imagine the fierce concentration Leaf mustered in order to scrape away sand grains between fingers and feathers. Each figure’s movement is precise, down to tiny groups of grains. Coupled with this dexterity, however, is an openness to meet sand halfway – to work with its constant state of flux. Leaf improvises alongside sand’s errancy to transiently form and dissolve objects.

OPEN IN VIEWER

Figure 1.

Peter cartwheels into the frame, mimicked by the bird, near the beginning of Sand or Peter and the Wolf (Caroline Leaf, 1969). Screen grab from DVD (Caroline Leaf: Out on a Limb: Handcrafted Cinema, 2010, Caroline Leaf. Montreal: National Film Board of Canada). Reproduced courtesy of the Caroline Leaf Collection, Harvard Film Archive, Harvard College Library.

As this analysis shows, the emotive power of motion in Sand does not derive from a sense of a magical wind – but it also does not come from the sense of early cinema’s happy accidents that captured ‘the wind in the trees’. For Jordan Schonig (2017: 20), such contingent movements are not mere unplanned curiosities; they are ‘motion forms’ activated by the framing of the camera as ‘tools for thinking’ that can redress habitual ways of seeing and making sense. Following Schonig, I take Leaf’s carefully arranged sand grains as an opportunity for fine-grained analysis of her engagement with materials in animation, seeing in it a tactile approach to perceive what the next frame will look like. ⁴

Considered more broadly, Leaf’s animation process showcases two main aspects of her work: first, it highlights the time and effort it took her to develop the motor skills needed to work with sand, and second, it shows how she performed those skills as an improviser. Leaf manipulates sand with fine motor skills honed through thoughtful practice. Consider a violinist, who must place a fingertip on an exact point on the string to play the correct pitch. To learn to identify how notes correspond to the strings, young violinists practice with tape lines on the fingerboard underneath the strings. After a few years of consistent practice, students develop an ear for pitch and muscle memory for the proper placement of their fingers on the strings, and they take off the tape. Just as a violinist learns to produce exact notes through extensive training, so Leaf developed a sense of how to plan and perform miniscule alterations of each sand figure. She also shaded those changes like violinists hone dynamics and timbres. As she told Nag Vladermersky (2003), ‘I refined and complicated the sand imagery by pressing the opaque sand into different densities and creating shadings.’ In this way, Leaf can modulate shape, mood, texture, and tone, grain by grain.

The musician metaphor extends further, explicating the second key aspect of her animation process – that Leaf’s films record the improvisatory performances of minute movements. Underneath the camera, Leaf works

straight forward. You can’t go back and change things. There’s no point to testing movement. You do it once with intensity, feeling the movement, the best you can. It’s a kind of performance, like animating puppets, only slower because you have to draw and redraw the images.

She will develop this metaphor, calling her work ‘a performance in slow motion. You develop a system for remembering movement as you are making it, and you have to feel it intensely’. Leaf’s robust muscle memory, learned from improvising with sand’s ethereal motion, endows figures with emotional intensity and defies critics who naturalized her talent as merely ‘the gentle fingers of the wind’ (all quotes cited in Ajanovic, 2002).

Near the end of Sand, for example, Leaf channels sand’s composite nature into emotive force. Night has fallen; the screen is dark with sand. Leaf etches out Peter’s white form as negative space when he creeps up to the wolf, catches hold of its tail, and swings it around (Figure 2). His rapid, circular motion resembles a phenakistoscope that creates a smear of the wolf moving through the air. When the animals eaten by the wolf catapult out of its stomach whole, their undigested forms indicate sand’s agency as a dis- and reassembling aggregate: it shapes characters as much as Leaf does. In a playful wink to the sand that slips between Leaf’s fingers, Peter nudges himself out of the shell of smeared sand that the wolf has become and blows it away in a single puff.

OPEN IN VIEWER

Figure 2.

Peter vanquishes the wolf near the end of Sand or Peter and the Wolf (Caroline Leaf, 1969). Screen grab from DVD (Caroline Leaf: Out on a Limb: Handcrafted Cinema, 2010, Caroline Leaf. Montreal: National Film Board of Canada). Reproduced courtesy of the Caroline Leaf Collection, Harvard Film Archive, Harvard College Library.

Leaf’s practiced performances before the camera balance playful winks and weighty gravitas, nowhere better illustrated than in the wolf’s virtuosic shape-shifting powers. Any mention of shape-shifting in animation often heralds Eisenstein’s (2006: 101) concept of the plasmatic, where an animated being is a ‘primal protoplasm . . . capable of assuming any form and which, skipping along the rungs of the evolutionary ladder, attaches itself to any and all forms of animal existence’. Thomas Lamarre (2008), Daisy Du (2019) and others have observed how plasmaticness is often applied to animated animals because audiences tend to be ‘more tolerant of violent deformation and transformation of animal body forms than of human characters’ (Du, 2019: 177). But Sand’s metamorphoses depart from familiar uses of animal plasmaticness in two ways.

First, Leaf faithfully imprints the animals’ forms, countering a long-held tradition of anthropomorphizing animals in animated cartoons. Anthropocentric ideals of plasmaticness seem baked into many cartoons, where animals transform only to resemble humans or human fantasies of them. Leaf’s granular transformations are not humanoid fantasies like Eisenstein’s example of Mickey’s arm stretching for a piano key. Eisenstein’s case studies fall short of the complete transformation invoked in his above definition; the octopuses that take the shape of elephants in Merbabies (1938) remain octopuses, and spectators’ belief in their animality momentarily bends but does not break. However, much as Miriam Hansen (1993) describes the utopian potential for Mickey’s plasmaticness through Walter Benjamin’s views on technology, Leaf’s own technology – her improvisatory muscle memory and stop-motion – pushes the plasmatic past the anthropomorphic. Leaf animates the animality of animals in their sensuous materiality, not merely caricaturing their behaviors as cartoons do. Her wolf transforms into a tree to surveil the animals, a worm in order to entice the bird, and a flower for the cat to eat. These are not mere extensions of limbs, as in the temporary squash-and-stretch similes of Merbabies, but transformations into entirely new objects – even non-animal.

Part of the aesthetic shift has to do with material. In typical cases, animators bend the outlines of animals to their will, treating them as inert material. Leaf, however, deftly improvises with sand grains in a way that highlights the tension between sand’s fluctuating nature and the creator’s hand. ⁵ Whereas plasmatic contours eventually spring back to form like a rubber band, Leaf engages granularity: she gleans animals’ distinctive qualities from the shifting makeup of their grains. The wolf, for example, shape-shifts extensively, which communicates its threat to the other animals. Leaf’s performances are sensitive to each animal in performance: as the sand shifts, without cuts, spectators see animals transform in a way that is intimately related to their real-life movements, and to those of sand.

As Leaf’s animals transform and tail each other around the screen in all directions, Sand makes the sense of a stable ground not merely plasmatic, but granular. Both Aylish Wood (2006) and Ryan Pierson (2015) have detailed how Leaf’s The Metamorphosis of Mr. Samsa (1977) transforms spectators’ sense of space. They focus on the maximal details packed into the frame that turn seemingly simple settings into environments where characters emote and travel in surprising ways across screened space. Leaf’s transformations of figures negotiate our view onto the world beyond the temporary stretch of plasmaticness. As Pierson (2015: 17) observes of Sand, ‘we detach the sand from what it represents and wait for it to form another picture, at which point the story can continue’. Due to Leaf’s smooth and often otherworldly transformations, the sand is what our eyes follow – we lack ‘a ground against which we can attribute change or constancy to a figure’ (p. 13). Whereas the ground is normally preserved in cases of plasmaticness, Leaf’s spaces and characters metamorphose in an unstable background. By dissolving figure and ground, Leaf effects the ‘musicality of landscape’ that Eisenstein desired in Bambi, or what Daniel Morgan (2020: 299) calls activated backgrounds: interactions between background and foreground that showcase the mutable forces of nature. In Sand, animals emerge from, and merge with, a nature that no longer plays second fiddle to plasmatic foregrounds.

If spectators view Sand without sound, the harmony of grains that cohere to move each animal’s limbs seems even more spectacular. In the intricate walk cycles of Peter running away from the wolf, or the wolf creeping toward him, Leaf’s refined grains have an emotional attraction – a magnetism, to pun on the iron oxide in black sand that attracts magnets (Jeffers, 2007: 3). At the end of the film, that magnetism becomes animal when every creature’s limbs fuse with Peter’s – the bird on top of his back, the duck’s feet protruding between his, Peter’s hand extending to pet the cat. Each animal is visible in a human–animal assemblage before Peter hugs the cat and the duck’s limbs disappear. The sand grains of the bird also enfold into Peter’s back at this moment, and all creatures gather in a round embrace. These are not merely companion species (Haraway, 2003) but beings molded from and subject to sand’s granular agency. The duck and the wolf are hewn from the same material, no longer predator and prey but co-constituted in grainy detail. As we have seen, Leaf treats sand not as weightless matter that she controls – as in characters’ plasmatic contours – but as a weighty material force that displaces habitual ways of perceiving.

Interlude: The source material of Peter and the Wolf

Much as sand makes possible the contingencies Leaf portrays among animals, human, and environment, an analog synthesizer allows Riesman to intermingle previously unheard sounds with patently familiar ones. Through Riesman’s attention to sounds’ grainy details, spectators discover how sounds exceed a 1:1 mimetic relationship with images. Riesman contests a common cartoon scoring practice of recurring musical themes – called leitmotifs – that correspond with one character or mood. Leitmotifs prevail in Leaf’s source of inspiration, Peter and the Wolf (Sergei Prokofiev, 1891–1953), one of the most ubiquitous pieces of classical music at educational concerts. In 1936, Moscow Children’s Theater director Natalia Sats (1985: 216) asked Prokofiev to collaborate on a piece that blended musical and spoken-word narration. Sats wanted to help children understand which sound went with each instrument, so Prokofiev wrote instrument-based leitmotifs that matched characters: the flute for the bird, oboe for the duck, clarinet for the cat, French horn for the wolf, strings for Peter, bassoon for his grandfather, and drums for the hunters. To establish these pairings before the piece begins, the narrator asks each performer to play their leitmotifs – sometimes even dressed up as their character. Then, the performers launch into the tale of Peter, who, despite his grandfather’s warnings, defeats the wolf and rescues the animals.

But Leaf’s sand grains do not tell this tale. While Peter is still the hero who captures the wolf that has eaten the duck, bird, and cat, Leaf’s shape-shifting creatures shift the story to focus more on animals. She omits Peter’s elder, Grandfather, and the hunters who help Peter dispose of the wolf. Her tale decenters humans and even human ways of understanding, as she also dispenses with the narration that is half of Prokofiev’s piece. When Leaf adapts literary texts – as she has done in several of her films – her goal is ‘to eliminate as much of the text as possible, and put it into visuals’ (quoted in Ajanovic, 2002). Without text, non-human interactions come to the fore. For example, Peter gives a reassuring hug to the frightened duck, who reciprocates with a grateful kiss. This may resemble anthropomorphic behavior at first blush, but Leaf’s grain arrangements crane the duck’s neck to emphasize the textures of sand and creaturely skin. Riesman’s score also discourages a reading of Sand as character animation. His music emphasizes the graininess of sand and the ethereal movements of creatures within it.

Prokofiev aurally reinforced characters with leitmotifs but, in pairing the sprightly flute with the bird and the reedy oboe with the duck, he also emphasized each instrument’s timbre – its distinctive tone color apart from pitch and volume. Timbre is tied to the materials used to make instruments and performers’ ways of playing them. While timbre has been the focus of many studies of classical music (Dolan, 2013; Eidsheim, 2019) and electronic music (Fales, 2005; Lavengood, 2019), scholars have neglected how timbre lends material qualities to both sounds and images in animation. Notably, Eisenstein (2006: 151) describes Prokofiev’s Peter leitmotifs as ‘instrumental timbres . . . comically localised throughout representational, concrete realms’. Because timbre can ‘imply a sound source through the qualities of the sound itself’ (Garcia, 2015: 66), it is a crucial window into the concrete realms of sound. So, too, is Roland Barthes’ (1977) concept of the ‘grain of the voice’, which exhibits material traces of the body. Barthes describes singers whose lips, teeth, and tongue become audible as they sing. Grain encompasses not only voice, but material apparatuses that produce sound, such as Riesman’s synthesizer. Or, as Buddy Collette described jazz saxophonists, each performer has signature tones ‘just like when you pick up the phone and somebody says, “Hi, Steve”, and right away you know instantly, the first word, who it is’ (cited in Bryant et al., 1998: 147–148).

For viewers of Sand who are familiar with Prokofiev’s piece, resonances of the animals’ leitmotifs surface in the mind’s ear as the tell-tale title and animals appear on-screen. A viewer’s recollection of Prokofiev’s earworms creates a memory montage effect that is not only aural but also visual; seeing Leaf’s figures, for example, could recall images and sounds from the 1946 Disney cartoon or a childhood concert. This memory montage effect becomes jarring, however, when Riesman’s score departs from Prokofiev’s character/instrument pairings and adds new timbres. He uses a violin, a synthesizer, somewhat unusual woodwinds, and the percussion-like prepared piano. Much as Leaf emphasizes the kinship between Peter, the bird, duck, and cat, Riesman’s score emphasizes the symbiosis of humans and nonhumans with acoustic and electronic timbres. ‘I didn’t think about adapting Prokofiev but rather about doing something that would work for the film’, Riesman (2020, personal communication) reflected. He quoted Peter’s theme ‘just as a joke, to reference it and get audiences to think, “Oh yeah, Peter and the Wolf, I know that tune”, but then to immediately do something else.’ As a result, spectators’ memories of the piece are triggered and then confounded. Viewers cling to the sound grains that Riesman creates, like lines of breadcrumbs, to perceive the emotional and environmental resonances inherent in the sand grains of Leaf’s imagery.

Grains of sound

Riesman’s method echoed Leaf’s two-step process of attuning to the textures of her materials and then improvising with them to carve out new forms. When Caroline Leaf was searching for a composer for Sand, Riesman was a graduate student in music composition at Harvard. Riesman was impressed by the film and excited to collaborate with the visual arts department. In his goal to compose something that ‘worked’ for the film, the close attention he paid to sonic timbres heightened the material qualities of Leaf’s imagery.

With his ability to play multiple instruments and improvise, Riesman materially differentiates each animal using timbres that vaguely resemble the logic behind Prokofiev’s instrumentation. Riesman played the bamboo flute and a bagpipe chanter for the bird and duck, respectively. The bagpipe chanter suits the throaty timbre of a duck’s quacks and sounds similar to the oboe. Importantly, however, the sounds Riesman made on those birdlike instruments are much closer to bird calls than Prokofiev’s leitmotifs. Riesman’s improvised sounds resemble animals in nature more than leitmotivic, human approximations of a duck or bird. The only instrument he did not play in the score is violin, which was played by his friend Tison Street. Street opens the film with a quote of the Peter melody, but the familiar theme quickly dissolves into new territory. Street’s unfinished Peter theme instead opens out into a trill (a repeated waver between two notes) that ushers in a pastoral outpouring of different sounds – improvised layers of warbling duck and bird sounds on Riesman’s woodwinds, and fuzzy bass synthesizer chords. Riesman’s granular score fleshes out animals with contingent sounds, not reiterative themes.

Across Sand, Riesman uses a different musical strategy than Prokofiev to evoke the animals. Prokofiev’s themes possess mnemonic staying power for listeners due to the repetitive nature of leitmotifs – they reiterate character traits over the course of a film and rely on music that is culturally familiar to listeners in order to fulfill ‘quick and efficient signification to a mass audience’ (Gorbman, 1987: 4). Whereas Prokofiev – and many classical Hollywood films – use leitmotifs to typecast characters, Riesman dispenses with these culturally determined signs for animals. Riesman does not attribute anthropomorphic airs to Leaf’s animals but distinguishes them with subtle shadings of timbre, pitch, rhythm, and tempo. For example, the duck’s reedy bleat is lower and slower than the bird’s high, lilting phrases; these qualities reflect the animals’ differing movements, with the duck’s low-to-the-ground waddle and the bird’s lofty flight.

In tune with the animals’ mercurial movements, Riesman’s sustained use of the Buchla synthesizer produced chance meetings with sounds. His acoustic and electronic improvisations parallel Leaf’s muscle-memory performances with found objects; his process also originated from the random instruments he had at home and at Harvard. ‘While there was basically no budget for the soundtrack’, Riesman (2020, personal communication) recalls, ‘I had the facilities of Harvard’s fledgling electronic studio at my disposal. It had a Buchla synthesizer and two tape machines, so I could do layering by overdubbing onto a second tape machine’. He largely improvised the score using the Buchla, an analog modular synthesizer that has touch-sensitive pads instead of keys – a learning curve for musicians used to keyboard interfaces. ⁶ As Trevor Pinch and Frank Trocco (2004: 44) explain, Don Buchla had sought to break musicians out of the confines of the keyboard’s 12 discrete pitches so that they could generate unheard-of timbres: ‘Buchla wanted something more imaginative as a controller, something that would connect the performer to this new source of sound. No doubt his life-long interest in human–machine communication was part of his motivation.’

Buchla’s ‘box’ – a cabinet studded with knobs, plugs, wires, and touch pads – inspired musicians to create sounds for a new, electrified medium. Composers have embraced synthesizers in science fiction and horror film scores to depict aliens or high-tech sounds; as Philip Brophy (2004: 99) writes of the ‘70s and ‘80s horror film scores of John Carpenter, synthesizers make ‘exceedingly inhuman’ sounds by stimulating ‘electrical energy’. With an interface designed for exploring sounds’ electrical qualities, Buchla’s machine was well suited for composers who wanted to home in on sonic textures: they could combine different modules of oscillators, amplifiers, and sequencers with wires to make their own patches. They could also trigger varied pitches on each touch-sensitive pad. Buchla users had to practice the amount of pressure to put on the pad to produce desired tones and, in the process, they developed a similar kind of muscle memory to Leaf’s displacements of sand grains.

Like many Buchla users, Riesman was no stranger to electronic music; he had done some tape editing before Harvard and took a computer science course while there. He was given the keys to the electronic music studio and composed Sand’s score by night in a few weeks’ time. Much of Riesman’s challenge was discovering the Buchla’s capabilities and how it could work for the score: ‘I was working against the limits of the machine to do something creative, while at the same time knowing that I had to make it fit into the film, too.’ He worked by looking at the film and experimenting on the Buchla. He admired the continually fluid movement of Leaf’s animation, which in part may have inspired his largely improvised approach. But the Buchla’s malleable interface also enticed him to try out sounds by connecting wires between new sets of modules. Similar to Leaf’s painstaking sand animation process, programming the Buchla took a lot of practice and effort. ‘I’d play something and think, “that’s cool, I hope it fits at some point”. The struggle was to get something attractive and usable out of that synthesizer’ (Riesman, 2020, personal communication).

Without the means to bring in a full orchestra, Riesman challenged himself to compose with the Buchla and a few acoustic instruments. He was fascinated by the touch-sensitive pads and the possibility to create gestures that would translate into sounds, such as glissandi (an upward or downward slide between pitches). As he explained, he could not plan out sounds in advance ‘because it never really comes out quite the way you imagined it when you get it on the hardware’. What was needed was a kind of muscle memory on the touch pads that paralleled Leaf’s sand choreography. By first learning how to use the synthesizer, and then following the possibilities it opened up, Riesman could summon sounds suggested by Leaf’s images. ‘I was happy to find something on the Buchla to work for the duck or the bird. I was searching to make something appropriate for a wolf’, he says, describing the process as ‘a kind of desperation’ that also spurred his creativity (2020, personal communication).

For example, Riesman plays the bamboo flute in the film’s opening for the bird’s high-pitched tweets. However, he plays the Buchla when the bird is chased by the wolf – he uses high oscillator sounds approaching the range of a dog whistle to evoke the bird’s fright as the wolf draws near. His variable sound grains shape the contours of the animals’ fear. But he denies the 1:1 match of melody to symbol that occurs in leitmotifs, which leave little room to show the dynamic array of the animals’ emotions. Riesman guides interpretations of Leaf’s film beyond character animation: his sonic textures amplify the granular movements of sand, into which character motifs inevitably disappear.

This is a crucial point in thinking about sound and animation. Grains are ‘the matter of sound’ that we can perceive in multimodal domains; for Luis-Manuel Garcia (2015: 69), who builds on a treatise by musique concrète composer Pierre Schaeffer, ‘aural, visual, and tactile perceptions of grain all refer to the way that small elements form a textured surface.’ For example, the sound of wood against wood paints an image – and a texture – of a woodblock. As Garcia elaborates,

Sonic grain does not only resemble material grain, it also has a direct relation to real-world textures, allowing the ear to perceive physical textures usually associated with haptic experience. Thus, the perceptual mapping between sound and touch that Schaeffer achieves through the use of adjectives such as ‘rough’ and ‘smooth’ is not arbitrary; rather, it is a descriptive tracking of how the perception of texture arises across multiple modes at once. (p. 69)

Garcia’s evocation of real-world textures is helpful to grasp the grains of the bird’s oscillator sound in Sand. Synthesizer oscillators produce audio waveforms – including sine, triangle, and sawtooth – that contain different amounts of harmonics and overtones; a sine wave lacks overtones and sounds like a straight-toned flute or whistle. By combining oscillators, musicians can control the frequency (pitch), amplitude (volume), and shape of the waveform, and they are able to replicate the waveforms of acoustic instruments or to create new sounds. Composers sculpt sound grains. Riesman’s bird sound works by drawing on textures that are familiar to audiences, then combining and defamiliarizing textures by layering (patching) different oscillators. As a result, it has some of the bird-like whistle of the sine tone yet is grittier and more strident than a flute. These sound grains evoke the bird with a flute-like sound but also describe its panic with textures that convey its struggle to escape the wolf.

Much as Leaf’s sweeping motions engender total transformations of sand, Riesman’s sound grains make characters fluid outside of pinned-down leitmotifs. Besides a departure from leitmotifs, his improvised approach also creates indeterminate connections to the animals’ movements that are not locked into the tight synchronization of mickey mousing. Sand is not a Silly Symphony, where xylophones play ascending scales in time with a character’s movement and impart a sense of sonic plasmaticness – sounds that follow contour lines. Unlike the claims of David O Selznick and Chuck Jones, who maligned music in early Disney as heavy-handed duplications of images (see Goldmark, 2007: 6 and Taberham, 2018: 135–136), Riesman’s score loosens the synchronization between movement and music so that sound is more subtly incorporated into the film’s world. There is indeed a moment when Riesman couldn’t seem to resist mickey mousing – ‘there’s a cliché hi-hat cymbal hit when the wolf gets flung’, he recalls (Riesman, 2020, personal communication) – but even then it is mixed with other sounds and is difficult to detect as cliché. But largely, rather than having image in lockstep with musical beats, his approach is closer to the ‘illusion-of-life aesthetic’ (Taberham, 2018: 136) that Disney animators adopted by the mid-1930s, when, as Lea Jacobs (2015: 66) argues, mickey mousing evolved to become ‘the clever synthesis of music and movement, not the musical mimesis of the depicted action’. Riesman will stretch this model even further with his emphasis on timbre over timing, as he enhances the sand’s fluid motions with his own experimental sounds, creating a granular friction between sound and image by shading motion with variable textures.

Elsewhere I have laid out a contour theory of mise-en-scène that relies on spectators’ memories to detect the shapes of melodies that they have heard previously in other social settings, or even in other films (Skjerseth, 2020). With the idea of ‘granular modulation’, I propose that groups of similarly textured sound grains form flexible, contoured motifs for characters. This term riffs on Gilles Deleuze’s (2003: 95) view of modulation in analog synthesizers that ‘establish an immediate connection between heterogeneous elements . . . [and] introduce a literally unlimited possibility of connection between these elements’. Examples of granular modulation include the onomatopoeic sound–image relationships one learns from leitmotifs in Peter and the Wolf or verbalized duck sounds in children’s books. But granular modulation especially occurs when Riesman uses both the bamboo flute and oscillators for the bird – acoustic and electronic instruments with clashing sonic associations. Flutes evoke birds in many classical music pieces, whereas synthesizers signal horror, suspense, or even alien entities. Riesman’s association of both instruments with the bird requires granular thinking: listening closely to the oscillator tones reveals their proximity to a flute-like timbre, but also conjures the bird’s fright instead of peace. A close attention to grains of sound and their granular modulation helps spectators to hear how sound in animation might not always anthropomorphize animals as culturally repeatable cues, as in leitmotifs or mickey mousing. Scores can also evoke animals’ sounds with grainy textures that attune to their different states of being.

Our ability to pinpoint how sounds’ textures and contours flexibly evoke characters in animation is crucial when it comes to the unfamiliar sounds of synthesizers in Riesman’s score. As Deleuze suggests that analog synthesizers create manifold connections between disparate elements, granular modulation can help us perceive links between the Buchla’s strange sounds and everyday noises. When Riesman uses high-pitched oscillators for the bird, our memory of the high contour and similar texture of the bamboo flute helps us to understand how these high pitches still represent the bird, even though the sonic material has changed. Riesman also creates a grainy group of inhuman sounds with the Buchla to distinguish the wolf from other animals. While Riesman doesn’t recall whether he used the Buchla’s ring modulator for the wolf’s inhuman sounds, to my ear this was the likely producer. According to Pinch and Trocco (2004: 38), ring modulators multiply two frequencies to generate only the sum and difference between them: ‘the output frequencies have no musical relationship to the input frequencies. They produce very strange bell-like sounds because of the altered harmonic structure.’ Spectators might recognize this sound in the Dalek voices in Dr. Who or David Bowie’s ‘Scary Monsters’ (1980), where metallic, grating sounds ‘threaten to obscure or even replace the vocal’s human origins altogether’ (Holm-Hudson, 2018: 219). Significantly, ring modulation produces inharmonic sounds, or frequencies that deviate from the tuneful harmonic series: its output of frequencies is alien to its input.

Riesman uses these inharmonic, inhuman sounds to bring a distinctive but highly variable character to the sandy wolf. The granular modulation of a ring modulator or a similar synthesizer manipulation renders sounds metallic, thus adding menace to the wolf’s sounds. An excess of ring modulation is like hitting a metal trash can: percussive strikes on metal create several overtones that are hard for humans to hear (unlike a tuneful violin). Riesman’s performance for the wolf squeezes the tunefulness out of low sounds to signal an invader. His improvisation heightens this sensation: when the duck falters behind Peter, Riesman leaves several beats ominously silent. Before the wolf enters the frame, distorted sounds herald its presence that also accompanied its debut. As spectators remember their familiar contour, these wolf sounds project a threat onto the empty, white background like an acousmêtre – when a sound’s source is unknown to spectators and seems omnipotent and omniscient. ⁷ Unlike a true acousmêtre, the wolf is the known source, but when it enters the frame and chases the duck, it morphs into a ball with hungry eyes. Its shape-shifting powers seem more terrifying when the music crackles with synthetic buzzes, almost like McLaren’s synthetic sound. Shortly after, the wolf snaps up and swallows the duck, which protests with limbs sticking out of the negative space of the wolf’s body. In another grainy, superimposed contour, Riesman performs the bagpipe chanter associated with the duck but filters it through the wolf’s metallicized timbres.

When spectators attune to grainy sonic contours and textures – the granular modulation of both sonic and visual material – they can associate the cacophonous sounds with the wolf’s hungry pursuit as well as the sandy vortices that shape its wake. Whereas spectators have heard metallic rattles before in battle sounds, horror films, and the like, in Sand those general associations are heightened by the synthesizer’s alien frequencies that jar with acoustic ones. Riesman’s mixtures of acoustic and electronic sounds create grainy contingencies. The loud twangs when the cat struggles inside the wolf’s stomach, for example, are prepared piano sounds, where metallic objects placed on piano strings muffle resonances when players strike corresponding keys. ‘I was messing around with that kind of sound’, Riesman (2020, personal communication) remembers. ‘I would’ve prepared it at home and brought it in to mix into the score.’ He searched for any instrument – and any sound, on the Buchla – to suit the motions and emotions of Leaf’s sand paintings. In tandem with Leaf’s care to touch and be touched by the sand, Riesman improvised and refined sounds in a labor of grain they both shared.

From grain to world

Sand and sound enrich Sand’s world, but not by accident. Leaf and Riesman molded these raw materials in flexible ways that transiently ground figures and figure grounds. Their improvisations, informed by skilled practice and muscle memory, evoke characters but also their affects and surroundings. Leaf’s use of sand for both human and animal, friend and foe, stresses the material connections between beings and environment. Her art exemplifies what Ursula Heise (2014: 303) calls animation’s ‘refusal to treat either natural or human-made environments as mere inert materials and . . . its insistence that these environments are alive and populated by all manner of nonhuman agents’. Riesman crafts motifs for characters that dissolve or disappear to equally evoke the grains that animate them. His score has both the subtlety and terror of The Birds (Alfred Hitchcock, 1963), which uses an electronic instrument that predates modular synthesizers, the Mixturtrautonium, derived from the 1928 trautonium (Pinch and Trocco, 2004: 54). ⁸ Attention to granular modulation – of grainy visual and sonic materials – extends to a range of media, from experimental animation to Hollywood cinema, and from familiar acoustic sounds to novel electronic ones.

This study of Sand highlights the malleability of sound and vision especially present in experimental animation, and the ways in which granular modulation emphasizes audio-visual friction and interplay. Whether in the synthetic sound experiments of Oskar Fischinger and Norman McLaren, or the sound collages of Robert Breer, Lewis Klahr, and Jodie Mack, seeing and hearing grains helps us to identify the real-world materials that make up seemingly ephemeral forms. Granular modulation also stretches our imaginations about how sounds index materials. As Frank (forthcoming, 2021) argues for sonic indexicality, ‘The sounds we hear, no matter how alien, have a potential and particular referent.’ But sounds can trick us when they rub against the fabric of the visually animated world, which is made of a different material. Whether sounds are produced with Foley, acoustic or electronic instruments, or dialogue, sounds have complex indexical relationships to images – such as famous voice actors who impede the audience’s ability to suspend disbelief for the separate identity of an animated character. The grains of voices and instruments index their worlds of origin while simultaneously pointing to emotional, relational, and environmental effects onscreen. For Rebecca Coyle (2010: 5), sound can ‘order the world depicted in film in a different way – one that both references (and simulates) “reality” and yet surpasses it, presenting scenarios that we recognize as simultaneously other and the same’. The friction between sound and vision stems from such blends of reality and fiction, of source and transformation.

Sound in animation, then, has the potential to bring drawings to life by virtue of its material friction – and symbiosis – with the visual world. The textures of sound and image create material impressions for spectators that retain a semblance of reality and stretch it just enough to create new ways of experiencing the world. By homing in on the granular modulation of textures, grains, and contours, we can witness new meetings of worlds that nevertheless keep differences intact. Moment to moment, grains of visual and sonic forms collide, improvising new arrangements out of seemingly moored materials.