The Effects of Temporal Action-Sound Congruence on Evaluations of Conductor Quality

Introduction

Vision serves a fundamental role in the human experience of music performance. While its propensity for broad audio/visual influence is well established (McGurk & MacDonald, 1976), the feature-rich context that music presents highlights the pervasive universality of its influence and the wide variety of manifestations it takes. Examples of this influence are as varied as expressivity (Dahl & Friberg, 2007; Davidson, 1993), tone duration (Schutz & Lipscomb, 2007), and performance quality of individuals (Tsay, 2013) and groups (Tsay, 2014). Even within these, vision’s influence extends to both the mechanical (Fendrich & Corballis, 2001) and affective or expressive (Vines, Krumhansl, Wanderley, Dalca, & Levitin, 2006) within musicians’ performance behaviors. The observed perceptual interaction of sound and sight is situated within a narrowed view of embodied music cognition (Leman & Maes, 2014), where action’s effect on music is central to the perception and understanding of it. Interestingly, the overwhelming majority of music performance bolsters this view by providing a directly linked action-sound pairing of performer gesture to instrument response save for one case: that of the conductor.

Seen as central to the function of instrumental ensembles, the conductor is unique in considerations of music as their sole offering is non-auditory. Through gesture alone, they convey both the affect and temporal coordination of ensemble musicians, leading them by their deployment of motions that are thought of as alternately “navigational” or “evocative” (Leinsdorf, 1981), with the latter steering affect and the former guiding precision. While some offer that conductors are, often as not, “selected from among the worst pianists to be found, or [those that] cannot play the pianoforte at all” (Berlioz, 1926, p. 257), this individual’s contributions are universal within Western instrumental ensemble performance, lending credence to the conductor’s role in the function of the ensemble. To wit, former conductor of the New York Philharmonic Alan Gilbert offers that “There is a connection between the gesture, the physical presence, the aura that a conductor can project, and what the musicians produce” (2012). This link between conductor and performer is further bolstered by the physical location of the conductor—often directly in the view of all performers at the front of the ensemble—which allows performers to easily see their contributions.

Despite their soundless contribution, the conductor’s gestures are intended to inform a wide range of musical function and sonic outcomes within the ensemble (Rudolf, 1980). Pedagogically focused conducting texts are explicit in centralizing the conductor’s coordinative role and treating expressivity as a necessary extension (see Pasquale, 2008 for a review). Often, this is presented as “mere time-beating is not enough; the appropriate gesture for each musical expression must be mastered before we can speak of conducting” (Rudolf, 1980, p. xiv). While the communication of time can be viewed pejoratively, its necessity is embedded in the requisite nature of its hierarchical position articulated by Rudolf. Interestingly, this implicitly graduated bifurcation of time (e.g., navigation) into affect (e.g., evocation) in conductor gesture is also supported by empirical reports. The temporal coordination of the ensemble performers (D’Ausilio et al., 2012; Hilt et al., 2013; Luck & Nte, 2008; Luck & Sloboda, 2007; Luck & Toivianien, 2006) as well as the communication of affect within the work performed (Kumar & Morrison, 2016; Morrison, Price, Geiger, & Cornacchio, 2009; Morrison, Price, Smedley, & Meals, 2014) have both been shown significant influence by the conductor’s gesture.

Expressive gesture influences perception and subsequent evaluation of music performance with near universality (Dahl & Friberg, 2007; Platz & Kopiez, 2012). Noted at the level of the individual musician (Davidson, 1993; Tsay, 2013; Vines et al., 2006), it has also been identified in the gestures of instrumental (Montemayor & Silvey, 2019; Morrison et al., 2009, 2014; Silvey, Wacker, & Felder, 2017) and choral (Morrison & Selvey, 2014; Napoles, 2013) conductors. The ability of these soundless gestures to filter and influence perceptions even in the absence of supporting sonic information bolsters the statement that “the visual component is not a marginal phenomenon in music perception, but an important factor in the communication of meaning” (Platz & Kopiez, 2012, p. 75) as it applies to this context. Within and around this, however, the tendrils of ensemble coordination are tightly woven in the activity of the conductor.

Ensemble coordination is effected by the use of a prescribed gestural vocabulary that prominently features points of elision between intersecting horizontal and vertical vectors (Luck & Toivianien, 2006). Within conducting literature, each point of elision is referred to as an ictus (Green & Gibson, 2004; Rudolf, 1980), serving as the indicator of isochronous pulse within the conductor’s gesture (Clayton, 1986). Performers extract temporal information through visual beat induction, where temporal signals are extracted from a combination of gestural acceleration and deceleration paired with changes in the value of parameters that define the gesture’s trajectory (Luck & Sloboda, 2008). Put another way, performers extract time-bearing information from both the changes in direction, as well as the rate of that change, within the conductor’s patterns. Once encoded, this serves as a point of reference for each performer’s analytical and future-oriented attending (Jones & Boltz, 1989), allowing for the coordination of individual-level performance of complex musical material with others via the conductor.

This process joins a range of coordinative interperformer signaling present in small groups (Chang, Livingston, Bosnyak, & Trainor, 2017; Glowinski et al., 2019; Schiavio & Høffding, 2015) and in larger ensemble configurations (Goebl & Palmer, 2009; Hilt et al., 2013; Timmers, Endo, Bradbury, & Wing, 2014) that complement the conductor’s communication. D’Ausilio and colleagues (2012) investigated leadership in orchestral performance through analysis of nonverbal communication along both conductor-to-musician and musician-to-musician channels. Using audio- and motion capture data, they investigated sonic and kinematic interactions both between performers and with the conductor, identifying a temporally infused “driving force” of each in a given performance. From this they noted a flexibility between individual musicians and the conductor, finding an emergent hierarchy in higher-rated performances where the driving force of the conductor was more salient. In this, the authors noted that “aesthetic appreciation of music orchestras’ performance was associated to the concurrent increase of conductor-to-musician influence and a reduction of musician-to-musician information flow” (p. 4). The ensemble, then, presents a dynamic communication network within which the conductor’s gesture is highly salient to, but by no means exclusively responsible for, musical coordination.

This musical ecosystem represents an instance of social, multi-focal entrainment. Rhythmic detection, production, as well as phase and period correction (Clayton, 2012; Levitin, Grahn, & London, 2018; Philips-Silver, Aktipis, & Bryant, 2010; Repp, 2005) are present throughout the ensemble’s activities, highlighting the potential for coordinative flexibility rather than strict synchrony. Taken alongside the findings of D’Ausilio and colleagues (2012) and bolstered by related work by Glowinski and colleagues (2019), the externalities of temporal detection, production, and correction actually represent two overlapping systems of entrainment: one between musicians and the other between the conductor and each musician.

These complementary and coexistent systems represent an example of social (Philips-Silver et al., 2010) entrainment existing in an asymmetrical, or one-to-many (Clayton, Sager, & Will, 2005), relationship that is crossmodal in its function (Phillips-Silver & Trainor, 2007). The balance of that complexity varies between ensembles, performances, and sometimes within given works (D’Ausilio et al., 2012; Glowinski et al., 2019; Luck & Toivianien, 2006). Musicians’ ancillary, or non-sound-bearing, gestures vary both in quantity and complexity depending on the musical context in which they are deployed, as highlighted by Glowinski and colleagues (2019). Similarly, conductors have been anecdotally observed to alter their gestural vocabulary depending on the size of the group they are conducting, often adhering to an inverse relationship between complexity and size of ensemble (Acklin, 2009; Leinsdorf, 1981; Rudolf, 1980). Interestingly, the resultant flexibility within the entrained system appears to take on unique qualities across some performance contexts, as the congruence of the conductor to ensemble action-sound relationship is at times seen as, in fact, incongruent (Bell, 2004).

This phenomenon has been noted by many performers, audience members, and conductors (Bell, 2004; Bennett, 2017; Johnson, 2014), but has received limited scholarly or empirical attention. The presence of variability in synchronization with conductor gesture (Luck & Sloboda, 2008) suggests that correction of phase and period are present in this relationship (Repp, 2005). While anecdotal accounts speak to flexibility contrafactually, as conductor-to-ensemble relationships are not identified as qualitatively orthogonal, examples of severe misalignment seem to produce a negative or incredulous response (Arhombus, 2014). Given the role and importance of the conductor’s rhythmic signaling and entrainment’s ability to engender emotion (Trost, Labbé, & Grandjean, 2017), this presents an interesting case for consideration. Mirrored activities as simple as tapping fingers (Hove & Risen, 2009) or the coordinated movement of an appendage (Sparenberg, Topolinski, Springer, & Prinz, 2012) produce preference between individuals. Evidence also suggests that organized dance and movement (Dean, Byron, & Bailes, 2009; McNeill, 1995) report generally similar outcomes, even in group settings. When viewed alongside the development and persistence of crossmodal integration of body movement and auditory encoding (Phillips-Silver & Trainor, 2007; Spence, 2011), and the propensity toward extracting coordinative information from gesture (Luck & Sloboda, 2007), the potential for temporal perturbations to influence evaluations of conductor efficacy becomes salient.

Anecdote suggests that the temporal incongruity in conductor gesture to ensemble response is common in orchestral settings (Roberts, 2017), but that it is a variable property that depends on the musical content being performed and the preferences of the conductor themself (Johnson, 2014). Given the flexibility inherent in entrained systems, especially those as large as a musical ensemble, and the propensity for entrainment to promote affinity (Sparenberg et al., 2012) and preference (Hove & Risen, 2009), temporal action-sound congruence outside a range of affordance may impact evaluations of conductor quality, regardless of their skill in conducting. To this end, we selected a large ensemble context where synchrony is almost exclusively the expectation in conductor-to-ensemble coordination (e.g., wind band), and posed the following research question: “Does the temporal displacement of conducting gesture from its sonic correlates have any bearing on evaluations of conductor efficacy and quality in a large ensemble setting?”

Method

Stimuli

Stimuli were taken from intact performance videos collected from five experienced conductors (2 female). In all videos, conductors appeared from the perspective of the performers, with a variance of approximately 15 degrees to the right or left of ensemble center. The similarity of evaluations of conductor gesture from the performer’s perspective to the audience’s perspective has been demonstrated in similar studies (Price & Mann, 2011). Across all videos, performing musicians and audience members are viewable in the fore- and background, respectively. Video quality across collected samples met or exceeded baseline requirements for high definition (720p HD, 720 x 480 pixels). Video artifacts were addressed using color correction and cropping where they did not detract from the focus of attention (e.g. automatic white balancing and cropping to center conductor in viewable field). Audio resolution for all videos was determined to be 16-bit WAV (CD quality) or better and represented professionally recorded audio captured during the performance in question.

Two musically cohesive excerpts per conductor (fast/slow) were chosen from the collected videos (fast: M_fast = 153 beats per minute (BPM), SD_fast = 15.4 BPM; slow: M_slow = 76 BPM, SD_slow = 27.18 BPM). These 10 samples (see Table 1) were selected for their consistent tempo throughout each excerpt and for the balanced presence of consistent and clear gestural communication within a fully intact and coherent musical phrase, although tempo variance among slow excerpts was greater than that of fast. Consideration was also given to the presence of contextually appropriate, expressive gesture shown by conductors. Excerpt length (M = 29.2 seconds) was consistent with previous research in conductor evaluation (Morrison et al., 2009, 2014).

OPEN IN VIEWER

Table 1.

Works selected and performance information.

Conductor and Excerpt	Work Performed	Composer	Tempo (BPM)
1.A (Fast)	Fantasia for Euphonium	G. Jacob	142
1.B (Slow)			89
2.A (Fast)	Canzona	P. Mennin	144
2.B (Slow)	Prelude and Fugue in G minor	J. S. Bach / arr. Moehlmann	58
3.A (Fast)	Flourishes and Meditations on a Renaissance Theme	M. Gandolfi	150
3.B (Slow)			118
4.A (Fast)	Firefly	R. George	150
4.B (Slow)			60
5.A (Fast)	Molly on the Shore	P. Grainger	180
5.B (Slow)	Old Wine in New Bottles	G. Jacob	55

Intact stimuli videos were manipulated within Adobe Premiere Pro (Adobe, Inc.). Excerpts were extracted and manipulated to allow for experimental manipulation through the separation and adjustment of audio and video tracks. This created a set of excerpt iterations encompassing an intact sample with no manipulation, a ±30% audio/visual offset condition, and a ±15% audio/visual offset condition for a total of 50 manipulated stimuli. The degree of offset reflected consideration of previous research highlighting similar offset procedures in conducting task synchronicity (Luck & Toiviainen, 2006), synchrony in music performance (Vatakis & Spence, 2006), and error detection in musical performance (Danz & Janyan, 2009). Offsets in the current study were chosen to allow for potential differences to emerge within given tempo conditions (fast/slow) rather than in a fixed offset relationship (as used by Vatakis & Spence, 2006 and discussed by Repp, 2005) where excerpt tempo might be a less salient variable. Much of the existing literature where static offset values were employed focuses on temporal order judgments (see Repp, 2005), a related but not identical task to the focus of our investigation.

These manipulations did not alter content quality or speed, but rather shifted the audio to either a lead (-) or lag (+) condition as compared to the intact video (see Figure 1). This temporal distance, measured in milliseconds (ms), was determined by analyzing the tempo of each excerpt and determining the quantity of offset required (M_fast = ±138 ms (±30%), ±69 ms (±15%); M_slow = ±262 ms (±30%), ±131 ms (±15%)). For clarity, we labeled stimuli by using the position of the audio stream related to the video where “minus” indicated audio positioned before video and “plus” indicated audio positioned after video.

OPEN IN VIEWER

Figure 1.

Audio/video offset orientation.

The generated offsets were theorized to fall within a range of manipulation that would remain above the threshold of just noticeable difference (JND) by capitalizing on the ensemble experience of our participants (M = 6.1 years) and their familiarity with the multiple affordances present within ensemble performance (e.g., co-performer entrainment, conducting practice). While the offset selection exceeds the findings of others (Drake & Botte, 1993; Large, 2008), and may be seen to extend beyond what Weber’s law suggests regarding a perceivable change in stimulus intensity, it is important to remember that in many settings participants may experience a priming effect due to the nature of the task undertaken. In this experiment, musically experienced participants were asked to evaluate conductor quality rather than detect synchrony—and the flexible nature of conductor-to-ensemble timing paired with the complexities of ensemble performance constitute a perceptual affordance that is otherwise untested in this specific setting. Additionally, the complex web of communication in naturalistic performance combined with the crossmodal nature of the excerpts supports a perceptual adaptation of temporal incongruity similar to that reported by Keetels and Vroomen (2008), as well Large’s observation that “there is a tendency for pulses to gravitate toward event onsets in a way that produces (approximate) synchrony when a stimulus rhythm is purely periodic” (2008, p. 193).

Conversely, stimulus manipulations fall outside the critical range of crossmodal integration reported by Fendrich and Corballis (2001). In their study, temporal fusing of combinations of visual flashes and auditory clicks and flutters presented in close temporal proximity to one another was observed below a threshold of 50 ms. Mean offset values across present stimuli were designed to fall outside this range. We theorize that this provides participants with variance that is meaningful enough to elicit a significant effect in evaluations yet not so severe that manipulations are perceived as a product technical error.

A 45-second “Please Respond” prompt was added to the end of each excerpt and each was rendered with high-definition audio and video (H.264, Apple, Inc.). From these stimuli five fully crossed orders were organized where no conductor, offset value, or excerpt condition (fast/slow) was seen consecutively. Stimuli were individually uploaded to YouTube (Alphabet, Inc.) as unlisted videos. ²

Procedure

Via Psychdata.net, respondents viewed an instructional video and were asked to complete a sample evaluation of a conductor not included in the created stimuli. To obscure the emphasis of the experimental manipulation, embedded videos were presented with 5-point Likert-type scales (Poor to Excellent), asking the participant to rate the conductor, ensemble, and overall performance (see Figure 2). In keeping with related work on conductor evaluation (Morrison et al., 2009; Morrison & Selvey, 2014; Price & Chang, 2005), the participant prompts were intended to foster general assessments in order to capture authentic responses on the part of these experienced musicians. It was theorized that introducing greater specificity to evaluations might increase a nascent likelihood for error detection on the part of the conductors and thereby nullify the aim of the temporal manipulation.

OPEN IN VIEWER

Figure 2.

Sample participant interface.

The survey instrument allowed for the tracking of elapsed and active time by each participant (M = 20.52 minutes, SD = 4.85 minutes), which we used to identify participants who did not complete their session (n = 2). Participants were assigned to each condition randomly after indicating agreement to the IRB protocols, and completed the survey on desktop or laptop computers within each university’s computer lab. As YouTube (Alphabet, Inc.) is a web application that renders equally well on the vast majority of available browsers and systems, hardware or software information was not collected for analysis.

Results

A significant main effect was found for ensemble evaluation such that ratings for the slow tempo condition (M = .124, SE = .042) were significantly lower than the fast tempo condition (M = .308, SE = .036) (F(1,1090) = 11.25, p < .001, η _p ² = .04). The main effect for offset condition was not significant F(4,1090) = 2.35, p > .05. However, the interaction effect between tempo condition and offset condition was significant F(4,1090) = 6.89, p < .001, η _p ² = .02, and post hoc comparisons using the Tukey HSD test indicated that the mean score for slow excerpts in the +30% offset condition (M = -.190, SE = .103) was significantly different than those in the -15% slow (M = .288, SE = .085), intact slow (M = .273, SE = .107), -30% fast (M = .305, SE = .085), intact fast (M = .314, SE = .078), +15% fast (M = .471, SE = .075), and the +30% fast (M = .421, SE = .079) offset conditions. The pervasiveness of this interaction is understandable, however, since ensemble repertoire was not controlled for in the study design and both fast (M = 152 BPM, SD = 16.21 BPM) and slow (M = 76 BPM, SD = 27.1 BPM) tempo conditions varied by conductor and piece.

There was a significant main effect for conductor evaluation such that all offset conditions (audio- and video-lead) were rated lower than intact stimuli (F(4,1090) = 4.668, p < .001, η _p ² = .02). Post hoc comparisons using the Tukey HSD test revealed a significant difference between the intact (M = -.087, SE = .075) and the +30% (M = -.467, SE = .078) conditions. The main effect for tempo condition was not significant (F(1,1090) = .004, p > .05), and there was no significant interaction effect between offset condition and tempo condition in conductor evaluations.

As can be seen in both overall mean participant ratings (see Figure 3a) and participants ratings by excerpt condition (Figure 3b), an inverted U shape exists broadly across evaluations of both conductor and ensemble, where ratings were lower in any offset condition and conductor ratings were lower than ensemble ratings. Differences in both conductor and ensemble evaluations were observed even when considering the idiosyncratic styles and gestural magnitude demonstrated by each individual conductor.

OPEN IN VIEWER

Figure 3.

a-b. Standardized participant ratings (a) overall and by (b) tempo condition (fast/slow).

Discussion

Conducting represents a unique element of music performance, as its gesture does not have a direct sonic consequent. Instead, gesture serves as a coordinating signal from conductor to ensemble that conveys both temporal and affective information. Within this, musicians entrain to a number of rhythmic signals in ensemble performance (D’Ausilio et al., 2012), with the conductor’s central location allowing for their communication to remain salient to all performers. From that, this signaling combines with other layers of cooperative information exchange (e.g., complementary and imitative) to leaven the predictive qualities of musical joint action (Hilt et al., 2013). The conductor provides temporal information to the ensemble and these time-bearing gestures share a flexible relationship to the ensemble’s activity and output (Bell, 2004; Johnson, 2014), although there exists a clear directionality to the embedded information from conductor to ensemble (Hilt et al., 2013). As entrainment has been shown to engender preference (Hove & Risen, 2009) and affinity (Sparenberg et al., 2012), we sought to determine the aesthetic effects of manipulation to temporal action-sound congruence in conductor gestures.

Our results indicate that differences exist within both conductor and ensemble evaluations where gestural action-sound congruence has been manipulated, although the qualities of these differences are themselves different. Interestingly, a broad view of our data revealed that the majority of offsets were rated lower regardless of orientation of action to sound (e.g., video- or audio-lead) (see Figure 3a). Exceptions to this are revealed through division of ratings by tempo condition (e.g., fast and slow), but the generally inverted U shape to the distribution—especially with regard to slow excerpt evaluations—is intriguing. In this same vein, no participants reported their videos appearing to be altered, suggesting that the manipulation of the stimuli fell within an acceptable range of affordance for the participants, who are experienced ensemble musicians (M_XP = 6.10 years). While this range of affordance varies from other literature on participants’ identification of “just noticeable difference” (JND) (Drake & Botte, 1993), the degree to which ensemble performance integrates multiple, multimodal streams of communication opens the possibility for a wider range of affordance in this setting (Hilt et al., 2013).

In conductor evaluations, all offset conditions were rated significantly different than intact video, with the majority rated lower than intact. The same did not hold true for ensemble evaluations, although a similar, inverted U shape appears in slow excerpt ensemble evaluations that generally mirrors their analogous conductor evaluations (Figure 3b). That evaluations of slow excerpts were generally rated lower than fast stands to reason, as offset values were greater due to the experimental protocols (as suggested by Large, 2008). A complicating aspect to this, however, is that the accompanying audio also features material of a longer duration across the musical spectrum (e.g., note duration, interonset interval, harmonic motion), which may serve as a mediating factor in these evaluations. Additionally, a consistent trend for conductor ratings to be lower than those of the ensemble pervaded evaluations, which is notable in its own right and bears further investigation, potentially building upon the framework offered by Kumar and Morrison (2016).

These findings raise two critical questions. First, would the response profile between slow and fast excerpts be more similar if each was adjusted using the same absolute (rather than relative) magnitudes of deviation? If so, this would reveal either (1) a potential duration-general window of “affordance” within which observers reconcile disjunct audio and video music performance information (as offered by Drake & Botte, 1993), or (2) a “blind spot” that lies outside the range of discernment of decontextualized stimulus but may be facilitated by a more content-rich perceptual environment, an eventuality supported by work in crossmodal perceptual binding (Engel, Roelfsema, Fries, Brecht, & Singer, 1997).

Second, given a direct comparison between intact and adjusted excerpts, would a more apparent positive response to intact performances emerge for fast excerpts? Likewise, would a correspondingly more pronounced positive response emerge for slow excerpts? It is difficult to determine this from the data at hand, as stimuli were not organized in a fashion to centralize direct comparison, instead focusing on the influence of offset quantity. In contrast to fixed offset values, such a finding would confirm that the adjustments used in the present study were sufficiently discernible and that the responses observed reflect an active accommodating of temporal misalignment.

Ensemble performance presents us with a complex, multimodal communication network in which conductors are a critical, influential, but not isolated node. The results of this study suggest that the temporal congruence of their gesture bears consequence not only in their coordinative role, but in evaluations of quality in their performance, as well as the ensemble’s. Additionally, these results reinforce existing findings within synchrony preference and attendant prosocial behavior and reward center activity. Although this study adds broad support to the crossmodal influence of sight and sound in musical contexts, further research is needed to foster a more robust understanding of the interactions that dominate the dense mesh of activity, communication, and interaction within ensemble music.