Using time series analysis to evaluate skin conductance during movement in piano improvisation

Abstract

During musical improvisation, performers’ skin conductance (SC, a measure of psychological arousal) may respond to movement and to events whose timing is beyond control. SC has not been studied in these difficult conditions. Our purpose was to establish a procedure and analysis that would permit the meaningful use of continuous SC measures while pianists play. Consequently, two case studies of SC during piano performances develop an effective method. SC was measured at the left ankle and movement was monitored nearby. Two musicians performed manipulations of movement (flexing legs, hand motion), and performance content (playing scales versus improvisation) and type (actual, silent and imagined). Time series analysis modeled SC in relation to supplied improvisational referents. We could interpret SC during performance, provided that we accounted for the impact of movement. We detected genuine SC changes around moments of transition between musical segments; these could reflect the mental effort of planning and generating music. In a subsequent validation study, we demonstrated the applicability of our method for SC analysis to performances by nine professional piano improvisers.

Keywords

movement music performance skin conductance time series analysis

We wished to establish a method for the informative use of continuous skin conductance (SC) measures while pianists improvise or interpret notated music. Body movements create major signals in the skin conductance detection systems, especially when measured on the forehead or arms/hands (Ham & Tronick, 2008; MacIntosh, Mraz, McIlroy, & Graham, 2007). This article develops a solution to this problem.

We use SC to refer to the continuous measure of skin conductance, and SCR (SC responses) to refer specifically to transient SC events that may be elicited by external stimuli or by psychological processes. SC¹ is an expression of electrodermal activity closely related to the operation of sweat glands (Benedek & Kaernbach, 2010) and to arousal and attention (Mandryk & Atkins, 2007). It is an autonomic nervous system function which is not under conscious control (Brunsdon & Skinner, 1987; Gruzelier & Venables, 1973). SCR was formerly called the galvanic skin response. We first describe previous studies on SC and music; then the difficulties associated with current methods of measuring SCR. We next introduce the appropriate statistical technique, time series analysis; and finally detail our aims.

SC studies are useful in the context of decision-making (such as interpersonal appraisal, Kylliäinen & Hietanen, 2006). Thus SC measures are frequently interpreted as an index of not only emotional response (Khalfa, Peretz, Blondin, & Manon, 2002; Mandryk & Atkins, 2007), but also task effort and attention (MacIntosh et al., 2007; Nakahara, Furuya, Francis, & Kinoshita, 2010).

Many studies have concerned physiological responses to music listening (reviewed by Bartlett, 1999), where the participants can avoid movement. However, studies of psychophysiological change in performing musicians, where such constraint is difficult if not impossible, are rare. Exceptionally, Nakahara et al. (2010), Nakahara, Furuya, Obata, Francis, and Kinoshita (2009), and De Manzano, Theorell, Harmat, and Ullén (2010) made cardio-respiratory measures during piano performance. SC is a notable exclusion from their work.

It is normal to distinguish between on-going “tonic” SC level and slow changes therein, which may not be informative, and rapid “phasic” changes (SCR). Phasic changes can be externally or internally evoked. For example, moderate acoustic loudness changes can evoke SCR. SCR may also be due to psychological processes in performers. Phasic responses usually peak within about 5 seconds and decline to the on-going (tonic) baseline within about 10 seconds (Benedek & Kaernbach, 2010; Gruzelier & Venables, 1973).

There is continuing debate as to satisfactory analytical methods for assessing both tonic levels and SCR (Levinson & Edelberg, 1985), even in relation to precisely-timed stimuli (e.g., Alexander et al., 2005; Bach, Flandin, Friston, & Dolan, 2009; Lim et al., 1997). Traditional “peak picking” (identification of phasic responses by visual inspection) in an SC trace has been criticized as arbitrarily selective. This approach may be made more consistent by defining a criterion for an SCR. However, most work has assumed that the time of onset of stimuli is known, and that the SCR response has a canonical temporal form (Alexander et al., 2005; Lim et al., 1997). Bach et al. (2009) achieve the best modeling of SC yet by convolving the temporal pattern of the stimuli with a canonical SC response, to form a model for the SC data. This permits analysis of SC peaks even when they overlap in time, and improves discrimination.

Linear convolution is unfortunately unsuitable to address SC during keyboard performance, because the influential factors (including movement) are not defined in advance, nor are they controllable events. In addition, the approach of Bach et al. (2009) disregards the impact of serial correlation in the data. As is long known (Brunsdon & Skinner, 1987), SC shows substantial serial correlation, like most continuous physiological measures. Serial correlation renders data unsuitable for most common forms of statistical analysis, which rely on independence of successive samples. This is neglected in much literature on SC, which is thus possibly compromised. We use the highly-developed methods of Time Series Analysis (Hamilton, 1994) to deal with this problem in music perception studies (Bailes & Dean, 2012; Dean & Bailes, 2010b;). Here we show that time series analysis can also be used to assess the impact of movement on SC, while separately distinguishing SC changes that originate from other factors. We also show that time series analysis can reveal differences between SC in successive musical segments of a performance, and across the performer-generated transitions between those segments. Time series analysis treats SC without assuming a canonical response form or requiring removal of tonic changes (by high pass filtering in Bach et al., 2009).

Our context is the need to apply a continuous psychophysiological measure to keyboard players actually during performance. Our longstanding interest in improvisation (Smith & Dean, 1997) now concerns several psychological aspects of the process (Bailes & Dean, 2009; Dean, 2009; Dean & Bailes, 2010a). For example, it is likely that mental imagery is involved in the planning stages of improvisation, and again during realization (Bailes, 2009), while decision making is required at the points in an improvisation at which musicians create structural or affective contrast. These psychological processes may be reflected in psychophysiological measures.

We also needed to consider the impact of auditory stimuli themselves on SC during performance: both the possible influence of exogenous events such as an auditory command to move to a new section of an improvised piece; and the possible influence of hearing oneself play during the performance, given that loudness change can influence SC in passive perception experiments. For example, short acoustic stimuli can produce an SCR (Bach et al., 2009; Lim et al., 1997).

Having established an analytical approach that was useful for our case studies, we undertook a validation study with nine professional pianist-improvisers, to test the appropriateness of the approach.

Methods

Participants

The authors were the pianists in our two case studies, having no previous experience of performing with an SC detector. Both are extensively trained in classical music, with Ollen Musical Sophistication Indices of 952 and 983 (index ranging from 0 to 1000, where a score greater than 500 indicates “more musically sophisticated,” Ollen, 2009), and one is also an expert in improvisation. When necessary, they are distinguished as A and B.

In the subsequent validation study, we studied nine professional keyboard improvisers (not including the authors) unaware of the purpose of the studies in which they engaged. There were three women and six men.

Apparatus and data acquisition

The participants performed on a Yamaha Disklavier 3 MIDI grand piano, seated in a quiet studio. The physiological and movement measures were made under typical conditions, except for the choice of body position for the detectors to minimize the likely movement experienced at the site. Skin conductance was recorded via a pair of dry electrodes, with a surface of 2.5 cm², placed at the lateral and medial hypomalleolus (left foot) once the skin had been washed in soapy water, rinsed and dried. The bipolar electrode plates were held against the skin with Velcro straps, and they fed into an amplifier connected to a PowerLab 16/30 system (ADInstruments). The electrodes were excited by a constant-voltage AC current of 22mV_rms at 75 Hz. A Piezo transducer was also strapped to the top of the same foot to detect ankle surface movement. The pulse transducer functions by converting force against its surface into an electrical signal, reflecting muscular effects on the skin surface. These are more closely related to SC and sweat gland activity than is stiff translation of the leg, which is more directly assessed by accelerometry. We refer to the pulse transducer data series as the “pressure” series.

SC and the signal from the pulse transducer were recorded continuously in Chart V5 running on a Lenovo PC (Windows XP), which was situated in an adjacent recording booth, and connected to the PowerLab by USB. The skin conductance was measured at a 40 µS range. Physiological data were sampled every 50 ms: SC (500 mV µS) and pressure (5 V), together with the amplitude (5 V) of microphone signal.

Procedures

Each participant performed a series of trials at the piano. Each trial (other than a free improvisation) comprised an ABA structure: [cue: begin task] 1 min [cue: change task] 1 min [cue: change back to first task] 1 min [cue: stop]. The participant was instructed to avoid moving the left foot or leg, and to use the piano pedals with the right foot only.

Baseline conductivity was measured once the electrodes were strapped into place, and while the participants were still and relaxed. Performances were videoed (two cameras) and the audio was recorded, as well as the performed MIDI data, containing information about note, key press velocity, onset and offset times, and right pedal use.

For the case studies, a recording of a male speaker (not one of the authors) saying the word “change” was used as a section cue to the performer, to begin, change, or stop the performance. The sequence of required events was known to the performer at the outset of each performance. Spoken verbal cues were used in the case studies to avoid providing another musical sound, and to avoid a requirement to watch a screen for instructions, contrary to normal playing conditions.

For the validation study, the experiment firstly and lastly requested free improvisations (no referent); while another eight improvisations responded to three part referents describing musical processes such as sparse-dense-sparse or tonal-atonal-tonal provided ahead of the performance. There were no spoken instructions during a performance. Each of the 10 improvisations was requested to be about 3 minutes. We used one block within the overall experiment for the purposes of the validation study.

Analysis

Time Series Analysis (TSA) was undertaken to investigate the influence of external factors such as movement on SC, and to characterize the autoregressive properties of SC under different conditions. We have published a simple introduction to TSA in the context of music cognition (Dean & Bailes, 2010b) and also provided a glossary (Bailes & Dean, 2012). We used standard methods to discriminate between different time series models of a process. All models used here are autoregressive moving average models which assess the influence of chosen exogenous (independent) variables (X): ARMAX models (Enders, 2004; Hamilton, 1994). In this article, movement was the main X, but once we had taken account of this, the influences of other factors (such as loudness, psychological processes) could be considered as additional X.

TSA normally requires that the series under analysis show constant mean and variance: what is called weak stationarity. To this end, in several cases we used the procedure of “differencing” to convert non-stationary into stationary series. “Differencing” creates a new series one shorter than the original whose values are the differences between the successive original values. We selected TSA models by minimizing the Bayesian Information Criterion (BIC), and models were only accepted if they generated white noise residuals, a necessary criterion for secure statistical significance estimates. BIC value differences between models of the same series which are less than 4.6, correspond to Bayes Factor (exp[0.5 × deltaBIC]) values less than the 10 recommended for “strong” discrimination; these are considered “ambiguous” (Wasserman, 2000). In such cases, the Likelihood Ratio test was applied if the models being compared were nested. Unless either the Bayes Factor or the Likelihood Ratio test implied a significant preference for the simpler model, the more complex, with higher log likelihood (but sometimes slightly worse BIC), was accepted. This was to establish models for different segments that were similar in form to each other, facilitating further comparisons.

In the case studies, when segments were distinct as judged by the extent of SC change, further analyses were conducted by TSA to determine their dynamic properties. In these cases, cross-validation was also used (see below). Models to be shown here were required to be highly significant, and to have most or all of the individual parameter coefficients significant at p < .05.

Autocorrelation in SC

It is fairly easy to detect the presence of serial correlation in a series of successively obtained measures of some parameter (for example, perceptual ratings of music (Schubert, 2001; Schubert & Dunsmuir, 1999). When these are plotted sequentially, a series of independent measures will oscillate around its mean value, with no more than one or two successive values showing a directional trend. Conversely, a serially correlated set may show many successive points constituting an upward trend, followed by many downward points, with this pattern repeated. There may also be an overall trend. Most physiological and movement data share this feature; in the case of movement, it is a physical necessity. There are formal tests of autocorrelation.

It is already known (e.g. Brunsdon & Skinner, 1987) that SC shows serial correlation. Figure 1 shows the autocorrelation and partial autocorrelation functions² of two representative SC traces, one taken from a free improvisation (from case study 1), and the other from an imagining improvisation condition (from case study 2). Figure 1(a) shows the large autocorrelations of successive 50 ms measures of SC across a free improvisation, and Figure 1(b) shows this comprises partial autocorrelations which are significant over the preceding 6 lags. “Lags” is a term used to describe the data points preceding the one being statistically modeled. Figures 1(c) and 1(d) show the analogous graphs for the static imagining condition. While the autocorrelations are comparably impressive, the partial autocorrelations show significance over a greater number of lags (up to about 10) with none at lag 2. This confirms that autocorrelation is core to SC traces, including all that we study, and should not be neglected.

Figure 1.

Autocorrelation and partial autocorrelation functions of two SC traces. Figures 1(a) and 1(b) were recorded from A during a free improvisation (case study 1), while Figures 1(c) and 1(d) were recorded from A while still, imagining an improvisation (case study 2). Coefficients outside of the confidence limits (shaded grey) show significant levels of autocorrelation.

Case study 1

A range of factors was investigated in this study. To assess the impact of body movement on the SC measures, the effect of flexing either leg, or moving either hand side to side on the keyboard was determined without playing concurrently. Simple playing of scales and improvising conditions were then compared, culminating in a free improvisation. It was considered important to move from the non-playing to the playing, then from simple scalar performance to slightly more flexible scalar improvisation, and finally to free improvisation, so that serial effects of the complex activities on the simpler would be minimized. Pedaling (with the right foot alone) was only permitted in one of the silent studies (trial 2), and in the free improvisation (trial 7). The possible effect of the auditory cue was evaluated. In conditions in which performance was without auditory feedback (silent), the piano hammers were disconnected from the piano strings and the sound of the fingers hitting the keys was minimized by the use of close-fitting headphones. Our overall purpose was to establish whether even in the presence of movement and/or auditory input it would still be feasible to assess the impact of musical task processes on SC.

The instructions were as follows, with each trial and each of its subsequent segments initiated by the standard auditory cue “change,” and each having the tripartite ABA structure:

1. CUE – flex left leg while otherwise stationary; CUE – flex right leg while otherwise stationary; CUE – flex left leg while otherwise stationary.

No pedaling is used in this trial.

2. CUE – move right hand horizontally as if across keys, no music; CUE – add right pedal movement; CUE – stop pedal (continue right hand horizontal movement).

No pedaling is used in this trial.

3. Play scales without sound, right hand, isochronous, step by step up and down the scales without jumps (we use the term monotonic below to repeat this condition). Throughout: legato, played expressively (no pedaling).

CUE – C major; CUE – change key to A minor; CUE – change back to C major.

4. Improvise without sound, isochronous, right hand, monotonic in interval progression. Throughout: legato, played expressively (no pedaling).

CUE – C major; CUE – change key to A minor; CUE – change back to C major.

5. Play scales aloud, right hand, isochronous, monotonic in interval progression. Throughout: legato, played expressively (no pedaling).

CUE – C major scale; CUE – change key to A minor; CUE – change back to C major.

6. Improvise aloud, isochronous, right hand, monotonic in interval progression. Throughout: legato, played expressively (no pedaling).

CUE – C major; CUE – change key to A minor; CUE – change back to C major.

7. CUE – Free improvisation (both hands and right pedal); CUE – change something; CUE – change back to style of first section.

We ensured that both participants understood that items 5 and 6, while very restricted, were still to be undertaken expressively, and the monotonic interval progression requested that successive isochronic notes be no more than a tone and a half away from the surrounding ones, and within the key in force. Turns in direction were permitted at any stage in 6, and both harmonic and melodic minor components could be used.

Results and discussion of case study 1

Flex leg: left, right, left

Figure 2 shows alignment of periods of left leg movement with phasic bursts of SC, and much smaller phasic changes while the right leg is moving (and the left hardly moved). Similar results were obtained for both participants.

Figure 2.

Movement (pressure) and SC (µS) during the flex leg condition for A. The first segment corresponds to left leg movement, and the subsequent changes into right then left leg flexing are marked by vertical lines.

SC during the left leg flexure was modeled by TSA on the basis of the continuous pressure measures. For one participant, the SC was highly autocorrelated, and showed significant partial autocorrelations over the first six 50ms lags. The SC series was stationary during segment one. The best ARMAX model for the first 60 s of left leg movement involved lags 3 and 4 of the pressure series, and autoregressive lags 1–6 of the SC itself; the model prediction was excellent (correlation 0.99 with the observations). Table 1 illustrates some of the models defined in this paper.

Table 1.

ARMAX models from case study 1.

Condition	Participant	Modeled endogenous (dependent) variable	X (predictor exogenous factor lags)	AR (predictor autoregressive lags)	Comments
1. Left leg flexure	A/B	sc	l3,4.press	ar(1–6)	Correlation model: observed for the whole series was 0.99
2. RH movement with pedaling only in middle section	A/B	dSC	None	ar(1–7)
3. Silently playing a C major scale, then A minor, then C major	A/B	dSC	None	ar(1–10)
4. Silent very simple tonal improvising	B	absdSC	None	ar(1,2,4,8,10,11)
5. Playing scales	B	dSC	None	ar(1, 3–11)
7. Free improvisation	A	dSC	None	ar(1–4, 9)	The piece was performed loud-soft-loud, and there was considerable movement in segment 3 (see Figure 4)
Segment 1		dSC	None	ar(2/5,8,9,11)
Segment 2		dSC	l(1-5, 7,9).	ar(1–3,5,6)
Segment 3			dpress
7. Free improvisationSegment 3	B	dSC	l(4/8).dpress	ar(1–7)	The piece was soft-loud-soft. Relatively little movement, but again most in the loud segment (2)
Segment 1		dSC	None	ar(1–4,6,7)
Segment 2		dSC	None	ar(1,3–7)

Note. SC: skin conductance response. Pressure: Piezo transducer signal representing flexure/movement; l1, l2 . . . ln: lags of the exogenous variable; ar(1. . . n) autoregressive lags of the errors in the endogenous (dependent) variable. Note that the autocorrelation and partial autocorrelation functions shown earlier (Figure 1) refer directly to the measured SC, whereas the autoregressive coefficients in these models apply to the “error” in the dependent variable, that is the difference between the modeled and measured values in that variable (be it differenced SC, undifferenced SC, etc. as specified). Abs refers to the absolute value of a parameter (the number with sign removed). Statistical terminology defines the putative relationship between inputs and output as exogenous vs. endogenous (closely analogous to the psychological concepts of independent and dependent variables). Lag one means the value measured in the time interval immediately preceding that being predicted or modeled, and lags 2 . . . n are progressively earlier values in the series. d: differenced series. In this work first differences of series were often required in order to make them statistically stationary (see text), and so suitable for time series analysis. Higher differencing (d2 . . . dn) was not required.

This model could predict the second left-leg and the right-leg flexure periods comparably well, though the pressure terms were not essential to the model for the right-leg flexure. The model could also predict the behavior of the other participant. These data were encouraging, since there were no other significant events in this condition, and the effect of movement on SC could be modeled successfully.

Right hand across keys: no pedal, right pedal, no pedal

There was less overt pattern in this second condition, and SC and pressure responses were much smaller. The whole data series was used first for modeling. Partial autocorrelations again suggested the importance of eight autoregressive lags for both participants, but the SC series were not stationary. A single differencing stationarized both series. Modeling dSC (where d indicates the first difference) showed as expected that seven autoregressive lags of dSC were significant. Correlation between the predictions of the model and actual for Participant A was only 0.62, largely because the model predictions repeatedly lagged behind the actual. Participant B’s SC series lacked segments but showed tonic change (opening c. 1.0 µS; ending c. 1.6 µS).

Thus, leg movement in this condition was not sufficient to impinge on the SC trace. This remained true even during pedaling with the right foot. The incomplete prediction of the SC indicates that there are factors generating this response that are not as yet addressed.

Conditions 3–6

In conditions 3 and 4, the three successive sections involved respectively playing isochronous scales in silence (C major, A minor, C major: C3), or improvising isochronously in the same succession of keys (C4). Figure 3 illustrates the SC result for A in C3, revealing the occurrence of a significant peak shortly after each auditory cue (shown by vertical lines). There was no corresponding change in the very small pressure responses, which continued throughout. This SC pattern is most probably a “startle” response (Ham & Tronick, 2008; Witvliet & Vrana, 1996). It is known that sudden auditory stimuli can evoke a specific phasic response even if they are expected (Gruzelier & Venables, 1973). So using the peak-picking approach discussed above, we defined a “cue response” in our experiments with A, as a well shaped peak of SC involving an increment from a trough shortly after the “change” point of at least 1.5 standard deviations, peaking within 10 s and subsiding within about 15 s to a level near to that at the time of the auditory cue. It is well known that people differ in their sensitivity to sudden acoustic stimuli (Bach et al., 2009; Gruzelier & Venables, 1973; Lim et al., 1997), and there were few cue responses for participant B. Where the cue response occurred, analyses were restricted to seconds 30 to 60 of the relevant segment, thus excluding the change response (an approach also used by Bach et al., 2009). In the preceding leg flex and right hand movement conditions, such cue responses were not detectable alongside the much larger SC induced by the movement patterns, and only in stationary spots, such as the beginning of segment 2 (Figure 2, at 71.1 s, shown by a vertical line).

Figure 3.

SC trace of A silently playing isochronous scales. Vertical lines mark the onset of the auditory “change” cue, and peaks in the SC occur shortly after each.

Further analyses of the silent conditions were similar to those for the right hand movement condition, with no clear distinction between segments (once the cue response is discounted in the case of A), modest dSC changes, and no impact of dpress. Very similar autocorrelation and partial autocorrelation patterns were observed. One case (A-silently playing a scale) was modeled, with very similar outcomes to that described for the right hand movement condition, except that up to 10 lags (i.e. up to 0.5 s) of autoregression were required to give white noise residuals.

Analysis of segmentation in silent improvisation conditions

There were small and inconsistent tonic changes, but no significant differences in the total SC change for the different segments. We assessed the mean absolute values of dSC data series, by constructing a TSA model, from which the mean can be calculated appropriately. Tonic change was estimated by taking a linear regression gradient across the segment. The summed absolute dSC value for a segment with no tonic change reflects the phasic changes, and when there is tonic change, the difference between the sum and the tonic change reflects the phasic changes.

There were no significant differences between segments for either participant Table 2, and all segments showed a mean SC change that was not significantly different from zero. Thus no segmentation of SC responses was detectable in this condition.

Table 2.

Attempted segmentation of SC during B’s silent improvisation.

Segment 1		Segment 2		Segment 3
Tonic SC change (µS/min)	Mean SC changes/50 ms, delta µS (95% CI)	Tonic SC change	Mean SC changes/50 ms, delta µS (CI)	Tonic SC change	Mean SC changes/50 ms, delta µS (CI)
−0.0859	0.00019 (−.00037 to .00075)	−0.03216	0.00016 (−.00041 to .00074)	−0.03168	0.00024 (−.00042 to .00090)

Note. Tonic change values are estimated as the linear gradient of SC with respect to time from a regression analysis. The total changes are the mean absolute change per 50 ms together with confidence limits, derived from the TSA ARMA model of the whole absolute-dSC series applied to each segment.

Analysis of segmentation during sounded improvisation conditions

Modest levels of pressure change were detected, intermediate between those of the silent conditions and the leg flex condition, and similar to those in the right hand movement condition. The data in Table 1 suggest that as long as movement remains slight, it is a relatively small influence on SC during playing. Conversely, during the extensive movement of the pre- and post-performance sections, pressure was a strong predictor of SC.

Condition 7: Free improvisation

Both participants used both hands throughout as assessed from the videos. For A, the middle section was much softer than the surrounding sections. The first two sections had minimal left leg pressure changes, and slight SC change; while the last (loud) had considerable pressure and SC change (see Figure 4). As predicted, pressure modeled part of the SC during this third section, a part not directly related to the musical events per se or their conception.

Figure 4.

Movement (pressure) and SC (µS) during A’s free improvisation (case study 1), which comprised a loud-soft-loud structure. Solid vertical lines indicate the cued segment boundaries, while the vertical dotted line is a measured datum, though apparently an artifact.

Table 1 also confirms dpress as a predictor of SC in segment 1 of B’s free improvisation, which was soft-loud-soft, in contrast to A. The sections comprised successively an SC tonic fall, a tonic rise and a tonic fall. There was more pressure change in the middle (loud) section than in the surrounding sections. Yet dpress impacted on the dSC response only in segment 1 (see Table 1). Thus pressure change can affect SC but it is not necessarily in the conditions with pronounced movement that it is most important.

We deduce that even during free improvisation, where left leg movement was substantial in spite of the request to keep it as stationary as possible, movement was sometimes a detectable influence on the SC. Overall, monitoring the leg movement at the site of the SC measurement is a useful precaution to deal with leg flexure, the impact of which can be modeled. Otherwise, the (sub)segments in which there is a significant motion impact would have to be removed from subsequent analyses.

Does the SC distinguish the three segments of the free improvisation?

The SC responses for the free improvisation condition were next analysed as before, taking account of the co-existence of tonic and phasic changes. Each player showed some changes in the tonic SC value, but these were not related to the segmental loudness changes. The mean change values are much higher in this condition than in the silent improvisation condition, and much greater than the tonic change, confirming the important contribution of phasic events. For B, segment 3 showed significantly different mean SC change values from segment 2 (i.e. confidence limits did not overlap), and segments 1 and 2 were also distinct by this criterion. A did not show any significant difference in tonic or phasic SC amongst the segments (see Table 3).

Table 3.

Segmentation of SC during free improvisation.

	Segment 1		Segment 2		Segment 3
	Tonic SC change (µS /min)	Mean SC changes/50 ms, delta µS (95% CI)	Tonic SC change	Mean SC changes/50 ms, delta µS (95% CI)	Tonic SC change	Mean SC changes/50 ms, delta µS (95% CI)
A soft-loud-soft	−0.00071>(–.0024 to .00099)	0.0010(.00065 to .0027)	−.0021(–.0054 to .0012)	0.0040(.0022 to .0057)	.0022(.0002 to .0043)	0.0017(.00005 to .0034)
B loud-soft-loud	0.0012(0.00049 to .00200)	0.0017(.00071 to .0027)	−0.0018(−.00236 to −.00116)	0.0011(−.00027 to .0024)	0.0034(0.0015 to .00526)	0.0075(.0054 to .0096)

Note: Mean SC change represents the accumulation of tonic and phasic changes; l, lags; ar, autoregressive error term. For A, given the observed cue response, the last 30 seconds of each segment of the experiment was used for the analyses. The overall ARMA model for the whole data series for B was l(1,3-10).dpress ar(1-4,6), and for A it was l(1).dpress ar(1-7, 10,11): cf. Table 1, for the best models for individual segments. The overall models were used to analyse each individual segment in order to obtain the mean SC values.

Further analysis of segmentation in the free improvisation condition – Participant B

For B, the distinction between the three segments of the stationarized series was characterized further by time series models and cross-validation. As already shown (Table 1), models of the different segments were all slightly different from each other. Since they were not nested, we assessed the Bayes Factor for the difference between the two models from any pair of segments each applied to a single segment (see Table 4). Prima facie the three segments are significantly distinct. There was no cue response in these data, but segment 3 differed from the others in the significant influence of dpress. Differences between segments 1 and 2 therefore indicate differences between SC responses therein, while those between 2 and 3 might merely reflect movement effects.

Table 4.

Comparison of segment models for B’s free improvisation.

	Segment 1	Segment 2	Segment 3
BIC	By Segment 1: −9859.82	By Segment 2: −10346.85	By Segment 3: −9134.09
	By Segment 2: −9836.44	By Segment 1: −10339.54	By Segment 2: −8909.75
Delta BIC	23.38	7.31	124.34
Bayes Factor	131926.47	38.48	1.00e+27

This was investigated further by likelihood ratio tests, making the model comparisons into nested pairs. For this the optimal model for a “test” segment was supplemented by any other parameters contained in the model for the “comparison” segment, forming a “parent” model. The parent was always slightly better in fit than the optimal model in terms of log likelihood, though worse in BIC, because of the additional parameters. The nested model was simply the optimal model for the “comparison” segment. The parent model was compared with the nested model for the ability to model the “test” segment. In this way, differences between the segment models could be attributed primarily to the autoregressive components. Each segment was used as the “test” segment for comparison with its neighbors. For every comparison the likelihood ratio test was p < .0001, confirming that the three segments were each distinct. In the case of segment 3 and of segment 2 as the “test,” this means that the dpress lags were included. The positive likelihood ratio test for both these comparisons indicates that the dpress terms were not the only significant difference between the two: in other words, that SC segments 2 and 3 are intrinsically distinct, possibly reflecting changing musical activity.

Figure 5 confirms this by using rolling 2.5 s window (50 points) ARMAX models of the change in one of the largest autoregressive parameters (ar2) (segments 2–3, participant B). The audible “change” command is shown as a vertical line. Changes in the autoregressive mechanisms do occur across this boundary; most of the autoregressive coefficients, even determined on these short windows, were statistically significant. In comparison, the dpress coefficients were often not individually significant in these windows. Thus evidence for informative changes in the autoregressive mechanisms was obtained.

Figure 5.

Rolling lag 2 autoregressive component from the model of B’s free improvisation (case study 1) across the change from segment 2 to segment 3 (marked by the vertical line).

Case study 2

Since interesting segmentation was observed in free improvisation in case study 1, we investigated whether this might be a response to hearing the dynamic contrasts between the segments, rather than a feature of conceiving and executing the improvisation. Given the association of loudness with arousal (Dean, Bailes, & Schubert, 2011; Schubert, 2004), we took the opportunity to include a condition in which loudness changes were imagined. We also assessed possible differences between the movement impact of LH and RH movement.

Methods were the same as in case study 1, but with the following conditions and instructions:

1. Improvise, playing without sound, isochronous, RH/monotonic.

CUE – C major; CUE – change key to A minor; CUE – change back to C major.

2. Improvise in imagination (without playing) isochronous, RH/monotonic, with dynamic contrast.

CUE – C major, soft; CUE – change key to A minor and imagine loud; CUE – change back to C major and imagine soft.

3. Playing scales aloud, RH, isochronous (a repetition of C5, case study 1).

4. Free improvisation without sound, expressive.

CUE – soft; CUE – change key to loud; CUE – change back to soft.

5. Free improvisation aloud as previously (a repetition of C7, case study 1).

6. Left-hand movement: one segment of LH one of RH one of LH to complement C2, case study 1.

7. Imagine (without playing) free improvisation.

CUE – soft; CUE – change key to loud; CUE – change back to soft.

8. Change cued at altered times, viz. at 30 s, 90 s, and 120 s while playing expressive scales aloud in C major/A minor/C major.

If audible acoustic intensity impacts on SC, the difference between conditions 4 and 5 (without and with auditory feedback) should reveal this, particularly when taken in conjunction with the seventh condition of imagining a free improvisation.

Results and discussion of case study 2

The conditions that closely related to precedents in case study 1 (that is, present 1–3, 5) gave confirmatory results. Neither left nor right hand movements impacted on SC generation. The free improvisations tested whether the audible impact of soft-loud-soft transitions might be responsible for the distinct SC segments that accompany them. If so, then segmentation should disappear when the performance is silent. We found that with or without auditory feedback segments 1 and 3 differed from segment 2 even with pressure included in the models. This suggests that SC segmentation cannot be solely due to accompanying acoustic intensity changes.

Given that case study 1 had shown little sign that playing simple tonal improvisations in silence could reveal dynamic mechanistic differences in SC between segments, condition 7 extended this analysis to a condition of more complex silent improvisation, without playing movement. The overall model for absolute dSC was ar(1,2,7–14) with a constant (dpress terms were not required), suggesting more protracted autoregression than in the other conditions (c.f. Table 1). However, the coefficients from the time series model as applied to each successive segment were not distinct from each other. Thus in study two, only in performing free improvisation was there evidence of segmental distinctions in SC.

The final condition confirmed that B showed no pressure or SC responses to the audible “change” command. Thus some people will show no detectable response to auditory cueing, while others do, and this must be taken into account.

Validation study

Our validation study tests whether the use of the movement parameter within the models of SC, or to “clean” that SC data, was both necessary and appropriate within a more ecological improvisation setting. The experiment, to whose purposes participants were blind, assessed whether musical referent ABA structures could be detected computationally in the performed pieces, and if so what relation they bore to the psychophysiological variable SC. Thus the analysis required the meaningful segmentation of SC data, and this forms the basis of our validation study.

In previous work on time series analysis of relations between musical, perceptual and production features, we have analysed data at a lower sampling rate than used in the case studies, because these relations show considerable lags. For example, the influence of acoustic intensity of a piece on a listener operates over 1–5 seconds (Bailes & Dean, 2012; Dean & Bailes, 2010b). So for the validation study we chose a compromise sampling rate, 5Hz (by upsampling our original data). In this context, the necessity of assessing the role of movement was tested by considering whether the addition of this exogenous variable to models improved their quality, judged by the BIC. The appropriateness of our approach was judged by showing that the resultant models correlated well with the original (or cleaned) data, and that the cleaned series could be segmented in accord with the purpose of the experiment.

Nine professional keyboard improvisers performed self-paced repeated c. 3-minute improvisations. From the complete set of 45 performances included in one block, we took the 5 with the lowest mean undifferenced pressure values, the 5 with the highest means, and the 5 with the highest standard deviations of pressure, to represent the full range of movement patterns (Table 5). These 15 selections included two duplicates, hence 13 separate SC series from the performances were analysed; and these came from 6 of our 9 participants.

Table 5.

Summary data for the SC validation study on professional improvisers.

Participant ID	Mean dPressure	SD dPressure
Chosen during means of the pressure series
P1RATU	−1.3605e-05	0.0028
P8RDNU	−2.2248e-05	0.0242
P1RTNT	−3.1315e-05	0.0066
P8RTNN	−3.7423e-05	0.0127
P2RATU	−4.0718e-05	0.0140
P1RDNN	−0.0007	0.0303
P4RFRE	−0.0010	0.0352
P5RPUN	0.0021	0.3854
P8RPUT	−0.0025	0.0450
P7RPUN	0.0066	0.8690
Chosen using SDs of the pressure series
P5RATN	0.0001	0.0780
P5RDNT	0.0006	0.1215
P4RDNN	0.0004	0.1337
P5RPUN	As above	As above
P7RPUN	As above	As above

Note. The performances from the specified block of our experiment (in which nine improvisers each performed five improvisations) were those with the five lowest and five highest absolute means, and the five highest SDs of the pressure parameter (undifferenced; representing movement). Six of our participants qualified through this selection. The first two characters of the ID define the participant, the last three the referent they were undertaking (see text).

In establishing necessity, we required that a substantial proportion of the SC series be modeled best by including movement as a predictor. This was tested by optimizing the AR-alone model of each SC series (using BIC), and then assessing possible enhancement of the model by the addition of lags of movement. The autoregressive order required for most AR-alone models was similar in duration to those of the case studies. Table 6 shows that modeling 9 of the selected 13 series required movement predictors. As before, it was not solely series accompanied by large movement which required the movement predictor. All 6 performers in the validation study showed at least one performance in which the use of movement was necessary. Three performers contributed 3 series each, and none were uniform in their movement patterns, and whether it impacted on SC. This confirms that an analysis has to consider the relationship between the SC and movement series, and not just the magnitude and variability of either independently.

Table 6.

TSA modelling of dSC (stationary first differenced skin conductance) by autoregression plus the eXternal Predictor dPressure (the first differenced movement series).

Participant ID: in order of ascending mean and then ascending high SD movement series	AR model order chosen	BIC for AR model	ARX model: lag order of the dPressure series added	BIC for ARX model	Spearman correlation best model: dSC series	Number of Changepoints detected in the dSC series (corrected for the influence of movement when appropriate)
P1RATU	3	−8477.1	None	NA	0.71	3
P8RDNU	4	−23.5	3	−59.7	0.83	3
P1RTNT	2	−6578.7	1	−6585.1	0.81	3
P8RTNN	5	−2753.8	3	−2959.2	0.95	3
P2RATU	3	−7523.2	2	−7617.8	0.56	3
P1RDNN	3	−4416.5	1	−4486.9	0.84	3
P4RFRE	5	−5412.4	2	−5418.8	0.77	3
P5RPUN	3	−4326.8	None	NA	0.43	3
P8RPUT	4	537.9	None	NA	0.33	3
P7RPUN*	4	−2989.6	5	−3762.2	0.81	3
P5RATN	3	−5205.2	5	−5238.4	0.40	3
P5RDNT	3	−7392.8	None	NA	0.31	3
P4RDNN	4	−2822.8	5	−3045.5	0.74	3

Note. * Fig 6 shows the impact of cleaning the dSC time series in this case.

The AR models were permitted to be up to order 5; similarly up to lag 5 of the eXternal variable, dPressure (representing movement) was permitted, lag 0 included. Thus, lag order 2 means the use of lag 0, 1 and 2 as predictors. Models were optimized by BIC (lower is better). The modeling was done in two phases: first the AR; then the possible addition of the eXternal variable to form ARX. Note that it is only meaningful to compare BIC values for models of an individual series, not between series; thus the AR and ARX models for an individual data series may be compared in the Table. An absolute deltaBIC difference > 4.6 between pairs of models of a series indicates a substantial probability difference, as discussed in the text. Here we allowed the AR order to be simply that of lowest BIC (not always surpassing this 4.6 distinction). Thus, sometimes models with fewer AR lags might have been as good as the one we chose (we were not necessarily maximally parsimonious in the AR portion of the model). This was appropriate for the present purposes of method validation because it means we were applying a more stringent test of whether the movement parameters were required for the model than had we also maximized parsimony. This is consequently also a more conservative test of whether it was necessary to “clean” the SC data series by removing their impact. Spearman correlations were always p < 0.01. Changepoints in the time series are detected using Killick’s “Changepoint” package in R 2.1.14. For this, we assessed changes in variance using the “BinSeg” method, and with the Cumulative Sum of Squares error method (which is free of a distributional assumption). While changes in mean in the series would be largely due to tonic changes, those in variance more reflect the phasic changes in which we are interested.

Our first criterion of appropriateness was that the resultant best TSA model of the SC series gave a significant correlation with the original data. Such correlations are meaningful when data and model share autoregressive features (but not otherwise). Because SC data are often not normally distributed (e.g. Gruzelier & Venables, 1973) we used a non-parametric correlation (Spearman) appropriate for that situation.

The second criterion of appropriateness was that the resultant SC series (“cleaned” of the influence of movement when necessary), show at least 2 segmentation points, such that we could test in future work whether the SC segmentation points relate to musical segments. This assessment was made in a principled way with the R “changepoint” package (developed by R. Killick). We allowed the algorithm to choose as many segmentation points as fulfilled an alpha of 0.05. In every case 3 such changepoints were found. We concluded that considering movement is necessary to analyse SC data from keyboard performers; and that our method using time series models is appropriate for our intended application.

Figure 6 shows an example of a dSC trace before and after cleaning. The improvements are subtle, but can be readily judged for example in terms of the number of values which slightly exceed 0.07; and by viewing the data of the time points between about 750 and 850. Major SC peaks are retained, and may represent SCRs. The correlation between model and data before cleaning was 0.43, and afterwards was 0.8. Similarly, the dSC series s.d. was reduced by cleaning, while as expected for a differenced and stationarized series, the mean was unchanged.

Figure 6.

A comparison of the dSC trace for performance P7RPUN (see Table 6) before and after cleaning by application of the movement series as described in the text. The Time axis displays time count in 200 msec units (5Hz sampling); the total performance duration is 179 s, of which 1 s is not visible, because of the use of 5 lags (each 200 msec) in the model.

General discussion

Our purpose was to establish methods for assessing the SC/SCR of piano performers who may move substantially, and who may hear intermittent auditory stimuli. The results confirm that it is possible to measure and interpret such SC, provided that pressure changes adjacent to the SC electrode are monitored to determine the influences of movement. Our results also show the need for care with auditory cues, but provide approaches to deal with them. Thus one could use a model of the standard “cue response” to correct for its impact, or model it as a transient predictor variable; closely related approaches have been applied in SC studies (e.g. Bach et al., 2009; Benedek & Kaernbach, 2010).

Our participants did not mention feeling any constraint in wearing the monitoring devices; they had no prior exposure to this condition. On the other hand, there are circumstances in which musicians, particularly jazz musicians, move their left feet with the pulse. Yet the problem would be more acute with electrodes attached to the right ankle, and commonly used alternative sites for SC electrodes are either very intrusive, or susceptible to much greater movement artifacts than the hypomalleolus. It is quite likely that the hypomalleolus can be used for SC-monitoring of performers of other instruments using our approach. We found that the operational definition of this approach has to be a time series assessment of whether pressure is a significant predictor for any individual segment. The validation experiment resoundingly confirms the success of our method.

Interestingly, we found that hearing musical output is not required for the SC segmental response. Changes in SC in the absence of auditory feedback, and not attributable to movement, could reflect the mental processes associated with planning and generating music in real-time. Such a psychological tool is an important window onto the processes of improvisation. Of course, changes in SC in the absence of sound do not mean that hearing the performed dynamics is unimportant and this can be assessed in future work.

Other technical issues in the analysis of continuous SC data

TSA allows the modeling to provide the summary statistics for SC series or segments thereof. The approach also avoids the temptation to use theoretically inapplicable tests of differences between series datasets, such as t tests (inappropriate because of the lack of independence of the multiple data points).

One very interesting analysis of continuous SC data has been presented recently: the non-negative deconvolution approach of Benedek and Kaernbach (2010). This method is particularly suitable for physiological investigations, as it decomposes the SC data stream on the basis of a model of two-compartment diffusion of sweat, with a characteristic predicted SC shape, together with an additional component representing putative pore opening. Our purpose instead was psychophysiological, to use SC changes as an index of autonomic changes and potentially of arousal in relation to cognitive events that we cannot dictate in time as part of the experimental procedure. Our simpler, yet representative and unbiased time series analysis approach was more appropriate. Our use of absolute changes in dSC has parallels with non-negative deconvolution.

Future studies

It will be important to establish whether differences between responses are significant when there are changing experimental conditions on a segment-by-segment basis. Experiments in which continuous independent variables (such as acoustic intensity) are manipulated will provide input data streams which can be quantized to provide predictors for such analyses. In the absence of such independent variables, musical features of the performance may be statistical predictors of the SC (cf. Schubert, 2001).

Concluding remarks

SC has previously been neglected in performance studies, probably due to the problematic artifacts associated with movement. This study has shown the feasibility of measuring skin conductance during musical performance, and applied time series analysis to illustrate ways to understand informative changes in autonomic arousal during improvisation. We have been able to measure and model the ankle movement of our performers, and analyse its relative contribution to the skin conductance signal. The method thus treats all SC-data as potentially informative. Our comparison of silent with sounded performances indicated that SC is more than a listening response and reflects more than physical movement, confirming its potential as an index of arousal and attention. Moreover, time series analysis techniques can unpack the lag structure of continuous response variables, and so begin to disentangle preparatory from responsive behaviors.

Footnotes

Acknowledgements

We thank Jon Drummond for his technical assistance with data collection.

Funding

This work was supported by an Australian Research Council grant.

Notes

References

Alexander

D. M.

Trengove

Johnston

Cooper

August

J. P.

Gordon

(2005). Separating individual skin conductance responses in a short interstimulus-interval paradigm. Journal of Neuroscience Methods, 146, 116–123.

Bach

D. R.

Flandin

Friston

K. J.

Dolan

R. J.

(2009). Time-series analysis for rapid event-related skin conductance responses. Journal of Neuroscience Methods, 184, 224–234.

Bailes

(2009). Translating the musical image: Case studies of expert musicians. In Chan

Noble

(Eds.), Sounds in translation: Intersections of music, technology and society (pp. 41–59). Canberra, Australia: ANU E-Press.

Bailes

Dean

R. T.

(2009). Empirical studies of computer sound. In Dean

R. T.

(Ed.), The Oxford handbook of computer music (pp. 473–492). New York, NY: Oxford University Press.

Bailes

Dean

R. T.

(2012). Comparative time series analysis of perceptual responses to electroacoustic music. Music Perception, 29(4), 359–375.

Bartlett

D. L.

(1999). Physiological responses to music and sound stimuli. In Hodges

D. A.

(Ed.), Handbook of music psychology (2nd ed., pp. 343–385). San Antonio, TX: IMR Press.

Benedek

Kaernbach

(2010). Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology, 47, 647–658.

Brunsdon

T. M.

Skinner

C. J.

(1987). The analysis of dependencies between time series in psychological experiments. British Journal of Mathematical and Statistical Psychology, 40(2), 125–139.

De Manzano

Ö.

Theorell

Harmat

Ullén

. (2010). The psychophysiology of flow during piano playing. Emotion, 10(3), 301–311.

10.

Dean

R. T.

(2009). Envisaging improvisation in future computer music. In Dean

R. T.

(Ed.), The Oxford handbook of computer music (pp. 133–147). New York, NY: Oxford University Press.

11.

Dean

R. T.

Bailes

(2010a). A rise-fall temporal asymmetry of intensity in composed and improvised electroacoustic music. Organised Sound, 15(2), 147–158.

12.

Dean

R. T.

Bailes

(2010b). Time series analysis as a method to examine acoustical influences on real-time perception in music. Empirical Musicology Review, 5(4), 152–175.

13.

Dean

R. T.

Bailes

Schubert

(2011). Acoustic intensity causes perceived changes in arousal levels in music. PLoS One, 6(4), e18591. doi:10.1371/journal.pone.0018591

14.

Enders

(2004). Applied econometric time series (2nd ed). Hoboken, NJ: Wiley.

15.

Gruzelier

J. H.

Venables

P. H.

(1973). Skin conductance responses to tones with and without attentional significance in schizophrenic and nonschizophrenic psychiatric patients. Neuropsychologia, 11, 221–230.

16.

Ham

Tronick

(2008). A procedure for the measurement of infant skin conductance and its initial validation using clap induced startle. Developmental Psychobiology, 50(6), 626–631.

17.

Hamilton

J. D.

(1994). Time series analysis. Princeton, NJ: Princeton University Press.

18.

Khalfa

Peretz

Blondin

J.-P.

Manon

(2002). Event-related skin conductance responses to musical emotions in humans. Neuroscience Letters, 328, 145–149.

19.

Kylliäinen

Hietanen

J. K.

(2006). Skin conductance responses to another person’s gaze in children with autism. Journal of Autism and Developmental Disorders, 36(4), 517–525.

20.

Levinson

D. F.

Edelberg

(1985). Scoring criteria for response latency and habituation in electrodermal research. Psychophysiology, 22(4), 417–426.

21.

Lim

C. L.

Rennie

Barry

R. J.

Bahramali

Lazzaro

Manor

Gordon

(1997). Decomposing skin conductance into tonic and phasic components. International Journal of Psychophysiology, 25, 97–109.

22.

MacIntosh

B. J.

Mraz

McIlroy

W. E.

Graham

S. J.

(2007). Brain activity during a motor learning task: An fMRI and skin conductance study. Human Brain Mapping, 28(12), 1359–1367.

23.

Mandryk

R. L.

Atkins

M. S.

(2007). A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4), 329–347.

24.

Nakahara

Furuya

Francis

P. R.

Kinoshita

(2010). Psycho-physiological responses to expressive piano performance. International Journal of Psychophysiology, 75, 268–276.

25.

Nakahara

Furuya

Obata

Francis

P. R.

Kinoshita

(2009, December). Cardio-respiratory responses to expressiveness in piano performance. Paper presented at the International Symposium on Performance Science, Auckland, New Zealand.

26.

Ollen

(2009, August). Cross-validation of a model for classifying musical sophistication. Paper presented at the Society for Music Perception and Cognition Conference, Indianapolis, IN.

27.

Schubert

(2001). Continuous measurement of self-report emotional response to music. In Sloboda

J. A.

Juslin

P. N.

(Eds.), Music and Emotion (pp. 393–414). Oxford, UK: Oxford University Press.

28.

Schubert

(2004). Modeling perceived emotion with continuous musical features. Music Perception, 21(4), 561–585.

29.

Schubert

Dunsmuir

(1999). Regression modelling continuous data in music psychology. In Yi

S. W.

(Ed.), Music, Mind, and Science (pp. 298–352). Seoul, Korea: Seoul National University Press.

30.

Smith

Dean

R. T.

(1997). Improvisation, hypermedia and the arts since 1945. London, UK: Harwood Academic.

31.

Wasserman

(2000). Bayesian model selection and model averaging. Journal of Mathematical Psychology, 44, 92–107.

32.

Witvliet

C. V.

Vrana

S. R.

(1996). The emotional impact of instrumental music on affect ratings, facial EMG, autonomic measures, and the startle reflex: Effects of valence and arousal. Psychophysiology, 33, S91–S91.