Reliability issues regarding the beginning,middle and end of continuous emotion ratings to music

Abstract

Continuous self-reported emotion expressed by four pieces of music were collected on a two-dimensional (valence and arousal) emotion space in a repeated measures (test-retest conditions) design. Initial orientation time (IOT), test-retest reliability and afterglow were examined. Median IOT was 8 seconds. Valence ratings took up to 25 (median 4), and for arousal up to 35 (median 12) seconds. Slower tempi seemed to require longer IOT. Test-retest reliability examined correlation coefficients, and compared periods of sample-by-sample good agreement in response between Test and Retest condition. About 80% of responses were reliable in both the Test and Retest conditions regardless of response dimension. Pearson correlations demonstrated better test-retest reliability for arousal responses than for valence. Retest condition ratings were within 8% of Test condition rating within participant. Average standard deviations for ratings collapsed across dimension, stimulus and conditions was 12.2% of the ratings scale range. Afterglow effects – large outliers in spread of scores just after the end of a piece – were identified. The reliability of continuous emotional response is therefore considered to be quite good, but caution must be taken as to how to deal with the opening and ending of continuous emotional response data.

Keywords

afterglow arousal continuous response emotion in music orientation time reliability time series valence

Introduction

Both emotional responses to music, and the music which triggers these emotions, unfold in time. With relatively cheap and fast computers, it is possible to collect emotional response to music and other temporally dependent stimuli in real time, using little more than a computer and a mouse (Cowie et al., 2000; Nagel, Kopiez, Grewe, & Altenmüller, 2007; Schubert, 1999, 2007b) or more specialized devices (Geringer, Madsen, & Gregory, 2004; Stevens et al., 2009). As a result we have been able to investigate aspects of emotion in music that were not afforded to researchers previously because they were restricted to measuring emotional response to music after the musical stimulus was heard, the so-called ‘postperformance’ or retrospective measure (e.g., Asmus, 1985; Gabrielsson & Lindström, 2001; Hevner, 1936). The continuous approach has allowed us to verify that emotions can vary from moment to moment as a piece of music unfolds, and that these variations can only be captured with postperformance methods in a way that is necessarily reductive: the richness of a time-varying emotional experience is potentially lost.

While the number of papers published on continuous emotional response to music are rising, there are relatively few that focus specifically on analytic issues. For example, when a continuous response is reported, how certain can we be that a statistically identical profile will emerge if the experiment were run again at a different time? That is, replication is rare, and the present study aims to address this. Exceptions are Schubert (1999) and Grewe et al. (2007a) but the former only reports general, overall test-retest reliability using Pearson correlations, and the latter took repeated measures to one participant, albeit seven times (see also, Grewe, Nagel, Kopiez, & Altenmüller, 2007b). Further, there are several complications when dealing with time-series data. One of them is the reliability of the initial responses made to a piece of music – a period of ‘settling-in’ during which responses are unreliable. Another complication occurs at the end of a continuous rating collection period, when the participant’s emotion-expressed rating continues despite the absence of the just completed music. Schubert and Dunsmuir (1999) catalogued various kinds of outliers in the continuous music-emotion system. The initial response outlier was classified by them as ‘orientation’ time. The variability in response just after the music had ended was called ‘afterglow’. However, no statistical analysis was provided to diagnose what an initial orientation time might be, and afterglow was described in terms of outliers from a musical feature predictive model. The present study aims to revisit these issues and extend recent research by Bachorik et al. (2009) from the perspective of the variability of multiple participant ratings to the same piece, rather than from a musical feature modelling point of view (for which, see also Korhonen, Clausi, & Jernigan, 2006; Schubert, 2004). In addition, this paper will report how reliable emotion ratings are using a test-retest experimental paradigm.

Methods used to analyse time-series data range from pure visual inspection to sophisticated time-series and functional data analysis methods (see Box, Jenkins, & Reinsel, 1994; Engle, 2001; Gottman, 1981; Ramsay & Silverman, 2002). According to the review of the field by Schubert (2001), most analytic techniques applied to emotion in music data lie on the less sophisticated, visual inspection end of the spectrum, and do not always show awareness of the complications to which time-series data are prone, such as serial correlation (Schubert, 2002). Further, the apparent complexities of the sophisticated analytic techniques discourage some researchers from applying legitimate ways of processing data. In this paper, a simple method of identifying rating agreement in a time series, the Second Order Deviation Threshold, will be applied. It will be described in the analysis section of this paper.

A study by Schubert (1999) (the data of which is the basis of the present study) presented results of two Pearson correlation analyses comparing the Test and Retest conditions. A correlation coefficient of 0.75 for valence (the positive-negative dimension of emotion) and 0.71 for arousal (the arousal-sleepiness dimension of emotion) were reported. Since then, research into self-report continuous emotional response has developed sufficiently that a more sophisticated analysis is timely. For example, there have been several developments in the comparison of such time-series data (Grewe, Kopiez, & Altenmüller, 2009a, 2009b; Grewe et al., 2007a, 2007b; McAdams, Vines, Vieillard, Smith, & Reynolds, 2004; Schubert, 2002; Vines, Krumhansl, Wanderley, & Levitin, 2006). Grewe et al. (2007a) examined responses of a participant who provided continuous responses to the same pieces of music seven times. In that study moment by moment median responses for the pieces were reported, as well as the differenced (explained in more detail later in this paper) responses, and they used the nonparametric Wilcoxon test to identify whether peak moments were significant. McAdams et al. (2004) compared two sets of responses using running (moment by moment) F-tests. The risk of inflated Type I error detection due to multiple testing (because time-series data typically have many samples) were dealt with by setting the error probability threshold to p = 0.01 or 0.001, rather than the more usual p = 0.05 found in conventional parametric data analysis. These approaches are generally appropriate for the analyses of interest in those studies. However, none of the more recent studies have specifically examined test-retest reliability with multiple participants. For example, the F-test and Wilcoxon test mentioned are typically used to identify significant differences between time points, rather than significant similarities across time-series. Schubert (1999) applied Pearson correlation coefficients as a measure of similarity of two time-series data sets, but this only demonstrated dependence, rather than absolute differences, and further, Pearson correlations were reported as producing inflated values unless the data sets were transformed, for example, through differencing (Schubert, 2002; Vines et al., 2006) which is what Schubert reported in the 1999 paper. In the absence of an established, absolute correlation coefficient about which test-retest reliability can be quantified, the approach in the present study is to compare two kinds of responses – valence and arousal – and determine which, if any, is more reliable.

Aim

This aim of the study was to explore the extent to which a continuous emotion rating of music is stable, and to what extent it remains stable when the same piece is rated again at a later date. The focus will be on ratings of expressed emotion made near the beginning of the music, those made at the end of the music, as well as more general test-retest reliability. Further, the study examines whether arousal or valence ratings of emotion (as defined below) are reported with equal reliability.

Method

Participants

Fourteen participants completed the test and retest conditions as part of a larger study (Schubert, 1999). There were six males and eight females. Ages ranged from 19 to 48 years (mean = 29.40, SD = 10.36). Ten participants reporting having some music-playing experience, including eight who reported more than ten years of playing. Four reported never playing a musical instrument.

Stimuli

Four pieces were selected with the intention of depicting a wide variety of emotions. They were chosen from a largely Romantic, orchestral idiom. These pieces were Slavonic Dance Op. 46, No. 1, by Antonin Dvorak (Slavonic Dance), Pizzicato Polka by Johann (Jr) and Josef Strauss (Pizzicato), Morning Mood from Peer Gynt by Edvard Grieg (Morning) and the Adagio Movement from Concierto de Aranjuez for guitar and orchestra by Joaquin Rodrigo (Adagio). Abbreviations used for referring to the pieces are shown in parentheses. Full recording details are provided in the Discography.

Procedure

The study used specially written software that controlled all aspects of participant interaction, interfacing and data collection. The study was conducted twice, first in the ‘Test’ condition, and later in a ‘Retest’ condition, as explained below. The software was run on a Macintosh LC520 computer with one participant tested at a time. In the test condition, participants received training to familiarize themselves with the Two-Dimensional Emotion-Space (2DES), which consisted of valence indicated on the x-axis (positive emotions to the right, negative to the left) and arousal on the y-axis (aroused emotions toward the top, and sleepy emotions towards the bottom), based on the circumplex model and as found on the affect grid of Russell (Russell, 1980; Russell, Weiss, & Mendelsohn, 1989). Such a layout implies that the structure of emotion is dimensional, meaning that emotions are thought of as occupying various locations along a limited number of dimensions (in this case valence – positive vs negative, and arousal – high vs low). However, there are other models, such as a discrete units structure (sad, happy, angry etc.) with no necessary link along underlying dimensions. The debate as to which model is suited to which circumstances is to be found elsewhere (Barrett, 1998; Christie & Friedman, 2004; Eerola & Vuoskoski, 2011; Laukka, 2008; Zentner, Grandjean, & Scherer, 2008). However, for this research the structure of emotion was based on a dimensional model based on valence and arousal. This model was considered appropriate because it suited the logic of moving a mouse around a two-dimensional surface (computer screen). The x (valence) and y (arousal) axes were presented on the computer screen and mouse movements within the bounds of the axes were recorded on a 201 point scale (−100 to +100), with axes crossing at the origin.

The training consisted of three stages, each involving the presentation of a selection of words which encompassed a wide range of valence and arousal values. In one stage, only the arousal axis was displayed, in another only the valence axis was displayed and in the final stage, both arousal and valence axes were displayed as a 2DES. The order of initial presentation of valence and arousal stages was selected randomly across participants. High arousal words used were Afraid, Angry and Happy, while low arousal words of Relaxed and Sad were used. An animation placed each of these words at an appropriate region of the emotion space. This was followed by a practice session, where participants placed these and other words and pictures of faces (based on Ekman & Friesen, 1975) on the emotion space by clicking with the computer’s mouse. After a little practice, participants were able to do this reliably. The results of these responses are reported in Schubert (1999).

Then the participants were presented with the first of the four music stimuli selected at random for each participant. They moved the mouse to the middle of the emotion space and then the music, played over headphones, commenced. The software controlled the audio, which was all contained on one audio CD (see Discography) via the built-in CD-ROM player of the computer. Participants listened to the audio via headphones in a quiet room at a comfortable listening level. The participant was instructed to move the mouse around the emotion space to reflect the emotion that the music was expressing (and not how the music made the participant feel. See Evans & Schubert, 2008; Gabrielsson, 2002; Grewe et al., 2007a; Salimpoor, Benovoy, Longo, Cooperstock, & Zatorre, 2009). When no emotion was detected, participants were instructed to move the pointer to the middle of the emotion space. The four pieces were presented in random order, and participants were encouraged to take a short break between each piece.

Participants were not aware that they would be asked to take part in the study again, and six to twelve months elapsed before they returned to complete the Retest condition. This strategy was intended to minimize the possibility that participants would deliberately try to memorize their responses in the test condition. The exact timing of the second data collection phase could not easily be controlled because not all participants were available at the same time. However, it has been argued elsewhere that variations in time in repeating this kind of task of the duration indicated is unlikely to affect results considerably (Cepeda, Pashler, Vul, Wixted, & Rohrer, 2006; Lucas, Schubert, & Halpern, 2010). In the Retest condition, the participants received minimal training which was intended to provide a reminder of the way the emotion space worked. The pieces were presented in random order. In all continuous response recording, participant responses were recorded once per second (Schubert, 2001).

Analysis

Schubert (2007c, 2010; Schubert et al., in press) proposed a general analytic technique that made minimal assumptions about the distribution characteristics of the continuous self-report emotional data. The technique was developed to provide a simple approach to analysing time-series data such as that of the present study. While the method is flexible and can be optimized for different situations, it can be performed on a conventional spreadsheet program using nothing more sophisticated than mean and standard deviation calculation, though other approaches are mentioned in some of the analyses that follow, and in the Discussion section. Further, the method has a visual orientation for easy identification of regions consisting of reliable ratings of the time series. Consider the general time-series X(t, ζ_i)_p,d,c where X is a matrix representing Arousal or Valence (subscript d) ratings in time t (each increment of this index progresses time by one sample) by i participants (where i varies from 1 to 14 for a given sample, and taken together is referred to an ensemble of ratings) to piece (or stimulus) p in condition c (Test vs Retest) at time t. The sample-by-sample ensemble mean is represented as a new time series, M(t)_p,d,c, the sample-by-sample ensemble median is represented by a new time series Mdn(t)_p,d,c and the sample-by-sample ensemble rating standard deviation time series as SD1(t)_p,d,c. This series will be referred to as the first order standard deviation, and abbreviated SD1(t). Lower case subscripts indicate that the series is calculated for each combination (‘p,d,c’ for example, indicates 4 pieces by 2 dimensions by 2 conditions), whereas a capital letter subscript denotes the level of the variable (for example, M _p,d,T(t) indicates the sample-by-sample ensemble mean score in the test condition for each combination of piece by dimension). Individual elements of a time series are indicate by an index, such as X(t₅, ζ₇) which is the fifth time sample rating for participant number seven in the time series rating matrix X. A single real number evaluation of a time series is represented in lower case. For example, the standard deviation of the mean series, M(t), will evaluate to sd1.

The mean of SD1(t), $\bar{S D 1}$ is:

\bar{S D 1} = \frac{1}{T} \sum_{t = 1}^{T} S D 1 (t)

(1)

The standard deviation of the SD1(t) series, which, like its mean, is a real number and will be referred to as the second order standard deviation, abbreviated sd2:

s d 2 = \sqrt{\frac{1}{T - 1} \sum_{t = 1}^{T} {(S D 1 (t) - \bar{S D 1})}^{2}}

(2)

where T is the total number of samples recorded for a given piece in one condition and dimension. This total was dependent on the number of seconds of the CD track, and included about three seconds of silence after the stimulus finished playing.

These values were then used to assign a threshold range, or threshold level. The range or level of the threshold is labelled τ and is defined as

τ_{\pm n} = \bar{S D 1} \pm n s d 2 or τ_{n} = \bar{S D 1} + n s d 2

(3), (4)

where n is a real number multiplier of sd2 ¹. τ_±n refers to a range within which elements of the SD1(t) series are to fall to be considered in ‘good agreement’, and otherwise they are considered in ‘poor agreement’ (Equation 3). For the threshold level τ_n (Equation 4), elements of SD1(t) are coded as ‘poor agreement’ when greater than or equal to τ_n (because the deviation score for element t in SD1(t) is relatively large) and otherwise as ‘good agreement’ (because the deviation score of the element t in SD1(t) is relatively small). These terms will be applied and further developed in the reporting of the results.

Results

Initial orientation time

Initial orientation time (IOT) in the present study was defined as the time taken for an emotional rating SD1(t) series to settle to within τ_±n as described above. This definition is analogous to ‘settling time’ used in off-line controller design in engineering (Tay, Mareels, & Moore, 1998, p. 93).² As with settling time, IOT was set as a range rather than a threshold because of the nature of the emotion space procedure used: Participants commenced their emotion ratings at the neutral centre of the emotion space. This means that the start of each piece produces good agreement. However, instructing participants to start in the neutral centre point of the emotion space is a pragmatic decision, and not a reflection of any true underlying agreement. In this case an estimate can be obtained by finding the time taken for the ratings to fall within the ensemble standard deviation range, such as ±n sd2 of the mean SD1(t) value: τ_±n. Table 1 shows the results of the analysis conducted with three range criteria: τ_±1, τ_±1.5 and τ_±2.

Table 1.

Initial orientation time by stimulus and dimension.

Stimulus	Dimension	Condition	Time (in seconds) to settle to within τ_±n
			n = 1	n = 1.5	n = 2
Adagio	Arousal	Test	18	17	5
Adagio	Arousal	Retest	35	18	11
Adagio	Valence	Test	15	13	5
Adagio	Valence	Retest	25	15	7
Pizzicato	Arousal	Test	10	9	8
Pizzicato	Arousal	Retest	14	11	3
Pizzicato	Valence	Test	4	4	4
Pizzicato	Valence	Retest	4	4	3
Slavonic Dance	Arousal	Test	3	2	2
Slavonic Dance	Arousal	Retest	5	5	4
Slavonic Dance	Valence	Test	2	2	2
Slavonic Dance	Valence	Retest	2	2	2
Morning	Arousal	Test	5	13	4
Morning	Arousal	Retest	9	9	5
Morning	Valence	Test	7	6	4
Morning	Valence	Retest	4	4	3
	Overall	Median	8	7.5	4
		Min	2	2	2
		Max	35	18	11

As the interval about gets wider due to the larger τ_±n criterion, we expect to see the initial response settle into the good agreement region sooner (i.e., the estimate will be more liberal). The most dramatic examples of this are in the Arousal response for Adagio (18 s to 5 s in Test condition and 35 s to 11 s in Retest condition for τ_±1 and τ_±2 respectively – see also Figure 1) and for Pizzicato in the Retest condition (for τ_±1 it is 14 s, but τ_±1 drops to 3 s). In some cases the IOT is extremely resilient to different τ criteria, such as the valence times for Pizzicato in both conditions (about 4 s), and for the Slavonic Dance: 2 to 5 seconds regardless of dimension and condition. Within this piece Arousal has the longest initial orientation time at 4 to 5 seconds, depending on the threshold criterion. These ‘resilient’ results may be a reflection of the generally smaller SD1(t) movements from the origin throughout the piece: for example, if SD1(t) remained very low throughout the stimulus-dimension-condition combination. Inspection of the eight SD1(t) time-series (Figure 2) indicates quasi-exponential rise, overshoot and decay toward the ‘operating’ region of SD1(t) from the beginning in each case, verifying the presence of an initial orientation (start up) time, analogous to settling time. The Adagio Arousal response is presented as a close-up example in Figure 1. It was chosen because it allows closer inspection of a result with a large disparity in IOT as a function of τ_±n, with n = 1, 1.5 and 2. While the τ_±2 range produces a smaller, more liberal estimate of the initial orientation time (5 s in Test condition and 11 s in the Retest condition), the τ_±1 criterion allows us to see the markedly different effect of initial orientation time across the two conditions (17 s in the Test condition, and 35 s in the Retest condition).

Figure 1.

Initial orientation time estimates for Adagio Arousal for Test and Retest conditions using two threshold criteria.

Figure 2.

M(t) and SD1(t) time series of each stimulus, dimension and condition combination, showing regions of good agreement at threshold τ₁.

Table 2 provides a summary of these results. It shows that across the four stimuli there is no major difference in median initial orientation time across Test and Retest. At τ_±1, the arousal ratings oriented one second earlier (11.5 versus 12.5 s) in the Retest, and the valence IOT remained unchanged (4 s) across Test and Retest. It is noted, however, that the maximum IOT tended to be longer in the Retest condition, perhaps reflecting the shorter training session that was performed for that condition regardless of the τ threshold range selected. For arousal, the maximum value increased from 18 s to 35 s, and for valence, from 15 s to 25 s (Test to Retest) for τ_±2. The trend was the same for the other two threshold criteria reported, but was most consistent for τ_±1.5, in that the maximum values across conditions changed the least for arousal and for valence.

Table 2.

Initial orientation time (in seconds) by response dimension collapsed across stimuli.

τ_±1				τ_±1.5				τ_±2
		A	V			A	V			A	V
Test	Median	12.5	4	Test	Median	11	4	Test	Median	4.5	4
	Min	3	2		Min	2	2		Min	2	2
	Max	18	15		Max	17	13		Max	8	5
Retest	Median	11.5	4	Retest	Median	10	4	Retest	Median	4.5	3
	Min	5	2		Min	5	2		Min	3	2
	Max	35	25		Max	18	15		Max	11	7

A = Arousal; V = Valence.

Test-retest reliability

The reliable responses in each time series (four pieces by two dimensions) were identified by comparing the τ₁ threshold of SD1(t)_p,d,T and SD1(t)_p,d,R conditions. This meant that when elements in an SD1(t) series fall below τ₁, the ratings are considered to be in good agreement with each other, and therefore, in part, reliable. This provided an indication of reliability within (i.e., during) the response to each piece and dimension. The τ₁ threshold was chosen to allow a reasonably large number of response samples to be considered in good agreement for reasons that will become clear shortly.

Visual inspection of the mean responses, M(t)_p,d,c, in Figure 2 comparing each Test-Retest time series pair shows how similar the profiles appear to be for each stimulus-dimension combination. The conventional Pearson test-retest reliability is indicated in Table 3, with all values at 0.83 or higher. However the serial correlation commonly found in continuous response data can inflate the coefficients generated by Pearson correlations. Each of the participant responses were therefore first-order differenced to reduce the effect of serial correlation (as suggested by Schubert, 2002). The differencing transformation of the ensemble mean times series, ΔM(t)_p,d,c, produces changes in response made by each participant for a given piece-dimension-condition combination from sample to sample, and is defined in general form as:

Δ M (t) = [M (t_{2}) - M (t_{1}), M (t_{3}) - M (t_{2}), M (t_{4}) - M (t_{3}), \dots M (t_{T}) - M (t_{T - 1})

(5)

Table 3.

Comparison of test-retest condition correlation and good agreement match methods for untreated (X(t)) and differenced (ΔX(t)) ratings.

Stimulus^a	Dimension^b	Pearson’s r of		Spearman’s rho of		CA/Any based on ^c		CA/All based on ^d
		M(t)_T&R	ΔM(t)_T&R	Mdn(t)_T&R	ΔMdn(t)_T&R	X(t)	ΔX(t)	X(t)	ΔX(t)
P	A	0.88	0.70	0.75	0.38	0.94	0.87	0.82	0.81
P	V	0.83	0.30	0.38	0.19	0.85	0.83	0.82	0.80
M	A	0.94	0.74	0.78	0.41	0.83	0.84	0.79	0.79
M	V	0.85	0.76	0.74	0.32	0.76	0.77	0.72	0.74
S	A	0.96	0.90	0.93	0.68	0.72	0.86	0.69	0.78
S	V	0.89	0.67	0.73	0.35	0.84	0.82	0.74	0.75
A	A	0.96	0.61	0.90	0.21	0.79	0.86	0.76	0.83
A	V	0.79	0.40	0.78	0.21	0.88	0.85	0.79	0.79

Notes:

P Pizzicato, M. Morning, S. Slavonic Dance, A. Adagio

A Arousal, V. Valence

Corresponding sample good agreement (at τ₁) match count between test and retest as ratio of any (Test or Retest) samples of good agreement (i.e., no agreement at corresponding samples in both test and retest ignored).

Corresponding sample good agreement (at τ₁) match count between test and retest as a ratio of all samples (T) of the piece.

If coefficient of one of a valence-arousal pair is greater, it is shown in bold font.

The Test and Retest sample-by-sample ensemble average ΔM(t)_p,d,T and ΔM(t)_p,d,R were subjected to a Pearson product–moment correlation analysis. This produced a test-retest reliability statistic that was markedly lower, as shown in Table 3, ranging from 0.40 for Adagio Valence, to 0.90 for Slavonic Dance Arousal. Each of these correlation coefficients still produced large effect sizes, except for Adagio Valence which produced a medium effect size (Cohen, 1992). The result demonstrates a clearer picture of relatively better reliability for arousal responses in each case. Only Morning Valence produced a lower Pearson’s r (= 0.76), but this was relatively close to the Morning Arousal Test-Retest correlation coefficient (r = 0.74). To further examine test-retest correlation, a nonparametric approach was applied, where the median of the ensemble ratings were generated to produce the new Mdn(t)_p,d,T and Mdn(t) _p,d,R series. A Spearman’s rho analysis was performed on these pairs, and is reported in Table 3, as is the differenced version of the median series, ΔMdn(t). In general, all rho values are lower than their corresponding Pearson’s r. However, the comparison is similar, with arousal ratings producing relatively higher rho values. Only the Spearman rho for Adagio produced a tie between arousal and valence when the median ensemble time series was differenced, but in this case the value was low (0.21) and therefore both dimensions have dubious test-retest reliability according to the Spearman rho criterion. This analysis, too, suggests that arousal responses are more reliable. However this is only an indication of reliability from the perspective of the statistical dependency of mean rating time series. It does not provide information, about differences between rating levels nor about regions in time, which may be spurious noise rather than reliable responses. Therefore, two further analyses were performed, one examining difference in test-retest paired ratings, and the other based on good agreement across test-retest conditions.

Paired differences in ratings between corresponding Test-Retest samples for each stimulus-dimension combination were performed by generating the time series that is a result of subtracting each Test sample element of the series from the corresponding Retest condition sample element, participant by participant, as according to the formula for each sample t, of participant i:

X {(t, ζ_{i})}_{p, d, T} - X {(t, ζ_{i})}_{p, d, R}

(6)

producing a new time series for each stimulus-dimension combination, for each participant. The resulting Arousal Test-Retest rating difference series for the four stimuli by fourteen participants were pooled, as were the Valence Test-Retest rating difference series for the four stimuli by fourteen participants.

The median of the Test and Retest difference time series was 0.5 for arousal and -1.0 for valence, with 25th and 75th percentile (and 10th to 90th percentile in parentheses) values of −2 (−6) and 4 (7.85) for arousal, and -5 (−10) and 2.5 (5.5) for valence respectively. This is on a rating scale ranging from -100 to +100 – that is 201 units. Positive values indicate that the rating made in the Test condition was higher than that made in the Retest condition. So, while the differences in paired ratings between Test and Retest were on the whole small, valence ratings were slightly higher in the Retest (by a median of 5 units, compared to 2 units for arousal). Considering the 10th to 90th percentile ranges: 4/5ths of the all pairwise Test-Retest difference time-series elements fell within 13.85(= 7.85 + 6)/210 = 6.89% for arousal and 15.5(= 5.5 + 10)/201 = 7.71% for valence of the 201 point rating scale.

Next, samples with good agreement at τ₁ were identified for X(t)_p,d,T and X(t)_p,d,R (for each stimulus-dimension combination) for the purpose of providing a measure of reliability that would take into account the sample-by-sample agreement of ratings. This was performed over a two-stage process. In the first stage the total number of points that fell within the τ₁ threshold were located for each of the sixteen stimulus-dimension-condition combinations. These are shown graphically as the shaded regions in Figure 2. As mentioned, this technique identifies the points that are in good agreement compared to other (unshaded) regions. Although standard deviation is not normally distributed, central limit theory predicts that with a sufficiently large number of samples the distribution will begin to resemble normality (Salkind, 2010), in which case the ratio of points identified would be 0.84 (84%) using a threshold of τ₁. Across the sixteen stimulus-dimension-condition combinations, the proportion of high-agreement elements selected ranged from 0.79 (for Adagio Valence) to 0.94 (for Pizzicato Valence). The median of these ratios is 0.84, suggesting an effect of the central limit theory with just 16 distribution samplings (albeit with a positive skew). However, what is important is where these regions of good agreement occur.

In the second stage the corresponding element numbers in the SD1(t)_p,d,T and SD1(t)_p,d.R series were compared and a new series generated where an element number was set to 1 if both the corresponding element numbers in test and retest condition had a value below τ₁ (good agreement) and otherwise 0 (poor agreement). That is, the new corresponding sample agreement match time series is generated by producing a value of 1 when both an element of SD1(t) _p,d,T is less than τ₁, and the corresponding element of SD1(t) _p,d,T is also less than τ₁. Otherwise the corresponding element of the new series is set to zero. In Figure 2 this is equivalent to locating the points in a time series where shaded regions of good agreement overlap (intersect) across test-retest plots. The analysis is summarized using two ratios. One ratio reports the count of 1 elements of the corresponding sample agreement match time series (equivalent of summing the elements) denoted by CA, as a proportion of the number of good agreement samples in either condition (number of samples with any shading in the overlapped Test and Retest plot). This means that a sample had to be considered in good agreement in at least one condition, and would ignore samples that are in poor agreement in both conditions (CA/Any columns in Table 3). Such a calculation is not biased by the total number of high-agreement samples; poor-agreement samples will not contribute to the ratio. But it also means that a higher ratio of between-condition good-agreement matches would be reported if there were a larger number of poor-agreement matching samples in both conditions. Therefore a second ratio reports CA as a proportion of the total number of samples (CA/All columns in Table 3, where ‘all’ here means T the total number of time samples for the rating duration of the stimulus). For the moment, we will restrict our analysis to the matches identified from the undifferenced SD1(t) series.

As can be seen in Table 3, the fewest relative matches (regardless of ratio calculation) occur in Slavonic Dance Arousal (72% and 69% of samples with matches for CA/Any and CA/All, respectively). The highest ratio of matches occurred for Pizzicato Arousal, again regardless of the ratio (94% and 82% of any good match and of total samples respectively). It will now become clear why a threshold (τ₁) was chosen that identified a reasonably large number of samples in good agreement — when comparing across conditions: a more stringent τ_n criterion would have led to a loss of good-agreement points matching in both conditions. Hence, issuing an overly conservative threshold risked having no Test-Retest matches at all, thus preventing meaningful comparison.

The question at hand is to compare the reliability, according to this analysis, of arousal versus valence. The overall median for arousal match is 81% and for valence 84% as a proportion of any good agreement samples. As a proportion of all samples the overall median is 77% for both. Given the fluctuations of this proportion across stimuli, it is not possible to conclude that there is a difference in corresponding agreement between arousal and valence using this approach. In general, participants agree with each other’s responses in the same regions of a piece when responding on different occasions. According to the present calculations, based on a τ₁ threshold, this occurs at least two thirds of the time (69% worst case), is typically about 80% of the time (average of the median results for all ratios) and up to 94% of the time among the stimuli investigated. However, corresponding agreement in Valence Test-Retest is proportionally higher than Arousal for Slavonic Dance and Adagio. When the same agreement matching analysis is conducted on the differenced data – that is ΔX(t)_p,d,T with ΔX(t)_p,d,R – Arousal consistently has a higher proportion of Test-Retest matches in agreement for all stimuli compared to Valence (Table 3). Since changes in arousal are sensitive to changes in loudness (Schubert, 2004) it seems reasonable that pieces with many intense changes in dynamics (such as Slavonic Dance and Adagio) will lead to a higher number of changing responses. So, this finding is likely to be related to the Romantic style, orchestral instrumentation of the selected music.

The final analysis concerning test-retest reliability involves an examination of the standard deviation time series across each stimulus-dimension-condition combination: SD1(t)_p,d,c. Figure 3 shows a comparison of various statistics (box-plot, minimum, maximum, mean and τ_±1) applied to the SD1(t) time series. The height of each median and mean can be thought of as a ‘disagreement’ or ‘confusion’ estimate, because high mean/median indicates larger SD1(t) elements in the course of the emotional response dimension for that piece. Mean SD1(t) (i.e., ) values ranged from 21 (Slavonic Valence – both test and retest, median also 21 for both), to 28 (Adagio – both test and retest, median also 28 for both) units on a 201 point scale, which translates to 10% and 14% of the response scale range respectively. Median values produce similar results (median line of box plots are close to or overlap the large crosses representing mean, with mean displaying a small positive skew in all cases). Slavonic Dance Arousal Retest SD1(t)_S,A,R had the smallest interquartile range (6.42 units), while Adagio Arousal Test had the largest (11.5 units). The τ_±1 and interquartile range changed little from Test to Retest within stimulus and dimension, however a small trend could be observed with Retest variability being slightly smaller than Test variability (e.g., compare interquartile range of Slavonic Dance Valence, noting that it is smaller in the Retest than the Test condition). No clear trend could be observed across dimension, but overall stimulus-wise differences could be observed, with Slavonic Dance producing higher agreement, with least heteroskedasticity (see the relatively low , and small interquartile and τ_±1 range in Figure 3) of the four stimuli, and Adagio producing more poor agreement than the other stimuli (both according to τ_±1 and interquartile distances). Adagio Valence had the highest deviation scores, with a peak (maximum) of 49 units for both Test and Retest conditions (24% of the 201 point scale). Across the entire responses, collapsed across stimulus-dimension-condition, the mean standard deviation was 26.6 units, which is 12.2% of the response scale range (median = 12.6%).

Figure 3.

Central tendency (mean and median) and spread (sd and box plot) of SD1(t)_p,d,c.

Afterglow

Afterglow was calculated by examining the SD1(t_T)_p,d,c values, the ensemble standard deviations of the last sample, calculated for each stimulus-dimension-condition combination, which was approximately three seconds after the offset of the last note of the piece (see Discography for t_T values of each stimulus). It was not possible to obtain a precise offset time as different recordings had different amounts of reverberation. To put the final sample deviation score into perspective, it was divided by sd2. Table 4 lists SD1(t_T)_p,d,c for each stimulus-dimension-condition combination, both in rating scale units and as a multiple of sd2. For comparison, the initial sample standard deviation, SD1(t₁)_p,d,c is shown, which was expected to be small because all participants commenced their response from the same, central region on the emotion space. The peak SD1(t) value of each series is also shown. The overall highest peak SD1(t)_p,d,c value was 9.64 sd2 units which occurred in the last sample of the Slavonic Dance Arousal. This is symptomatic of the final, SD1(t_T)_p,d,c, values, with values close to or matching the peak SD1(t) value. The average of the peak values across stimuli and dimensions was 6.72 sd2 units, whereas the final, afterglow SD1(t_T) values were 6.00 sd2 units. For comparison, the initial SD1(t₁)_p,d,c averaged only 0.3 sd2 units across stimulus-dimension-condition combinations. No obvious trends could be identified in afterglow for emotion dimension, or for condition (Test versus Retest). But the afterglow effect, indicated by large disagreement in ratings, is clearly present.

Table 4.

SD1(t) values at beginning, peak and ending (afterglow) of ensemble ratings and as proportions of sd2.

Morning	arousal test	arousal retest	valence test	valence retest	Mean	Min	Max
Opening: SD1(t₁)	0.98	0.96	1.6	1.02	1.14	0.96	1.6
Peak SD1(t)	43.44	42.05	40.8	39.49	41.44	39.49	43.44
Afterglow: SD1(t_T)	37.69	30.11	35.44	33.33	34.14	30.11	37.69
sd2	7.78	6.18	5.45	5.95	6.34	5.45	7.78
Opening: SD1(t₁)/sd2	0.13	0.16	0.29	0.17	0.19	0.13	0.29
Peak SD1(t)/sd2	5.58	6.8	7.49	6.63	6.63	5.58	7.49
Afterglow: SD1(t_T)/sd2	4.84	4.87	6.51	5.6	5.45	4.84	6.51
Pizzicato	arousal test	arousal retest	valence test	valence retest	Mean	Min	Max
Opening: SD1(t₁)	0.77	1.42	0.7	0.77	0.92	0.7	1.42
Peak SD1(t)	35.93	31.53	34.41	29.91	32.94	29.91	35.93
Afterglow: SD1(t_T)	29.35	25.35	30.37	23.94	27.25	23.94	30.37
sd2	5.08	4.98	4.1	3.17	4.33	3.17	5.08
Opening: SD1(t₁)/sd2	0.15	0.29	0.17	0.24	0.21	0.15	0.29
Peak SD1(t)/sd2	7.07	6.33	8.39	9.44	7.81	6.33	9.44
Afterglow: SD1(t_T)/sd2	5.78	5.09	7.4	7.56	6.46	5.09	7.56
Slavonic Dance	arousal test	arousal retest	valence test	valence retest	Mean	Min	Max
Opening: SD1(t₁)	4.37	1.83	4.13	3.59	3.48	1.83	4.37
Peak SD1(t)	43.12	43.86	37.63	36.99	40.4	36.99	43.86
Afterglow: SD1(t_T)	39.13	43.86	37.02	36.99	39.25	36.99	43.86
sd2	5.71	4.55	5.13	4.08	4.87	4.08	5.71
Opening: SD1(t₁)/sd2	0.77	0.4	0.8	0.88	0.71	0.4	0.88
Peak SD1(t)/sd2	7.55	9.64	7.33	9.07	8.4	7.33	9.64
Afterglow: SD1(t_T)/sd2	6.85	9.64	7.21	9.07	8.19	6.85	9.64
Adagio	arousal test	arousal retest	valence test	valence retest	Mean	Min	Max
Opening: SD1(t₁)	0.92	0.96	1.22	0.98	1.02	0.92	1.22
Peak SD1(t)	42.74	40.45	49.08	46.92	44.8	40.45	49.08
Afterglow: SD1(t_T)	41.68	36.19	28.03	24.71	32.65	24.71	41.68
sd2	6.41	4.34	6.67	5.4	5.7	4.34	6.67
Opening: SD1(t₁)/sd2	0.14	0.22	0.18	0.18	0.18	0.14	0.22
Peak SD1(t)/sd2	6.67	9.32	7.36	8.69	8.01	6.67	9.32
Afterglow: SD1(t_T)/sd2	6.5	8.34	4.2	4.58	5.91	4.2	8.34
Means across stimuli	arousal test	arousal retest	valence test	valence retest	Mean	Min	Max
Opening: SD1(t₁)	1.76	1.29	1.91	1.59	1.76	0.7	4.37
Peak SD1(t)	41.31	39.47	40.48	38.33	41.31	29.91	49.08
Afterglow: SD1(t_T)	36.96	33.88	32.71	29.75	36.96	23.94	43.86
sd2	6.25	5.01	5.34	4.65	6.25	3.17	7.78
Opening: SD1(t₁)/sd2	0.3	0.27	0.36	0.37	0.3	0.13	0.88
Peak SD1(t)/sd2	6.72	8.02	7.64	8.46	6.72	5.58	9.64
Afterglow: SD1(t_T)/sd2	5.99	6.98	6.33	6.7	5.99	4.2	9.64

Discussion

Initial orientation time

Initial orientation time varies from 2 to 35 seconds, with an overall median of 8 seconds for the stimuli investigated, consistent with Bachorik et al. (2009). This means that the first seconds of response in a continuous response task must be treated with caution. It is likely that the present estimates are low because substantive training was provided, using examples from all quadrants of the emotion space, with worked examples, and asking the participants to perform the task with words and pictures of faces before commencing continuous responses to music. The reliability of continuous responses can therefore be improved by disregarding an opening period of response. Caution is also required when no training is given. Further studies should explore whether there is a training–orientation-time trade off, for if little instruction is provided, perhaps the participant will find their way around the response space or scale, but to reach a point of reliability will take longer (longer initial orientation time) which can later be diagnosed and discarded if necessary.

The present study does not, however, provide a dictum as to how long the initial orientation time will be in all or even similar continuous emotional response paradigms. The variability of the results provides evidence for this. It seems that initial orientation time is more complex than a participant needing a cognitive adjustment and processing time. It could be that some music facilitates consistent emotional response ratings more rapidly, and it might be that certain musical characteristics or musical style affect this time requirement (Bachorik et al., 2009). The overall finding that valence had a shorter initial orientation time than arousal (4 s versus 12 s median times) suggests that valence judgements are things with which we are well experienced – evaluating whether something is positive or negative. But this is a dubious conclusion because arousal may be even more privileged in speed of response (e.g., see Cuthbert, Schupp, Bradley, Birbaumer, & Lang, 2000). A more plausible explanation is that there are more clear and unambiguous indicators of valence, in particular mode, than arousal for the examples used. Morning, Pizzicato and Slavonic Dance all produce a clear expression of major tonality at the very opening of the piece, and hence a clear positive valence (Pittenger, 2003; Webster & Weir, 2005), although Morning may produce a slight ambiguity as the melody outlines a descending minor third from the fifth degree until it reaches the end of its descent at the tonic which is clearly in the major mode. Adagio is in a natural minor mode, and this provides a strong sense of negative valence. Yet, it produces long initial orientation times of 15 s in the test and 25 s in the retest condition (though still shorter than the corresponding arousal response). It is therefore somewhat a surprise that Morning produced a median initial orientation time of 4 s in the retest (same as the overall median) instead of something extended as in Adagio. Tempo may provide some explanation as the shortest initial orientation time was for the fastest piece, Slavonic Dance, and the slowest was for Adagio, regardless of emotional response dimension or Test-Retest condition. So, musical characteristics may well have a role to play, and selection of music may therefore have a bearing on the reliability of responses at the start of a continuous emotional response task.

Test-retest reliability

The emotion ratings were examined in different ways to investigate the reliability of continuous emotional ratings. Because there are some moot issues regarding the reporting of test-retest reliability for continuous data, the focus was on comparative reliability, and in particular whether arousal responses are more, less, or equally reliable to valence responses. We have seen, above, that valence response has some advantage in terms of smaller initial orientation time. However, Test-Retest correlation of first-order differenced data suggested that arousal reliability is higher than valence reliability (average r = 0.79 and 0.54 respectively, undifferenced r = 0.97 and 0.92 respectively). But is this sufficient to conclude that continuous arousal response is more reliable than continuous valence response? Does it mean that when the two are combined onto an emotion-space at right angles that valence responses suffer? Correlations are sensitive to the amount of systematic variance in the variables under comparison. For example, the variability in the longest piece, Adagio – lasting over ten minutes – is relatively small in valence response, certainly compared to the responses to the other pieces (see Figure 2). So, it could be that the lack of variability was responsible for the lowest correlation coefficient, of 0.37, for differenced Adagio Valence response, rather than a lack of reliability per se. To conclude that arousal responses are more reliable than valence responses would therefore necessarily be tentative.

A method that takes into account the within-participant variability was applied as another way of examining test-retest reliability. By comparing samples of relatively high agreement using the second order standard deviation method between Test-Retest conditions, no major differences could be identified, and no difference could be identified between valence and arousal. While further work is needed to determine what an adequate measure and level of reliability is, the present study proposes a technique that provides an alternative, but more so an addendum, to Pearson correlations in those instances where the correlation coefficient may be susceptible to rogue effects. For the method used, the typical level of reliability was quantified at 80%, meaning that participant responses were in relatively high agreement at the same points in time across test and retest conditions (using a τ₁ threshold criterion). This reliability varied from 69% to 94% of the time across the various stimulus-dimension combinations.

The final comparative analysis of reliability directly examined the mean of the standard deviations for each stimulus-dimension combination for Test and Retest conditions. The results indicate similar standard deviation (agreement) in the Retest condition. This result does not support a cognitive load explanation, which predicts that, because of practice, the participant does not require as much extraneous cognitive load for operating the interface and can focus more on the experimental task (Paas, Tuovinen, van Merrienboer, & Darabi, 2005; Sweller, 1988; Sweller, 2006). It would be interesting to see whether subsequent retests would further change agreement. If emotional ratings of music change due to multiple listenings (Balkwill & Thompson, 1999; Ritossa & Rickard, 2004), then it is likely that deviation scores will also change. However, it is more likely that this will be the case when it is the felt emotion that is rated, rather than (as in the present study) that which the music appears to be conveying (Grewe et al., 2007b; Salimpoor et al., 2009; Schubert, 2007a).

Afterglow

Participants are meant to return the cursor to the centre of the emotion space when the piece ends. They should do this because they were instructed to do so, but also, logically, if rating the emotion expressed by the music, and the music stops, the music cannot be expressing further emotion. The analysis provided evidence for the presence of an afterglow effect, caused by some participants not returning to the centre at the conclusion of the music. The lowest relative spread of scores at the end of any stimulus-dimension combination was 4.2 times the sd2 of Adagio Valence, and the highest relative spread of scores was 9.64 times the sd2 for Slavonic Dance Arousal which corresponds to the largest ‘outlier’ point for that entire piece. While there were no systematic differences in median afterglow values across dimensions, the values were slightly higher for the median Retest condition. There is, therefore, an unreliable response at the end of the continuous emotion rating task and this presents a message to continuous response researchers that the last portion of a response, like the initial orientation time, should be treated with caution, and possibly deleted from analysis if appropriate.

The explanation of the afterglow effect can only be speculated upon here. Some of the participants seem to become unaware of the requested task – unable to report the emotion that the music is expressing, and confusing the request (expressed emotion) rating with their own feelings. Another (though not independent) explanation is that the serial correlation of the recent responses has bled through to after the end of the piece. If this is the case, it is a further vindication of the differencing technique, which can mathematically reduce the effect of such inertial responses. In non-music literature, a similar effect is referred to as response inhibition – the time it takes to stop performing a task upon presentation of a ‘stop task’, though this inhibition time delay is of a smaller magnitude than the durations found in the present study (see Pessoa, Padmala, Kenzer, & Bauer, 2011; Verbruggen & Logan, 2009). Differencing ratings is an alternative to deleting data in cases where it is desirable to use as much of the time series as possible.

The second order deviation threshold method

The second order deviation threshold method was used as a tool to help induce criteria for sample-by-sample agreement in a time series. It is a flexible method, and while the present study used only mean and standard deviation calculations to determine the good agreement threshold criteria, other statistics should be considered in the future. For example, instead of the mean, a less biased central tendency estimator, such as the median, could be used (see also Grewe et al., 2007a; Grewe et al., 2009a; Korhonen et al., 2006). Further, alternative estimates of the spread of scores should be considered, such as the interquartile distance (Grewe et al., 2009a; Grewe et al., 2007a) and the median absolute distance (Schubert et al., in press). So, for example, for the present study, the median, first order interquartile and second order interquartile could be applied in an analogous manner instead of the mean, first order standard deviation and second order deviation respectively. Some examples of these nonparametric statistics were reported in this study.

However the second order deviation threshold does have limitations. It is a relative measure of reliability. Since SD1(t) data are presented as a ratio of sd2 the method is identical in principle to effect size (Cohen, 1992), and is appropriate for comparing conditions, in the present case arousal and valence responses. In the present study, the technique, used along with other approaches, provides a tool for diagnosing initial orientation time, reliability and afterglow effects in continuous emotional responses to music. It demonstrates the presence of potentially shorter initial orientation time in valence, with respect to arousal, no difference (or equal reliability) across Test-Retest conditions for both arousal and for valence responses, and the presence of an unreliable or misleading ‘afterglow’ at the conclusion of the piece.

Conclusion

Test-retest reliability of four pieces of non-vocal, orchestrals music made by fourteen participants separated by a long period of time (six to twelve months) suggests that there is good agreement in identifying emotion expressed by music. The method of analysis employed confirmed and identified the presence of some important characteristics of time-series data collected from emotional response tasks to music. While it can be concluded that test-retest responses are generally reliable – the current approach reporting agreement across conditions about 80% of the time – the opening moments of the music and the end of the music in general produce unreliable results. Responses to the start of a piece of music are marked by a period of orientation time that, in the present study, ranged from a couple of seconds to over thirty seconds. It was longer for arousal response than for valence response, and was typically eight seconds long. The conclusion drawn is that the initial emotional responses need to be treated with caution and perhaps eliminated from some analyses. It is also speculated that training may reduce the orientation time, but repeated exposures may alter emotional responses, though these issues were not an explicitly manipulated in the present study.

Researchers are aware of the unreliability of the responses to the opening section of a piece of music – for example, Grewe et al. (2007a) omitted for first 10 seconds of their response data. However, the present study provides a technique for diagnosing the duration of this orientation time. This will increase the amount of ‘useful’ response time that can be analysed, and highlights the variability of initial orientation time, which seems to be a function of musical features, with tempo being speculated as an important factor – slow pieces used in the present study had longer orientation times, and fast pieces had shorter ones.

Afterglow effects were also quantified in this study, and indicate the potential lack of reliability at the end of a continuous rating of expressed emotion by music task. Some of the largest deviation scores were identified at the responses just after the music ended, despite the instruction (returning to the neutral point) which should have produced very small ensemble rating deviations.

Continuous response methods offer exciting insights into how emotion in music works. The present study demonstrated how variations in responses can be identified visually, and can therefore be adapted to more sophisticated problems such as modelling the response as a function of other variables (e.g., Korhonen et al., 2006; Schubert, 2004) in the future. Further work is needed to investigate the nature of these various factors that influence reliability, and a wider range of musical styles (Bachorik et al., 2009) will determine whether the results are limited to Western orchestral music from part of the last two centuries. As continuous response techniques are adopted by emotion in music researchers, the implications of issues such as reliability at the beginning, middle and end of the time dependent data sets need also be considered if the results of the data sets and subsequent conclusions are to be optimized.

Footnotes

Acknowledgements

This research was conducted with support from the Australian Research Council (DP0452290 and DP0986153). The author is grateful for the comments of two anonymous reviewers, and for the continuing support of Visiting Associate Eric Sowey and Professor William Dunsmuir.

Notes

Author biography

Emery Schubert is Associate Professor in the School of the Arts and Media at the University of New South Wales in Sydney, Australia and co-leader of the Empirical Musicology Group. His research interests are concerned with the ‘scientific’ study of aesthetics, including measuring and predicting emotional responses to music continuously. He is on the Editorial Board for Empirical Musicology Review and Journal of New Music Research, and was President of the Australian Music and Psychology Society (AMPS).

References

Dvorak

Anton

. Slavonic Dance, Op.46. No.1. Slovak Philharmonic Orchestra, Zdenek Kosler (conductor). Discover Classical Music, CD2. (1993) HNH International Ltd. 8.550008–9, track 9. Duration 3 minutes 45 seconds (t_T = 225s).

Strauss

Johann

(Jr.) Strauss

Joseph

. Pizzicato Polka. CSR Symphony Orchestra, Ondrej Lenard (conductor). Discover Classical Music, CD2. (1993) HNH International Ltd. 8.550008–9, track 11. Duration 2 minutes 30 seconds (t_T = 150s).

Grieg

Edvard

. Peer Gynt – Morning. Czecho-Slovak Radio Symphony Orchestra, Stephen Gunzenhauser (conductor). Discover Classical Music, CD2. (1993) HNH International Ltd. 8.550008–9, track 13. Duration 3 minutes 38 seconds (t_T = 218s).

Rodrigo

Joaquin

, Concierto De Aranjuez – Adagio. Gerald Garcia (guitar), CSSR State Philharmonic Orchestra, Peter Breiner (conductor). Discover Classical Music, CD2. (1993) HNH International Ltd. 8.550008–9, track 17. Duration 10 minutes 52 seconds (t_T = 652s).

Asmus

E. P.

(1985). The development of a multidimensional instrument for the measurement of affective responses to music. Psychology of Music, 13, 19–30.

Bachorik

J. P.

Bangert

Loui

Larke

Berger

Rowe

. (2009). Emotion in motion: Investigating the time-course of emotional judgments of musical stimuli. Music Perception, 26(4), 355–364.

Balkwill

L.-L.

Thompson

W. F.

(1999). A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues. Music Perception, 17(1), 43–64.

Barrett

L. F.

(1998). Discrete emotions or dimensions? The role of valence focus and arousal focus. Cognition, & Emotion, 12(4), 579–599.

Box

G. E. P.

Jenkins

G. M.

Reinsel

G. C.

(1994). Time series analysis: Forecasting and control (3rd ed). New Jersey: Prentice-Hall.

10.

Cepeda

N. J.

Pashler

Vul

Wixted

J. T.

Rohrer

(2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological bulletin, 132(3), 354–380.

11.

Christie

I. C.

Friedman

B. H.

(2004). Autonomic specificity of discrete emotion and dimensions of affective space: A multivariate approach. International Journal of Psychophysiology, 51(2), 143–153.

12.

Cohen

(1992). A power primer. Psychological Bulletin, 112(1), 155–159.

13.

Cowie

Douglas-Cowie

Savvidou

McMahon

Sawey

Schroeder

(2000). FEELTRACE: An instrument for recording perceived emotion in real time. In Cowie

Douglas-Cowie

Schroeder

(Eds.), Speech and Emotion: Proceedings of the ISCA workshop (pp. 19–24). Newcastle, Co. Down, UK.

14.

Cuthbert

B. N.

Schupp

H. T.

Bradley

M. M.

Birbaumer

Lang

P. J.

(2000). Brain potentials in affective picture processing: Covariation with autonomic arousal and affective report. Biological Psychology, 52(2), 95–111.

15.

Ekman

Friesen

W. V.

(1975). Unmasking the face: A guide to recognizing emotions from facial clues. Oxford: Prentice-Hall.

16.

Eerola

Vuoskoski

J. K.

(2011). A comparison of the discrete and dimensional models of emotion in music. Psychology of Music, 39(1), 18–49.

17.

Engle

R. F.

(2001). GARCH 101: The use of ARCH/GARCH model in applied economics. Journal of Economic Perspectives, 15(4), 157–168.

18.

Evans

Schubert

(2008). Relationships between expressed and felt emotions in music. Musicae Scientiae, 12(1), 75–99.

19.

Gabrielsson

Lindström

(2001). The influence of musical structure on emotional expression. In Juslin

P. N.

Sloboda

J. A.

(Eds.), Music and emotion: Theory and research (pp. 223–248), Oxford: Oxford University Press.

20.

Gabrielsson

(2002). Perceived emotion and felt emotion: Same or different? Musicae Scientiae, 6(1; Special Issue), 123–148.

21.

Geringer

J. M.

Madsen

C. K.

Gregory

(2004). A fifteen-year history of the Continuous Response Digital Interface: Issues relating to validity and reliability. Bulletin of the Council for Research in Music Education(160), 1–15.

22.

Gottman

J. M.

(1981). Time-series analysis: A comprehensive introduction for social scientists. Cambridge: Cambridge University Press.

23.

Grewe

Kopiez

Altenmüller

(2009a). The chill parameter: Goose bumps and shivers as promising measures in emotion research. Music Perception, 27(1), 61–74.

24.

Grewe

Kopiez

Altenmüller

(2009b). Chills as an indicator of individual emotional peaks. The Neurosciences and Music III: Disorders and Plasticity, 1169, 351–354.

25.

Grewe

Nagel

Kopiez

Altenmüller

(2007a). Emotions over time: Synchronicity and development of subjective, physiological, and facial affective reactions to music. Emotion, 7(4), 774–788.

26.

Grewe

Nagel

Kopiez

Altenmüller

(2007b). Listening to music as a re-creative process: Physiological, psychological, and psychoacoustical correlates of chills and strong emotions. Music Perception, 24(3), 297–314.

27.

Hevner

(1936). Experimental studies of the elements of expression in music. American Journal of Psychology, 48, 246–268.

28.

Korhonen

M. D.

Clausi

D. A.

Jernigan

M. E.

(2006). Modeling emotional content of music using system identification. IEEE Transactions on Systems Man and Cybernetics Part B-Cybernetics, 36(3), 588–599.

29.

Laukka

(2008). Research on vocal expression of emotion: State of the art and future directions. Emotions in the Human Voice, 1, 153–169.

30.

Lucas

B. J.

Schubert

Halpern

A. R.

(2010). Perception of emotion in sounded and imagined music. Music Perception, 27(5), 399–412.

31.

McAdams

Vines

B. W.

Vieillard

Smith

B. K.

Reynolds

(2004). Influences of large-scale form on continuous ratings in response to a contemporary piece in a live concert setting. Music Perception, 22(2), 297–350.

32.

Nagel

Kopiez

Grewe

Altenmüller

(2007). EMuJoy: Software for continuous measurement of perceived emotions in music. Behavior Research Methods, 39(2), 283–290.

33.

Paas

Tuovinen

J. E.

van Merrienboer

J. J. G.

Darabi

A. A.

(2005). A motivational perspective on the relation between mental effort and performance: Optimizing learner involvement in instruction. Etr&D-Educational Technology Research and Development, 53(3), 25–34.

34.

Pessoa

Padmala

Kenzer

Bauer

(2011). Interactions between cognition and emotion during response inhibition. Emotion. Advance online publication. doi: 10.1037/a0024109

35.

Pittenger

R. A.

(2003). Affective and perceptual judgments of major and minor musical stimuli. Dissertation Abstracts International: Section B: the Sciences, & Engineering, 63(7-B), 3491.

36.

Ramsay

J. O.

Silverman

B. W.

(2002). Applied functional data analysis: Methods and case studies. New York: Springer.

37.

Ritossa

D. A.

Rickard

N. S.

(2004). The relative utility of ‘pleasantness and liking’ dimensions in predicting the emotions expressed by music. Psychology of Music, 32(1), 5–22.

38.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Social Psychology, 39, 1161–1178.

39.

Russell

J. A.

Weiss

Mendelsohn

G. A.

(1989). Affect grid: A single-item scale of pleasure and arousal. Journal of Personality and Social Psychology, 57(3), 493–502.

40.

Salimpoor

Benovoy

Longo

Cooperstock

Zatorre

(2009). The rewarding aspects of music listening are related to degree of emotional arousal. PloS one, 4(10), 29–49.

41.

Salkind

N. J.

(Ed.). (2010). Encyclopedia of research design. Thousand Oaks, CA: Sage Publications, Inc.

42.

Schubert

(1999). Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space. Australian Journal of Psychology, 51(3), 154–165.

43.

Schubert

(2001). Continuous measurement of self-report emotional response to music. In Juslin

P.N.

Sloboda

J. A.

(Eds.), Music and emotion: Theory and research (pp. 393–414). Oxford: Oxford University Press.

44.

Schubert

(2002). Correlation analysis of continuous emotional response to music: Correcting for the effects of serial correlation. Musicae Scientiae, Spec Issue, 213–236.

45.

Schubert

(2004). Modeling perceived emotion with continuous musical features. Music Perception, 21, 561–585.

46.

Schubert

(2007a). Locus of emotion: The effect of task order and age on emotion perceived and emotion felt in response to music. Journal of Music Therapy, 44(4), 344–368.

47.

Schubert

(2007b). Real time cognitive response recording. In Schubert

Buckley

Eliott

Koboroff

Chen

Stevens

(Eds.), Proceedings of the Inaugural International Conference on Music Communication Science (pp. 139–142). Sydney, Australia: ARC Research Network in Human Communication Science (HCSNet), University of Western Sydney.

48.

Schubert

(2007c). When is an event in a time-series significant? In Schubert

Buckley

Eliott

Koboroff

Chen

Stevens

(Eds.), Proceedings of the Inaugural International Conference on Music Communication Science (pp. 135–138). Sydney, Australia: ARC Research Network in Human Communication Science (HCSNet), University of Western Sydney.

49.

Schubert

(2010). Continuous self-report methods. In Juslin

P. N.

Sloboda

J. A.

(Eds.), Handbook of music and emotion: Theory, research, applications (pp. 223–253). Oxford: OUP.

50.

Schubert

Dunsmuir

(1999). Regression modelling continuous data in music psychology. In Yi

S. W.

(Ed.), Music, mind, and science (pp. 298–352). Seoul: Seoul National University.

51.

Schubert

Vincs

Stevens

C. J.

(in press). Identifying regions of good agreement among responders in engagement with a piece of live dance. Empirical Studies of the Arts.

52.

Stevens

Schubert

Haszard Morris

Frear

Chen

Healey

. (2009). The portable Audience Response Facility (pARF): PDAs that record real-time and instantaneous data during live or recorded performance. International Journal of Human-Computer Studies, 67, 800–813.

53.

Sweller

(1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.

54.

Sweller

(2006). Discussion of ‘Emerging topics in cognitive load research: Using learner and information characteristics in the design of powerful learning environments’. Applied Cognitive Psychology, 20(3), 353–357.

55.

Tay

T. T.

Mareels

Moore

J. B.

(1998). High performance control. Boston: Birkhauser.

56.

Verbruggen

Logan

G. D.

(2009). Models of response inhibition in the stop-signal and stop-change paradigms. Neuroscience, & Biobehavioral Reviews, 33(5), 647–661.

57.

Vines

B. W.

Krumhansl

C. L.

Wanderley

M. M.

Levitin

D. J.

(2006). Cross-modal interactions in the perception of musical performance. Cognition, 101(1), 80–113.

58.

Webster

G. D.

Weir

C. G.

(2005). Emotional responses to music: Interactive effects of mode, texture, and tempo. Motivation and Emotion, 29(1), 19–39.

59.

Zentner

Grandjean

Scherer

K. R.

(2008). Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion, 8(4), 494–521.