What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody

Abstract

The current study investigated what can be understood from another person’s tone of voice. Participants from five English-speaking nations (Australia, India, Kenya, Singapore, and the United States) listened to vocal expressions of nine positive and nine negative affective states recorded by actors from their own nation. In response, they wrote open-ended judgments of what they believed the actor was trying to express. Responses cut across the chronological emotion process and included descriptions of situations, cognitive appraisals, feeling states, physiological arousal, expressive behaviors, emotion regulation, and attempts at social influence. Accuracy in terms of emotion categories was overall modest, whereas accuracy in terms of valence and arousal was more substantial. Coding participants’ 57,380 responses yielded a taxonomy of 56 categories, which included affective states as well as person descriptors, communication behaviors, and abnormal states. Open-ended responses thus reveal a wide range of ways in which people spontaneously perceive the intent behind emotional speech prosody.

Keywords

appraisal theory emotion process emotion recognition free responses speech prosody vocal expression

The study of emotion perception has been a long-standing and fundamental research topic within psychology (e.g., Darwin, 1872/1965; Schlosberg, 1952). A better understanding of how individuals perceive nonverbal cues of emotion has bearing on theoretical issues at the heart of what we understand about emotion as well as practical issues in our understanding of interpersonal effectiveness and functioning (Buck, 2014; Hall et al., 2009; Rosenthal et al., 1979). The most common research in this tradition has investigated still photographs of facial expressions (Calvo & Nummenmaa, 2016; Elfenbein & Ambady, 2002). In addition to facial behavior, the voice is another important channel of nonverbal communication. The voice provides a rich source of effective information—as the old expression goes, it is not what you say but how you say it. Indeed, vocal cues are the most frequently reported cues to emotion in naturally occurring situations (Planalp et al., 1996). The current investigation attempts to add to what we know about perceiving vocal emotions and asks the basic question: What can be understood from another person’s tone of voice? In order to get a nuanced picture of the ways in which people spontaneously perceive emotional speech prosody, we investigate open-ended judgments of a wide range of both positive and negative emotions.

Open-Ended Judgment of Emotional Expression

In the most common source of evidence for the perception of emotion, participants are presented with stimuli that are intended to express various emotional states—usually performed by actors—and judgments are recorded by instructing participants to select the most appropriate label from a fixed list of options. This is called the forced-choice paradigm. Although it has provided much insight, the dominant paradigm has also been the subject of critique. Typical forced-choice studies include a relatively small number of options, which may artificially inflate recognition rates due to various reasons. Studies on facial expression suggest that perceivers tend to use informed guessing strategies (Russell, 1994). Some states may have clear signal characteristics, whereas other states do not and yet still appear to be well recognized after other response choices are eliminated. In implementing such strategies, perceivers may judge the same stimulus differently depending on which response options are available (Frank & Stennett, 2001; Russell, 1993). DiGirolamo and Russell (2017) even found that participants could reach consensus on nonsense options when other options were made available to them strategically (see also Nelson & Russell, 2016). The current work attempts to expand the body of evidence outside of this dominant paradigm. In particular, we use open-ended responses that do not limit participants to experimenter-imposed selections.

The forced-choice design is the most common because it is convenient and readily analyzed, and yet it does not capture all the nuances and complexities that may be present in people’s emotion inferences. Because the vast majority of existing research uses the forced-choice design, we know little about how listeners interpret emotion expressions naturalistically. The use of open-ended responses may be the most suitable for assessing participants’ spontaneous impressions, but this free response method is relatively rare due to the work required for coding, and the challenge in defining accurate performance.

Given this extra difficulty, typically free-response studies have added the constraint that respondents should only provide answers using single-word emotion labels—that is, they are free-labeling, rather than completely open-ended judgments. Several studies have compared recognition accuracy between free-labeling and forced-choice methods for facial expressions (e.g., Boucher & Carlson, 1980; Limbrecht-Ecklundt et al., 2013; Rosenberg & Ekman, 1995), vocal expressions (e.g., Abelin & Allwood, 2000; Gendron et al., 2014; Greasley et al., 2000), and music performance (Juslin, 1997). These studies show that free labeling results in accuracy that is lower than that of forced-choice but still acceptable. Restricting comparisons to samples judging stimuli from their own culture, accuracy with exact word matches ranged from 20.7% (Greasley et al., 2000) to 78.4% (Rosenberg & Ekman, 1995).

In the current study, we use free responses rather than free labeling, which has been used only in a few studies. Frijda (1953) asked participants to rate film clips and film still images of facial expressions, and to describe what might be going on with the person who was being shown. He found that judges scored relatively well (about 50% received scores of 3 or 4 on an accuracy scale of 0–4 based on how close the response was to the intended state). Interestingly, he also found that they frequently described the situations that might elicit emotion rather than emotional states per se. Haidt and Keltner (1999) showed participants’ photographs of facial expressions and asked them to describe what happened to make a person feel that way. They also found that participants recognized the emotions with a wide range of success rates (from 16% for shame to 94% for happiness), and that the open-ended responses often included a range of situational descriptions. We expand on this small amount of existing work by studying the open-ended judgments of emotional prosody.

Emotional Prosody Across Emotions

This research focuses on one particular kind of vocal expression of emotion, namely vocal or speech prosody—that is, the tone of voice used while speaking words. Emotional prosody holds a special place among channels of nonverbal communication, because it is the only channel that necessarily involves simultaneous verbal communication.

Most research on the recognition of emotion via the voice has focused on a small number of emotions that are relatively well recognized (see Juslin & Laukka, 2003), usually, those that have historically been considered “basic” on the basis of research with facial expression—namely, anger, fear, disgust, happiness, sadness, and surprise (e.g., Ekman, 1992). Meta-analyses of forced-choice studies have shown that basic emotion categories are generally recognized at accuracy rates well above chance, both within and across cultures (Elfenbein & Ambady, 2002; Juslin & Laukka, 2003). Recent research in the area of vocal bursts (e.g., crying, laughter, grunts, and sighs), which are brief non-linguistic vocalizations that communicate emotion (Scherer, 1994), has started to expand the number of emotions. For example, Cordaro et al. (2016) examined vocal bursts of 16 emotional states and found above-chance accuracy in the recognition of the positive emotions of amusement, awe, contentment, desire for food and sex, interest, relief, serenity, and triumph, and the negative emotions of anger, contempt, disgust, embarrassment, fear, pain, sadness, and surprise.

The current work tests a far larger number of states than most other prosody research, including nine positive states, nine negative states, and neutral. The positive emotions are affection, amusement, happiness, interest, lust, pride, relief, serenity, and (positive) surprise. Although the small number of basic emotions tested typically includes only one positive state, namely, happiness, there has been more recent interest in expanding the number of positive states being tested in general (Sauter, 2017). Research suggests that a variety of positive emotions can be recognized from vocal bursts (Cordaro et al., 2016; Simon-Thomas et al., 2009), and the current study on vocal prosody therefore also samples a large number of positive states. The negative emotions are anger, contempt, disgust, distress, fear, guilt, sadness, shame, and (negative) surprise. Overall, these 18 emotions were selected to represent a diverse set that includes basic emotions, self-conscious emotions, interpersonal emotions, low arousal emotions, and emotions that are rarely studied in emotion recognition research. They vary in the extent to which they are considered prototypical (Fehr & Russell, 1984). It is important to include a wide range of emotions in open-ended studies to be able to capture the possible variety of judgments.

Research Questions

We attempt to expand the existing body of work in a number of ways. This study focuses on the underemphasized nonverbal channel of emotional prosody, attempts to expand the methodological repertoire beyond the dominant research paradigm, and samples from a wide variety of positive and negative emotions. It is a large-scale attempt to understand what listeners can hear in the voice. Given the relative dearth of research on the open-ended understanding of emotional prosody, we organize our investigation around three research questions, rather than around traditional hypothesis testing.

These research questions revolve around the more general question: What can people hear in the voice? Participants were provided with an open-ended task, namely to describe what they thought the speaker in each prosody stimulus was trying to express to the listener. We approach the analysis of participant responses in two different ways. In the first analysis, we examined the location of their response within the emotion componential process (Frijda, 2007; Scherer & Moors, 2019), which is described in the section that follows. In the second analysis, we examined the content of the response. That is, we coded the underlying meaning of the text that they wrote, notably the emotion that participants listed if any. This meaning was examined for its accuracy vis-à-vis the intended emotional state as well as for developing a typology of these responses.

Question 1: What Aspects of Emotion Do We Hear in Vocal Prosody?

In this research, we explore what reaction participants generate naturally with little constraint. Without instructing participants as to what the stimuli should contain, we ask them to describe what the speaker is trying to express to the listener.

Most research on the perception of effect takes as given that the stimuli will be judged as expressing a feeling state. This assumption is reflected in the nature of the response choices, which typically refer to discrete emotional categories. However, providing only feeling states as response choices may be presumptive (Russell, 1994). Vocal expression has the ability to convey not only categories but also emotion dimensions such as activation, valence, and potency (Belyk & Brown, 2014; Goudbeek & Scherer, 2010; Laukka et al., 2005). Listeners may also hear other phenomena in emotional prosody, including affect-related person attributes such as dominance and politeness (Hönemann et al., 2015; Shochi et al., 2009). The current study steps back from the feeling state assumption and actively evaluates what people think about emotional expressions.

Emotional feeling states are considered just one stage of the larger emotion process that consists of multiple components (Frijda, 2007; Scherer & Moors, 2019). These are: (a) the Situation, which is the underlying stimulus, event, or cause that elicits an emotion (e.g., “good news,” “someone did something to him”), (b) Cognitive appraisal, which is the evaluation of a situation, notably along the dimensions predicted by appraisal theories (e.g., Ellsworth & Scherer, 2003; Laukka & Elfenbein, 2012; Nordström et al., 2017), such as a stimulus’ relevance, goal conduciveness, normative significance, predictability, and the individual’s coping potential for the situation (e.g., “expected,” “important”), (c) Feeling state, which is the individual’s subjective emotional state (e.g., “angry,” “anxious”), (d) Physiological arousal, which is the individual’s level of arousal experienced (e.g., “excited,” “calm”), (e) Expressive behavior, which is any verbal or nonverbal behavior that conveys an emotional state either intentionally or unintentionally (e.g., “crying,” “laughing”), (f) Emotion regulation, which is any attempt to change an individual’s feeling states or their expression (e.g., “concealing,” “hiding”), and (g) Influence, which are any actions taken with respect to another person (e.g., “commanding,” “empathizing”). It has been proposed that emotion expressions may provide information about several of the components of the emotion process (e.g., Ekman, 1997). The current study provides the first descriptive data on this issue based on spontaneous impressions.

Question 2: How Accurate Are Open-Ended Judgments of Emotions in Vocal Prosody?

The degree of accurate recognition is a question that looms large in the study of emotion. The current study takes four perspectives on what it means for an open-ended response to be correct. First, we examine whether the intended emotional category matches the category that was reported by the participants. The other three perspectives involve dimensions rather than discrete categories. The circumplex model of affect maps distinct emotional categories into a two-dimensional space. One axis refers to valence or hedonic tone—the intrinsic pleasantness of a stimulus—and the other axis refers to the level of intensity of arousal (Russell, 1980, 2003; Russell & Feldman Barrett, 1999). Along these lines, we examine accuracy in terms of whether the judgment matches the intended category in terms of its valence, its arousal, or its quadrant within the 2×2 of valence and arousal. We note that our analyses do not further distinguish the 18 emotional categories along the dimension of potency, for two reasons. First, the model of valence and arousal has been the dominant model in modern writing. Second, including potency would create an unwieldy 2×2×2 set of 8 cells, not all of which would be populated by the 18 emotional categories represented in this study.

In examining the accuracy of open-ended judgments of vocal prosody, we note that the term “accuracy” can be hard to define in a free-response format. Not only do judges need to be able to recognize emotional states from the acoustical cues but they also need to find those emotional states to be the most relevant feature of the stimulus when reporting their impressions to researchers. For this reason, one can argue that the content of such a judgment is as much a matter of salience as it is a matter of accuracy. For example, a response that a participant is “talking to a young child” is not necessarily right or wrong, because it does not contain a judgment of an emotional state. As an attempt to assuage this concern, we include instructions to the participants that direct their attention to the notion that actors are trying to express something in these stimuli, and their stated task is to report what the actors are attempting to express. Furthermore, below we limit the analysis of accuracy to those responses that were coded as feeling states.

Question 3: What Else Do People Hear in Vocal Prosody Other Than Emotional States?

The final goal of this free-response study is to develop a nuanced portrait of the inferences people make from others’ emotional expressions. These inferences are not limited to emotional categories and, indeed, we use participant responses to understand the nature of spontaneous impressions of vocal expressions. After coding the content of open-ended judgments, we rely on expert emotions researchers—full members of the International Society for Research in Emotion (ISRE)—in order to develop a taxonomy from these codes.

The Current Study

This empirical work uses data collected in five distinct nations—namely, Australia, India, Kenya, Singapore, and the United States. To prevent confounds across linguistic backgrounds, each of these groups is from an English-speaking country. Even with this linguistic commonality, the selected countries vary greatly with respect to their cultures. Considering the five dimensions in Hofstede’s model (www.geert-hofstede.com; Hofstede, 2001), (a) Australia and the United States are often considered individualistic societies, whereas India, Kenya, and Singapore are more collectivistic, (b) India, Kenya, and Singapore are considered high in power distance, whereas the United States and Australia are low, (c) Singapore is low in uncertainty avoidance, with the other four cultures having moderate levels, (d) in terms of masculinity, levels from lowest to highest are Singapore, India, Kenya, the United States, and Australia, and (e) long-term orientation is lower in Australia, Kenya, and the United States, and higher in India and Singapore. Collecting data in multiple nations allows us to increase diversity in the participant samples.

Stimuli come from the largest available database of vocal expressions (Vocal Expressions of Nineteen Emotions across Cultures [VENEC]; Laukka et al., 2010, 2016), which contains 18 different emotional states (9 positive and 9 negative), as well as neutral states, portrayed by 20 actors from each of five English speaking nations. In an effort to maximize the quality of vocal portrayals in producing diagnostic cues, stimuli were collected from professional actors. The developers sampled a number of speakers that is larger than usual from each culture, in order to reduce the possibility of specific item effects that are idiosyncratic to particular individuals. Overall, this stimulus set is of a far larger scale than those used in previous research on emotional prosody, with a rich set of emotional states and diverse set of national groups represented.

Taken together, this study is designed to examine the perception of emotion through the voice in a setting that imposes the lowest level of constraint feasible for participant responses. It is exploratory in that it embraces the diversity of potential responses and catalogs the open-ended impressions of the emotional prosody stimuli. We attempt to increase greatly the amount of data available to understand the open-ended judgment of affective states expressed through emotional prosody.

Method

The data that support the findings of this study are openly available via the Center for Open Science at: https://osf.io/8x6s7/?view_only=b41e545daca6412c82a5324b8dfe80dd.

Stimulus Materials

Vocal stimuli were sampled from the VENEC database, which consists of emotion portrayals by 100 professional actors across five English-speaking countries (Australia, India, Kenya, Singapore, and United States; Laukka et al., 2010), and which was developed by the same author team as the current article. For the sake of consistency, we defined a “professional actor” as an individual who had been paid on at least one occasion for their acting. The database contains recordings from 20 actors from each culture (50% women). Each actor conveyed 18 affective states (affection, anger, amusement, contempt, disgust, distress, fear, guilt, happiness, interest, lust, negative surprise, positive surprise, pride, relief, sadness, serenity, and shame) with moderately high level of emotion intensity, and also recorded emotionally neutral expressions. To maintain consistency across portrayals, the words spoken were one of two emotionally neutral phrases: “Let me tell you something” or “That is exactly what happened.”

The actors were first provided with scenarios based on the appraisal theory of emotion (e.g., Ellsworth & Scherer, 2003; Lazarus, 1991; Ortony et al., 1988). These scenarios describe typical situations in which each of the above emotions may be elicited, and actors were instructed to try to enact finding themselves in these situations (see Laukka et al., 2016). The protocol further asked the speakers to try to remember similar situations that they had experienced personally and that had evoked the specified emotion. They were asked, if possible, to try to put themselves into the same emotional state of mind. The method of acting out emotional episodes by reactivating past emotional experiences is common among actors (e.g., Stanislavski, 1936), and these instructions were used to encourage consistency in the methods used across the individuals providing stimuli.

The recordings were conducted on location in each country (Brisbane, Australia; Pune, India; Nairobi, Kenya; Singapore, Singapore; and Los Angeles, United States) with conditions standardized across locations. The actors’ speech was recorded directly onto a computer with 44 kHz sampling frequency using a high-quality microphone (sE Electronics USB2200A, Shanghai, China). To enable a wide dynamic range while avoiding clipping, each actor first produced sample stimuli, and recording levels were optimized and held constant based on the loudest sample. The sound level of the stimuli was normalized separately for each speaker.

The stimulus set consisted of 380 items from each of the five cultures (total number of stimuli = 1,900), with 20 actors posing 18 emotional states plus neutral. To reduce fatigue, each participant listened to a subset of half of these 380 stimulus items. Two equivalent sets of 190 stimuli were created that contained five male and five female actors each posing these states.

Participants

A total of 302 participants took part in the study, from Australia (N = 61; 26 females), India (N = 60, 30 females), Kenya (N = 58, 29 females), Singapore (N = 51, 27 females), and the United States (N = 72, sex not recorded). In each nation, participants were undergraduate university students who were born and raised in their respective countries, and self-reported that they spoke English at native fluency. For determining the sample size, there are no specific power tests for open-ended qualitative data, and so we conducted a power test for the ANOVA comparing quantitative results across 18 emotions for one culture at a time. Using G*Power (Faul et al., 2009), assuming a small effect size (r = 0.15), a sample of N=50 was predicted a priori to have power of .85 for a within-subjects design with 18 repeated measures. Each nation’s sample exceeded this number.

Procedure

Participants listened to each stimulus and provided open-ended judgments of the stimuli from their own cultural group. MediaLab software (Jarvis, 2008) presented stimuli in a randomized order for each participant and recorded responses. Instructions were as follows:

In this study we want to examine what kinds of information people can tell from another person’s vocal tone. You will hear a number of phrases that were spoken by professional actors trying to show how they would express themselves in various situations. These expressions might involve emotions, intentions, or other kinds of feelings that a person could have. For each phrase expressed, we want you to think about what kind of feeling would likely have elicited such a vocal expression. For each phrase, we ask you to describe what you think the actor is trying to express to you, the listener. You can use a single word for your response, up to a maximum of one line of text.

A total of 57,380 responses were generated from the 302 participants judging 190 stimuli each. These consisted of 18,217 unique text strings.

Coding for Emotion Process

We coded the open-ended responses for seven stages of the componential emotion process. Based on theoretical models, these stages were: Situation, Cognitive appraisal, Feeling state, Physiological arousal, Expressive behavior, Emotion regulation, Influence attempts, and Other could not be coded. The second author (PL) coded all stimuli, and a research assistant provided independent ratings of all stimuli. Each text string could be assigned to multiple emotion process categories. The coder could list categories in order of their first, second, and even third choice. Before calculating reliability, partial credit was assigned to the top two process codes if more than one code was listed. The average (r) and total (R) inter-rater reliability varied for each stage of the emotion process: Stimulus r = .50, R = .66, Cognitive appraisal r = .63, R = .77, Feeling state r = .71, R = .83, Physiological arousal r = .79, R = .88, Expressive behavior r = .29, R = .45, Emotion regulation r = .53, R = .70, and Influence attempts r = .71, R = .83. Based on these reliability figures, data on expressive behavior appear for completeness, but it is noted that coding did not reach conventional reliability in the tables in which it appears. Disagreements were handled by assigning proportional credit to the different codes chosen. This meant that any particular text string could be considered half part of one category and half part of another.

Coding for Stimulus Content

To capture the diversity of the 18,217 unique text strings within a manageable yet comprehensive set of categories, the first author (HAE) coded 5,000 strings from the first three countries in which data were collected (India, Kenya, and the United States). These 5,000 included every text string that appeared at least twice. Beginning with the originally intended categories portrayed by the actors, she added a new category any time that a frequently-used text string could not be coded into the existing categories. She also added a new category any time that at least three separate text strings were related to each other and could not be coded into an existing category. When data arrived from the other two countries (Australia and Singapore), she coded the 115 new text strings that appeared 5 or more times, and confirmed that there were no further changes to the coding system. These 5,115 unique text strings used to generate the coding system represented 43,441 (75.7%) of the total participant responses. This process resulted in a total of 56 unique categories, of which 32 referred to affective states. Both these 32 affective states and the remaining 24 categories are described below.

Four raters used this coding system. In addition to the first author, two research assistants entered independent responses for each text string from all five nations, and an additional research assistant entered independent responses for the first three countries that had complete data. The average inter-rater reliability (r) was .75, and total Reliability (R) was .90 on the basis of an average of 3.6 ratings per text string. As with the emotion process coding, disagreements were handled by assigning proportional credit to the different categories chosen.

Results

Research Question 1: What Aspects of Emotion Do We Hear in Vocal Prosody?

Table 1 summarizes the categorization of open-ended judgments of vocal prosody into stages of the emotion process, broken down by the intended states (18 emotions and neutral). In general, feeling states were implicated in nearly half of all responses (43.2%), followed by attempts to influence others (20.0%), cognitive appraisals (16.6%), and physiological arousal (15.0%). Less common but still present were references to situations (2.2%), expressive behavior (1.7%, which was not a sufficiently reliable variable), and a very small number of entries related to emotion regulation (0.2%).

Table 1.

Categorization of Open-Ended Judgments of Vocal Tones Into Stages of the Emotion Process (N = 302).

	Situation	Cognitive Appraisal	Feeling State	Physiological Arousal	Emotion Regulation	Expressive Behavior	Influencing Others	Other	Total
Positive emotion
Affection	2.1%	16.0%	36.5%	16.9%	.2%	1.2%	25.5%	1.5%	100.0%
Amusement	2.9%	10.0%	50.4%	16.9%	.1%	2.9%	15.9%	0.8%	100.0%
Happiness	2.1%	12.9%	45.8%	20.8%	.2%	1.4%	16.0%	0.8%	100.0%
Interest	2.3%	15.5%	33.6%	16.0%	.1%	2.1%	29.2%	1.2%	100.0%
Lust	2.3%	13.6%	37.9%	14.1%	.4%	1.5%	29.0%	1.3%	100.0%
Positive surprise	2.4%	10.4%	44.7%	19.4%	.1%	3.0%	19.2%	0.8%	100.0%
Pride	2.1%	24.1%	32.2%	14.4%	.1%	1.0%	24.9%	1.0%	100.0%
Relief	2.5%	15.4%	43.0%	17.8%	.2%	1.9%	18.1%	1.2%	100.0%
Serenity	2.2%	16.6%	37.9%	18.4%	.1%	1.4%	21.7%	1.6%	100.0%
Total	2.3%	15.0%	40.2%	17.2%	.2%	1.8%	22.2%	1.1%	100.0%
Negative emotion
Anger	1.6%	22.1%	50.3%	7.6%	.1%	1.1%	16.6%	0.6%	100.0%
Contempt	2.1%	25.0%	35.0%	11.1%	.2%	0.8%	24.5%	1.2%	100.0%
Disgust	2.1%	18.6%	43.1%	13.6%	.3%	1.4%	19.8%	1.2%	100.0%
Distress	2.2%	13.8%	52.0%	13.9%	.3%	2.3%	14.8%	0.8%	100.0%
Fear	1.8%	11.6%	57.8%	12.0%	.1%	2.3%	14.0%	0.5%	100.0%
Guilt	2.6%	19.5%	44.3%	13.2%	.2%	1.3%	17.4%	1.4%	100.0%
Negative surprise	2.3%	16.5%	41.4%	13.7%	.1%	2.0%	22.6%	1.1%	100.0%
Sadness	2.5%	11.6%	59.2%	10.0%	.3%	3.3%	12.1%	0.9%	100.0%
Shame	2.4%	19.4%	46.3%	12.6%	.2%	1.1%	16.9%	1.1%	100.0%
Total	2.2%	17.6%	47.7%	12.0%	.2%	1.7%	17.6%	1.0%	100.0%
Neutral	1.4%	23.4%	28.6%	22.4%	.2%	1.0%	21.4%	1.5%	100.0%
Total	2.2%	16.6%	43.2%	15.0%	.2%	1.7%	20.0%	1.1%	100.0%

Note. Expressive behavior appears in the table for completeness; however, coding did not reach conventional reliability and coefficients appear in italics.

Positive vs. negative emotions are more likely to be seen as physiological reactions, whereby negative vs. positive emotions are more likely to be seen as cognitive appraisals and feeling states. We note that many of the social/moral emotions (anger, contempt, pride, guilt, and shame) had the highest percentages of responses coded as the cognitive appraisal. The emotions most frequently coded as influencing others were interest, lust, and affection.

Supplemental Tables 1a through 1g contains these data for every combination of emotion process, emotional state, and nation. Judgments of stimuli as representing situations averaged 2.2% and ranged from 0.5% for Kenya to 5.0% in Singapore. Cognitive appraisals averaged 16.6%, ranging from 15.1% in India to 18.8% in Singapore. Feeling states had an average of 43.2%, with a wide range from 33.2% in Singapore to 51.9% in Australia. For physiological arousal, the average was 15.0%, ranging from 12.9% in India to 17.6% in Singapore. Attempts at emotion regulation were rarely used, at 0.2% in total. As mentioned above, coding of expressive behavior did not reach conventional reliability, but we note here that it averaged 1.7% of responses. From 0.9% in Australia to 2.5% in India. Finally, attempts to influence others were coded at 20.0% of responses, and ranged from 12.9% in Australia to 26.2% in India.

Research Question 2: How Accurate Are Open-Ended Judgments of Emotions in Vocal Prosody?

The convergence between emotion perceptions and intended states has been a central question within research on vocal tone, and we attempt to assess it here. To address this question, we focus on the 43.2% of open-ended judgments that referred to feeling states, because strictly speaking only these responses can be considered accurate vs. inaccurate. If a participant did not label any of the 20 trials for a given emotion as a feeling state, the mean was substituted for that combination of emotion and culture. This occurred in only 1.43% of cases, ranging from 0% for distress, fear, happiness, and sadness, up to 7% for neutral.

Table 2 contains a confusion matrix that lists the 19 intended emotional categories in the columns and the 32 feeling states that emerged from stimulus content coding in the rows. Supplementary Tables 2a through 2e contains these matrices separately for each of the five nations sampled. In these tables, accuracy is based on the convergence between the open-ended responses and the intended state, which is indicated in bold and underlined typeface. Whether or not this convergence is greater than that predicted by chance guessing is not tested formally, because there is no multiple-choice response that a participant could choose at random. As such, we treat all convergence with intended responses as if it is better than that based on chance alone.

Table 2.

Categories Coded for 19 Affective States That Were Judged as Feeling States Based on Open-Ended Responses From Five Nations (N = 302).

Intended States
		AFF	AMU	HAP	INT	LUS	PRI	PSU	REL	SER	ANG	CON	DSG	DST	FEA	GUI	NSU	SAD	SHA	NEU	Average
Judged states
Positive states
1.	Affection	3.3%	1.0%	1.2%	1.1%	1.3%	1.2%	0.5%	1.0%	2.5%	0.1%	0.3%	0.3%	0.2%	0.3%	0.5%	0.3%	0.3%	0.7%	1.1%	0.9%
2.	Amusement	8.2%	23.2%	17.0%	3.6%	3.0%	3.8%	6.6%	2.4%	3.4%	0.6%	2.4%	1.3%	1.3%	1.5%	0.8%	1.9%	1.5%	1.2%	0.9%	4.5%
3.	Happiness	17.6%	27.8%	30.9%	10.1%	6.2%	14.6%	21.6%	6.4%	4.8%	2.3%	5.2%	3.3%	2.4%	2.2%	1.8%	4.3%	1.9%	2.0%	3.3%	8.9%
4.	Interest	3.9%	3.3%	3.5%	11.3%	2.4%	5.5%	6.0%	1.6%	1.6%	0.9%	2.7%	2.8%	0.9%	1.0%	1.2%	5.3%	0.3%	0.9%	2.6%	3.0%
5.	Lust	2.4%	0.4%	0.6%	1.2%	9.6%	0.9%	0.7%	2.1%	3.3%	0.3%	1.0%	0.8%	0.2%	0.1%	0.7%	0.3%	0.5%	0.6%	0.6%	1.4%
6.	Pride	1.8%	1.4%	2.1%	1.3%	1.9%	5.4%	0.9%	1.2%	1.0%	1.1%	3.4%	1.2%	0.6%	0.6%	0.6%	0.6%	0.3%	0.6%	1.3%	1.4%
7.	Positive surprise	0.2%	0.7%	0.8%	1.5%	0.0%	0.3%	1.9%	0.5%	0.2%	0.0%	0.2%	0.3%	0.2%	0.2%	0.2%	1.4%	0.0%	0.0%	0.0%	0.5%
8.	Relief	1.8%	0.5%	1.0%	0.8%	1.7%	1.9%	1.0%	7.5%	2.4%	0.5%	1.5%	0.9%	0.6%	0.2%	0.8%	0.6%	0.4%	0.6%	0.9%	1.3%
9.	Serenity	3.9%	2.0%	1.8%	1.7%	1.9%	3.6%	0.9%	3.8%	4.9%	0.4%	1.7%	1.8%	0.2%	0.5%	1.6%	1.5%	1.0%	1.3%	4.2%	2.0%
10.	Excitement	2.8%	3.8%	5.4%	3.3%	0.8%	2.9%	5.6%	1.3%	0.5%	1.4%	1.3%	1.4%	1.4%	1.5%	0.1%	1.7%	0.2%	0.2%	0.9%	1.9%
11.	Hope	0.1%	0.0%	0.3%	0.4%	0.1%	0.2%	0.2%	0.1%	0.2%	0.4%	0.3%	0.1%	0.1%	0.0%	0.1%	0.3%	0.0%	0.2%	0.4%	0.2%
12.	Inspired	0.3%	0.4%	0.4%	1.0%	0.2%	0.5%	0.5%	0.5%	0.4%	0.0%	0.4%	0.3%	0.0%	0.1%	0.1%	0.4%	0.1%	0.1%	0.2%	0.3%
13.	Other positive state	0.7%	0.8%	0.7%	0.8%	0.4%	1.0%	0.6%	0.2%	0.3%	0.2%	0.2%	0.5%	0.1%	0.2%	0.1%	0.2%	0.0%	0.2%	0.3%	0.4%
Negative states
14.	Anger	6.8%	5.5%	8.2%	9.8%	8.2%	18.7%	10.0%	14.9%	6.0%	61.2%	32.5%	29.5%	16.2%	8.5%	8.0%	20.8%	4.6%	7.3%	11.7%	15.2%
15.	Contempt	1.1%	0.6%	0.6%	0.8%	1.5%	3.0%	0.4%	1.1%	1.6%	2.6%	3.8%	2.0%	0.6%	0.8%	0.9%	1.1%	0.5%	0.7%	2.9%	1.4%
16.	Disgust	0.3%	0.1%	0.3%	0.2%	0.2%	0.3%	0.1%	0.4%	0.2%	0.4%	0.5%	2.3%	0.2%	0.2%	0.3%	0.4%	0.1%	0.4%	0.1%	0.4%
17.	Distress	3.7%	4.6%	4.7%	5.3%	6.6%	5.0%	6.4%	6.8%	4.6%	7.4%	5.9%	11.6%	17.0%	14.8%	10.0%	11.4%	14.0%	9.9%	5.7%	8.2%
18.	Fear	3.9%	2.8%	1.5%	3.9%	8.3%	1.0%	2.8%	5.6%	3.8%	1.2%	2.6%	5.3%	14.3%	29.8%	12.5%	6.6%	12.2%	9.0%	2.2%	6.8%
19.	Guilt	0.6%	0.1%	0.2%	0.9%	1.2%	0.4%	0.4%	1.4%	1.4%	0.3%	0.4%	0.8%	1.0%	0.7%	2.8%	0.5%	1.4%	3.2%	1.1%	1.0%
20.	Negative surprise	1.1%	1.7%	1.1%	3.0%	2.1%	0.7%	3.2%	2.3%	1.2%	1.3%	1.4%	1.8%	2.5%	3.2%	1.4%	4.3%	1.0%	1.4%	0.9%	1.9%
21.	Sadness	7.6%	3.9%	2.4%	5.2%	14.2%	4.9%	4.4%	11.5%	21.5%	2.1%	5.1%	7.5%	16.3%	10.6%	21.1%	5.0%	38.6%	23.5%	15.3%	11.6%
22.	Shame	1.4%	0.3%	0.2%	0.1%	1.8%	0.3%	0.2%	1.4%	1.9%	0.3%	0.7%	1.3%	1.2%	0.8%	3.2%	0.4%	2.1%	3.7%	2.6%	1.3%
23.	Anxiety	1.6%	1.6%	1.8%	3.0%	2.2%	1.9%	3.4%	2.0%	2.5%	1.1%	2.1%	3.5%	5.9%	7.0%	3.6%	4.9%	3.0%	4.1%	2.8%	3.1%
25.	Confusion	0.4%	0.4%	0.3%	1.9%	0.4%	0.6%	1.5%	0.7%	0.4%	0.2%	0.1%	0.5%	1.0%	0.4%	1.9%	1.9%	0.5%	1.3%	0.4%	0.8%
26.	Hatred	0.7%	0.2%	0.5%	0.4%	1.7%	1.1%	0.3%	1.1%	1.1%	1.3%	2.2%	1.3%	0.4%	0.3%	0.6%	0.3%	0.3%	1.1%	1.0%	0.8%
24.	Low energy	4.2%	0.8%	0.8%	1.5%	3.2%	3.0%	0.9%	5.3%	7.3%	1.0%	3.6%	2.4%	1.4%	0.7%	5.2%	1.4%	3.1%	6.9%	11.4%	3.4%
27.	Regret	0.5%	0.1%	0.3%	0.3%	1.7%	0.5%	0.1%	1.7%	2.1%	0.2%	0.9%	0.6%	0.8%	0.5%	2.2%	0.4%	1.3%	2.0%	1.3%	0.9%
28.	Other negative state	0.8%	0.2%	0.5%	0.8%	1.0%	0.8%	0.2%	1.7%	1.9%	0.4%	1.0%	1.0%	1.0%	0.8%	2.3%	1.2%	1.4%	1.5%	1.9%	1.1%
Emotional states without valence
29.	Neutral	2.5%	0.7%	1.2%	2.2%	1.2%	2.8%	0.7%	2.0%	2.8%	1.0%	2.6%	1.0%	0.8%	0.6%	1.6%	1.2%	0.8%	1.9%	9.3%	1.9%
30.	Disbelief	0.7%	1.0%	0.6%	2.7%	0.7%	0.5%	2.3%	0.6%	0.3%	0.1%	0.8%	0.7%	0.5%	0.5%	0.7%	3.6%	0.5%	0.4%	0.0%	0.9%
31.	Surprise, no valence specified	2.0%	3.0%	1.7%	8.9%	0.8%	1.5%	6.9%	1.0%	0.7%	0.4%	0.5%	1.5%	1.1%	2.2%	0.9%	5.9%	0.1%	0.2%	0.7%	2.1%
32.	Emotional, no valence specified	0.9%	0.8%	1.2%	0.7%	1.0%	1.2%	1.7%	2.0%	1.4%	0.6%	1.4%	1.4%	1.7%	1.7%	1.4%	1.2%	2.0%	1.7%	0.8%	1.3%
Other		12.1%	6.3%	6.4%	10.2%	12.2%	9.9%	7.0%	7.9%	12.0%	8.7%	11.6%	8.8%	7.6%	7.3%	10.8%	8.8%	6.2%	11.2%	11.2%	9.3%
Total		100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%	100.0%

Note. Trials were included only if at least one rater coded the response as representing a feeling state, as opposed to another stage of the emotion process. AFF = affection; AMU = amusement; HAP = happy; INT = interest; LUS = lust; PRI = pride; PSU = positive surprise; REL = relief; SER = serenity; ANG = anger; CON = contempt; DSG = disgust; DST = distress; FEA = fear; GUI = guilt; NSU = negative surprise; SAD = sad; SHA = shame; NEU = neutral.

We examined four different types of accuracy: based on emotional categories (categorical), based on affective valence (valence-based), based on affective arousal (arousal-based), and based on the quadrant in the affective circumplex marked by valence and arousal (quadrant-based). These values for the five nations and overall appear in Table 3. For each type of accuracy, we provide formal tests of differences in accuracy levels from 5 (Culture Between-Subjects) × 19 (Emotion Within-Subjects) ANOVAs.

Table 3.

Judgment Accuracy for 19 Affective States That Were Judged as Feeling States Based on Open-Ended Responses From Five Nations (N = 302).

	Categorical Accuracy						Valence Accuracy						Arousal Level Accuracy						Affective Quadrant Accuracy
	AUS	IND	KEN	SING	USA	Total	AUS	IND	KEN	SING	USA	Total	AUS	IND	KEN	SING	USA	Total	AUS	IND	KEN	SING	USA	Total
Positive emotion
Affection	3.7%	2.3%	1.7%	0.7%	7.0%	3.3%	46.1%	46.9%	45.6%	47.1%	49.2%	47.1%	27.8%	25.5%	30.6%	17.7%	33.9%	27.6%	9.2%	6.6%	5.2%	5.5%	16.9%	9.1%
Amusement	24.5%	13.9%	9.9%	29.5%	36.0%	23.2%	68.5%	63.9%	53.9%	64.0%	73.2%	65.1%	88.1%	76.3%	76.1%	80.1%	78.5%	79.8%	63.4%	58.3%	49.9%	61.7%	67.3%	60.5%
Happiness	23.6%	51.2%	25.6%	25.6%	28.3%	30.9%	66.8%	83.1%	50.1%	57.8%	68.4%	65.7%	81.3%	86.9%	75.2%	81.2%	76.1%	80.0%	59.1%	78.2%	47.8%	54.6%	60.6%	60.3%
Interest	18.2%	16.6%	4.5%	12.0%	5.6%	11.2%	46.4%	42.0%	28.5%	30.8%	40.1%	37.9%	70.4%	68.6%	63.2%	59.6%	61.9%	64.8%	38.1%	39.7%	25.1%	26.8%	30.1%	32.1%
Lust	20.4%	1.2%	4.0%	3.3%	16.4%	9.6%	47.4%	11.5%	31.3%	19.8%	35.1%	29.6%	68.4%	50.0%	54.7%	46.2%	52.1%	54.5%	40.9%	7.1%	28.3%	14.4%	26.8%	24.0%
Positive surprise	4.3%	2.2%	0.5%	1.3%	1.1%	1.9%	51.2%	55.8%	44.4%	44.2%	39.6%	46.9%	85.6%	84.4%	66.6%	75.1%	65.2%	75.1%	47.2%	52.7%	41.4%	41.0%	35.0%	43.2%
Pride	9.4%	7.8%	2.0%	5.8%	2.1%	5.3%	47.3%	37.2%	31.3%	37.4%	51.9%	41.7%	74.0%	65.2%	65.8%	60.8%	63.3%	65.9%	39.6%	28.0%	28.1%	30.3%	38.9%	33.3%
Relief	9.9%	11.0%	0.8%	2.7%	11.3%	7.5%	36.1%	38.6%	19.9%	14.6%	31.4%	28.7%	34.0%	41.9%	38.3%	34.2%	39.2%	37.7%	14.3%	16.1%	4.9%	4.5%	19.2%	12.4%
Serenity	6.2%	6.0%	1.6%	2.6%	6.9%	4.8%	38.5%	17.1%	14.6%	14.0%	36.7%	25.1%	46.1%	53.3%	55.0%	49.2%	42.3%	48.9%	15.8%	8.8%	5.0%	3.0%	14.6%	9.9%
Total	13.3%	12.5%	5.6%	9.3%	12.8%	10.9%	49.8%	44.0%	35.5%	36.7%	47.3%	43.1%	63.9%	61.4%	58.4%	56.0%	56.9%	59.4%	36.4%	32.8%	26.2%	26.9%	34.4%	31.6%
Negative emotion
Anger	52.8%	76.0%	35.7%	71.3%	69.5%	61.2%	83.4%	84.3%	65.5%	86.3%	84.4%	80.9%	84.4%	87.7%	69.7%	86.5%	86.0%	83.0%	76.9%	82.9%	50.2%	83.0%	82.5%	75.3%
Contempt	9.4%	1.1%	1.5%	4.3%	2.7%	3.8%	70.5%	63.9%	55.7%	53.5%	67.6%	62.8%	75.3%	68.2%	61.7%	59.6%	67.6%	66.8%	59.4%	49.4%	34.5%	36.6%	59.9%	48.9%
Disgust	3.3%	3.2%	2.4%	0.4%	1.9%	2.3%	71.1%	71.4%	70.1%	72.3%	72.7%	71.5%	72.4%	68.3%	70.9%	65.6%	63.1%	67.9%	55.9%	54.3%	50.9%	54.2%	54.0%	53.9%
Distress	24.4%	15.1%	9.3%	15.8%	19.5%	17.0%	82.1%	85.0%	72.8%	81.2%	79.7%	80.2%	70.6%	55.7%	64.1%	54.4%	59.6%	61.0%	60.2%	48.2%	45.7%	47.7%	53.6%	51.3%
Fear	23.1%	51.8%	22.0%	30.3%	23.2%	29.8%	83.9%	83.6%	68.8%	83.0%	77.4%	79.2%	75.7%	72.6%	68.4%	63.1%	63.2%	68.6%	65.0%	65.0%	44.4%	57.1%	56.1%	57.6%
Guilt	6.4%	4.0%	0.1%	1.6%	1.9%	2.8%	77.2%	88.9%	69.8%	70.7%	71.3%	75.6%	44.6%	47.3%	39.0%	41.5%	40.0%	42.5%	37.5%	46.1%	34.3%	37.9%	34.3%	37.9%
Negative surprise	5.8%	8.1%	0.8%	4.1%	2.8%	4.3%	58.3%	53.3%	68.7%	62.6%	60.4%	60.5%	75.4%	71.0%	66.0%	69.8%	56.5%	67.3%	47.3%	36.2%	47.5%	49.2%	44.4%	44.8%
Sadness	28.5%	45.9%	30.9%	44.1%	43.4%	38.6%	88.6%	87.0%	75.3%	86.4%	82.0%	83.8%	45.6%	56.2%	48.2%	54.0%	54.8%	51.8%	42.6%	55.2%	44.4%	52.7%	51.6%	49.3%
Shame	9.0%	3.0%	0.8%	2.9%	2.5%	3.7%	81.3%	78.1%	66.4%	79.4%	74.9%	76.0%	47.3%	48.6%	47.6%	48.4%	48.1%	48.0%	41.0%	45.9%	40.8%	45.9%	43.3%	43.3%
Total	18.1%	23.1%	11.5%	19.4%	18.6%	18.2%	77.4%	77.3%	68.1%	75.1%	74.5%	74.5%	65.7%	64.0%	59.5%	60.3%	59.9%	61.9%	54.0%	53.7%	43.6%	51.6%	53.3%	51.4%
Neutral	6.9%	15.9%	4.9%	4.9%	12.5%	9.3%	7.2%	17.4%	7.5%	5.5%	14.8%	10.8%	60.7%	53.1%	47.6%	51.2%	40.9%	50.4%	-	-	-	-	-	-
Total
With neutral	15.3%	17.7%	8.4%	13.8%	15.5%	14.2%	60.6%	58.4%	49.5%	53.2%	58.5%	56.3%	64.6%	62.2%	58.4%	57.8%	57.5%	60.1%	-	-	-	-	-	-
Without neutral	15.7%	17.8%	8.6%	14.3%	15.7%	14.5%	63.6%	60.6%	51.8%	55.9%	60.9%	58.8%	64.8%	62.7%	59.0%	58.2%	58.4%	60.6%	45.2%	43.3%	34.9%	39.2%	43.8%	41.5%

Note. Values may sum to less than 100% due to responses that are neutral or that could not be coded into the four quadrants. Bold font indicates judgments of emotional states in the intended quadrant. Some values in Supplemental Table 3 differ from corresponding values in the one-tenth of one-percent place because the values in this table are weighted by culture. AUS = Australia; IND = India; KEN = Kenya; SING = Singapore, USA = United States.

Categorical accuracy

There was a main effect for Emotion, F(18, 5346) = 276.25, η² = .48, p < .001, a main effect for Culture, F(4, 297) = 43.57, η² = .37, p < .001, and an interaction effect of Culture × Emotion, F(72,5346) = 9.58, η² = .11, p < .001. It is worth noting that the overall categorical accuracy is relatively modest, at 14.2%. However, the results of the ANOVA emphasize that this varied by emotional state, and some individual emotions received substantial recognition rates. In general, negative states (18.2%) were better recognized than positive states (10.9%). The states with highest accuracy were anger (61.2%), sadness (38.6%), happiness (30.9%), and fear (29.8%). Those with the lowest accuracy were positive surprise (1.9%), disgust (2.3%), guilt (2.8%), and affection (3.3%). Participants from India were the most accurate (17.7%), followed by those from the United States (15.5%), Australia (15.3%), Singapore (13.8%), and Kenya (8.4%).

Valence-based accuracy

There was a main effect for Emotion, F(18, 5346) = 295.65, η² = .50, p < .001, a main effect for Culture, F(4, 297) = 19.74, η² = .21, p < .001, and an interaction effect of Culture × Emotion, F(72,5346) = 5.31, η² = .07, p < .001. Valence accuracy was overall substantially higher than that for individual categories, and averaged 56.3%. In general, negative states (74.5%) were far better recognized than positive states (43.1%). The best recognized states were sadness (83.8%), anger (80.9%), distress (80.2%), and fear (79.2%). Those with the lowest accuracy were serenity (25.1%), relief (28.7%), lust (29.6%), and affection (47.1%). The most accurate participants were from Australia (60.6%), followed by those from the United States (58.5%), India (58.4%), Singapore (53.2%), and Kenya (49.5%).

Arousal-based accuracy

Based on Scherer (2005) and Gillioz et al. (2016), arousal levels were coded as low for the positive emotions affection, relief, and serenity, and the negative emotions guilt, sadness, and shame. The remainder were coded as high arousal.

There was a main effect for Emotion, F(18, 5,346) = 128.28, η² = .30, p < .001, a main effect for Culture, F(4, 297) = 9.441, η² = .11, p < .001, and an interaction effect of Culture × Emotion, F(72,5,346) = 2.90, η² = .04, p < .001. Arousal-based accuracy was also substantially higher than that for individual categories, with an average of 60.1%, and was approximately equal for negative (59.4%) and positive (61.9%) states. The most highly recognized states were anger (83.0%), happiness (80.0%), amusement (79.8%), and positive surprise (75.1%). The most poorly recognized states were affection (27.6%), relief (37.7%), guilt (42.5%), and shame (48.0%). Note that the effect size for cultural differences was relatively smaller than that for categorical and valence-based accuracy. Values were highest in Australia (64.6%), India (62.2%), Kenya (58.4%), Singapore (57.8%), followed by the United States (57.5%).

Affective quadrant accuracy

There was a main effect for Emotion, F(17, 5049) = 201.39, η² = .40, p < .001, a main effect for Culture, F(4, 297) = 19.51, η² = .21, p < .001, and an interaction effect of Culture × Emotion, F(68,5049) = 5.40, η² = .07, p < .001. Affective quadrant accuracy was again higher than that for categorical accuracy, and averaged 41.5%. As with categorical and arousal-based accuracy, negative states (51.4%) were better recognized than positive states (31.6%). Those with the highest accuracy were anger (75.3%), amusement (60.5%), happiness (60.3%), and fear (57.6%). Those with the lowest accuracy were affection (9.1%), serenity (9.9%), relief (12.4%), and lust (24.0%), which tended to be low arousal emotions. Accuracy was highest in Australia (45.2%), followed by the United States (43.8%), India (43.3%), Singapore (39.2%), and Kenya (34.9%).

To understand these data further, we present a confusion matrix in Table 4, which illustrates to what extent the erroneous judgments were random vs. systematic.¹ The most visible trend is that low activation emotions—whether positive or negative—appear to be judged most often as negative states. In particular, low activation positive states appear to be challenging to judge, with 10.0% of judgments appearing in the intended quadrant, and the remaining judgments split about equally among the three others (22.3%, 22.9%, and 25.5% for high activation positive, high activation negative, and low activation negative, respectively).

Table 4.

Confusion Matrix of Affective Circumplex Quadrants Judged for Each Emotional State (N = 302).

	Positive	Valence	Negative	Valence
	High Activation	Low Activation	High Activation	Low Activation
Positive emotion
High activation
Amusement	60.3%	3.4%	15.6%	7.1%
Happiness	60.0%	4.1%	17.3%	5.7%
Interest	32.1%	3.8%	23.6%	11.4%
Lust	23.5%	4.9%	29.0%	24.5%
Positive Surprise	43.5%	2.5%	23.4%	9.5%
Pride	33.1%	6.5%	30.1%	11.4%
Total	42.1%	4.2%	23.2%	11.6%
Low activation
Affection	37.3%	8.7%	17.5%	16.0%
Relief	15.6%	11.8%	32.2%	23.7%
Serenity	14.2%	9.5%	18.8%	36.9%
Total	22.3%	10.0%	22.9%	25.5%
Negative emotion
High activation
Anger	6.6%	1.4%	75.3%	5.0%
Contempt	16.6%	3.7%	48.0%	13.1%
Disgust	11.3%	3.0%	53.8%	16.1%
Distress	7.1%	1.1%	51.1%	27.0%
Fear	7.3%	1.1%	57.5%	20.5%
Negative Surprise	15.8%	2.6%	44.9%	12.7%
Total	10.8%	2.1%	55.1%	15.7%
Low activation
Guilt	5.6%	3.0%	33.4%	38.1%
Sadness	4.9%	1.6%	32.6%	49.2%
Shame	5.5%	2.8%	29.7%	43.4%
Total	5.3%	2.5%	31.9%	43.6%

Overall findings

Taken together, the three forms of accuracy that draw from the affective circumplex—that is, valence-based, arousal-based, and quadrant-based—tended to have far higher accuracy than discrete emotional categories. Those that tended to be best recognized were the basic emotions of anger, sadness, happiness, and fear. These are also the emotions that Research Question 1 found to be the most often coded as feeling states.

Supplemental analyses by gender

In the case of data from Australia, India, and Singapore, individual participant data could be matched with self-reported gender, and accuracy by gender appears in Supplemental Tables 3a, 3b, 3c, and 3d for categorical, valence, arousal, and quadrant accuracy, respectively.² Given findings of gender differences in emotion recognition (Hall et al., 2016; Lausen & Schacht, 2018), we analyzed these data using a 2 (Gender) × 3 (Culture) × 19 (Emotion) ANOVA separately for each dependent measure. The advantage for women that is apparent by visual inspection of the tables was significant for categorical accuracy, F(1, 166) = 5.24, η² = .03, p=.023, reached marginal significance for arousal-based accuracy, F(1, 166) = 2.93, η² = .02, p=.089, and quadrant-based accuracy, F(1, 166) = 3.38, η² = .02, p=.068, and was not significant for valence-based accuracy, F(1, 166) = 1.10, η² = .01, p=.295. There were no significant two-way interaction terms of Gender × Culture or Gender × Emotion for any of the four types of accuracy.

Research Question 3: What Else Do People Hear in Vocal Prosody Other Than Emotional States?

Table 5 summarizes the frequency in which the intended states were categorized into the entire set of 56 open-ended categories. Separate versions of this table for each of the five nations appear in Supplemental Tables 4a through 4e.

Table 5.

Categories Coded for 19 Affective States Based on Open-Ended Responses From Five Nations (N = 302).

	AFF	AMU	HAP	INT	LUS	PRI	PSU	REL	SER	ANG	CON	DSG	DST	FEA	GUI	NSU	SAD	SHA	NEU
Emotional states
Positive states
1. Affection	2.26	1.18	1.17	1.60	1.70	1.13	0.66	1.03	1.95	0.26	0.71	0.58	0.30	0.33	0.75	0.62	0.52	0.88	1.22
2. Amusement	4.13	15.82	9.65	1.52	1.70	1.79	4.14	1.37	1.40	0.30	0.95	0.70	0.85	0.97	0.46	1.12	1.03	0.63	0.33
3. Happiness	8.66	17.52	18.71	4.41	3.19	6.39	12.33	3.35	2.51	1.15	2.36	1.95	1.69	1.55	1.21	2.24	1.37	1.28	1.52
4. Interest	2.59	2.33	2.38	6.19	1.67	2.67	4.09	1.50	1.35	0.72	1.77	2.08	0.94	0.97	1.08	3.49	0.41	0.77	1.23
5. Lust	2.51	0.61	0.67	1.16	10.59	0.79	0.63	2.09	3.29	0.19	0.69	0.81	0.32	0.20	0.76	0.35	0.73	0.70	0.39
6. Pride	1.37	0.99	1.42	0.66	1.19	3.77	0.78	1.13	0.81	1.01	2.73	1.04	0.54	0.41	0.55	0.75	0.27	0.73	0.89
7. Positive surprise	0.14	0.49	0.53	0.94	0.06	0.23	1.37	0.39	0.11	0.02	0.08	0.11	0.18	0.23	0.21	0.78	0.01	0.02	0.01
8. Relief	0.81	0.36	0.71	0.58	1.03	0.92	0.55	4.44	1.55	0.40	0.59	0.68	0.42	0.20	0.57	0.38	0.30	0.44	0.36
9. Serenity	6.26	2.68	2.66	3.15	3.84	4.11	1.61	5.03	7.17	0.80	2.68	2.27	0.87	0.85	3.32	2.41	2.20	3.07	6.88
10. Excitement	4.36	11.57	14.53	7.34	2.18	5.82	15.62	2.77	1.23	3.57	2.35	5.11	5.38	5.35	1.37	5.80	1.16	0.76	1.20
11. Hope	0.07	0.08	0.21	0.35	0.10	0.22	0.18	0.08	0.22	0.27	0.24	0.09	0.09	0.06	0.15	0.23	0.08	0.13	0.28
12. Inspired	0.23	0.26	0.31	0.50	0.17	0.23	0.33	0.28	0.25	0.06	0.20	0.18	0.04	0.09	0.10	0.29	0.10	0.09	0.15
13. Other positive state	0.39	0.36	0.44	0.41	0.23	0.50	0.38	0.17	0.17	0.30	0.21	0.30	0.13	0.10	0.12	0.21	0.09	0.18	0.25
Negative states
14. Anger	3.36	3.64	4.96	4.66	3.93	7.80	6.25	8.36	3.14	38.51	15.72	16.75	10.60	6.20	4.65	11.16	3.32	4.42	4.87
15. Contempt	0.88	0.89	0.86	1.18	1.00	2.41	0.60	1.24	1.14	2.29	3.84	1.97	0.83	0.78	0.91	1.22	0.52	0.80	1.16
16. Disgust	0.12	0.07	0.24	0.09	0.12	0.18	0.09	0.30	0.15	0.34	0.40	1.23	0.13	0.13	0.17	0.24	0.04	0.22	0.04
17. Distress	2.16	2.93	2.82	2.40	3.45	2.20	3.76	3.94	2.54	4.73	3.15	6.89	11.95	11.18	6.15	6.40	11.03	6.38	2.46
18. Fear	1.75	1.68	0.79	1.78	4.28	0.59	1.74	3.24	1.96	0.88	1.13	2.59	9.00	20.92	6.95	3.72	8.47	5.39	1.07
19. Guilt	0.74	0.33	0.49	0.82	1.09	0.41	0.37	1.50	1.28	0.39	0.64	0.81	1.23	1.25	3.21	0.70	2.09	3.10	0.91
20. Negative surprise	0.79	1.30	0.59	1.31	0.88	0.34	2.07	1.73	1.00	0.73	0.73	1.22	1.84	2.34	1.30	2.76	0.81	1.45	0.67
21. Sadness	4.19	2.67	1.49	2.27	7.70	2.31	2.62	6.49	10.58	1.46	2.44	4.27	10.86	7.74	12.31	2.66	28.79	14.45	6.45
22. Shame	0.84	0.22	0.15	0.09	1.03	0.20	0.15	0.81	1.04	0.22	0.37	0.72	0.87	0.66	1.93	0.32	1.77	2.72	1.08
23. Anxiety	0.92	1.02	1.08	1.46	1.12	0.89	2.00	1.27	1.24	0.63	0.97	2.07	3.80	4.78	2.14	2.56	2.20	2.58	1.02
25. Confusion	0.32	0.37	0.17	0.93	0.19	0.25	0.90	0.25	0.25	0.09	0.20	0.37	0.64	0.30	1.11	0.99	0.35	0.80	0.32
26. Hatred	0.49	0.22	0.56	0.39	1.18	0.70	0.30	0.50	0.61	1.20	1.17	1.00	0.39	0.28	0.31	0.32	0.29	0.52	0.47
24. Low energy	2.41	0.59	0.76	0.78	4.87	1.62	0.33	7.98	6.52	0.56	2.24	2.26	1.57	0.88	4.33	1.09	3.56	5.70	4.90
27. Regret	0.34	0.06	0.14	0.17	0.89	0.20	0.09	0.89	0.98	0.15	0.35	0.38	0.50	0.38	1.39	0.18	0.98	1.19	0.51
28. Other negative state	0.45	0.18	0.30	0.46	0.56	0.28	0.19	0.91	0.80	0.38	0.47	0.56	0.68	0.58	1.07	0.70	0.83	0.98	0.74
Emotional states without valence
29. Neutral	9.55	4.05	5.37	6.54	4.31	8.01	1.88	5.98	8.62	3.32	7.56	5.06	2.92	2.36	5.88	4.39	4.11	6.24	19.08
30. Disbelief	0.78	1.56	0.62	3.09	0.79	0.39	2.93	0.59	0.44	0.15	0.64	0.82	0.72	1.30	1.10	3.84	0.39	0.74	0.12
31. Surprise, no valence specified	0.70	1.98	0.88	4.11	0.29	0.56	4.31	0.67	0.44	0.24	0.26	0.83	0.75	1.45	0.52	3.37	0.03	0.22	0.16
32. Emotional, no valence specified	0.50	0.67	0.76	0.68	0.67	0.52	1.24	1.16	0.70	0.51	0.70	0.94	1.49	1.47	0.84	0.82	1.84	0.92	0.39
Person descriptors
34. Dominant	8.57	5.63	8.45	9.46	6.67	18.80	6.60	7.19	8.27	20.35	18.93	12.49	7.61	4.83	5.54	8.59	3.30	5.98	12.21
35. Hesitant	1.19	0.68	0.27	2.56	1.55	0.58	1.05	1.28	1.97	0.25	0.66	1.77	1.79	1.85	6.52	2.82	1.88	5.28	1.28
36. Humble	0.04	0.06	0.06	0.07	0.09	0.06	0.03	0.01	0.04	0.03	0.00	0.02	0.04	0.03	0.05	0.05	0.03	0.07	0.16
37. Polite	0.34	0.18	0.12	0.29	0.14	0.17	0.22	0.19	0.17	0.12	0.18	0.07	0.10	0.12	0.23	0.30	0.28	0.16	0.34
38. Impatient	0.50	0.77	0.97	1.42	0.21	0.95	1.69	0.81	0.39	1.05	1.13	1.08	3.10	3.11	0.79	2.07	0.28	0.72	1.49
39. Secretive	4.05	2.73	1.49	4.26	8.91	2.51	1.53	2.48	4.14	0.82	1.99	2.54	1.61	4.01	2.78	2.62	1.85	2.98	1.92
40. Sincere	2.19	0.75	1.05	1.18	0.79	2.01	0.73	1.40	1.29	1.12	1.70	1.31	1.08	0.48	1.23	1.31	0.70	1.30	2.95
41. Serious	1.61	0.81	0.92	1.08	1.62	1.98	0.66	1.34	2.41	2.35	2.08	1.64	1.01	1.09	1.74	1.60	1.11	1.62	4.31
42. Sympathetic	0.92	0.28	0.48	0.72	0.40	0.58	0.51	0.78	0.91	0.32	0.65	0.46	0.51	0.45	0.58	0.44	0.68	0.58	0.80
43. Thoughtful	0.44	0.23	0.29	0.73	0.52	0.65	0.20	0.35	0.66	0.22	0.70	0.44	0.16	0.05	0.49	0.40	0.30	0.25	0.46
33. Wanting attention	0.89	0.70	0.90	1.03	0.78	1.35	1.12	0.39	0.52	0.69	0.84	0.70	0.88	0.38	0.64	0.87	0.35	0.27	0.59
44. Wanting to be understood	0.38	0.31	0.43	0.77	0.49	0.59	0.52	0.48	0.48	0.51	0.59	0.59	0.62	0.50	0.36	0.65	0.50	0.48	0.50
45. Weak	1.09	0.73	0.50	0.73	2.20	0.50	0.45	2.25	2.74	0.32	0.68	1.24	1.44	1.19	2.74	0.99	1.83	2.75	1.22
Communication behaviors
49. Agreeing	2.36	1.32	1.66	1.87	0.80	2.28	1.22	1.41	0.99	1.60	2.58	1.82	0.92	0.58	0.88	1.17	0.57	0.70	1.96
50. Asking a question	0.65	0.59	0.49	4.10	0.56	0.35	3.45	0.15	0.27	0.12	0.23	0.27	0.35	0.35	0.57	2.45	0.17	0.44	0.25
52. Being sarcastic	0.28	0.21	0.37	0.36	0.23	0.20	0.14	0.54	0.29	0.14	0.95	0.18	0.09	0.03	0.08	0.53	0.02	0.09	0.24
47. Complaining	0.01	0.00	0.07	0.03	0.02	0.08	0.03	0.02	0.04	0.03	0.06	0.15	0.04	0.07	0.27	0.03	0.16	0.06	0.05
48. Confessing	0.41	0.21	0.09	0.23	0.61	0.13	0.09	0.58	0.50	0.12	0.22	0.48	0.37	0.18	1.06	0.18	0.83	0.84	0.59
51. Requesting	0.73	0.27	0.45	0.94	0.54	0.27	0.85	0.32	0.44	0.41	0.61	0.73	1.49	0.84	0.87	1.08	1.31	1.08	0.40
46. Telling	5.04	2.98	2.71	3.61	2.48	4.85	2.37	3.62	4.11	2.08	4.02	3.18	2.12	1.65	2.93	2.89	2.00	2.71	4.66
53. Warning	0.39	0.21	0.10	0.32	0.27	0.66	0.22	0.31	0.50	0.46	0.55	0.20	0.41	0.32	0.21	0.49	0.21	0.26	0.50
54. Whispering	0.06	0.02	0.01	0.17	0.35	0.05	0.03	0.17	0.29	0.00	0.01	0.02	0.08	0.13	0.09	0.08	0.19	0.21	0.17
Abnormal states
55. Crazy	0.79	0.38	0.33	0.39	1.29	0.18	0.41	0.28	0.76	0.16	0.39	0.34	0.47	0.54	0.48	0.28	0.38	0.68	0.13
56. Under narcotic influence	0.45	0.33	0.52	0.18	0.64	0.30	0.37	0.52	0.62	0.11	0.21	0.21	0.19	0.09	0.42	0.23	0.33	0.41	0.09
Could not be coded
Ambiguous entry	0.47	0.15	0.19	0.48	0.43	0.42	0.28	0.35	0.45	0.18	0.28	0.30	0.26	0.18	0.55	0.67	0.35	0.61	0.47
Missing response	1.06	0.80	0.67	1.02	1.37	1.06	0.68	1.35	1.28	0.61	1.22	1.13	0.72	0.64	1.02	1.07	0.70	0.96	1.09
Total	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0

Note. Entries in underlined bold font represent the intended categories. AFF = affection; AMU = amusement; HAP = happy; INT = interest; LUS = lust; PRI = pride; PSU = positive surprise; REL = relief; SER = serenity; ANG = anger; CON = contempt; DSG = disgust; DST = distress; FEA = fear; GUI = guilt; NSU = negative surprise; SAD = sad; SHA = shame; NEU = neutral.

To group these categories into larger themes, we recruited expert emotions researchers, who were all full members of the ISRE. They participated in a “card sorting” task, which provided participants with 56 “cards” that they could sort into as many categories as they wish, and they provided their own label for each category. The resulting data produced a 56×56 similarity matrix, which consisted of the number of times that any given codes appear in the same user-generated category. These data were examined using participant-centered analysis (https://support.optimalworkshop.com/en/articles/2626877-interpret-the-optimalsort-participant-centric-analysis-pca), which displays the sorting methods that converged most closely with those of multiple participants. To serve as a power test for card-sorting tasks, a simulation model determined that N=15 achieves a correlation of .90 between similarity matrices and those using a sample over ten times that number (Tullis & Wood, 2004). Sixteen scholars took part in this card sorting task.

The resulting grouping of the 56 categories is indicated in the headings of Table 5. It is noteworthy that affective states accounted for 32 of the 56 categories, which suggests that such states account for a substantial proportion of open-ended judgments. However, 24 of the categories were grouped into themes that the experts did not consider affective states, even though these responses were generated by participants after listening to stimuli designed to represent such states. In particular, additional categories could be classified under the heading of person descriptors, communication behaviors, and abnormal states. Person descriptors included wanting attention, being dominant, hesitating, being humble, being polite, rushing, being secretive, sincerity, seriousness, sympathy, thoughtfulness, wanting to be understood, and being weak. Communication behaviors included advising, complaining, confessing, confirming, asking questions, making requests, sarcasm, warning, and whispering. Abnormal states included being crazy or under the influence of narcotics.

Several patterns are worth mentioning that consisted of 9% or more responses. In terms of feeling states, amusement was often attributed to happiness and excitement. Happiness was often attributed to amusement and excitement. Serenity was often attributed to sadness and neutrality. Contempt and disgust were often attributed to anger. Distress was often attributed to anger, fear, and sadness. Fear was often attributed to distress. Guilt was often attributed to sadness. Negative surprise was often attributed to anger. Sadness was often attributed to distress. Shame was often attributed to sadness. In terms of person descriptors, the negative emotions of anger, contempt, disgust, and negative surprise were frequently attributed to dominance, as were the positive emotions of affection, interest, and pride. Lust was often attributed to being secretive. None of the communication behaviors or abnormal states was part of major confusion patterns.

Discussion

We started this investigation with a basic question: What do we hear in the voice? Taking a large-scale approach to examining the judgment of emotional prosody, we extended a long-standing tradition on emotion recognition research that has focused primarily on multiple-choice judgments of a small number of emotions expressed via still photographs of the face. Emotional prosody is unique among communication channels, in that it necessarily conveys verbal and nonverbal content simultaneously. Emotional expressions came from 100 professional actors from five English-speaking nations that span four continents. These actors portrayed 18 distinct emotional categories, particularly including more positive states than typically sampled in research on emotion perception. Participants had little constraint on their judgment process³ and were asked only what the speaker was trying to convey to the listener. In conducting a project of this unusual magnitude, the goal was to provide additional data to fill in the basic science of understanding the human voice in expressing emotion.

Rather than organizing the paper around hypotheses, we examined three research questions within the larger question of what people hear in the voice.

First, we examined what aspects of emotion listeners hear in vocal prosody. Participants spontaneously reported responses that cut across the chronological emotion process (Frijda, 2007). This includes not only feeling states—which have been the focus of response choices in conventional studies—but also the situation that elicited the emotion, the cognitive appraisals that analyze the situation, the physiological arousal associated with the feeling state, the expressive behavior that intentionally or unintentionally conveys information about feeling states, attempts at regulating emotions, and also attempts at influencing social partners. The current study provides the first descriptive data of this kind. Some components were more frequently mentioned for some emotions, which suggests that not all emotions are equally linked to all components in the participants’ perceptions. For example, social/moral emotions had the highest percentages of responses coded as cognitive appraisal; and interest, lust, and affection had the highest percentages of responses coded as influencing others. There were also some relative differences across cultures. Singaporean participants provided judgments that were less often coded as feeling states, and more often as situations, cognitive appraisals, and physiological arousal. Participants from India judged stimuli less often as cognitive appraisals and more often as attempts to influence others. These results should be replicated to see if similar patterns can also be observed in other samples of expressive stimuli and participants. Research on how people spontaneously infer different components of the emotion process from expressive behavior is still emerging, but has the potential to widen our understanding of the social function of emotion.

Second, we asked how accurate open-ended judgments of emotions in vocal prosody are. For this, we examined only those responses that consisted of feeling states—which are the only ones strictly speaking that can be considered accurate vs. inaccurate. Doing so, we found modest average accuracy of 14.2% in responding with the same intended emotional category. Although average categorical accuracy was low, it was fairly high for some emotions with anger reaching the highest rates at 61.2%. We suggest that accurate open-ended vocal emotion recognition requires both that the emotion in question has a distinct acoustic pattern that is being recognized, and that the emotion concept in question is salient and readily accessible when recording the written response. (In addition, participants also need to find emotional states to be the most relevant feature of the stimulus when reporting their impressions to researchers, and therefore accuracy is here based only on responses that were coded as feeling states in the first place). This seems to be the case for at least some emotion categories—anger, fear, happiness, and sadness—which received fairly high categorical recognition rates. We note that these emotions all belong to those that are historically considered as “basic” (Ekman, 1992). Future research is required to find out if the low recognition rates for the other emotions were caused by less distinct acoustic patterns, lower salience of the emotion concepts in question, or a combination of both.

Accuracy in terms of affective dimensions was overall more substantial. Participants matched the valence of the intended emotion an average of 56.3% of the time and the arousal level of the intended emotion an average of 60.1% of the time. When valence and arousal were combined into four quadrants, participant responses were considered accurate an average of 41.5% of the time. Arousal-based accuracy also showed the smallest effect size for cultural differences, compared with categorical and valence-based accuracy. These findings are in accordance with previous rating studies, which show that especially arousal may modulate many vocal cues and is well recognized in the voice (e.g., Laukka et al., 2005). When participants’ responses do not directly match the intended emotion, these findings suggest that they still often match another emotion that is similar to the intended emotion in terms of arousal and valence. This suggests that listeners’ perceptions are not as fine-grained as simply falling into emotion categories.⁴ Although these findings provide potentially suggestive data, more research is needed to address explicitly the controversial question of to what extent vocal expressions are perceived in terms of emotion categories or dimensions (see Cowen et al., 2019).

It was interesting to note that women outperformed men in categorical accuracy but not in arousal-based or quadrant-based accuracy. This suggests that women in our sample were more attuned to nuance in recognizing specific emotions, but had no greater sensitivity to the underlying dimensions of core affect.

Finally, we examined what else people hear in vocal prosody other than emotional states. We took a holistic approach to explore what types of responses emerged when listeners had no constraints on their reactions to a diverse pool of vocal emotional expressions. In doing so, we found that there was a wide range of ways in which people spontaneously perceive the intent behind effective vocal cues. Like Frijda (1953) and Haidt and Keltner (1999) had previously shown for facial expressions, our results showed that participants did not only describe feeling states when given the opportunity to provide open-ended responses of vocal expressions. Our taxonomy of these responses thus included not only emotional categories but also cues about person descriptors, communication behaviors, and abnormal states. These categories emerged from a rating task completed by expert scholars in the field of emotion. Whereas participants are typically directed to discrete categories of affective experience in research studies, we found that participants who could freely convey what they hear in the voice produced a diverse set of judgments. The responses were coded by emotions research experts into 56 categories that not only included 32 affective states but also included 13 person descriptors, 9 communication behaviors, and 2 abnormal states. The complex taxonomy inherent in these findings points to the complexity of social cues that listeners hear in vocal tone. The inclusion of communication behaviors speaks to the debate about whether emotional stimuli should be considered “expression” vs. “communication” (Ekman, 1997). That is, the internal experience of emotion can surface in terms of involuntary expressive behavior, but in addition, we may have evolved to imitate these involuntary cues to convey deliberate messages to others. To the extent that listeners perceive communication behaviors in the vocal signals of others, this supports the notion that emotional displays represent more than spontaneous reflexes. The inclusion of person descriptors suggests potential links between emotion perception and broader aspects of social perception (e.g., Todorov, 2017), which could be investigated in future studies.

Overall, our findings of open-ended accuracy and the diversity of judgments may also provide new insights for research on the real-life outcomes of emotion recognition skills (e.g., Schmid Mast & Hall, 2018). Future studies could, for example, investigate which kind of inferences (other than emotion categories) are associated with successful social interactions.

Limitations

A number of important limitations qualify the findings reported above.

First, we note the wording of our stimulus sentences. Our goal was to have sentences that were long enough for variance in prosody and that could apply to 19 different affective states. However, participants could have interpreted the particular sentences (“Let me tell you something” and “That is exactly what happened”) as implying influence attempts. This would have influenced the results for the first and third research questions. In particular, it may have increased the proportion of influence attempts appearing in Table 1, which accounted for an average of 20.0% of all responses, and may have increased the proportion of communication behaviors appearing in Table 4. Future research could use nonsense words, content filtering, or sentences that are more truly neutral in content with respect to emotional processes.

A related limitation is to the wording of the instructions. Listeners were told that the “expressions might involve emotions, intentions, or other kinds of feelings that a person could have,” and were instructed to think about “what kind of feeling would likely have elicited such a vocal expression.” This may also have influenced the results for the first and third research questions. It could have increased the proportion of feeling states in participant responses, which accounted for an average of 43.2% of the text strings in Table 1, and for the emotional states listed in Table 4, which account for 32 of the 56 categories coded.

The coding process also introduced error, as it involved reading details about participants’ intended meanings that are limited by the precision of everyday emotional vocabulary in relatively brief text strings. For example, participants may have used the same words to refer to feeling states as they do to refer to physiological arousal, e.g., calm, and may have used more generalized vocabulary to refer to more specific emotions (e.g., happy for affection, and sadness for guilt).⁵

A potential limit to generality is that all five nations represented in the study are English-speaking. This was chosen deliberately so that linguistic properties would not be confounded with paralinguistic ones. This also meant that the emotion categories were consistent in their meaning across the five nations. It would be worthwhile also to conduct large-scale research on vocal prosody with other language groups. We note that all data were collected for individuals’ judgments of members of their own cultural group, that is, within-cultural data versus cross-cultural data. A cross-cultural judgment study using open-ended responses would be welcome. Such a study could provide novel information about possible differences in spontaneous impressions of emotion expressions from one’s own vs. other cultural groups.

Another concern is that the studies made use of posed expressions instead of spontaneous speech—consistent with most research on the perception of vocal tone. We made this decision for the feasibility of including 18 different emotions and neutral portrayals. However, actors may enact vocal tone in a way that differs from spontaneous speech. For example, in a comparison of acoustic patterns in spontaneous and posed expression databases, Juslin et al. (2018) reported small but systematic differences. We attempted to assuage this concern by using stimuli developed via the Stanislavski (1936) technique of method acting, which increases authenticity by inviting actors to relive past experiences (Scherer & Bänziger, 2010). Furthermore, we note Scherer’s (2013) challenge to the notion that using acted samples in empirical research is problematic, based on several sources of evidence (see also Scherer & Bänziger, 2010). Notably, Scherer (2013) found that speech samples from mood induction and acting were comparable in terms of their acoustic properties. Taken together, we believe that the limitation of using posed expressions is an important one, but that it does not invalidate the research presented here. Related to this concern is the fact that we informed participants that the stimuli were produced by actors, which may have increased their awareness that these were intended to be emotional expressions.⁶

Although maintaining the argument that these stimuli are valid representations of emotional expression, we acknowledge that it is not possible to compare directly the accuracy rates in this open-ended study to those in forced-choice studies for which stimuli were hand-selected to be prototypical representations. The current stimulus set was not hand selected, as all actor portrayals were included, unlike other studies that use pre-screening to ensure a baseline intensity of emotional cues.

Finally, our data on accuracy come from only those responses in which participants judged stimuli as representing a feeling state. This is because in other responses the feeling state was not the most salient feature to the listener, and so their response could not be judged as accurate vs. inaccurate per se. In further work, researchers may choose to capture responses as generated spontaneously by participants being prompted only to describe what the actor is trying to express, or they may limit participants to emotion labels in order to analyze the entire data set.

Conclusion

In asking the question of what we hear in the voice, we conclude that people hear quite a bit. Beyond emotional states, people hear a wide range of messages. This diversity in what people perceive in prosody indicates the far-reaching value of emotion as a means of communication, including information about the environment, other people’s characteristics, and their behavioral intentions. People make a wealth of spontaneous inferences from expressive cues, and many of these inferences were captured in the current open-ended study and contribute to a fuller understanding of vocal emotion perception. We note that the usable nature of this information is at the heart of the social function of emotion—after all, perceiving is for doing (Gibson, 1979). The recognition of emotional content embedded within nonverbal behavior is critical for real-life outcomes (Schmid Mast & Hall, 2018), and the perception of emotion in prosody is crucial within this important human process.

Supplemental Material

sj-docx-1-psp-10.1177_01461672211029786 – Supplemental material for What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody

Supplemental material, sj-docx-1-psp-10.1177_01461672211029786 for What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody by Hillary Anger Elfenbein, Petri Laukka, Jean Althoff, Wanda Chui, Frederick K. Iraki, Thomas Rockstuhl and Nutankumar S. Thingujam in Personality and Social Psychology Bulletin

Supplemental Material

sj-docx-2-psp-10.1177_01461672211029786 – Supplemental material for What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody

Supplemental material, sj-docx-2-psp-10.1177_01461672211029786 for What Do We Hear in the Voice? An Open-Ended Judgment Study of Emotional Speech Prosody by Hillary Anger Elfenbein, Petri Laukka, Jean Althoff, Wanda Chui, Frederick K. Iraki, Thomas Rockstuhl and Nutankumar S. Thingujam in Personality and Social Psychology Bulletin

Footnotes

Acknowledgements

We thank Ann Liu and Alex Morel for research assistance and Alan Cowen for helpful suggestions. Yochi Cohen-Charash, Shlomo Hareli, Marcello Mortillaro, James Russell, Disa Sauter, Craig Smith, and 10 anonymous colleagues contributed expert ratings.

Author Contributions

The third, fourth, fifth, sixth, and seventh authors contributed equally, and appear in alphabetical order.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We acknowledge United States National Science Foundation BCS-0617624 to Hillary Anger Elfenbein and Swedish Research Council 2006-1360 to Petri Laukka.

ORCID iDs

Hillary Anger Elfenbein

Petri Laukka

Supplemental Material

Supplemental material is available online with this article.

Notes

References

Abelin

Å.

Allwood

(2000). Cross linguistic interpretation of emotional prosody. In Cowie

Douglas-Cowie

Schröder

(Eds.), Proceedings of the ISCA Workshop on Speech and Emotion (pp. 110–113). International Speech Communication Association.

Belyk

Brown

(2014). The acoustic correlates of valence depend on emotion family. Journal of Voice, 28, 523.e9–523.e18. https://doi.org/10.1016/j.jvoice.2013.12.007

Boucher

J. D.

Carlson

G. E.

(1980). Recognition of facial expression in three cultures. Journal of Cross-Cultural Psychology, 11, 263–280. https://doi.org/10.1177/0022022180113003

Buck

(2014). Emotion: A biosocial synthesis. Cambridge University Press.

Calvo

M. G.

Nummenmaa

(2016). Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cognition and Emotion, 30, 1081–1106. https://doi.org/10.1080/02699931.2015.1049124

Cordaro

D. T.

Keltner

Tshering

Wangchuk

Flynn

L. M.

(2016). The voice conveys emotion in ten globalized cultures and one remote village in Bhutan. Emotion, 16, 117–128. https://doi.org/10.1037/emo0000100

Cowen

A. S.

Laukka

Elfenbein

H. A.

Liu

Keltner

(2019). The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures. Nature Human Behaviour, 3, 369–382. https://doi.org/10.1038/s41562-019-0533-6

Darwin

(1965). The expression of the emotions in man and animals. University of Chicago Press. (Original work published 1872)

DiGirolamo

M. A.

Russell

J. A.

(2017). The emotion seen in a face can be a methodological artifact: The process of elimination hypothesis. Emotion, 17, 538–546. https://doi.org/10.1037/emo0000247

10.

Ekman

(1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200. https://doi.org/10.1080/02699939208411068

11.

Ekman

(1997). Should we call it expression or communication? Innovation: The European Journal of Social Science Research, 10(4), 333–344. https://doi.org/10.1080/13511610.1997.9968538

12.

Elfenbein

H. A.

Ambady

(2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128, 203–235. https://doi.org/10.1037/0033-2909.128.2.203

13.

Ellsworth

P. C.

Scherer

K. R.

(2003). Appraisal processes in emotion. In Davidson

R. J.

Scherer

K. R.

Goldsmith

H. H.

(Eds.), Handbook of affective sciences (pp. 572–595). Oxford University Press.

14.

Faul

Erdfelder

Buchner

Lang

A.-G.

(2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41, 1149–1160. https://doi.org/10.3758/BRM.41.4.1149

15.

Fehr

Russell

J. A.

(1984). Concept of emotion viewed from a prototype perspective. Journal of Experimental Psychology: General, 113, 464–486. https://doi.org/10.1037/0096-3445.113.3.464

16.

Frank

M. G.

Stennett

(2001). The forced-choice paradigm and the perception of facial expressions of emotion. Journal of Personality and Social Psychology, 80, 75–85. https://doi.org/10.1037/0022-3514.80.1.75

17.

Frijda

N. H.

(1953). The understanding of facial expression of emotion. Acta Psychologica, 9, 294–362. https://doi.org/10.1016/0001-6918(53)90021-3

18.

Frijda

N. H.

(2007). The laws of emotion. Lawrence Erlbaum.

19.

Gendron

Roberson

van der Vyver

J. M.

Barrett

L. F.

(2014). Cultural relativity in perceiving emotion from vocalizations. Psychological Science, 25, 911–920. https://doi.org/10.1177/0956797613517239

20.

Gibson

J. J.

(1979). The ecological approach to visual perception. Houghton Mifflin.

21.

Gillioz

Fontaine

J. R. J.

Soriano

Scherer

K. R.

(2016). Mapping emotion terms into affective space: Further evidence for a four-dimensional structure. Swiss Journal of Psychology, 75, 141–148.

22.

Goudbeek

Scherer

K. R.

(2010). Beyond arousal: Valence and potency/control cues in the vocal expression of emotion. Journal of the Acoustical Society of America, 128, 1322–1336. https://doi.org/10.1121/1.3466853

23.

Greasley

Sherrard

Waterman

(2000). Emotion in language and speech: Methodological issues in naturalistic approaches. Language and Speech, 43, 355–375. https://doi.org/10.1177/00238309000430040201

24.

Haidt

Keltner

(1999). Culture and facial expression: Open-ended methods find more expressions and a gradient of recognition. Cognition and Emotion, 13, 225–266. https://doi.org/10.1080/026999399379267

25.

Hall

J. A.

Andrzejewski

S. A.

Yopchick

J. E.

(2009). Psychosocial correlates of interpersonal sensitivity: A meta-analysis. Journal of Nonverbal Behavior, 33, 149–180. https://doi.org/10.1007/s10919-009-0070-5

26.

Hall

J. A.

Gunnery

S. D.

Horgan

T. G.

(2016). Gender differences in interpersonal accuracy. In Hall

J. A.

Schmid Mast

West

T. V.

(Eds.), The social psychology of perceiving others accurately (pp. 309–327). Cambridge University Press.

27.

Hofstede

(2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations across nations (2nd ed.). SAGE.

28.

Hönemann

Mixdorff

Rilliard

(2015). Classification of auditory-visual attitudes in German. In Proceedings of the 1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing (FAAVSP 2015) (pp. 202–207). International Speech Communication Association.

29.

Jarvis

B. G.

(2008). MediaLab (Version 2008) [Computer Software]. Empirisoft Corp.

30.

Juslin

P. N.

(1997). Can results from studies of perceived expression in musical performances be generalized across response formats? Psychomusicology, 16, 77–101. https://doi.org/10.1037/h0094065

31.

Juslin

P. N.

Laukka

(2003). Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin, 129, 770–814. https://doi.org/10.1037/0033-2909.129.5.770

32.

Juslin

P. N.

Laukka

Bänziger

(2018). The mirror to our soul? Comparisons of spontaneous and posed vocal expression of emotion. Journal of Nonverbal Behavior, 42, 1–40. https://doi.org/10.1007/s10919-017-0268-x

33.

Laukka

Elfenbein

H. A.

(2012). Emotion appraisal dimensions can be inferred from vocal expressions. Social Psychological and Personality Science, 3, 529–536. https://doi.org/10.1177/1948550611428011

34.

Laukka

Elfenbein

H. A.

Chui

Thingujam

N. S.

Iraki

F. K.

Rockstuhl

Althoff

(2010). Presenting the VENEC corpus: Development of a cross-cultural corpus of vocal emotion expressions and a novel method of annotating emotion appraisals. In Devillers

Schuller

Cowie

Douglas-Cowie

Batliner

(Eds.), Proceedings of the LREC 2010 Workshop on Corpora for Research on Emotion and Affect (pp. 53–57). European Language Resources Association.

35.

Laukka

Elfenbein

H. A.

Thingujam

N. S.

Rockstuhl

Iraki

F. K.

Chui

Althoff

(2016). The expression and recognition of emotions in the voice across five nations: A lens model analysis based on acoustic features. Journal of Personality and Social Psychology, 111, 686–705. https://doi.org/10.1037/pspi0000066

36.

Laukka

Juslin

P. N.

Bresin

(2005). A dimensional approach to vocal expression of emotion. Cognition and Emotion, 19, 633–653. https://doi.org/10.1080/02699930441000445

37.

Lausen

Schacht

(2018). Gender differences in the recognition of vocal emotions. Frontiers in Psychology, 9, Article 882.

38.

Lazarus

R. S.

(1991). Emotion and adaptation. Oxford University Press.

39.

Limbrecht-Ecklundt

Scheck

Jerg-Bretzke

Walter

Hoffman

Traue

H. C.

(2013). The effect of forced choice on facial emotion recognition: A comparison to open verbal classification of emotion labels. GMS Psycho-Social-Medicine, 10, 1–8. https://doi.org/10.3205/psm000094

40.

Nelson

N. L.

Russell

J. A.

(2016). A facial expression of pax: Assessing children’s recognition of emotion from faces. Journal of Experimental Child Psychology, 141, 49–64. https://doi.org/10.1016/j.jecp.2015.07.016

41.

Nordström

Laukka

Thingujam

N. S.

Schubert

Elfenbein

H. A.

(2017). Appraisal dimensions inferred from vocal emotion expressions are consistent across cultures: A comparison between Australia and India. Royal Society Open Science, 4, Article 170912. https://doi.org/10.1098/rsos.170912

42.

Ortony

Clore

G. L.

Collins

(1988). The cognitive structure of emotions. Cambridge University Press.

43.

Planalp

DeFrancisco

V. L.

Rutherford

(1996). Varieties of cues to emotion in naturally occurring situations. Cognition and Emotion, 10, 137–153. https://doi.org/10.1080/026999396380303

44.

Rosenberg

E. L.

Ekman

(1995). Conceptual and methodological issues in the judgment of facial expressions of emotion. Motivation and Emotion, 19, 111–138. https://doi.org/10.1007/BF02250566

45.

Rosenthal

Hall

J. A.

DiMatteo

M. R.

Rogers

P. L.

Archer

(1979). Sensitivity to nonverbal communication: The PONS Test. The Johns Hopkins University Press.

46.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178. https://doi.org/10.1037/h0077714

47.

Russell

J. A.

(1993). Forced-choice response format in the study of facial expression. Motivation and Emotion, 17, 41–51. https://doi.org/10.1007/BF00995206

48.

Russell

J. A.

(1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115, 102–141. https://doi.org/10.1037/0033-2909.115.1.102

49.

Russell

J. A.

(2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172.

50.

Russell

J. A.

Feldman Barrett

(1999). Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology, 76, 805–819.

51.

Sauter

D. A.

(2017). The nonverbal communication of positive emotions: An emotion family approach. Emotion Review, 9, 222–234. https://doi.org/10.1177/1754073916667236

52.

Scherer

K. R.

(1994). Affect bursts. In Goozen

S. H. M.

van de Poll

N. E.

Sergeant

J. A.

(Eds.), Emotions: Essays on emotion theory (pp. 161–193). Lawrence Erlbaum.

53.

Scherer

K. R.

(2005). What are emotions? And how can they be measured? Social Science Information, 43, 591–598. https://doi.org/10.1177/0539018405058216

54.

Scherer

K. R.

(2013). Vocal markers of emotion: Comparing induction and acting elicitation. Computer Speech and Language, 27, 40–58. https://doi.org/10.1016/j.csl.2011.11.003

55.

Scherer

K. R.

Bänziger

(2010). On the use of actor portrayals in research on emotional expression. In Scherer

K. R.

Bänziger

Roesch

E. B.

(Eds.) Blueprint for affective computing: A sourcebook (pp. 271–294). Oxford University Press.

56.

Scherer

K. R.

Moors

(2019). The emotion process: Event appraisal and component differentiation. Annual Review of Psychology, 70, 719–745. https://doi.org/10.1146/annurev-psych-122216-011854

57.

Schlosberg

(1952). The description of facial expressions in terms of two dimensions. Journal of Experimental Psychology, 44, 229–237. https://doi.org/10.1037/h0055778

58.

Schmid Mast

Hall

J. A.

(2018). The impact of interpersonal accuracy on behavioral outcomes. Current Directions in Psychological Science, 27, 309–314.

59.

Shochi

Rilliard

Aubergé

Erickson

(2009). Intercultural perception of English, French and Japanese social affective prosody. In Hancil

(Ed.), The role of prosody in affective speech (pp. 31–60). Peter Lang.

60.

Simon-Thomas

E. R.

Keltner

D. J.

Sauter

Sinicropi-Yao

Abramson

(2009). The voice conveys specific emotions: Evidence from vocal burst displays. Emotion, 9, 838–846. https://doi.org/10.1037/a0017810

61.

Stanislavski

(1936). An actor prepares. Theatre Arts Books/Methuen.

62.

Todorov

(2017). Face value: The irresistible influence of first impressions. Princeton University Press.

63.

Tullis

Wood

(2004). How many users are enough for a card-sorting study? [Conference session]. Proceedings of the Usability Professionals Association (UPA) 2004 Conference, Minneapolis, MN, June 7–11, 2004.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.31 MB