Human Stress Detection: Cortisol Levels in Stressed Speakers Predict Voice-Based Judgments of Stress

Abstract

Despite recent evidence of a positive relationship between cortisol levels and voice pitch in stressed speakers, the extent to which human listeners can reliably judge stress from the voice remains unknown. Here, we tested whether voice-based judgments of stress co-vary with the free cortisol levels and vocal parameters of speakers recorded in a real-life stressful situation (oral examination) and baseline (2 weeks prior). Hormone and acoustic analyses indicated elevated salivary cortisol levels and corresponding changes in voice pitch, vocal tract resonances (formants), and speed of speech during stress. In turn, listeners’ stress ratings correlated significantly with speakers’ cortisol levels. Higher pitched voices were consistently perceived as more stressed; however, the influence of formant frequencies, vocal perturbation and noise parameters on stress ratings varied across contexts, suggesting that listeners utilize different strategies when assessing calm versus stressed speech. These results indicate that nonverbal vocal cues can convey honest information about a speaker’s underlying physiological level of stress that listeners can, to some extent, detect and utilize, while underscoring the necessity to control for individual differences in the biological stress response.

Keywords

stress detection voice response coherence nonverbal communication pitch fundamental frequency formant cortisol

Introduction

Vocal communication of stress has been a central topic of research for decades (Kreiman & Sidtis, 2011b; Scherer, 1986). Yet, due to the inherent complexities of defining, evoking, and measuring stress (Murray et al., 1996), and individual differences in the magnitude of the stress response and in its behavioral manifestation (Pisanski et al., 2018; Scherer, 1986), the effects of stress on vocal production and perception remain unclear.

Attempts to identify the nonverbal vocal correlates of stressed speech have most consistently shown an increase in fundamental frequency (F0, perceived as voice pitch) during stress (for reviews, see, Buchanan et al., 2014; Giddens et al., 2013; Kirchhübel et al., 2011). Fundamental frequency is positively related to the rate of vocal fold vibration, and thus increases as the vocal fold’s tense or as subglottal pressure increases, as often occurs under stress. Past research has shown less consistent effects of stress on other nonverbal vocal characteristics including voice perturbation and noise parameters, such as jitter (a measure of instability in vocal fold vibration causing small asymmetries in periodicity), shimmer (instability in voice amplitude), and harmonics-to-noise ratio (HNR, representing the degree of additive noise in the voice signal) (Kreiman & Sidtis, 2011a; Titze, 1994). The influence of psychological stress on resonances of the vocal tract (formant frequencies) and their relative spacing (ΔF, Pisanski et al., 2014; Reby & McComb, 2003) is also not known.

Importantly, in two recent papers, we show that increases in cortisol levels under lab-induced psychosocial stress (Trier Social Stress Test) and under naturally occurring stress (oral examination) predict increases in voice fundamental frequency (F0, pitch) among women (Pisanski, Nowak, et al., 2016; Pisanski et al., 2018). Similarly, Buchanan et al. (2014) showed that pauses in speech during stress are more pronounced in speakers with a relatively high cortisol stress response. Together, these studies indicate a degree of response coherence between vocal and endocrinological stress systems (Andrews et al., 2013) but, crucially, also show that vocal changes under stress may occur only or predominantly among speakers who show a corresponding biological stress response.

Researchers examining voice-based stress perception have focused largely on developing computer algorithms for speaker identification and speech recognition (both of which are degraded in stressed speech, Hansen & Patil, 2007) and related technologies for lie detection (for review, see, Hollien et al., 2014). Few studies have examined human listeners’ capacity to assess stress from the voice, and to the authors’ knowledge, no previous study has tested accuracy in voice-based stress detection, which here, we test by measuring how closely listeners’ judgments of stress map onto a key biological marker of stress in the speaker (cortisol). Earlier playback studies have, however, examined which vocal parameters predict listeners’ ratings of stress using either re-synthesized speech or natural speech stimuli recorded in real-life stressful scenarios, or under induced or imagined stress. These studies have generally shown a positive correlation between the mean or maximum F0 of speech and listeners’ perceptions of stress (Protopapas & Lieberman, 1997; Streeter et al., 1983) but have produced inconsistent results regarding the influence of other vocal parameters on perceived stress. While re-synthesized speech allows for careful control of vocal parameters, natural speech from real-life situations benefits from ecological validity, and critically, allows for direct tests of accuracy in listeners’ stress detection.

In the present study, we tested whether changes in the salivary cortisol levels and vocal parameters of speakers who were recorded under real-life stress predict listeners’ voice-based judgments of those speakers’ stress levels. Importantly, in light of recent evidence that vocal changes under stress covary with the magnitude of the biological stress response (Buchanan et al., 2014; Pisanski, Nowak, et al., 2016; Pisanski et al., 2018), we test the prediction that listeners’ stress ratings will correlate positively with the underlying cortisol levels of speakers, and with their stress-linked vocal parameters, particularly voice pitch.

Methods

Speakers and Stimuli

Twenty voice recordings were taken from a sub-sample of 10 female students registered in an upper-level psychology course who were audio-recorded in a quiet room immediately before an oral examination (stress context) and approximately 2 weeks prior to the exam (M= 12.3, SD = 2.2 days; baseline context), as part of a study examining response coherence in stress systems (see Pisanski, Nowak, et al., 2016 for detailed sample description and recording procedures). In both contexts, speakers were asked to familiarize themselves with and subsequently read the first five sentences of the Rainbow Passage (translation) from which we extracted the third sentence, translating to: “The rainbow is a division of white light into many beautiful colors.” Audio was digitally encoded using an M-Audio Fast Track ultra interface at a sampling rate of 44.1 kHz and 16-bit amplitude quantization and stored onto a computer as WAV files. Voice stimuli were amplitude normalized to 70 dB RMS SPL in Praat (Boersma & Weenink, 2020) for playback.

Saliva samples were collected in duplicate from each speaker into 2 ml microtubes using the passive drool method (Gröschl, 2008) and were taken 15 min after the baseline and stress voice recordings to allow for cortisol produced by the adrenal glands to manifest in saliva (Kirschbaum et al., 1993). Saliva samples were frozen at –70°C until analysis. Free cortisol levels were measured with enzyme-linked immunosorbent assay (ELISA) commercial kits (DEMEDITEC®, Germany; assay sensitivity 0.014 ng/ml, intra-assay CV 5.9%, inter-assay CV 9.4%). For additional details, see Pisanski et al. (2016).

Acoustic Analysis

Acoustic analysis was performed in Praat (Boersma & Weenink, 2020) on voiced segments of speech. Fundamental frequency parameters included F0 mean, minimum, maximum, and standard deviation (SD) measured using Praat's autocorrelation algorithm (search range 100–600 Hz). Formant frequencies (F1–F4) were measured using Praat’s Burg Linear Predictive Coding algorithm (ceiling 5500 Hz); formant number was adjusted to fit predicted values to observed formants. Formant spacing (ΔF) was taken as a measure of the distance among adjacent formants using equations described in Reby and McComb (2003) and Pisanski et al. (2014). We also measured noise (harmonics-to-noise ratio, HNR), frequency perturbation (jitter: local, local absolute, rap, ppq5, ddp), and amplitude perturbation (shimmer: local, local dB, apq3, apq5, apq11, dda) using Praat's cross-correlation algorithm. Due to strong collinearity, jitter measures and shimmer measures were respectively grouped into two principal components that explained 89% (Jitter PC) and 77% (Shimmer PC) of the variance in each perturbation parameter. Finally, speed of speech was measured as duration and words-per-minute (WPM).

Listeners

Fifty listeners (aged 19–28, M= 23.5, SD= 2.4; 52% female) were recruited from two universities in a large European city to rate voice stimuli. All participants provided informed consent. The study was performed in accordance with the Declaration of Helsinki on Biomedical Studies Involving Human Subjects and was approved by the Institutional Ethics Review Board (project no. 2014/13/B/HS6/02636).

Playback Procedure

Participants completed the experiment in individual sessions in a designated lab space. Voice stimuli were presented via a custom computer interface and Sennheiser HD 201 professional headphones at a constant, pre-set volume. Participants rated all 20 voice stimuli (10 baseline, 10 stress) in a randomized order on a scale from 1 (this person is not at all stressed) to 10 (this person is very stressed).

Results

Changes in Voice and Cortisol Under Stress

Linear mixed models (LMMs; see Table 1 footnotes for model structure) fit by restricted maximum likelihood confirmed that speakers had a higher voice pitch (F0), greater formant spacing (ΔF), and spoke faster (more WPM, shorter duration) under stress relative to their unstressed, baseline speech. There were no changes in speakers’ voice perturbation or noise parameters (jitter, shimmer, or HNR). A separate LMM confirmed that speakers’ cortisol levels were, on average, significantly higher under stress relative to baseline (Table 1).

Table 1.

Increased Cortisol Levels and Changes in Vocal Parameters During Stress.

			Linear mixed models^a
Speaker parameter	Baseline M (SD)	Stress M (SD)	Change M diff (SEM)	F (df = 1,9)	P
Cortisol (ng/ml)	5.3 (2.3)	8.8 (4.8)	+3.51 (0.2)	213.9	.001**
F0 mean (Hz)	210.83 (12.9)	221.75 (13.3)	+10.92 (2.2)	25.6	.001**
F0 min (Hz)	134.80 (33.3)	150.00 (33.9)	+15.17 (1.3)	1.3	.281
F0 max (Hz)	292.00 (28.8)	305.67 (34.7)	+13.7 (6.9)	3.9	.078†
F0 SD (Hz)	32.57 (12.1)	29.39 (8.8)	–3.2 (3.8)	0.7	.430
ΔF (Hz)	1,177.82 (55.6)	1,191.34 (53.3)	+13.5 (4.1)	10.9	.009*
Words-per-minute	174.05 (17.8)	188.03 (15.5)	+13.98 (4.17)	11.2	.008*
Duration (s)	3.13 (0.3)	2.89 (0.2)	–0.24 (0.08)	8.5	.017*
Jitter PC	–0.05 (0.92)	–0.20 (0.7)	+0.15 (0.3)	0.2	.691
Shimmer PC	0.21 (1.4)	0.26 (1.3)	+0.05 (0.5)	0.01	.915
Harmonics-to-noise ratio (dB)	15.08 (2.0)	16.06 (1.8)	+0.98 (0.6)	2.4	.153

Descriptive statistics and results of linear mixed models.

^aLMMs: Each speaker parameter was entered as the dependent variable, and context (baseline, stress) as a fixed factor. Speaker ID was included as a random subject variable with random intercept.

**p < .001. *p < .01. †p < .10.

Voice-Based Judgments of Stress

On average, listeners’ ratings of stress were not higher for voices under stress (M ± SEM = 5.2 ± 0.07) than at baseline (5.4 ± 0.07). To examine whether speakers’ cortisol levels predicted listeners’ stress ratings, stress ratings were then entered into two separate linear regression models, with speakers’ cortisol levels as the predictor. These significant models confirmed that listeners’ ratings of stress increased with the speakers’ cortisol levels (stress model: standardized β = 0.38, S.E. = 0.03, t = 9.1, p < .001; baseline model: β = 0.09, S.E. = 0.01, t = 2.1, p = .04). Simple zero-order Spearman’s correlations, pooling listeners’ stress ratings for each individual speaker, confirmed a positive relationship between mean stress ratings and speakers’ cortisol levels (r_s = 0.27). Stress ratings were unimodal but slightly positively skewed; however, regression models with log-transformed stress ratings produced comparable results.

To examine which voice parameters predicted listeners’ stress ratings, all 10 voice parameters were first subjected to principal component analysis (PCA) with varimax rotation to control for collinearity (see Table S1). The PCA produced four components explaining 88% of the variance in voice parameters, corresponding respectively to vocal perturbation and noise (PC1: jitter, shimmer, and HNR); speech rate (PC2: duration and WMP); voice pitch parameters (PC3: F0 mean, max, and SD); and formant spacing (PC4: ΔF).

These acoustic principal components were then entered into a LMM with context (baseline, stress) as a fixed factor, controlling for Speaker ID and Listener ID (Table S2, see footnotes for full model structure). The model showed a significant main effect of voice pitch parameters (PC3) on stress ratings overall. Crucially, however, the model revealed significant interactions between stress context and acoustic principal components, indicating that different vocal parameters predicted listeners’ stress ratings in the baseline and stress contexts.

To investigate this further, separate LMMs were conducted for each context. In the Stress model (F = 4354.2_1,495, p < .001), the acoustic principal components corresponding to voice pitch parameters (PC3: F = 29.5_1,495, p < .001) and to voice perturbation and noise (PC1: F = 22.8_1,495, p < .001) significantly predicted listeners’ ratings of stress. Zero-order correlations, averaging across listeners’ ratings (n = 50) within speakers and contexts, confirmed moderate positive relationships between listeners’ stress ratings and the F0 mean (r = .0.37), F0 min (r = .45), F0 SD (r = .39), and jitter (r = .52) of speakers’ voices in the stress context. Relatively weaker correlations were observed between listeners’ ratings and the shimmer (r = .29) and HNR (r = –.19) voice parameters of stressed speakers (see Table S3). In the Baseline model (F = 3682.3_1,495, p < .001), the voice principal components corresponding to formant spacing (PC4: F = 13.7_1,495, p < .001) and again to voice pitch (PC3: F = 7.5_1,495, p = .006) explained significant variance in listeners’ stress ratings. Zero-order bivariate correlations were, however, small in effect size, with the largest correlation observed for mean F0 (r = .28; see Table S3 for all correlations). Although the LMMs indicated that speech rate parameters (PC2) did not predict listeners’ stress ratings in either a stressful or baseline context, bivariate correlations showed moderate relationships between duration (r = –.38) and words-per-minute (r = .33) on ratings of stressed speech.

Discussion

Our results show that listeners’ voice-based judgments of stress can be predicted both by the underlying biological stress response of speakers and by their vocal parameters.

Increases in the free cortisol levels of university students recorded during an oral examination, and 2 weeks prior, positively predicted listeners’ voice-based ratings of stress. Acoustic analyses further showed that fundamental frequency (pitch), vocal tract resonances (formants) and speed of speech all increased in speakers during stress, and that higher voice pitch consistently predicted higher stress ratings by listeners. However, the influence of other vocal parameters on stress ratings was less consistent and varied by context. Taken together, these results suggest that listeners can, to some extent, judge a speakers’ physiological stress response from their voice.

Critically, as listeners’ stress ratings were, on average, not predicted by the presence or absence of a stressful context (i.e., oral examination), our results further underscore the importance of controlling for individual differences in speakers’ biological stress responses when conducting research on stressed speech (Pisanski, Nowak, et al., 2016). Indeed, not all speakers in a ‘stressful’ condition will show a physiological stress response nor corresponding vocal changes, and thus, listeners’ perceptions of stress should not be expected to vary as a mere function of context, but rather, as our results suggest, may vary instead with individual differences in biological and behavioral markers of stress, regardless of context.

With the exception of voice pitch (F0) that consistently predicted stress ratings, different combinations of vocal parameters predicted listeners’ stress ratings of speech recorded in baseline and stress contexts, suggesting that listeners utilize different perceptual strategies when assessing calm versus stressed speech, as recently demonstrated for listeners’ assessments of different pain intensities from nonverbal vocalizations (Raine et al., 2018). Listeners’ ratings of stress in stressed speech were also predicted by vocal perturbation and noise parameters, despite a lack of strong evidence that these parameters changed under stress in our sample of speakers. Controlled psychoacoustic experiments on a larger and more diverse sample of voice stimuli and listeners are now needed to systematically test and model the relationships among specific vocal parameters and listeners’ judgments of stress, mapping both onto the physiological stress responses of speakers, to better inform multimodal models of stress detection.

Public speaking and social evaluation evoke among the strongest stress responses in humans (Dickerson & Kemeny, 2004), often manifesting both physiologically and behaviorally, with some limited evidence of response coherence (covariation) in psychophysical responses to stress (Andrews et al., 2013). The capacity to measure an underlying physiological or endocrinological stress response from a noninvasive behavioral marker such as the voice could improve the efficacy of real-time stress detection, yet has posed a challenge for researchers (Alberdi et al., 2016; Andrews et al., 2013). Our results indicate that human listeners can successfully utilize nonverbal parameters of the voice to gauge stress, and that they may do so by exploiting correspondences between vocal and biological markers of stress (e.g., pitch and cortisol: Pisanski, Nowak, et al., 2016). This supports the hypothesis that vocal communication in humans, like many other mammals (Morton, 1977), functions to communicate ecologically and socially relevant information about speaker traits and motivational or emotional states (Pisanski & Bryant, 2019). While humans have a unique capacity to volitionally modulate nonverbal parameters of the voice (e.g., to feign or conceal emotional states, Pisanski, Cartei, et al., 2016), the current findings suggest that the human voice conveys honest cues to stress, most likely due to the relationship between the physiological stress response and vocal anatomy, that may be difficult to conceal. At the same time, our results show that only 7% of the variance in listeners’ stress ratings was explained by speakers’ cortisol levels, indicating that the capacity to accurately assess stress from the voice in humans is far from perfect.

Supplemental Material

sj-xlsx-1-pec-10.1177_0301006620978378 - Supplemental material for Human Stress Detection: Cortisol Levels in Stressed Speakers Predict Voice-Based Judgments of Stress

Supplemental material, sj-xlsx-1-pec-10.1177_0301006620978378 for Human Stress Detection: Cortisol Levels in Stressed Speakers Predict Voice-Based Judgments of Stress by Katarzyna Pisanski and Piotr Sorokowski in Perception

Supplemental Material

sj-pdf-2-pec-10.1177_0301006620978378 - Supplemental material for Human Stress Detection: Cortisol Levels in Stressed Speakers Predict Voice-Based Judgments of Stress

Supplemental material, sj-pdf-2-pec-10.1177_0301006620978378 for Human Stress Detection: Cortisol Levels in Stressed Speakers Predict Voice-Based Judgments of Stress by Katarzyna Pisanski and Piotr Sorokowski in Perception

Footnotes

Acknowledgements

We thank Judyta Nowak (Institute of Genetics and Microbiology, Faculty of Biological Sciences, University of Wroclaw) for conducting hormone analysis.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a research grant from the National Science Centre to K.P. (2014/13/B/HS6/02636).

ORCID iD

Katarzyna Pisanski

Supplemental Material

Supplemental material for this article is available online.

References

Alberdi

Aztiria

Basarab

(2016). Towards an automatic early stress recognition system for office environments based on multimodal measurements: A review. Journal of Biomedical Informatics, 59, 49–75.

Andrews

Ali

Pruessner

J. C.

(2013). Reflections on the interaction of psychogenic stress systems in humans: The stress coherence/compensation model. Psychoneuroendocrinology, 38, 947–961.

Boersma

Weenink

(2020). Praat: Doing phonetics by computer v 6.1.21 (6.1.21) [Computer software]. www.praat.org

Buchanan

T. W.

Laures-Gore

J. S.

Duff

M. C.

(2014). Acute stress reduces speech fluency. Biological Psychology, 97, 60–66.

Dickerson

S. S.

Kemeny

M. E.

(2004). Acute stressors and cortisol responses: A theoretical integration and synthesis of laboratory research. Psychological Bulletin, 130, 355.

Giddens

C. L.

Barron

K. W.

Byrd-Craven

Clark

K. F.

Winter

A. S.

(2013). Vocal indices of stress: A review. Journal of Voice, 27, 390.e21–390.e29.

Gröschl

(2008). Current status of salivary hormone analysis. Clinical Chemistry, 54, 1759–1769.

Hansen

J. H.

Patil

(2007). Speech under stress: Analysis, modeling and recognition. In C. Müller (Ed.), Speaker classification I (pp. 108–137). Springer.

Hollien

Bahr

R. H.

Harnsberger

J. D.

(2014). Issues in forensic voice. Journal of Voice, 28, 170–184.

10.

Kirchhübel

Howard

D. M.

Stedmon

A. W.

(2011). Acoustic correlates of speech when under stress: Research, methods and future directions. International Journal of Speech Language and the Law, 18, 75–98. https://doi.org/10.1558/ijsll.v18i1.75

11.

Kirschbaum

Pirke

K. M.

Hellhammer

D. H.

(1993). The ‘Trier Social Stress Test’—A tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology, 28, 76–81. https://doi.org/119004

12.

Kreiman

Sidtis

(2011a). Foundations of voice studies: An interdisciplinary approach to voice production and perception. Wiley-Blackwell. http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0631222979.html

13.

Kreiman

Sidtis

(2011b). Perception of emotion and personality from voice. In J. Kreiman & D. Sidtis (Eds), Foundations of voice studies: An interdisciplinary approach to voice production and perception (pp. 302–360). John Wiley & Sons.

14.

Morton

E. S.

(1977). On the occurrence and significance of motivation-structural rules in some bird and mammal sounds. The American Naturalist, 111, 855–869.

15.

Murray

I. R.

Baber

South

(1996). Towards a definition and working model of stress and its effects on speech. Speech Communication, 20, 3–12.

16.

Pisanski

Bryant

G. A.

(2019). The evolution of voice perception. In Eidsheim

N. S.

Meizel

K. L.

(Eds.), The Oxford handbook of voice studies. Oxford University Press.

17.

Pisanski

Cartei

McGettigan

Raine

Reby

(2016). Voice modulation: A window into the origins of human vocal control? Trends in Cognitive Sciences, 20, 304–318.

18.

Pisanski

Fraccaro

P. J.

Tigue

C. C.

O’Connor

J. J. M.

Röder

Andrews

P. W.

, . . . Feinberg

D. R.

(2014). Vocal indicators of body size in men and women: A meta-analysis. Animal Behaviour, 95, 89–99. https://doi.org/10.1016/j.anbehav.2014.06.011

19.

Pisanski

Kobylarek

Jakubowska

Nowak

Walter

Błaszczyński

, . . . Piątek

(2018). Multimodal stress detection: Testing for covariation in vocal, hormonal and physiological responses to Trier Social Stress Test. Hormones and Behavior, 106, 52–61.

20.

Pisanski

Nowak

Sorokowski

(2016). Individual differences in cortisol stress response predict increases in voice pitch during exam stress. Physiology & Behavior, 163, 234–238.

21.

Protopapas

Lieberman

(1997). Fundamental frequency of phonation and perceived emotional stress. The Journal of the Acoustical Society of America, 101, 2267–2277.

22.

Raine

Pisanski

Simner

Reby

(2018). Vocal communication of simulated pain. Bioacoustics, 1–23. https://doi.org/10.1080/09524622.2018.1463295

23.

Reby

McComb

(2003). Anatomical constraints generate honesty: Acoustic cues to age and weight in the roars of red deer stags. Animal Behaviour, 65, 519–530. https://doi.org/10.1006/anbe.2003.2078

24.

Scherer

K. R.

(1986). Voice, stress, and emotion. In Appley

M. H.

Trumbull

(Eds.), Dynamics of stress: Physiological, psychological, and social perspectives (pp. 157–179). Plenum Press.

25.

Streeter

L. A.

Macdonald

N. H.

Apple

Krauss

R. M.

Galotti

K. M.

(1983). Acoustic and perceptual indicators of emotional stress. Journal of the Acoustical Society of America, 73, 1354–t360.

26.

Titze

I. R.

(1994). Principles of vocal production. Prentice-Hall.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB

0.17 MB