The Effects of Hearing-Aid Amplification and Noise on Conversational Dynamics Between Normal-Hearing and Hearing-Impaired Talkers

Abstract

There is a long-standing tradition to assess hearing-aid benefits using lab-based speech intelligibility tests. Towards a more everyday-like scenario, the current study investigated the effects of hearing-aid amplification and noise on face-to-face communication between two conversational partners. Eleven pairs, consisting of a younger normal-hearing (NH) and an older hearing-impaired (HI) participant, solved spot-the-difference tasks while their conversations were recorded. In a two-block randomized design, the tasks were solved in quiet or noise, both with and without the HI participant receiving hearing-aid amplification with active occlusion cancellation. In the presence of 70 dB SPL babble noise, participants had fewer, slower, and less well-timed turn-starts, while speaking louder with longer inter-pausal units (IPUs, stretches of continuous speech surrounded by silence) and reducing their articulation rates. All these changes are indicative of increased communication effort. The timing of turn-starts by the HI participants exhibited more variability than that of their NH conversational partners. In the presence of background noise, the timing of turn-starts by the HI participants became even more variable, and their NH partners spoke louder. When the HI participants were provided with hearing-aid amplification, their timing of turn-starts became faster, they increased their articulation rate, and they produced shorter IPUs, all indicating reduced communication effort. In conclusion, measures of the conversational dynamics showed that background noise increased the communication effort, especially for the HI participants, and that providing hearing-aid amplification caused the HI participant to behave more like their NH conversational partner, especially in quiet situations.

Keywords

conversational dynamics turn-taking hearing impairment hearing aid

Introduction

Hearing aids aim to compensate for the loss of hearing by presenting an amplified and processed acoustic signal to impaired ears. Traditionally, the development of new processing features and designs has focused on the reception of speech and has been evaluated by applying listening tests that measure speech intelligibility. Common for these tests are that they solely focus on listening, and traditionally operate at intelligibility levels that are much lower than would be expected for comfortable communication, such as at a speech reception threshold of 50%. Recently, both the academic and industrial communities have increased focus on developing and using test methods reflecting real-life hearing-related function, activity, and participation (for a review, see Keidser et al., 2020). The purpose of this study was to explore the effect of hearing-aid amplification in a natural face-to-face conversation with and without the presence of background noise between hearing-impaired (HI) and normal-hearing (NH) participants. The conversations were quantified by different measures of the prosody and pattern of the speech produced by the interlocutors, constituting the conversational dynamics.

As opposed to traditional listening tests, real-life communication involves an interactive overlap between speech comprehension (listening) and production (talking). During a conversation, interlocutors (conversational partners) can conduct repairs, ask for clarifications, and signal that they are experiencing difficult, verbally or through body language, all of which may lead to changes in the communication. During listening, the interlocutor must simultaneously comprehend what the other person is saying while planning their own upcoming response (for a review, see Levinson & Torreira, 2015). It has been argued that turn-taking between people in a conversation is cognitively demanding because language production and comprehension partly take up the same cognitive resources (Barthel & Sauppe, 2019; Hagoort & Indefrey, 2014). Whether seated face-to-face or in separate locations (e.g., telephone call), multiple studies have found that the typical floor-transfer offset (FTO), i.e. the interval from when one person stops talking to when the next person starts, is slightly positive with modal response times around 200 ms in dialogue (Aubanel, Cooke, Villegas, & Garcia Lecumberri, 2011; Brady, 1968; Heldner & Edlund, 2010; Levinson & Torreira, 2015; Norwine & Murphy, 1938; Stivers et al., 2009). This timing of turns is universal across languages and cultures (Stivers et al., 2009), and it has been suggested that people choose to optimize for socially appropriate timing of responses at the expense of increased cognitive effort (Barthel & Sauppe, 2019). The time course involved in preparing and uttering speech is at least 600 ms for a single word and over one second for multiple words (Indefrey & Levelt, 2004; Magyari et al., 2014). To deliver a verbal response 200 ms after the interlocutor stops speaking, a person cannot wait until the interlocutor is silent, but must predict the content of their interlocutor's speech so as to start planning their response in overlap with the ongoing turn (Barthel et al., 2016; Bögels et al., 2015; Corps, Gambi, & Pickering, 2018; Gisladottir et al., 2015; Levinson & Torreira, 2015). The prepared response must also be launched at an appropriate time, which can be identified using a variety of turn-end cues (Bögels & Torreira, 2015; Brusco, Vidal, Beňuš, & Gravano, 2020; De Ruiter, Mitterer, & Enfield, 2006; Gravano & Hirschberg, 2011). If these turn-end cues are not correctly interpreted, an interlocutor could launch a response much earlier (overlapping speech) or later (gap) than intended, causing the turn-taking timing (FTO) to become more variable, without necessarily affecting the average FTO.

While HI listeners often complain about not being able to hear, their difficulties often manifest as miscommunications in social interactions (Kiessling et al., 2003). Miscommunications, or signs thereof, cause changes in the conversational dynamics for both the HI talker and their interlocutor. It has been observed that when communicating with a HI person, their NH interlocutor adapts the level and spectral content of their speech in face-to-face conversation (Beechey, Buchholz, & Keidser, 2020b; Hazan et al., 2019). This effect has also been identified when interlocutors were not seated face-to-face, i.e. in two different rooms, where the NH person speaks at positive SNRs when communicating with HI interlocutors (Sørensen, 2021a), whereas similar communication between two NH interlocutors happened at negative SNRs (Sørensen, Fereczkowski, & MacDonald, 2021). Such increases in speech level and fundamental frequency is similar to the Lombard effect, e.g., the involuntary change of speech in the presence of background noise (Lombard, 1911).

Increased noise levels can also cause NH talkers to adapt their articulation rates to match that of their HI interlocutors (Sørensen, 2021a), which has been interpreted as a sign of the NH talkers trying to alleviate the communication load posed on the HI listeners by their hearing loss and the background noise. Furthermore, Sørensen (2021a) found that, with increasing noise levels, both NH and HI interlocutors started their turns later and with more variability, and the turn-starts of HI talkers were even later and more variable than those of the NH talkers. This suggests that when exposed to challenging conditions, the changes in FTOs can be used as a measure of difficulty in the conversation (communication effort). Interlocutors have also been observed to increase the duration of their utterances in noise (Beechey et al., 2018; Sørensen, 2021a; Sørensen, Fereczkowski & MacDonald, 2021; Watson et al., 2020), and it has been speculated that interlocutors will, when possible, attempt to start their turn around 200 ms by adding filler words, such as “uhm” and “ahh”, to their not-fully-prepared utterance (Barthel & Sauppe, 2019; Sørensen, 2021a). The usage of filler words will lead to longer inter-pausal units (IPUs, i.e., stretches of connected speech surrounded by pauses). Thus, the effects of altered communication effort on the median of FTO distributions may not be as large as the effects seen on the variability of FTOs and lengths of the IPUs. In addition to alterations in the speech level, articulation rate, turn-taking behaviour, and IPU length, HI interlocutors have been observed to take up more speaking time when communicating in noise (Lu, McKinney, Zhang, & Oxenham, 2021; Sørensen, 2021a), potentially because they adapt a strategy of talking, in order to avoid listening (Jaworski & Stephens, 1998; Stephens & Zhao, 1996). In this context, it is of interest to investigate how providing hearing-aid amplification affects the conversational dynamics of the HI, as well as the NH, interlocutor.

The goal of the present study was to investigate how the measures of conversational dynamics outlined above are affected by hearing-aid amplification and background noise in face-to-face communication between a younger NH and an older HI interlocutor. Participant pairs solved a collaborative spot-the-difference task, i.e., the DiapixDK task (Sørensen, 2021b), a Danish version of the DiapixUK task (Baker & Hazan, 2011). As adding background noise to the communication situation affects both talkers, we hypothesized that this will have large effects on the communication. It was expected that the addition of background noise would make the communication especially trying for the HI participants. We hypothesized that, in comparison to the addition of background noise, providing hearing-aid amplification to the HI interlocutor would result in smaller changes in the communication, as this alteration only directly affects the HI talker. We expected an increase in communication effort induced by (I) adding background noise, (II) having a hearing loss, and (III) not providing hearing-aid amplification to the HI interlocutor. We expected that changes in the three factors (I-III) would cause the following alterations in the conversational dynamics: (1) Slower, (2) more variable, and (3) fewer floor transfers (Aubanel et al., 2011; Sørensen, 2021a; Sørensen et al., 2021), as well as (4) increased speech levels (Beechey et al., 2018, 2020b; Sørensen, 2021a; Sørensen et al., 2021; Watson et al., 2020) with (5) slower articulation rates (Hazan et al., 2018; Tuomainen, Hazan, & Taschenberger, 2019) and (6) longer IPUs (Beechey et al., 2018; Sørensen, 2021a; Sørensen et al., 2021; Watson et al., 2020). For the DiapixDK task, we also expected to find (7) longer task completion times (Hazan & Baker, 2011; Sørensen, 2021a, Sørensen et al., 2021) and that (8) the proportion of time that HI participants spoke would increase (Jaworski & Stephens, 1998; Lu et al., 2021; Stephens & Zhao, 1996; Sørensen, 2021a). To investigate how the above hypothesized changes are experienced by the interlocutors, subjective ratings of the conversations were obtained. From these evaluations, we expected to find (9) lower subjective ratings of conversational success and (10) increased ratings of wanting to improve the situation, as well as (11) increased ratings of listening effort (Tuomainen et al., 2019) and (12) talking effort.

Methods

Participants

For the current study, 11 native-Danish conversational partners were recruited. All pairs consisted of an older HI participant (mean age 74.1, sd = 3.5, range 67.8 – 79.1 years), and a younger NH participant (mean age 25.3, sd = 6.1, range 19.9–39.1 years) who were not previously acquainted. The HI group was significantly older than the NH group [t(10) = -19.7, p < .001]. Six (54.5%) of the 11 HI participants and three (27.3%) of the NH participants were female. The pairs were matched at random, without considering sex, resulting in five mixed and six same-sex pairs. The experimental design was originally meant to include 12 pairs; however, due to COVID-19 restrictions, the data collection had to be stopped after 11 pairs.

All HI participants had symmetrical mild-to-moderate hearing loss with typical, high-frequency sloping N2/N3 audiograms (Bisgaard, Vlaming, & Dahlquist, 2010). Pure-tone thresholds were determined for the HI participants prior to the experimental visit (Figure 1A, mean PTA across 0.5, 1, 2, and 4 kHz = 35.1, sd = 7.5, range 23.1–48.8 dB HL). No significant correlation was found between age and PTA of the HI participants (r = -0.17, p = .6). All HI participants were experienced hearing-aid users (>1 year of hearing-aid usage) and reported using their hearing aids all day (81.8%) or for specific purposes such as watching television, work, or social events (18.2%). The hearing status of the NH participants was assessed by confirming that 20 dB HL tones at the PTA frequencies were audible in both ears.

Figure 1.

A) Individual pure-tone hearing thresholds for the HI participants averaged across ears in thin grey lines. The bold black line indicates the average hearing thresholds across subjects and the shaded area the standard deviation. The dotted line indicates the criteria for the NH listeners hearing threshold. B) Experimental setup of the sound-proof listening booth. Loudspeakers were positioned half-way between the participants, pointing perpendicularly into the room. Background noise was presented from the loudspeakers in half of the experimental conditions.

All participants gave their written informed consent and the study was approved by the regional ethical committee of the Capital Region of Copenhagen, Denmark (Board of Copenhagen, Denmark, reference H-20068621).

Experimental Conversational Task

Communication between the conversational partners was initiated using the DiapixDK spot-the-difference task (Sørensen, 2021b). In each trial, pairs were given a new DiapixDK picture and instructed to identify 10 of the 12 differences existing between the two near-identical pictures and to do this as fast as possible. If 10 differences were not found within 10 min, the trial was terminated to reduce the overall test time. The pairs were seated face-to-face with 2 meters between them in a soundproof booth (see Figure 1B).

The pairs were asked to solve DiapixDK tasks in a 2 × 2 design, with conditions varying the presence of background noise (quiet and noise) and hearing-aid amplification provided to the HI participant (unaided and aided, see section Hearing-Aid Fitting). The experimental task required involvement from both interlocutors to be solved and does not allow for the HI participant to withdraw from the communication situation. The pairs were also subjected to a fifth condition with an alternative hearing-aid signal processing scheme in noise. However, due to technical issues, the implemented beamforming was not as narrow as desired, resulting in only a small change in SNR compared to the omnidirectional condition. The data from this condition are not included in the following as no significant differences between the omni- and directional experimental conditions were identified.

In the conditions with noise, a 20-talker speech babble consisting of gender-balanced excerpts of NH interlocutors solving the DiapixDK from Sørensen et al. (2021) was presented from two loudspeakers (JAMO D400, Klipsch, IN, USA) positioned between the two participants at an angle of 45- and 315-degrees azimuth relative to each participant (see Figure 1B). The background noise was presented at a level of 70 dB SPL (unweighted), calibrated using a sound-level meter (Type 2250, Brüel & Kjaer, Nærum, Denmark) positioned at the expected ear-height of the participants on the empty chairs of the experimental setup.

After the HI participant was fitted with a pair of hearing aids, the two conversational partners were instructed about the DiapixDK task and were given one training run where a single DiapixDK picture pair was solved under the supervision of the test leader. The participants solved two DiapixDK tasks for each condition, separated into two experimental blocks with a pause in between. The conditions were counterbalanced across blocks and pairs, while the DiapixDK images were counterbalanced across conditions and pairs. At the end of the second block, the participants were asked to evaluate their experience in a free conversation performed in quiet. Data from this condition will not be presented here. Before beginning each block, a calibration signal was recorded for each participant (for details see section on Sound Recordings and Analysis).

Inspired by Picou & Ricketts (2018), the participants were asked to answer four questions after finishing each Diapix task, relating to the communication during the task-solving. The questions were, in the order of presentation (translated from Danish): 1) How successful do you think the conversation was? 2) If this situation occurred in your everyday life, how likely would it be that you would try and improve the situation (e.g., by moving to a different room, ask your partner to speak louder)? 3) How effortful was it to speak? 4) How effortful was it to listen? All questions were answered by marking (pen and paper) a 0-10 visual-analog scale ranging from “Not at all” to “A lot”.

Hearing-aid Fitting

The HI participants were fitted with Signia Pure 312x RIC devices with a closed standard dome (click-sleeve), with the gain determined by the NAL-NL2 rationale (Keidser, Dillon, Carter, & O’Brien, 2012). Closed fittings were chosen with the expectation that applying directionality to a closed fitting would yield a greater effect. Unfortunately, closed-fittings cause occlusion which can result in an altered own-voice perception. For this reason, the Own-Voice Processing technology in the Signia hearing aids was enabled (Powers, Froehlich, Branda, & Weber, 2018). The Own-Voice Processing feature was trained individually for each HI participant during the hearing-aid fitting procedure to detect the wearer's own voice based on spectral content and the direction of arrival, making it robust to changes in the speech, e.g., resulting from Lombard effects. Upon detecting the wearer's own voice during the experiment, the gain provided by the hearing aid was reduced to obtain a more natural perception of their own voice (Høydal, 2017).

The hearing aids were equipped with a static omni-directional program, without any digital noise reduction, speech enhancement features, transient noise reduction, or directional pattern approximating the pinnae effect. As various hearing-aid parameters were logged throughout the experiment (data not presented here), a designated program was made for the unaided condition where the gain in all frequency bands was reduced as much as possible (from 0 dB for low frequencies up to 8 dB for high frequencies). In the unaided experimental condition, the test leader changed to the near-0 dB-gain program and removed the receivers from the HI participants ears but left the hearing aids in the same position on the ears.

Sound Recordings and Analysis

Each participant was equipped with a cheek-mounted directional microphone (DPA 4088. DPA Microphones, Allerød, Danmark) connected to a Mackie 402 VLZ4 pre-amplifier (Mackie, WA, USA) and an ECHO Audiofire12 soundcard (ECHO Digital Audio, CA, USA) through a Neutrik patchbay (NYS-SPP-L 48, Neutrik, Lichtenstein). All sounds were recorded at a sampling frequency of 44.1 kHz, using Matlab 2018a, and saved in wav-format for offline processing.

To estimate the speech level of each participant, a calibration measurement was performed at the beginning of each experiment block. In turn, each participant was seated in their chair and asked to introduce themselves, while a 5-s speech signal was recorded from the cheek-mounted microphone and an omni-directional reference microphone (B-5, Behringer, Willich, Germany) placed on the empty chair where the conversational partner would sit during the experiment at the expected height of the partner's ears. With these two signals, the attenuation, in RMS, from the cheek-mounted microphone to the microphone placed at the chair of the conversational partner could be calculated. To convert this into dB SPL (unweighted), a calibration signal recorded prior to the experimental visit was used. The calibration signal was recorded from the reference microphone placed on the empty chair near a sound-level meter (Type 2250, Brüel & Kjaer, Virum, Denmark). A white noise signal was then presented from one of the two loudspeakers at a level of 75 dB SPL, confirmed by the sound-level meter, and recorded by the reference microphone. For each experimental condition, the speech level at the position of the conversational partner could then be estimated by subtracting the attenuation from the cheek to the reference microphone and converting it into dB SPL by normalizing by the RMS level to that of the calibration signal.

Offline, power-based Voice Activity Detection (VAD) was performed on each cheek-microphone recording to identify and categorize individual utterances, following the approach in Heldner and Edlund (2010) and Sørensen et al. (2021). In each 5-ms window, with 1-ms overlap, the segment was labeled as containing speech if it was above an individually set power threshold. Speech intervals with gaps shorter than 180 ms were merged, and intervals shorter than 90 ms were removed to avoid categorizing transient sound bursts as speech. The SNR between the speech identified by the VAD and the non-speech segments, containing silence and speech from the conversational partner, was on average 27.8 dB for the quiet conditions and 28.4 dB when noise was presented. A t-test revealed no significant effect of the presence of background noise on the SNR of the microphone recordings [t(174) = -1.2, p = .3].

The resulting detected utterances from each talker of each pair and condition were fed to a communicative state classification algorithm (Sørensen et al., 2021), labeling utterances into the following categories: Overlaps-between (acoustic overlaps between the turns of interlocutors during a floor transfer), gaps (acoustic gaps between the turns of interlocutors during a floor transfer), pauses (pauses in one person's speech stream that did not result in a floor-transfer), overlaps-within (stretches of speech that occurred completely within the other interlocutor's turn), and IPUs (stretches of continuous speech surrounded by 180 ms of silence, not including overlaps-within). Together, overlaps-between and gaps make up the FTOs, where negative FTOs are overlaps-between, and positive FTOs are gaps. Further, we calculated the speaking time of both participants in the conversation, as well as the speech level of each participant, defined as the RMS of all utterance scaled by the calibration offset. Finally, a Praat script was used for detecting syllables with a silence threshold of −25 dB, a minimum dip between peaks of 2 dB, and a minimum pause duration of 180 ms (De Jong & Wempe, 2009). In a software interface between Praat and MATLAB (Bořil & Skarnitzl, 2016), we extracted the detected syllables and normalized by the duration of the utterances to compute the articulation rate for each person in each condition and block.

Statistical Analysis

The lme4 package for R (Bates et al., 2015) was used to build mixed-effects regression models for each of the variables of interest. Unless otherwise stated, the starting model, before model-reduction, included fixed effects of amplification (HI unaided, HI aided), background (quiet, noise), hearing (normal, impaired), and block (1, 2) with up to second-order interactions, as well as a random intercept of pair and person varying within pair, i.e.: x ∼ amplification + background + hearing + block + amplification:background + amplification:hearing + amplification:block + background:hearing + background:block + hearing:block + (1|pair/person). The backward stepwise elimination of non-significant factors (step function in the lmerTest package, Kuznetsova, Brockhoff, & Christensen, 2017) was used to reduce the models with an alpha level set to 0.1 to avoid stepping out borderline significant factors. The anova function from the stats package and residuals plots was used to compare models before and after reduction to ensure that the one with the better fit was selected. Q-Q plots and the Anderson-Darling test for normality were used to confirm that the residuals of the final model were normally distributed, which was the case for all variables. Post-hoc analyses were done using the ls_means function from the lmerTest package, computing pairwise differences of least-squares means using a Satterthwaite method for estimating the degrees of freedom (Satterthwaite, 1946).

For each of the 12 outcome measures listed in the Introduction, the group-level statistical test described above was conducted. P-values ≤ 0.05 were considered statistically significant. No adjustments for multiple comparisons were made, but the Althouse guidelines were followed by reporting each statistical test, p-values, regression coefficients, and 95% confidence intervals (Althouse, 2016).

Results

This section presents the statistical effects of the four fixed-effects factors (background noise, hearing-aid amplification, hearing status, and experimental block) and their interactions on the eight objective, and four subjective measures of the conversation. For information on the factors included into the final models and the corresponding test statistics, regression coefficient, and confidence intervals, please see Table 1. Statistically significant effects of experimental block will be described, but not visualized. For the visualized results, lines and asterisks will indicate the statistical significance levels (*** p < .001, ** p < .01, * p < .5).

Table 1.

Statistical Results. For Each Outcome Measure, the Stepped Model is Indicated in Grey. For Each of the Fixed Main and Interaction Effects the Statistical Test-Values, Corresponding p-Values, Regression Coefficients, and 95% Confidence Intervals are Indicated. Statistically Significant Factors are Indicated in Bold Writing.

Final stepped model
Fixed effect	F-statistics	p-value	Regression coefficient	95% confidence interval
FTO median ∼ background + hearing + amplification + hearing:amplification + (1\|person:pair)
background	F(1,151) = 45.2	< 0.001	69.4	[49.3 : 89.4]
hearing	F(1,20) = 0.1	0.8	10.5	[-68.7 : 89.6]
amplification	F(1,151) = 0	1.0	20.3	[-8.2 : 48.8]
hearing: amplification	F(1,151) = 3.8	0.05	-40.6	[-80.9 : -0.4]
FTO IQR ∼ background + hearing + amplification + block + hearing:background + hearing:amplification + background:block + (1\|person:pair)
background	F(1,148) = 59.4	< 0.001	118.0	[59.9 : 176.1]
hearing	F(1,20) = 13.1	0.002	125.5	[35.9 : 215.2]
amplification	F(1,148) = 1	0.3	16.6	[-30.9 : 64.2]
block	F(1,148) = 0.1	0.7	27.2	[-20.2 : 74.6]
hearing:background	F(1,148) = 8.4	0.005	101.1	[34.0 : 168.2]
hearing:amplification	F(1,148) = 3.7	0.06	-66.6	[-133.7 : 0.6]
background:block	F(1,148) = 4.0	0.05	-69.3	[-136.5 : -2.1]
FT rate ∼ background + block + background:block + (1\|pair)
background	F(1,162) = 45.5	< 0.001	-2.9	[-3.7 : -2.0]
block	F(1,162) = 5.4	0.02	-1.5	[-2.4 : -0.7]
background:block	F(1,162) = 9.7	0.002	1.8	[0.6 : 3.0]
Speech level ∼ background + hearing + amplification + hearing:background + amplification:background + (1\|person:pair)
background	F(1,149) = 2310	< 0.001	13.1	[12.3 : 14.0]
hearing	F(1,20) = 0.1	0.7	1.1	[-1.3 : 3.2]
amplification	F(1,149) = 3.8	0.05	-1.1	[-1.8 : -0.4]
hearing:background	F(1,149) = 31.2	< 0.001	-2.8	[-3.8 : -1.8]
amplification:background	F(1,149) = 5.4	0.02	1.1	[0.2 : 2.1]
Articulation rate ∼ background + hearing + amplification + block + hearing:amplification + background:block + (1\|person:pair)
Background	F(1,149) = 11.8	< 0.001	-0.1	[-0.28 : -0.12]
hearing	F(1,20) = 0.6	0.4	0.02	[-0.2 : 0.2]
amplification	F(1,149) = 2.2	0.1	-0.02	[-0.1 : 0.1]
block	F(1,149) = 0.1	0.7	-0.1	[-0.2 : -0.01]
hearing:amplification	F(1,149) = 3.8	0.05	0.1	[0.001 : 0.3]
background:block	F(1,149) = 7.9	0.006	0.1	[0.06 : 0.2]
IPU median ∼ background + hearing + amplification + block + hearing:amplification + background:block + (1\|person:pair)
background	F(1,149) = 9.5	0.002	90.8	[44.5 : 137.0]
hearing	F(1, 20) = 2.9	0.1	133.8	[19.5 : 248.0]
amplification	F(1,149) = 0.0	1.0	39.6	[-6.7 : 86.0]
block	F(1,149) = 1.2	0.3	57.7	[11.4 : 104.0]
hearing:amplification	F(1,149) = 5.5	0.02	-79.1	[-144.6 : -13.6]
background:block	F(1,149) = 5.2	0.02	-77.4	[-142.9 : -11.8]
Completion time ∼ background + block + (1\|pair) + (1\|Diapix)
background	F(1,148) = 20.5	< 0.001	45.7	[25.9 : 65.5]
block	F(1,148) = 8.0	0.005	-28.4	[-48.1: -8.7]
Speaking time - 50 ∼ background + amplification + amplification:background + (1\|person)
background	F(1,74) = 1	0.3	-0.4	[-3.7 : 3]
amplification	F(1,74) = 3.5.	0.07	-3.9	[-7.2 : -0.5]
amplification:background	F(1,74) = 1.7	0.2	3.3	[-1.5 : 8]
Conversational success ∼ background + (1\|pair/person)
background	F(1,152) = 98.2	< 0.001	-1.9	[-2.4 : -1.6]
Desire to remove ∼ background + hearing + hearing:background + (1\|person:pair)
background	F(1,151) = 657	< 0.001	7.5	[6.7 : 8.1]
hearing	F(1,20) = 0.4	0.5	1.4	[0.01 : 2.6]
hearing:background	F(1,151) = 12.2	< 0.001	-1.7	[-2.8 : -0.7]
Talking effort ∼ background + hearing + (1\|pair)
background	F(1,162) = 421	< 0.001	5.3	[4.9 : 5.8]
hearing	F(1,162) = 16.9	< 0.001	1.1	[0.6 : 1.6]
Listening effort ∼ background + hearing + hearing:background + (1\|person:pair)
background	F(1,151) = 430	< 0.001	4.3	[3.6 : 5]
hearing	F(1,20) = 11.1	0.003	1.6	[0.2 : 2.9]
hearing:background	F(1,151) = 5.5	0.02	1.0	[0.1 : 2.1]
Conversational success ∼ completion time + desire to remove + talking effort + listening effort + 1\|person:pair
completion time	F(1,159) = 9.6	0.002	-0.004	[-0.006 : -0.002]
desire to remove	F(1,163) = 4.3	0.04	0.09	[0.005 : 0.2]
talking effort	F(1,163) = 16.1	< 0.001	-0.2	[-0.4 : -0.1]
listening effort	F(1,161) = 9.8	0.003	-0.1	[-0.4 : -0.08]

Floor-Transfer Offsets

From the recorded conversations, the offsets were extracted for each floor transfer. On average, the FTO distributions (Figure 2A) peaked at 230 ms, indicating that participants tend to initiate their turn after a small gap. For each participant, block, and condition, the median and interquartile range (IQR) of the FTO distributions were extracted to evaluate the effects of the experimental contrasts on the timing of turn-taking.

Figure 2.

FTO distributions and medians. A) The distributions of FTOs across blocks for all NH (solid lines) and HI (dotted lines) participants. Positive FTO values indicate an acoustic gap between one talker stopping and the other starting, while a negative FTO indicates an acoustic overlap their turns. B) The median FTO for all conditions, averaged across blocks, for the NH and HI participants. C) Visualization of the significant interaction effect between hearing status and whether the HI participant received amplification on the median FTO, averaged across background and blocks. Here, and in the following plots, the boxes indicate the 25th to 75th percentile and the horizontal lines the median. The whiskers extend the range of the data, and outliers are indicated with dots.

The median FTO was affected by background noise [F(1,151) = 45.2, p < .001, Figure 2B], causing the participants to initiate their turn 69 ms later in noise. A significant interaction between hearing status and hearing-aid amplification [F(1,151) = 3.8, p = .05] indicates that when the HI participant received amplification the median FTO was reduced by 41 ms for the HI, relative to the NH participants [t(70) = -1.9, p = .05, Figure 2C].

The variability of the turn-taking timing (FTO distribution), as measured by the IQR, increased by 118 ms in noise for both groups [F(1,148) = 59.4, p < .001, Figure 3A], and the IQR of the HI participants was 125 ms larger than that of the NH participants [F(1,20) = 13.1, p < .01].

Figure 3.

FTO variability (IQR) and rate of floor transfers. A) The interquartile range (IQR) extracted from the FTO distribution of each participant, see Figure 2A. B) The significant interaction effect between hearing status and background, averaged across amplification and blocks, extracted for visualization. C) The number of floor transfers per minute between interlocutors averaged across block.

A significant interaction between background noise and hearing status [F(1,148) = 8.4, p < .01, Figure 3B] revealed that the NH had an 83-ms increase in variability in noise compared to quiet [t(148) = -3.4, p < .001] whereas it was 184 ms for the HI participants [t(148) = -7.5, p < .001]. The increase in variability with the addition of noise was significantly larger for the HI participants than for the NH listeners [t(83.8) = -2.9, p < .01].

An interaction between experimental block and background noise [F(1,148) = 4.0, p = .05] indicates that the variability increased by 160 ms when adding background noise in the first experimental block [t(148) = -6.8, p < .001], but only by 99 ms when adding noise in the second block [t(148) = -4.1, p < .001].

The rate of floor transfers (FT rate) between speakers per minute, decreased by 2.9 turns per minute in the presence of background noise [F(1,162) = 45.5, p < .001, Figure 3C], and by 1.5 turns per minute between the first and second experimental block [F(1,162) = 5.4, p = .02]. Further, there was an interaction effect between background noise and experimental block [F(1,162) = 9.7, p < .01], indicating that the presence of noise caused interlocutors to decrease their FT rate by 2.9 per minute in the first experimental block [t(162) = 6.9, p < .05], but in the second experimental block the reduction was only 1.1 per minute [t(162) = 2.5, p < .05].

Speech Level, Articulation Rate, and IPU Duration

The outcome measures associated with the participants’ speech pattern are presented in Figure 4 and Figure 5.

Figure 4.

Estimated speech levels. A) Estimated speech levels at the average position of their conversational partner in dB SPL, averaged across blocks. B) Visualization of the main effect of amplification, averaged across all participants, background noise, and blocks. C) Visualization of the significant interaction effect between hearing and background on the speech level, averaged across amplification and blocks. D) Visualization of the significant interaction effect between amplification and background on the speech level, averaged across NH and HI participants and blocks.

Figure 5.

Articulation rates and length of inter-pausal units (IPUs). A) The articulation rate measured as the number of syllables per second, averaged across blocks. B) Visualization of the significant interaction effect between hearing and amplification on the articulation rate, averaged across background and blocks. C) Median inter-pausal unit (IPU) in milliseconds averaged across blocks. D) Visualization of the significant interaction effect between hearing and amplification on the IPUs, averaged across background and blocks.

The speech levels estimated at the interlocutor's position showed that in noise, the talkers increased their speech levels by 13.1 dB [F(1,149) = 2310, p < .001, Figure 4A], such that they communicated at an average SNR of 0.03 dB (sd = 2.9 dB) in noise. When the HI participants were aided, the speech levels were 1.1 dB lower for both HI and NH participants [F(1,149) = 3.8, p = .05, Figure 4B], compared to when no amplification was provided to the HI participant.

A significant interaction between hearing status and background noise [F(1,149) = 31.2, p < .001, Figure 4C], revealed that when adding background noise, the NH participants, on average, raised their speech levels by 13.6 dB [t(149) = 18.5, p < .001], whereas the HI participants only raised their speech levels by 10.8 dB [t(149) = -29.9, p < .001]. This increase in speech level was significantly higher for the NH, than for the HI participants [t(59.4) = 5.4, p <.001].

The background noise also affected the speech levels in interaction with amplification [F(1,149) = 5.4, p = .02, Figure 4D], with the post-hoc analysis revealing that when the HI participant was unaided, both the NH and HI interlocutor spoke 1.1 dB louder in quiet [t(149) = 3.0, p < .01], whereas the hearing-aid amplification did not affect the speech levels produced in noise [0.09 dB, t(149) = -.26, p = .8].

The articulation rate decreased by 0.1 syll/s in the presence of background noise, [F(1,149) = 11.8, p < .001, Figure 5A]. However, an interaction effect between background noise and experimental block [F(1,149) = 7.9, p < .01] revealed that while participants spoke 0.2 syll/s slower in noise compared to quiet in block 1 [t(149) = 4.4, p < .001], no significant effect of noise was observed in the second block [0.02 syll/s, t(149) = .5, p = .7]. This was caused by both interlocutors speaking 0.1 syll/s slower in quiet in the second block [t(149) = 2.2, p < .05], while articulation rates were similar in noise between blocks [0.07 syll/s, t(149) = 2.2, p = .08].

The articulation rate was also affected by a significant interaction between hearing status and amplification [F(1,149) = 3.8, p = .05, Figure 5B] indicating that the HI participants spoke 0.1 syll/s faster when aided [t(149) = -2.4, p < .05], while the articulation rates of the NH participants were not affected by the HI being aided [0.01 syll/s, t(149) = .3, p = .8].

The median IPU duration, i.e., the median lengths of connected speech surrounded by 180 ms of silence, is seen in Figure 5C. Background noise caused participants to increase their IPU durations by 90 ms [F(1,149) = 9.5, p < .01]. An interaction with experimental block [F(1,149) = 5.2, p = .02] revealed that while there was no difference in the median IPU durations in noise between blocks [19 ms, t(149) = .8, p = .4], participants held their turn for 58 ms longer in quiet in block 2 compared to block 1 [t(149) = -2.4, p = .02].

An interaction between hearing and amplification [F(1,149) = 5.5, p = .02, see Figure 5D] revealed that when unaided, the HI participants talked for 134 ms longer than their NH conversational partners [t(23) = -2.3, p = .04], while there was no difference between the two talkers when the HI participant was aided [54 ms, t(23) = -.9, p = .4].

Task Completion and Speaking Time

The time it took the pairs to identify 10 differences between the Diapix can be used as an indicator of the efficiency of the conversation (Baker & Hazan, 2011). For time considerations, the pairs were stopped after 10 min if they had not yet found 10 differences, which occurred three times (3.4% of the 88 trials, two in quiet and one in noise). These completion times were not included in the statistical analysis.

The task completion time increased by an average of 45 s for conditions with background noise [F(1,148) = 20.5, p < .001, Figure 6A]. A significant learning effect was observed between the two blocks [F(148) = 8.0, p < .01], such that the task was solved on average 28.4 s faster in the second block.

Figure 6.

A) Task completion time and B) percentage speaking time between the two interlocutors of each pair. A completely balanced partaking in the conversation of 50% is indicated by the dotted line.

From the detected utterances, the proportion of speaking time of each participant was calculated, see Figure 6B. As the sum of the proportion of speaking time is always 100%, or a little above if speech overlaps occur, the statistical analysis was performed on data from the HI participants only. None of the included fixed effects significantly affected the speaking time.

Subjective Ratings of the Conversations

The subjective ratings made after each DiapixDK task, Figure 7, were all affected by the presence of background noise, resulting in: 1) The conversational success being rated 1.9 points lower [F(1,152) = 98, p < .001], 2) the desire to improve the situation being rated 7.5 points higher [F(1,151) = 657, p < .001], 3) the talking effort being rated 5.3 points higher [F(1,162) = 421, p < .001], and 4) the listening effort being rated 4.3 points higher [F(1,151) = 430, p < .001].

Figure 7.

Subjective ratings of the conversation. On a scale from 0–10, each subject independently rated the conversational success (upper left), the desire to improve the situation (upper right), as well as the effort related to talking (lower left) and listening (lower right).

The HI participants generally rated their talking [1.1 points, F(1,162) = 16.9, p < .001] and listening effort [1.6 points, F(1,20) = 11.1, p < .01] higher than their NH partner. Furthermore, an interaction between hearing status and background noise [F(1,151) = 5.5, p = .02], reveals that the difference in listening-effort ratings between HI and NH participants in quiet were 1.6 points [t(25.5) = -2.4, p < .05], while it was 2.7 points in noise [t(25.5) = -4.0, p < .001]. A similar interaction between background noise and hearing status [F(1,151) = 12.2, p < .001], revealed that while the 1.3-point difference in the desire to improve the situation between NH and HI participants approached statistical significance in the quiet situation [t(27.3) = -2.0, p = .06], no differences were observed between the participants in noise [0.4 points, t(27.5) = .7, p = .5].

It is interesting to observe that hearing status did not affect the ratings of conversational success, Figure 7 upper left, despite the HI participants indicating higher talking effort, listening effort, and desire to improve the situation in quiet. One common criterion on which conversational success could be judged is the task completion time. To investigate this and whether the rating of conversational success was linked to any of the other subjective ratings, a model predicting the conversational success from the three other subjective ratings and completion time was built (no interaction effects included). The result showed that a higher rating of conversational success was related to shorter task completion time [F(1,159) = 9.6, p < .01], and lower desire to improve the situation [F(1,163) = 4.3, p = .04], but also to increased perceived talking [F(1,163) = 16.1, p < .001] and listening effort [F(1,161) = 9.8, p < .01].

Discussion

By exploring the dynamics of a conversation between a younger NH and an older HI talker, the goal of the current study was to investigate the effects of the three factors (background noise, hearing status, and hearing-aid amplification provided to the HI participants) on twelve outcome measures quantifying the conversational dynamics. In the following, the effect of each factor will be discussed separately, together with the observed effects of experimental block. Finally, a cross-study comparison will be provided to address the potential differences of being seated face-to-face compared to sitting in different rooms.

Background Noise Impacts Multiple Aspects of the Conversations

As hypothesized, the presence of background noise caused alterations in the communication, as evidenced by longer, more variable, and fewer turn-takings (FTOs, Figure 2 - Figure 3), as well as louder speech (Figure 4A), slower articulation (Figure 5A), longer IPUs (Figure 5C), and longer task completion times (Figure 6A). In the presence of noise, the participants rated the conversations as being less successful, they had an increased desire to improve the situation, and they experienced higher talking and listening effort (Figure 7).

Clear signs for Lombard speech (Lombard, 1911) were found when participants were communicating in the presence of background noise, evident from the increased speech levels and reduced articulation rate. With babble noise at a level of 70 dB SPL, participants raised their speech levels to communicate at 0.03 dB SNR (NH spoke at 1.1 dB and HI at −0.7 dB SNR, Figure 4C). This SNR level is very close to the level of 1 dB SNR found in a previous study with similar experimental setup and participants, but with interlocutors seated in two different rooms (Sørensen (2021b). Considering that providing visual information improves the speech understanding significantly (Remez, 2012; Sumby & Pollack, 1954), it might have been expected that participants in the current study would speak at lower SNRs when seated face-to-face in order to reduce the talking effort. However, it should be noted that the speech levels were calibrated to the expected average position of the interlocutor (see Sound Recordings and Analysis), which does not account for potential alterations occurring if participants leaned forward/backward and/or turned their better ear towards their interlocutor. Recent studies describe, however, how interlocutors seated 1.5 m apart, only decreased their distance by a maximum of 10 cm in noise, resulting in a negligible SNR improvement of less than 1 dB (Hadley, Brimijoin, & Whitmer, 2019). Listeners also tend to turn their heads to favor their better ear (maximize speech level), even if it does not necessarily maximize the SNR (Brimijoin, McShefferty, & Akeroyd, 2012). Although these movements only result in minor changes in SNR, they might signal to the interlocutor to speak up, indirectly improving the SNR. Whether leaning in or turning the better ear was done in the current study, and whether this indirectly affected the SNR, is unknown.

Despite participants in the current study adapting their speech level and articulation rate to improve the communication situation, adding background noise resulted in later (larger FTO median, Figure 2) and more variable (larger FTO IQR, Figure 3A) turn-starts. As highlighted in the introduction, achieving rapid turn-starts, around 230 ms, requires correctly predicting the end of the talking interlocutor's turn, while at the same time preparing a verbal response (Corps et al., 2018), both of which rely on being able to hear and interpret the incoming speech. In suboptimal listening conditions, the model for Ease of Language Understanding (ELU) describes how explicit processing is activated, involving inference-making and semantic integration, which requires longer processing time before obtaining speech understanding (Rönnberg et al., 2013). This increased internal speech-processing time might explain the increased FTO median observed in noise. The fact that the increased median FTO is accompanied by increased FTO variability suggests that participants make errors in identifying turn-ends, causing both larger overlaps and gaps in the turn-taking in noise.

In this context of turn taking, producing longer utterances could give a conversational partner more time to understand and prepare an adequate response. Indeed, Sørensen (2021a) found that longer IPUs were proceeded by faster and less variable FTOs, while multiple studies have found that interlocutors increased their IPU durations in noise (Beechey et al. (2018); Sørensen et al., 2021; Watson et al. (2020)). Sørensen (2021a) speculated that when the input signal is degraded, additional resources must be spent on planning a response, and the entirety of the response might not have been planned before launching it at the socially appropriate time of around 200 ms. Thus, talkers may continue planning their utterance while producing it, a process that can be helped by inserting filler-words such as “uhm” or “ahh” (Clark & Fox Tree, 2002; Sjerps & Meyer, 2015) at the boundary and/or during the turn to allow for more time to prepare a complete response, thereby lengthening their utterances.

Considering that we observed that participants spoke for longer (larger IPU median) and the timing of their turn-starts were later (larger FTO median), it is not surprising that the FT rate was reduced in noise (Figure 3C). If we were to assume that the same amount of information was conveyed between the conversational partners in noise and quiet, this will consequently lead to the observed increase in completion times (Figure 6B). However, this assumption does not hold, as an exploratory analysis showed that the influence of FT rate on completion time was not significant [p = .09], and the estimated coefficients revealed that a 1% decrease in FT rate only caused a 0.2% increase in the completion time. A similar analysis for articulation rate also showed that speaking slower did not significantly influence the completion time [p = .2]. Hence, it is more likely that the prolonged completion times were caused by miscommunications than by alterations in the conversational dynamics between interlocutors. Indeed, it has been shown how a miscommunication, besides requiring correction/clarification on its own, will cause interlocutors to add more clarifications in the speech following the miscommunication (Hazan & Baker, 2011), which could have potentially increased the task completion time beyond the miscommunication itself.

The various detrimental effects that noise has on the communication efficiency are experienced by the participants, who all subjectively rated the conversational success to be lower, while they rated their desire to improve the situation as well as their talking and listening effort higher (Figure 7). The increase in listening effort is in agreement with a previous study that observed a similar effect of adding background noise (Tuomainen et al., 2019).

Hearing-Impaired Participants Experience Increased Communication Effort, Especially in Noise

The communication behavior differed significantly between the two conversational partners based on their hearing status and these differences were most prominent in the presence of background noise. The most noticeable difference between the participants was observed in the change of speech level between noise and quiet (Figure 4C) where the NH participants increased their speech level by 2.9 dB more than the HI participants. This finding confirms previous observations that NH talkers speak louder than HI participants (Sørensen, 2021a), indicating that they must make up for the communication difficulty experienced by the HI participants in the presence of noise. Interestingly, the NH participants do not seem to notice this behavior, as their subjective ratings of talking effort is significantly lower than that of their HI interlocutors (Figure 7 lower left). The rating of talking effort was meant to address the vocal strain experienced by the talkers, but the fact the HI participants rated experiencing higher talking effort could potentially be explained by them feeling that the preparation of their upcoming verbal response was more effortful.

Hearing impairment also affected the ability to consistently time turn-starts, as the FTO variability was 142 ms higher for the HI participants than the NH participants (Figure 3A). This is consistent with Sørensen (2021a) finding that the FTO IQR, but not median, was increased for HI interlocutors. The observed increased variability in the FTO (IQR) suggests that the HI participants’ ability to predict the end-of-turn timing based on acoustic cues of the speech such as pitch increases, final-word lengthening, speech level changes, and voice quality (Bögels & Torreira, 2015; Brusco et al., 2020; De Ruiter et al., 2006; Gravano & Hirschberg, 2011) is reduced. This ability is further impaired for the HI participants in the presence of background noise (Figure 3B), highlighting how detrimental background noise is to communication when hearing is impaired.

The HI participants indicated experiencing increased communication difficulty, desire to improve the quiet situations, and talking and listening effort. In background noise, HI participants reported an additional increase in listening effort compared to their NH conversational partners. In an exploratory analysis, we investigated whether the listening and talking effort ratings were affected by the speech level produced by the conversational partner and the talkers themselves, respectively, but no statistical relationship was found (not shown in Results). Hence, we cannot know what internal criteria drive the subjective ratings of listening and talking effort, or rule out that the ratings partly reflect participants’ experience of chronic stress or fatigue (McGarrigle et al., 2014; Pichora-Fuller et al., 2016), which are known to be elevated in HI listeners (Hornsby, Naylor, & Bess, 2016; Nachtegaal et al., 2009).

The participants were also asked to rate the conversational success to investigate if they subjectively experienced alterations in the communication (Figure 7). Compared to the NH participants, the HI participants reported experiencing higher talking and listening effort, as well as a higher desire to want to improve the situation. Surprisingly no effect of hearing status was found on the rating of conversational success. The explanation for this is most likely that the task completion time served as a common internal criterion by which the conversational success was judged. We found that the ratings of conversational success were also significantly related to the ratings of the desire to improve the situation and the talking and listening effort. However, we must conclude that conversational success is not a well-understood concept; indeed, in the scientific literature, conversational success has been evaluated as the lack of conversational breakdown (Beechey et al., 2020b; McInerney & Walden, 2013), but also as depending on the conversational topic, social contact, fluency, and other factors (Lind, 2012). As such, it is potentially more beneficial to ask participants to rate more well-defined subsets of a conversation, rather than the overarching success of it.

Hearing-Aid Amplification Affects the Conversation

Providing amplification resulted in reduced speech levels of both talkers (Figure 4B), especially in quiet (Figure 4D), while the HI participants had faster turn-starts (larger median FTO, Figure 2C), increased their articulation rate (Figure 5B), and reduced the IPU duration to become more similar to the NH participants (Figure 5D).

Amplification made the HI participants 41 ms faster in initiating their turn (FTO median). Indeed if, as argued previously, the timing of turn-starts relies on cognitive processing of the speech input, hearing-aid amplification seemed to improve the HI participants’ ability to consistently predict the end-of-turn (FTO median). This is in contrast to standardized listening tests, where social pressure to formulate a well-times response may be substantially reduced or absent.

The increased speech levels observed for both participant groups when the HI participants were unaided, suggests that the loss of hearing causes the HI participants to speak louder either to get adequate auditory feedback (Junqua, 1996) and/or to signal difficulty understanding their interlocutor (Beechey et al. 2018). This leaves it up to the NH talker to compensate for the HI persons’ lost audibility by also speaking louder. Amplification, especially in quiet, makes up for the lost audibility of the HI participant and improves the speech intelligibility, causing a reduction in the speech level of the HI participants and of their NH interlocutor. In a previous study by Beechey and colleagues, NH talkers were found to speak approximately 1.5 dB lower in both 69.8 dBA church noise and 67.3 dBA living-room noise, when the HI interlocutor was aided compared to unaided (Beechey, Buchholz, & Keidser, 2020a). In contrast to Beechey et al. (2020a), the effect of providing amplification to the HI participant was primarily driven by the changes observed when communicating in quiet (Figure 4D).

The phenomenon ampclusion (Painton, 1993) describes how the hearing-aid wearers’ perception of their own voice is changed, not only by the amplification, but also by altered auditory feedback arising from closed-dome hearing aids occluding the ear canal. A recent study found that experienced hearing-aid users rated their voice as being more dominating after receiving new hearing aids, even more so than a group of participants receiving their first hearing aids (Hengen, Hammarström, & Stenfelt, 2020). In the current study, the hearing aids were programmed with active occlusion cancellation, which reduces the gain upon detecting the wearer's own voice, hence trying to compensate for the ampclusion and occlusion effects experienced while talking. It is, however, possible that the active occlusion cancellation causes a smaller decrease in speech level between unaided and aided, than would otherwise be expected without the feature.

When unaided, the HI participants had significantly longer median IPUs (Figure 5C), confirming the previous observation by Sørensen (2021a) that (unaided) HI participants had longer IPUs compared to their NH interlocutor. In a similar manner, an increase in articulation rate was also observed when HI participants received amplification. The observations of the current study suggest that providing the HI participants with amplification can reduce the communication effort that they experience because of their hearing impairment.

It is unusual to find significant effects of hearing-aid amplification on speech understanding in quiet for listeners with mild to moderate hearing impairment, because the ability to repeat sentences/words presented above hearing-threshold levels is close to perfect. To indirectly quantify the benefit of hearing-aid processing in close-to-ideal listening situations, secondary cognitive effects, such as word recall (Kuk, Slugocki, Ruperto, & Korhonen, 2021; Lunner, Rudner, Rosenbom, Agren, & Ng, 2016), or changes in pupil size (Wendt, Hietkamp, & Lunner, 2017), have been investigated. In this study, we were able to show the direct effects of amplification on the HI interlocutor wearing the hearing aid, as well as indirect effects on their NH partner when communicating in quiet.

Participants Adapt Their Conversation Over Time

The experimental DiapixDK task was used to ensure a flowing conversation between the two previously unacquainted participants. In line with previous studies (Sørensen et al., 2021a), we also observed a significant learning effect over time, resulting in the task being solved 30 s faster in the second experimental block. In the development of the DiapixUK task, no significant improvements in the task completion time were observed after having completed a single initial training trial (Baker & Hazan, 2011). However, as Baker and Hazan tested the Diapix task on previously acquainted pairs of NH participants, it is possible that the effects of block observed in the current study are caused by a familiarization between the interlocutors, rather than with the task itself.

The articulation rate, FT rate, and FTO variability were more affected by the addition of background noise in the first experimental block, compared to the second. The effects were caused by general lowering of all three measures in the quiet conditions of the second experimental block. Although this observation cannot be easily explained, it might suggest that the interlocutors are more familiar with each other's speaking patterns and can time their turn-taking better (lower FTO IQR) and require fewer FTs (lower FT rate) to solve the DiapixDK task when the communication is not challenged by background noise.

Insights into the Relative Importance of Face-To-Face Communication

The effect of sitting face-to-face (audio-visual communication), compared to being seated in two different rooms (auditory communication over headphones, denoted remote communication), can be evaluated by comparing the unaided condition of the current study with the unaided communication in the study by Sørensen (2021a). The two studies used the same DiapixDK task, background noise, and inclusion criteria for the NH and HI participants, making a cross-study comparison possible.

In comparing results, the FTO median, FTO IQR, FTO rate, articulation rate, and IPU duration were all less affected by noise in face-to-face communication, compared to remote communication. This is in line with previous studies observing that visual information improves communication, including the timing of turn-taking, through the facilitation of body language including hand gestures (ter Bekke, Drijvers, & Holler, 2020), eye contact (Kiessling et al., 2003), and lip movement (Fitzpatrick, Kim, & Davis, 2015). The FT rate decreased more in noise in remote, compared to face-to-face communication, potentially driven by the fact that in noise the HI participants increased their IPUs length by 18% in remote communication, but only 4% in face-to-face communication.

This increased length of turn-holding for the HI participants in noise for remote communication influenced the percentage of speaking time, with a greater contribution from the HI (57%) in remote communication compared to NH participants. Although the proportion of speaking time is not a measure of communication success, it is believed that an equal distribution thereof indicates a more cooperative conversation (Beechey, Buchholz, & Keidser, 2019). Persons with HI are known to sometimes adopt the face-saving strategy of dominating a conversation in order to avoid listening (Jaworski & Stephens, 1998; Stephens & Zhao, 1996). In the current study, HI participants did not dominate the conversation (Figure 6B). An opposite strategy the HI talker may adopt in challenging situations is to withdraw from conversation (Jaworski & Stephens, 1998). This was not observed in the current study, likely because the DiapixDK task forced both interlocutors to contribute to the conversation.

Potential Confounding Effect of Age

Several effects of hearing status on the conversational dynamics were observed in the present study. However, the study's design does not allow us to disentangle the potential effects of age from those of hearing status. Studies have previously observed that older persons slow down their speech (articulation rate), produce longer sentences (IPUs), make longer pauses (median FTO), and are more affected by increases in task complexity than younger participants (Hazan et al., 2018; Mortensen, Meyer, & Humphreys, 2006). Although the studies mentioned in the review by Mortensen et al. (2006) do not compensate for potential changes in hearing status with age, their findings could suggest that effects of hearing status observed in the current study might be driven by the age difference between the NH and HI groups. In a recent study investigating communication between NH and HI interlocutors, no significant effect of the age of the HI participants was found on their own, or their NH interlocutors’ speech levels and changes in the formant frequencies (Beechey et al., 2020a; age range 53–85 years).

If the differences between the HI and NH participants of the current study are driven by age, rather than hearing status, the review by Mortensen et al. (2006) predicts that we would find the older (HI) participants to have slower articulation rates, longer IPUs, and larger median FTOs. However, in the current study, none of these measures were affected directly by hearing status (equivalent to age). On the contrary, all measures showed significant, or very near-significant, interactions between hearing status and amplification, indicating that the differences between groups were altered by hearing-aid amplification. These interactions indicate that effects observed in the current study are not solely driven by age but are affected by hearing loss.

Conclusion

The current study explored the conversational dynamics between young NH and older HI interlocutors and found that noise increased communication difficulty, especially for HI participants. Providing the HI interlocutors with amplification, through a hearing aid with active occlusion cancellation, caused them to behave more like their NH conversational partners. This was especially evident in quiet where the amplification ensured audibility. These results indicate that HI listeners benefit from hearing-aid amplification in quiet, even if they do not often report listening difficulties in these situations. The method seems promising for evaluating the benefits of hearing-aid amplification in more realistic and interactive communication scenarios.

Footnotes

Author Contributions

EBP, ENM, and AJMS all contributed to the scoping and designing of the study. EBP conducted the data collection. AJMS and EBP performed the data processing and statistical analysis. EBP, ENM, and AJMS did the interpretation of the results. EBP and AJMS prepared the manuscript.

Declaration of Conflicting Interests

The authors declared the following potential conflicts of interest concerning the research, authorship, and/or publication of this article: Eline Borch Petersen is employed at WS Audiology, Lynge, Denmark. No conflicts were declared for the remaining authors.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Eline Borch Petersen

Ewen N. MacDonald

A. Josefine Munch Sørensen

References

Althouse

A. D.

(2016). Adjust for multiple comparisons? It's Not that simple. Annals of Thoracic Surgery, 101(5), 1644–1645. https://doi.org/10.1016/j.athoracsur.2015.11.024

Aubanel

Cooke

Villegas

Garcia Lecumberri

M. L.

(2011). Conversing in the presence of a competing conversation: Effects on speech production. InTwelfth Annual Conference of the International Speech Communication Association., INTERSPEECH, (May 2014), 2833–2836.

Baker

Hazan

(2011). DiapixUK: Task materials for the elicitation of multiple spontaneous speech dialogs. Behavior Research Methods, 43(3), 761–770. https://doi.org/10.3758/s13428-011-0075-y

Barthel

Sauppe

(2019). Speech planning at turn transitions in dialog is associated with increased processing load. Cognitive Science, 43(7), 1–16. https://doi.org/10.1111/cogs.12768

Barthel

Sauppe

Levinson

S. C.

Meyer

A. S.

(2016). The timing of utterance planning in task-oriented dialogue: Evidence from a novel list-completion paradigm. Frontiers in Psychology, 7(DEC), 1–13. https://doi.org/10.3389/fpsyg.2016.01858

Bates

Mächler

Bolker

B. M.

Walker

S. C.

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01

Beechey

Buchholz

J. M.

Keidser

(2018). Measuring communication difficulty through effortful speech production during conversation. Speech Communication, 100(April), 18–29. https://doi.org/10.1016/j.specom.2018.04.007

Beechey

Buchholz

J. M.

Keidser

(2019). Eliciting naturalistic conversations: A method for assessing communication ability, subjective experience, and the impacts of noise and hearing impairment. Journal of Speech, Language, and Hearing Research, 62(2), 470–484. https://doi.org/10.1044/2018_JSLHR-H-18-0107

Beechey

Buchholz

J. M.

Keidser

(2020a). Hearing aid amplification reduces communication effort of people with hearing impairment and their conversation partners. Journal of Speech, Language, and Hearing Research, 63(4), 1299–1311. https://doi.org/10.1044/2020_JSLHR-19-00350

10.

Beechey

Buchholz

J. M.

Keidser

(2020b). Hearing impairment increases communication effort during conversations in noise. Journal of Speech, Language, and Hearing Research, 63(i), 1–24. https://doi.org/10.1044/2019_JSLHR-19-00201

11.

Bisgaard

Vlaming

M. S. M. G.

Dahlquist

(2010). Standard audiograms for the IEC 60118-15 measurement procedure. Trends in Amplification, 14(2), 113–120. https://doi.org/10.1177/1084713810379609

12.

Bögels

Magyari

Levinson

S. C.

(2015). Neural signatures of response planning occur midway through an incoming question in conversation. Scientific Reports, 5(1), 1–11. https://doi.org/10.1038/srep12881

13.

Bögels

Torreira

(2015). Listeners use intonational phrase boundaries to project turn ends in spoken interaction. Journal of Phonetics, 52, 46–57. https://doi.org/10.1016/j.wocn.2015.04.004

14.

Bořil

Skarnitzl

(2016). Tools rPraat and mPraat. In Text, speech, and dialogue (pp. 367–374). Springer International Publishing.

15.

Brady

P. T.

(1968). A statistical analysis of on-off patterns in 16 conversations. Bell System Technical Journal, 47(1), 73–91. https://doi.org/10.1002/j.1538-7305.1968.tb00031.x

16.

Brimijoin

W. O.

McShefferty

Akeroyd

M. A.

(2012). Undirected head movements of listeners with asymmetrical hearing impairment during a speech-in-noise task. Hearing Research, 283(1–2), 162–168. https://doi.org/10.1016/j.heares.2011.10.009

17.

Brusco

Vidal

Beňuš

Gravano

(2020). A cross-linguistic analysis of the temporal dynamics of turn-taking cues using machine learning as a descriptive tool. Speech Communication, 125(September), 24–40. https://doi.org/10.1016/j.specom.2020.09.004

18.

Clark

H. H.

Fox Tree

J. E.

(2002). Using uh and um in spontaneous speaking. Cognition, 84(1), 73–111. https://doi.org/10.1016/S0010-0277(02)00017-3

19.

Corps

R. E.

Gambi

Pickering

M. J.

(2018). Coordinating utterances during turn-taking: the role of prediction, response preparation, and articulation. Discourse Processes, 55(2), 230–240. https://doi.org/10.1080/0163853X.2017.1330031

20.

De Jong

N. H.

Wempe

(2009). Praat script to detect syllable nuclei and measure speech rate automatically. Behavior Research Methods, 41(2), 385–390. https://doi.org/10.3758/BRM.41.2.385

21.

De Ruiter

J. P.

Mitterer

Enfield

N. J.

(2006). Projecting the end of a speaker’s turn: A cognitive cornerstone of conversation. Language, 82(3), 515–535. https://doi.org/10.1353/lan.2006.0130

22.

Fitzpatrick

Kim

Davis

(2015). The effect of seeing the interlocutor on auditory and visual speech production in noise. Speech Communication, 74, 37–51. https://doi.org/10.1016/j.specom.2015.08.001

23.

Gisladottir

Levinson

Chwilla

(2015). EEG Data for ‘conversation electrified: ERP correlates of speech act recognition in underspecified utterances’. Plos One, 10(3), 1–24. https://doi.org/10.5281/zenodo.13003.

24.

Gravano

Hirschberg

(2011). Turn-taking cues in task-oriented dialogue. Computer Speech and Language, 25(3), 601–634. https://doi.org/10.1016/j.csl.2010.10.003

25.

Hadley

L. V.

Brimijoin

W. O.

Whitmer

W. M.

(2019). Speech, movement, and gaze behaviours during dyadic conversation in noise. Scientific Reports, 9(1), 1–8. https://doi.org/10.1038/s41598-019-46416-0

26.

Hagoort

Indefrey

(2014). The neurobiology of language beyond single words. Annual Review of Neuroscience, 37, 347–362. https://doi.org/10.1146/annurev-neuro-071013-013847

27.

Hazan

Baker

(2011). Acoustic-phonetic characteristics of speech produced with communicative intent to counter adverse listening conditions. The Journal of the Acoustical Society of America, 130(4), 2139–2152. https://doi.org/10.1121/1.3623753

28.

Hazan

Tuomainen

Kim

Davis

Sciences

Sydney

(2019). The effect of visual cues on speech characteristics of older and younger adults in an interactive task. In Proceedings of the 19th International Congress of the Phonetic Sciences (Vol. 19, pp. 815-819). Australasian Speech Science and Technology Association Inc.

29.

Hazan

Tuomainen

Kim

Davis

Brungart

Sheffield

(2018). How do aging and age-related hearing loss affect the ability to communicate effectively in challenging communicative conditions? Hearing Research, 369, 33–41. https://doi.org/10.1016/j.heares.2018.06.009

30.

Heldner

Edlund

(2010). Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4), 555–568. https://doi.org/10.1016/j.wocn.2010.08.002

31.

Hengen

Hammarström

Stenfelt

(2020). Perception of one’s own voice after hearing-aid fitting for naive hearing-aid users and hearing-aid refitting for experienced hearing-aid users. Trends in Hearing, 24, https://doi.org/10.1177/2331216520932467

32.

Hornsby

Naylor

Bess

(2016). A taxonomy of fatigue concepts and their relation to hearing loss. Ear and Hearing, 37(Suppl 1), 136S–144S. https://doi.org/10.1097/AUD.0000000000000289

33.

Høydal

(2017). A new own voice processing system for optimizing communication. Hearing Review, 24(11), 20–22. Retrieved from https://www.hearingreview.com/practice-building/marketing/new-voice-processing-system-optimizing-communication.

34.

Indefrey

Levelt

(2004). The spatial and temporal signatures of word production components. Cognition, 92(1–2), 101–144. https://doi.org/10.1016/j.cognition.2002.06.001

35.

Jaworski

Stephens

(1998). Self-reports on silence as a face-saving strategy by people with hearing impairment. International Journal of Applied Linguistics (United Kingdom), 8(1), 61–80. https://doi.org/10.1111/j.1473-4192.1998.tb00121.x

36.

Junqua

J. C.

(1996). The influence of acoustics on speech production: A noise-induced stress phenomenon known as the Lombard reflex. Speech Communication, 20(1–2), 13–22. https://doi.org/10.1016/S0167-6393(96)00041-6

37.

Keidser

Dillon

Carter

O’Brien

(2012). NAL-NL2 Empirical adjustments. Trends in Amplification, 16(4), 211–223. https://doi.org/10.1177/1084713812468511

38.

Keidser, G., Naylor, G., Brungart, D., Caduff, A., Campos, J., Carlile, S., Carpenter, M.,Grimm, G., Hohmann, V., Holube, I., Launer, S., Lunner, T., Mehra, R., Rapport, F., Slaney, M., Smeds, K. (2020). The quest for ecological validity in hearing science: What it is, why it matters, and how to advance it. Ear and Hearing, 41(Suppl 1), 5S–19S. https://doi.org/10.1097/AUD.0000000000000944

39.

Kiessling

Pichora-Fuller

M. K.

Gatehouse

Stephens

Arlinger

Chisolm

Von Wedel

(2003). Candidature for and delivery of audiological services: Special needs of older people. International Journal of Audiology, 42(Suppl 2), 92–101. https://doi.org/10.3109/14992020309074650

40.

Kuk

Slugocki

Ruperto

Korhonen

(2021). Performance of normal-hearing listeners on the repeat-recall test in different noise configurations noise configurations. International Journal of Audiology, 60(1), 35–43. https://doi.org/10.1080/14992027.2020.1807626

41.

Kuznetsova

Brockhoff

P. B.

Christensen

R. H. B.

(2017). Lmertest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–28. https://doi.org/10.18637/jss.v082.i13

42.

Levinson

S. C.

Torreira

(2015). Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology, 6(JUN), 1–17. https://doi.org/10.3389/fpsyg.2015.00731

43.

Lind

(2012). Living Well: Everyday Conversation and Hearing Impairment. Hearing Review, May. Retrieved from https://www.hearingreview.com/hearing-products/implants-bone-conduction/cochlear-implants/living-well-everyday-conversation-and-hearing-impairment-2.

44.

Lombard

(1911). Le signe de le elevation de la voix. Ann. Malad. l’Orielle Larynx Nez Pharynx, 37(7), 101–119.

45.

McKinney

M. F.

Zhang

Oxenham

A. J.

(2021). Investigating age, hearing loss, and background noise effects on speaker-targeted head and eye movements in three-way conversations. The Journal of the Acoustical Society of America, 149(3), 1889–1900. https://doi.org/10.1121/10.0003707

46.

Lunner

Rudner

Rosenbom

Agren

E. H. N.

(2016). Using speech recall in hearing aid fitting and outcome evaluation under ecological test conditions. Ear and Hearing, 37(Suppl 1), 145S–54S. https://doi.org/10.1097/AUD.0000000000000294.

47.

Magyari

Bastiaansen

M. C. M.

De Ruiter

J. P.

Levinson

S. C.

(2014). Early anticipation lies behind the speed of response in conversation. Journal of Cognitive Neuroscience, 26(11), 2530–2539. https://doi.org/10.1162/jocn_a_00673

48.

McGarrigle

Munro

K. J.

Dawes

Stewart

A. J.

Moore

D. R.

Barry

J. G.

Amitay

(2014). Listening effort and fatigue: What exactly are we measuring? A British society of audiology cognition in hearing special interest group ‘white paper’. International Journal of Audiology, 53(7), 433–440. https://doi.org/10.3109/14992027.2014.890296

49.

McInerney

Walden

(2013). Evaluating the use of an assistive listening device for communication efficiency using the diapix task: A pilot study. Folia Phoniatrica et Logopaedica, 65(1), 25–31. https://doi.org/10.1159/000350490

50.

Mortensen

Meyer

A. S.

Humphreys

G. W.

(2006). Age-related effects on speech production: A review. Language and Cognitive Processes, 21(1–3), 238–290. https://doi.org/10.1080/01690960444000278

51.

Nachtegaal

Kuik

D. J.

Anema

J. R.

Goverts

S. T.

Festen

J. M.

Kramer

S. E.

(2009). Hearing status, need for recovery after work, and psychosocial work characteristics: Results from an internet-based national survey on hearing. International Journal of Audiology, 48(10), 684–691. https://doi.org/10.1080/14992020902962421

52.

Norwine

A. C.

Murphy

O. J.

(1938). Characteristic time intervals in telephonic conversation. Bell System Technical Journal, 17(2), 281–291. https://doi.org/10.1002/j.1538-7305.1938.tb00432.x

53.

Painton

S. W.

(1993). Objective measure of low-frequency amplification reduction in canal hearing aids with adaptive circuitry. Journal of the American Academy of Audiology, 4(3), 152–156.

54.

Pichora-Fuller M. K., Kramer S. E., Eckert M. A., Edwards B., Hornsby B. W. Y., Humes L. E., … Wingfield A. (2016). Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear and Hearing, 37, 5S–27S. https://doi.org/10.1097/AUD.0000000000000312.

55.

Picou

E. M.

Ricketts

T. A.

(2018). The relationship between speech recognition, behavioural listening effort, and subjective ratings. International Journal of Audiology, 57(6), 457–467. https://doi.org/10.1080/14992027.2018.1431696

56.

Powers

Froehlich

Branda

Weber

(2018). Clinical study shows significant benefit of own voice processing. Hearing Review, 25(2), 30–34.

57.

Remez

R. E.

(2012). Three puzzles of multimodal speech perception. In Audiovisual Speech Processing, 4–20. https://doi.org/10.1017/CBO9780511843891.003

58.

Rönnberg

Lunner

Zekveld

Sörqvist

Danielsson

Lyxell

, … Rudner

(2013). The ease of language understanding (ELU) model: Theoretical, empirical, and clinical advances. Frontiers in Systems Neuroscience, 7(JUNE), 1–17. https://doi.org/10.3389/fnsys.2013.00031

59.

Satterthwaite

F. E.

(1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. https://doi.org/https://doi.org/10.2307/3002019

60.

Sjerps

M. J.

Meyer

A. S.

(2015). Variation in dual-task performance reveals late initiation of speech planning in turn-taking. Cognition, 136, 304–324. https://doi.org/10.1016/j.cognition.2014.10.008

61.

Sørensen

A. J. M.

(2021b). PDF and PSD files of DiapixDK picture materials - adapted Danish version [Data set], http://doi.org/10.5281/zenodo.4638693.

62.

Sørensen

A. J. M.

(2021a). The effects of noise and hearing loss on conversational dynamics. Technical University of Denmark. Retrieved from https://www.hea.healthtech.dtu.dk/-/media/centre/hea_hearing_systems/hea/english/research/phd-thesis-pdf/00_47_-sorensen.pdf?la=da&hash=BA1FD9620387CA7A7C0F284C1089554CDFD10331.

63.

Sørensen

A. J. M.

Fereczkowski

MacDonald

(2021). The effects of noise and second language on conversational dynamics in task dialogue. Trends in Hearing, 25, 1–17. https://doi.org/https://doi.org/10.1177/23312165211024482

64.

Stephens

Zhao

(1996). Hearing impairment: Special needs of the elderly. Folia Phoniatrica et Logopaedica, 48(3), 137–142. https://doi.org/10.1159/000266400

65.

Stivers

Enfield

N. J.

Brown

Englert

Hayashi

Heinemann

Levinson

S. C.

(2009). Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences of the United States of America, 106(26), 10587–10592. https://doi.org/10.1073/pnas.0903616106

66.

Sumby

W. H.

Pollack

(1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26(2), 212–215. https://doi.org/10.1121/1.1907309

67.

ter Bekke

Drijvers

Holler

(2020). The predictive potential of hand gestures during conversation: An investigation of the timing of gestures in relation to speech. PsyArXiv Preprints, 1–6. https://doi.org/10.31234/osf.io/b5zq7

68.

Tuomainen

Hazan

Taschenberger

(2019). Speech communication in background noise: Effects of aging.

69.

Watson

Sørensen

A. J. M.

MacDonald

(2020). The effect of conversational task on turn taking in dialogue. In Proceedings of ISAAR 2019: Auditory Learning in Biological and Artificial Systems. 7th International Symposium on Auditory and Audiological Research. Nyborg, Denmark. (Vol. 43, pp. 61–68).

70.

Wendt

Hietkamp

R. K.

Lunner

(2017). Impact of noise and noise reduction on processing effort: A pupillometry study. Ear and Hearing, 38(6), 690–700. https://doi.org/10.1097/AUD.0000000000000454