Objective Prediction of Hearing Aid Benefit Across Listener Groups Using Machine Learning: Speech Recognition Performance With Binaural Noise-Reduction Algorithms

Experiment

The stimuli and the empirical SRTs were taken from Völker et al. (2015). There, SRTs of 10 listeners with normal hearing and 12 listeners with aided impaired hearing were adaptively measured with the German matrix sentence test in a binaural cafeteria environment that was simulated with head-related impulse responses from Kayser et al. (2009) with eight different binaural preprocessing strategies. Three different noises were employed to determine SRTs in a condition where the target speaker was located in front of the listener: (a) A 20-talker babble noise (20T), (b) the ambient sound in the cafeteria in which the head-related impulse responses were recorded (cafeteria ambient noise [CAN]), and (c) a single interfering talker (SIT). In this study, the focus was on the natural, recorded ambient sound the cafeteria, which was assumed to represent the most realistic condition. The preprocessing algorithms included adaptive differential microphones (ADM), a coherence filter, single-channel noise reduction (SCNR), and binaural beamformers and are listed in Section Binaural Preprocessing Algorithms. In addition, a condition with no preprocessing (NoPre) served as a reference. To provide listeners with impaired hearing with audible signals in all conditions, the multiband dynamic compressor of the Master Hearing Aid (MHA) was used to compensate their HL with the nonlinear fitting procedure by the National Acoustic Laboratories (Byrne, Dillon, Ching, Katsch, & Keidser, 2001), even in the reference condition with no binaural preprocessing. For the group of listeners with normal hearing, no amplification was provided. The group of listeners with impaired hearing consisted of eight male and four female listeners with ages ranging from 21 to 66 years. Their hearing thresholds were 43.9 dB higher on average over normal-hearing listeners between frequencies of 500 Hz and 4 kHz and ranged from 20 dB to 55 dB HL. For more details on the audiometric data and the compensation of HL, please refer to Völker et al. (2015).

Prediction of SRTs With FADE

The outcome of the German matrix test was predicted with FADE (Schädler et al., 2016b). The condition here is defined by the task (matrix sentence recognition), the speech signals, the noise signal, the individual signal processing, and the individual audibility. Therefore, the ASR system is trained and tested for each condition separately, that is, there is no interdependence regarding the conditions. It is particularly noteworthy that no prior information about any empirical SRT is required to obtain a predicted SRT. In other words, no reference to another condition is required, and an SRT can be predicted for an isolated condition.

The procedure to obtain the predicted outcome in one condition with FADE is as follows:

Gaussian Mixture Model (GMM) and HMM-based ASR systems are trained on all available matrix sentences; one for each considered SNR (−24 dB to 6 dB in 3-dB steps). Therefore, random portions of the noise signal were chosen to generate the training data. The mixed recordings were then processed with the (individual) HA using the MHA (Grimm, Herzke, Berg, & Hohmann, 2006) with the algorithm that corresponds to the condition. From the processed signals, features were extracted that incorporate the individual hearing thresholds as explained in Section Model of Impaired Hearing. The feature extraction is based on separable Gabor filter bank (SGBFB) features with mean-and-variance normalization. The SGBFB feature set was shown to robustly extract information of speech signals from audio recordings (Schädler & Kollmeier, 2015) and to be suitable for modeling tone detection and speech recognition experiments with normal-hearing listeners (Schädler et al., 2016b). SGBFB features encode a range of spectrotemporal modulation patterns, where mean-and-variance normalization removes information about the absolute level and the overall dynamic range of the recording. For the acoustic model, word models with six states were used. One GMM/HMM word model was trained for each word and SNR. For each SNR, the 50 word models were combined with a language model that describes the (language-independent) syntax of the matrix test; that is, 5 words with 10 options each.

The ASR systems are then used to recognize all available matrix sentences at all considered SNRs (−24 dB to 6 dB in 3-dB steps). Like for the training data, random portions of the noise signal were chosen to generate the testing data. The mixed recordings were, again like for the training data, processed with the (individual) HA using MHA with the algorithm that corresponds to the condition. From the processed signals, features that incorporate the individual hearing thresholds were extracted, like for the training data. This training/testing scheme is also known as matched training, an ideal training condition. It is employed to get an estimate of the lowest achievable SRT in the given condition. Recognition performance of the ASR system is subject to the same limitations that trained human listeners encounter due to the task, the speech material, the noise signal, the individual signal processing, and the individual audibility. From the transcripts, the word recognition rates in percent are determined for each combination of training and testing SNR and compiled into a square matrix, with training SNR in one dimension and testing SNR in the other dimension. This matrix is referred to as the recognition result map (cf. Schädler et al., 2016b). Usually the testing SRT depends on the training SNR. The lowest testing SNR at which 50% of the words were correctly recognized was interpolated between the 3 dB-spaced samples and taken as the predicted SRT. For further details about the simulations, please refer to the original publications (Schädler et al., 2015, 2016b). The concept of how the predictions with FADE were individualized is illustrated in Figure 2. The feature extraction and HAs were individualized based on the audiogram, which is explained in Sections “Model of Impaired Hearing” and “Model of Binaural Hearing.” FADE is free software and available online. ¹ OPEN IN VIEWER

Model of Impaired Hearing

The feature extraction of the original publication from Schädler et al. (2016b) was modified to implement an absolute hearing threshold. The implementation of the hearing threshold is identical to the one from Kollmeier et al. (2016), where the absolute hearing threshold and a suprathreshold parameter were successfully used to predict individual outcomes of the German matrix sentence test in noise for listeners with impaired hearing. In this work, only the hearing thresholds were considered because no suitable empirical data existed to determine individual estimates of the suprathreshold component. To account for the loss of sensitivity, which is usually assessed by the audiogram, the individual hearing threshold was applied to the spectrotemporal representation, the log Mel-spectrogram, by setting any input spectrogram level to the audiogram level if it was below this value. The effect of this implementation of the hearing threshold on the log Mel-spectrogram of a noisy speech sample is illustrated in Figure 3. Applying an individual threshold has the effect that any patterns below have no effect on the feature vector anymore. Hence, part of the speech information is concealed from the recognizer, which naturally results in individually lower recognition performance and increased SRTs. OPEN IN VIEWER

Model of Binaural Hearing

The measurements in Völker et al. (2015) were dichotic, which to model required a stage that integrated the two acoustic signals from both ears. Different models of binaural hearing and signal processing were proposed in the literature (e.g., Beutelmann et al., 2010); however, due to using intrusive classification, none of them works blindly. In other words, these models require information that is not accessible in the current setup, where only noisy, processed signals are available, and hence could not be used. Here, to limit the complexity of the approach, independent feature vectors were calculated for each ear, which were then concatenated to form one binaural feature vector. This very simple binaural model assumes a late, that is, cortical, integration of binaural information in speech processing. Because the recognizer can learn a better ear during the training stage, we denominate it as automatic better ear listening. An implementation of the feature extraction is available online. ²

Binaural Preprocessing Algorithms

The eight binaural signal preprocessing strategies were taken from Völker et al. (2015) and included (1) No preprocessing (NoPre), (2) ADM (Elko & Pong, 1995), (3) ADM with coherence-based noise reduction (ADM + coh; Grimm, Hohmann, & Kollmeier, 2009), (4) SCNR (e.g., Gerkmann, Breithaupt, & Martin, 2008), (5) fixed binaural minimum variance distortionless response (MVDR) beamformer (e.g., Van Veen & Buckley, 1988), (6) adaptive binaural MVDR beamformer (e.g., Griffiths & Jim, 1982), (7) common postfilter (com PF) based on fixed MVDR (e.g., Simmer, Bitzer, & Marro, 2001), (8) com PF based on adaptive MVDR (e.g., Simmer et al., 2001), and (9) individual postfilter (ind PF) based on adaptive MVDR (e.g., Simmer et al., 2001). The algorithms were described and instrumentally evaluated by Baumgärtel et al. (2015) in the same experimental conditions. ADM and SCNR are monaural algorithms and do not require a binaural link, that is, they work independently on both ears. The remaining six algorithms are binaural preprocessing strategies, where the signal input on one side can influence the output on the other side. All algorithms aim to improve speech intelligibility; however, they achieve that goal only to a varying degree (Völker et al., 2015). The goal of the current study is to predict this benefit, which can be measured in improvement in SRT. For details on the individual algorithms please refer to Völker et al. (2015), Baumgärtel et al. (2015), or the corresponding original publications. It is assumed that this selection provides a good representation of modern binaural speech and audio signal processing algorithms, and hence also reflects typical effects, nonlinearities, and artifacts involved in such processing. In the current study, the signal processing was performed by using the MHA, which was also used by Völker et al. (2015) to carry out the measurement of the empirical data. For the individual predictions, the same individual configuration that was used for the empirical measurements was used to process the noisy training and testing data (cf. Section “Prediction of Speech Reception Thresholds With FADE”) before individual feature vectors that considered the individual hearing thresholds were extracted (cf. Section “Model of Impaired Hearing”).

Predicted SRTs

The predicted SRTs are plotted against the empirical SRTs in Figure 4 for the 20T condition and the CAN condition, and median SRTs for the groups of listeners with normal hearing and aided impaired hearing in all conditions are presented in numerical form in Table 2. It should be noted that the FADE approach in its current form is not suitable to predict the recognition thresholds for the conditions with an SIT. The model seems to be unable to differentiate between the two competing speakers, and no a priori information was provided to the algorithm which speaker is the target and which is the interferer. Without a cue to differentiate the speakers, the FADE model can only perform the recognition task at much higher SNRs than achieved by human listeners. This is reflected by the finding that the predicted SRTs for listeners with normal hearing in the SIT condition were about 10 dB higher than the empirical ones. Reasonable predictions for the generally very homogeneous group of listeners with normal hearing can be considered a prerequisite to performing reasonable predictions for the generally less homogeneous group of listeners with impaired hearing, which is why no predictions were performed for listeners with impaired hearing in this condition. Hence, in the following, the extreme (SRTs below −20 dB) SIT condition is not considered, and the analysis focuses on the artificial 20T condition and, primarily, on the realistic CAN condition. However, for completeness, the available data for the SIT condition are provided in numerical form. OPEN IN VIEWER

OPEN IN VIEWER

Table 2.

Median Empirical and Predicted SRTs for the Group of Listeners With Normal Hearing (NH) and Impaired Hearing (HI) Aided With Eight Different Binaural Preprocessing Algorithms in Three Conditions: 20-Talker Babble Noise (20T), Cafeteria Ambient Noise (CAN), and Single Interfering Talker (SIT).

Cond.	Data	NoPre	ADM	1) + coh	SCNR	fixed MVDR	adapt MVDR	5) + com PF	6) + com PF	6) + ind PF
		1)	2)	3)	4)	5)	6)	7)	8)	9)
20T	NH emp.	−7.6	−12.0	−11.8	−7.2	−11.7	−11.4	−11.3	−10.8	−11.3
20T	NH pred.	−6.6	−9.2	−9.3	−4.9	−10.2	−9.6	−10.4	−9.6	−9.2
20T	HI emp.	−6.7	−10.2	−9.6	−5.4	−9.8	−10.1	−9.3	−8.9	−8.9
20T	HI pred.	−3.9	−6.6	−6.6	−3.8	−9.6	−8.2	−9.2	−8.1	−8.0
CAN	NH emp.	−9.7	−13.1	−13.5	−9.4	−14.2	−12.7	−12.8	−12.1	−11.4
CAN	NH pred.	−7.2	−10.8	−11.1	−6.4	−11.4	−9.3	−10.4	−8.3	−8.3
CAN	HI emp.	−8.6	−11.8	−11.6	−7.4	−12.6	−12.2	−11.7	−10.4	−11.1
CAN	HI pred.	−5.7	−8.7	−9.7	−5.1	−10.9	−8.3	−9.5	−7.4	−7.5
SIT	NH emp.	−21.6	−23.8	−23.1	−19.9	−23.6	−25.9	−24.1	−23.8	−25.1
SIT	NH pred.	−9.4	−13.5	−14.0	−6.4	−13.1	−8.5	−13.1	−8.3	−8.1

Note. Individual predictions for listeners with impaired hearing in the SIT condition were not performed. The numbers in the header identify the algorithms: (1) no preprocessing (NoPre), (2) adaptive differential microphones (ADM), (3) ADM with coherence-based noise reduction (coh), (4) single-channel noise reduction (SCNR), (5) fixed binaural minimum variance distortionless response (MVDR) beamformer, (6) adaptive binaural MVDR beamformer, (7) common postfilter (com PF) based on fixed MVDR, (8) com PF based on adaptive MVDR, and (9) individual postfilter (ind PF) based on adaptive MVDR.

SRT = speech reception threshold.

For the 20T and CAN conditions, the median empirical SRTs of both groups (normal and impaired hearing) show an effect size of about 5 dB due to the different binaural preprocessing algorithms. The median predicted SRTs show a very similar effect size. The median empirical SRTs for listeners with normal hearing are lower than the ones for listeners with impaired hearing, which is also correctly predicted by the model. The median predictions for both groups (black symbols) were mostly underestimated, that is, the predicted SRTs were higher than the empirical ones. This deviation is more pronounced in the CAN than in the 20T condition. However, the predicted values seem to be scattered around a parallel to the diagonal, which indicates an overall offset (bias) but correct relations of the predictions. For the median prediction of the group of listeners with impaired hearing, the deviation from the diagonal is more pronounced in the 20T than in the CAN condition. Individual predictions (cf. colored symbols in Figure 4) were overestimated as well as underestimated, where the predictions for a specific listener, indicated by the same color, seem to tend to either one. A statistical analysis of the median and individual SRTs is provided in Table 3. The high bias of 13 dB in the SIT condition quantifies the extensive gap between the empirical and predicted group performance with a pure speech masker, and the p value > .05 shows that the predicted and the empirical data were not significantly correlated. In contrast, in the 20T and CAN condition, all predicted outcomes (individual and group) were found to be significantly correlated with the empirical data. However, only the group predictions explained more than 50% of the variance of the data (R², assuming normally distributed data) due to the use of the different preprocessing algorithms. For the group data of listeners with normal hearing, the predictions explained 87% of the variance in the 20T condition and 92% of the variance in the CAN condition. For the—usually less homogeneous—group of listeners with impaired hearing, the group predictions still explained 63% and 83% in the 20T and the CAN condition, respectively. The individual predictions, although significantly correlated, explained less than 20% of the variance in the individual data. The bias of the predictions, which also determines the lower limit of the root-mean-square (RMS) prediction error, mainly depended on the noise condition and was found to be about 1.5 dB in the 20T condition and less than 3 dB in the CAN condition.

OPEN IN VIEWER

Table 3.

Statistical Analysis of the Predicted SRTs for the Normal-Hearing Group (NH Median), the Hearing-Impaired Group (HI Median), and the Individual Outcomes of Listeners With Impaired Hearing (HI Individual) With Eight Different Binaural Preprocessing Strategies (Plus the No Processing Reference Condition) in Three Acoustic Conditions: 20-Talker Babble Noise (20T), Cafeteria Ambient Noise (CAN), and Single Interfering Talker (SIT).

Cond.	Prediction	R 2	Interval	p	B	RMS
20T	NH median	0.87	0.51–0.97	<.01	1.8	1.9
20T	HI median	0.63	0.08–0.91	<.05	1.6	2.1
20T	HI indiv.	0.16	0.06–0.31	<.01	1.4	3.2
CAN	NH median	0.92	0.66–0.96	<.01	2.9	2.9
CAN	HI median	0.83	0.40–0.96	<.01	2.7	2.8
CAN	HI indiv.	0.10	0.02–0.23	<.01	2.4	3.9
SIT	NH median	0.04	0.00–0.59	>.05	13.0	13.3

Predicted Benefits in SRT

The benefit in SRT was calculated as the inverted difference between the SRT in a condition and the SRT in the reference (NoPre) condition and reflects the improvement due to using an algorithm (higher values are better). Note that this calculation was performed independently for the predicted and the simulated data, and each individual. Hence, the predictions of benefits did not depend on empirical data (apart from the individual audiograms), and the median values are calculated from differences of SRTs, that is, they are not differences of median SRTs. While the analysis in this section focuses on the realistic CAN condition, the results for the 20T and SIT condition are reported in numerical form. In Figure 5, the predicted individual and group benefits are plotted against the empirical individual and group benefits in the CAN condition for listeners with normal and aided impaired hearing. For listeners with normal hearing (left panel), the data points were closely aligned to the diagonal, which indicates highly accurate predictions. For the group of listeners with impaired hearing (right panel), the deviations from the diagonal were more pronounced than for listeners with normal hearing. This was an expected result due to less homogeneous nature of this group. Nonetheless, the group’s median values were found to be well aligned to the diagonal, which indicates still accurate predictions of the benefit for the group of listeners with impaired hearing in the CAN condition. The predicted group benefits with the different algorithms are directly compared with the empirically determined ones in Figure 6 for the group of listeners with normal hearing and in Figure 7 for the group of listeners with aided impaired hearing. For listeners with normal hearing, the predicted benefits were found to be in very good agreement with the median of the empirical data. The largest deviations were observed for the algorithms with the adaptive MVDR beamformer, which is the most complex algorithm and also resulted in a relatively large empirical interquartile range of about 2 dB. It is notable that the model correctly predicted that the SCNR did not improve speech intelligibility. For listeners with impaired hearing, the median of the individually predicted benefits is in agreement with the empirical data, however, not to the same degree than it was found for listeners with normal hearing. Interestingly, the interquartile ranges of the predictions were found to be consistently larger than those of the empirical data. This is probably due to the interaction of the hearing threshold, which is implemented as a hard threshold as opposed to the probably smoother transition area from inaudible to audible in human listeners, with the compressive amplification scheme. Small deviations of the hard threshold will have stronger interactions with compressed signal levels than human listeners with smooth transition zones. This effect is reinforced by the low accuracy ( ± 5 dB ) of threshold measurements in audiograms, which adds an additional noise source to the predictions not present in the empirical data. The benefit of the best and worst performing algorithms, fixed MVDR and SCNR, respectively, was overestimated by about 1 dB, while the deviations in the other conditions were smaller. Again, the predicted results did not suggest a benefit from using the SCNR algorithm. The group benefits in SRT for all conditions are presented in numerical form in Table 4, and a statistical analysis of the individual and group benefits in SRT is provided in Table 5. The proportion of explained variance of the empirical group data was similar in the 20T and the CAN condition. It was higher (about 90%) for the group of listeners with normal hearing than for the group of listeners with impaired hearing (about 75%). The bias and RMS error were particularly low in the CAN condition, with less than 0.5 dB and less than 1.0 dB, respectively. These were found to be slightly higher in the babble noise condition, with about 1 dB and less than 2 dB, respectively. In the SIT condition, the median predicted benefits were not significantly correlated with the empirical data. The predicted benefits for the individual listeners with aided impaired hearing were found to be significantly correlated with the empirical data. However, the proportion of explained variance was, with less than 25%, low, and the RMS prediction error, with more than 2.4 dB, comparatively high. OPEN IN VIEWER

OPEN IN VIEWER

Figure 6.

Median predicted and empirical benefits in SRT (difference to reference condition) for the group of listeners with normal hearing in a cafeteria ambient noise condition with eight different binaural preprocessing algorithms: ADM) adaptive differential microphones, + coh) with coherence-based noise reduction, SCNR) single-channel noise reduction, MVDR) binaural minimum variance distortionless response beamformer, com PF) common postfilter based, and ind PF) individual postfilter based. The error bars of the empirical data indicate the first and third quartile of the corresponding individual data, and their length the interquartile range. The error bars of the predicted data indicate an assumed symmetric interquartile range of 1 dB.

OPEN IN VIEWER

Figure 7.

Median predicted and empirical benefits in SRT (difference to reference condition) for the group of listeners with aided impaired hearing in a cafeteria ambient noise condition with eight different binaural preprocessing strategies (cf. Figure 6 for an explanation of the abbreviations). The error bars the indicate the first and third quartile of the corresponding individual data, and their length the interquartile range.

OPEN IN VIEWER

Table 4.

Median Empirical and Predicted Benefits in SRT (Difference to Reference Condition) in dB for the Group of Listeners With Normal Hearing (NH) and Aided Impaired Hearing (HI) With Eight Different Binaural Preprocessing Algorithms in Three Conditions: 20-Talker Babble Noise (20T), Cafeteria Ambient Noise (CAN), and Single Interfering Talker (SIT).

Cond.	Data	ADM	1) + coh	SCNR	fixed MVDR	adapt MVDR	5) + com PF	6) + com PF	6) + ind PF
		1)	2)	3)	4)	5)	6)	7)	8)
20T	NH emp.	4.5	4.1	−0.6	4.1	3.7	3.7	3.2	3.4
20T	NH pred.	2.6	2.7	−1.7	3.6	3.0	3.8	3.0	2.5
20T	HI emp.	3.5	3.4	−0.7	4.1	3.8	3.4	3.1	3.2
20T	HI pred.	3.4	3.7	0.7	6.2	4.7	5.8	4.8	4.9
CAN	NH emp.	3.5	4.0	−0.5	4.1	2.7	3.1	2.3	2.2
CAN	NH pred.	3.6	3.9	−0.7	4.2	2.1	3.3	1.1	1.1
CAN	HI emp.	2.9	3.2	−1.0	3.8	3.2	3.3	1.7	2.6
CAN	HI pred.	3.1	4.3	0.2	5.1	2.6	4.3	1.8	1.9
SIT	NH emp.	2.4	3.0	−1.4	3.2	4.1	1.3	3.2	3.2
SIT	NH pred.	4.1	4.6	−3.0	3.7	−0.8	3.7	−1.1	−1.3

Note. Individual predictions for listeners with impaired hearing in the SIT condition were not performed. The numbers in the header identify the algorithms: (1) adaptive differential microphones (ADM), (2) ADM with coherence-based noise reduction (coh), (3) single-channel noise reduction (SCNR), (4) fixed binaural minimum variance distortionless response (MVDR) beamformer, (5) adaptive binaural MVDR beamformer, (6) common postfilter (com PF) based on fixed MVDR, (7) com PF based on adaptive MVDR, and (8) individual postfilter (ind PF) based on adaptive MVDR. Please note that the group predictions of the benefits in SRT do not have to coincide with the corresponding differences in the group predictions of SRTs provided in Table 2.

SRT = speech reception threshold.

OPEN IN VIEWER

Table 5.

Statistical Analysis of the Predicted Benefits (Difference to Reference Condition, Higher Values Are Better) in SRTs for the Normal-Hearing Group (NH Median), the Hearing-Impaired Group (HI Median), and the Individual Outcomes of Listeners With Impaired Hearing (HI Individual) in Three Acoustic Conditions: 20-Talker Babble Noise (20T), Cafeteria Ambient Noise (CAN), and Single Interfering Talker (SIT).

Model	Cond.	Prediction	R 2	Interval	p	B	RMS
iSNR a	20T	NH median	0.62	–	<.05	−3.0	3.3
FADE	20T	NH median	0.87	0.46–0.98	<.01	−0.8	1.0
FADE	20T	HI median	0.75	0.17–0.95	<.01	1.3	1.5
FADE	20T	HI indiv.	0.23	0.10–0.39	<.01	1.3	2.4
iSNR a	CAN	NH median	0.35	–	>.05	−2.7	3.1
FADE	CAN	NH median	0.91	0.57–0.98	<.01	−0.4	0.6
FADE	CAN	HI median	0.77	0.20–0.96	<.01	0.4	0.9
FADE	CAN	HI indiv.	0.13	0.03–0.27	<.01	0.3	2.4
FADE	SIT	NH median	0.07	0.00–0.67	>.05	−1.1	3.1

Note. Pearson’s correlation coefficients (R²) are reported including their 95% confidence intervals according to Fisher (1958) along with the root-mean-square (RMS) prediction error and the bias (B) in dB for the predicted SRTs.

SRT = speech reception threshold; RMS = root-mean-square; iSNR = intelligibility-weighted signal-to-noise ratio; FADE = framework for auditory discrimination experiment.

^aThe data of the iSNR-based predictions were taken from Baumgärtel et al. (2015).

Correlation of Prediction Errors Across Test Conditions

To test if the pronounced individual prediction errors (difference between the predicted and the corresponding empirical SRT) for listeners with aided impaired hearing in the 20T and the CAN condition (cf. left panel in Figure 5) are systematic, that is, due to a common cause that the model did not consider, the individual prediction errors are plotted against each other in Figure 8. The prediction errors were found to be significantly correlated (R²= .69, p < .01), clearly indicating a systematic error. In other words, when the model over/underestimated an individual SRT in one condition, it also over/underestimated it in the other conditions. OPEN IN VIEWER

Intrusive Classification and Objectivity

Völker et al. (2015) modeled the same empirical data with the BSIM (Beutelmann et al., 2010) and found that the difficult—but required for BSIM predictions—post hoc separation of the processed signal into a speech and noise part was a major problem for those conditions where nonlinear signal processing algorithms were used. The problems of Intrusive (1) classification concerns many speech intelligibility models (cf. Table 1). To overcome this problem in modeling, Völker et al. (2015) proposed to use FADE, which was successfully employed in this study. Another—often neglected—problem with many other prediction methods for human speech recognition performance is the lack of objectivity due to being based on empirical reference data, which also makes the interpretation of prediction errors difficult. To the best of our knowledge, no other model with nonintrusive classification that allows the objective, that is, empirical reference-free, prediction of SRTs in aided conditions with fluctuating noise maskers exists at this time of writing.

Predicted Outcomes

Predicted outcomes of the German matrix sentence test were compared with empirical outcomes, whose differences are the prediction errors. An interpretation of the prediction error is a prerequisite for well-founded speculations on their causes, and hence on the causes of potential shortcomings of the model. Objective SRT predictions in combination with nonintrusive classification are presumably helpful for deriving a meaningful interpretation of the prediction errors. For example, if FADE cannot explain some aspect, the recognition performance gets worse and the predicted SRTs increase, hence underestimating the empirical recognition performance. This performance limitation is similar to most ASR systems that perform worse than listeners with normal hearing when confronted with situations for which they were not designed. Hence, a significant overestimation of the empirical performance (not observed here) indicates that the model was (a) provided with more information or (b) made better use of the information that is required to perform the corresponding recognition task. A significant underestimation of the empirical performance of listeners with normal hearing, on the other hand, indicates a suboptimal—or missing—implementation of at least one of the required principles to perform the corresponding recognition task.

A small underestimation of the median empirical performance was observed in Section “Predicted SRTs” for the 20T and, slightly more pronounced, in the CAN condition. This could be due to the simplistic binaural better-ear strategy adopted by the current approach that might not be able to separate the target speaker from the noises as efficiently as listener with normal binaural hearing. The interaction of the spatial scene with binaural signal processing and binaural (impaired) hearing could also be the cause of the variability in the deviation of predicted median SRTs for listeners with impaired hearing in the 20T condition. A more sophisticated, true binaural processing stage with a direct across-ear processing of the input signal (e.g., Spille, Kollmeier, & Meyer, 2017) might improve the model behavior here.

A larger systematic underestimation of the recognition performance was observed in Section “Predicted SRTs” for the SIT condition, which resulted in large positive biases in Table 3 for the performance of listeners with normal hearing in all conditions. The considerable bias of 13 dB indicates that the model had a severe problem with voiced speech maskers. The system was neither able to learn nor provided with information about which speech portion is considered as target and which has to be considered as nonattended interferer. Human listeners achieve very low SRTs ( < - 20 dB) primarily by being able to listen in the dips of the masking speech. Probably, human listeners are better able to segregate the talkers based on their pitch and spatial cues than the model, an ability that is often referred to as the cocktail-party effect. The necessary signal processing to perform such a task by machines has been studied under the term computational acoustic scene analysis (CASA) and is largely missing in current ASR systems as well as in FADE. Emerging technical solutions to this problem could potentially improve ASR systems and lead to improved predictions in the SIT condition.

Because the predicted SRTs in the SIT condition were not found to be in agreement with the empirical data for normal-hearing listeners, no individualization, that is, prediction for listeners with impaired hearing was performed. In the 20T and CAN condition, the predictions for the group of listeners with impaired hearing showed the same bias (±0.2 dB) and hence higher SRTs than for the group of listeners with normal hearing (cf. Table 3) in agreement with the empirical data. According to the empirical data, some listeners with impaired hearing outperformed the average listener with normal hearing, by up to ≈ 2 dB. However, the individual predicted SRTs were not found to be more than 0.5 dB lower than the corresponding predictions without individualization, that is, for listeners with normal hearing. This can be partly observed in Figure 4 comparing the lowest individual predictions for listeners with impaired hearing (colored stars) to the lowest prediction for listeners with normal hearing (black squares) and also holds for the other algorithms (not shown). An explanation might be the inevitable uncertainty or statistical measurement error in the individual empirical data that could sum up to ≈ 2 dB depending in the noise condition. This expected error in the empirical data is due to small training effects, limited and varying attention of the listeners, and the small number of presented sentences (at most, 20) on which the SRT calculation is based. It should be noted that this statistical error is not—or to a much lesser extent—found in the predictions because SRTs are calculated based on results from all 120 sentences.

Apart from the already discussed bias, which also dominated the RMS prediction errors, the predicted group SRTs correlated significantly with the empirical data, explaining between 63% and 92% of the variance across algorithms in the 20T and CAN condition, respectively. The individual predictions for listeners with impaired hearing did not explain more that 16% of the variance across individuals and algorithms. These results already hint at good group predictions in general, but unsatisfactory individual predictions for listeners with impaired hearing. However, individual offsets originating from nonacoustical causes which are not modeled here (e.g., central neurosensory component of the HL or varying cognitive abilities) may affect all algorithms alike. They may cause a spread of the individual empirical SRTs and hence result in a lower correlation between the predicted and empirical data. To test this hypothesis and remove an individual bias, the benefits in SRT with respect to the reference condition (NoPre) were calculated individually for all listeners (to be discussed later).

Objective Instrumental Evaluation of Algorithms

The high prediction accuracy (RMS prediction error ≤ 1 dB) and the low absolute biases (< 1 dB) of the benefits in SRT for the group of listeners with normal hearing (cf. Figures 5 and 6, and Table 5, respectively) indicate a solid basis for individualized predictions. Further, the predicted benefits for listeners with normal hearing do not depend on any empirical data, which is why FADE provided a near-optimal objective evaluation in this regard. Notably, in the realistic CAN condition, the model correctly predicted the fixed MVDR to result in the highest benefit and the SCNR in the lowest, where the benefit of the latter was correctly predicted to be negative, that is, it increased the SRT. While the ADM was predicted to perform worse than the fixed MVDR in the artificial 20T condition, which was not observed empirically, the negative benefit of the SCNR was still correctly predicted. The differentiation between improvement and deterioration is very valuable to algorithm developers and users alike.

In the instrumental evaluation of the algorithms from Baumgärtel et al. (2015), the iSNR, STOI, and perceptual evaluation of speech quality (PESQ; ITU-T Recommendation, 2001), which can be considered representative for the methods that are used to objectively assess the performance of (speech) signal processing algorithms, were used to predict the benefit. None of them predicted the negative effect of the SCNR algorithm in the artificial 20T condition and the realistic CAN condition for listeners with normal hearing. There, benefits in SNR were only predicted with the iSNR because the output of the other measures were indexes that could not be mapped to SNRs or benefits in SNR due to the lack of empirical reference data in that study. This reflects a common, fundamental problem of reference-based models. For objective predictions of the benefit with the iSNR, a working point, that is, an input SNR at which the benefit was predicted, had to be chosen independently from the empirical data, which was 0 dB. If the empirical working point is unknown, which usually is the case when objective measures are used, a guess of 0 dB seems still reasonable, particularly if the predictions should also represent the behavior of listeners with impaired hearing or cochlear implants. These limitations, that is, the requirement of guessing or knowing the outcome prior to any prediction, clearly shows one advantage of FADE that can predict an SRT without any empirical reference. The statistical analysis of the predicted outcomes with the iSNR from Baumgärtel et al. (2015) for the group of normal-hearing listeners in the 20T and CAN condition is shown in Table 5. For the artificial 20T condition, the iSNR predictions explained about 60% of the variance in empirical data and the RMS prediction error exceeded 3 dB. For the realistic CAN condition, the iSNR predictions were not significantly correlated with the empirical data and the RMS prediction error also exceeded 3 dB. Compared with these results, the FADE predictions of benefits in SRT for normal-hearing listeners provided a much more reliable objective instrumental measure with RMS prediction errors of 1 dB and less in the respective 20T and CAN conditions.

The mediocre performance of the iSNR, STOI, and PESQ could be partly due to the method (phase inversion according to Hagerman & Olofsson, 2004) that was used to separate the speech from the noise signal after processing and that might not achieve a perfect separation. This seems probable because no true separate processed signals exist whereby the goodness of the estimation cannot be assessed. The phase inversion could be possibly replaced by a more sophisticated separation method, such as shadow filtering. However, even if this would lead to better predictions, it would not overcome the fundamental limitation that this kind of separation of the mixed signals can lead to uncontrolled biases, which are typically concealed by a calibration to empirical reference data. Because the classification with FADE is nonintrusive and works on mixed signals, predictions can be performed with virtually any signal preprocessing without additional modifications to the model. It seems worthwhile to further validate the suitability of FADE for the objective evaluation of algorithms on a broader basis of applications.

Individual Aided Performance Predictions

Individualization was performed using the audiograms, that is, the individual hearing thresholds of both ears, where listeners with normal hearing were assumed to have no HL. This is also why there were individual measurements from listeners with normal hearing but no individual predictions for them. The individualization for listeners with impaired hearing was reflected in the personalized HA setting: Here, it had consequences for the fitting of the multiband compressor of the MHA, where the HL was (partially) compensated (cf. Section “Binaural Preprocessing Algorithms”) and the feature extraction of FADE, where the individual hearing threshold was simulated (cf. Section “Model of Impaired Hearing”). Therefore, assuming an appropriate compensation for the main component of the moderate HLs encountered in our rather homogeneous group of listeners with impaired hearing, no strong individual deviation from the compensated main component of the HL should be expected.

However, despite similar audiograms and individually compensated HLs, the empirical results show (a) that the variance of the individual benefits in SRT were in the same order of magnitude as the effect size of the algorithms (cf. Figure 5) and (b) that the median SRTs for the group of listeners with impaired hearing were about 1 to 2 dB higher than for the group of listeners with normal hearing. The individual variance of the listeners with impaired hearing could not be explained by the model considering individual absolute hearing thresholds, where at most 23% of the variance could be explained with a considerable RMS prediction error of 2.4 dB (cf. “HI indiv.” predictions in the CAN condition in Table 5). However, the general trend, that the SRTs for the group of listeners with impaired hearing were higher than those for listeners with normal hearing, was predicted in accordance with the empirical data (cf. Table 2). Also, the predictions of benefits in SRT for the group of listeners with impaired hearing (median of the corresponding benefits in SNR) explained about 75% of the variance in the empirical data with RMS prediction error of 1.5 dB and 0.9 dB in the 20T and CAN condition, respectively.

A hypothesis is that the listeners with impaired hearing are not sufficiently characterized by the audiogram, as already suggested by Kollmeier et al. (2015). According to Kollmeier et al. (2015), a (monaural) suprathreshold distortion component was important to accurately model the individual effect of HL on speech recognition tasks, and according to Plomp (1978), such a distortion component would affect the individual SRTs similarly in different conditions. This behavior could be observed in Figure 8, where the prediction errors in the 20T and CAN condition were plotted against each other and showed a significant and strong correlation (R²= .69, p < .01). Any effect that was not explained by the model is accumulated in the prediction error, that is, deviations due do not considering a suprathreshold distortion component, missing binaural interaction, or neglected cognitive effects, as well as any measuring inaccuracies (apart from the audiograms). Errors in the measurement of the audiograms would affect the predictions in all conditions similarly, while errors in the measurement of SRTs can be assumed to be independent and hence would not lead to systematic errors. One possible explanation for the strong correlation of prediction error across conditions could be systematic biases due to errors in the measurement of the audiograms. Another possibility is that there could be another individual factor—apart from the audiogram—missing to accurately predict individual benefits in SRT with FADE. Such an uncontrolled individual distortion component could be responsible for up to 4 dB (cf. point spread in Figure 8), which would translate to an average group effect of up to approximately 2 dB. This is in agreement with the average differences in SRT between the group of listeners with impaired hearing and the group of listeners with normal hearing, which were empirically found to be 1.8 dB in the 20T and 1.3 dB in the CAN condition, and predicted to be 1.7 dB and 1.2 dB, respectively.

In its current form, FADE was able to accurately predict the benefit of algorithms for the groups of listeners with normal hearing and impaired hearing. Individual (aided) performance, however, could not be predicted based on an individualization with the audiogram alone. In future work, a possible individual distortion component, for example, as suggested by Kollmeier et al. (2015) in the form of a level uncertainty, should be considered to possibly enable accurate individual predictions of the aided performance of listeners with impaired hearing.

Another (related) explanation for the low correlation of the individual predictions could be an insufficient consideration of the (possibly even individual impaired) binaural interaction. The current approach considers better ear listening that does not include true binaural interactions across the signals presented to both ears. An individual component regarding the efficiency of the binaural processing would have the same effect as the (monaural) suprathreshold distortion component discussed before. In the end, a combination of both seems likely.

Limitations and Opportunities of the Approach

The FADE modeling approach has been extensively evaluated with the matrix sentence test—mainly because of its very good test–retest reliability—but still not with other speech tests. Because ASR systems are trained for each prediction on the corresponding speech material, there are no fundamental reasons why the same approach should not work with other speech material or vocabulary. In fact, it was already shown that the very same model predicts the outcome of the matrix sentence test in other languages (Schädler et al. 2016a) in different noise conditions without additional assumptions. However, larger vocabularies and more complex grammatical structures will probably increase the already high computational demands; currently, about 5 min per prediction can be achieved on high-end ³ desktop PCs. It should be investigated in the future if the approach generalizes in this regard, that is, if it is able to predict the effect of the complexity of a speech test.

Also, these properties allow the interpretation of the prediction error. This might help to better understand which cues are missing in the feature vector for different kinds of tasks in comparison with full-blown complex ASR experiments where single factors cannot be isolated as easily, let alone the required resources to perform these experiments. A natural evolution of FADE could be the stepwise integration of methods/models from research on ASR, CASA, psychoacoustics, and even human physiology. Such extensions might be valid as long as the predicted SRTs for listeners with normal hearing do not increase. This appears to be a simple criterion that does not depend on empirical data and preserves the objectivity of FADE while being compatible with research on robust ASR.

If objectivity and interpretability of prediction errors are of no concern, FADE can be turned into a reference-based model by calibrating the predictions with empirical reference data. For example, if SRTs are empirically found to be 4 dB higher for a more complex speech test than for the matrix sentence test in a reference condition, the reference-free predictions for the matrix sentence test can be increased by 4 dB to predict the outcome of the more complex speech test. This approach is very similar to the calibration of reference-based models. It would allow performing reference-based predictions with FADE for data sets which otherwise could only be predicted with other reference-based models. Evaluations of predictions with FADE for other speech tests should consider testing the validity of a constant-offset assumption.

For individual predictions, the SRTs are allowed to increase depending on (dys-)functional parameters such as the audiogram or putative suprathreshold processing deficiencies. It is open for further research, which modification of the standard signal preprocessing and feature extraction best reflects the properties of listeners with hearing impairment. However, traditional auditory models are not expected to provide a sufficiently robust signal representation to achieve human performance levels with FADE in a fluctuating interferer (Schädler et al. 2016b), a prerequisite for impaired predictions. This was attributed to the assumption of independent frequency bands and the resulting lack of across-frequency integration.