Immediate improvement of speech-in-noise perception through multisensory stimulation via an auditory to tactile sensory substitution

1 Introduction

Over 5% of the world population has disabling hearing loss (i.e., a loss <40 dB in the better hearing ear; World Hearing Organisation 2018). Furthermore, its prevalence is expected to grow in the society, both among the elderly, where it is already the most common sensory deficit, as well as among younger people due to heavy noise exposure (World Hearing Organisation 2015). People affected by hearing loss, including age-related hearing loss (presbyacusis), experience specific difficulties with understanding rapid speech, speech presented with background noise or in case of two or more speakers talking simultaneously (Schneider et al., 2010; Agrawal et al., 2008; Imam et al., 2017; Davis et al., 2016). In the elderly, a hearing deficit not only impairs exchange of information, causing isolation and dependency on others, but its presence has been found to correlate with cognitive decline related to aging (Uchida et al., 2019; Davis et al., 2016; Fortunato et al., 2016). Recent studies even showed that hearing impairments increase the chance of developing dementia and other neurodegenerative diseases (Ford et al., 2018; Gurgel et al., 2014; Livingston et al., 2017; Loughrey et al., 2018; among others).

Hearing function and hearing-related quality of life have been shown to improve by using both hearing aids (HAs) and/or cochlear implants (CIs). Hearing aids only improve audibility of sound and thus can provide benefit to patients with mild to severe hearing loss. Nevertheless, severe damage to the inner ear, namely sensorineural hearing loss (SNHL), is not compensated by a hearing aid. Patients with severe to profound SNHL can receive a cochlear implant, an invasively inserted neuronal prosthesis that bypasses damaged hearing cells in the inner ear and directly stimulates auditory nerve fibers (Gaylor et al., 2013). Nevertheless, a number of patients are unwilling to use hearing devices [the numbers range from 4.7% (Hougaard & Ruf, 2011) to 24% (Hartley et al., 2010), due to problems with handling and maintaining them, as well as the related social stigma. Invasiveness of the cochlear implantation procedure is another reason why some patients are reluctant towards this solution (e.g, Maki-Torkko et al., 2015). Among actual users of HAs and CIs the most fundamental problem that is reported is not being able to understand speech, especially when presented in background noise (McCormack and Fortnum, 2013; Hickson et al., 2014). Patients have difficulty perceiving speech in acoustically hard conditions despite their good performance in quiet. Noisy environments are specifically challenging due to patients’ inability to segregate different speech streams and thus discriminate among talkers (Gaylor et al., 2013; Carol et al., 2011; Moradi et al., 2017). In all these populations, the underlying reason for struggling in noisy environments is that the auditory input reaching the brain is deprived of temporal fine structure information: in hearing loss due to the damage to the inner ear, and in CI-users due to the limitations of the algorithms applied for speech coding (Moore et al., 2008; Moon and Hong, 2014).

The abovementioned difficulties, with understanding speech in challenging acoustic situations, are however also experienced by healthy adults. Indeed, many of us have been in a situation when we were unable to understand the person speaking on the other end of the phone line, especially when using a language that is not our native. Many of us had struggled to understand a person talking to us if there was another person talking at the same time. All these occurrences are very demanding for the central nervous system as they require “glimpsing” in the impoverished acoustic signal to recover the information, and such process engages a considerable amount of cognitive effort, also in the hearing impaired (Rosen et al., 2013; Erb et al., 2013; Huyck and Johnsrude, 2012; Banks et al., 2015; Wendt et al., 2018; Rosemann et al., 2017; Krueger et al., 2017).

In the current study we hypothesized that we would be able to improve speech understanding in noise by applying multisensory input combining auditory and tactile stimuli. To enhance the multisensory benefit of our set up we designed our experiment according to two fundamental rules governing multisensory integration, i.e. the temporal rule (stimuli are temporarily congruent) and the inverse effectiveness rule. The latter rule states that the size of the multisensory enhancement is inversely proportional to the signal to noise ratio (SNR) between the two senses; clear stimuli are more reliable and thus information from another sensory modality is redundant, resulting in reduced benefit from multisensory stimulation (see e.g., Meredith and Stein, 1986; Holmes, 2007; Otto et al., 2013).

To deliver tactile stimulation, we designed a minimalistic auditory-to-tactile Sensory Substitution Device (SSD) which transforms auditory signal into tactile vibration. Indeed, SSDs are ideal tools to test the benefits of multisensory stimulation on perception, as they can convey specific features typically provided by one sensory modality using a different one (Heimler & Amedi, in press; Heimler, Striem-Amit & Amedi, 2015). There have been many works showing that blind people are able to perform a great variety of visual tasks when visual input is conveyed via visual-to-tactile SSD, namely via mapping images from a video camera to a vibrotactile device worn on the subject’s back, forehead or tongue (Bach Y Rita, 2004; Chebat et al., 2011; Kupers and Ptito, 2011). However, to observe benefits of SSD stimulations in performance, extensive training programs had to be implemented or at least prolonged use was necessary, due to the complexity of the SSD algorithms and the cognitive load required from new SSD users (Striem-Amit et al., 2012; Bubic et al., 2010; Maidenbaum et al., 2014; Kupers and Ptito. 2011; Yamanaka et al., 2009). Here we hypothesized that due to the numerous similarities between the auditory and the tactile system (see Discussion) our set-up would require minimal training in order to observe improvement in performance (and in fact we started testing with no training whatsoever).

Several attempts have already been undertaken to improve speech understanding in patients with hearing loss, and mainly children, with the use of tactile devices. The aids provided single- or multichannel electric pulses or vibrations and were worn on various parts of the body (Galvin et al., 2000; Plant and Risberg, 1983; Weisenberger and Percy, 1995; Bernstein et al., 1998; Galvin et al., 1991). All these early studies demonstrated that the whole complex speech signal cannot be effectively translated into tactile vibration. With that in mind and also following some more recent findings in that field (Carol et al., 2011; Young et al., 2016; Huang et al., 2017), we decided to convey via touch only the low-frequency (<500Hz) information which is shared by touch and audition. Specifically, we transformed the auditory speech fundamental frequency (f0), namely the lowest frequency of a periodic waveform which is typically in the range of 80-250 Hz, into the corresponding vibrotactile frequency. Availability of fundamental frequency in the speech signal has been found crucial for understanding speech in noise, as well as segregation of talkers (Carol et al., 2011; Young et al., 2016). In addition, adding it as a tactile input has already been shown to have an enhancing effect on recognition of speech in cochlear implant users (Huang et al., 2017).

In sum, we performed a proof-of-concept study first in normal hearing individuals, by 1) applying a very degraded auditory input, deprived of temporal fine structure information and 2) using non-native speech. This procedure was selected to increase the difficulty of the task, and thereby increase the chance of taking advantage of the inverse effectiveness law for multisensory integration. Only one study thus far applied a similar multisensory audio-tactile set-up but only in the native language of a group of normal hearing participants (Fletcher et al., 2018). Participants showed minor improvement of 10% in sentence-in-noise recognition but only at fixed SNR rates and, moreover, only after a dedicated training. Demonstrating that our multisensory set-up pairing impoverished auditory signal with a vibrotactile input, delivered via a minimalistic auditory-to-tactile SSD, would enhance perception of speech-in-noise even without specific training, would have crucial implications for a number of domains. These would include research in sensory perception and integration, design of SSDs, as well as rehabilitative programs tailored for auditory recovery. In addition, showing an improvement in normal hearing non-native speakers would mean we could offer our set-up as a sensory augmenting technology to normal hearing individuals struggling in challenging acoustic situations.

2 Methods

2.3 Experimental set-up

A dedicated 3T MR-compatible Vibrating Auditory Stimulator (VAS) was developed by the Warsaw-based Neurodevice company (http://www.neurodevice.pl/en) The main parts of the system were a vibrating interface with two piezoelectric plates to simultaneously stimulate two fingers (Fig. 1 A) and a controller. The controller was powered from a socket of 230V, and vibration was delivered from a PC via an audio input. The vibration frequency range of the device is 50–500 Hz.

OPEN IN VIEWER

Fig.1

A) vibrating interface of the Vibrating Auditory Stimulator; B) Matlab GUI for stimuli presentation and control; C) Speech Reception Threshold values obtained for auditory and auditory-tactile speech in noise stimulation, at the group level and in individual subjects [subject 6 showed an improvement from 0.3 to –3.0 SRT(dB)].

The auditory stimulation was delivered via noise-cancelling headphones (BOSE QC35 IIA). Noise-cancelling was needed to attenuate noise produced by the tactile device when vibrating. Nevertheless, a follow-up study is envisaged with a free-field sound presentation, which set-up would be more ecologically valid for the hearing impaired population using hearing aids and/or cochlear implants. Headphones and the VAS system were connected to a PC via a sound-card (Creative Labs, SB1095). Sound intensity and harmonic distortions were regularly monitored with a GRASS calibration system (Audiometer Calibration Analyzer HW1001).

A MatLab (version R2016a, The MathWorks Inc., Natick, MA, USA) application with a user-friendly GUI was developed by dr Tomasz Wolak to provide very precise auditory and/or vibratory stimulation (Fig. 1 B). The application offers a number of parameter configurations to be used, including, type of stimulation to be presented (an earlier defined target vs noise), type of background noise (eg. speech or white noise), channel of stimulation (headphones vs VAS), type of the adaptive procedure to be applied to estimate SNR (for 25%–75% understanding) and the step size (1–4 dB), among others.

3 Results

The results of the experiment are presented in (Fig. 1 C-D). The mean SNR for 50% understanding (SRT) of speech in noise for the A (auditory only) condition was 18.6 dB (SD = 7.9 dB), and for the AT (auditory-tactile) condition it improved to 12.6 dB (+ /–8.5 dB). The mean benefit of adding vibration in terms of the SRT value was 6 dB (SD = 4 dB). The outcomes in the two experimental conditions were compared using the non-parametric Wilcoxon Signed-Ranks test. It was found that when vocoded sentences were accompanied with vibration, the SNR for 50% understanding was significantly lower (z = –3.06; p <  0.005) (IBM SPSS Statistics 20). Participants were able to correctly repeat on average 21.25% (+ /–9%) of the sentences in the A condition and 25% (+ /–9.3%) in the AT condition (percent of understood sentences out of 20) (Wilcoxon, p = 0.047). The outcomes of speech understanding in noise, with and without accompanying vibration, were found significantly correlated (Spearman’s Rank-Order correlation; rho = 0.6; p <  0.05). In terms of understanding vocoded sentences presented in quiet (Audio in Q; percent of understood sentences out of 20) participants obtained a mean of 55% (SD = 19%). There was no correlation found between these latter outcome and the outcomes obtained when listening to sentences presented in noise (Spearman’s Rank-Order correlation; A and Audio in Q: rho = 0.18; p = 0.57; AT and Audio in Q: rho =&thinsp-0.08; p = 0.82).

4 Discussion

The present study shows that when an auditory signal is degraded, understanding of speech in noise can be significantly improved by adding complementary information via tactile vibration, thus ultimately providing further support to the inverse effectiveness law (Otto et al., 2013; Meredith & Stein, 1986; Holmes, 2007). One crucial aspect of the current result is the automaticity of such improvement, which has very interesting consequences both for rehabilitation as well as for basic research on multisensory processing and its benefits on perception. Our results were obtained by using a minimal custom-made auditory-to-tactile Sensory Substitution Device (SSD) that conveys extracted fundamental frequency (f0) of the speech signal through vibration. All our subjects showed automatic and immediate improvement when perceiving speech-in-noise with tactile stimulation, compared to when perceiving speech-in-noise alone, resulting in a significant group mean benefit of 6dB. This effect is quite remarkable if one considers that: (1) every increase of 3 dB represents a doubling of sound intensity and every increase of 10 dB represents a doubling of the perceived loudness (Stevens, 1957), (2) no training aimed at matching audition and touch was applied, (3) the direction of the effect was consistent across all individuals. To our knowledge, this is the first study to show such a systematic improvement across participants during multisensory audio-tactile perception of speech-in-noise.

Two other recent studies tested a similar question, though both obtaining somewhat less convincing results (Huang et al., 2017; Fletcher et al., 2018). Specifically, Huang and colleagues (2017) tested cochlear implant patients and showed an improvement in speech-understanding when adding low-frequency vibration corresponding to the f0 extracted from the presented sentences, without any specific training. The improvement was, however, significantly more modest (mean 2.2 dB) than in our experiment. Fletcher and colleagues (2018) complemented vocoded speech with tactile vibration in normal hearing subjects using their native language. After a dedicated training, participants showed an improvement of only 10% in sentence-in-noise recognition. In addition, Fletcher and colleagues used fixed SNRs, as opposed to an adaptive procedure for SNR estimation as we applied in our study. The latter is more optimal as it matches task difficulty across participants preventing the floor and the ceiling effects (Levitt, 1978). We hypothesize that the fact that we used non-native and thus less intelligible auditory input enhanced the effects of the inverse effectiveness law. Nevertheless, future studies should investigate which experimental settings used in the other two works made their outcomes less impressive.

The findings of the current study are especially interesting if embedded within the more general literature regarding the benefits in behavior when using SSDs. Most SSD solutions have been thus far aimed for the blind population, with visual input conveyed via audition or touch (Bach-Y-Rita et al., 1969; Meijer, 1992; Bach-y-Rita et al., 2004; Abboud et al., 2014). All these works showed that blind participants can perform a variety of ”visual” tasks with SSDs but only after an extensive training (e.g., Striem-Amit et al., 2012; Bach-Y-Rita, 2004; Chebat et al., 2011; Chebat et al., 2015). Training was also needed to observe benefits in the sighted popultion learning to perceive visual information via SSDs (e.g., Amedi et al., 2007; Hamilton-Fletcher et al., 2016).

Why did the current SSD set-up show such an automatic effect? One possible reason might be that the aforementioned SSD solutions use quite complex algorithms to convey a variety of visual features simultaneously, such as shape, location and even colors of objects (Striem-Amit et al., 2012; Maidenbaum et al., 2014; Levy-Tzedek et al., 2014; Abboud et al., 2014). Indeed, when the amount of information was reduced, also visual SSDs required shorter training programs (e.g., for navigation with the EyeCane; Maidenbaum et al., 2014). The current auditory-to-tactile SSD transfers “only” the fluctuating frequency information, and therefore maybe requires less cognitive effort from the user, ultimately speeding the learning process.

In addition, in many everyday situations the auditory and the tactile system often work together (e.g. when a mobile phone is ringing and vibrating at the same time or when listening to music), thus potentially increasing the chances of an immediate multisensory integration. Furthermore, audition and tactile vibration share several physical properties. In both types of stimulation information is conveyed through mechanical pressure generating oscillatory patterns, ultimately constructing frequency percepts (e.g., Soto-Franco and Deco, 2009; Ro et al., 2013; Good et al., 2014; Auer et al., 2007). Moreover, within a certain frequency range, the very same oscillatory pattern can be perceived simultaneously by the peripheral receptors of both sensory modalities (i.e., the basilar membrane of the cochlea and the skin, respectively; e.g., Von Békésy, 1959; Gescheider, 1970; Soto-Franco & Deco, 2009; Heimler et al., 2014). Finally, there seems to be a privileged neuronal coupling between auditory and sensory-motor brain regions (Suarez et al., 1997; Burton et al., 2004; Kayser et al., 2005; Beauchamp et al., 2008; Bellido et al., 2018; Arenada et al., 2017; Igucji et al., 2007; Caetano and Jousmaki, 2006; Auer et al., 2007; Fu et al., 2003; Hoefer et al., 2013). All these similarities probably contribute to why both hearing and deaf participants consistently report to perceive simultaneous vibrotactile and auditory stimulation as an interleaved signal (Wilson et al., 2012; Bernstein et al., 1998; Russo et al., 2012). Future studies may further elucidate the easiness and intuitiveness of SSD learning in multisensory contexts, thus unraveling rules of multisensory integration depending on the used sensory modalities.

Our results carry important implications for further research, as well as possible clinical and practical solutions. In rehabilitation, benefits of unisensory task-specific trainings for recovery of sensory or cognitive functions have been demonstrated in a number of domains, such as in the ageing brain (Cheng et al., 2017; Smith et al., 2009; Anderson et al., 2013; Bherer, 2015) or following brain lesions occurred during adulthood (Kerr et al., 2011; Xerri et al., 1998). In addition, successful multisensory interventions have been reported in patients after stroke (Tinga et al., 2016), as well as in hemianopia (i.e., with loss of vision in only one part of the visual field and/or one eye) and spatial neglect (Keller and Lefin-Rank 2010; Bolognini et al., 2017). Importantly, improvements in speech skills have been reported in CI patients following extensive audio-visual trainings combining the restored auditory input with speech-reading or sign language (Leybaert and LaSasso, 2010; Fu and Galvin, 2007; Stevenson et al., 2017).

The present work extends the current approaches in rehabilitation even further, by showing benefits of multisensory stimulations on performance even without any training. This is an important advantage because training programs are generally time consuming to both patients and caregivers and discourage potential users to adopt SSDs in everyday life (Elli et al., 2014). Our minimal set-up that requires only two fingertips is also especially attractive in relation to the more cumbersome tactile SSDs described in literature (Kupers and Ptito, 2011; Novich and Eagleman, 2015).

We suggest that the designed auditory-to-tactile SSD could serve as a valid assistive technology for several populations with various degrees of hearing deficits, either with or without a hearing aid/a cochlear implant. As an example, CIs provide great benefits for understanding speech in quiet but do not transmit the low-frequency cues effectively, thereby making speech comprehension in noise extremely hard. The latter is related to inherent technological limitations of the device, i.e. the restricted number of independent frequency-specific channels, spread of excitation, as well as the fixed rate of pulses delivered to the auditory nerve (Wilson, 2012; Cullington and Zeng, 2010). Therefore, conveying the missing low-frequency information through vibration seems a promising approach to improve speech understanding in this population (e.g., Huang et al., 2017). This is even more convincing, if one considers patients with partial deafness who use an electro-acoustic hearing prosthesis, combining electrical hearing via partially inserted CI electrode array with low-frequency acoustic signal delivered naturally or via a HA in the same ear. This population consistently shows superior speech performance in noise, when compared to profoundly deaf users of CIs, probably due to the access to low-frequency acoustic cues (Skarzynski et al., 2003; Gstoetner et al., 2004; Gifford et al., 2013; Skarzynski et al., 2009; von Illberg, 2011; Zhang et al., 2010).

Although the current results show an immediate benefit of speech in noise perception with audio-vibratory stimulation, we suggest that perhaps some specific training might be required when aiming at achieving an improvement in unisensory auditory performance. In this case, we predict that multisensory stimulation might facilitate understanding of the auditory signal, ultimately boosting its further recovery (see Isaiah et al., 2014 for a successful example in deaf ferrets). Such an approach may be most effective if training is started before cochlear implantation by delivering stimulation via touch that, nevertheless, maintains features typical of the auditory modality (i.e. periodic information with fluctuating frequency and intensity). We suggest that such a training could prepare the auditory cortex for future reafferentation of its natural sensory input (Heimler et al., 2018; see analogous suggestions for the blind population in Heimler and Amedi in press). Interestingly, even though conclusive research data supporting this approach are still lacking, some hearing aid manufacturers already propose solutions delivering vibration to profoundly deaf individuals prior to cochlear implantation (see, e.g., http://www.horentekpro.com).

Importantly, we also predict that our proposed SSD might serve as an assistive aid for the elderly population whose both cognitive and sensory abilities (and most often hearing) have deteriorated (Amieva et al., 2015; Murray et al., 2018). Solutions for the elderly seem crucial in the nowadays aging society, with hearing loss becoming an epidemic. Since this population might have issues with complying with extensive training programs, we believe that our intuitive set-up might prove helpful.

Finally, we see the potential use of our device to support second language acquisition, as we have shown in the current experiment that users do benefit from vibration when trying to decipher sentences in a non-native language. The device we have developed provides low-frequency tactile stimulation which conveys much of the cues that have been shown hardest to detect in a foreign speech signal, such as e.g. duration, rhythm and voicing (Kuhl et al., 1992; Riviera et al., 2005). Interestingly, multisensory approaches have already been successfully applied in foreign language teaching, such as e.g. within the Multisensory Structural Language Education framework, which combines visual, auditory and tactile modalities (Schams and Seitz, 2008; Lidestam et al., 2014). Other possible applications of our setup in normal hearing subjects (but also the hearing impaired) can include voice rehabilitation, improving appreciation of music, as well as assistance when talking on the phone.

Future studies should investigate the feasibility of our set-up for all the aforementioned applications and potentially implement slight modifications to adapt it to the needs of the specific population of interest. Our aim in describing possible applications of our multisensory audio-tactile set-up was to highlight its flexibility and therefore, the wide spectrum of rehabilitative conditions it can be used for. On a final note, we are planning to elucidate the neural correlates of the demonstrated improved speech understanding during multisensory stimulation. Indeed, our SSD has been designed to be MR-compatible, thus making it immediately suitable for testing in the scanner with functional magnetic resonance imaging (fMRI).