Prosodic modulation in the babble of cochlear implanted and normally hearing infants: A perceptual study using a visual analogue scale

Adult raters

Thirty naive native Dutch-speaking adults (mean age 23 years, range 18–42 years) made the prominence judgements in the perceptual experiment. The raters were naive since they were not informed about the purpose of the study. Moreover, they were not familiar with CI and NH children’s speech. None of the raters reported problems with hearing or health, nor developmental disorders.

Experimental stimuli

For the present study, all eligible disyllabic utterances were selected from the CCLC. Each recording lasted 60–90 minutes and the portions of the recordings in which the child was vocally most active were selected for further processing. These selections were restricted to approximately 20 minutes. From the selections only disyllabic babbled utterances were selected when they consisted of two vocalic phases minimally separated by a clear consonantal phase. Moreover, disyllabic babble was included if there was no concurrent speech or noise and if they were not produced with a creaky, breathy, excessively loud or whispery voice. A total of 524 disyllabic babble utterances (165 CI babble, 359 NH babble) was included as stimuli for the perceptual experiment (see Table 2).

Since maturation effects might influence the production prosodic modulation, two variables were investigated in this study. The first maturation variable is the infants’ babble experience. Since infants start babbling at different ages and since there is a large discrepancy between the chronological age at onset of babbling of the CI and NH infants, it is more meaningful to take into account the chronological age counted from the onset of babbling onwards. More specifically, the month in which the child started babbling was labelled as 0, the first month after the onset of babbling as 1, the second month as 2, etc. The second maturation variable is the phonological complexity of the babble. The canonical babble utterances were categorised into reduplicated and variegated babble. The prelexical utterances in the recordings of the CCLC are transcribed in broad phonemic categories (detailed information on the transcription is provided in Molemans, 2011; Van den Berg, 2012; Van Severen, 2012). These broad phonemic transcriptions were used to automatically categorise babble into variegated or reduplicated babble. When the same segmental content was repeated in the two syllables the utterance was classified as reduplicated (e.g. ‘baba’). Whereas in variegated babble the syllables differ at the segmental level by consonantal and/or vocalic variation (e.g. ‘baka’). This means that there were two intermediary steps: the sound files were first transcribed as part of a previous research project (Molemans, 2011; Van den Berg, 2012; Van Severen, 2012), and then these transcriptions were automatically coded as reduplicated or variegated babble. Of the disyllabic babble included in the present study 44% was reduplicated babble (reduplicated CI babble: 45%, reduplicated NH babble 43%) and 56% of the stimuli was variegated babble (variegated CI babble: 55%, variegated NH babble: 57%). Table 2 contains an overview of the number of reduplicated and variegated babble utterances per child.

As a reliability check 20% of the automatically categorised stimuli (106 babble: 53 NH and 53 CI utterances) was manually recategorised into reduplicated and variegated babble by the first author. The raw sound files were used to judge whether the babble has repeated segmental content or not. In 91 out of 106 utterances categorisation was identical, resulting in an agreement of 86% and a Cohen’s kappa of 0.68.

Experimental design

In the rating task, the raters indicated the more prominent syllable. Moreover, by moving the slider on the VAS they indicated the relative degree of prominence of the syllable. Studies have shown that perceptual ratings on a VAS are informative for capturing the perceptual effects of fine phonetic detail in child speech (Julien & Munson, 2012; McAllister Byun, Harel, Halpin, & Szeredi, 2016; Munson et al., 2012). Munson and Carlson (2016) showed that a VAS is a useful method for experimental and observational studies of within-category speech sound perception as it allows gradual responses.

The VAS used in the present study was a sliding bar of which a screencast can be found in supplementary material 1 and a print screen of three different rating positions can be found in Figure 1. Two circles above the sliding bar served as a visualisation of the two syllables of each stimulus. The initial position of the slider was the midpoint of the VAS (Figure 1(a)). When sliding to the left, the left circle became larger while the right one became smaller, indicating a more prominent first syllable (Figure 1(b)). In order to indicate a more prominent second syllable, the slider was moved to the right, creating a larger right circle and a smaller left circle (Figure 1(c)). The position of the slider moved along a scale ranging from 0 (extreme left) to 100 (extreme right). Thus, a range between 0 to 49 means a more prominent first syllable (the lower the number, the more prominent the first syllable). Point 50 indicates equally prominent syllables, and a position in the range from 51 to 100 indicates a more prominent second syllable (the higher the number, the more prominent the second syllable).

OPEN IN VIEWER

Figure 1.

Print screens of three different rating positions on the sliding bar. (a) Equally prominent syllables. (b) Prominent first syllable. (c) Prominent second syllable.

Experimental procedure

Before the start of the experiment, the raters went through an information phase and a familiarisation phase. During the information phase, the purpose of the experiment was explained in a section with instructions (see supplementary material 2) and raters were told that they could replay an utterance two times (i.e. each utterance could be played maximally three times).

The familiarisation phase consisted of an example and trial phase. During the example phase utterances with clear prominence relationships (i.e. clear equal prominence or a clearly prominent first or second syllable) were presented. The raters saw an animation of six clear examples of how the sliding bar was moved to the appropriate position. During the trial phase the raters had to indicate the prominence on similarly clear utterances. The task in the trial phase was the same as in the actual experiment, i.e. ‘indicate the prosodic modulation and stress pattern on the VAS’. The description and design of the example and trial utterances can be found in supplementary material 3. In the actual experiment 524 disyllabic babble utterances produced by CI and NH infants were presented in random order to the raters. The total duration of the experiment was approximately 60 minutes. For every stimulus we recorded the rate and how many times raters listened to the stimulus.

The experimental interface was created by means of Functional Reactive Programming (Czaplicki, 2012), a tool developed for creating responsive graphical interfaces. The programming language used to create the front end was Elm, and Haskell was used for the back end. The experiment was designed as an online application and ran on an iMac (2.9 GHz Intel Core i5, on a 21.5 inch screen) in a quiet room for all 30 participants. During the experiment the selected disyllabic babble utterances were presented one by one to the raters via SONY MDR-1R headphones. Every stimulus was preceded and followed by one second of surrounding silence.

Generalised mixed models

To analyse the data generalised mixed models (GLMM) (Baayen, 2008) were run in R (R Core Team, 2013) with the lme4 package (Bates, Maechler, Bolker, & Walker, 2014). GLMM is an appropriate tool to examine data that are hierarchically structured: the stimuli in the present study (N = 524) are embedded in different infants (N = 18) and all utterances are rated by different raters (N = 30). Moreover, GLMM is robust to missing data and different numbers of data points per participant. These models consist of two parts: a random and a fixed part. The random part takes into account the variation caused by the random effects, e.g. the variation between individuals. The fixed part consists of the independent variables that may have an effect on the dependent variables, i.e. participant group (CI or NH babble), utterance type (variegated or reduplicated babble) and age at onset of babbling.

The statistical models are constructed iteratively: first the random effects are added one by one and then the fixed effects are added one at a time. After adding a random or fixed effect the new model is compared to the previous model by means of likelihood ratio tests. Only the effects that improve the fit of the model are included. The best fitting model explains the largest amount of variance with the smallest set of predictors.

The dependent variables

The ratings on the VAS resulted in a continuous score from 0 to 100, representing the degree of prosodic modulation. A set of analyses with this continuous dependent variable were conducted (see supplementary material 4). However, in neither of the analyses were the residuals normally distributed, meaning that a GLMM analysis of the continuous variable was not permitted. As an alternative the data were recoded into binomial dummy variables that are informative of the rating zone on the VAS instead of using the raw score (from 0 to 100) as dependent variable.

As shown in Figure 2, the collected ratings form a multimodal distribution: a distribution on the left and right side of the VAS and a peak at the midpoint of the VAS. These variables were ‘trochee’, ‘iamb’, ‘midpoint’. ‘Trochee’ (score: 0–49) represents the utterances that are rated as having a prominent first syllable, which is the predominant Dutch pattern. ‘Iamb’ (score: 51–100) represents the utterances that are rated as having a prominent second syllable. ‘Level’ (score: 50) represents the utterances that are rated at the midpoint of the axis. The midpoint was also the default position of the sliding bar before being moved by the raters. When an utterance is rated at the midpoint this is an indication that there is no clear prosodic differentiation between syllables.

OPEN IN VIEWER

Figure 2.

Distribution of all ratings per participant group.

The binomial dependent variables were analysed by means of logistic regression analyses in the form of GLMM. The estimates (E), standard errors (SE), t- and p-values of the fixed effects of the four best fitting models (i.e. one for every dependent variable) are reported in the results section. These values are computed in logits, but to facilitate the interpretation they are converted to the likelihood or probability (expressed in percentages) to be rated in this area of the VAS.

The aim of the analyses was to investigate for every rating zone whether (1) CI utterances were more likely to be rated in the respective zone as compared to NH utterances and vice versa, (2) chronological age at onset of babbling has an impact and (3) whether reduplicated babble is more likely to be rated in a particular zone as compared to variegated babble and vice versa.

The first research question focuses on the prosodic modulation in the babble of CI and NH infants. Three analyses were performed to answer this research question. The first analysis looked at the probability of an utterance being rated at the midpoint of the VAS (score: 50). Since the midpoint of the VAS is considered to be the zone that represents the least prosodic modulation, this analysis aims to answer the question whether the babble of infants with CI is more likely to be rated in this zone. The analysis controls for a possible effect of babble experience (i.e. chronological age counted from the onset of babbling) and phonological complexity (i.e. the babble type: reduplicated versus variegated babble) on prosodic modulation.

The second analysis that investigates raters’ perception of prosodic modulation focused on those utterances that were indicated as having the most modulated prominence. This analysis aims to answer the question whether the utterances of the NH group are more likely to be rated as having a clearly prominent syllable than the utterances of infants with CI. Again age at onset of babbling and the babble type are added as predicting variables.

The utterances that are rated more towards the extremes of the VAS are considered to have the most prosodic modulation. However, looking at the extremes of the raw VAS (e.g. the stimuli rated in zone 0–20 and zone 80–100) would not be informative since individual raters may well apply different strategies to indicate prominence on a VAS: some use the entire scale (from 0 to 100), but most of them do not use the entire VAS and tend to centre their ratings in a smaller range (e.g. from 19 to 82). Therefore it is more informative to convert the ratings of every rater to z-scores. The z-scores are used to determine the extreme ratings per individual rater. A stepwise approach was used to determine these extreme ratings. First of all the range of the ratings of each rater was determined: e.g. for rater 1 the leftmost z-score was −2.33, whereas the rightmost rating was a z-score of 3.01. In a next step this range was divided into five equal parts in order to determine the 20% leftmost and 20% rightmost parts of the VAS for each individual rater: e.g. for rater 1 the extremes range from −2.33 and −1.26 (left) and 1.94 and 3.01 (right). The utterances rated in the first and fifth part of the selected range were coded as being rated at the extremes of the scale: e.g. for rater 1 the left extreme included 45 utterances and the right extreme 24 utterances. The dependent variable in the third analysis was the number of times an utterance was replayed by the raters. If the prosodic modulation of a stimulus is hard to judge, raters are expected to replay it more often than when prominence is clear.

The second research question was whether listeners perceive the babble to be predominantly trochaic and whether there is a difference between CI and NH infants in this respect. This was investigated by two analyses. The first analysis looked at the probability that an utterance was rated on the left, trochaic side (score: 0–49) of the VAS. The second analysis looked at the probability that an utterance was rated on the right, iambic side (score: 51–100) of the VAS. Both analyses control for possible maturation effects by adding age at onset of babbling and utterance type to the models.

Midpoint

The first analysis looked at the probability of an utterance being rated at the midpoint of the VAS (score: 50). Since the midpoint of the VAS is considered to be the zone that represents the least prosodic modulation, this analysis aims to answer the question whether the babble of infants with CI is more likely to be rated in this zone. Moreover, it is investigated whether maturation variables (i.e. utterance type and babble experience) influence the likelihood of babble being rated with no modulation.

The best model with ‘midpoint’ as dependent variable had participant group and utterance type as fixed effects (supplementary material 5). Adding age at onset of babbling did not improve the fit model. The random part controlled for the variance of the different infants, utterances and raters. The intercept of the model was estimated at −3.222 logits or 4% (SE = 0.297, z = −10.833, p < 0.001), meaning that babble is significantly less likely to be rated at the midpoint, than anywhere else on the VAS. This finding is not surprising since this variable focuses on only one point on the VAS, in comparison to all other points. The finding indicates that the cursor was very likely being moved when making a judgement on the prominence pattern of babble. Regarding the question whether there is a difference between CI and NH infants, the results show that CI utterances are significantly more likely to be rated at the midpoint: reduplicated CI babble utterance has a probability of 3.5% of being rated at the midpoint, whereas the probability is only 2.6% for reduplicated NH utterance (−3.222 – 0.401 logits; E = −0.401, SE = 0.188, z = −2.127, p = 0.033). Reduplicated babble of CI infants is judged as having less modulated prominence. The same holds for variegated babble. Variegated CI babble utterance has a probability of 2.2% (−3.222 – 0.527 logits; E = −0.527, SE = 0.136, z = −3.878, p < 0.001) to be rated at the midpoint and that of a NH infant has a probability of 1.5% (−3.222 – 0.401 + 0.527 logits) to be rated at the midpoint. All these differences are shown to be significant. These results indicate that babble of CI infants is significantly more likely judged as having no clearly prominent syllable. A similar result is found for the difference between variegated and reduplicated babble. In both infant groups reduplicated babble had a higher probability (i.e. 3.5% for a CI- and 2.6% for a NH-reduplicated babble) to be rated at the midpoint compared to segmentally variegated babble (i.e. 2.2% for a CI- and 1.5% for a NH-reduplicated babble). The age at onset of babbling did not improve the fit of the statistical model, meaning that it did not impact the probability of being rated with no clearly stressed syllable.

It can be concluded that babble of CI children is more likely to be rated as having no clearly prominent syllable. Regarding the effect of the maturation variables it can be concluded that phonetic complexity influences prosodic modulation: reduplicated babble is more likely to be rated as having no clearly prominent syllable and thus as having less prosodic modulation, irrespective of the infant group or the age at onset of babbling at which they are produced.

Extremes

The best model with ‘extreme’ as dependent variable had participant group and utterance type as fixed effects (supplementary material 6). Again the random part controlled for the variance of the different infants, utterances and raters and the fixed effects were participant group and babble type. The intercept of the model was estimated at −0.826 logits or 30% (SE = 0.116, z = −7.133, p < 0.001), indicating that babble is significantly less likely to be rated at the extremes than anywhere else on the VAS. Regarding the question whether there is a difference between CI and NH infants in prosodic modulation in babble, the results indeed show that NH utterances are significantly more likely to be rated at the extremes: there is a probability of 30% for reduplicated babble of a CI infant being rated at the extremes, whereas the probability that reduplicated NH babble is rated at the extremes is 36% (−0.826 + 0.255 logits; E = 0.255, SE = 0.129, z = 1.981, p = 0.048). When looking at variegated utterances it appears that variegated CI babble has a probability of 38% (−0.826 + 0.308 logits; E = 0.308, SE = 0.084, z = 3.667, p < 0.001) and that variegated NH babble has a probability of 43% (−0.826 + 0.255 + 0.308 logits) to be rated at the extremes. All these differences are shown to be significant. The age at onset of babbling has no significant effect on the probability of being rated at the extremes (E = 0.004, SE = 0.010, z = 0.426, p < 0.670).

It can be concluded that the canonical utterances of the NH infants are more likely to be rated at the extremes of the ratings compared to the CI infants’ utterances. This is an indication that the prosodic modulation within NH babble is more salient. This difference between groups is apparent in both variegated and reduplicated babble. Regarding the effect of the maturation variables it can be concluded that phonetic complexity influences prosodic modulation: variegated babble is more likely to be rated at the extremes than reduplicated utterances irrespective of the infant group or the age at onset of babbling at which they are produced.

Analysis of number of replays

A third analysis investigates how many times an utterance was replayed by the raters. The random effects again account for the variance caused by the different infants, raters and stimuli. The fixed effects in this analysis were infant group, babble type and extreme (i.e. part of the extreme ratings or not). The fixed effects in the best fitting model were participant group and the variable ‘extreme’ (i.e. the utterance is rated at one of the extremes or not). Adding babble type did not improve the fit of the model, meaning that there was no significant difference in the number of replays of reduplicated versus variegated babble. The results of this analysis are displayed in supplementary material 7.

The intercept of this statistical model was estimated at 0.423 (SE = 0.046, t = 9.156, p < 0.001), meaning that an average CI utterance that is rated anywhere on the VAS except at the extremes is replayed on average 0.423 times. Utterances of the NH infants are repeated significantly less than those of the infants with CI (E = −0.058, SE = 0.017, t = −3.349, p = 0.004), indicating that it was easier to judge the prominence of NH babble than CI babble. The stimuli rated at the extremes were repeated significantly less than those on the rest of the VAS (E = −0.126, SE = 0.009, t = −14.613, p < 0.001). The age at onset of babbling did not improve the fit of the statistical model, meaning that it did not impact the number of replays. These results are supporting evidence for the findings of the VAS analysis that showed that NH stimuli are rated as having clearer prominence and as those utterances that were rated at the ends of the continuum are repeated less often and are thus easier to judge.

Trochee

The first analysis looked at the probability that an utterance was rated on the left, trochaic side (score: 0–49) of the VAS. The random part of the best fitting model controlled for the variance explained by participant identity (i.e. the 18 infants), utterance identity (i.e. a unique number for each of the 524 babble utterances) and rater identity (i.e. the 30 adult raters). The only fixed effect required in the best fitting model was infant group (CI versus NH). Adding babble type (reduplicated versus variegated babble) and age at onset of babbling as fixed effects did not improve the fit of the model. An overview of the best fitting model with ‘trochee’ as dependent variable can be found in supplementary material 8.

The intercept of the model was estimated at 0.157 logits or a likelihood of 54% (SE = 0.268, z = 0.586, p = 0.558). This positive intercept indicates a higher probability for utterances to be rated as a trochee than at any other zone on the VAS. However this effect is not significant. There is no significant difference between the ratings of the CI and NH babble (E = −0.041, SE = 0.345, z = −0.120, p = 0.904). These results show that babble is not significantly more likely to be rated at the trochaic side of VAS, nor that the NH infants are more likely to produce trochees than CI infants or vice versa. The babble type did not improve the fit of the statistical model, indicating that the segmental variation of the utterance does not lead to more ratings on the trochaic side of the VAS. Age at onset of babbling did not improve the fit of the model either, meaning that there is no age effect on the probability to be rated as a trochee. Regarding the second research question it can be concluded that the predominant trochaic pattern is not significantly more often perceived by raters and this is the case for both infant groups.

Iamb

The aim of the second analysis was to see whether there is a difference between groups and babble types regarding the probability to be rated on the right side of the VAS (score: 51–100). The same fixed and random effects as in the analysis with trochee as dependent variable were added. The statistical model with the best fit (supplementary material 9) had participant identity, utterance identity and rater identity as random effects and participant group and utterance type as fixed effects. The intercept was estimated at −0.929 or a likelihood of 28% (SE = 0.320, z = −2.905, p = 0.004). This significant negative intercept indicates that utterances are less likely to be rated on the right side of the VAS than in all the other rating zones together. As for the analysis of trochees, there was no significant difference between the ratings of the CI and NH babble (E = 0.094, SE = 0.389, z = 0.243, p = 0.808) suggesting that there is no difference between groups in the probability that an utterance is rated as having a more prominent second syllable. The analysis showed that variegated babble is significantly more likely to occur at the iambic side of the VAS (E = 0.401, SE = 0.192, z = 2.088, p = 0.037). For the CI infants the likelihood of reduplicated babble occurring on the iambic side of the VAS is 28% (−0.929 logits) whereas it is 37% for variegated babble (−0.929 + 0.401 logits). For the NH infants the likelihood of reduplicated babble occurring on the iambic side of the VAS is 30% (−0.929 + 0.094 + 0.401 logits) and 39% for variegated babble (−0.929 + 0.401 logits). It can be concluded that variegated babble is more likely to be rated as having a more prominent second syllable than babble with a repetitive segmental content, irrespective of the children’s hearing status.