Abstract
The Early Communication Indicator (ECI) is a progress monitoring measure designed to support intervention decisions of the home visitors and early educators who serve infants and toddlers. The present study sought to add to the criterion validity claims of the ECI in a large sample of children using measures of language and preliteracy not previously investigated. Early Head Start service providers administered and scored ECIs quarterly for infants and toddlers in their caseload as part of standard services. In addition, a battery of language and early literacy criterion tests were administered by researchers when children were 12, 24, 36, and 48 months of age. Analyses of this longitudinal data then examined concurrent and predictive correlational patterns. Results indicated that children grew in communicative proficiency with age, and weak to moderately strong patterns of relationship emerged that differed by ECI scale, age, and criterion measure. The strongest positive patterns of relationships were between Single Words and Multiple Words and the criterion at older ages. Gestures and Vocalizations established a pattern of negative relationships to the criterion measures. Implications for research and practice are discussed.
Language and social communication are desired outcomes in early childhood (Priest et al., 2001). Communication and language skills enable children to express their wants and needs, and they are predictive of young children’s emergent literacy skills in preschool and subsequent success learning to read in school (e.g., NICHD Early Child Care Research Network, 2005; Shanahan & Lonigan, 2008). This knowledge is reflected broadly in the expected outcome goals of early intervention programs serving young children (National Early Childhood Accountability Task Force, 2007). Communication outcomes are expected by IDEA Part C programs serving infants and toddlers with developmental delays and disabilities (ECO Center, 2011; Individuals With Disabilities Education Improvement Act, 2004), and language and literacy are school readiness goals for infants and toddlers served by Early Head Start (Early Head Start National Resource Center, 2012). As a recommended practice (Division for Early Childhood, 2014; “Frameworks for Response to Intervention in Early Childhood: Description and Implications,” 2014), programs and practitioners are increasingly expected to use children’s individual performance data to monitor progress, guide provision of the intervention services they receive, and report results for the children and families served. Thus, the validity evidence of measures used to inform these decisions are highly important.
Currently, few progress monitoring measures of growth in early expressive language skills are available to infant/toddler service providers to support this kind of individualization and program accountability (Akers et al., 2015; Greenwood, Carta, & McConnell, 2011). Compared with traditional assessment (Caesar & Kohler, 2008), progress measurement is focused on the growth in children’s performance based on frequent, brief assessments of a few targeted skills that estimate individual children’s proficiency at specific points in time as well as the rate of improvement over time (Fuchs & Deno, 1991). Progress monitoring measures are specifically designed for use by home visitors, early educators, early interventionists, and developmental specialists responsible for determining when children are not making adequate progress, plan individualized change in intervention, and monitor progress in response to intervention (Deno, 1997).
The Early Communication Indicator (ECI) was developed to address the need for monitoring children’s acquisition of expressive language during the 6 to 42 months age span (Greenwood et al., 2011; Priest et al., 2001). The ECI measures the rate of growth in four foundational communication skills for children that are alterable given early intervention. They are gestures, vocalizations, and single and multiple word utterances representing the prelinquistic and linguistic domains of language (Vallotton & Ayoub, 2010; Vandereet, Maes, Lembrechts, & Zink, 2011).
The ECI is a play-based assessment typically administered in the home, center, or other child care setting (e.g., pediatric clinic) by a practitioner who observes the child in play with a familiar adult partner using a standard toy set (Walker & Carta, 2010). The assessor counts the occurrence of each ECI key skill during a standard 6-min administration. The data are entered into a website where rate per minute scores are automatically calculated and charted for each individual child. From these data, level and rate of improvement (slope) are calculated for each key skill and a composite rate of total communication and compared with benchmarks for interpretation and decision making (Walker & Buzhardt, 2010). For universal screening, the ECI is administered once quarterly or monthly for progress monitoring.
The ECI is intended for use with all children in the age range. Children included in ECI validation research to date have been primarily from low-income eligible families in Early Head Start and children with special needs (Individualized Family Service Plans [IFSPs]) served in IDEA Part C (Infant/Toddler) services. With respect to ruling out subpopulation measurement bias, the ECI has been shown to yield equivalent estimates for boys and girls, and for those whose home languages are English, Spanish, and other languages. However, the ECI does discriminant level and slope differences between children with versus without an IFSP (Greenwood, Walker, & Buzhardt, 2010), as would be expected given that the vast majority of children this age receiving IFSP services are targeting speech and language delays (Hebbeler et al., 2007). We reported that children with an IFSP started using words and sentences ~5 months later than typically developing children, and in the meantime continue using gestures and vocalizations (not recognized words) in social communication months longer that typically developing children (Greenwood et al., 2010).
A number of validity claims of the ECI have been reported based on ECI data collected by practitioners as designed and not by researchers. For example, one report found that the pattern of change in the ECI’s key skills was consistent with a theoretical continuum of growth in expressive language (construct validity) evidenced by an overlapping sequential progression in initial growth and decline of gesture and vocalization skills over the 6- to 36-month age span in synchrony with growth in single and multiple word utterances after 12 months (Greenwood, Walker, et al., 2013). Invariance testing (Widaman, Ferrer, & Conger, 2010) supported the claim that the ECI measures the same constructs in different time-displaced samples (Greenwood, Buzhardt, Walker, McCune, & Howard, 2013). Benchmark growth trajectories have been reported for children served in Early Head Start programs for use as comparatives in intervention decision making (Greenwood et al., 2010). Additional reports demonstrated the ECI’s sensitivity to the child effects of parent-implemented, language intervention (treatment validity) (e.g., Buzhardt et al., 2018).
However, the criterion validity of ECI has not yet been investigated beyond initial development in a small sample (Luze et al., 2001; Greenwood & Walker, 2010). The purpose of the study was to update the ECI’s claim of concurrent validity and investigate the measure’s predictive validity in a 4-year longitudinal study. Concurrent validity is important to one’s confidence that the ECI actually measures young children’s expressive communication skills. Predictive validity is important to the claim that one’s future performance in language and literacy depends on current performance. Both claims are important to infant-toddler programs seeking to deliver and report the impact of the intervention services they provide. The research also sought to generate new knowledge of the technical adequacy of language measures for very young children that has been less than those used with older children (e.g., Larson, 2016). Because of the documented changes in children’s early communication skills with age (Greenwood, Buzhardt, et al., 2010; Greenwood et al., 2013), we investigated the following validity claims within and across age cohorts at 12, 24, 36, and 48 months of age.
We hypothesized finding growth in ECI Key Skills and composite Total Communication rate over time and age based on prior research.
We hypothesized replicating previous concurrent relationships between the ECI Total Communication rate and discovering new findings between the ECI Key Skills and criterion measures.
We hypothesized finding stronger concurrent relationships among measures in older children with greater use of spoken language skills compared with younger children.
We hypothesized finding that the ECI measures of early language at 12, 24, and 36 months would predict later language and early literacy up to 48 months.
We hypothesized finding that the ECI’s prelinguistic skills in younger children would predict linguistic skills in older children based on recent literature.
Method
Participants
Programs
Participants were recruited and enrolled from the population of Early Head Start (EHS) programs (N = 15) in one Midwestern state. EHS programs served low-income eligible infants and toddlers and their families (Office of Head Start, 2012). In addition, 10% of the EHS openings were available for children receiving Part C early intervention services under the Individuals With Disabilities Education Act (IDEA). The service model in the majority of programs was home-visiting, wherein families were visited in homes. Programs had previously adopted the ECI and were using it universally as an accountability measure and in some cases for universal screening and progress monitoring (Greenwood, Buzhardt, et al., 2013).
We recruited program directors to participate in an additional battery of language and early literacy criterion measures administered by our research team. Seven of the 15 directors/programs agreed to participate. Overall, a total of 71 EHS practitioners in these programs were trained, certified, and conducted ECI administrations with the children and families they served. Each assessor conducted an average of 10 ECIs, ranging from 1 to 44.
Children
Children and families (caregivers) were recruited to receive the battery of language and early literacy criterion measures. Because program and institutional review board (IRB) procedures prohibited the researchers from directly contacting parents, program staff provided parents with initial information about the study. Interested parents/caregivers contacted the researchers for information regarding requirements and benefits. Caregivers (N = 749 with 1,007 children) contacted the researchers and were provided information. Consent was obtained from 424 (57%) caregivers who enrolled their children (N = 570) in the study.
The mean age of children at the first assessment was 15.6 months (SD = 3.6). The distribution of children by gender was 248 (43.5%) female, 267 (46.8%) male, with 55 (9.6%) unspecified. Forty-eight (8.4%) had an IFSP, 466 (81.8%) were typically developing, and 56 (9.8%) did not specify. The distribution of languages spoken at home was 81.3% English, 21.6% Spanish, and 1.5% other (e.g., Arabic, Vietnamese, Somali, etc.). Given that only about half of interested caregivers consented and enrolled in the study, we explored the possibility of sample selection bias between enrolled versus not-enrolled families. There were no significant differences in gender, first ECI age, disability status, primary language, and mean ECI TC.
Design and Measurement
A longitudinal, cross-sectional design was used. In addition to the ECI assessments collected by EHS staff, participants were assessed with a sociodemographic survey at enrollment and with the language and literacy criterion measures at 12, 24, and 36 months.
Sociodemographic measures
Caregivers completed a 16-item sociodemographic survey developed by the researchers (see Table 1). For the child, birth date, age, gender, race/ethnicity, and disability status (IFSP) were collected. For the family, the caregivers’ relationship to the children, marital status, language spoken at home, size of family, highest level of education, and family income were collected.
Measures Administered Over Time.
Completed by parent/caregiver. bAdministered by Early Head Start (EHS) staff. cAdministered by research staff.
Predictor variables
The ECI is a 6-min, play-based measure of children’s growth in expressive communication (Carta, Greenwood, Walker, & Buzhardt, 2010). Brief definitions of the ECI’s key skills follow: Gestures were defined as physical movements made by the child in an attempt to communicate with the partner. Examples included giving or taking an object from the play partner, shrugging shoulders, pushing away, pointing, showing, nodding head, and so forth. Vocalizations were nonword verbal utterances voiced by the child. Examples included babbling, cooing, “ah,” “da,” animal sounds, and so forth. Single Words were either single voiced or signed words (i.e., sign language) by the child that are recognized and understood by the coder. Multiple Word utterances consisted of two or more different voiced or signed words understood by the coder. Multiple Word utterances did not need to be grammatically correct (Walker & Carta, 2010). The ECI was administered and scored quarterly by EHS staff. In some cases, the lowest performing children were assessed frequently to monitor response to intervention.
ECI administration
Administration of the ECI involved one familiar adult, either the assessor or parent, who played with a child using one of two toys: the Fisher-Price Barn or Fisher-Price House (Walker & Carta, 2010). This occurred in children’s homes (96%), infant-toddler center (3%), or elsewhere (0.9%, e.g., library). Play partner familiarity avoided “stranger effect” reactivity in young children (Brooker et al., 2013). The play partner’s role during an ECI session was to encourage the child’s communication by following the child’s lead and commenting on the child’s actions and words. Because the goal was to capture the child’s typical communication performance, play partners followed the child’s lead and supported the child’s communicative behavior through responsive interactions and interest in the child’s play. The administration was coded live by a trained assessor or later from videotape.
ECI accommodations
Accommodations were made for children who were non-English speakers and with sensory and/or physical impairments (Walker & Buzhardt, 2010). In the case of non-English speakers, the play partners and coders of the ECI data were speakers of both English and the home language. In this study, 15.5% of ECI administrations included Spanish and English, 84.5% were English only. Accommodations for children with sensory and/or physical impairments included moving toys closer to the child; utilizing supported positioning for the child to orient the child toward the toys to enable best access; and where needed using more identifiable, larger toys that made recognizable sounds. In these situations, the adult play partner might introduce the toys by allowing touching and manipulation, provide more movement of toys, and tell the child where the toys were placed if needed. Assessors were not required to document accommodations used during assessments.
ECI recording and scores
The occurrence of each key skill was recorded on a standard score sheet by trained EHS staff. These scores were entered into the ECI online data system. Calculation of ECI key skill scores was automated online, wherein the raw frequencies of skill occurrences were converted to rate per minute by dividing each by the duration of administration (i.e., 6 min). The ECI Total Communication rate, however, was a weighted composite of the four key skills to account for the curvilinear pattern of change in key skills, thereby creating a positively accelerating, composite score reflective increasing proficiency in early communication (Walker & Carta, 2010). Calculation of Total Communication rate involved weighting the frequency of Single Words by 2 and Multiple Words by 3. For example, if a child produced frequencies of 5 (Gestures), 6 (Vocalizations), 3 (Single Words), and 1 (Multiple Words), the Total Communications rate was equal to (5 + 6+ [3 × 2] + [1 × 3]) = 20/6 min, or 3.3 communications per minute.
ECI assessor training and certification
All EHS staff members were trained and certified to administer and code the ECI. Initial training involved a partial day workshop in which trainees learned the ECI’s administration protocol and the skills definitions for coding communicative behaviors. Using two ECI videos, trainees calibrated their coding at 85% agreement or higher with master codings. Trainers provided individualized feedback to trainees who did not achieve 85%, and trainees coded additional videos until they achieved 85%. Administration fidelity was checked by the ECI developers who reviewed and scored each trainee’s videotape of an ECI administration. Fidelity was defined as 13 out of 16 correct administration steps completed. Failing this, trainees were provided feedback and repeated the procedure until meeting the criterion. Untrained staff did not administer, code, or enter ECI scores.
Interobserver agreement and ECI score reliability
ECI assessors within programs also completed paired interrater reliability checks with a second assessor either live or via videotape to maintain reliability and check for drift. Both scorings were entered into the online system and an automatic percentage agreement was reported. An agreement was defined as perfect match in counts, otherwise a disagreement was counted. Results for a sample of 97 paired scorings indicated a mean agreement score of 94.8%, ranging from 70.6% to 100.0%. Agreement on the individual key skills ranged from 89.4% for Vocalizations to 93.7% for Multiple Words. We also checked the reliability of the ECI rate scores used for making interferences. Pearson r correlations for the four key skills and Total Communication rate ranging from .96 (Gestures) to 1.0 (Multiple Words) between primary and reliability assessments.
Criterion measures
Criterion measures were assessed in the child’s home at 12, 34, 36, and 48 months of age by trained research staff (see Table 1). The language outcome measures assessed receptive and expressive language, gestures, words (vocabulary), and sentences. The Preschool Language Scale (PLS-4; Zimmerman, Steiner, & Pond, 2002) is a measure of auditory comprehension and expressive communication used with children birth through 6 years. Norms are nationally representative in a sample of 1,500 children, including ethnic minorities and children with disabilities. Test–retest reliabilities are reported in the .90 range; interrater reliability correlation = .95. PLS-4 validity was referenced to the PLS-3 and Denver II.
Children’s communication using gestures, words, and sentences was measured using the MacArthur-Bates Communicative Development Inventories (CDIs, Second Edition). The Gestures and Words test form was used with children at the 12-month assessment that was replaced by the Words and Sentences test form at the 24-month assessment. In the Gestures and Words form, parents reported the Total Gestures that they had seen the child attempt or use regularly. Parents also reported the words the child (a) understands and/or (b) uses, generating a separate index of Words Understood and Words Produced. In the Words and Sentences form, parents also reported the child’s Words Produced. An additional score in this test form was mean length of utterance (MLU), a measure of linguistic productivity. Test–retest reliability estimates for the Gestures and Words test form were .80 for Words Produced and Words Understood scores, except at 12 months, where it was only .60. Cronbach’s alpha for Words and Sentences was reported .86 for WP scores. Validity correlations ranging from .40 to .88 were referenced to the PPVT-4 and PLS-4 (Fenson et al., 2007).
Receptive vocabulary was measured using the Peabody Picture Vocabulary Test–Fourth Edition (PPVT-4) (Dunn & Dunn, 2007). Reliability for the PPVT-4 is reported to be .92 to .98 (alpha) and .87 to .97 (split-half). The criterion validity of the PPVT-4 ranges from .66 to .77 with the OWLS Test of Listening Comprehensions, and .67 to .83 on OWLS Test of Oral Expression. The representative norm sample was linked to 1994 Census data including African American and Hispanic children, and children with disabilities.
The early literacy criterion measures assessed several skill domains that included vocabulary, alphabet knowledge, phonological skills, and print awareness. To assess vocabulary, we used the Picture Naming Early Literacy—Individual Growth and Development Indicators (EL-IGDIs) (Missal & McConnell, 2010) with children at 36 and 48 months of age. Test–retest reliability for Picture Naming across 3 weeks was 0.67. Alternate form reliability estimates for Picture Naming ranged from 0.44 to 0.78. Criterion validity correlations ranged from .75 to .79. To assess children’s Definitional Vocabulary, Phonological Awareness, and Print Knowledge, the Test of Preschool Early Literacy (TOPEL: Lonigan, Wagner, & Torgesen, 2007) was used. Split-half reliability is reported to range from .87 to .96. The criterion validity of the TOPEL–Definitional Vocabulary with the One Word P Vocabulary Picture Test was r = .71. Similar estimates for TOPEL–Phonological Awareness, and TOPEL–Print Knowledge scales with the PPVT-4 were r = .70, and .65, respectively, and with the Pre-K version of the Comprehensive Evaluation of Language Function CELF-P2 were r = .61 and .45 in the fall (Greenwood, Carta, et al., 2013).
Criterion test administration and collection procedures
Twenty-nine assessors administered the criterion measures. Assessors were hired and trained to administer measures within or near the communities and programs serving participants. Assessors held academic degrees ranging from the BS to the MS in a range of fields (e.g., family science, clinical psychology, speech/language pathology, etc.). Completed assessment forms were returned to the research office or shipped via preaddressed FedEx packets. Supervision included monthly conference calls, examination of data scored, and annual reliability sessions.
The research team’s Lead Assessor trained all assessors in person or at distance using a video conferencing application (Skype/Polycom) during a 90- to 120-min training session. Trainees also received an instructional CD that included model videos of each assessment administration, video description of scoring forms, and a training manual. Trainees were certified as assessors after administering and scoring each assessment with at least 85% administration fidelity and 85% agreement with the Lead Assessor’s scoring. The Lead Assessor checked administration fidelity and scored reliability either live or via videos that the trainee submitted to the research team. Reliability assessments with assessors were conducted annually. Assessors were paid US$80 for each completed assessment battery.
Statistical Analysis
ECI data were downloaded from the online database as a coma-separated variables (*.csv) file for statistical analysis. Because the ECI data were measured more frequently by program staff than were the criterion measures, ECI mean estimates were calculated to match the 12, 24, and 36 month age bins (i.e., the bin for 12 month olds was 9–15 months using 6 month [binned] blocks around each age point). The ECI observations that fell outside these bins were removed. As a result, 483 children of the N = 570 enrolled remained in the sample; the final dataset included 483 children and 39 variables. Descriptive statistics for the predictor and criterion variables are shown in Table 1 of the supplemental online materials including the N number of children with data on each variable, ranging from 339 with ECI data at 12 months of age to 49 with TOPEL–Phonological Awareness data at 48 months of age. Missing data of all types was 61% (11,491/18,837) with 32 unique patterns of missing variables. The statistical approach needed to address missing data is determined by considering the nature of missingness described as (a) missing completely at random, (b) missing at random, and (c) not missing at random. Given the diversity in missing data patterns and the small percentage accounted by each pattern (range, 0.2%–15.0%), we assumed data were missing at random as recommended by Enders (2010). Thus, full information maximum likelihood (FIML) was used (Asparouhov & Muthén, 2008; Enders, 2010) given data missing at random (Graham, Olchowski, & Gilreath, 2007; Zhang & Walker, 2008). A strength of FIML was use of auxiliary variables to help retrieve missing data information loss. These variables included home language, IFSP status, and caregiver characteristic variables (i.e., marital status, hours per week spent away from child, education level, income, employment status, and receipt of assistance).
To address the research questions, we used the Pearson correlation coefficient. Because the statistical significance of a correlation is influenced by both its magnitude and its standard error, we interpreted results using effect size ranges for correlations (small effect [from r ≥.1 to <.3], medium effect [from r ≥.3 to <.5], and large [r ≥ .5]). Because the ECI predictor variables were growth-based measures, we used raw scores to represent growth on the criterion rather than their relativistic standardized score values. For concurrent validity, correlations were computed within the same measurement age bins. For predictive validity, correlations were computed across the different age bins. Because ECI Multiple Words at 12 months was zero for 99% of children, no Pearson r was possible. Because ECI Total Communication and key skills were included, we examined nomological 1 networks of relationships between predictor and criterion.
Results
Sample Description
Children in the sample proved to be in the normative range on criterion measures (see Table 1 in the supplemental online materials). For example, the PLS-4 mean values were at or above the norm for auditory comprehension, expressive communication, and composite total language at 12, 24, and 36 months of age. Similarly, the PPVT-4 receptive vocabulary scores at 36 months also were at the normative level. TOPEL Definitional Vocabulary, Phonological Awareness, and Prink Knowledge standard scores fell below the normative mean but within the average range at 36 and 48 months of age. Mean levels of ECI Total Communication and key skills also were consistent with levels previously reported for children at these ages, as were the patterns of growth in ECI key skills as children became more proficient using Single and Multiple Words (Greenwood et al., 2010) (see Figure 1, Table 1 in supplemental online materials). Together, results indicated that children were functioning at expected levels of proficiency.
Trends in Early Communication Indicator (ECI) key skills means 12 to 36 months.
ECI at 12 Months
ECI repertoire
Children’s communication repertoire at 12 months was primarily prelinguistic. Children’s ECI communication consisted of Gestures and Vocalizations with Single Word use just emerging (see Table 2 in the supplemental online materials, and Figure 1). The ECI Total Communication rate was 5.6 per minute.
Concurrent validity
The strongest concurrent relationships were observed between the 12 month ECI Total Communication rate and the PLS–Auditory Comprehension and Total Language scores and the CDI–Words Produced and Total Gestures language criterion scores. ECI key skills showing positive relations were Vocalizations and Single Words (see Table 2 in the supplemental online materials).
Predictive validity (24, 36, and 48 months)
The strongest and most consistent patterns of prediction were between 12-month ECI Total Communication rate and the 36-month criterion measures (see Table 2 in the supplemental online materials). These included all of the PLS-4 language scales as well as the PPVT-4 Receptive Vocabulary score. ECI key skills at 12 months with moderately strong positive 36-month predictions were Vocalizations and Single Words. ECI prediction of the criterion at 24 and 48 months was near zero and negative. The exception was a –.36 prediction of 48 month TOPEL–Phonological Awareness by 12 month ECI Gestures. The more gestures used at this age, the lower the phonological awareness.
ECI at 24 Months
ECI repertoire
A year later, children’s ECI communication profile was a combination of prelinquistic skills (Gestures and Vocalizations) with emerging spoken language (Single and Multiple Words) (see Table 2 in the supplemental online materials, and Figure 1). The ECI Total Communication rate had improved by more than double at 13.8 communications per minute.
Concurrent validity
The strongest current relationships were between ECI Total Communication and the 24 month PLS–Expressive Communication and Total Language scores and the CDI–Words Produced and MLU language criterion measures. Notably different from the 12 month findings was the stronger correlation with PLS-4 Expressive Communication and the new relations with CDI–MLU. The ECI key skills with positive concurrent relations with the CDI were Single and Multiple Words (see Table 2 in the supplemental online materials), while Gestures and Vocalization at this age were mostly zero or negative concurrent correlates.
Predictive validity (36 and 48 months)
The moderately strong and most consistent pattern of prediction from the 24 month ECI Total Communication rate was the 36-month PLS-4 scales, PPVT-4, and EL-IGDI Vocabulary. In a rare case, 24-month Gestures (.31) was a positive predictor of 36-month TOPEL–Phonological Awareness, while Vocalizations was a negative predictor (–.35). Predictive relations were the strongest and most diverse between the 24-month Single and Multiple Words, and Total Communication rate and the 48-month PPVT and TOPEL measures.
ECI at 36 Months
ECI repertoire
Children’s communication repertoire at 36 months was comprised primarily by spoken language with prelinquistic skills waning. Children’s ECI communication was now primarily Multiple Words now exceeding use of Single Word Utterances at 24 months. Gestures and Vocalizations were now at their lowest levels (see Table 2 in the supplemental online materials, and Figure 1). The ECI Total Communication mean rate at 36 months had improved again to 22.6 communications per minute.
Concurrent validity
The strongest concurrent relationships at 36 months occurred between the ECI Multiple Words rate and the 36 month PLS–Expressive Communication and Total Language scores. For the first time, the ECI Multiple Word key skill exceeded that of the ECI Total Communication rate in strength of concurrent relations with both the PLS-4 scales and EL-IGDI Vocabulary. ECI Gestures, Vocalizations, and Single Words were negative correlates (see Table 2 in the supplemental online materials).
Predictive validity (48 months)
The strongest and most consistent pattern of prediction was from the 36 month ECI Multiple Words to the 48 month PPVT-4 Vocabulary, EL-IGDI Vocabulary, and TOPEL–Definitional Vocabulary. All other ECI key skills (Gestures, Vocalizations, and Single Words) were negative predictors. Gestures maintained its moderately strong negative relationship with 48-month PPVT receptive vocabulary.
Discussion
The criterion validity claims of the ECI were investigated using estimates at 12, 24, and 36 months of age. The hypothesis regarding observed growth in ECI Key Skills and Total Communication rate was accepted. At 12 months, children’s communication was prelinquistic, comprised of vocalizations and gestures with no use of single or multiple words (Figure 1). A year later, children’s prelinquistic communication profile was in transition with rapid growth seen in Single Word use with Multiple Word phrases emerging. Two years later at 36 months, children’s communications consisted primarily of multiple word phrases and single word utterances. Vocalizations and Gestures remained in the repertoire but at their lowest rates of occurrence. These patterns were consistent with prior reports based on monthly growth trajectories also in low socioeconomic status (SES) samples (Greenwood et al., 2010).
Each skill evidenced a uniquely shaped growth trajectory based on (a) age at skill onset, (b) rate of growth, and (c) a maximum inflection point changing from acceleration, to deceleration in some cases. When considered together, they reflected temporally ordered change within overlapping skills through 36 months of age, with use of all remaining in the 36-month repertoire. These finding are consistent with other reports of growth in early communication (e.g., Warren & Walker, 2005) and studies reporting the emergence of simpler language skills prior to more advanced skills (e.g., Bates & Dick, 2002; Greenwood, Buzhardt, et al., 2013). The findings inform the concept of develpmental continuum in the sense that what was observed was not just simply temporally ordered emerging steppingstones but rather a pattern of spoken language emerging with prelinguistic language attenuating. Knowledge of these age-linked skill patterns is likely to be of greater value in intervention decision making than only just a single composite score (Good, Simmons, & Kameenui, 2001), because potential intervention targets for children with delays and who are not responsive to interventions are provided.
The hypothesis related to replicating and finding new concurrent relationships between the ECI and criterion measures also was true. However, correlations were weak to moderately strong depending on ECI scale and child age. ECI Total Communication formed the most consistent positive relationships with the criterion at each age, replicating prior findings with similar measures (Luze et al., 2001), r = .56 (parent rating) and .62 (PLS-3), supporting its use as leading indicator of proficiency.
The ECI Gestures was not a concurrent correlate of the criterion at any age. ECI Vocalizations at 12 months, however, was a positive correlate of CDI–Gestures, but not related to other criterion thereafter. In contrast, the ECI Single Words skill was a positive correlate of CDI–Words Produced at 12 months, and PLS–Expressive Communication and Auditory Comprehension at 36 months. The strongest concurrent correlation was r = .50 between Multiple Words and PLS–Expressive Communication rate at 36 months.
The ECI concurrent correlations for Single Words and Total Communication at 24 and 36 months of age were similar to prior studies that investigated relationships between the ECI and early language and preliteracy measures. For example, Friedman (2012) at 24 months reported Single Word rate correlations ranging from .33 to .45 with PLS–Expressive Communication, a caregiver language sample, and the Communication and Symbolic Behavior Scales (Wetherby, Allen, Cleary, Kublin, & Goldstein, 2002). At 36 months, Friedman (2012) reported ECI Total Communication rate correlations of .44 with a language sample and .62 with PLS–Expressive Communication.
These moderately strong ECI correlations were consistent with reported values in studies of the psychometric properties of other measures. For example, the CDI concurrent validity correlations at 24 months was in the r = .32 to .49 range with PPVT Receptive Vocabulary, and the McCarthy Cognitive and Verbal scales (Feldman et al., 2005). ECI correlations also compared favorably in magnitude to correlations observed among the criterion measures used in this study (r = .30 to .67). Thus, the ECI concurrent findings for its linguistic indicators were comparable in size to many other measures.
The hypothesis that the ECI scales would predict language and literacy criterion measures was partially true depending on the ECI subscales and time between measurement occasions. Twelve-month ECI predictions were the strongest and most consistent with criterion at 36 months; but surprisingly there were no relationships at 24 and 48 months. The only exception was a negative Gestures prediction of TOPEL–Phonological Awareness at 48 months. The pattern of prediction based on 24 month ECI Single and Multiple Words estimates were comparatively stronger and more consistent particularly years later. Multiple Words was the strongest predictor of 36 and 48 month criterion. The pattern of predictions based on the 36 month ECI was the strongest between Multiple Words and the 48 months criterion.
The hypothesis of finding stronger patterns of relationships in older than in younger children was confirmed in the case of concurrent and predictive validity. Relationships were strongest when children were using Single and Multiple Words when compared with 12 months when children were using prelinquistic skills and Single Words was just emerging. The expectation that the ECI’s prelinguistic skills would predict linguistic skills was partially true. Twelve month Vocalizations but not Gestures was a moderate positive predictor of the PLS-4 auditory comprehension and expressive communication scales at36 months. A uniform pattern of negative to zero relationships between the prelinquistic and linguistic skills after 12 months of age was observed, and children were using Single and Multiple Words. These findings of stronger predictions from 24- and 36-month data compared with 12 months were consistent with prior reports (Larson, 2016). Overall, these were important new findings suggesting the value of the ECI’s prelinguistic measurements as indicators of risk in older children. Children continuing prelinquistic rather than linguistic communication is a characteristic of children with IFSPs and language delays (Greenwood et al., 2010; Greenwood, Walker, et al., 2013).
Accounting for These Findings
Why only weak to moderately sized correlations were observed was not clear and may be the result of a number of factors/challenges in the criterion assessment of very young children. These include psychometric weakness; differences in content, method, and informant factors; context of administration; and reactivity due to the stranger reaction in infants and toddlers. The ECI has adequate reliability and assesses both prelinguistic and linguistic skills in naturalistic play context with a familiar adult recorded by an observer using an observational protocol. Thus, considerable differences existed between the ECI and the criterion measures in these factors potentially affecting the strength of correlational relationships.
For example, a weakness in some of the widely used language criterion measures was low reliability and validity, particularly for the youngest children. For example, Fenson et al. (1994) reported test–retest reliabilities of only .60 at 12 months compared with .80 for older children on the CDI–Words Produced and Words Understood scores. Variability in performance of children at 36 and 48 months also has been reported based on the 0.67 reliability finding for EL-IGDI Vocabulary.
Differences in content assessed (i.e., definitions) was another factor. Most criterion measures tapped child vocabulary size as its central construct developing initially when children were between 12 and 24 months of age and older. This is in contrast to growth in actual expressive use of words in the ECI and not vocabulary size. It also was the case that the vocabulary constructs assessed in the criterion measures ranged from surface to deep with advancing age, from reports of word use (CDI), to one word picture naming (PLS-4, PPVT-4, EL-IGDI), and identifying a word and recalling its definition (TOPEL). These changing content trends were not aligned with the ECI’s continuing measurement of prelinquistic skills, especially in the first 12 months, but perhaps more so with the oral linguistic skills.
Correlational patterns between the ECI and the criterion measures trended stronger with increasing age as children made advances in use of words and sentences, alongside declines in gestures and vocalizations. These differences in children’s skill profiles may have accounted for the observed discontinuity in 12-month ECI predictions at 24 and 48 months as compared with 36 months of age. The differences also may explain the better continuity shown by the 24 and 36 month ECI predictions 1 and 2 years into the future as all measures consistently tapped spoken language.
The weakness in the cross-measure prelinquistic skills relationships might have been due to the differences in how similar constructs are actually assessed by the criterion measures at this age compared with the ECI. For example, CDI–Total Gestures was a moderate correlate of ECI Total Communication and Vocalization rates but not ECI Gestures. None of the criterion measures assessed Vocalizations as directly as did the ECI. The criterion measures at this age (the PLS-4 and CDI) did measure a gesture construct but each defined and assessed differently. For example, PLS–Gestures were included in the auditory comprehension scale and not the expressive scale (Zimmerman & Castilleja, 2005). The CDI asked parents to rate their child’s use of pointing, showing, giving, and so forth, as “often,” ‘sometimes,’ or “not yet,” whereas the ECI tapped children’s observed use of nodding, shrugging shoulders, and pushing away an object offered to the child. These weak correlational findings are in accord with other studies that have reported limited or no correlations between standardized measures of gestures for young children also due to variations in definitions and measurement approaches (Ellawadi & Weismer, 2014).
Other measurement challenges unique to children this young and weaker psychometrics may be due to differences in the context of administration and reactivity to the assessor. The available contexts for assessing children this age are fewer than for older children because of the limited ability to sit, focus attention, use language, and communicate with an adult assessor—skills only just emerging (Gibbs & Teti, 1990; Larson, 2016). Thus play-based measures with familiar adults used in the ECI are an ecologically valid alternative. Very young children also are subject to the reactive “stranger effect” (Brooker et al., 2013), and consequently, parent-informed measures (questionnaires, ratings) are the most widely used in early language screening, even though parent reports have been questioned for both under- and overestimating their own child’s skills (Larson, 2016).
Strengths
A strength of the study was the longitudinal design with a large sample of children that allowed analyses of the concurrent and predictive validity of the ECI at four age points in association with age-appropriate, widely used language and early literacy criterion variables/constructs. Another strength was the observational approach to children’s early prelinguistic and linguistic skills represented in the ECI in contrast to parent reports of child vocabulary size. Because the development of spoken language undergoes a transitional process readily captured in the ECI and in some cases linked to other widely used measures of language and literacy development, this work uniquely deepened our understanding of communication development in the first 12 months that was primarily comprised of gestural and vocal utterances, not yet spoken words, and later relationships to language and literacy.
Limitations
The research faced a number of challenges in recruiting, enrolling, and following children over 4 years. Because the team was not able to directly contact EHS families, recruitment relied on parents contacting the research team for information. Because the number of contacts varied widely across home visitors, home visitors may not have been reliable recruiters. Thus, we had a low rate of family enrollment. While it was possible to rule out sampling bias in participants’ gender, age at first ECI, disability status (IFSP or not), primary language spoken at home, and mean ECI Total Communication rate, there may have been differences between groups on other variables (e.g., parent’s highest level of education, parent responsivity/motivation/involvement in their children’s education, etc.) that were not examined.
The longitudinal design was based on the natural rolling enrollment of children in EHS programs, resulting in cross-sectional, longitudinal data that are known not to predict fully longitudinal data (Maxwell & Cole, 2007). Another rolling enrollment challenge in the design was that children at all age bins were not all assessed 3 years ahead for predictive analysis.
Missing data are a particular threat to the internal validity of longitudinal research, one that increases incrementally with the time between measurement occasions. The greatest loss of information in this study was due to difficulties scheduling and administering the criterion measures, particularly in rural programs. Another contributing factor was turnover in the part-time testing staff, so that replacement and training rates were high. Particularly challenging was testing at 48 months when a family had moved or the child aged out of EHS program after 36 months of age. These challenges required consideration of the pattern of missing data leading and selection of FIML (Enders, 2010; Enders & Bandalos, 2001). Because prior ECI research did not find cluster effects at the levels of ECI observations or assessors, and because the data were “binned,” we did not include these effects. We also did not examine program-level clustering.
While intended for use with all children in this age range, the vast majority of findings are based on low-income eligible children in EHS and children with special needs also served by practitioners in these programs. It should be noted that under federal policy, the eligibility of children with IFSP in EHS is conditional only on IFSP status and low income is not a requirement. Thus, findings in this and other ECI reports (Greenwood et al., 2010) including comparative benchmarks are based on findings from these subpopulations of children. Even though the current at-risk sample of children performed at the norm on the language criterion measures suggesting that overall the sample was not delayed in language, it remains unclear what estimates would emerge if based on a fully random sample of children.
Implications for Future Research
Young children’s language measurement research remains a need and challenge. Questions remain regarding the strength and consistency of the multiyear ECI predictions and continuity with criterion relationships over time. Future work could benefit from efforts to overcome this and the other limitations discussed including improvements in the criterion measures in future ECI validation. Other improvements are needed to reduce the amount of missing data in future research. Future research could also benefit from a stronger longitudinal design. For example, an age-cohort sequential design with 12-, 24-, and 36-month cohorts each followed longitudinally for 3 years would be such an improvement (Miyazaki & Raudenbush, 2000).
Experimental studies offer stronger alternative approaches to future ECI validity research. For example, experiments using the ECI as a basis for measuring progress in response to intervention when combined with distal criterion outcomes such as the CDI or the PLS-4 is a promising approach to ECI validation. Experimental studies also may help improve what we know about the benefits of intervening with very young children’s gestural and vocal production in relationship to the acquisition of spoken communication (Warren, Yoder, Gazdag, Kim, & Jones, 1993; Yoder & Warren, 2001).
While the uptake in use of the ECI has been in programs serving low-income eligible children and/or with documented special needs, how growth estimates would differ in programs serving children without these risks is unknown. To date, this research has been challenging to undertake at reasonable cost because such programs typically have lower accountability demands and financial consequences given that children are not meeting progress benchmarks. Future ECI research is needed in these programs serving low-risk, typically developing children.
Implications for Practice
Early childhood practitioners are rarely included in psychometric studies of measurement validity because they are considered biased or less objective. While this may call into question the rigor of the findings, it is also a strength that practitioners collected the ECI data in the natural conditions in which these assessments were meant to be used for progress monitoring and data-driven decision making. The findings provided new reasons why practitioners serving infants and toddlers should consider the ECI. For example, the ECI uniquely supports direct measurement of Gestures and Vocalization in relationship to single and multiple word growth over time. Gestures, Single and Multiple Words and composite Total Communication rate are known predictors of typical language development as well as early delay. Even though the concurrent and predictive validity findings were uneven in some regards, they provided a complex, first look at the relationships with these widely used criterion measures.
The ECI is uniquely suited for use in a response to intervention approach to early intervention decision making and has been demonstrated sensitive to the effects of language promoting interventions delivered to parents by home visitors (Buzhardt et al., 2018). The ECI also can involve parents as play partners as well as collaborating intervention partners. Considering this, prior psychometric knowledge of the ECI, and web-based access (Carta et al., 2010; Greenwood et al., 2011), the ECI offers help to programs and practitioners interested in the response to intervention approach in early intervention.
Supplemental Material
Supplemental_Online_Material – Supplemental material for Criterion Validity of the Early Communication Indicator for Infants and Toddlers
Supplemental material, Supplemental_Online_Material for Criterion Validity of the Early Communication Indicator for Infants and Toddlers by Charles R. Greenwood, Jay Buzhardt, Dale Walker, Fan Jia and Judith J. Carta in Assessment for Effective Intervention
Footnotes
Acknowledgements
We gratefully acknowledge the contributions of colleagues Debra Montagna, Barbara Terry, Christine Muehe, Susan Higgins, April Fleming, Constance Beecher, Alana Schnitz, and Dwight Irvin. We are grateful for the participation of the Kansas Early Head Start programs, children and families, and Mary Weathers for her support in facilitating partnerships with Early Head Start.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported here was supported by grants from the Institute of Education Sciences, National Center for Special Education Research (R324A070085), and the Office of Special Education Programs (H324C040095, H327A060051), U.S. Department of Education to the University of Kansas. Additional support was provided by the Kansas Intellectual and Developmental Disabilities Research Center, National Institutes of Health (HD002528), Schiefelbusch Institute for Life Span Studies, Kansas Social Rehabilitation Services, and the regional Early Head Start Association. The opinions expressed are those of the authors and do not represent views of the Institute, the U.S. Department of Education, the National Institutes of Health, or other agencies.
Notes
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
