Accuracy of Telehealth-Administered Measures to Screen Language in Spanish-Speaking Preschoolers

Abstract

Introduction: There is a critical need for telehealth language screening measures for use with Spanish-speaking children because of the shortage of bilingual providers and the current lack of psychometrically sound measures that can be administered via telehealth. The purpose of the current study was to describe the classification accuracy of individual telehealth language screening measures as well as the accuracy of combinations of measures used with Spanish-speaking preschoolers from rural and underserved areas of the country. Materials and Methods: This study applied a hybrid telehealth approach that implemented synchronous videoconferencing, videocasting, and traditional pen and paper measures. Screening measures included a processing efficiency measure (Spanish nonword repetition [NWR]), language sampling, and a developmental language questionnaire. Eighty-two mostly Spanish-speaking preschool-age children and their parents participated. Thirty-four children had language impairment (LI), and 48 had typical language development. Results: Although many of the individual measures were significantly associated with standardized language scores (r=0.27–0.55), not one of the measures had classification values of 0.8 or higher, which is recommended when screening for LI. However, when NWR scores were combined with language sample or parent survey measures, promising classification accuracy values that approached or were higher than 0.8 were obtained. Conclusions: This research provides preliminary evidence showing the effectiveness of a hybrid telehealth model in screening the language development of Spanish-speaking children. A processing efficiency measure, NWR, combined with a parent survey or language sample measure can provide informative and accurate diagnostic information when screening Spanish-speaking preschool-age children for LI.

Introduction

Language impairment (LI) is a dynamic and complex disorder that affects multiple domains of language, including morphosyntax, lexical, and phonological processing.^1

–4 Epidemiological studies indicate that approximately 7% of white monolingual English-speaking U.S. kindergarteners have LI that cannot be attributed to other disorders.^5,6 Children with LI are at high risk for psychosocial, academic, and other problems that negatively impact development and quality of life.^7

–10 Consistent with the overall prevalence rates of LI in monolingual English speakers, we can reasonably anticipate that a subset of Spanish-speaking preschoolers (approximately 7%) will have LI, which requires timely screening, identification, and clinical rehabilitative services to ameliorate negative long-term effects. Recent demographic trends and projections indicate that Spanish-speaking children under the age of 5 years are one of the fastest growing segments of the population, and continued growth of Hispanic populations has occurred over the last 10 years in rural counties of the Western region of the country, including Wyoming, Colorado, and New Mexico.^11,12 The result is a significant need to identify Spanish-speaking children at risk for LI and deliver screening and other speech–language pathology services in regions with few professionals in close proximity.

The U.S. Department of Health and Human Services¹³ recently (in 2014) compiled a compendium of psychometrically sound screening measures that can be used to screen for LI in young children. Early and frequent screening of language development is recommended to help determine if additional language assessment is needed, and this screening should use reliable and valid screening tools appropriate to the age, culture, and language background of the child.¹³ According to the compendium, there are over 10 acceptable screening tools for use with English-speaking children, but not one screening tool was deemed appropriate for use with Spanish-speaking children.

Screening the language development of Spanish-speaking children is a complicated process for several reasons. In several surveys, speech–language pathologists (SLPs) reported that they lacked clinical and cultural competence, as well as a set of easily administered measures to screen and assess the language development of children from linguistically diverse backgrounds.^14
–16 This lack of clinical competence may be a result of the shortage of bilingual SLPs. In the United States, there is approximately 1 Spanish-English bilingual SLP available for every 13,395 Spanish-speaking individuals.^4,17 Moreover, a recent nationwide study has revealed that developmental screenings do not occur as recommended by the American Academy of Pediatrics, and there are large discrepancies in access to developmental screenings based on family demographic and geographic factors.¹⁸

Due to the extreme shortage of bilingual SLPs and the growing number of young Spanish-speaking children, there is a need for accessible and informative telehealth measures that accurately screen the language development of Spanish-speaking children. A telehealth approach will offer children and families access to services that may otherwise be unavailable and importantly bridge the clinician–client mismatch, by granting Spanish-speaking families access to Spanish-speaking SLPs. Also, empirically tested telehealth screening measures may dramatically improve early identification and result in enhanced access to early intervention services.

This research was designed to evaluate the classification accuracy of hybrid telehealth measures when used to screen the language development of Spanish-speaking preschool-age children from rural and underserved communities. A hybrid telehealth approach that implemented synchronous videoconferencing and videocasting technology, as well as traditional pen and paper forms, was applied in the current study. The purpose of the study was (1) to determine the relationship between screening measures and standardized language assessment scores, (2) to examine the accuracy of individual screening measures, and (3) to describe the accuracy of combined screening measures presented using the hybrid telehealth approach. Based on previous studies, the researchers hypothesized that hybrid telehealth screening measures would be significantly associated with standardized language scores and that combinations of screening measures would yield acceptable classification accuracy.

Materials and Methods

The research team recruited participants from Head Start and state-funded preschool programs in rural and underserved communities in Wyoming, Colorado, and New Mexico. Families who indicated interest in the study and met the selection criteria (stated below) scheduled study visits with a bilingual site coordinator from their local early childhood center.

Participants

Eighty-two children between 37 and 69 months of age (mean=53.65, standard deviation [SD]=8.84) and their families participated in this study. Only families that met the inclusion criteria of speaking only or mostly Spanish were included in the study. In addition, children met the following inclusionary criteria: spoke only or mostly Spanish acording to parent report and had normal hearing, no known neurological impairment, and lack of severe speech impairment. Maternal level of education ranged from 1 year of formal schooling completed to some college (mean=9.94 years, SD=3.25 years). Nine of the families were single mother households. Paternal level of education ranged from no formal schooling to some college (mean=9.29 years, SD=3.31 years). A questionnaire based on the interview protocol of Gutiérrez-Clellen and Kreiter¹⁹ was used to obtain family language usage profiles. The information gathered showed that for 91.5% of the families their children were exposed to Spanish 90–100% of the time in the home, whereas 8.5% indicated their children hear Spanish between 80% and 89% of the time in the home. In addition, the researchers asked parents if children had experience with or access to electronic tablets or smartphones; according to parent response, 89% of the children had electronic tablet or smartphone experience.

LI Status

LI is a dynamic and complex disorder, and thus converging sources indicating disorder, or a composite reference standard, is recommended over a single reference standard used in isolation.²⁰ Given the complexity of LI and the fact that there is currently no single gold standard for identification of LI in Spanish-speaking children, converging sources, based on recommended best practices for identifying LI in Spanish-speaking children, were applied.^4,21,22 Thus three sources of information were obtained to establish LI status: (a) identification of LI by a bilingual SLP; (b) report of parent concerns about the child's language development; and (c) expressive language standard scores on the Spanish Preschool Language Scale, 4th edition (SPLS-4) of <77 (1.5 SD below the mean). Children were placed in the typically developing (TD) group if (a) they had not been previously diagnosed as having LI, (b) their parents did not report concerns about their language development, and (c) they had expressive language SPLS-4 standard scores of >85.

Based on these criteria, the LI group included 34 children (15 boys and 19 girls), and the TD group included 48 children (23 boys and 25 girls); there were no significant group differences in terms of gender (χ²=0.013, p=0.91), children's age (t=0.42, p=0.67), or children's percentage exposure to Spanish in the home (t=−0.43, p=0.26).

Processing Efficiency Measure

A Spanish nonword repetition (NWR) task developed by Ebert et al.²³ was selected as the processing efficiency measure for this study for several reasons. First, a recent study of Spanish-speaking preschoolers showed that the 4–5-syllable NWR items yielded acceptable classification accuracy values when administered face to face.²⁴ The NWR task consisted of 20 nonword stimuli that follow the sound patterns of Spanish and that increased in syllable length (from 1 to 5 syllables), with four items for each syllable length presented. A complete list of the NWR items and more detailed information about item development and criteria are provided by Ebert et al.²³ The NWR items were presented to children using an iPad^® (Apple, Cupertino, CA) and a videoconferencing application. Prior to data collection, five bilingual professionals were trained in the administration and scoring of the NWR items using the videoconferencing application.

An item-level scoring approach was applied, in which children's repetition of entire nonwords were scored as either correct or incorrect. Conventions for item level scoring from earlier studies were applied (for a review, see Guiberson and Rodriguez²⁴), but also are briefly described here. The Spanish bilingual research professionals scored each item in vivo as a correct or incorrect repetition. Once all items were scored, the number of correct items was tallied for each syllable length, and the scores for syllable length were summed to calculate total item-level score. This particular scoring approach has yielded significant classification accuracy models (p<0.01) with suggestive classification accuracy levels (sensitivity=0.71, specificity=0.74). In addition, 10% of the visits were videotaped in order to establish interrater reliability of NWR scoring. Interrater agreement (point by point) for NWR scoring was 97%.

Spanish Developmental Language Questionnaire

The Spanish Developmental Language Questionnaire (SDLQ) was developed based on Rasch analyses of the Pilot Inventario-III, a Spanish language questionnaire, which was tested in earlier studies.^25
–27 It is based on a developmental model of Spanish language, rather than a translation of an English tool. The SDLQ is composed of items that were shown to differentiate children with typical language development from children with LI and includes 84 age-appropriate vocabulary items and 35 language questions focusing on content, form, and use of language. In addition, parents reported the child's three longest utterances as estimated by number of words (M3L-W). Native Spanish-speaking SLPs and early-childhood professionals assisted in reviewing items and wording used on the SDLQ to ensure it was sensitive and appropriate for dialects of Spanish spoken in the southwestern and western regions of the country. For the current study, the SDLQ was given to parents in a traditional pen and paper format.

Language Sample Measures

To elicit language samples, children were shown a story presented in an e-book format, entitled Pato está sucio (Duck is dirty). The story was created for the study and was recorded by a native Spanish speaker using a screencast application and then played to children on iPads. The story was presented in a format similar to a video, with narration accompanying digital picture stimuli. After the story was told, the children heard recorded prompts to retell the story as the digital picture stimuli used in the story were displayed on the screen. Children's retells were video recorded with a high-definition video flip camera and then transcribed using the Systematic Analysis of Language Transcription (SALT) program. SALT was used to calculate the mean length of utterance in words (MLU-W) and an ungrammaticality index (percentage of utterances with grammatical errors). Five bilingual SLP graduate students were trained to use SALT and achieved 90% or higher interrater reliability (point by point agreement), during their training. Subsequently, interrater reliability checks were completed on 25% of the narrative retell samples. Interrater reliability for the narrative retell samples was 92%.

SPLS-4

The SPLS-4 was standardized on 1,188 Spanish-speaking children living in the United States. The expressive language section of the SPLS-4 has adequate psychometric qualities for language assessment and was used to examine concurrent validity and as one of the sources to establish language status (LI or typical).

Procedures

A synchronous hybrid telehealth model (including synchronous videoconferencing, screencasting, and traditional pen and paper forms) was applied. When families arrived, the bilingual site coordinator reminded parents that the purpose of the study was to test out the use of technology in screening young children. The parents and children were escorted to a preschool assessment room and instructed on how to use the teleconferencing application on the iPad. A bilingual research associate in a separate room or building also had an iPad and initiated the videoconference for the presentation and scoring of the NWR task. Once the NWR was completed, the bilingual research associate joined the family in the assessment room and set up the prerecorded e-book, on the iPad, and the video flip camera. Children's language behaviors were recorded and later transcribed into SALT as described earlier. Parents were then given the SDLQ (pen and paper) and asked to complete the questionnaire. After these measures were collected, the research team administered the SPLS-4 using traditional face-to-face delivery.

Results

Relationship Between Screening Measures and SPLS-4

One of the goals of this study was describe the relationship between screening measures (processing efficiency, language sample, and parent report measures) and standardized language assessment scores (SPLS-4 expressive language scores). Pearson product moment correlations were calculated²⁸ and are presented in Table 1. NWR scores were significantly correlated with SPLS-4 expressive language standard scores (r=0.55, p<0.01). Of the narrative retell sample measures, the ungrammaticality index was the only variable with practical and statistically significant association with SPLS-4 expressive language scores (r=−0.38, p<0.01), whereas all three of the SDLQ measures were significantly associated with SPLS-4 expressive language standard scores (vocabulary, r=0.46, p=0.01; language questions, r=0.45, p=0.01; M3L-W, r=0.42, p=0.01). A large effect size was observed for NWR association, whereas a medium effect size was observed for the ungrammaticality index and the SDLQ measures.²⁹ These positive correlations and medium–large effect sizes indicate that NWR, ungrammaticality, and the SDLQ measures corresponded with the standardized expressive language measure, a promising finding that these measures may be useful in telehealth language screening sessions.

Table 1.

Pearson Correlations Between SPLS-4 Expressive Language Standard Scores and Study Measures

MEASURES	1	2	3	4	5	6	7	8
1. SPLS-4 expressive language	—
2 NWR 4–5-syllable items	0.55^b	—
3. MLU-W	0.15	0.48^b	—
4. NDW	0.27^a	0.55^b	0.82^b	—
5. Ungrammaticality	−0.38^b	−0.16	0.21	0.01	—
6. SDLQ: vocabulary	0.46^b	0.49^b	0.34^b	0.49^b	−0.28^a	—
7. SDLQ: language questions	0.45^b	0.41^b	0.35^b	0.43^b	−0.12	0.64^b	—
8. SDLQ: reported M3L-W	0.42^b	0.41^b	0.30^b	0.38^b	−0.22^a	0.54^b	0.50^b	—

The Spanish Preschool Language Scale-4 (SPLS-4) expressive language subtest was one of the sources to establish language impairment in the current study (n=82). The data given here show how the processing efficiency (nonword repetition [NWR] scores), language sample measures (mean length of utterance in words [MLU-W], number of different words [NDW], and ungrammaticality) and Spanish Developmental Language Questionnaire (SDLQ) measures (vocabulary, language questions, and reported three longest utterances as estimated by number of words [M3L-W]) were associated with SPLS-4 scores.

p≤0.05, ^b p≤0.01.

Classification Accuracy of Individual Screening Measures

To establish the classification accuracy of single predictors, we calculated statistics recommended by the Standards for the Reporting of Diagnostic Accuracy,^30,31 including sensitivity, specificity, positive (+LR) and negative (−LR) likelihood ratios (LRs), and area under the curve of receiver operating characteristics curves. In screening for LI, diagnostic accuracy values greater than 0.90 are considered good, values between 0.80 and 0.89 are considered desirable, values between 0.70 and 0.79 are suggestive but inadequate when used alone, and values less than 0.7 are undesirable.^31
–33 As Table 2 shows, NWR had the highest area under the curve and classification accuracy values, but sensitivity and specificity were below 0.8, and +LR and −LR were suggestive but inadequate alone to make a screening determination. The ungrammaticality index alone did not have desirable diagnostic accuracy. Both vocabulary items and the language questions had near desirable classification values, but the +LR and −LR showed that these measures were inadequate to make a screening determination, and M3L-W alone had inadequate classification values.

Table 2.

Classification Accuracy of Measures

	AUC	SENSITIVITY	SPECIFICITY	+LR (95% CI)	−LR (95% CI)
NWR	0.83^a	0.74	0.75	2.94 (1.73–5.0)	0.35 (0.20–0.63)
Ungrammaticality Index	0.68^a	0.59	0.67	1.76 (1.08–2.88)	0.62 (0.39–0.97)
Vocabulary	0.78^a	0.79	0.77	3.47 (2.0–5.98)	0.27 (0.14–0.53)
Language questions	0.75^a	0.74	0.69	2.35 (1.48–3.75)	0.39 (0.21–0.70)
MLU3-W	0.75^a	0.66	0.91	7.70 (2.56–23)	0.37 (25–0.56)

p≤0.05.

AUC, area under the curve; CI, confidence interval; +LR and −LR, positive and negative, respectively, likelihood ratios; MLU3-W, mean length of utterance in words; NWR, nonword repetition.

Classification Accuracy of Combined Screening Measures

Given the promising results obtained from the NWR measure, we completed additional analyses to evaluate whether combinations of NWR with other variables resulted in stronger diagnostic accuracy values. We estimated a series of logistic regression models to establish the most accurate diagnostic model that predicted language status (LI or typical). Predictors included individual measures, as well as their pairwise combinations. Table 3 shows that combining NWR with other variables resulted in increased sensitivity and specificity and stronger LR values with narrower confidence intervals. This reflects improved precision when NWR is combined with other measures. NWR and the ungrammaticality index had the most promising values. Given our sample size, models with three or more variables were not estimated.

Table 3.

Logistic Regression R ² and Diagnostic Accuracy of Combined Variables

	ADJUSTED R ²	SENSITIVITY	SPECIFICITY	+LR (95% CI)	−LR (95% CI)
NWR+MLU-W	0.52^a	0.82	0.77	3.59 (2.09–6.18)	0.23 (11–0.48)
NWR+Ungrammaticality Index	0.54^a	0.79	0.79	3.81 (2.14–6.79)	0.26 (0.13–0.51)
NWR+vocabulary	0.54^a	0.79	0.77	3.47 (2.0–5.98)	0.27 (0.14–0.53)
NWR+language questions	0.54^a	0.82	0.75	3.29 (1.97–5.51)	0.24 (0.11–0.50)

p≤0.01.

CI, confidence interval; +LR and −LR, positive and negative, respectively, likelihood ratios; MLU-W, mean length of utterance in words; NWR, nonword repetition.

Discussion

Despite the fact that combinations of synchronous telehealth and traditional (pen and paper or face-to-face administered) measures may assist in screening for LI in Spanish-speaking children, there are several additional considerations that need to be addressed. First, stronger diagnostic values are needed. Not one of the measures or combinations of measures evaluated had both sensitivity and specificity values that were in the desirable range, although they approached the desirable range when combined. This means that measurement refinement and additional telehealth studies examining multiple screening measures are needed. Second, the measures from the preliminary study were collected through a synchronous hybrid telehealth model (including synchronous videoconferencing, screencasting, and traditional or pen and paper forms). This model may work by bringing the bilingual resources available to areas of the country where these services are needed. Based on the results of the current study, several recommendations can be made. First, combining two screening measures yields stronger classification accuracy results than single measures. Second, the processing efficiency measure, NWR, accounted for the most variability in language status and unique variability when combined with other screening measures. And, finally, either language sample measures (e.g., ungrammaticality or MLU-W) or measures from parent surveys (vocabulary or language questions) appear to yield similar diagnostic information when combined with NWR. These results indicate that a screening protocol that includes NWR and either a language sample or parent questionnaire can yield clinically informative results.

There are limitations to the hybrid model applied in this study. First, pen and paper forms increase the paperwork burden placed on families and practitioners who must track, remind, and request these forms from families. When forms are not collected, this information is lost and cannot be factored into screening decisions. Another limitation is that language sampling requires a Spanish-speaking professional, and there is a known shortage of bilingual SLPs. This transcription problem could be addressed by training translators and Spanish-speaking aides how to collect brief language samples. Language samples could then be shared with screening professionals, and this information could be factored into screening decisions. Although there are logistical challenges to be addressed, these approaches represent a significant step in providing services in rural and underserved areas of the country.

At the same time, a major problem in rural communities is limited or poor Internet access or connectivity as well as limited cellular or satellite connections. This makes using synchronous technology, such as live videoconferencing or electronic forms that require a connection, unfeasible. Numerous studies have shown the effectiveness of “store and forward” and other asynchronous telehealth approaches used with populations in rural and poorly connected communities (for a review, see Mashima and Doarn, 2008). Store-and-forward electronic forms collected through a computer tablet would also help with the loss of forms or parents forgetting to send forms in to providers. The current study showed that screening measures collected through a hybrid telehealth model yielded promising classification accuracy when measures were combined. This work provides motivation for further studies and provides preliminary evidence showing the effectiveness of telehealth models in screening the language development of Spanish-speaking children.

Footnotes

Acknowledgments

This research was funded by the National Institute of General Medical Sciences (grant U54GM104944-01A1).

Disclosure Statement

No competing financial interests exist.

References

Leonard

. Children with specific language impairment. Cambridge: MIT Press, 2000.

Bartlett

, Flax

, Logue

, Vieland

, Bassett

, Tallal

, Brzustowicz

. A major susceptibility locus for specific language impairment is located on 13q21. Am J Hum Genet, 2002; 71:45–55.

Bishop

. What causes specific language impairment in children? . Curr Dir Psychol Sci, 2006; 15:217–221.

Kohnert

. Language disorders in bilingual children and adults. San Diego: Plural, 2013.

Rice

, Tomblin

, Hoffman

, Richman

, Marquis

. Grammatical tense deficits in children with SLI and nonspecific language impairment: Relationships with nonverbal IQ over time. J Speech Lang Hear Res, 2004; 47:816–834.

Tomblin

, Records

, Buckwalter

, Zhang

, Smith

, O'Brien

. Prevalence of specific language impairment in kindergarten children. J Speech Lang Hear Res, 1997; 40:1245–1260.

Rescorla

, Ross

, McClure

. Language delay and behavioral/emotional problems in toddlers: Findings from two developmental clinics. J Speech Lang Hear Res, 2007; 50:1063–1078.

Catts

, Adolf

, Hogan

, Ellis Weismer

. Are specific language impairment and dyslexia distinct disorders?. J Speech Lang Hear Res, 2005; 48:1378–1396.

Johnson

, Beitchman

, Brownlie

. Twenty-year follow-up of children with and without speech-language impairments: Family, educational, occupational, quality of life outcomes. Am J Speech Lang Pathol, 2010; 19:51–65.

10.

Rescorla

. Age 17 language and reading outcomes in late-talking toddlers: Support for a dimensional perspective on language delay. J Speech Lang Hear Res, 2009; 52:16–30.

11.

U.S. Census Bureau. Overview of race and Hispanic population 2010. 2013. Available at www.census.gov/ (last accessed October 27, 2014 ).

12.

U.S. Department of Health and Human Services. Healthy people 2010 focus area 1: Access to quality health services. Washington, DC: U.S. Department of Health and Human Services, 2013.

13.

U.S. Department of Health and Human Services. Birth to 5: Watch me thrive. A compendium of screening measures for young children. Washington, DC: U.S. Department of Health and Human Services, 2014.

14.

Guiberson

, Atkins

. Speech-language pathologist's preparation, practices and perspectives on serving culturally and linguistically diverse children. Commun Disord Q, 2012; 33:169–180.

15.

Hammer

, Detwiler

, Blood

, Qualls

. Speech-language pathologists' training and confidence in serving Spanish-English bilingual children. J Commun Disord Q, 2004; 37:91–108.

16.

Roseberry-McKibbin

, Brice

, O'Hanlon

. Serving English language learners in public school settings: A national survey. Lang Speech Hear Serv Sch, 2005; 36:48–61.

17.

American Speech-Language Hearing Association. Membership survey CCC-SLP survey summary report: Number and type of responses, updated. Rockville, MD: American Speech-Language Hearing Association, 2014.

18.

Rice

, Van Naarden Braun

, Kogan

, Smith

, Kavanagh

, Strickland

, and Blumberg

. Screening for developmental delays among young children—National Survey of Children's Health, United States, 2007. MMWR Surveill Summ, 2014; 63(Suppl 2):27–35.

19.

Gutiérrez-Clellen

, Kreiter

. Understanding child bilingual acquisition using parent and teacher reports. Appl Psycholinguistics, 2003; 24:267–288.

20.

Rutjes

, Reitsma

, Coomarasamy

, Khan

, Bossuyt

. Evaluation of diagnostic tests when there is no gold standard: A review of methods. Health Technol Assess, 2007; 11:iii–ix.

21.

Bedore

, Pena

. Assessment of bilingual children for identification of language impairment: Current findings and implications for practice. Int J Bilingual Educ Bilingual, 2008; 11:1–29.

22.

Banerjee

, Guiberson

. Evaluating young children from culturally and linguistically diverse backgrounds for special education services. Young Except Child, 2012; 15:32–44.

23.

Ebert

, Kalanek

, Cordero

, Kohnert

. Spanish nonword repetition: Stimuli development and preliminary results. Commun Disord Q, 2008; 29:67–74.

24.

Guiberson

, Rodriguez

. Classification accuracy of nonword repetition when used with preschool-age Spanish-speaking children. Lang Speech Hear Serv Sch, 2013; 44:121.

25.

Guiberson

, Rodriguez

. Rasch analysis of a Spanish language-screening parent survey. Res Dev Disabil, 2014; 35:646–656.

26.

Guiberson

, Rodriguez

. Measurement properties and classification accuracy of two Spanish parent surveys of language development for preschool-age children. Am J Speech Lang Pathol, 2010; 19:225–237.

27.

Guiberson

. Concurrent validity of a parent survey measuring communication skills of Spanish-speaking preschoolers with and without delayed language. Perspect Commun Disord Culturally Ling Div Popul, 2008; 15:54–72.

28.

Gliner

, Morgan

, Leech

. Research methods in applied settings: An integrated approach to design and analysis, 2nd ed. New York: Routledge Academic, 2009.

29.

Cohen

. Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Erlbaum, 1988.

30.

Bossuyt

, Reitsma

, Bruns

, Gatsonis

, Glasziou

, Irwig

, Lijmer

, Moher

, Rennie

, de Vet

. Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. Fam Pract, 2004; 21:4–10.

31.

Dollaghan

. The handbook for evidence-based practice in communication disorders. Baltimore: Paul H. Brookes Publisher, 2007.

32.

Law

, Boyle

, Harris

, Harkness

, Nye

. The feasibility of universal screening for primary speech and language delay: Findings from a systematic review of the literature. Int J Lang Commun Disord, 2000; 35:165–188.

33.

Plante

, Vance

. Selection of preschool language tests: A data-based approach. Lang Speech Hear Serv Sch, 1994; 25:15.

34.

Mashima

, Doarn

. Overview of telehealth activities in speech-language. Telemed E Health, 2008; 14:1101–1117.