Does Use of Text-to-Speech and Related Read-Aloud Tools Improve Reading Comprehension for Students With Reading Disabilities? A Meta-Analysis

Abstract

Text-to-speech and related read-aloud tools are being widely implemented in an attempt to assist students’ reading comprehension skills. Read-aloud software, including text-to-speech, is used to translate written text into spoken text, enabling one to listen to written text while reading along. It is not clear how effective text-to-speech is at improving reading comprehension. This study addresses this gap in the research by conducting a meta-analysis on the effects of text-to-speech technology and related read-aloud tools on reading comprehension for students with reading difficulties. Random effects models yielded an average weighted effect size of ( $\bar{d}$ = .35, with a 95% confidence interval of .14 to .56, p < .01). Moderator effects of study design were found to explain some of the variance. Taken together, this suggests that text-to-speech technologies may assist students with reading comprehension. However, more studies are needed to further explore the moderating variables of text-to-speech and read-aloud tools’ effectiveness for improving reading comprehension. Implications and recommendations for future research are discussed.

Keywords

reading comprehension text-to-speech reading disabilities technology meta-analysis

Reading comprehension, which is defined as the ability to construct meaning from interacting with a text, is critical for students to succeed in today’s educational settings (Snow, 2002). For students with reading disabilities, reading comprehension is often difficult (Kim, Linan Thompson, & Misquitta, 2012). Most current theories argue that one of the primary causes of reading disabilities is a struggle to decode written text. This has a direct negative effect on reading comprehension by decreasing word reading accuracy and speed. Inefficient decoding may also tax cognitive resources, leaving fewer resources available for comprehension (Smythe, 2005).

Presenting reading material orally in addition to presenting it in a traditional paper format removes the need to decode reading material and therefore has the potential to help students with reading disabilities better comprehend written texts. There are several technologies for presenting oral materials (e.g., text-to-speech, reading pens, audiobooks). Previously text was available orally through books-on-tape and human readers. More recently, text-to-speech technology has been used widely in educational settings from elementary school through college. Unfortunately, its implementation has outpaced the lagging research base on the effects of using text-to-speech to support comprehension. The research literature is characterized by contradictory results, with some studies reporting improved reading and others not (Dalton & Strangman, 2006; Stetter & Hughes, 2010; Strangman & Hall, 2003). Due to these mixed findings, we wanted to conduct a current review of the literature. The goal of this meta-analysis is to synthesize the research literature on the effects of text-to-speech and related tools for oral presentation of material on reading comprehension for students with reading disabilities.

Individuals with reading disabilities have unexpected significant deficits in reading and its component skills (e.g., decoding, fluency) despite potential educational opportunities. This reading deficit is believed to have a neurobiological basis and is also characterized by a failure to respond to appropriate instruction and intervention (Fletcher, 2009; Lyon, Shaywitz, & Shaywitz, 2003; Wagner, Schatschneider, & Phythian-Sence, 2009).

Common problems for individuals with reading disabilities include inaccurate and slow word reading and reading of connected text, making comprehension challenging (LaBerge & Samuels, 1974; Perfetti, 1985). Mastery of lower level decoding skills is essential for being able to use higher level language skills to understand text (Cain, Oakhill, & Bryant, 2004).

Intervention Versus Compensation

Intervention studies seek to improve students’ reading skills independent of the technology. In contrast, compensation studies seek to provide students with an assistive tool to help them with their reading. Intervention-oriented studies use text-to-speech tools to improve unassisted reading skills, whereas compensation-oriented studies address the use of text-to-speech tools to compensate for word-level skill deficits and gain access to written material. It is often unclear how to tell when intervention has failed and when students should begin to use compensation (Edyburn, 2007). Although educators would like to improve students’ basics skills as much as possible, not all students are able to reach proficient reading skills. The goal of compensation is designed to help students access texts. If comprehension problems of students with reading disabilities stem at least in part from decoding deficits, then reducing or eliminating the decoding requirement should improve reading comprehension. Oral presentation of material including text-to-speech helps eliminate the decoding requirement by reading the words aloud to the student, thus enabling comprehension (Olson, 2000). Text-to-speech and read-aloud text presentation tools are used in both interventions and compensation settings. We included both intervention and compensation studies in this review but also added a moderator variable to represent this distinction to determine whether comparable effects emerged from both kinds of studies.

Additionally, some students with reading disabilities are relatively accurate at decoding but are slow readers and may be unable to keep up with reading assignments. This becomes increasingly true as students move into more advanced course work (i.e., college reading) (Jones, Schwilk, & Bateman, 2012). Text-to-speech may enable students to read material that is beyond their basic word-reading level while matching their interests and listening comprehension abilities.

Use of Text-to-Speech and Other Read-Aloud Tools in Education

Text-to-speech technology development began in the 1980s and has been rapidly increasing. With technological advances, practitioners and education researchers have begun to use text-to-speech and related tools to help students with reading disabilities. In text-to-speech, the text is paired either with a synthesized computer or human recording of that same text (Biancarosa & Griffiths, 2012). As computer technology has improved over the past 20 years, the prevalence and quality of electronic versions of books and computer software with text-to-speech capabilities have also increased. Text-to-speech software development is occurring worldwide in diverse locations, including the United States, Egypt, and Europe (Smythe, 2005). Examples of such programs are DecTalk, ClassMate Reader, Texthelp Read & Write Gold, and Kurzweil 3000 (Berkeley & Lindstrom, 2011). The number of free and easily accessible text-to-speech software programs is increasing (Berkeley & Lindstrom, 2011). There are many statewide implementations, including examples from Kentucky and Missouri (Goddard, Kaplan, Kuehnle, & Beglau, 2007; Hasselbring & Bausch, 2005). These programs commonly contain voice options, custom pronunciation, creation of synthetic audio files, and other assistive tools such as text highlighting (Peters & Bell, 2007). Text-to-speech technology features can greatly affect the users’ experience and thus may affect its effectiveness. Some of these features include reading rate, voice type, document tagging (which affects reading order), and dynamic highlighting. Reading rate can greatly affect the users’ experience (Lionetti & Cole, 2004). When users choose text-to-speech software, a consistent comment is the desire to have software that follows natural speech and prosody. These subtle differences of reading rate, voice, document formatting, and dynamic highlighting between text-to-speech systems may affect overall user experience and thus may contribute to the mixed findings in the literature.

In addition to different features of text-to-speech tools, individual differences in users may affect the users’ text-to-speech experience. Individuals with clinical diagnoses of reading disabilities are heterogeneous. Some may be more skilled at decoding than others and thus may benefit differently from text-to-speech/read-aloud tools. Also, many have comorbid diagnoses, such as attention deficit/hyperactivity disorder, which may affect their reading performance. In addition, personality and social factors interacting with their disabilities may either facilitate or decrease text-to-speech use (Parette & Scherer, 2004; Parr, 2012). Finally, having supportive and knowledgeable technical help with the software, particularly at school, increases students’ ability to use and integrate this technology effectively into their daily lives (King Sears, Swanson, & Mainzer, 2011; Newton & Dell, 2009). These differences may lead to different compensation patterns, which may affect interactions with read-aloud/text-to-speech accommodations.

As previously mentioned, the results of several reviews of the literature have not yielded consistent findings about whether text-to-speech improves comprehension (e.g., Hall, Hughes, & Filbert, 2000; MacArthur, Ferretti, Okolo, & Cavalier, 2001; Moran, Ferdig, Pearson, Wardrop, & Blomeyer, 2008; Stetter & Hughes, 2010; Strangman & Dalton, 2005). More recently, two meta-analyses investigated the compensatory effects of read-aloud accommodations on assessments (Buzick & Stone, 2014b; Li, 2014b). These studies investigated read-aloud accommodations for students with and without disabilities (focused more broadly than just reading disabilities) on English and math assessments. Comprehensive comparative comments on the two studies can be found in (Buzick & Stone, 2014a; Li, 2014a); however, we will provide a brief description of the two studies to illustrate how the current study extends the previous findings.

Li (2014b) was focused on two general questions of the effects of read-aloud accommodations for students with and without disabilities on reading assessments and which factors would influence those effects of read-aloud accommodations. Those factors were explored through moderators (e.g., disability status and content area), and effect sizes were compared utilizing hierarchical linear modeling. Therefore, they focused on using these moderators to explain the variability between student groups and topics. Having broad inclusionary criteria such as including both quasi-experimental and experimental studies as well as published and unpublished studies created a large enough pool, which enabled moderator analysis. The results revealed smaller effect sizes for read-aloud accommodations for math assessments compared to reading assessments regardless of student disability status. Additionally, students with disabilities did not benefit differentially on math assessments compared to students without disabilities.

Buzick and Stone (2014b) carried out a meta-analysis that addressed the effects of read-aloud accommodations for students with and without disabilities on standardized assessments, whether read-aloud accommodations improve test scores more for students with disabilities compared to students without disabilities, and what factors contribute to differences between the effects of students with and without disabilities for read-aloud accommodations. Their inclusionary criteria were stricter than those used by Li (2014a), resulting in a smaller number of studies included in the meta-analysis. The results supported those of Li (2014a) in that smaller effects were found for math assessments compared to reading assessments regardless of student disability status. Additionally, students with disabilities did not benefit differentially on math assessments compared to students without disabilities.

The two previous meta-analyses included students regardless of disability type, and their focus was on large-scale standardized tests that were not specific to reading comprehension. The focus of the current meta-analysis was the effects of text-to-speech and related tools on reading comprehension for individuals with reading disabilities. Additionally, we considered small-n studies, sometimes known as single-case studies, which are commonly used in the field of special education. Recently new methods have become available to calculate a standardized effect for single-case studies, which can be combined with between-subjects studies (Shadish & Lecy, 2015). These methods are now recommended for use as described in a recent paper commissioned by the National Center for Education Research and the National Center for Special Education Research (Shadish, Hedges, Horner, & Odom, 2015).

We hope to extend the findings from the previous meta-analyses by (a) calculating effect sizes for just students with reading disabilities using read-aloud tools on reading comprehension assessments and (b) including additional moderator variables (e.g., delivery method, intervention vs. compensation, amount of reading material presented using the read aloud). Three questions were addressed in this meta-analysis:

What is the average weighted effect size of the use of text-to-speech and related tools on reading comprehension for students with reading disabilities?

Are there identifiable moderators of the effects of text-to-speech and related tools on reading comprehension?

What is the overall quantity and quality of the current research base on students with reading disabilities using text-to-speech software to assist with reading comprehension, and are there important gaps in the literature?

Method

Inclusionary Criteria

Inclusionary criteria were developed to reflect the diversity of technology and settings in which participants used oral presentations of text. The below were used in this meta-analysis:

Reading comprehension must be measured at the sentence, paragraph, or passage level. Both researcher-designed and norm-referenced assessments were included.

Only studies in which the effect sizes could be calculated for students with dyslexia, reading disabilities, or learning disabilities (subtype reading) were included. Due to the diversity and lack of consensus in the definition of reading disabilities (Quinn & Wagner, 2013), an inclusive definition was used. Students could have more than one disability but had to have a reading disability as well.

All studies must include a condition with oral presentation of the reading material. Presentation type may be any of the following: human recorded audio, human readers, or a variety of technology, including synthesized text-to-speech, reading pens, or other eBooks with text-to-speech or oral capabilities.

Studies must be reported in English.

Additionally, in order to gain a complete quantitative synthesis with minimal publication bias, this meta-analysis included all types of literature: peer-reviewed studies, unpublished reports, theses, dissertations, conference proceedings, and poster presentations. Because of variability in the grade at which students with reading disabilities are identified, we restricted our meta-analysis to studies of third grade and above, the grade by which most students with reading disabilities have been identified. The search was not limited by quantitative study design (experimental, quasi-experimental, or observational), utilization of preexisting data, or year; however, qualitative studies were excluded.

Search Procedures

Studies were gathered through a comprehensive and systematic search that utilized multiple methods: searching databases, checking previously published studies’ bibliographies, and contacting experts in the text-to-speech research field. The following databases were used: Education Resources Information Center, Dissertation and Theses, PsycINFO, Linguistic and Language Behavior Abstracts, and ComDisDom provided by Proquest; Education Full Text and Academic Search Complete provided by EBSCO; Web of Science provided by Thomson Reuters; Expanded Academic ASAP provided by GALE; and Google Scholar provided by Google.

Three groups of search terms describing the outcome variable (reading comprehension), the population of interest (students with reading disabilities/dyslexia), and the oral modality of reading material (read aloud/text-to-speech) were used in various combinations, depending on each database’s thesaurus. The topic of “reading comprehension” was searched with the following terms: “reading comprehension,” or “comprehension tests,” or “reading comprehension tests.” Next, various terms referring to dyslexia were connected to the reading comprehension topic using the Boolean AND operator including “response to intervention,” or “learning dis*,” “reading disabilities,” “dyslex*,” or “special education students,” or “disabilities.” These search statements were then combined using the Boolean AND operator with various terms referring to the pairing of audio and visual representations of the text simultaneously. These terms reflect the diversity of text delivery including text-to-speech technology, human readers, and other read-aloud accommodations: “text-to speech synthesizer,” or “text to speech,” or “text-to-speech,” or “speech synthes*,” or “synthes* speech” or “speech synthesizers,” or “oral reading,” or “read aloud,” or “book on tape,” or “audio books,” or “assistive technology,” or “examinations,” or “academic accommodations,” or “testing accommodations.” Collectively all of the various search combinations yielded 2,933 records, which were reviewed, and finally duplicates were deleted. Initial screening of the search results involved reading titles and abstracts to assess their fit with this review of text-to-speech technology for individuals with reading disabilities. “Reading disabilities” was broadly defined so as not to exclude potentially useful studies. This review of titles and abstracts excluded 2,874 records, leaving 59 records remaining.

Additionally, “backwards mapping” was utilized by identifying 14 other relevant studies from the references list. Of these studies 3 did not have the appropriate population or disability categories (Crawford & Tindal, 2004; Koretz & Hamilton, 2000; Randall & Engelhard, 2010). The 11 remaining studies were retained for further review. Also, experts in the field were contacted for additional studies. This resulted in finding 6 additional studies, of which 3 were already included (Boyle et al., 2003; Hodapp, Judas, Rachow, Munn, & Dimmitt, 2007; Roberts, Takahashi, Park, & Stodden, 2013). The last 2 studies of the 6 identified were not included because 1 was not about reading comprehension (Orr & Parks, 2007) and we were not able to extract data on the treatment groups for the other study (Swan, Kuhn, Groff, & Roca, 2005).

Collectively, all of the search methods yielded 81 articles, which were reviewed in full text. Studies were excluded from the meta-analysis based on the following six exclusionary criteria: qualitative studies, synthesis studies (literature reviews and meta-analyses), multiple interventions without reading comprehension reported separately from other interventions, missing a silent reading condition, not conducted in the students’ native language, and inaccessible full-text versions of studies. More details are available upon request from the first author about methods, including which specific studies were excluded for which reasons. The remaining 43 records were then reviewed for possible inclusion within the meta-analysis.

Coding Procedure

Before coding began, studies for which the effect sizes could not be calculated were removed (see Note 1). In an attempt to get more information about the studies, we contacted the studies’ authors. Next, the following characteristics of the studies were coded: What was read aloud (divided between partial versus entire passages and whether comprehension questions were read aloud), type of oral modality (categorized into one of the three terms: synthesized text-to-speech, human readers, recorded human voice), reading material difficulty (categorized into at the participants’ grade levels or reading levels), intervention (defined as using oral presentation to improve students’ basic deficits so they can become better readers independent of technology), compensation (defined as utilizing technology as a long-term reading solution, like glasses for fixing vision problems), ability to repeat oral reading material, decoding (defined as students’ decoding skills being needed for any part of the passage or comprehension assessment), participant grade (defined as a student’s grade level represented in the sample [see Note 2]), and type of publication (categorized into either published or unpublished). For between-subjects studies, the following three additional characteristics were coded: randomization (defined as an equal chance of participants’ receiving random treatment or control condition), differential sample attrition, and group equivalence (defined as intervention and control groups’ performing similarly on measured pretest variables). To ensure coding accuracy a second rater coded 20% of the studies. Overall reliabilities were 0.95 for total interrater agreement and 0.94 for overall Kappa. The range of the interrater agreement across the codes was 0.90 to 1.00 (except for decoding at 0.60). Kappa was 1.00 for all codes except decoding (0.28), intervention (0.65), and what was read aloud (0.73), respectively. Coding discrepancies were resolved through discussion except for the decoding moderator, which was coded by a third rater. The third rater was in agreement with the first rater with Kappa = 1.00 and not in agreement with the second rater with Kappa = 0.40. We went with the codes provided by the first and third raters as the second rater appeared to have trouble rating this variable.

Results

Effect Size Calculations

Effect sizes were calculated for the 22 remaining studies (see Note 3). We calculated effect sizes for all studies, except single-case studies, using the equations in Cooper, Hedges, and Valentine (2009). We used the standardized mean difference (Hedges g, which is Cohen’s d with a correction, noted in this paper as $\bar{d}$ ) as our effect size measure as it corrects for small sample size bias. For the within-subject studies, the paired samples correlations were estimated using other information provided in the publication. Effect sizes that are comparable to those of group studies are available for some case study designs, such as multiple baseline and AB^k designs (Shadish, 2014a, 2014b), and these were included where possible.

Data Analyses

The analyses were conducted using the metafor package (Viechtbauer, 2010) as implemented in R. The package reports restricted maximum likelihood (Harville, 1977) estimates of the pooled effect size and also reports several measures of heterogeneity including Q (Q represents the amount of total heterogeneity in the study and follows a χ² distribution with k − 1 degrees of freedom) (J. P. T. Higgins & Thompson, 2002) and remaining unexplained heterogeneity (for the moderator analysis) (see Note 4), which we reported. We also estimated the effects of potential publication bias using the trim and fill method (Duval & Tweedie, 2004) as well as funnel plots (Cooper et al., 2009).

The results show that the use of text-to-speech tools has a significant impact on reading comprehension scores with $\bar{d}$ = .35, 95% confidence interval (CI) (.14, .56), p < .01 (see Table 1 and Figure 1, respectively). The sampling variance was significant Q (22) = 2051.19, p < .001. Visual confirmation of this heterogeneity using a funnel plot is presented in Figure 2. A trim and fill analysis suggested that four studies were missing on the left side of the mean effect size. Accounting for these studies, the trim and fill analysis suggested that the effect size would be $\bar{d}$ = .24, 95% CI (.02, .45), p = .03. Additionally, a series of sensitivity analyses were conducted.

Table 1.

Summary of Included Articles.

Citation	$\bar{d}$	v	n	Average grade	Delivery method	Goal of study	What was read aloud	Reading material difficulty	Decoding needed	Peer reviewed	Measure
Between-subjects study design
Bielinski, Thurlow, Ysseldyke, Freidebach, and Freidebach (2001)	0.28	0.003	1,261	3.0	HR	C	CP and Q/T	GL	No	No	MAP
Boyle et al. (2003)	0.65	0.108	38	10.5	HR and RHV	C	CP and Q/T	GL	No	Yes	RD
Fasting and Lyster (2005)	0.92	0.083	52	6.0	TTS	I and C	CP	GL	Yes	Yes	LS60
Fletcher, Francis, Boudousquie, and Copeland (2006)	0.90	0.048	91	3.0	HR	C	PP and Q/T	GL	Yes	Yes	TAKS
Fletcher et al. (2009)	0.50	0.065	62	7.0	HR	C	PP and Q/T	GL	Yes	Yes	TAKS
Lange, McPhillips, Mulhern, and Wylie (2006)	0.77	0.083	54	9.5	TTS	C	CP and Q/T	GL	No	Yes	NARA^a
Lundberg and Olofsson (1993)	−0.32	0.241	15	5.6	TTS	I	PP	RL	Yes	Yes	RD
Meloy, Deville, and Frisbie (2002)	1.10	0.073	62	7.0	HR	C	CP and Q/T, RA	GL	No	Yes	ITBS
Roberts, Takahashi, Park, and Stodden (2013)	0.45	0.039	164	9.0	TTS	I	CP	GL	Yes	No	GMRT
Shany and Biemiller (1995)	0.81	0.209	18	3.5	HR	I	CP	RL	Yes	Yes	SAT^a
Within-subject study design
Dolan, Hall, Banerjee, Chun, and Strangman (2005)	1.02	0.575	9	11.5	TTS	C	CP and Q/T	GL	No	Yes	NAEP
Elkind, Black, and Murray (1996)	0.10	0.139	50	17.0	TTS	C	CP and Q/T	GL	No	Yes	NDRT
Floyd and Judge (2012)	1.09	0.102	6	17.0	TTS	C	CP and Q/T	GL	No	Yes	SAT^a
E. L. Higgins and Raskind (1997)	−0.12	0.019	37	13.0	TTS	C	CP and Q/T, RA	GL	No	Yes	FRI
Hodapp, Judas, Rachow, Munn, and Dimmitt (2007)	−0.74	0.034	27	7.5	TTS	C	CP and Q/T	GL	No	No	JRS
Kosciolek and Ysseldyke (2000)	0.53	0.073	14	4.0	RHV	C	CP and Q/T	GL	No	No	CAT/5
Laitusis (2010a), Sample 1	0.57	0.000	527	4.0	RHV	C	CP and Q/T, RA	GL	No	Yes	GMRT
Laitusis (2010b), Sample 2	0.32	0.000	376	8.0	RHV	C	CP and Q/T, RA	GL	No	Yes	GMRT
McCove (2012)	0.53	0.337	3	10.0	TTS	I and C	CP	GL	Yes	No	STAR
Meyer and Bouck (2014)	−0.21	0.143	3	7.6	TTS	C	CP and Q/T	GL	No	Yes	SWP
Schmitt, Hale, McCallum, and Mauck (2011)	0.23	0.063	25	7.0	TTS	C	CP	GL	Yes	Yes	TRL
Sorrell, Bell, and McCallum (2007)	−0.16	0.078	4	4.0	TTS	C	CP	RL	Yes	Yes	AR
Thurlow, Moen, Lekwa, and Scullin (2010)	−0.32	0.011	44	6.7	RP	C	PP and Q/T	GL	Yes	No	GSRT

Note. $\bar{d}$ = effect size; v = variance; n = number of subjects; HR = human reader; C = compensation; CP = complete passages; Q/T = questions/test(s); GL = grade level; MAP = Missouri Assessment Program; RHV = recorded human voice; RD = researcher developed; TTS = text-to-speech; I = intervention; LS60 = Test of Silent Sentence Reading; PP = partial passages; TAKS = Texas Assessment of Knowledge and Skills; NARA = Modified Neale Analysis of Reading Ability II; RL = student’s reading level; ITBS = Iowa Test of Basic Skills the reading comprehension part; RA = explicitly stated that they could request oral reading material on questions, passages, or comprehension tests; GMRT = Gates-MacGinitie Reading Tests; SAT = SAT Critical Readings; NAEP = NCES U.S. History and Civics tests; NDRT = Nelson-Denny Reading Test; FRI = Formal Reading Inventory; JRS = Jamestown Reading Series; CAT/5 = California Achievement Test 5th Edition comprehension part; STAR = STAR Reading Assessment Test; SWP = Six Way Paragraphs Middle Level; TRL = Timed Readings in Literature; AR = Accelerated Reader; RP = reading pen; GSRT = Gray Silent Reading Test.

Researcher modified.

Figure 1.

Forest plot of effect sizes.

Figure 2.

Funnel plot showing (A) the heterogeneity of all studies included in the meta-analysis and (B) the residual heterogeneity after controlling for if the study was between subjects or within subject.

Our effect size is consistent with the previous meta-analyses (Buzick & Stone, 2014b; Li, 2014b) although with a somewhat smaller average weighted effect size. A possible contributing factor to our smaller average effect size was that we had three within-subject studies with negative effect sizes in our pool that were not in either of the previous meta-analyses (E. L. Higgins & Raskind, 1997; Hodapp et al., 2007; Thurlow, Moen, Lekwa, & Scullin, 2010). Without these studies the results are more positive and remain significant, $\bar{d}$ = .49 (.34, .58), p < .001.

In contrast to previous meta-analyses, our study contained postsecondary and adult subjects, but if we only analyzed studies for K–12 we still obtained similar results, $\bar{d}$ = .36 (.13, .58) p < .01. Also, the previous meta-analyses included studies measuring reading for students with disabilities published in or after the 2000s. In this meta-analysis, when only the studies published after 2000 were analyzed the average weighted effect size became $\bar{d}$ = .40 (.17, .64), p < .001. This pattern remained true for both intervention and compensation studies, $\bar{d}$ = .54 (.20, .89), p < .001, $\bar{d}$ = .35 (.12, .59), p < .001, respectively. Overall, the effect of read-aloud tools on reading comprehension found here was significant and positive. However, consistent heterogeneity was found and explored through moderator analyses.

For the moderator analyses, only a single, significant moderator emerged, namely, whether the study design was a between-subjects or within-subject study. The average weighted effect size for between-subjects studies alone was $\bar{d}$ = .61, 95% CI (.39, .83), p < .001. For within-subject studies, the average weighted effect size dropped to $\bar{d}$ = .15, 95% CI (−.13, .43), p > .05. Previous meta-analyses asking what is the impact of presentation accommodations (segmented text and read aloud) on high-stakes tests for students with disabilities also found a lower effect size for within-subject study effect sizes as compared to between-subjects studies. Li (2014b) also found this pattern, but it did not remain significant after all of the predictors were added into the model.

Discussion

The impact of text-to-speech and related read-aloud tools for text presentation on reading comprehension for individuals with reading disabilities was explored in this meta-analysis. Three questions were explored and will be discussed in turn.

Average Effect Size of the Use of Text-to-Speech on Measures of Reading Comprehension

Our meta-analysis provides a more focused approach than previously taken as we looked exclusively at students with reading disabilities using read-aloud tools on reading comprehension assessments. This is motivated by a previous meta-analysis finding that indicated that presentation accommodations (segmented reading and read aloud) had a significant positive effect for students with learning disabilities but did not have an effect for students requiring special education services (Vanchu-Orosco, 2012). Both meta-analyses published in 2014 included many types of disabilities without distinguishing among them, whereas in the current study we focus exclusively on students with reading disabilities. Another difference is that the previous meta-analysis (Buzick & Stone, 2014b) focused mostly on state assessments and other large standardized assessments. Many large-scale assessments test general English skills, not solely reading comprehension. Here, we specifically chose to examine reading comprehension tests. Through this narrow focus on students with reading disabilities and reading comprehension assessments, our model hopes to expand the previous studies to provide new information about moderators affecting the effectiveness of read-aloud tools on reading comprehension.

The effects of text-to-speech and related read-aloud tools indicate that oral presentation of text for students with disabilities helps their reading comprehension test scores. Our finding of an average weighted effect size of .35 is consistent with the recent meta-analysis results on read-aloud accommodations for students with disabilities (Buzick & Stone, 2014b; Li, 2014b). Taken together, this meta-analysis is useful as a starting point to begin quantifying the wide range of published and unpublished read-aloud literature.

Amount of Variability Present and Identifiable Moderators in Effect Sizes

There was significant heterogeneity present. The moderator of between-subjects versus within-subject study design was found to be significant. Possible reasons for this may include regression to the mean, attrition, and order effects due to lack of counterbalancing treatments (Li, 2014b). Interestingly, none of the other moderators were significant in the random effects model. Potential carry-over or training effects of reading with assistive technology may contribute to these differences. Nonsignificant moderators included what was read aloud, type of oral modality, reading material difficulty, intervention, compensation, ability to repeat oral reading material, decoding needed, and grade level. We were surprised that grade level did not moderate the effect of the text-to-speech/read-aloud presentation. This is in contrast to the previous meta-analyses, which found grade level a significant moderator (Buzick & Stone, 2014b). However, depending on how the variation in effect sizes between studies is conceptualized, it affects the amount of variance accounted for by the grade-level moderator (see Table 2; see Note 5).

Table 2.

Moderator Analysis.

Moderator	K	N	B	QM	QE
Between-subjects	23	2,942	.48*	6.17	2049.60**
Grade level⁵	23	2,942	<.01	0.01	168.13**
Intervention	23	2,942	.20	0.47	2049.02**
Difficulty	23	2,942	.30	0.67	2049.14
Delivery method (2 modes below)	23	2,942		4.45	2009.41
a. Recorded voice			.29
b. Human reader			.44
Decoding needed	23	2,942	−.02	0.01	2041.46**
Partly oral passages	23	2,942	−.16	0.32	2032.51**
Ability to repeat oral reading	23	2,942	.11	0.16	2040.92**
Peer reviewed	23	2,942	.41	3.49	2028.56
Randomized	23	2,942	.28	1.34	2046.64
Baseline	3	1,337	.59	1.86	1.46

Note. K = number of studies included; B = change in effect size of one-unit increase in moderator; QM = heterogeneity explained by the moderator; QE = remaining unexplained heterogeneity.

p < .05. **p < .01.

In addition, we were surprised that whether the study used text-to-speech/read-aloud tools in an intervention setting did not emerge as a significant moderator. Using the text-to-speech/read-aloud tools as intervention is theoretically and practically different than using text-to-speech/read-aloud tools in a compensation setting. There were very few studies that were truly interventions despite many studies claiming to be interventions. More research comparing intervention versus compensation approaches needs to be conducted.

The Quantity and Quality of Research on Students Using Text-to-Speech and Related Read-Aloud Tools for Reading Comprehension

Initial search results suggest that this is an active and growing area of research. We found 81 articles for full-text review while applying exclusionary criteria. We found a small number of studies from the 1980s and 1990s. The literature is growing, and the rate of new studies being conducted is increasing with time. Interestingly, the amount of studies using the computerized text-to-speech read-aloud tool is increasing. For example, the studies from the 2010s all used text-to-speech technology. This may reflect the trend of wider access to improved text-to-speech technology.

However, study quality remains a concern, with many studies missing key components such as proper controls and adequate statistical reporting. Additionally, many studies used convenience samples (e.g., Boyle et al., 2003). It is important to have randomized treatment-controlled studies to make causal inferences about the effect of text-to-speech/read-aloud tools in an educational context, thus increasing ecological validity by capturing the diversity of the current classroom. However, there are trade-offs between internal and external validity that are difficult to balance. It is ideal to have large sample sizes, but they should also be as homogeneous as possible, allowing the best possible assessment of the read-aloud/text-to-speech presentation. However, small-sample studies could be useful for identifying students’ disabilities and then allowing intensive study of these individual differences. The downside to them is external validity because you are not able to have a large enough sample size to extend to various types of learning disabilities. However, the studies’ high internal validity is beneficial to this field. Taken together, new studies with clear and complete reporting of both between-subjects and within-subject study designs will be beneficial.

Limitations and Future Directions

There are a few limitations that are worth noting. First, our study included a relatively small sample size (n = 22 studies); however, this is still larger than previous findings. Second, reading comprehension measures and text-to-speech interventions were diverse and were applied at different intensities. Analyzing the importance of intensity of interventions seems particularly interesting, but due to inconsistent reporting, we were unable to develop a meaningful measure of intensity. Third, placebo effects and motivation have yet to be explored or explicitly controlled for. Currently, there is a lack of randomized control trials; these studies will help provide evidence as to whether and how assistive technology affects students’ reading comprehension test scores.

In future studies, measurements of participants’ reading skills such as decoding, working memory, vocabulary, and listening and reading comprehension should be assessed. Text-to-speech technology features including speech speed, text highlighting, and intensity need to be explicitly addressed. With the advent of systemwide user tracking (e.g., length of use, user preferences) of text-to-speech programs, such as Read & Write Gold and Kurzeweil 3000, users’ actions can be recorded. This data can be combined with student characteristics like reading level and disability status as well as their reading test outcomes to see how students should be utilizing the software. These data could then be compared to the students’ reading comprehension measures with text-to-speech/read-aloud tools in compensatory settings such as a statewide reading comprehension test. This enables large quantities of data to be collected, but the quality of the data this monitoring provides is questionable. In summary, future research should investigate the following: features of text-to-speech tools, participant reading characteristics and disabilities, study design considerations, and larger scale usage studies.

Conclusion

Text-to-speech/read-aloud presentation positively affects reading comprehension for individuals with reading disabilities, with average weighted effect sizes of $\bar{d}$ = .35 (p < .001). There is more variability than would be expected due to random error alone. Significant variance was accounted for by design type (e.g., within-subject studies versus between-subjects studies). The overall quantity and quality of studies investigating the effectiveness of text-to-speech and related read-aloud tools on reading comprehension is increasing. However, more studies are needed to explore diverse types of text presentation, for example, different software packages and displays as well as reading disability profiles. These future research studies can then be used to explore how students are interacting with these technologies and their impact on reading comprehension. Taken together, our results indicate that text-to-speech/read-aloud presentation may help students, although the mechanisms remain unknown. Better study designs are required to control for these potential effects.

Footnotes

Authors’ Note

Thank you to the following experts in assistive technology for responding to our request for additional data and/or providing access to unpublished data: Cynthia M. Okolo, Sara M. Flanagan, Emily Bouck, Maria Earman Stetter, Marie Tejero Hughes, Richard E. Ferdig, Tracey Hall, E. A. Draffan, Terry Thompson, Margaret Bausch, Ted S. Hasselbring, Jane Hauser, Melanie Kuhn, Dave Edyburn, and Chuck Hitchcock. Elizabeth L. Tighe is now at Georgia State University, Atlanta, USA.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Support for this project was provided by grant number P50 HD052120 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development.

Notes

References

Berkeley

Lindstrom

J. H.

(2011). Technology for the struggling reader: Free and easily accessible resources. Teaching Exceptional Children, 43(4), 48–55.

Biancarosa

Griffiths

G. G.

(2012). Technology tools to support reading in the digital age. Future of Children, 22(2), 139–160. doi: 10.1353/foc.2012.0014

*Bielinski

Thurlow

Ysseldyke

Freidebach

(2001). Read-aloud accommodation: Effects on multiple-choice reading and math items (Technical Report 31). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://education.umn.edu/NCEO/OnlinePubs/Technical31.htm

*Boyle

E. A.

Rosenberg

M. S.

Connelly

V. J.

Washburn

S. G.

Brinckerhoff

L. C.

Banerjee

(2003). Effects of audio texts on the acquisition of secondary-level content by students with mild disabilities. Learning Disability Quarterly, 26(3), 203–214. doi: 10.2307/1593652

Buzick

H. M.

Stone

E. A.

(2014a). A comment on Li: The same side of a different coin. Educational Measurement: Issues and Practice, 33(3), 34–35. doi: 10.1111/emip.12042

Buzick

H. M.

Stone

E. A.

(2014b). A meta-analysis of research on the read aloud accommodation. Educational Measurement: Issues and Practice, 33(3), 17–30. doi: 10.1111/emip.12040

Cain

Oakhill

Bryant

(2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96(1), 31–42. doi: 10.1037/0022-0663.96.1.31

Cooper

Hedges

L. V.

Valentine

J. C.

(2009). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage Foundation.

Crawford

Tindal

(2004). Effects of a read-aloud modification on a standardized reading test. Exceptionality, 12(2) 89–106. doi: 10.1207/s15327035ex1202_3

10.

Dalton

Strangman

(2006). Improving struggling readers’ comprehension through scaffolded hypertexts and other computer-based literacy programs. In McKenna

M. C.

(Ed.), International handbook of literacy and technology (Vol. 2, pp. 75–92). Mahwah, NJ: Lawrence Erlbaum.

11.

*Dolan

Hall

T. E.

Banerjee

Chun

Strangman

(2005). Applying principles of universal design to test delivery: The effect of computer-based read-aloud on test performance of high school students with learning disabilities. Journal of Technology, Learning, and Assessment, 3(7), 4–32.

12.

Duval

Tweedie

(2004). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. Retrieved from http://doi.org/10.1111/j.0006-341X.2000.00455.x

13.

Edyburn

D. L.

(2007). Technology-enhanced reading performance: Defining a research agenda. Reading Research Quarterly, 42(1), 146–152. doi: 10.1598/RRQ.42.1.7

14.

*Elkind

Black

M. S.

Murray

(1996). Computer-based compensation of adult reading disabilities. Annals of Dyslexia, 46(1), 159–186. doi: 10.1007/BF02648175

15.

*Fasting

R. B.

Lyster

S. H.

(2005). The effects of computer technology in assisting the development of literacy in young struggling readers and spellers. European Journal of Special Needs Education, 20(1), 21–40. doi: 10.1080/0885625042000319061

16.

Fletcher

J. M.

(2009). Dyslexia: The evolution of a scientific concept. Journal of the International Neuropsychological Society, 15, 501–508. doi: 10.1017/S1355617709090900

17.

*Fletcher

J. M.

Francis

D. J.

Boudousquie

Copeland

(2006). Effects of accommodations on high-stakes testing for students with reading disabilities. Exceptional Children, 72(2), 136–150. doi: 10.1093/acref/9780195173697.001.0001/acref-9780195173697

18.

*Fletcher

J. M.

Francis

D. J.

O’Malley

Copeland

Mehta

Caldwell

C. J.

Vaughn

(2009). Effects of a bundled accommodations package on high-stakes testing for middle school students with reading disabilities. Exceptional Children, 75(4), 447–463. Retrieved from http://doi.org/10.1177/001440290907500404

19.

*Floyd

K. K.

Judge

S. L.

(2012). The efficacy of assistive technology on reading comprehension for postsecondary students with learning disabilities. Assistive Technology Outcomes and Benefits, 8(1), 48–64.

20.

Goddard

Kaplan

Kuehnle

Beglau

(2007). Voice recognition and speech-to-text pilot implementation in primary general education technology-rich emints classrooms. Retrieved from http://www.emints.org/wp-content/uploads/2012/02/TtS-VRpilot-qualitative.pdf

21.

Hall

T. E.

Hughes

C. A.

Filbert

(2000). Computer assisted instruction in reading for students with learning disabilities: A research synthesis. Education and Treatment of Children, 23(2), 173–193.

22.

Harville

D. A.

(1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72(358), 320–338. doi: 10.1080/01621459.1977.10480998

23.

Hasselbring

T. S.

Bausch

M. E.

(2005). Assistive technologies for reading. Learning in the Digital Age, 63(4), 72–75.

24.

*Higgins

E. L.

Raskind

M. H.

(1997). The compensatory effectiveness of optical character recognition: Speech synthesis on reading comprehension of postsecondary students with learning disabilities. Learning Disabilities a Multidisciplinary Journal, 8(2), 1–15.

25.

Higgins

J. P. T.

Thompson

S. G.

(2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. doi: 10.1002/sim.1186

26.

*Hodapp

Judas

Rachow

Munn

Dimmitt

(2007, October 20). Iowa text reader project Year 3: Longitudinal results. Paper presented at the 26th annual Closing the Gap conference, Minneapolis, MN.

27.

Jones

M. G.

Schwilk

C. L.

Bateman

D. F.

(2012). Reading by listening: Access to books in audio format for college students with print disabilities. In Aitken

Fairley

Carlson

(Eds.), Communication technology for students in special education and gifted programs (pp. 249–272). Hershey, PA: Information Science Reference. doi: 10.4018/978-1-60960-878-1.ch020

28.

Kim

Linan Thompson

Misquitta

(2012). Critical factors in reading comprehension instruction for students with learning disabilities: A research synthesis. Learning Disabilities Research & Practice, 27(2), 66–78. doi: 10.1111/j.1540-5826.2012.00352.x

29.

King Sears

M. E.

Swanson

Mainzer

. (2011). TECHnology and literacy for adolescents with disabilities. Journal of Adolescent & Adult Literacy, 54(8), 569–578. doi: 10.1598/JAAL.54.8.2

30.

Koretz

Hamilton

(2000). Assessment of students with disabilities in Kentucky: Inclusion, student performance, and validity. Educational Evaluation and Policy Analysis, 22(3), 255–272. doi: 10.3102/01623737022003255

31.

*Kosciolek

Ysseldyke

J. E.

(2000). Effects of a reading accommodation on the validity of a reading test (Technical Report 28). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved from http://education.umn.edu/NCEO/OnlinePubs/Technical28.htm

32.

LaBerge

Samuels

S. J.

(1974). Toward a theory of automatic information processing in reading. Cognitive Psychology, 6(2), 293–323. doi: 10.1016/0010-0285(74)90015-2

33.

*Laitusis

C. C.

(2010). Examining the impact of audio presentation on tests of reading comprehension. Applied Measurement in Education, 23(2), 153–167. doi: 10.1080/08957341003673815

34.

*Lange

A. A.

McPhillips

Mulhern

Wylie

(2006). Assistive software tools for secondary-level students with literacy difficulties. Journal of Special Education Technology, 21(3), 13–22.

35.

(2014a). Comment on Buzick and Stone: Two tales of one city. Educational Measurement: Issues and Practice, 33(3), 31–33. Retrieved from http://doi.org/10.1111/emip.12043

36.

(2014b). The effects of read-aloud accommodations for students with and without disabilities: A meta-analysis. Educational Measurement: Issues and Practice, 33, 3–16. doi: 10.1111/emip.12027

37.

Lionetti

T. M.

Cole

C. L.

(2004). A comparison of the effects of two rates of listening while reading on oral reading fluency and reading comprehension. Education Treatment of Children, 27(2), 114–129.

38.

*Lundberg

Olofsson

. (1993). Can computer speech support reading comprehension? Computers in Human Behavior, 9, 283–293. doi: 10.1016/0747-5632(93)90012-H

39.

Lyon

G. R.

Shaywitz

S. E.

Shaywitz

B. A.

(2003). A definition of dyslexia. Annals of Dyslexia, 53(1), 1–14. doi: 10.1007/s11881-003-0001-9

40.

MacArthur

C. A.

Ferretti

R. P.

Okolo

C. M.

Cavalier

A. R.

(2001). Technology applications for students with literacy problems: A critical review. Elementary School Journal, 101(3), 273–301.

41.

*McCove

(2012). The use of the kindle to access textbook resources by secondary students in automotive technology. Doctoral dissertation. Retrieved from ProQuest Dissertations and Theses database (UMI No. 3555373).

42.

*Meloy

L. L.

Deville

Frisbie

D. A.

(2002). The effect of a read aloud accommodation on test scores of students with and without a learning disability in reading. Remedial and Special Education, 23(4), 248–255. doi: 10.1177/07419325020230040801

43.

*Meyer

N. K.

Bouck

E. C.

(2014). The impact of text-to-speech on expository reading for adolescents with LD. Journal of Special Education Technology, 29(1), 21–33.

44.

Moran

Ferdig

Pearson

P. D.

Wardrop

Blomeyer

Jr. (2008). Technology and reading performance in the middle-school grades: A meta-analysis with recommendations for policy and practice. Journal of Literacy Research, 40(1), 6–58. doi: 10.1080/10862960802070483

45.

Newton

D. A.

Dell

A. G.

(2009). Issues in assistive technology implementation: Resolving at/it conflicts. Journal of Special Education Technology, 24(1), 51–56.

46.

Olson

R. K.

(2000). Individual differences in gains from computer-assisted remedial reading. Journal of Experimental Child Psychology, 77(3), 197–235. doi: 10.1006/jecp.1999.2559

47.

Orr

Parks

(2007, August 1). Assisted reading software—Teachers tell it like it is. Tech&Learning. Retrieved from http://www.techlearning.com/showArticle.php?articleID=196604599

48.

Parette

Scherer

(2004). Assistive technology use and stigma. Education Training in Developmental Disabilities, 39(3), 217–226.

49.

Parr

(2012). The future of text-to-speech technology: How long before it’s just one more thing we do when teaching reading? Procedia—Social and Behavioral Sciences, 69, 1420–1429. doi: 10.1016/j.sbspro.2012.12.081

50.

Perfetti

C. A.

(1985). Reading ability. New York, NY: Oxford University Press.

51.

Peters

Bell

(2007). Choosing and using text-to-speech software. Computers in Libraries, 27(2), 26–29.

52.

Quinn

J. M.

Wagner

R. K.

(2013). Gender differences in reading impairment and in the identification of impaired readers: Results from a large-scale study of at-risk readers. Journal of Learning Disabilities. doi: 10.1177/0022219413508323 (Advance online publication.)

53.

Randall

Engelhard

(2010). Performance of students with and without disabilities under modified conditions: Using resource guides and read-aloud test modifications on a high-stakes reading test. The Journal of Special Education, 44(2), 79–93. doi: 10.1177/0022466908331045

54.

*Roberts

Takahashi

Park

H. J.

Stodden

(2013, January). Does text-to-speech use improve reading skills of high school students? Paper presented at the Assistive Technology Industry Association, Orlando, FL. Abstract retrieved from http://www.cds.hawaii.edu/steppingstones/downloads/presentations/pdf/TTS_ATIA2013.pdf

55.

*Schmitt

A. J.

Hale

A. D.

McCallum

Mauck

(2011). Accommodating remedial readers in the general education setting: Is listening-while-reading sufficient to improve factual and inferential comprehension? Psychology in the Schools, 48(1), 37–45. doi: 10.1002/pits.20540

56.

Shadish

W. R.

(2014a). Analysis and meta-analysis of single-case designs: An introduction. Journal of School Psychology, 52(2), 109–122. Retrieved from http://doi.org/10.1016/j.jsp.2013.11.009

57.

Shadish

W. R.

(2014b). Statistical analyses of single-case designs: The shape of things to come. Current Directions in Psychological Science, 23(2), 139–146. Retrieved from http://doi.org/10.1177/0963721414524773

58.

Shadish

W. R.

Hedges

L. V.

Horner

R. H.

Odom

S. L.

(2015). The role of between-case effect size in conducting, interpreting, and summarizing single-case research (No. NCER 2015-002, pp. 1–109). Washington, DC: National Center for Education Research.

59.

Shadish

W. R.

Lecy

J. D.

(2015). The meta-analytic big bang. Research Synthesis Methods, 6(3), 246–264. Retrieved from http://doi.org/10.1002/jrsm.1132

60.

*Shany

M. T.

Biemiller

(1995). Assisted reading practice: Effects on performance for poor readers in Grades 3 and 4. Reading Research Quarterly, 30(3), 382–395.

61.

Smythe

(2005). Provision and use of information technology with dyslexic students in university in Europe (EU-funded Welsh Dyslexia Project). Retrieved from http://eprints.soton.ac.uk/264151/1/The_Book.pdf

62.

Snow

C. E.

(2002). Reading for understanding: Toward a research and development program in reading comprehension. Arlington, VA: RAND.

63.

*Sorrell

C. A.

Bell

S. M.

McCallum

R. S.

(2007). Reading rate and comprehension as a function of computerized versus traditional presentation mode: A preliminary study. Journal of Special Education Technology, 22(1), 1–12.

64.

Stetter

M. E.

Hughes

M. T.

(2010). Computer-assisted instruction to enhance the reading comprehension of struggling readers: A review of the literature. Journal of Special Education Technology, 25(4), 1–16.

65.

Strangman

Dalton

(2005). Using technology to support struggling readers: A review of the literature. In Edyburn

Higgins

(Eds.), Handbook of special education technology research and practice (Vol. 2, pp. 545–570). Milwaukee, WI: Knowledge by Design.

66.

Strangman

Hall

(2003). Text transformations. Wakefield, MA: National Center on Accessing the General Curriculum. Retrieved from http://aim.cast.org/learn/historyarchive/backgroundpapers/text_transformations

67.

Swan

Kuhn

M. R.

Groff

Roca

(2005). Evaluating the effectiveness of Learning Through Listening® program on the reading abilities of students with learning disabilities. Unpublished paper.

68.

*Thurlow

M. L.

Moen

R. E.

Lekwa

A. J.

Scullin

S. B.

(2010). Examination of a reading pen as a partial auditory accommodation for reading assessment. Minneapolis: University of Minnesota, Partnership for Accessible Reading Assessment.

69.

Vanchu-Orosco

(2012). A meta-analysis of testing accommodations for students with disabilities: Implications for high-stakes testing (doctoral dissertation). Denver, CO: Electronic Theses and Dissertations.

70.

Viechtbauer

(2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–243.

71.

Wagner

R. K.

Schatschneider

Phythian-Sence

(2009). Beyond decoding. New York, NY: Guilford.