Effects of TWA-Supported Digitally on Comprehension of Students With Autism Spectrum Disorder,Level 1

Participants

Participants were recruited by contacting the director of special education of a large urban school district in the northeastern United States. The director then recruited principals, who then recruited special education teachers whose students met the inclusion criteria. Inclusion criteria for participants were as follows: (a) diagnosed with ASD, (b) had reading comprehension goals listed in their individual education plans, and (c) had a full-scale IQ higher than 85 as measured by the Woodcock-Johnson III (WJ III; Woodcock, McGrew, & Mather, 2007). Six eligible children were identified at two different schools, this study describes the data for three who all received English language arts (ELA) instruction in the same resource room in the same school. The other three students were not included in the study.

All participants demonstrated an average level of word identification coupled with below average fluency levels according to the criteria set forth by Hasbrouck and Tindal (2006). Further examination of their records revealed a discrepancy between decoding and reading comprehension skills (decoding > comprehension). This pattern provided evidence of significant comprehension difficulty relative to decoding skills (Nation, Clarke, Wright, & Williams, 2006). Participants’ decoding skills fell in the average to significantly below-average range and comprehension skills in the average to significantly below range. Participants did not have additional intellectual disabilities, nor were classified as students with learning, emotional, or behavioral disabilities. All participants were male, 11 years of age, and in sixth grade: two were African American and one Caucasian. One participant had a speech and language disorder in addition to ASD, Level 1.

Table 1 represents the characteristics of the participants involved in this study. The classroom teacher initially contacted parents of participants and written consent was obtained before working with the children. In addition to parental consent, participant assent was obtained via researcher-led interviews and having the participants circle a smiling face to indicate assent with a “yes” response and a frowning face icon to indicate a “no” response to each question.

OPEN IN VIEWER

Table 1.

Participant Demographic Information.

Participant	Ethnicity	DIBELS-Fluency (Winter)	WJ Subtest: Comprehension	WJ Subtest: Decoding
Michael Age 11	African American	45 wpm (<10th percentile)	55 Significantly below average	78 Low average
David Age 11	African American	91 wpm (<25th percentile)	86 Average	97 Average
Jaime Age 11	Caucasian	77 wpm (<10th percentile)	56 Significantly below average	67 Significantly below average

Note. Reading comprehension and decoding scores as indicated by Woodcock-Johnson III (WJ) subtests (Woodcock et al., 2007). Fluency scores as indicated by DIBELS and benchmarks according to Hasbrouck and Tindal (2006).

Setting

After institutional review board approval, this investigation was conducted in a middle school resource room during ELA instruction. The school was located in an urban, low socioeconomic city in the northeast United States. Although teachers were present while the researcher conducted the study, they were asked not to use the intervention materials with their students until the generalization phase to ensure fidelity of strategy instruction. Baseline, intervention, and maintenance sessions took place during 1:1 sessions in the participants’ resource room with the researcher serving as instructor. Maintenance data were collected 10 days after the completion of the intervention phase in the same setting, but with novel science and social studies text passages within the participants’ instructional Lexile level.

Outcome Measures

Instructional-level reading expository science passages were selected based on the participants’ performance on the Qualitative Reading Inventory-5 (QRI-5; Leslie & Caldwell, 2011). Accuracy, sequence, discussion of main ideas, and supporting details have been previously associated with high levels of expository text comprehension (Magliano, Millis, Levinstein, & Boonthum, 2011; Mason, 2004, 2013; Mason, Meadan, et al., 2006). Therefore, an oral retell summary statement was used to measure participants’ comprehension of text passages at baseline, intervention, maintenance, and generalization phases (Carnahan & Williamson, 2013; Mason, 2004; Mason, Snyder, et al., 2006). The retell was scored according to a rubric (Table 2) including measures of accuracy, main ideas with supporting details, and prompting. Two upper elementary and one middle school writing teacher reviewed the rubric for content validity. In addition, it was shown to three doctoral-level reading experts who all agreed the description of content on the rubric was valid.

OPEN IN VIEWER

Table 2.

Oral Retell Rubric.

	4	3	2	1	Circle 1
Main Fries
Accurate in ideas and sequence	Precise gist of the passage in same exact sequence as the story Extremely accurate and discusses all the main points at length	Precise gist of the passage In roughly the same sequence as the story Mostly accurate and discusses most of the main points	Unclear explanation of main idea Unclear sequence of the story Many inaccuracies and Unclear discussion of main points	No retell of the main ideas	4 3 2 1
Details
Accurate in major and minor details	Four or more text-based details Two or more details not explicitly slated, but inferred Embellished with reader’s prior knowledge Finer points that enrich the context of the passage provided (compares to existing knowledge)	Two or more text-based details One inferred detail Some reference to reader’s prior knowledge	Minimal text-based details No inferences No reference to the text	No details given to support the main idea	4 3 2 1
Prompting
	Retell is independent—no prompting	1–2 Verbal prompts (i.e., What else?)	3–4 Verbal prompts	4+ Verbal prompts	4 3 2 1
Total					/3=

Note. Adapted from Tindal and Marston (1990); and Haworth, Lopata, Thomeer, and Rodgers (2016)

After participants completed each passage, they were asked to independently read and answer 10 reading comprehension questions during baseline, intervention, and maintenance sessions. These questions were within their instructional Lexile level and generated by a free, web-based source (https://www.readworks.org/) for expository text passages according to grade and Lexile level. For all phases of the study, topics of the texts included natural resources, rocks and minerals, volcanoes, glaciers, and natural energy sources. These topics aligned with what was being taught by the general education science teacher in the participants’ science class. These questions were used, not only because they pertained to the passages being read but because they also assessed key information about the participants’ comprehension. Specifically, 10 questions elicited information on participant understanding of (a) text structure, (b) explicit information presented in text, (c) inferential understanding, (d) main idea, (e) vocabulary (two questions), (f) sentence-level syntax questions, (g) author’s purpose, (h) text implicit information, and (i) supporting details (Andreassen & Bråten, 2011; Baker, Gersten, & Scanlon, 2002; Berninger, Vermeulen, & Abbott, 2003).

Reliability

Interobserver agreement (IOA) was 100% on 30% of baseline retells and 100% on 30% of intervention retells. IOA was obtained from both the classroom teacher and an independent doctoral-level graduate student who was not otherwise involved in the study. Training consisted of listening to an audio recording of a participant giving a retell and the researcher, classroom teacher, and graduate student independently scoring the audio retell. Training continued to 100% agreement, using point-by-point agreement.

Design and Procedures

Digital supports

Participants were taught, using the steps of explicit instruction, how to digitally highlight the main ideas in pink using the mnemonic prompts, “I pink this is the main idea,” details in green using the mnemonic “Do you a-green this is an important detail?,” and rhetorical information in orange using the mnemonic prompt, “Orange you glad this isn’t important?.” In addition, participants were taught, using the steps of explicit instruction, how to add digital “sticky notes” as reminders for each of the three steps of the TWA strategy and to record their audio retell summaries using the audio note feature. For this study, all of these digital supports were found in the iAnnotate application for iPad (see Figure 1). Criterion for participants to reach mastery of the TWA-SD strategy was independent use of eight out of nine steps of the TWA-SD strategy (Mason, 2004).

OPEN IN VIEWER

Figure 1.

Example comprehension questions (https://www.readworks.org/).

Data Collection After Training

Once all participants had met mastery criteria, oral retell data were collected for five to six sessions without researcher instruction, and only verbal prompting (maximum of two) of “What do you do next?” until a stable trend was evident in the data (Howorth et al., 2016; Mason, 2013). Oral retells were collected via the audio note tool, and scored by both the researcher, and the classroom teacher. Again, no time limit was given for the participants to give a retell. The length of retells ranged from 1 to 6 min during the intervention phase. Performance for each participant was graphed. According to the criteria for exemplary single-case research (Horner et al., 2005), a small percentage (30% or seven participant retells) was also scored independently by a doctoral-level graduate student to ensure reliability. Agreement was 100% using point-by-point comparison (Horner et al., 2005).

Maintenance and generalization

Five additional maintenance probes were administered to each group to measure oral retells, and performance on comprehension questions 10 days after each participant had completed the intervention phase. Instruction in the strategy was not given, only oral prompts (maximum of two): “What do you do next?” Furthermore, as a measure of generalization, the classroom teacher gave participants expository digital text passages as PDF files to read on iPads during their normal social studies instruction. Participants were reminded they could use the TWA strategy, but no further prompts to use the strategy were provided. The classroom teacher chose text passages according to the independent reading levels appropriate for each participant. Participant-annotated PDF files were then given to the researcher in a password-protected file. The oral retells embedded in these files were evaluated for quality and accuracy of main ideas and details. Comprehension question accuracy was also recorded (See Figure 2). These measures were taken to evaluate whether participant performance was reliant on the presence of the researcher or whether participants’ TWA-SD strategy use could generalize to other content area expository reading tasks.

OPEN IN VIEWER

Figure 2.

TWA-digitally supported (SD) available digital annotation supports example.

Michael

Michael’s baseline performance on the retell rubric was represented by a mean of 1.44, median of 1.33 (range = 1.33–1.67) across three sessions. During the intervention phase, Michael’s performance on the retell rubric increased to a mean of 2.44, median of 2.66 (range = 1.33–3) across six sessions. His retell performance stabilized during the maintenance phase (range = 3–3.17) across four sessions and also across the generalization phase (range = 3–3.33) across five sessions.

Michael’s baseline performance on the reading comprehension questions was represented by a mean score of 34% across three sessions (range = 13–50%); median of 40% (range = 13–50%). His comprehension question scores increased markedly and immediately during the intervention phase to a mean of 71% (range = 50–100%). Interestingly, the accuracy of Michael’s comprehension questions continued to increase during maintenance (mean = 80%, range = 40–100%) but dropped sharply during the generalization phase (mean = 40%, range = 0–60%).

Jaime

Jaime’s baseline performance on the retell rubric was represented by a mean of 2.33, median of 2.33 (range = 2.24–2.33) across three sessions. His performances on the oral-retell measure increased markedly and immediately during the intervention phase to a mean retell rubric score of 3.5 (range = 3.33–3.67) across five intervention sessions. His retell accuracy scores remained stable during the maintenance phase with a mean score of 3.5 (range = 2.67–4).

Jaime’s accuracy on the baseline reading comprehension questions was a mean of 67%, a median of 65% (range = 60–75%) across three baseline sessions. His mean score on comprehension questions increased immediately and markedly during the intervention phase to a mean score of 78% (range = 50–100%) across six sessions. His performance stayed stable during the maintenance phase at a mean score of 78% (range = 60–100%) during the maintenance phase. No retell or comprehension question generalization data were gathered for Jaime during the generalization sessions, as he was absent during these sessions.

David

David’s baseline performance on the retell rubric was represented by a mean of 2.25, median of 2.0 (range = 2.33–2.67) across four sessions. His retell rubric scores increased immediately and markedly to a mean of 3.28 (range = 3–3.67) during the intervention phase. During the maintenance phase, David’s scores remained above baseline at a mean of 3.11 (range = 2.67–3.67) across three sessions. During the generalization phase, however, David’s scores trended back toward baseline with a mean of 2.67 across three sessions.

During the baseline phase, David’s accuracy on the reading comprehension questions was a mean of 40%, median of 35% (range = 10–55%) across four sessions. His comprehension question scores also increased immediately and markedly during the intervention phase to a mean of 71% (range = 60–100%) across six sessions. David’s scores also dropped sharply during the maintenance phase to a mean comprehension accuracy score of 47% (range = 30–70%) across three maintenance sessions. His scores on comprehensions question assessments continued to decline during the generalization phase to a mean score of 30% across three sessions (range = 0–80%).

Measures of Effect

Table 3 shows that all single-case design measures of effect (i.e., Percent Exceeding Mean [PEM], PND, and TAU-U) indicate positive effects of use of the TWA with digital tools across all participant groups for oral retell and variable effects on comprehension question answer accuracy. According to PND, 83% of the retell quality data for Michael did not overlap with baseline data. This indicates that the intervention was effective according quality indicators established by the Institute of Educational Sciences (Kratochwill et al., 2010). According to TAU-U, 88% of Michael’s retell quality intervention data indicate an improvement over baseline, which indicates an effective and significant outcome (p = .039, α = .05). Finally, 100% of Michael’s retell intervention data exceed the mean of baseline data (PEM; very effective). Similarly, 100% of Michael’s comprehension question intervention data exceed the mean of baseline data (PEM; very effective). According to PND, 67% of the comprehension question accuracy intervention data for Michael did not overlap with baseline data (questionable effect). Interestingly, the intervention was shown to be effective for comprehension question accuracy. According to TAU-U, 89% of Michael’s comprehension question accuracy data within the intervention phase indicate improvement over his baseline performance, which is also significant (p = .014, α = .05; Hollander & Wolfe, 1999; Parker et al., 2010; Scruggs & Mastropieri, 1998). Finally, 100% of Michael’s comprehension question accuracy data exceed the mean of baseline data (PEM; very effective).

OPEN IN VIEWER

Table 3.

Single-Case Design Measures of Effect.

Participant	Retell PEM	Retell PND	Retell TAU-U
Michael	100%Very effective	83%Effective	88%Effective
Jaime	100%Very effective	100%Very effective	100%Very effective
David	100%Very effective	100%Very effective	100%Very effective
Participant	Comp. PEM	Comp. PND	Comp. TAU-U
Michael	100%Very effective	67%Questionable	61%Questionable
Jaime	67%Questionable	67%Questionable	37%Ineffective
David	100%Very effective	50%Ineffective	58%Ineffective

Note. Effectiveness indicators from Hollander and Wolfe (1999); Parker, Vannest, Davis, and Sauber (2010); and Scruggs and Mastropieri (1998). PND = percent of nonoverlapping data.

Jaime also showed overall improvement in comprehension during the intervention phase when compared to baseline phase for both retell quality and comprehension accuracy, although only the retell quality improvement was significant. According to PEM and PND, 100% of Jaime’s intervention retell data did not overlap with baseline data (very effective). According to TAU-U, 100% of Jaime’s retell data indicate an improvement between phases (very effective). Interestingly, comprehension accuracy data for Jaime, according to both PEM and PND indicate 67% of the data between baseline and intervention phases do not overlap (questionable effect). TAU-U measures also indicate the intervention was ineffective for Jaime’s ability to answer comprehension questions accurately with only 37% of the comprehension accuracy data showing improvement between phases (Hollander & Wolfe, 1999; Parker et al., 2010; Scruggs & Mastropieri, 1998).

David also showed general improvement in retell quality and questionable improvement in comprehension question accuracy between baseline and intervention phases. According to PEM, PND, and TAU-U, 100% of David’s retell data did not overlap between phases during intervention. Although 100% of David’s comprehension question accuracy did not overlap according to PEM, only 50% of his comprehension accuracy data did not overlap according to PND, and only 58% of the data in the intervention phase indicated improvement in comprehension accuracy according to TAU-U (Hollander & Wolfe, 1999; Parker et al., 2010; Scruggs & Mastropieri, 1998).

Social Validity

All participants indicated the TWA-SD strategy helped them to become better readers. Regarding what he had learned, one participant responded he had learned to take his time, add audio summaries, and slow down to make sure he understood what he was reading. Another participant responded he had learned the goal of reading was to understand, not to “get done.” Regarding how the strategy could help other students, responses included, “They need to think before, while, and after they read” and “It will help them on their English Language Arts (ELA) exams.” None of the participants offered information to add to the strategy. Finally, the participants listed the following as what helped them the most from the strategy: (a) the thinking while reading stage (i.e., linking knowledge, reading speed, and rereading), (b) “I didn’t know I had to do any of that” (c) “Adding audio notes on what the reading reminded me of,” and (d) “Highlighting and looking first at the main idea and then the details.”

In addition, after the completion of the maintenance phase, all participants were interviewed by the first author who read questions from a technology use survey. This survey used Likert-type style questions with a high score of four containing the following questions: (a) Indicate how much you enjoyed using the iPads during your reading activities, (b) How easy was it for you to use the iPads? (c) How easy for you was it to highlight using iAnnotate? (d) How easy was it for you to use the TWA strategy? (e) How easy for you was it to use the TWA checklist on the iPad? (f) How likely are you to recommend the iAnnotate app to another student? and (g) How likely are you to recommend reading on the iPad to your teacher? The participants all responded that they found it easy to use the technology. The mean response for Likert-type style questions was 3.57 (range = 3.14–4). Participants found it particularly difficult to copy and paste text and to save their work using the iAnnotate application. This is important to know because it may make classroom management of student data a challenge for a teacher who may have to manually save each student’s annotated text.

Implications

The results indicate that the use of TWA-SD by participants with ASD, Level 1 who struggle with reading comprehension increased their comprehension as measured by retells. Results for comprehension questions answered correctly are more variable. Contrary to what theory and current research suggests, participants were not more likely to answer inferential questions inaccurately any more than explicit questions. More research with a larger participant sample is needed before any generalizations can be made from this result.

In addition, all participants indicated they enjoyed using the TWA-SD strategy, and it had changed their attitudes in a positive way toward reading. Bastug (2014) indicated positive attitudes toward reading were highly correlated to higher reading comprehension. This study therefore adds much needed information to the list of research-based effective content area interventions for students with an ASD diagnosis. Despite the general low accuracy in answering individual comprehension questions, it is important to note the significant overall increase in retell measures across intervention, maintenance, and generalization phases compared to baseline data, indicating a generalized effect on retell.

The TWA strategy has been long validated as an evidence-based instructional strategy for students with learning disabilities. The results of this study indicate this strategy can be used with digital supports to provide for visual and auditory self-regulation of participants’ own reading comprehension. In lieu of rewording the steps of the strategy in their own words, as is common in any self-regulated strategy instruction, each of these participants drew what looked like a “symbol” or “symbol sentence” to represent the steps of the strategy. The use of such graphical (rather than alphabetical or word based) mnemonics may be unique to students with autism who have often been described as visual thinkers. Furthermore, the use of TWA-SD was investigated as an intervention package. Further research is warranted on what specific digital annotation features yielded the most impact on students’ learning of the strategy and the effects of those tools on the reading comprehension outcomes. Component analysis of each of the steps of the TWA strategy and how the digital tools support each component could contribute to a greater understanding of maintaining and generalizing the effects of this intervention.

Limitations

Although this study was successful on multiple measures, there are many limitations that warrant discussion. Namely, due to the public school setting of this study and variability of middle-school schedules, a stable baseline was not reached for the comprehension question accuracy measure. This may contribute to the variability of the results and limits the conclusions that can be made to the functional relationship between use of the TWA-SD intervention package and the ability to answer prewritten comprehension questions. For instance, the quality and accuracy of participants’ retells improved with their use of the TWA-SD strategy; however, their performance and accuracy in answering multiple-choice and short-answer comprehension questions were more variable. Another limitation is the small male sample, which restricts generalization of findings, and the administration protocol. Since many students receive their instruction in a large group setting, it is unknown whether similar outcomes would be achieved if the TWA-SD was delivered in a group format. Finally, this study examined the effectiveness of this intervention for expository reading comprehension, its efficacy for narrative reading comprehension cannot be inferred from the findings.

Need for Future Research

Based on this study, future studies should seek to evaluate this protocol with a larger, more diverse group of students with ASD. It might also be useful to study the intervention in a randomized group study, so the relative efficacy of TWA-SD can be determined compared to other reading interventions. Future researchers might investigate the types of short answer and comprehensions questions missed by students with ASD as they may show patterns that provide more information to guide instruction. For example, what types of comprehension questions are most frequently answered incorrectly? Furthermore, in order to increase the ecological validity of this intervention, future research should examine use of the TWA-SD in more inclusive classrooms, with classroom texts, and taught by teachers rather than researchers. As such, instructional, behavioral, and technological logistics of use of this strategy on a larger scale should be investigated. Also, in order to further generalize these findings, participants of both genders should be included in future studies.