Improving Academic Vocabulary for Adolescent Students With Disabilities: A Replication Study

Abstract

Creating Habits That Accelerate the Academic Language of Students (CHAAOS) is a vocabulary intervention developed by O’Connor et al. to improve the academic vocabulary of middle school students with disabilities. This study was designed as a replication of O’Connor et al.’s study; CHAAOS lessons were taught to 33 sixth graders who received special education services for disabilities in English/Language Arts classes. Researcher-designed vocabulary and comprehension assessments for the taught words were administered pre- and post-instruction. The performance of students in this study was compared with the performance of students in O’Connor et al.’s study. Results demonstrated that the present students made similar gains in vocabulary and comprehension compared with the previous CHAAOS students and scored significantly higher than the previous comparison students who did not receive any instruction on the words. These findings support the use of CHAAOS for improving the academic vocabulary of students with disabilities.

Keywords

academic vocabulary middle school intervention students with disabilities replication research

Approximately 63% of eighth-grade students with disabilities perform below the basic reading achievement level on the National Assessment of Educational Progress (NAEP), and only 9% perform at the proficient level (U.S. Department of Education, 2019). These data indicate that adolescents with disabilities experience significant challenges in reading. At the same time, Common Core State Literacy Standards (CCSS) and instructional expectations require students to read, interpret, integrate, analyze, and evaluate complex and authentic academic texts in the secondary grades (National Governors Association [NGA] Center for Best Practices & Council of Chief State School Officers, 2010).

These academic texts are often characterized by high frequencies of cross-disciplinary academic vocabulary, dense syntactic structures, and a variety of text structures (Schleppegrell, 2001; Snow, 2010; Uccelli et al., 2015). Such characteristics can be problematic for adolescent students who have difficulties in reading; among adolescent students with below-average reading comprehension, the majority also experienced concurrent difficulties in word reading, text-reading fluency, vocabulary, or across multiple areas at once (Catts et al., 2006; Clemens et al., 2017; Hock et al., 2009). Therefore, adolescent students with difficulties in reading need instructional support to read and interact with academic texts sufficiently. One promising area in which educators can help students build reading proficiency is the development of vocabulary (Oslund et al., 2018).

The Importance of Vocabulary to Reading Comprehension

It is well established that vocabulary plays a key role in reading comprehension (Cromley & Azevedo, 2007; Gough & Tunmer, 1986; Nagy et al., 2012; Perfetti & Stafura, 2014). Multiple theoretical frameworks highlight the importance of language knowledge and its contribution to reading comprehension. According to the Simple View of Reading (Gough & Tunmer, 1986), reading comprehension is the product of decoding and listening comprehension. The Direct and Inferential Mediation Model (Cromley & Azevedo, 2007) suggests that word reading and vocabulary, coupled with background knowledge, reading strategies, and inferences, produce reading comprehension. The Reading Systems Framework (Perfetti & Stafura, 2014) stresses the importance of the link between the reading processes and knowledge systems, which involve knowledge of words. While the conceptualization of how students attain comprehension differs in each of these frameworks, each one acknowledges the contribution of understanding word meanings.

Moreover, multiple researchers have studied the empirical role of vocabulary in comprehension. These studies have included elementary students (Hjetland et al., 2019; Lonigan et al., 2018), adolescent students (Ahmed et al., 2016; Cromley & Azevedo, 2007; Oslund et al., 2018), linguistically diverse students (Proctor et al., 2005), and students with differences in comprehension achievement (Brasseur-Hock et al., 2011; Catts et al., 2006; Clemens et al., 2017). Across these studies, vocabulary was a consistent predictor of reading comprehension (Ahmed et al., 2016; Cromley & Azevedo, 2007) and accounted for significant proportions of variance in reading comprehension when combined with other factors, such as word reading or broader language skills (Hjetland et al., 2019; Holahan et al., 2018; Oslund et al., 2018). These results support the relevance of providing instruction that targets the vocabulary of secondary students with disabilities given the relationship between vocabulary and comprehension.

Effective Vocabulary Instruction

Vocabulary is a key literacy standard addressed in the CCSS with the expectation that students will acquire and use academic and domain-specific words (NGA Center for Best Practices & Council of Chief State School Officers, 2010). One way students develop their repertoire of words is through incidental or indirect experiences with words while reading (Nagy et al., 1987). However, while some students may learn copious amounts of words incidentally over time, students who experience reading challenges have difficulties acquiring words independently (Bryant et al., 2003; Ferrer et al., 2010). In a descriptive study, Hock et al. (2009) determined that adolescent students who struggled with reading and those with disabilities scored significantly below proficient readers on measures of vocabulary. Furthermore, a later study showed that students with the most severe reading impairments also had the lowest vocabulary scores (Brasseur-Hock et al., 2011).

These findings imply that students with reading challenges need another method to support their acquisition of vocabulary, which can come in the form of vocabulary instruction, such as teaching generative vocabulary strategies (Brown et al., 2016; Reed, 2008) or teaching words directly (McKeown et al., 2018). Researchers have revealed the effectiveness of interventions that teach vocabulary directly in school-wide programs like Word Generation (Snow et al., 2009), in general education classes with Academic Language Instruction for All Students (Lesaux et al., 2014), and in content-specific areas, such as science (August et al., 2009) and social studies (O’Connor et al., 2015; Vaughn et al., 2009).

Perhaps the best known instruction for teaching vocabulary is Beck et al.’s (2013) robust vocabulary (RV) instruction, in which students receive multiple exposures to words and are given multiple practice opportunities. Beginning in the 1980s, Beck and McKeown developed RV instruction and studied its effects on learning. In a series of studies with fourth graders, RV instruction improved the students’ knowledge of target words and their comprehension of passages that included target words (Beck et al., 1982; McKeown et al., 1985). Beck and McKeown (2007) developed further support for their instructional approach when working with kindergarten and first-grade students from low-income schools. The kindergarteners and first graders learned more sophisticated words than a control group, and the students performed better when they received more instructional days compared with fewer instructional days.

More recently, McKeown et al. (2018) designed Robust Academic Vocabulary Encounters (RAVE) to improve the academic vocabulary and comprehension of sixth- and seventh-grade students. The sixth and seventh graders who received RAVE instruction outperformed control students on post-tests of word knowledge and lexical access. Furthermore, the students who received RAVE instruction displayed an advantage over control students on a standardized measure of reading comprehension. While RAVE instruction was successful for sixth- and seventh-grade students in general, McKeown et al. (2018) did not specifically assess how RAVE instruction impacted students with disabilities.

Effective Vocabulary Instruction for Students with Disabilities

Other researchers have examined how directly teaching vocabulary impacts the learning of students with disabilities (Bos & Anders, 1990; Fore et al., 2007; O’Connor et al., 2019; Seifert & Espin, 2012). Bos and Anders (1990) found that vocabulary instruction designed to elicit deeper processing of word meanings and contexts resulted in greater learning for students with learning disabilities (LD) in middle school than did memorization of definitions. Similarly, Fore et al. (2007) found that seventh-grade students with LD experienced greater learning with instruction that taught definitions, characteristics, examples, and non-examples of word meanings compared with students who independently looked up words in the dictionary. In addition, when Seifert and Espin (2012) taught science vocabulary to adolescent students with LD, the students performed better on vocabulary tests when they were directly taught the vocabulary terms. The findings of these studies illustrated that learning words directly and interacting with words in multiple ways are effective practices for students with LD.

Building on the previous research of vocabulary instruction for students with LD, as well as Beck et al.’s (2013) RV instruction, O’Connor et al. (2019) developed a vocabulary intervention, Creating Habits That Accelerate the Academic Language of Students (CHAAOS), to teach academic vocabulary to middle school students with learning disabilities using direct and explicit instructional routines. Special education teachers provided the CHAAOS vocabulary instruction to students in sixth-grade special education English/Language Arts (ELA) classes for 15 min, 4 days per week for 12 weeks; during this time, they taught 48 CHAAOS academic words. Students who received CHAAOS instruction were compared with a business-as-usual (BAU) group of students who did not receive vocabulary instruction on the target words. At post-test, the CHAAOS students outperformed the BAU students in terms of vocabulary knowledge. Importantly, the CHAAOS students also maintained this knowledge 4 to 24 weeks after instruction.

Context of the Current Study

The present research focused on supporting the academic vocabulary of secondary students with disabilities who face considerable reading challenges. Research has indicated that teachers should provide direct vocabulary instruction, repeated exposures to new words, and sufficient opportunities to use the words in activities to build vocabulary knowledge (Beck et al., 2013; McKeown et al., 2018; Stahl & Fairbanks, 1986). The CHAAOS intervention meets all of these criteria and is important because it was designed for use with students with disabilities in middle school. Therefore, the present research used the CHAAOS intervention with a new group of students to determine whether the findings of O’Connor et al. (2019) could be replicated. Replication research is crucial to the field of special education because it provides converging evidence of specific instructional practices that are beneficial for students with disabilities (Cook, 2014; Coyne et al., 2016; Travers et al., 2016).

For this study, the CHAAOS sixth-grade intervention was implemented with a different group of students from the original O’Connor et al. (2019) study. The results were then compared with the results of the previous study. The following questions guided the research:

RQ1. How will the replication group perform on measures of vocabulary knowledge, and how will that performance compare with that of the O’Connor et al. CHAAOS treatment and BAU comparison groups?

RQ2. How will the replication group perform on a measure that assesses how students comprehend vocabulary in context, and how will that performance compare with that of the O’Connor et al. CHAAOS treatment and BAU comparison groups?

RQ3. How will the replication group perform on generalized measures of reading comprehension, and how will that performance compare with that of the O’Connor et al. CHAAOS treatment and BAU comparison groups?

Method

The sixth-grade CHAAOS vocabulary intervention was taught to sixth-grade students in their special education ELA class. Students were assessed on their vocabulary knowledge and comprehension before and after each cycle of instruction using the same measures as O’Connor et al. (2019). Results from this trial were compared with the results of O’Connor et al. (2019) where the same sixth-grade CHAAOS lessons were used.

Replication CHAAOS Treatment Participants

This replication study included one middle school in one school district in the Los Angeles County area. This school district served approximately 6,000 students, and the middle school enrolled 1,200 students, including students who were socioeconomically disadvantaged (55%) and students who were learning English (25%). One special education teacher provided CHAAOS instruction to 33 sixth-grade students (male = 22; female = 11) in their special education ELA classes. The majority of the students in the class received services under the designation of Specific Learning Disability (82%); however, other disabilities were also represented among the students (15% Autism, 3% Other Health Impairment). The students’ ethnicity was as follows: 64% Hispanic/Latino, 15% Asian American, 12% European American, and 9% African American. The majority of the participating students were socioeconomically disadvantaged (85%), and many students were also English learner (EL) students (58%). The primary language of the 19 EL students was Spanish (79%), Mandarin (10.5%), or Vietnamese (10.5%).

O’Connor et al.’s Participants

O’Connor et al. (2019) worked with three middle schools in a southern California school district. This school district served approximately 20,000 students. The three participating middle schools enrolled 950 to 1,300 students and had a large population of students who were socioeconomically disadvantaged.

O’Connor et al.’s CHAAOS treatment

In the study by O’Connor et al. (2019), two special education teachers provided CHAAOS instruction to 22 sixth-grade students (male = 20; female = 2) in their special education ELA classes. Most of the students had a primary special education designation of Specific Learning Disability (82%), but other disabilities were also represented (14% Other Health Impairment, 4% Autism). Of these students, 59% were Hispanic/Latino, 32% were European American, 5% were African American, and 4% did not specify their ethnicity. Furthermore, 12 students were designated as EL students (55%), and their primary language was Spanish. O’Connor et al. (2019) reported an average standard score of 62.77 (SD = 14.97) on the Woodcock–Johnson (WJ) Total Reading cluster and an average standard score of 77.95 (SD = 9.88) on the Comprehensive Receptive and Expressive Vocabulary Test (CREVT).

O’Connor et al.’s BAU comparison

In O’Connor et al.’s (2019) study, a special education ELA teacher taught one BAU class with 21 sixth-grade students (male = 14; female = 7). This BAU class did not receive CHAAOS vocabulary instruction or a researcher-provided alternate vocabulary approach. Their primary special education designations were as follows: 76% Specific Learning Disability, 10% Autism, 10% Other Health Impairment, and 4% Speech/Language Impairment. Of the students, 95% were Hispanic/Latino and 5% were African American. Furthermore, 14 students (67%) were designated as EL students with Spanish as their primary language. O’Connor et al. (2019) reported an average standard score of 60.89 (SD = 14.59) on the WJ Total Reading cluster and an average score of 75.86 (SD = 7.26) on the CREVT.

CHAAOS Instruction

The sixth-grade CHAAOS vocabulary intervention was developed by O’Connor et al. (2019) to be used with students with disabilities. In O’Connor et al.’s (2019) study, CHAAOS teachers provided instruction to whole classes of around 15 to 19 students with disabilities per class period. Similarly, the CHAAOS replication teacher provided instruction to whole classes with around 12 to 16 students with disabilities per class period.

In this study, CHAAOS lessons and materials were provided to the teacher; they are available to download for free at https://gsoe.education.ucr.edu/CHAAOS/index.php. CHAAOS includes 48 scripted lessons that target 48 grade-appropriate academic vocabulary words. It is divided into three cycles of instruction that are designed to be taught throughout the school year. Each cycle includes a sequence of sixteen 15-min lessons focused on 16 academic vocabulary words over a 4-week period. Within a week, the teacher provides instruction on four words over 4 days, and this pattern repeats the next week, continuing until the 4-week cycle of instruction is completed. All the CHAAOS instructional lessons follow a systematic and predictable routine that utilizes explicit instruction coupled with peer interaction and group responses. A specific outline of the scripted instructional procedures across a week is illustrated in the following section as a sample to contextualize the instruction.

Day 1: Introduction of the four words of the week

On Day 1, the teacher introduced the four words of the week (e.g., persuade, elaborate, compare, publish) by stating each word individually and having the students repeat the word. Then, the teacher introduced the meaning of the words one at a time. The teacher stated a short, student-friendly definition and synonym and asked the students to repeat the definition. (e.g., “Persuade means to convince someone to do something; to talk into. What does persuade mean?”). Then, to prompt a discussion, the teacher used the illustrative example that included an image and a short sentence that described the image (e.g., an image of a woman selling a car to a couple was presented: “The saleswoman persuaded us to buy the car. How does persuade apply to this scenario?”). Once this sequence of activities was completed for one word, the teacher would move to the next word and repeat the process until all four words of the week were covered.

Days 2 and 3: Deep practice of words (two words per day)

Lessons on Days 2 and 3 began with a brief review of the four words of the week. During the remainder of the lesson, the teacher guided students through detailed practice for two of the four words (e.g., persuade and elaborate on Day 2; compare and publish on Day 3). The teacher presented Word 1 (e.g., persuade) and restated the definition and sample context to confirm that students understood how to use the word. Next, the teacher presented Word 1 in a new context, asking the students to apply their knowledge of the vocabulary word to answer a question. Illustrations paired with sentence frames were used to support student responses (e.g., images of skydiving, holding an anaconda, and eating a frog were presented: “Which could you be persuaded to do? Complete the sentence: ‘I could be persuaded to ____.’”). Then, the teacher presented the definition and sample context of Word 2 (e.g., elaborate), followed by application of Word 2 in a new context. The lesson would conclude with an activity that encouraged the students to interact with the words in a new context.

Day 4: Cumulative practice

On Day 4, teachers and students reviewed all four of the vocabulary words and definitions of the week, plus additional words from previous weeks of the cycle. Following the review, varied types of practice activities encouraged the students to play with the words and produce responses in different contexts.

BAU Comparison Instruction

In O’Connor et al. (2019), one special education teacher taught the BAU class. The BAU teacher did not provide any direct explicit instruction on the 48 CHAAOS words throughout the year. Typical instruction included Corrective Reading Decoding Strategies (Engelmann et al., 1999). During a break from Corrective Reading, a unit study on Greek mythology was used instead. O’Connor et al. (2019) observed the BAU class twice, during which vocabulary instruction lasted an average of 5 to 9 min and involved matching words with definitions.

CHAAOS Training and Fidelity

The CHAAOS teacher in this study was a consultant for the development of the O’Connor et al. (2019) CHAAOS intervention. Therefore, no specific training was provided to this teacher. Video recordings were collected for eight instructional sessions and were checked for fidelity using the O’Connor et al. (2019) Observation Tool, which rates eight features of vocabulary instruction from 1 (poor) to 3 (great). Inter-rater reliability (IRR) was established between two raters for four sessions at 95%. On average, the replication teacher taught CHAAOS lessons for 17 min and scored high on the 3-point quality rating (M = 2.6). This was comparable with reports of treatment fidelity from O’Connor et al. (2019), who conducted 46 observations of CHAAOS treatment teachers with established IRR between two raters for eight sessions reported at 92%. On average, the O’Connor et al. treatment teachers taught CHAAOS lessons for 23 min and scored high on the 3-point quality rating scale (M = 2.5).

Measures

In this study, the students’ reading comprehension was assessed with two measures to describe the sample. Then, the students’ knowledge of CHAAOS word meanings and their ability to comprehend those meanings in untaught contexts were assessed using the measures developed by and used in O’Connor et al. (2019).

Test of Silent Reading Efficiency and Comprehension

The Test of Silent Reading Efficiency and Comprehension (TOSREC) is a norm-referenced test intended to measure silent reading fluency and comprehension (Wagner et al., 2010). The TOSREC was group-administered during the fall and spring of the students’ sixth-grade year. For this assessment, students are given 3 min to read statements silently and to determine whether the statement is truthful. The TOSREC includes standard scores with a mean of 100 and a standard deviation of 15. Reported alternate-form reliability for sixth grade ranges from .88 to .89, inter-scorer reliability is .99, and coefficients with alternate measures of reading exceed .70.

AIMSweb Maze

The AIMSweb Maze is a curriculum-based assessment tool used to measure reading comprehension (Shinn & Shinn, 2002). For this assessment, students are given 3 min to silently read a passage in which every seventh word, after the first complete sentence, is deleted. Students select the word that correctly completes the sentence among a choice of three words. In this study, students completed two maze passages, and the average number of correct selections was recorded and used in analyses. The AIMSweb Maze was group-administered during the fall and spring of the sixth-grade year. Reported AIMSweb Maze reliability ranges from .86 to .99, and its validity ranges from .75 to 81.

CHAAOS definitions

O’Connor et al. (2019) developed a 16-item multiple-choice test for each of the three instructional cycles. This test assessed student knowledge of the taught vocabulary definitions. Students were presented with the taught word and then selected the correct definition among four choices. The assessment was administered pre- and post-cycle to the students. O’Connor et al. (2019) reported test–retest reliability ranges from .88 to .96 across the cycles and also reported validity with a standardized measure of vocabulary, the CREVT (r = .445, p = .002).

CHAAOS comprehension

O’Connor et al. (2019) also developed an 8-item test for each of the cycles to determine how well students could transfer their knowledge of the words taught to untaught sentences and passages. The sentence task included four cloze sentences for which students had a word bank of vocabulary words to choose from to correctly complete the sentence. The passage task included a short paragraph with vocabulary words embedded. After reading the paragraph, students answered four multiple-choice comprehension questions. This assessment was administered pre- and post-instruction during Cycles 2 and 3. O’Connor et al. (2019) reported validity with the WJ Passage Comprehension subtest (r = .552, p = .000) and the TOSREC (r = .501, p = .001).

Data Analysis

First, the reading comprehension achievement scores from the Maze and TOSREC were analyzed to describe and compare the present replication students with O’Connor et al.’s (2019) treatment and BAU students. Next, a series of multivariate analyses of covariances (MANCOVAs) were used to answer the research questions related to CHAAOS word learning, CHAAOS comprehension, and generalized comprehension achievement among the present replication students and the O’Connor et al. (2019) treatment and BAU students. In each analysis, differences among post-test scores for the three groups were compared while controlling for pre-test differences. MANCOVA was selected as the method of analysis because it considers multiple dependent variables while controlling for type I errors. As in the study by O’Connor et al. (2019), partial eta-squared was reported as the measure of effect size.

Results

Reading Comprehension Achievement

Descriptive statistics for fall TOSREC and Maze scores are provided in Table 1 for the Replication CHAAOS group and for the O’Connor CHAAOS Treatment and BAU groups. Scores on the TOSREC were close to 2 SDs below the mean (Replication: M = 75.76; O’Connor Treatment: M = 74.00; O’Connor BAU: M = 68.38). No significant differences were found between the Replication CHAAOS group and the O’Connor Treatment group (p = .261), but the Replication CHAAOS group scored significantly higher than the O’Connor BAU group (p = .042). Scores on the Maze for all three groups of students were below the 15th percentile (Replication: M = 14.67; O’Connor Treatment: M = 12.40; O’Connor BAU: M = 12.76), with no statistically significant differences among the groups (p = .327).

Table 1.

Mean Performance Scores of Sixth Graders on Vocabulary and Reading Measures.

Vocabulary and reading measures	Replication CHAAOS treatment		O’Connor et al.’s CHAAOS treatment		O’Connor et al.’s BAU
Vocabulary and reading measures	M (SD)	%	M (SD)	%	M (SD)	%
Cycle 1 Definitions
Pre	6.61 (2.9)	43.0	6.58 (3.4)	41.1	5.30 (2.1)	33.1
Post	12.93 (2.6)	80.8	13.42 (4.2)	83.9	4.79 (3.3)	29.9
Cycle 2 Definitions
Pre	9.00 (3.1)	56.0	7.90 (2.8)	49.4	5.62 (2.0)	35.1
Post	13.25 (3.0)	82.8	13.89 (2.9)	86.8	5.42 (1.9)	33.9
Cycle 3 Definitions
Pre	8.69 (2.7)	53.1	9.20 (3.9)	57.5	5.42 (1.9)	33.8
Post	13.54 (3.3)	84.6	14.11 (2.7)	88.2	5.32 (3.1)	33.3
Cycle 2 Comp
Pre	4.38 (1.8)	55%	3.86 (1.5)	48%	3.71 (1.7)	46%
Post	5.38 (1.7)	67%	4.56 (2.2)	57%	2.95 (1.3)	37%
Cycle 3 Comp
Pre	4.53 (2.0)	57%	4.80 (1.4)	60%	3.53 (1.3)	44%
Post	5.83 (1.8)	73%	6.28 (1.4)	79%	4.21 (1.7)	53%
TOSREC
Fall	75.76 (11.4)	–	74.00 (10.4)	–	68.38 (9.0)	–
Spring	71.84 (11.7)	–	67.33 (11.2)	–	65.38 (10.4)	–
AIMSweb Maze
Fall	14.67 (5.8)	–	12.40 (7.0)	–	12.76 (5.1)	–
Spring	17.19 (7.5)	–	11.69 (7.1)	–	11.17 (5.7)	–

Note. Taught Words = Abandon, Access, Alter, Apply, Apprehensive, Attain, Compare, Consent, Consequence, Conserve, Context, Contrast, Contribute, Controversy, Credible, Elaborate, Eliminate, Emerge, Establish, Evaluate, Evidence, Indicate, Individual, Infer, Interpret, Major, Negative, Negotiate, Perceive, Persuade, Point of View, Precise, Predict, Prejudice, Preliminary, Primary, Publish, Restrict, Reveal, Rigid, Significant, Stable, Sustain, Symbol, Target, Unify, Unique, Vary. BAU = business-as-usual; TOSREC = Test of Silent Reading Efficiency and Comprehension.

CHAAOS Definitions Performance

MANCOVA for CHAAOS definitions showed that treatment had a significant effect on vocabulary word knowledge, Λ = 0.326; F(6, 120) = 15.027, p = .000, $η_{p}^{2} = . 429$ . Pillai’s trace, Hotelling’s, and Roy’s tests all converged with Wilks’s lambda. Analyses of covariance (ANCOVA) results were significant for multiple-choice responses at all cycles—Cycle 1: F(2, 62) = 24.358, p = .000, $η_{p}^{2} = . 440$ ; Cycle 2: F(2, 62) = 41.683, p = .000, $η_{p}^{2} = . 573$ ; Cycle 3: F(2, 62) =30.718, p = .000, $η_{p}^{2} = . 498$ . The Replication CHAAOS group and the O’Connor Treatment group outperformed the O’Connor BAU group across all cycles (p = .000), and no significant differences were found in the performance of the Replication CHAAOS and O’Connor Treatment groups at Cycle 1 (p = .301), Cycle 2 (p = .133), or Cycle 3 (p = .273). Table 1 presents the reported means of all groups across cycles.

CHAAOS Comprehension Performance

MANCOVA for CHAAOS comprehension indicated that treatment had a significant effect on comprehension of CHAAOS words in context, Λ = 0.695; F(4, 112) = 5.591, p = .000, $η_{p}^{2} = . 166$ . Pillai’s trace, Hotelling’s, and Roy’s tests all converged with Wilks’s lambda. ANCOVA results were significant for the comprehension measure at both cycles—Cycle 2: F(2, 57) = 7.127, p = .002, $η_{p}^{2} = . 200$ ; Cycle 3: F(2, 57) = 5.848, p = .005, $η_{p}^{2} = . 170$ . The Replication CHAAOS group and the O’Connor Treatment group outperformed the O’Connor BAU group at Cycle 2 (p = .000 and p = .023, respectively) and Cycle 3 (p = .028 and p = .001, respectively). No significant differences were found between the Replication CHAAOS group and the O’Connor Treatment group across cycles. Table 1 contains the CHAAOS comprehension–reported means for all groups across cycles.

General Reading Comprehension Outcomes

Descriptive statistics for the spring Maze and TOSREC scores are also reported in Table 1 for the Replication CHAAOS, O’Connor Treatment, and BAU groups. MANCOVA for spring TOSREC and Maze scores at the end of the school year established that treatment did not have a significant effect on comprehension, Λ = 0.870; F(4, 132) = 2.387, p = .054, $η_{p}^{2} = . 067$ . Pillai’s trace and Hotelling’s tests converged with the nonsignificant result. At the end of the year, all three groups of students still remained below the 15th percentile on the Maze and 2 SDs below the mean on the TOSREC.

Discussion

Many adolescent students with disabilities still struggle with reading in secondary school (U.S. Department of Education, 2019) and require additional support in one or more skill areas (Brasseur-Hock et al., 2011; Clemens et al., 2017; Hock et al., 2009; Oslund et al., 2018). While multiple avenues for promoting reading development exist, one promising avenue is through the development of vocabulary, especially given the intimate link between vocabulary and comprehension (Oslund et al., 2018). The results of this replication study provide preliminary support for the use of the CHAAOS vocabulary intervention as a method for improving the academic vocabulary of students with disabilities. The study had three major findings. First, the results indicated that the Replication CHAAOS students made similar gains to the O’Connor et al. (2019) Treatment group in learning the definitions of the taught words. Second, the Replication CHAAOS students also improved in comprehension of sentences and passages that included CHAAOS words, and this improvement was similar to the O’Connor et al. (2019) Treatment group. Finally, as was the case in O’Connor et al. (2019), the Replication CHAAOS students did not improve in generalized measures of reading comprehension.

Replication Research

Replication studies are a critical component of educational research and are necessary for the advancement of the field; many social science researchers have advocated for an increase in the number of replication studies (Asendorpf et al., 2013; Brandt et al., 2014; Valentine et al., 2011). Although this study was not an exact replication, it was an attempt at a closely aligned conceptual replication (Brandt et al., 2014; Coyne et al., 2016). At the onset, the goal was to explore whether the success of CHAAOS would generalize to an additional group of students with disabilities. The study followed the methods and implementation of O’Connor et al. (2019) closely, utilized the same researcher-developed measures, and used the original data for analysis. All of these components placed the study in an advantageous position for completing a closely aligned replication.

Improving CHAAOS Vocabulary Knowledge

The first research question addressed whether the learning of word meanings would improve in the replication students. Using the CHAAOS lessons resulted in improvement of academic word learning across all cycles for students at the replication site, and these improvements were comparable with the improvements of the CHAAOS treatment group in the study by O’Connor et al. (2019). These findings support the effectiveness of explicit vocabulary instruction for teaching academic words to middle school students with disabilities, such as LD, who face significant challenges in reading (Bos & Anders, 1990; Bryant et al., 2003; Fore et al., 2007; Seifert & Espin, 2012). Providing opportunities for students to learn, practice, and develop knowledge of word meanings was a meaningful practice that enriched the students’ word learning. The findings also demonstrate that replicating the original results with a new group of students with disabilities was possible. Therefore, it shows that CHAAOS could serve as a potential way to develop a core knowledge of academic words that students with disabilities must know when encountering academic texts.

Improving CHAAOS Comprehension

The second research question addressed whether the replication students would comprehend the CHAAOS vocabulary words in untaught sentences and passages. Indeed, using the CHAAOS lessons resulted in greater comprehension of CHAAOS words in context as measured at post-test; this improvement was consistent across cycles and comparable with O’Connor et al. (2019). These findings suggest that adolescent students who need significant support in reading benefited from multiple opportunities to develop their understanding of words; CHAAOS instructional activities provided illustrative and varied contexts and explicit practice and feedback on how to use the words in those contexts, which helped students learn and retain academic words so that they could transfer that knowledge immediately to text reading. This finding supports the existing research, which has shown that multiple encounters with words that elicit active processing support text comprehension (Elleman et al., 2009; McKeown et al., 2018; Stahl & Fairbanks, 1986). Studies have also shown that this type of instruction is especially beneficial for students with reading challenges (Elleman et al., 2009).

Using the BAU Comparison

Throughout this study, O’Connor et al.’s (2019) BAU students served as a comparison for the replication students. Across cycles, the replication students scored significantly higher than the BAU students on measures of word learning and comprehension. However, the BAU students did not receive comparable vocabulary instruction, so the positive effects reported herein may be inflated. Still, although an alternate vocabulary approach was not designed for BAU students, typical instruction still serves as a worthwhile comparison. The CHAAOS words were representative of academic words that frequently occur in secondary texts and across disciplines (see O’Connor et al., 2019, for word selection). Students are expected to understand words like evaluate, evidence, compare, contrast, and infer, which are included in the CCSS for literacy (NGA & Center for Best Practices & Council of Chief State School Officers, 2010). The BAU students knew approximately 30% of those words and failed to improve in knowledge concerning the definitions and comprehension of those words. Thus, exposing adolescent students with disabilities to these types of academic words incidentally might not be enough to support their vocabulary development (Bryant et al., 2003).

Generalized Reading Comprehension

The last research question addressed how the replication students performed on two generalized measures of reading comprehension, TOSREC and Maze. Learning CHAAOS words did not result in improvement in these measures. At the end of the school year, the present students, in addition to the O’Connor et al. (2019) treatment and BAU students, remained approximately 2 SDs below the mean on the measure of silent reading efficiency and comprehension. Furthermore, they all remained below the 15th percentile on the Maze measure of reading comprehension. In light of a vocabulary-only intervention, these results were understandable, especially because CHAAOS lessons focused on discussion and application of academic vocabulary with no attempt to increase the speed of reading. However, these scores truly underscore the severity of the reading challenges that all three groups of students experienced in middle school while reading grade-level text.

Limitations

Several limitations in the areas of experimental design and measurement should be considered when evaluating the present results. The research design would have been strengthened if a control group had been identified at the replication site and had received comparable vocabulary instruction. Doing so would have allowed CHAAOS lessons to be compared with an alternate format. Second, the special education teacher who provided replication instruction was a developer of the CHAAOS lessons and was involved with the O’Connor research team. This instructor may have produced an experimenter effect that threatened the external validity of the results. This limited the generalizability of the findings, and similar results may not have been evidenced with a teacher who was more removed from the CHAAOS instructional team.

Next, incorporating a standardized measure of vocabulary and comprehension may have strengthened the study. O’Connor et al. (2019) administered standardized measures of vocabulary and comprehension to assess students’ initial skill levels and to document whether the students exhibited any change in skills. However, no significant differences were found in vocabulary or reading comprehension over the course of the sixth-grade year. We hypothesize that similar results would have emerged in this study, likely because vocabulary interventions in general or special education settings rarely demonstrate gains on standardized measures. This could be because standardized vocabulary measures probe a vast set of words and are therefore insensitive to growth on a limited set of words taught during vocabulary interventions (Elleman et al., 2009; Pearson et al., 2007). Alternatively, vocabulary learning that occurs during interventions may simply not transfer beyond the local context (Elleman et al., 2009; McKeown et al., 2018; Pearson et al., 2007).

Future Considerations for Research and Practice

Even with limitations, this study revealed positive results for consideration. Replicating similar results in word learning and comprehension added support for CHAAOS instruction and bolstered the extensive literature supporting direct teaching of vocabulary to adolescent students (Lesaux et al., 2014; McKeown et al., 2018; Snow et al., 2009; Vaughn et al., 2009). Furthermore, it added to the literature that supports direct teaching of vocabulary specifically to students with disabilities who experience severe issues in reading efficiency and comprehension (Bos & Anders, 1990; Fore et al., 2007; O’Connor et al., 2015, 2019; Seifert & Espin, 2012).

However, a number of directions should be considered to help support the effectiveness of CHAAOS in the future. Additional investigations of CHAAOS would benefit from a strengthened longitudinal research design that includes a much larger sample, adequate comparison groups, and standardized vocabulary and reading measures. A larger design would require professional development and coaching support to assist teachers with instruction. Furthermore, these findings lead to further questions about the type and quality of vocabulary instruction that occurs in middle school special education classrooms. More deeply contextualizing vocabulary instruction prior to the use of CHAAOS routines would be beneficial for understanding the vocabulary instructional practices in special education classrooms.

In addition, the results of this study have a few implications for middle school educators. First, systematic vocabulary instruction can improve word learning. Special education teachers should take time to address academic vocabulary because students in their classes are likely in need of vocabulary support, and these students can learn targeted words with appropriate instruction (Bryant et al., 2003; Clemens et al., 2017). Second, the instructional routines in CHAAOS supported the transfer of word knowledge to immediate contexts in which the words were used. Using routines that required conversations about the words in context and that provided opportunities to read words in text supported students’ understanding of those words. Finally, the CHAAOS lessons are freely available for teachers to use, and while they only address 48 academic words, a small number compared with all the words students need to know, teachers could use CHAAOS lessons as a template for incorporating additional vocabulary instruction (see Beach et al., 2015; Beck et al., 2013, for instructional guidelines).

Conclusion

Without specialized instructional support in reading, adolescent students with disabilities run an exceptional risk of remaining behind their peers (Gilmour et al., 2019). This replication study provided evidence that the success of the CHAAOS intervention is generalizable to an additional group of students with similar disabilities. Importantly, it provided additional evidence that explicitly teaching key academic vocabulary can lead to valuable vocabulary word learning for adolescents and that such word learning transfers to the use of words in immediate contexts. We find this trial of CHAAOS instruction to be valuable, especially given the reading challenges of adolescent students with disabilities in secondary grades and the need to substantiate effective instructional practices that can raise student achievement in reading.

Footnotes

Associate Editor: Margaret King-Sears

Authors’ Note

This article is based on data published in .

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The research described in this paper was supported by Grant R324A160019 from the U.S. Department of Education, Institute of Education Sciences to the University of California at Riverside. The content is solely the responsibility of the authors and does not necessarily represent the official views of the U.S. Department of Education or the Institute of Education Science.

References

Ahmed

Francis

D. J.

York

Fletcher

Barnes

Kulesz

(2016). Validation of the direct and inferential mediation (DIME) model of reading comprehension in Grades 7 through 12. Contemporary Educational Psychology, 44–45, 68–82. https://doi.org/10.1016/j.cedpsych.2016.02.002

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J. A.

Fiedler

Funder

D. C.

Kliegl

Nosek

B. A.

Perugini

Roberts

B. W.

Schmitt

van Aken

M. A. G.

Weber

Wicherts

J. M.

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. https://doi.org/10.1002/per.1919

August

Branum-Martin

Cardenas-Hagan

Francis

D. J.

(2009). The impact of an instructional intervention on the science and language learning of middle grade English language learners. Journal of Research on Educational Effectiveness, 2(4), 345–376. https://doi.org/10.1080/19345740903217623

Beach

K. D.

Sanchez

Flynn

L. J.

O’Connor

R. E.

(2015). Teaching academic vocabulary to adolescents with learning disabilities. TEACHING Exceptional Children, 48(1), 36–44. https://doi.org/10.1177/0040059915594783

Beck

McKeown

(2007). Increasing young low-income children’s oral vocabulary repertoires through rich and focused instruction. The Elementary School Journal, 107(3), 251–271. https://doi.org/10.1086/511706

Beck

McKeown

Kucan

(2013). Bringing words to life: Robust vocabulary instruction (2nd ed.). Guilford Press.

Beck

I. L.

Perfetti

C. A.

McKeown

M. G.

(1982). Effects of long-term vocabulary instruction on lexical access and reading comprehension. Journal of Educational Psychology, 74(4), 506–521. https://doi.org/10.1037/0022-0663.74.4.506

Bos

C. S.

Anders

P. L.

(1990). Effects of interactive vocabulary instruction on the vocabulary learning and reading comprehension of junior-high learning disabled students. Learning Disability Quarterly, 13(1), 31–42. https://doi.org/10.2307/1510390

Brandt

M. J.

Ijzerman

Dijksterhuis

Farach

F. J.

Geller

Giner-Sorolla

Grange

J. A.

Perugini

Spies

J. R.

van’t Veer

(2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. https://doi.org/10.1016/j.jesp.2013.10.005

10.

Brasseur-Hock

I. F.

Hock

M. F.

Kieffer

M. J.

Biancarosa

Deshler

D. D.

(2011). Adolescent struggling readers in urban schools: Results of a latent class analysis. Learning and Individual Differences, 21(4), 438–452.

11.

Brown

S. H.

Lignugaris-Kraft

Forbush

D. E.

(2016). The effects of morphemic vocabulary instruction on prefix vocabulary and sentence comprehension for middle school students with learning disabilities. Education and Treatment of Children, 39(3), 301–338. https://doi.org/10.1353/etc.2016.0014

12.

Bryant

D. P.

Goodwin

Bryant

B. R.

Higgins

(2003). Vocabulary instruction for students with learning disabilities: A review of the research. Learning Disability Quarterly, 26(2), 117–128. https://doi.org/10.2307/1593594

13.

Catts

H. W.

Adlof

Ellis Weismer

(2006). Language deficits in poor comprehenders: A case for the simple view of reading. Journal of Speech, Language, and Hearing Research, 49(2), 278–293. https://doi.org/10.1044/1092-4388(2006/023)

14.

Clemens

Simmons

L. E.

Wang

Kwok

(2017). The prevalence of reading fluency and vocabulary difficulties among adolescents struggling with reading comprehension. Journal of Psychoeducational Assessment, 35(8), 785–798. https://doi.org/10.1177/0734282916662120

15.

Cook

B. G.

(2014). A call for examining replication and bias in special education research. Remedial and Special Education, 35(4), 233–246. https://doi.org/10.1177/0741932514528995

16.

Coyne

M. D.

Cook

B. G.

Therrien

W. J.

(2016). Recommendations for replication research in special education: A framework of systematic, conceptual replications. Remedial and Special Education, 37(4), 244–253. https://doi.org/10.1177/0741932516648463

17.

Cromley

J. G.

Azevedo

(2007). Testing and refining the direct and inferential mediation model of reading comprehension. Journal of Educational Psychology, 99(2), 311–325. https://doi.org/10.1037/0022-0663.99.2.311

18.

Elleman

A. M.

Lindo

E. J.

Morphy

Compton

D. L.

(2009). The impact of vocabulary instruction on passage-level comprehension of school-age children: A meta-analysis. Journal of Research on Educational Effectiveness, 2(1), 1–44. https://doi.org/10.1080/19345740802539200

19.

Engelmann

Johnson

Carnine

Meyer

Becker

Eisele

(1999). Corrective reading decoding strategies. SRA/McGraw-Hill.

20.

Ferrer

Shaywitz

B. A.

Holahan

J. M.

Marchione

Shaywitz

S. E.

(2010). Uncoupling of reading and IQ over time: Empirical evidence for a definition of dyslexia. Psychological Science, 21(1), 93–101. https://dx-doi-org.web.bisu.edu.cn/10.1177/0956797609354084

21.

Fore

C. I. I. I.

Boon

R. T.

Lowrie

(2007). Vocabulary instruction for middle school students with learning disabilities: A comparison of two instructional models. Learning Disabilities: A Contemporary Journal, 5(2), 49–73.

22.

Gilmour

A. F.

Fuchs

Wehby

J. H.

(2019). Are students with disabilities accessing the curriculum? A meta-analysis of the reading achievement gap between students with and without disabilities. Exceptional Children, 85(3), 329–346. https://doi.org/10.1177/0014402918795830

23.

Gough

P. B.

Tunmer

W. E.

(1986). Decoding, reading, and reading disability. Remedial and Special Education, 7(1), 6–10. https://doi.org/10.1177/074193258600700104

24.

Hjetland

H. N.

Lervag

Lyster

S. H.

Hagtvet

B. E.

Hulme

Melby-Lervag

(2019). Pathways to reading comprehension: A longitudinal study from 4 to 9 years of age. Journal of Educational Psychology, 111(5), 751–763. https://doi.org/10.1037/edu0000321

25.

Hock

M. F.

Brasseur

I. F.

Deshler

D. D.

Catts

H. W.

Marquis

J. G.

Mark

C. A.

Stribling

J. W.

(2009). What is the reading component skill profile of adolescent struggling readers in urban schools? Learning Disability Quarterly, 32(1), 21–38. https://doi.org/10.2307/25474660

26.

Holahan

J. M.

Ferrer

Shaywitz

B. A.

Rock

D. A.

Kirsch

I. S.

Yamamoto

Michaels

Marchione

K. E.

Shaywitz

S. E.

(2018). Growth in reading comprehension and verbal ability from Grades 1 through 9. Journal of Psychoeducational Assessment, 36(4), 307–321. https://doi.org/10.1177/0734282916680984

27.

Lesaux

N. K.

Kieffer

M. J.

Kelley

J. G.

Harris

J. R.

(2014). Effects of academic vocabulary instruction for linguistically diverse adolescents: Evidence from a randomized field trial. American Educational Research Journal, 51(6), 1159–1194. https://doi.org/10.3102/0002831214532165

28.

Lonigan

C. J.

Burgess

S. R.

Schatschneider

(2018). Examining the simple view of reading with elementary school children: Still simple after all these years. Remedial and Special Education, 39(5), 260–273. https://doi.org/10.1177/0741932518764833

29.

McKeown

M. G.

Beck

I. L.

Omanson

R. C.

Pople

M. T.

(1985). Some effects of the nature and frequency of vocabulary instruction on the knowledge and use of words. Reading Research Quarterly, 20(5), 522–535. https://doi.org/10.2307/747940

30.

McKeown

M. G.

Crosson

A. C.

Moore

D. W.

Beck

I. L.

(2018). Word knowledge and comprehension effects of an academic vocabulary intervention for middle school students. American Educational Research Journal, 55(3), 572–616. https://doi.org/10.3102/0002831217744181

31.

Nagy

Townsend

Lesaux

Schmidt

(2012). Words as tools: Learning academic vocabulary as language acquisition. Reading Research Quarterly, 47(1), 91–108. https://doi.org/10.1002/RRQ.011

32.

Nagy

W. E.

Anderson

R. C.

Herman

P. A.

(1987). Learning word meanings from context during normal reading. American Educational Research Journal, 24(2), 237–270. https://doi.org/10.3102/00028312024002237

33.

National Governors Association Center for Best Practices & Council of Chief State School Officers. (2010). Common core state standards for English language arts and literacy in history/social studies, science, and technical subjects. http://www.corestandards.org/wp-content/uploads/ELA_Standards1.pdf

34.

O’Connor

R. E.

Beach

K. D.

Sanchez

V. M.

Bocian

K. M.

Flynn

L. J.

(2015). Building BRIDGES: A design experiment to improve reading and United States history knowledge of poor readers in eighth grade. Exceptional Children, 81(4), 399–425. https://doi.org/10.1177/0014402914563706

35.

O’Connor

R. E.

Beach

K. D.

Sanchez

V. M.

Knight-Teague

Orozco

Jones

(2019). Teaching academic vocabulary to sixth grade students with disabilities. Learning Disability Quarterly, 42(4), 231–243. https://doi.org/10.1177/0731948718821091

36.

Oslund

E. L.

Clemens

N. H.

Simmons

D. C.

Simmons

L. E.

(2018). The direct and indirect effects of word reading and vocabulary on adolescents’ reading comprehension: Comparing struggling and adequate comprehenders. Reading and Writing, 31, 355–379. https://doi.org/10.1007/s11145-017-9788-3

37.

Pearson

P. D.

Hiebert

Kamil

M. L.

(2007). Vocabulary assessment: What we know and what we need to learn. Reading Research Quarterly, 42, 283–296. https://doi.org/10.1598/RRQ.42.2.4

38.

Perfetti

Stafura

(2014). Word knowledge in a theory of reading comprehension. Scientific Studies of Reading, 18(1), 22–37. https://doi.org/10.1080/10888438.2013.827687

39.

Proctor

C. P.

Carlo

August

Snow

(2005). Native Spanish-speaking children reading in English: Toward a model of comprehension. Journal of Educational Psychology, 97(2), 246–256. https://doi.org/10.1037/0022-0663.97.2.246

40.

Reed

D. K.

(2008). A synthesis of morphology interventions and effects on reading outcomes for students in grades K–12. Learning Disabilities Research & Practice, 23(1), 36–49. https://doi.org/10.1111/j.1540-5826.2007.00261.x

41.

Schleppegrell

M. J.

(2001). Linguistic features of the language of schooling. Linguistics and Education, 12(4), 431–459. https://doi.org/10.1016/S0898-5898(01)00073-0

42.

Seifert

Espin

(2012). Improving reading of science text for secondary students with learning disabilities: Effects of text reading, vocabulary learning, and combined approaches to instruction. Learning Disability Quarterly, 35(4), 236–247. https://doi.org/10.1177/0731948712444275

43.

Shinn

M. R.

Shinn

M. M.

(2002). AIMSweb training workbook: Administration and scoring of reading MAZE for use in general outcome measurement. Edformation.

44.

Snow

C. E.

(2010). AL and the challenge of reading for learning about science. Science, 328(5977), 450–452. https://doi.org/10.1126/science.1182597

45.

Snow

C. E.

Lawrence

J. F. W.

White

(2009). Generating knowledge of academic language among urban middle school students. Journal of Research on Educational Effectiveness, 2(4), 325–344. https://doi.org/10.1080/19345740903167042

46.

Stahl

S. A.

Fairbanks

M. M.

(1986). The effects of vocabulary instruction: A model-based meta-analysis. Review of Educational Research, 56(1), 72–110. https://doi.org/10.3102/00346543056001072

47.

Travers

J. C.

Cook

B. G.

Therrien

W. J.

Coyne

M. D.

(2016). Replication research and special education. Remedial and Special Education, 37(4), 195–204. https://doi.org/10.1177/0741932516648462

48.

Uccelli

Galloway

E. P.

Barr

C. D.

Meneses

Dobbs

C. L.

(2015). Beyond vocabulary: Exploring cross-disciplinary academic-language proficiency and its association with reading comprehension. Reading Research Quarterly, 50(3), 337–356. https://doi.org/10.1002/rrq.104

49.

U.S. Department of Education. (2019). The nation’s report card. National Center for Education Statistics. https://www.nationsreportcard.gov/reading/?grade=8

50.

Valentine

J. C.

Biglan

Boruch

R. F.

Gonzalez Castro

Collins

L. M.

Flay

B. R.

Kellam

Mościcki

E. K.

Schinke

S. P.

(2011). Replication in prevention science. Prevention Science, 12, 102–117. https://doi.org/10.1007/s11121-011-0217-6

51.

Vaughn

Martinez

L. R.

Linan-Thompson

Reutebuch

C. K.

Carlson

C. D.

Francis

D. J.

(2009). Enhancing social studies vocabulary and comprehension for seventh-grade English language learners: Findings from two experimental studies. Journal of Research on Educational Effectiveness, 2(4), 297–324. https://doi.org/10.1080/19345740903167018

52.

Wagner

R. K.

Torgesen

J. K.

Rashotte

C. A.

Pearson

N. A.

(2010). Test of Silent Reading Efficiency and Comprehension (TOSREC). Pro-Ed.