Abstract
This mixed methods study explores the pilot implementation of a Response to Intervention framework in the first grade classrooms in three urban schools. Two schools in a fully implemented condition (FI) with a facilitator and a partially implemented condition (PI) without a facilitator were investigated using student achievement data, field notes, teacher questionnaires, and teacher focus groups. Repeated measures ANOVA indicate that student risk levels measured by DIBELS phonemic awareness and decoding tasks were reduced in both conditions but favored the FI condition. Qualitative evidence indicates that programmatic changes were implemented in one year, but more complex paradigmatic changes did not occur.
Keywords
The 2004 reauthorization of the Individuals with Disabilities Education Act (IDEA 2004) provided school districts with a new alternative for the identification of children with learning disabilities. Chronic nonresponsiveness to effective instruction, Response to Intervention (RTI), may now be used as evidence of learning disabilities. The purpose of RTI is twofold: (a) to provide early intervention for students who are at risk for reading difficulties and (b) to develop a more valid procedure for identifying students with learning disabilities in the area of reading (Gersten & Dimino, 2006).
Previously, districts were required to use an IQ-achievement discrepancy model as the defining characteristic of reading disability (RD) and qualification for special education services. Robust scientific evidence demonstrates that IQ scores are irrelevant to the identification of learning disabilities and many children are not properly served or identified by relying on the IQ model (Fletcher et al., 2001; Fuchs & Deschler, 2007; Lyon et al., 2001; Siegel, 1989; Stanovich & Siegel, 1998; Vellutino, Scanlon, & Lyon, 2000). Additionally, research has revealed that referrals and assessment decisions are often arbitrary (Blanchett, Klingner, & Harry, 2009; Donovan & Cross, 2002; Klingner & Harry, 2006; Shinn 2007). Not only have there been increases in the numbers of children identified with reading disabilities, but also students of color and English Language Learners (ELLs) are disproportionately identified as needing special education (Blanchett et al., 2009; Donovan & Cross, 2002).
Components of RTI
Most conceptualizations of RTI incorporate three common components: (a) multiple tiers of instruction, (b) evidence-based instruction, and (c) systemic collaboration and coordination of schoolwide resources. Although we describe each component separately, in implementation they dynamically influence and inform each other in often seamless ways.
Tiered Instruction Based on Assessment Data
A premise of RTI is that children should not need to fail before receiving additional instructional support. The lowest performing children must be identified early and monitored as successively intense tiers of research-based literacy instruction are provided. While the specific number of tiers before due process may vary by implementation site, the three-tiered model is popular. For an elaborated explanation, see Johnson, Mellard, Fuchs, and McKnight (2006).
Based on screening assessment results, students who are likely to have difficulty learning to read are identified as at-risk. These at-risk students are closely monitored for responsiveness to their general education instruction (Tier 1). At a designated time point, a standardized test is administered to determine how their responsiveness to classroom instruction compares to an established norm or standard. One method of determining responsiveness is to compare student performance to a particular benchmark (Good, Simmons, Kame’enui, Kaminski, & Wallin, 2002; O’Connor & Klingner, 2010). Children who are determined to be unresponsive are given more intense instruction. Additional successive tiers of intervention ideally increase in intensity, frequency, duration and diagnostic individualization (Mellard, McKnight, & Jordan, 2010). During the Tier 2 intervention, progress monitoring occurs to determine their response to this more intense instructional intervention. Research indicates that intense instruction and progress monitoring can increase early reading achievement of at-risk students and reduce the numbers of children requiring special education services (McMaster, Fuchs, Fuchs, & Compton, 2005; Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994; Velutino et al., 2000).
Schools may employ a standard protocol or problem-solving model of intervention. In the problem-solving approach, a school team evaluates the struggling reader’s literacy data, identifies the problem(s), implements an appropriate intervention and then monitors the child’s progress (Vaughn & Denton, 2008). Most efficacy studies of RTI have been performed using standardized protocols applying fixed interventions for a fixed duration in the early grades (McMaster, et al., 2005; Shapiro, Zigmond, Wallace, & Marston, 2011; Velutino et al., 2000). In such programs, fidelity (the degree to which the instructors are delivering the program as intended) is easier to accomplish, assess, and ensure (Fuchs & Fuchs, 2006). This is particularly important in districts that are large or plan to scale-up their RTI model. However, the majority of the studies have been conducted with graduate students under the ongoing supervision of university researchers or in other highly controlled environments. It is important to investigate how variations in the ecology of a school setting might impact the use of resources and the standardized protocol, which in turn could have implications for outcomes (Fuchs & Deshler, 2007; Gersten & Dimino, 2006). Additionally, it is imperative that we explore cultural and linguistic implications of using a standardized RTI framework (Klingner & Edwards, 2006; Linan-Thompson, Cirino, & Vaughn, 2007; Orosco & Klingner, 2010).
Evidence-Based Instruction
To maximize the likelihood of student achievement, all tiers of instruction should apply instructional practices that have been substantiated by empirical research. Schools need to select programs and instructional techniques that have been validated through research with samples of participants that are similar to the school population. Schools might use the What Works Clearinghouse, the Best Evidence Encyclopedia, or other websites that review the effectiveness of commercial programs and procedures. Similarly, research reviews that discuss the effectiveness of particular instructional practices are now published by professional organizations and federal agencies (e.g. Shanahan et al., 2010). Additionally, one of the new explicitly stated criteria in determining eligibility for special education under the category of specific learning disability established that states “must permit the use of a process based on the child’s response to scientific, research-based intervention” (IDEA, 2004).
Systemic Collaboration and Coordination
Collaboration and coordination are required at all levels of schoolwide implementation and pose large challenges to the success of RTI (Gersten & Dimino, 2006). Systemic resource allocation also takes different forms by school. Due to the complexity of school dynamics, the existing research has predominantly focused on a single component of RTI such as assessment, a particular tier or type of intervention, or student risk reduction. To date, the projects that have investigated systemic implementation of RTI relied on federal funds and extensive support from university experts (e.g. Fuchs, 2009; Shapiro et al., 2011). Most urban schools do not have the financial or human resources afforded by grant-funded research teams. As a result, each school’s interpretation and implementation of RTI is likely to look different than an RTI framework that was shaped and facilitated by a research team.
Ecological Systems Theory (Bronfenbrenner, 1979; Bronfenbrenner & Morris, 1998) is a means of viewing the ways that the interaction of multiple systems directly and indirectly influence student development. These sytems include microsystems (the student’s immediate relationships), mesosytems (more far-reaching relationship interactions), exosystems (external settings and decisions that directly affect the child) and macrosystems (societal influences such as policy, culture, and economics). An ecological perspective provides a lens for viewing the complexities of the implementation of an RTI framework in three very different urban public schools.
Purpose
The purpose of this study was to determine the effectiveness of the RTI framework initiated by the special education office in a large urban school district. In particular, the study investigated how the district and schools approached the transition to an RTI model. A salient element of this study is the absence of a university research team directing and supporting this urban district’s efforts. We observed and documented the schools’ translations of RTI policy during this district’s pilot implementation of those policies. Our investigation addressed the following two research questions:
Research Question 1: Did this RTI framework reduce the number of first grade students at risk for reading difficulty?
Research Question 2: Within one school year, how effectively were these schools able to implement the three components (data-driven tiered instruction, evidence-based practice, systemic collaboration and coordination) associated with an RTI paradigm?
Method
A larger study was conducted in the primary classrooms (K-2) in three schools whose applications for participation were selected by the district Office of Special Education (OSE) based on their high percentages of special education referrals. This article will report on the outcomes in the first grade classrooms in three schools.
The first author became involved in the project during the year preceding the pilot implementation and study. The district OSE invited her to serve as a university representative on a 12-person RTI advisory panel. The advisory panel met three times to hear the ongoing plans of the OSE but without any official authority or decision-making power. The director of the OSE took the advisory panel’s comments and suggestions under consideration in creating the pilot RTI implementation plan for the district. All other advisory panel members had special education backgrounds. The first author has 30 years of experience as an elementary general educator and reading specialist. As a result, her views of RTI implementation often were viewed from the lens of a classroom teacher. After receiving the Internal Review Board approval of both her university and the District’s research review board, she entered the schools in the role of university researcher and reading specialist without any district affiliation. All teachers and principals understood that she neither reported to nor was affiliated with the district OSE. While our research team was allowed access to the schools data, allowed to visit the classrooms in the schools, and permitted to attend grade level meetings, we were observers. The district OSE made all decisions regarding RTI implementation at each school.
This is a mixed-method study primarily incorporating quantitative data collection and analysis with prolonged engagement and observation (one school year). In the schools, we performed in the role of observers and documenters without giving advice or assistance (Glesne, 1999). We employed triangulation involving observations, questionnaires, teacher self-report check sheets, transcripts of conversations with principals, teachers and facilitators to create a collective case study to explain the RTI implementation process and to elaborate on the quantitative findings. According to Orosco and Klingner (2010), mixed-methods studies are essential to help us understand the contextual variables in complex systems such as the implementation of RTI. Rossman and Wilson (1984) guided the ways that we linked qualitative and quantitative data: (a) to enable confirmation or corroboration of each other via triangulation; (b) to elaborate or develop analysis, providing richer detail; (c) to initiate new lines of thinking through attention to surprises or paradoxes.
Participants
School Settings
Schools were purposefully sampled from a pool of volunteers by the OSE, not the researchers. The OSE director chose three Title I schools in which more than 90% of the children receive free or reduced lunch. Additionally, the three schools were purposefully selected to represent the wide range of differences found in this urban district. Overall, the schools reflected ethnic and linguistic diversity characteristic of our urban schools, although they did not mirror each other demographically. These schools were neighborhood urban schools. Therefore, they reflected their neighborhood demographic; see Table 1.
School Demographic Information.
These three schools shared the challenge of referring relatively high numbers of children for special education in the primary grades (K-2), 8.7%. However, by state and district standards these schools were considered effective schools. Each school received a grade of A on its citywide school report card rating. Only the top 25% of all schools in the city receive an A. Grades are based on student achievement and formal parent and teacher evaluations of the school. All three schools met their state-mandated adequate yearly progress goals for reading (and math) achievement at all tested grade levels and disaggregated subgroups for several consecutive years. Parents and teachers in all three schools reported the highest levels of satisfaction compared to other schools on the city’s School Survey in the categories of academic expectation, communication, engagement and safety.
The literacy block in each school was 90 min. Reflective of the philosophic diversity in this large district, the schools did not share a common core literacy program. The OSE director intentionally sought schools with the programmatic variation that mirrored the district at large. As a result, she did not deem it desirable to demand that pilot schools implement a common Tier 1 core literacy program. Such demands could not be met in the planned scale-up in a district with more than 1,700 schools, so it was not required in this pilot implementation.
Before the RTI implementation, School A and School B had never used any formally agreed upon phonics or word study program. School A and School B used a reading and writing workshop model. Both lesson plans and classroom observations revealed that teachers at School B also used the stories from a basal reader for whole group shared reading on average twice a week. School C was a Reading First school that used a basal reading series, including its word study program. In November 2007 School C withdrew from the district’s pilot RTI project because Reading First requirements conflicted with OSE requirements, but they agreed to participate in our research. The effect of School C’s withdrawal from the district’s pilot project resulted in the removal of an RTI facilitator. In this article, Full-Implementation (FI) will be the designation assigned to School A and School B, not because we consider them to have fully implemented all RTI components, but because they received the full package of implementation resources provided by the district. Partial-Implementation (PI) will be the designation assigned to School C because it received all of the financial benefits and product resources afforded to FI schools, but they simply did not receive the aid of an OSE RTI Facilitator after November of the implementation school year.
Teacher Participants
First grade teachers at each school participated in the study. There were 2 teacher participants in School A (19 and 9 years of teaching experience). Of the 4 teachers in School B, 1 teacher had 2 years of experience, 1 teacher had 5 years of experience and 2 teachers had 6 years of experience. One of these teachers also had ELL certification. The 3 teacher participants in School C had 1, 10, and 24 years of experience. All teachers in the study had master’s degrees in childhood education. Additionally, the OSE RTI facilitators working with School A and School B participated. Both facilitators had master’s degrees in special education and more than 25 years of experience in the school system as teachers and administrators.
Student Participants
Looking closely at first grade enabled us to focus on the children’s progress on several foundational early reading skills. Additionally, first grade abilities are often seen as important indicators of continued reading progress (Good et al., 2002; Juel, 1988). The student participants were 160 first graders. 110 students were from two classrooms in School A and four classrooms in School B, including 45 ELLs. The largest percentage of ELLs spoke Spanish as their first language. Three first grade classrooms in School C consisted of 50 English Only (EO) African American students born in this country.
Data Sources
Student Achievement Data
To determine levels of student achievement, we used student data that was collected by their teachers during the course of typical instruction and the routine assessments given at each grade level as part of the district’s RTI framework. The prescribed grade level tasks from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) were administered three times (T1, T2, T3) throughout the year as a standardized measure of student progress (Good & Kaminski, 2002). The tasks are
Phoneme Segmentation Fluency (PSF): Given a word orally (without seeing the word), the child says each phoneme in the word separately. One point is scored for each individual sound identified. (T1, T2, T3)
Correct Letter Sound Fluency (CLSF): The child reads a list of nonsense words. One point is scored for each correct letter sound read. (T1, T2, T3)
Oral Reading Fluency (ORF): The child is allowed 1 min each to read three grade level passages. The score is the median number of words read correctly in 1 min. (T2, T3)
Although each school’s comprehensive Tier 1 primary literacy program included attention to broader literacy development such as writing, comprehension and conceptual vocabulary, the principal focus of the pilot RTI framework and this study was on word recognition and alphabetics. These are the areas that are most likely to be used as indicators at this developmental level of a possible RD. Both the DIBELS assessments and the interventions were aligned in their exclusive focus on word recognition and alphabetics.
Other Student Data
In School B and School C, the Tier 2 intervention was administered during the before-school extended day program by the classroom teachers. In School A, certified teachers conducted the interventions during the school day. However, support staff without their own classrooms are often called upon by administrators for tasks such as test proctoring, covering classes, and even substitute teaching in emergencies. Therefore, Tier 2 attendance data was viewed as important.
Additionally, we looked at the first grade retention data and special education referrals at each school. While we didn’t expect to see changes in special education referrals during the preliminary year of implementation, we did collect school referral data to be part of an ongoing longitudinal view of the referral trajectory.
Teacher Questionnaires and Focus Groups
In the fall and late spring, the teacher particpants responded to the Instructional Practices Questionnaire (IPQ) to document their views and practices related to literacy instruction. The first section of IPQ required teachers to describe their professional background and their classroom literacy program. The remainder of the questionnaire used Likert scales with space for additional teacher comments regarding the frequency of classroom literacy practices and their degree of confidence with different types of learners. The IPQ was developed by the researchers and had been piloted and revised during two summer institutes with similar groups of teachers. Each teacher’s answers were later recorded individually by question on an Excel spreadsheet. This enabled us to view teacher and school similarities and differences at a glance. Since the questions on the questionnaire were organized by topic, using the Excel spreadsheet enabled us to compare things such as teacher training, minutes devoted to particular instructional components, and instructional techniques favored by particular teachers. These data was used to explain the summative, quantitative achievement data.
Focus group conversations were conducted with each team of first grade teachers at the end of the school year. A semistructured interview protocol was designed to elicit a discussion of the teachers’ experiences and beliefs regarding the implementation of RTI. Questions addressed resource allocation, professional development, communication among faculty regarding instructional decisions, and how decisions were made regarding special education referral and grade level retention. The first author served as facilitator. The second author took notes, audio-recorded and transcribed the audiotapes. The first and second author coded the transcripts collaboratively during discussion meetings within the month following the focus group meeting.
Observations
We took descriptive field notes during at least one formally scheduled 30- to 45-min observation in each participating teacher’s classroom during literacy instruction. Additionally, we visited the classrooms more informally and took field notes during a portion of the literacy lesson on a monthly basis. Our aim was to observe the classroom literacy curriculum and procedures, not to insure fidelity to the RTI framework or to evaluate the instruction. Our observations always took place during general literacy instruction, not during the implementation of the commercial phonics program. Additionally, the teachers completed a weekly check sheet of classroom literacy activities. Our field notes, the teacher check sheets, and the teacher questionnaires provided a portrait of the comprehensive literacy program in Tier 1. Since the schools did not share a common literacy program, this collection of qualitative data was essential in identifying the instructional practices employed by the teachers during the 90-min literacy block in each school. It was used to explain the student achievement data and to determine whether students had received evidence-based instruction in Tier 1.
RTI facilitators assigned to each FI school by the OSE used check sheets to monitor each teacher’s fidelity to the scripted commercial phonics program that was adopted for use in Tier 1 and Tier 2. The producer of the commercial phonics program provided the check sheets that directed attention to specific teacher and student behaviors during 15 lesson components. The facilitators observed the teachers three times throughout the year formally using the check sheet and conducted informal bimonthly drop-ins.
Finally, during our monthly visits to the schools we observed and took field notes during the grade-level planning meetings. These meetings provided a lens for viewing how the teachers planned instruction. During the meetings, the teachers also addressed the progress or lack of progress of particular students. Therefore, these field notes provided insights into the ways that teachers at each school made instructional decisions and the role of collaboration. The first and second author coded the field notes collaboratively during a debriefing session on the day following the observation.
Procedures
Instructional Interventions
The FI schools in this study used a double-dose (Tier 1 and Tier 2) standard protocol intervention. A commercial phonics program, Wilson Fundations®, was added to each school’s existing comprehensive literacy program. It is a lock-step, scripted phonics program based on a systematic sequence of instruction that calls for 30 to 45 min of instruction per lesson. The district special education administration justified this decision by reasoning that a scripted program increased the likelihood that school personnel could apply the intervention with consistency in format, style, and intensity within widely diverse school-selected literacy programs. Additionally, standardized protocols are easier to implement, less expensive, and often incorporate a professional development model that yields fidelity and effectiveness. Administrators hoped that applying a double-dose of a common intervention in both Tier 1 and Tier 2 would build consistency between the two settings.
All schools had a 90-min literacy block and a 30-min phonics block everyday in Tier 1. FI schools used Fundations® as their phonics program in Tiers 1 and 2, PI schools used their basal phonics materials in Tier 1. In all three schools, certified teachers formally trained by Wilson supplemented Tier 1 instruction with a small group Tier 2 intervention of Fundations® four days a week for 30 min.
Professional Development
All participating teachers in both conditions, FI and PI, attended a full-day Wilson Fundations® professional development session prior to the start of the school year. All Tier 2 teachers in both conditions attended a half-day professional development session in the fall to learn how to adapt the Fundations® lesson plans to the Tier 2 intervention group. Only the FI Tier 2 teachers participated in a January professional development session on progress monitoring offered by Wilson. Trained OSE facilitators observed each teacher twice a month, provided ongoing feedback, and modeled lesson components to insure fidelity. After November, the PI schools did not receive any facilitator support. All three schools received complete sets of teacher and student Fundations® materials for the primary grades and access to online resources.
Implementing a Tiered Model
All three schools used formative assessments to inform classroom instruction in the primary grades. Formative assessments were adaptations of the tasks found in Clay’s Observation Survey (2006). Preliminary classroom screening assessments included untimed measures of children’s knowledge of letter recognition, print concepts, high-frequency word recognition, and leveled text reading accuracy. The classroom teacher administered the informal assessments during the first month of the school year. Based on preliminary assessment results, students who seemed likely to have difficulty learning to read were identified as at-risk. For 8 weeks, these at-risk students were monitored using teacher observation and informal classroom assessments for responsiveness to their comprehensive general education instruction (Tier 1). Comprehensive Tier 1 instruction always included attention to comprehension, conceptual-oral vocabulary and writing processes.
In late October, the prescribed first grade DIBELS assessment tasks were administered as a universal screening tool to determine how each child’s responsiveness to classroom instruction compared to established risk predictors (Good et al., 2002; Good & Kaminski, 2002). Children who were determined to be unresponsive to general classroom instruction and at-risk for literacy difficulties were given more intense instruction in a small group Tier 2 setting. After eight weeks, the winter benchmark DIBELS tasks were administered to all children. At this decision point, students could have exited Tier 2 or continued receiving Tier 2 support during Cycle 2. Children in the classroom who did not meet DIBELS winter benchmarks entered the the second cycle of Tier 2 intervention at this time point. At each decision point, a review team consisting of the principal, Tier 2 teacher, and OSE facilitator met to review test data and to evaluate the progress of each child. An additional 8 weeks of instruction in Tier 1 and Tier 2 took place. At decision point 3, the spring DIBELS assessment tasks were administered. Teachers identified nonresponders based on DIBELS benchmarks and classroom performance data.
In most descriptions of RTI in the literature (see Johnson et al., 2006), children who do not respond to the Tier 2 intervention are recommended for Tier 3, a more intense level of instruction. In some RTI models, this is when a due-process evaluation considering the possibility of special education services might take place (Fuchs, Stecker, & Fuchs, 2008; Johnson et al., 2006). However, during the pilot year this district had not clearly defined a tertiary course of action for students who did not respond to the standard protocol in Tier 2.
Results
In this section, we adress each research question and report the findings. Student data was analyzed to invetsigate student achievement. Our qualitative data was used to evaluate the implementation of the district’s RTI framework in the three schools.
Did Early Intervention Increase Reading Achievement and Reduce Risk?
We used student test data to gauge reading achievement and risk status shifts. We analyzed DIBELS data to determine gains on each DIBELS task. Additionally, we wanted to determine whether the percentage of children at risk for reading difficulties was decreasing across the school year. Finally, in an urban setting it was important to determine whether ELLs were making achievement gains on the foundational skills that were the focus of this district’s intervention.
Gains
The means and standard deviations for the DIBELS in the fall (Time 1), winter (Time 2), and spring (Time 3) are reported in Table 2. To test whether literacy gains occurred equally for the FI and PI conditions a repeated measures ANOVA was conducted in which implementation level was the between subjects factor, and pre- and posttest scores for each testing time were the within subjects factor on each DIBELS task.
Means for Repeated Measures of DIBELS.
The good news is that in these high poverty urban schools, first graders showed statistically significant gains between each testing period for each foundational skill tested by DIBELS. As displayed in Table 3, both groups showed significant gains in PSF over time, both between fall and winter, F(1, 158) = 113, p < .001, and between winter and spring, F(1, 158) = 18.3, p < .001. The rate of improvement was greater for students in the PI school in the fall-winter time period, F(1, 158) = 9.8, p = .002, but gain rates were comparable between winter and spring, F(1, 158) = .34, p = .56. The rate of improvement was greater for the PI condition from fall to winter on PSF. The greater gain by students in the PI school represented a “catching up” with the students at the FI schools over the fall-winter time period.
Repeated Measures ANOVA of Literacy Gains.
Second, both groups showed significant gains in CLSF over time, between fall and winter, F(1, 158) = 64.2, p < .001, and between winter and spring, F(1, 158) = 41.4, p < .001. The rate of improvement was not different at a level of statistical difference for the two groups between the fall and winter, F(1, 158) = 1.1, p = .3, but the FI condition made greater gains from winter to spring, F(1, 158) = 11.6, p = .001.
Oral reading fluency was only tested in the winter and spring. As displayed in Table 4, both groups showed significant gains in ORF over time, F(1, 165) = 140, p < .001; however the rate of gain in both conditions was comparable, F(1, 165) = .40, p = .53).
DIBELS PSF Prepost Percentage of Children at Each Proficiency Level.
Note: Level 1 is high risk, Level 2 is some risk, Level 3 is low risk category for reading difficulties.
Risk Level Reduction
Each school’s primary goal was to reduce the number of first graders at risk for reading failure. We compared the percentage of children assigned to each risk level based on their performance on the DIBELS tasks in the fall to the percentage of children in each level in the spring. See Table 4. Good et al. (2002) present a discussion of these levels.
Risk categories were reduced for the majority of children on PSF regardless of condition. All children who were in the high-risk level at baseline on the PSF task moved into lower risk levels at Time 3. On CLSF, the FI condition was more effective at risk reduction. Although the mean reading rate increased significantly from winter to spring, elevated developmental expectations yielded greater numbers of children shifting to higher risk categories on ORF in both conditions.
Comparing EO and ELL Gains
A repeated measures ANOVA was conducted to determine if EO and ELL students made comparable gains on all three DIBELS measures. Language group was the between subjects factor, and pre- and posttest scores for each testing time (fall, winter, spring) were the within subjects factor on each DIBELS task. On the tests of PSF, both groups showed statistically significant gains, F(1, 108) = 8.5, p < .001, from baseline to outcome, and the magnitude of gains over times did not differ, as tested by the Group × Time interaction, F(1, 108) = .99, p = .32. Similarly, tests of CLSF showed that both groups made statistically significant gains, F(1, 108) = 1.5, p < .001, from fall to spring and the magnitude of the gains did not differ, F(1, 108) = .04, p = .84. Finally, both groups demonstrated statistically significant gains from winter to spring on oral reading fluency, F(1, 108 = 8.6, p < .001, and the Group × Time interaction indicated that there was not a statistical difference between the magnitude of gains made by EO and ELL students, F(1, 108) = 2.08, p = .15. See Table 5.
Repeated Measures Analysis of Variance Comparing Gains by Language Group.
p < .05.
Within one school year, how effectively were these schools able to implement the three components (data-driven tiered instruction, evidence-based practice, systemic collaboration and coordination) associated with an RTI paradigm?
We made extensive use of the field notes that we took during classroom visits, grade team meetings, and focus groups to address this question (Rossman & Wilson, 1984). Qualitative data analysis was ongoing. The first two authors debriefed after each school visit to compare and contrast our notes and observations (Strauss & Corbin, 1998). Analytic notes were recorded during these debriefing sessions. First, we coded the data by RTI components. For example, the classroom observation field notes were primarily coded to inform whether Tier 1 instruction was evidence based. However, data from the questionnaires addressed evidence-based practice and school systems. Second, the notes were analyzed and coded for emerging themes. Themes that emerged had to do with stakeholder roles, scheduling, and meeting student needs. We triangulated multiple sources of data to get a more complete picture of the instruction, RTI implementation process, and the interaction of multiple systems (Rossman & Wilson, 1984). For example, we noted discrepancies between information reported in the teacher questionnaires and our observations of classroom practice. Additionally, information transcribed and coded from grade team meetings and focus group conversations was used to support interpretations of the quantitative data and school processes relating to the RTI framework.
Tiered Instruction Based on Assessment Data
The DIBELS assessments were used to determine whether children needed the additional intensity provided by the double dose of Tier 2 Fundations instruction. There was a clear connection between the phonics program and the skills that were assessed on DIBELS, PSF and CLS. Children were taught and tested a limited set of items (letters and sounds) in ways that were similar and made the transfer from instruction to assessment outcome fairly direct. Things proceeded smoothly when this transfer occurred as expected. However, none of the schools had constructed a decision tree to dictate what would happen if a child did not achieve the benchmark goal after one or more rounds of the standard protocol Tier 2 intervention.
According to our field notes, this lack of specificity seemed to result in a dual-pronged approach. In one prong, dedicated, well-intended Tier 2 teachers often continued to make modifications to the Tier 2 experiences for poor responders, including adjusting the level of instruction or adding a third session during the day so that additional repetitions of the lessons could be conducted in dyads, triads, or individual settings. In the second prong, classroom teachers acted independently and traditionally in making decisions about the future of their students. In many cases this meant retention.
Evidence-Based Instruction
Observations and questionnaires provide evidence that each school’s comprehensive primary level literacy program met the basic standards established by Johnson et al. (2006, p. 39) and the NRP for judging a high quality core program. Instruction in each school emphasized the five critical elements of reading, the core program occurred for more than 90 min a day and professional development focused on improving instruction. In particular, the PI school was a Reading First school with all of the resources, professional development support and machinery in place that is intended to foster instructional rigor.
Missing pieces
However, our classroom observation field notes, lesson plans, and focus group conversations provided evidence that although the instruction addressed the five critical elements of instruction the manner that students engaged with text may not have been optimal for promoting reading growth. The increased emphasis on phonics compromised the amount of connected text reading by the children during Tier 1 instruction in the FI schools. Additionally, we observed that children in the FI setting were only minimally held accountable for accurately reading little books in their reading workshop. Teachers held 3- to 5-min conferences with children once a week. Often the 100 to 200 word texts were below grade level. Children in the PI condition were exposed to more difficult grade level texts in the basal reader, but they too were only minimally held accountable for reading performance on a daily basis. Without verbal coaching and accountability, children in both settings seemed unable to connect and apply newly acquired word recognition skills into focused, meaningful (or fast and accurate) reading of connected text. In our urban setting, this transfer was particularly difficult for ELLs. Despite achieving DIBELs PSF and CLSF benchmarks, many ELLs still struggled to fluently read and understand grade level texts. As a result, teachers retained disproportionately higher numbers of ELLs than other children in the cohort.
Professional development
Although the schools provided four days of general professional development during the course of the school year, these sessions addressed content areas across the curriculum and were not a sustained study of an instructional strategy set with classroom follow-up. However, ongoing professional development of the phonics program did occur with the facilitators in the FI schools. Facilitators visited each FI school one full day each week during the first semester and two or three days a month during the second semester. The facilitator provided ongoing support and coaching for the FI schools in the form of organizing materials, acting as a Wilson Fundations® liaison, modeling lessons, observing lessons, providing lesson feedback, conducting fidelity checks, managing data collection, and spearheading the Tier 2 decision process as expressed by these teachers in our focus groups.
What role did Susan (pseudonym) play this year?
She gave us a lot of feedback.
She did a lot of modeling.
She showed us what to do and how to do it.
The PD at the beginning of the year wasn’t great. Susan put it together for us and helped us get materials.
As experienced school leaders, the facilitators were also able to find creative and effective ways to manage program implementation challenges. For example, when Tier 2 students in one school did not have the prerequisite skills for the lock-step lessons, the facilitator rearranged grouping and adjusted the instructional level of the materials. This support resulted in clear instructional differences between the FI schools and the PI school. Based on our triangulation of teacher check sheets and classroom observation field notes, there is evidence that phonics instruction in the PI schools lacked the intensity, explicitness, and systematic sequence found in the FI schools. We believe that facilitator support may partially explain the differences in results. The PI school made much stronger progress between the Fall and Winter testing. A facilitator worked with the PI school early in the year to help initiate the process. However, she discontinued her work with them late in November when they withdrew from the district’s pilot. In the FI schools the facilitator worked with the teachers all year to help them refine their instruction and guide the process of data-driven instruction. Their gains continued and increased across the year.
This was the first experience teaching phonics for most teachers in the FI schools. They welcomed both the program and the facilitator’s support. In both FI schools, the teachers were grateful for the insights and help provided by these respected, experienced mentors.
What supports were instrumental to learning the program this year?
Susan gave very personal professional development.
Susan watched us and gave us immediate feedback.
We had meetings about what’s coming up to prepare us.
And some modeling. In a very nonthreatening way.
Systems of collaboration and coordination
Attendance in Tier 2
The three schools were all Title 1 schools and each had funded 37-min before-school extended day allocations. Originally, it made sense for the Tier 2 instruction to occur during this time slot. This ensured that Tier 2 supplemented the full comprehensive literacy block. However, many of the children who needed Tier 2 instruction frequently arrived late for school. Midyear, School A arranged to conduct Tier 2 intervention groups during the school day. School B and School C were unable to make any kind of adjustment midyear, so in both conditions Tier 2 attendance was not optimal. See Table 6.
Attendance Means for Students Assigned to Each Tier 2 Cycle.
RTI and retention
Retention rates in both conditions were comparable. The PI school retained 6% of their EO first grade population. The FI schools retained 7% of the first grade population at the end of the first year of the RTI pilot. Although only 41% of the first graders were ELLs, 75% of those retained were ELLs. This indicates that disproportionally high numbers of ELL students were retained in first grade. When questioned about the high percentage of ELL retainees during the focus group conversation, a teacher responded,
I didn’t realize that it was that high but we do realize that it’s an issue. At this point, we’re doing so much with literacy I don’t know how much more can be done for these children. Tier 2 instruction, writing, reading, vocabulary. I don’t know how much further you can go when these children are not meeting the standards. We do have 60% Hispanics. It seems to be more in first grade. They haven’t met the standards. It’s never just reading because it’s everything, across the board.
Communication systems
Grade level team structures, including common team planning times, existed in all three schools. Data teams, consisting of the principal, assistant principal, and school data managers met regularly to review the school assessment data. Academic Intervention Services (AIS) teams reviewed students’ progress and determined next steps for supporting the children who were encountering learning difficulties. The AIS team also determined which children were candidates for formal testing by the school psychologist. However, during this pilot year the data team, the AIS team, the RTI team and the classroom teacher all functioned separately making decisions about children’s lives without sharing data with each other.
Special education referral rates
According to OSE, the three schools were selected for the RTI pilot project based on their high initial referral rates. OSE reports collectively group all special education referrals together without identifying specific student need. Although we did not expect to see any shift in special education referrals in the first year of implementation, we collected data so that we would be able view shifts over time. Table 7 displays the percentage of the school population initially referred for special education services during each time period within the last two years as reported by OSE. The numbers provide evidence of a decrease in the FI school referral rates.
Special Education Referral Rates.
In end-of-year focus group conversations, several teachers conveyed a vague understanding that RTI had taken the place of special education referrals, viewing special education referral by classroom teachers as an extinct option. One first grade teacher explained, “Special Education has to be requested by the parents. The decision has to be made by more people than the teacher. It has to be discussed with administrators, teachers, specialists.” We are cautiously suggesting that the drastic decrease in special education referrals in FI schools displayed on Table 7 may be the result of the teacher’s reported misconceptions about a referral prohibition. The lack of definition of Tier 3 and post-Tier 2 procedures, may have resulted in a reduction of referrals for appropriate initial referral candidates perpetuating the “wait-to-fail” scenario for the children with the most severe needs.
Discussion
As a complex process RTI may be illuminated and better understood in terms of ecological systems theory (Bronfenbrenner, 1979; Bronfenbrenner & Morris, 1998). Certainly the altruistic goal of RTI is to insure that children get the support they need without experiencing extended failure. However, politically and financially the impetus for RTI is to find a means to reduce the steadily increasing numbers of children qualifying for special education (macrosystem). In urban schools, we are also battling the special education classification of disproportional numbers of children of color and linguistic diversity (Blanchett et al., 2009). A series of studies supported the notion that providing intense evidence-based instruction to struggling novice readers does result in a reduction in the numbers of children requiring special education support (McMaster et al., 2005; Pinnell et al., 1994; Velutino et al., 2000).
RTI requires the application of effective, empirically based instructional models in the classroom and intervention setting (microsystem). It also requires the implementation of a sensitive, ongoing assessment system that is easily interpreted by teachers and used to drive instruction. It requires the collaboration of school faculty and resource personnel who have traditionally not worked together closely (mesosystem). Externally, it requires masterful scheduling and systems organization to provide opportunities for faculty to meet together for collaboration and to ensure needs-based, multilevel professional development (exosystem). Individually, each of these goals could require more than one year of school reform effort and sustained professional development.
Considering such complexities, all three schools were selected to participate in the district’s pilot RTI project because they were healthy schools. The OSE Director knew how difficult it would be to affect any worthwhile change in one year in a troubled school or in a school with faculty who were resistant to change. This urban district has more than 1,700 schools. Before allowing a new RTI policy to impact thousands of children, the OSE Director wisely decided to start small in the primary grades in three schools. It was important to introduce RTI in a setting that could serve as a petri dish, making it possible to establish, observe and adjust the preliminary standardized RTI framework before scaling up at the district level. Using the systems defined by Bronfenbrenner (1979; Bronfenbrenner & Morris, 1998), we will discuss the pilot implementation of RTI in three urban schools.
The Microsystem: Direct Interactions With the Children
Achievement and Risk Status Shifts
Like many new RTI projects, this district’s program was initiated and financially supported by the district OSE. Our research provides evidence that within one year the FI schools who worked with facilitators and ongoing OSE support were able to make continuous improvement in their effective implementation of a program that yielded increased student achievement and decreasing numbers of children in high risk categories in the district’s area of concentration, phonemic awareness and decoding. However, the provision of effective tertiary services for the students at highest risk for reading problems was unable to be accomplished in one year by these urban schools.
The district’s focus for supplementary support was limited to phonemic awareness and decoding instruction in the primary grades. Since the two FI schools had not previously incorporated a systematic phonics program as part of their comprehensive literacy curriculum or as a supplemental intervention, the introduction of a phonics program made an immediate positive difference. Consistent with earlier studies, a Tier 2 intervention that focused on constrained skills (phonemic awareness, phonics, oral reading fluency) yielded positive outcomes with at-risk early readers, including ELLs (Gunn, Biglan, Smolkowski, & Ary, 2000; Linan-Thompson, Vaughn, Hickman-Davis, & Kouzekanani, 2003). The outcomes achieved in the FI schools within one year are commendable. Children who were at risk for reading difficulties were quickly identified and provided with small group support. The number of students in high-risk categories on essential foundational reading skills was reduced. ELL students made comparable gains to EO students on skills that are the basic building blocks of reading. Stakeholders in urban schools view this as great news. The reduction of children in an at-risk status on foundational basic reading skills increases the likelihood that they can be successful in complex literacy processes. These schools celebrated and publicized these changes while neglecting the following concerns that we shared with them in our reports.
First, we have to wonder if the focus on highly constrained skills was a contributing factor to the lack of risk reduction on DIBELS ORF, the only measure of connected text reading. It makes sense that an increase in the foundational phonemic awareness and phonics skills measured by PSF and CLSF should translate into improved reading fluency. Instead, risk categories increased on ORF between the midyear and end-of-year tests. Our qualitative data leads us to surmise that the quality and quantity of connected text reading by the children during Tier 1 instruction did not prepare them to handle the text difficulty of the grade level DIBELS ORF passages. Additionally, we never observed teachers coaching children to seamlessly apply decoding strategies during the reading of connected text.
Second, there were a few children who were considered nonresponders, children who were not achieving the benchmark goals. According to IDEA (2004), the children who have not responded to these tiers of evidence-based instruction may truly need the more individualized, sustained, specialized expertise offered by the special education program. So much emphasis had been placed on the implementation of the phonics program, that there was not clarity at any level about the steps that needed to be taken with the children who did not respond well to the Tier 2 standardized phonics intervention. During the first year of implementation, these schools had neither the foresight nor the resources to provide more intense interventions or interventions that addressed needs other than alphabetics.
Instructional Needs of ELLs
The growth of ELLs on the word recognition abilities taught and tested in this project was parallel to the growth of EOs. To us, it implies that all children benefit when 30 min a day is devoted to systematic, explicit phonics instruction in the early grades. More importantly, it implies that the majority of the literacy block must be devoted to the more complex literacy components, like comprehension, conceptual vocabulary, and writing processes that tend to be the ongoing reading challenges for the ELL student. Based on conversations with teachers, the decision to retain a high percentage of ELLs despite their success with foundational skills was typically based on their struggles with broader classroom literacy expectations. Our findings are consistent with Orosco and Klingner (2010), who determined that the teachers working with ELL students as part of an RTI model lacked both theoretical and pedagogical knowledge needed to support the reading growth of ELLs. In our study, only one first grade teacher had ELL certification. Additionally, consistent with Klingner & Harry (2006), we found that decision-making teams had serious voids in expertise regarding appropriate instructional considerations for ELLs. In schools with ELL populations, sustained professional development must be provided to all school faculty to enhance this knowledge (Blanchett et al., 2009; Boute, 2012; Klingner & Harry, 2006).
The Mesosystem: Interactions Between People Directly Involved With the Children
Overall, the weakness of the pilot RTI framework in this district seemed to be an emphasis on program rather than process. Classroom teachers were at the center of the shift to initiate Fundations® (microsystem), but they were at the periphery of the decision-making process throughout the year (broader systems). The facilitator and the Tier 2 teachers made the decisions about children’s progress and interventions with only informal conversations with the classroom teachers. The lack of clear definition about how to define a nonresponder and what to do with nonresponders resulted in children being retained or receiving intense, repetitive phonics instruction.
An important mission of the RTI legislation and RTI research is to get students the help they need as early as possible. Born in that mission is the idea that all school personnel will work collaboratively to view Tier 2’s nonresponders individually to provide whatever it takes to maximize each child’s academic potential. There was a lack of cohesion in the school’s efforts to work together for the good of the individual child. Voices were left out of the collaboration and school faculty often lost sight of the individual child, resorting instead to generalized solutions.
The Exosystem: External Settings and Decisions That Impact the Children
The Role of the Principal
Scheduling, faculty collaborations, and professional development are crucial aspects of RTI that fall under the direct jurisdiction of the school principal and his or her administration. Each school was led by an effective principal with decades of classroom experience as well as a strong track record as the school’s administrator. In each school, the principal represented a maternal professional mentor trusted and respected by the teachers. These were women who got things done in their school and for their school. In establishing a new RTI paradigm, a strong, visionary administrator seems to be an essential element. These principals advocated at the district level for their school to be the recipient of the OSE support. Being selected was a financial windfall.
However, these principals turned over too much of the responsibility for implementing RTI to the OSE Director and her representatives, the facilitators. These were strong principals but they didn’t seem to own the RTI implementation in their schools. They relied exclusively on defining and implementing RTI as introduced by the district OSE. There was a misplaced trust that the OSE would own this process, when in fact the OSE supported the implementation of the phonics program but didn’t have the inside knowledge needed to transform the in-house systems required for an RTI transition. To move beyond viewing RTI as an instructional program, school leadership must create the structures that fit the basic tenets of the RTI paradigm with the unique resources and personnel existing in their school. The leadership team creates the vision of the destination and maps the journey. The complexity of the transition to RTI seems to merit mapping implementation phases over a three to five year period. School leadership needs to oversee and orchestrate the overall process.
The Role of the Facilitator
Evidence in this study seems to indicate that a coach or facilitator is necessary to operationalize the instructional details of the process. One key element differentiating the FI schools and the PI school was the support of the facilitator. The facilitators were special education teacher leaders who had expertise in Wilson Fundations® and coaching. They played a crucial role that may help to explain the difference in the ability of the FI schools and the PI school to apply the Fundations® program, especially in Tier 2. Fuchs (2009) determined that facilitators did not affect the outcomes in an RTI project in kindergarten classrooms. However, the facilitators in that study were graduate students working with experienced teachers. In our study, experienced facilitators seemed to have a strong positive impact on the fidelity of implementation.
Scheduling Tier 2
Scheduling is an exosystem challenge for all schools. Lunch and physical education requirements often dictate school schedules. A central tenet of RTI is that Tier 2 instruction supplements, never supplants, Tier 1 instruction. Although there were significant achievement gains, it is likely that the intervention did not reach its full potential for effectiveness due to poor attendance in Tier 2 sessions that were conducted before the official start of the school day in both FI and PI conditions. By conducting Tier 2 during the extended day timeslot, the teachers were able to guarantee that none of the children missed any portion of their comprehensive general education literacy instruction. However, some of the children who were assigned to Tier 2 instruction had trouble getting to school before the official starting time.
In most RTI models, the general education teachers will need to be the gatekeepers ensuring that components of the comprehensive literacy program are not sacrificed for Tier 2 instruction. The reality is that when Tier 2 sessions are scheduled during the school day, supplementary reading instruction inevitably requires missing something else that is being taught. Collaboration, creative scheduling, and classroom teachers who are confident enough to advocate for the value of the general education curriculum are going to be essential elements of successful RTI implementations.
Organizational Structure
A flexible, yet supportive school organizational structure is also crucial to the implementation of the RTI paradigm. All three schools had particular structures of organization in place that seemingly supported the transition to an RTI model. These structures indicate that the schools were poised to make the transition to RTI to identify children who needed increasingly intense instruction and to determine what form that instruction should take. The first step in the development of the phased, RTI roadmap should be the implementation of a readiness survey or “state of the school” analysis regarding the existing intervention, remediation and referral process. Schools need to perform a needs survey, including an assessment audit, to determine how the current organizational structures function and how they might need to be adjusted to serve an RTI paradigm. For example, what assessments are currently administered, for what functions, where are there redundancies, where are there voids, who administers each assessment, and how is the data shared? Also, what are the decision points for Tier movement? Who will be involved in conversations at each decision point? How will the necessary stakeholders increase the likelihood that each child receives the appropriate level of support necessary for accelerated achievement? These and other questions must be asked and answered at the school level. The school RTI design team might be conducted as a study group that reads about the RTI paradigm and applies that knowledge to the needs and resources found in their own school. The design team might consist of principal, literacy coach, special education teacher, general education grade level representatives, ELL teacher, and school psychologist.
Implications for Future Research
Future research needs to investigate the longitudinal impact of RTI implementation in naturalistic settings. These schools received additional resources from the OSE. Studies need to investigate how schools will bear the economic burdens associated with RTI in an era of budget and personnel cuts. The development of teacher expertise, increased assessment and data analysis, and supplementary interventions with low teacher-student ratios require funds. Many urban schools have large percentages of children who are functioning at levels that would require multiple support services in keeping with an RTI paradigm. Additionally, studies of successful RTI models in urban settings that operate within the constraints of a strong union presence are needed (Boute, 2012).
Most of the studies investigating RTI are short-term intervention studies. Long term studies that investigate the systemic changes that are compatible with an RTI paradigm and the long-term impact of those changes are needed. Specific areas of study related to the school reform process and professional development are essential in operationalizing the paradigm in ways that marry general education and special education for the good of all learners. Finding the time and faculty with specialized training for the provision of tertiary services is a challenge. It seems likely that within an effective RTI framework, special education services will need to evolve into something different over time (Blanchett et al., 2009; Fuchs et al. 2008; Johnson et al., 2006). If children have received effective instruction at increasingly intense levels before they enter special education, special education will need to provide something individualized and more intense than was provided in the preceding tiers. Research is needed to inform such an evolution. New structures that allow for fluidity in both faculty conversations and student support services must be created and nurtured.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.