Abstract
The present study investigated the effectiveness of an equity-based inclusive school reform model nested within a multitiered system of support (MTSS) framework on the improvement of math and reading performance of students with Individualized Education Programs (IEPs). Descriptive statistics revealed that math state assessment scores of students with IEPs increased over the implementation period. Results of multilevel modeling demonstrated that the model’s fidelity of implementation scores positively and significantly predicted state assessment math scores. A further analysis examining the effectiveness of the model in three schools that implemented with adequate fidelity compared with nonimplementing schools indicated students with IEPs in implementing schools increased their math scores at a greater rate than their peers in comparison schools; however, effects on reading scores were equivocal. Findings are discussed in the context of inclusion and efforts to support high fidelity implementation of MTSS.
Our study addressed the extent to which implementing a multicomponent intervention for inclusive educational practices within a tiered instructional framework was related to educational outcomes for students with disabilities. Our theoretical model postulated that an equity-based operationalization of inclusion, derived from the work of Kozleski et al. (2004), would reveal better results than traditional place-based service delivery wherein students in special education with more extensive needs for extra supports are moved from self-contained settings to full participation in general education classrooms (Sailor, 2017). Furthermore, the theoretical model postulated that a tiered, whole school, instructional delivery approach, that is, a multitiered system of support (MTSS) could, if implemented with fidelity, serve as a driver for inclusion. MTSS, in this case, consisted of a framework of Positive Behavior Interventions and Support (PBIS) and Response to Intervention (RTI) applied to all students (McIntosh & Goodman, 2016).
This multicomponent intervention was under taken in a large urban school district following about a decade of research in a variety of smaller school districts under the descriptor, Schoolwide Applications Model (SAM; Sailor & Roger, 2005). SAM in 2012 morphed into the Schoolwide Integrated Framework for Transformation (SWIFT) following the award of a large cooperative agreement from the federal Office of Special Education Programs (OSEP) and the subsequent establishment of a research and technical assistance center serving 64 schools across five states (Sailor et al., 2018). The present study follows up from a previous publication that examined the inclusive MTSS framework in the same district on math and reading results from all students. Here we examine the relationships for students served under special education, or students with IEPs. Inclusive MTSS represents a very complex intervention. In the sections that follow, we delineate the key components of our work and present its context and purpose.
Inclusive Education
The Individuals with Disabilities Education Act (IDEA, 1975, 1997; IDEIA, 2004) sets a minimum expectation for U.S. public schools to effectively educate students with disabilities in general education classrooms, to the maximum extent appropriate, to increase their access to the general education curriculum—which is often referred to as the “least restrictive environment” (LRE) provision of the law. Access to LRE for students with disabilities is often discussed in the research literature using the term “inclusion.”
The proportion of students with disabilities in general education settings has increased over time from 31% in 1990 to 58% by 2008 (U.S. Department of Education, 2013). Positive outcomes of inclusive education for students with disabilities, including some large-scale databases, have been reported. For example, multilevel modeling analysis of data from 1,300 students in the Pre-Elementary Education Longitudinal Study (PEELS) showed the hours spent in general education were positively and significantly related to achievement gains across all disability categories (Cosier, 2010). Peetsma et al. (2001) found that students educated in inclusive settings demonstrated greater academic progress than their matched peers in special education schools. Blackorby et al. (2007) found from analyses of the Special Education Elementary Longitudinal Study (SEELS) database, that students with disabilities who took more academic classes in general education had greater academic success than peers who took fewer. These and other findings have contributed to students with disabilities spending greater percentages of the school day in general education settings. Other investigations, however, indicated much more room remains for examination of academic outcomes of students with disabilities accruing to inclusive education (Farrell et al., 2007; McLeskey, 2011).
MTSS
A more recent inclusion narrative is shifting away from physical classroom placement considerations and toward equitable distribution of available supports and services for all students within whole school applications of tiered instructional options (i.e., MTSS, including PBIS and RTI; Kozleski & Huber, 2012; Kozleski & Waitoller, 2010; Sailor, 2017). The outcomes of students with disabilities (including students considered at risk) in schools implementing MTSS have been investigated and reported through a number of studies. The positive findings include a reduced gap between actual reading performance and grade-level standards for children at risk (i.e., kindergarten through third grade) (Simmons et al., 2008); reduced identification rate of students with reading disabilities and increased passing rate on assessments (Harn et al., 2011); and increased time on tasks and academic outcomes through an integrated academic and behavior MTSS framework (Algozzine et al., 2012; Lassen et al., 2006; McIntosh et al., 2008). Mathematics improvement was investigated for students at risk and students with disabilities, and positive effects were observed (Bryant et al., 2008; Gersten et al., 2009; Montague & Jitendra, 2012). Those studies, however, were focused on effective math instruction and interventions, and few examined the combined effectiveness of a schoolwide framework including inclusive education, MTSS, and collaborative professional efforts.
SAM
The SAM offered an example of an “equity-based” inclusion model that clearly targeted outcomes for all students, including students with all ranges and types of disabilities, while their access to general education content in integrated environments was driven by an integrated MTSS. The equity definition of inclusion shifts the narrative away from the difficulty of educating students with disabilities in the general education classroom, to recognizing that various school environments, including the regular grade-level or content classrooms, can be pressed into service with small group or one-on-one interventions as needed. SAM, in effect, alleviates the need for segregating students in self-contained categorical programs (e.g., special education, gifted programs, English as an additional language education). This practice allows all students to be adequately instructed with equitable distribution of supports and services to meet the individually measured educational needs of each, while securing their voice, participation, and opportunities to learn in a positive school culture (Ainscow, 2005; Hehir & Katzman, 2012). A framework offering collaborative instructional support and a problem-solving system that incorporates general, special education, and intervention staff (Coburn & Turner, 2012; Denton et al., 2006; D. Fuchs et al., 2012); data-based decision making (Coburn & Turner, 2012; McLeskey et al., 2012; Newton et al., 2011); and strong administrative leadership (Billingsley et al., 2018; Waldron et al., 2011) adds components that have been shown to cultivate an inclusive school culture.
The SAM concept was developed through exploration of the possibility of combining several discrete, high-impact interventions into a single coherent framework with the understanding that all interventions would be available to all students, and self-contained service delivery to discrete populations would be phased out. Specific interventions included evidence-based curricula and tiered instructional methods in math and reading, and such inclusive educational practices as peer-assisted learning, parent engagement practices and positive behavior interventions and supports (Sailor & Roger, 2005). Those integrated MTSS/RTI and inclusive efforts were supported by administrative leadership, family supports, and district-level supports (Choi et al., 2017; Sailor & Roger, 2005). The framework was first implemented in Kansas City, Kansas (Sailor et al., 2009); was next replicated at scale in the Ravenswood City School District in East Palo Alto, California (Sailor et al., 2006, 2009); and subsequently in a large, east coast district (Choi et al., 2017), where the present study took place. All these studies had similar positive trend results.
SAM pursued, as does its successor SWIFT, an equity-based definition of inclusion that shifts the unit of analysis from the general education classroom to the whole school environment (Choi et al., 2017). The tiered instructional delivery system enables this arrangement by using all available school space, personnel, and resources to accommodate small instructional grouping arrangements for students at risk (i.e., Tier 2) and one-on-one interventions without categorical segregation for students with high support needs (i.e., Tier 3) along with a strong universal preventive support layer for every student (i.e., Tier 1) (Burrello et al., 2001). The term “equity,” as mentioned, refers to dynamic allocation and match of instruction or intervention resources to measured student need for all students, not just for categorically identified students. In the present study, we look at the definition as extended to students in special education and not other student subgroups historically experiencing inequity. This form of inclusive education can be accomplished within existing IDEA and Every Student Succeeds Act of 2015 statutory and regulatory authority through resource allocation and changes made to a school’s master schedule (Sailor & Burrello, 2013).
Context and Purpose
The present study is a follow-up to previous research on the effectiveness of the SAM intervention to all students participating in the selected schools. Our earlier findings revealed that math score growth was significantly larger for students in SAM implementation schools than in comparison schools (Choi et al., 2017). Although no significant reading growth difference was observed between implementation and comparison groups, schools that implemented with the highest scores on a measure of fidelity made better improvements on both reading and math while scores of matched comparison schools actually decreased (Choi et al., 2017). Readers seeking a broader description of specific practices under the SAM system may wish to examine the previous publication.
The purpose of the present investigation was to further document relationships between SAM implementation and academic gains (Study 1) and relationships between high levels of implementation and academic performance in comparison with a matched set of control schools (Study 2) for students with Individualized Education Programs (IEPs) who took the same standardized tests as their peers without IEPs (i.e., annual state assessments). We hypothesized that the students with IEPs in higher implementation schools would make better academic progress (Study 1) and that students with IEPs in full implementation schools would make better academic progress than those in comparison schools (Study 2).
Method
Participants and Sampling
Fourteen public schools serving PreK-5th and PreK-8th grade configurations from a large east coast school district participated in the study. This urban district, circa 2008, faced federally mandated school improvements due to the high rate of failure in meeting adequate yearly progress (AYP) targets and a lawsuit citing inadequate education of students with disabilities. Specifically, about 75% of schools had failed to meet their AYP targets for at least two consecutive years. The implementation of SAM was largely led by the need to improve academic performance for all students and resulted from the lawsuit brought against the district.
Seven schools participated in the SAM implementation. The implementation schools were selected through volunteer and purposive sampling with emphasis on the need for more inclusive education. District officials sent out a letter to all elementary school Principals (PreK-8th grade) and introduced SAM. Among those who showed an interest in the approach, eight schools were initially selected. Based on prior experience and given the district’s funding model, researchers considered it appropriate to have about eight schools in a cohort for intensive support for SAM implementation. Preference was given to the lowest performing schools among those who signaled interest. One school was purposely selected because of its persistent segregation of students with IEPs and involvement in the district’s legal dispute. One school opted to defer their implementation at the beginning, leaving seven schools in the implementation study. Thus, five PreK-5th grade and two PreK-8th grade schools participated in SAM implementation between 2008 and 2012.
Seven comparison group schools were also purposively selected to match characteristics of schools in the implementation group using propensity-score matching at the school level. The criteria were set and school-level data were reviewed to match each implementation school to a comparable school in the district. The criteria included school size (total students enrolled) and proportion of students in major ethnic groups, free and reduced meals (FARM) status, special education population, and English as an additional language learner population. Race and ethnicity, socioeconomic status, and language were used as criteria because they are common factors associated with achievement (Chatterji, 2006; Cronin et al., 2005; Hochschild & Scovronick, 2003; Reardon & Galindo, 2009). The comparison group also consisted equally of five PreK-5th grade and two PreK-8th grade schools.
In the analysis, annual summative state assessment results from a total of 621 students with IEPs were included. Three hundred fifty-five students were from the implementation schools, and 266 students were from the comparison schools. This analysis compares all students with IEPs taking the state assessment (all but about 1%) between the 2009–2010 Academic Year (AY) and the 2011–2012 AY.
During the study period, implementation and comparison schools improved on measures of their inclusive education. The average proportion of time in general education was 32.6% for the implementation schools and 35.4% for the comparison schools in 2010–2011 AY. The proportion increased to 65% and 64.6%, respectively, in the 2011–2012 AY. These data suggest that no difference existed for the extent of inclusive education progress between the two groups during the analysis. The comparison group’s improvement in inclusion came as no surprise because de-segregation of students with disabilities was a district-wide priority at the time in response to ongoing litigation.
Procedure
SAM employed six guiding principles to support schools in implementing evidenced-based practices in the integrated instructional system (the framework). Those principles were (a) general education guiding all instruction, (b) resource configuration, (c) social/behavior development, (d) data-based problem-solving, (e) family engagement, and (f) district support (Sailor & Roger, 2006). The principles are described in detail in Sailor and Roger (2006) and Choi et al. (2017).
SAM implementation included five stages of technical assistance provided by SAM developers under contract with the district, and occurred simultaneously at district and school levels. In the first stage at the school level, the principles of SAM were introduced to staff. Schools discussed strengths and opportunities for improvement and created implementation plans based on a mutually agreed upon vision for improvement. Next, baseline scores, together with fidelity of implementation scores using the SAM Analysis System (SAMAN) tool (to be described), and student academic and behavior outcomes were collected. Data were used to plan and prioritize action to improve SAMAN core features. In the third stage, school leadership teams were established to efficiently support SAM implementation and monitor progress. Fourth, school leaders investigated available resources and discussed their reallocation to accomplish the action plan. Finally, professional learning opportunities were designed, including facilitation of professional learning communities in each school. The second through fifth stages were repeated every implementation year to facilitate each school’s differentiated implementation support.
After the support foundation was established with the five stages, professional development and coaching in Year 2 started focusing on academic and behavior support within MTSS. This support included instructional strategies for inclusive education such as flexible grouping and differentiated instruction, classroom management, and formative assessment for learning to enhance the universal-level support system. Trained instructional coaches were utilized to deliver effective coaching for educators. Tier 2 and 3 components were also reviewed and supported by refining the data-based decision system, interventions, and leadership team functions. SAM technical assistance (TA) for math within MTSS provides a good example. Since math screening and progress monitoring tools were not available at the beginning, TA and coaching were provided for utilizing the district’s benchmark assessment and teacher-created progress monitoring for math MTSS. SAM implementation also involved intensive professional learning and ongoing support for all tiers of behavioral MTSS, which consisted of positive behavior interventions and supports (PBIS; e.g., check-in check-out, function-based behavior management). Due to the high incidence of discipline referrals, behavioral MTSS was highly prioritized in all schools. In implementation schools, PBIS involved all adults in the building in providing behavior support. All school staff, including nonteaching staff, were trained for schoolwide expectations, and reward systems were expanded to nonclassroom settings. In Year 3, the TA emphasis moved to coaching for instructional supports and interventions for educators while MTSS and other SAM features were continuously supported. Activities also focused on assisting school leadership teams to develop the capacity to build sustainable implementation practices and strengthen family and community partnerships. Volunteers more actively engaged in after-school programs to support academic needs in reading and math.
TA activities included 2-day professional learning for coaches and school leadership teams every semester, and summer professional learning institutes for coaches, administrators, and teachers. Direct TA was provided to schools for inclusive practices especially for students with special needs. Quarterly TA and other visits were scheduled based on their needs. District-level SAM coordinators were assigned to provide coaching, and those SAM coaches made weekly visits to schools. The district-level SAM professional development focused on common needs identified from the aggregated SAMAN assessments and qualitative data collected by the TA providers.
No SAM TA activities or SAMAN assessments were conducted for the comparison schools. Schools in the comparison group educated students under business-as-usual practices with regular support provided by the District. Therefore, no curriculum or instructional material difference existed between the SAM implementation and comparison schools except SAM TA with fidelity assessments, as the independent variable.
The Schoolwide Applications Model Analysis System (SAMAN)
SAMAN (Sailor & Roger, 2008) was developed to measure the fidelity of implementation of SAM. SAMAN administration involved staff interviews, document reviews, and school/classroom observations conducted by trained assessors. A preliminary technical adequacy study examined its interrater reliability, internal consistency, and convergent validity (Choi, 2007). Interrater reliability was calculated with six paired data sets between two independent trained assessors, and 80% average agreement was reported. For internal consistency, Cronbach’s alpha for all 15 items was 0.95, which was strong and exceeded the general cutoff for research purposes (Henson, 2001). Cronbach’s alphas for each subscale (i.e., guiding principle) ranged from 0.80 to 0.91. Convergent validity was examined only for SAMAN Item 7 (positive behavior intervention and support) because no other assessment tool was available with which to compare with other SAMAN items. The correlation between the Schoolwide Evaluation Tool (SET: Sugai et al., 2001) and SAMAN (Item 7) showed a statistically significant and positive relationship, r (51) = 0.78, p < .01 (Choi, 2007). These findings suggest that SAMAN could be considered a valid and reliable fidelity tool for SAM implementation.
Annual summative state assessment
Annual summative state assessment data were collected for the academic outcomes of students with IEPs. Third- to eighth-grade students participated in the assessment each year during the SAM implementation. Most students with IEPs, with the exception of about 1% of students taking the alternate assessment, were tested. According to published reports, the state assessment system met technical adequacy standards. Reliabilities for the assessment averaged .92 for the stratified alpha. Average stratified alpha for reading assessments was .93, and was .93 for math, which exceeded the acceptable coefficient, .8.
Research Design and Data Analysis
We used a multilevel modeling approach to document the relationship between SAM implementation (i.e., SAMAN score and SAM implementation status groups) and annual state assessment scores (Study 1). We used a similar approach to document the longitudinal score difference between the three fully implementing schools and their matched comparison schools (Study 2).
Study 1
A multilevel modeling approach, using the linear mixed models function SPSS was employed to examine the relationship between SAM implementation and annual state assessment scores during the implementation years. Data in the analysis had two levels of structure. The first level was within-person (Level 1), and it defined each individual student’s measurements over time with individual growth trajectory and random error. The second level, between-person (Level 2), was included to examine the differences in trajectories between groups of individuals (Raudenbush & Bryk, 2002). This level could explain variability in the random parameters across individuals, and addressed the question: Are there differences in state assessment score changes across groups with different SAM implementation status? Separate analyses were conducted to examine the effect of two aspects of SAM implementation on reading and math assessment scores. First, with the schools implementing SAM only, SAM fidelity (i.e., SAMAN scores) was included to examine any related assessment score changes. Second, among all 14 schools, the assessment score changes across three SAM implementation status groups (i.e., no SAM; SAM implemented but did not meet the fidelity criteria; SAM implemented and met the fidelity criteria) were examined (see Supplemental Material Appendix for equations and variables of the multilevel modeling).
The schools meeting the fidelity criteria (i.e., full implementation) were selected when the school (a) administered SAMAN for at least 2 years; (b) had a total scale score of SAMAN higher than 2.50 (i.e., between 2.50 to 3.00); and (c) had all average scores of individual guiding principles higher than 2.50. The total scale score was calculated by averaging the 15 features, and each feature had a score range between 0 to 3. These fidelity criteria were determined by experts and researchers who were experienced in SAM and implementation science.
Study 2
The second study was conducted to answer the research question: Do the schools implementing SAM with fidelity improve their state assessment scores more over time than the comparison schools? This can be better answered with the analysis of schools that implemented with adequate fidelity because it is expected that student outcome improvement can occur when a high level of fidelity is achieved (McIntosh & Goodman, 2016; Sugai & Horner, 2008). The question was examined by a treatment–time interaction, that is, different growth trends were expected for the control and treatment groups. The within-person (Level 1) growth across SAM implementation group (Level 2) was examined with a model including a dichotomous variable representing SAM (i.e., SAM implementation group and comparison group), Year (i.e., 3 years), and their interaction (equations and variables of the multilevel modeling are in Supplemental Material Appendix).
Results
Study 1: Relationship of Implementation and Academic Scores
Schools continuously implementing SAM increased their fidelity of implementation as measured by SAMAN. The overall average of the 15 SAMAN critical features was 1.79 out of 3 (60%) in the 2009–2010 school year and increased to 2.36 (79%) in 2011–2012. For the average critical feature (CF) scores, large score changes were made on CFs about “inclusion of students with special needs” (M = 1.00, SD = 0.58 to M = 2.71, SD = 0.49), “flexible and non-categorical service delivery” (M = 0.86, SD = 0.38 to M = 2.29, SD = 0.76), and “engagement of all school personnel in teaching and learning” (M = 1.57, SD = 0.53 to M = 2.71, SD = 1.14) over the three years of implementation. At the end of the implementation year, high average scores were found in CFs regarding inclusion, data-driven and collaborative decision making, engagement of all personnel, empowered site leadership team (SLT), and district supports. For the three schools that met the fidelity criteria, their average SAM total scale score increased from 1.73 (SD = 0.26) to 2.67 (SD = 0.12).
The aggregated reading and math scores on the annual state assessment are depicted in Figure 1. The mean score change showed that schools with SAM implementation made stronger improvement on math relative to comparison schools. Implementation schools’ mean scores on math improved from 36.94 for the baseline in the 2009–2010 school year (N = 201, SD = 15.89) to 40.28 in the 2011–2012 school year (N = 180, SD = 15.43). In contrast, schools in the comparison group decreased their math scores from 34.44 (N = 186, SD = 16.03) to 33.01 (N = 102, SD = 18.03). The gap between the implementation and comparison groups grew bigger over the implementation period. When results of students with IEPs who consistently took the math state test for 3 years were analyzed, the same pattern was observed (see Supplemental Material Figure S1). The baseline mean score of the implementation schools was 34.02 (N = 48, SD = 19.38), and it increased to 40.71 (N = 48, SD = 13.76) while comparison schools obtained a relatively small increase at the end of the implementation. No score change difference was observed between implementation and comparison groups for reading (see Figure 1).
State assessment mean score changes on reading and math for all schools, and on math for the three SAM implementation schools with fidelity and matching comparison school.
The multilevel modeling results showed that SAM implementation affected math score growth more than reading scores (see Supplemental Material Table S1). The significant Year effects in model 1 revealed that math scores significantly increased over time in both the 7 SAM schools (β10 = 1.37, p < .001) and all 14 schools (β10 = 1.85, p < .001). For the analysis of math score change with SAMAN in Model 2, the results showed that both the SAMAN and quadratic Year significantly and positively predicted math scores while other variables were held constant. The interaction between SAMAN and Year was also significant but in a negative direction. Students’ math intercept (β00) was 36.24, which was students’ achievement scores adjusted for Year and SAMAN. This statistic can be interpreted to mean that the estimated math score was 36.24 for students who were at the first year of the SAM implementation (coded 0) and their schools’ SAMAN score was average (i.e., SAMAN was centered with the grand mean). Implementation year (Year) was significantly and positively related to the math scores (β10 = 1.52, p < .001). This indicates that the math scores of students in the SAM implementation schools significantly increased over time in a quadratic fashion. Regarding an explanation of the variability in math growth between individuals, the quadratic interaction (Year × SAM) was significant at p < .01 (β11 = −3.57, p < .01). This indicates that students in implementation schools who scored higher on SAMAN demonstrated less growth over time compared with students in schools scoring low on SAMAN. There was significant residual variance in intercepts to be explained (Wald Z = 6.76, p < .001). However, no significant residual variance in slopes were left to be explained across individuals (Wald Z = 1.45, p = .15). The covariance between the intercept and slope was negative (−20.46) and significant (p < .05). This statistic represents the same result of the interaction between Year and SAM, which indicated that students in schools that started with higher achievement demonstrated less growth over time and vice versa.
When math scores were analyzed with the three SAM status groups and comparison schools were included (i.e., no SAM, did not meet SAM fidelity, met SAM fidelity), no effects of Year (β10 = 0.53, p = .47) or SAM (β01 = −0.14, p = .88) were observed on the math scores in the final model; however, the interaction effect (Year × SAM) was positive and significant at p < .05 (β11 = 1.55, p < .05) while other variables were controlled, indicating that students made greater growth on math scores when their school met SAM implementation with fidelity, and students in schools implementing SAM with low fidelity had lower math score growth. The lowest growth was in comparison schools.
Similar results were observed when the same analyses were conducted with students with IEPs who attended their school for the three consecutive years of the study period. For the analysis of math scores and SAMAN, the implementation year (i.e., Year) was still positively and significantly related to the math scores in the quadratic fashion (β10 = 1.57, p < .05) while other variables were held constant. The interaction effect between Year and SAM was also significant and negative (β11 = −1.59, p < .01), which indicates that student math score growth in high SAMAN fidelity schools was smaller than the growth in low fidelity schools. However, the interaction effect was positively significant when the SAMAN scores were replaced with SAM status group (β11 = 1.82, p < .01), indicating that the math score growth was better when schools implemented SAM and best when schools meet the fidelity criterion. The effects of Year (β10 = 0.87, p = .41) and SAM (β01 = 1.52, p = .39) were not significant while other variables were controlled. These results replicate the math analysis results for all students with IEPs.
The reading scores, however, did not reveal a significant association with SAM implementation status. In the final model for reading (see Supplemental Material Table S1), Year was significantly and positively related to the reading score (β10 = 0.84, p = < .05), indicating that the reading scores of schools implementing SAM significantly increased over time; however, no significant effects of SAM (i.e., SAMAN score) were observed, nor was the interaction effect between Year and SAM (β11 = −0.27, p = .77) significant. Similar results were found when the SAM status group was analyzed instead of using SAMAN average total scores. Although the reading scores significantly increased over time (β10 = 1.95, p < .05) while other variables were controlled, the SAM status group (β01 = 0.43, p = .66) and its interaction effect with Year were not significant (β11 = −0.73, p = .28).
Study 2: Relationship of Full Implementation and Academic Scores
Separate multilevel modeling was conducted to examine the effectiveness of SAM with the three schools implementing SAM at the criteria for implementation fidelity (78% average mean score across all features) and their matching comparison schools. The bottom chart of the Figure 1 illustrates the growth trajectories in the two groups.
The fixed-effect estimates suggest that the students in the comparison group started with a math mean score of 33.19 (see Supplemental Material Table S2). Over each year, individuals’ math scores in the comparison group increased by 0.17 on average but not significantly (β10 = 0.17, p = .90). It was also confirmed that there was no difference between the SAM implementation and comparison group schools initially (β01 = −2.10, p = .33). Over time, however, it was observed that students whose schools underwent SAM implementation increased their math scores over each interval at a greater rate (3.99 points) than their peers in the comparison group schools (p < .05).
With the covariance parameter estimates, it was found that the correlation between initial status and growth was negative (β = −0.51, p < .01), which suggests that students who started higher in achievement demonstrated significantly less growth over time and vice versa. An additional model was specified to determine the difference between the treatment and control groups at each occasion, instead of assuming a polynomial growth curve over the entire temporal sequence. This model fits the data better than the previous model, using Akaine Information Criterion (AIC) as an index of model fit (AIC reduced from 3,647.04 to 3,637.29) (see Supplemental Material Table S3). The intercept is defined as “ending” math achievement status (33.23) for the comparison group (coded 0). The SAM implementation group would be estimated as 39.25. It was found that students demonstrated considerable (but differing) growth at each time occasion as represented by the Year effects. The Year coefficients refer to how much lower the control group was at each interval preceding the ending status intercept. The coefficients can be used to estimate math achievement for each group at any occasion. For example, the comparison group’s achievement at the beginning of the study was 33.23 + 0.14 = 33.37. The results also suggested that growth for the comparison group decreased over successive intervals. A positive effect was noted for the SAM implementation group (β = 6.02, p = .06) although the effect was not statistically significant. It was also found that the amount of difference at each occasion between the implementation and comparison groups, which was modeled as the Year × SAM Implementation interactions, was significant at the first occasion (p < .05). For reading, the interaction effect Year × SAM was not significant (β = −1.54, p = .39), which suggests that growth trajectories for individuals in schools undergoing SAM implementation were not significantly different from those in the comparison group.
Discussion
This study is a quasi-experimental work depicting the relationship between MTSS implementation status and outcomes. In two analyses, we examined (a) the longitudinal relationship between SAM implementation (i.e., SAMAN score and SAM status group) and state assessment scores; and (b) the score changes between the three full implementation schools (i.e., schools met the fidelity of implementation criteria) and comparison schools during three implementation years. Since SAM no longer exists in its original form, the dissemination value is that it adds to the developing knowledge base on whole school applications of MTSS for students with IEPs supported by inclusive school reform. Findings indicated that the math scores of students in the analysis (i.e., scores of the 7 SAM implementation schools) demonstrated significant quadratic growth, and the SAM implementation fidelity was positively and significantly associated with the math scores while other variables were controlled. A significant negative interaction effect between SAM fidelity and state assessment scores suggests that students with IEPs in high fidelity schools were already approximating their maximum levels of performance and the possibility of additional growth was less in comparison to the group in the low fidelity schools. When SAM implementation status group was analyzed, our findings showed that students made greater growth on math scores when their schools are fully implemented than schools with low fidelity or comparison schools. The comparison study confirmed that math score growth of students with IEPs are significantly different from their matched comparison schools when SAM was fully implemented.
The findings support the benefit of an inclusive MTSS framework for enhancing math academic outcomes of students with IEPs. The rationale for the academic growth lies in the balance between a strong commitment to educate all students inclusively and to implementation of MTSS with fidelity. While a mandate exists to effectively educate all students, including students with disabilities, in the general education curriculum and in grade-level or content classrooms to the maximum extent appropriate (Huefner, 2000; U.S. Department of Education, Office of Special Education Programs, 2003), it is hard to expect students to achieve quality outcomes with only a commitment to physically placing students in general education settings (i.e., place-based inclusion). The means to provide quality of instruction for all may better progress through whole school applications of tiered intervention (e.g., MTSS) that integrates academic and behavior support with data-driven problem-solving to reorganize and reallocate school space, desegregate special populations, and integrate all resources to benefit all students, in this case students with IEPs, based on their needs (i.e., equity-based inclusion).
The significant relationship between SAM implementation and state assessment score was not observed for reading, an interesting and unexpected finding. SAM implementation for math initially had some disadvantages over reading. MTSS is a leading driver of equity-based inclusive educational transformation models like SAM (Choi et al., 2017); however, fewer resources and limited information (e.g., universal screeners, progress-monitoring tools, and evidence-based interventions for Tier 2 and 3 support) were available for math MTSS (Lembke et al., 2012). MTSS for reading, in contrast, was supported relatively well across the district with universal screening and progress monitoring, and supplemental instruction with trained reading interventionists. Those MTSS core components for reading were equally supported for all schools by the district even before the SAM implementation. The results revealed that reading scores were improved in both implementation and comparison schools and no difference was observed between implementation and comparison groups. This might have been the result of the existing supports provided for all schools in the district.
The lack of resources in math MTSS increased the need to enhance its instructional delivery system, and SAM implementation may have provided a creative way to enhance the system with the given resources. Although research-based universal screening or progress monitoring tools were not available for math MTSS, the District benchmark assessment system, a curriculum-based measure, was administered three times a year. Tier 2 and Tier 3 supplemental supports were initiated for math MTSS with existing resources, but no particular evidence-based intervention curriculum was newly applied in SAM implementation schools. Instead, the use of data for instructional decisions, need-based supports, and problem-solving oriented leadership were emphasized in the SAM implementation. Therefore, each school initiated team-driven discussions to explore student instructional needs, an adhocratic approach, which may have been substantially different from the top-down mandated and standard approach (Skrtic, 1995). Thus, each school could have installed the core components of MTSS to the greatest extent possible with available resources. In the meantime, SAM facilitated better access to the general education curriculum for students with IEPs through effective inclusive educational strategies such as application of function-based behavior supports in the classroom, differentiated instruction with flexible grouping, and universal design for learning (UDL), as well as the supplemental support matched to student need via MTSS. As the result of these efforts, low-performing students in SAM implementation schools might have had better opportunities to obtain necessary math skills through supplemental math instruction based on student skill levels. Unlike reading, math has domains with different concepts and skills to be learned (e.g., numbers, operations, measurement and data, geometry). The different branches of math may differentially challenge students, so instructional approaches providing more frequent dynamic decision making may benefit students who need learning support in each math domain. The problem-solving protocol in MTSS is a more individualized decision-making model than the standard protocol approach (D. Fuchs & Fuchs, 2006) and might be a more appropriate way to adopt the problem-solving protocol for math. SAM’s adhocratic culture in combination with the problem-solving protocol and inclusive instructional strategies might well have fostered the flexibility to improve math outcomes within the MTSS framework. More research is needed to investigate the differences between reading and math under inclusive MTSS conditions associated with measured outcomes.
The controlled comparison analysis included SAM implementation schools that met the fidelity of implementation criteria and their matching comparison schools. Although the SAM implementation schools made significant improvement on math scores (i.e., the Year effect in the Study 1) when analyzed for all SAM implementation schools, it was not expected that student outcomes would be significantly different from comparison schools unless implementation of SAM reached the SAMAN fidelity criteria. The efficacy of implementation depends on implementation fidelity (i.e., treatment integrity). The findings in the present study demonstrated the effects of MTSS in the context of high fidelity, especially for students with disabilities. This finding also supports the postulation that special education support is more efficient to deliver within a MTSS structure that is implemented with fidelity (Balu et al., 2015; L. S. Fuchs & Vaughn, 2012).
Limitations and Future Directions
Several limitations should be considered when interpreting these findings. First, participants were not randomly selected and assigned to groups. Schools that participated in implementation were recruited by the school district, and comparison schools in the analyses were conveniently sampled from the same school district based on matching characteristics of implementation schools. Although implementation and comparison schools were balanced, the recruiting process could not exclude the possibility that implementation schools possibly had stronger willingness to make an extended commitment to engage in district-initiated activities.
The role and effect of the school district was not carefully considered in the present study. All schools in the study were located in the same district. Those schools might have benefited from spillover effects of improved district capacity resulting from the SAM technical assistance. District support is critical for the successful implementation of system change efforts. Future studies need to investigate the success of equity-based inclusive school reform in combination with district support for the purpose of scaling up innovation. In addition, the number of participating schools needs to be larger to produce reliable results.
Finally, the academic support needs of students with IEPs can vary depending on the nature of their disabilities and prior learning experiences. The present study only investigated overall academic outcomes of all students with IEPs who participated in the state assessment, and not variation within a range of support needs. A future study that controls for differentiated needs for levels and types of special education would provide a more robust analysis. The effects of equity-based inclusion on students who take alternate assessment based on modified or alternate achievement standards also needs to be investigated in future studies.
Conclusion
This study lends support to a conception of equity-based inclusive education through MTSS enhancing some academic outcomes (math in this case) for students with IEPs. SAM implementation fidelity, measured by SAMAN, was positively and significantly related to math outcomes measured by the state assessment; and schools with high quality implementation demonstrated better progress on math outcomes for students with IEPs than schools in the business-as-usual comparison group. One possible goal of inclusive school transformation is to provide student support through an integrated and positive school culture in which all adults in the building have shared responsibility for academic and/or social outcomes of all students and respond to students collaboratively. Equity-based inclusion models cultivate a school culture that can potentially benefit all marginalized student groups, particularly students with disabilities in the context of full access to the general education curriculum and instruction, within an inclusive educational environment. Along with a previously published study on SAM effectiveness (Choi et al., 2017), the present study examined implications of equity-based inclusion for students in special education.
Supplemental Material
JSED-19-10-187.R1_Supplemental_Materials – Supplemental material for Achievement of Students With IEPs and Associated Relationships With an Inclusive MTSS Framework
Supplemental material, JSED-19-10-187.R1_Supplemental_Materials for Achievement of Students With IEPs and Associated Relationships With an Inclusive MTSS Framework by Jeong Hoon Choi, Amy B. McCart and Wayne Sailor in The Journal of Special Education
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
