Randomized Evaluation of a Supplemental Grade-Wide Mathematics Intervention

Year-end statewide accountability measure (Pearson, 2010)

The statewide accountability measure was developed by the department of education in the state where the study was conducted and CTB/McGraw-Hill based on the framework for accountability monitoring in the state. The statewide accountability math test included five primary competencies (i.e., number and operations, algebra, geometry, measurement, and data analysis and probability). Intercorrelations among competencies have been reported to range from r = .45 to r = .67 and from r = .50 to r = .65 for fourth and fifth grade, respectively. Cronbach’s alpha of .88 was reported for both fourth- and fifth-grade math scores. Mean adjusted item-total point biserial correlations of .35 and .34 were reported for fourth- and fifth-grade math scores (Pearson, 2010). The 2008 and 2009 administration of the statewide accountability assessment were the same assessment; however, the scale was adjusted between 2008 and 2009. The 2009 scale scores are standardized to a mean of 150 and standard deviation of 10.

CBMs of mathematics for computation

Three computation probes were selected for each grade level that assessed computational skills considered prerequisite to mastery of instructional objectives at a given grade level. Each probe looked like a worksheet and contained problems of a single type on the front and back side of a single sheet of paper. Probes were generated using Mathematics Worksheet Factory (Schoolhouse Technologies, 2003). A sufficient number of problems were provided on each probe such that students would not be able to complete all problems before the timed administration was over. All students completed three probes (three timed worksheets constructed to meet CBM standards) yielding three scores. Fourth-grade students completed a three-digit addition and subtraction with and without regrouping probe, multiplication facts 0–12 probe, and fact families multiplication and division facts 0–12 probe. Fifth-grade students completed a fact families multiplication and division facts 0–12 probe, two- and three-digit multiplication with and without regrouping probe, and reducing fractions to their simplest form probe. Previous studies have estimated score reliability of computation CBM probes for mathematics from r = .67 to above r = .90 (Foegen et al., 2007) for alternate form, test-retest, interscorer agreement, and internal consistency. Delayed alternate form reliability of the scores obtained on the probes used in this study was r = .85 (Burns, VanDerHeyden, & Jiban, 2006). Scores obtained on the computation CBMs used in this study have been found to correlate with Stanford Achievement Test (Harcourt, 1997) scores in the low to moderate range of r = .27 to r = .40 (VanDerHeyden & Burns, 2008).

Intervention Rating Profile-15 (IRP-15)

The IRP-15 samples perceived effectiveness, practicality, ease of implementation, potential risks, and a teacher’s willingness to use recommended interventions. Teachers were asked to rate 15 items on a Likert-type scale ranging from 1 to 6. Lower scores indicated poor acceptability. Higher scores indicated greater acceptability. Item ratings were summed and then divided by the total number of items to yield an average acceptability rating for each teacher. Average ratings lower than 3 suggest that the interventions were unacceptable to the teacher, whereas average ratings greater than 3 suggest that the procedures were generally acceptable (Witt & Martens, 1983).

Training

Teachers, RtI coordinators, and administrators were trained to implement the intervention using a combination of antecedent and live coaching strategies. Following a series of trainings specific to principals, the first author traveled to each school to conduct a 1-hour training with teachers whose classes were assigned to the intervention condition. Additionally, each school had an RtI coordinator and that person was charged with receiving and organizing weekly data to provide to the first author via an electronic spreadsheet program designed to organize the data and present graphs of class progress each week during the intervention. In the didactic training session, an overview of the rationale for the intervention program was shared with teachers using the district’s data reflecting low mathematics achievement. Details of the intervention were provided, including sharing the intervention protocol, describing how the intervention would progress based on student mastery of skills within a preestablished hierarchy of skills, showing effects on mathematics achievement obtained in other districts using the same intervention, and showing short video clips of the intervention being implemented in classrooms in other districts. An opportunity to discuss and troubleshoot intervention implementation was provided to teachers at this time. Teachers were provided all materials needed to implement the intervention each week by the on-site RtI coordinators. School principals agreed to conduct implementation integrity checks via direct observation as part of the intervention plan (described in greater detail in the next section).

The consultant organized feedback on district progress for district administrators and school principals bimonthly during the year. Graphed feedback on each class’s progress with the intervention was provided to principals and district administrators. The consultant met in person with the district leaders and principals, reporting the number of skills mastered by teacher and identifying implementation errors. Additionally, the consultant communicated directly with principals and RtI coordinators providing a list of teachers whose classes were growing at a slower pace relative to other classes in the same school and encouraged an intervention integrity check in those classes. Finally, on a bimonthly basis (four total occasions), the consultant conducted integrity observations in classrooms with each principal and modeled for school principals how to troubleshoot intervention implementation with the classroom teacher.

School principals agreed to conduct implementation integrity checks via direct observation as part of the intervention plan. Principals or on-site RtI coordinators agreed to conduct four integrity observations each week with approximately half of those occurring during regular mathematics instruction within control classrooms in an attempt to capture contamination between control and intervention conditions. The intervention integrity checklist listed each step of the intervention in observable terms and administrators were trained to observe and note the occurrence of each step of the intervention. The trained observer used the scripted intervention protocol to note correctly and independently completed steps of the intervention. Where deviations from the protocol were observed, principals and/or RtI coordinators provided corrective feedback on implementation and assisted the teacher to troubleshoot barriers to effective implementation. A total of 406 observations were conducted. Average percentage of treatment steps completed was 96.69% (range = 83.5%–100%) in treatment classrooms and 2.69% (range = 0%–20.33%) in control classrooms. No interrater reliability data were collected to verify the accuracy of the integrity observation data, so these data were not used in further analyses.

Monitoring intervention integrity

Intervention integrity was examined in two ways in this study. The number of skills mastered was used as an indicator of correct intervention use because classes were equivalent at the start of intervention and the intervention used was the same in all intervention classes (i.e., the same intervention steps were followed). On average, classes mastered 9.12 skills of 14 but there was wide variability across classes (range = 2–14). Second, the degree to which the intervention occurred each week as planned and the degree to which the decision rules were followed could be examined by review of permanent products. The spreadsheet file for each school contained a weekly score on the targeted intervention skill. For the intervention to be correctly conducted, teachers had to follow the skills in order and advance to higher skills only when the class median reached criterion. The weekly intervention scores could be examined by class to document number of deviations from the intervention plan. Twenty-nine weeks were available for intervention. On average, teachers completed 26.1 weeks of intervention (range = 20–29). Intervention teachers deviated from the decision rules on average 1.4 times during the intervention (range = 0–5). On average, correct decisions were made for 94% of decision-making occasions (range = 80%–100%).

Monitoring reliability of assessment procedures

An assessment team was trained to administer the CBMs. Assessment teams were provided with scripted instructions, CBM probes, and a digital timer. Two assessment team members administered each probe in each classroom. One assessor administered the probes following the scripted instructions while the second assessor followed along on a separate set of instructions to ensure that each step of the CBM assessment protocol was correctly followed. Prior to scoring, 25% of the completed probes were removed from each class for each computation probe, copied onto colored paper, and reattached to the original. Later, the probes were separated and scored independently by two trained scorers. The smaller score was divided by the larger score and the resulting quotient was multiplied by 100% to estimate interscorer agreement. For fourth grade, average interscorer agreement exceeded 97% (range = 0%–100%). For fifth grade, average interscorer agreement exceeded 95% on all occasions for all probes (range = 44%–100%) except for the fall administration of the multidigit multiplication probe. In the fall, average interscorer agreement for the multidigit multiplication probe was in the acceptable range at 88% (range = 0%–100%) but increased on subsequent screenings (96% and 95% average agreement) as the number of problems completed by students increased. All cases of low agreement occurred on probes with fewer than two digits attempted.

Year-end statewide accountability test scores

The statewide test analyses were designed to investigate a treatment difference on students’ 2009 statewide test math scores while controlling for 2008 statewide test score. Follow-up analyses examined the difference between groups on students’ numbers and operations subscale of the statewide test while controlling for 2008 statewide test score. Students with a 2008 and 2009 statewide test math score were included in these analyses. A 2008 statewide math score was not available for 95 fourth-grade students and 68 fifth-grade students; this resulted in a reduced sample for these analyses (see Table 1). Descriptive data for intervention and control groups are shown in Table 3.

We used two different dependent variables for each of the fourth-grade and fifth-grade samples. The following descriptions apply to both samples and both dependent variables. The initial model for the data was

Y i j k = γ + γ 1 T + γ 2 X i j k + γ 3 T × X i j k + e i j k + r 0 j k + u 00 k + r 1 j k X i j k + u 10 k X i j k + u 01 k T ,

where i is the index for students; j is the index for classes; k is the index for teachers; T is a dummy code for the intervention; X_ijk is the class-mean centered statewide test score in the spring of 2008; e_ijk is the within-class residual for the Level 1 model (i.e., the regression of the statewide test score in the spring of 2009 on statewide test); r_0jk and u_00k are the class-level and teacher-level residuals, respectively, for the intercept in the Level 1 model; r_1jk and u_10k are the class-level and teacher-level residuals, respectively, for the slope of the Level 1 model; and u_10k is the teacher-level residual for the effect of treatment. The last term was included because treatment and teachers were crossed for a subset of the teachers. The estimation procedure would not converge when the residual for treatment was included. Excluding the residual for the treatment resulted in convergence, but the estimated variance components for the slope for the 2008 statewide test score were zero or very nearly zero at the teacher and classroom with teacher levels. (The largest variance component across the eight combinations of dependent variable, grade, and level was .0000093). Estimating a model with the residual for the intercept at the teacher and classroom within teacher levels resulted in an estimated intercept variance for the classroom within teacher level that was zero or nearly so. (The largest variance component was 2.34E-16). Our final specification of residuals included a residual for the teacher level for the intercept and a within-class residual for the Level 1 model. Therefore our final model was

Y i j k = γ 0 + γ 1 T + γ 2 X i j k + γ 3 T × X i j k + e i j k + u 00 k .

For each of the four combinations of dependent variable and grade, this model resulted in the smallest Akaike Information Criterion (AIC) fit index (Akaike, 1974) among the models for which estimation converged. Group-mean centering (GPMC) based on classroom mean was used at Level 1 to protect against spurious cross-level interactions (Enders & Tofighi, 2007).

CBMs

A multilevel repeated measures analysis, including treatment and occasion as factors, was used to test for a Treatment × Time interaction and account for scores nested within students across occasions, students nested within classrooms, and classrooms nested within teachers for each CBM in each grade. The initial model for the data was

Y i j k m t = µ + α m + β t + α β m t + u 00 k m t + r 0 j k m t + ε i j k m t ,

This is a split-plot ANOVA model with a multilevel residual. Specifically, µ is the grand mean, α _m is the effect of treatment m (m = 1,2), β _t is the effect of occasion t (t = 0,1,2), αβ _mt is the Treatment × Occasion interaction, u_00kmt, r_0jkmt, and ϵ _ijkmt are the teacher-level, class-level, and student-level components of the residual for treatment m at time t. An unstructured covariance matrix (UCM) was specified for each component. That is, for each component the variances were permitted to be unequal for different occasions and the covariances were permitted to be unequal for different pairs of occasions. For models that did not converge with unstructured covariance matrices for student, class, and teacher components, we estimated new models that allowed either UCMs for student and teacher components of the residual and a compound symmetric covariance matrix (CSCM) for the class component or UCMs for the student and class components and a CSCM for the teacher component. We used the AIC fit index (Akaike, 1974) to select the model with the most appropriate variance-covariance structure. For models with a significant Treatment × Occasion interaction, we estimated treatment effects at each occasion. When the Time × Treatment interaction was significant, we estimated an MLM with a treatment effect and a Treatment × Pretest interaction at each of the last two occasions to evaluate if the treatment effect varied with the pretest score. The covariance structure was selected by following the same steps that we used for the repeated measures analysis.

Intervention integrity effects

To evaluate the relationship of student scores to intervention integrity, the number of skills mastered and the percentage of correctly followed decision rules were used as indicators of correct intervention implementation. To investigate the relationship of 2009 statewide test scores to intervention integrity, we estimated two-level means-as-outcomes (MAO) models (Raudenbush & Bryk, 2002) for students in the treatment group only. To investigate the difference in CBM scores by intervention integrity, we estimated two three-level intercepts-and-slopes-as-outcomes (ISAO) models for each CBM probe for students in the treatment group only. Occasion was the predictor at Level 1, and an intervention integrity indicator was used as the predictor at Level 2 for each model. In the first model, the number of skills mastered was the intervention integrity indicator; in the second, percentage of correctly followed decision rules was the indicator.

Core instruction in mathematics is highly variable across classrooms presenting research and practical challenges

An incidental finding of this study involved the substantial variation in instructional practices that occurred between classrooms within schools and within the district. Prior to the study, administrators reported a more consistent picture of mathematics instruction whereby a standard number of minutes was allocated to mathematics instruction and a standard core curriculum was used across schools, for example. Teacher survey data reflected substantial variation in time allocated to mathematics instruction in general and to specific instructional strategies in mathematics (e.g., fluency-building procedures).

Treatment and control classrooms in the same grade and school had the same amount of time allocated for mathematics instruction. Teachers in treatment classrooms replaced 15 minutes per day of their typical mathematics instruction with the fluency-building intervention for skills that the students could accurately perform and teachers in control classrooms continued with whatever instructional procedures they wished to use. Teacher-reported allocations of instructional time varied slightly between treatment and control classrooms, with teachers in treatment classrooms reporting allocating more time to fluency-building instruction and slightly less time to mathematics instruction in general (about 20 minutes per week).

We suspect there might have been substantial variation in both instructional time and strategy allocations across classrooms, some of which was documented using teacher surveys. Variation in instructional time allocations might have been a meaningful source of variance that was not adequately measured in this study and might be in future studies. To measure instructional time allocations adequately, extensive videotaping or observation of instruction in the classroom would be needed to document the variations in instructional time allocations across time and classrooms. Teacher report of instructional time allocations might be an error-prone way to quantify instructional time allocations given the nature of schools where interruptions are frequent and often unpredictable. One interesting question prompted by the findings of our study is why so many children were so low performing when teachers reported spending nearly 89 minutes per day on mathematics instruction. Instructional time allocations provide one way of estimating instruction, but questions about instructional quality will require more nuanced metrics (Hattie, 2009; Rivken, Hanushek, & Kain, 2005) that reflect not only the amount of time allocated to instruction but use of particular instructional strategies and child engagement and student performance. It is interesting to note that the strategies used in this randomized controlled trial (RCT) were commonly used instructional strategies in classrooms and in fact were strategies that teachers in control classrooms reported using. The most probable difference between typical classroom use of these strategies and the RCT intervention was that student data were used to ensure that the intervention content was matched to student proficiency.

Intervention implementation integrity requires direct measurement, and trials to criterion data might be a useful metric

Integrity of intervention implementation was a substantial potential threat in this study, as can be the case in an effectiveness study where the intervention is carried out by individuals who are not paid research project staff (O’Donnell, 2008). Because the threat of insufficient intervention integrity was anticipated, a number of procedures were undertaken to both estimate and facilitate correct intervention usage. Direct observation data seemed to be of questionable value in the absence of a second, trained independent observer to estimate integrity agreement. Direct observation integrity scores fell within a restricted range with a modal score of 100% of treatment steps correctly completed upon direct observation. Permanent products could be evaluated to estimate intervention integrity and reflected greater variation in correct intervention implementation. Contamination was documented via direct observation and by teacher report. Two teachers assigned to the control condition reported using the actual intervention and two teachers assigned to the control condition did not answer that question on a written anonymous survey. Some contamination was expected because the intervention contained procedures that are commonly used in classrooms during instruction (guided practice, independent timed trial with corrective feedback, incentives for improved performance). Because of the limitations associated with observing in control classrooms (reactivity, lack of independence between teachers, and small sample), it is not possible to estimate contamination in this study; rather, we can only conclude that contamination likely occurred to some extent.

The use of trials-to-criterion data as a reflection of intervention integrity when standard interventions are applied merits further scrutiny as a meaningful estimate of intervention integrity. In this data set, classes mastering more than 75% of intervention skills correctly followed 98% of decision rules and had 26.38 weeks of intervention data available. Classes mastering fewer than 50% of intervention skills correctly followed only 86% of decision rules and had available only 25.28 weeks of intervention data available. Despite efforts to assure adequate intervention integrity, it is clear that some classes in the intervention group received a lower-integrity intervention than did others. Given the substantial research base documenting integrity challenges with intervention implementation (McIntyre et al., 2007) and the data linking fidelity of intervention implementation to student learning gains during academic intervention (Witt et al., 1997), integrity of intervention implementation merits further scrutiny in practice, research, and policy.

When RtI is used in a school, it leaves a data source that might be exploited to provide a more sensitive estimation of actual integrity than random or nonrandom direct observation. Sanetti and Kratochwill (2009) assert that treatment integrity is a multidimensional construct that includes content, quality, quantity, and process and offer this definition: “Treatment integrity is the extent to which essential intervention components are delivered in a comprehensive and consistent manner by an interventionist trained to deliver the intervention” (p. 448). Teacher report of intervention usage alone is not an adequate basis for measuring actual integrity (Noell et al., 2005). However, direct observation and calculation of percentage of treatment steps observed during an intervention period might not be adequate either. The metrics used in this study to estimate treatment integrity represent a different method of estimating integrity that better captures the quality and process dimensions of treatment integrity using a data source that results when the intervention is used (i.e., permanent products). Two novel metrics were used in this study to reflect intervention integrity, trials to criterion given a standard intervention protocol and percentage of correct intervention decisions/actions as estimated by permanent products. These metrics merit further scrutiny and may contribute to a deeper understanding of intervention implementation integrity and assist researchers to estimate variable dosage effects of interventions (in terms of content, quality, quantity, and process) on student learning outcomes.

Ensuring all children learn well requires differentiated instruction

The intervention showed differential effects for students who were lower performing at baseline in fourth grade. Researchers have suggested that accountability pressures may promote the use of “educational triage” where decision makers allocate resources to students who are near the proficiency criterion at a cost to students (and schools) who are well above or well below the proficiency criterion, particularly where a static criterion is used for accountability (as opposed to growth) (Lauen & Gaddis, 2012). Research findings suggest that educational triage is not an inevitable consequence of accountability systems (Lauen & Gaddis, 2012), but identifying interventions that show stronger effects for students at particular points in a performance distribution may prove useful to systems that wish to improve learning outcomes for all students. One way that systems may be more strategic in their consumption of student assessment data is to use assessment data to verify that the right types of interventions (those shown to be most effective) are matched to students for whom those interventions are likely to be most effective.

RtI offers schools an opportunity to use data to drive instructional practices, evaluate the effects of instructional actions and decisions on their own local data set, and make midstream adjustments to their practices to ensure all students master essential learning objectives. Selecting an intervention that has been shown to work in research is an important first step but does not ensure that comparable effects will be obtained in all settings. In understanding evidence-based practices in education, it is important to quantify not only the practice, but the conditions within which the practice is effective (Gersten et al., 2005). Educators should adopt interventions that are well aligned with their specific system needs. Selecting instructional practices that are well aligned with system needs has two implications. First, the system must understand its needs (that is, what students have learned and how well they have learned it relative to grade-level expectations). Second, system decision makers must be able to consume their own data to identify instructional adjustments that will produce the desired improvements. RtI provides a rich data set that can be used to drive instructional decision making, but a common and fatal flaw of many RtI implementations is a failure for adults to consume the collected data and do something differently in classrooms as a result of the data. Collecting student data is not sufficient. If systems wish to see steady and universal performance improvements in basic mathematics proficiencies, systems must consume RtI data with an eye toward (a) improving stable use of high-quality, evidence-based core instructional practices in mathematics; (b) selecting and deploying interventions that are well aligned with student learning needs; and (c) using implementation data to make implementation adjustments needed to ensure the intervention will work as intended in all settings.