Modifying a Research-Based Problem-Solving Intervention to Improve the Problem-Solving Performance of Fifth and Sixth Graders With and Without Learning Disabilities

Abstract

The purpose of the present study was to test the efficacy of a modified cognitive strategy instructional intervention originally developed to improve the mathematical problem solving of middle and high school students with learning disabilities (LD). Fifth and sixth grade general education mathematics teachers and their students of varying ability (i.e., average-achieving [AA] students, low-achieving [LA] students, and students with LD) participated in the research study. Several features of the intervention were modified, including (a) explicitness of instruction, (b) emphasis on meta-cognition, (c) focus on problem-solving prerequisites, (d) extended duration of initial intervention, and (e) addition of visual supports. General education math teachers taught all instructional sessions to their inclusive classrooms. Curriculum-based measures (CBMs) of math problem solving were administered five times over the course of the year. A multilevel model (repeated measures nested within students and students nested within schools) was used to analyze student progress on CBMs. Though CBM scores in the intervention group were initially lower than that of the comparison group, intervention students improved significantly more in the first phase, with no differences in the second phase. Implications for instruction are discussed as well as directions for future research.

Keywords

problem solving/calculation strategy instruction cognitive strategies

The difficulty students experience with mathematical word problems has long been recognized, both in research and in practice (Hudson & Miller, 2005; Jitendra & Xin, 1997). Students with or at risk for learning disabilities (LD) find word problems to be particularly daunting, as these types of problems require not only reading and computation proficiency but also problem representation and solution planning (Xin, Jitendra, & Deatline-Buchman, 2005). These skills represent consistent areas of deficit for students with LD (Montague & Applegate, 1993), especially when word problems are linguistically complex (Fuchs & Fuchs, 2002). Yet problem solving is a critical skill, as evidenced by its prominence on national and international assessments (e.g., National Assessment of Educational Progress [NAEP], Program for International Student Assessment [PISA]), evaluations of workplace competency (e.g., Workforce Readiness Report), and college preparatory coursework (Common Core State Standards [CCSS] Initiative, 2014). Research on math problem solving has increased over the past 20 years in response to these changes and the policy reports that have urged greater emphasis on problem-solving skills (Bottge & Cho, 2013).

In the field of special education, research has identified strategy instruction as a powerful means of developing proficiency in complex skills such as problem solving (Griffin & Jitendra, 2009; Montague, Krawec, Enders, & Dietz, 2013; Pressley & Hilden, 2006; Swanson, Hoskyn, & Lee, 1999). In this approach, students learn how to apply strategic tools during a task (e.g., note-taking, summarizing, paraphrasing, estimating) and monitor their thinking during task execution using meta-cognitive processes such as self-questioning, self-checking, and self-correcting. Although the specific cognitive components and structure differ across interventions, there is general consensus about the cognitive strategies important for students with LD who struggle in math problem solving, including paraphrasing for problem comprehension (Swanson, Moran, Bocian, Lussier, & Zheng, 2013), visualizing relationships among problem parts (Jitendra, Griffin, Deatline-Buchman, & Sczesniak, 2007; Krawec, 2014; van Garderen, 2006), and creating a viable solution plan (Fuchs & Fuchs, 2007). Yet although there is a strong body of research on middle and high school grades (e.g., Jitendra et al., 2009; Jitendra et al., 2014; Montague et al., 2013) as well as in Grades 1 through 3 (e.g., Bryant, Bryant, Williams, Kim, & Shin, 2013; Jitendra et al., 2007; Owen & Fuchs, 2002), there is much less research at the pivotal fifth and sixth grades, when students must bridge arithmetic to algebra, that domain of math long recognized as the gatekeeper to postsecondary academic trajectories (Paul, 2005). Problem-solving strategies provide a means to solving algebraic problems and thus aiding students in developing algebraic proficiency.

The purpose, then, of the present study was to determine the effectiveness of specific modifications for fifth and sixth graders to an intervention shown to be effective in improving the problem-solving performance of students with LD in Grades 7 through 12. In 1986, the efficacy of the original intervention was first established with high school students with LD; participants improved both their problem-solving performance as well as their meta-cognitive knowledge of effective strategy use (Montague & Bos, 1986). A follow-up study tested the same intervention with middle school students with LD (Montague, 1992). Using a single-subject design, the author found that although the intervention was successful in improving the problem-solving performance of seventh and eighth grade students, sixth grade students never achieved mastery on the problem-solving measures utilized despite additional treatment sessions. Montague suggested that because “the sixth grade students remained noticeably deficient in their knowledge, use, and control of problem representation strategies, an adaptation of strategy instruction that … focuses on concept development, use of manipulatives during practice sessions, and a progression from simple to complex problems may be more appropriate” (p. 240). Subsequent research using increasingly rigorous designs, including quasi-experimental and randomized controlled trials, validated Montague’s (1992) findings on the effectiveness of the intervention for seventh and eighth graders with LD and found that it also improved the performance of both AA and LA math students without LD (Montague, Enders, & Dietz, 2009, 2011; Montague et al., 2013). Practical effects of studies were consistently large (d = .674 to .979). However, in the 22 years since Montague’s (1992) initial findings and despite the positive impact of the intervention on older students, no research has investigated the modifications necessary to improve the problem-solving performance of younger students (i.e., those in fifth and sixth grades).

There are two reasons why this particular study is important. First, as described earlier, prominent researchers in the field of special education who have focused their efforts on problem solving have tended to target either Grades 2 and 3 (e.g., Bryant et al., 2013; Fuchs & Fuchs, 2007; Griffin & Jitendra, 2009; Jitendra et al., 2007; Powell & Fuchs, 2010) or middle grades (e.g., Jitendra et al., 2009; Montague et al., 2013). Second, results of the 2013 NAEP showed that almost half of fourth graders with disabilities performed at the below basic level compared to 14% of their nondisabled peers; in eighth grade, the percentage of students with disabilities performing below basic rose to 65% compared to 21% of eighth graders without disabilities. Though students with disabilities in fourth grade are performing significantly more poorly than their nondisabled peers, that achievement gap further widens by eighth grade. As math problem solving is a critical skill to determine mathematical proficiency, it is imperative that interventions in math focus on problem solving and target students between fourth and eighth grade with the intention of reducing (and, ultimately, eliminating) that achievement gap. The modified problem-solving intervention in the present study was designed to do just that. The following section begins with a definition of problem solving, followed by a detailed description of the cognitive and meta-cognitive components of Montague’s (2003) original problem-solving intervention and its instructional characteristics. The modifications made to address the specific needs of students in fifth and sixth grades are then provided.

Problem Solving

Because the focus of this intervention is on computational problem solving, it is necessary to first provide a definition of the term. According to the Organization for Economic Cooperation and Development (OECD, 2013), problem solving “involves a set of critical control processes that guide an individual to effectively recognize, formulate and solve problems. This skill is characterized as selecting or devising a plan or strategy to use mathematics to solve problems arising from a task or content, as well as guiding its implementation” (p. 31). Though this definition is somewhat circular (defining problem solving as solving problems), it does highlight how it is done. The present intervention explicitly teaches these “critical control processes” to which OECD refers. For this intervention, problem solving consists of single- and multistep problems using the four operations with whole numbers and decimals. Open-ended problems were not included.

It is important to note that problem solving is often characterized as “engaging in a task for which the solution method is not known in advance” (National Council of Teachers of Mathematics [NCTM], 2000, p. 52). This statement implies that to label a task as problem solving, one must first consider the individual attempting the problem. Problems that are straightforward and procedural to a proficient problem solver may, for a struggling student, meet NCTM’s definition of problem solving as a task with an unknown solution method (Sharpe, Fults, & Krawec, 2014).

The Problem-Solving Intervention

The intervention implemented in this study aligns with the problem-solving model used by OECD (2013) for PISA. It identifies four phases: (1) exploring and understanding, (2) representing and formulating, (3) planning and executing, and (4) monitoring and reflecting. Figure 1 shows the relationships among the phases with the seven cognitive strategies of the problem-solving intervention embedded within Phases 1 through 3. Phase 4 represents the meta-cognitive component, which supersedes the other phases, monitoring their accuracy. Phase 4 reflects the meta-cognitive components of the intervention that are embedded within each cognitive strategy.

Figure 1.

A Problem-Solving Model Based on the PISA Framework (OECD, 2013) and Montague’s (2003) Problem-Solving Processes.

Cognitive processes

As depicted in Figure 1, the intervention is comprised of cognitive processes that are integral to the development and application of strategic knowledge of mathematics and essential to successful math problem solving. The ultimate goal of the program is to have students internalize the cognitive processes so that they become automatic during problem solving. Because struggling students lack the knowledge and/or the utilization of key strategies (Krawec, Huang, Montague, Kressler, & Alba, 2013; Montague & Applegate, 1993), this intervention makes their application explicit for students. Keeping in mind that problem solving is a recursive activity (i.e., meta-cognitive cues may direct the student to start again, revisit a previous step, or jump ahead to a later one), the intervention teaches students the following:

Read the problem by reading, rereading, and identifying relevant/irrelevant information.

Paraphrase the problem by putting it into their own words while maintaining its meaning.

Visualize the problem by forming a schematic representation of critical problem parts.

Hypothesize a problem solution, determining the operation(s) and number of steps.

Estimate the answer as a way to confirm the solution plan and the outcome.

Compute the answer by using the correct algorithmic procedures.

Check the accuracy of the solution process and product.

Meta-cognitive strategies

Whereas cognitive processes are proactive in nature, meta-cognitive strategies require reflectivity and reactivity. Problem solvers reflect on what they are doing and react to what they have done. These meta-cognitive (i.e., self-regulation) strategies emphasize self-awareness of cognitive knowledge, deployment of cognitive processes and strategies during problem solving, and control of processes and strategies for purposes of regulating, evaluating, and monitoring performance (Berardi-Coletta, Dominowski, Buyer, & Rellinger, 1995). Problem solvers use self-regulation strategies to tell themselves what to do, ask themselves questions, recall what they know, detect and correct errors, and monitor performance. As such, in Figure 1, the meta-cognitive processes (i.e., say, ask, and check) are reflected in the monitoring and reflecting phase and activated throughout the problem-solving process.

Instructional characteristics

In addition to the instructional content of the intervention, there are key instructional characteristics aligned with best practices for students with LD. The methodology for teaching students the intervention routine is explicit instruction, which is based on developmental theories and incorporates scientifically demonstrated instructional principles and procedures (for a review, see Fuchs, Fuchs, Schumacher, & Seethaler, 2013; Grouws, 1992; Montague, 2003). Explicit instruction utilizes validated teaching strategies such as cueing, modeling, rehearsal, and feedback and is characterized by structured and organized lessons, scaffolded supports, guided and distributed practice, immediate and corrective feedback on learner performance, positive reinforcement, overlearning, and mastery (Montague, Warger, & Morgan, 2000).

Modifications for Grades 5 and 6

Several features of the intervention were targeted based on previous research, current understanding of the developmental characteristics of fifth and sixth grade students, and a pilot study, resulting in the following five areas targeted for modifications: (1) explicitness of instruction, (2) emphasis on meta-cognition, (3) focus on problem-solving prerequisites, (4) duration of the initial intervention lessons, and (5) the use of visual supports. We increased the explicitness of instruction for fifth and sixth grade students by expanding the original 1-day lesson on the comprehensive problem-solving routine into three separate lessons: modeling the routine, paraphrasing, and visualizing. Each lesson built on the previous one, modeling the interactive and recursive nature of the strategies during problem solving. The role of meta-cognition in the intervention was also expanded. As internalization of meta-cognitive skills is typically established between the ages of 11 and 14 (Kass & Maddux, 2006), for students in fifth and sixth grades, additional support was necessary to scaffold the development of meta-cognition and then gradually shift to its independent and flexible use. Students were given additional meta-cognitive prompts in a logical sequence within each cognitive strategy (e.g., instead of the self-instructive prompt to “make a drawing or a diagram,” students first asked themselves, “What am I looking for? Am I looking for the total?” before continuing with prompts to make a drawing). Figure 2 displays the modified cue cards.

Figure 2.

The Modified Cue Cards for Grades 5 and 6. From Solve It! Teaching Mathematical Problem Solving in Inclusive Classrooms–Grades 5-6 (p. 30), by J. Krawec and C. Warger, 2015, Reston, VA: Exceptional Innovations, Inc. Copyright 2015 by Exceptional Innovations, Inc. Reprinted with permission.

The pilot study also identified deficits in fifth and sixth grade students with LD related to the prerequisite skill of concepts of operations. As such, an additional lesson was developed to precede the initial intervention. Prior to any specific problem-solving instruction, students received one lesson on the conceptual basis of each of the four operations, with subsequent lessons revisiting these concepts to reinforce learning (i.e., reinforcing the definitions by incorporating them into think alouds, directing students to the operation decision tree, etc.). Finally, several visual cues were added to the intervention to support students’ paraphrasing, visualizing, and planning. An online appendix includes the Concepts of Operations lesson as well as the visual cues used.

In summary, we extended the initial instructional sessions from 3 to 6 days, expanded the focus on paraphrasing and visualizing strategies, increased the explicitness of the meta-cognitive components, taught and then reinforced concepts of the four arithmetic operations, and added visual supports to aid students’ use of the intervention components. (The script for the weekly practice sessions, which followed initial instructional sessions, was not altered.) Table 1 provides an overview of the specific changes to the intervention. As with the original intervention, the practice sessions were conducted weekly for the 6 months following initial instructional sessions.

Table 1.

Modifications to the Original Intervention.

Component	Original intervention: 7th–12th grades	Modified intervention: Fifth and sixth grades
Initial instructional sessions^a	1: Introduction (50 min) 2: Modeling the routine (50 min) 3: Application (50 min)	1: Concepts of operations (50 min) 2: Introduction^b (50 min) 3: Modeling the routine (50 min) 4: Paraphrasing (50 min) 5: Visualizing (50 min) 6: Application (50 min)
Visual cues	Student cue cards Class charts	Student cue cards Class charts Operations decision tree Paraphrasing framework^c Visualizing framework^c
Meta-cognitive prompts	“Say,” “ask,” “check” for each cognitive strategy	More explicit prompts (i.e., multiple “say” and/or “ask” prompts within each cognitive strategy)

Initial instructional sessions do not include the 50-min weekly practice sessions that follow. The weekly practice session script was not modified from the original intervention and is included as is in both interventions.

The concepts of operations lesson is included in an online appendix.

The paraphrasing and visualizing frameworks are included in an online appendix.

The following research questions guided the study:

What are the effects of the modified problem-solving intervention on students’ problem-solving progress over time, as measured by the curriculum-based measures (CBMs)?

Do these effects differ by ability (AA, LA with and without LD)?

Do teachers perceive the intervention as being socially valid?

Method

Participants

Initially, four schools with kindergarten through eighth grade (K8) were selected to participate in this research study. Each school was matched to another on variables of school performance (as measured by the school grade assigned by the state), socioeconomic status (SES; as measured by percentage of school population receiving free/reduced lunch), and ethnic diversity. One school from each matched pair was randomly assigned to the intervention condition. However, prior to the start of the study, one school decided to withdraw from participation. It was replaced with one elementary and one middle school, which were similar in terms of student SES, school performance grade, and ethnic diversity. Thus, five schools (three K8 schools, one elementary school, one middle school) participated in the study. Because the withdrawal of the original K8 school occurred after the summer professional development for intervention teachers, the replacement elementary and middle schools had to be assigned to the comparison condition; thus, random assignment was part of the intended design but not upheld.

Two teachers participated at each K8 school (one fifth grade and one sixth grade teacher), along with a fifth grade teacher at the elementary school and a sixth grade teacher at the middle school; in total, eight teachers participated. Teachers within participating schools were selected by the school principal based on the following criteria: the teacher must be “high quality” as determined by the nominating administrator, certified in mathematics education (i.e., having passed the state subject area math exam for teaching Grades 5 through 9), teaching at least one class period that included LA students and students with LD, and for the intervention teachers, able to attend the one and one-half day professional development workshop on the intervention. Two fifth and two sixth grade teachers participated in the intervention condition (n = 4) and two fifth and two sixth grade teachers participated in the comparison condition (n = 4).

A research assistant presented the study to students from each participating class and handed out student assent forms and parent consent forms. All students in the participating inclusive classrooms were eligible to participate in the larger project; for the present study, only students with LD (district-identified with LD and below grade-level score of 1 or 2 of possible 5 on the state math assessment the previous year), LA students (no disability and below grade-level score of 1 or 2 on the state math assessment the previous year), and AA students (no disability and on grade-level score of 3 on the state math assessment the previous year) were included in the analyses. Across all schools, 386 students participated in the overall project, but 79 were removed for the following reasons: disability status other than LD (n = 25), high-achieving/gifted (n = 31), and insufficient demographic information (n = 23). Demographic data for the 307 participating students are presented in Table 2.

Table 2.

Demographic Information of Participating Students.

	Intervention (n = 191)	Comparison (n = 116)
Variable	n (%)	n (%)
Grade
Fifth	55 (29)	68 (59)
Sixth	136 (71)	48 (41)
Ability level
AA	69 (36)	60 (52)
LA	105 (55)	42 (36)
LD	17 (9)	14 (12)
Gender
Male	100 (52)	53 (46)
Female	91 (48)	63 (54)
Ethnicity
Hispanic	147 (77)	93 (80)
Black	10 (5)	6 (5)
White	34 (18)	17 (15)
Free/reduced lunch
Yes	111 (58)	90 (78)
No	80 (42)	26 (22)

Note. AA = average achieving; LA = low achieving; LD = learning disabilities.

Measures

Five CBMs of math problem solving were developed by the research team and calibrated using Item Response Theory methods to achieve equivalent difficulty level. Each CBM was created by combining two 5-item blocks from a pool of nine blocks composed of 45 discrete word problems in all. Each five-item block contained approximately one 1-step problem, three 2-step problems, and one 3-step problem (for a more thorough description, see Montague, Penfield, Enders, & Huang, 2010). The internal consistency of the measures ranged from .82 to .84. Each of the five measures consisted of 10 one-, two-, and three-step textbook-type math problems, which used all four operations and included whole numbers and decimals.

The social validity scale is a researcher-developed 4-point Likert-style questionnaire with 17 items that were developed following Carter’s (2010) guidelines. Teachers responded to statements such as “The target problem-solving skills emphasized in this intervention are important” and “My ability to teach math problem solving increased as a result of this intervention.” Intervention teachers completed the questionnaire during the final month of the school year. In addition to the questionnaire, they were provided space to expand on any of the 17 questions or to add comments.

Procedures

The modified intervention includes seven scripted lessons (the six initial instructional sessions: concepts of operations, intervention introduction, modeling the routine, paraphrasing, visualizing, and application; and the script for the weekly practice sessions), class charts of the cognitive/meta-cognitive routine and the operations decision tree, and student cue cards. All intervention materials including class sets of practice problems were provided for the school year. The scripted lessons were intended to cover a 50-min class period and incorporated explicit instructional procedures for helping students acquire, apply, maintain, and generalize problem-solving processes, strategies, and skills (see Montague, 2003). The six initial lessons in the intervention were implemented over 6 consecutive school days in mid-October, followed by weekly practice sessions using word problems aligned with the district pacing guide. The focus of the weekly practice sessions varied depending on student need. Early in the year, students worked through only one or two word problems, with substantial time dedicated to elements of the routine (e.g., paraphrasing, visualizing) and discussion of alternative solution paths. Over the course of the year, teachers were able to include four or five problems in a session as students became more proficient in their problem solving. Likewise, student groupings changed over time. Teachers had students work in small groups early in the year but transitioned over time to pairs and eventually independent work. Except for holidays, conflicts with other school activities, and testing, practice sessions occurred each week for the duration of the intervention; teachers taught between 12 and 16 practice sessions. All instructional sessions, including the six initial sessions and the subsequent weekly practice sessions, were observed for treatment fidelity. Students in the comparison group received typical classroom instruction in math problem solving. Research assistants also observed in comparison classrooms to determine the specific nature of instruction as well as to identify any overlap with the intervention condition. CBMs were administered to each teacher’s participating math class periods five times: prior to intervention, 1 month following, and then every other month for the remainder of the school year.

Intervention Fidelity

Intervention classrooms

Observation checklists for the initial 6-day intervention plus the subsequent practice sessions were developed based on the content, components, and procedures associated with the intervention and thus include criteria for both teacher adherence to the lesson content (e.g., provides the definition for paraphrasing) and teacher competence of implementation related to the critical teaching techniques (e.g., provides immediate and corrective feedback). Each checklist contained between 13 and 16 items that were scored as either “yes” or “no.” That is, the behavior was either demonstrated or was not. Intervention teachers were given verbal feedback following the observation and expected to use that feedback to improve subsequent instruction. Level of treatment fidelity and interrater agreement were averaged across the observations for the intervention group and were calculated by dividing number of agreements by agreements plus disagreements multiplied by 100. For the intervention group observations, fidelity averaged 92.0% (interrater agreement = 94.0%).

Comparison classrooms

In order to determine whether any of the critical intervention components were also present in comparison classroom instruction, detailed observations were conducted four different times over the course of the year to determine comparison teacher instructional practices across the school year. Qualitative analyses of the observations revealed that most incorporated at least one of the intervention strategies into instruction, with paraphrasing (underlining and restating), rereading, and planning the most frequently utilized instructional content, and performance feedback, practice with peers, and active student participation the most frequently used instructional approach. Only one comparison teacher consistently emphasized the superficial and ineffective keyword strategy as a primary teaching tool.

Analyses

Statistical analyses were conducted using the Mplus 7.3 (Muthen & Muthen, 2014; Muthen & Muthen, 2012) and the SPSS 17.0 software packages. Only five schools, with two classrooms in K8 schools and one each in the elementary and middle school, participated in this study; little school-level variance could be modeled, so we dropped it from the model. We also excluded from subsequent analysis those students who had missing data on the student-level predictors measured (e.g., gender, race, free/reduced lunch, etc.) The data were consistent with a two-level growth curve model in which repeated measures (Level 1) were nested with students (Level 2). A Level 2 dummy variable that represented participation in the study (1 = intervention school and 0 = comparison school) was added as the predictor. Our primary interest was to investigate whether the two conditions differentially improved during the school year. Furthermore, the interaction terms were added to examine whether the ability group (LA, AA, and LD) moderated the impact of the intervention effect. Internal consistency of the CBMs ranged from .82 to .84. The equated CBM scores were used in this study (Montague et al., 2010).

Mplus allows for estimation of models with missing data using maximum likelihood estimation under missing completely at random and missing at random assumption. The goodness of fit of the estimated models was evaluated with five indicators, χ-square test, comparative fit index (CFI), Tucker Lewis Index (TLI), root mean square error of approximation (RMSEA), and standardized root-mean square residual (SRMR). These statistical indices were used to compare model fit between different models. Good fit values were considered greater than .95 for CFI and TLI and less than .08 for RMSEA and SRMR (Browne & Cudeck, 1993; Hu & Bentler, 1999).

Results

The purpose of this study was to examine whether the intervention improved the mathematical problem solving of fifth and sixth grade students of varying ability. Two-level piecewise growth model analyses were used to accommodate the nested data structure and determine the effects of the intervention on students’ performance on problem solving. The fit indices were used to access the overall latent growth model fit, including χ-square test, SRMR, RMSEA, CFI, ad TLI.

The following model-building process was conducted. First, unconditional models were estimated in order to identify sources of the score variation and two sources were identified, within- and between-student variability. Then both within-student and between-student variability were examined to evaluate between-student variability in growth trajectories. Finally, analyses focused on the fit of linear, quadratic, and piecewise models of growth and the comparison of these competing models (see Table 3). Initial examination of time-specific CBM means suggested that a linear latent growth model might not be appropriate as mean values increased until the fourth time point and then showed a downturn after this point, as seen in Figure 3. Student scores in both comparison and intervention groups dropped on the last CBM, which was administered in the final week of the academic school year. The results of the fit indices for three growth models (i.e., latent, quadratic, and piecewise) prompted us to pursue nonlinear growth models with slopes at the student level, which fit the data significantly better than the other two models.

Table 3.

Comparison of Three Growth Models.

Model type	χ²	p	RMSEA	CFI	TLI	SRMR
Linear growth	109.447	<.001	.078	.892	.844	.028
Quadratic growth	150.228	<.001	.131	.810	.564	.022
Piecewise growth	42.703	.03	.044	.976	.952	.024

Note. RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker Lewis Index; SRMR = standardized root-mean square residual.

Figure 3.

Data-Implied Growth Trajectories of Treatment and Comparison Groups Across the Five Curriculum-based Measures (CBMs).

Piecewise trajectory modeling is applied to capture the nonlinear function through the use of two or more linear piecewise splines by dividing a time series into two meaningful phases to capture key features of change in each phase. Because the study’s four estimated time points were not equally spaced prior to the transition, we set the value of our first time point at 1 and then set values for the subsequent time points in relation to their relative distance to the first time point. After the transition, the slope for the fourth time point was fixed at 0 and the fifth point at 1.

We estimated a two-level growth model with the repeated measures over time being the first level and students the second level to evaluate whether the comparison and intervention groups differentially improved in math problem solving over the course of the school year. We used the variable $T i m e_{t i}$ to create two predictor variables $T i m e O n e_{t i}$ and $T i m e T w o_{t i}$ , which enabled us to capture individual growth rate in two phases. The two-level growth model expresses the outcome variable as a function of temporal predictor variable that captures the passage of time after controlling for student-level covariates. The first model is expressed as follows:

\begin{array}{l} L e v e l 1 : C B M_{t i} = π_{0 i} + π_{1 i} (T i m e O n e_{t i}) \\ + π_{2 i} (T i m e T w o_{t i}) + e_{t i} \end{array}

\begin{array}{l} L e v e l 2 : π_{0 i} = β_{00} + β_{01} (C o n d i t i o n_{i}) + β_{02} (L A_{i}) \\ + β_{03} (L D_{i}) + β_{04} (L u n c h_{i}) + β_{05} (G e n d e r_{i}) \\ + β_{06} (H i s p a n i c_{i}) + β_{07} (A f r i c a n_{i}) \\ + β_{08} (F C A T R e a d_{i}) + β_{09} (F C A T M a t h_{i}) + r_{0_{i}} \end{array}

\begin{array}{l} π_{1 i} = β_{10} + β_{11} (C o n d i t i o n_{i}) + β_{12} (L A_{i}) \\ + β_{13} (L D_{i}) + β_{14} (L u n c h_{i}) \\ + β_{15} (G e n d e r_{i}) + β_{16} (H i s p a n i c_{i}) \\ + β_{17} (A f r i c a n_{i}) + β_{18} (F C A T R e a d_{i}) \\ + β_{19} (F C A T M a t h_{i}) + r_{1_{i}} \end{array}

\begin{array}{l} π_{2 i} = β_{20} + β_{21} (C o n d i t i o n_{i}) + β_{22} (L A_{i}) \\ + β_{23} (L D_{i}) + β_{24} (L u n c h_{i}) + β_{25} (G e n d e r_{i}) \\ + β_{26} (H i s p a n i c_{i}) + β_{27} (A f r i c a n_{i}) \\ + β_{28} (F C A T R e a d_{i}) + β_{29} (F C A T M a t h_{i}) + r_{2_{i}}, \end{array}

where $C B M_{t i}$ is the outcome score for student i at time t and $T i m e_{t i}$ is the value of the temporal predictor for student i at time t. The parameters π_0i, π_1i, and π_2i are growth parameters. π_0i is the intercept, π_1i is the expected change in the outcome variable for an increment in the Time variable before the transition, π_2i is the expected change in the outcome variable for an increment in the Time variable after the transition, and $e_{t i}$ is a residual. The growth parameters contained in the within-student model are treated as outcomes in a between-student model. The covariates included two dummy codes representing three-category ability variables (LA vs. AA, LD vs. AA), a dummy code representing free or reduced lunch, a gender dummy code, two dummy codes representing a three-category ethnicity variable (African American vs. White, Hispanic vs. White), state math assessment achievement scores, and state reading assessment achievement scores (state assessment scores were standardized to have a mean of 0 and standard deviation of 1). These variables were grand mean centered in order to control for student-level differences in the covariates (Ender & Tofighi, 2007).

In the first phase, the intervention group improved at a significantly higher rate than the comparison group (see Table 4). The effect size comparing intervention and comparison groups using Cohen’s (1988) d was .43. In the second phase, there was no significant difference in rate of growth between comparison and intervention groups. In the second model, the interaction term was introduced in order to investigate whether ability group (AA, LA, and LD) moderated the impact of the intervention effect. The moderating influence of ability group on the intervention was evaluated after investigating the intervention effect. The results showed the interactions between experimental condition and ability group were not significant, indicating that ability group had no significant impact on the intervention effect. In other words, AA students did not respond differently to the intervention than LA students or those with LD.

Table 4.

Two-Level Parameter Estimates Controlling for Student-Level Covariates.

Parameter	Est.	SE	p	95% LCL	95% UCL
Slope 1
Condition	.084	.033	.012	.029	.138
LA vs. AA	.102	.097	.293	−.058	.262
LD vs. AA	.046	.108	.672	−.132	.223
Free/reduced lunch	−.059	.014	.000	−.082	−.036
Gender	.098	.020	.000	.066	.131
African American vs. White	−.001	.154	.993	−.255	.252
Hispanic vs. White	.032	.058	.578	−.063	.127
FCAT reading	−.002	.001	.044	−.003	.000
FCAT math	−.003	.003	.240	−.008	.001
Slope 2
Condition	−.090	.144	.531	−.327	.146
LA vs. AA	−.274	.235	.244	−.660	.113
LD vs. AA	−.060	.331	.857	−.603	.484
Free/reduced lunch	−.058	.177	.745	−.349	.234
Gender	.044	.173	.801	−.240	.327
African American vs. White	.039	.161	.809	−.225	.303
Hispanic vs. White	.182	.115	.112	−.006	.371
FCAT reading	−.002	.009	.801	−.016	.012
FCAT math	−.003	.012	.778	−.023	.017

Note. Est. = estimate; SE = standard error; LCL = lower control limit; UCL = upper control limit; LA = low achieving; AA = average achieving; LD = learning disabilities; FCAT = Florida Comprehensive Assessment Test.

Social Validity

Teachers rated the social validity of the intervention using the social validity questionnaire, which addressed their perception of the value of the intervention in terms of feasibility, usability, and its potential to be incorporated into the larger curricular structure. Responses across teachers were very consistent. Overall, teachers found the intervention to be effective for their students and a useful addition to their math instruction. All recognized that the intervention targeted critical problem-solving skills and complemented the existing curriculum. All four intervention teachers strongly agreed that the professional development prior to the school year helped them implement the intervention with fidelity. Of particular value to these participating teachers were the processes of paraphrasing and visualizing; they reported that students responded well to the visualizing instruction and that the intervention improved students’ problem representation abilities. Further, 100% of the teachers found the intervention easy to implement in their classrooms, and three of four said they would recommend the intervention to other teachers of students with LD. However, teacher responses did vary on the degree to which the intervention fit their existing math schedules. Not surprisingly, the two teachers on block scheduling (i.e., students receive math instruction every other day for longer periods) found it easier to implement than teachers on traditional scheduling who saw students every day for shorter periods. Interestingly, only one of the teachers agreed that the intervention had improved her own ability to teach problem solving. Because we had no measure of teachers’ initial instructional style or problem-solving approach, we could not corroborate this perception.

Discussion

Overall, significant findings indicated the effectiveness of the modifications to the problem-solving intervention to improve the performance of students of varying ability. However, findings of the present study do not fully replicate previous research on this intervention (e.g., Montague et al., 2013); specifically, the performance of the comparison group in the present study showed growth over time that was similar to that of the intervention group. Similar to previous studies, though, no significant Growth × Ability interactions were detected, indicating that the intervention was equally effective across LA and AA students and those with LD. The focus of the discussion will center on the following two findings: the significant improvement of the intervention group over the course of the year and growth over time of the comparison group.

Improved Problem-Solving Performance

Despite growth over time in both experimental conditions, it must be noted that the intervention group, whose performance at the outset was below that of the comparison group, improved significantly (d = .43) through the first phase of the results (i.e., CBMs 1 through 4). Although the number of students with LD participating in the study was too low to conduct reliable statistical analyses, mean performance growth showed that students with LD in the intervention group performed 20% better than students in the comparison group on the final CBM. Despite this difference, students with LD in the intervention group still did not, on average, meet mastery criterion (set at 70%). That is, the intervention improved their problem-solving performance but not enough to reach a level of mastery widely accepted as proficient.

These results, particularly that students with LD responded to intervention but did not meet mastery levels, are not uncommon in special education research implemented by teachers themselves; some well-established interventions have demonstrated significant student growth but low overall accuracy (e.g., 45%) (Jitendra, Dupuis, Star, & Rodriguez, 2014; Owen & Fuchs, 2002). Thus, an important first step is identifying interventions that improve performance over that of the comparison group; the second and equally important step, then, is to raise that performance to grade-level proficiency. Further research is needed on the present intervention, focusing specifically on students with LD, to determine the specific components of the modified intervention that contribute to student growth as well as whether increasing the intensity of instruction for students with LD will result in problem-solving proficiency.

Comparison Group Growth

Previous studies that implemented the original problem-solving intervention with middle and high school students showed relatively flat growth in the comparison group over time (Montague et al., 2013). Though we anticipated a similar trend in the present study, an explanation for comparison group growth may be provided through analyses of comparison classroom observation measures over time. As noted previously, comparison classrooms were observed to determine the type of problem-solving instruction delivered. Results showed that various effective problem-solving strategies were being taught in some classrooms. Previous research that utilized treatment fidelity checks in comparison classes noted only a 2.8% fidelity to the treatment across comparison teachers (Montague et al., 2009); however, in the present study, at least one aspect of the instructional content as well as the instructional approach were utilized by three of the four comparison teachers. Although a percentage-of-overlap value would be helpful to determine the degree of impact, the treatment fidelity checklists for the practice sessions did not sufficiently reflect specific components of the intervention (e.g., prompting use of cognitive strategies of paraphrasing, modeling the solution plan, etc.). Although we cannot assign a discrete value to the percentage of overlap, comparison student performance may have increased over time due to some instructional features (identified in the detailed observations of practice in the comparison group) that were shared between the two experimental conditions. The underlying cause of this substantial increase of intervention components in comparison classrooms across years may be related to the changing curricular demands in math education; that is, the 6-year span between implementation of the two studies represents a great deal of change in the national approach to the teaching of mathematics. Development of the CCSS was initiated in 2009, with adoption by states beginning in 2011 (CCSS Initiative, 2014). The emphasis of the CCSS on problem solving (e.g., the prominence of problem-solving skills and processes throughout the eight standards for mathematical practice) and the current ubiquity of the CCSS in the curriculum may now be making an impact on general teaching practice.

Recent special education research in other academic domains, including fractions and reading comprehension, has shown a similar change in the comparison group profile (Lemons, Fuchs, Gilbert, & Fuchs, 2014). The authors identified a dramatic decrease in the effectiveness of their intervention across five randomized controlled trials spanning 9 years and determined that performance of students after intervention was maintained or improved across the years, whereas the trend across the comparison groups dramatically increased in the most recent years. They cautioned that these “data highlight the importance of considering time and place and the possibility of a changing counterfactual when interpreting experimental and quasi-experimental research, thereby leading to more nuanced understandings of education science” (p. 244).

Limitations

Several limitations of the study are apparent. First, due to one school’s last-minute decision not to participate, we were unable to randomly assign schools to conditions. Although math ability across conditions was equal, student performance on the first problem-solving CBM was not, with the intervention group scores significantly lower than those of the comparison. Second, critical characteristics of the problem-solving intervention were present in the comparison condition. Although this finding is actually positive in its implications for quality instruction in typical classrooms, it complicated analyses and made clear conclusions difficult. In the first phase, the intervention group showed significant growth over that of the comparison group, but we cannot confidently attribute that growth to the problem-solving intervention due to the shared elements across conditions. The final limitation of the study is that we administered the fifth CBM during the final week of the school year. Although students showed a relatively steady increase over the course of the year, performance on the final CBM was noticeably poorer. In fact, teachers in both conditions warned us of a potential fall-off in student performance, citing a lack of student academic engagement and focus due to the end of the school year (which was riddled with assemblies, field trips, and fun class activities). Future research should position the final assessment no later than 1 month prior to the close of the school year.

Instructional Implications

From a policy standpoint, it is exciting to see that research-validated teaching strategies and techniques are finding their way into typical classroom instruction and that student performance is responding. Yet the results of the present study indicate that these changes are not comprehensive enough, as significant growth over time in the intervention group demonstrates the effectiveness of the intervention and its modifications to improve students’ problem solving over and above that which occurs in typical classroom instruction. However, that many of the students with LD did not meet mastery indicates that further research is necessary. A component analysis that identifies the instructional modifications that improve practice will help to clarify ways in which instruction can be made more intensive for students not responding to the intervention as designed. As this intervention was implemented in a whole class setting with content integrated into the general curriculum and pacing guide, it reflects the structure of a Tier 1 intervention in a response to intervention (RTI) model of instruction. In the same way that RTI provides nonresponders in the first tier with intensified quality, research-based instruction, this intervention may best serve the needs of all students (but particularly those with LD) if it also offers more intensive small group intervention that focuses on the modified elements most critical to improved performance.

Conclusion

Conducting school-based research is a major endeavor not only for researchers but also for administrators and teachers. The purpose of the present study was to conduct a relatively small-scale study in general education math classes in five schools to investigate the efficacy of an intervention that had been validated with students with LD. Although the results were generally positive, several limitations were noted that should be addressed in future studies. The intervention improved students’ math problem solving and the growth of students with LD was particularly encouraging following intervention, but many of these students still did not meet mastery. Thus, although the modifications to the intervention were effective in improving the performance over that of the comparison group, more intensive intervention may be necessary to increase performance of students with LD to mastery level (i.e., 70%).

Although this study cannot identify the intervention as a research-validated package that teachers can immediately implement, it does provide insight into the strengths of the modifications made and it highlights areas that must be further differentiated from typical classroom instruction in order to bring about the necessary improvement in performance. This intervention provides a solid argument for whole class instruction, with the understanding that future research should examine more intensive instruction using smaller groups of students who do not sufficiently respond to the general intervention. The goal remains to improve fifth and sixth grade students’ math problem solving in order to perform better overall in mathematics, which should have a positive impact on their grades, success in school, graduation rate, and ultimately on their postsecondary outcomes.

Footnotes

Acknowledgements

The authors acknowledge Dr. Marjorie Montague, who initiated the research project described in this paper. The authors are grateful to the students as well as the teachers, administrators, and other school personnel in the Miami-Dade County Public Schools for their cooperation and support.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is supported by Grant No. R324A110009 from the Institute for Education Sciences (IES), U.S. Department of Education. Opinions expressed herein are those of the authors and do not represent the position of the U.S. Department of Education.

Supplemental Material

The online appendix is available at .

References

Berardi-Coletta

Buyer

L. S.

Dominowski

R. L.

Rellinger

E. R.

(1995). Metacognition and problem solving: A process-oriented approach. Journal of Experimental Psychology, 21, 205–223. doi:10.1037/0278-7393.21.1.205

Bottge

B. A.

Cho

S.-J.

(2013). Effects of enhanced anchored instruction on skills aligned to Common Core Math Standards. Learning Disabilities: A Multidisciplinary Journal, 19, 73–83.

Browne

M. W.

Cudeck

(1993). Alternative ways of assessing model fit. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 136–162). Thousand Oaks, CA: Sage.

Bryant

D. P.

Bryant

B. R.

Williams

Kim

Shin

(2013). Instructional practices for improving student outcomes in solving arithmetic combinations. In Chard

Cook

Tankersley

(Eds.), Research-based strategies for improving outcomes in academics (pp. 58–69). Boston: Pearson.

Carter

S. L.

(2010). The social validity manual: A guide to subjective evaluation of behavior interventions in applied behavior analysis. Boston: Elsevier.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Common Core State Standards Initiative. (2014). Development process. Retrieved December 12, 2014 from: http://www.corestandards.org/about-the-standards/development-process/

Enders

C. K.

Tofighi

(2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12, 121–138. doi:10.1037/1082-989x.12.2.121

Fuchs

L. S.

Fuchs

(2002). Mathematical problem-solving profiles of students with mathematics disabilities with and without comorbid reading disabilities. Journal of Learning Disabilities, 35, 564–574. doi:10.1177/00222194020350060701

10.

Fuchs

L. S.

Fuchs

(2007). Mathematical problem solving: Instructional intervention. In Berch

D. B.

Mazzocco

M. M.

(Eds.), Why is math so hard for some children? The nature and origins of mathematical learning difficulties and disabilities (pp. 397–414). Baltimore: Paul H. Brookes Publishing.

11.

Fuchs

L. S.

Fuchs

Schumacher

R. F.

Seethaler

P. M.

(2013). Instructional intervention for students with mathematics learning disabilities. In Swanson

H. L.

Harris

K. R.

Graham

(Eds.), Handbook of learning disabilities (2nd ed., pp. 388–404). New York: Guilford.

12.

Griffin

C. C.

Jitendra

A. K.

(2009). Word problem-solving instruction in inclusive third grade mathematics classrooms. Journal of Special Education, 102, 187–201.

13.

Grouws

(1992). Handbook of research on mathematics teaching and learning. New York: MacMillan.

14.

Hegarty

Kozhevnikov

(1999). Types of visual-spatial representations and mathematical problem solving. Journal of Educational Psychology, 91, 684–689. doi:10.1037//0022-0663.91.4.684

15.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. doi:10.1080/10705519909540118

16.

Jitendra

A. K.

Dupuis

D. N.

Star

J. R.

Rodriguez

M. C.

(2014). The effects of schema-based instruction on the proportional thinking of students with mathematics difficulties with and without reading difficulties. Journal of Learning Disabilities, 87, 18–32. doi:10.1177/0022219414554228

17.

Jitendra

A. K.

Griffin

C. C.

Deatline-Buchman

Sczesniak

(2007). Mathematical word problem solving in third-grade classrooms. The Journal of Educational Research, 100, 283–302. doi:10.3200/joer.100.5.283-302

18.

Jitendra

A. K.

Griffin

C. C.

McGoey

Gardill

M. C.

Bhat

Riley

(1998). Effects of mathematical word problem solving by students at risk or with mild disabilities. The Journal of Educational Research, 91, 345–355. doi:10.1080/00220679809597564

19.

Jitendra

A. K.

Star

J. R.

Starosta

Leh

J. M.

Sood

Caskie

… Mack

T. R.

(2009). Improving seventh grade students’ learning of ratio and proportion: The role of schema-based instruction. Contemporary Educational Psychology, 34, 250–264. doi:10.1016/j.cedpsych.2009.06.001

20.

Jitendra

A. K.

Xin

Y. P.

(1997). Mathematical word-problem-solving instruction for students with mild disabilities and students at risk for math failure: A research synthesis. The Journal of Special Education, 30, 412–4383. doi:10.1177/002246699703000404

21.

Kass

C. E.

Maddux

C. D.

(2005). A human development view of learning disabilities: From theory to practice. Springfield, IL: Charles C. Thomas.

22.

Krawec

(2014). Problem representation and mathematical problem solving of students of varying math ability. Journal of Learning Disabilities, 47, 1030–1115. doi:10.1177/0022219412436976

23.

Krawec

(2015). [Identifying process improvement through error analysis of math word problem solving: The impact of a cognitive strategy intervention.] Unpublished raw data.

24.

Krawec

Warger

(2015). Solve it! Teaching mathematical problem solving in inclusive classrooms–Grades 5-6. Reston, VA: Exceptional Innovations, Inc.

25.

Lemons

C. J.

Fuchs

Gilbert

J. K.

Fuchs

L. S.

(2014). Evidence-based practices in a changing world: Reconsidering the counterfactual in education research. Educational Researcher, 43, 242–252. doi:10.3102/0013189x14539189

26.

Montague

(1992). The effects of cognitive and metacognitive strategy instruction on mathematical problem solving of middle school students with learning disabilities. Journal of Learning Disabilities, 25, 230–248.

27.

Montague

(2003). Solve it! A mathematical problem-solving instructional program. Reston, VA: Exceptional Innovations.

28.

Montague

Applegate

(1993). Mathematical problem-solving characteristics of middle school students with learning disabilities. The Journal of Special Education, 27, 175–201.

29.

Montague

Bos

(1986). The effect of cognitive strategy training on verbal math problem solving performance of learning disabled adolescents. Journal of Learning Disabilities, 19, 26–33.

30.

Montague

Enders

Dietz

(2009). The effects of Solve It! on middle school students’ math problem solving and math self-efficacy. Unpublished manuscript, Department of Teaching and Learning, University of Miami, Coral Gables, Florida.

31.

Montague

Enders

Dietz

(2011). Effects of cognitive strategy instruction on math problem solving of middle school students with learning disabilities. Learning Disabilities Quarterly, 34, 262–272. doi:10.1177/0731948711421762

32.

Montague

Krawec

Enders

Dietz

(2013). The effects of cognitive strategy instruction on math problem solving of seventh-grade students of varying ability. Journal of Educational Psychology. Advance online publication. doi:10.1037/a0035176

33.

Montague

Penfield

R. D.

Enders

Huang

(2010). Curriculum-based measurement of math problem solving: A methodology and rational for establishing equivalence of scores. Journal of School Psychology, 48, 39–52. doi:10.1016/j.jsp.2009.083.002

34.

Montague

Warger

Morgan

(2000). Solve it! Strategy instruction to improve mathematical problem solving. Learning Disabilities Research and Practice, 15, 110–116.

35.

Muthén

L. K.

(2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Author.

36.

Muthén

L. K.

(2014). Mplus 7.3 (Computer program). Los Angeles, CA: Author.

37.

Organization for Economic Cooperation and Development. (2013). PISA 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.1787/9789264190511-en

38.

Owen

R. L.

Fuchs

L. S.

(2002). Mathematical problem-solving strategy instruction for third-grade students with learning disabilities. Remedial & Special Education, 23, 268–278. doi:10.1177/07419325020230050201

39.

Paul

F. G.

(2005). Grouping within Algebra I: A structural sieve with powerful effects for low-income, minority, and immigrant students. Educational Policy, 19, 262–282. doi:10.1177/0895904804274056

40.

Powell

Fuchs

L. S.

(2010). Contribution of equal-sign instruction beyond word-problem tutoring for third-grade students with mathematics difficulty. Journal of Educational Psychology, 10, 381–394. doi:10.1037/a0018447

41.

Pressley

Hilden

(2006). Cognitive strategies. In Kuhn

Siegler

R. S.

Damon

Lerner

R. M.

(Eds.), Handbook of child psychology: Vol. 2. Cognition, perception, and language (6th ed., pp. 511–556). Hoboken, NJ: Wiley.

42.

Swanson

H. L.

Hoskyn

Lee

(1999). Interventions for students with learning disabilities: A meta-analysis of treatment outcomes. New York: Guilford.

43.

Swanson

H. L.

Moran

A. S.

Bocian

Lussier

Zheng

(2013). Generative strategies, working memory, and word problem solving accuracy in children at risk for math disabilities. Learning Disability Quarterly, 36, 203–214. doi:10.1177/0731948712464034

44.

van Garderen

(2006). Spatial visualization, visual imagery, and mathematical problem solving of students with varying abilities. Journal of Learning Disabilities, 39, 496–506. doi:10.1177/00222194060390060201

45.

Xin

Y. P.

Jitendra

A. K.

Deatline-Buchman

(2005). Effects on mathematical word problem-solving instruction on middle school students with learning problems. The Journal of Special Education, 39, 181–192. doi:10.1177/00224669050390030501

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.45 MB