An Intelligent Tutor-Assisted Mathematics Intervention Program for Students With Learning Difficulties

Abstract

The Common Core Mathematics Standards have raised expectations for schools and students in the United States. These standards demand much deeper content knowledge from teachers of mathematics and their students. Given the increasingly diverse student population in today’s classrooms and shortage of qualified special education teachers, computer-assisted instruction may provide supplementary support, in conjunction with the core mathematics instruction, for meeting the needs of students with different learning profiles. The purpose of this study was to explore the potential effects of the Please Go Bring Me-Conceptual Model-Based Problem Solving (PGBM-COMPS) intelligent tutor program on enhancing the multiplicative problem-solving skills of students with learning disabilities or difficulties in mathematics.

Keywords

mathematics problem solving intervention programs learning disabilities at-risk

About 5% to 10% of school-age children have been identified as having mathematics disabilities (Fuchs, Fuchs, & Hollenbeck, 2007), and students whose math performance was ranked at or below the 20th to 35th percentile are often considered at risk for learning disabilities or for having learning difficulties in mathematics (LDM; Bryant et al., 2011; Fuchs et al., 2007). Students with LDM lag behind their peers beginning in early elementary school and continue to fall further behind as they transition from elementary to secondary school. According to National Assessment of Educational Progress (NAEP; 2015) mathematics assessment data, during the past 10 years, score gains were seen at Grades 4 and 8 for higher performing students at the 75th and 90th percentiles, but there were no significant changes over the same period for lower performing students. For instance, according to fourth-grade NAEP mathematic achievement data, the percentage of students without disabilities who performed below the basic level significantly decreased from 17% in 2005 to 14% in 2015. However, this improvement trend is absent for students with disabilities. In fact, the percentage of students with disabilities who performed below the basic level slightly increased from 43% in 2005 to 45% in 2015. It seems that the gap between students with disabilities and their same-age peers has become wider and certainly does not seem to be closing.

In conjunction with this lack of growth in mathematics learning among students with disabilities, expectations for all students, including those with LDM, have been elevated in today’s educational climate. In particular, the Common Core State Standards for Mathematics (CCSSM; Common Core State Standards Initiative [CCSSI], 2012) emphasize conceptual understanding of ideas and the connections between mathematical ideas. The CCSSM also emphasizes that students “model with mathematics.” In particular, the Common Core emphasizes higher order thinking and reasoning as well as algebra readiness throughout elementary mathematics. The challenge of meeting the expectations of the Common Core Standards is compounded by the shortage of teachers certified to teach mathematics (Hutchison, 2012) and more so, the shortage of special education teachers that is “severe, chronic, and pervasive” and presents threats to the quality of educational services provided to these students (Billingsley & McLeskey, 2004, p. 2). As such, there is a need to explore potential intervention support that addresses the new emphasis of the Common Core to facilitate all students’ access to higher order thinking and meeting the standards.

To this end, computer-assisted instruction (CAI) may provide additional support in meeting individual students’ needs in the inclusive classroom (Fuchs & Allinder, 1993; Seo & Bryant, 2012). In particular, when the CAI incorporates effective intervention components as well as instructional design principles, it can serve as an effective delivery system to support regular core classroom instruction (Seo & Bryant, 2012). However, according to a meta-analysis conducted by Seo and Bryant (2009), most of the CAI intervention programs in the field of special education have been driven by behavioral learning theories, which were not in line with recent “pedagogical shifts” toward constructivist-oriented learning theory (Seo & Bryant, 2009, p. 926). Current reform in mathematics education calls for reasoning and communication, and today’s inclusive classrooms promote discourse-oriented instructional approach (Baxter, Woodward, Voorhies, & Wong, 2002). The good news is that students with learning disabilities or difficulties have demonstrated success with thinking behaviors such as asking questions, disagreeing, explaining, suggesting solutions (Berry & Kim, 2008), and reasoning with critical thinking (Xin, Liu, Jones, Tzur, & Si, in press).

During the last decade, CAI programs have incorporated cognitive and metacognitive strategies, on the basis of information processing theories of learning, to promote math problem solving of students with LDM. For instance, Seo and Bryant (2012) developed and evaluated the effectiveness of a CAI program, Math Explorer, with four students with LDM (second or third graders) using a single-subject design. Math Explorer was designed to apply cognitive and meta-cognitive strategies for teaching one-step addition and subtraction word problem solving. The four-step cognitive strategy was as follows: reading (the problem), finding (important information), drawing (a picture), and computing. The metacognitive strategy involved “do” the activity, “ask” whether the task is completed, and “check” for the accuracy. As a result, the participants in this study demonstrated growth in solving addition and subtraction word problems after 5 to 7 weeks of training (20–30 min each day).

Huang, Liu, and Chang (2012) examined the effects of a computer-assisted mathematical problem-solving system that was developed as a tool for remedial education. The design of the system was based on Polya’s (1945) heuristic four-step problem-solving procedure (understanding the problem, devising a plan, carrying out the plan, and looking back). In addition, schematic representation of additive word problem types (Fuson & Willis, 1989) was used as a graphical representation strategy to facilitate students’ problem solving. Huang and colleagues found that 17 second- and third-grade students with LDM, who attended this afterschool CAI program, significantly outperformed the control group in mathematics problem solving. However, in the study by Huang et al., students in the control group did not receive any intervention, while the experimental group received intervention in the afterschool program.

To address the need for more CAI programs in response to recent education reform as well as the new curriculum standards (e.g., the Common Core Standards) that emphasize higher order reasoning and algebra readiness, the authors of this study began a collaborative work that integrates research-based practices from mathematics education and special education. The researchers in this study have developed an intelligent tutor, “Please Go Bring Me-Conceptual Model-Based Problem Solving” (PGBM-COMPS), designed to promote multiplicative reasoning and problem solving of elementary students with LDM. Multiplicative reasoning is one of the most important concepts children develop progressively throughout math education years (Harel & Confrey, 1994).

Conceptual Framework

The PGBM-COMPS intelligent tutor draws on three research-based frameworks: a constructivist view of learning from mathematics education (Steffe & D’Ambrosio, 1995), data (or statistical) learning from computer sciences (Sebastiani, 2002), and Conceptual Model-Based Problem Solving (COMPS; Xin, 2012) that generalizes word-problem underlying structures from special education. In particular, the PGBM-COMPS tutor is made of two parts: (a) “Please Go Bring Me . . . ” (PGBM) turn-taking games designed to nurture a learner’s construction of fundamental ideas in multiplicative reasoning and (b) COMPS that emphasizes understanding and representation of word-problem structures in mathematical model equations.

A basic version of the PGBM platform game involves sending a student to a box with Unifix Cubes to produce and bring back a tower made of a few cubes. After taking two to nine “trips” for bringing same-sized towers, students are asked how many towers (i.e., composite units; CU) they brought, how many cubes each tower has (i.e., unit rate; UR), and how many cubes (1s) there are in all. The PGBM game was devised to promote learners’ anticipated creation of and differentiation among 1s and CU (Tzur et al., 2013). These two anticipations are crucial if the learner is to construct the mental operation of multiplicative double counting (mDC; see Steffe & Cobb, 1988), which is fundamental to multiplicative reasoning (Vergnaud, 1983). The mDC integrates two counting sequences in a multiplicative problem situation (e.g., “Please bring me a tower with six cubes in each . . . If you brought me five such towers; how many cubes in all?”): one sequence that quantifies how many CUs (i.e., towers) were produced and one sequence that monitors the corresponding accumulation of 1s (i.e., total number of cubes) contained within those CUs (i.e., towers). Double counting is considered to be “an advance over the more basic direct representation because it requires more abstract processing” (Kouba, 1989, p. 152). A variety of activities following a PGBM format were designed to promote students’ construction of basic multiplicative concepts on the basis of continuous assessment of their existing knowledge and experiences. The learner will progress from a low to high level of tasks along the dimensions of (a) numerical numbers (e.g., 2, 5, or 10—Level 1; 3 or 4—Level 2; and 6, 7, 8, or 9—Level 3) involved in the problem and (b) cognitive demands of the task (i.e., operating with visible objects or invisible/covered objects with mental system).

However, COMPS generalizes the understanding of multiplicative reasoning to the level of mathematical models. At this stage, students no longer rely on “concrete” models (such as cubes and towers) or drawing pictures or tally marks; the mathematical models directly drive the solution plan. The COMPS program emphasizes (a) the connection between the PGBM games (in the contexts of cubes and towers, for instance) and the symbolic mathematical model equations, (b) students’ representation of various multiplicative problem situations in the mathematical model equations, and (c) development of the solution plan that is directly driven by the model equations. COMPS supports students’ thinking process as they advance from concrete and figurative representations (e.g., cubes or fingers) to abstract mathematical model equations that depict the mathematical relationships within the problems (see Figure 1 in the “Method” section under “The PGBM-COMPS Condition”). This unique feature is in line with the Common Core Standards (CCSSI, 2012), which emphasize mathematical modeling for abstract level of conceptual understanding in particular.

Figure 1.

Sample screenshots of the PGBM-COMPS intelligent tutor system.

The purpose of this study was to explore the potential effects of the PGBM-COMPS intelligent tutor program, as a supplementary intervention program, on enhancing the multiplicative reasoning and problem-solving skills of students with LDM. The specific research question is as follows:

Research Question 1: What are the effects of PGBM-COMPS intelligent tutor-assisted intervention program, in reference to traditional teacher-delivered intervention (TDI), on participating students’ multiplicative problem-solving performance measured by a researcher-developed criterion test and a norm-referenced standardized measure?

Method

Participants and Setting

This study was conducted within the larger context of the National Science Foundation-funded, Nurturing Multiplicative Reasoning in Students With Learning Disabilities/Difficulties project (NMRSD; Xin, Tzur, & Si, 2008). Participants included 17 elementary students with LDM from one elementary school in the Midwestern United States. This elementary school is part of an urban school corporation that serves 7,616 students, with 62% qualifying for free or reduced lunch, 13% in English as a New Language (ENL) programs, and 19% receiving special education services.

Participant selection was based on (a) school identification of students experiencing substantial problems in mathematics word-problem solving and (b) scores below the 35th percentile on the Mathematics Problem Solving subtest of the Stanford Achievement Test (SAT-10; Harcourt Assessment, 2004). A total of 23 students were recommended by the school to participate in this study and institutional review board (IRB) consents were obtained from the parents as well as assents from the students. However, six students were not included in the study because their pre-assessment scores were above 60% correct on the criterion tests. Table 1 presents the demographic information of the participants who completed the PGBM-COMPS program or the TDI program as part of their school’s afterschool programs. The average age of the students in the PGBM-COMPS group was 9.8 years and the average age of the students in the TDI group was 9.8 years as well.

Table 1.

Participating Student Demographics by Condition.

Variable	PGBM-COMPS	TDI
Gender
Male	6	4
Female	3	4
Grade
3	5	3
4	4	5
M age in months (SD)	117.89 (12.19)	117.50 (7.76)
Race
EA	4	2
AA	1	2
Hispanic	3	2
Multiracial	1	2
Classification
LD	3	1
ADHD	1	1
NL	1	5
ENL	3	1
Mild intellectual disability	1	0
Intelligence
M	78	83
Range	70–87	67–94
Test used	OLSAT	OLSAT
Math problem solving
M	27	31
SD	9.92	15.37
Test used	SAT-10	SAT-10
Geographic region local (urban, rural)	Midwest urban	Midwest urban

Note. PGBM-COMPS = Please Go Bring Me-Conceptual Model-Based Problem Solving; TDI = teacher-delivered intervention; EA = European American; AA = African American; LD = learning disabilities; ADHD = attention deficit hyperactivity disorder; NL = not labeled; ENL = English as a New Language; OLSAT = The Otis–Lennon School Ability Test; SAT = Stanford Achievement Test (SAT-10; normal curve equivalent scores).

Design

A pretest–posttest comparison group design with random assignment of participants to groups was used to examine the potential effect of the PGBM-COMPS tutor program in reference to the TDI program. In addition to pre- and post-tests, both groups also took two maintenance tests scheduled 1 and 2 weeks following the termination of the intervention. To determine an appropriate sample size for this study, the researchers conducted a power analysis using an alpha level of .05 and an effect size of 1.25 based on existing related research studies (e.g., Xin, Zhang, et al., 2011), which indicated that a minimum of eight or nine participants in each group was sufficient to obtain a power of .87 to .91 for a 2 × 4 or 2 × 3 repeated-measures ANOVA (Friendly, 2000). All the students (n = 17) who met the selection criterion (see the “Participants” section above) were randomly assigned (through the flipping of a coin) to one of the two comparison conditions, PGBM-COMPS or TDI.

Dependent Measures

We used a researcher-developed, 11-item criterion test (Purdue Research Foundation, 2011) to assess students’ multiplicative concepts and problem solving. Table 2 presents sample items in the criterion test that assesses the concepts of double counting, same unit coordination, mixed unit coordination (with a focus on the differentiation among 1s and CU; Tzur et al., 2013), and quotitive and partitive division. As part of the NMRSD project (Xin et al., 2008), we developed the criterion test based on literature in mathematics education as well as feedback/input from researchers/educators (i.e., experts) in math education (e.g., Guershon Harel, Leslie Steffe, and Patrick Thompson) and special education (e.g., Anne Foegen, Paula Maccini, and John Woodward). In particular, we provided these experts with all the test items and the conception we assumed to be assessed via each of the test item/task. We asked the experts (via email survey) the following: “Please tell us if you think the task (item) should be maintained in the instrument, and suggest changes to the task as you see proper (wording, numbers, contexts, etc.).” In addition, we encouraged the experts to “suggest additional tasks for consideration.” We finalized this criterion test in 2011; we then established the reliability of the test by assessing a group of third and fourth graders (non-participants of this study) from the same school. The Pearson’s correlation coefficient for the test–retest reliability of the criterion test was .89 and the internal consistency (split-half reliability) of the criterion test was .86.

Table 2.

Sample Tasks Included in the Criterion Test.

Problem type	Sample problems situations
Multiplicative double counting	Pretend that you have made many towers, each made of 7 cubes. How many cubes are in every tower? Your Answer: _________ How many cubes are in the first 4 towers? Your Answer: _____ So, we can count those by seven, “7, 14, 21, 28 . . .” Do you think you will say the number 70 if you continue counting cubes in the towers? Your Answer: _____________ Why? _____________________________________________ Do you think you will say the number 84 if you continue counting cubes in the towers? Your Answer: _____________ Why? _____________________________________________
Same unit coordination	Rachael has built 13 towers with 2 cubes in each. Mary has built 7 towers with 4 cubes in each. Who has more towers, Rachael or Mary? Your Answer: ______ How many more towers does she have?
Unit differentiation and selection	Tom’s father bought 6 pizzas. Each pizza had 4 slices. Tom’s mother bought a few more pizzas. Then, there were 9 pizzas. How many more slices did Tom’s mother bring?
Mixed unit coordination	Maria made birthday bags. She wants each bag to have 6 candies. After making 3 bags, she still had 12 candies left. How many bags will she have altogether after putting these 12 candies in bags?
Quotitive division	There are 28 students in Ms. Franklin’s class. During reading, she puts all students in groups of 4. She asked a student (Steve): “How many groups will I make?” Steve said: “32. Because 28+4 is 32.” Do you think that Steve is correct? Your Answer: __________ Why? _____________________________________________
Partitive division	After an art class, there were 78 crayons out on the tables. There are 6 boxes for the crayons. Ms. Brown puts the same number of crayons in each box. How many crayons would she put in each box?

In addition to the criterion test, we used the Mathematics Problem Solving subtest of SAT-10 (Harcourt Assessment, 2004) as a standardized measure for identifying students who were struggling with mathematics problem solving, as well as a transfer measure for evaluating the impact of the interventions. The SAT-10 is a norm-referenced and criterion-referenced, standardized achievement test with established reliability and validity. All items are presented in a multiple-choice format. Per the technical manual of the test, its internal consistency reliability for Primary 3 (Grade 3, spring) and Intermediate 1 (Grade 4, spring) for the Mathematics Problem Solving subtest were .91 and .90. Alternate form reliabilities for Primary 3 and Intermediate 1 for the Mathematics Problem Solving subtest were .85 and .74, respectively (Harcourt Assessment, 2004).

Scoring

For the criterion test, each correctly solved problem was awarded one point. When a problem involved a set of sub-questions, the points were evenly distributed to each of the sub-questions. For those problems that include a follow-up “why” question, the scoring was based on the analysis of students’ written answer. For instance, for the first problem listed in Table 2, “ . . . Do you think you will say the number 70 if you continue counting cubes in the towers?” If a student answered “Yes” and the student wrote “7 × 10 = 70,” for instance, to answer the follow-up “why” question, full credit was given (for answering “Yes” and for his or her follow-up correct reasoning). If a student put “I do not know” or other incorrect reasoning, no credit was given for the follow-up portion of the question. The scoring for the SAT was consistent with the administration and scoring procedure specified by the SAT-10 (Harcourt Assessment, 2004). Two research assistants independently scored all tests using the answer keys. Then, they computed interrater reliability for 34% of the scored tests by dividing the number of agreements by the total number of agreements and disagreements and multiplying by 100, which resulted in a median interrater reliability of 100% (range = 91%–100%).

Procedure

Across both the PGBM-COMPS and TDI conditions, students worked on similar tasks pertinent to multiplication and division word problems. Students worked with either the teacher (the TDI condition) or the intelligent tutor (the PGBM-COMPS condition) 4 times a week with each session lasting about 25 min over a total of 36 sessions. It should be noted that students in both groups were allowed to use calculators during the intervention and assessment conditions.

The PGBM-COMPS Condition

Each participant worked with the web-based computer program “one-on-one” during the same time slot; sessions were monitored by two supervisors (trained graduate research assistants or participating school personnel). Supervisors’ roles included administering pre-post assessment, fixing/recording the computer program’s “bugs” when necessary, and redirecting students to appropriate parts of the program after any unexpected interruptions in students’ work due to technical difficulties with the software.

“Prior to working one-on-one” with the intelligent tutor, the participating students engaged in the PGBM game (please refer to the PGBM game described in the “Conceptual Framework” section), conducted by research assistants outside of the computer system, with a purpose of getting participants familiar with the rules for this game, so that later they could be more easily adapted it to a situated game in the tutoring system. The PGBM-COMPS tutoring program includes four modules (A, B, C, and D). Each participant in the PGBM-COMPS condition went through all four modules in sequence. Module A focuses on mDC. When working with mDC tasks (e.g., PGBM seven towers with three cubes in each [i.e., 7T₃], how many cubes in all?), students were expected to explicitly keep track of two quantities (i.e., number of towers and total number of cubes) while counting two number sequences to find the answer to the total number of cubes. In Module A, students were also engaged in solving the same unit coordination tasks (see Table 2, for sample problems), which are designed to help students make the distinction between the unit of one and the CU with a focus on the CU. Module B involves tasks designed to further develop skills in unit differentiation and selection (UDS) and multiplicative mixed unit coordination (MUC). UDS tasks (e.g., 7T₃ + 6 cubes; how many cubes in all) and MUC tasks (6T₃ + 12 cubes; how many towers of 3 in all) develop students’ sense of which unit they are working on, whether it is the number of cubes (the 1s) or number of towers (the CU).

Module C presents quotitive division tasks (e.g., “There are a total of 28 cubes; if you put them in towers of 7 cubes in each, how many towers can you make?”) where students solve the problems either through mDC or segmenting cubes into equal-sized groups for solution. Module D deals with partitive division problems (e.g., “There are a total of 28 cubes; if you put them in 4 equal sized towers, how many cubes will be in each tower?”). The program presents concrete models that demonstrate the concept of equally distributing 1s to the given number of CUs for the solution. In all these modules, concrete modeling (with the cubes/towers context, for instance) was always connected to mathematical symbols and expressions to facilitate concept–symbol coordination.

At the end of each of the PGBM parts of the modules (except for Module B), the COMPS component facilitates students’ transition from “concrete” (e.g., “cubes” and “towers” in the computer program) and figurative (e.g., “fingers”) to symbolic abstract mathematical modeling for solutions. Specifically, students were engaged in representing various real-world problem situations symbolically in the COMPS model equation (e.g., UR × Number of Units = Product; Xin, 2012), and then solving the problem by finding the unknown value in the equation. Figure 1 presents sample screen shots from the PGBM-COMPS program. The upper panel shows how the program engages students in making the connection between the “concrete” modeling (“cubes” and “towers”) and the mathematical expression; the lower panel shows how the problem should be represented in the COMPS model for finding a solution.

Across the PGBM and the COMPS components, the intelligent tutor system provided feedback and corresponding scaffolding based on continuous assessment of the students’ performance. Prior to providing specific instruction of mathematical conventions (e.g., showing students how to apply algebraic model equation for finding a solution), the PGBM-COMPS program consists of general or indirect hints in conjunction with mathematical modeling experiences (e.g., PGBM situations) to facilitate students’ construction of knowledge of multiplicative reasoning; such constructive progression serves to involve the learner in the process of concept development. Table 3 presents a sample of progressive scaffolding schedule for promoting the understanding of unit segmentation that leads to the concept development of quotitive division.

Table 3.

Level of Scaffolding for Concept Construction at the Concrete Level of Operation.

Task: I have 28 cubes under the cover. How many towers of 7 can I make of these before I run out of cubes?
Level of prompting	Correct response	Incorrect response	Incorrect response	Incorrect response	Incorrect response	Incorrect response
Prompt Level 1		Please make a tower (T) of 7; now do you know the answer?
Prompt Level 2			Please make another T of 7; now do you know the answer?
Prompt Level 3				Please make another T of 7; now do you know the answer?
Prompt Level 4					Please make another T of 7, how many towers of 7 you have built from the 28 total cubes?
Correct answer: Seven towers	Student generated correct answer	Student generated correct answer	Student generated correct answer	Student generated correct answer	Student generated correct answer	The system gives the correct answer by showing the number of Ts built

The TDI Condition

The instructors of this group were two licensed schoolteachers who taught third or fourth grade, general education math classes (one with 10 years of teaching experiences, one with 1 year of teaching experience); they worked together as a team in teaching one group. The instructors of the TDI condition used similar word problem tasks to the PGBM-COMPS condition. However, teachers in the TDI condition implemented the instructional strategies/practice that were consistent with what they would do in teaching their regularly scheduled math class periods, which were consistent with the mathematics curricula adopted by the school district. The mathematics textbooks used by the math teachers in the participating school included enVisionMath: Common Core Edition (Charles et al., 2012), Math in Focus: The Singapore Approach (Ramakrishnan & Soon, 2009), and Harcourt Math (Maletsky et al., 2004).

A survey was conducted to document the strategies the teachers used during the TDI. Table 4 presents survey questions that the instructors of the TDI condition were asked to complete immediately following the intervention. The strategies these teachers used in teaching this group included (a) drawing pictures to represent the problem situation (e.g., equal groups of objects described in the problem), (b) using repeated addition or subtraction to teach the concept of multiplication or division, (c) using “guess and check” in making decisions about the operation or the missing factor for solution, and/or (d) using multiplication or division directly.

Table 4.

Teacher’s Survey.

Survey questions
1. What strategies did you use to teach your students (in the research group) in solving multiplication and division problems? Please explain and show how you would teach your students to solve each of the problems listed below (if needed you can draw pictures or make tables or present whatever you used to teach your students)It takes 32 oranges to make one gallon of orange juice. How many oranges would you need to make 15 gallons of orange juice?Edwin received a total of US$374 to buy basket balls for the football team. Each basketball costs US$34. How many basket balls can he buy?There are 126 spring rolls to be placed on 42 platters. How many spring rolls will be on each platter?
2. If you are a regular math classroom teacher and you teach math, what strategies you use to teach your regular classroom students in solving these problems?(if you used similar strategies, please indicate so, and you do not have to repeat the same strategies here. However, if different, then please describe the strategies you used in teaching your regular math class.)It takes 32 oranges to make one gallon of orange juice. How many oranges would you need to make 15 gallons of orange juice?Edwin received a total of US$374 to buy basket balls for the football team. Each basketball costs US$34. How many basket balls can he buy?There are 126 spring rolls to be placed on 42 platters. How many spring rolls will be on each platter?
3. What math textbooks and/or programs the school currently use in teaching third-grade and fourth-grade math?Third grade:Fourth grade:
4. During this past spring semester, what was taught in the regular math classes that involve the students in your research group? Please list the goals and objectives of the chapters or lessons (You can make a photo copy of the table of contents of the book and check off those lessons that were taught during this spring semester).For third graders in your group?For fourth graders in your group?

Treatment Fidelity

For the PGBM-COMPS condition, the intervention for each of the students was prescribed by the intelligent tutor, and the student had to follow the sequence of the four modules as described for the PGBM-COMPS condition. The session supervisors ensured students’ completion of all four modules of the PGBM-COMPS program. In contrast, the enacted curriculum/intervention portrayed by the two schoolteachers through survey/interview defines the TDI condition. Throughout the TDI intervention, the researchers interviewed the teachers about their teaching strategies. These teachers discussed their teaching strategies with the researchers often diagramming on paper how they supported students’ learning during the TDI condition. The teaching strategies reported during these interviews were consistent with the strategies these teachers reported on the surveys.

Results

Table 5 presents the mean and standard deviation (SD) of two groups of students’ performance on the criterion test before and after the intervention. We used an independent samples t test to examine pre-intervention performance on the criterion test. The results indicated no statistically significant difference between the two groups (PGBM-COMPS and TDI) on the criterion test, t = −1.77, p = .10.

Table 5.

Means and Standard Deviations by Condition With Effect Sizes.

Test (raw scores)	M		SD		N		ES^a
Test (raw scores)	PGBM-COMPS	TDI	PGBM-COMPS	TDI	PGBM-COMPS	TDI	ES^a
MR pretest	1.94	3.80	1.64	2.62	9	8	−0.85
MR Posttest 1	8.97	7.00*	0.95	1.96	9	8	+1.28
MR Posttest 2	8.93**	7.56**	1.15	1.58	9	8	+0.99
MR Posttest 3	8.89**	6.94*	0.96	1.98	9	8	+1.25
SAT pretest	18.78	21.33	3.93	8.80	9	6	−0.37
SAT post	29.00**	23.67	6.32	9.67	9	6	+0.65

Note. PGBM-COMPS = Please Go Bring Me-Conceptual Model-Based Problem Solving; TDI = teacher-delivered intervention; MR = Multiplicative Reasoning; SAT = Stanford Achievement Test; ES = effect size.

Effect size was calculated as the two conditions’ mean difference divided by the pooled standard deviation (Hedges & Olkin, 1985). A positive ES indicates a favorable effect for the PGBM-COMPS condition; a negative ES indicates a favorable effect for the TDI condition.

p < .05; **p < .01.

Acquisition and Maintenance Effects of the PGBM-COMPS Tutoring Intervention

To assess the effect of the PGBM-COMPS intelligent tutor, in reference to the TDI, on students’ multiplicative problem solving, we conducted an ANOVA (2 Groups × 4 Times of Testing) with repeated measures on time (pretest, posttest, and two follow-up tests) on the criterion test. The one-way ANOVA is considered a robust test against the normality assumption (Laerd Statistics, 2013a). The results indicated that there was a statistically significant effect of time (F = 41.62, p < .001), which indicated that both groups of students improved their performance following the intervention, respectively. More importantly, there was a statistically significant Group × Time Interaction effect (F = 5.36, p = .013). That is, although both groups improved their performance from pre- to post-test, the improvement rate of the students in the PGBM-COMPS group was much greater than that of the students in the TDI group. Figure 2 presents the two groups’ performance on the criterion measure during pretest, posttest, and two additional follow-up tests. As shown in Figure 2, the PGBM-COMPS group surpassed the TDI group following the intervention. Using pretest to immediate posttest gain score as the measure, the effect size (Cohen’s d) was 1.99, favored the PGBM-COMPS condition.

Figure 2.

Performance of the two groups on the criterion test before (Time = 1) and after the intervention (Time = 2, 3, and 4).

In addition, post hoc analyses of the repeated-measure ANOVA (2 Groups × 3 Times of Post-Testing, Posttests 1, 2, and 3) indicated a significant difference between the two groups’ post-intervention tests, favoring the PGBM-COMPS condition (F = 6.87, p = .019). Furthermore, a non-significant effect on time (from Posttest 1 to Posttest 3, F = 1.908, p = .185) indicated that both groups of students maintained their posttest performance. As shown in Figure 2, the two groups’ performance on the criterion test during three posttests was relatively stable.

Potential Transfer Effect on the Standardized Measure

We conducted an ANOVA (2 Groups × 2 Times of Testing) with repeated measures on time (pretest and posttest) on SAT raw scores. As only six (out of eight) students in the TDI condition completed SAT tests both before and after the intervention, this part of the analyses included nine students in the PGBM-COMPS condition and six students in the TDI condition. The results indicated a significant main effect for time, F = 11.921, p = .004, and no significant main effect for group, F = 0.183, p = .676. In addition, the results indicated a significant interaction effect between group and time of testing, F = 4.706, p = .049. That is, the PGBM-COMPS group improved relatively more than the TDI group from pre- to post-test on the transfer measure.

Further post hoc analyses by paired-samples tests indicated that the PGBM-COMPS group significantly improved its performance from pre- to post-test (mean difference = 10.222, t = 3.707, p = .006). In contrast, the TDI group did not significantly improve its performance from pre- to post-test (mean difference = 2.333, t = 1.513, p = .191). Using pre- to post-intervention gain score as the measure, the effect size between the two groups was 1.23, favoring the PGBM-COMPS group.

Discussion

The purpose of this study was to explore the impact of an intelligent tutor-assisted intervention program (PGBM-COMPS), as a supplementary intervention program, on enhancing the multiplicative reasoning and problem-solving skills of students with LDM. Overall, the findings seem to support the potential for using intelligent tutor-assisted instructional program to address the knowledge gap of the students with LDM and enhance their problem-solving performance. It is encouraging that students in the PGBM-COMPS group not only outperformed the TDI group on the criterion test, but more importantly, they also showed significant improvement on the Problem Solving subtest of a norm-referenced standardized assessment, SAT-10, following the intelligent tutor-assisted intervention.

Distinctive Features of the Two Intervention Programs

To better understand the findings from this study, we will discuss and elaborate on multiple distinctive features of the PGBM-COMPS intelligent tutor program and the teacher-delivered instruction (TDI).

PGBM-COMPS intelligent tutor

First, the program has a focus on nurturing conceptual understanding of concepts that are essential to enabling these students to access mathematics problem solving in a meaningful way (Tzur, Xin, Si, Kenney, & Guebert, 2010). In particular, the program engaged students in the construction of mathematical ideas (e.g., a number as an abstract CU; mDC) through heuristic prompting, such that the reasoning behind the mathematics was made explicit to students.

Second, the PGBM-COMPS program was designed to emphasize a mathematical-model-based approach for promoting students’ generalized problem-solving skills. Modeling involves translation or representation of a real problem situation into a mathematical expression or model. Mathematical models are an essential part of all areas of mathematics, including arithmetic; researchers have recommended introducing models to all age groups, including elementary students (Mevarech & Kramarski, 2008). It should be noted that engaging students in the modeling process does not necessarily mean engaging students in the discovery or invention of mathematical models or complex notational systems; however, according to Lesh, Doerr, Carmona, and Hjalmarson (2003), it does mean that when such models or systems are given to the students, they likely have better opportunities to unpack the meaning of the system, represent the problem situation in a mathematical expression or model, and flexibly use the model to solve real-world problems. The PGBM-COMPS was designed with the intention of making the mathematical models explicit to students with LDM and was programmed to facilitate students’ understanding/reconstruction of the models and, therefore, strategic use of the models for generalized problem solving.

The PGBM-COMPS program incorporates instructional components that have been demonstrated to be effective for students with LDM. These include use of heuristics to facilitate students’ construction/representation of the mathematics relations as well as explicit instruction of mathematical models (Gersten et al., 2009; National Mathematics Advisory Panel, 2008). Also, to promote skill transfer, the system introduced a range of problem situations to the student to help him or her recognize an invariant problem structure under very different cover stories, which was an application of using a “sequence and/or range of examples” (Gersten et al., 2009, p. 1219). Furthermore, the PGBM-COMPS program utilized the computer’s flexibility in simulating concrete (e.g., unfix cubes) and figurative (e.g., fingers and pictures) levels of representation to help students construct a symbolic/abstract level of representation (mathematical models), which was an application of using visual representations (Gersten et al., 2009).

TDI

One of the distinctive features of the TDI was its focus on the choice of operation when dealing with problem solving. Excerpt 1 below, from surveying the participating schoolteachers who taught the TDI condition, reflected this focus on the choice of operation.

Excerpt 1 (from Teacher 1, May 29):

Each problem that I went through with the children I began by having a student read aloud the problem. From that point on, we always had a conversation about whether or not we should be using multiplication or division.

Other strategies applied by the TDI condition included drawing a picture and using repeated addition and subtraction. See examples in Excerpts 2 and 3 below.

Excerpt 2 (from Teacher 1, May 29):

For this type of problem, I would sometimes draw a picture similar to this one [see Figure 3] so that students could visually see and understand whether or not we were doing multiplication or division. Since many students know that multiplication is somewhat like repeated addition, a picture likes this one helped them to see this. Once I began to show students a problem using a picture like the one below, they saw they did in fact need to multiply.

Excerpt 3 (from Teacher 2, June 4):

Task: Edwin received a total of $374 to buy basketballs for the basketball team. Each basketball costs $34. How many basketballs can he buy?

We talked about what they were asking and how we could find out how many basketballs there would be. Then, we figured out we could do repeated addition until we got to 374. We knew we couldn’t go over it. We also decided we could do repeated subtraction until we didn’t have any left. We then would count up the number of sets of 34 that were either added or subtracted depending on the strategy they used.

Figure 3.

“Draw a picture” to solve the problem.

In addition, “Guess and Check” was also a strategy used by the TDI condition. Excerpt 4 supported the use of this strategy:

Excerpt 4 (from Teacher 1, May 29):

Since this problem [i.e., the same problem as in Excerpt 3] needed to use division that involved two-digits, it posed quite a challenge for my fourth graders. We did a lot of guessing and checking [emphasis added] as we worked through a division problem of this sort . . .

Another thing I try to stress to them is whether or not our numbers for our answer are going to be increasing (multiplication) or decreasing (division).

Differences in Problem-Solving Strategies During the Posttests

The unique features of the PGBM-COMPS program, in particular, its mathematical model-based problem solving that was built on students’ understanding of fundamental mathematical ideas (e.g., a number as an abstract CU, mDC), might have contributed to better acquisition and transfer effect as shown in this study. Figure 4 (left panel) presents an example of the problem-solving process exhibited on the immediate posttest by a third-grade participant in the PGBM-COMPS group. It seems that she used the model equation to solve the quotitive division problem and/or checked her answer.

Figure 4.

Sample student work during the posttest by participants in the PGBM-COMPS group (left panel) or the TDI group (right panel).

In contrast, the strategies of repeated addition/subtraction, drawing pictures, or “guess and check” may easily become cumbersome and error-prone. Although repeated addition and subtraction may help students in understanding the concept of multiplication and division in a concrete way, Schwartz (1988) expressed concern with an overly simple perception of multiplication as “repeated addition” and highlighted the importance of modeling in teaching and learning mathematics to promote conceptual understanding of the mathematical relationships within problems. Figure 4 (right panel) presents the problem-solving process of a third-grade participant in the TDI group, who used repeated subtraction to find the answer to a quotitive division problem on the immediate posttest. She either started with an incorrect number or left out the first four she subtracted mentally perhaps, and, therefore, reached an incorrect answer. It should be noted that when the numbers in the problem are small, as in the case of this problem, it might be manageable to solve the problem using such strategies. However, when the numbers become large, this problem-solving process is often not effective and/or efficient.

The strategies taught in the TDI group reflected the emphasis of the curriculum adopted by the participating school. Examining the third-grade textbook (i.e., enVisionMath: Common Core Edition), when teaching about the meaning of multiplication, “Multiplication as Repeated Addition” was listed on the top, followed by “Arrays and Multiplication” and “Commutative Property” (Charles et al., 2012, p. vi). Similarly, when teaching the concept of division, “Division as Sharing” and “Division as Repeated Subtraction” were listed as the subheadings for the topic (Charles et al., 2012, p. ii). These were followed by “Finding Missing Numbers in a Multiplication Table,” which promotes using multiplication to solve division problems—one of the unique features commonly found in math textbooks in the United States (Xin, Liu, & Zheng, 2011). Again, in the “Problem Solving” section, repeated addition was promoted to solve for quotitive and partitive division problems by using objects or “drawing a picture” (Charles et al., 2012, p. 182). The textbook does not really make the distinction between quotitive and partitive division problems or the concept of segmentation (into equal-sized groups) and equal distribution.

Limitations and Future Research

One limitation of this study was the limited number of participants involved in this study, which is a common challenge when conducting randomized-control trial (RCT) studies involving students with learning disabilities or students at-risk for being identified for special education services in one school, due to the fact that only about 5% to 10% of school-age children are usually identified as having mathematics disabilities (Fuchs et al., 2007). Although the repeated-measures ANOVA used in this study should lead to an increase in the power of the test to detect significant differences (Laerd Statistics, 2013b), future studies with larger sample sizes will enhance the external validity of the PGBM-COMPS program. Conducting studies with large samples size across multiple schools will further test the feasibility of implementing this web-based, intelligent tutor-assisted intervention program in the inclusive classroom settings (during and/or after the school hours) and its impact on students’ learning.

Practical Implications

As the Common Core Standards are adopted and integrated into curricula and state assessments, schools and teachers are expected to be responsible for all students, including those with learning disabilities or difficulties, meeting the standards and achieve success in mathematics. To date, however, the problem of how to improve mathematical achievements of students with LDM is far from being settled, as shown in the most recent Nation’s Report Card (NAEP, 2015). Given the advancement and increase in use of technology in all aspects of our lives, CAI tools have been, and will be, playing more important roles in education, including providing support to students with diverse needs as well as their teachers. Intelligent tutor systems, such as the one examined in this study (i.e., PGBM-COMPS), developed on the basis of effective instructional practices from mathematics education and special education, are likely to provide educators with a research-based tool for supporting their teaching of students with LDM as they make key adaptations to their instructional methods to align with the demands of the Common Core Standards.

It should be noted that when using this high-tech intelligent tutor, the main role of the teacher is a problem solver and facilitator of learning. Just like in a typical reform-oriented classroom, where children carry out much of the small-group activities independently of teacher input, the PGBM-COMPS program will be characterized by much independent exploration by students. To this end, the PGBM-COMPS program has an embedded “alert” system that will get the human teacher’s attention under specified conditions so that the teacher will be able to support student learning when needed. To help teachers understand the PGBM-COMPS program and therefore better use of this program, we have also developed a teacher’s manual to accompany the PGBM-COMPS program. This manual provides guidance for using the PGBM-COMPS program, which includes the theoretical framework and the concept map of the program as well as tasks involved in each of the modules.

Conclusion

The PGBM-COMPS intelligent tutor program was designed to foster children’s multiplicative reasoning, a central theme in mathematics for Grades 3 to 5 (CCSSI, 2012). The PGBM-COMPS intelligent tutor promotes students’ learning that is consistent with the emphasis of the Common Core Standards (CCSSI, 2012). Promoting these highly linked conceptual and procedural understandings in all children is vital not only for multiplication and division but also for constituting a foundation for fractional, proportional, and algebraic reasoning (Harel & Confrey, 1994).

Given that the CCSSM (CCSSI, 2012) demands much deeper content knowledge from teachers of mathematics, the preliminary findings from this study are encouraging. The PGBM-COMPS intelligent tutor, which integrates the best practices from general mathematics education and special education, seems to yield better outcomes in enhancing participating students’ multiplicative problem solving. Through the integration of heuristic instruction (that facilitates concept construction) and the explicit model-based problem-solving instruction, it seems that the PGBM-COMPS programs have demonstrated the potential to promote generalized problem-solving skills of students with LDM.

Footnotes

Acknowledgements

The authors would like to thank the administrators, teachers, and staff at Lafayette School Cooperation.

Authors’ Note

The opinions expressed do not necessarily reflect the views of the Foundation.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Science Foundation, under Grant DRL 0822296.

References

Baxter

Woodward

Voorhies

Wong

(2002). We talk about it, but do they get it? Learning Disabilities Research & Practice, 17, 173–185. doi:10.1111/1540-5826.00043

Berry

R. A. W.

Kim

(2008). Exploring teacher talk during mathematics instruction in an inclusion classroom. The Journal of Educational Research, 101, 363–377. doi:10.3200/JOER.101.6.363-378

Billingsley

S. B.

McLeskey

(2004). Critical issues in special education teacher supply and demand: Overview. The Journal of Special Education, 38, 2–4. doi:10.1177/00224669040380010101

Bryant

D. P.

Bryant

B. R.

Roberts

Vaughn

Pfannenstiel

K. H.

Porterfield

Gersten

(2011). Early numeracy intervention program for first-grade students with mathematics difficulties. Exceptional Children, 78, 7–23. doi:10.1177/001440291107800101

Charles

R. I.

Caldwell

J. H.

Cavanagh

Copley

Crown

Fennell

… Tate

(2012). enVisionMath: Common core edition. Glenview, IL: Pearson Education.

Common Core State Standards Initiative. (2012). Introduction: Standards for mathematical practice. Retrieved from http://www.corestandards.org/the-standards/mathematics

Friendly

(2000). Power analysis for ANOVA designs. Retrieved from http://davidmlane.com/hyperstat/power.html

Fuchs

L. S.

Allinder

R. M.

(1993). Computer applications in the schools for students with mild disabilities: Computer-assisted instruction and computer-managed instruction. In Gable

R. A.

Warren

S. F.

(Eds.), Advances in mental retardation and developmental disabilities (pp. 49–70). Philadelphia, PA: Jessica Kingsley.

Fuchs

L. S.

Fuchs

Hollenbeck

K. N.

(2007). Expanding responsiveness to intervention to mathematics at first and third grade. Learning Disabilities Research & Practice, 22, 13–24. doi:10.1111/j.1540-5826.2007.00227.x

10.

Fuson

K. C.

Willis

G. B.

(1989). Second graders’ use of schematic drawings in solving addition and subtraction word problems. Journal of Educational Psychology, 81, 514–520. doi:10.1037/0022-0663.81.4.514

11.

Gersten

Chard

D. J.

Jayanthi

Baker

S. K.

Morthy

Flojo

(2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional component. Review of Educational Research, 79, 1202–1242. doi:10.3102/0034654309334431

12.

Harcourt Assessment. (2004). Stanford Achievement Test Series: Tenth edition technical data report. San Antonio, TX: Author.

13.

Harel

Confrey

(1994). The development of multiplicative reasoning in the learning of mathematics. Albany: State University of New York Press.

14.

Hedges

L. V.

Olkin

(1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

15.

Huang

T. H.

Liu

Y. C.

Chang

H. C.

(2012). Learning achievement in solving word-based mathematical questions through a computer-assisted learning system. Journal of Educational Technology & Society, 15, 248–259.

16.

Hutchison

L. F.

(2012). Addressing the STEM teacher shortage in American schools: Ways to recruit and retain effective STEM teachers. Action in Teacher Education, 34, 541–550. doi:10.1080/01626620.2012.729483

17.

Kouba

V. L.

(1989). Children’s solution strategies for equivalent set multiplication and division word problems. Journal for Research in Mathematics Education, 20, 147–158. doi:10.2307/749279

18.

Laerd Statistics. (2013a). One-way ANOVA. Lund Research. Retrieved from https://statistics.laerd.com/statistical-guides/one-way-anova-statistical-guide-3.php

19.

Laerd Statistics. (2013b). Repeated measures ANOVA. Lund Research. Retrieved from https://statistics.laerd.com/statistical-guides/repeated-measures-anova-statistical-guide.php

20.

Lesh

Doerr

H. M.

Carmona

Hjalmarson

(2003). Beyond constructivism. Mathematical Thinking and Learning, 5, 211–233. doi:10.1080/10986065.2003.9680000

21.

Maletsky

E. M.

Andrews

A. G.

Burton

G. M.

Johnson

H. C.

Luckie

L. A.

(2004). Harcourt math (Indiana ed.). Chicago, IL: Harcourt.

22.

Mevarech

Z. R.

Kramarski

(2008). Mathematical modeling and meta-cognitive instruction. Retrieved from http://www.icme-organisers.dk/tsg18/S32MevarechKramarski.pdf

23.

National Assessment of Educational Progress. (2015). Trend in fourth-grade NAEP mathematics average scores for students who are identified as students with disabilities. Retrieved from http://www.nationsreportcard.gov/reading_math_2015/#mathematics/groups?grade=4

24.

National Mathematics Advisory Panel. (2008). Foundation for success: The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.

25.

Polya

(1945). How to solve it: A new aspect of mathematical method. Princeton, NJ: Princeton University Press.

26.

Purdue Research Foundation. (2011). Multiplicative Reasoning Assessment Instrument. West Lafayette, IN: Author.

27.

Ramakrishnan

Soon

G. K.

(2009). Math in focus: The Singapore approach. Singapore: Marshall Cavendish Education.

28.

Schwartz

(1988). Intensive quantity and referent transforming arithmetic operations. In Hiebert

Behr

(Eds.), Number concepts and operations in the middle grades (pp. 41–52). Hillsdale, NJ: Lawrence Erlbaum.

29.

Sebastiani

(2002). Machine learning in automated text categorization. ACM Computing Surveys, 34, 1–47. doi:10.1145/505282.505283

30.

Seo

Y. J.

Bryant

D. P.

(2009). Analysis of studies of the effects of computer-assisted instruction on the mathematics performance of students with learning disabilities. Computers & Education, 53, 913–928. doi:10.1016/j.compedu.2009.05.002

31.

Seo

Y. J.

Bryant

D. P.

(2012). Multimedia CAI program for students with mathematics difficulties. Remedial and Special Education, 33, 217–225. doi:10.1177/0741932510383322

32.

Steffe

L. P.

Cobb

(1988). Construction of arithmetical meanings and strategies. New York, NY: Springer-Verlag.

33.

Steffe

L. P.

D’Ambrosio

B. S.

(1995). Toward a working model of constructivist teaching: A reaction to Simon. Journal for Research in Mathematics Education, 26, 146–159. doi:10.2307/749206

34.

Tzur

Johnson

H. L.

McClintock

Kenney

R. H.

Xin

Y. P.

… Jin

(2013). Distinguishing schemes and tasks in children’s development of multiplicative reasoning. PNA, 7(3), 85–101.

35.

Tzur

Xin

Y. P.

Kenney

Guebert

(2010). Students with learning disability in math are left behind in multiplicative reasoning? Number as Abstract Composite Unit is a likely “culprit.” ERIC database and abstract journal, Resources in Education. (ED510991)

36.

Vergnaud

(1983). Multiplicative structures. In Lesh

Landau

(Eds.), Acquisition of mathematical concepts and processes (pp. 127–174). Orlando, FL: Academic Press.

37.

Xin

Y. P.

(2012). Conceptual model-based problem solving: Teach students with learning difficulties to solve math problems. Rotterdam, The Netherlands: Sense Publishers.

38.

Xin

Y. P.

Liu

Jones

Tzur

(2016). A preliminary discourse analysis of constructivist-oriented math instruction for a student with learning disabilities. The Journal of Educational Research, 109(4), 436-447. DOI:10.1080/00220671.2014.979910

39.

Xin

Y. P.

Liu

Zheng

(2011). A cross-cultural lesson comparison on teaching the connection between multiplication and division. School Science and Mathematics, 111, 354–367. doi:10.1111/j.1949-8594.2011.00098.x

40.

Xin

Y. P.

Tzur

(2008). Nurturing multiplicative reasoning in students with learning disabilities in a computerized conceptual-modeling environment (National Science Foundation funded project). Purdue University, West Lafayette, IN.

41.

Xin

Y. P.

Zhang

Park

J. Y.

Tom

Whipple

(2011). A comparison of two mathematics problem-solving strategies: Facilitate algebra-readiness. The Journal of Educational Research, 104, 381–395. doi:10.1080/00220671.2010.487080