The Effect of Assistance on Learning and Affect in an Algebra Tutor

Abstract

We investigated the impact of assistance on learning and affect during problem-solving activities with a computer tutor we built using the Cognitive Tutor Authoring Tools framework. The tutor delivered its primary form of assistance in the form of worked-out examples. We manipulated the level of assistance the examples in the tutor provided, by having similar problem-example pairs in one version of the tutor (high-assistance condition) and reduced similarity problem-example pairs in the other version (reduced-assistance condition). The reduced-assistance condition resulted in significantly higher learning, without increasing negative affect like frustration.

Keywords

worked examples assistance learning affect

Educational technologies have tremendous potential to support student learning, approaching in some cases the effectiveness of human tutors (VanLehn, 2011), but this potential depends on how the technologies are designed. In this article, the focus is on the design of technologies that support problem-solving activities for students who are in the intermediate stage of cognitive skill acquisition (i.e., are not complete novices because they have received instruction on the domain, but are not yet experts). In this intermediate stage, problem-solving is beneficial for learning (Kalyuga, Ayres, Chandler, & Sweller, 2003) but can be hindered by impasses due to incomplete or fragmented student knowledge (VanLehn, 1998). Educational technologies can provide assistance to help students overcome the impasses. This assistance can take various forms, such as feedback for correctness on solution entries (Koedinger & Aleven, 2007), examples showing worked-out solutions that are similar to the problem the student is solving (Muldner & Conati, 2010; Najar & Mitrovic, 2013; Weber & Brusilovsky, 2001), hints on the domain (Muir & Conati, 2012; VanLehn et al., 2005), encouragement (Arroyo et al., 2014; Baylor, 2011), meta-cognitive support (Aleven & Koedinger, 2002; Conati & VanLehn, 2000), and instructional videos (Craig, Gholson, Brittingham, Williams, & Shubeck, 2012; McNamara, 2017).

An open question, however, is how explicit the assistance should be in order to foster learning. This question, generally referred to as the assistance dilemma (Koedinger & Aleven, 2007), has been the subject of much debate (Alfieri, Brooks, Aldrich, & Tenenbaum, 2011; Anderson, Corbett, Koedinger, & Pelletier, 1995; Schwartz & Martin, 2004). Explicit assistance, for instance in the form of a hint specifying what to do, can help students make progress (e.g., generate a problem solution). However, this type of assistance may encourage passive processing, resulting in less learning than constructive engagement (Chi, 2009). In contrast, reduced assistance, for instance in the form of general prompts that nudge the student toward the answer without giving that answer away, may block shallow strategies and thus encourage constructive behaviors. However, the challenge with reduced assistance is that it needs to provide sufficient scaffolding to enable problem-solving progress and learning, without being detrimental to affect.

We investigated the assistance dilemma in the context of a computer-based tutor that we built using the Cognitive Tutor Authoring Tools (CTAT) framework (Aleven et al., 2016). The tutor provided problems for students to solve and assistance in the form of worked-out examples, which are problems that include a step-by-step solution. In general, students commonly refer to examples when problem-solving (Chi, Bassok, Lewis, Reimann, & Glaser, 1989; Renkl, 2014; VanLehn, 1999), but to date there is little work on what types of examples best promote learning in this context, and even less work on how assistance delivered by examples influences student affect. Our work takes a step in filling these gaps. Specifically, we manipulated the level of assistance the examples provided in the tutor according to how similar a given example was to its corresponding problem and measured the subsequent effect on learning and affect. We describe the study we conducted and the results from that study after we present the related work.

Examples as Assistance

One way that examples can provide assistance to learning is through an example-studying context that does not involve problem-solving (Adams et al., 2014; Atkinson & Renkl, 2007; Conati & VanLehn, 2000; Najar & Mitrovic, 2013). Studying examples is well suited to the early stages of cognitive skill acquisition, but as students gain more knowledge, solving problems becomes more effective for learning (Kalyuga et al., 2003). There are a number of ways to operationalize the problem-solving context. For instance, in a series of seminal studies, students alternated between studying examples and solving problems (Cooper & Sweller, 1987; Sweller & Cooper, 1985). Some subsequent research has investigated the effect of problem or example sequencing, for example, to determine if students learn more if they first study an example and then solve a problem, or vice versa (Leppink, Paas, van Gog, van der Vleuten, & van Merrienboer, 2014; Reisslein, Atkinson, Seeling, & Reisslein, 2006; van Gog, 2011). In these studies, the examples and problems were not presented together (e.g., students studied an example, closed it, opened a problem, and worked on it without access to the example). In contrast, other studies presented the problem and example(s) together at the same time, for instance, as problem-example pairs (Chi et al., 1989; Lee, Betts, & Anderson, 2015; Muldner & Conati, 2010; VanLehn, 1999). This latter context is the focus of our work.

When an example is present while a student is solving a problem, the example can provide assistance by illustrating a step-by-step solution. Ideally, students who refer to the example should learn the domain principles (i.e., rules) that generated the example solution steps. While information on the domain principles is typically not included in the worked-out example, students can infer the principles through self-explanation, the process of explaining instructional material to one self (Chi & VanLehn, 1991; VanLehn, 1998). Unfortunately, many students choose to copy the example solution over to the problem they are solving without trying to self-explain or constructively process the example (Chi et al., 1989; Muldner & Conati, 2010; VanLehn, 1998, 1999). Copying may help the student make progress in solving the problem, but it also interferes with learning (Muldner & Conati, 2010; Reed, Dempster, & Ettinger, 1985; VanLehn, 1998).

A key factor that impacts the type of processing an example facilitates (shallow copying vs. constructive self-explanation) is the similarity between it and the problem the student is solving. Examples that are highly similar to the problem provide high assistance because they illustrate exactly how the corresponding problem should be solved. While even highly similar examples typically include some superficial differences with the problem, students can resolve these differences by aligning the example to the problem, and replacing example variables and constants with ones needed for the problem solution (Reed, 1987; VanLehn, 1998). Thus, highly similar examples facilitate copying, a passive behavior that does not foster learning (Muldner & Conati, 2010). In contrast, examples that include differences with the problem that block copying provide reduced assistance, because the student cannot copy the example solution to obtain a correct problem solution. While this is a good thing, problem-example differences may block not only copying but potentially also learning (Reed, 1987; Reed, Ackinclose, & Voss, 1990) if students cannot reconcile the differences. Thus, traditionally, tutoring systems that provide examples to aid problem-solving have selected highly similar examples (Weber & Brusilovsky, 2001).

Recently, there has been renewed interest in the potential of problem-example differences to foster learning. Muldner and Conati (2010) developed a tutoring system that provided assistance to students solving physics problems by selecting examples tailored to student knowledge and the problem being solved. The examples included certain types of differences that blocked superficial copying. Students who received these examples produced more self-explanations and copied less than students in a control condition who had highly similar examples. While this is encouraging, the within-subject design made it difficult to assess learning due to carry-over effects. Lee et al. (2015) did investigate the effect of problem-example differences on learning, based on participants’ ability to generate algebraic expressions represented as data flow diagrams. Examples that blocked superficial copying resulted in more learning than examples that enabled copying. Jennings and Muldner (2018) varied the similarity between problem-example pairs within a single problem session, with a focus on how fading of assistance influenced learning. The reverse fading mechanism that provided initially low assistance (i.e., reduced similarity problem-example pairs) but that transitioned to providing high assistance toward the end of the instructional sequence (i.e., high similarity problem-example pairs) improved learning more than the reverse sequence (initially high assistance faded to reduced assistance). This result highlights the benefits of reduced assistance that fades to high assistance. However, since the focus in that work was on manipulating the fading of assistance, we did not compare learning from reduced assistance against learning from high assistance, something we do in this work. Also in contrast to the main manipulation in Jennings and Muldner (2018), here we do not vary the level of assistance in a given condition, instead holding it constant.

Affect and Learning

Affect is a broad term that can be used to characterize emotional states (e.g., frustration, boredom) and moods (emotional states sustained over a period of time) (Gross, 1998). We focus on emotion in this study (the terms emotion and affect are used interchangeably in this article). In problem-solving situations involving interactions with educational technologies, the prominent emotions appearing during problem solving include frustration, boredom, anxiety, confusion, and delight (Graesser & D’Mello, 2011). Predominantly, these are negative emotions. This is concerning as certain negative emotions interfere with learning, including boredom (Bosch & D’Mello, 2017; Baker, D’Mello, Rodrigo, & Graesser, 2010) and anxiety (Ashcraft & Krause, 2007). For instance, Bosch and D’Mello (2017) analyzed affective data from students learning to program and found that boredom and frustration were negatively associated with learning. Interestingly, they also found that transitions between confusion and frustration (and vice versa) were positively correlated with learning, highlighting that some confusion and frustration may be beneficial. D’Mello, Lehman, Pekrun, and Graesser (2014) confirmed that under certain conditions confusion can be beneficial. This direct link between emotion and learning, however, is not always present. For instance, Arroyo et al. (2017) tested an intervention designed to reduce boredom by having a tutoring system promote collaboration between students. The intervention successfully reduced boredom but did not impact student learning. Arroyo, Woolf, Cooper, Burleson, and Muldner (2011) showed that virtual learning companions that delivered encouraging messages in a tutoring system lowered frustration and boredom but did not influence learning. The lack of a learning benefit, however, does not mean that the affective interventions failed, since they did help foster positive affect and potentially had long-term effects, such as improved attitudes toward the domain, that were not captured by the posttest administered immediately after the learning session in that study.

To date, work on educational technologies has focused on building models that can automatically detect student affect (Arroyo et al., 2009; Bosch & D’Mello, 2017; Botelho, Baker, & Heffernan, 2017; Muldner, Burleson, & VanLehn, 2010), so less research exists on the impact of interventions or assistance on affective variables. As an example of a notable exception, however, Corbalan, Paas, and Cuypers (2010) manipulated whether students would receive step-by-step feedback on their solutions (which could be seen as high assistance) or delayed feedback (reduced assistance). The results showed that high assistance in the form of step feedback resulted in increased motivation. More broadly in the classroom context, however, high assistance can lead to boredom if students do not feel challenged (Martin, Hands, Lancaster, Trytten, & Murphy, 2008; Pekrun, Goetz, Titz & Perry, 2007). While reduced assistance can increase challenge, this can also cause boredom when students feel overwhelmed (Pekrun et al., 2007). These examples illustrate the complexity related to identifying relationships between assistance and affect.

The Present Study

Prior work suggests that when students use examples to assist problem-solving (where both the example and problem is present at the same time), reduced assistance can be beneficial, but more work is needed to replicate and extend this research. Moreover, an open question is how the level of assistance, reduced versus high, impacts how students feel in this context (for instance, does reduced assistance raise frustration or anxiety?). Our work fills these gaps by addressing the following two research questions:

Does the level of assistance impact learning or problem-solving behaviors?

Does the level of assistance influence how students feel during problem-solving?

To address these questions, we built a computer tutor that provided students with problems to solve; each problem was presented with one corresponding example that showed a step-by-step solution, thus providing assistance for problem solving. We manipulated the level of assistance in the tutor so that it was either high assistance or reduced assistance, based on the similarity between the example and its corresponding problem. Our hypothesis was that reduced assistance would result in more learning as compared with high assistance, because it would encourage constructive processing by blocking copying. We did not have explicit hypotheses about affect but were especially interested in whether reduced assistance would increase negative affect like frustration. We chose to focus on three negative emotions, namely, frustration, boredom, and anxiety because they commonly occur during problem-solving situations (Graesser & D’Mello, 2011) and can be detrimental to learning (Ashcraft & Krause, 2007; Baker et al., 2010, Graesser & D’Mello, 2011).

The study protocol was reviewed and approved by the university research ethics committee.

Target Domain and Theoretical Framework

The study involved algebra problems of the type shown in Figure 1, left (with the exception that the problem solutions were longer than shown in the figure, 3–4 steps). The problems were composed of variables instead of numeric constants (e.g., similar to the approach used in Cooper and Sweller, 1987). For illustrative purposes, Figure 1 shows the problem’s worked-out solution, but this would not be given to the student.

Figure 1.

A problem (left) and two examples (middle and right); problem-example₁ pair have high similarity, while problem-example₂ pair have reduced similarity. Note that also shown for illustrative purposes are labels for the algebraic operations (domain rules), that were applied to a given line (ADD implies adding a variable to both sides to eliminate it from one side; MULT implies multiplying both sides by a variable to eliminate it from one side).

As the theoretical framework, we used the cognitive architectures ACT-R and Cascade (Anderson, Fincham, & Douglass, 1997; VanLehn, 1999), both of which state that problem-solving is accomplished by a series of rule applications. In our study, a given rule embodied an algebraic operation. For instance, to isolate the variable a to solve the problem in Figure 1 (left), one has to first eliminate the variable c from the right side, accomplished by adding c to both sides of the equation (ADD rule in Figure 1, left), and then eliminate the variable d from the right side by multiplying both sides of the equation by it (MULT rule in Figure 1, left).

This theoretical framework provides a useful mechanism for quantifying the level of assistance an example provides for solving a problem (Jennings & Muldner, 2018; Muldner & Conati, 2010). Specifically, if the example does not share any rules with the corresponding problem, then the example does not assist problem-solving, because students cannot use it to learn any of the rules needed for the problem’s solution. In contrast, problem-example pairs that differ only with respect to variable names offer high assistance because they facilitate solution generation through copying. This is the case for the problem-example₁ pair in Figure 1. One way to use example₁ to help solve the problem involves replacing example-specific constants by ones needed for the problem solution. This can be accomplished by first generating a mapping between the problem and example₁ variables (e.g., the problem variable b maps to the example₁ variable y), and then using that mapping to guide the replacement of example variables by ones needed for the problem solution. Students do not find this process challenging and so can use it to copy from the example to produce a correct solution (Muldner & Conati, 2010; VanLehn, 1998). Of course, students could also use self-explanation to learn the underlying rule after copying (e.g., example₁ provides that opportunity), but students who copy tend to not self-explain (Muldner & Conati, 2010; VanLehn, 1999), possibly because self-explanation requires effort.

Problem-example pairs that differ with respect to the order of rule applications needed for their respective solutions offer reduced assistance, because they block copying of the example solution and thus require self-explanation. This is illustrated by the problem-example₂ pair in Figure 1, which shows that the example₂ solution requires the application of the MULT rule and then the ADD rule, while the reverse order of rule applications is needed to generate the problem solution. This type of difference reduces the assistance the example affords because copying will not produce a correct problem solution. Critically, however, the example still affords the student the opportunity to infer the rules from the example solution by self-explanation; once these are inferred they can be applied to generate the problem solution. However, self-explanation means the student has to infer the rule on their own by reasoning about the example solution, and thus the example’s assistance is reduced.

We relied on this characterization of assistance to match examples to problems, producing either high-assistance pairs or reduced-assistance pairs (details later in this article). In contrast to prior work (Jennings & Muldner, 2018; Muldner & Conati, 2010), in this study, we hold the assistance constant, in order to investigate the pure effect of high versus reduced assistance.

CTAT Computer Tutor

Tutor construction

To address our research questions, we created a computer tutor using the CTAT framework (Aleven et al., 2016). In general, tutor construction is a challenging process that can require hundreds of hours of development time for a single hour of instruction. The CTAT framework facilitates the construction of tutoring systems by providing tools for the process (e.g., interface manager and problem sequencing).

The tutor was built using the CTAT graphical user interface that includes various widgets (e.g., textboxes, buttons, and labels). To investigate the impact of assistance, we created two versions of the tutor. Each version was populated with a set of problem-example pairs and each problem was presented with an example that was always visible (see Figure 2). The only difference between the two versions corresponded to the examples that the tutor provided. Specifically, we populated one tutor with examples that offered high assistance because the similarity between a given problem-example pair was always high, and populated a second tutor with examples that offered reduced assistance because the example solution required a different order of rule applications. Details on the problem-example sets are described later in this article.

Figure 2.

CTAT tutor used to enter problem solutions (left window) and refer to examples (right window); the example solution was not editable by the students.

To enable the tutor to provide feedback for correctness on problem-solving entries, we used the CTAT functionality to create a solution graph for each problem. The solution graph is an internal representation of a problem solution used by CTAT to evaluate student’s problem-solving entries. We implemented additional functionality to allow the tutor to recognize all correct versions of a given entry, so students had flexibility in terms of the form of their entries. To do so, we used the algorithm proposed in Shapiro (2005), which does not require prestoring all possible solutions and thus saves significant development effort and processing time while students are using the system. Once the tutor was implemented, it was deployed online, which facilitates its dissemination.

Interacting with the tutor

The interaction with the tutor was the same in the two tutor versions (reduced assistance and high assistance). Each problem was presented alongside a corresponding example (see Figure 2). Students were required to enter the problem solution step-by-step using the provided text boxes in the tutor interface (see white regions in the problem solution, Figure 2). The tutor provided feedback for correctness by coloring a given entry green or red, for correct and incorrect steps, respectively (this was done after a student pressed the enter key to indicate they were done with a step). Students had to complete a problem before moving on to the next one and once they moved on, they could not return to a previous problem. Students moved on to a new problem by pressing the Done button in the CTAT interface. All student interface actions were automatically logged, facilitating analysis.

Method

Materials

The study materials included algebra problem-example pairs used to populate the CTAT tutors, a pretest and a posttest, and affective questionnaires embedded in the tutor.

Algebra problems and examples

We created two sets of problem-example pairs (12 pairs per set) as follows:

The high-assistance set consisted of highly similar problem-example pairs, with the only differences in a given pair corresponding to variable names. This set provided high assistance because the examples enabled copying.

The reduced-assistance set consisted of reduced similarity problem-example pairs, with the example in a given pair requiring a different order of rule applications for its solution compared with the corresponding problem solution. Importantly, the set of rules needed for the problem solution was isomorphic to the set of rules needed for its corresponding example’s solution (this condition is necessary so that the example affords the opportunity to learn the rule needed for the problem). Thus, while this set provided reduced assistance because it blocked copying, it still afforded the opportunity to use the example to learn the rules needed for the corresponding problem’s solution.

The assistance in each set was constant (i.e., in the high-assistance set, all the examples provided high assistance and in the reduced-assistance set, all the examples provided reduced assistance). The problems in the two sets were identical (same 12 problems in each set, each requiring three to four steps for their solution), but the sets had different types of examples paired with the problems.

To identify appropriate problem-example pairs, we relied on content analysis, by (a) solving by hand on paper a range of algebra problems requiring three to five steps for their solutions and identifying the rule needed for each solution step (this produced a rule vector for each solution specifying the sequence of rule applications needed to generate the solution) and (b) creating problem-example pairs based on that content analysis. For the high-assistance set, an example was matched to a problem if its rule vector was identical to the problem’s (length and sequence), but the problem and example used different variable names. For the reduced assistance set, an example was matched to a problem if its rule vector was isomorphic to the problems (same length and same set of rules) but with different rule orders (i.e., the example required a different order of rule applications as compared to the problem).

Pretest and posttest

To assess learning, we used a paper and pencil pretest and posttest that included 11 questions each (the same questions were used for the two tests, but the variable names were varied). Examples were not included on the tests because we wanted to measure students’ proficiency in the absence of assistance. The test was created based on the content analysis described earlier and accordingly included problems whose solutions required the same knowledge (rules) as the problems used to populate the CTAT tutor. Some of the test problems involved the same order of rule applications as the problems in the CTAT tutor, while others required a different order of rule applications.

Affect questions

Affective information was obtained using the following three questions: How frustrated are you feeling?/How anxious are you feeling?/How bored are you feeling? To answer the question, participants selected an option from a 5-point Likert-type scale for each emotion (1 = not at all, 5 = extremely). This method is commonly used in studies obtaining affective information from students interacting with educational technologies (e.g., Arroyo et al., 2009; Conati & MacLaren, 2009). The three questions were presented on a single, separate screen in the CTAT tutor on three occasions, namely, every four problems.

Participants

The participants were undergraduate students who were compensated with bonus course credit. The only exclusion criterion was that participants had not taken (or were currently enrolled in) any university-level math courses. The analysis is based on students who did not score at ceiling on the pretest (N = 47).¹

Design and procedure

The study involved two conditions: high assistance (the only difference in a given problem-example pair was variable names) and reduced assistance (a given problem-example pair required different order of rule applications for their respective solutions). We used a between-subject design, with participants assigned to one of the two study conditions in a round-robin fashion.

The procedure for both conditions was the same. Each session was conducted individually in a quiet room. Participants first completed a pretest (15 minutes) as well as a brief questionnaire (10 minutes). Next, the experimental portion of the study began. Participants were introduced to the CTAT tutor and shown how to enter the solution to a problem—At this stage, they were free to ask any clarification questions. They were informed that the tutor provided feedback for correctness, and in order to move on to the next problem, they had to finish the current problem. Participants were given up to 45 minutes for this phase. Following the experimental phase, participants completed a posttest (15 minutes) and a brief questionnaire (5 minutes). The brief questionnaires measured participants’ beliefs about mindset and perseverance beliefs and are not included in this analysis.

The pretest and posttest were scored out of 40, with the points for a given question corresponding to the number of rule applications needed for the question’s canonical solution (e.g., if a question required three rule applications to generate the solution, its point value was three). We used this scoring method as it is more sensitive than just marking a question as correct or incorrect, given that each question required multiple rule applications (2–5 applications).

Results

Our analysis was guided by our two research questions related to the effect of assistance on learning and behaviors as well as affect.

Impact of assistance on learning and behaviors

The descriptives related to the pretest, posttest, and gain scores (posttest – pretest) are presented in Table 1. As far as the pretest, as expected the pretest scores were similar and there was no significant difference in pretest scores between conditions (p = .56), meaning the groups were comparable in terms of the target algebra knowledge prior to the intervention. Since algebra is taught in middle and high school and all participants were undergraduate students, they had been exposed to algebra in the past, but as shown by the pretest scores, their initial level of competence was on the low end of the scale.

Table 1.

Descriptives Related to the Pretest %, Posttest %, and % Gain for Each Condition.

	High assistance (n = 27)		Reduced assistance (n = 20)
Variable	M	SD	M	SD
Pretest %	37.2	36.2	43.1	31.2
Posttest %	62.0	33.5	78.4	22.3
Gain	24.8	23.4	35.3	24.8

Did level of assistance significantly impact learning? To answer this question, we conducted an analysis of covariance with posttest as the dependent variable, pretest as the covariate, and level of assistance (reduced, high) as the between-subject variable. As shown in Table 1, the reduced-assistance group had higher gains from pre to posttest and this effect of assistance was significant, F(1, 44) = 4.5, p = .04, η_p²= .09. Thus, reducing the level of assistance, in the form of reduced problem-example similarity, improved learning.

We next report on the within-tutor variables, including time on task and error rate. As expected, reduced assistance increased time on task during the experimental phase: On average, participants took 26.2 minutes in the reduced-assistance condition (SD = 12.0) as compared with 20.2 minutes in the high-assistance condition (SD = 6.5), t(27.2) = 2.0, p = .05, d = .60. Participants also made marginally more errors while generating the problem solution in the reduced-assistance condition (reduced assistance: M = 24.8, SD = 25.5; high assistance: M = 13.3, SD = 16.8), t(45) = 1.9, p = .07, d = .55. Reduced assistance did not interfere with participants’ ability to eventually generate a final correct solution as there was little difference between the two conditions in terms of the number of problems completed (reduced assistance: M = 12.0, SD = 0; high assistance: M = 11.22, SD = 2.35).

Impact of assistance on affect

The level of assistance a tutor provides has the potential to impact student emotions, for instance, to increase frustration if assistance is low and the student is struggling. Recall that students were prompted to self-report on the three target emotions every four problems, below labeled as points p1, p2, and p3, respectively. Figure 3 shows the average ratings for each emotion at these 3 points in the two conditions. For each of the three target emotions, the average score at each of the 3 points was below 3 (on a scale from 1 to 5, with 1 indicating not at all [frustrated, anxious, and bored]), indicating that students did not self-report high levels of frustration, anxiety, or boredom.

Figure 3.

Mean self-reported ratings for each of the three target affective states at points p1 (after four problems), p2 (after 8 problems), and p3 (after 12 problems); Mean self-reported score is shown on the y axis, where 1 = not at all and 5 = Extremely.

For frustration, in the high-assistance condition, the mean rating was low and fairly flat across the three assessment points. In the low-assistance condition, frustration did start off slightly higher (see point p1, Figure 3) and this difference between conditions at point p1 was significant, t(44) = .04. However, the frustration in the low-assistance condition decreased over time and the effect of condition was not significant in either points p2 or p3. Anxiety decreased slightly over time, with little difference between the conditions (while it was slightly higher from a descriptive standpoint in the low-assistance condition at point p1, this was not significant). Boredom increased somewhat over time for the high-assistance condition, but exhibited a more quadratic trend in the low-assistance condition, with an increase in the middle of the session followed by a decrease at the end.

We conducted three mixed analyses of variance, one for each target emotion. To account for potential trends over time in addition to overall main effects, each analysis of variance included point-in-sequence (p1, p2, p3) as the within-subject variable and condition (high assistance and reduced assistance) as the between-subject variable. If condition influenced self-reported emotion, this would have been indicated by either a main effect of condition (this analysis collapses across p1, p2, p3) or by a condition x point-in-sequence interaction (this analysis checks if condition disproportionally influences affect across the three points). We did not find evidence of such effects, in that none of the analyses were significant (p > .1) and with one exception, all the effect sizes were very small (η_p² = <.02; the one exception was for the main effect of frustration: η_p² = <.057).

Discussion

Our work makes a number of contributions: (a) it demonstrates the benefit of reduced assistance over high assistance for learning from problem-solving when examples are present and (b) it examines the impact of assistance level on affect, one of the few studies to do so. Importantly, the benefit of reduced assistance did not come at the cost of inducing negative student affect: Self-reported frustration, anxiety, and boredom were similar in the reduced- and high-assistance conditions, and we did not find evidence that condition influenced affect (as indicated by the lack of a significant main effect of condition on the target emotions and by the lack of a significant interaction between condition and emotion). While we acknowledge that conclusions cannot be drawn from nonsignificant results, the corresponding effect sizes for the affective analyses were small or very small, indicating that from a practical stand point, reduced assistance did not increase negative affect. Prior work does exist examining the effects of assistance operationalized through examples (Atkinson & Renkl, 2007; Cooper & Sweller, 1987; McLaren, Lim, & Koedinger, 2008; McLaren, van Gog, Ganoe, Karabinos, & Yaron, 2016), but it focuses on comparing examples against other approaches such as tutored problem-solving or faded examples. Thus, to date, there is little work examining assistance operationalized as the degree of similarity between problems and examples on student learning and affect, something we did in this study.

The fact that reduced assistance did not cause more negative affect (frustration, anxiety, boredom) is an encouraging finding, particularly since the reduced-assistance made copying an unviable strategy for obtaining the correct answer, and so was the more challenging condition of the two (as indicated by increased error rate and time on task). Aligned with our results, there is prior precedent for challenging students by giving them low assistance in studies with human tutors working with students. For instance, results from observational studies indicate that prompting students to produce an answer, that is, low assistance, is associated with more learning than telling them the answer, that is, high assistance (Chi, Roy, & Hausmann, 2008). Along similar lines, Chi, Siler, Jeong, Yamauchi, and Hausmann (2001) found that students who were given high-level prompts by human tutors had similar learning gains as students who were given more detailed information. A recent review by Chi (2009) stated that in order for students to learn, they have to be constructive agents in the learning process, meaning that they had to engage in strategies that went over and beyond the surface level of the instructional materials. Our results demonstrate the same pattern, but with a computer tutor rather than a human tutor, providing promising indications that the findings transfer to educational technology settings.

A relevant question relates to how level of assistance interacts with prior knowledge to influence learning outcomes. We propose the answer depends on several factors, including students’ stage of cognitive skill acquisition, how constructively students process the assistance, and how “reduced” the assistance is. In the context of our study, all participants had prior exposure to algebra, so they were not at the earliest stage of skill acquisition, which is when the highest assistance is needed (Cooper & Sweller, 1987; Sweller & Cooper, 1985). This, combined with the fact that the reduced-assistance condition was intended to promote constructive processing while providing sufficient scaffolding, leads us to conjecture that even the low knowledge students would benefit from the reduced-assistance condition. While our study was not designed to test this conjecture as we did not select low versus high knowledge students, an exploratory analysis provides preliminary support for our conjecture. Specifically, we divided students into a low versus high knowledge groups based on a median split and added this as a second factor to an analysis of covariance that also included condition as the other factor and pretest as the covariate. As shown in Figure 4, there is a little difference between the low and high knowledge students in posttest scores in the high-assistance condition. In contrast, there is a large difference between the low and high knowledge students in posttest scores in the reduced-assistance condition, with low knowledge students scoring higher. This interaction between condition and prior knowledge is significant, F(1, 56) = 6.5, p = .01, η_p²= .11. In retrospect, this result is not surprising since the low knowledge group has more room to “gain,” but it is particularly encouraging that reduced assistance was the most beneficial to that group. This suggests that the reduced assistance was at the right level of scaffolding in terms of supporting learning. However, it is not clear if and how these findings would transfer to students who are in the early stages of cognitive skill acquisition. This awaits future research.

Figure 4.

Interaction between knowledge (low and high) and posttest score for each type of assistance (high and reduced).

Should examples be shown alongside a problem? As we mentioned earlier, to date, the majority of studies present problem-example pairs sequentially, one after the other, rather than showing both the problem and example together (Leppink et al., 2014; Reisslein et al., 2006; van Gog, 2011). A sequential presentation format may encourage students to study the example, since direct copying is not possible. When an example is presented alongside the problem, copying is facilitated, but this type of presentation encourages problem-example comparison, which in turn fosters schema development. To illustrate, Rittle-Johnson and Star (2007) had students either study multiple examples, where comparison between the examples was encouraged or study the examples sequentially (all students subsequently solved similar problems). Comparing and contrasting solutions led to higher learning. However, in this study, the similarity between the examples was held constant. To the best of our knowledge, only one study manipulated similarity and type of presentation (Lee et al., 2015). The findings showed that the highest transfer performance was obtained when the example was presented alongside the problem (with critical features annotated; see Lee et al., 2015, p. 949). Thus, there are indications that a side-by-side presentation format has benefits.

Limitations and Future Work

In this study, the examples were always visible to the student. However, our results still do show that examples had a significant effect on learning, due to the experimental design we used. Specifically, because the type of assistance was the only difference between two conditions, and because the reduced-assistance condition significantly improved learning over the high-assistance condition, this indicates that students did use the examples to support problem-solving, and, importantly, that the type of example had a significant effect on their learning. In other words, while students did not have to request an example, if the examples were not used, then we would not have observed a conditional effect. However, our design meant we do not have data on how students used the examples, which could enrich our results. For instance, this could involve analyzing if students used examples to correct mistakes, to verify answers, or to help solve the problem. These kind of data could be obtained using an eye tracker or a masking interface (Conati & VanLehn, 2000), where example solution steps are covered and can be uncovered by moving the mouse over them. Both these alternatives add complexity to the study design and in the case of the masking interface, make the interface more cumbersome to use. Thus, for this iteration of the study, we did not incorporate functionality to enable logging of example-usage actions, but this is something we will consider in the future. We also plan to analyze student strategies as they solve problems with different types of examples.

We investigated the impact of the type of assistance without having the tutor tailor that assistance to the student’s knowledge or meta-cognitive strategies. While we believe that adaptation has potential to improve student learning, it adds complexity to the system and so it is important to evaluate the effects of more basic interventions as did here. Thus, this functionality also awaits future research.

Footnotes

Acknowledgments

The authors thank the anonymous reviewers for their comments.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded under NSERC Discovery Grant (no. 1507).

Note

Giuliana Borracci is a recent graduate of Carleton University, having received her Bachelor of Cognitive Science. She is currently in her first year of her Master of Science in Occupational Therapy at Western University.

Erica Gauthier obtained her Bachelor of Cognitive Science with Honors. Her thesis work focused on the impact of assistance in educational technologies.

Jay Jennings is a PhD student at Carleton University studying Cognitive Science. His primary research interests revolve around improving and understanding human learning, including ways that examples assist learning and how examples can promote transfer between disparate domains, which he hopes can inform educational policy development.

Kyle Sale is a graduate of Carleton University's Cognitive Science program. His thesis in educational psychology investigated the influence of assistance on learning with educational technologies; his other interests include personality psychology and the impact of individual differences on performance and learning.

Kasia Muldner is an assistant professor in the Institute of Cognitive Science at Carleton University. Her research program is focused on student learning and creativity, both in terms of investigating factors that influence outcomes of interest (such as performance, learning, and affect) as well as the design of interventions or technologies that support students in the instructional process.

References

Adams

D. M.

McLaren

B. M.

Durkin

Mayer

R. E.

Rittle-Johnson

Isotani

van Velsen

(2014) Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior 36: 401–411.

Aleven

McLaren

B. M.

Sewall

van Velsen

Popescu

Demi

Ringenberg

Koedinger

K. R.

(2016) Example-tracing tutors: Intelligent tutor development for non-programmers. International Journal of Artificial Intelligence in Education 26: 224–269.

Aleven

V. A.

Koedinger

K. R.

(2002) An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science 26(2): 147–179.

Alfieri

Brooks

P. J.

Aldrich

N. J.

Tenenbaum

H. R.

(2011) Does discovery-based instruction enhance learning? Journal of Educational Psychology 103(1): 1–18.

Anderson

J. R.

Corbett

A. T.

Koedinger

K. R.

Pelletier

(1995) Cognitive tutors: Lessons learned. Journal of the Learning Sciences 4(2): 37–41.

Anderson

J. R.

Fincham

J. M.

Douglass

(1997) The role of examples and rules in the acquisition of a cognitive skill. Journal of Experimental Psychology: Learning, Memory, and Cognition 23(4): 932–945.

Arroyo

Cooper

D. G.

Burleson

Woolf

B. P.

Muldner

Christopherson

(2009) Emotion sensors go to school. In Proceedings of the Conference on Artificial Intelligence in Education (pp.17–24).

Arroyo, I., Wixon, N., Allessio, D., Woolf, B., Muldner, K., & Burleson, W. (2017). Collaboration improves student interest in online tutoring. In Proceedings of the conference on artificial intelligence in education (pp. 28–39). Cham, Switzerland: Springer.

Arroyo

Woolf

B. P.

Burleson

Muldner

Rai

Tai

(2014) A multimedia adaptive tutoring system for mathematics that addresses cognition, metacognition and affect. International Journal of Artificial Intelligence in Education 24: 387–426.

10.

Arroyo, I., Woolf, B. P., Cooper, D. G., Burleson, W., & Muldner, K. (2011). The impact of animated pedagogical agents on girls’ and boys’ emotions, attitudes, behaviors and learning. Proceedings of the IEEE International Conference on Advanced Learning Technologies (pp. 506–510). Piscataway, NJ: IEEE.

11.

Ashcraft

M. H.

Krause

J. A.

(2007) Working memory, math performance, and math anxiety. Psychonomic Bulletin & Review 14(2): 243–248.

12.

Atkinson

R. K.

Renkl

(2007) Interactive example-based learning environments: Using interactive elements to encourage effective processing of worked examples. Educational Psychology Review 19(3): 375–386.

13.

Baker

D’Mello

S. K.

Rodrigo

M. T.

Graesser

A. C.

(2010) Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68(4): 223–241.

14.

Baylor

A. L.

(2011) The design of motivational agents and avatars. Educational Technology Research and Development 59(2): 291–300.

15.

Bosch

D’Mello

(2017) The affective experience of novice computer programmers. International Journal of Artificial Intelligence in Education 27: 181–206.

16.

Botelho, A. F., Baker, R. S., & Heffernan, N. T. (2017). Improving sensor-free affect detection using deep learning. In Proceedings of the conference on artificial intelligence in education (pp. 40–51). Cham, Switzerland: Springer.

17.

Chi

M. T. H.

(2009) Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science 1(1): 73–105.

18.

Chi

M. T. H.

Bassok

Lewis

M. W.

Reimann

Glaser

(1989) Self- explanations: How students study and use examples in learning to solve problems. Cognitive Science 13(2): 145–182.

19.

Chi

M. T. H.

Roy

Hausmann

R. G.

(2008) Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. Cognitive Science 32(2): 301–341.

20.

Chi

M. T. H.

Siler

Jeong

Yamauchi

Hausmann

(2001) Learning from human tutoring. Cognitive Science 25(4): 471–533.

21.

Chi

M. T. H.

VanLehn

(1991) The content of physics self-explanations. Journal of the Learning Sciences 1(1): 69–105.

22.

Conati

VanLehn

(2000) Toward computer-based support of meta-cognitive skills: A computational framework to coach self-explanation. International Journal of Artificial Intelligence in Education 11: 389–415.

23.

Conati

Maclaren

(2009) Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction 19(3): 267–303.

24.

Cooper

Sweller

(1987) Effects of schema acquisition and rule automation on mathematical problem-solving transfer. Journal of Educational Psychology 79(4): 347–362.

25.

Corbalan

Paas

Cuypers

(2010) Computer-based feedback in linear algebra: Effects on transfer performance and motivation. Computers & Education 55(2): 692–703.

26.

Craig

S. D.

Gholson

Brittingham

J. K.

Williams

J. L.

Shubeck

K. T.

(2012) Promoting vicarious learning of physics using deep questions with explanations. Computers & Education 58(4): 1042–1048.

27.

D’Mello

Lehman

Pekrun

Graesser

(2014) Confusion can be beneficial for learning. Learning and Instruction 29(1): 153–170.

28.

Graesser

D’Mello

S. K.

(2011) Theoretical perspectives on affect and deep learning. In: Calvo

D'Mello

(eds) New perspectives on affect and learning technologies, New York, NY: Springer, pp. 11–23.

29.

Gross

J. J.

(2018) The emerging field of emotion regulation: An integrative review. Review of General Psychology 2(3): 271–299.

30.

Jennings, J., & Muldner, K. (2018). From dissimilar to similar: Reverse fading assistance improves learning. Proceedings of the Cognitive Science Society Conference (pp. 560–565). Retrieved from mindmodeling.org/cogsci2018/papers/0122/index.html.

31.

Kalyuga

Ayres

Chandler

Sweller

(2003) The expertise reversal effect. Educational Psychologist 38(1): 23–31.

32.

Koedinger

K. R.

Aleven

(2007) Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review 19(3): 239–264.

33.

Lee

H. S.

Betts

Anderson

J. R.

(2015) Not taking the easy road: When similarity hurts learning. Memory & Cognition 43(6): 939–952.

34.

Leppink

Paas

van Gog

van der Vleuten

C. P.

van Merrienboer

J. J.

(2014) Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction 30: 32–42.

35.

Martin

J. H.

Hands

K. B.

Lancaster

S. M.

Trytten

D. A.

Murphy

T. J.

(2008) Hard but not too hard: Challenging courses and engineering students. College Teaching 56(2): 107–113.

36.

McLaren, B. M., Lim, S. J., & Koedinger, K. R. (2008). When is assistance helpful to learning? Results in combining worked examples and intelligent tutoring. In Proceedings of the conference on intelligent tutoring systems (pp. 677–680). Berlin, Germany: Springer.

37.

McLaren

B. M.

van Gog

Ganoe

Karabinos

Yaron

(2016) The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in computer-based learning environments. Computers in Human Behavior 55: 87–99.

38.

McNamara

D. S.

(2017) Self-explanation and reading strategy training (SERT) improves low-knowledge students science course performance. Discourse Processes 54(7): 479–492.

39.

Muir, M., & Conati, C. (2012). An analysis of attention to student—Adaptive hints in an educational game. In Proceedings of the conference on intelligent tutoring systems (pp. 112–122). Springer Berlin Heidelberg.

40.

Muldner, K., Burleson, W., & VanLehn, K. (2010). “Yes!”: Using tutor and sensor data to predict moments of delight during instructional activities. In Proceedings of the conference on user modeling, adaptation and personalization (pp. 159–170). Berlin, Germany: Springer.

41.

Muldner

Conati

(2010) Scaffolding meta-cognitive skills for effective analogical problem solving via tailored example selection. International Journal of Artificial Intelligence in Education 20: 99–136.

42.

Najar, A., & Mitrovic, A. (2013). Examples and tutored problems: How can self-explanation make a difference to learning? In Proceedings of the conference on artificial intelligence in education (pp. 339–348).

43.

Pekrun

Goetz

Titz

Perry

(2007) Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist 37(2): 91–105.

44.

Reed

S. K.

(1987) A structure-mapping model for word problems. Journal of Experimental Psychology: Learning, Memory, and Cognition 13(1): 124–139.

45.

Reed

S. K.

Ackinclose

Voss

(1990) Selecting analogous problems: Similarity versus inclusiveness. Memory & Cognition 18(1): 83–98.

46.

Reed

S. K.

Dempster

Ettinger

(1985) Usefulness of analogous solutions for solving algebra word problems. Journal of Experimental Psychology: Learning, Memory, and Cognition 11(1): 106–125.

47.

Reisslein

Atkinson

R. K.

Seeling

Reisslein

(2006) Encountering the expertise reversal effect with a computer-based environment on electrical circuit analysis. Learning and Instruction 16(2): 92–103.

48.

Renkl

(2014) Toward an instructionally oriented theory of example-based learning. Cognitive Science 38(1): 1–37.

49.

Rittle-Johnson

Star

J. R.

(2007) Does comparing solution methods facilitate conceptual and procedural Knowledge? An experimental study on learning to solve equations. Journal of Educational Psychology 99(3): 561–574.

50.

Shapiro

J. A.

(2005) An algebra subsystem for diagnosing students’ input in a physics tutoring system. International Journal of Artificial Intelligence in Education 15: 205–228.

51.

Sweller

Cooper

G. A.

(1985) The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction 2(1): 59–89.

52.

Schwartz

Martin

(2004) Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction 22(2): 129–184.

53.

van Gog

(2011) Effects of identical example-problem and problem-example pairs on learning. Computers & Education 57(2): 1775–1779.

54.

VanLehn

(1998) Analogy events: How examples are used during problem solving. Cognitive Science 22(3): 347–388.

55.

VanLehn

(1999) Rule-learning events in the acquisition of a complex skill: An evaluation of Cascade. Journal of the Learning Sciences 8(1): 71–125.

56.

VanLehn

(2011) The relative effectiveness of human tutoring, intelligent tutoring systems and other tutoring systems. Educational Psychologist 46(4): 197–221.

57.

VanLehn

Lynch

Schulze

Shapiro

J. A.

Shelby

Taylor

Treacy

Weinstein

Wintersgill

(2005) The Andes physics tutoring system: Lessons learned. International Journal of Artificial Intelligence in Education 15: 147–204.

58.

Weber

Brusilovsky

(2001) Elm-Art: An adaptive versatile system for web-based instruction. International Journal of Artificial Intelligence in Education 12: 351–384.