Abstract
We report three experiments that examine whether immediate versus delayed feedback produce differential concept learning. Subjects were shown hypothetical experiment scenarios and were asked to determine whether each was a true experiment. Correct-answer feedback was used for all three experiments; Experiments 2 and 3 also included detailed explanations. In all three experiments, subjects who received immediate feedback were shown the correct answer after each response. In Experiments 1 and 2, subjects in the delayed feedback condition were shown feedback after responding to all of the scenarios. All subjects then completed a posttest with novel scenarios. Experiment 3 was three parts (each session was 2 days apart). Subjects in the immediate feedback condition completed the posttest on the second session; subjects in the delayed feedback condition were given feedback on the second session and completed the posttest on the third session. Although no posttest differences were observed between the feedback conditions in Experiments 1 and 2, a delayed feedback advantage was found in Experiment 3. We propose that longer intervals in delayed feedback (relative to shorter intervals) might allow learners to forget the incorrect hypotheses they form during learning, which might thereby enhance the processing of feedback.
Feedback plays a critical role in learning and is often necessary for concept acquisition (Benassi et al., 2014; Butler et al., 2007; Corral & Carpenter, 2020; Corral et al., 2019). Behaviourists posit that for learning to occur, it is critical to provide learners feedback immediately after a response, as even minor delays can significantly impair acquisition (Saltzman, 1951). In line with this idea, various studies have demonstrated that immediate feedback produces better learning and retention than delayed feedback (Azevedo & Bernard, 1995; Kulik & Kulik, 1988; White, 1968). However, other studies have shown the opposite pattern (Butler et al., 2007; Butler & Roediger, 2008; Carpenter & Vul, 2011; Mullet et al., 2014; Schroth & Lund, 1993; Sturges, 1978; Swindell & Walls, 1993).
In a seminal meta-analysis, Kulik and Kulik (1988) carefully examined these contradictory findings and concluded that although delayed feedback can lead to better learning than immediate feedback, this effect is primarily limited to artificial laboratory settings that consist of relatively simple stimuli (e.g., letter strings), and under more ecologically valid conditions (e.g., classroom studies with complex concepts) immediate feedback typically produces superior learning. Various researchers (e.g., Azevedo & Bernard, 1995; Kulik & Kulik, 1988; Mory, 2004) have thus proposed that to optimise learning in the classroom, instructors should provide students feedback immediately after they respond to a given question.
However, Mullet et al. (2014) argue that most of the studies on this topic have typically been confounded by various factors (e.g., different retention intervals between feedback conditions; Butler et al., 2007), many of which often favour learning in the immediate feedback conditions. For example, as noted by Mullet et al., students in classroom studies are typically required to view immediate feedback for each question, whereas this requirement is optional when feedback is delayed (Pressey, 1950). On the contrary, many laboratory studies often confound the time between when immediate and delayed feedback are presented and when the posttest is completed, such that delayed feedback is often presented closer to the posttest than immediate feedback (e.g., Butler et al., 2007; Swindell & Walls, 1993). As a result, information that is acquired through delayed feedback might be less likely to be forgotten than when it is acquired through immediate feedback, and might therefore enable subjects who receive delayed feedback to achieve better posttest scores than those who receive immediate feedback.
Interestingly, various researchers have posited that the timing of feedback (immediate vs. delayed) that best aids learning depends on multiple factors. For example, Clariana et al. (2000) predict that delayed feedback is more beneficial than immediate feedback for learning of simple items, whereas the opposite is true for learning of difficult items. Other researchers have hypothesised that immediate feedback better benefits lower order learning (e.g., memorising facts) than delayed feedback, whereas delayed feedback better benefits higher order learning (e.g., application of knowledge to novel scenarios or transfer; Shute, 2008; Van der Kleij et al., 2015). Although to our knowledge there is no particularly strong support for these hypotheses, some studies have produced non-significant trends that are in line with these predictions (e.g., Clariana et al., 2000).
One particularly noteworthy example that is in line with this latter hypothesis comes from Mullet et al. (2014), wherein students completed class assignments that consisted of ecologically valid, complex concepts; half of the students were given feedback immediately after each assignment was due, whereas the other half were given feedback 1 week after the assignment deadline. All students were then given exams that tested the concepts that were covered in the assignments, but which required students to apply their knowledge about those concepts in a novel way to novel scenarios (i.e., transfer). Mullet et al. found that students who received feedback after a longer delay achieved better exam performance than students who received feedback after a shorter delay.
Although this finding is promising, Mullet et al. (2014) stress that it should be interpreted cautiously, as the study was conducted in a class setting that did not permit strict experimental control over some factors that could have influenced the observed findings. First, feedback was presented closer to the exam for students who were in the longer delay condition than for students who were in the shorter delay condition, which might favour learning and retention in the former condition (as explained above). Second, strict control over when students viewed feedback after it was made available, and how long the feedback was examined by students, could not be enforced, nor could the students be prevented from seeking their own feedback through external sources (e.g., looking up answers with the course material or asking their peers who did receive feedback to share it with them). Finally, students in both groups received a form of delayed feedback, which was either made available immediately after the assignment was due or 1 week later. As such, it is unknown how performance in these two groups would fare in comparison to a group that received feedback immediately after providing responses during learning.
In the present article, we investigate the effects of immediate versus delayed feedback while controlling for the duration and exposure to the feedback itself, as well as the time interval between feedback and final test. We also examine the extent to which each of these conditions facilitates learning in comparison to a baseline control group. Although many studies on this topic do not typically include a control group, doing so might improve the interpretability of a null result, as a control group can be used to ascertain the extent to which subjects in the feedback conditions are able to learn. Specifically, if no posttest differences are observed between the feedback conditions, but these subjects outperform a control group, one might conclude that both types of feedback have a similar effect on learning. However, if no posttest differences are found among subjects in these conditions, this would point to a general lack of learning and might suggest that the results are restricted to the specific materials or procedures that were used in the study. In such cases, a null finding may simply be due to subjects not being able to learn the material (i.e., a functional floor effect). This type of result would thus not be particularly useful for determining whether immediate and delayed feedback produce differential learning.
We report three laboratory experiments in which subjects were asked to learn ecologically valid, complex concepts on research methodology. During training, subjects in the feedback conditions were quizzed and asked to distinguish true from non-true experiments; some of these subjects were shown feedback after each response, whereas others were shown feedback after a given delay. Later, all subjects completed a final test consisting of novel experimental scenarios that required them to transfer the knowledge they acquired during training. This posttest thus assessed the extent to which subjects were able to learn and comprehend the concept of a true experiment.
Experiment 1
In Experiment 1, all subjects were first given a short tutorial on research methodology. Subjects in the feedback conditions were then presented hypothetical scenarios about experimental studies and were asked to determine whether each was a true experiment. True experiments consist of at least one variable that is manipulated by the experimenter, experimental control such that all non-manipulated variables are held constant (i.e., lack of confounds), and random assignment of subjects to conditions. Figure 1 shows an example of a true and non-true experiment scenario. Some subjects were shown the correct answer immediately after responding to each scenario (Immediate Feedback), whereas others were shown this feedback after responding to all of the scenarios (Delayed Feedback). After training, all subjects were tested on a novel set of hypothetical scenarios (without any feedback) so as to examine the extent to which subjects could apply the concepts from training to novel scenarios.
An example of a (a) true and (b) non-true experiment scenario. The scenario in (a) includes random assignment, manipulation of a variable, and is free from confounds, whereas the scenario in (b) includes a confound.
Method
Participants
A total of 111 subjects participated for course credit in an introductory psychology course at Iowa State University. All of the studies presented in this article were approved by the institutional review board at Iowa State University.
Materials
All stimuli and instructions were presented on a 24-in. high-definition monitor on a black background at the centre of the screen and subjects used a keyboard to enter their responses. All of the stimuli were taken from Corral and Carpenter (2020).
Pretest quiz
A pretest quiz was used to assess subjects’ prior knowledge on research methodology, which covered concepts such as confounds, experimental control, and causal inference. This quiz consisted of eight multiple-choice questions with five answer options (e.g., a variable that is not controlled and covaries with an independent variable is known as a (n): (a) dependent variable, (b) quasi-independent variable, (c) quasi-dependent variable, (d) confounding variable, and (e) uncontrolled variable).
Question types
There were three question types that were used in this study: (a) training questions, (b) classification questions, and (c) application questions. The training questions were used for the training portion of the study and the other two question types were used for the posttest. There were 12 training questions; half were true experiments and half were non-true experiments (see Figure 1 for an example of each type of scenario). The non-true experiment scenarios for each question type consisted of two scenarios that had a confound, two scenarios that lacked random assignment, and two scenarios that lacked a manipulation.
The posttest consisted of novel hypothetical scenarios that were not presented during training. On the first half of the posttest subjects were required to determine whether a given scenario was a true experiment or not, as they did during training. We refer to these posttest items as classification questions. The classification questions consisted of 12 scenarios, half of which were true experiments and half of which were non-true experiments (two that contained a confound, two that lacked random assignment, and two that lacked a manipulation).
The second half of the posttest involved application questions, which tested how well subjects understood the material, as they were required to use the knowledge that was covered in training and apply it in a way that had not been explicitly tested during training. These questions were multiple-choice (with options ranging from a to e) and only consisted of non-true experiment scenarios. Application questions were partitioned into two question types: (a) six diagnostic questions and (b) six reparative questions. Diagnostic questions required subjects to determine what was wrong with a given experiment scenario and reparative questions required them to specify how to fix it (see Figure 2 for an example of each of these question types).
An example of a (a) diagnostic and (b) reparative question.
It is important to note that both classification and application questions involve transfer of knowledge, because these question types consist of novel scenarios that comprise different surface and contextual features than the training questions. For classification questions, subjects must comprehend the concept of a true experiment well enough to reason about the elements within a given scenario to determine whether it meets the criteria of a true experiment. For application questions, subjects must know and comprehend threats to internal validity well enough to apply that knowledge to novel scenarios and identify (diagnostic questions) or fix (reparative questions) those issues. Thus, to correctly respond to these question types, subjects must transfer the knowledge they acquired during training to novel problems, wherein they must apply and reason about this knowledge in a dynamic manner that is specific to the constraints of each novel scenario.
It is important to note that subjects did not receive explicit training on identifying specific threats to internal validity. Thus, the application questions amount to a more distal form of transfer than the classification questions. Successfully applying and transferring knowledge in ways that have not been explicitly trained can be fairly challenging for learners and often fails to occur altogether (Barnett & Ceci, 2002; Detterman, 1993; Melby-Lervåg & Hulme, 2013). For this reason, the application questions were used as a secondary dependent measure.
Design and procedure
Subjects were randomly assigned to one of two groups: immediate feedback (n = 54) and delayed feedback (n = 57). First, subjects were given an eight-item multiple-choice pretest quiz on research methodology. Next, subjects were given a 7-min tutorial on research methodology and were presented five PowerPoint-style slides (one at a time) to study. The study time for each slide was self-paced and subjects were shown a prompt instructing them to press “N” to view the next slide or “B” to view to the previous slide; a counter was presented at the bottom of the screen to inform subjects which slide they were on (e.g., Slide 4 out of 5). The first slide covered independent and dependent variables; the second and third slides covered true experiments, non-experimental studies, correlations, and causal inference problems; the fourth slide covered confounds and experimental control; and the fifth slide covered random assignment. If subjects attempted to move past the fifth slide before 7 min, they were shown the study time that remained and were instructed to continue to study for that duration.
Training session
After 7 min, the tutorial screen was cleared and subjects in the feedback conditions were given a training task. Subjects were presented training questions (one at a time), which consisted of hypothetical experiment scenarios and were shown a prompt instructing them to indicate whether each was a true experiment by pressing “Y” if the scenario was a true experiment or “N” if it was not a true experiment.
Feedback timing
Feedback was identical for subjects in both conditions and only varied based on when it was presented. For subjects who received immediate feedback, after they entered a response for a given scenario, the screen was cleared and the scenario was presented again along with the correct answer, which was presented in green text directly beneath the scenario (e.g., This is a true experiment). For subjects who received delayed feedback, after they entered a response for a given scenario the screen was cleared and a “Thank You” prompt was presented at the centre of the screen for 500 ms. After these subjects responded to all of the training scenarios, each scenario was presented again (one at a time) along with the correct answer.
During the feedback presentation in both the immediate and delayed feedback conditions, each scenario and the corresponding feedback were presented together and remained on the screen for 30 s. After 30 s, subjects were shown a prompt instructing them to press the spacebar when they were ready to continue, after which the screen was cleared for 800 ms. Following the training trials and feedback, subjects completed the posttest.
Posttest. The first half of the posttest consisted of classification questions and subjects were required to indicate whether each scenario was a true experiment or not (as in the training session). The second half of the posttest consisted of application questions (comprising diagnostic and reparative questions) and for each item subjects were instructed to select the correct multiple-choice option by typing in the corresponding letter option on the keyboard; because application questions were used as a secondary dependent measure, they were presented after the classification questions. For all posttest items, after subjects entered a response the screen was cleared for 800 ms. The order that all pretest quiz, training, and posttest items were presented was randomised for each subject.
Results and discussion
Performance between the diagnostic and reparative questions was almost identical. For this reason, these two question types were combined into a single measure. Cronbach’s alpha was .420 and .610 for the classification and application questions, respectively. 1 No performance differences were found between the two feedback groups on the pretest quiz (p = .807). This factor was therefore not included as a covariate in the analyses reported below.
Immediate versus delayed feedback
Figure 3 shows the mean performance on the training, classification, and application questions for subjects in each feedback condition in Experiment 1. A mixed analysis of variance was conducted (ANOVA), with condition as a between-subjects factor (immediate vs. delayed feedback) and question type as a within-subjects factor (training vs. classification vs. application). The results revealed no main effect of feedback condition and no interaction between feedback condition and question type (both ps > .379). 2 Thus, no differences were observed between the feedback conditions on any of the question types (all ps > .319).
Mean proportion correct on the question types along with the standard error of the mean in each condition in Experiment 1.
Although these results suggest that both immediate and delayed feedback equally benefit learning, further research is needed to better interpret these results. One possibility is that the materials that were used in the present study were too challenging, and as a result subjects failed to learn. Thus, it is possible that the null findings reported here were due to a floor effect. To address this possibility, a second experiment was conducted.
Experiment 2
Experiment 2 was identical to Experiment 1, except that the feedback during training was modified to include detailed explanations. Specifically, some subjects were shown explanatory feedback—the correct answer along with a detailed explanation—immediately after responding to each scenario (Immediate Feedback), whereas others were shown this feedback after responding to all of the training scenarios (Delayed Feedback); explanatory feedback was used over correct-answer feedback to maximise learning and thereby reduce the likelihood of a floor effect, as previous work has shown that explanatory feedback is more effective for concept learning than correct-answer feedback (Butler et al., 2013; Corral & Carpenter, 2020).
In addition, to determine whether learning occurs in the feedback conditions, a control condition was included in which subjects completed the training tutorial but did not see any of the experiment scenarios during training. This control condition allows us to directly assess whether subjects in the feedback conditions are able to learn the study material. If subjects in the feedback conditions are indeed learning the material, then they should outperform control subjects on the posttest. However, if the null findings between the two feedback conditions in Experiment 1 were due to a floor effect, wherein subjects simply failed to learn, no posttest differences should be expected to emerge between the feedback conditions and the control group.
Method
A total of 171 subjects participated in this experiment for course credit in an introductory psychology course at Iowa State University. Subjects were randomly assigned to one of three conditions: immediate feedback (n = 57), delayed feedback (n = 58), and control (n = 56). Feedback presentation was identical to Experiment 1, except that for each scenario, a detailed explanation was shown in green text, directly under the scenario’s description (see Figure 4). As in Experiment 1, the feedback for each scenario remained on the screen for 30 s. Subjects were instructed to read the scenario and feedback carefully and were notified that they would be prompted when it was time to move on. After the training trials and feedback (or immediately after the training tutorial for subjects in the control group), subjects completed the posttest. All other procedures were identical to Experiment 1.
An example of a (a) true and (b) non-true experiment scenario, along with explanatory feedback (shown in green text).
Results and discussion
Because performance on the diagnostic and reparative questions was almost identical, these two question types were combined into a single measure. Cronbach’s alpha was .562 and .683 for the classification and application questions, respectively. A non-significant trend was found in pretest quiz performance between the feedback conditions, wherein subjects in the immediate feedback condition (M = .577, SE = .024) performed numerically better than subjects in the delayed feedback condition (M = .524, SE = .023), t(113) = 1.57, p = .118, d = .295. Because pretest quiz performance did not differ reliably between conditions, it was not included as a covariate in subsequent analyses.
Immediate versus delayed feedback
Figure 5 shows the mean performance on the training, classification, and application questions for subjects in each feedback condition in Experiment 2. A 2 × 3 mixed ANOVA was conducted, with condition as a between-subjects factor (immediate vs. delayed feedback) and question type as a within-subjects factor (training vs. classification vs. application). A marginal effect of feedback condition was observed, F(1, 113) = 3.91, p = .050, MSE = .066,
Mean proportion correct on the question types along with the standard error of the mean in each condition in Experiment 2.
Receiving immediate feedback therefore seemed to aid learning during training relative to not receiving any feedback, as subjects in the delayed feedback condition did not receive feedback until after they responded to all of the training questions. However, once subjects in the delayed feedback condition were presented feedback, they were able to learn the material as well as subjects who received immediate feedback, as evidenced by the similar level of performance that was observed between these conditions on the classification and application questions. That is, the immediate feedback advantage during training disappeared once subjects in the delayed feedback condition received feedback, suggesting that both types of feedback benefit learning equally well.
Feedback conditions versus control condition
To further examine whether the null findings on the posttest (i.e., classification and application questions) between the feedback conditions were driven by subjects in both conditions learning the material equally well, we conducted a mixed ANOVA with condition as a between-subjects factor (feedback conditions vs. control condition) and posttest questions as a within-subjects factor (classification vs. application).
A main effect of condition was observed, F(1, 169) = 5.85, p = .017, MSE = .062,
Nevertheless, these findings indicate that subjects in the two feedback conditions were indeed able to learn the materials (otherwise no performance differences on classification questions should have been observed between subjects who received feedback and control subjects). Taken together, the findings presented here suggest that subjects in the two feedback conditions are learning the material equally well.
It is important to note, however, that the feedback conditions differed from the control condition in numerous ways, besides feedback. Specifically, subjects in the feedback conditions viewed examples of experiment scenarios and completed a classification task during training (in addition to receiving feedback on those classification judgements), whereas control subjects did not engage in any of these activities. One possibility is that although learning occurred in the two feedback conditions, it was not due to feedback itself but to one of these other factors. Indeed, the control condition in Experiment 2 was not designed to answer this specific question. However, we do note that in recent work, Corral and Carpenter (2020; Experiment 1) used the present set of materials and compared an explanatory feedback condition (identical to the immediate feedback condition in Experiment 2) with a training condition that was identical, but which did not receive feedback. Corral and Carpenter found that subjects in the feedback condition were better able to learn these materials on a posttest than subjects who did not receive feedback. This study thus directly isolated feedback as a learning mechanism, and its results provide direct evidence that explanatory feedback indeed facilitates the learning of the materials used here.
In sum, the findings across the first two experiments are not consistent with the trends reported in the literature showing that immediate feedback is more beneficial than delayed feedback for learning difficult, complex material (Clariana et al., 2000). It is important to note that Experiments 1 and 2 involved a fairly short interval between subjects’ responses and feedback, however. Critically, short delays in feedback might not reflect how longer delays affect concept learning. To the extent that the benefits of delayed feedback depend upon a longer delay interval (Butler et al., 2007), the relatively short delay provided in the first two experiments may not have been sufficient to allow the mechanism(s) that might underlie a delayed feedback advantage to fully take hold.
Experiment 3
Experiment 3 therefore explored the effects of immediate versus delayed feedback under conditions in which the delayed feedback interval was extended. In Experiments 1 and 2, the retention interval between the feedback conditions was not controlled, as we followed a similar conceptual design to previous studies on this topic (e.g., Butler et al., 2007; Mullet et al., 2014). One possibility is that this difference might have counteracted the benefits of each type of feedback. Specifically, the design of the first two experiments was such that subjects in the delayed feedback condition were shown feedback closer to the posttest than subjects in the immediate feedback condition. Thus, it is possible that learners were better able to retain the knowledge they acquired through feedback when feedback was delayed than when it was immediate, because the former occurred closer to the posttest than the latter. However, on the contrary, it can be argued that the timing of the posttest favours learning in the immediate feedback condition, as subjects have more time to think about, process, and reflect on the content and feedback from the learning phase than subjects in the delayed feedback condition, who are instead presented feedback and are shortly thereafter given a posttest. Thus, it is possible that when the retention interval is not controlled, these elements counteract one another and produce a null result. Experiment 3 was designed to address these possibilities.
Experiment 3 consisted of three parts, each spaced 2 days apart. Subjects in the immediate feedback condition completed training and feedback during the first session, followed by the posttest during the second session, and then were given an unrelated task on the third session. Subjects in the delayed feedback condition completed training without feedback during the first session, followed by feedback on the second session, and then completed the posttest on the third session. Thus, each group completed the posttest exactly 2 days after receiving feedback. Based on Experiment 2 showing a significant advantage for the feedback groups over the control group, and given that our primary interest was between the immediate versus delayed feedback groups, we did not include the control condition in Experiment 3.
Method
A total of 77 subjects at Iowa State University participated in Experiment 3 and were randomly assigned to either the immediate or delayed feedback groups. Each subject was paid US$20 for completing all three sessions. Subjects were paid US$5 at the end of each session and were given an additional US$5 bonus for completing all three sessions. At the end of the first two sessions, subjects were reminded to return for the next session in 2 days. Subjects who missed one session were disqualified from continuing in the study, which resulted in five subjects being removed. An additional three subjects were removed for scoring over 2.5 standard deviations below the mean on the posttest, leaving 69 total subjects in the immediate feedback group (n = 36) and delayed feedback group (n = 33). With the exceptions noted in this and the previous section, Experiment 3 was identical to Experiment 2.
Results and discussion
Performance on the diagnostic and reparative questions was again nearly identical, so these two question types were combined into a single measure. Cronbach’s alpha was .367 and .710 for the classification and application questions, respectively. No differences in pretest quiz performance were found between the feedback conditions (p = .795). Performance on the pretest quiz was therefore not included as a covariate in the following analyses.
Figure 6 shows mean posttest performance partitioned by question type for subjects in each condition in Experiment 3. A 2 × 3 mixed analysis of variance was run, with condition as a between-subjects factor (immediate vs. delayed feedback) and question type as a within-subjects factor (training vs. classification vs. application). A marginal main effect of feedback condition was found, F(1, 67) = 3.27, p = .075, MSE = .054,
Mean proportion correct on the question types along with the standard error of the mean in each condition in Experiment 3.
General discussion
The present experiments address a topic that is important to learning and education but is currently not well-understood from the existing literature. Although there has been some work on immediate versus delayed feedback, many of these studies consist of methodological issues that favour learning in one condition over the other, such as exposing subjects in one condition to more feedback than the other condition, requiring subjects to view feedback in one condition but not the other, or varying when subjects in each condition are shown feedback and when they are tested. It is thus not surprising that the literature on this topic is littered with contradictory findings, with some studies showing that immediate feedback leads to better learning and retention than delayed feedback (e.g., Pressey, 1950) and other studies showing the opposite (e.g., Mullet et al., 2014; Swindell & Walls, 1993).
In the present article, we examined immediate versus delayed feedback across three experiments and carefully controlled for these methodological issues. For instance, all subjects were always shown the same type of feedback for the same duration, and in Experiment 3 all subjects completed the posttest 2 days after they were presented feedback, thus the interval between when feedback was presented and when the posttest was completed was held constant between conditions.
In Experiments 1 and 2 subjects were either presented feedback after each response or after they responded to all of the training scenarios. However, no differences in the classification or application questions were observed between the feedback conditions. Critically, in Experiment 2, subjects in the feedback conditions outperformed control subjects on the classification questions, indicating that feedback subjects were indeed able to learn the material. Taken together, these results suggest that the null findings between the feedback conditions in Experiments 1 and 2 were not due to floor effects.
However, in Experiments 1 and 2 the interval between when subjects were given feedback and when they completed the posttest was shorter when feedback was delayed than when it was immediate. On the one hand, this difference might benefit the delayed feedback condition because these subjects were exposed to feedback right before the posttest, and thus what was learned from feedback might be more accessible to them than to subjects who receive immediate feedback. On the other hand, however, this difference might benefit the immediate feedback condition because these subjects have more time to think about and process feedback than subjects who receive delayed feedback. As a result, these differences might have counteracted the benefits of either type of feedback.
Experiment 3 therefore held this interval (i.e., when feedback was provided and when subjects completed the posttest) constant between conditions, which led to a delayed feedback advantage on both classification and application questions. Thus, when the two conditions were rigorously compared by holding all other factors constant, longer delays in feedback appear to be more potent than shorter delays for concept acquisition.
It is important to note that the present set of experiments required subjects to learn ecologically valid, complex concepts. Specifically, subjects were asked to distinguish true from non-true experiments and were then required to apply and transfer that knowledge to novel scenarios (classification questions) in ways that were not explicitly tested during training (application questions). Thus, the present findings show that delayed feedback leads to better concept learning and transfer than immediate feedback. These results are in line with those from Mullet et al. (2014) and with the idea that delayed feedback produces better higher-order learning than immediate feedback (Shute, 2008; Van der Kleij et al., 2015). However, to our knowledge, previous work has only found modest support for this hypothesis in the form of numerical trends that have not reached statistical significance (e.g., Shute, 2008; Van der Kleij et al., 2015). Mullet et al. (2014) used a classroom-based design and demonstrated a delayed feedback advantage with complex materials, but in doing so were unable to control for various factors due to the real-world class in which the study was conducted (as discussed in the ‘Introduction’). Our findings build on this work and add further support to the idea that delayed feedback produces superior higher order learning than immediate feedback, as we demonstrate this effect under experimentally rigorous conditions (Experiment 3).
Although Experiment 2 revealed an immediate feedback advantage on the training questions, this advantage during training did not occur in Experiments 1 and 3. It appears this effect in Experiment 2 reflects random fluctuations in performance during training, as the immediate feedback advantage did not occur on the first training trial (p = .497) and only occurred on a few trials in an apparently non-systematic fashion. No such advantage occurred in Experiments 1 and 3, or in a previous experiment that used the same learning materials and a similar training procedure. Specifically, Corral and Carpenter (2020, Experiment 1) had subjects classify examples of true and non-true experiments during training. One group received explanatory feedback after each response (as in the immediate feedback condition) and the other group received no feedback (as in the delayed feedback condition). Corral and Carpenter found no performance differences on the training questions between subjects who received explanatory feedback (M = .669, SE = .027) and subjects who received no feedback (M = .638, SE = .028), t(65) = .785, p = .435, d = .195. The same results occurred in a pilot study that was conducted prior to the studies reported in this article, wherein the feedback procedure during training was identical to the current Experiment 2 and revealed no performance differences between the immediate (n = 21, M = .611, SE = .048) and delayed (n = 20, M = .704, SE = .032) feedback groups, t(39) = −1.604, p = .117, d = −.514.
Thus, the performance differences during training in Experiment 1 appear to be neither systematic nor reliable. It appears that a key factor underlying the delayed feedback benefit is the delay interval of the feedback, rather than performance during training per se. All three experiments presented here showed a consistent pattern, in which the benefits of delayed feedback do not occur when feedback is delivered within the same session as training and the posttest is immediate (Experiments 1 and 2), but the benefits of delayed feedback do occur when both feedback and the posttest are delayed by a matter of days (Experiment 3). Below we discuss the implications of this pattern of results for potential theoretical mechanisms underlying the benefits of delayed feedback.
Learning mechanisms of delayed feedback
Interference perseveration hypothesis
The interference perseveration hypothesis (Kulhavy & Anderson, 1972) has been proposed as a theoretical mechanism to account for a delayed feedback advantage. Formulated on the basis of studies using fairly simple materials such as cued recall of word pairs, this hypothesis posits that delayed feedback leads to better learning and retention than immediate feedback because the delay helps learners forget their incorrect responses from the learning phase. When subjects make an error (e.g., retrieve the incorrect target from the cue) and then receive correct answer feedback, those errors are more likely to interfere with encoding the correct answer when feedback is immediate, in which case the error and the correct answer are active at the same time and could compete with one another.
Spacing effects
In contrast, a delayed feedback advantage might be explained by spacing effects. The spacing effect refers to the robust finding that two or more learning episodes that are spaced over time lead to better learning and retention than the same episodes that occur back to back, or massed (for recent reviews, see Carpenter, 2017; Gerbier & Topppino, 2015). When subjects answer a test question and then receive feedback, they have had two opportunities to engage with the material. When feedback occurs immediately, the two opportunities (first, answering the test question and then seeing the answer through feedback) occur back to back and can be considered massed. When feedback is delayed, however, the two opportunities are separated in time and can be considered spaced. The advantage of delayed feedback over immediate feedback, therefore, could be driven by the same mechanisms that underlie the spacing effect.
Various mechanisms have been proposed to explain the spacing effect. According to the deficient processing account, the attention devoted to items declines more rapidly across repeated items that are massed compared with spaced, due to the high familiarity of massed items (e.g., Delaney et al., 2012; Hintzman et al., 1973). According to the encoding variability account, because of the extra time that passes in-between spaced repetitions relative to massed repetitions, spaced repetitions are more likely to be associated with a variety of contextual features that benefit memory (Glenberg, 1979). In support of this account, some studies have shown that changing the context of repeated presentations weakens or eliminates the spacing effect. For example, Glover and Corkill (1987; see also Krug et al., 1990) found that the spacing effect in memory for paragraphs was significantly reduced when the paragraphs were changed slightly, rather than repeated verbatim, across multiple presentations. Finally, a study-phase retrieval account has also been proposed (Benjamin & Tullis, 2010), wherein spacing benefits memory because subsequent spaced presentations (compared with massed presentations) are more likely to involve retrieval of the previous item, and thereby benefit memory through the well-documented benefits of retrieval practice (e.g., Dunlosky et al., 2013).
Importantly, these theoretical accounts have been derived primarily from studies involving memory for direct repetitions of the same stimuli. It is not immediately clear how these existing theories would account for the benefits of delayed feedback in the current study involving more complex materials in which direct repetitions of the same stimuli never occur. Although spacing effects do occur for more complex types of learning that involve transfer and comprehension (e.g., see Vlach, 2014; Vlach et al., 2012, 2008; Zulkiply et al., 2012), these results too would suggest modifications or alternatives to existing theories of spacing. We turn next to a potential theoretical proposal that may account for the present results.
Explaining concept learning and transfer
To learn a given concept, it is presumably necessary to learn the properties that define it (Corral & Jones, 2014). For example, consider the stimuli in the present article where subjects were shown hypothetical study scenarios during training and a novel set of scenarios during the posttest. Although theories based on interference perseveration or spacing might explain why subjects who receive delayed feedback would better recall a given scenario or correct answer from training than subjects who received immediate feedback, neither theory explains why retrieval of such information would aid subjects in classifying novel scenarios (i.e., transfer). Such classification seemingly relies on learners abstracting and comprehending the principles of a true experiment and then applying that knowledge to a given scenario. As such, successfully retrieving a previous scenario or response from the learning phase would appear to be insufficient for concept learning and transfer to occur.
These theories could explain why delayed feedback might lead to better memory retention than immediate feedback, but do not seem to account for why delayed feedback might better benefit transfer that involves concept learning and comprehension. Thus, there is not presently a cohesive theory that accounts for why delayed feedback would lead to better concept learning and transfer than immediate feedback. Although it is beyond the scope of the present article to fully develop and test such a theory here, we offer one potential explanation to account for a delayed feedback advantage.
Recent work suggests that incorrect hypotheses about a to-be-learned concept can interfere with subsequent learning, even when the to-be-learned concept is fully learned (Corral et al., 2020; Corral & Jones, 2020). One possibility is that when learners are initially learning a given concept, they form incorrect hypotheses about it. When feedback is immediate, these hypotheses might proactively interfere with processing that feedback. This interference is important, because previous work has shown that explanatory feedback can highlight the properties that define the to-be-learned concept and can thereby aid learning and transfer (Butler et al., 2013; Corral & Carpenter, 2020). However, when feedback is delayed, these hypotheses might be forgotten. As a result, learners might be able to better process and comprehend feedback when it is delayed than when it is immediate.
This explanation might account for why the benefits of delayed feedback appear to be stronger when the delay interval is increased (Butler et al., 2007), as incorrect representations that learners form about the to-be-learned concept might still be active during shorter feedback intervals and might consequently interfere with the processing and comprehension of feedback. In line with this proposal, a delayed feedback advantage was observed in Experiment 3, once the feedback interval was extended, but not in Experiments 1 and 2, which involved a far shorter feedback interval. Indeed, the delay between testing and feedback was only a few minutes in Experiments 1 and 2 (after subjects responded to all of the training scenarios), which might not have been enough time for the incorrect hypotheses that subjects may have formed during the learning phase to decay from memory, whereas this delay was 2 days in Experiment 3. These results thus provide preliminary support for this idea. Nevertheless, follow-up work will be required to further test and refine this proposal.
Of course, it is important to stress that learning still likely occurs when feedback is immediate. Indeed, immediate feedback is used in many of the studies that have shown the benefits of feedback on learning (e.g., Benassi et al., 2014; Butler et al., 2013; Corral et al., 2019). In a recent paper, Corral and Carpenter (2020) used the same materials as in the present set of experiments and demonstrated that immediate feedback leads to better concept learning than no feedback. Moreover, many studies on category learning typically involve feedback after each response (i.e., immediate feedback), and subjects are nevertheless able to learn (e.g., Corral et al., 2018; Corral & Jones, 2014, 2020). When these findings are coupled with the present results, it seems to follow that both immediate and delayed feedback aid learning. Immediate feedback might thus simply hinder or slow learning (relative to delayed feedback) but learning still likely occurs under both types of feedback.
Future directions
The present findings raise important questions that future research should seek to address. Although Experiment 3 revealed a delayed feedback advantage, this effect occurred with explanatory feedback and it is an open question as to whether this finding holds with other types of feedback (e.g., correct-answer feedback). In addition, in the present set of experiments subjects were tested on concepts from research methodology, but future work is necessary to ensure that the present findings extend to complex concepts from other domains as well (e.g., mathematics, physics, chemistry, biology). From an applied perspective, it is also important to examine whether the present findings translate to a real classroom environment over the course of an entire semester.
Conclusion
In conclusion, the present article finds that when the feedback interval between immediate and delayed feedback is not controlled, both types of feedback seem to aid learning equally well. However, once this interval is held constant, we find evidence of a delayed feedback advantage on a complex concept learning task. Although there does not appear to be a present theory that can directly explain a delayed feedback advantage for concept learning and transfer, we hope that the present results and discussion will help spur further thinking about spacing effects and how they might interact with feedback and concept learning.
From a practical perspective, the current results also shed important light on the timing of feedback in educational situations. In particular, they suggest that teachers need not be concerned with potential deleterious consequences of failing to provide feedback to students immediately while they are learning something new. In real classroom situations, particularly with learning complex concepts, teachers often need the time to read and evaluate students’ work and provide relevant comments and corrections, necessitating a matter of days before students might receive this feedback. The current results suggest that this practice is not harmful, and in fact appears to be beneficial, for learning complex concepts of the type that students would encounter in a real course.
Footnotes
Author’s note
Portions of this work were presented at the 91st Annual Meeting of the Midwestern Psychological Association, Chicago, IL.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the James S. McDonnell Foundation 21st Century Science Initiative in Understanding Human Cognition, Collaborative Grant No. 220020483. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the McDonnell Foundation.
Data accessibility statement
None of the experiments were preregistered and all of the data will be made available upon request.