Do Multiple Homework Attempts Increase Student Learning? A Quantitative Study

Abstract

Learning must inherently involve practice. It is true for athletes, for musicians, for almost any area of learning. However, this commonsense approach stops short when it comes to student homework where the debate about whether to allow multiple attempts continues. Proponents claim the value of practice and learning by reworking. Opponents cite grade inflation, student guessing behaviors, and superficial learning rather than true mastery. Meanwhile, the increased use of online homework management systems that easily allow for multiple graded attempts intensifies the need for a data-based answer to the question. It is further complicated by the increase in adult learners and online education. Would multiple homework attempts increase student learning? Would the effect be the same for adult learners in a fully online environment as found in previous studies of traditional students? This natural experiment divided a sample of 917 online students in an entry-level university economics course into two groups to look at the relationship between multiple graded homework attempts and exam scores. The results show mean exam scores increased from 60.34% to 70.77%, when multiple homework attempts were allowed. In addition, 76.89% of variance in exam scores was explained by variance in homework scores. This strong relationship suggests that multiple graded homework attempts increased student learning as measured by the increased exam scores.

JEL Classifications: A22, C93, D01, 121

Keywords

multiple attempts homework student performance online homework management systems entry-level economics student learning

Introduction

Learning must inherently involve practice. No one would expect an athlete or a musician to perform perfectly without practice. This commonsense approach stops short, however, when it comes to student homework where the debate about whether to allow multiple graded attempts continues. Proponents claim the value of practice and learning by reworking (Palocsay & Stevens, 2008; Titard, DeFranceschi, & Knight, 2014). Opponents cite artificially inflated grades, student guessing behaviors, and superficial learning rather than true mastery (Fish, 2015; Rhodes & Sarbaum, 2015). Meanwhile, the increased use of online homework management systems that easily allow for multiple graded attempts intensifies the need for a data-based answer to the question.

Previous studies are far from unanimous in their support of multiple graded homework attempts and web-based homework management systems in general. Supporters contend the flexibility of extensive practice, and immediate feedback lead to improved student performance (Arora, Rho, & Masson, 2013) as well as increased student enthusiasm and motivation (Halcrow & Dunnigan, 2012). Opposing studies found that multiple attempts improve homework grades, but not exam grades, leading to artificial grade inflation without increased learning (Rhodes & Sarbaum, 2015). The question is further complicated by the increase in adult learners and online education: Would the effect of multiple graded homework attempts be the same for adult learners in a fully online environment as found in previous studies of traditional students?

Literature Review

The term web-based homework management system is not as precise as one might hope, with multiple definitions emerging in the literature. For purposes of this discussion, web-based homework management system refers to a system that is accessible from any standard Internet browser that includes password authentication, transmission of assignments to students, collection of student answers, and automatic grading and recording (Bonham, Deardorff, & Beichner, 2003). Commonly embedded within these systems is an option to allow students multiple graded attempts at the homework assignment with feedback after each attempt. This raises the question: Do multiple graded homework attempts increase student learning or merely inflate grades?

Student learning is a complex construct, not easily measured. This study relied upon exam scores as a proxy for student learning due to the quantitative nature of the course material. This approach is in keeping with other previous studies in this line of research (Arora et al., 2013; Bowman, Gulacar, & King, 2014; Halcrow & Dunnigan, 2012; Kontur, de La Harpe, & Terry, 2015; Rhodes & Sarbaum, 2015; Titard et al., 2014).

In this context, two dominant themes emerge from the debate about the value of allowing multiple graded homework attempts: Do multiple homework attempts result in increased student learning as reflected in improved exam scores? Is any increased student learning superficial in nature, or does it include deeper understanding of the content?

Improved Exam Scores

The literature is crowded with studies showing web-based homework management systems increase student learning as measured by improved homework scores. However, the effect of multiple graded homework attempts and its relationship to increased exam scores are more ambiguous. The pedagogical goal of multiple graded homework attempts, as stated by Butler and Zerr (2005), is to increase student engagement outside the classroom by replicating the “. . . attempt-feedback-reattempt sequence of events which often occurs in a teacher’s presence . . .” (p. 51-58). The theoretical framework for this idea comes from the concept of deliberate practice, a regimen of effortful activities (deliberate practice) designed to optimize improvement (Ericsson, Krampe, & Tesch-Römer, 1993). Deliberate practice requires adequate feedback. Without that feedback, improvement is only minimal (Trowbridge & Cason, 1932).

To the extent that such increased student engagement can be accomplished by deliberate practice through multiple graded homework attempts, evidence from a broad spectrum of disciplines suggests that student learning, as measured by exam scores, may increase in some circumstances. Titard et al. (2014) found that exam scores in their undergraduate accounting course increased with multiple attempts in a web-based homework management system, but this effect was not universal. The exam score increase was greater for students who scored at least 70% on the homework, compared with students who scored less than 70% on the homework.

A similar effect was noted by Kontur et al. (2015) in undergraduate engineering classes. In their study, students using the homework management system were given five attempts on each question, with no point deduction for multiple attempts. Their research showed that successfully completing the homework correlated to increased exam scores for their high aptitude students. However, low and middle aptitude students showed no change in exam scores, or in some cases reduced exam scores. Although five attempts were allowed for each question, the study found no significant correlation between number of attempts on homework questions and exam scores.

Bowman et al. (2014) measured time spent on homework through a web-based homework management system used in an introductory chemistry class. They found a positive correlation between time spent using the web-based homework management system, and both exam grades and course grades. But they found a negative correlation between exam scores and multiple homework attempts.

Halcrow and Dunnigan (2012) looked at two groups of undergraduate calculus students studying under two different instructors. Instructor A was very enthusiastic about the web-based system with the multiple attempts, and his students showed improved exam scores with the use of the system and the multiple attempts. Instructor B was less enthusiastic, and his students showed no significant change in exam scores with the web-based system and multiple homework attempts. Halcrow and Dunnigan (2012) attributed this difference to instructor attitude transmitted to students.

These studies generally conceded that allowing multiple homework attempts tended to increase exam scores in some circumstances. However, the studies also raised questions centered around whether students truly mastered the material in the process.

Superficial Learning or True Mastery

While exam scores are a convenient measure of the level of student learning, true mastery of the material is more difficult to measure, and the literature is more mixed in its support of web-based homework management systems as a means to true mastery.

Arora et al. (2013) found that the undergraduate engineering students did better on exams in the first course in the sequence with multiple homework attempts administered through a web-based homework management system. The study followed these same students through subsequent courses in the program, and found that those who had been allowed multiple homework attempts through a web-based homework management system continued to perform better on exams. The authors concluded that this indicates a high level of concept mastery and retention.

Fish (2015) showed a similar increase in exam scores for undergraduate operations management students using a web-based homework system allowing multiple attempts. However through further investigation, the study also found this effect to be particularly strong on scaffolded questions, where the question sequence gives the logical problem development, and much weaker on open-ended questions. Fish concluded that web-based systems and multiple attempts help students learn the problem-solving method but do not sufficiently require students to develop the critical thinking skills necessary for open-ended questions. Fish suggested that this indicates a lower level of student learning with multiple graded homework attempts administered through a web-based system.

Results were similar in a study of intermediate accounting students conducted by Fatemi, Marquis, and Wasan (2015). They found that students who were allowed multiple graded homework attempts through a web-based system performed better on problems on the exam but worse on multiple-choice exam questions. The authors suggest that with multiple attempts through the web-based system, students grasp the mechanics of the problem better but do not develop the critical thinking required for multiple-choice questions that were designed to test for a deeper level of learning.

One theory as to why students may not master the material when using web-based homework management systems focuses on guessing behavior. Allowing multiple homework attempts allows students to guess at the problems without negative consequences. In a study of 100 economics students, Rhodes and Sarbaum (2015) showed that multiple homework attempts administered through a web-based homework management system sparked guessing behavior. By gaming the homework management system, students could earn higher homework scores without spending additional time on the homework. This leads to improved student homework scores without increased student learning, resulting in artificial grade inflation.

This guessing behavior was also observed by Kortemeyer (2009) and was found to be associated with gender. Through student interviews, Kortemeyer found that male students in this study of undergraduate physics students were more apt to engage in guessing behaviors than the female students. As a result, exam scores increased for female students but remained the same for male students in the study.

Kortemeyer (2015) also found that the number of homework attempts allowed affects the homework completion rate. His study of physics students looked at the difference in outcomes when students were allowed various numbers of attempts on each question. The results showed that problem solving became unproductive after the first attempt, and that an increase in the allowed number of attempts was associated with a reduction in the homework completion rate.

Method

This exploratory analysis was based on a natural experiment in which a sample of 917 online students in an entry-level university economics course was divided into two groups to look at the relationship between multiple graded homework attempts and exam scores. This study compared student outcomes in the spring semester 2015 with student outcomes in the spring semester 2016. Students were assigned to Group A or Group B based on when they took the economics class, as shown in Table 1. Changes in the curriculum dictated by the university between 2015 and 2016, not random selection, dictated the makeup of the groups. In the spring of 2016, changes to the curriculum included the addition of a web-based homework management system and the elimination of the standard textbook in favor of a custom e-book with limited summaries of each topic written by university faculty. The intent was to encourage students to rely on the web-based homework management system for practice and clarification of topics.

Table 1.

Summary of Student Groups and Dates.

Group	Dates	N
A	JAN 2015-MAY 2015	406
B	JAN 2016-MAY 2016	511

Full demographic information for every student in this study was not readily attainable. However, demographics for the university student body as a whole remained constant over this period. Furthermore, a comparison of random samples of 50 students from each group showed no statistically significant difference in the demographics of age, gender, and race between the two groups.

The homework completion rate was slightly higher for Group B compared with Group A (97.65% vs. 96.31%). This small difference is likely the result of increased motivation related to use of the web-based homework management system, as reported in previous studies (Peng, 2009; Riffell & Sibley, 2003; Zerr, 2007). However, it must be noted that this small increase in homework completion rate contradicts the findings of Kortemeyer (2015), which showed that multiple graded homework attempts correlated to reduced homework completion rates.

Group A completed homework assignments on simple Microsoft Excel templates. The templates were structured, so that the cell changed color when the correct answer was entered, so students had the opportunity to change their answers. The change in color in the cell alerted students to correct or incorrect answers but provided no additional feedback. It was the student’s responsibility to identify the error and correct it. Not all students were successful in this endeavor. The mean homework score for Group A was 77.97%, indicating that students often turned in assignments with answers that they knew were wrong. The assignments were graded upon final submission and feedback was returned a few days later, but there was no opportunity for the student to repeat the assignment. Students were allowed only one graded attempt for each assignment.

Group B used an online homework management system which allowed up to three graded attempts on each homework question. Each attempt was graded immediately. The system provided detailed feedback after each attempt, then presented the student with another similar problem. Although the problems were similar, numbers and other details were changed to allow the student a fresh problem with each attempt. The system recorded only the highest score of all attempts, so there was no penalty for multiple attempts.

Even though all students were offered three attempts at each question, only 63.55% took advantage of the second attempt and 35.94% attempted the question a third time. Some of this attrition is explained by the fact that as students get full points on a question, they do not typically continue to further attempts. However, 9.68% of students who were incorrect on the first attempt did not continue to a second attempt, and 11.31% of students who were incorrect on the second attempt did not continue to the third attempt. This may not only reflect a general lack of student effort, but it may also reflect Kortemeyer’s (2015) findings that after the first attempt problem solving becomes unproductive and was associated with a reduction in the homework completion rate.

Both Group A and Group B completed weekly exams testing their knowledge of the material covered in that week. There were a total of seven exams in each course. None of the exams for either group were cumulative. For purposes of this analysis, exam scores were combined, and average exam scores overall for the two classes were compared.

Several versions of the exam were administered to the classes, to reduce the opportunity for cheating. Test questions were changed for each of the versions to help prevent unauthorized circulation of test answers. However, questions on all the exam versions remained similar to allow for comparison. An analysis of questions across all versions, based on Bloom’s (1956) Taxonomy, showed the exams to be equivalent with approximately 25% of questions at the knowledge level, 25% at the comprehension level, and 50% at the application level. The homework consisted entirely of application-level problems. Students were required to solve problems, then answer questions about the results. These questions were predominately multiple-choice questions, along with some graphing.

The course that was the focus of this study is a fully online introductory economics course with adult learners ranging in age from 16 to more than 70 years. The class, which covers both microeconomics and macroeconomics topics, is 7 weeks long. The sample of 917 students used for this study included every student enrolled in the course from January to May 2015 and from January to May 2016.

The university from which the study sample was taken relied upon a centrally controlled curriculum for all its courses, so all course materials for all sections were identical. All courses were taught by one of two instructors. A comparison of exam scores, homework grades, and course grades found no significant difference in student outcomes for the two instructors.

The methodology for this study relied upon ANOVA and regression analysis to develop inferential statistics to allow comparison of results between the two groups. The process began with a comparison of means for exam scores and homework scores between the two groups based on descriptive statistics alone. This was followed by ANOVA to confirm that the observed difference in means was statistically significant. Finally, regression analysis was used to better define the relationship between homework scores and exam scores.

Results

This study found that student learning, as measured by exam performance, increased with multiple graded homework attempts administered through a web-based homework management system. Contrary to some previous studies, this research found no evidence of grade inflation (increased grades not supported by increased learning) as students were allowed additional attempts at the homework assignments. There was no significant difference between the mean homework scores for Group B, where students were allowed multiple graded homework attempts, and Group A, where students were allowed only one graded homework attempt. However, Group B, which was allowed multiple homework attempts, showed significantly better exam scores, and variance in homework scores explained an increased percentage of variance in exam scores. What follows is a discussion of specific results for the research question that guided this study.

Research Question 1: Is there a significant relationship between allowing multiple homework attempts and improved student learning as measured by exam scores for adult learners in a fully online environment?

As shown in Table 2, Group B, where students were allowed multiple homework attempts, showed a mean exam score of 70.77%. This compares with a mean exam score of 60.34% for Group A where students were allowed only one graded homework attempt. ANOVA shows that the difference between the mean scores for the two treatment groups is statistically significant, as shown in Table 3.

Table 2.

Descriptives Group A Versus Group B.

Exam grade		HW grade
Group A
Mean	60.34%	Mean	77.97%
Median	63.69 %	Median	90.16%
Count	406	Count	406
Group B
Mean	70.77%	Mean	79.84%
Median	75.46%	Median	88.07%
Count	511	Count	511

Note. Group A: Microsoft Excel templates with a single graded homework submission. Group B: Multiple homework attempts administered through a web-web based homework management system. HW = homework.

Table 3.

ANOVA Single Factor—Exam Scores Group A Versus Group B.

Summary
Groups	Count	Sum	Average	Variance
Group A Quiz Grade	406	244.9902	0.603424	0.048917
Group B Quiz Grade	511	361.6115	0.707655	0.04344
ANOVA
Source of variation	Sum of Squares	df	Mean Square	F	p value	F crit
Between groups	2.4579	1	2.4579	53.5910	5.4E-13	3.85164
Within groups	41.9657	915	0.04586
Total	44.4236	916

Note. Group A: Microsoft Excel templates with a single graded homework submission. Group B: Multiple homework attempts administered through a web-web based homework management system.

Homework grades remained relatively unchanged with the mean homework score for Group A at 77.97% compared with 79.84% for Group B, as seen in Table 2. This difference is not statistically significant, as seen in the ANOVA summarized in Table 4. Yet even though the homework scores did not change significantly from Group A to Group B, the relationship between homework scores and exam scores grew significantly stronger. As seen in Table 5, variance in homework scores explained 76.93% of variance in exam scores for Group B compared with less than 63.49% for Group A.

Table 4.

ANOVA Single Factor—Homework Grades Group A Versus Group B.

Summary
Groups	Count	Sum	Average	Variance
Group A	406	316.5731	0.779737	0.091502
Group B	511	407.9869	0.798409	0.051312
ANOVA
Source of variation	Sum of Squares	df	Mean Square	F	p value	F crit
Between groups	0.0788	1	0.0788	1.14149	.2856	3.8516
Within groups	63.2277	915	0.0691
Total	63.3065	916

Note. Group A: Microsoft Excel templates with a single graded homework submission. Group B: Multiple homework attempts administered through a web-web based homework management system.

Table 5.

Regression Analysis Homework Grades (IV) Versus Exam Grades (DV).

Regression statistics
Group A
Multiple R	.7974
R²	.6357
Adjusted R²	.6349
SE	0.1336
Observations	406
Group B
Multiple R	.8771
R²	.76934
Adjusted R²	.7689
SE	0.1001
Observations	511

Note. DV = dependent variable; IV = independent variable.

Discussion

The results of this study clearly indicate that student learning, as measured by exam performance, increases with multiple graded homework attempts administered through a web-based homework management system. Students allowed multiple graded homework attempts administered through a web-based homework management system earned higher mean homework scores (79.84%) and higher mean exam scores (70.77%). In addition, variance in homework scores explained a greater percentage of variance in exam scores (76.89%) for students who were allowed multiple graded homework attempts compared with students who allowed only one graded homework attempt (63.48%). This strong relationship suggests that allowing multiple graded homework attempts results in increased student learning as measured by exam performance.

Although the mean homework score for students who were allowed multiple graded homework attempts was higher, it was not significantly different from the mean for students who allowed a single graded homework attempt using simple Microsoft Excel-based templates. The difference in student performance on homework for the two groups was statistically insignificant. The fact that the mean homework score changed so little suggests that allowing multiple graded homework attempts did not stimulate guessing behaviors or result in grade inflation without underlying student learning. However the increase in mean exam scores for students with multiple graded homework attempts was significant, and homework scores explained a significantly higher percentage of variance in exam scores for this group.

This contradicts findings from previous studies that found web-based homework management systems to be associated with guessing behaviors, grade inflation without underlying student learning, and superficial learning leading to poor exam performance (Bowman et al., 2014; Fatemi et al., 2015; Rhodes & Sarbaum, 2015). Neither guessing behaviors nor grade inflation without underlying learning was seen with multiple graded homework attempts in this study, as evidenced by the fact that there was no significant change in homework scores. In this study, multiple graded homework attempts were associated with increased student learning as measured by exam scores, and a stronger relationship between homework scores and exam scores. This difference in findings may be explained by the fact that the web-based homework management system used in this study presented a different version of the problem with each attempt, reducing the opportunity for guessing. It may be further explained by the adult population from which this study sample was drawn, who may be less inclined to guessing behaviors.

The timing of the homework feedback could also explain the findings of this study. The deliberate practice theory holds that without adequate feedback, improvement resulting from practice will be minimal (Trowbridge & Cason, 1932). For both groups, the homework and the exams were graded, and feedback provided. For both groups in this online class, homework and exams were due on the same day. The difference for the two groups came in the timing of the homework feedback. Group A completed the homework on Microsoft Excel templates, submitted for grading and returned with feedback a few days later. These students had no opportunity to review homework feedback before attempting the exam. Students in Group B enjoyed multiple graded homework attempts, with full immediate feedback after each attempt. This immediate homework feedback is an essential component of deliberate practice, and as seen in the results of this study, is associated with increased student learning.

In addition, as there are no prerequisites for this course, and the university has an open admissions policy for the online student population, students come to the course with widely varying levels of preparation and ability. Allowing multiple homework attempts, with detailed feedback after each attempt, can help to mitigate those differences through deliberate practice.

Opportunities for further study in this area are many. Of particular interest are studies that look at student demographics, studies that look at time spent on homework, and studies that explore how these factors affect low-performing students compared with high-performing students.

Student demographics offers a rich source of data for further explaining the effects seen here. Gender, age, previous experience with the content area, and previous grade-point average may all be factors that contribute to the results seen in this study. Understanding how and to what extent each contributes would increase the usefulness of these findings.

The web-based homework management systems track time spent on assignments for each student. A study exploring the relationship between time spent on homework and either homework scores or exam scores would shed additional light on the findings of this research.

High-performing students are different from low-performing students in many ways. Research that explores how these differences interact with the effects seen in this study would be a valuable contribution to the field. Further understanding is needed into the role that multiple homework attempts play in each of these student groups, and how each group might be better served by adjustments to the overall system.

Conclusion

The findings of this study indicate that multiple graded homework attempts administered through a web-based homework management system are a useful tool that increases student learning for adult online learners. Given the rate of expansion of online education, these results provide a springboard for further studies to better define ways to increase student learning among the newer and increasingly important population of online adult learners.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author Biography

Kathy Archer, Assistant Professor of Economics at Grand Canyon University holds a Doctor of Business Administration (DBA). Dr. Archer teaches entry level economics classes in both the online and traditional modalities. Additionally, she has chaired four dissertation committees. Her personal research interests include teaching economics, particularly in the online environment, and wage inequality.

References

Arora

M. L.

Rho

Y. J.

Masson

(2013). Longitudinal study of online statics homework as a method to improve learning. Journal of STEM Education: Innovations and Research, 14, 36-44.

Bloom

B. S.

(1956). Taxonomy of educational objectives, Vol. 1: Cognitive domain. New York, NY: David Mckay.

Bonham

S. W.

Deardorff

D. L.

Beichner

R. J.

(2003). Comparison of student performance using web and paper-based homework in college-level physics. Journal of Research in Science Teaching, 40, 1050-1071.

Bowman

C. R.

Gulacar

King

D. B.

(2014). Predicting student success via online homework usage. Journal of Learning Design, 7, 47-61.

Butler

M. B.

Zerr

R. J.

(2005). The use of online homework systems to enhance out-of-class student engagement. International Journal for Technology in Mathematics Education, 12, 51-58.

Ericsson

K. A.

Krampe

R. T.

Tesch-Römer

(1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100, 363-406.

Fatemi

D. J.

Marquis

Wasan

(2015). Student performance in intermediate accounting: A comparison of the effectiveness of online and manual homework assignments. The Accounting Educators’ Journal, 24, 1-19.

Fish

L. A.

(2015). Undergraduate students computer-managed homework versus in-class performance for different testing formats. Business Education Innovation Journal, 7, 5-14.

Halcrow

Dunnigan

(2012). Online homework in calculus I: Friend or foe? Primus, 22, 664-682.

10.

Kontur

de La Harpe

Terry

(2015). Benefits of completing homework for students with different aptitudes in an introductory electricity and magnetism course. Physical Review Special Topics-Physics Education Research, 11, 010105.

11.

Kortemeyer

(2009). Gender differences in the use of an online homework system in an introductory physics course. Physical Review Special Topics-Physics Education Research, 5(1), 010107.

12.

Kortemeyer

(2015). An empirical study of the effect of granting multiple tries for online homework. American Journal of Physics, 83(7), 646-653.

13.

Palocsay

S. W.

Stevens

S. P.

(2008). A study of the effectiveness of web-based homework in teaching undergraduate business statistics. Decision Sciences Journal of Innovative Education, 6, 213-232.

14.

Peng

J. C.

(2009). Using an online homework system to submit accounting homework: Role of cognitive need, computer efficacy, and perception. Journal of Education for Business, 84, 263-268.

15.

Rhodes

M. T.

Sarbaum

J. K.

(2015). Online homework management systems: Should we allow multiple attempts? American Economist, 60, 120-131.

16.

Riffell

S. K.

Sibley

D. H.

(2003). Learning online. Journal of College Science Teaching, 32, 394-399.

17.

Titard

P. L.

DeFranceschi

J. E.

Knight

(2014). Using online homework to improve exam scores. Journal of Business and Educational Leadership, 5, 58-63.

18.

Trowbridge

M. H.

Cason

(1932). An experimental study of Thorndike’s theory of learning. The Journal of General Psychology, 7, 245-260.

19.

Zerr

(2007). A quantitative and qualitative analysis of the effectiveness of online homework in first-semester calculus. The Journal of Computers in Mathematics and Science Teaching, 26, 55-73.