Abstract
Background:
Most students rely on ineffective study habits to learn material instead of using evidence-based strategies known to improve learning.
Objective:
The purpose of this study was to examine the effectiveness of a novel in-class quizzing strategy (i.e., Make & Take Quizzes) that was intentionally designed to engage students in evidence-based study strategies.
Method:
Final exam grades between sections of identical courses where students completed (N = 74) or did not complete (N = 73) Make & Take Quizzes were compared. Changes in the use of study strategies from the beginning to the end of the semester in two courses where students completed the quizzes were also examined.
Results:
Students in Make & Take Quiz sections significantly outperformed those in sections without the quizzes. Students’ overall strategies for studying did not significantly change; however, there were significant improvements in the specific strategies of deciding on the main ideas when studying and putting ideas into one’s own words when studying.
Conclusion:
Make & Take Quizzes appear to help students learn course material, while also engaging them in the use of specific study strategies.
Teaching Implications:
An easy-to-implement strategy, instructors can engage students in the effective use of evidence-based study strategies by incorporating Make & Take Quizzes in their courses.
There are striking differences in the mindsets of psychologists versus students when it comes to discussing effective study strategies. While psychologists agree that cramming, highlighting, and rereading are not effective study strategies (especially for long-term retention and understanding of material; Dunlosky et al., 2013), students regularly engage in these popular behaviors and often mistake them as being effective and efficient study strategies (Bjork et al., 2013). Indeed, Putnam and colleagues (2016) describe the plight of college students who must learn a large amount of material, lack an understanding of what behaviors lead to durable learning, and are overconfident in evaluating their own learning. Of course, we cannot necessarily fault students for having this mindset; after all, many students using ineffective strategies tend to “get by” on tests and have even been taught to study in this manner by their earlier teachers, their parents, and/or their friends.
Many students become accustomed to cramming because they lack time management skills (Geller et al., 2018). They may become accustomed to highlighting because it helps them stay focused while reading, and it has the visual appearance of progress and effort. Similarly, rereading notes and textbooks gives students (false) confidence that they already know the material because it is familiar. Thus, the challenge is to convince students to shift from using these well-engrained and seemingly effective study strategies to those that are evidence-based and lead to the genuine learning of material. For instance, a rich body of empirical literature provides evidence for the effectiveness of spaced practice, retrieval practice, and elaboration, specifically, in promoting student learning (see Dunlosky et al., 2013, for a review).
Spaced practice (sometimes called distributed practice) involves reviewing material on many different occasions; studying is distributed across time and happens in a number of short sessions over a longer period (Ebbinghaus, 1885/1964; see also Delaney et al., 2010, for a review). Although many students believe that massed practice (i.e., studying all material in a single session) is more effective than spaced practice (McCabe, 2011), research indicates that even if the same amount of time is spent studying overall, spaced practice results in more material being retained over the long-term than massed practice (Cepeda et al., 2006).
Retrieval practice (sometimes called practice testing, test-enhanced learning, or the testing effect) is the act of deliberately trying to recall to-be-learned information in an effort to enhance long-term retention of the material (Roediger & Karpicke, 2006a). Retrieval practice works in two ways: It directly affects learning by enhancing students’ ability to recall information from memory, and it indirectly affects learning by alerting students to what they do and do not know (Roediger & Karpicke, 2006b). Dunlosky and colleagues (2013) rated spaced practice and retrieval practice as the two most effective study strategies in a comprehensive review of a variety of learning strategies. Of particular importance, these two strategies were rated highly because of their applicability to aid in the learning and understanding of different kinds of material and their ability to help boost student achievement.
Dunlosky and colleagues (2013) also rated elaboration as an effective study strategy with “much promise” (from Dunlosky, 2013, p. 18). Elaboration (also known as elaborative interrogation) is a broad concept that involves having students explain material in their own words and/or making connections between new material and what they already know (Pressley et al., 1987). One practice that requires elaboration is the creation of questions from recently or to-be-learned information for the purpose of studying. Called the generation effect (Slamecka & Graf, 1978), this practice has been shown to effectively enhance learning in a number of laboratory and classroom studies (e.g., Bertsch et al., 2007; Van Blerkom et al., 2006). Most recently, Kelley and colleagues (2019) describe their use of an online platform (PeerWise) specifically designed for students to create and share multiple-choice questions with their classmates. They showed that students performed significantly better on exam questions in which they had generated a question covering that topic compared to exam questions in which they had not.
In contrast, when Lloyd and colleagues (2018) directly compared the efficacy of question generation to retrieval practice as an in-class exam review strategy, they found that question generation led to lower exam grades and was perceived less favorably by students. In this study, students performed better on the exam immediately following the review session when they took part in an instructor-led guiding testing activity (answering approximately 30 instructor-created review questions) versus when they spent that session generating their own review questions (each student created three questions). As students in the question generation session did not review their questions and answers in class, but were provided the questions from all of their classmates after the session, this study appears to only show the differential effects of retrieval practice separate from elaboration (or question generation). How powerful would these techniques be if used in combination?
It is clear that these evidence-based strategies can help students succeed in their coursework and, perhaps most importantly, lead to the genuine learning of course material. But these strategies may also take more time and effort and feel harder or less efficient than other more popular studying behaviors such as cramming, highlighting, and rereading. So how can we encourage students to actually use these evidence-based strategies? Instead of describing or demonstrating effective study strategies, what if we could actively engage students in spaced practice, retrieval practice, and elaboration in the classroom? Moreover, what if we could harness the additive power of these three evidence-based study strategies?
I developed a novel, in-class quizzing strategy (Make & Take Quizzes) to address the significant gap between student perceptions of what constitutes effective study strategies and the evidence-based study strategies of spaced practice, retrieval practice, and elaboration. Make & Take Quizzes foster engagement in each of these study strategies in the classroom setting. Designed to be easy to implement for instructors and low-stakes/low-anxiety producing for students, Make & Take Quizzes intentionally combine spaced practice and retrieval practice (known as successive learning; Rawson et al., 2013) with elaboration. As the name implies, students both “make” and “take” quiz questions created by themselves and their classmates each week. The present study evaluates the effectiveness of Make & Take Quizzes by addressing two questions. First, will students who complete weekly Make & Take Quizzes score significantly higher on their final course exam than students who do not complete the quizzes? Second, do students who complete weekly Make & Take Quizzes report improved study strategies from the beginning to the end of the semester?
Method
Participants
Participants were 147 undergraduate students (111 females, 36 males) from seven sections of two different psychology courses at a private liberal arts institution in the Northeast region of the United States. In four sections (N = 74; two sections of social psychology with 40 students and two sections of health psychology with 34 students), students completed the quizzes as part of the course requirements. Students in the other three sections (N = 73; two sections of social psychology with 56 students and one section of health psychology with 17 students) did not complete the quizzes. Sections of the same course were identical in instructor, syllabus, textbook, and lecture material, and all sections were comparable in class size and composition (e.g., major, class level). All sections met in a face-to-face format for two class sessions per week.
Measures
Exam performance
The effectiveness of the Make & Take Quizzes was evaluated by comparing final exam grades across sections of both courses that did and did not include the quizzes throughout the semester. All sections of both courses were administered a final exam that was comprehensive in nature at the end of the semester. Furthermore, all sections of the same course completed an identical final exam. The final exam in each course contained a combination of multiple-choice (∼60%), short answer (∼25%), and essay questions (∼15%). The final exams were worth approximately 20% of the course grade and contained either 40 questions (health psychology exam) or 60 questions (social psychology exam). Exam grades were converted to a percentage out of 100 for analyses.
Study strategies
The 13-item Cognitive Strategy Use subscale of the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich & De Groot, 1990) was used to examine changes in the use of strategies for studying throughout the semester. Example items from this subscale include “When reading, I try to connect the things I am reading about with what I already know” and “When I study for a test, I try to put together the information from class and from the book.” Students in one section of each course that included the Make & Take Quizzes (N = 36; social psychology N = 21 and health psychology N = 15) were asked to complete the MSLQ at the beginning (prequiz implementation; α = .640) and end (postquiz implementation; α = .750) of the semester. Participants indicated their strength of agreement/disagreement on a 1 (not at all characteristic of me) to 5 (very characteristic of me) Likert-type scale for all scale items. Scale items were averaged to create a prequiz implementation score and a postquiz implementation score for each student.
Procedure
Make & Take Quizzes
During the first week of the course, all students in the courses that included the Make & Take Quizzes received a stack of approximately 30 colored index cards. Half of the index cards were “magenta” colored, and students were told that these cards would be their make cards. The other half of the index cards were “teal” colored, and students were told that these cards would be their take cards. The colors, while irrelevant to the actual quizzing procedure, were strategically chosen and named to help students remember and distinguish between the two components of the quizzes (magenta = make; teal = take).
Once per week (usually at the end of a week of course material), students were asked to review their class notes and develop two potential quiz questions from them—one multiple-choice question and one short answer question. Students were required to include the correct response for their questions. Students wrote their quiz questions on the magenta-colored make cards and turned them into the instructor who assigned a maximum of two points for complete and accurate questions. The goal of the make process was to engage students in the elaboration strategy of self-generation (the generation effect; Slamecka & Graf, 1978).
The next class, the instructor selected and displayed five student-generated quiz questions on a single PowerPoint slide. Selected quiz questions were most often those that embodied a main idea or topic from the lesson, were applied in nature, and/or presented a novel example/illustration. Strictly factual questions (e.g., “define X”), unless critical to the lesson, were usually not selected. As the student-generated questions ranged in difficulty, the instructor aimed to balance each quiz in this area. Students were given 5 min to respond to the questions on their teal-colored take cards. Once completed, the instructor collected the cards and immediately reviewed the correct answers with the entire class. The cards were returned to the students the next class period with a grade—a maximum of five points could be earned for correctly answering the quiz questions. The goal of the take process was to engage students in the strategy of retrieval practice (Roediger & Karpicke, 2006a). Together, the make and take sessions required students to engage in spaced practice (Ebbinghaus, 1885/1964) as they reviewed previously learned information twice—once to make their quiz questions and once to study for their quiz. The make and take procedure was repeated each week of the semester, on average 12 times. In total, the quizzes were worth between 7% and 8% of students’ final grade in the course.
Control condition
Students enrolled in identical courses that did not complete the Make & Take Quizzes acted as the comparison group for the analyses involving final exam grades. Sections of the same course were identical in instructor, syllabus, textbook, and lecture material. In lieu of the class time spent reviewing notes to make quiz questions and to study for the take quiz, students in the no-quiz sections participated in a brief (5–10 min), informal, instructor-led review. This informal review always took place at the beginning of class and covered the same set of material that would have been included in a Make & Take Quiz. A typical instructor-led review consisted of two–three open-ended questions shared with the class verbally, in which students were invited to respond in an informal discussion setting. The questions presented always focused on the main ideas or topics of the previous lesson and were used to transition into the new lesson for that class period. No points were awarded during the review.
Results
Exam Performance
Independent samples t tests were conducted to compare final exam grades between sections of the courses in which students did or did not complete Make & Take Quizzes. Students in the Make & Take Quiz sections significantly outperformed those in the sections without the quizzes when combining both courses, t(145) = −4.80, p < .001, 95% CI [−11.15, −4.65], d = .79, 1 as well as when considering each course individually, social psychology, t(95) = −3.39, p = .001, 95% CI [−10.35, −2.70], d = .73; health psychology, t(48) = −2.51, p = .015, 95% CI [−14.61, −1.62], d = .76. As displayed in Figure 1, the mean final exam grade of students in the combined Make & Take Quiz sections was 85.4% (SD = 10.0%; B grade) compared to 77.5% (SD = 9.9%; C+ grade) in the combined no-quiz sections. Similarly, the mean final exam grades of students in the social psychology and health psychology Make & Take Quiz sections compared to the no-quiz sections, respectively, were 83.4% (SD = 8.5%; B grade) versus 76.8% (SD = 9.9%; C grade) and 87.9% (SD = 11.2%; B+ grade) versus 79.8% (SD = 10.0%; C+ grade).
Mean final exam grade of Make & Take Quiz (N = 74) versus control sections (N = 73). Note. All comparisons were significant at or below the p < .05 cutoff. Error bars represent standard errors of the mean.
Study Strategies
Paired samples t tests were conducted to examine changes in the use of cognitive strategies for studying from the beginning to the end of the semester in one section of each course that completed the Make & Take Quizzes. Students’ cognitive strategies for studying improved overall, though not significantly, t(35) = 1.89, p = .07, 95% CI [−0.01, 0.28], d = .33; Mpre = 3.61, SDpre = 0.42; Mpost = 3.75, SDpost = 0.51. Exploratory analyses were also conducted, examining the individual scale items. These analyses revealed that students reported significant improvements in two of the 13 individual study strategies. Specifically, students reported having significantly less trouble deciding on the main ideas when studying, t(35) = −4.79, p < .001, 95% CI [−0.79, −0.32], d = .81; Mpre = 2.53, SDpre = 0.81; Mpost = 1.97, SDpost = 0.81; and were significantly more likely to put ideas into their own words when studying, t(35) = 2.67, p = .011, 95% CI [0.11, 0.78], d = .44; Mpre = 3.50, SDpre = 0.77; Mpost = 3.94, SDpost = 0.86, by the end of the semester.
Discussion
In an effort to encourage students to regularly engage in evidence-based study strategies, rather than relying on ineffective study habits, the novel in-class quizzing strategy presented in this study was developed and empirically assessed. The results of the assessment revealed that the quizzes were effective at helping students to increase their final exam grades, by as much as or more than a full letter grade, while also suggesting that students improved in their ability to use specific study strategies. Rather than merely describing or demonstrating effective study strategies, which copious amounts of anecdotal instructor evidence and some research suggests will not change students’ actual study habits (e.g., Chew, 2005), Make & Take Quizzes require students to actually engage in spaced practice, retrieval practice, and elaboration on a regular basis.
Indeed, the significant difference in end-of-semester final exam grades between the quiz and no-quiz sections is consistent with the plethora of research on the efficacy of spaced practice when it comes to the long-term retention of learned material (see Cepeda et al., 2006, for a comprehensive review). While many college students engage in “cramming”—waiting until the last minute to study for an exam—students completing the Make & Take Quizzes participated in mini-study sessions each time they reviewed their notes to either make or take a quiz (i.e., at least twice per week). These frequent quizzes also engaged students in retrieval practice, one of the most well-researched and well-established learning strategies in the cognitive and educational sciences communities (Roediger et al., 2011). Finally, the generation of quiz questions necessarily required students to elaborate upon, or self-explain, concepts in such a way that they could rework their understanding into a new or original format for their classmates. The finding that students in the Make & Take Quiz sections were significantly more likely to put ideas into their own words when studying supports the premise that the quizzes helped students learn to use elaboration to study more effectively.
Despite these promising findings, it remains unclear if students continued to utilize these study habits once the course (and the mandated “making” and “taking”) was over. In other words, how “transferable” are these skills to other courses and environments? Will students continue to make their own quiz questions and test themselves as an independent study strategy? Some research (e.g., Cho & Powers, 2019; Soderstrom & Bjork, 2014) suggests that engaging students in the experience of retrieval practice, specifically, can lead to the development of more effective study strategies. It would be beneficial to take a longitudinal approach to the study of Make & Take Quizzes to assess the long-term benefits of this in-class quizzing strategy, especially to see whether the strategies of spaced practice and elaboration continued.
Moreover, adding Make & Take Quizzes to a course does take away some class time from instruction (somewhere between 5 and 15 min at the beginning or end of a class period), so instructors may want more evidence that these quizzes are effective in the long-term before making that commitment. Instructors who teach larger sections might also find the time commitment of Make & Take Quizzes challenging, though lab sections or breakout sessions with students from larger lectures might be appropriate venues to engage students in the “making” and “taking” of their own review quizzes. Another challenge for instructors to consider is how much time and effort they want to spend outside of class selecting appropriate questions for the take quiz. Some research suggests that student-generated examples of concepts may be flawed or inaccurate (e.g., Zamary et al., 2016), thus instructors may fear the time investment required to thoroughly vet the make questions. However, given that students appear to enjoy seeing their quiz questions appear in the take component (almost as if it is a compliment), most work hard to generate questions that are thoughtful, creative, and generally of a high quality. In the end, instructors may adopt and modify this quizzing strategy to fit their teaching style, personal preferences, and any logistical constraints they may face.
Future research should continue to examine the effectiveness of engaging students in evidence-based study strategies in the classroom since a number of methodological limitations were present in this study. For example, a more equitable comparison group would provide stronger evidence for the efficaciousness of Make & Take Quizzes. An intriguing idea for a more direct comparison might involve reviewing student-generated questions and answers that have been completed in a Make & Take Quiz section each week with a no-quiz section. These students would therefore be doing the same amount of review and hearing the same information as the Make & Take students but would not necessarily be engaged in elaboration or retrieval practice. Future research should also measure student-level demographics (e.g., GPA, credits completed) to try to control for issues of nonrandom assignment that are inherent in field studies such as this. Finally, a more thorough exploration of changes in students’ study strategies as a result of completing Make & Take Quizzes for a semester is needed. A more explicit measure of evidence-based study strategies used by students, rather than a broad measure such as the MSLQ, might better address the question of whether Make & Take Quizzes can positively impact the use of specific study strategies.
Conclusion
Make & Take Quizzes offer a number of benefits to both students and instructors. From the instructor side, Make & Take Quizzes are easy to implement since students create the questions (instructors just pick the ones to include in the actual quiz), and students can even grade the quizzes themselves (either their own or a classmate’s quiz). Students rarely complain about the quizzes being too hard or too easy because they are comprised of their own questions, and students are genuinely excited to see their question on a quiz. The questions that students make are also of instructive value—examining trends in the topics of quiz questions each week allows an instructor to revisit a topical area that was misunderstood and/or redirect students when questions seem to overlook the most important concepts from a lesson. Further, beginning each class session with either a make or take task is a (surprisingly) pleasant way to ease into class and get students focused on the course material. Students actually report that they enjoy Make & Take Quizzes since they are low stakes and thus low anxiety-provoking for most students. Quizzes become about learning, rather than about grades since students are involved in the entire process. Make & Take Quizzes offer a new twist on in-class quizzing that both students and instructors can appreciate.
Footnotes
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
