Paying Attention in Class: Using In-Class Quizzes to Incentivize Student Attention

Abstract

Background:

Methods are needed to incentivize student attention to class material.

Objective:

The purpose of this study was to examine the effects of in-class quizzing to incentivize student attention to class material to boost exam performance.

Method:

A randomized, alternating treatments design embedded in an introductory psychology class compared learning benefits of two types of quiz-based engagement activities, mastery quizzes, and concepts checks, as compared to a no-engagement activity control.

Results:

Students performed significantly better on exam content linked to classes with quiz-based engagement activities. Learning benefits of engagement activities extended across levels of cognitive complexity indexed to Bloom levels, although were stronger for low-level items. The effects of engagement activities were also stronger for content directly discussed in class. There were no significant differences in learning outcomes between the two engagement activities.

Conclusion:

Students performed better on exam content linked to class sessions with quiz-based engagement activities that incentivized attention to class material compared to those that did not.

Teaching Implications:

Using in-class quizzes as attentional cues offers instructors a means of incentivizing student attention to class material with minimal disruption of class time and lecture flow, while also improving exam scores.

Keywords

classroom quizzing attention engagement exam performance

The ability to focus on task-relevant information is a key element in the learning process, as cognitive psychologists recognize that selective attention is a critical factor in memory encoding (Kuhl & Chun, 2014; Schmidt, 2001). Consider a typical classroom situation: While an instructor discusses a particular concept or topic, students are bombarded with many distracting stimuli that can impair effective learning, such as ambient noise, screen images and pings from personal devices, movements or rumblings of other students, and perhaps a leafy view outside the classroom window. The mere presence of cell phones can have distracting effects on student attention in class (Markowitz et al., 2019; Ward et al., 2017). Students in class are also prone to internal distractors, such as mind wandering, especially during long class periods. It is little wonder that instructors face the daunting challenge of capturing and holding student attention.

Drifting consciousness during class is likely to interfere with sustained attention. Many students have difficulty maintaining attention in class, especially in classes lasting 50 min or more, despite the best efforts of the most stimulating instructors. In his influential primer on teaching tips, McKeachie suggested that student attention in class is likely to drift unless instructors break up long lectures by using activities that engage student participation, such as minute papers or small group exercises (McKeachie & Svinicki, 2006). Prior research suggests that more active learning strategies can increase student attention in class (Deslauriers et al., 2019; Freeman et al., 2014; Nilson, 2010; Yazedjian et al., 2007). For example, Burke and Ray (2008) reported that the use of student-generated questions enhanced concentration. Other investigators reported that more interactive pedagogical techniques, such as the use of clickers and demonstrations, mitigated self-report attention declines in students in college chemistry classes (Bunce et al., 2010). Hlas et al. (2017) also found that specific pedagogical practices, such as discussion questions and calling on students randomly, increased self-reported student attention in class.

The present study extends the focus of research from pedagogical techniques designed to enhance student attention in class to incentivizing students to pay attention to class material to boost learning outcomes. The use of quizzes to improve long-term retention, which is often referred to as the testing effect, is well established (Karpicke & Grimaldi, 2012; Roediger & Butler, 2011; Rowland, 2014; Sotola & Crede, 2021). However, quizzes may serve other purposes, such as providing incentives for students to attend more closely to material discussed in class. There remains a need for using brief quizzes as engagement activities that can be seamlessly integrated during class with minimal disruption to class flow.

Recent research demonstrated the learning benefits of brief, incentivized engagement activities during synchronous online classes in teaching introductory psychology (Nevid et al., 2022). In this study, sporadic attendance checks and brief writing assignments improved learning outcomes relative to a no-engagement activity or teaching-as-usual control. The present study extends this work to using quizzes as incentivized engagement activities for in-person classes in introductory psychology.

The utility of in-class engagement activities may depend on how much class time they require and whether students are incentivized to participate. Instructors may be reluctant to use engagement activities that require more than a few minutes of precious class time and students may not engage in activities not tied to course grades. In the present study, in-class quizzes required only a few minutes of class time and were incentivized by providing students with course credit for correct answers. These engagement activities were designed to cue students to pay close attention in class to learn the answers to the in-class quizzes. The study compared the learning benefits of two types of quizzes, mastery quizzes, and concept checks, in comparison to a teaching-as-usual condition with no in-class quizzes.

The two types of quizzes used in the present study cued students to pay close attention to class material but in different ways. A mastery quiz gives students the opportunity to master a particular concept discussed at some point during the class period (Nevid & Mahon, 2009). The present study used a variation of mastery quizzing in which a single multiple-choice question testing knowledge of a concept discussed during the class was posted at the beginning of class and which students had the opportunity to answer at the end of class. With concept checks, students were informed at the beginning of class that the instructor would stop the class at some unannounced point during the class period and direct them to answer a single multiple-choice question testing knowledge of a concept the instructor had just been discussing at that point in class.

In the present study, mastery quizzing and concepts checks incentivized students to pay attention in class to learn the answer to the quiz question but did not cue them when during class the tested concept would be discussed. A no-engagement activity, teaching-as-usual condition served as a control within a randomized, alternating treatments design over the course of the semester.

We hypothesized that students would perform better on exam items linked to topics discussed in classes with in-class quizzes (mastery quizzes or concept checks) than classes without these engagement activities (no-quiz control). We also explored two potential moderators of these effects: (1) level of cognitive complexity of exam items as indexed to Bloom's taxonomy of educational objectives (Bloom et al., 1956); and (2) whether concepts tested by exam items were discussed directly in class.

Method

Participants

The study sample comprised 54 students in a class in introductory psychology at a large private metropolitan university in the northeastern U. S. The university serves a racially/ethnically and socioeconomically diverse population and enrolls a large proportion of international students and first-generation college students. The ethnic/racial background of undergraduate students at the university is as follows: African American/Black (n = 2,066, 10.51%), Asian, (n = 2,703, 13.75%), Hispanic/Latino (n = 3,558, 18.10%), White (n = 9,297, 47.29%), two or more races (n = 710, 3.61%), and other ethnicities or races, unknown, or not reported, (n = 674, 6.76%). A relatively high percentage (35%) of undergraduates are Pell grant recipients. As a core course that can be used to fulfill a general social science requirement, introductory psychology attracts a range of students from different majors as well as those interested in pursuing a major in psychology.

Measures

We measured learning outcomes based on student exam performance on three regularly scheduled, noncumulative, closed-book, multiple-choice exams covering chapters in the accompanying textbook along with related class material. Exams were administered online in the regular classroom setting using the Canvas course management system and accessed through Respondus® LockDown Browser, which prevents students from accessing other websites or online textbook materials.

We linked each exam item to a class period in which the general topics covered in the assigned textbook chapter were discussed. We then computed three scores for each student representing the percentage of correct answers to exam items linked to classes with either mastery quizzes or concept checks or without any accompanying quizzes (no-activity control).

We also coded exam items by Bloom level by dichotomizing Bloom's taxonomy into high and low levels of cognitive complexity. Low-level items corresponded to the first Levels 1–2 in Bloom's taxonomy, comprising questions assessing basic knowledge and comprehension, such as concept definitions and descriptions. High-level items comprised questions indexed to Levels 3 through 6 in Bloom's taxonomy, which measure the ability to apply concepts to examples, to analyze concepts, and to explain underlying processes. Each exam comprised 15 questions drawn from each chapter that represented topical areas covered in the chapter and were weighted toward higher levels of Bloom's taxonomy with 10 high-level items and 5 low-level items per chapter.

The two authors independently rated the dichotomized Bloom level of all exam items used in the analysis of engagement conditions (N = 206), which yielded near-perfect agreement (Cohen's Kappa, k = 0.86). In the relatively few cases in which raters disagreed, we used the first author's ratings in the analysis.

Procedure

An introductory psychology course taught by the first author served as the in-person classroom setting for the study. The research protocol was approved on an exempt basis by the university Institutional Review Board as it adhered to regular educational practices in the course. All course requirements including the use of in-class engagement activities and regularly scheduled exams were listed in the course syllabus distributed to all students and student responses on engagement quizzes and course exams were deidentified for data analysis purposes.

Course grades were based on a 100-point total, comprising performance on three in-class exams (50 points, or 50% of course grade), online homework assignments using the accompanying integrated learning system (MindTap; 40 points, or 40% of course grade), and the two in-class engagement activities, mastery quizzes and concept checks (10 points total, or 10% of course grade). We credited students with fractional points (.71 points) for each correctly answered question on the seven mastery quizzes and seven concept checks, which amounted to 10 points in total, rounded.

Engagement Conditions

The study compared three engagement conditions: (1) mastery quizzes; (2) concept checks; and (3) a no-engagement activity, or teaching-as-usual, control. Students were informed at the beginning of each class whether there would be an engagement activity during class and what type of engagement activity it would be (mastery quiz or concept check). Performance on mastery quizzes and concept checks was weighted equally in the student's course grade.

Mastery Quizzes

This comprised a single multiple-choice question posted on the course management system (Canvas) at the start of class testing knowledge of a concept covered during the class. Students had a 2-min time limit for answering the question using the course management system and were not permitted to collaborate or to access other materials. Students were advised at the start of class that the mastery quiz question was posted on Canvas and the concept it tested would be discussed at some point during class. However, students were not told when during class the concept tested by the mastery quiz would be discussed.

Concept Checks

Like mastery quizzes, concept checks comprised a single multiple-choice question posted on the course management system that tested knowledge of a concept discussed at that point during class. Students were informed at the start of class that a concept check would occur sometime during class and they would have 2 min to answer the question before class resumed. Concept checks occurred at sporadic intervals during class sessions, excepting the first 15 min of class.

No-Engagement Activity Control

This no-activity control condition represented a teaching-as-usual condition. Students were informed at the start of class that no engagement activity was scheduled for that day.

To control content differences across the semester, we used a within-subjects, randomized alternating treatments design in which engagement conditions were randomized across classes. We randomized the three engagement conditions to seven class sessions each. We excluded the first and last class periods from the randomized sequence, and these classes were not included as part of the data analysis.

Results

Manipulation Check

Mastery quizzes and concept checks were intended as knowledge checks that students would be able to answer so long as they were paying attention in class. As a check on the engagement manipulation, we computed percentages of correct responses to both types of in-class quizzes, mastery quizzes and concept checks, applying an inclusion criterion that restricted analysis to students completing most of the quizzes. One student each was dropped from each analysis for failure to complete at least four of the seven quizzes. On average, student performance on both types of quizzes showed a high level of accuracy (mean scores of 84.26% on Mastery Scores and 80.77% on Concept Checks). A paired t-test analysis showed no significant difference in student performance between the two types of quizzes, t(52) = 1.27 p = .208, d = 0.18.

Learning Outcomes of Engagement Conditions

We first excluded exam items from the analysis of learning effects that tested the same concept as the mastery quizzes and concept checks (e.g., extinction). In total, four items across exams were dropped from the analysis, leaving 206 usable items in the analysis of exam performance. Analyses were based on a final sample of 54 students who completed all three-course exams.

Analysis of differences in exam performance as a function of engagement conditions (Mastery Quizzes/Concept Checks/No-Activity Control) showed a significant overall effect, F_{(1.80, 95.34)} = 17.19, p < .001, η² = .05 The degrees of freedom reported are Greenhouse–Geisser corrected, as Mauchly's test indicated that the assumption of sphericity had been violated, W = .89, χ²(2) = 6.17, p = .046. Post hoc comparisons using the Bonferroni correction showed significantly better exam performance with a large effect size, on topical content linked to classes with mastery quizzes, t(53) = 7.54, p < .001, d = 1.03, and concept checks, t(53) = 6.45, p < .001, d = 0.88, as compared to the teaching-as-usual (no-activity) control condition. Mean exam scores were 65.73% (SD = 10.26) for content linked to classes with concept checks, 66.24% for content linked to classes with mastery quizzes (SD = 10.67), and 58.23% (SD = 9.72) for content covered in classes without engagement activities (no-activity control condition).

Moderator Analysis: Bloom Level

Supporting a moderator effect of Bloom level, repeated measures analysis of variance (ANOVA) showed a significant interaction between the repeated measures factors of Engagement Condition and Bloom Level, F_{(2, 106)} = 17.63, p < .001, η² = .07 (see Figure 1).

Figure 1.
Exam performance as a function of engagement condition and bloom level.

Post hoc analysis of pairwise comparisons using Bonferroni correction for low-level Bloom items showed significantly better performance with a large size of effect on items linked to class days with mastery quizzes than those linked to classes without engagement activities (control condition), t(53) = 9.57, p < .001, d = 1.29. Students also performed significantly better, with a medium size of effect, on high Bloom-level items linked to classes with mastery quizzes than those with no-engagement activities (control condition), t(53) = 4.94, p < .001, d = 0.67.

Bonferroni-corrected pairwise comparisons also showed significantly better exam performance on items linked to class days with concept checks than those without any engagement activity (no-engagement control) for both low Bloom-level items, t(53) = 7.53, p < .001, d = 1.02, and high Bloom-level items, t(53) = 4.56, p < .001, d = 0.62. Again, the analysis of low Bloom-level items produced a large effect size as compared to a medium magnitude of effect in the analysis of high Bloom-level items. There were no significant differences between the two engagement activities for either the low, p = .029, or high Bloom-level items, p = .745.

Moderator Analysis: Class Coverage

We also examined class coverage of topical content of exam items as a potential moderator. We disaggregated exam items based on whether item content was discussed directly in class or reviewed in the text but not covered in class. Supporting a moderator effect of class coverage, repeated measures ANOVA on the repeated measures factors of Engagement Condition and Class Coverage revealed a significant interaction, F_{(2, 106)} = 9.74, p < .001, η² = .05 (see Figure 2).

Figure 2.
Exam performance as a function of engagement condition and class coverage.

Post hoc analysis of pairwise comparisons using Bonferroni correction showed that students performed significantly better, with a large size of effect, on exam items covered in class that were linked to class days with mastery quizzes than those without engagement activities (control), t(53) = 8.75, p < .001, d = 1.19. Similarly, but with a smaller effect size, students performed significantly better on text-only exam items drawn from chapters covered in classes on mastery quiz days than those without any engagement activity (control), t(53) = 3.23, p = .002, d = 0.44.

Bonferroni-corrected pairwise comparisons also showed significantly better exam performance, with a large size of effect, on exam items covered in class that were linked to class days with concept checks than those without engagement activities (control), t(53) = 9.453, p < .001, d = 1.29. However, there was no significant difference in student performance on text-only exam items drawn from chapters covered in classes on concept check days than classes without engagement activities (control) (p = 1.00) There were also no significant differences between the two engagement activities for either class-based, p = 1.00, or text-based exam items, p = .274.

Discussion

Course instructors face a twofold challenge of holding student attention in class in the face of both distracting stimuli and mind wandering. Student attention naturally ebbs and flows during class no matter how engaging an instructor may be or may believe that they are. The question posed in the present study was whether incentivizing student attention to material discussed in class yields a learning benefit as measured by performance on exam questions linked to topics covered in class. Two types of engagement activities, mastery quizzes and concept checks, provided incentives for students to pay closer attention during an introductory psychology class. These engagement activities, along with the no-engagement control, were randomized to class periods to control for content differences in topical content and order effects.

The results provided direct evidence supporting the learning benefits of in-class engagement activities. Students performed significantly better on topics covered on exams when those topics were linked to classes with mastery quizzes or concept checks than when they were taught under usual conditions without engagement activities. Relative to classes without engagement activities, student performance on course examinations increased an average of 8.0% for content linked to classes with mastery quizzes and 7.5% on content linked to classes with concept checks. These differences correspond to nearly a full grade-level difference.

Exam questions from regularly scheduled course exams surveyed knowledge of content drawn from topics discussed throughout each class period, so the learning benefits of interspersed engagement activities suggest that students not only attended to the targeted concept tested by the engagement quiz but to class content in general. A common feature of both engagement activities was that we did not cue students to the specific time during class when the targeted concept in the engagement quiz was discussed. Consequently, the use of an unscheduled cue may have kept students “on their toes” to be more attentive to class material throughout class.

The results of the moderator analysis showed learning effects for exam items representing both low and high levels of cognitive complexity according to Bloom's taxonomy of educational objectives. Low-level items tested basic knowledge, such as definitions and descriptions of key terms and concepts, and identification of major figures in psychology. High-level items required students to demonstrate knowledge relating to applications of concepts to examples and explanation and analysis of concepts. However, the results showed stronger effects of engagement conditions for exam items linked to lower levels in Bloom's taxonomy. It appears that using engagement quizzes to incentivize students to pay closer attention in class has stronger effects on student exam performance for more fact-based knowledge than higher levels of knowledge.

The analysis of moderator effects also showed the learning benefits of mastery quizzing for topical content discussed in class as well as related topical material drawn from the accompanying textbook but not directly covered in class. However, the effects of mastery quizzing were stronger for content discussed in class. Although concept checks showed a significant effect for class-based material as compared to the no-engagement activity control, they did not yield a significant benefit on textbook material not discussed in class. These results support the greater generalizability of mastery quizzing to text-based material; however, the smaller size of the effect in this analysis mitigates the significance of this finding.

Using in-class quizzes as attentional cues offers instructors a means of capturing student attention with minimal disruption of class time and lecture flow. A major advantage of these incentivized quizzes is that they were minimally intrusive with respect to class time, and unlike pop quizzes, students can demonstrate knowledge of concepts by paying close attention during class rather than drawing upon knowledge acquired before class. Both types of in-class quizzes required only a 2-min response period for students to answer a single question posed on the course management system.

Several limitations of the present study should be noted. For one, although engagement activities were randomized to classes, instructor bias may have favored how cued concepts were taught in class. However, mitigating this potential confound was that we omitted exam items from the analysis that tested concepts from in-class quizzes. So, the dependent measure comprised performance on course examinations of concepts not cued by in-class quizzes. Another limitation of the study was the absence of any direct measure of student attention, so we cannot assume that more focused attention in class mediated improvement in exam performance. Future research would be helpful in determining whether these types of engagement activities increase student engagement and attention in class and whether changes in attention mediate learning benefits of in-class quizzing. Overall, the findings from the present study suggest that brief in-class quizzes that incentivize student attention during class may have a beneficial payoff with respect to student performance on course examinations.

The results of the present study support the learning benefits of these brief, quiz-based engagement activities in class. However, the present findings on mastery quizzing differ from those of a previous study of engagement activities in online synchronous classes in which in-class mastery quizzing failed to demonstrate a significant learning benefit (Nevid et al., 2022). The lack of a learning benefit of mastery quizzing in the earlier study may be explained by the greater salience of distracting stimuli during remote synchronous classes, especially the presence or use of nearby electronic devices. It is possible that student attention to cell phones during online classes mitigated the learning benefits associated with cueing attention via mastery quizzing.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bloom

B. S.

Engelhart

M. D.

Furst

E. J.

Hill

W. H.

Krathwohl

D. R.

(1956). Taxonomy of educational objectives: Handbook 1, cognitive domain. Longman.

Bunce

D. M.

Flens

E. A.

Neiles

K. Y.

(2010). How long can students pay attention in class? A study of student attention decline using clickers. Journal of Chemical Education, 87(12), 1438–1443. https://doi.org/10.1021/ed100409p

Burke

L. A.

Ray

(2008). Re-setting the concentration levels of students in Higher Education: An exploratory study. Teaching in Higher Education, 13(5), 571–582. https://doi.org/10.1080/13562510802334905

Deslauriers

McCarty

L. S.

Miller

Callaghan

Kestin

(2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116

Freeman

Eddy

S. L.

McDonough

Smith

M. K.

Okoroafor

Jordt

Wenderoth

M. P.

(2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. https://doi.org/10.1073/pnas.1319030111

Hlas

A. C.

Neyers

Molitor

(2017). Measuring student attention in the second language classroom. Language Teaching Research, 23(1), 107–125. https://doi.org/10.1177/1362168817713766

Karpicke

J. D.

Grimaldi

P. J.

(2012). Retrieval-based learning: A perspective for enhancing meaningful learning. Educational Psychology Review, 24(3), 401–418. https://doi.org/10.1007/s10648-012-9202-2

Kuhl

B. A.

Chun

M. M.

(2014). Memory and attention. In Nobre

A. C.

Kastner

(Eds.), The Oxford handbook of attention (pp. 806–836). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199675111.013.034

Markowitz

D. M.

Hancock

J. T.

Bailenson

J. N.

Reeves

(2019). Psychological and physiological effects of applying self-control to the mobile phone. PLoS One, 14(11), e0224464. https://doi.org/10.1371/journal.pone.0224464

10.

McKeachie

W. J.

Svinicki

(2006). McKeachie’s teaching tips: Strategies, research, and theory for college and university teachers. Houghton Mifflin.

11.

Nevid

J. S.

Keating

L. H.

Lieblich

(2022). Effects of in-class engagement activities in online synchronous classes. Scholarship of Teaching and Learning in Psychology, 8(4), 304–314. https://doi.org/10.1037/stl0000312

12.

Nevid

J. S.

Mahon

(2009). Mastery quizzing as a signaling device to cue attention to lecture material. Teaching of Psychology, 36(1), 29–32. https://doi.org/10.1080/00986280802529152

13.

Nilson

L. B.

(2010). Teaching at its best: A research-based resource for college instructors. Jossey-Bass.

14.

Roediger

H. L.

Butler

A. J.

(2011). The critical role of retrieval practice in long-term retention . Trends in Cognitive Sciences, 15(1), 20–27. https://doi.org/10.1016/j.tics.2010.09.003

15.

Rowland

C. A.

(2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463. https://doi.org/10.1037/a0037559

16.

Schmidt

(2001). Attention. In Robinson

(Ed.), Cognition and second language instruction (pp. 3–32). Cambridge University Press.

17.

Sotola

L. K.

Crede

(2021). Regarding class quizzes: A meta-analytic synthesis of studies on the relationship between frequent low-stakes testing and class performance. Educational Psychology Review, 33(2), 407–426. https://doi.org/10.1007/s10648-020-09563-9

18.

Ward

A. F.

Duke

Gneezy

Bos

M. W.

(2017). Brain drain: The mere presence of one’s own smartphone reduces available cognitive capacity. Journal of the Association for Consumer Research, 2(2), 140–154. https://doi.org/10.1086/691462

19.

Yazedjian

Kolkhorst

B. B.

(2007). Implementing small-group activities in large lecture classes. College Teaching, 55(4), 164–169. https://doi.org/10.3200/ctch.55.4.164-169