Abstract
We examined the effect of quizzing on students’ ability to apply lecture content to clinical examples on an in-class exam. Fifty-six occupational therapy students in a graduate-level course completed three online modules that included lectures, slide presentations, demonstrations, and video clips. Throughout each module, students were given “Learning Moments,” in which they were asked a question (Question condition); were asked to read a statement confirming the content presented (Study condition); or were not asked to do anything (Not Asked condition). The Question condition resulted in better exam performance than the combined Study and Not Asked conditions (p < .01), whereas performance on Study and Not Asked questions did not differ (p = .67). Students performed better on material on which they had been previously quizzed (i.e., testing effect). Quizzing may be a good strategy to prepare students for clinical practice.
Occupational therapy educators prepare students to enter the profession with the knowledge and skills to support practice in a variety of contexts. Preparation includes basic theoretical knowledge of the core construct of occupation (Hooper, 2006) as well as models of practice and frames of reference to guide decisions about evaluation and intervention approaches (American Occupational Therapy Association [AOTA], 2012). Occupational therapy graduates must acquire the skills necessary for critical analysis and clinical reasoning so they can apply their knowledge to address client problems (AOTA, 2010).
Occupational therapy education reflects the full continuum of cognitive processes described by Anderson and Krathwohl (2001) in their revision of Bloom’s (1956) original taxonomy: remembering, understanding, applying, analyzing, evaluating, and creating. Students must learn new concepts, apply them to clinical examples, and analyze and evaluate nuances to select from appropriate options. In addition, they must generate novel ways of organizing information to create programs that address the needs of specific groups of people. Facilitating these cognitive processes in a short time frame requires careful planning on the part of educators. One strategy to achieve learning, retention, and transfer of new material is quizzing (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013).
Gupta and Bilics (2014) called for research in occupational therapy education that addresses strategies to facilitate learning and support practice competency. In their review of 129 research articles on occupational therapy education, Hooper, King, Wood, Bilics, and Gupta (2013) found that most studies were descriptive, addressed a specific contextual need in education, and lacked conceptual frameworks or educational issues that cross educational settings. They urged researchers to frame future studies according to educational issues and research questions that cut across programs and to use conceptual frameworks to explain why and how an educational approach works and why it is suitable for occupational therapy education.
In their review of teaching and learning strategies, Dunlosky et al. (2013) reported on the learning benefits of practice quizzing across a variety of learning conditions, student characteristics, teaching materials used, and outcome measure criteria. Although multiple studies in controlled laboratory experiments (see Roediger & Karpicke, 2006) and several studies in college courses (e.g., McDaniel, Wildman, & Anderson, 2012) have demonstrated the learning benefits of quizzing, students do not typically report using it to support or enhance their learning (Karpicke, Butler, & Roediger, 2009). Both students and educators traditionally have considered tests as a way to measure learning. Interim testing, or quizzing, prior to a major exam, however, can be used as a powerful intervention to enhance learning, retention, and transfer of knowledge (Carpenter & Pashler, 2007; Dunlosky et al., 2013; Karpicke, 2012; McDaniel et al., 2012; Roediger & Karpicke, 2006). The learning benefit of interim quizzing, relative to rereading the same material during learning, is known as the testing effect.
In typical studies, participants are either quizzed on or asked to reread targeted material. Performance on a later, final test for the two groups is then compared. The testing effect has been shown to be robust in a variety of settings, with various age groups, quizzing materials, and quizzing paradigms. Interim quizzing has been shown to improve long-term retention of factual information in areas of foreign language word pairs; understanding of text passages; and performance in statistics, science, and map learning (Carpenter et al., 2016; Roediger & Karpicke, 2006).
In one example of the testing effect in a face-to-face course on human stress offered in an occupational therapy department, students completed five weekly online quizzes before a midterm exam (Benassi et al., 2014). Students were randomly assigned to one of two groups; each student received six questions, unique to the group, per quiz. Performance on the midterm demonstrated a testing effect for the quizzed items: Students performed better on items on which they had been previously quizzed. In a second, within-group study in a separate offering of the same course, students were exposed to all possible exam topics during the quizzing. Half the questions were recall questions (testing trials), and half were summary statement exposures (study trials). Again, students performed better on transfer exam questions for those items on which they had been quizzed, relative to the summary study items.
Despite what we know about the learning benefits of quizzing, students report that they prefer to reread their notes or course materials (Karpicke et al., 2009). Research, however, informs us that rereading provides only limited benefits (Carpenter, Pashler, Wixted, & Vul, 2008; McDaniel et al., 2012). Students who choose rereading as a study strategy are likely to demonstrate lack of metacognitive insight regarding their learning (Karpicke et al., 2009); their assessments of their judgments of learning are often biased by the availability of material during study, which will be absent during an exam (Koriat & Bjork, 2005). The resulting overconfidence in learning impedes students’ choice to engage in more effective, cognitively based learning strategies such as practice tests. Educators are in a position to promote quizzing as a learning tool by crafting occasions for quizzing into their courses. Incorporating quizzing into a course design can take multiple forms, such as in-class or self-quizzing, problem-solving activities, and end-of-chapter questions.
The purpose of the current study was to evaluate the transfer of material presented in a lecture capture platform that students watched outside of class in preparation for in-class activities and an in-class exam. At strategic locations throughout the lecture capture lessons, students either reread statements on presented material or were quizzed.
On the basis of evidence in the literature in both laboratory (Roediger & Karpicke, 2006) and course-based (e.g., McDaniel et al., 2012) studies supporting the testing effect, we predicted no difference in students’ exam performance on questions in the restudying condition and the no-exposure condition. Moreover, we predicted that students would perform better on exam transfer questions on which they were previously quizzed, compared with their combined performance on exam transfer questions that were linked to the restudy and no-exposure conditions.
Method
Research Design
Because our sample size was 56, had we randomly assigned students to experimental groups, as would be done in a randomized between-subjects design, our group sample sizes would have been relatively small. Thus, we used a within-subject comparison of question conditions to examine the benefits of quizzing as a learning strategy. An additional advantage of this type of design is the provision of greater statistical power and experimental control. The within-subject design ensured that we would eliminate between-condition variance in exam performance because each participant served as her own control. The dependent measure was student performance on in-class exam transfer questions. This experiment was approved by the university’s institutional review board.
Participants
Participants were 56 female students (mean [M] age = 23.09 yr, range = 22–30 yr) from a moderate-sized public university in the northeastern United States, enrolled in a 14-week course in a professional master’s program in occupational therapy. The course focused on pediatric evaluation and intervention, and its structure included in-class lecture, out-of-class lecture capture, demonstrations, class activities, and laboratory experiences. Lecture capture lessons provided students with essential information in preparation for activities during the subsequent class. From the original pool of 61 enrolled students, 2 students opted out, and 3 were eliminated because they did not meet the minimum requirements regarding the number of completed quizzes associated with the three lecture capture lessons.
Materials
The course instructor (the first author) recorded a total of three lecture capture lessons leading up to the in-class exam. Embedded within the lessons were two or three Learning Moments, which consisted of question sets covering the just-presented lesson content. Recorded lessons included combinations of lecture, demonstrations and, in some cases, videos. Lesson topics included standardized assessment, infant neurodevelopment, observation, and assessment concerns. Students viewed lecture capture lessons outside class via a lecture capture platform (Tegrity; McGraw-Hill Education, Columbus, OH), available through the course learning management system (Blackboard Learn; Blackboard, Washington, DC).
Instructor-developed learning objectives for each lecture formed the basis for the multiple-choice quiz questions. The second author randomized these questions to take the form of one of three conditions: (1) Question, in which the question was kept as the original multiple-choice question; (2) Study, in which the question was converted into a statement that students read and documented that they had read; or (3) Not Asked, a control condition, in which students were not asked to revisit the material (Table 1). Questions were then uploaded in their randomized condition onto the test platform in the course Blackboard site. The instructor was blinded to all question conditions. For the in-class exam, the instructor composed transfer questions for each Learning Moment question, including those that were not presented to the students. All transfer exam questions were in multiple-choice format (see Table 2 for an example).
Sample Questions for the Three Conditions
Note. Correct answers are in
Example of Quiz Question and Transfer Exam Question
Note. Correct answer is in
Procedure
All students watched the same three recorded lecture capture lessons out of class and completed the Learning Moment activities as directed. At three or four strategic points during the presentation (depending on the lesson watched), the instructor told students to pause the lecture and access the test platform on Blackboard to complete a Learning Moment. The Learning Moments were randomized into one of the three conditions (Question, Study, Not Asked); all students responded to questions in each of the conditions in the Learning Moments within the lecture capture recordings.
On completion of each Learning Moment, students returned to the recorded lecture capture lesson until instructed to respond once again to a Learning Moment. This cycle continued until lesson completion. Students received no corrective feedback regarding their performance on Learning Moment questions. At the end of the first unit in the course, which covered one-third of the course content, students completed the first of three in-class exams. Exam questions were coded for condition for purposes of analysis. Students’ responses to coded questions served as the basis for analysis.
Results
In addition to standard null-hypothesis testing techniques (with α set to .05), we assessed the differences between the mean paired scores using the estimation approach and considered confidence intervals (CIs; Cumming, 2013). We conducted all analyses using IBM SPSS Statistics (Version 23; IBM Corp., Armonk, NY).
We first performed the Mauchly test to assess for possible violation of the sphericity assumption, which was nonsignificant, Mauchly’s W = 0.95, χ2(2) = 2.72, p < .26 (Warner, 2008, pp. 909–910). We next performed a repeated-measures analysis of variance. The means (and standard deviations [SDs]) for the Not Asked, Study, and Question conditions were 76.56 (SD = 14.11), 77.68 (SD = 15.58), and 81.75 (SD = 12.96), respectively, Wilks’s Λ F(1, 54) = 3.26, p < .05, ηp 2 = 0.11 (Warner, 2008, p. 916). Finally, we conducted a set of a priori planned contrasts using the difference approach in SPSS (Warner, 2008, pp. 916, 920). As expected, the means for the Not Asked and Study questions did not differ, t(55) = 0.42, p = .67. Therefore, we combined these two conditions into a new variable: Not Asked/Study. We then conducted a planned contrast between the mean of the Question condition and the mean of the newly combined Not Asked/Study condition. As predicted, the mean of the Question condition was significantly greater than that for the combined condition, t(55) = 2.54, p < .007 one-sided, d = 0.34.
For the two planned contrasts, we calculated the mean difference between each of the two condition performances (Figures 1 and 2). A difference score of 0 would indicate no difference between the mean test condition performances. We predicted no difference in the Study and Not Asked mean difference scores; therefore, we computed a two-tailed confidence level for this pairing. The 95% CI contains 0, and our mean difference score falls within that interval (see Figure 1). Because there was no difference between the mean test performances in the Study and Not Asked exam question pairing, we combined the two to compute a new variable: Not Asked/Study (M = 77.12, SD = 11.13).
Mean percentage correct on exam transfer questions in the Study and Not Asked conditions. On the right axis is the two-tailed 95% confidence interval with mean difference (Diff) scores between the Study and Not Asked conditions.
Mean percentage correct on exam transfer questions in the combined Not Asked and Study conditions and Question condition. On the right axis is the one-tailed 95% confidence interval with mean difference (Diff) scores between the Not Asked/Study and Question conditions.
Because we predicted a learning benefit for the Question condition, we computed a one-tailed CI for the pairing that included the newly created Not Asked/Study condition and the Question condition. The lower boundary confidence level describing the differences for the Not Asked/Study and Question conditions comparison did not reach 0; that is, our mean difference fell within the interval that describes a difference between the mean scores. If our study had been conducted an infinite number of times, 95% of our intervals would be expected to contain the true effect, with a 4.63% difference being our best estimate of the population parameter.
Discussion
Our results demonstrate a testing effect. Students performed better on transfer exam questions on which they had been previously quizzed compared with those on which they had not been quizzed. The view that the testing effect may be due to the additional exposure to course material that quizzing provides has received little empirical confirmation in laboratory-based research; in fact, research suggests that reexposure to previously presented material may not have a benefit over no reexposure to the material (Carpenter et al., 2008; Roediger & Karpicke, 2006). Students in our study performed better on the in-class exam transfer questions for which they answered quiz questions compared with exam questions for which they had no reexposure and, more pointedly, for those questions for which they had been reexposed through reading study statements. Consistent with prior research (Carpenter et al., 2008; McDaniel et al., 2012), students in our study received virtually no benefit from reexposure to the material. In fact, performance in the Study condition did not vary from performance in the control (Not Asked) condition (in which questions were generated but, by random assignment, unavailable to students).
Our study was conducted in the natural context of learning in a university course. In addition, our study may be the first to examine the testing effect in the context of course material presented during a recorded lesson. Butler and Roediger (2007) examined the testing effect whereby students watched videotaped lectures on art history in a laboratory context. We examined whether asking questions during a recorded lecture in an academic course might benefit students’ learning. Although students in our study were permitted to return to the recorded lecture capture lessons as desired, they were able to access the Learning Moments only once, after which they received no feedback; this approach differs from McDaniel et al.’s (2012) study, in which students received immediate feedback and had unlimited quiz access. Of interest is that our students’ unlimited access to the recorded lectures did not augment their performances in the Study or Not Asked question conditions.
The in-class exam questions required students to transfer their learning and to apply their knowledge to clinical examples, reflecting high-level cognitive processing (Anderson & Krathwohl, 2001). Thomas and McDaniel (2007) reported that college students performed better on exams when their study activity was congruent with the category (detail, conceptual, relational) of questions that were asked on the exam. However, in our study the Question condition within the Learning Moment activities quizzed students on content, with the intent of promoting retention. The in-class exam required students to transfer their knowledge and apply it to clinical examples, which Anderson and Krathwohl (2001) reported as located higher on Bloom’s taxonomy.
Limitations and Future Research
Our study was based on one course and one content area for occupational therapy practice. Participants were a convenience sample of students already enrolled in the course. Moreover, students in this sample were all women and in the same age range. Replicating this study in other courses and occupational therapy programs would support the generalizability of the findings. At the same time, we have no reason to expect a different finding than we obtained, given that testing effects are robust and have been found in a variety of academic courses (e.g., introductory psychology, statistics, chemistry, geography; Carpenter et al., 2016; McDaniel et al., 2012; Roediger & Karpicke, 2006).
In addition to replication of this study in other areas of occupational therapy course content, future research might include exploring whether embedding questions into the lecture affects learning differently than presenting all questions at the end of the lecture. One might also explore the impact different types of quiz questions have on learning, for example, questions that focus on retrieval of facts versus questions that require the application of information. Researchers might also examine the influence of the testing effect by means other than exam questions, such as written intervention plans or hands-on application.
Implications for Occupational Therapy Education
Our findings have the following implications for occupational therapy education:
Rereading content may not be an effective learning strategy for students.
Asking students questions about course content as they learn that content supports their learning.
Asking questions during learning may facilitate transfer of knowledge to application in client examples.
Conclusion
The results of this study provide evidence on the effective use of asking quiz questions during a recorded lecture to promote students’ acquisition of knowledge and the transfer of the knowledge for application to clinical examples on an in-class exam. Although the testing effect has been widely documented in the educational psychology literature, to our knowledge this is the first examination of quizzing in a recorded lecture capture lesson in occupational therapy education. The results suggest the potential for using in-class quizzing as a valuable strategy in other occupational therapy courses to promote learning.
Footnotes
Acknowledgments
This research was supported by a grant from the Davis Educational Foundation (Yarmouth, ME). The foundation was established by Stanton and Elizabeth Davis after Stanton Davis’s retirement as chairman of Shaw’s Supermarkets, Inc. We thank Elizabeth Tappin for her contributions in compiling data necessary for completion of this project.