Effects of embedded questions in pre-class videos on learner perceptions,video engagement,and learning performance in flipped classrooms

Abstract

This study investigates the impact of embedded questions in pre-class instructional videos on learner perceptions (cognitive load, emotional engagement, satisfaction, judgement of learning), video engagement (total views, total viewing time), and learning performance (retention, transfer). The research occurred in a real flipped classroom environment. We designed a quasi-experiment in which 86 university students from two natural classes watched pre-class instructional videos featuring procedural knowledge with or without interpolated true or false questions. Students were asked to practice the operation steps introduced in the videos. While they practiced operations, they could either pause the videos or let the videos continue playing. Face-to-face contact time was utilised to consolidate and extend previewed content with student-centred, instructor-facilitated problem-solving activities. Results revealed no discernible effects from embedded questions in pre-class videos on cognitive load, emotional engagement, satisfaction, judgement of learning, total views, knowledge retention or knowledge transfer. We speculate that the various in-class practice activities and frequent access to procedural knowledge videos offset the cognitive benefits derived from question-embedded videos. Learners who viewed question-embedded videos presented significantly reduced total viewing time, likely because the embedded questions scaffolded them in sustaining attention and efficiently pinpointing the exact information needed. Future research should identify boundary conditions for embedding questions in instructional videos (e.g. learning mode, type of knowledge) rather than indiscriminately applying this design strategy.

Keywords

educational video flipped classroom flipped learning instructional video teaching and learning

Introduction

Meta-analyses have revealed that flipped classroom approaches are more effective than traditional teaching approaches in enhancing cognitive (Strelan et al., 2020a) and affective outcomes (Strelan et al., 2020b). In flipped classrooms, pre-recorded instructional videos are considered the most important learning resource to prepare students for in-class participation (Guo, 2019) and the key to success in flipped learning (Förster et al., 2022). Students’ attitudes towards pre-class videos are generally positive (Xiu et al., 2019), and they prefer videos to assigned readings as preparatory materials (Bordes et al., 2021). Student performance has also been found superior in weeks where videos were used as preparatory materials compared to weeks where readings were used for pre-class learning (Lee and Choi, 2019). However, using pre-class videos with limited technical and pedagogical features can negatively impact student learning in flipped classrooms (Akçayır and Akçayır, 2018). Given the importance of pre-class videos in the flipped learning model, improving the design of such videos to help students learn more effectively in flipped classrooms is critical.

Generative activities improve learning by prompting students to actively make sense of to-be-learned information. Examples of generative activities known to enhance learning performance include imitating the instructor’s actions and writing explanations while watching instructional videos (Mayer et al., 2020). Self-testing is among the generative activities that facilitate information retrieval and help structure existing knowledge (Brod, 2020). Self-testing involves answering test questions about previously learned materials (Fiorella and Mayer, 2015). Embedding questions in instructional videos is a practical application of self-testing and is considered a promising strategy to facilitate learning. Research indicates that interpolating videos with multiple-choice (Vural, 2013) and open-ended questions (van der Meij and Bӧckmann, 2020) can significantly improve learning performance. Although those studies are potentially useful, most were conducted in laboratory or online settings where face-to-face instruction was unavailable to students. To our knowledge, a study conducted by Haagsman et al. (2020) is the only exploration into the effects of embedded questions in videos on student performance in flipped classrooms. Their study featured the transmission of declarative knowledge using question-embedded pre-class videos. Specifically, knowledge is twofold, consisting of declarative and procedural knowledge (Hiebert and Lefevre, 1986). Declarative knowledge involves factual information and is considered static (Ohlsson, 2012). Procedural knowledge involves how to perform tasks and is intrinsically related to action (Hiebert and Lefevre, 1986). Given that declarative knowledge has been the primary focus of past research, this research bridges the knowledge gap by extending the investigation to question-embedded pre-class videos disseminating procedural knowledge.

This research conducts a quasi-experimental study to investigate the utilisation of question-embedded videos in a flipped learning environment where face-to-face instruction is available during in-class sessions designed to help students consolidate and extend knowledge garnered from pre-class videos. By conducting the quasi-experimental study, it addresses the following research question: How do embedded questions in pre-class videos transmitting procedural knowledge impact learner perceptions, video engagement, and learning performance in flipped classrooms?

Literature review

This section reviews the existing literature on learner perceptions, video engagement, and learning performance variables.

Learner perceptions

Learner perceptions are defined as individuals’ subjective reactions to the experience of watching instructional videos (Wang et al., 2020). In this study, learner perceptions are conceptualised to comprise cognitive load, emotional engagement, satisfaction and judgement of learning.

Cognitive load is a multidimensional construct comprising three aspects: intrinsic load imposed by the complexity of the to-be-learned materials, extraneous load imposed by the design of the instructional materials and germane load representing the amount of working memory resources learners devote to schema acquisition and modification (Sweller, 2010). In the literature, there are studies that discuss the influence of question-embedded videos on students’ cognitive load. Researchers’ opinions are polarised. Some studies contend that interpolating questions into videos splits the content into smaller chunks, where the partitioning of educational materials makes learning more manageable and lowers cognitive load (Desai and Kulkarni, 2022; Pulukuri and Abrams, 2021; Szpunar et al., 2013). They further mentioned that this could trigger reflection and lead to learning gains without overwhelming students. Torres et al. (2022) argued that question-embedded videos can decrease extraneous cognitive load and increase germane cognitive load. However, their study did not present empirical evidence to justify this assumption. In addition to the affirmative point of view, some contend that question-embedded videos could have a negative impact. Rice et al. (2019) pointed out that the embedded questions in a video could become a distraction, where learners might lose interest and focus on the questions rather than the educational content. Tweissi (2016) further raised a concern about whether the embedded questions could overwhelm learners with too much information and cause anxiety as a result. Given that the effectiveness of question-embedded videos featuring procedural knowledge has not been empirically verified in flipped learning settings, we aim to investigate whether embedded questions in pre-class videos influence learners’ perceived cognitive load.

Emotional engagement is operationally defined as the affective connections learners make with the course content (Deng et al., 2020b). Emotional factors potentially explain the processes that facilitate video-based learning and should not be ignored in multimedia learning research (Deng and Gao, 2022). Tweissi (2016) speculated that question-embedded videos may influence emotional factors such as the learner’s appreciation and attitude. Previous research has investigated the effects of embedded questions on learners’ emotional experience. The study by van der Meij and Bӧckmann (2020) revealed that learners watching the educational videos with and without embedded questions uniformly displayed positive emotions. However, the focus of van der Meij and Bӧckmann’s (2020) study was on video-recorded lectures in laboratory settings. The effect of question-embedded pre-class videos on emotional engagement has not yet been investigated in flipped learning settings. To provide evidence, we used emotional engagement as a dependent variable for the intervention – pre-class videos interpolated with true or false questions.

Satisfaction is conceptualised as an affective outcome and describes learners’ short-term attitudes resulting from the evaluation of educational experience (Cheng et al., 2016). A laboratory study by van der Meij and Bӧckmann (2020) revealed that inclusion of open-ended embedded questions did not contribute to learner satisfaction, whereas Spanjers et al. (2015) found that blended learning environments with quizzes were generally more attractive to students. Vural (2013) argued that question-embedded videos could enrich the learning process with interactivity and lead to higher learner satisfaction; however, this argument has not been verified in empirical studies. One may speculate that question-embedded interactive videos provide additional support to students in the pre-class stage, which makes learning a more satisfying experience. The influence of question-embedded pre-class videos on learner satisfaction has not been quantitatively investigated in the flipped learning context. To provide evidence on this issue, we used satisfaction in this study as a dependent variable for the intervention.

Video engagement

Video engagement is conceptualised differently among scholars. It is often considered as the number of views per student (Kovacs, 2016) and pausing and rewinding events (Davis et al., 2018). While such metrics cannot verify the degree to which students paid attention to the learning content they encountered (Hu et al., 2020), measuring true engagement with pre-class videos requires direct observation and is often not feasible in authentic learning environments (Dart, 2020). Laboratory research has shown that open-ended questions can result in greater video engagement, as measured by total time played and replay duration (van der Meij and Bӧckmann, 2020). Log files are files that record events and user interactions and often used in combination with other research methods, particularly surveys, to investigate the online learning process (Deng and Benckendorff, 2017). In the present study, we use the total views and total viewing time recorded by the learning management system (LMS) as proxies for video engagement and investigate whether embedded questions in pre-class videos increase total views and total viewing time.

Learning performance

Researchers investigating the effects of question-embedded videos on learning performance have obtained mixed results. While Vural (2013) found that embedded questions can positively affect learning performance, van der Meij and Bӧckmann (2020) found that this effect was limited to the information asked about in the embedded questions. Embedded questions that closely resemble post-test questions can counteract the positive effects of such questions on learning achievement (Rice et al., 2019). To minimise such practice effects, we ensured that the questions embedded in the videos in this study were not identical to any of the questions used in the prior knowledge test or the learning performance test. We assessed learning performance at the retention and transfer levels and investigated whether embedded questions in pre-class videos enhance knowledge retention and transfer. Apart from actual learning, students self-monitor their study by making a judgement of learning, a prediction of the likelihood of eventual memory performance for recently studied items (Dunlosky and Nelson, 1992). Some scholars contend that the presence of questions in instructional videos can support learners’ confidence in their capacity to comprehend the information conveyed in the videos and positively affect their assessments of the amount of knowledge they have acquired (Pulukuri and Abrams, 2021). Accordingly, in addition to actual learning performance, we consider perceived learning and investigate whether embedded questions in pre-class videos improve the judgement of learning.

Materials and methods

Research design

The study was designed as a quasi-experiment, and two natural classes of sophomore students at a public university in China participated. The educational intervention was administered during an introduction-level course titled Modern Educational Technology. The course was designed by the same instructor and delivered at the same pace for both classes.

The study manipulated the presence of questions in pre-class videos (question-embedded vs question-absent) as a between-subjects variable. The only difference between the experimental and control conditions is that true or false questions were embedded in the pre-class videos viewed by the experimental group. The independent variable comprises the pre-class videos interpolated with true or false questions (with feedback provided). The dependent variables are threefold: learner perceptions, video engagement, and learning performance. Participants’ age, sex, actual and perceived prior knowledge and motivation were controlled in the study.

Participants

Participants were 86 Chinese sophomore students, 42 were in the experimental group and 44 were in the control group. A priori power analysis using G*Power 3.1 (Faul et al., 2009) indicated that a sample of at least 68 participants was required to detect a large effect size with a power of 0.90, and alpha of 0.05. This study included 55 female and 31 male students (M_age = 20.42, SD = 0.80). Most participants majored in mathematics, physics and primary education. They received no monetary compensation for participating. Ethics approval was received from the university’s ethics committee before conducting the research.

Materials

The instructional videos comprise eleven presentations on how to use Microsoft PowerPoint which features procedural problem-solving tasks. The medium of instruction was Chinese. The videos were drawn from the education channel of an online video platform and characterised by a media-centric style. The speaker’s image was not presented in the videos because only marginally positive effects of instructor presence on learning performance in instructional videos disseminating procedural knowledge have been found (Alemdag, 2022). The videos adhered to design principles proven effective in empirical studies, including coherence (removing text, audio and graphics that do not support instructional goals), signalling (highlighting important information), temporal continuity (presenting animation and narration simultaneously), and spatial continuity principles (placing text and corresponding graphics next to one another) (Mayer and Fiore, 2014).

Under the experimental condition, questions were embedded at points in the videos after a procedure had been performed. A total of 50 true or false questions were embedded in 11 instructional videos (Supplemental Appendix A). Video durations ranged from 5 m 5 s to 11 m 50 s. The control group watched videos with no embedded questions. The embedded questions were based on the learning objectives of each video and developed by two subject-matter experts. For example, the learning content of the first video is PowerPoint Basic Functions, and the video contained four true or false questions. Here an example question is provided: after animating multiple elements on a slide, numbers appear next to the animated elements, representing each animation’s time duration.

Neither student performance in answering the embedded questions nor their viewership of videos were included in calculating the course grade. Participants were allowed unlimited attempts to answer all questions and to likewise watch the videos as many times as desired. Special attention was paid to the experimental design to ensure that the students read the embedded questions and answered them when watching the videos. When an instructional video is playing and the viewing progress reaches the time node of embedded questions, the video will be automatically paused, and the participants need to either select the answer (true/false) and click ‘submit’ or click ‘submit’ and then ‘continue playing’ to continue the video without providing answers. Although answering questions was optional, participants in the experimental group chose to answer questions instead of avoiding them. This is not only because the video will automatically pause when the embedded question is on display, but also because the course instructor emphasised in the first week the importance of pre-class video learning for the subsequent flipped classroom and told the participants that the course assessments involve video content. This ensures that the learners answered the embedded questions while watching the videos before class. All videos were hosted in a secure LMS. Both the control and experimental groups could log in to the LMS and watch the videos at their convenience from any location. Additionally, the knowledge retention and transfer tests were closed-book exams. During the tests, learners were not allowed to consult the pre-class videos.

Procedures

The study was conducted in the Spring of 2022. The experimental procedures employed are outlined in Figure 1. During week 1, participants were given guidelines for excelling in a flipped classroom and told that their knowledge of video content would be assessed in future weeks. They were instructed to watch the videos before class, practice the sequences of operations shown in the videos while learning the preparatory video materials, and note any questions they had. Participants filled out an informed consent form and completed a 15-minute prior knowledge test monitored by the instructor. Then they were asked to report their age and sex and their prior interest in and familiarity with the topic in a pre-questionnaire (Supplemental Appendix B). During weeks 2–5, participants were asked to watch the videos at their own pace before class. Face-to-face contact time was used to review video content and consolidate the knowledge delivered in the videos through instructor-facilitated problem-solving activities. Each face-to-face session was 2 hours long. In-class learning activities were constant between the two groups. In week 6, participants completed a post-questionnaire to report their subjective experiences with the videos, including cognitive load, emotional engagement, satisfaction, and judgement of learning (Supplemental Appendix C). They were asked to complete the questionnaire based on their actual learning experiences with the videos. The retention test was administered in week 6 and the transfer tests a week later.

Figure 1.

Outline of the experimental procedures.

Measures

Prior knowledge test

The prior knowledge test developed by the instructor comprised 17 multiple choice questions (e.g., ‘What is the quickest way to set three images the same distance from the top edge of the slide?’) and three true or false questions (e.g. ‘The duration of the switch between two slides cannot be adjusted’). As the questions represented knowledge delivered in multiple videos, the test was not designed to be internally consistent (Ulrich et al., 2019). Its purpose was to determine whether or not the participants in the two different groups had an equal understanding of PowerPoint knowledge. Five points were given for each correct answer (max score = 100 points).

Pre-questionnaire

To understand whether participants had the same level of perceived knowledge and motivation to study the topic across the two conditions, they were asked to rate their knowledge on an 11-point Likert scale (i.e. ‘How much do you know about the topic of PowerPoint’?) and motivation (i.e. ‘How interested are you in the topic of PowerPoint’?), with 0 indicating not very much and 10 indicating very much. Participants also reported sex and year of birth in the pre-questionnaire. Sex was a binary categorical variable with men coded as 1 and women coded as 2. The birth year was converted to age.

Post-questionnaire

Cognitive load

We adapted the scale developed by Leppink et al. (2013) to measure cognitive load. In this study, the cognitive load was operationalised as a multidimensional construct comprising the intrinsic, extraneous, and germane load. We adopted eight items to measure cognitive load: three for the intrinsic load (e.g. ‘The content covered in the instructional videos was very complex’), three for the extraneous load (e.g. ‘The instructions and explanations in the instructional videos were very unclear’), and two for the germane load (e.g. ‘The instructional videos really enhanced my ability to perform relevant procedures’). A complete list of the items for measuring cognitive load is provided in Supplemental Appendix D. Cronbach’s alpha values for each dimension were 0.63, 0.81, and 0.87, respectively. Aggregated scales were constructed by calculating the sum of the items comprising each dimension. For example, the composite score of the intrinsic load in the cognitive load scale is the sum of the values of the three included statements. Composite values were retained for data analysis.

Emotional engagement

To measure emotional engagement, we adapted the scale developed by Deng et al. (2020a). It comprised three items rated on a 6-point Likert scale (e.g. ‘The instructional video stimulated my interest in PowerPoint’), with 1 representing strongly disagree and 6 representing strongly agree. A complete list of the items for measuring emotional engagement is provided in Supplemental Appendix E. Cronbach’s alpha was .83. An aggregated scale was created to substitute the original three items.

Satisfaction

Participants rated their satisfaction with the videos on a 6-point Likert scale (i.e. ‘Overall, I was satisfied with the instructional videos’), from strongly disagree (1) to strongly agree (6).

Judgement of learning

To measure the judgement of learning, we adapted the scale designed by Alqurashi (2018). It contained two items rated on a 6-point Likert scale (e.g. ‘I am confident that I will be able to independently perform the procedures shown in the instructional videos’). A complete list of the items for measuring judgement of learning is provided in Supplemental Appendix F. Cronbach’s alpha was .86. An aggregated scale was constructed to replace the original two items.

Video engagement

The number of views and viewing time for each video were extracted from the LMS for each participant. Total views and total viewing time were calculated for each participant by adding the number of views and viewing time for all videos. To accurately capture their video engagement, participants were asked to watch the videos through the LMS and were unable to download videos to their personal devices.

Learning performance tests

The retention test consisted of 40 multiple choices (e.g. ‘When editing the points of a shape, what kind of points can be controlled independently with the two control sticks’?) and 10 fill-in-the-black questions (e.g. ‘You need to perform multiple steps to create a section zoom, filling in the space with numbers that represent the correct order of actions’). As these questions tackled different sub-topics across multiple videos, the retention test was not designed to be internally consistent (Ulrich et al., 2019). Its purpose was to determine whether the intervention affected the amount of knowledge retained. Each correct answer earned two points (max score = 100 points).

The transfer test consisted of an open-ended task. Participants were asked to design a new collection of slides using the knowledge they acquired from the pre-class videos. We used a rubric to articulate the expectations for the task, which comprised task expectation, dimensions (i.e. operation, creativity, presentation) that participants were to attend to completing the task, mastery levels for each performance dimension (i.e. distinction, credit, pass, fail), and performance descriptors. The rubric was available to participants when completing the task. As the transfer test is open-ended, a second independent evaluator scored all the answers using the rubric. The two evaluators resolved scoring differences through face-to-face consultations. The maximum score was 100 points.

Results

The assumptions of normality, homogeneity of variance, and absence of outliers were checked before analysis. IBM SPSS Statistics 28 was used for data processing and analysis. We first performed Levene’s test for equality of variances. The results indicated that the variability of scores for each group was similar and homoscedasticity was met (p > .05). We then assessed skewness and kurtosis values and histograms. Skewness and kurtosis values for all dependent variables were within acceptable ranges between +2 and −2. We, therefore, performed independent-samples t-tests to compare the means of the dependent variables for the experimental and control groups. We then inspected boxplots for outliers. One participant was identified as an outlier due to having a very low satisfaction score (z-score < −2.5). Two other participants were identified as outliers due to having reported a very low germane load (z-score < −2.5). One participant was identified as an outlier due to a very long total viewing time (z-score > 2.5). After excluding outliers, data from 40 and 42 participants in the experimental and control groups were retained for analysis.

We checked whether participants in the two groups differed significantly in age, sex, actual, and perceived prior knowledge, and motivation. The results were considered significant at the p < .05 level. A chi-square test of independence revealed no significant association between condition and sex, X² (1, N = 82) = .98, p = 0.32. Independent t-tests indicated no significant difference in participant age between the experimental (M = 20.35, SD = 0.90) and control groups (M = 20.50, SD = 0.74), t(80) = −0.83, p = 0.41. No significant difference in actual prior knowledge was detected between the experimental (M = 75.63, SD = 9.88) and control groups (M = 76.67, SD = 11.88), t(80) = −0.43, p = 0.69.

In addition, we detected no significant difference in perceived prior knowledge between the experimental (M = 4.25, SD = 2.22) and control groups (M = 4.21, SD = 2.19), t(80) = .07, p = .94. Furthermore, the experimental (M = 6.20, SD = 2.17) and control groups (M = 6.74, SD = 2.34) showed no significant difference in motivation to study the PowerPoint topic, t(80) = −0.1.08, p = .28. The similarity between groups indicated that we could proceed to examine the effect of the intervention on learner perceptions, video engagement, and learning performance.

What impact do embedded questions in pre-class videos have on learner perceptions?

A series of independent t-tests were performed to identify differences in cognitive load between the experimental and control groups (Table 1). No statistically significant difference in intrinsic load, t(80) = 1.06, p = 0.29, extraneous load, t(80) = 1.29, p = 0.20, or germane load, t(80) = −0.81, p = 0.42, was observed between groups. No significant differences in emotional engagement, t(80) = −0.85., p = 0.40, or satisfaction between groups, t(80) = −0.04, p = 0.97 was found. Additionally, we found no significant difference between groups in the judgement of learning, t(80) = −0.07., p = 0.95. In sum, the embedded questions did not affect learner perceptions, including cognitive load, emotional engagement, satisfaction, or judgement of learning.

Table 1.

Differences in learner perceptions between the groups.

	Mean	SD	95% CI for mean
	Mean	SD	Lower	Upper
Intrinsic load
Experimental	11.45	2.51	10.65	12.25
Control	10.88	2.34	10.15	11.61
Extraneous load
Experimental	7.30	2.50	6.50	8.10
Control	6.57	2.61	5.76	7.38
Germane load
Experimental	10.08	1.14	9.71	10.44
Control	10.29	1.22	9.91	10.66
Emotional engagement
Experimental	11.33	3.17	10.31	12.34
Control	11.93	3.26	10.91	12.95
Satisfaction
Experimental	4.88	.72	4.64	5.11
Control	4.88	.67	4.67	5.09
Judgement of learning
Experimental	7.35	1.90	6.74	7.96
Control	7.38	2.21	6.69	8.07

How do embedded questions in pre-class videos impact video engagement?

We also performed an independent t-test to identify differences in total views and total viewing time between groups (Table 2). In Table 2, total views and total viewing time are the data recorded in the LMS. Total views refer to the average number of times that the 11 instructional videos were clicked in by each participant in either the experimental or control group. Total viewing time refers to the average number of minutes that the 11 instructional videos were watched by each participant in either the experimental or control group.

Table 2.

Differences in video engagement between the groups.

	Mean	SD	95% CI for mean
	Mean	SD	Lower	Upper
Total views
Experimental	322	248	243	401
Control	385	251	307	463
Total viewing time
Experimental	152.75	92.61	123.13	182.37
Control	201.60	93.66	172.41	230.78

The results indicated no significant difference in total views between the experimental and control groups, t(80) = −1.15., p = 0.25. Participants who viewed question-embedded videos spent significantly less time watching videos than participants in the control group who did not, t(80) = −2.37., p = 0.02. Cohen’s d effect size was .52, indicating a medium-sized effect (Cohen, 1992). In sum, embedded questions did not influence the number of views but significantly reduced the total viewing time for participants in the experimental condition.

The number 322 in Table 2 indicates that each participant in the experimental group clicked 322 times on the 11 instructional videos on average, which does not mean that the 322 times were full-length views. That is, the participant in the experimental group clicked on each video about 29 times on average. A plausible explanation for the participants’ frequent access is that procedural knowledge acquisition requires repeated reference and application of the targeted knowledge (Ohlsson, 2012). This would cause the learners to frequently refer to certain information in the instructional videos when completing tasks. The number 152.75 in Table 2 implies that each participant in the experimental group watched the 11 instructional videos for 152.75 minutes on average. That is, each participant in the experimental group watched each video for an average of 14 minutes.

How do embedded questions in pre-class videos impact learning performance?

To identify differences in knowledge retention between groups, we performed an independent t-test (Table 3). No significant difference in knowledge retention between the experimental and control groups, t(80) = −0.59, p = 0.55, was apparent. The minimal difference between the pre- and post-knowledge test scores for the experimental and control groups occurred because the post-knowledge test contained a higher number of difficult questions. Although they all had a total score of 100, the prior- and post-knowledge tests consisted of 20 and 50 questions, respectively. As participants had never learned this topic, the prior-knowledge test was designed to be easier than the post-knowledge test.

Table 3.

Differences in learning performance between the groups.

	Mean	SD	95% CI for mean
	Mean	SD	Lower	Upper
Knowledge retention
Experimental	74.05	9.12	71.13	76.97
Control	75.19	8.25	72.62	77.76
Knowledge transfer
Experimental	78.72	5.15	77.07	80.36
Control	77.58	4.01	76.33	78.83

Discussion

This study investigated whether embedded questions in pre-class instructional videos have a detectable effect on learner perceptions, video engagement, and learning performance in an authentic flipped learning environment where learners could watch the videos repeatedly and at their own pace and were informed that learning performance tests were upcoming. We found that participants under the question-embedded condition spent significantly less time watching pre-class videos. A plausible explanation is that the embedded questions optimised attentional distribution and improved visual search efficiency. Research measuring eye movement indicates that providing cues in instructional videos can prompt learners to look at the referenced information significantly faster, rather than longer (Chisari et al., 2020; Jarodzka et al., 2012). In this study, the embedded questions may have served as cues and efficiently directed participants’ attention to corresponding content areas for further processing. This speculation is partially substantiated by recent research showing that individuals who watched videos under the question-embedded condition demonstrated a shorter time to first fixation on content areas corresponding to embedded questions (Yang et al., 2021). Time to first fixation signifies the amount of time it takes someone to first pay attention to an area of interest on the screen and provides information about visual search speed and which aspects of a scene are prioritised (Neta et al., 2017). Other research has revealed that interpolated testing in video lectures can reduce mind wandering and increase task-relevant behaviour such as note-taking (Schacter and Szpunar, 2015). In this vein, it is likely that the embedded questions scaffolded learners in sustaining attention and efficiently pinpointing the exact key information they needed, thereby substantially reducing the amount of time they spent watching the videos.

The present study revealed that learning performance did not differ between participants assigned to the experimental and control conditions. This finding diverges from past research showing that the presence of questions in videos can enhance learning performance (Haagsman et al., 2020). Although speculative, a possible explanation for this difference is that the knowledge type moderated the effects of question-embedded videos on learning performance. While videos transmitting declarative knowledge have been the primary focus of past research (Haagsman et al., 2020; Rice et al., 2019; van der Meij and Bӧckmann, 2020), the pre-class videos in our study fostered learning of procedural knowledge. Knowledge type moderates academic performance in video-based learning. For instance, Hong et al. (2018) observed that presenting the instructor’s image in videos facilitated students’ declarative knowledge learning but not their procedural knowledge learning. Success in learning procedural knowledge may require frequent access to the pre-class videos, as reflected in the fact that total views were equally high for the experimental and control groups, with no statistically significant difference observed between them in that regard. Frequent access may have suppressed the beneficial effects of embedding questions in pre-class instructional videos. To verify this assumption, future research could manipulate pre-class videos to include the two knowledge types as within-subject variables and the presence of questions as a between-subjects variable.

An alternative explanation for the lack of influence of the intervention on learning performance involves the flipped classroom in-class sessions compensating for the absence of embedded questions in pre-class videos with the control group. Most previous studies found that question-embedded videos had positive effects on learning and were conducted in the laboratory (van der Meij and Bӧckmann, 2020) and online education settings (Vural, 2013). In flipped learning, in-class sessions can help learners recognise knowledge gaps and build more complex cognitive schemata of course content (Förster et al., 2022). Unlike declarative knowledge which is acquired via discourse and observation, the acquisition of procedural knowledge is refined via practice (Ohlsson, 2012). The various in-class practice activities coupled with frequent learner-instructor interactions may have offset the cognitive benefits derived from the question-embedded videos watched before class. Future research could investigate this possibility by manipulating the online mode of learning where students have no access to face-to-face sessions facilitated by instructors as an additional experimental condition.

In this study, we found no obvious differences in learner perceptions between the experimental and control groups. The reasons could be twofold. First, the pre-class video-embedded questions were all true-false questions. Although the embedded questions impart useful information and knowledge points, merely inserting true-false questions might not be effective enough to reduce learners’ cognitive load and improve their emotional engagement, the judgement of learning, and satisfaction. It is argued that boundary conditions be considered to determine how video design principles work with different learners and teaching contexts (Deng and Gao, 2022). Question type, for example, could be potential boundary conditions that affect the effectiveness of embedded questions (van der Meij and Bӧckmann, 2020). Second, a series of pedagogical activities were designed in the flipped classroom to help participants understand the content of the pre-class videos. These pedagogical activities may have compensated for the absence of embedded questions in videos watched by the control group, resulting in few differences in learner perceptions between the experimental and control groups.

There were certain limitations of the present study that can be improved on in future research. First, allowing students to continuously access videos before, during, and after class could mitigate the effects of embedded questions on student learning and learner perceptions. As instructional videos are designed to be repeatedly used by learners, we cannot limit the number of times learners watch a video. A future research direction is to explore how to enhance learning performance and experience by implementing appropriate educational interventions in flipped classrooms when learners have continuous access to pre-class videos. Second, the results on the prior knowledge of the experimental and control groups were 75.63 and 76.67 respectively. Although the difficulty level of the pre-test was set lower than that of the post-test, this may leave readers with an impression that the participants had fair levels of prior knowledge. Future research should consider reusing pre-test questions in the post-test or keeping the difficulty of the prior- and post-knowledge tests at the same level.

Conclusions

This study offers new insights into the effects of embedded questions in pre-class videos transmitting procedural knowledge on learner perceptions, video engagement, and learning performance in the flipped learning environment. We found that videos interpolated with true or false questions had a limited effect on student learning in flipped classrooms – embedding questions in pre-class videos significantly reduced total viewing time but exerted no detectable effect on cognitive load, emotional engagement, satisfaction, judgement of learning, total views, knowledge retention, or knowledge transfer. No empirical studies to date have observed that question-embedded videos have any detrimental effects on student learning. The positive effects of embedded questions in videos, on the other hand, appear to vary by learning mode (laboratory, traditional, online, flipped) and type of knowledge transmitted in the videos (declarative, procedural). It is paramount that researchers identify boundary conditions for interpolating instructional videos with questions rather than indiscriminately applying this design strategy.

Supplemental Material

sj-docx-1-alh-10.1177_14697874231167098 – Supplemental material for Effects of embedded questions in pre-class videos on learner perceptions, video engagement, and learning performance in flipped classrooms

Supplemental material, sj-docx-1-alh-10.1177_14697874231167098 for Effects of embedded questions in pre-class videos on learner perceptions, video engagement, and learning performance in flipped classrooms by Ruiqi Deng and Yifan Gao in Active Learning in Higher Education

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by the National Natural Science Foundation of China (Grant number: 72204072).

ORCID iDs

Ruiqi Deng

Yifan Gao

Supplemental material

Supplemental material for this article is available online.

Author biographies

Ruiqi Deng has a PhD from University of Queensland, an MSc in Education from University of Oxford, and a BITHM (Hons) from University of Queensland. His research interests are massive open online courses (MOOCs), educational multimedia design and technology-enhanced learning.

Yifan Gao earned a PhD from University of Auckland and a BS from Chongqing University. His primary research interests include occupational psychology, virtual reality-based construction safety training and multimedia learning.

References

Akçayır

(2018) The flipped classroom: A review of its advantages and challenges. Computers & Education 126: 334–45.

Alemdag

(2022) Effects of instructor-present videos on learning, cognitive load, motivation, and social presence: A meta-analysis. Education and Information Technologies 27: 12713–42.

Alqurashi

(2018) Predicting student satisfaction and perceived learning within online learning environments. Distance Education 40(1): 133–48.

Bordes

Walker

Modica

, et al. (2021) Towards the optimal use of video recordings to support the flipped classroom in medical school basic sciences education. Medical Education Online 26(1): 1841406.

Brod

(2020) Generative learning: Which strategies for what age? Educational Psychology Review 33(4): 1295–318.

Cheng

Taylor

Williams

, et al. (2016) Student satisfaction and perceptions of quality: Testing the linkages for PhD students. Higher Education Research & Development 35(6): 1153–66.

Chisari

Mockevičiūtė

Ruitenburg

, et al. (2020) Effects of prior knowledge and joint attention on learning from eye movement modelling examples. Journal of Computer Assisted Learning 36(4): 569–79.

Cohen

(1992) A power primer. Psychological Bulletin 112(1): 155–9.

Dart

(2020) Khan-style video engagement in undergraduate engineering: Influence of video duration, content type and course. In: Proceedings of the 31st annual conference of the Australasian Association for Engineering Education, Sydney, Australia, 6–9 December 2020. Australia: Engineers Australia.

10.

Davis

Chen

Hauff

, et al. (2018) Activating learning at scale: A review of innovations in online learning strategies. Computers & Education 125: 327–44.

11.

Deng

Benckendorff

(2017) A contemporary review of research methods adopted to understand students’ and instructors’ use of Massive Open Online Courses (MOOCs). International Journal of Information and Education Technology 7(8): 601–7.

12.

Deng

Benckendorff

Gannaway

(2020a) Learner engagement in MOOCs: Scale development and validation. British Journal of Educational Technology 51(1): 245–62.

13.

Deng

Benckendorff

Gannaway

(2020b) Linking learner factors, teaching context, and engagement patterns with MOOC learning outcomes. Journal of Computer Assisted Learning 36(5): 688–08.

14.

Deng

Gao

(2022) A review of eye tracking research on video-based learning. Education and Information Technologies. Epub ahead of print 7 December 2022. DOI: 10.1007/s10639-022-11486-7

15.

Desai

Kulkarni

(2022) Assessment of interactive video to enhance learning experience: A case study. Journal of Engineering Education Transformations 35: 74–80.

16.

Dunlosky

Nelson

(1992) Importance of the kind of cue for judgments of learning (JOL) and the delayed-JOL effect. Memory & Cognition 20(4): 374–80.

17.

Faul

Erdfelder

Buchner

, et al. (2009) Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods 41(4): 1149–60.

18.

Fiorella

Mayer

(2015) Eight ways to promote generative learning. Educational Psychology Review 28(4): 717–41.

19.

Förster

Maur

Weiser

, et al. (2022) Pre-class video watching fosters achievement and knowledge retention in a flipped classroom. Computers & Education 179: 104399.

20.

Guo

(2019) The use of an extended flipped classroom model in improving students’ learning in an undergraduate course. Journal of Computing in Higher Education 31(2): 362–90.

21.

Haagsman

Scager

Boonstra

, et al. (2020) Pop-up questions within educational videos: Effects on students’ learning. Journal of Science Education and Technology 29(6): 713–24.

22.

Hiebert

Lefevre

(1986) Conceptual and procedural knowledge in mathematics: An introductory analysis. In: Hiebert

(ed.) Conceptual and Procedural Knowledge: The Case of Mathematics, vol.2. Hillsdale, NJ: Erlbaum, pp.1–27.

23.

Hong

Yang

(2018) Learning declarative and procedural knowledge via video lectures: Cognitive load and learning effectiveness. Innovations in Education and Teaching International 55(1): 74–81.

24.

Zhang

Gao

, et al. (2020) Big data analytics for MOOC video watching behavior based on Spark. Neural Computing and Applications 32(11): 6481–89.

25.

Jarodzka

Balslev

Holmqvist

, et al. (2012) Conveying clinical reasoning based on visual observation via eye-movement modelling examples. Instructional Science 40(5): 813–27.

26.

Kovacs

(2016) Effects of in-video quizzes on MOOC lecture viewing. In: The third ACM conference on learning at scale, Edinburgh, Scotland, 25–26 April 2016. New York, NY: ACM.

27.

Lee

Choi

(2019) Rethinking the flipped learning pre-class: Its influence on the success of flipped learning and related factors. British Journal of Educational Technology 50(2): 934–45.

28.

Leppink

Paas

Van der Vleuten

, et al. (2013) Development of an instrument for measuring different types of cognitive load. Behavior Research Methods 45(4): 1058–72.

29.

Mayer

Fiore

(2014) Principles for reducing extraneous processing in multimedia learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity principles. In: Mayer

(ed.) The Cambridge handbook of multimedia learning. 2nd ed. Cambridge: Cambridge University Press, pp.279–315.

30.

Mayer

Fiorella

Stull

(2020) Five ways to increase the effectiveness of instructional video. Educational Technology Research and Development 68(3): 837–52.

31.

Neta

Tong

Rosen

, et al. (2017) All in the first glance: First fixation predicts individual differences in valence bias. Cognition and Emotion 31(4): 772–80.

32.

Ohlsson

(2012) Procedural knowledge. In Seel

N. M.

(ed.) Encyclopedia of the sciences of learning. Berlin, Germany: Springer, pp.677–81.

33.

Pulukuri

Abrams

(2021) Improving learning outcomes and metacognitive monitoring: Replacing traditional textbook readings with question-embedded videos. Journal of Chemical Education 98(7): 2156–66.

34.

Rice

Beeson

Blackmore-Wright

(2019) Evaluating the impact of a quiz question within an educational video. TechTrends 63(5): 522–32.

35.

Schacter

Szpunar

(2015) Enhancing attention and memory during video-recorded lectures. Scholarship of Teaching and Learning in Psychology 1(1): 60–71.

36.

Spanjers

IAE

Könings

Leppink

, et al. (2015) The promised land of blended learning: Quizzes as a moderator. Educational Research Review 15: 59–74.

37.

Strelan

Osborn

Palmer

(2020a) The flipped classroom: A meta-analysis of effects on student performance across disciplines and education levels. Educational Research Review 30: 100314.

38.

Strelan

Osborn

Palmer

(2020b) Student satisfaction with courses and instructors in a flipped classroom: A meta-analysis. Journal of Computer Assisted Learning 36(3): 295–314.

39.

Sweller

(2010) Element interactivity and intrinsic, extraneous, and germane cognitive load. Educational Psychology Review 22(2): 123–38.

40.

Szpunar

Khan

Schacter

(2013) Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences USA 110(16): 6313–7.

41.

Torres

Pulukuri

Abrams

(2022) Embedded questions and targeted feedback transform passive educational videos into effective active learning tools. Journal of Chemical Education 99(7): 2738–42.

42.

Tweissi

(2016) The effects of embedded questions strategy in video among graduate students at a middle eastern university. Doctoral dissertation, Ohio University, OH.

43.

Ulrich

Helms

Frandsen

, et al. (2019) Learning effectiveness of 360° video: Experiences from a controlled experiment in healthcare education. Interactive Learning Environments 29(1): 98–111.

44.

van der Meij

Bӧckmann

(2020) Effects of embedded questions in recorded lectures. Journal of Computing in Higher Education 33: 235–54.

45.

Vural

(2013) The impact of a question-embedded video-based learning tool on e-learning. Educational Sciences: Theory and Practice 13(2): 1315–23.

46.

Wang

Antonenko

Dawson

(2020) Does visual attention to the instructor in online video affect learning and learner perceptions? An eye-tracking analysis. Computers & Education 146: 1–16.

47.

Xiu

Moore

Thompson

, et al. (2019) Student perceptions of lecture-capture video to facilitate learning in a flipped classroom. TechTrends 63(4): 369–75.

48.

Yang

Zhang

, et al. (2021) Students’ achievement motivation moderates the effects of interpolated pre-questions on attention and learning from video lectures. Learning and Individual Differences 91: 1–9.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB