Abstract
The use of multimedia-driven instruction in college courses is an emerging practice designed to increase students’ knowledge. However, limited research has validated the effectiveness of using multimedia to teach students about functional behavioral assessments (FBAs). To test the effectiveness of a multimedia tool called Content Acquisition Podcasts (CAPs), this study utilized a pretest–posttest design across two groups of students. One group received instruction on FBAs in the form of a CAP, whereas the other group received a typical lecture (control comparison). Results revealed that members of the CAP group performed better on the posttest compared to the students who received the lecture when the pretest scores were controlled for previous knowledge. In addition, students in the CAP group had lower self-reported levels of cognitive load. Implications for the use of CAPs and future research are discussed.
Keywords
University instructors responsible for teaching undergraduates in introductory-level courses face many challenges. For one, student background knowledge for some topics is understandably absent. However, the lack of existing knowledge puts a burden on the instructor with respect to filling gaps before new content can be mastered (Gaff & Lambert, 1996). Instructors receive little, if any training on how to meet the needs of undergraduates taking their first introductory courses (Loughran, 2014). On a related point, student interest and engagement in courses are likely to be variable given the interaction between their career aspirations and the content of the course. In other words, university instructors frequently find themselves providing instruction to students who are not motivated to dedicate the needed time and energy toward learning content (Sales et al., 2007). As noted above, instructors are experts in their field but not necessarily in how to get undergraduates to share their passion. Therefore, augmenting student engagement can be a challenge. Third, instructors face challenges in deciding the scope and breadth of content within introductory courses. Introductory courses are a good recruiting tool for bringing undergraduates into majors and programs. Thus, ensuring introductory courses provide a positive experience for students without sacrificing rigor can be a priority for programs or departments. In sum, university instructors who teach introductory-level courses would benefit from instructional tools known to help engage learners and provide high-quality instruction (Zeichner, 2005).
Content Acquisition Podcasts
University instructors would benefit from access to multimedia-based instructional materials that are grounded in high-quality instructional design theories and are supported by empirical evidence. The intervention introduced and used in this empirical study, Content Acquisition Podcasts (CAPs; Kennedy & Thomas, 2012), is grounded in evidence-based instructional design principles (Mayer, 2008) and relevant theory (Mayer’s Cognitive Theory of Multimedia Learning [CTML], 2009). CAPs are short, multimedia-based instructional vignettes that can provide high-quality instruction in a short amount of time and help build students’ essential knowledge of core content in any course (Kennedy, Alves, & Rodgers, 2015). CAPs combine still images and occasional text with carefully crafted narration for various topics spread across relatively short periods of time (e.g., average of 5–10 min vignettes). CAPs are easy to create (see Kennedy, Alves, & Rodgers, 2015 for printed and online instructions) and for students to use during learning depending on user preferences and needs. This manuscript reports results from the 12th empirical replication of CAPs’ capacity to support student learning (Kennedy, Hart, & Kellems, 2011; Kennedy & Thomas, 2012; Kennedy, Ely, Thomas, Pullen, Newton, Ashworth, Cole, & Lovelace, 2012; Kennedy, Driver, Pullen, Ely, & Cole, 2013; Driver, Pullen, Kennedy, Williams, & Ely, 2014; Ely, Pullen, Kennedy, Hirsch, & Williams, 2014; Kennedy, Thomas, Aronin, Newton, & Lloyd, 2014; Sayeski, Kennedy, de Irala, Clinton, Hamel, & Thomas, 2015; Hirsch, Kennedy, Haines, Thomas, & Alves, 2015; Kennedy, Wagner, Stegall, Lembke, Miciak, Alves, Brown, Driver, & Hirsch, 2015; Hart & More, 2013; Sayeski, Kennedy, Clinton, de Irala, & Thomas, 2015). This article is the first to include a measure of user’s perceived cognitive load to help explain why CAP viewers outperform peers who learn using other methods on tests of knowledge and skill application.
Like many introductory university courses, including psychology, the introductory course in special education is jam-packed with content. Therefore, instructors must prioritize certain content and design learning experiences accordingly. Although our setting is an introductory course in special education, university instructors in psychology and other domains face similar responsibilities and challenges with respect to supporting student-learning issues within introductory-level coursework.
Weaknesses of generic podcasts
Upon review of the emerging literature on podcasts in higher education (see Hew & Cheung, 2013 for a recent review), it is clear that most researchers evaluate social validity instead of empirical impact on learning. To illustrate, many researchers make student satisfaction surveys the key dependent measure in podcasting studies instead of directly measuring of how podcasts impact learning (Heilesen, 2010). This phenomenon is well known to readers of Teaching of Psychology across numerous domains and interventions (see Griggs & Collison, 2013). In addition, audio-only, enhanced (visuals in time with audio), or video podcasts do not inherently reflect any specific learning theory and are not automatically shaped by evidence-based instructional design principles (Kennedy, Thomas, Aronin, Newton, & Lloyd (2014); Heilesen, 2010). CAPs differ in that Mayer’s CTML (2009) and accompanying evidence-based instructional design principles (2008) are used to shape the looks and sounds of CAPs. A sample CAP that also introduces Mayer’s principles is available at: https://vimeo.com/89716786.
Multimedia instruction is a known perpetrator for overwhelming viewers with fast-paced, visually rich, instructionally redundant features that make no explicit effort to structure content so that viewers have time for processing (Clark, 2009; Mayer, 2009). The CTML and accompanying instructional design principles are designed to help instructors do a better job of creating instruction that is a match for how people learn (Mayer, 2008). Although there is an excellent empirical record demonstrating the impact of Mayer’s model on student learning outcomes (see Mayer, 2009), less is known about the impact of multimedia created using a theory intended to address induced cognitive load. Logic holds that learning is suppressed when induced cognitive load is higher and vice versa (Sweller, Ayers, & Kalyuga, 2011), but this should be tested experimentally and replicated across multimedia-based interventions being used to teach different content.
Utility of CAPs
CAPs are not intended as a replacement for lecture or assigned weekly readings. Instead, instructors can assign CAPs to supplement weekly homework assignments. To illustrate, students can opt to review a CAP as an advance organizer prior to reading the course textbook or review the CAP after reading (Kennedy, Ely, Thomas, Pullen, Newton, Ashworth, et al., 2012).
Three independent groups of researchers have published empirical results in numerous topic areas (Kennedy, Wagner, Stegall, et al., 2015; Hart & More, 2013; Sayeski et al., 2015). In each of the previous CAP studies, participants were randomly assigned to either watch a CAP or read a practitioner-friendly article containing the same information (see Kennedy, Kellems, Thomas, & Newton, 2015 for a review). A weakness of this line of research is the homogeneity of instruction for students in the comparison condition of each study. Students in the comparison condition read a practitioner-friendly article/chapter to receive content.
The current study
In the current study, we compared a CAPs condition to a live lecture condition, the latter chosen given the common use of lecture during university teaching. As a topic for the CAPs and live lectures, we chose an important topic in special education (and in psychology), functional behavior assessments (FBAs). FBA is a topic studied and implemented by behavior analysts who are frequently trained within courses offered by psychologists and their departments. As one example, Applied Behavior Analysts use FBAs regularly in their practice. Previous studies did not attempt to evaluate students’ perceived cognitive load as a way to help understand performance following watching a CAP or reading. In this study, we used the NASA-Task Load Index (NASA-TLX; Hart & Staveland, 1988) as a proxy for cognitive load in order to evaluate possible differences between participants in the two groups.
Method
Participants
Fifty-six students (10 men, 46 women) enrolled in an introduction to special education course at a large public university in America’s mid-Atlantic region participated in the study. The participants included 50 undergraduate and 6 graduate students. Students are required to have at least a 3.0 grade point average prior to admittance to the program following their second year at the university.
Prior to completing the experiment, 44 (78.6%) participants reported that they had not received any previous training on FBAs, the topic of this unit. A smaller percentage (N = 12; 21.4%) reported that they had received information on FBAs previously through another university course or participated in an in-service on FBAs. Students who reported previous FBA experience were equally distributed across the two conditions prior to randomization. The Results section contains additional detail on the need for and success of randomization.
Measures
During the study, students completed two dependent measures. The first is a researcher-created measure of FBA knowledge and ability to apply skill. The second is a proxy measure of cognitive load where students self-report their perceived load following instruction.
FBA knowledge and application measure
To measure participant knowledge of FBA practices, we created and administered a 24-item instrument based on FBA practices that were disseminated through instruction (CAP and lecture). The instrument contained multiple-choice, open-response, and application questions. This assessment was administered as a pretest 1 week prior to the experiment and as a delayed posttest approximately 2 hr after the experiment. A reliability analysis of the items demonstrated Cronbach’s α for the instrument was .74, which is in the acceptable range for social science research and given the limited number of participants in this experiment.
Cognitive load measure
To evaluate participant’s cognitive load, researchers asked participants to complete the NASA-TLX (Hart & Staveland, 1988) immediately following their mode of instruction. Researchers have used the NASA-TLX in over 300 published studies (Hart, 2006), and the instrument continues to be regularly used in research (e.g., Wastlund, Norlander, & Archer, 2008; Wiebe, Roberts, & Behrend, 2010). At the conclusion of a task, the participant completes the NASA-TLX, which contains six questions. The student rates himself or herself on a performance scale for each domain ranging from low to high. The online computer system (www.NASATLX.com) generates a score for each of the domains based on the location of the students’ clicks along the scale. The computer application version provides an overall measure of workload or mental load on a scale of 0–100 (Sharek, 2009).
Procedures
All research activities occurred during regularly scheduled class time. Following consent, students were asked to complete a pretest in one sitting independently without class materials. One week later, the students were randomly assigned (using a random number generator) to one of two conditions: (a) CAP or (b) lecture. Students who reported previous experience learning about FBAs were stratified equally into the two groups.
The CAP was created using Mayer’s principles (2008) and divided into two parts to adhere to Mayer’s segmenting principle: definitions (8 min 5 sec) and FBA details (11 min 42 sec). The CAPs are available online, Part 1: https://vimeo.com/111015222 and Part 2: https://vimeo.com/111015778. The lecture (37 min 15 sec) contained the same material in CAP and was delivered using a traditional PowerPoint with text (M = 24.08 words per slide, range = 1–56 words, SD = 13.5). A research assistant with 5 years of experience as a special education teacher and 2 years of experience as a teaching assistant delivered the lecture. The slides used by the research assistant in the lecture matched the content of the CAPs. Students in the CAP condition were relocated to a separate room and completed their task at individual laptop terminals with headphones. After completing the lecture or watching the CAPs, students came back together and received a 45-min lecture on a different topic, which served as a distractor. At the conclusion of the distracting lecture, all students completed the posttest in class.
All measures were completed and scored electronically. The open-response questions were scored independently by the second and third authors. Reliability was conducted on 100% of assessments. Interobserver agreement was calculated by dividing the number of agreements by the sum of the number of agreements and disagreements and multiplying by 100 (agreements/(agreements + disagreements) × 100), with a 94.79% score recorded.
Results
This study used an experimental, pretest–posttest design. Scores reported herein for the combined knowledge and application measure are unadjusted raw scores. In addition, Levene’s statistic for evaluating homogeneity of results was not significant for any analysis.
FBA Knowledge and Application
Pretest results
As noted in the Method section, students who reported prior experience with FBA were randomly stratified into the two conditions. This decision was essential, as students who reported prior FBA training (N = 12, M = 16.9, SD = 2.3) scored significantly higher on the pretest than students without prior FBA training (N = 42, M = 10.6, SD = 3.2), F(1, 53) = 40.0, p < .001, d = 2.26. This large effect size should be interpreted with caution due to the small and unequal sizes of the groups; however, the point remains that these groups were significantly different from one another and stratification on prior FBA training was necessary.
Results from the pretest also show there was no significant difference between those who had prior FBA training in the CAP group (N = 6, M = 16.0, SD = 2.2) and in the lecture group (N = 6, M = 17.8, SD = 2.2), F(1, 11) = 2.1, p = .17, d = −.82. Thus, randomization was successful in distributing students with prior knowledge across the two groups. When the students with prior FBA experience were included in the two groups, students in the CAP group (N = 27, M = 11.5, SD = 3.8) were not significantly different from students in the lecture group at pretest (N = 27, M = 12.5, SD = 4.2), F(1, 53) = .844, p = .362, d = −.25. However, because of the magnitude of difference between students with and without prior FBA experience, we control for this variable in posttest analyses.
Posttest results
To evaluate results at posttest, a mixed-model repeated-measures analysis of variance (ANOVA) was conducted. Repeated-measures ANOVA allows for the evaluation of growth in performance across two time points while controlling for other important variables. Prior experience with FBA (continuous variable) was used as the covariate. For the main analysis, the author evaluated group performance on the combined measure of FBA knowledge and application. There was not a significant main effect for group, F(1, 51) = 1.37, p = .248, ω2 = .03; however there was a significant main effect for time, F(1, 51) = 143.7, p < .001, ω2 = .74, and the interaction between group and time, F(1, 51) = 25.4, p < .001, ω2 = .33. To further examine the directionality of the significant interactions, simple effects tests were conducted.
A Bonferonni correction was used to evaluate the individual group differences at the two time points (.05/2 = .025). Aforementioned results note the nonsignificant group differences at pretest. However, there was a significant group difference at posttest, F(1, 53) = 8.3, p = .006, d = .26. Expressed another way, the CAP group (M = 17.9, SD = 2.1) had significantly higher scores on the posttest than the lecture group (M = 15.0, SD = 4.7). In addition, Cohen’s d indicates a small to medium effect size. This finding demonstrates that both groups made gains on FBA knowledge and application from pretest to posttest, but the gains by the CAP group were more robust.
Students with prior FBA experience in the CAP group (N = 6, M = 19.7, SD = 2.1) scored higher than students with prior FBA experience in the lecture group (N = 6, M = 18.0, SD = 1.4) at posttest, but the results were not statistically significant, F(1, 11) = 2.7, p = .131, d = .95 (interpret effect size cautiously due to small N). Thus, both groups made gains from pretest to posttest, although students in the CAP group learned at a faster rate. Regardless of group assignment, students with prior FBA training (N = 12, M = 18.8, SD = 1.9) continued to significantly outperform students without FBA training at posttest (N = 42, M = 15.8, SD = 4.1), F(1, 53) = 84.0, p = .016, d = .94, however the magnitude of difference closed significantly from pretest to posttest. All students made gains from pretest to posttest.
Results for Perceived Cognitive Load
To help determine whether the students’ reported cognitive load can help explain observed differences in learning, all students completed the NASA-TLX scale online following their group’s instruction. The lecture group (M = 40.3, SD = 19.2) had a significantly higher overall workload score than the CAP group (M = 23.9, SD = 12.7), F(1, 58) = 15.1, p < .001, d = 1.00. Given the significant difference in performance on the dependent measure of learning, this finding suggests higher cognitive load may function to suppress performance for students in the lecture group compared to those who watched the CAPs (see Figure 1).
Comparison of pretest and posttest scores for participants in lecture and Content Acquisition Podcast (CAP) conditions with overlaid regression line of self-reported cognitive load data using the NASA-Task Load Index (NASA-TLX) instrument (Hart & Staveland, 1988).
To further explore this relationship between cognitive load and learning, we conducted a simple linear regression. Results suggest that the total score on the NASA-TLX predicted a significant proportion of the total variation in posttest scores: β = 19.9, 95% confidence interval [15.5, 20.3], p = .007, adjusted R 2 = .107. In other words, regardless of experimental condition, a student’s score on the NASA-TLX is a good predictor of posttest score, F(1, 58) = 8.0, p = .007. Results show posttest scores are reduced by .35 points for every 1 additional point of cognitive load recorded on the NASA-TLX instrument.
Discussion
The results of this study support the use of CAPs as an effective method of instruction to increase student knowledge of FBA procedures in an introductory course. It is reasonable to expect similar results across a range of topics within other fields given the empirical record of CAPs as an instructional tool (Kennedy, Kellems, Thomas, and Newton, 2015). All students increased their knowledge of FBA procedures from pretest to posttest; however, the CAP group significantly outperformed the lecture group on the posttest measure. In addition, results from the NASA-TLX demonstrate that students in the lecture group experienced higher levels of perceived cognitive load compared to students in the CAP group, and that higher perceived cognitive load predicted lower posttest learning regardless of experimental condition.
Although CAPs are derived from Mayer’s CTML (2009), which is based on cognitive load theory (Chandler & Sweller, 1991), researchers have not measured the impact of multimedia-driven instruction on perceived cognitive load. Previous research using CAPs did not make any attempt to measure user’s self-reported cognitive load. Therefore, the results of this study advance how Mayer’s theory can be applied to create multimedia-based interventions such as CAPs. The cognitive load findings are preliminary and should also be replicated with a larger sample and with different content to provide additional data to support this interesting result.
The students with prior FBA knowledge significantly outscored their classmates at pretest and posttest, regardless of instructional condition. This result speaks to the importance of prior knowledge in introductory courses and the role CAPs can play in supporting preliminary learning. To elaborate, although students with prior knowledge had higher scores, the gap was closed at posttest, thus providing further support for CAPs as an important learning tool. In addition, students with prior knowledge in the CAP group scored higher than classmates with prior knowledge in the lecture group. CAPs can therefore be useful for any student in an introductory course and can help them improve knowledge.
Limitations and Future Directions
It is important to note that, although these data suggest positive effects for the intervention, some limitations of the study should be considered. First, the study did not evaluate the students’ application of knowledge in real-world educational settings. Thus, we are unable to determine whether the students would actually be able to perform the necessary steps to conduct an FBA. Second, this study compared CAPs to a traditional lecture with PowerPoint; by nature of PowerPoint’s inherent format, it may seem obvious that this technique would have higher perceived cognitive load. However, it is generally better to conduct an experiment to challenge such assumptions rather than allow perceptions to guide commonly accepted truths in the field. PowerPoint is broadly used in university teaching (see Craig & Amernic, 2006; Hardin, 2007), and slides typically contain content delivered via text (Masie, 2006). It was for this reason we selected lecture with PowerPoint as our comparison condition. In future studies, there are other permutations of instruction with and without PowerPoint to compare learning using CAPs.
Conclusion
Although the participants in this study are future teachers, and the content is specific to that aim, CAPs and our findings have relevance to instructors and researchers in the field of psychology. As noted, instructors who teach introductory courses, whether in psychology or teacher preparation, face similar challenges in terms of providing high-quality instruction that meets the needs of their novice students. CAPs is an instructional tool grounded in theory from the field of cognitive psychology, and this process can be adapted for delivering instruction within psychology coursework. In addition, FBA is a topic included in some psychology courses and in the training of Applied Behavior Analysts.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.