Abstract
We report the findings of a randomized controlled trial examining the use of content acquisition podcasts for teachers (CAP-Ts) with 94 preservice teacher participants from two public universities. This study is an extension of a previous CAP-T study. We examined whether active embedded questions in CAP-Ts increased participant knowledge of functional behavioral assessments (FBAs). Participants completed a pretest, treatment, and posttest assessment to measure the extent to which CAP-Ts with active embedded questions supported participant knowledge, improved application of FBA skills, and impacted social validity as compared with traditional CAP-Ts. Findings indicate that CAP-T with embedded questions did not produce significant changes in preservice teacher knowledge and application of skills. Limitations and future directions are discussed.
Over the past 20 years, a growing number of students with disabilities access inclusive settings and the general education curriculum (McLeskey et al., 2012). Notably, in 2015–2016, approximately 81% of students with disabilities spent at least 40% of their school time inside a general education classroom (U.S. Department of Education, National Center for Education Statistics, 2017b). In the same year, 63% of the 347,000 students receiving services under the category of emotional disturbance (ED) were served in general education classes for at least 40% of the school day (U.S. Department of Education, National Center for Education Statistics, 2017b). Although students receiving special education services under the category of ED are a small portion of the overall student population (less than 1%), researchers estimate 12% of students exhibit moderate-to-severe emotional and behavioral disorders (EBD) and 20% with mild-to-severe manifestations at a given point in time (Forness et al., 2012; U.S. Department of Education, National Center for Education Statistics, 2017a). These students exhibit some of the poorest outcomes in our education system, including persistent negative interactions with teachers, peer rejection, severe impairments with language and communication, mental health concerns, community isolation, substance abuse, and experiences with the juvenile justice system (Blake et al., 2012; Chow, 2018; Chow & Wehby, 2018, 2019; Dunlap et al., 2006; National Research Council and Institute of Medicine, 2009; Wagner et al., 2005).
Given students with or at risk for EBD are likely to receive their education in a general education setting, the need for knowledgeable educators who understand practices to improve the behavior of students with or at risk for EBD is imperative. In this article, we evaluate the effects of two variants of a multimedia instructional tool designed to teach preservice teachers foundational content related to the functional behavioral assessment (FBA) process.
FBA
The FBA process is grounded in applied behavior analysis and designed to identify the function (i.e., purpose) of problem behavior, as well as the actions that predict the occurrence of the problem behavior (J. O. Cooper et al., 2020). A behavior intervention plan based on an FBA is a promising individualized intervention for addressing challenging behaviors and school engagement (U.S. Department of Education, 2016). The information collected from the FBA is used to drive an individualized, data-based behavior intervention aimed at improving specific outcomes for one targeted student.
An FBA is often considered a Tier III intervention in the context of multitiered systems of support (MTSS), a framework that promotes the use of a continuum of evidence-based practices. MTSS starts with Tier I as universal prevention and high-quality instruction for all students and Tier II as targeted strategies for small groups of students. Tier III provides the most intensive and individualized level of support to address challenging behaviors (Sugai & Horner, 2009). These supports are utilized for addressing persistent behavior that have been unresponsive to less intensive (i.e., Tiers I and II) tactics. As part of the FBA process, direct observational and indirect data are collected to determine a functional understanding of a student’s behavior (J. O. Cooper et al., 2020). Direct observation data include the recording of antecedent–behavior–consequence along with measures of behavior (e.g., frequency, duration, intensity). Indirect measures such as interviews and rating scales provide additional details and contextual information for support teams to analyze and form a hypothesis for the function of a challenging behavior (J. O. Cooper et al., 2020).
FBA-based interventions are effective supports for students with EBD and challenging behaviors. For example, Gage et al. (2012) conducted a meta-analysis of 69 FBA single-case design studies with students with or at risk for EBD. Results indicated that FBA-based interventions were an effective practice for students with EBD in schools with a statistically significant reduction in problem behaviors with an average reduction 70.5%. This underscores that FBA-based interventions conducted in the general education setting are particularly effective in general education settings (Gage et al., 2012). Walker et al. (2017) conducted a meta-analysis of FBA-based interventions in inclusive school settings. They identified 27 studies indicating that FBA-based interventions led to the reduction of challenging behavior. Using two single-case effect size calculations (i.e., nonoverlap of all pairs and Tau-U), Walker et al. (2017) identified the statistically significant improvements in appropriate behavior and reductions in challenging behaviors when classroom teachers implemented a FBA-based interventions intervention within a whole group arrangement.
Although FBAs have been studied and their efficacy evaluated for decades (e.g., Bijou & Baer, 1961; Carr et al., 1999), teachers report minimal knowledge of FBA practices, as many do not receive formal training on the use of FBAs and FBA-based interventions. In a review of state accreditation and teacher preparation programs, strategies for behavior assessment (i.e., FBA) were present in only 16% of elementary and 9% of secondary programs (Freeman et al., 2014). In course syllabi from 26 special education teacher preparation programs, Oliver and Reschly (2010) found only 27% of university programs included an entire course devoted to classroom management. Findings indicated that only a small percentage of preservice teachers receive training related to behavioral interventions and FBAs. Research suggests that lack of training on classroom management and behavioral interventions such as FBAs, can affect in-service teacher knowledge and implementation (J. T. Cooper et al., 2018; Gable et al., 2012; Reinke et al., 2011). For example, Gable et al. (2012) surveyed 3,060 teachers to determine their perceptions of their preparation related to working with students who engage in challenging behavior. Out of the 15 practices surveyed, teachers indicated they were least prepared to use FBA-based interventions. Specifically, only 43.5% for special educators and 34% of general educators indicated they were well prepared to implement formal procedures to develop FBA-based interventions. This is of particular concern as FBAs, and subsequent behavior intervention plans, are critical behavior supports for students with challenging behavior. As a field, we must identify and promote research-based methods that increase knowledge and use of FBAs and FBA-based interventions.
Recent calls have been made for researchers to consider cost-effective interventions, and federal agencies have recently required them for projects aimed at testing the promise of evaluating efficacy and effectiveness (U.S. Department of Education, 2018). This will allow for active, prospective planning and consideration of the cost relative to the benefit of an intervention or practice. At the same time, increasing the likelihood that interventions will work to improve the outcomes for educators and students with attention to resources and resource allocation. In regards to higher education, teacher educators are tasked with covering a wide range of critical content over a short period of time. This prompts one to consider flexible and effective technology tools for delivering content (Kennedy & Thomas, 2012). To address this call, researchers have created and evaluated cost-effective methods such as multimedia. Multimedia instruction is a promising approach for improving preservice teacher knowledge on academic instruction as well as behavioral practices due to the pragmatic and flexible nature of delivery (Ely et al., 2014; Kennedy et al., 2017). Instructors can assign the content outside of scheduled face-to-face time, allocating more time to discussion-based and interactive learning. Multimedia instruction also provides high-quality content delivery for remote and online learning environments. The current research examines the promise of a multimedia tool designed to address gaps in teacher knowledge on FBA.
Content Acquisition Podcasts for Teachers (CAP-Ts)
CAP-Ts are multimedia-based instructional videos highlighting key steps or aspects of a specific evidence-based practice (Kennedy & Thomas, 2012). CAP-Ts are not intended to replace a lecture, reading, or training; instead, they provide foundational content as an advance organizer (Kennedy & Thomas, 2012). The brief CAP-T videos (average 5–10 min) are designed to maximize retention by reducing the cognitive load of the learner (Mayer, 2009). Key CAP-T components include high-quality instruction with relevant on-screen pictures, occasional text, and narration with the goal of building the viewer’s declarative knowledge of a topic (Kennedy, Alves, et al., 2015; Kennedy, Wagner, et al., 2015).
CAP-T videos are grounded in Mayer’s (2008) instructional design principles and cognitive theory for multimedia learning (CTML; Mayer, 2009). Mayer’s (2008, 2009) CTML and 12 evidenced-based design features provide guidelines to reduce extraneous cognitive processing and maximize active learning processes. According to Mayer (2009) learners utilize their visual and auditory input concurrently within working memory, therefore learning is maximized when text on the screen matches the auditory content verbatim. In addition, Mayer (2008) provides instructional design principles aligned with the CTML. Each of the 12 principles has been tested and demonstrate effect sizes ranging from .52 to 1.39. For example, CAP-Ts are created to adhere to the coherence principle by only containing relevant content. For a detailed description of Mayer’s instructional design principles as implemented within CAP-Ts see Kennedy, Wagner, et al., 2015 as well as Kennedy et al. (2011).
To align with the CTML theoretical framework, the CAP-T content is broken into segments (i.e., Mayer’s segmenting principle) as noted by passive pauses in the videos. The segmenting principle emphasizes pauses in content to assist with processing by providing time between sections, embedding questions, or splitting content into separate CAP-Ts. This design principle asserts “people learn better when the multimedia message is presented in user-paced segments rather than a continuous unit” (Mayer, 2009, p. 175).
To demonstrate, a CAP-T may be divided into four sections (each 2–3 min in length). At the end of each section, the learner is asked to passively respond to one to five questions. In previous studies, their response is not required or recorded. Rather, CAP-Ts provide questions with a pause. The learner is able to proceed to the next section with or without responding to the embedded questions. Interestingly, few studies have explored the effects of the segmenting principle (embedded questions vs. passive pause), and much of the work has been outside preservice teacher education. For example, Cheon et al. (2014) explored active-pause (active embedded questions) versus passive-pause with 96 undergraduate students in a computer literacy course. The results of the randomized controlled trial indicated that the participants in the active-pause condition outperformed the participants in the passive-pause condition with an instructional animation film. Despite this promising finding, no research has evaluated the CAP-Ts’ segmenting principle with active embedded questions versus passive-pause questions between sections within preservice teacher education.
Empirical Support for CAP-Ts
Research to date on the use of CAP-Ts with preservice teachers has shown CAP-Ts as an effective tool to promote knowledge of a topic related to teaching students with disabilities (e.g., Hirsch et al., 2015; Kennedy et al., 2011, 2015). In 2014, CAP-Ts were designated as an evidence-based practice for improving preservice teachers’ knowledge and learning (Dieker et al., 2014). To date, there are 15 published CAP-T randomized controlled trials with significant effects aimed at improving preservice teachers’ knowledge about topics such as phonological awareness (Kennedy et al., 2013), curriculum-based measures (Kennedy et al., 2015), vocabulary instruction (Peeples et al., 2018), positive behavior supports (Kennedy et al., 2011), and FBAs (Hirsch et al., 2015; Kennedy et al., 2016). Kennedy et al. (2016) served as a pilot study for the current research.
Kennedy et al. (2016) used Mayer’s (2008) principles to develop and test the efficacy of CAP-Ts with respect to preservice teacher knowledge and application of FBA practices as well as perceived cognitive load as measured by NASA–Task Load Index (NASA-TLX). In this study, the authors compared the learning outcomes of participants randomly assigned to the CAP-T condition or a traditional lecture control condition. The treatment group watched two CAP-Ts, the first focused on definitions, while the second focused on the basics of FBA procedures. The control group received the same content via traditional lecture, which was a PowerPoint with text and pictures. Although there was a significant main effect (d = .26), the authors report a significant time by group interaction, suggesting that both groups made average gains in FBA knowledge, but participants in the CAP-T group made greater gains and learned faster. The lecture group reported a significantly higher overall workload score than the CAP-T participants, suggesting increased cognitive load could inhibit performance on the dependent measure of learning. Cognitive load was a predictor of outcomes for both conditions, and posttest scores were found to decrease by .35 points for every 1-point increase in NASA-TLX score. The results of this study suggest Mayer’s (2009) CTML theory can be used in multimedia driven instruction in the form of CAP-Ts to address issues of perceived cognitive load. Results also indicate CAP-Ts can support learners in additional types of introductory coursework by activating and enhancing prior knowledge.
In a replication, Hirsch et al. (2015) tested the efficacy of CAP-Ts to teach key concepts of the FBA process to preservice teachers across three universities. The procedures were the same as Kennedy et al. (2016); however, modifications to CAP-Ts were made based on expert feedback. Specifically, experts suggested adjustments to the CAP-Ts scripts (i.e., vocabulary, sequence of material). The participants who watched the CAP-Ts did slightly better than those who learned via live lecture, with a moderate effect size effect (d = .45), and participants also reported both CAP-T and lecture formats were acceptable forms of learning. Although the effect size was moderate, it is promising and prompts one to continue to pursue this avenue of research.
The Present Study
Although CAP-Ts have improved preservice teacher knowledge in the area of FBAs, there is still much to learn. Mayer’s segmenting principle provides learners with passive-pauses in the video content along with checks for understanding. Active embedded questions (or opportunities to respond [OTR]) are considered an evidence-based practice for supporting student academic outcomes (MacSuga-Gage & Simonsen, 2015; Partin et al., 2010) and within multimedia instruction (Reiser & Tabak, 2014). An active embedded question prohibits a learner from advancing through content until an active response to questions as attempted, acting as scaffolds that prompt learners to activate the new content, helping them to construct deep knowledge (Sawyer, 2014). Having learners stop periodically allows for critical reflection and engagement in metacognitive processes where they can assess their understanding before moving forward (Reiser & Tabak, 2014). Including these questions in CAP-Ts provides students with opportunities to reflect on their understanding and engage in self-regulated learning.
As reviewed earlier, researchers have tested and replicated CAP-T procedures across a variety of topics and settings (e.g., Dieker et al., 2014). However, to date, studies have yet to examine the effects of specific components, such as the segmenting principle with active embedded questions. Thus, this study aims to extend previous studies (e.g., Hirsch et al., 2015; Kennedy et al., 2016) by isolating the effects of a specific, theory-driven component of the CAP-T that aligns with calls to conduct studies as a method to better understand the features of an intervention (Coyne et al., 2016; Therrien et al., 2016).
In this study, we hypothesized that adding active embedded questions would have beneficial effects on preservice teacher learning in the context of FBA content. As such, we asked the following research questions: To what extent do CAP-Ts with active embedded questions support preservice teacher knowledge about FBAs compared with CAP-Ts with passive embedded questions? Are there differences in the preservice teachers’ social validity regarding the use of CAP-Ts?
Method
Participants and Setting
All participants were undergraduate preservice teacher candidates in general education and special education programs enrolled in an introductory special education course at two public universities in the United States. University A (n = 37) is a public flagship university in the Southeast, and University B (n = 57) is a public flagship university in the Mid-Atlantic. All activities were approved by both university institutional review boards (IRB) prior to the beginning of the study. One student from University A opted out of data collection. The final study sample (N = 94) included 80 females (85%) and 14 males (15%). The age range was from 18 to 36 years with an average age of 21.75 years (SD = 3.62). Only 2.13% (n = 2) of participants reported previous FBA experience. All researchers for the present study either have a PhD in special education or are enrolled in a special education doctoral program. See complete participant demographic data in Table 1.
Participant Demographics.
Note. Percentages are based on the number of participants who completed the demographic survey. CAP-T = content acquisition podcasts for teachers; OTR = opportunities to respond; FBA = functional behavioral assessment.
Secondary education = English, science, mathematics, social studies education. bOther major = architecture, business, art, psychology.
We selected introductory special education courses for the present study sample because it was a requirement for the teacher preparation program at both Universities and the only special education content requirement. Thus, it was likely that participants had minimal knowledge of FBAs. In each University, the instructor of the course did not teach related content, nor did they provide specific details about the research until the beginning of the study. To protect against potential bias, graduate students not affiliated with the courses introduced the study to the students. Students were informed that the study activities were a part of the course curriculum and a requirement, but they could opt out of data being included in the study analysis. Participants were provided with a unique identification number. Instructors were blinded to whether a student opted to participate.
Procedures
Design
We used a randomized two-group pretest–posttest-maintenance design. Pretest measures were collected 1 week prior to CAP-T administration, posttest measures were collected immediately following the CAP-T, and follow-up measures were collected 1 month after CAP-T administration. Researchers at each site randomly assigned students to a condition using Excel™’s random number generator. After randomization, the participants completed a pretest to assess baseline understanding of FBAs. The results of an analysis of variance (ANOVA) showed there were no significant differences in FBA knowledge between the groups at pretest, F(1, 92) = .871, p =.35, indicating successful randomization. In addition, there were no significant differences between participants’ reported technology attitudes, F(1, 90) = .322, p =.672.
Independent variables
In the present study, participants were randomly assigned to two groups. Participants in Condition 1 watched two CAP-Ts; participants in Condition 2 watched the two CAP-Ts with active embedded OTR. For the purposes of the present study, we will refer to Condition 1 as CAP-T and Condition 2 as CAP-T + OTR. Both conditions contained the same FBA content.
For Condition 1 (CAP-T), we used a previously developed CAP-T (Hirsch et al., 2015) which utilized Mayer’s (2008) multimedia design principles and have been shown to improve FBA content knowledge and application skill in a similar sample (see Table 1 in Hirsch et al. [2015] for a detailed overview of development and content). The 25 min, 17 s content was divided into two CAP-Ts: FBA definitions (10 min, 15 s) and FBA basics (14 min, 52 s). This division allowed adherence to the segmenting principle (Mayer, 2008). Participants were able to pause the video, rewatch segments, and record notes. This condition also included 22 instances of passive questions for participants and did not require a response. With that, an OTR occurred at a rate of .88 per minute (less than 1 per minute). The average time participants in Condition 1 took to complete CAP-Ts was 32 min (SD = 4.64). The CAP-Ts are available at https://edpuzzle.com/open/enetmue .
For Condition 2, we adapted the CAP-Ts used in Condi-tion 1 to include required active embedded questions (i.e., OTRs) throughout the videos. Both conditions included the same content, combination of still images (e.g., graphs, pictures, text), audio narration, and 22 embedded instances where the audio narration paused, and participants were provided with an OTR. The 22 questions were the same in both conditions, however the CAP-T + OTR condition, required participants to answer a content area question prior to advancing to the next slide. Two members of the research team reviewed the CAP-T + OTRs to ensure embedded prompts functioned appropriately. Identical to Condition 1, participants in Condition 2 were able to pause the video, rewatch segments, and record notes. The average time participants in Condition 2 took to complete CAP-T + OTR was 41 min (SD = 5.69). The CAP-Ts + OTR are available at https://edpuzzle.com/open/enetmue .
Research protocols
All research activities, including data collection, during three in-person class sessions, were intentionally scheduled to occur the same week of the semester at each participating University. To ensure consistency across sites, RAs read the same brief script for administration. This brief introduction oriented students to the CAP-Ts and told students to begin.
During Session 1, participants in both conditions completed outcome and technology attitudes measures as well as participant demographics. In Session 2, we administered CAP-T videos via EdPuzzle which allowed us to ensure participants accessed the correct CAP-T for their condition. Edpuzzle is an online platform that allows instructors to assign videos and monitor whether students watch the video. All participants were instructed to watch the CAP-Ts (Condition 1) or CAP-T + OTR (Condition 2), take notes, pause and replay any sections of the videos. Although both groups watched the CAP-Ts, Condition 2’s response items were embedded using the EdPuzzle tool. The embedded questions were a mixture of 11 open ended and 11 multiple choice items. The prompts would automatically stop at specified intervals, and the response item would appear directly below the video. Participants had the option to respond to the item or rewatch the previous segment of the video before proceeding. Response feedback was provided to the CAP-T + OTR group (Condition 2).
After watching the videos, participants completed the posttest outcome measure. At both sites, we allocated 1 hr for participants to complete their assigned CAP-T, and doctoral-level research assistants (RAs) monitored and encouraged students to work at their own pace. Session 3 was our maintenance session 1 month (4 weeks) after Session 2, and participants all completed the same outcome measure as they previously did in Sessions 1 and 2.
It is important to note that while we made many efforts to ensure consistency across sites, there were a few differences between the administration procedures at each site. First, at University A, all students brought personal computers and Session 2 was conducted in one location. Students at University B did not all have access to personal computers for this study. Thus, at University B, students who had their own personal computers brought them to the class session, and we reserved the computer lab for students who did not have their own computers. This resulted in University B having two locations for Session 2 administration. Related to this issue, because all students had computers at University A, they completed three data collection activities (Sessions 1–3) electronically. At University B, Session 1 and Session 3 data were recorded via paper-pencil, and data were entered electronically by RAs at University B. RAs at University B entered the data collected via paper-pencil using the same electronic forms as participants at University A to ensure consistency in data entry. We randomly selected 25% of these paper-pencil data from University B and double entered them for reliability purposes. Agreement between these 25% of the data sample was 100%. We compared University A and B’s pre- and posttest findings for the intervention group (Condition 2). There were no statistically differences in FBA knowledge on the pretest between students from University A (M = 21.67, SD = 8.33) and University B (M = 18.88, SD = 9.10), F(1, 40) = 1.16, p = .286 or at posttest for University A (M = 70.28, SD = 15.52) and University B (M = 64.39, SD = 16.10), F(1, 45) =1.52, p = .224.
Measures
All measures were collected in the same order in a similar format across sites during the same week of the semester. We collected standard demographic information as well as pretest, posttest, and follow-up data to assess the extent to which preservice teacher candidates improved knowledge and application skills related to the FBA process. Social validity data were collected to gather the participants’ views. We also collected a measure of preservice teachers’ attitudes toward technology to provide a descriptive measure of participants and to better describe the present study sample. We used this assessment of attitudes to further establish pretest group equivalence on a variable more directly related to our intervention.
Outcome measure
This study measured the relative effects of CAP-T versus CAP-T + OTR using a 25-item assessment of FBA knowledge and application. We administered this same measure three times: at pretest, posttest, and 1 month following the end of intervention. We adapted the measure from a previous CAP-T FBA study. See Kennedy et al. (2016) for information on the original measure. Modifications included an increased number of application questions with scenarios and decreased the number of multiple-choice items. Prior to conducting the study, three doctoral level scholars reviewed the modified 25-item measure. Reliability for the measure in the present study was α = .78. For a copy of the FBA knowledge measure, contact the first author.
The multiple-choice section (knowledge and application); included 16 items, each scored one point per correct item. The following is an example of a multiple-choice knowledge question, A review that determines whether or not a child’s behavior that led to the disciplinary action is linked to his or her disability is known as a__________. (a) Manifestation determination review, (b) disciplinary adjustment review, (c) determination for assessment review, and (d) functional behavior determination review. The following is an example of a multiple choice application question, Analyze the following scenario and identify the target behavior: In Earth Science, when asked to read aloud, Noah often gets out of his seat, walks around the room, and jokes with peers. Noah’s peers laugh and talk to him as he walks by: (a) Read aloud, (b) Peers laugh and talk to Noah, (c) Out of seat, walks around the room, jokes with peers, (d) Noah’s parents brought him to school.
The open-ended assessment (application) had nine items that were eligible for scores ranging from 0 to 2 points. Example items include: (a) Explain the purpose of an FBA, (b) Name three reasons to complete an FBA, (c) Please tell us what you know about “A–B–C data collection,” and (d) Please tell us what you know about “operational definitions of behavior.” Items were scored independently by two authors. To ensure consistency, a scoring guide with acceptable responses was used to score each open ended response. Responses were compared, and any discrepancies were resolved to ensure 100% scoring accuracy. Several participants did not complete the entire 25 item measure, therefore a percentage correct was calculated.
Technology attitudes
To describe our included sample, we assessed participants’ experiences with and attitudes toward technology for learning. We also used this measure to test for group equivalence to rule out the possibility that attitudes toward and use of technology influenced our findings. During pretest, participants completed four Likert-type items on statements related to their reliance on and attitudes toward technology in the context of college learning environments. Using a 5-point Likert-type scale where 1 = never and 5 = always participants were asked to report whether they: (a) believe they can learn from online modules presenting new content, (b) believe the use of technology improves learning in the college classroom (e.g., electronic polls, clickers, interactive presentation), (c) enjoy when professors use technology in the class, and (d) rely on technology to learn outside of school/work (e.g., Coursera, YouTube, Khan Academy). Reliability for the measures in the present study was a = .72.
Social validity
To assess the acceptability of the CAP-Ts and CAP-T + OTR, participants completed six Likert-type items on statements related to the appropriateness, suitability, and effectiveness of the intervention. This is the same measure employed by Hirsch et al. (2015). Immediately following the study, participants were asked to assess the extent to which their activity (i.e., CAP-Ts or CAP-Ts + OTR): (a) worked well for their learning preferences, (b) was appropriate for teachers learning about FBAs, (c) was worth recommending to other students, (d) provided them with confidence in their entry-level knowledge of FBAs, and (e) was an effective way to learn new content. Reliability for the measure in the present study was α = .91.
Data Analysis
We replicated the data analysis procedures from Hirsch et al. (2015) and conducted all analyses using SPSS Version 25 (IBM Corp, 2016). Due to a large percentage of participants missing one item, participant scores were calculated to provide a percentage of items scored correct. We used an ANOVA to test for significant group differences in FBA knowledge and application. Next, we calculated standardized mean difference effect sizes (Cohen’s d) and conducted a descriptive analysis of participants’ modal responses to the social validity items.
Results
FBA Knowledge and Application of Skills
The first and second research question addressed whether there were significant differences in participants’ knowledge and application of FBAs between our study conditions. We used a one-way ANOVA to evaluate group differences at three timepoints: pretest, posttest, and follow-up. Initially, we randomized 94 participants; because of class absences, the final pool of participants who completed the posttest was 91. Two absent participants were from the CAP-T + OTR (experimental) condition, and one was from the CAP-T (control) condition. The results of the ANOVA did not reveal significant differences between the participants in CAP-T + OTR (n = 50, M = 20.88, SD = 10.86) and CAP-T groups (n = 44, M = 19.01, SD = 8.10) on the pretest, F(1, 92) = .871, p = .35.
Both groups increased their mean knowledge scores after intervention. However, mean scores for the CAP-T + OTR group (Mpost = 65.81, SD = 17.3) were not significantly different from the CAP-T group (Mpost = 71.95, SD = 12.6) at posttest, F(1, 89) = 3.67, p = .059. One month later, 75 participants (CAP-T [n = 39]; CAP-T + OTR [n = 36]) completed the same measure. The mean scores for the CAP-T + OTR group (Mfollow-up = 36.3, SD = 13.7) were not significantly different from the CAP-T group (Mfollow-up = 37.8, SD = 15.03) at posttest, F(1, 74) = .212, p = .646 (see Note 1).
We elected to include a descriptive measure of the participants’ technology attitudes at pretest to rule out the possibility that the participant’s technology attitude could influence results. Results indicated there was not a significant difference (p = .572) between the participants technology attitudes in CAP-T + OTR (M = 16.2, SD = 2.9) and CAP-T groups (M = 15.8, SD = 3.1). To interpret, we can rule out technology attitudes as a mechanism through which intervention effects would have differed. This also indicates our randomization worked.
Social Validity
As noted in the “Method” section, immediately following the study, participants were asked to complete a brief social validity survey about their experiences watching the CAP-T + OTR or CAP-T. A total of 77 participants completed the social validity measure. Participants ranked their perceptions of their condition (i.e., CAP-T or CAP-T + OTR) on a 6-point, forced choice Likert-type scale with 1 = strongly disagree and 6 = strongly agree. To determine how participants viewed their activity, we computed reports of modal responses for each item along with the frequency of modal responses, and range of responses. Table 2 presents each item. For Items 1 to 2, participants in the CAP-T group most frequently “slightly agreed” that (a) the format of the FBA activity worked well for their learning preferences and (b) most teachers would find this an appropriate way for learning about FBAs. The same items were frequently rated higher by the participants in the CAP-T + OTR group with the most frequent score “agreed.” On Item 3, regarding suggesting the use of the FBA activity to other students, participants in the CAP-T + OTR group reported they “strongly agreed” whereas the CAP-T group most frequently reported they “agree.” For Items 4 and 5, participants in both groups most frequently “agreed” (a) following the activity, they felt confident in their entry-level knowledge and (b) the format of the instruction was an effective way for them to learn new content.
Modal Distribution of Social Validity Data.
Note. Participants responded to items given a 6-point Likert-type scale with 1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = slightly agree, 5 = agree, 6 = strongly agree. Frequency is the number of times participants gave the modal response option. Range is the lowest-to-highest rating response. CAP-T = content acquisition podcasts for teachers; OTR = opportunities to respond; FBA = functional behavioral assessment;
Discussion
Given the recent emphasis on replication within special education (Coyne et al., 2016; Makel et al., 2016), we conducted a systematic extension of the methods used in previous CAP-T studies. In this randomized controlled trial, we compared the effects of CAP-Ts with CAP-Ts plus embedded OTRs. In addition, we elected to include a technology attitudes measure for descriptive information about our sample, as no prior CAP study has included a measure of technology use and/or attitudes.
We intended to find differences between groups as seen in prior studies (i.e., Hirsch et al., 2015; Kennedy et al., 2016). However, pre- and posttest FBA knowledge and application measures completed by the participants indicated no significant differences between the two groups. In the following section, we outline possible explanations for the findings along with future research.
Although both groups increased their knowledge and application of FBA concepts, there was not a clear effect based on the condition. The results of this study are unlike results from other CAP-T studies. We reviewed Hirsch et al. (2015) to compare the participant scores. The Hirsch et al. (2015) measure included 26 items whereas ours included 25. Due to a large percentage of participants missing one item, we elected to calculate the percentage of correct answers out of the total number of questions the participant answered. Although the Hirsch et al. (2015) CAP-T and lecture participants had higher pretest scores (32% and 33%, respectively), their posttest scores were lower than our sample (61.53% and 57.73% respectively). It is important to note the 7-day window between Hirsch and colleague’s experiment and posttest.
For example, Kennedy et al. (2016) compared CAP-Ts with a lecture while studying participants’ FBA knowledge and application along perceived cognitive load. In Kennedy et al. (2016), the CAP-T outperformed the lecture group while demonstrating that participants in the CAP-T group reported a lower perceived cognitive load. Previous CAP-T studies adhere to Mayer’s (2009) CTML and design principles including limiting extraneous processing (Kennedy & Thomas, 2012). In addition, findings from other studies (i.e., Cheon et al., 2014), indicate the active pauses (i.e., embedded questions) enhance recall and scheme construction as learners have to encode information from previous segments. Although both groups were presented with questions throughout the videos, only the CAP-T + OTR group was required to stop and respond. The CAP-T group could proceed (continue to watch the CAP-Ts) without responding. Therefore, the active responses may unintentionally increase cognitive load for the CAP-T + OTR group. This study prompts one to consider whether the CAP-T + OTR group violated Mayer’s CTML. That is, frequently pausing to respond to questions may have provided, irrelevant or extraneous information as described in the coherence principle. Although we did not measure cognitive load, requiring the CAP-T + OTR participants to respond to questions may have caused the group to experience higher levels of cognitive load.
Another explanation for the lack of between-condition differences could be a large amount of potential heterogeneity in how students completed the embedded OTR. We did not instruct students explicitly on what to do at each OTR. There were instructions as a part of the CAP-T + OTR delivery, but students could have approached them differently. For example, we could have instructed students to do their best to remember the content or application presented to them right before each forced pause. Students could have ignored the prompt and elected to proceed with the CAP-T; we did not monitor participants on whether they actually attempted to use the forced pause to better understand the content. Future research on embedded OTRs should include observational or interview procedures to ensure or monitor that intentionally learning is taking place, or to offer an explanation for why embedded OTRs did not produce greater learning gains than CAP-Ts without them. Despite no statistically-significant group differences, the present study suggests that both conditions effectively increased participant FBA knowledge. This is promising and in line with the practical implications of using inexpensive multimedia instructional methods to increase preservice teacher knowledge in important content areas in special education.
Relative to social validity, the overall reports deemed the intervention a positive one for the mode of delivery and for student learning. A noteworthy observation of these data, however, was that participants reported ranges of 3 to 6 for “format worked well for my learning preferences” and “effective way for me to learn,” but reported a wider range of 1 to 6 for “suggest use for others.” It may be that participants believed the CAP-Ts supported their own learning, but perhaps the material was not as engaging for some participants and subsequently reported lower ratings when asked about suggesting the use for others.
Limitations
One of the primary limitations of the present study was the absence of a no-treatment control group. Because of the complexities involved with conducting experimental research and the parameters around our IRB’s approval, all participants received some version of the intervention. Specifically, one of the participating university’s IRB required us to provide the instructional content to all participants, but allowed us to embed the OTRs. We relied on previous efficacious results of the CAP-T research literature to date and we posited that contrasting either CAP-T or CAP-T + OTR group with a no-treatment control group would have yielded statistically significant and practically meaningful differences. Thus, we proceeded with two contrasting intervention conditions where all participants would receive the CAP-T instruction. However, the active experimental between-condition difference of embedded OTRs within the CAP-T platform might have been too similar to yield meaningful differences.
In terms of site differences, our study had to use different methods of data collection and entry. At one site, all students had computers that they had access to and brought them to class every session. Thus, at University A, all data were collected electronically at all time points. At University B, which was the larger of the two, not all students had personal computers. To administer the CAP-Ts, the research team reserved laptop computers for intervention delivery. However, at pretest and follow-up, researchers collected data via paper-and-pencil and hand-entered the data. Although this difference and data collection did not likely bias any between-group comparisons due to successful random assignment, it is possible that this difference in method influenced how students responded to answers at University B.
Finally, this study did not measure whether watching a CAP-T or CAP-T + OTR improved implementation of FBA practices. Future research in this area should examine whether watching a CAP-T improves preservice teachers’ implementation of FBA practices. In a recent study, Peeples et al. (2018) evaluated whether multimedia plus performance feedback improved participants application of practices related to vocabulary instruction. Researchers could explore topics such as identifying the function of behavior and matching the intervention to behavioral function.
Future Directions
Results from the present study suggest several avenues for future work. First, more targeted research in designing powerful embedded content within CAP-T’s will provide researchers and teacher educators the methods and resources to produce educationally effective multimedia methods of content area instruction. Related, future studies should consider combining online and offline forms of instruction. To date, researchers have tested the effects of multimedia versus multimedia conditions (the present study) and multimedia versus lecture conditions (Hirsch et al., 2015). Given the increases in hybrid models of higher education programming, are these CAP-T instructional modules effective if delivered as content during an online course? In addition, would a unit following the CAP-Ts that include enriched examples and discussion of the knowledge and application of FBAs significantly enhance performance? These questions should be experimentally evaluated to add to the knowledge base and help determine the boundaries of the effectiveness of multimedia instruction in teacher education.
Given the general efficacy of primary applications of CAP-T iterations, future studies should also examine the relative effects of CAP-Ts versus lecture/reading administered via online-only instruction—and they should do this in the context of cost-effectiveness research. Relatedly, funding agencies of educational research are now requiring explicit cost-effectiveness studies to be a part of educational research on efficacy and effectiveness. Therefore, it is important to better understand how multimedia interventions and programs fit into the landscape of cost analysis will help provide important information on the value of technologically enhanced instruction as well as the effectiveness of remote or distance educational formats (i.e., completely online course).
Another potential line of inquiry is to supplement the quantitative work with rigorous qualitative and mixed methods; this should also include involving students, future learners, and current teachers in the development of CAP-Ts. Enhancing the product and design around the method of delivery by directly including learners and stakeholders may maximize the utility of instruction overall.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
