Abstract
Antecedent interventions are often used preventatively to address engagement, but few studies have evaluated their effectiveness with students at risk for disability. This study evaluated the effectiveness of two commonly recommended antecedent interventions that have limited empirical support for use with students at risk: social stories and visual supports. Both interventions were evaluated separately in the context of two single-case alternating treatments designs across five elementary students at risk for emotional and behavioral disorders. Social stories were not effective for improving students’ levels of engagement during targeted activities; visual supports resulted in increases in students’ engagement relative to baseline conditions. Future use of visual supports for students at risk should include evaluations of the feasibility of implementation by teaching staff.
Students who exhibit problem behavior are at risk for poor academic and social outcomes (Bulotsky-Shearer, Bell, Romero, & Carter, 2012; Crick et al., 2006). The presence of disruptive behavior in preschool is associated with lower engagement (Harden et al., 2000; Olson & Huza, 1993) that may continue in kindergarten (Searle, Sawyer, Miller-Lewis, & Baghurst, 2014). Conversely, high levels of engagement are generally associated with developmental and academic gains (Hofer, Farran, & Cummings, 2013; Howes, Sanders, & Lee, 2008; Vitiello & Williford, 2016; Williford, Vick Whittaker, Vitiello, & Downer, 2013). Furthermore, task engagement may mitigate the negative future academic outcomes associated with problem behavior for young students (McWayne & Cheung, 2009). Early intervention to support students’ engagement in classroom activities is particularly important for students exhibiting persistent problem behavior, as these students are at risk for disabilities such as emotional and behavioral disorders (EBD; Searle et al., 2014). Despite positive outcomes associated with early intervention, students identified as at risk have less access to early intervention services than those with identified developmental and physical disabilities (Fox, Dunlap, & Powell, 2002).
In addition, unlike students with developmental disabilities, preventive interventions for students at risk for EBD are likely to be implemented by general rather than special education staff, via multitiered systems of support (MTSS) models. Although use of MTSS models is supported by research, use of increasingly intensive interventions by general educators is associated with some barriers compared with use of these interventions by specialists (e.g., special educators). General education teachers have reported they are unfamiliar with most evidence-based practices for students with EBD (Stormont, Reinke, & Herman, 2011) and consequently select interventions based on convenience or familiarity (Gable, Park, & Scott, 2014). Thus, interventions designed to be implemented in general education classrooms (a) need to prevent the occurrence of problem behavior by increasing engagement and (b) require minimal specialized training or knowledge on behalf of the teacher. Antecedent interventions include practices designed to prevent the occurrence of problem behavior by increasing the likelihood of engagement (Blair, Umbreit, Dunlap, & Jung, 2007). Two antecedent interventions that are commonly recommended for use to improve engagement and that are feasible for use in general education classrooms are social stories and visual supports (Blair et al., 2007; Breitfelder, 2008; Lane, Barton-Arwood, Spencer, & Kallberg, 2007).
Social Stories and Visual Supports
Social stories (SS) are narratives written to make expected behaviors and setting characteristics more salient for specific routines, activities, or events students experience in which they do not engage in appropriate prosocial behaviors (Gray, 1994; Gray & Garand, 1993). Stories are read immediately before the activity in which a student is expected to display the targeted behaviors. SS may support students exhibiting problematic behaviors by providing precorrective feedback to identify expected behaviors for an upcoming activity or routine (Lane et al., 2007).
Visual supports are drawings, images, or materials (e.g., timers) added to the physical environment to provide information about the expected sequence or steps in an activity (Wong et al., 2014). Two common types of visual supports are visual activity schedules (VASs; Krantz, MacDuff, & McClannahan, 1993) and structured visuals (SVs; Hume & Odom, 2007). SVs include structured work boxes (SWB; Hume & Odom, 2007) and other materials for structuring participation in a variety of tasks (e.g., hundreds charts for participation in large group counting, menu of pictures depicting songs for a child to choose during circle time, folder system for organizing independent work). SVs can be provided in conjunction with VASs as part of a multicomponent intervention (Bennett, Reichow, & Wolery, 2011) or in isolation as a tool to organize expected tasks and increase opportunities to respond to improve engagement behaviors (Zimmerman, Ledford, & Severini, 2019).
Both SS and visual supports are designed to make expected behaviors more salient during activities and routines. However, visual supports differ from SS because they are present in the environment while the activity is ongoing. In addition, visual supports (e.g., SVs) may produce a predictable format for task organization and make opportunities to respond more salient, whereas SS state expected behaviors without organizing materials or opportunities to respond. Either of these formats may result in increases in the engagement of students at risk for EBD by making expectations, ways for students to engage with instruction, and materials more explicit before or during activities.
Some reviews suggest SS are an evidence-based practice (Wong et al., 2014), whereas others conclude they are not (Leaf et al., 2015; Zimmerman & Ledford, 2017). SS have been evaluated for use to improve the performance of typically developing young students displaying problem behaviors (Benish & Bramlett, 2011) and students with autism spectrum disorders (ASDs; Wong et al., 2014), but have not been evaluated with students displaying problematic behaviors who are at risk for disability. Inconsistent conclusions across reviews (Leaf et al., 2015; Wahman, Pustejovsky, Ostrosky, & Santos, 2019; Zimmerman & Ledford, 2017) may be due to procedural differences between studies (i.e., use of multicomponent interventions), differences across participants, and limited or variable study quality. Visual supports have been identified as evidence based for some populations (e.g., students with ASD; Wong et al., 2014) but are widely recommended for use with students with a variety of disabilities (Lawry, Danko, & Strain, 1999; Lequia, Machalicek, & Rispoli, 2012). However, only a few evaluations of VAS interventions for young students have included typically developing students (Watson & DiCarlo, 2016) or students at risk for disability (Zimmerman, Ledford, & Barton, 2017).
Both SS and visual supports have been assessed using a variety of formats and in combination with a number of other intervention components. For example, SS have been evaluated with and without comprehension questions (Schneider & Goldstein, 2009), and in paper (Lane et al., 2007) and electronic (Vandermeer, Beamish, Milford, & Lang, 2015) presentation. VAS presentation formats have varied and include book layouts (pages connected in a binder; Bryan & Gast, 2000), linear left to right traditional layouts (Zimmerman et al., 2017), and digital presentation (iPad©; Brodhead, Courtney, & Thaxton, 2017). Heterogeneity in outcomes due to these procedural variations is not well studied for either intervention, but multiple variations have been found to be effective (Benish & Bramlett, 2011; de Mers, Tincani, van Norman, & Higgins, 2009; Lequia et al., 2012; Schneider & Goldstein, 2009). Given there may be multiple effective variations, including student choice in intervention format may increase the likelihood students at risk for EBD will engage with the content delivered in the story or visuals.
Despite mixed evidence for improving outcomes for students, teachers frequently use SS, report they are easy to implement, and identify them as an acceptable intervention to improve student outcomes (Fees, Kaff, Holmberg, Teagarden, & Delreal, 2014; Wikete Lee, 2016). Similarly, teachers report frequent use of visual supports (Wikete Lee, 2016). Although teachers report using them, the conditions under which both of these interventions may be effective for students at risk for EBD are unknown.
Purpose and Research Questions
The purpose of this investigation is to evaluate two commonly used interventions designed to improve engagement: SS and visual supports. Specifically, the research questions to be addressed in the study are as follows:
Method
Participants
Target student participants
Participants included 5- to 7-year-old students who were at risk for EBD due to elevated levels of problem behavior and low levels of engagement. Seven students were nominated for participation by their teachers based on their inability to complete daily classroom activities and routines due to a performance deficit rather than a skill deficit (i.e., teachers reported students had demonstrated an ability to answer questions or gather center materials, but the students did not perform the tasks when requested during classroom activities or transitions). Two 30-min observations, a structured teacher interview, and teacher reports of engagement and problem behavior (via the Social Skills Improvement System–Rating Scales, SSIS-RS; Gresham & Elliott, 2008) were used to determine whether students met study inclusion criteria. The SSIS-RS has been used in prior studies to determine disability risk status for social delays (Zimmerman et al., 2017) and been demonstrated as an internally consistent and valid measure for problem behavior (Gresham, Elliott, Vance, & Cook, 2011).
Students were eligible for inclusion if they met the following criteria: (a) displayed below average engagement compared with age and gender norms on the SSIS-RS (Gresham & Elliott, 2008), (b) exhibited problem behavior at above average levels compared with age and gender norms on the SSIS-RS, (c) demonstrated consistent school attendance (no more than two absences per month on average), (d) demonstrated object–picture correspondence by matching an image of an object to a tangible representation of the same object (e.g., a picture of a pencil to a physical pencil), and (e) received instruction in a general education classroom during the targeted activity. Students were excluded if they met one or more of the following criteria: (a) diagnosis of an ASD or intellectual disability, (b) use of a VAS or SV in current support plan, and (c) aversion to physical prompting by an adult.
Five of the seven nominated students were deemed eligible to participate. One student was excluded from study participation due to above average engagement on the SSIS-RS, and one student was excluded due to teacher report of substantial improvements in engagement when class size was reduced from 26 to 18 students before the initiation of screening procedures. Two eligible first graders (Xander and Raven) and two eligible preschoolers (Marc and Michael) did not receive special education services. One eligible kindergartener (Jason) received special education services as a student with a developmental delay, although he did not meet criteria as a student with an intellectual disability or autism. One student received individual counseling once a week and utilized check-in check-out as a daily intervention (Raven), one was referred to receive counseling services during the study (Xander), and another was referred for special education evaluation during the last 2 weeks of the study (Marc). All students met age-level normative scores on the cognitive screening measure of the Battelle Developmental Inventory–Second Edition Normative Update (BDI-2 NU; Newborg, 2016). Teachers reported Xander, Raven, and Jason performed below grade level in all academic subjects; Marc and Michael were on or above grade level in all preacademic subjects.
Xander, Raven, and Michael displayed low levels of engagement during whole group activities, Jason displayed low levels of engagement during independent reading centers, and Marc did not engage in the classroom morning routine. All students eloped from designated locations; Jason and Michael also engaged in inappropriate peer interactions. The Functional Analysis Screening Tool (FAST; Iwata & DeLeon, 1996) was completed by the researcher in collaboration with the classroom teacher to identify the function of each student’s problem behavior. Results of the FAST indicated all students exhibited socially maintained problem behavior. Additional participant information can be found in Table 1.
Participant Descriptions.
Note. All participants spoke English as their primary language; Michael also spoke Spanish. FAST = Functional Assessment Screening Tool (Iwata & DeLeon, 1996); SSIS = Social Skills Improvement System (Gresham & Elliott, 2008); PB = problem behavior; K = kindergarten; PK = preschool; M = male; F = female; AA = African American; A = attention; T = tangible; E = escape; DD = developmental delay; W = White; H = Hispanic.
Scores presented as percentile ranks. bAge presented in years. cFamily income relative to the poverty line (U.S. Department of Health and Human Services Poverty Guidelines, 2017). dTeacher report of overall performance relative to grade level standards. eScores not calculated for preschool students.
Implementers
The first author, a doctoral candidate in early childhood special education and applied behavior analysis, and the third author, a first-year master’s student in the same program, implemented sessions. The first author had 10 years of experience, and the third author had less than 1 year of experience working with students at risk for EBD. The first author was the primary presession (see procedures below for specific details) implementer for Xander, Raven, and Jason, and the session implementer for Marc and Michael. The third author was trained by the first author and conducted presessions for Marc and Michael and sessions for Xander, Raven, and Jason. If either implementer was not available (this occurred on two occasions), a third graduate student in special education with less than 1 year of experience working with students at risk for EBD implemented sessions during both comparisons after observing sessions via video and practicing procedures with the first author. All implementers were trained to fidelity via modeling and role-play prior to study initiation.
Setting
Sessions were conducted in general education classrooms at two Title I schools (one early learning center serving 3- to 5-year-old students and one elementary school serving students in Grades PK-4) in a large, urban, public school district in the Southeastern United States. Intervention sessions occurred during whole group instruction (reading, Xander; math, Raven), independent reading centers (Jason), the morning-arrival routine (Marc), or morning meeting (Michael; activity selection procedures described below). One general education teacher and a researcher were present in the first-grade classroom (Xander and Raven); a general education teacher, a paraprofessional, and a researcher were present in the preschool (Marc and Michael) and kindergarten (Jason) classrooms.
All classmates not participating in research activities were in the targeted instructional area during Xander, Raven, and Michael’s sessions (12–20 total). Jason completed reading centers with one peer partner, and two to four students were present in the morning-arrival area during Marc’s sessions. Participants remained in close proximity to nonparticipating adults and students during sessions; participants were removed from the classroom during presessions. Presessions and preference assessments occurred in the hallway seated on the floor adjacent to the classroom door. One student and one researcher were present during presessions and preference assessments, although nonparticipating students and adults frequently walked nearby.
SS materials were only present during presessions. Visual supports were present during presessions and located in the area in which students were expected to complete targeted instructional activities. Exact positioning of the visual support was determined in conjunction with classroom teachers to increase the likelihood teachers would continue the intervention after study completion.
Kindergarten and first grade
Classroom centers including computers, writing cubbies, books, and a calm-down area were present in both classrooms. A kidney table and a large-group carpet area were also present in the classrooms. The kindergarten classroom had a large-group carpet below a SMARTboard in the front of the classroom with four rectangular student tables and child-sized chairs on the perimeter of the carpet. The first-grade classroom had the same organization with hexagonal student tables and child-sized chairs.
Preschool
The physical layout of the room included centers created via the arrangement of bookshelves, wooden cubbies, and tables. Student cubbies lined the entry of the classroom; this was the designated location of the morning-routine center. The large-group carpet area was adjacent to the back of the cubbies under a pull-down projector screen. An easel was placed at the back of the carpet to display the daily question and attendance sign-in for students.
Materials
Preference assessment, BL, and intervention sessions were recorded using a Canon Vixia Mini camera. Sessions were recorded in two video segments (presession and session), which allowed for coding by observers blind to study condition for the SS sessions. Visual supports sessions occurred in the same manner, although observers were not blind to study condition because the visual supports were visibly present or absent in the environment. The timer on the camera was used to monitor session length.
Social Stories
The content for each story was selected by the researcher in collaboration with the classroom teacher to reflect behaviors required to complete the targeted activity. Each story was created using guidelines from Gray (1994). Three comprehension questions were printed on a page accessed only by the researcher to assess comprehension of the content (Schneider & Goldstein, 2009). Descriptions of SS materials (preference assessment formats and intervention materials), including images of each material, can be obtained on Open Science Framework (https://osf.io/6jhgy/).
Visual supports
The visual supports included a VAS or SV. Interventions were selected by the researcher in conjunction with the classroom teacher based on observations and the nature of the target activity or routine using the decision-making diagram in Figure 1. Selected interventions are reported in Table 2. Descriptions of materials (all preference assessment formats and intervention materials), including images of each material, can be obtained on Open Science Framework (https://osf.io/6jhgy/).
Decision-making diagram for selecting visual support interventions.
Intervention Descriptions.
Note. Formats were selected by students via preference assessments. WG = whole group; CVC = consonant vowel consonant.
Jason completed one center per day, each with three activities (Task 1, Task 2, and reading books). bSelected format; no clear preference.
Response Definitions and Measurement Systems
The primary dependent variable across all comparisons was engagement. Although not reported in this manuscript, data for secondary variables including problem behavior, correct use of visual supports, and visual supports generalization can be obtained on Open Science Framework (https://osf.io/6jhgy/).
Choice during preference assessment
Choice during the preference assessment was defined as the participant grasping an item with one or both hands for at least 5 s, bringing an item to the participant’s body, or pointing to an item with a single finger or full hand. Primary data were collected in situ; reliability data were collected via video. Choice responses during the preference assessment were measured on a trial-by-trial basis using a paper-and-pencil data recording sheet. The researcher recorded item selection for each trial; if no selection was made, the response “NR” was recorded.
Engagement
Duration of engagement was estimated using a 5-s momentary time sampling (MTS) procedure across all conditions (Ledford, Lane, & Gast, 2018). MTS is highly accurate when compared with continuous recording (Cook & Snyder, 2019), particularly when intervals are smaller than or equal to the average duration per occurrence of the targeted behavior (Ledford, Ayres, Lane, & Lam, 2015). Visual analysis comparisons of engagement data collected using MTS with 20-s intervals and continuous duration resulted in similar conclusions (Cook & Snyder, 2019), thus a 5-s MTS procedure was selected rather than continuous recording, given the complexities of measuring engagement onset and offset (Ledford et al., 2018).
All data were collected via video using ProcoderDV (Tapp, 2003). Engagement definitions were adapted from previous evaluations of VASs and SWB (Bryan & Gast, 2000; Zimmerman et al., 2017; Zimmerman et al., 2019). Operational definitions, examples, and nonexamples of engaged and unengaged behaviors can be found in Table 3. The total percentage of intervals in which each participant was engaged was calculated for each session using the following formula: (total number of intervals in which the student was engaged/total number of intervals) × 100.
Engagement Operational Definitions, Examples, and Nonexamples.
Note. Engaged and unengaged definitions adapted from previous visual supports research (Bryan & Gast, 2000; Zimmerman, Ledford, & Barton, 2017; Zimmerman, Ledford, & Severini, 2019). Participants may exhibit any of the behaviors (a)–(e) to meet criteria for engaged or unengaged. If participant exhibited engaged and unengaged behavior simultaneously (looking at teacher while kicking peer), then unengaged behavior was recorded for the interval. Examples and nonexamples are nonexhaustive.
Experimental Design
Two sequential alternating treatments single-case research designs (ATDs; Barlow & Hayes, 1979) were used to evaluate the (a) SS comparison and (b) visual supports comparison. Although ATDs are typically used to compare two interventions, they can also be used to rapidly alternate between variations of a single intervention to compare components of an intervention and to demonstrate intervention effectiveness relative to a business as usual or BL condition if a reversible behavior (e.g., engagement) is being measured (Wolery, Gast, & Ledford, 2018). Components of the SS intervention were isolated to evaluate the differential effects of attention, reading a book, reading a book with behavioral expectations, and reading a book with behavioral expectations and answering comprehension questions in the context of the first ATD. A visual support intervention was compared with a BL condition in the context of the second ATD. Specific condition descriptions can be found below.
Conditions were randomized in blocks of four in the SS comparison using a blocked random sequence and random number generator. Conditions were semirandomly ordered in the visual supports comparison; sessions were limited to only occur twice in a row in the visual supports comparison. The presence of a functional relation was evaluated using visual analysis by assessing differentiation in level and overlapping data between conditions (Barton, Lloyd, Spriggs, & Gast, 2018). Stability of data paths within conditions and immediacy of behavior change were also evaluated to guide decisions for moving from the SS comparison ATD to the visual supports comparison ATD. Initial BL (before the SS comparison) and best alone conditions (after the SS and visual supports conditions) were conducted to detect possible multitreatment interference.
Procedures
Sessions occurred daily for all participants except Raven; sessions occurred 4 days a week due to Raven’s counseling schedule. Presessions were approximately 2 min in duration; sessions were 5 min (Xander, Raven, Marc) or 10 min (Jason, Michael) in duration. Session length was chosen based on teacher report or observations of activity duration. Verbal assent was obtained each day prior to experimental activities.
Teacher interview and assessment
A structured interview was completed with each classroom teacher to identify (a) activities or routines in which the target student displayed low engagement, (b) expected behaviors or tasks to be completed during the activities or routines, and (c) demographic information about the student (e.g., age, gender, race, disability and language status, family socioeconomic status [SES], attendance history). The SSIS-RS (Gresham & Elliott, 2008) was completed by the classroom teacher for each participating student in collaboration with the researcher.
Preference assessments
A multiple stimulus without replacement (MSWO; DeLeon & Iwata, 1996) preference assessment was conducted before initiation of the study to assess student preference for intervention format. In consultation with the classroom teacher, the researcher created one SS and one VAS in three formats: (a) book, (b) traditional, and (c) digital (see Open Science Framework for materials descriptions: https://osf.io/6jhgy/). A paired stimulus preference assessment (Fisher, Piazza, Bowman, Hagopian, Owens, & Slevin, 1992) was conducted if the classroom teacher reported students did not have access to an iPad (Marc and Michael). A SS preference assessment was conducted for each participant. Visual supports preference assessments were conducted for all participants who used a VAS (all participants except Jason). Preference assessments were not conducted for SVs (Jason) because multiple formats of the intervention were not present in the extant literature.
General
Two research activities occurred daily across all conditions and comparisons: a presession and a session. After student assent was obtained, the presession researcher took the student to the designated location in the hallway adjacent to the classroom door, turned on the camera and placed it adjacent to the student and researcher, and told the student the condition for the day (e.g., today is a story and questions day). The presession researcher told the student they would stay in the hallway until the timer said 02:00. After presession procedures were implemented for the designated condition, the presession researcher said, “It’s time for (targeted activity)” and walked the student to the classroom door.
The presession researcher stayed in the hallway during Xander, Raven, and Jason’s sessions. The session camera was mounted on a classroom bookshelf during the sessions. The presession researcher entered the classroom during Marc and Michael’s visual support sessions to move the camera when needed to capture each student during his targeted activity. The presession researcher did not interact with either student and ignored bids for attention. At the end of each session, the camera was stopped by the researcher who remained in the hallway. Specific procedures across conditions are detailed below.
Baseline
Presession and session procedures were the same across both comparisons for the BL condition. During BL presessions, the presession researcher told the student it was a “talk day.” The presession researcher responded to student statements but did not ask the student any questions or make any statements related to expected behaviors during the upcoming activity or routine. If the student did not talk, the presession researcher made general statements (e.g., I’m going on a trip this weekend) approximately once every 30 s.
During BL sessions, no additional verbal, gestural, or physical prompts were delivered to the student. The session researcher told the teacher to conduct the activity as she typically would with the exception of Marc. The researcher asked the teacher not to follow Marc into the hallway if he failed to enter the classroom. The presession researcher stayed in the hallway to monitor Marc’s safety but did not interact with him. The session researcher was present while videoing all BL sessions with the exception of Raven. Raven was observed leaving her instructional area to walk to novel adults during the classroom observations conducted prior to study initiation. As a result, the session researcher started the camera recording and left the classroom until the instructional activity ended.
Social Stories intervention
The SS comparison included one no-intervention condition (book alone [BA]) and two SS intervention conditions (SS and SS plus reading comprehension questions [SS-reading comprehension]). Each condition was selected based on previously conducted studies evaluating the effectiveness of SS for students without ASD (Schneider & Goldstein, 2009). The format of the SS across conditions was the same and determined by the results of the preference assessment. The presession behaviors varied by comparison condition, whereas the session procedures were identical to BL (i.e., for this comparison, session procedures were identical throughout; only presession procedures differed). During SS presessions, the presession researcher told the student, “we’re going to read a story about (book content, for example, spiders; BA condition)” or “we’re going to read a story about what to do during (target activity/center; SS and SS-reading comprehension conditions).” Next the presession researcher read the book corresponding with the appropriate condition.
A different leveled reader (guided reading level A–B; kindergarten level text) was read during each BA session. The texts were selected using a random number generator from a set of 10 possible books. Texts at this level typically have one sentence per page with a single image and are no longer than 15 pages, roughly estimating the time it would take to read a SS. The presession researcher did not ask questions during or after reading the text.
The target-activity SS was read during the SS and SS-reading comprehension conditions. Three comprehension questions were asked after reading the SS during the SS-reading comprehension condition only. The three questions were modeled after previous research implementing SS with students without ASD (Schneider & Goldstein, 2009). Constant time delay (CTD; Wolery, Ault, & Doyle, 1992) procedures were implemented to assist students in responding to questions in the SS-reading comprehension condition; 0 s delays were used in the first two SS-reading comprehension sessions and a 3-s terminal delay was used for remaining sessions. The controlling prompt was a verbal model of the correct answer to the question. If the student answered the question correctly, the presession researcher said “yes, that’s right” and repeated the answer (e.g., raise hand). If the student incorrectly answered the question, the presession researcher said “no” and provided the correct response. If the student asked questions during the book relevant to the book content, the presession researcher provided an answer to the question. Researchers responded to irrelevant questions with, “We’re reading a book. We can talk at the end,” and then answered the question after reading. At the end of reading (BA, SS) or comprehension questions (SS-reading comprehension), the presession researcher checked the time. If less than 2 min had elapsed, the researcher told the student they could look at the book or story until the timer said 02:00. When the timer on the camera reached 02:00, the presession researcher stopped the camera, said “it’s time for (target activity),” left the SS or book in the hallway, and walked the student to the classroom door. Researchers did not inform classroom teachers of which condition was implemented during presessions in the hallway. Teachers were blind to the daily presession type given that session procedures were identical.
Visual supports intervention
The visual supports comparison included one visual support intervention condition compared with a BL condition. During visual support presessions, the presession researcher told the student, “today is an iPad© schedule (Xander, Raven), work box (Jason), book (Michael), or schedule (Marc, Michael) day.” Then the presession researcher modeled how to manipulate the visual support (e.g., swipe the icon to the right when a task is complete; iPad schedule) with verbal directions when required (e.g., choice means you can pick to ask to get water or stay and earn a ticket). The presession researcher did not ask the student any questions about using the visual support but answered any questions posed by the student. At the end of the model, the presession researcher checked the time on the camera. If less than 2 min had elapsed, the researcher told the student they could look at the visual support until the timer said 02:00. When the timer on the camera reached 02:00, the presession researcher stopped the camera, gave the student the visual support, told the student “it’s time for (target activity),” and walked the student to the classroom door. CTD procedures were used by the session researcher to prompt steps required to complete visual support task analyses (available on Open Science Framework [https://osf.io/6jhgy/]). The tasks displayed on each student’s visual support are listed in Table 2.
Interobserver Agreement (IOA) and Procedural Fidelity (PF)
IOA data were collected for 91%–100% of sessions for engagement behaviors using point-by-point agreement and kappa. Secondary observer data were graphed alongside primary data to detect possible observer bias. IOA data were collected for at least 33% of sessions across participants for preference assessments. Reliability data were collected via video using ProcoderDV (Tapp, 2003). Reliability data were collected by observers blind to study purpose, hypothesis, and condition for designs evaluating SS interventions; observers were not blind to condition for designs evaluating visual support interventions. IOA was calculated using point-by-point agreement using the following formula: agreements/(agreements + disagreements) × 100. Disagreements were resolved via consensus. Average agreement across participants and conditions met contemporary standards (M = 86%–100% agreement across participants). Kappa was calculated for engagement (the only dependent variable for which this calculation was possible); across participants, the mean value for kappa ranged from 0.73 to 0.86 (categorized as “substantial” [0.6–0.8] to “almost perfect” [0.81–1.0] agreement; Cohen, 1960).
PF data were collected using direct systematic observational recording via video for at least 33% of sessions across participants, conditions, implementers, and behaviors. Overall fidelity for each session was calculated by dividing the number of correctly implemented behaviors by the total number of expected behaviors and multiplying the quotient by 100. Average fidelity across conditions met contemporary standards (M = 95%–100% fidelity across participants). Additional detailed IOA and PF data, including graphed second-observer data are available on Open Science Framework (https://osf.io/6jhgy/).
Social Validity
The social validity of the SS and visual supports interventions were evaluated via student preference for intervention format and information gathered from teachers assessing the acceptability and feasibility of a series of interventions commonly used in classrooms.
Stakeholder attitudes
The acceptability and feasibility of both interventions were evaluated by asking teachers of participants to assess SS and visual supports in relation to other commonly used classroom interventions: weighted blankets, SS, VASs, headphones, weighted vests, alternative seating, token boards, first/then boards, timers, work boxes, response cards, point sheets, and choice boards. Teachers were provided with a brief description of each intervention and asked to (a) sort interventions into three categories: effective, ineffective, and effectiveness unclear; and (b) identify the three interventions they would be most likely to use and least likely to use. Teachers were also asked to place the interventions from easiest to hardest to implement on a number-line continuum. Teachers completed these tasks after the initial BL data collection (n = 1) or during the best alone data collection (n = 2), but before results data were reviewed with the researcher.
Results
Preference Assessments
Participant preference of intervention format across both SS and visual supports interventions are displayed in Table 2. Xander, Jason, and Raven preferred the digital format SS; Xander and Raven also preferred the digital format VAS. Michael preferred the traditional SS format. Marc did not indicate a clear preference for SS format, thus the teacher selected the format she would be more likely to use: a single-page story. Marc and Michael both preferred the traditional VAS format. Preference for SV format was not assessed for Jason or Michael.
Social Stories Comparison
Engagement data during the SS comparison are presented across participants in Figures 2–4. Xander, Jason, and Raven displayed variable levels of engagement during the initial BL condition between 0%–40%, 20%–80%, and 0%–60% engagement, respectively. Marc engaged with the morning routine during one session of the initial BL condition; during all other sessions, he displayed 0% engagement. Michael had a stable level of engagement around 30%–45% during the initial BL condition. When SS were implemented with all participants, engagement levels remained variable across all conditions with no differentiation between the SS intervention conditions (SS, SS-reading comprehension) and the no-intervention conditions (BL, BA). Although Marc’s level of engagement increased during the third SS and SS-reading comprehension sessions, his level of engagement returned to zero in the subsequent sessions and remained undifferentiated from the no-intervention BA condition. Functional relations between SS interventions and students’ engagement in activities were not present for any participant.
Xander engagement (top panel) and Jason engagement (bottom panel).
Raven engagement (top panel) and Marc engagement (bottom panel).
Michael engagement data.
Visual Supports
Engagement data for these comparisons are also displayed in Figures 2–4. Xander and Jason displayed an immediate increase in the level of engagement when the visual support intervention was implemented to approximately 55% (Xander; VAS) and 90% (Jason; SWB) of intervals for the first two sessions followed by minimal overlap between the no-intervention (BL) and intervention conditions during the third sessions. Engagement continued an increasing trend (Xander; VAS) or increased and remained stable (Jason; SWB) across the remainder of the condition to approximately 90% of intervals. Engagement was variable during the remaining no-intervention (BL) sessions between 0%–60% (Xander) and 10%–60% of intervals (Jason). Engagement remained stable around 80% of intervals during the best alone condition for both Xander and Jason. A functional relation between the visual support intervention and engagement was present due to differentiation between conditions with higher and more stable levels of engagement occurring in the visual support condition compared with the no-intervention condition.
Raven’s level of engagement immediately increased in level and remained stable around 75% of intervals during the VAS condition and continued during the best alone condition. Engagement was stable between approximately 35%–45% of intervals during the no-intervention (BL) condition. There was no overlap between VAS and no-intervention BL conditions. A functional relation between VAS intervention and engagement behaviors was present due to clear differentiation between conditions with higher levels of engagement occurring in the VAS condition compared with the no-intervention condition.
Marc’s level of engagement was at levels similar to the SS comparison (around 25% of intervals engaged) when the VAS intervention was implemented. Engagement increased in level after the second VAS session but remained variable between 45% and 90% of intervals. Engagement was low and stable during the no-intervention BL condition (0%–20% of intervals). There was no overlap between VAS and no-intervention BL conditions. A functional relation between VAS intervention and engagement was present due to clear differentiation between conditions with higher levels of engagement occurring in the VAS condition compared with the no-intervention condition. When the VAS + preferred images modification was implemented, engagement immediately increased in level to approximately 90% of intervals, similar to the highest levels of the VAS intervention. Engagement remained high and stable in the VAS + preferred images condition. Levels of engagement in the no-intervention (BL) condition were stable at low levels (less than 40% of intervals), then decreased to near 0% engagement. There was no overlap between VAS + preferred images and no-intervention BL conditions. A functional relation between VAS + preferred images intervention and engagement behaviors was present due to clear differentiation between conditions with higher levels of engagement occurring in the VAS + preferred images condition compared with the no-intervention condition.
Michael displayed stable levels of engagement around approximately 55% of intervals when the VAS intervention was implemented. BL levels of engagement were consistently lower than VAS levels around approximately 25%–40% of intervals. There was no overlap and clear differentiation in levels of engagement between VAS and no-intervention BL conditions. A functional relation between the VAS intervention and engagement was present due to clear differentiation between conditions with higher levels of engagement occurring in the VAS condition compared with the no-intervention condition. When the SV modification was implemented, Michael’s level of engagement immediately increased compared with the VAS condition to around 80% of intervals. Engagement decreased during the fourth SV session, but immediately increased during the remaining sessions to around 70% of intervals. Engagement during the no-intervention BL condition was stable around approximately 20%–40% of intervals. There was no overlap between SV and BL conditions. A functional relation between the SV intervention and engagement was present due to clear differentiation between conditions with higher levels of engagement in the SV condition compared with the no-intervention BL condition.
Social Validity
Three general education teachers in preschool, kindergarten, and first-grade classrooms completed the surveys. Teachers had been in their current positions for 0.5–11 years and had 7–11 years of experience teaching in schools. Responses to the survey are available on Open Science Framework (https://osf.io/6jhgy/). All three respondents rated SS and VASs as effective. Two teachers rated SWB as effective; one indicated she was unsure if SWB were effective. Teachers also indicated SS and visual supports were feasible for use in their classrooms. Two teachers further noted they would be very likely to implement SS relative to other commonly used antecedent interventions. No teachers rated either visual support (VAS or SWB) as an intervention they were most likely to implement compared with the other antecedent interventions. In addition, SWB were rated by all three teachers in the third and fourth quartiles of feasibility, indicating teachers may not find implementation of SV feasible in the context of general education classrooms. Finally, one teacher indicated VASs would be the last antecedent intervention she would implement in her classroom. Overall, teachers rated SS as more feasible and more likely to be implemented than visual supports.
Discussion
This study contributed to the existing literature by providing the first evaluation of the effectiveness of SS for students at risk for EBD, the first evaluation of the effectiveness of VAS and SVs for students at risk for EBD, and the first incorporations of participant preference in intervention format for both SS and visual supports. Conclusions are discussed below in relation to the existing literature and the impact of instructional settings and intervention format on intervention effectiveness.
Results of this study suggest SS are not an effective intervention for improving engagement behaviors during ongoing instructional activities and routines for students at risk for EBD in general education preschool and elementary settings. These conclusions mirror those of some prior reviews evaluating their effectiveness for students with (Leaf et al., 2015) and without (Zimmerman & Ledford, 2017) ASD. Conversely, results from this study differ from previous research in which SS with comprehension questions improved on-task behaviors of students with language impairments during lunchtime routines (Schneider & Goldstein, 2009). SS may be more effective when a child would benefit from brief reminders and/or antecedent attention before a difficult activity, whereas visual supports may be needed when students require more permanent reminders, or materials that increase opportunities or allow alternative methods for responding. Future studies should evaluate the effectiveness of SS with comprehension questions in improving routines for students with and at risk for disability given their lack of effectiveness in improving engagement in this study compared with positive effects found in previous research (Schneider & Goldstein, 2009).
In contrast, visual supports may be an effective intervention for improving engagement in students at risk for EBD. These outcomes are consistent with the only other evaluation of visual supports for students at risk for disability (Zimmerman et al., 2017). Variability in students’ engagement decreased for all participants except Marc, suggesting consistency of students’ performance may increase when visual supports are present.
Although the overall level of Marc’s engagement increased in the presence of the VAS, his performance was more variable relative to no-intervention conditions. Differences in Marc’s performance relative to the other participants may be explained by his limited previous experiences at school and the need for modifications to visual supports for some students. Unlike the school-aged participants, intervention did not begin for Marc until the second semester of the school year. His limited 5-month learning history did not include the morning routine as a component of the school day (his teacher reported he had never been observed to engage in the routine prior to the study). Thus, presenting a visual support to a young student with little experience engaging in the targeted activity may have been insufficient to result in meaningful behavior change. The variability observed in Marc’s performance could have been a result of the intermittent success of the intervention in changing the contingencies associated with school arrival. Students with limited learning histories, particularly those for whom school has not been associated with successful completion of activities or routines, may require modifications to existing visual support protocols. Incorporating student preference as an antecedent modification stabilized Marc’s engagement at high enough levels for him to successfully and independently complete the morning routine.
Differences in students’ performance were also observed relative to the type of activity targeted by the visual support. The presence of VASs resulted in increases in engagement for Xander and Raven during whole group reading and math instruction, respectively. However, VASs did not result in sufficient improvements in Michael’s levels of engagement during the whole group morning meeting activity. The variability in the effectiveness of the VAS may be due to the age of participants (Michael was in preschool, whereas Xander and Raven were first-grade students) or due to the format of each whole group activity. Whole group instruction in Xander and Raven’s class was relatively short and served to briefly introduce content, whereas whole group instruction in Michael’s class was longer in duration and involved multiple instructional activities. Simply providing visual structure to the order of the activities may have been insufficient for meaningful behavior change in the latter context. Thus, VASs may be better suited for a single whole group activity of a short duration rather than multiple activities that culminate in a longer duration instructional session. Researchers should continue to investigate the conditions under which the use of VASs during large group activities is likely to result in improved outcomes.
SVs, however, resulted in improvements in students’ engagement across reading centers and whole group instruction. SVs in the form of SWB functioned as an organizational tool for Jason to collect center materials, whereas they functioned as visual supports to increase Michael’s methods to respond to instruction during morning meeting. Increasing students’ opportunities to respond has been demonstrated to be an effective way to improve academic outcomes (Common et al., 2019), thus SVs may be a low-effort way to improve engagement for students at risk for disability as well as students with ASD (Zimmerman et al., 2019). Future studies investigating the utility and flexibility of SV formats (e.g., boxes versus books) are needed to guide selection of SV interventions across multiple instructional arrangements.
The variable effectiveness of the three interventions examined (SS, VASs, and SVs) might be explained by the type of support provided by each approach. It may be that VASs result in superior levels of engagement because they represent the temporal order of activities as an in situ support as opposed to the transient support offered by SS. When students require temporal organization and additional methods to organize their responding, SVs may be a more appropriate intervention choice. Further applications of VASs and SVs across activities and routines in general education settings may further clarify intervention selection decisions, particularly given teachers’ reports that both visual supports interventions were less likely to be implemented than SS, despite rating them all as acceptable interventions. Additional research is needed to determine in what contexts SS interventions are effective, and dissemination of these constraints to teachers is critical, given this study adds to the literature suggesting they are feasible and likely to be used.
Limitations and Future Directions
Results of the current study should be considered in light of some limitations. First, SS and visual supports were not directly compared in the current study; thus, comparative conclusions cannot be experimentally drawn. Moreover, all SS comparisons occurred prior to visual supports comparisons. It is possible that the history of intervention had some impact on later behavior. However, the consistency of continuing BL conditions throughout both comparisons makes this possibility less likely. Students in the current study were also not appropriately engaging in instructional activities and routines, and missing instructional content for at minimum 15 days prior to the visual supports condition. As a result, researchers decided not to continue to evaluate conditions in which meaningful behavior change had not occurred for experimental purposes. Because neither intervention had been evaluated and shown to be effective in the context of general education classrooms for this population of students, comparing the interventions to each other would have been premature. Comparing each intervention to a no-intervention BL condition, rather than each other, allowed us to examine the potential effectiveness of each intervention. Future studies should be conducted to directly compare the interventions or evaluate the effectiveness of the interventions as a combination package similarly to Schneider and Goldstein (2010); this comparison was outside the scope of the current study.
Moreover, although information regarding the function of problem behaviors was gathered across all participants, this information was not used in developing the interventions. The opportunity to leave the classroom was provided at the end of Xander and Raven’s schedules, so one could argue escape (one function of each of their behaviors) was contingently provided in the context of the intervention. All participants displayed socially maintained problematic behaviors that impacted their engagement, yet visual supports were differentially effective without modifications (Marc and Michael). Future studies should investigate how the function of students’ problematic behaviors could be used to design visual supports to tailor the interventions to create function-based interventions to support engagement. For example, a VAS intervention might include the conditions under which escape from the activity can be expected (i.e., complete three steps then leave) and an SS intervention might include information about the conditions under which social attention will be provided (i.e., raise your hand and wait up to three times during a large group activity). Additional analyses are needed to determine the extent to which these considerations are necessary for effectiveness.
In addition, all sessions were implemented by research staff rather than teachers or paraprofessionals in the classrooms. The feasibility of implementing VASs or SVs with students at risk for disability has not yet been determined, nor has the feasibility of appropriate tool selection. Furthermore, stakeholder survey results indicated some teachers rated VASs as difficult to implement relative to other antecedent interventions. When VASs were used, teachers preferred traditional paper formats, whereas students preferred digital formats. Additional surveys of teacher preferences in intervention format as well as intervention selection may provide information about the likelihood of teacher use of interventions in nonpreferred formats. Perhaps teachers select interventions for reasons other than effectiveness, thus teacher preference for SS over visual supports should also be further investigated to determine the components of each intervention that may be preferred by teachers. As further information about teacher preferences for antecedent interventions is gained, guidance about teacher selection of effective interventions like visual supports may begin to be created.
Despite these limitations, the current study provided five failed demonstrations of the effectiveness of SS and five successful demonstrations of the effectiveness of visual supports for improving engagement behaviors compared with no-intervention BL conditions during ongoing instructional activities and routines for students at risk for EBD in public, general education preschool and elementary classrooms. Although teachers identified both SS and visual supports as feasible and acceptable low-effort interventions, teachers might consider using visual support interventions rather than SS interventions for improving engagement behaviors for students at risk for EBD in general education settings.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported in this article was supported in part by Grant H325D120091 from the Office of Special Education Programs, U.S. Department of Education. The opinions expressed are those of the authors and do not represent the views of the Office or the U.S. Department of Education.
