Individualized Instructional Delivery Options: Adapting Technology-Based Interventions for Students With Attention Difficulties

Individualized supports

As implemented in the intervention phase of the current study, KinderTEK has customizable settings to fit varied educational contexts and students. Some settings help teachers fit KinderTEK to their class-level lesson plans and content goals (e.g., session duration and learning mode; i.e., app directed, teacher directed, or student directed). Others offer much finer-grained manipulation of the student experience and can be adjusted on an as-needed, student-by-student basis (e.g., audio and visual cues to act, optional midsession reward time, higher sticker award rate, on screen timer, and visualization of session structure). In practice, teachers can track student progress and adjust students’ settings via the app and accompanying web dashboard; however, all instructional cueing and self-regulation supports were activated by research personnel, as this served as the independent variable in the current study.

KinderTEK research base

Data from pilot studies conducted during KinderTEK development indicate that KinderTEK produces positive student outcomes. As part of an Institute of Education Sciences development grant, 11 kindergarten classes participated in a randomized control trial study (Shanley et al., 2013, 2015; Cary et al., 2014b). After attrition, the final analytical sample included 45 KinderTEK students and 49 control students. Two substantively important (Hedge’s g > .25) effect sizes were found in favor of the KinderTEK group: Magnitude Comparison = .36 and Number Line Estimation = .36 (Cary et al., 2014c). For students who were able to complete a significant portion of the lessons (i.e., 75% of the activities introduced), six meaningful effect sizes were detected: Magnitude Comparison = .43, Number Line Estimation = .29, Missing Number = .26, Oral Counting = .29, Number Sense Brief = .26 (Jordan, Glutting, & Ramineni, 2008), and Test of Early Mathematics Ability (TEMA-3) = .27 (Ginsburg & Baroody, 2003).

Setting

The study took place in two public schools located in a suburban district in the Pacific Northwest. School A was an elementary school serving approximately 400 students in Grades K through 5, 100% of whom received free or reduced-price lunch services and 48% of whom were from minority backgrounds. School B was an elementary school serving approximately 530 students in Grades K through 5, 66% of whom were eligible for free or reduced-price lunch services and 38% of whom were from minority backgrounds. All observations took place in a quiet space apart from the general education classroom (e.g., commons, hall, separate room). During observation sessions, students worked one-on-one with observers to use an iPad-based mathematics program. The research team is not aware of any other behavioral interventions offered to participating students.

Participants

The five participants in the present study were selected from a larger group of children participating in a study funded by the Office of Special Education and Rehabilitative Services aimed at exploring the feasibility and usability of a kindergarten iPad-based mathematics program. Prior to identifying potential research participants, human subjects approval was obtained from Research Compliance Services in accordance with institutional guidelines. In all, five kindergarten students were nominated for participation by their classroom teacher based on teacher-reported difficulties attending to instruction as compared to their classroom peers. Participants 1 and 4 (identified here with the pseudonyms Nicole and Samuel, respectively) attended School A and were in the same class taught by a White, female teacher. Participants 2, 3, and 5 (identified with the pseudonyms Edwin, Ruby, and Michael, respectively) attended School B. Ruby and Michael had the same White, female teacher, and Edwin’s teacher was female, and of more than one race. All participating teachers were fully licensed and had between 5 and 10 years of teaching experience.

All participants completed a prestudy screening battery in midfall that consisted of a brief curriculum-based measurement battery (Assessing Student Proficiency in Early Number Sense [ASPENS]) and an engaging, popular measure of self-regulation for preschool-aged children (head–toes–knees–shoulders [HTKS]). These measures are described in more detail in the Materials section below, and participant scores are summarized to provide additional context about each participant’s initial mathematics and self-regulation skills.

Baseline

During baseline, participants used the basic KinderTEK iPad mathematics intervention daily in 15-min structured sessions without the instructional cueing and self-regulation supports activated. Under staff supervision, students entered their personal password, worked on at least two different mathematics instructional activities as assigned by the program, and had reward time at the end of the session.

Intervention phase

During intervention phases, participants used KinderTEK in 15-min structured sessions with instructional cueing and self-regulation supports activated and no other instructional modifications. Instructional cueing and self-regulation supports consisted of visual and audio cues to act, a progress bar, and a countdown timer that displayed remaining session time. The visual indicator consisted of a red “stop” hand and a green “thumbs up” sign indicating to wait and respond, respectively. Modeled from common choral response procedures (Adams & Carnine, 2003), a “ding” sound serves as the auditory cue to indicate when it is the student’s turn to respond. To ensure students were aware of the instructional cueing and self-regulation supports, staff demonstrated the audio and visual cues and showed students the progress bar and countdown timer before students began the first intervention phase. Figure 1 shows these cues as experienced by KinderTEK users.

OPEN IN VIEWER

Figure 1.

KinderTEK screenshots of inactive (top left) and active instructional cueing and self-regulation supports including countdown timer, progress path, visual indicators, and auditory cues (not pictured).

Maintenance

One maintenance data point was also collected for all but one participant. Each maintenance observation occurred 4–7 days after the end of the final intervention phase and was conducted in an identical manner to all previous observations. Students were able to access the KinderTEK program with settings activated as implemented by their classroom teacher between the end of the intervention phase and the maintenance observation.

Social validity

A student survey was administered to all participants to assess the extent to which students were satisfied with and enjoyed participating in the KinderTEK intervention. Questions were crafted to query student agreement with statements about KinderTEK and included statements like “How much did you like using KinderTEK?” “Did KinderTEK teach you math?” and “Did KinderTEK make math fun?” These surveys were group administered with teachers reading each statement and students responding by circling a picture (i.e., smiling/frowning face or thumbs up/down) representing their agreement with each statement.

Participating teachers were also surveyed to assess the extent to which they were satisfied with the KinderTEK intervention. Survey questions were used to gauge teacher impressions of KinderTEK, teacher perceptions of the student experience, and the extent to which KinderTEK was feasible for use in their classrooms. Satisfaction surveys were distributed after the conclusion of the study and all responses were anonymous.

Observer training

Prior to beginning data collection, four university staff familiar with the KinderTEK mathematics program and with conducting student assessments were trained to facilitate and observe students’ use of the KinderTEK program without (i.e., baseline) and with (i.e., intervention) instructional cueing and self-regulation supports and to record behaviors related to response accuracy. In an in-person training day, staff were trained how to describe the study to students and to identify and code the target behavior. Observers were given guidelines about (a) the extent to which they should and should not interact with students during the study, (b) how to activate KinderTEK’s instructional cueing and self-regulation supports (i.e., the intervention), and (c) a script for introducing these the instructional cueing and self-regulation supports to students.

Interobserver agreement

During training, observers practiced coding while the trainer played the KinderTEK program with the display mirrored on a large screen and the audio turned up so that all could hear. All observers completed a checkout coding exercise to meet a 90% interobserver agreement criterion on the target behaviors. Interobserver agreement was not calculated in the field because coding participant behavior required that observers sit close to the participant and share an audio source, and having multiple adults sitting with one student proved to be untenable. However, observers were required to conduct monthly reliability checks using training materials and simulated sessions to insure ongoing reliability across observers. Observers also participated in follow-up trainings, if the reliability check data did not meet the 90% interobserver agreement threshold.

Fidelity of implementation

Once activated in KinderTEK, instructional cueing and self-regulation supports were “on” until deactivated; thus, no formal fidelity measures of the implementation of the instructional cueing and self-regulation features were needed. To assess fidelity of implementation of the iPad sessions, observers were instructed to take notes of anything unusual during the session that might affect the child’s behavior. Because data collection was conducted on iPads and uploaded for review each day, these notes provided valuable contextual information to help inform study decision-making. For example, an observer noted that a participating student’s parent was present for a study session and the parent frequently interacted with the student during the session, so the research team was able to account for this anomaly in the data. Similarly, because the outcome of interest was an average count across intervals, observer notes were used to determine whether the session length was sufficient for inclusion in the study. Ultimately, analyzed data included only those observations for which students were present for at least 80% of the session (i.e., 12 min of KinderTEK use) and sessions for which there were no additional observers (i.e., parents, school psychologists) present. Across all participants, 85 of the 94 (i.e., 90%) attempted sessions were successfully conducted.

Response measurement

Direct observation data were collected by trained observers using an iPad-based data collection system, and all direct observations of student behavior took place as students used KinderTEK in a quiet space in the school (e.g., commons, hallway, unused room). Coding required an understanding of the instructional context (e.g., what student was seeing on the iPad screen, what the student was hearing, and whether it was the student’s turn to respond), so observers sat next to the student within view of the screen and used a headphone splitter so both the student and the observer could wear headphones and hear the program audio. Direct observation of student behavior was conducted 3–5 days per week per student across 15-min sessions. All responses were scored using partial interval recording across 20-s intervals. As described in the Observer Training section, observers were required to meet a 90% interobserver agreement criterion on the target behavior before entering the field

Analysis

Data were analyzed to assess the extent to which the intervention was functionally related to change in response accuracy. Data related to accurate responding were analyzed using traditional single-case design procedures that included visual inspection to examine (a) level (i.e., average behavior), trend (i.e., increasing or decreasing data points), and variability (i.e., spread of data points) within phases; (b) immediacy of effects (i.e., change in level), overlap (i.e., proportion of data points that overlap) between phases; and (c) consistency of data patterns within and across phases (Barton, Lloyd, Spriggs, & Gast, 2018; Gast & Spriggs, 2010; Kennedy, 2005; Kratochwill et al., 2010). Participant behavior was considered responsive to intervention if observable and sustained reductions in problem behavior and increases in desired behaviors were observed during the intervention phases (Fairbanks, Sugai, Guardino, & Lathrop, 2007). Accurate responding was the dependent variable upon which demonstration of functional control was predicted. Graphs (see Figure 2) depict the percentage of observation intervals with at least one instance of accurate responding.

OPEN IN VIEWER

Figure 2.

Percentage of intervals with accurate responding during baseline, intervention, withdrawal, and maintenance phases for study participants. B = baseline; I = intervention; W = withdrawal; M = maintenance.

Nicole

During baseline, intervals scored with accurate responding averaged 46.6% (range = 37.0–53.0%). As is shown in Figure 2, implementation of the intervention resulted in an immediate increase in accurate responding, with the behavior occurring in an average of 81.3% of intervals (range = 77.0–87.0%). As illustrated in Figure 2, the data document a slight increasing trend during this intervention phase, with low variability.

A withdrawal of the intervention was implemented to establish functional control over accurate responding and, as seen in Figure 2, resulted in a slight but immediate decrease in accurate responding. During this phase, intervals scored with accurate responding averaged 67.7% (range = 56.0–73.0%) and displayed a decreasing trend. Upon return to intervention, an immediacy of effect and improvement of response accuracy was again observed, with the behavior occurring in an average of 79.3% of intervals (range = 73.0–85.0%). Data collected during a maintenance phase documented that accurate responding continued, with accurate responding occurring in 87.0% of intervals. Nicole’s data demonstrated evidence of a very effective treatment with both effect size metrics (i.e., PND and PEM) indicating 100% nonoverlap.

Edwin

During baseline, intervals scored with accurate responding averaged 51.0% (range = 40.0–63.0%). As is shown in Figure 2, implementation of the intervention resulted in an increasing trend in accurate responding. Edwin also demonstrated an increase in level of accurate responding in the intervention phase with accurate responding occurring in an average of 64.3% of intervals (range = 60.0–70.0%).

Upon return to the baseline condition, Edwin demonstrated an immediate decrease in level of accurate responding. Intervals scored with accurate responding averaged 35.7% with a range of 27.0–43.0%. Upon return to intervention, an increase in accurate responding was again observed, with an immediacy of effect displaying a rise in level from the previous phase. During this phase, accurate responding was consistent with the similar intervention phase and occurred in an average of 66.7% of intervals (range = 60.0–70.0%). Maintenance data document that Edwin’s accurate responding continued to occur at a similar rate to intervention phases, with accurate responding occurring in 83.0% of intervals. PND effect size calculations for Edwin’s data indicated a potentially questionable treatment effect with PND = 50%; however, the PEM statistic accounted for the desirable increasing trend in the intervention phases and reflected a very effective treatment measure with PEM = 100%.

Ruby

In general, Ruby demonstrated less consistent performance within each phase and there were more overlapping data points across phases. However, her average performance within phases was quite consistent. During baseline, Ruby demonstrated accurate responding during an average of 49.2% of intervals (range = 33.0–60.0%). As shown in Figure 2, these data document a decreasing trend with moderate to high rates of inaccurate responding and some variability across sessions. In the first intervention phase, the introduction of the added instructional cueing and self-regulation supports resulted in an immediate increase in accurate responding from 40.0% to 73.0%, with accurate responding occurring in an average of 71.5% of intervals across sessions (range = 60.0–80.0%).

With the withdrawal of intervention in Session 10, Ruby demonstrated an immediate decrease in accurate responding and during this phase, intervals scored with accurate responding averaged 51.0% with a range of 40.0–63.0%. Figure 2 indicates that these data show a slight decreasing trend. An immediate increase in intervals with accurate responding occurred upon return to intervention. During this phase, Ruby continued to demonstrate more instances of accurate responding, with the behavior occurring in an average of 79.2% of intervals across sessions (range = 70.0–87.0%). As seen in Figure 2, rates of accurate responding remained high during the maintenance probe. Additionally, effect sizes measures indicated evidence of treatment effectiveness for Ruby with PND = 88% and PEM = 100%.

Michael

Like Ruby, Michael had overlapping data points across phases, but he demonstrated consistent trends of performance in similar phases. In the baseline phase, Michael exhibited accurate responding in an average of 58.2% of intervals (range = 43.0–80.0%) and these data showed a decreasing trend. With the addition of the instructional cueing and self-regulation supports, the percentage of intervals with accurate responding rose from 47.0% during Session 4 to 70.0% in Session 5. Accurate responding during the first intervention condition averaged 78.0% across intervals (range = 70.0–83.0%) and followed an increasing trend.

After the withdrawal of the instructional cueing and self-regulation supports, Michael exhibited accurate responding during an average of 66.6% of intervals (range = 57.0–80.0%), demonstrating a decrease in level compared to the intervention phase, with a decreasing trend. Last, with intervention reimplementation, accurate responding again increased in level (average = 79.7%, range = 73.0–93.0%) and showed an increasing trend. A follow-up maintenance probe revealed similarly high levels of accurate responding. Like Edwin, Michael’s effect size metrics were highly impacted by the trends across phases. The high initial point in the baseline phase of Michael’s data resulted in PND = 38%, which suggests an ineffective treatment, but when evaluating overlap based on median baseline behavior PEM = 100%, suggesting that the intervention was also generally effective for Michael.

Samuel

During baseline, Samuel exhibited accurate responding in an average of 55.0% of intervals (range = 50.0–63.0%). With the introduction of the intervention, Samuel’s rate of accurate responding increased from 53.0% to 87.0% of intervals. As displayed in Figure 2, rates of accurate responding remained high and stable during the first intervention phase, with an average of accurate responding occurring in 83.3% of intervals (range = 80.0–87.0%).

With the removal of the added instructional cueing and self-regulation supports following the intervention phase, rates of accurate responding decreased and were highly variable across sessions (average = 59.6%, range = 34.0–80.0%). Last, with the reintroduction of the intervention, levels of accurate responding increased to an average of 72.8% of intervals (range = 60.0–87.0%). However, data during this phase were moderately variable, and this was presumably due to the discipline issues occurring outside of the study as detailed above. Due to the eventual suspension of this participant, no maintenance data are available. Nonetheless, effect size measures supported treatment effectiveness for Samuel with PND = 88% and PEM = 100%.