Strengthening Progress Monitoring Procedures for Individual Student Behavior Support

Step 1: Develop Fidelity Measure(s)

A BSP like Allison’s outlines specific research- or evidence-based strategies for staff to implement that will positively impact student behavior. However, the effectiveness of the BSP is impacted by the fidelity with which the strategies are implemented. Without explicit attention to fidelity, BSP implementation is inconsistent at best (Noell, 2008; Sanetti et al., 2014). Improving fidelity is not only related to better student outcomes (Noell, Gresham, & Gansle, 2002; Pinkelman & Horner, 2017) but is also necessary for evaluation of the validity of intervention results (Shadish, Cook, & Campbell, 2002). Thus, measuring fidelity becomes as important as measuring student outcomes (Detrich, 2014; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; McIntosh & Av-Gay, 2007). Teams must determine how they will measure the extent to which the BSP strategies are implemented as planned within daily routines.

To develop a fidelity measure, teams must first identify, prioritize, and carefully define critical plan components, based on knowledge of the student and the context. Although all BSP components might be beneficial, teams should emphasize monitoring parts of the plan that are most likely to impact student behavior. There are multiple aspects of fidelity that can be measured. They include content (what), quantity (how much), quality (how well), and conveyance (how, when, by whom; Weiss, Bloom, & Brock, 2014). Teams can start with the simplest indicator, one that is easy to measure and likely to indicate overall plan fidelity. An example of such an indicator is a checklist indicating that three critical prompting and consequence-based procedures were implemented on a given treatment day. Over time, teams can adapt or enhance the fidelity measure to improve its acceptability to implementers, sensitivity to change, and utility for support team review (e.g., rating scale vs. a checklist).

The perspective of fidelity is also important to consider. Direct observation of plan implementation by an impartial observer is considered the gold standard. However, it may only be feasible for implementers to self-monitor their fidelity using a checklist or rating scale (Gresham, 2014). A team leader may instead choose to meet with implementers and come to consensus on the level of fidelity. Permanent products may also be used as indicators of fidelity (e.g., forms completed, visual prompts available, lesson notes). Although there is mixed evidence on the reliability and accuracy of self-reported fidelity, school climate and resources (e.g., time, staff willingness to be observed) may impact a team’s ability to incorporate direct observation measures or even interview and discussion–based measures. Self-report may be a more feasible and socially acceptable measure, at least to start with (Mouzakitis, Codding, & Tryon, 2015; Plavnick, Ferreri, & Maupin, 2010). If using self-report, the measure should include a quantitative measurement, such as yes/no checklists or a rating scale (e.g., 0–4) for each strategy/component of the strategy and space for anecdotal information (e.g., success, barriers, suggestions, specific adaptations, and rationale). The team will identify a goal (i.e., a minimum threshold) and timeline for meeting that goal, so that implementers and the team are able to make adjustments to best meet the goal or to change the goal if it becomes clear that barriers need to be addressed.

When completing this first step, it will be helpful for teams to consider the following questions:

Allison’s team discussed each BSP component, and members agreed that the fourth-grade teacher, classroom aide, and special education teacher would each use a checklist that outlines the critical prevention, teaching, and reinforcement components of Allison’s BSP to assess fidelity (6 components total). Some team members wanted to assess each of the 10 components separately to identify which components were most critical, but the fourth-grade teacher and classroom aide were concerned that this was not realistic given their other responsibilities during math period. The checklist lists the six critical components of Allison’s BSP in one column, and in another column, staff mark a “yes” if that BSP component was implemented, a “no” if it was not implemented, and “N/A” if there was no opportunity to implement the component (e.g., response to problem behavior if problem behavior did not occur). From this, a percentage was calculated by dividing the number of components scored as “yes” by the total number of components possible and multiplying by 100. On the checklist, there was also a place for staff to write any additional information that might be relevant to Allison’s behavior for that day. The team agreed that each of them would use the checklist to self-monitor their implementation daily and that another team member would use the checklist to directly observe other team members’ implementation weekly.

Allison’s team identified two initial fidelity goals. The first fidelity goal was that each individual would implement the plan with 80% fidelity per day for two consecutive weeks. The second fidelity goal was for staff to implement her BSP at 90% or higher per day for two consecutive weeks. The team also agreed that members would note when (a) fidelity percentage dropped below the goal, (b) there was a change in routine, (c) information arose that would be helpful in making decisions about the current or future phase of the BSP, or (d) if they would like additional modeling or coaching on implementation.

Step 2: Develop Outcome Measure(s)

Once team members have determined a fidelity measure that will allow for monitoring of BSP implementation, they can then explore how they will assess the extent to which the plan is impacting student behavior. Outcome measures indicate BSP effectiveness in (a) increasing prosocial behaviors and alternative/replacement behaviors and (b) decreasing behaviors that are undesired or negatively impact a student’s social and academic success (Dunlap, Iovannone, Wilson, Kincaid, & Strain, 2010; O’Neill et al., 2015). As with fidelity measures, it is recommended that teams start by developing one or more simple outcome measures that are easy to collect and sensitive to small changes in behavior. Maintaining staff commitment to consistent and accurate data collection is more feasible when the goals and timelines have been aligned to the school context (e.g., staff capacity and schedule, existing data collection procedures). Teams, with input from implementers, can modify the system to meet identified gaps when data are reviewed and discussed.

Outcome measures should provide data that indicate whether the student is making steady progress toward the short-term and long-term goals at a pace that is likely to match the proposed timeline. Continuous progress monitoring informs the team if students receiving individualized supports are responding to interventions or if adaptations are needed (“National Center on Intensive Intervention,” n.d.; O’Neill et al., 2015). It is likely that two or more outcome measures will be needed to capture nuances of student behavior change: a measure to capture student problem behavior and a measure to capture student replacement behavior(s). To develop measures for these behaviors, teams start by operationally defining each behavior, ensuring that each description is specific, observable, and measurable. Once operational definitions have been drafted, team members can decide how staff can measure each of the behaviors. The same aspects (e.g., content, quantity, quality, conveyance) and perspectives (e.g., self-report, direct observation, permanent products) that were considered in developing fidelity measures can apply to outcome measures.

Teams can use baseline student performance (often referred to as the present level of performance, or PLOP), to set short-term and long-term goals, identify timelines, and determine the student’s response to the intervention. Long-term goals are often desired behaviors or an ambitious estimate of what can be accomplished over 1 year. Short-term objectives are smaller snapshots that can be used to trigger a phase change or increments of change toward the long-term behavior. Teams should consider the available tools and procedures that are already in place in the classroom or school, and the preferences of those who will be involved in collecting and reviewing the data. It is worth restating that individualized and intensive supports require individualized and intensive monitoring (Lee et al., 2011). Teams should think critically about the efficiency and effectiveness of measures over time, so that short-term and long-term goals can be monitored without unnecessary burden on staff. Introducing new measures and methods of data collection requires training, coaching, and behavior change across implementers and should be acknowledged and supported by the team and school administrators. Implementers who are not team members (i.e., are not regularly involved in decisions) should be included early in the process to identify the goals of the plan and to ensure that proposed goals and plan components are clear and realistic.

When identifying outcome measures, teams should consider answering the following questions:

Allison’s team members decide to first focus on short-term goals and measure replacement behaviors that they want to see increase: using a break routine and asking for help when needed. For Allison, a four-step “break” routine was operationally defined as (a) using a verbal or gestural request to pause from the current activity, (b) setting an egg timer to a preset time, (c) engaging in preapproved quiet break activities at her desk or at a special chair at the back of the classroom, and (d) quietly returning to the task or activity within 30 sec of the timer signal. Asking for help was similarly broken down into observable and measurable steps.

While measuring the number of breaks and the number of steps completed accurately for each break would provide greater detail, this was not deemed to be feasible by the primary implementers. Instead, the team developed a behavior point card to include a rating scale for each routine during math period. Procedures similar to Check-In, Check-Out (Swoszowski, 2014) would be used to facilitate a check-in at the beginning and check-out at the end of math with a rating scale to measure whether Allison followed both the break and help steps as defined. The scale options were defined as follows: (4) Amazing! Allison followed all the steps every time or she didn’t need any breaks or help; (3) Pretty good. Allison missed one or two steps; (2) Progressing. Allison missed three, four, or five steps (across opportunities); (1) Attempted. Allison tried out one or more steps but missed six or more; and (0) Keep trying. Allison did not use any of the steps but we will try again tomorrow.

The fourth-grade teacher and classroom aide agreed that this was feasible to incorporate into their routines if the special education teacher would teach and practice the routines and point card procedures with Allison. Next, Allison’s team members turned their focus to behaviors of concern they wanted to see decrease: time off-task during math. This was operationally defined as number of minutes out of the 40-min math period when Allison was engaging in any activity or behavior that was not offered or approved by a teacher as part of the lesson. The team determined that it would be helpful to know how many incidents occur and how much time was spent away from instruction. The team also wanted to include Allison and her family in discussions about her behavior, so at the bottom of the behavior point card, there was a space allotted (a) to tally up the number of incidents (including breaks) when Allison was off-task for more than 1 min, (b) for staff to indicate the total number of minutes that Allison was off-task, and (c) for Allison, her teacher (or aide), and a family member to initial to indicate review. Space for comments would be allotted. Implementers were concerned about accurately monitoring the number of minutes off-task but agreed to try for at least 2 weeks. The PLOP for incidents and time off-task was estimated at 10 incidents and 90 min per week. The team developed two long-term goals for the next 3 months: Allison will receive a daily rating of three for each desired behavior for two consecutive weeks and will take fewer than four breaks (with less than 30 min of off-task behavior) per week. Team members agreed that all goals were specific, measurable, and achievable but agreed to revisit classroom procedures.

Step 3: Develop an Action Plan

Developing a comprehensive BSP that includes a decision system with fidelity and outcome measures is a major accomplishment for the support team and should be celebrated. However, the investment of time and energy in building a plan is based on the premise that there will be beneficial returns (i.e., the plan will be implemented with fidelity and there will be improvement in student behavior). The success of a plan is contingent on the fidelity with which it is implemented into daily routines and procedures (Detrich, 2014; Weiss et al., 2014). Even when the team adequately represents (or includes) individuals who will be responsible for implementing the BSP, underappreciation of the implementation costs (e.g., training needed, time and effort to implement, materials, etc.) may undermine a plan’s success before it has had a chance to prove its effectiveness (O’Neill et al., 2015). An action plan is needed to document all tasks and resources associated with implementation of the BSP. Some of the changes will be immediately obvious (e.g., train staff on new reinforcement procedure, create lesson plan), whereas others may be more subtle (e.g., designate confidential location for blank and completed data forms, secure time for staff to enter data into spreadsheet). Time spent considering how the BSP (including data collection) will be put into action will help the team anticipate and address potential barriers and stumbling blocks (Sanetti, Collier-Meek, Long, Byron, & Kratochwill, 2015).

The action plan should bring together components of the BSP including each fidelity and outcome measure identified for progress monitoring. Action planning discussions focus on who will be involved in implementation, what they need to do, and how they can efficiently embed data collection into daily routines (Pinkelman & Horner, 2017). In isolation, each component may appear to be reasonable, but putting all the parts together and comparing with current resources and routines may reveal a different perspective and new barriers to implementation. In developing measures, the focus is on what will be collected, by whom, and how. Goals and timelines are set that may specify how often data are collected (e.g., daily or weekly). During action planning, the team may realize that the time and resources needed to collect data in addition to implementing the critical BSP components would be unrealistic. In that circumstance they may collectively agree to alter goals and data collection schedules that prioritize one or two key measures and maintain a less intensive goal and schedule for remaining measures.

Throughout action planning, a theme for teams to embrace is parsimony. Choose the smallest and simplest change(s) that will provide the largest effect(s) toward goals. Parsimonious action plans require discipline when a group of individuals are sharing ideas and experience from different perspectives. Appointing meeting roles such as “facilitator” and “time keeper” can make this process more clear and allow team members to hold each other accountable to contribute respectfully and maintain focus on the task (Newton, Horner, Algozzine, Todd, & Algozzine, 2012).

When teams build action plans, members should consider answering the following questions:

Allison’s team built an action plan, starting with a review of current routines and procedures during Allison’s math period and identification of where the BSP components made the most sense. For example, Allison was pulled out for reading just before math. The special education teacher scheduled 10 min after the reading lesson to provide an overview of the upcoming math lesson and complete one or two example problems together. The action plan identified that a weekly meeting was needed to review the lesson plans and identify the information that would be most helpful to Allison. When the schedule changed or Allison was absent, Allison’s classroom teacher would email changes. The team proposed that fidelity ratings and outcome data (behavior card points and breaks) be collected Monday through Friday to show changes from day to day. However, once all the implementation tasks were outlined in the action plan, members were concerned that there were too many changes happening at the same time. After some discussion about what would be realistic and sensitive to change, the team decided that individuals would collect fidelity data on Monday, Wednesday, and Friday each week and submit those data to the special education teacher via email at the end of the week. Point card and break data would be collected and submitted daily, sent home for parent signature, and returned the following day for data entry. The special education teacher organized data using an online data system, the Individual School-Wide Information System (I-SWIS; May et al., 2017). Discipline referrals would be collected in accordance with school procedures. The final action plan provided detailed information, so that each team member understood their initial tasks and timelines to implement the BSP as well as the ongoing tasks to maintain implementation and data collection.

Step 4: Establish Decision-Making Routines

A decision system outlines a process to help teams efficiently identify and respond to changes in fidelity or student outcome data. A decision system allows the team to efficiently use data to make important decisions that promote continuous quality improvement (Blase, Van Dyke, & Fixsen, 2013). Teams should establish regular schedules and meeting agendas for where and how often to meet, and what data will be reviewed. Data should be organized and formatted before the meeting, so that its messages or stories are meaningful to team members and are easily translated into actionable tasks (Hojnoski et al., 2009; Van Norman & Christ, 2016). This may require expertise in graphing technologies or the adoption of a system that automatically generates useful reports.

A consistent monitoring schedule is the foundation for individualized supports. Commitment to active, team-based progress monitoring requires that stakeholders meet consistently. Additional meetings may also be necessary when specific needs or situations are identified that require more immediate action. A written meeting agenda can assist the team in using meeting time wisely and help ensure that all needed discussion points are addressed in the meeting. Starting each meeting with data review will facilitate the identification of urgent discussion items and problems to be addressed. When reviewing the data, teams may ask the following questions:

Specific criterion should be clearly defined that will indicate overall level and trend. These indicators form decision rules or guidelines keep discussions focused, so that actions are consistent over time. To build, the decision rules start with possible actions such as (a) continue current plan, (b) modify the plan, (c) add or intensify the plan, and (d) fade supports to build self-management skills. Each action is then broken into one or more data criterion (e.g., fidelity level of A and outcome trend of B over X of Y days). Decision rules are not meant to be blindly followed. When combined with critical thinking and professional judgment, attending to decision rules can be helpful to teams in reducing effort and time needed to make final decisions.

When establishing decision-making routines, teams can answer the following questions:

Allison’s team already had a meeting schedule that worked for the core members, with reports communicated weekly. A more specific agenda was adopted for consistency and the team adopts general guidelines on formatting the data (e.g., graphs that summarize data weekly as opposed to monthly). The team agreed upon decision rules to guide data review and action planning. For example, if fidelity data were below the short-term goal for two consecutive weeks, teachers and the classroom aide met to review the procedures and provide feedback to the team regarding barriers to implementation. When the short-term goal was met, the fidelity goal would increase. All components of the plan were intensified if Allison’s problem behavior continued and the number of requested breaks remained lower than the number of teacher prompted breaks for over 1 week. The team members all committed to their individual tasks and agreed that the changes would strengthen the BSP.