Engagement as an Alternative to Noncompliance Measurement: Promoting Validity,Accuracy,and Student Outcomes

Measurement of Challenging Behavior

Choosing a system for measuring challenging behavior depends on two key considerations. The first is what the behavior looks like. Some challenging behaviors may occur for an extended time, in which case duration or time-based measurement would be best suited. Other challenging behavior may occur across multiple instances with a clear start and stop, in which case count or frequency would be best suited. For example, when measuring out-of-seat behavior, one may choose to measure the length (duration) of each out of seat occurrence. When measuring aggression, total count (frequency) may be deemed most appropriate. Second, measurement system choice can depend on classroom staffing and resource availability (LeBlanc et al., 2016). If a staff member can collect data without interruptions, an exact measurement system can be used, such as frequency or duration. However, if staff are limited and need to multitask, an estimation method of measurement (Ledford et al., 2015) may be more appropriate.

Challenging behavior can be continuously measured, with every instance counted or the time of each instance recorded. It can also be estimated using time sampling procedures that only require recording challenging behavior at time-based intervals. It is of utmost importance that the measure is valid and accurate (Cooper et al., 2019). That is, it must accurately capture the challenging behavior (the total time [duration] or count [frequency]) and be reasonable for an educator to use during ongoing instruction. For example, if a data collection system results in challenging behavior being undercounted, an ineffective intervention may continue to be implemented long after adjustments are required. However, a data collection system that overcounts challenging behavior may result in the implementation of an unnecessary intervention (Ledford et al., 2015).

Measurement of Noncompliance

While validity and accuracy are crucial, it can be especially difficult to select a sensitive measurement for certain types of challenging behavior. Sensitive measurement systems may be critically important for challenging behavior that has broad definitions or examples and/or unclear start and stop moments that can make counting difficult. Noncompliance is a challenging behavior typically defined as an individual doing anything other than what has been directed by an authority figure, such as an educator, within a specific time frame (Kalb & Loeber, 2003). Because of the broad definition of noncompliance it may be clear that noncompliant behavior has started, but it may be unclear when noncompliant behavior has stopped. It may also be difficult to know if noncompliance is occurring because each instance depends upon the existing skills of the student and educator behavior, explained further below. Thus, a noncompliance measurement system must be sensitive to both the student’s existing skills and the behavior of various educators, who may present demands differently. Given the difficulty with defining and capturing noncompliance across students and classrooms, its inaccurate reporting is highly likely.

Asa created a datasheet to track Audrey’s noncompliant behavior. At the top of the datasheet, he wrote a definition of noncompliance that reads, “Following a directive from an adult, count one instance of noncompliance if Audrey does not begin engaging in the behavior directed by the educator within 5 s.” Within 10 min, Asa noticed the teacher, Mr. Cade, gave Audrey three directives, including “get to work,” “get your materials,” and “get started.” Each instance included three to four repetitions of the direction. Asa marked 10 instances, but he felt that this did not accurately capture Audrey’s noncompliant behavior when Mr. Cade only gave a total of three different directives.

Limitations of Noncompliance Measurement

We believe that the measurement of noncompliant behavior is limited by compliance, educator, and ethical issues.

Engagement as an Alternative Measure

Selecting a valid and accurate measurement system that addresses the issues of the measurement of noncompliant behavior is essential because compliance within certain instructional settings is required for accessing learning opportunities. No literature exists to support the validity and accuracy of the measurement of noncompliance (Losinski et al., 2017; Malone & Zimmerman, 2023). Thus, rather than adjusting the definition of compliance, practitioners may consider measuring engagement, as participating in adult-led instruction is the goal of reducing noncompliant behavior. Estimating engagement through time sampling methods, specifically momentary time sampling, provides educators with a valid and accurate alternative measurement (Cook & Snyder, 2020; J. D. Lane & Ledford, 2014).

Engagement is most accurately measured as duration or time. However, such measurement may be difficult because a student’s engagement can change in brief moments of time (Cook & Snyder, 2020). For example, a student may be completing academic work, then wander around the room, then return to academic work. Although it ensures accuracy, collecting total duration data can be a cumbersome, error-prone system that is not always feasible among educators for whom multitasking is a necessity. Thus, estimating engagement provides a reasonable alternative to directly measuring the number of seconds or minutes that a student may be engaged during ongoing instruction.

Momentary time sampling (MTS) is the only interval-based measurement system designed to estimate duration-based behaviors that marks the occurrence of the behavior at the end of a designated interval (Ledford et al., 2015). It allows for determination of an estimated percentage of intervals of student engagement (i.e., engaged 70% of the time) in an accurate and reliable way that is comparable with total duration measurement (Cook & Snyder, 2020). Momentary time sampling also allows educators to capture student performance quickly alongside their instruction every 60 s, rather than continuously watching student noncompliant behavior until the student engages in instruction or becomes unengaged in instruction. Thus, MTS is a feasible and accurate system available to educators during ongoing instruction in a classroom.

Individualizing Engagement Definitions

Although classwide engagement behaviors may provide a general description of how students participate in instruction, accessing instruction varies across students depending on student strengths, support needs, and environmental contexts (e.g., activity expectations). For example, during a discussion activity with peers, a student with extensive support needs may be considered engaged for all the following behaviors: rocking back and forth, handling relevant materials, and responding to a peer with a voice output device (Cosbey & Johnston, 2006). A student with language and behavioral support needs (e.g., students with or at risk of EBD) may be considered engaged when transitioning independently to a new location, visually attending to instruction, or completing activities with proper supports such as visual activity schedules (Zimmerman et al., 2017), within-activity choice (Cole & Levinson, 2002), or academic supports (e.g., hundreds chart; Zimmerman et al., 2020).

Individual differences in how students access instruction, and thereby demonstrate engagement, often necessitate an individualized definition of engagement. Individualized definitions should capture the student’s unique participation in the instructional context (i.e., if they are participating, they are engaged) to allow for diversity in ways to access and participate in learning. If a standard definition of engagement is used across students, educators risk applying narrow requirements that may not be inclusive of all students (i.e., students who may not be comfortable making eye contact or prefer to engage in repetitive behaviors while simultaneously participating.)

“Unengaged” can be defined as the absence of engagement or participation, which also varies across students. For some, unengaged may look like using materials unrelated to instruction (e.g., reading a library book instead of participating in math instruction), unresponsiveness when given supports and opportunities to respond, or engaging in other challenging behavior such as elopement or throwing academic materials (Zimmerman et al., 2020). When defining “unengaged,” it is important to note that engagement in the instructional environment is not always a necessary behavior. For example, a student may appear to be doing “nothing” during a long writing period when they are critically thinking about the writing task. As another example, a student may engage in an unrelated activity if they have completed their work.

The broad range of behaviors that can be encompassed in the definitions of engaged and unengaged are a key distinction between measuring engagement and noncompliance. Whereas noncompliance requires educator behavior to occur, engagement behaviors can occur independently of educator behavior and do not require the educator to assume the demands a student may be able to complete. Engagement behaviors can also be tailored to student strengths and support needs (e.g., using text to speech to complete writing tasks, completing chunks of a math worksheet rather than the entire worksheet). Furthermore, no conceivable ethical concerns exist regarding teaching or promoting individualized engagement within instructional contexts. In fact, data suggest that engagement is related to later improvements in student achievement (Lei et al., 2018; Lindström et al., 2021). Thus, if the overall goal of decreasing noncompliant behavior is to support a student’s access to learning across a variety of contexts, engagement measurement provides a more valid and accurate measure of these skills.

As shown in Figure 1, when students are engaged, they access learning opportunities through appropriate supports and high-quality instruction, which leads to achievement (Greenwood, 1996; Lindström et al., 2021). Whereas measuring noncompliance may provide invalid data (e.g., the number of adult demands delivered or student support needs as opposed to true instances of noncompliance), measuring engagement provides information about participation in instructional contexts.

OPEN IN VIEWER

Figure 1.

Conceptual Framework of Engagement and Achievement.

Engagement in classroom activities is more closely related to the goal of reducing noncompliant behavior, and therefore is more valid, because it may indicate to what extent a student is accessing educator instruction. If a student is frequently unengaged, subsequent interventions to increase engagement do not have the same ethical concerns as interventions to increase compliance with demands. Strategies to enhance engagement include instructional adjustments such as increasing opportunities to respond, using materials to encourage active participation, or teaching the student to self-monitor (CAST, 2018). Measuring students’ engagement levels using individualized engagement definitions can inform important environmental adjustments and unique interventions needed to promote positive outcomes.

A Guide to the Measurement of Individual Engagement

The data collector must make several logistical decisions to prepare for data collection to increase the likelihood of accurate estimates of student engagement. First, decisions for when and how often to collect data need to be made. Specifically, at least 3 to 5 samples of at least 10 to 15 min duration need to be collected to ensure consistent estimates of engagement (Briesch et al., 2010; Cook & Snyder, 2020). More data will likely produce a more stable estimate. Considerations of the total number of samples should be made based on time and resource considerations (e.g., who is available to collect data, when, and how often).

Second, the setting of data collection must be determined. It may be useful to collect data in settings where the student is reported to be unengaged and settings where they are engaged in order to understand the range of the student’s current engagement. If possible, the same number of samples should be collected in both unengaged and engaged settings. Third, an interval size—the number of seconds between recording data—must be selected. Given teachers often collect data while simultaneously delivering instruction, interval sizes of 12 to 60 s are recommended for measuring student or class engagement using MTS procedures (Ledford et al., 2015; Zimmerman et al., 2022). Because smaller interval sizes are more accurate (Ledford et al., 2015), select the smallest interval that is most feasible for your setting and data collectors.

Data may be recorded electronically on a spreadsheet (see Figure 2, Supplemental File https://osf.io/hqk8f/) or using paper-and-pencil forms. A running stopwatch is required for data collection, which could be a smartphone stopwatch, a traditional stopwatch, or a free interval timer app, such as Beepwatch2^© that vibrates at the end of every 60 s interval. A step-by-step guide for collecting engagement data can be found in Table 1.

OPEN IN VIEWER

Figure 2.

Electronic Engagement Data Sheet.

OPEN IN VIEWER

Table 1.

Task Analysis: Collecting Momentary Time Sampling Data for Engagement.

Step	Description
Preparation
1	Determine observation settings for data collection. Choose at least 1 setting where the student is reported to be engaged, and 1 setting where the student is reported to be unengaged.
2	Determine the number of observation samples and length of samples that are feasible to conduct for the data collector (aiming for 3–5 samples of between 10 and 30 min in each setting, longer sample time = more accurate).
3	Select interval size (between 10 and 60 s; shorter interval = more accurate).
4	Gather materials: data collection sheet (electronic or paper/pencil), stopwatch, or interval app).
5	Individualize operational definitions through collaboration with educators and/or service providers. Create examples and/or non-examples for engaged and unengaged based on the student, their support needs, and the context.
Data Collection
6	Start stopwatch or interval timer app.
7	Look away from student; as soon as the interval ends, look at the student.
8	Record either engaged (1) or unengaged (0) according to operational definitions.
9	Continue Steps 7 and 8 until observation ends.
10	Calculate a percentage of intervals for engagement (total number of engaged/total number of intervals × 100).

Prior to data collection, the definitions of both engaged and unengaged should be individualized and exhaustive, with examples and non-examples of each definition specific to the context of instruction and to the student. For example, for a student who engages in off-task behavior by being out of their seat during math class, engagement may include manipulating instructional materials, retrieving necessary materials, or staying in location (e.g., around desk, table, or carpet). A non-example may be out-of-seat behavior to talk with peers about non-related topics.

Once all materials have been gathered, the data collector can begin collecting data by starting the stopwatch or interval timer app. During the interval, the data collector should look anywhere except for the target student (e.g., provide instructional feedback to other students, look at instructional materials, or scan student responses on whiteboards). When the interval ends (i.e., stopwatch reads 60 s or app beeps or vibrates), the data collector should look at the target student and immediately mark a “1” for engaged or a “0” for unengaged, using the predetermined definitions and examples. Data collection continues in this manner until the observation sample time is complete. The percentage of student engagement can be calculated by summing the number of intervals where the student was engaged (all intervals marked 1), dividing by the total number of intervals (1’s and 0’s), and multiplying by 100. For example, if a student was engaged for 14 intervals of a 20-min observation (20 intervals), the student’s estimated engagement was 70%.

Engagement data are valuable because they may provide an estimate of the percentage of instructional time accessed by a student. However, the amount of engagement (i.e., the percentage of intervals) required to impact academic achievement is unknown. Student engagement can vary across the day and activities, depending on the quality of instruction, student preferences, and student support needs. Therefore, it may be best for the expected percentage of engagement to remain dynamic and situational, depending on the individual student, instructional context, and other factors (e.g., comparison with peers, student scoring on formative assessments indicating skill acquisition, response effort requirements for the activity).

How Much Engagement Is Enough?

Standardized amounts of engagement have not been established in the literature. However, measuring and understanding overall engagement in the classroom may support educators in understanding if a target student is “engaged enough” to access instruction and social relationships in the classroom. Following the procedures detailed above, educators can identify an overall classwide expectation of engagement (e.g., 90% of students are engaged during literacy centers). These data can be used to support decisions about how much engagement is “enough” for student success.

Once a target student is identified and classwide levels of engagement and achievement are acceptable, determining how much engagement is enough may be best achieved by comparing the target student’s engagement performance to that of an average peer in the same classroom. This method, often called “normative peer comparisons,” has been found to be an adequate comparison method (Johnson et al., 2016; Riley-Tillman et al., 2008) for decision-making regarding cascading supports for students (Simonsen et al., 2019). To identify a normative peer, the data collector may choose to randomly select one to two peers and collect engagement data simultaneously alongside the target student. The data collector may identify (or if unfamiliar with the classroom, ask for support in identifying) a student who, in their estimation, displays average levels of engagement and simultaneously collect data for that student (Zimmerman et al., 2020).

Comparing a target student’s data to a normative peer sample can support decisions regarding next steps for the target student. For example, if a target student’s percentage of engagement is quite low compared to that of a normative peer (e.g., 30% compared with 80%), this may support decisions to increase the intensity of their supports to Tier 2 (e.g., check-in/check-out; McDaniel et al., 2015) or Tier 3 (e.g., an individualized behavior support plan; Liaupsin & Cooper, 2017). If the target student’s engagement levels nearly match or exceed that of the peer comparison, then an educator may have confidence that the supports in place during the observation were sufficient for the student to access instruction.

Recommendations for Increasing Engagement

If data indicate relatively low levels of engagement for a target student, the amount of high-quality instructional opportunities (Simpson et al., 2020) and the availability of meaningful student supports should be examined. One way to enhance the quality of instruction is to provide a positive learning environment through clear expectations and utilizing frequent behavior-specific praise (Ennis et al., 2018; Haydon et al., 2020), which can lead to trusting, positive student–teacher relationships. Incorporating student preference and choice throughout instruction, such as utilizing preferred materials or providing varying forms of opportunities to respond (Rila et al., 2019; Simpson et al., 2020), are other ways to facilitate high-quality instruction and can lead to increased academic engagement (Gage et al., 2018).

Universal Design for Learning (UDL; Dunn & Pérez, 2012) includes recommendations for both high-quality instructional practices and academic supports for diverse learners, all of which encourage engagement in instructional contexts. Some ways to increase engagement include providing a variety of ways for students to interact with instructional materials to support learning preferences, teaching goal-setting and progress-monitoring skills and providing access to alternative technology for those who may benefit. These strategies provide avenues for increasing engagement and support student learning preferences and promote autonomy within learning. In addition, the use of visuals, scaffolding, or chunking may be considered during instruction. Behavioral supports may also be necessary for some students, such as visuals of behavioral expectations, reinforcement systems that align with a student’s values and preferences, and self-monitoring interventions via paper/pencil (see Figure 3) or using an app such as iConnect^© (Beckman et al., 2019; Wills & Mason, 2014). Further preventive behavior supports for students may be found through the ibestt Project^© (University of Washington, 2017).

OPEN IN VIEWER

Figure 3.

Self-Monitoring for Engagement Data Sheet.

After talking to Mr. Cade and Audrey’s speech-language pathologist about Audrey’s support needs, Asa decides to collect MTS data on her engagement across four classroom activities (literacy centers, math centers, circle time, and whole group instruction.) As included in Table 1, first, Asa individualizes Audrey’s definitions for engaged and unengaged with Mr. Cade and the speech-language pathologist, then he edits the datasheet: At the end of each interval, Mark 1 for engaged if Audrey is participating in ongoing classroom activities. Examples include talking to a partner during group work, sitting in a designated area during instruction, retrieving necessary materials, playing with relevant materials during large group, or completing a classroom routine. Mark 0 for unengaged if Audrey is clearly not participating in ongoing classroom activities. Examples include walking around the classroom during instruction (not retrieving materials), hitting/pushing peers or throwing materials, or other off-task behavior that is clearly not related to instruction, such as playing with a superhero figure on the floor during handwriting tasks.

Conclusion

The measurement of noncompliance presents many concerns. The subjective nature of the definition can generate invalid and inaccurate data collection, which can result in inappropriate interventions and a variety of other ethical concerns for students who engage in frequent noncompliance. Teaching compliance in place of noncompliance also comes with ethical concerns. Thus, we advocate that the proposed solution is not to adjust the definition of noncompliant behavior, but instead measure engagement. Whereas noncompliance data provide information about the instructional context, engagement data may provide an estimation of the amount of instruction accessed by the student. Engagement data can be utilized by educators to ensure that high-quality instructional opportunities are readily available and student supports are in place as needed to promote engagement, thus promoting positive long-term outcomes for all students.

Supplemental Material