Direct Behavior Rating

Abstract

French

Direct behavior rating (DBR) offers users a flexible, feasible method for the collection of behavioral data. Previous research has supported the validity of using DBR to rate three target behaviors: academic engagement, disruptive behavior, and compliance. However, the effect of the base rate of behavior on rater accuracy has not been established. In addition, although teachers report frequent use of DBR-like tools, little is known about what type of training is necessary to ensure accurate usage. The present study sought to determine (a) whether training with practice and feedback improves rater accuracy when rating academic engagement, disruptive behavior, and compliance and (b) whether rater accuracy varies at low, medium, and high levels of these behaviors. Participants (N = 67) were randomly assigned to one of three training conditions: brief familiarization, brief training with practice and feedback, or extensive training with practice and feedback. Following training, participants watched videos of elementary students and rated behavior with DBRs. Results indicated that practice with feedback improved rater accuracy for disruptive behavior but not for the behaviors of academic engagement and compliance. In addition, rater accuracy was found to vary significantly at different levels of behavior. Academic engagement was rated most accurately when high rates of the behavior were displayed. Disruptive behavior and compliance were rated most accurately at low or high base rates, while raters had more difficulty rating medium levels of behaviors. Limitations, implications, and future directions for research are discussed.

Keywords

behavior assessment training direct behavior ratings DBRs accuracy

With the increase in academic and behavioral school-wide problem-solving models, training in the use of empirically validated formative assessments is critical for educational professionals. Although measures appropriate for academic progress monitoring (e.g., curriculum-based measurement) have garnered a large literature base, research into social-behavioral progress-monitoring tools has been sparse (Riley-Tillman, Kalberer, & Chafouleas, 2005). This is problematic because school-wide behavioral problem-solving models require that school professionals be able to use reliable behavioral monitoring tools with fidelity. To be effective, such tools must be feasible both in terms of usage and training. Direct behavior rating (DBR) has been identified as an emerging behavioral assessment method for student progress monitoring (Chafouleas, Kilgus, & Hernandez, 2009; Riley-Tillman, Methe, & Weegar, 2009). However, few studies have yet evaluated the type and intensity of training that is needed to ensure that individuals can accurately rate student behavior. In addition, little is known about how rates of student behavior may affect rater accuracy. Research is needed to further develop defensible behavior assessment tools such as the DBR, as well as to determine the training conditions necessary for users to use these tools with fidelity.

DBR refers to a class of behavioral assessment tools in which student behavior is rated following a specified observation period and then resulting information is shared across parties to make decisions. The procedures and instrumentation of DBR combine the strengths of both rating scales and systematic direct observation to produce a unique tool for rating a directly observed target behavior immediately following an observation period (Chafouleas, Riley-Tillman, & Christ, 2009). DBR has a long history of use within applied settings and has included tools such as the daily behavior report cards, check in/check out notes, home-school notes, good behavior notes, behavior report cards, and so on (Chafouleas, Riley-Tillman, & Sassu, 2006). Such tools may best be characterized as formative assessments as they allow for repeated measurement of behavior over time and can be used by educators to evaluate and tailor interventions as needed. DBR may be used both as a means of progress monitoring and as a component of an intervention (Chafouleas, Riley-Tillman, & McDougal, 2002). For example, a teacher concerned about a student’s high rate of disruptive behavior may use DBR to collect data on a student’s behavior. Every day at a particular time (e.g., immediately following the student’s guided reading block), the teacher would complete a DBR to briefly rate the student’s disruptive behavior. Once collected, the data would be analyzed, graphed, and shared across professionals to determine next steps. The DBR could also serve as a component of the student’s intervention plan. For instance, the student could earn incentives for receiving low ratings for disruptive behavior or the teacher could use the DBR to teach the student self-monitoring skills by having both student and teacher complete the DBR and then conferencing about the student’s behavior.

One particular version of DBR that has garnered empirical support is the DBR “Single Item Scale” (DBR-SIS). The DBR-SIS is a standardized rating tool designed to measure the three target behaviors of academic engagement, disruptive behavior, and compliance/respectfulness (LeBel, Chafouleas, & Sugai, 2006). While some behavioral rating tools contain multi-item scales, in which a rater would rate many specific behaviors within a common behavior class (e.g., talking back, arguing with teacher, refusing tasks, yelling out, etc.), the DBR-SIS instead requires the rater to rate single broad behaviors (e.g., disruptive behavior; Christ, Riley-Tillman, & Chafouleas, 2009). By focusing on broad classes of behavior, the DBR-SIS is able to target general outcome measures important for social-behavioral success at school. The three target behaviors for the DBR-SIS were selected with input from several sources, including a review of the literature regarding school success, definitions and data from the School Wide Information System (www.swis.org), the National Center on Positive Behavior Supports (www.pbis.org), and consultation with national experts. Research has provided support for the validity of this tool in tracking a child’s social behavior in a classroom or other educational environment (Chafouleas, Riley-Tillman, et al., 2009; Christ et al., 2009). In addition, a number of studies have indicated that DBR is a defensible formative assessment measure that can accurately rate student behavior (Briesch, Chafouleas, & Riley-Tillman, 2010; Chafouleas, McDougal, Riley-Tillman, Panahon, & Hilt, 2005; Riley-Tillman, Chafouleas, Christ, Briesch, & LeBel, 2009; Riley-Tillman, Chafouleas, Sassu, Chanese, & Glazer, 2008). Past research has supported that DBRs can be used to rate student behavior with accuracy following a wide variety of observation durations (e.g., 5-, 10-, 20-, and 40-min periods; Riley-Tillman, Christ, Chafouleas, Boice, & Briesch, 2010). Previous research has also focused on other instrumentation aspects of administration such as behavior definitions (Chafouleas, Jaffery, Riley-Tillman, Christ, & Sen, 2013; Christ, Riley-Tillman, Chafouleas, & Jaffery, 2011; Riley-Tillman, Chafouleas, et al., 2009). Recent research has also begun to investigate whether DBR-SIS may be an appropriate tool to screen students for school social risk (Chafouleas, Kilgus, et al., 2009; Chafouleas, Kilgus, et al., 2013), as well as its role in multi-tiered behavioral problem-solving models (Chafouleas, Riley-Tillman, et al., 2009).

Practitioners report frequent use and high acceptability of behavior assessment methods that fall in the DBR class. In a national survey of teachers ranging from the preschool to high school level, 64% reported that they had previously used some form of DBR within their classrooms (Chafouleas et al., 2006). In addition, teachers reported using DBR for a variety of purposes including monitoring or observing student behavior, changing student behavior, and communicating with others about behavior. School psychologists also view DBR as a highly feasible method for classroom data collection. A survey of members of the National Association of School Psychologists suggested that school psychologists view DBR as a highly acceptable formative assessment tool, yet report significantly less training in DBR compared with systematic direct observation (Riley-Tillman, Chafouleas, & Eckert, 2008). Because of the limited demands on time and rater, DBR may offer an easier and more feasible method for classroom behavioral data collection than many of the systematic direct observation systems such as State-Event Classroom Observation System (SECOS; Saudargas & Creed-Murrah, 1980) and Behavioral Observation of Students in Schools (BOSS; Shapiro, 2008). However, there are still many aspects of the DBR tool that raters must be familiar with in order to use the tool in a reliable manner, including (a) the defined target behaviors, (b) the frequency with which ratings should be made, and (c) instances when the rater should abstain from recording. To maximize the accuracy of ratings, training in the use of DBR should address these elements. Raters may also benefit from additional training components, including the opportunity to practice rating target behaviors and receive corrective feedback. Despite the prevalent use of DBR tools in educational settings, one cannot assume that the majority of users have engaged in such training. This may be problematic given that progress-monitoring data are increasingly being used for high-stakes decision making (e.g., special education eligibility, recommendation for Tier III, etc.). Therefore, research is needed to determine whether training improves rater accuracy, potentially resulting in better decision making regarding student behavior.

Several methods for training individuals in the use of a new procedure or tool have been developed. Such trainings can be broadly classified as either direct or indirect in nature. Direct training typically involves trainer and trainee interaction, and may include activities such as role-playing, modeling, and practice with feedback (Sterling-Turner, Watson, Wildmon, Watkins, & Little, 2001). In contrast, indirect training relies on the didactic presentation of material, often in written format; typically lacks a responsive element; and provides few, if any, means of corrective feedback. Thus trainees who participate in direct training may be at an advantage as they learn a new procedure under the supervision of an “expert” (i.e., the trainer) who is able to provide corrective feedback if needed. In this way, direct training may enhance the likelihood that a trainee will be able to perform a procedure with treatment integrity once under “real world” constraints. Although several studies have demonstrated that direct training is superior to indirect training for teacher and parent trainings (Flanagan, Adams, & Forehand, 1979; Watson & Kramer, 1995), only a few studies have begun to address which training is most effective for DBR.

In a preliminary investigation of the impact of training on DBR accuracy, classroom teachers were randomly assigned to either a training or no training condition. They then watched 2-min video clips of students displaying mild or severe behavioral problems and used DBR to rate behaviors. Although a moderate association was found between the teachers’ DBR ratings and systematic direct observation conducted by an external observer, training did not significantly improve the accuracy of ratings (Chafouleas et al., 2005). Researchers expanded upon this study by examining whether individuals with limited prior experience with behavioral assessment would rate behavior more accurately following a training that included a practice and feedback component or a training that involved only a brief familiarization procedure. In the training condition, participants viewed a short presentation on DBR-SIS, watched six 1-min video clips, and practiced rating student behavior after each clip. Following each rating, a researcher immediately provided corrective feedback to the participants regarding the accuracy of their ratings. In the brief familiarization condition, participants were presented with an informational presentation, viewed a video on behavioral assessment, and then received a brief overview of DBR-SIS. The impact of training on rater accuracy was examined through difference scores and differential accuracy scores that were calculated by comparing participants’ ratings of behavior to momentary time sampling data. Although participants in both conditions rated student behavior in a similar pattern, those in the training condition were more accurate and displayed less overall variability within ratings, providing support that training with practice and feedback may result in more accurate DBR ratings than brief informational training (Schlientz, Riley-Tillman, Briesch, Walcott, & Chafouleas, 2009).

Training and accuracy were further examined in a study in which secondary teachers were randomly assigned to one of the three conditions: direct training, indirect training, or no training. Trainings differed in the extent of instruction, amount of modeling provided to participants, and number of practice opportunities. Teachers assigned to the direct training received directions on how to complete a DBR to rate academic engagement and disruptive behavior. Teachers then watched three 2-min training video clips, practiced rating the target behaviors, and received corrective feedback and explanations for expert DBR ratings made on the same clips. Teachers in the indirect training condition received directions on how to complete the DBR, watched three 2-min training video clips, and received explanations for expert ratings; however, indirect training participants did not practice making ratings. Teachers in the no training condition received only general directions for completing the DBR prior to watching clips and making ratings. Following training, participants in all conditions watched a novel video clip and made ratings for both target behaviors. No differences in accuracy were found between the no training and indirect training groups, and surprisingly, participants in the no training and indirect training conditions rated academic engagement more accurately than those in the direct training condition. No differences were found among the three conditions with regard to disruptive behavior. However, a significant limitation existed for this study in that the experimental video clip viewed by participants featured a child displaying only extreme behaviors (i.e., 0% academically engaged, 87.5% disruptive) that may be more easy to rate. Given the preliminary nature of the study, the authors suggested the need for further research to establish optimal training conditions for DBR (LeBel, Kilgus, Briesch, & Chafouleas, 2010). In addition, studies that examine whether it is easier to rate behaviors at low, medium, or high levels are needed so that the future trainings may address this potential issue.

The current study expands upon prior DBR research in several ways. First, as the DBR-SIS now includes three standard behaviors, additional research is needed to examine rater accuracy for the three DBR-SIS target behaviors: academic engagement, disruptive behavior, and compliance/respectfulness. Second, the study offers the first examination as to how rater accuracy might vary depending on the base rate of the target behavior (i.e., low, medium, high). Third, in the current study, researchers created two direct training conditions, a brief and an extensive training, to evaluate whether additional practice with feedback sessions may result in even greater accuracy.

Method

Participants and Setting

Participants included 67 undergraduate students enrolled in introductory psychology courses at a large Southeastern university, with most participants reporting a class status of freshmen (67%) or sophomore (25%). Although 17 participants (25%) reported that they had experience working in a school setting, only one participant (2%) indicated previous experience using a DBR-like tool within that setting.

Materials

Experience questionnaires

All participants first completed a one-page questionnaire that presented a brief description of DBR and asked participants about their previous experience using such a tool.

Behavior definitions and sample ratings

All participants were provided with an instructional sheet that contained behavior definitions for three target behaviors (academically engaged, disruptive behavior, compliance), as well as examples and nonexamples of each (see Figure 1). For instance, disruptive behavior was defined as “a student action that interrupts regular school or classroom activity,” while compliance was defined as “initiating or completing a response to an adult request in a timely and socially acceptable manner.” The sheet also contained directions for how to complete the DBR. Participants were instructed to make a rating by placing a dot along a line divided into 10 intervals (e.g., 0-10 scale) to reflect the percentage of time a child exhibited the specified behavior during the observation. A sample rating was provided to illustrate this concept.

Figure 1.

Behavioral definitions for DBR-SIS and example ratings.

Direct behavior ratings

Participants were also provided with a packet of blank DBR forms to rate student behavior. Each DBR contained the name of the target student for whom the rating was to be made and three scales to rate the target student’s academic engagement, disruptive behavior, and compliance. Brief definitions of the target behaviors were provided. The scale was divided into 10 intervals and demarcated with numbers 0 to 10. Percentages were also provided on the scale (i.e., 0%, 50%, 100%). Directions instructed participants to place a dot on the line that best reflected the percentage of total time that the target student exhibited the specified behavior during the observation session.

Video clips

Researchers produced 10 min of video footage of simulated instruction in a second-grade classroom. The clips featured four target students who engaged in a number of specific behaviors of interest (e.g., inappropriate motor behavior, response to teacher question, compliance with teacher request, etc.). Two minutes of video footage were used for practice and feedback sessions. The remaining 8 min were used for the rating portion of the study.

After the creation of the video clips, two advanced doctoral students in school psychology coded the behaviors of the four target students in each 1-min video clip using the Multiple Option Observation System for Experimental Studies (MOOSES; Tapp, 2004), a computerized program that allows for real-time event coding of up to 14 behaviors using 1-s coding intervals. The three DBR-SIS behaviors of academic engagement, disruptive behavior, and compliance were selected as target behaviors. However, because the MOOSES system rates noncompliance rather than compliance, compliance was calculated by subtracting the percent of noncompliance from 100. Interrater agreement was calculated for 30% of the available footage to ensure that doctoral students were using the MOOSES system in a reliable manner. Agreement was calculated by dividing the number of agreements (i.e., 1-s intervals in which both observers coded the presence of the target behavior) by the total opportunities for agreement, and then multiplying by 100. Interrater reliability was found to equal 95.8%.

Expert DBR ratings were then generated for each 1-min video clip by converting the percentage of time a target behavior was displayed to a DBR scale. This was accomplished by dividing the percentage of time a behavior was displayed (as measured by the MOOSES scoring system) by 10 and rounding to the nearest whole number. For example, if MOOSES data indicated that a target student was disruptive for 47% of the clip, this was converted to a DBR score of 5. This process resulted in 120 expert DBR ratings (4 target students × 3 target behaviors × ten 1-min clips). Next, each 1-min video clip was categorized as displaying either a low, medium, or high base rate of target behavior. If a target behavior was displayed for 0% to 33% of the clip, it was categorized as a “low rate” clip. Behaviors exhibited for 34% to 66% of a clip resulted in the clip being labeled “medium rate”; behaviors exhibited for 67% to 100% of the clip yielded a classification of “high rate.” Using this procedure, every 1-min video clip was classified as displaying either low, medium, or high rates of academic engagement, disruptive behavior, and compliance for each target student.

Procedure

Participants, blind to the conditions of the study, self-selected to attend one of three available experimental sessions. Twenty participants attended the first session, which researchers had determined a priori would be the brief familiarization condition. The second session was the brief training with practice and feedback condition and was attended by 22 participants. The final session was the extensive training with practice and feedback condition and was attended by 25 participants. Participants returned one week later to complete a second round of ratings. All sessions were conducted in a controlled setting consisting of a large classroom on the university campus. Study procedures were run in accordance with the university’s Institutional Review Board procedures.

Brief familiarization

Researchers presented participants in the brief familiarization condition with a 5-min presentation that provided a short overview of DBR delivered through a combination of PowerPoint slides and lecture. The presentation informed participants of the definition of DBR (i.e., “DBR is a tool that involves a brief rating of a target behavior following a specified observation period”). Participants were next shown a sample DBR and provided with directions on how to use a DBR to rate student behavior. Finally, they were provided with behavioral definitions of the three target behaviors that were to be rated: academic engagement, disruptive behavior, and compliance. Once the presentation was complete, participants viewed a brief instructional video (Another Set of Eyes; Wurzburg, 1987) that detailed the use of observation within classroom settings. Participants were provided with an opportunity to ask general questions about observation techniques; however, no specific information regarding DBR was provided.

Brief training

Researchers provided participants in the brief training condition with the same 5-min informational slide show and lecture on DBR. However, following the presentation, brief training participants were given the opportunity to practice using DBR to rate student behavior. Participants were provided with pictures of four target students who they would be viewing in video clips. They were then allotted 4 min to read behavioral descriptions of the three target behaviors and provided with a brief description of how to complete the DBR. Following this introduction, researchers initiated the practice with feedback portion of the training. Participants were shown a 1-min video clip and instructed to focus their attention on a specific target student. Following the clip, participants were told to use the DBR to rate the student’s academic engagement. Once ratings were complete, participants were asked to share their ratings. After three participants had volunteered responses, the researcher reviewed the definition of the target behavior, stated the actual percentage of time the target student had displayed academic engagement, and explained what DBR rating should have been assigned (e.g., “Well, actually when we go through this clip and look second by second at how Jack is behaving, we find that he is academically engaged 98% of the time. Therefore you should have marked the 10 on the DBR”). The researcher then provided examples of behaviors the target student had demonstrated that represented the category of academic engagement (e.g., raising his hand to answer a question, engaged in seat work). This practice and feedback process was repeated for the remaining two target behaviors of disruptive behavior and compliance. Participants then practiced rating three additional 1-min video clips, receiving feedback after each one. Thus, participants in the brief training condition completed practice and feedback exercises for four 1-min clips, yielding a training that lasted approximately 20 min.

Extensive training

The extensive training was identical to the brief training with the exception that participants in the extensive condition completed twice the amount of practice and feedback sessions. That is, participants practiced rating target behaviors for eight 1-min video clips and received corrective feedback from the researcher after each one. The extensive training lasted approximately 30 min.

Dependent measure

On completion of the training procedure, participants in all conditions were told that they would next be asked to watch several video clips that featured second-grade students participating in regular classroom activities. Participants were instructed to focus their attention on two target students in each clip to be able to rate the students’ behavior using a DBR. Participants were shown a slide and provided with a handout that contained then names, pictures, and physical descriptions of the two target students. Participants were told to abstain from making ratings until instructed to do so by the trainer. They were also told to make their ratings based only on behavior observed within the 1-min clip. Eight 1-min video clips were shown, with 30 s allotted in between clips for participants to make their ratings independently.

One week later, participants returned to the laboratory to make a second round of ratings. No training was provided. Rather, participants were reminded that they would be making ratings of student behavior using a DBR. Protocol packets were returned to participants although participants were not allowed to view their first set of ratings. Participants were given 4 min to reread descriptions, examples, and nonexamples of the target behaviors. This time, participants were provided with physical descriptions, names, and photos of two new target students. Eight new 1-min video clips were again shown. After each clip, participants were given 30 s to rate the three target behaviors for the two new targets students.

Results

Difference scores were created to examine rater accuracy. This was accomplished by subtracting the expert DBR from each participant’s DBR rating and taking the absolute value. Thus, difference scores close to 0 suggest that a participant’s rating was highly accurate while larger difference scores indicate less accurate ratings. Differences between training conditions were examined first. Participants in the brief and extensive trainings rated all three target behaviors with similar levels of accuracy. For the behavior of academic engagement, the mean difference score for the brief training group was 2.00 (SD = 1.83); mean difference score for extensive training was 1.93 (SD = 1.84). For disruptive behavior, mean difference score for participants in the brief training was 1.58 (SD = 1.79), while participants in the extensive training produced a mean difference score of 1.57 (SD = 1.79). Finally, for compliance, mean difference score for the brief training group was 1.54 (SD = 2.19), while mean difference score for extensive training was 1.77 (SD = 2.38). Results of independent samples t tests assuming unequal variances indicated that no significant differences existed between brief and extensive training groups for accuracy in rating academic engagement, t(45) = 0.13, p = .90, disruptive behavior, t(45) = 0.02, p = .99, and compliance, t(45) = 0.34, p = .73. Thus, the brief and extensive training conditions were collapsed to produce one overall training condition consisting of participants who were provided with practice and feedback sessions. Participants from both the brief familiarization and the training group rated overall target behaviors with a high degree of accuracy, with mean ratings for all target behaviors falling within two DBR points of the expert scores. Descriptive statistics for target behaviors are provided in Table 1.

Table 1.

Descriptive Statistics.

	Brief familiarization difference score		Training difference score
Behavior	M	SD	M	SD
Academic engagement	1.90	1.81	1.97	1.84
Disruptive behavior	1.95	1.97	1.57	1.79
Compliance	1.90	2.50	1.67	2.30

Academic Engagement

A 2 × 3 factorial ANOVA was used to determine the impact of training condition and base rate of academic engagement on participants’ systematic accuracy difference scores. A .05 criterion of statistical significance was used for all tests. Results indicated a significant interaction between training condition and base rate of behavior, although not in the expected direction, F(2, 2144) = 3.821, p = .022, $η_{p}^{2}$ = .004. The effect of base rate of behavior on difference scores was dependent on the training condition; however, participants who completed only the brief familiarization training rated academic engagement significantly more accurately at low levels than those who participated in trainings with practice and feedback components (see Table 2). Nonsignificant results were found for the main effect of training condition, while the main effect of base rate of behavior was significant, F(2, 2144) = 69.252, p < .001, $η_{p}^{2}$ = .061. Participants in both the brief familiarization (M = 1.46) and the training condition (M = 1.31) rated most accurately when base rates of academic engagement were high. Accuracy was significantly decreased for both the brief familiarization and training conditions at medium (M = 2.33, M = 2.38) and low rates (M = 2.18, M = 2.56) of academic engagement. Overall, training did not appear to improve accuracy in rating the behavior of academic engagement; participants in the brief familiarization condition demonstrated less error in rating low levels of academic engagement (M = 2.18) than did participants in the training condition (M = 2.56).

Table 2.

Descriptive Statistics for Academic Engagement.

	Brief familiarization difference score		Training difference score
Base rate	M	SD	M	SD
High	1.46	1.58	1.31	1.39
Medium	2.33	1.54	2.38	1.37
Low	2.18	2.14	2.56	2.34

Disruptive Behavior

A 2 × 3 factorial ANOVA was used to determine the effect of training condition and base rate of disruptive behavior on accuracy using a .05 criterion of statistical significance. The main effect of condition was significant, F(1, 2144) = 12.393, p < .001, $η_{p}^{2}$ = .006, as was the main effect of base rate of behavior, F(2, 2144) = 154.070, p < .001, $η_{p}^{2}$ = .126. Nonsignificant results were found for the interaction effect of the independent variables. Participants in both the control and training conditions rated disruptive behavior most accurately when rates of disruptive behavior were low (M = 1.61, M = 1.25) or high (M = 1.92, M = 1.48) as opposed to mid-level (M = 3.61, M = 3.31). Training improved the accuracy of participants’ ratings of disruptive behavior at all levels of behavior (low, medium, and high). Descriptive statistics for disruptive behavior at different base rates are found in Table 3.

Table 3.

Descriptive Statistics for Disruptive Behavior.

	Brief familiarization difference score		Training difference score
Base rate	M	SD	M	SD
High	1.92	1.49	1.48	1.05
Medium	3.61	1.41	3.18	1.48
Low	1.61	1.96	1.25	1.74

Compliance

A 2 × 3 factorial ANOVA was also used to determine the effects of training condition and base rate of compliance on accuracy using a .05 criterion of statistical significance. Results indicated a significant main effect for base rate of behavior, F(2, 2144) = 21.550, p < .001, $η_{p}^{2}$ = .020. However, the main effect of training condition and the interaction between base rate of behavior and training condition were found to be nonsignificant. Because of this, difference scores for the brief familiarization and training groups were examined together. Participants rated compliance significantly less accurately when students displayed medium rates of compliance (M = 2.75). Participants rated compliance more accurately when students displayed low (M = 1.73) or high (M = 1.57) base rates of the behavior (see Table 4).

Table 4.

Descriptive Statistics for Compliance.

	Combined conditions difference score
Base rate	M	SD
High	1.57	2.49
Medium	2.75	1.63
Low	1.73	1.67

Discussion

The overall purposes of this study were twofold: (a) to identify whether training with practice and feedback improved rater accuracy for DBR and (b) to evaluate whether different base rates of target behavior (i.e., low, medium, high) were associated with differences in rater accuracy. Although comparison of the amount of direct training (i.e., brief vs. extensive) was proposed, participants from the two training conditions rated behaviors with similar accuracy; thus the two training conditions were combined to create one overall training with practice and feedback condition. Participants in all conditions were able to produce fairly accurate ratings of student behavior in that overall mean ratings fell within two DBR points of the expert score for each target behavior. Researchers hypothesized that training procedures that included a practice and feedback component would result in increased rater accuracy; however, this occurred only with ratings for disruptive behavior. Contrary to expectations, trainees who were provided with indirect, brief familiarization training on DBR rated academic engagement and compliance as accurately as well or better than those who participated in direct trainings that included a practice with feedback component. This is fairly consistent with previous research that found similar DBR rating patterns among trainees who participated in either brief familiarization (indirect) or practice with feedback (direct) trainings (Schlientz et al., 2009). Similarly, LeBel et al. (2010) found that trainees rated student behavior with similar levels of accuracy after participating in either brief familiarization or indirect trainings. The DBR has been designed to be a tool that is simple, easy to understand, and quick to complete, thus providing a feasible method for the collection of behavioral data. This simplicity of design and ease of use may also set it apart from other assessment methods (e.g., systematic direct observation) in that less training may be needed to implement the tool with integrity.

Although training appears to have had a limited overall effect on rater accuracy, important differences were revealed in regards to the effect of base rate of behavior on accuracy. Accuracy varied substantially depending on whether a target behavior was displayed at a low, medium, or high rate. For disruptive behavior and compliance, participants rated most accurately at either low or high base rates of behavior (e.g., behavior displayed for less than 33% or more than 67% of the rating period). Accuracy dropped substantially at medium base rates of behavior (e.g., behavior displayed for 34%-66% of clip). However, for academic engagement, trainees rated most accurately at high base rates of behavior, but had more difficulty rating at both middle and low rates. In addition, contrary to expectations, participants who received only a brief familiarization training rated low base rates of academic engagement more accurately than those who had participated in direct training procedures. Interestingly, this mirrors the findings of LeBel et al. (2010) who also found that direct training led to lower accuracy in rating academic engagement. Yet, in that study, base rate of behavior was not considered; thus, results must be interpreted cautiously. Overall, findings indicate that extensive training may not be necessary for raters to produce accurate DBR ratings. Trainings that provide a brief overview of DBR, directions for how to make ratings, and information on target behaviors may be sufficient to enable raters to use DBR to accurately rate student behavior. Importantly, the study provides preliminary evidence that accuracy varies at different base rates of behavior although future research is needed to confirm this. If this is supported, trainings for DBR may need to provide opportunities for trainees to practice rating at varying rates of behavior, with particular focus on those behavior and rate combinations that appear to be most difficult for raters to assess (i.e., low to medium rates of academic engagement, medium rates of disruptive behavior and compliance).

Limitations

Several limitations to the current study warrant consideration. First, the study was limited through its use of a convenience sample of undergraduate college students. Participants in the study reported limited prior exposure to DBR, and most did not have experience working in educational settings. Thus the sample differed in important ways from typical users of DBR (i.e., teachers), and the response to training and the rating accuracy of the sample may have limited generalizability to real world settings. An effort should be made to conduct future training studies with teachers to determine specific training needs for this target population. Second, although the use of videotapes was critical to ensure that all participants viewed identical samples of behavior, the artificial nature of the design may limit the external validity of the study. In the laboratory setting, participants were able to focus full attention on individual target students for brief, 1-min intervals. This observation differs significantly from what may occur in a classroom setting where a teacher is likely juggling multiple tasks and is responsible for supervising dozens of students at any given moment. The short duration of video clips (1 min) may be problematic in that DBR is typically used to rate student behavior following a longer period of time. However, past research has examined whether observation duration (i.e., 5-, 10-, 20-, and 40-min periods) affects rating accuracy, with results suggesting that accuracy is not reduced when ratings follow shorter observation periods (Riley-Tillman et al., 2010). Future training studies may be improved by providing videos of varying durations for the practice with feedback components of training.

Implications for Future Research

The findings of this study support previous research into optimal training conditions and accuracy of DBR. Specifically, the current study provides additional evidence that training improves raters’ accuracy for measuring some student behaviors, specifically disruptive behavior. In addition, it also supports past findings that trainees who participate only in brief familiarization exercises are able to use DBR to rate student behavior with high levels of accuracy (LeBel et al., 2010; Schlientz et al., 2009). An important finding is that additional practice with feedback opportunities may be unnecessary to ensure that DBR users are able to accurately rate target behaviors during brief observation periods. Future research may be needed to determine whether this finding holds when rater demands are increased. For instance, would practice sessions be beneficial prior to using DBR in actual classrooms where the rater may be charged with observing multiple students at once for longer observation periods? Finally, a unique finding from this study is that training packages may need to address the difficulty of rating behaviors that are displayed at medium rates. As such, future research should investigate other DBR training components to identify what elements are most effective in training raters to observe and measure mid-level behavior. In addition, future studies need to move beyond convenience samples to target relevant school-based professionals. Nonetheless, the finding that even brief trainings can provide individuals with the skills needed to rate student behavior with a fair level of accuracy provides further support for the feasibility of the DBR-SIS as a social-behavioral assessment tool that can provide teachers with a reliable, quick, and easy method for the collection of student behavioral data.

Footnotes

Authors’ Note

The opinions expressed herein do not necessarily reflect the position of the U.S. Department of Education, and such endorsements should not be inferred.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Preparation of this article was supported by a grant from the Institute for Education Sciences, U.S. Department of Education (R324B060014).

Author Biographies

Sayward E. Harrison, PhD, is a school psychologist at Guilford County Schools Psychological Services, where she provides consultation to parents and teachers, completes psychological and behavioral assessments, and assists with intervention planning. She received a PhD in health psychology with a specialization in pediatric school psychology from East Carolina University. Areas of professional interest include behavioral assessment and intervention, chronic health conditions, and school-based mental health services.

T. Chris Riley-Tillman, PhD, is currently a professor at the University of Missouri. He received his PhD in school psychology from Syracuse University. In his current position, he provides training in assessment, intervention, and consultation. His research interests and expertise involve development and validation of assessment and intervention methodologies, single case design, and consultation. He has authored more than 75 articles, book chapters, and books. Finally, he is a fellow in the American Psychological Association and an invited member of the Society for the Study of School Psychology.

Sandra M. Chafouleas, PhD, is an associate dean of The Graduate School and professor in the Neag School of Education at the University of Connecticut. She has authored more than 100 refereed articles, book chapters, and books—topics of which are primarily related to school-based behavioral issues. She is a fellow in both the American Psychological Association and Association for Psychological Science and an invited member of the Society for the Study of School Psychology. Prior to becoming a university trainer, she worked as a school psychologist and school administrator in a variety of settings for children with behavior disorders.

References

Briesch

A. M.

Chafouleas

S. M.

Riley-Tillman

T. C.

(2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408-421.

Chafouleas

S. M.

Jaffery

Riley-Tillman

T. C.

Christ

T. J.

Sen

(2013). The impact of target, wording, and duration on rating accuracy for direct behavior rating. Assessment for Effective Intervention, 38(4), 1-15.

Chafouleas

S. M.

Kilgus

S. P.

Hernandez

(2009). Using direct behavior rating (DBR) to screen for school social risk: A preliminary comparison of methods in a kindergarten sample. Assessment for Effective Intervention, 34, 224-230. doi:10.1177/1534508409333547

Chafouleas

S. M.

Kilgus

S. P.

Jaffery

Riley-Tillman

T. C.

Welsh

M. E.

Christ

T. J.

(2013). Direct Behavior Rating as a school-based behavior screener for elementary and middle grades. Journal of School Psychology, 51, 367-385.

Chafouleas

S. M.

McDougal

J. L.

Riley-Tillman

T. C.

Panahon

C. J.

Hilt

A. M.

(2005). What do daily behavior report cards (DBRCs) measure? An initial comparison of DBRCs with direct observation for off-task behavior. Psychology in the Schools, 42, 669-676.

Chafouleas

S. M.

Riley-Tillman

T. C.

Christ

T. J.

(2009). Direct behavior rating (DBR): An emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 195-200. doi:10.1177/1534508409340391

Chafouleas

S. M.

Riley-Tillman

T. C.

McDougal

J. L.

(2002). Good, bad, or in-between: How does the daily behavior report card rate? Psychology in the Schools, 39, 157-169. doi:10.1002/pits.10027

Chafouleas

S. M.

Riley-Tillman

T. C.

Sassu

K. A.

(2006). Acceptability and reported use of daily behavior report cards among teachers. Journal of Positive Behavior Interventions, 8, 174-182. doi:10.1177/10983007060080030601

Christ

T. J.

Riley-Tillman

T. C.

Chafouleas

S. M.

(2009). Foundation for the development and use of direct behavior rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention, 34, 201-213. doi:10.1177/1534508409340390

10.

Christ

T. J.

Riley-Tillman

T. C.

Chafouleas

S. M.

Jaffery

(2011). Direct behavior rating: An evaluation of alternate definitions to assess classroom behaviors. School Psychology Review, 40, 181-199.

11.

Flanagan

Adams

H. E.

Forehand

(1979). A comparison of four instructional techniques for teaching parents to use time-out. Behavior Therapy, 10, 94-102.

12.

LeBel

T. J.

Chafouleas

S. M.

Sugai

G. M.

(2006). Project VIABLE technical report for Direct Behavior Rating Scales: Rationale for selection of behaviors to be investigated. Storrs: University of Connecticut.

13.

LeBel

T. J.

Kilgus

S. P.

Briesch

A. M.

Chafouleas

S. M.

(2010). The impact of training on the accuracy of teacher-completed direct behavior ratings (DBRs). Journal of Positive Behavioral Interventions, 12, 55-63. doi:10.1177/1098300708325265

14.

Riley-Tillman

T. C.

Chafouleas

S. M.

Christ

T. J.

Briesch

A. M.

LeBel

T. J.

(2009). The impact of item wording and behavioral specificity on the accuracy of Direct Behavior Ratings (DBRs). School Psychology Quarterly, 24, 1-12. doi:10.1037/a0015248

15.

Riley-Tillman

T. C.

Chafouleas

S. M.

Eckert

(2008). Daily behavior report cards and systematic direct observation: An investigation of the acceptability, reported training and use, and decision reliability among school psychologists. Journal of Behavioral Education, 17, 313-327. doi:s10864-008-9070-5

16.

Riley-Tillman

T. C.

Chafouleas

S. M.

Sassu

K. A.

Chanese

J. A. M.

Glazer

A. D.

(2008). Examining the agreement of direct behavior ratings and systematic direct observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions, 10, 136-143.

17.

Riley-Tillman

T. C.

Christ

T. J.

Chafouleas

S. M.

Boice

C. H.

Briesch

A. M.

(2010). The impact of observation duration on the accuracy of data obtained from direct behavior rating (DBR). Journal of Positive Behavior Interventions. Advance online publication. doi:10.1177/1098300710361954

18.

Riley-Tillman

T. C.

Kalberer

S. M.

Chafouleas

S. M.

(2005). Selecting the right tool for the job: A review of behavior monitoring tools used to assess student response to intervention. The California School Psychologist, 10, 81-91.

19.

Riley-Tillman

T. C.

Methe

S. A.

Weegar

(2009). Examining the use of direct behavior rating methodology on classwide formative assessment: A case study. Assessment for Effective Intervention, 34, 242-250. doi:10.1177/1534508409333879

20.

Saudargas

R. A.

Creed-Murrah

(1980). State-event classroom observation system. Knoxville: University of Tennessee, Department of Psychology.

21.

Schlientz

M. D.

Riley-Tillman

T. C.

Briesch

A. M.

Walcott

C. M.

Chafouleas

S. M.

(2009). The impact of training on the accuracy of direct behavior ratings (DBRs). School Psychology Quarterly, 24, 73-83. doi:10.1037/a0016255

22.

Shapiro

E. S.

(2008). Behavioral Observation of Students in Schools (BOSS). Minneapolis, MN: Pearson Assessments.

23.

Sterling-Turner

H. E.

Watson

T. S.

Wildmon

Watkins

Little

(2001). Investigating the relationship between training type and treatment integrity. School Psychology Quarterly, 16, 56-67.

24.

Tapp

(2004). MOOSES (Multi-Option Observation System for Experimental Studies). Retrieved from http://kc.vanderbilt.edu/mooses/mooses.html

25.

Watson

T. S.

Kramer

J. J.

(1995). Teaching problem solving skills to teachers-in-training: An analogue experimental analysis of three methods. Journal of Behavioral Education, 5, 281-293. doi:10.1007/bf02110316

26.

Wurzburg

(1987). Another set of eyes: Techniques for classroom observation [Video]. Alexandria, VA: Association for Supervision and Curriculum Development.