Direct Behavior Rating Instrumentation

Scale Development of DBR-SIS

DBR-SIS were developed as a systematic approach to monitor the progress of student behavior. DBR-SIS evolved from a myriad of user-created DBR tools (behavior report cards, homeschool notes, point sheets), where “home grown” assessments were created to fill the void of available behavioral progress monitoring tools. Yet, the psychometric properties of these tools remain unknown. To fill this gap, DBR-SIS were designed as a hybrid behavioral assessment method, combining the benefits of systematic direct observation (SDO) with an efficient rating scale format (Chafouleas, Christ, Riley-Tillman, Briesch, & Chanese, 2007; Chafouleas, Riley-Tillman, & Sugai, 2007). Specifically, teachers observe students during a pre-specified observation period and immediately following the observation, record ratings of the target behaviors on a 0 to 10 scale. As outlined by Chafouleas (2011), considerable work was undertaken to inform the development of the scales, including reviews of prior research on scale development and validation studies. In regard to scale development, various options were considered in terms of polarity (unipolar or bipolar), the scale of measurement (level of measurement or range of scores), the scale format (continuous, pseudocontinuous, or categorical), and number of gradients included (see Christ & Boice, 2009). Prior research examining variations in scale format (e.g., number of gradients, continuous or categorical scales) supported the flexibility of the scales and suggested negligible differences in rating accuracy as a function of scale format (Briesch, Kilgus, Chafouleas, Riley-Tillman, & Christ, 2012; Christ, Riley-Tillman, & Chafouleas, 2009; Riley-Tillman, Christ, Chafouleas, Boice-Mallach, & Briesch, 2011). However, questions remain regarding the impact of other key features of the scales that have yet to be studied empirically.

Although DBR-SIS demonstrates flexibility in some aspects, it is not clear whether other features of the scale affect measurement error. In particular, duration was selected as the behavioral dimension of interest in developing DBR-SIS, and the scale includes explicit instructions referencing duration (i.e., place a mark along the line that best reflects the percentage of total time the student exhibited each target behavior). Furthermore, percentage of time anchors is included on the scale (0%, 50%, and 100%). Several bodies of literature, briefly described below, informed this selection including (a) measurement research related to scale construction and (b) behavioral observation research (e.g., Hartley, Trueman, & Rodgers, 1984; Newstead & Arnold, 1989; Skinner, Rhymer, & McDaniel, 2000).

Although the structure of DBR-SIS resembles a Likert-type scale, the inclusion of percentage anchors results in equal interval measurement not typically associated with Likert-type scales. For example, a 4-point Likert-type scale might include the following response options: never, sometimes, often, always, and are often considered to be at the ordinal level of measurement. Conversely, DBR-SIS includes a numerical range from 0 to 10 that can be considered to be at the interval level of measurement. The result is a pseudocontinuous scale, as ratings are rounded to the nearest whole number for interpretation. This presumably results in more desirable measurement characteristics, given increased precision of equal interval measurement compared with the ordinal measurement associated with traditional Likert-type scales (Hartley et al., 1984). Prior measurement research on the issue of scaling has supported the use of percentage anchors when possible, as opposed to numerical anchors or verbal anchors (such as those typically utilized on Likert-type scales) as they are often less accurate (Hartley et al., 1984; Newstead & Arnold, 1989). In addition, it has been argued that including percentage of time rather than general frequency descriptors allows for a more accurate comparison of results between individuals (Diener, Smith, & Fujita, 1995).

Although global recommendations have been made regarding scale construction, an additional contextual consideration in the measurement of behavior involves the target behaviors of interest. The process of selecting and operationalizing behavior targets and selecting measurement approaches is inherently related (Skinner et al., 2000). That is, in behavioral observational research, operationalizing the target behavior informs the dimension of interest and subsequent measurement of that behavior. Efforts have been made to assist in the classification of behaviors and selecting the appropriate measurement approach.

Altmann (1974) and Saudargas and Lentz (1986) conceptualized behavior as falling along a state-event continuum. Within this framework, state behaviors are considered to be behaviors with meaningful duration (e.g., on-task, out of seat, etc.) whereas event behaviors are of brief duration and have clear start and end points (e.g., calling out, swearing, etc.). Many behaviors can be conceptualized as either states or events, but some fall within both categories. In fact, it can be argued that some of the core behavioral competences targeted on DBR-SIS forms do not clearly fall into one category—while academically engaged (AE) behavior can be readily categorized as a state, disruptive (DB) and respectful (RS) behavior could be conceptualized as either states or events. The categorization of state versus event behaviors becomes increasingly important when deciding how to measure different behaviors. According to Saudargas and Lentz (1986), state behaviors are best studied by examining the percent of total time the behavior occurred across an observation session. Event behaviors, on the contrary, are best studied by examining their frequency. Because some of the behaviors on the DBR-SIS scale are not clearly interpreted as either states or events, it remains unclear how the scale format might affect rating accuracy.

Purpose

In light of these considerations, additional research is needed to evaluate the instrumentation of DBR-SIS, and the scaling in particular. Consequently, this study was designed to evaluate whether or not including percentage of time anchors and an explicit reference to duration on a DBR-SIS 0 to 10 scale affects the accuracy with which individuals rate student behavior. The following research question guided our inquiry:

It was hypothesized that including percentage of time anchors would affect ratings only when the target behavior was a clear state behavior (AE) but not for behaviors that did not clearly fall into one category (i.e., DB and RS).

Method

Results

In Table 1, the descriptive statistics for the DBR-SIS ratings for all nine behaviors by group (percent and no percent) are displayed. To examine differences in rater error scores, a series of Mann–Whitney U tests were performed to evaluate differences in mean ranks between groups. Holm–Bonferroni corrections were used to adjust for multiple comparisons across the nine clips. Results from the analyses revealed statistically significant differences between groups for both low AE and high AE clips. For the low AE clip, significant differences were found between groups using both SDO error scores (|DBR_Part − SDO|; U = 1,185.5, p = .002) and expert error scores (|DBR_Part − DBR_Exp|; U = 889.5, p < .001). Similarly, significant differences were found between groups on the high AE clip using both SDO error scores (U = 1279.0, p = .004) and expert error scores (U = 1279, p = .004). No statistically significant differences were found for the other clips.

OPEN IN VIEWER

Table 1.

Descriptive Statistics.

Behavior	Level	SDO rating	Expert rating	Group	n	Mdn DBR-SIS	IQR
							Q1	Q3
Academically engaged	Low	1.7	1.0	Percent	64	1.00	0.25	2.00
				No percent	55	3.00	2.00	5.00
	Medium	4.8	4.0	Percent	64	6.00	5.00	8.00
				No percent	55	6.00	4.00	8.00
	High	9.8	10.0	Percent	64	10.00	9.00	10.00
				No percent	55	9.00	8.00	10.00
Disruptive	Low	1.7	2.0	Percent	64	3.00	1.00	5.00
				No percent	55	2.00	1.00	5.00
	Medium	5.7	7.0	Percent	64	8.00	6.00	9.00
				No percent	55	8.00	6.00	9.00
	High	8.5	9.0	Percent	64	9.00	8.25	10.00
				No percent	55	9.00	8.00	10.00
Respectful	Low	1.7	1.0	Percent	64	1.00	1.00	2.00
				No percent	55	2.00	1.00	3.00
	Medium	5.3	3.0	Percent	64	2.00	1.00	4.00
				No percent	55	3.00	2.00	5.00
	High	10.0	10.0	Percent	64	9.00	7.00	9.00
				No percent	55	9.00	7.00	10.00

Note. SDO = systematic direct observation; DBR-SIS = Direct Behavior Rating–Single Item Scale; IQR = interquartile range.

To explore the practical significance of the significant findings, effect sizes were generated using the probability of superiority (PS) metric recommended by Nussbaum (2014). This metric is used to determine the probability that a randomly sampled score from one group would be higher than the comparison group. According to interpretive guidelines developed by Grissom (1994) and Grissom and Kim (2005), a PS value of 56% represents a small effect, 65% represents a medium effect, and 71% represents a large effect. For the low AE clip, PS values ranged from 66% (|DBR_Part − SDO|) to 75% (|DBR_Part − DBR_Exp|), suggesting a medium to large effect; a 66% to 75% chance of randomly drawing a higher error score for the no percent group on that clip. For the high AE clip, PS values were 64% for both |DBR_Part − SDO| and |DBR_Part − DBR_Exp| error scores, indicating a small to medium effect; a 64% chance of randomly drawing a higher error score for the no percent group on that clip.

To derive robust estimates of confidence intervals for the sample, bootstrapping procedures were used. To this end, participants’ scores are used to generate bootstrap samples, and a median is calculated for each sample. The bootstrapped medians are then put in order, from lowest to highest, and the central 95% of values are used to form the confidence interval (Keselman, Algina, Lix, Wilcox, & Deering, 2008). Bootstrapping procedures were used on the sample of rater error scores for all nine clips using IBM SPSS statistical computing software version 22. Ten thousand samples were created with replacement to generate 95% bootstrapped confidence intervals about the median error scores for each group (see Tables 2 and 3).

OPEN IN VIEWER

Table 2.

Results of Nonparametric Tests and 95% Bootstrapped CIs: SDO Error Scores.

Behavior	Level	Group	Mann–Whitney U	p	Mean rank	Mdn |DBRPart − SDO|	Bootstrapped 95% CI |DBRPart − SDO|
							Lower	Upper
Academically engaged	Low	Percent	1,185.50	.002*	51.02	0.70	0.70	1.30
		No percent			70.45	1.70	1.30	2.30
	Medium	Percent	1,496.50	.157	64.12	2.20	1.80	3.20
		No percent			55.21	2.20	1.20	2.20
	High	Percent	1,279.00	.004*	52.48	0.20	0.20	0.20
		No percent			68.75	0.80	0.20	1.80
Disruptive	Low	Percent	1,658.50	.584	61.59	1.70	0.70	2.30
		No percent			58.15	1.30	0.70	1.70
	Medium	Percent	1,622.50	.455	62.15	2.30	2.30	2.30
		No percent			57.50	2.30	1.30	2.30
	High	Percent	1,758.50	.993	60.02	1.50	0.50	1.50
		No percent			59.97	1.50	0.50	1.50
Respectful	Low	Percent	1,688.50	.695	58.88	0.70	0.70	1.30
		No percent			61.30	1.30	0.70	1.30
	Medium	Percent	1,384.50	.043	65.87	3.30	2.30	4.30
		No percent			53.17	2.30	2.30	3.30
	High	Percent	1,652.50	.557	61.68	1.00	1.00	2.00
		No percent			58.05	1.00	1.00	2.00

Note. SDO = systematic direct observation; DBR = Direct Behavior Rating; CI = confidence interval.

OPEN IN VIEWER

Table 3.

Results of Nonparametric Tests and 95% Bootstrapped CIs: Expert Error Scores.

Behavior	Level	Group	Mann–Whitney U	p	Mean rank	Mdn |DBRPart − DBRExp|	Bootstrapped 95% CI |DBRPart − DBRExp|
							Lower	Upper
Academically engaged	Low	Percent	889.50	.000*	46.40	1.00	0.00	1.00
		No percent			75.83	2.00	2.00	3.00
	medium	Percent	1,608.00	.411	57.24	3.00	2.00	3.00
		No percent			62.38	2.00	2.00	3.00
	high	Percent	1,279.00	.004*	52.48	0.00	0.00	0.00
		No percent			68.75	1.00	0.00	2.00
Disruptive	Low	Percent	1,657.50	.575	61.60	2.00	1.00	2.00
		No percent			58.14	1.00	1.00	2.00
	medium	Percent	1,576.50	.305	62.87	1.00	1.00	2.00
		No percent			56.66	1.00	1.00	2.00
	high	Percent	1,595.50	.319	57.43	1.00	1.00	1.00
		No percent			62.99	1.00	1.00	1.00
Respectful	Low	Percent	1,411.50	.049	54.55	1.00	0.00	1.00
		No percent			66.34	1.00	1.00	1.00
	medium	Percent	1,726.00	.852	60.53	2.00	1.00	2.00
		No percent			59.38	1.00	1.00	2.00
	high	Percent	1,652.50	.557	61.68	1.00	1.00	2.00
		No percent			58.05	1.00	1.00	2.00

Note. DBR = direct behavior rating; CI = confidence interval.

Discussion

Overall, findings from the present investigation suggest small but potentially important differences in rating accuracy between groups. Specifically, Low and High AE were the only two behavior clips that produced statistically significant differences in rating accuracy across the percent and no percent groups. In both cases, the percent group produced more accurate ratings than the no percent group. This finding was consistent across both SDO error scores and expert error scores, providing important corroboration in the absence of a clear gold standard by which to measure the accuracy of DBR-SIS ratings.

Considering previous research on state versus event behaviors, it is reasonable to expect that AE behavior would be the most affected by the scale format on the rating scale due to its clear classification as a state behavior (Saudargas & Lentz, 1986). That is, state behaviors are best measured by duration. Furthermore, academic engagement does not lend itself well to other possible dimensions of interest, such as frequency or intensity. That is, duration is unequivocally the most important dimension to measure, thus the scale referencing duration was associated with more accurate scores. Conversely, significant differences were not observed between scale formats for DB and RS behavior, both of which do not clearly fall into either category in the state-event dichotomy. As such, including the percent anchors and referencing duration did not appear to affect the accuracy of ratings, despite research pointing to inaccuracies associated with low-specificity descriptors such as those used in the no percent group (Lucas, Diener, & Larsen, 2009). Although we can only speculate as to the reasons for these discrepancies, it seems reasonable to expect that scale format may have greater impacts on some behaviors than others. Specifically, either scaling approach may be adequate for the measurement of behaviors not clearly falling within the state/event dichotomy.

This study provides an important contribution to the literature on the instrumentation of DBR-SIS scales. Particularly, the inclusion of percentage of time anchors may be more important for some behaviors than others, aligning to Saudargas and Lentz’s (1986) conclusion about the differences in measuring state versus event behaviors. Therefore, the scaling utilized for behaviors is a particularly relevant consideration when practitioners use DBR-SIS or DBR-like tools. That is, given the genesis of DBR tools and a history of user-created measures, consideration needs to be given regarding the construction of the scale and scaling of specific behaviors to maximize the desirable psychometric properties of these instruments.

With regard to the effect size estimates, moderate to large effects were observed for those contrasts yielding significant differences between groups. In terms of data-based decision making, it is plausible that such differences could have modest impacts on goal setting and determinations made regarding goal attainment. This is important if the goal is to enhance the psychometric adequacy of rating data (Saal et al., 1980). Consistent with previous research, results from this study suggest that DBR-SIS may be a flexible tool with regard to instrumentation design, while also suggesting that decisions regarding scale construction may need to be considered within the context of specific behaviors (Briesch et al., 2012; Riley-Tillman et al., 2011; Skinner et al., 2000).

Conclusion

Given DBR’s intended use for data-based decision making, it is important to understand factors that influence rating accuracy. Thus, calls have been made for further investigations regarding the development and evaluation of DBR instrumentation (Christ & Boice, 2009). This study provides preliminary data regarding how the instrumentation of DBR-SIS may affect rating accuracy. Specifically, the inclusion of percentage of time anchors and references to duration were associated with more accurate ratings of AE behavior. In light of these findings, we recommend that behavioral assessment researchers and developers continue to attend to scaling formats utilized and study potential impacts on rating accuracy.