Comparability of Fidelity Measures for Assessing Tier 1 School-Wide Positive Behavioral Interventions and Supports

Sample

To determine eligible schools and fidelity assessments, we first selected the last fidelity assessment of the school year (2005–2006 through 2014–2015) for each of five measures for schools that reported fidelity data to the Office of Special Education Programs (OSEP) National Technical Assistance Center on PBIS. For each pair of fidelity measures, we then selected schools that had completed both fidelity assessments within 30 days in the same academic year. Thus, the sample for each pairwise comparison of fidelity measures differed (see Table 3), and school demographic information also differed to some extent across each of the 10 fidelity measure comparisons. The following characteristics, based on schools that completed at least one PBIS fidelity measure in 2012–2013, are representative of schools reporting data to the OSEP National Technical Assistance Center on PBIS. Characteristics for these schools are presented for 1 year only because some schools were included in samples across multiple years. The schools were located in urban (31%), suburban (29%), town (14%), or rural (26%) areas in 42 different U.S. states. Regarding grade levels served, 67% were elementary schools, 20% were middle schools, and 11% were high schools. Of these schools, 72% were eligible for Title I programs based on student economic need. On average, 51% of students in the schools were eligible for free or reduced price meals (SD = 25%), and the average school racial and ethnic student composition was the following: 57% White (SD = 33%), 18% Hispanic or Latino (SD = 24%), 18% Black or African American (SD = 25%), 4% Asian (SD = 7%), 3% Two or more races (SD = 4%), and less than 1% American Indian or Alaskan Native and Native Hawaiian or Other Pacific Islander.

OPEN IN VIEWER

Table 3.

Descriptive Information and Tests of Mean Differences for Pairs of Fidelity Assessments.

Measure 1	Measure 2	n	%Fid. 1	%Fid. 2	M 1	M 2	Difference	SD 1	SD 2	d	r
SET	BOQ	1,103	78	78	89.34	79.60	9.74***	11.78	16.27	.69	.63***
	SAS	2,055	75	49	87.23	77.08	10.15***	15.03	13.10	.72	.64***
	TIC	1,269	72	58	86.53	78.58	7.95***	14.00	15.79	.53	.59***
	TFI	36	61	58	73.89	62.03	−11.86***	28.20	32.31	.39	.92***
BOQ	SAS	3,705	82	56	81.17	79.20	1.97***	15.68	11.77	.14	.68***
	TIC	1,553	76	65	78.45	80.54	2.10***	18.39	16.89	.12	.71***
	TFI	200	90	90	84.83	85.39	0.57	13.87	15.11	.04	.65***
SAS	TIC	3,706	39	48	73.36	74.02	−0.63**	14.57	19.17	.04	.67***
	TFI	613	60	80	79.66	80.70	1.04	12.48	18.44	.10	.70***
TIC	TFI	119	56	76	75.87	77.03	1.16	21.84	21.74	.05	.96***

Note. % Fid. = percentage of schools in the sample at or above the fidelity criterion for the measure (see Tables 1 or 4 for criterion values). SET = School-Wide Evaluation Tool; BOQ = School-Wide Benchmarks of Quality; SAS = PBIS Self-Assessment Survey; TIC = Team Implementation Checklist; TFI = Tiered Fidelity Inventory.

**

p < .01. ***p < .001.

Measures

Procedures

Extant data were retrieved from PBIS Assessment, a database housed at the University of Oregon (http://pbisapps.org), a free online application system available to any U.S. school so long as they have (a) an identified PBIS coordinator and (b) agree to the use of their data for research purposes. PBIS Assessment allows users to enter and track fidelity of implementation and outcome data to facilitate data-based decision making and evaluation. The SWPBIS fidelity data used in this study were entered by school or district personnel into PBIS Assessment and extracted by the research team; thus, there was variability in the specific roles of respondents within and across fidelity measures, and data on fidelity of assessment procedures are not available.

Data Analyses

To address Research Question 1 regarding convergent validity, we calculated Pearson’s r for each pair of fidelity assessments. To address Research Question 2 regarding the presence of mean differences on fidelity assessments that could complicate judgments of whether or not a school is adequately implementing, we conducted paired sample t tests for each pair of measures and calculated standardized mean difference (d) effect sizes using the following formula (Dunlap, Cortina, Vaslow, & Burke, 1996):

d = ( M 1 − M 2 ) S D ,

where M₁ and M₂ are mean scores on fidelity assessments, and SD is the pooled standard deviation for the two fidelity assessments. To further address this research question, we compared the percentage of schools that were at or above the recommended fidelity criteria for the measures in the samples of paired fidelity assessments.

To address Research Question 3 regarding the extent to which the fidelity assessments are sensitive to variability in implementation below, near, and above recommended criteria for implementation status, we conducted equipercentile score linking with log-linear polynomial presmoothing (see Kolen & Brennan, 2004). In equipercentile linking, each school’s score on a target fidelity assessment is transformed such that the score has the same percentile rank as on a reference fidelity assessment (i.e., the linked scores have the same percentile rank relative to the group of schools). We selected equipercentile linking instead of other linear linking measures because we did not expect the relations among scores on different fidelity assessments to be consistent across the distribution of scores on the measures. Because scores in samples tend to be less smooth (i.e., score frequencies change more erratically across the score distribution) than scores in populations, it is customary to smooth scores before linking to reduce sample-specific irregularities. Based on comparisons of model fit for each pair of assessments, we used bivariate log-linear smoothing models that included polynomial terms up to the fourth power (thus preserving the mean, variance, skew, and kurtosis) and a two-way interaction term. Because observed score linking methods require large sample sizes, TFI scores were linked only with the SAS (n = 613); sample sizes for linking with other Tier 1 measures were all large (minimum n = 1,103).

Convergent Validity and Mean Differences

In general, convergent validity among all assessments was moderate, with rs ranging from .59 to .71 (p < .001), with the exception of two comparisons with the smallest sample sizes, the SET with the TFI (r = .92, p < .001, n = 36), and the TIC with the TFI (r = .96, p < .001, n = 119). Regarding mean differences in scores, scores on the SET were consistently higher compared with all other fidelity assessments (by 7.95–11.86 percentage points, all p < .001, d = .39–.72). Encouragingly, there were no statistically significant mean differences between the TFI and the BOQ, SAS, or TIC (d = .04–.10). Although there were statistically significant mean differences between scores on the BOQ, SAS, and TIC, most likely due to the very large sample sizes for these comparisons (smallest n = 1,553), the actual mean differences were of negligible to small magnitude (0.63–2.10 percentage points, d = .04–.14).

The higher scores obtained on the SET did not consistently manifest in more schools being classified as at or above the criteria for adequate implementation, however, because the recommended criteria for adequate implementation on the SET includes two scores (≥80% on both the total score and the Behavior Expectations Taught subscale, known as the 80–80 criterion). For fidelity assessments with a criteria of ≥70% on the total score (i.e., BOQ and TFI) and the SET with the 80–80 criterion, approximately similar percentages of schools were classified as adequately implementing. Specifically, 78% of schools were classified as adequately implementing on both assessments in the sample comparing the SET and BOQ (n = 1,103); 61% and 58% as adequately implementing on the SET and TFI, respectively (n = 36); and 90% adequately implementing on both the BOQ and TFI (n = 200). By contrast, more schools were consistently classified as adequately implementing on the SET, BOQ, and TFI when compared with assessments with greater than 80% criteria (i.e., the SAS and TIC). For example, 75% versus 49% of schools were classified as adequately implementing in the comparison of the SET and SAS (n = 2,055), and 72% versus 58% of schools were classified as adequately implementing when comparing the SET and TIC (n = 1,269).

Score Linking

Equipercentile linked scores for the measures are presented graphically in Figure 1. In addition, linked scores on each fidelity measure at the recommended implementation criterion (based on Tier 1 total scores) for each measure are presented in Table 4. If linked scores on one fidelity measure were perfectly equivalent to another measure, data points would be on the diagonal (dotted line); if higher scores on the second measure (as listed in the figure legend) are more likely to be obtained than on the first measure, data points are above the diagonal, with the opposite pattern (below diagonal) if higher scores are likely to be obtained on the first measure. Curved lines in the figure indicate that score differences between measures are not constant across levels of implementation (i.e., differences are more pronounced at particular levels of implementation). In the figure, linked scores for the comparisons involving the SET (as the second measure) were consistently above the diagonal (i.e., schools administered the SET and other fidelity measures had consistently higher SET scores), reflecting the higher scores on the SET that were reported in the tests of mean differences and indicating less sensitivity of the SET to capture differences in higher SWPBIS implementation levels relative to other fidelity measures.

OPEN IN VIEWER

Figure 1.

Linked scores on pairs of fidelity assessments.

OPEN IN VIEWER

Table 4.

Linked Scores at Criteria for Adequate Implementation Fidelity.

Measure 1	Recommended criterion	Measure 2	Linked score
SET a	80	BOQ	64
		SAS	68
		TIC	68
BOQ	70	SET	85
		SAS	70
		TIC	73
SAS	80	TFI	84
		SET	93
		BOQ	84
		TIC	83
TIC	80	SET	90
		BOQ	78
		SAS	78
TFI	70	SAS	72

Note. SET = School-Wide Evaluation Tool; BOQ = School-Wide Benchmarks of Quality; SAS = PBIS Self-Assessment Survey; TIC = Team Implementation Checklist; TFI = Tiered Fidelity Inventory.

a

The recommended criterion for the SET for adequate implementation is a score of at least 80 on the total and Behavioral Expectations Taught subscale—the linking analyses are based only on the SET total score.

For SAS scores compared with all other measures, there is asymptotic curvature at the upper end of the linked score distributions, indicating that there is less of a score ceiling on the SAS compared with other measures (including the TFI) and thus more sensitivity of the SAS to differences in higher levels of SWPBIS implementation compared with other measures. For measures other than the SET (which tends to yield higher scores across the distribution), higher scores were more likely to be obtained on the SAS than on other measures when scores were roughly below 70%, indicating less sensitivity to differences in lower levels of SWPBIS implementation. When scores were approximately in the range of 70% to 80%, the linked scores of the SAS with the TFI, BOQ, and TIC were approximately equal, and then lower scores were likely to be obtained on the SAS when above 80%, in part reflecting the score ceiling on the other measures described previously (i.e., SAS scores were consistently lower when the percent implementation was high). By contrast, the relation of TIC and BOQ scores was more consistent across the score distribution, as indicated by the line with very little curvature in Figure 1, with higher scores more likely to be obtained on the TIC.

Limitations and Future Research

The primary limitation of the study is that the measures were collected through an extant database, and therefore, the exact composition of the group completing the measures and level of adherence to administration rules are both unknown. It is likely that except for the SET and SAS, a similar group of individuals completed both pairs of assessments. This possibility may have inflated convergent validity estimates. It also indicates the need for future research to examine how respondent membership (e.g., with vs. without coach, with vs. without administrator) affects behavior during the assessment and scoring, and it would also be helpful to examine the consistency of these responses to objective observations of SWPBIS implementation. Finally, the ceiling effects seen for measures other than the SAS indicate the need to develop tools to assess fidelity of implementation for the highest levels of SWPBIS implementation.

Implications for Research

Because all of the SWPBIS measures are publicly available and familiar to implementers, it is likely that researchers will encounter schools using some combination of these measures in their prospective samples. The correlations and tests of mean differences show that the measures are related to one another and that the total scores can be used similarly to indicate level of implementation, although there appear to be substantive differences in classifications of whether a school is implementing adequately depending on the recommended criterion of the measure. This study provided evidence that the current criteria for some measures (BOQ, SET, and TFI) may be less stringent than others (SAS and TIC). These differences in classification rates have implications for a common research practice for assessing fidelity status across schools using different measures, i.e., using the implementation criteria for each measure (see Table 1) to indicate whether schools are implementing SWPBIS adequately (McIntosh et al., 2013; Nese et al., 2016) because fewer schools may be classified as adequately implementing on the SAS and TIC compared with the other measures. By contrast, the moderate correlations among total scores found in the current study are consistent with a recent study on a separate sample showing that these measures all load onto a single latent factor of Tier 1 SWPBIS fidelity of implementation with strong model fit (Turri et al., 2016). These findings add to the evidence that the total scores from these measures may be used relatively interchangeably (with the caveats described above) for research purposes, but researchers should be cautious about the application of implementation criteria to classify schools as adequately implementing or not when multiple measures are used or compared. Future research should continue to examine the empirical foundations for these criteria against alternate cut points.

Implications for Practice

Practitioners and evaluators may use these findings to see how scores on one measure may relate to scores on other measures, particularly for districts or states that have more variability in fidelity assessment use, or those considering the switch to newer measures such as the TFI. However, it is important to stress that the analyses we conducted are not true equivalence tests due to differences in item content and coverage of critical aspects of SWPBIS implementation, and using these results for score conversions would be inappropriate. The measures are not identical, and each was designed for a specific purpose. The difference between a self-assessment checklist for initial implementation (TIC), a survey and needs assessment for the entire staff (SAS), and an external evaluation (SET) should be obvious to experienced users and evaluators. As such, it would be counterproductive to try to select measures based on how easy it may be to obtain a minimum score for adequate implementation. As described previously, the key benefits from fidelity assessment come from their use as assessments of next steps for implementation, not simply meeting a criterion and then halting further implementation efforts. As a result, practitioners are advised not to rely too heavily on these criteria as precise discriminators between adequate and inadequate SWPBIS implementation. In addition, relying only on self-reported fidelity of implementation for district or state recognition systems seems unwise, as any potential bias toward inflated scores would reduce their utility for action planning. Instead, state and district teams are encouraged to use improvements in student outcomes (e.g., reduced use of suspensions, improved perceptions of school climate, equity in school discipline) as criteria.

Many district and state evaluators have relied on the SET as an objective assessment of implementation, but these results indicate that SET scores are generally higher than scores on the other measures, with a ceiling that may not provide teams with multiple years of suggested action planning items. Instead, those seeking a balance between external evaluation and utility for implementation action planning could consider using measures like the TFI and BOQ with an external coach or technical assistance provider present to ensure that responses are accurate (McIntosh et al., 2017). Such an assessment plan, especially when paired with a periodic (every few years) assessment of all staff perceptions with the SAS, may have the most promise for assessing high-quality implementation.