Additional Evidence for the Reliability and Validity of the Student Risk Screening Scale at the High School Level

Participants

Participants for the 2008–2009 school year were 1,845 students (930 males and 915 females) attending a rural:urban fringe high school in middle Tennessee. The school served students in 9th (28.8%, n = 532), 10th (24.20%, n = 447), 11th (24.53%, n= 453), and 12th (22.47%, n = 415) grades. Students were predominately Caucasian (90.12%, n = 1,642). Approximately 7.14% (n= 127) of students received special education services and 13% of students were classified as economically disadvantaged. There were 1,920 students in the following academic year, with nominal changes in demographic characteristics (see Table 1). The special education and economic demographic data for the 2009–2010 academic year were not yet reported at the time of submission (Tennessee Department of Education, 2009).

OPEN IN VIEWER

Table 1.

Student Demographics.

		2008–2009 (N = 1,845)		2009–2010 (N = 1,920)
Variable	Level	%	n	%	n
Gender	Male	50.41	930	50.47	969
	Female	49.59	915	49.53	951
Ethnicity	Caucasian	90.12	1,642	88.69	1,702
	African American	5.16	94	5.78	111
	Hispanic	3.57	65	4.06	78
	Asian	0.93	17	1.09	21
	Native American	0.16	3	0.16	3
	Other	0.05	1	0.21	4
Grade level	9th	28.80	532	26.88	516
	10th	24.20	447	27.40	526
	11th	24.53	453	23.02	442
	12th	22.47	415	22.71	436
Special education		7.14	127
Economically disadvantaged		13.00

Percentages are based on the number of participants for whom data were collected. There were less than 2% missing data in any one category. The special education and economic demographics were not yet reported at the time of submission (Tennessee Department of Education, 2009).

Procedures

During the 2007–2008 academic year, Liberty High School participated in a year-long training series at Vanderbilt University to build a comprehensive, integrated, three-tiered (CI3T) model of prevention with a focus on positive behavior intervention and support (PBIS). Throughout the academic year, the school’s leadership team utilized faculty input to develop their three-tiered model of prevention. The aims of the plan were to promote positive gains academically, behaviorally, and socially and included components to teach, reinforce, and monitor expectations. The school-site leadership team adopted the SRSS (Drummond, 1994) as a regular school practice, requiring second- and seventh-period teachers to complete the SRSS three times per year (details to follow). The team used SRSS data as well as other data collected as part of regular school practices (e.g., attendance, ODRs, GPA, and course failures) to monitor their plan’s effectiveness and plan supports for students for whom primary prevention efforts were insufficient (see Lane et al., 2009, for examples).

Measures

While multiple data sources were collected by the school to monitor students’ progress, SRSS, office discipline referrals (ODRs), grade point averages (GPAs), and course failures (CFs) were chosen as the primary data sources for this article.

Statistical Analyses

A series of analyses were conducted to determine the reliability of the SRSS. First, to establish internal consistency, alpha coefficients were computed for each of the five time points (2008–2009: Fall, Winter, and Spring; 2009–2010: Winter and Spring) for each rater (Period 2 and Period 7). Data were not aggregated across raters. Thus, separate alpha coefficients were computed for each rater at each of the five administrations. Alpha coefficients > .80 are considered acceptable (Nunnally & Bernstein, 1994). Second, test-retest stability was examined by computing Pearson correlation coefficients between each administration over 2 years for each rater. Again, Period 2 and Period 7 raters’ scores were not aggregated for test-retest stability analyses. Correlations ≥ .50 are considered large (Hopkins, 2002). Third, inter-rater reliability was calculated using Pearson correlation coefficients between Period 2 and Period 7 raters at each of the five time points.

Fourth, we computed a series of one-way, fixed-effects multivariate analyses of variance (MANOVA) to determine the extent to which students scoring in the low-, moderate-, and high-risk categories according to Period 2 and Period 7 raters could be differentiated on behavioral (ODR) and academic (GPA and CF) outcomes over time. Specifically, low-, moderate-, and high-risk groups according to Fall and Winter scores as rated by Period 2 or Period 7 teachers were contrasted on ODRs, GPA, and CF during the 2008–2009 and 2009–2010 academic years. Spring scores (2009) were also analyzed in terms of predicting year-end outcomes in the 2009–2010 academic year. We used SAS to compute each one-way MANOVA using the general linear model, with Wilks’s lambda used for significance testing. Significant MANOVAs were followed by univariate analyses of variance (ANOVAs). Significant ANOVAs were followed by post hoc Tukey-Kramer HSD comparisons with a criterion of p < .05. Namely, SAS automatically adjusts the Tukey test command for unequal sample sizes by performing the Tukey-Kramer test. This post hoc test does not assume homogeneity of variance across groups (the Tukey-Kramer test; SAS Institute). Effect sizes (ES_sm) were computed to determine the magnitude of difference between groups using the pooled standard deviation in the denominator. More specifically, effect sizes were computed to compare the magnitude of differences between two groups: (a) low to moderate risk, (b) low to high risk, and (c) moderate to high risk. For each computation, the differences between means (e.g., M_high – M_low) were divided by the pooled standard deviations for both groups being compared ([SD_high + SD_low]/2) to obtain the effect size.

Fifth, we conducted an additional set of analyses with the initial SRSS ratings to determine the extent to which predictive validity of the SRSS with extant school-wide data could be improved if students were identified as low, moderate, or high status by both raters (Period 2 and Period 7 teachers). Consistent with earlier studies (e.g., Ennis et al., in press; Menzies & Lane, 2010; Oakes et al., 2010), we used regression procedures to determine how well Fall 2008 SRSS scores (in this study, the SRSS categorical variable: low, moderate, or high, X_{SRSScategoricalscore}) as rated by only Period 2, only Period 7, and both Period 2 and Period 7 teachers (in combination) at the onset of the first school year (Fall 2008) predicted students’ year-end (a) ODRs (Y_ODR), (b) GPA (Y_GPA), and (c) course failures (Y_CF) during the 2008–2009 and 2009–2010 academic years. For this set of analyses, in addition to looking at Period 2 and Period 7 raters’ scores (low, moderate, or high) independently, we also explored the utility of both raters. We did not combine Period 2 and Period 7 raters’ scores into an aggregate score. Instead, we analyzed the data to explore common ratings from two perspectives, meaning both teachers rated a student as placing in the low-risk category, moderate-risk category, or high-risk category. Each data analysis step was completed based on the data provided; missing data were not imputed.

Reliability

Internal consistency

Cronbach’s α values exceeded .80 (Nunnally & Bernstein, 1994) for both raters at all five time points except for second period raters at the first time point (Fall 2008, α = .76; see Table 2). Specifically, alpha values for second period raters were .76, .81, .81, .81, and .82, respectively. Alpha values for seventh period raters were .85, .87, .85, .82, and .83, respectively. Overall, these seven items constituting the SRSS showed high internal consistency (see Table 2 for alphas values for deleted variables). The intercorrelations between items on the SRSS and the total score yielded statistically significant results at the p < .0001 level for both raters at all time points over the 2 academic years (see Tables 3 and 4).

OPEN IN VIEWER

Table 2.

Internal Consistency Estimates Periods 2 and 7 Over Time.

		Period 2				Period 7
				Standardized Variables				Standardized Variables
Time	Item	M	SD	r With Total	Alpha	M	SD	r With Total	Alpha
Fall					.76				.85
2008 a	1. Steals	0.01	0.09	.28	.77	0.03	0.21	.55	.84
	2. Lie, cheat, sneak	0.13	0.45	.55	.71	0.14	0.45	.70	.81
	3. Behavior problem	0.14	0.46	.58	.70	0.17	0.52	.63	.82
	4. Peer rejection	0.07	0.35	.37	.75	0.09	0.39	.49	.84
	5. Low academic achievement	0.28	0.67	.47	.73	0.33	0.74	.52	.84
	6. Negative attitude	0.14	0.48	.60	.70	0.16	0.51	.69	.81
	7. Aggressive behavior	0.03	0.21	.48	.73	0.05	0.30	.67	.82
Winter					.81				.87
2008 b	1. Steals	0.08	0.38	.49	.80	0.08	0.37	.66	.85
	2. Lie, cheat, sneak	0.23	0.57	.65	.77	0.23	0.58	.77	.83
	3. Behavior problem	0.15	0.46	.61	.78	0.18	0.53	.67	.85
	4. Peer rejection	0.11	0.42	.41	.81	0.13	0.39	.47	.87
	5. Low academic achievement	0.42	0.80	.53	.79	0.42	0.84	.58	.86
	6. Negative attitude	0.17	0.51	.64	.77	0.21	0.56	.75	.84
	7. Aggressive behavior	0.04	0.25	.49	.80	0.05	0.31	.62	.85
Spring					.81				.85
2009 c	1. Steals	0.06	0.28	.46	.80	0.09	0.38	.67	.82
	2. Lie, cheat, sneak	0.24	0.57	.67	.76	0.25	0.58	.74	.81
	3. Behavior problem	0.17	0.49	.69	.76	0.20	0.54	.69	.82
	4. Peer rejection	0.12	0.41	.41	.81	0.10	o.41	.45	.86
	5. Low academic achievement	0.42	0.82	.52	.79	0.42	0.82	.51	.85
	6. Negative attitude	0.18	0.54	.64	.77	0.20	0.53	.66	.82
	7. Aggressive behavior	0.03	0.20	.45	.80	0.05	0.25	.60	.83
Winter					.81				.82
2009 d	1. Steals	0.39	0.21	.36	.81	0.02	0.16	.36	.83
	2. Lie, cheat, sneak	0.13	0.45	.64	.76	0.09	0.36	.70	.77
	3. Behavior problem	0.09	0.38	.71	.75	0.11	0.43	.67	.78
	4. Peer rejection	0.06	0.31	.43	.80	0.05	0.30	.42	.82
	5. Low academic achievement	0.18	0.59	.50	.79	0.18	0.62	.47	.81
	6. Negative attitude	0.10	0.41	.62	.77	0.08	0.37	.73	.77
	7. Aggressive behavior	0.02	0.17	.54	.78	0.02	0.21	.59	.79
Spring					.82				.83
2010 e	1. Steals	0.07	0.31	.43	.82	0.07	0.34	.43	.83
	2. Lie, cheat, sneak	0.31	0.66	.72	.77	0.31	0.66	.71	.78
	3. Behavior problem	0.20	0.56	.68	.78	0.29	0.69	.70	.79
	4. Peer rejection	0.14	0.47	.41	.83	0.15	0.50	.48	.82
	5. Low academic achievement	0.50	0.88	.60	.79	0.47	0.90	.55	.81
	6. Negative attitude	0.27	0.66	.74	.77	0.24	0.62	.65	.79
	7. Aggressive behavior	0.05	0.25	.41	.83	0.08	0.35	.54	.81

Under the column labeled “Alpha,” the first coefficient is the overall alpha coefficient for the given rater and time point. The remaining alpha coefficients are the estimated alpha coefficient if the items were removed. The column labeled “r With Total” is the corrected item-total correlation.

a.

Period 2: n = 1,762; Period 7: n = 1,524.

b.

Period 2: n = 1,692; Period 7: n = 1,752.

c.

Period 2: n = 1,508; Period 7: n = 1,518.

d.

Period 2: n = 1,779; Period 7: n = 1,734.

e.

Period 2: n = 1,601; Period 7: n = 1,507.

OPEN IN VIEWER

Table 3.

Intercorrelations Period 2.

Time	Item	1	2	3	4	5	6	7
Fall	1. Steals	1.00
2008	2. Lie, cheat, sneak	0.30	1.00
n = 1,762	3. Behavior problem	0.17	0.50	1.00
	4. Peer rejection	0.16	0.18	0.24	1.00
	5. Low academic achievement	0.15	0.35	0.37	0.32	1.00
	6. Negative attitude	0.14	0.47	0.51	0.26	0.45	1.00
	7. Aggressive behavior	0.22	0.30	0.41	0.29	0.19	0.46	1.00
Winter	1. Steals	1.00
2008	2. Lie, cheat, sneak	0.62	1.00
n = 1,692	3. Behavior problem	0.36	0.48	1.00
	4. Peer rejection	0.17	0.26	0.30	1.00
	5. Low academic achievement	0.34	0.48	0.42	0.33	1.00
	6. Negative attitude	0.38	0.50	0.50	0.34	0.42	1.00
	7. Aggressive behavior	0.21	0.31	0.45	0.37	0.24	0.49	1.00
Spring	1. Steals	1.00
2009	2. Lie, cheat, sneak	0.51	1.00
n = 1,508	3. Behavior problem	0.37	0.58	1.00
	4. Peer rejection	0.20	0.28	0.35	1.00
	5. Low academic achievement	0.25	0.46	0.44	0.38	1.00
	6. Negative attitude	0.35	0.54	0.58	0.31	0.47	1.00
	7. Aggressive behavior	0.25	0.36	0.47	0.26	0.20	0.38	1.00
Winter	1. Steals	1.00
2009	2. Lie, cheat, sneak	0.58	1.00
n = 1,779	3. Behavior problem	0.30	0.55	1.00
	4. Peer rejection	0.15	0.28	0.38	1.00
	5. Low academic achievement	0.15	0.39	0.45	0.30	1.00
	6. Negative attitude	0.18	0.46	0.60	0.32	0.48	1.00
	7. Aggressive behavior	0.17	0.34	0.55	0.38	0.30	0.48	1.00
Spring	1. Steals	1.00
2010	2. Lie, cheat, sneak	0.50	1.00
n = 1,601	3. Behavior problem	0.34	0.65	1.00
	4. Peer rejection	0.23	0.29	0.30	1.00
	5. Low academic achievement	0.28	0.53	0.48	0.39	1.00
	6. Negative attitude	0.38	0.68	0.65	0.34	0.59	1.00
	7. Aggressive behavior	0.15	0.31	0.40	0.23	0.27	0.41	1.00

All significance levels are p < .0001.

OPEN IN VIEWER

Table 4.

Intercorrelations Period 7.

Time	Item	1	2	3	4	5	6	7
Fall	1. Steals	1.00
2008	2. Lie, cheat, sneak	0.50	1.00
n = 1,524	3. Behavior problem	0.39	0.55	1.00
	4. Peer rejection	0.32	0.41	0.32	1.00
	5. Low academic achievement	0.28	0.46	0.41	0.35	1.00
	6. Negative attitude	0.39	0.59	0.59	0.40	0.46	1.00
	7. Aggressive behavior	0.55	0.53	0.50	0.42	0.36	0.56	1.00
Winter	1. Steals	1.00
2008	2. Lie, cheat, sneak	0.65	1.00
n = 1,752	3. Behavior problem	0.51	0.62	1.00
	4. Peer rejection	0.30	0.41	0.30	1.00
	5. Low academic achievement	0.39	0.57	0.46	0.40	1.00
	6. Negative attitude	0.54	0.65	0.63	0.46	0.51	1.00
	7. Aggressive behavior	0.59	0.52	0.50	0.33	0.32	0.54	1.00
Spring	1. Steals	1.00
2009	2. Lie, cheat, sneak	0.69	1.00
n = 1,518	3. Behavior problem	0.52	0.61	1.00
	4. Peer rejection	0.36	0.32	0.32	1.00
	5. Low academic achievement	0.32	0.49	0.43	0.31	1.00
	6. Negative attitude	0.44	0.57	0.64	0.36	0.47	1.00
	7. Aggressive behavior	0.59	0.50	0.48	0.37	0.29	0.42	1.00
Winter	1. Steals	1.00
2009	2. Lie, cheat, sneak	0.50	1.00
n = 1,734	3. Behavior problem	0.36	0.58	1.00
	4. Peer rejection	0.06	0.30	0.35	1.00
	5. Low academic achievement	0.16	0.41	0.41	0.27	1.00
	6. Negative attitude	0.24	0.59	0.59	0.46	0.48	1.00
	7. Aggressive behavior	0.27	0.44	0.47	0.41	0.28	0.60	1.00
Spring	1. Steals	1.00
2010	2. Lie, cheat, sneak	0.46	1.00
n = 1,507	3. Behavior problem	0.36	0.63	1.00
	4. Peer rejection	0.25	0.35	0.37	1.00
	5. Low academic achievement	0.29	0.50	0.44	0.36	1.00
	6. Negative attitude	0.25	0.58	0.60	0.38	0.48	1.00
	7. Aggressive behavior	0.27	0.42	0.51	0.37	0.27	0.46	1.00

All significance levels are p < .0001.

Test-retest stability

Test-retest stability was computed for Period 2 and Period 7 teacher ratings provided at each of the five screening points, with durations between screenings ranging from 10 to 82 weeks (see Table 5). Coefficients of the resulting 20 comparisons ranged from .23 to .75, with all correlations statistically significant (p < .0001; see Table 5 for test-retest stability estimates). Test-retest stability estimates for Period 2 and Period 7 teachers ratings were comparable at each administration (e.g., Fall 2008 to Winter 2009: Period 2 r = .28 and Period 7 r = .29). The Spring 2009 to Spring 2010 comparisons were exact for both raters, meaning they both yielded the same correlation coefficients (r = .35). Correlations within a given year were higher (range: .47 to .75) than across academic years (range: .23 to .35).

OPEN IN VIEWER

Table 5.

Test/Retest Stability.

Time Frame Comparison			Period 2		Period 7
Time 1	Time 2	Weeks	n	Correlation	n	Correlation
Fall 2008	Winter 2008	10	1,685	0.60	1,500	0.68
	Spring 2009	29	1,461	0.51	1,265	0.58
	Winter 2009	67	1,185	0.28	982	0.29
	Spring 2010	82	1,102	0.27	865	0.29
Winter 2008	Spring2009	18	1,431	0.64	1,484	0.75
	Winter 2009	51	1,134	0.28	1,144	0.24
	Spring 2010	71	1,053	0.29	1,017	0.27
Spring 2009	Winter 2009	38	1,057	0.23	1,017	0.26
	Spring 2010	53	980	0.35	905	0.35
Winter 2009	Spring 2010	14	1,629	0.47	1,540	0.50

All significance levels are p < .0001.

Predictive Validity

In Table 6, we report outcomes of analyses conducted to determine the extent to which (a) Fall 2008 and (b) Winter 2008 SRSS scores as rated by Period 2 and Period 7 teachers predicted year-end ODR, GPA, and CF values for the 2008–2009 academic year. We also report analyses conducted to determine the extent to which (a) Fall 2008, (b) Winter 2008, (c) Spring 2009, and (d) Winter 2009 SRSS scores as rated by Period 2 and Period 7 teachers predicted year-end ODRs, GPAs, and CFs values for the 2009–2010 academic year.

OPEN IN VIEWER

Table 6.

Behavioral and Academic Characteristics of Risk Groups Using the SRSS.

			Risk
SRSS			Low	Moderate	High	Significance	Effect Size
Time Point	Rater	Variable	M (SD)	M (SD)	M (SD)	Testing	L:M	L:H	M:H
Predicting Year 1 2008-2009 Outcomes
Fall	Period 2		(n = 1592)	(n = 108)	(n = 18)
2008		ODR	0.0066 (0.01)	0.0213 (0.02)	0.0302 (0.04)	L < M, H; M = H	−0.98	−0.94	−0.30
		GPA	3.18 (0.78)	2.35 (0.89)	2.10 (0.97)	L > M, H; M = H	0.99	1.23	0.27
		CF	1.38 (2.28)	4.46 (5.46)	5.11 (5.99)	L < M, H; M = H	−0.80	−0.90	−0.11
	Period 7		(n = 1357)	(n = 100)	(n = 30)
		ODR	0.0062 (0.01)	0.0228 (0.02)	0.0297 (0.03)	L < M, H; M = H	−1.11	−1.18	−0.28
		GPA	3.20 (0.76)	2.24 (0.84)	2.34 (0.98)	L > M, H; M = H	1.20	0.99	−0.11
		CF	1.31 (2.77)	5.07 (5.40)	4.90 (4.84)	L < M, H; M = H	−0.92	−0.94	0.03
Winter	Period 2		(n = 1454)	(n = 162)	(n = 38)
2008		ODR	0.0059 (0.01)	0.0199 (0.03)	0.0293 (0.03)	L < M < H	−0.70	−1.17	−0.31
		GPA	3.25 (0.74)	2.27 (0.76)	1.96 (1.01)	L > M, H; M = H	1.31	1.47	0.35
		CF	1.13 (2.50)	4.64 (4.45)	7.03 (7.06)	L < M < H	−1.01	−1.23	−0.42
	Period 7		(n = 1522)	(n = 154)	(n = 59)
		ODR	0.0059 (0.01)	0.0200 (0.02)	0.0251 (0.03)	L < M, H; M = H	−0.94	−0.96	−0.20
		GPA	3.25 (0.73)	2.20 (0.79)	2.04 (0.91)	L > M, H; M = H	1.38	1.48	0.19
		CF	1.11 (2.42)	4.81 (5.09)	6.08 (5.41)	L < M < H	−0.99	−1.27	−0.24
Predicting Year 2 2009-2010 Outcomes
Fall	Period 2		(n = 1,149)	(n = 77)	(n = 13)
2008		ODR	0.0088 (0.02)	0.0393 (0.05)	0.0387 (0.05)	L < M, H; M = H	−0.87	−0.85	0.01
		GPA	3.21 (0.78)	2.32 (0.81)	2.25 (1.00)	L > M, H; M = H	1.12	1.08	0.08
		CF	1.27 (2.75)	3.38 (4.27)	2.77 (3.30)	L < M; L = H; M = H	−0.60	−0.50	0.16
	Period 7		(n = 976)	(n = 71)	(n = 22)
		ODR	0.0091 (0.02)	0.0299 (0.05)	0.0312 (0.05)	L < M, H; M = H	−0.59	−0.63	−0.03
		GPA	3.22 (0.78)	2.23 (0.81)	2.42 (1.04)	L > M, H; M = H	1.25	0.88	−0.21
		CF	1.23 (2.57)	3.83 (5.05)	3.68 (4.52)	L < M, H; M = H	−0.68	−0.69	0.03
Winter	Period 2		(n = 1,030)	(n = 123)	(n = 31)
2008		ODR	0.0080 (0.02)	0.0303 (0.05)	0.0378 (0.06)	L < M, H; M = H	−0.64	−0.75	−0.14
		GPA	3.27 (0.75)	2.30 (0.75)	2.08 (0.99)	L > M, H; M = H	1.29	1.37	0.25
		CF	1.10 (2.53)	3.46 (3.95)	4.45 (5.43)	L < M, H; M= H	−0.73	−0.84	−0.21
	Period 7		(n = 1,091)	(n = 109)	(n = 43)
		ODR	0.0075 (0.02)	0.0359 (0.05)	0.0368 (0.05)	L < M, H; M = H	−0.81	−0.84	−0.02
		GPA	3.27 (0.73)	2.24 (0.79)	2.16 (0.81)	L > M, H; M = H	1.36	1.44	0.10
		CF	1.11 (2.48)	3.64 (4.42)	3.33 (3.75)	L < M, H; M= H	−0.73	−0.71	0.08
Spring	Period 2		(n = 952)	(n = 124)	(n = 25)
2009		ODR	0.0079 (0.02)	0.0305 (0.04)	0.0455 (0.06)	L < M < H	−0.75	−0.94	−0.30
		GPA	3.30 (0.73)	2.17 (0.72)	1.79 (0.85)	L > M > H	1.56	1.91	0.48
		CF	1.08 (2.50)	3.76 (4.03)	5.04 (5.84)	L < M, H; M= H	−0.82	−0.95	−0.26
	Period 7		(n = 954)	(n = 115)	(n = 37)
		ODR	0.0074 (0.02)	0.0300 (0.04)	0.0496 (0.06)	L < M < H	−0.75	−0.06	−0.39
		GPA	3.22 (0.72)	2.43 (0.75)	1.76 (0.84)	L > M > H	1.07	1.87	0.84
		CF	1.10 (2.40)	3.10 (3.79)	5.62 (6.05)	L < M < H	−0.65	−1.07	−0.51
Winter	Period 2		(n = 1,656)	(n = 99)	(n = 20)
2009		ODR	0.0094 (0.02)	0.0273 (0.03)	0.0452 (0.06)	L < M < H	−0.72	−0.90	−0.40
		GPA	3.16 (0.81)	2.28 (0.90)	2.03 (0.88)	L > M, H; M = H	1.03	1.34	0.28
		CF	1.36 (2.87)	4.36 (5.01)	6.30 (6.13)	L < M < H	−0.76	−1.10	−0.35
	Period 7		(n = 1,640)	(n = 72)	(n = 18)
		ODR	0.0091 (0.02)	0.0355 (0.04)	0.0396 (0.05)	L < M, H; M = H	−0.88	−0.87	−0.09
		GPA	3.17 (0.81)	2.10 (0.87)	1.94 (0.98)	L > M, H; M = H	1.27	1.37	0.17
		CF	1.36 (2.86)	4.99 (5.12)	6.83 (6.82)	L < M < H	−0.91	−1.13	−0.31

ODR = office discipline referrals; GPA = grade point average; CF = course failures; L = low risk on the SRSS (0–3); M = moderate risk (4–8); H = high risk (9–21).

Multiple raters: Are they necessary?

In Table 7, we provided results of a series of regression analyses. Results indicated Fall 2008 SRSS categorical risk status scores according to only Period 2, only Period 7, and Period 2 and 7 teacher ratings combined (e.g., both teachers indicating low, moderate, or high risk) predicted ODR rates, GPA, and course failures for the 2008–2009 and 2009–2010 academic year. The proportion of explained variance was consistently low, regardless of raters, in each instance. Yet the proportion of explained variance was highest with Period 7 teachers as the raters. Requiring two teachers to indicate a cause for concern did not improve the predictive validity of the SRSS. This indicated that students with higher levels of risk, as measured by the SRSS—particularly as rated by Period 7 raters—just 6 weeks after the school year began, were likely to have higher ODR rates, lower GPAs, and more course failures relative to students with lower levels of risk.

OPEN IN VIEWER

Table 7.

Regression Outcomes.

			Predicting 2008–2009 Academic Year			Predicting 2009–2010 Academic Year
Time Point	Outcome Measure	Rater	F Value	R 2	Beta Weight	F Value	R 2	Beta Weight
Fall 2008	ODR	Period 2	F(1, 1757) = 112.26, p < .0001	.06	.01	F(1, 1243) = 108.75, p < .0001	.08	0.02
		Period 7	F(1, 1521) = 152.58, p < .0001	.09	.01	F(1, 1071) = 50.19, p < .0001	.04	.02
		Period 2 and 7	F(1, 1327) = 116.10, p < .0001	.08	.02	F(1, 942) = 44.25, p < .0001	.04	.03
	GPA	Period 2	F(1, 1718) = 141.92, p < .0001	.07	-.72	F(1, 1240) = 99.83, p < .0001	.07	-.72
		Period 7	F(1, 1068) = 95.70, p < .0001	.08	-.65	F(1, 1069) = 97.83, p < .0001	.08	-.65
		Period 2 and 7	F(1, 1306) = 49.12, p < .0001	.04	-.86	F(1, 941) = 30.65, p < .0001	.03	-.82
	CF	Period 2	F(1, 1748) = 111.88, p < .0001	.06	2.46	F(1, 1246) = 33.63, p < .0001	.03	.26
		Period 7	F(1, 1511) = 145.95, p < .0001	.09	2.55	F(1, 1073) = 57.79, p < .0001	.05	1.29
		Period 2 and 7	F(1, 1322) = 87.79, p < .0001	.06	4.16	F(1, 944) = 21.35, p < .0001	.02	2.21

The SRSS: A Reliable Measure at the High School Level

In the current study, internal consistency estimates ranged from .76 to .82 for Period 2 teacher ratings and from .82 to .87 for Period 7 teacher ratings. Thus, internal consistency was strong for both raters at each of the five administrations. These findings were highly consistent with the internal consistency estimates reported by Lane et al. (2008), which ranged from .78 to .86 for instructional raters and from .79 to .84 for noninstructional (e.g., study hall) raters. However, in the current study, ratings occurring after lunch had higher internal consistency estimates than those occurring in the morning. The time of day during which students were evaluated (e.g., before or after school) was not examined in the previous study. The exploration of time of day during which ratings occurred extends this line of inquiry and confirms the established internal consistency of the SRSS as used in the high school setting.

Next, test-retest stability estimates suggest Period 2 and Period 7 teacher ratings were highly similar, and in some cases exact (Spring 2009 to Spring 2010, r = .35). As expected, correlation coefficients for short test-retest intervals within a given year (e.g., 10 weeks to 29 weeks) were higher (r ranging from .47 to .75, with all but one above .50; Hopkins, 2002) than correlation coefficients for longer test-retest intervals across academic years (e.g., 38 weeks to 82 weeks; r ranging from .23 to .35). However, all test-retest stability estimates were statistically significant (p < .0001), even at the 82-week interval. Collectively, these findings again parallel results of the Lane et al. (2008) study, which reported (a) comparable test-retest stability estimates for both raters and (b) higher reliability estimates within academic years (instructional raters: r ranging from .44 to .71; noninstructional raters: r ranging from .39 to .73) than test-retest stability estimates across academic years (instructional raters: r ranging from. 22 to .30; noninstructional raters: r ranging from .30 to .43), with all statistically significant. Findings from these analyses confirm the stability of SRSS scores over time, adding evidence to the reliability of the SRSS for use as a benchmarking tool for behavioral performance at the high school level.

In addition to being internally consistent and stable across 2 academic years, SRSS scores were also identified as consistent across raters. Namely, inter-rater reliability results suggest SRSS scores were statistically significant (p < .0001) at each of the five screening time points, with correlation coefficients generally increasing within each academic year from the initial reliability estimates. For example, inter-rater reliability estimates increased from .28 at the initial Fall 2008 administration to .40 in Winter 2008 and .39 in Spring 2009. Similarly, the next year the inter-rater reliability estimate increased from .29 in Winter 2009 to .43 in Spring 2010. Again, these findings were comparable to the Lane et al. (2008) study, which reported inter-rater reliability estimates ranging from .35 to .46 in the first academic year and .19 to .50 in the second academic year. Thus, additional evidence is established for inter-rater reliability of the SRSS when used at the high school level.

Collectively, these analyses provide additional evidence to support the reliability of the SRSS for use in a rural fringe high school, as evidenced by high internal consistency estimates, test-retest stability up to 82 weeks, and inter-relater reliability. In addition, there is also evidence to support the validity of the SRSS in this context.

The SRSS: A Valid Measure at the High School Level

In predicting Year 1 outcomes, Fall scores were able to differentiate the low-risk from the moderate- and high-risk groups on ODR, GPA, and CF, with the low-risk group having significantly more ODRs and CFs as well as higher GPAs than the moderate- or high-risk groups. However, it was not possible to differentiate between the moderate- and high-risk groups on these variables. Using the Winter score, predictive validity improved as expected given (a) teachers had an increased amount of time to interact with the students, thereby learning more about their academic and behavioral performance patterns, and (b) the interval between the Winter screening scores and the year-end scores was smaller than the interval between Fall screening scores and year-end scores. In this case, both second- and seventh-period teacher ratings were able to differentiate between all three groups in terms of course failures, with each risk group having a statistically higher number of course failures. Period 2 teacher ratings could also differentiate the low-, moderate-, and high-risk groups in terms of ODRS; however, this was not true for Period 7 teacher ratings. When predicting GPA, Fall and Winter scores from Period 2 and Period 7 raters yielded similar results: Low-risk groups could be differentiated from moderate- and high-risk groups, but not between the moderate- and high-risk groups. The lack of distinction between the moderate- and high-risk groups is an important finding, suggesting any level of risk is cause for concern and warrants further consideration, a point we will address further in terms of educational implications.

Comparable findings were also noted when examining long-term predictive validity (across 2 academic years) of the initial Fall 2008 ratings in predicting Year 2 outcomes: The two risk groups (moderate and high) could not be differentiated. These findings were consistent with Lane et al. (2008), who also reported significant differences between the low-risk group relative to both the moderate- and high-risk groups, with the low-risk group having fewer ODRs and higher GPAs. Yet, as in the current study, the moderate- and high-risk groups could not be differentiated on these variables. However, it should be noted that the Lane, Kalberg et al. study only explored the predictive validity of the initial Fall ratings (Time 1) to Year 2 performance outcomes. Short-term predictive validity (within a single academic year) was not explored. The current study extends this line of inquiry by exploring short-term as well as long-term predictive validity.

In terms of long-term predictive validity, SRSS scores at the end of the first year provided by Period 7 teachers were able to differentiate all three groups of students (low-, moderate-, and high-risk status) on ODR, GPA, and CF. Specifically, according to these ratings, students with low-risk status earned significantly more ODRs, failed more classes, and had a lower GPA than students in the moderate-risk group, who earned more ODRs, failed more classes, and had a lower GPA than students in the high-risk group during the following academic year.

Although preliminary in nature, collective findings of the predictive validity studies suggest that for most time points, the SRSS—regardless of rater—distinguishes between students in the low-risk group from students in the moderate- and high-risk groups on important behavioral (ODR) and academic (GPA and CF) outcomes. Furthermore, given more time to interact with students, teachers’ Winter ratings improve in terms of predictive validity, particularly with respect to course failures. End-of-the-year scores (particularly those scores of teachers working with students later in the day) are highly accurate in predicting student outcomes in terms of ODR, GPA, and CF 1 year later.

In terms of educational implications, we offer three suggestions. First, when looking at initial Fall ratings, we encourage school-site leadership teams to consider supporting students in the moderate- and high-risk groups at the earliest possible juncture given the results of this study suggest students with either moderate- or high-risk status tend to experience negative outcomes. This additional support is particularly necessary if students’ risk status has not improved by the second behavior screening assessment (see Lane et al., 2008, for examples).We recognize it may be challenging to introduce secondary and tertiary levels of preventions for students in both the moderate- and high-risk ranges due to resource issues (e.g., time, money, personnel) as well as make the shift in thinking associated with general education teachers assuming the role of interventionists for students for whom primary prevention is insufficient. It may be helpful for teachers to conceptualize these additional supports as simply differentiating instruction in academic, behavioral, and social domains given that teachers are often quite skilled at differentiating academic instruction. Yet regardless of the challenges, given that outcomes are negative for both risk groups, as evidenced by lower GPAs, more course failures, and higher rates of ODRs, it is essential to intervene to prevent these deleterious outcomes from manifesting. However, as we will discuss later, it is important that these suggestions be interpreted with caution due to sample size issues. Specifically, there were very few students placing into the high-risk categories, which may have reduced the statistical power to detect differences between the moderate- and high-risk groups.

Second, given the collective findings of predictive validity analyses, it would appear that Spring screening data—particularly scores provided by teachers’ evaluations of students with whom they work later in the school day—hold particular promise for predicting students’ behavioral and academic performance during the following year. One highly practical application of this information is to use data to provide supports ideally over the course of the summer or at the onset of the following academic year. For example, it would be possible to analyze Spring screening data in conjunction with other data collected as part of regular school practices to identify students who may benefit from secondary or tertiary supports upon return from school during the following academic year. The school-site leadership team at the school presented in this article did just this, establishing a number of secondary and tertiary supports for returning students for whom primary prevention efforts were insufficient. One secondary support included a mentoring program for 10th-, 11th-, and 12th-grade students. Rising 10th-, 11th-, and 12th-grade students rated as moderate or high risk by the second- or seventh-period teachers during Spring 2010, who also earned a cumulative GPA ≤ 2.75 and more than two ODRs during the 2009–2010 school year, were invited to participate in this year-long support. Each student interested in participating was assigned an individual teacher mentor with whom he or she met each week. The mentors were volunteer teachers who were given specific parameters for working with their student; the goal of the program was to focus on academic achievement, character development, problem-solving skills, relationships with adults and peers, and school attendance.

Third, when considering issues of teacher time needed to complete screenings, if a school-site leadership team decides to involve only one rater, we recommend having teachers evaluate students with whom they teach later in the instructional day. Results of logistic regression procedures clearly indicate Period 7 teacher ratings were actually more predictive (although by a nominal amount) than Period 2 teacher ratings and Period 2 and Period 7 teacher ratings combined. It may be that Period 7 teachers were privy to behaviors that could be characterized as more colorful or lively behavior students tend to exhibit after lunch toward the end of the school day. Liberty’s school-site team was concerned that one rater may be insufficient in detecting students and perhaps students need to be identified as having a concern according to more than one teacher before support is offered. Results of this study indicate this is not the case. Ratings of students’ behavior later in the course of the school day are sufficient to distinguish between which students will—and will not—earn more ODRs, earn lower GPAs, and fail more classes.

Yet we strongly encourage each of these recommendations and considerations to be considered with caution in light of the limitations discussed in the following section.

Limitations and Future Directions

The findings of this study replicate and extend this line of inquiry, suggesting the SRSS may be an effective tool in screening students at-risk for antisocial behaviors at the high school level. However, the results must be interpreted in light of the following limitations. First, this study was only conducted in one moderate-to-large, rural fringe high school whose student body was predominately Caucasian. To establish generalizability, additional studies at the high school level must be conducted in more diverse schools, in other locales (e.g., suburban and urban areas), and in other geographic regions.

Second, as in the Lane et al. (2008) study, there were a relatively small number of students scoring in the high-risk category. For example, in Fall 2008, Period 2 teachers identified only 18 students in this high-risk category. The number for students scoring in the high-risk category varied by rater and time point, with numbers ranging from 18 (Fall 2008, Period 2 rater; Winter 2009, Period 7 rater) to 59 (Winter 2008, Period 7 rater). Given the low sample sizes for this risk category, definitive conclusions cannot be drawn with respect to the predictive validity in discriminating between risk groups. Additional replication is necessary to establish the generalizability of these findings. It is possible that this lack of differentiation between moderate- and high-risk groups may indicate the need for either including additional items to the current SRSS measure or a more sensitive measure for use at the high school level. However, we strongly recommend continuing this line of inquiry to explore the utility of the current instrument at the high school level with additional schools and greater numbers of students in the high-risk category before looking toward another tool. As mentioned previously, it is quite possible there may not have been sufficient power to detect differences between the moderate- and high-risk categories.

Third, as explained in the Method section, there were challenges associated with data collection, particularly during the 2009–2010 school year. Namely, the Fall 2009 screening data were not able to be analyzed given the first item (steals) was inadvertently omitted from the screening tool. Also, due to changes in the ODR data collection systems, Fall 2009 Quarter 1 ODR data were not collected and recording consistently. Therefore, the 2009–2010 ODR rates reflect only Quarters 2, 3, and 4 data rather than the entire academic year, as was reported for the 2008–2009 academic year. Consequently, it was not possible to examine changes in the rates of ODRs earned across academic years. Given the challenges with data collection noted in this study, we recommend future inquiry to explore methods of improving data collection at the high school level. In our own work, we have been working to establish new systems and structures for educating K-12 leadership teams on the importance and value of consistent, accurate data monitoring. We have also been working directly with superintendents and district-level data management teams to establish reliable, valid data collection systems to ensure the accuracy of the information collected, with the goal of providing precise answers to important questions (e.g., Which students are likely to earn office discipline referrals this year? Which students are in danger of failing courses?) and inform intervention efforts.

Fourth, although the school-site leadership team elected to institutionalize behavior screenings as part of regular school practices, some teachers were resistant and required multiple prompts from the team to complete the SRSS as requested. Future research is needed to explore teacher perceptions of the screening tools, with particular attention to issues of feasibility and utility. The field needs additional information on the bridges and barriers to implementing school-wide screening procedures at the high school level (Lane et al., 2009).

Fifth, this article presented psychometric evidence of the reliability and validity of the SRSS. Yet certain measurement issues such as dependency within the current sample and reliability of the existing cut scores should be considered when interpreting the results. Unlike traditional tests of individual differences, the SRSS respondent is the teacher, not the student. In addition to student characteristics, scores may reflect rater characteristics as well, possibly inflating internal-consistency reliability. Moreover, because teachers are rating all students in their Period 2 and Period 7 classes (in contrast to a design in which students provide independently completed self-report ratings), ratings within each class are not independent. It is possible that any rater bias would tend to increase the alpha coefficients. However, the precise impact of the nesting factor is unknown. In the future, multi-rater studies could determine whether the variance due to rater is negligible or large and whether differences among raters influence reliability. In the current study, we were analyzing data collected as part of regular school practices as part of a comprehensive, integrated, three-tiered model of prevention. The school used these data to (a) monitor the overall level of risk evident in a building over time and (b) identify students who might benefit from additional supports in the form of secondary (Tier 2) and tertiary (Tier 3) supports (see Lane et al., 2009). Consequently, these measurement issues could not be addressed. However, we do think the issues of dependency as well as reliability of the existing cutoff scores are important considerations warranting future inquiry. Namely, the intent of this article was to extend this line of inquiry at the high school level by presenting standard psychometric analyses of the SRSS implemented in one high and not to explore the accuracy of the existing cutoff scores for the SRSS as established by Drummond (1994). We encourage more sophisticated studies examining (a) reliability estimates for the scores analyzed (e.g., low-, moderate-, and high-risk categories) and (b) samples collected to avoid the nested nature of the data currently analyzed (e.g., psychometric studies in which a large number of teachers across multiple geographic regions randomly select and rate just one student in their class to allow for independent ratings). Clearly, such large-scale investigations would warrant significant grant funding. Yet given the costs associated with not supporting students with antisocial behavior tendencies, the investment is more than justified (Kauffman & Brigham, 2009; Wagner et al., 2006; Walker et al., 2004).

Finally, we also remind the reader that although Liberty High School conducted assessments—including the SRSS behavioral screening tool—as part of regular school practices, it did receive technical assistance in analyzing its data and identifying supports for students for whom primary prevention efforts were insufficient. As such, future research will need to examine the degree to which school-site leadership teams can design, implement, and evaluate such integrated programs of support in isolation from university partnership or other sources of support.