Abstract
We report findings of a validation study comparing two screening tools: the Student Risk Screening Scale–Internalizing and Externalizing (SRSS-IE, an adapted version of the Student Risk Screening Scale) and the Social Skills Improvement System–Performance Screening Guide (SSiS-PSG). Participants included 458 kindergarten through fifth-grade elementary students from one school in a southeastern state. Correlation coefficients indicated statistically significant, inverse relations between SRSS-IE scores (SRSS-IE 12 [total score], SRSS-E7 [subscale score which includes original seven items constituting the SRSS], and SRSS-I5 [subscale scores of the five items to address internalizing behaviors]) and Prosocial Behavior, Motivation to Learn, Reading Skills, and Math Skills subscale scores of the SSiS-PSG. Analysis of receiver operating characteristics (ROC) curves contrasting students with significant difficulty versus adequate progress suggested the SRSS-IE12 is more accurate for detecting Prosocial Behavior (area under the curve [AUC] = .972) and Motivation to Learn (AUC = .904) compared with Math (AUC = .817) and Reading Skills (AUC = .805) as measured by the SSiS-PSG. Educational implications, limitations, and future directions are offered.
Keywords
Many schools in the United States are constructing multi-tiered systems of support in an effort to become more efficient and effective in (a) preventing the development of learning and behavioral problems by offering primary prevention (Tier 1) efforts to all students and (b) identifying and assisting students with existing concerns, connecting these students to relevant secondary (Tier 2) and/or tertiary (Tier 3) supports according to student need (Lane, Menzies, Oakes, & Kalberg, 2012). There are different models, with some focusing mainly on academic performance as in response-to-intervention (RTI; Fuchs & Fuchs, 2006) models, others focusing mostly on behavioral and social performance as in positive behavior intervention and support (PBIS; Sugai & Horner, 2009) models, and others addressing academic, behavioral, and social domains as in comprehensive, integrated, three-tiered (CI3T; Lane, Menzies, Kalberg, & Oakes, 2012; McIntosh, Chard, Boland, & Horner, 2006) models of prevention.
In each model, there are typically three levels of prevention beginning with primary prevention efforts designed for all students attending a given school. For example, all students would access instruction addressing Common Core State Standards (e.g., National Governors Association Center for Best Practices [NGA Center] & the Council of Chief State School Officers [CCSSO], 2010); learn schoolwide expectations for behavior, including opportunities to practice and receive reinforcement for meeting expectations; and participate in a schoolwide violence prevention plan such as Second Step Violence Prevention (Sprague et al., 2001). Approximately 80% of the student body is anticipated to respond to these global prevention efforts, with secondary (e.g., small group and low intensity) and tertiary (e.g., individualized and high-intensity) supports in place to assist students for whom primary prevention efforts are insufficient in academic, behavior, and/or social domains. In other words, nonresponsiveness is expected. The key to these models is accurate detection of students who require assistance beyond primary prevention efforts to neither overlook students in need of additional support nor expend valuable resources (e.g., personnel time and money) unnecessarily.
While schools more often employ academic screening tools (e.g., AIMSweb® Reading–curriculum-based measure [CBM] probes; Pearson Education, 2008) to benchmark student performance, school-site and district-level leadership teams are now exploring the use of behavior screening tools to inform their decision making. For example, behavior screening tools can be used to accurately detect students with externalizing (e.g., aggressive, noncompliant) and internalizing (e.g., excessively shy, anxious, depressed) behavior patterns, with the latter group often overlooked if school systems rely on more traditional approaches for detecting students with behavior challenges (e.g., examining office discipline referral data; Bradshaw, Buckley, & Ialongo, 2008; McIntosh, Frank, & Spaulding, 2010). Behavior screening tools offer important data augmenting academic screening systems to inform educational programming and ensure students have equal access to secondary and tertiary supports. To illustrate, consider a third-grade student who is struggling with reading comprehension skills as measured by AIMSweb Reading MAZE CBM who also demonstrates higher than average inattention and impulsivity levels as measured by the Strengths and Difficulties Questionnaire (SDQ; Goodman, 2001). This student may be offered additional assistance in the form of a Tier 2 reading group to address his reading comprehension needs in conjunction with a self-monitoring intervention to increase the student’s level of engagement during the reading group. In essence, information from behavior screening tools can be used in conjunction with academic data to address students’ multiple needs, offering them the necessary combinations of Tier 2 or Tier 3 supports.
Several universal behavior screening tools are currently available, including the Systematic Screening for Behavior Disorders (SSBD; Walker & Severson, 1992), the Early Screening Project (ESP; Walker, Severson, & Feil, 1995), Student Risk Screening Scale (SRSS; Drummond, 1994), the SDQ, the BASC™-2 Behavior and Emotional Screening System (BASC-2 BESS; Kamphaus & Reynolds, 2007), and the Social Skills Improvement System–Performance Screening Guide (SSiS-PSG; Elliott & Gresham, 2007). While it is beyond of the scope of this study to detail each available screening tool, we note they vary widely in scope and associated costs. For example, some tools are commercially available and offer the benefit of corresponding intervention materials (e.g., BASC-2 BESS and SSiS-PSG). Other tools are free access, yet do not offer corresponding intervention recommendations (e.g., SRSS and SDQ).
Ideally, all schools would have the resources to access screening tools as well as (a) rating scales (e.g., BASC-2) to provide additional information on students’ strengths and areas of concern to inform intervention efforts and (b) evidence-based strategies, practices, and programs required within multi-tiered systems of support. Yet, the reality is many schools do not have the resources necessary to acquire and implement some of the commercially available tools. For some schools, free-access tools are the best option given budget reductions and restrictions. Therefore, it is particularly essential we ensure all screening tools—including free-access tools—be examined for psychometric rigor so school teams may have confidence that the screening tool they select is accurately identifying students for additional supports (i.e., Tiers 2 and 3).
As such, a series of studies have been conducted to determine the reliability and validity of the SRSS for use at the elementary level to determine the accuracy of this free-access tool in detecting students with behavioral challenges. The SRSS is a seven-item, teacher completed screening tool requiring approximately 10 to 15 min to rate a homeroom class. Developed to detect students with antisocial behavior, the tool includes the following items, each rated on a 4-point Likert-type scale (never = 0, occasionally = 1, sometimes = 2, frequently = 3): (a) steal; (b) lie, cheat, sneak; (c) behavior problem; (d) peer rejection; (e) low academic achievement; (f) negative attitude; and (g) aggressive behavior. Total scores are summed (range = 0–21, with higher scores indicating higher risk) and used to classify students into one of three risk categories established by Drummond (1994): low (0–3), moderate (4–8), or high (9–21). Several studies have established the reliability and validity of the SRSS at the elementary level, including establishing convergent validity between SRSS scores and (a) Child Behavior Checklist’s Aggressive Behavior subscale score (Achenbach, 1991; Drummond, Eddy, Reid, & Bank, 1994) as well as (b) SSBD scores (Lane, Kalberg, Lambert, Crnobori, & Bruhn, 2010; Lane, Little, et al., 2009). In terms of the SSBD, SRSS scores were equally sensitive and specific in identifying students with externalizing behaviors according to Stage 2 measures of the SSBD (improving chance estimates by approximately 45%) and, to a lesser extent, those with internalizing behaviors (improving chance estimates by 30%; Lane, Kalberg, et al., 2010; Lane, Little, et al., 2009). Although not designed to detect students with internalizing behaviors, it appears the SRSS holds promise in this area.
Recently, the SRSS was adapted to include additional items to expand the tool’s original purpose and enhance detection of students with internalizing behaviors. Lane, Oakes, et al. (2012) conducted an initial study with 2,460 elementary students in California and Arizona of an adapted version of the SRSS: the Student Risk Screening Scale–Internalizing and Externalizing (SRSS-IE). Using a data analytic plan grounded in Classical Test Theory, five of the originally tested seven items reflecting internalizing behaviors were retained, yielding the SRSS-IE12. These new items were (a) emotionally flat; (b) shy, withdrawn; (c) sad, depressed; (d) anxious; and (e) lonely. Findings of this first study also established convergent validity of the SRSS-IE12 with two existing screening tools: the SDQ and the SSBD. Given the low base-rates of students with high risk (1%–7% of the school population; Sugai et al., 1999) replication is important. Lane, Menzies, Oakes, Lambert, et al. (2012) conducted two additional studies to explore the utility of the SRSS-IE with students in rural (N = 982) and urban (N = 1,079) districts, offering additional evidence of the reliability and validity of SRSS-IE12. Results of item level, internal consistency, and factor structure analyses supported retention of these same five items. Furthermore, convergent validity was again established between SRSS-IE12 scores as well as the two subscales (SRSS-E7, seven original externalizing items; and SRSS-I5, five retained internalizing items) with the SSBD.
Collectively, these findings suggested the SRSS-IE may be a reliable, valid tool with comparable accuracy to the psychometric properties of the SDQ and SSBD in detecting students with externalizing and internalizing behavior patterns, offering schools another free-access tool requiring even less time than the SDQ and SSBD to complete and score. Our current objective is to compare the SRSS-IE (SRSS-IE12 full scale, SRSS-E7 original items, and SRSS-I5 retained internalizing items) with the SSiS-PSG, a commercially available, relatively low-cost, user-friendly screening tool. The SSiS-PSG is broader in scope than the SRSS-IE, developed to detect academic and behavioral challenges, requires only 20 to 30 min of teacher time per class, and offers the benefit of a family of accompanying tools (e.g., more detailed behavior ratings scales) and intervention materials to support intervention efforts. At the elementary level, teachers rate homeroom students on four domains: Prosocial Behavior, Motivation to Learn, Reading Skills, and Math Skills using a five-level criterion-related performance scale. In brief, test–retest stability has been established as has interrater reliability (details to follow). In this study, we explore convergent validity of the SRSS-IE and SSiS-PSG scores, building upon the work of Lane, Richards-Tutor, Oakes, and Connor (2014) that established convergent validity of the original SRSS scores and SSiS-PSG scores with a sample of 577 English learners (ELs) attending a large suburban elementary school. Their results suggested a negative relation between SRSS and SSiS-PSG scores, meaning increased behavioral risk is associated with decreased Prosocial Behavior (–0.53), Motivation to Learn (–0.63), Math Skills (–0.50), and Reading Skills (–0.50) as rated by teachers.
Purpose
This is a validation study comparing two screening tools: the SRSS-IE and SSiS-PSG as applied with elementary-age students attending a diverse, southeastern, rural school. We examined the degree to which the SRSS-IE screening tool (Externalizing subscale [SRSS-E7], Internalizing subscale [SRSS-I5], and Total scale [SRSS-IE12]) is equally accurate in identifying students with significant difficulties in (a) Prosocial Behavior, (b) Motivation to Learn, (c) Math Skills, and (d) Reading Skills as measured by the SSiS-PSG. We hypothesized SRSS-IE scores would be inversely related to Reading Skills, Math Skills, Prosocial Behavior, and Motivation to Learn scores as measured by the SSiS-PSG, yielding moderately-to-highly negative correlation coefficients (Lane et al., 2014). We predicted the highest convergent validity between SRSS-IE scores and Prosocial Behavioral and Motivation to Learn scores as these are the most proximal constructs.
Method
Participants and Setting
Participants were 458 kindergarten through fifth-grade students (239 [52.18%] boys) attending a diverse elementary school in a southeastern state (see Table 1). The school served students in pre-kindergarten through fifth grades; however, the present study does not include information on the preschool students (n = 32) due to the small sample size which would not allow for adequate exploration of convergent validity of the two measures of interest. The diverse student body was predominantly White (51.31%), with 57.51% of students qualifying for free and reduced-price lunches (U.S. Department of Education, 2011) and 21.18% receiving special education services.
Student Characteristics.
The Title 1 (schoolwide) school was located near a metropolitan area in a southeastern state, classified as rural fringe, meaning the school was located ≤ 5 miles from an urbanized area. The school did not meet Adequate Yearly Progress (AYP) benchmarks as specified in No Child Left Behind (NCLB; 2001), having met 13 of the 17 performance targets; a measure of the state’s progress toward NCLB’s mandate for improving student achievement and closing achievement gaps. Student attendance rates met the state average of 95% as well as attendance target rates for all student subgroups as part of AYP.
Participating teachers (n = 25, one of whom was a special education teacher) rated each student in their homeroom class on two measures: the SRSS-IE and SSiS-PSG (descriptions to follow). Twenty-four (96.00%) teachers were female, with 22 (88.00%) White. Teachers ranged in age from 23 to 63 years (M = 35.72, SD = 13.49). Teachers had 1 to 36 (M = 9.20, SD = 19.79) years of teaching experience, with 1 to 36 years (M = 7.71, SD = 9.98) at the current school. Twenty-four (96.00%) teachers were certified in the subjects they taught at the time of this study, with 18 (72.00%) having bachelor’s and 7 (28.00%) having master’s degrees.
Procedures
After securing university and district approvals, the participating elementary school was selected for possible inclusion based on the recommendation from the Associate Superintendent and her leadership team (Director of Elementary Education, Director of Secondary Education, and Behavior Specialist). The district-leadership team recommended the school as it had a schoolwide positive behavior interventions and supports (SW-PBIS) plan and school-site leadership team in place. The district-leadership team indicated the school faculty had the requisite skills and structures in place to potentially benefit from the addition of systematic screening tools.
The principal investigator (PI) contacted and met with the principal to determine if she had interest in participating in the current validation study, with principal expressing interest and scheduling a meeting with teachers to allow the PI to invite them to participate. All teachers attended this meeting after school during March, approximately 28 weeks after the school year began. The PI explained the purpose of the study, procedures, and time involved, and obtained consent. All teachers (n = 25) elected to participate, completing a brief self-report demographic form (5 min) and two screening tools, the SRSS-IE (10–15 min) and the SSiS-PSG (20–30 min). Student names were not written on any of the screening tools. Instead, students in each class were assigned a unique number from 1 to 25, which was written on both screening tools and linked to a student demographic form (details to follow). All teachers had a master list numbered 1 to 25 to represent each student in their classes. This master list was not shared with the PI nor returned to the university. Teachers retained the master lists, which were stored in a secure location by the principal until the project staff returned to the school site approximately 30 days later to present findings of the study. Teachers received results from the PI about their own students using the 1 to 25 student identification numbers.
Teachers rated each student on both screening measures (SRSS-IE and SSiS-PSG) during this meeting. Although it would have been optimal to counterbalance measures to control for potential order effects, we did not do so to ensure all teachers completed each of the screening tools correctly during this first administration by allowing time to address any questions on administration for each measure. Forms were checked for accuracy and completion by project staff. At no point were student names placed on any forms leaving the school site. Before leaving the school, teachers who completed screening tools and the demographic form were entered into a drawing to receive a US$100 gift card. Each teacher was assigned a random number for the drawing and a random numbers table was used to determine the winning teacher’s number. The teacher was selected and awarded the gift card before the project team left the school site.
Student demographic information was obtained using the same 1 to 25 numbering system described previously. The school secretary completed a student demographic form for each classroom, noting students’ gender, ethnicity, grade, birth month, birth year, whether or not they were receiving special education services, and if so, whether or not the student’s eligibility category was emotionally disturbed (ED) according to Individuals With Disabilities Education Improvement Act (2004).
All de-identified data were entered into Excel spreadsheets by project staff, with 100% checked for data-entry reliability. Data entry errors (<1%) were reconciled. Students had no missing SRSS-IE data. SSiS-PSG data were mostly complete, with two students each missing one item. Missing data were not imputed; instead, analyses were conducted on available cases. Data analyses were conducted for students in Grades kindergarten through fifth by the PI using SAS software (Institute, 2004), with reliability checks performed by the second author.
Measures
SRSS-IE
The SRSS-IE is the free-access measure currently under development. It is an adapted version of the SRSS, with seven additional items measuring behaviors characteristic of internalizing patterns. See Lane, Oakes, et al. (2012) for a detailed description of how items were selected using currently available evidence and tools. Salient behaviors of internalizing behaviors were included as new items, and the original items developed by Drummond (1994) were retained. New items were as follows: (a) emotionally flat; (b) shy, withdrawn; (c) sad, depressed; (d) anxious; (e) obsessive-compulsive behavior; (f) lonely; (g) self-inflicts pain. Teachers rated all 14 items (7 original items and 7 new items) on the 4-point Likert-type scale (never = 0, occasionally = 1, sometimes = 2, frequently = 3) used in the SRSS. Studies exploring initial evidence for reliability and validity of the SRSS-IE at the elementary level supported retention of five additional items (yielding the SRSS-IE12), eliminating items: obsessive-compulsive behavior and self-inflicts pain. Removal of these 2 items was confirmed in subsequent studies in rural and urban settings (Lane, Menzies, Oakes, Lambert, et al., 2012; Lane, Oakes, et al., 2012). Results also offered convergent validity of SRSS-IE12 scores with two established screening tools: the SDQ (Goodman, 1997; Lane, Oakes, et al., 2012) and the SSBD (Lane, Menzies, Oakes, Lambert, et al., 2012; Lane, Oakes, et al., 2012; Walker & Severson, 1992). In this study, we refer to the SRSS-IE as follows: the initial 7 items constituting the SRSS as the SRSS-E7, the retained set of 5 internalizing items as the SRSS-I5, and the overall 12-item scale as SRSS-IE12. Internal consistency estimates were as follows: SRSS-E7 = .85, SRSS-I5 = .71, and SRSS-IE12 = .79.
SSiS-PSG
The SSiS-PSG is a systematic screening tool developed to detect challenges in academic (i.e., Math Skills and Reading Skills) and behavioral (i.e., Motivation to Learn and Prosocial Behavior) domains. The SSiS-PSG is validated for use PK–12, offering three versions (preschool, elementary, and secondary). For the elementary and secondary versions, teachers rate each student on four items using a five-level criterion-related performance scale. On the elementary version, a score of 1 (red band) indicates students who experience significant difficulty, 2 or 3 (yellow band) signals students who may be experiencing moderate difficulty, and 4 or 5 (green band) indicates adequate performance. As reported in the technical manual, test–retest reliability was calculated for 302 elementary students with an average of 74 days between screening time points (Intraclass Correlations: .68 Math Skills, .74 Reading Skills, .74 Motivation to Learn, .69 Prosocial Behavior; Gresham & Elliott, 2008). Interrater reliability was calculated between two teachers for 215 elementary students (Intraclass Correlations: .68 Math Skills, .57 Reading Skills, .62 Motivation to Learn, .55 Prosocial Behavior; Gresham & Elliott, 2008).
Experimental Design and Data Analytic Plan
We examined convergent validity of the SRSS-IE scores using the SSiS-PSG scores as a criterion measure. Specifically, we examined convergent validity using two analytic approaches. First, we computed Pearson correlation coefficients between the SRSS-E7 (externalizing; range = 0–21), SRSS-I5 (internalizing; range = 0–15), and SRSS-IE12 (the reduced scale; range = 0–36, with higher scores indicating higher levels of concern) and the four items constituting the SSiS-PSG (Reading Skills, Math Skills, Motivation to Learn, and Prosocial Behavior) for kindergarten through fifth-grade students (see Table 2). Second, we utilized receiver operating characteristic (ROC) curves to evaluate accuracy (sensitivity and specificity) of the SRSS-IE (SRSS-E7 externalizing, SRSS-I5 internalizing, and SRSS-IE12 total scores) ratings compared with Reading Skills, Math Skills, Motivation to Learn, and Prosocial Behavior ratings on the SSiS-PSG (Petras, Chilcoat, Leaf, Ialongo, & Kellam, 2004; see Table 3). ROC curves allow for a comparison of two distributions: the distribution of scores on the screening test for those students classified as having a given condition (e.g., behavior challenge) against the distribution of scores on the screening test for those that are classified as not having a given condition. The farther apart these distributions are from each other, the better the screening test is able to distinguish those at risk for the given condition from those not at risk for the given condition. One useful index derived from ROC curves is called the AUC, or area under the curve. The AUC can be interpreted as the probability a randomly selected person from the group identified as having the given condition will have a higher score on the screener than a randomly selected person from the group that does not have the given condition. The AUC typically ranges from .50 to 1.0, with .5 indicating the screener is operating at chance levels in detecting a problem, to 1.0, which means the screener is operating perfectly (Schatschneider, 2013). The first set of analyses compared SSiS-PSG scores indicating any risk (scores = 1, 2, or 3) compared with adequate progress (scores = 4 or 5). The second set of analyses compared SSiS-PSG scores indicating significant difficulty (score = 1) compared with adequate progress (scores = 4 or 5), similar to procedures used to explore extreme groups in earlier validation studies of the SRSS and SSBD (Lane, Little, et al., 2009). ROC curves based on logistic regression revealed the predictive accuracy for all possible cutting scores: (a) 0 to 21 for the SRSS-E7 externalizing items, (b) 0 to 15 for the SRSS-I5 internalizing items, and (c) 0 to 36 for the SRSS-IE12 full scale. For K–12 students, we hypothesized moderate-to-high, significant negative correlations for each of the SSiS-PSG items as they reference academic and academic enabling skill strengths. Academic enabling skills are those needed to fully engage in and benefit from instruction (DiPerna & Elliott, 2002) such as Motivation to Learn and Prosocial Behaviors. In contrast, the SRSS-IE proposes to measure antisocial and behavioral skill-sets that impede learning—the opposite of Prosocial Behavior. Correlations were interpreted using guidelines specified in Kettler, Elliott, Davies, and Griffin (2012) as informed by Cohen’s (1992) classic Power Primer as follows: .00 to .10 were nonexistent, .10 to .30 were small, .30 to .50 were medium, .50 to .70 were large, .70 to .90 were very large, and .90 to 1.00 were close to perfect (Hopkins, 2002; Kettler et al., 2010). We also expected the SRSS-IE would be more predictive of Prosocial Behavior and Motivation to Learn scores than Reading Skills and Math Skills scores as the constructs measured by the SRSS-IE were more closely related to social behavior and motivation (Walker, Ramsey, & Gresham, 2004).
Convergent Validity: Correlation Coefficients - SRSS-E7, SRSS-I5, and SRSS-IE12 With the SSIS-PSG.
Note. Correlations were interpreted using the following guidelines specified in Kettler, Elliott, Davies, and Griffin (2012): .00 to .10 were nonexistent, .10 to .30 were small, .30 to .50 were medium, .50 to .70 were large, .70 to .90 were very large, and .90 to 1.00 were close to perfect (Hopkins, 2002; Kettler et al., 2010). SRSS-E7 = the original seven items constituting the SRSS; SRSS-I5 = the five items retained to measure internalizing behaviors; SRSS-IE12 = the original seven items from the SRSS developed by Drummond (1994) combined with the new five items constituting the SRSS-I5; SSiS-PSG = Social Skills Improvement System–Performance Screening Guide (Elliott & Gresham, 2007); SRSS-IE = Student Risk Screening Scale–Internalizing and Externalizing.
Convergent Validity: Receiver Operating Characteristics - SRSS-E7, SRSS-I5, and SRSS-IE12 With the SSiS-PSG.
Note. SRSS-E7 = the original seven items constituting the SRSS; SRSS-I5 = the five items retained to measure internalizing behaviors; SRSS-IE12 = the original seven items from the SRSS developed by Drummond (1994) combined with the new five items constituting the SRSS-I5; SSiS-PSG = the Social Skills Improvement System–Performance Screening Guide (Elliott & Gresham, 2007); SRSS-IE = Student Risk Screening Scale–Internalizing and Externalizing; ROC = receiver operating characteristics; AUC = area under the curve; Any Difficulty = SSiS scores of 1, 2, or 3; Significant Difficulty = SSiS-PSG Score of 1 and Adequate Progress to scores of 4 or 5.
Results
Convergent Validity With Prosocial Behavior as Measured by the SSiS-PSG
SSiS-PSG Prosocial Behavior scores were statistically significantly (p < .0001) and negatively correlated with the SRSS-E7 (r = −.71), SRSS-I5 (r = −.32), and SRSS-IE12 (r = −.72) scores, with externalizing and full scale correlation coefficients of very large magnitude.
In comparing scores of students with any difficulty in Prosocial Behavior with those making adequate progress according to SSiS-PSG Prosocial Behavior scores, 204 students were rated as having any difficulty (with scores of 1, 2, or 3) and 254 were making adequate progress (with scores of 4 or 5). Findings of ROC curve analyses with SRSS-E7 scores yielded an AUC of .845 in predicting Prosocial Behavior (0.345 better than chance, which is 0.50). The AUC was higher for the full scale SRSS-IE12 (AUC of .859), and lower for the SRSS-I5 (AUC of .652).
In comparing scores of students with significant difficulty in Prosocial Behavior with those making adequate progress according to SSiS-PSG Prosocial Behavior scores, 33 students were rated as having significant difficulty (with a score of 1) and 254 were making adequate progress (with scores of 4 or 5). Findings of ROC curve analyses with SRSS-E7 scores yielded an AUC of .956 in predicting Prosocial Behavior (0.456 greater than chance). The AUC was higher for the full scale SRSS-IE12 (AUC of .972), and lower for the SRSS-I5 (AUC of .750).
Convergent Validity With Motivation to Learn as Measured by the SSiS-PSG
SSiS-PSG Motivation to Learn scores were statistically significantly (p < .0001) and negatively correlated with the SRSS-E7 (r = −.61), SRSS-I5 (r = −.26), and SRSS-IE12 (r = −.62) scores, with the externalizing and full scale correlation coefficients of large magnitude.
In comparing scores of students with any difficulty in Motivation to Learn with those making adequate progress according to SSiS-PSG Motivation to Learn scores, 207 students were rated as having difficulty and 250 were making adequate progress. Findings of ROC curve analyses with SRSS-E7 scores yielded an AUC of .805 in predicting Motivation to Learn (0.305 better than chance). The AUC was comparable for the full scale SRSS-IE12 (AUC of .801) and lower for the SRSS-I5 (AUC of .629).
In comparing scores students with significant difficulty in Motivation to Learn with those making adequate progress according to SSiS-PSG Motivation to Learn scores, 30 students were rated as having significant difficulty. Findings of ROC curve analyses with SRSS-E7 scores yielded an AUC of .902 in predicting Motivation to Learn (0.402 better than chance). The AUC was slightly higher for the full scale SRSS-IE12 (AUC of .904) and lower for the SRSS-I5 (AUC of .630).
Convergent Validity With Math Skills as Measured by the SSiS-PSG
SSiS-PSG Math Skills scores were statistically significantly (p < .0001) and negatively correlated with the SRSS-E7 (r = −.39), SRSS-I5 (r = −.25), and SRSS-IE12 (r = −.43) scores, with the full scale correlation coefficients of medium magnitude.
In comparing scores of students with any difficulty in Math Skills with those making adequate progress according to SSiS-PSG Math Skills scores, 240 students were rated as having difficulty and 218 were making adequate progress. Findings of ROC curve analyses with SRSS-E7 scores yielded an AUC of .702 in predicting math (0.202 better than chance). The AUC was slightly higher for the full scale SRSS-IE12 (AUC of .710) and lower for the SRSS-I5 (AUC of .604).
In comparing scores of students with significant difficulty in Math Skills with those making adequate progress according to SSiS-PSG Math Skills scores, 20 students were rated as having significant difficulty. Findings of ROC curve analyses with SRSS-E7 scores yielded an AUC of .760 in predicting math (0.26 better than chance). The AUC was substantially higher for the full scale SRSS-IE12 (AUC of .817) and slightly higher for the SRSS-I5 (AUC of .767).
Convergent Validity With Reading Skills as Measured by the SSiS-PSG
SSiS-PSG Reading Skills scores were statistically significantly (p < .0001) and negatively correlated with the SRSS-E7 (r = −.39), SRSS-I5 (r = −.19), and SRSS-IE12 (r = −.41) scores, with the full scale correlation coefficients of medium magnitude.
In comparing scores of students with any difficulty in reading with those making adequate progress per SSiS-PSG Reading Skills scores, 264 students were rated as having difficulty and 193 were making adequate progress. Results of ROC curve analyses with SRSS-E7 scores yielded an AUC of .706 in predicting reading (0.206 better than chance). The AUC was comparable for the full scale SRSS-IE12 (AUC of .705) and lower for the SRSS-I5 (AUC of .587).
In comparing scores of students with significant difficulty in reading with those making adequate progress according to SSiS-PSG Reading Skills scores, 51 students were rated as having significant difficulty. Findings of the ROC curve analyses with SRSS-E7 scores yielded an AUC of .778 in predicting reading (0.278 better than chance). The AUC was higher for the full scale SRSS-IE12 (AUC of .805) and slightly lower for the SRSS-I5 (AUC of .656).
Summary
The SRSS-IE12, relative to the SRSS-E7 and the SRSS-I5, had the highest correlation coefficients with each of the SSiS-PSG scores, just slightly above SRSS-E7 correlation coefficients. As expected, the correlation coefficients were of the greatest magnitude with the Prosocial Behavior scores, which is the inverse of antisocial behavior. Collectively, these findings offer initial evidence of convergent validity between the three versions of the SRSS (SRSS-E7, SRSS-I5, and SRSS-IE12) and SSiS-PSG Reading Skills, Math Skills, Prosocial Behavior, and Motivation to Learn scores.
Discussion
As schools move toward prevention frameworks with tiered supports to address the learning and behavioral needs of students, accurate detection of those students who need more than the primary plan is essential. Universal screening procedures for behavior offer an efficient process for school-site leadership teams to ensure all students have equal access to secondary and tertiary interventions and supports, if needed (Lane, Menzies, Oakes, & Kalberg, 2012). Furthermore, these data allow leadership teams to monitor the level of student risk in a school building to assess and improve their primary prevention program (Lane, Kalberg, & Menzies, 2009). That is, if risk is increasing, the school team may examine the level of fidelity of the primary plan, assess the need for professional development, reevaluate the goals and responsibilities of the plan, and address identified areas of need with additional instructional programming, such as implementing a bullying prevention program (e.g., Bully Prevention in Positive Behavior Support; Ross & Horner, 2014). Because schoolwide screening procedures to detect students with behavioral concerns at the earliest possible juncture are so important for early intervening, all schools must have access to high-quality, socially valid screening tools.
There are behavior screening tools available PK to Grade 12, both commercially (e.g., BASC-2 BESS, SSiS-PSG) and free access (e.g., SDQ, SRSS) to meet the needs of all schools regardless of monetary resources available. Tools have been developed to provide schools a comprehensive approach for detecting students with increased behavioral needs as well as addressing those needs through instructional programs (e.g., BASC-2 BESS, SSiS); yet, not all schools have the resources to invest. The purpose of this article was to compare a newly adapted, user-friendly, free-access screening tool the SRSS-IE (Lane, Menzies, Oakes, Lambert, et al., 2012; Lane, Oakes, et al., 2012) to the existing user-friendly, commercially available measure, the SSiS-PSG.
The SRSS-IE scales (e.g., SRSS-E7, SRSS-I5, and SRSS-IE12) were all significantly and negatively correlated with the SSiS-PSG Scales (Reading Skills, Math Skills, Motivation to Learn, Prosocial Behavior). The correlation coefficients between the SRSS-E7 and SSiS-PSG scales were comparable with those reported by Lane et al. (2014), also suggesting a particularly strong relation between the SRSS-E7 and Motivation to Learn and Prosocial Behavior subscale scores as compared with the Reading Skills and Math Skills subscales scores. Furthermore, the newly adapted, combined scale (SRSS-IE12) had the strongest correlations with the SSiS-PSG scales indicating the combined scale would detect the largest proportion of students identified at risk on the SSiS-PSG for academic and behavioral concern. These findings add to previous preliminary studies of the technical qualities of the SRSS-IE12.
Consistent with the findings of Lane, Little, et al. (2009) in their study comparing the SRSS and SSBD, the utility of the SRSS-IE depends—at least partially—on how the tool is used. For example, when looking for students who struggle with respect to Prosocial Behavior, it appears the SRSS-IE is most useful when contrasting the low (adequate progress) and high risk (significant difficulties) groups. In this case, the AUC for the SRSS-IE12 is 97.20%, slightly better than the original 7 items (SRSS-E7, AUC .956). When contrasting the SRSS with other cutting scores (e.g., any difficulty vs. adequate progress), the predictive accuracy wanes (e.g., for Prosocial Behavior, SRSS-IE12 is 85.90%). ROC curve analyses provide initial evidence to suggest the SRSS-IE12 is an effective tool for predicting challenges associated with Prosocial Behavior and Motivation to Learn as measured by the SSiS-PSG. We note the five internalizing items (SRSS-I5) were less effective in predicting these outcomes than when combined with the SRSS-E7 items. This supports using the SRSS-IE12 total score when making these predictions.
A final noteworthy outcome is the accuracy of the SRSS-IE12 with respect to academic performance. Specifically, the accuracy for the SRSS-IE12 is 80.50% for Reading Skills and 81.70% for Math Skills, improving chance estimates substantially. This suggests the SRSS-IE is also an effective tool for predicting teacher ratings of academic performance in reading and math (highlighting the link between academic and behavioral performance), but to a lesser extent compared with predicting prosocial skills and motivation for learning.
Implications
First, a very strong, significant, negative relation exists between SRSS-IE12 scores and Prosocial Skills on the SSiS-PSG indicating students with more risk on the SRSS-IE12 may have fewer prosocial skills. In terms of predicting academic achievement, social skills are the second strongest predictor after motivation (DiPerna & Elliott, 2002). Malecki and Elliott (2002) found prosocial behavior to be a significant and independent predictor of academic achievement when examining teacher ratings of academic competence, achievement in math and reading, and problem behavior. The negative consequences of limited social skills cannot be overstated. Prosocial skills are essential for building and maintaining relationships with peer and teachers. Peer and teacher relationships, in turn, are related to students’ engagement, interest, motivation, and school attainment (Pianta, Hamre, & Allen, 2012; Wentzel & Watkins, 2002). For example, teachers may offer more learning opportunities to students who display strong social skills (e.g., by regularly calling on them) and social skills can impact academic practice and responding (e.g., raising hand to answer a question; Greenwood, Horton, & Utley, 2002). In terms of prevention efforts, instructional environments facilitating the development of students’ prosocial skills are characterized by organizational structures such as clear expectations, predictable routines, active supervision, and maximizing instructional time (Lane, Menzies, Bruhn, & Crnobori, 2011; Pianta et al., 2012).
Second, this study provides information on the relation between students who exhibit behavioral needs as detected by the SRSS-IE12 and Motivation to Learn. A very large magnitude, significant, negative relation between higher scores on the SRSS-IE12 and the SSiS-PSG Motivation to Learn scale shows that students with higher levels of behavioral risk have lower levels of Motivation to Learn. Motivation is one of the strongest predictors of academic achievement (DiPerna & Elliott, 2002). Therefore, particularly for classes with higher numbers of students with increased risk on the SRSS-IE12, strategies to increase student motivation to learn can be implemented by the teacher through low-intensity strategies such as offering choices in instructional activities, optimal levels of challenge, control over instructional programming, and contextualization of learning utilizing relevant contexts for problem solving and application of knowledge (Lane et al., 2011). In terms of Tier 2 supports, motivation can be developed through interventions that target improvements in self-determination skills both separately and in conjunction with academic skills (Oakes et al., 2012).
Third, a moderate, significant, negative relation was found between SRSS-IE12 scores and SSiS-PSG scores of Math Skills and Reading Skills indicating students with greater behavioral risk on the SRSS-IE12 are perceived by their teachers to have lower reading and math skill attainment. ROC curves also suggest chance estimates are improved by approximately 20–30% in these domains. Previous studies at the elementary level have found SRSS scores are inversely related to year-end reading (Oakes et al., 2010) and language arts (Menzies & Lane, 2011) achievement. At the secondary level, SRSS scores are predictive of course failures and grade point averages (Lane, Bruhn, Eisner, & Kalberg, 2010; Lane, Kalberg, Parks, & Carter, 2008; Lane et al., 2013; Lane, Parks, Kalberg, & Carter, 2007). The relation between academic underachievement and behavior problems is well documented (Nelson, Benner, Lane, & Smith, 2004). The importance of detecting and intervening with students with comorbid academic and behavioral difficulties is critical in that this group of students is our most at-risk population (Reinke, Herman, Petras, & Ialongo, 2008) and, often, the students deemed nonresponsive to secondary or tertiary academic efforts (Kamps & Greenwood, 2005). These findings offer additional evidence of the need for CI3T models of prevention to address behavioral and academic needs together (Kalberg, Lane, & Menzies, 2010) across all levels of prevention.
Future Directions and Limitations
We encourage readers to interpret these modest outcomes with caution in light of the following limitations, the first of which pertains to generalization concerns. Given these data were collected at just one school, it is imperative that additional inquiry be conducted to explore the convergent validity between the SRSS-IE and the SSiS-PSG before drawing any definitive conclusions. Additional studies are needed in other locales and with larger samples that would allow the nested nature of these data to be addressed (e.g., potential dependency issues resulting from students being nested in teachers’ classes).
Second, as with the Lane, Little, et al. (2009) study, this study compared only two screeners: the SRSS-IE and SSiS-PSG based on premise the SSiS-PSG is a very user-friendly tool, offering a wealth of information to inform primary, secondary, and tertiary supports using the family of SSiS tools available for screening, assessment, and intervention efforts. We elected to compare the relative value—psychometrically and in terms of feasibility—of the SRSS-IE12 as this expanded version of the SRSS continues to allow each student to receive a total score and contains only 12 items for completion, with the addition of five items measuring characteristic patterns of internalizing behaviors. We recommend additional studies be conducted to compare the SRSS-IE with available tools such as the BASC-2 Behavior and Emotional Screening Scale (Kamphaus & Reynolds, 2007), another commercially available screening tool. We also encourage additional inquiry into secondary schools, to examine the convergent validity of screening tools as applied in middle and high school settings (e.g., Lane et al., 2008; Lane et al., 2007). Such information would help school-site and district-level leadership teams in making decisions about which systematic screening tools to adopt as part of regular school practices based on screening goals and resource considerations.
Third, in this study, we did not explore specific cutting scores to establish low, moderate, and high risk status for the SRSS-IE12 and the SRSS-E5 nor did this study address the nested nature of the data (students nested in teacher’s classrooms) due to the small sample size (number of teachers). Next steps in the development of this measure are (a) to identify cutting scores at the elementary, middle, and high school levels (American Educational Research Association [AERA], American Psychological Association [APA], & National Council for Measurement in Education [NCME], 1999); (b) examine the predictive validity of the SRSS-IE12 with important learning and behavioral outcomes (e.g., office discipline referrals, attendance patterns, referrals for mental health supports, state achievement assessments); and (c) determine the social validity of the screening procedures (e.g., efficiency; addressing important goals; and resources for preparing, conducting, scoring, and interpreting the measure). School-site and district-level leadership teams will be able to use cutting scores to determine which students would benefit from secondary and tertiary supports currently available at the school sites and additional interventions and supports needed to fully address existing student needs. Cutting scores must demonstrate high sensitivity (the proportion of students detected that actually do present at-risk behaviors) and specificity (the proportion of students who fall below the at-risk cut point who do not present at-risk behaviors; AERA, APA, & NCME, 1999; Kraemer, 1992; Lane, Kalberg, et al., 2010). Future studies with larger samples should examine specific risk categories for the SRSS-IE12 and SRSS-I5 in predicting important behavioral, social, and academic outcomes for students, while addressing issues of dependency associated with the nested nature of the data (Elliott & Gresham, 2007; Lane, Oakes, et al., 2012). We currently have a multi-state validation study nearing completion to establish cutting scores, using the Child Behavior Checklist Teacher Report form (Achenbach, 1991) to inform the development of these scores. This is clearly a critical next step for researchers and practitioners alike as this information will be central for use in connecting students to Tier 2 and 3 supports to address internalizing issues. However, given that many schools have adopted the SRSS-IE, despite the absence of cutting scores for the SRSS-IE12 and SRSS-I5, we feel this modest study presented here is still important for educators in informing the decision-making processes involved with selecting behavior screening tools.
Fourth, as we mentioned previously, this study focused on comparing two screening tools and did not examine actual outcomes for these students. One purpose of screening is for the early detection of students who may, without intervention, have deleterious school and post school outcomes (e.g., low grades, failed courses, school dropout, justice system involvement; Wagner, Kutash, Duchnowski, Epstein, & Sumi, 2005). Therefore, it is important screening tools accurately predict, as early as possible, which students may experience these difficulties. Findings of the significant relation between SRSS-IE12 scores and SSiS-PSG Reading Skills and Math Skills Scales offer evidence as to the utility of the SRSS-IE12 in detecting students with comorbid risk; however, the SSiS-PSG scales are teacher ratings of student performance and therefore must be confirmed with further evidence of actual academic outcomes. Previous studies of the SRSS have examined the tool’s ability to predict students’ year-end outcomes. The SRSS-IE must now be examined in terms of its ability to predict outcomes for students with externalizing and internalizing behavioral risk as in other studies of the predictive validity of the SRSS (e.g., Menzies & Lane, 2011).
Finally, this study did not address issues of social acceptability. For any new practice to be fully implemented and sustained, teachers must have input into the decision-making process of selecting a universal behavior screener. Given teachers’ multiple demands, it is essential that they have a voice in the choosing of the screening tool for their school and in the procedures for its use (Harrison, Vannest, & Reynolds, 2013). Therefore, assessing the social validity of the screening tool selected and procedures is important. Future research should examine social validity of their process as well as the screening tools. As school contexts and goals are unique, this practice should be considered by each school site adopting universal screening. Nonetheless, research assessing social validity of screening would provide information to school systems and researchers developing tools on broader areas of teacher concern. Assessing and responding to teachers’ perspectives would provide valuable information in planning for the sustainable adoption of universal screening for behavior as a primary prevention practice within multi-tiered systems of support (McIntosh, Filter, Bennett, Ryan, & Sugai, 2010).
Summary
Despite the noted limitations, findings from this very modest psychometric study provide initial evidence to support the use of the SRSS-IE as an equally reliable tool as the SSiS-PSG when attempting to identify students with challenges in prosocial behaviors and motivation for learning. While the SRSS improves chance estimates of detecting which elementary-age students might have deficits in reading or math performance as measured by the SSiS-PSG and there are statistically significant relations between SRSS-IE (SRSS-IE12, SRSS-E7, and SRSS-I5) scores and SSiS-PSG (Reading Skills, Math Skills, Prosocial Behavior, and Motivation to Learn) scores, the SRSS-IE does not share the same level of predictive accuracy as the SSiS-PSG for students’ academic performance.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded in part by an internal grant from the University of North Carolina at Chapel Hill: Identifying and Supporting K–12 Students Within the Context of Three-Tiered Models of Prevention to Meet Students Multiple Needs: A Collaborative Effort, Research Triangle Schools Partnership Community/Schools Partnership Grant.