Development and Validation of a Parent Version of the Social,Academic,and Emotional Behavior Risk Screener in an Elementary Sample

Abstract

A line of research has supported the incremental and construct validity of multi-informant assessment. Accordingly, multiple universal screening systems have been designed to support the collection of information from multiple informants. The purpose of the study was to expand the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) suite of tools to include a novel parent form (SAEBRS-P). Specific aims of the study included the examination of SAEBRS-P factor structure, internal consistency reliability, and concurrent validity. Screening was conducted across four elementary schools in the Pacific Northwest with 212 students and their parents. Factor analytic results supported the retention of four factors, which demonstrated acceptable internal consistency reliability. Further analyses supported SAEBRS-P concurrent criterion-related validity, indicating moderate to high correlations between SAEBRS-P and Strengths and Difficulties Questionnaire (SDQ) scales. Limitations and implications for research are discussed.

Keywords

universal screening rating scales behavior assessment

The adoption of multi-tiered system of supports (MTSS) has increased efforts to identify and prevent further academic and behavioral difficulties in schools (Bruhn, Woods-Groves, & Huddle, 2014). MTSS utilizes a public health model, which emphasizes prevention through a three-tiered model of support. Tier 1 corresponds to prevention efforts targeted at the whole school. Tier 2 corresponds to group- or individual-level prevention efforts targeted at students who are at risk for academic or behavioral difficulties. Finally, Tier 3 corresponds to individual or intensive prevention efforts targeted for high-risk students who have not responded to supports at Tier 1 or Tier 2 level (Sugai & Horner, 2006). Within MTSS models, two common approaches to the identification of students who need more intensive behavioral support include office discipline referrals (ODRs; Sugai, Sprague, Horner, & Walker, 2000) and teacher referrals (Eklund et al., 2009). Despite their commonality, research suggests these approaches have questionable sensitivity and are prone to bias (Eklund et al., 2009; Miller et al., 2015; Predy, McIntosh, & Frank, 2014). In addition, ODR and teacher referral data tend to only identify students after they have experienced an extensive level of behavior difficulties over an extended time.

Universal Screening

Given the limitations associated with both ODRs and teacher referrals, researchers have attempted to identify alternative, more proactive methods by which to identify students in need of intervention. One of the most popular instances of such methods is found in universal screening, defined as the systematic evaluation of all students within a school using brief and efficient assessment tools to identify some condition of interest (Jenkins, Hudson, & Johnson, 2007). Research has suggested the implementation of a universal screening protocol supports the identification of at-risk students, thus enhancing the likelihood of intervention success (Eklund & Dowdy, 2014; Glover & Albers, 2007). Legislation concerned with student mental health has increased the need for schools to implement practices associated with early identification and early intervention through the use of MTSS and universal screening. Two major pieces of legislation require schools to address social, emotional, and behavioral needs for children in school. First, the Individuals With Disabilities Education Improvement Act (IDEA) of 2004 endorses the identification of children who may need additional support. Specifically, Part C of IDEA requires schools to identify students via referrals and screening to provide early intervention programs to children with disabilities (IDEA of 2004, 2004). In addition, in an effort to support school safety and student mental health, the Every Student Succeeds Act (2015) endorses the use of valid and reliable measures to identify students with academic and social-emotional and behavioral difficulties.

Multi-Informant Assessment

As a result of increased interest in MTSS prevention efforts generally and universal screening specifically, there is a need to extend screening-related research. Cook, Volpe, and Livanis (2010) identified gaps in the literature associated with the development, implementation, and outcomes of universal screening. One such gap is the need to identify the optimal informant of student behavior. Recognition of the potential contribution of various informant perspectives highlights the importance of multi-informant assessment, which has emerged as the gold standard for behavior assessment (Hunsley & Mash, 2007). Research to date has yielded preliminary support for the incremental validity of multi-informant approaches, suggesting the incorporation of multiple perspectives may enhance the validity of psychological assessments (De Los Reyes et al., 2015). Unfortunately, despite its enhanced validity, the time- and resource-intensive nature of multi-informant assessment commonly restricts its use to high stake decision making, such as that associated with classification and diagnosis. In contrast, applied assessment is typically mono-informant when used to drive lower stakes decisions, such as those related to universal screening (Dowdy & Kim, 2012). Within schools, universal screening data are commonly collected from a single individual, typically classroom teachers (Cook et al., 2010; Gerber & Semmel, 1984). By limiting universal screening procedures to a single informant, schools can (a) reduce the time and effort dedicated to data collection and analysis and (b) facilitate efficient decision making.

Yet, though teachers offer valuable perspectives on student behavior in school settings, parents possess unique viewpoints and may improve decision making within universal screening procedures. Parents have greater opportunities to observe their child’s behavior over time and across multiple settings (Smith, 2007). Such access is particularly true of younger students at the age of school entry, as these students still spend a sizable portion of their time with their parents. They are likely to be the sole source of information regarding a student’s behavior in the home and community (De Los Reyes et al., 2015). Findings further suggest parents provide key information regarding particular behaviors, such as externalizing behaviors (e.g., noncompliance, opposition; Dowdy & Kim, 2012). Collectively, this research suggests that by including the parent perspective within the screening process, schools might improve their ability to make timely and accurate decisions about student risk, particularly in relation to externalizing behaviors exhibited by younger students.

Support for the parental perspective is primarily found in the informant discrepancy literature. Discrepancy research suggests different informants are unlikely to fully agree in their assessment of student behavior. For instance, in their seminal study of informant agreement, Achenbach, McConaughy, and Howell (1987) found informants (e.g., parents and teachers) exhibited low to moderate correspondence in evaluating child behavior (e.g., .20–.60). Historically, scholars have attributed such disagreement to meaningful differences in child behavior across contexts, resulting in differential levels of informant exposure to the behaviors of interest (De Los Reyes & Kazdin, 2005). This conceptualization is in accordance with behavioral theory, which presupposes that behavior is determined by and linked to the environment within which it is displayed, resulting in behavioral variability across settings (Cooper, Heron, & Heward, 2007). Meta-analytic research has recently supported this presupposition, finding variability among informants to be at least partially explained by contextual differences in child behavior (De Los Reyes et al., 2015).

Parents as Screening Informants

In summary, research suggests that informants are unlikely to fully agree in their assessment of a student’s behavior. Modern perspectives of this phenomenon promote the interpretation of such disagreement as not reflective of measurement error, but rather as the unique perspective informants bring to the understanding of student behavior (De Los Reyes et al., 2015). Although it may differ from that of other informants, the information parents provide is still valuable in that it represents behavior that might not otherwise be viewed by other individuals (e.g., teachers). Accordingly, universal screening systems that do not permit parental perspectives might be considered somewhat limited.

Research has supported the development of small number of multi-informant screening systems that incorporate the parental perspective. One such commonly used screening systems the Ages and Stages Questionnaires (ASQ; Squires, Bricker, & Potter, 2009). Although research supports ASQ reliability, validity, and diagnostic accuracy (Hornman, Kerstjens, Winter, Bos, & Reijneveld, 2013; Schonhaut, Armijo, Schonstedt, Alvarez, & Cordero, 2013), the measure is only validated up to 5 years of age. Other screening systems have been designed to assess a wider range of individuals. For instance, the Behavioral and Emotional Screening System (BESS; Kamphaus & Reynolds, 2015) is intended for use with individuals 3 to 18 years old. The BESS is founded upon three forms, including those specific to teachers, parents, and student self-report. Research has supported BESS Parent Form’s criterion-related validity, structural validity, and reliability (Dowdy, Chin, Twyford, & Deverm, 2011; Dowdy, Kamphaus, Abdou, & Twyford, 2013). Finally, the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997) also represents a broader screening system inclusive of teacher, parent, and student forms. A large body of research has supported the SDQ Parent form, with a meta-analytic overview speaking to the screener’s internal consistency, test–retest reliability, interrater reliability, and validity (Stone, Otten, Engels, Vermulst, & Janssens, 2010).

Beyond these multi-informant screening systems, the majority of screeners are specific to a single informant. In particular, many screeners permit teacher report alone, including the Student Risk Screening Scale–Internalizing and Externalizing (SRSS-IE; Lane, Menzies, Oakes, & Kalberg, 2012), Systematic Screening for Behavior Disorders (SSBD; Walker, Severson, & Feil, 2014), and the Social Skills Improvement System—Performance Screening Guide (PSG; Gresham & Elliott, 2008). Until recently, the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS; Kilgus & von der Embse, 2014) was also specific to teacher report. Yet, research has supported expansion of the SAEBRS system to include a student self-report form (von der Embse, Kilgus, Iaccarino, & Levi-Nielsen, 2017). The purpose of this study was to further expand the SAEBRS system via development of a parent form, thereby providing an additional entry into the limited range of multi-informant screening systems.

Summary and Purpose

Taken together, research to date has established a precedent for the development of multi-informant universal screening systems inclusive of multiple forms, including those specific to teachers, parents, and students. A review of existing screening options suggests that although multiple screening systems support a multi-informant approach, others do not. The primary aim of this investigation was to support the expansion of one such system: the SAEBRS (Kilgus & von der Embse, 2014) suite of tools.

To date, multiple studies have supported the development and validation of the SAEBRS–Teacher Rating Scale (SAEBRS-T) and SAEBRS–Student Rating Scale (SAEBRS-S). Specifically, findings have spoken to each form’s internal consistency, criterion-related validity, structural validity, and diagnostic accuracy across the K–12 spectrum (Kilgus, Chafouleas, & Riley-Tillman, 2013; Kilgus, Sims, von der Embse, & Riley-Tillman, 2015; Kilgus, Sims, von der Embse, & Taylor, 2016; von der Embse, Iaccarino, Mankin, Kilgus, & Magen, 2017; von der Embse, Kilgus, et al., 2017). Each SAEBRS form taps into four factors, including an overarching broad factor (i.e., Total Behavior), and three narrow factors specific to domains of behavioral functioning (i.e., Social Behavior, Academic Behavior, and Emotional Behavior). Although its research is promising, the SAEBRS suite is notable for its omission of a parent form. Thus, the overarching purpose of the current investigation was to support the development and initial validation of the SAEBRS–Parent Rating Scale (SAEBRS-P).

Purposes of this study were threefold, with each being specific to psychometric properties deemed essential and necessary for any psychological measure (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). The first purpose of the current investigation was to examine SAEBRS-P structural validity through confirmatory factor analyses (CFAs). We hypothesized that CFAs would support a three-factor structure: Social Behavior, Academic Behavior, and Emotional Behavior. We further hypothesized that each factor would include adaptive and maladaptive behaviors, with (a) Social Behavior including items related to externalizing problems and social skills, (b) Academic Behavior including items regarding attention problems and academic enablers, and (c) Emotional Behavior including items related to internalizing problems and social-emotional skills. Finally, we anticipated that this three-factor structure would yield (a) superior fit relative to more a parsimonious factor structure (i.e., a unidimensional model) and (b) similar fit to that of a more complex factor structure (i.e., a four-factor structure, which specified factors specific to adaptive and maladaptive functioning), thereby supporting the more parsimonious three-factor structure. Our CFA-based, model comparison approach to examining structural validity was consistent with prior SAEBRS-T research (e.g., Kilgus, Sims, von der Embse, & Riley-Tillman, 2015). Furthermore, though the SAEBRS-P is a novel measure, our use of CFA (in lieu of exploratory factor analysis) was considered appropriate given the substantial expectations for SAEBRS-P factor structure, which were informed by prior CFA findings specific to the SAEBRS-T.

The second purpose was to examine the internal consistency of each resulting SAEBRS-P factor (as indicated by CFA findings) via the calculation of omega coefficients. It was hypothesized that each factor would be associated with adequate reliability (>.75; Reise, Bonifay, & Haviland, 2013). The third purpose was to examine evidence of the SAEBRS-P concurrent validity relative to a well-established screening tool. We anticipated that resulting SAEBRS-P factor scores would be highly correlated with both (a) the SDQ constructs with which each SAEBRS-P factor was most similar in terms of item content (e.g., SAEBRS-P Emotional Behavior and SDQ Emotional Symptoms) and (b) the SDQ Overall Difficulties score, given the enhanced reliability of this broad score and its inclusion of content from four of the five SDQ scales.

Method

Participants

Participants included 212 students (51.9% female) and their parents (91.5% female) who were attending four elementary schools in the Pacific Northwest. The sample consisted of students enrolled in kindergarten through fourth grade and is characterized by a homogeneous ethnicity profile (91% White). On average, children were 7.75 years old (SD = 1.49). About 78% of parents reported an alternate caregiver was present in the home and parents reported an average of 1.89 (SD = 1.11) kindergarten to 12th-grade children were living in the home. Parents reported their family had a child attending school within the school district for an average of 4.59 years (SD = 4.25). Three out of the four schools were Title 1 schools. All schools were in a midsize suburban area. Additional parent and student demographic data are in Table 1.

Table 1.

Demographic Information for Study Participants.

Demographic variable	Total		School 1		School 2		School 3		School 4
Demographic variable	N	%	n	%	n	%	n	%	n	%
Sample size	212	100	47	22	32	15	44	21	89	42
Child gender
Female	110	52	28	25	11	10	23	21	48	44
Child grade
Kindergarten	43	20	17	40	6	14	5	12	15	35
First	44	21	8	18	7	16	7	16	22	50
Second	48	23	13	27	7	15	4	8	24	50
Third	40	19	9	23	7	18	13	33	11	28
Fourth	35	17	0	0	4	11	14	40	17	49
Child race^a
White	192	91	44	23	27	14	41	21	80	42
Latino	30	14	5	17	8	27	5	17	12	40
American Indian	13	6	0	0	4	31	1	8	8	62
Black	10	5	1	10	2	20	1	10	6	60
Asian	10	5	2	20	0	0	4	40	4	40
Pacific Islander	6	3	1	60	1	60	0	0	4	67
Free or reduced-price lunch
Eligible	100	47	19	40	23	72	12	27	46	52
Parent gender
Female	194	92	44	23	28	14	41	21	81	42
Parent education level
Less than high school/GED	8	4	1	13	0	0	3	38	4	50
High school/GED	49	23	10	20	6	12	8	16	25	51
Partial college	53	25	8	15	9	17	10	19	26	49
Junior college/Associate’s	42	20	9	21	6	14	7	17	20	48
Standard college	38	18	10	26	10	26	8	21	10	26
Graduate/professional	31	15	8	26	4	13	11	35	8	26
Alternative caregiver present	165	78	41	25	23	14	42	25	59	36
Primary language spoken
English	198	93	44	22	30	15	41	21	83	42
Spanish	7	3	2	29	2	29	0	0	3	43

Note. GED = General Educational Development.

Parents could select multiple responses.

Measures

SAEBRS-P

Parents completed the pre-factor analyzed SAEBRS-P, which consisted of 21 items. The SAEBRS-P was designed to be consistent with the SAEBRS-T; thus, 19 SAEBRS-P items were generated that were highly similar to the 19 SAEBRS-T. These items spanned across the same three subscales from the SAEBRS-T, including Social Behavior, Academic Behavior, and Emotional Behavior. These 19 items were similar across measures in terms of content and structure, with some content modified to make items more relevant to parents and the home setting. For example, “difficulty working independently” was modified to “difficulty completing academic material independently.” In addition, two items were incorporated into the SAEBRS-P to assess the student’s completion of academic work in the home. To complete the SAEBRS-P, parents or other primary caregivers rated the frequency of behaviors during the previous month on a 4-point Likert-type scale (0 = never to 3 = almost always).

SDQ

The SDQ (Goodman, 1997) is a universal screening tool designed to identify behavioral strengths and deficits for students’ aged 3 to 17 years (Goodman, 2001). It was chosen as the criterion measure within this investigation given its theoretical alignment with the SAEBRS and its relevance to similar constructs and behaviors (e.g., SAEBRS Emotional Behavior and SDQ Emotional Problems). The SDQ contains 25 items that an informant rates regarding the extent to which each item is representative of student’s behavior (0 = not true to 2 = certainly true). Items are equally distributed across five subscales: Emotional Symptoms, Conduct Problems, Hyperactivity-Inattention, Peer Problems, and Prosocial Behavior. Negatively worded items are reverse scored and then all items are summed to generate scale scores. Higher scores indicate higher levels of behavioral and emotional risk. This is with the exception of the Prosocial Behavior subscale, for which higher scores are indicative of greater prosocial skills. Scores from the first four subscales are summed to create a Total Difficulties score. Prior research has supported SDQ Parent test–retest and internal consistency reliability, as well as the measures validity relative to general and specific measures of psychopathology (e.g., Child Behavior Checklist; Stone et al., 2010).

Procedures

This sample represented one of convenience, with the participating district expressing interest in the current research project. Schools were recruited at the district level. Interested schools distributed packets containing consents, questionnaires, and postage-paid envelopes to all families in kindergarten to fourth grade through each school’s typical home–school communication mechanism. Packets were distributed in May 2014 and the last packet was received by July 2014. Questionnaires were sent home with students in a weekly school folder. For three of the schools, parents returned surveys via a prepaid and stamped envelope directly to researchers. One school had parents return questionnaires to a confidential location at the school that only researchers could access. Parents had the option of participating in a drawing for one of five US$20 gift cards. School 4 chose to give students of families who returned a packet a school wrist band. Parents completed the SAEBRS-P and the SDQ on their oldest or only child in kindergarten to fourth grade. These grades were selected as part of a larger study investigating parent–teacher and home–school relationships. The overall response rate was 13.85%.

Data Analysis

Data were analyzed across two phases. First, a CFA was conducted using a maximum likelihood estimator. Three theory-driven factor models of increasing complexity were considered. The first was a unidimensional model, wherein all items loaded on a single factor indicative of broader social-emotional and behavioral functioning. The second model was the aforementioned three-factor model, which specified Social Behavior, Academic Behavior, and Emotional Behavior factors. The third model was a four-factor model, with constituent factors conceptualized as (a) Social-Emotional Skills (positively worded items from the original Social Behavior and Emotional Behavior subscales), (b) Academic Enablers (positively worded items from the original Academic Behavior subscale), (c) Externalizing Problems (negatively worded items from the Social Behavior and Academic Behavior subscales), and (d) Internalizing Problems (negatively worded items from the Emotional Behavior subscale). To note, if none of the above factor structures yielded adequate fit, more complex and theoretically relevant factor structures would be considered.

The latter two models represented correlated factor structures, wherein latent factors were permitted to covary. Each model’s fit was considered relative to multiple fit statistics, including root mean square error of approximation (RMSEA; with corresponding 90% confidence interval [CI]), comparative fit index (CFI), Tucker–Lewis Index (TLI), and standardized root mean square residual (SRMR). Criteria for acceptable fit were defined as RMSEA < .08, CFI and TLI > .90, and SRMR < .08 (Little, 2013). Standardized factor loadings, which range from 0 to 1 and are interpreted in a manner consistent with linear regression beta weights, were reported and interpreted for the factor model affording best fit. The use of CFA was considered appropriate within the current context, as researchers have called for samples of at least 200 when conducting structural equation modeling procedures like CFA (Kline, 2011; Wolf, Harrington, Clark, & Miller, 2013).

In the second phase, the internal consistency reliability was evaluated for each factor across both factor models via coefficient omega (with corresponding 95% CIs). Omega was preferred over the more commonly reported coefficient alpha given the presumed potential for item multidimensionality, as well as the latent nature of SAEBRS-P scores (McDonald, 1999). In accordance with previously specified interpretive guidelines, omega values >.50 were considered acceptable, whereas values >.75 were preferred (Reise et al., 2013). Finally, the concurrent criterion-related validity of each resulting SAEBRS-P factor, as it pertained to the SDQ, was analyzed using Pearson’s bivariate correlations. The correlations were evaluated in terms of statistical significance and relative to Cohen’s (1988) criteria for correlational magnitude (.10, small; .30, medium; and .50, large). All reliability and validity analyses were conducted with R Version 3.2.3 using the MBESS (Kelley, 2017) and psych (Revelle, 2018) packages, whereas CFAs were conducted with Mplus Version 7.3.

Results

A review of the data indicated that complete data were available for 201 of the 212 student participants, with 11 participants missing data on at least one SDQ or SAEBRS-P item. Two approaches were taken to the handling of this missing data. First, within each CFA, missing data were handled using full information maximum likelihood (FIML). Second, listwise deletion was used within reliability and correlational analyses. Given that the extent of missing data was small (i.e., <5%), listwise deletion was deemed an appropriate approach (Kline, 2011).

CFA

Results suggested the one-factor model yielded poor fit to the SAEBRS-P data, with none of the four fit statistics meeting acceptability thresholds (RMSEA = .12, 90% CI = [.11, .13]; CFI = .64; TLI = .60; SRMR = .10). The three-factor model also afforded poor fit, with none of the four fit statistics suggesting adequate fit (RMSEA = .10, 90% CI = [.09, .11]; CFI = .78; TLI = .75; SRMR = .09). The four-factor model was found to yield relatively superior fit, with two of four fit statistics exceeding their acceptability thresholds (RMSEA = .08, 90% CI = [.07, .09]; CFI = .85; TLI = .82; SRMR = .07). Modification indices were evaluated in determining whether any constrained parameters should be freely estimated toward the improvement of fit. The decision to freely estimate an item residual covariance was considered appropriate if (a) the items were within the same subscale, (b) the covariance is in accordance with theory (e.g., given that two items correspond to similar, highly related behaviors), and (c) the modification index suggests substantial benefit in model fit via estimation of the parameter (e.g., modification index > 50). Overall, results supported the modeling of four item residual covariances. This revised four-factor model afforded good fit, with three of four fit statistics exceeding acceptability thresholds (RMSEA = .07, 90% CI = [.06, .08]; CFI = .90; TLI = .89; SRMR = .07). Given this demonstrated fit, a decision was made to not consider any additional, more complex factor structures. See Table 2 for standardized factor loadings associated with this model.

Table 2.

Standardized Factor Loading Matrix.

	Social-Emotional	Academic Enablers	Externalizing Problems	Internalizing Problems
Cooperation with peers	.60
Polite and socially appropriate responses toward others	.70
Following instructions	.81
Adaptiveness to change	.49
Positive attitude	.54
Producing acceptable work		.81
Interest in academic topics		.78
Organizing academic materials		.71
Academic engagement		.78
Timely completion of academic work		.70
Disruptive behavior			.55
Impulsiveness			.55
Arguing			.62
Temper outbursts			.62
Difficulty completing work independently			.44
Distractedness when completing academic work			.57
Sadness				.65
Fearfulness				.65
Worry				.80
Withdrawal				.70
Difficulty rebounding from setbacks				.44

Note. All standardized factor loadings were statistically significant at the p < .001 level.

Internal Consistency

In light of the CFA findings, we elected to report coefficient omega internal consistency estimates specific to the supported four-factor model. Each of the factors within the four-factor model also met acceptability thresholds, including Social-Emotional Skills (.75; 95% CI = [.68, .82]), Academic Enablers (.86; 95% CI = [.82, .90]), Externalizing Problems (.74; 95% CI = [.67, .81]), and Internalizing Problems (.78; 95% CI = [.72, .84]).

Concurrent Validity

Bivariate correlations were calculated to examine the concurrent criterion-related validity of each of the four SAEBRS-P factors, as they pertained to the SDQ. Factor scores were calculated by taking the sum of all items within each of the four scales. All correlations between SAEBRS-P and SDQ were statistically significant at the p < .001 level. This was with the exception of the correlation between SAEBRS Academic Enablers and SDQ Peer Problems, which was not statistically significant at the p < .05 level. The SAEBRS-P factors tended to be most highly correlated with the SDQ Overall Difficulties scale (r = .40-.65; absolute values). See Table 3 for all bivariate correlations between SAEBRS-P and the SDQ.

Table 3.

Pearson Product-Moment Correlation Coefficients.

SDQ scale	SAEBRS scale
SDQ scale	Social-Emotional Skills	Academic Enablers	Externalizing Problems	Internalizing Problems
Hyperactivity	−.51*	−.39*	.61*	.29*
Emotional Symptoms	−.44*	−.28*	.38*	.64*
Prosocial	.48*	.23*	−.35*	−.31*
Conduct Problems	−.37*	−.24*	.44*	.29*
Peer Problems	−.27*	−.09	.26*	.38*
Overall Difficulties	−.61*	−.40*	.65*	.60*

Note. SDQ = Strengths and Difficulties Questionnaire; SAEBRS = Social, Academic, and Emotional Behavior Risk Screener.

p < .001.

Discussion

Previous studies have examined the use of the SAEBRS-T, and found three scales corresponding to social, academic, and emotional behavior problems (Kilgus et al., 2013; Kilgus et al., 2015). Research suggests the potential benefits of using multi-informants when identifying behavior difficulties in schools. Parents and teachers have specifically been found to be optimal informants of externalizing behavior in younger children (Smith, 2007). Therefore, this study sought to develop and initially validate the SAEBRS-P. It was anticipated that the measure would yield a small number of items falling under the original three factors, and that items retained would show adequate internal consistency.

Although the four retained factors are not identical to the original hypothesis of three factors, this interpretation provides similar information regarding student’s social (social skills, externalizing behavior), academic (academic enablers, attention problems), and emotional behavior (internalizing behavior). Interestingly, whereas factor analyses in previous SAEBRS-T studies suggested maladaptive and adaptive behaviors could be grouped together under broader domain factors, the current analyses suggested that such items be differentiated, resulting in factors that were adaptive or maladaptive. It is possible that the finding is representative of a measurement artifact. Prior research suggests raters are likely to respond to items differently depending on item valence (i.e., the positive or negative tone of the items; Schmitt & Stults, 1985). Regardless, resulting factor structure could still be said to be in accordance with the hypothesized structure, with factors instead corresponding to the narrower domains rather than broader domains.

The second purpose of this study was to examine the concurrent criterion-related validity of SAEBRS-P. Findings indicated moderate to high correlations between SAEBRS-P and the SDQ scales, suggesting the concurrent validity of the SAEBRS-P. Results suggested SAEBRS-P scores correlated best with the Overall Difficulties scale of the SDQ. Further examination of the remaining correlations indicated the highest correlations tended to be between subscales examining similar constructs. Overall, these findings suggest the SAEBRS-P correlates well with the SDQ. Furthermore, findings suggest the construct validity of the various SAEBRS-P subscales is promising, as the pattern of convergent relationships was found to be in accordance with theory-driven expectations.

Taken together, the overall purpose of this study was to provide initial support for a novel screening tool for parents of children in elementary school. Preliminary results suggest the screener might be used to identify behavior problems along five subscales: Academic Enablers, Internalizing Problems, Externalizing Problems, and Social-Emotional Skills. Given the initial support described above, the use of the SAEBRS-P as a universal screener for elementary students is promising. Regardless, additional research examining the validity of this measure is needed to further support the use of the SAEBRS-P in schools.

Implications for Practice

Parents serve as optimal informants of elementary children’s behavior (Smith, 2007). The current study is promising for the use of the novel universal screener, SAEBRS-P, for use with elementary students. The SAEBRS is unique in that it provides a parent’s perspective on academic enablers in addition to internalizing and externalizing problems. Specifically, school staff can use data from the SAEBRS-P to help further provide supports to students in more specific areas, such as skill support for students with difficulties in the classroom (i.e., academic enablers). Supports can also be provided in other specific areas such as social-emotional skill development and supports for externalizing and internalizing behavior difficulties. By collecting data that includes the parent perspective, schools have the unique opportunity to improve their MTSS prevention efforts because they will be able to make timely and accurate decisions about student risk, particularly in relation to externalizing behaviors exhibited by younger students, without having to wait until a teacher has time to get to know their students.

Another important implication of the SAEBRS-P is its ability to offer schools data from a unique perspective, the parent. Parents are often asked to complete ratings of their child’s behavior for special education services, but parents as informants of behavior do not occur frequently in the general education setting. This novel screening tool may provide an avenue for parental input of their child’s overall behavior and open up a line of communication between families and the school. Future research is needed to further support the use of the SAEBRS-P in schools, but currently, the results support preliminary evidence of the SAEBRS-P ability to identify students at risk for problem behavior across four subscales: Academic Enablers, Internalizing Problems, Externalizing Problems, and Social-Emotional Skills.

Limitations and Future Research

This study provides initial support for the use of the SAEBRS-P; however, it is not without limitations. First, it should be noted these results likely only apply to the current sample. Future research should examine the use of this measure across different geographic areas to support generalizability. Similarly, more research is needed to support applied use of the SAEBRS-P. Continued research is necessary for various psychometric characteristics. Second, the current response rate was relatively low (13.85%). This suggests the potential for sampling bias and the likelihood that findings and conclusions regarding SAEBRS-P psychometric defensibility might differ in other samples. Similar research (Becker, Woerner, Hasselhorn, Banaschewski, & Rothenberger, 2004; Dowdy et al., 2011; Whiteside, McCarthy, & Miller, 2007) does not report response rates. Therefore, it is difficult to compare our response rate with other studies of this nature. Regardless, both this study and our applied experience highlight such challenges with parent-based universal screening. Researchers have begun to consider strategies by which to promote parental response rates. A recent study reported high response rates (range = 62%–87%) when including parent screeners within student registration packets (Moore et al., 2016). Although yet to be specifically evaluated, we anticipate response rates would also improve within broader family–school partnership frameworks that include multidirectional communication across home and school (Dishion, 2011; Garbacz et al., 2016). Further research therefore remains necessary.

Finally, given that the current study considered SAEBRS-P performance in isolation, future studies might consider SAEBRS-P incremental validity, or the extent to which the measure contributes information above and beyond alternative screening tools. Third, both the predictor and outcome measures were parent rating scales, which makes this study susceptible to mono-informant and mono-method biases (Goodman, 1997, 2001). Finally, a CFA with a new sample is needed to support for the current structure identified in this study.

Summary

This study sought to develop and initially validate the SAEBRS-P within an elementary sample. Four factors were retained to provide information regarding student’s social-emotional skills, academic enablers, externalizing behavior, and internalizing behavior. Further evidence was found for the internal consistency of each of these scales, as well as the concurrent criterion-related validity of the SAEBRS-P relative to the SDQ Parent Form. Overall, this study provides initial support for a novel screening tool for parents of children in elementary school. Future research with larger and more representative samples remains necessary to support the use of the SAEBRS-P within applied settings.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Achenbach

T. M.

McConaughy

S. H.

Howell

C. T.

(1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232. doi:10.1037/0033-2909.101.2.213.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Becker

Woerner

Hasselhorn

Banaschewski

Rothenberger

(2004). Validation of the parent and teacher SDQ in a clinical sample. European Child & Adolescent Psychiatry, 13, 11–16. doi:10.1007/s00787-004-2003-5.

Bruhn

A. L.

Woods-Groves

Huddle

(2014). A preliminary investigation of emotional and behavioral screening practices in K-12 schools. Education and Treatment of Children, 37, 611–634. doi:10.1353/etc.2014.0039.

Cohen

(1988). Statistical power analysis for the behavior sciences (2nd ed.). New York, NY: Academic Press.

Cook

C. R.

Volpe

R. J.

Livanis

(2010). Constructing a roadmap for future universal screening research beyond academics. Assessment for Effective Intervention, 35, 197–205. doi:10.1177/1534508410379842.

Cooper

J. O.

Heron

T. E.

Heward

W. L.

(2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson.

De Los Reyes

Augenstein

T. M.

Wang

Thomas

S. A.

Drabick

D. A.

Burgers

D. E.

Rabinowitz

. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141, 858–900.

De Los Reyes

Kazdin

A. E

. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483–509.

10.

Dishion

(2011). Promoting academic competence and behavioral health in public schools: A strategy of systematic concatenation of empirically based intervention principles. School Psychology Review, 40, 590–597.

11.

Dowdy

Chin

J. K.

Twyford

J. M.

Deverm

B. V.

(2011). A factor analytic investigation of the BASC-2 Behavioral and Emotional Screening System Parent Form: Psychometric properties, practical implications, and future directions. Journal of School Psychology, 49, 265–280. doi:10.1016/j.jsp.2011.03.005.

12.

Dowdy

Kamphaus

R. W.

Abdou

A. S.

Twyford

J. M.

(2013). Detection of symptoms of prevalent mental health disorders of childhood with the parent form of the Behavioral and Emotional Screening System. Assessment for Effective Intervention, 38, 192–198. doi:10.1177/1534508412447009.

13.

Dowdy

Kim

(2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6, 1–10.

14.

Eklund

Dowdy

(2014). Screening for behavioral and emotional risk versus traditional school identification methods. School Mental Health, 6, 40–49. doi: 10.1007/s12310-013-9109-1.

15.

Eklund

Renshaw

T. L.

Dowdy

Jimerson

S. R.

Hart

S. R.

Jones

C. N.

Earhart

(2009). Early identification of behavioral and emotional problems in youth: Universal screening versus teacher-referral identification. The California School Psychologist, 14, 89–95. doi:10.1007/BF03340954.

16.

Every Student Succeeds Act. (2015). S.1177. Retrieved from https://www.gpo.gov/fdsys/pkg/BILLS-114s1177enr/pdf/BILLS-114s1177enr.pdf

17.

Garbacz

S. A.

McIntosh

Eagle

J. W.

Dowd-Eagle

S. E.

Hirano

K. A.

Ruppert

(2016). Family engagement within schoolwide positive behavioral intervention and supports. Preventing School Failure, 60, 60–69. doi:10.1080/1045988X.2014.976809.

18.

Gerber

M. M.

Semmel

M. I.

(1984). Teacher as imperfect test: Reconceptualizing the referral process. Educational Psychologist, 19, 137–148. doi:10.1080/00461528409529290.

19.

Glover

T. A.

Albers

C. A.

(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. doi: 10.1016/j.jsp.2006.05.005.

20.

Goodman

(1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. doi:10.1111/j.1469-7610.1997.tb01545.x.

21.

Goodman

(2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337–1345. doi:10.1097/00004583-200111000-00015.

22.

Gresham

F. M.

Elliot

S. N.

(2008). Social skills improvement system: Rating scales. Bloomington, MN: Pearson.

23.

Hornman

Kerstjens

J. M.

Winter

A. F.

Bos

A. F.

Reijneveld

S. A.

(2013). Validity and internal consistency of the ages and stages questionnaire 60-month version and the effort of three scoring methods. Early Human Development, 89, 1011–1015. doi: 10.1016/j.earlhumdev.2013.08.016.

24.

Hunsley

Mash

E. J.

(2007). Evidence-based assessment. Annual Review of Clinical Psychology, 3, 29–51.

25.

Individuals With Disabilities Education Improvement Act of 2004. (2004). Pub. L., 108-446, 118 Stat. 2647.

26.

Jenkins

J. R.

Hudson

R. F.

Johnson

E. S.

(2007). Screening for at-risk readers in a response to intervention framework. School Psychology Review, 36, 582–600.

27.

Kamphaus

R. W.

Reynolds

C. R.

(2015). Behavior assessment system for children, second edition (BASC-2): Behavioral and emotional screening system (BESS). Bloomington, MN: Pearson.

28.

Kelley

(2017). MBESS (Version 4.0.0 and higher) [Computer software and manual]. Available from http://cran.r-project.org

29.

Kilgus

S. P.

Chafouleas

S. M.

Riley-Tillman

T. C.

(2013). Development and initial validation of the Social and Academic Behavior Risk Screener for elementary grades. School Psychology Quarterly, 28, 210–226.

30.

Kilgus

S. P.

Sims

W. A.

von der Embse

N. P.

Riley-Tillman

T. C.

(2015). Confirmation of models for interpretation and use of the Social and Academic Behavior Risk Screener (SABRS). School Psychology Quarterly, 30, 335–352.

31.

Kilgus

S. P.

Sims

W. A.

von der Embse

N. P.

Taylor

C. N.

(2016). Technical adequacy of the Social, Academic, and Emotional Behavior Risk Screener in an elementary sample. Assessment for Effective Intervention, 42, 3–5. doi: 10.1177/1534508415623269.

32.

Kilgus

S. P.

von der Embse

N. P.

(2014). Unpublished technical manual of the Social, Academic, and Emotional Behavior Risk Screener.

33.

Kline

R. B.

(2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

34.

Lane

K. L.

Menzies

H. M.

Oakes

W. P.

Kalberg

J. R.

(2012). Systematic screening of behavior to support instruction: From preschool to high school. New York, NY: Guilford Press.

35.

Little

T. D.

(2013). Longitudinal structural equation modeling (methodology in social sciences). New York, NY: Guilford Press.

36.

McDonald

R. P.

(1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

37.

Miller

F. G.

Cohen

Chafouleas

S. M.

Riley-Tillman

T. C.

Welsh

M. E.

Faviano

G. A.

(2015). A comparison of measures to screen for social, emotional, and behavioral risk. School Psychology Quarterly, 30, 184–196.

38.

Moore

K. J.

Garbacz

Gau

J. M.

Dishion

T. J.

Brown

K. L.

Stormshak

E. A.

Seeley

J. R.

(2016). Proactive parent engagement in public schools: Using a brief strength and needs assessment in a multiple-gating risk management strategy. Journal of Positive Behavior Interventions, 18, 230–240. doi:10.1177/1098300716632590.

39.

Predy

McIntosh

Frank

J. L.

(2014). Utility of number and type of office discipline referrals in predicting chronic problem behavior in middle schools. School Psychology Review, 43, 472–489. doi:10.17105/SPR-13-0043.1.

40.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129–140. doi:10.1080/00223891.2012.725437.

41.

Revelle

(2018). Procedures for psychological, psychometric, and personality research (R package Version 1.4.5) [Computer software manual]. Available from http://personality-project.org/

42.

Schmitt

Stults

D. M.

(1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9, 367–373. doi:10.1177/014662168500900405.

43.

Schonhaut

Armijo

Schonstedt

Alvarez

Cordero

(2013). Validity of the Ages and Stages Questionnaires in term and preterm infants. Pediatrics, 131, 1468–1474. doi:10.1542/peds.2012-3313.

44.

Smith

S. R.

(2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Psychoeducational Assessment, 25, 139–149. doi:10.1177/0734282906296233.

45.

Squires

Bricker

Potter

(2009). Ages & Stages Questionnaires, Third Edition (ASQ-3) user’s guide. Baltimore, MD: Paul H. Brookes.

46.

Stone

L. L.

Otten

Engels

R. C. M. E.

Vermulst

A. A.

Janssens

J. M. A. M

. (2010). Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4 to 12 year-olds: A review. Clinical Child and Family Psychology Review, 13, 254–274. doi:10.1007/s10567-010-0071-2.

47.

Sugai

Horner

R. H.

(2006). A promising approach to expanding and sustaining school wide positive behavior support. School Psychology Review, 35, 245–256.

48.

Sugai

Sprague

J. R.

Horner

R. H.

Walker

H. M.

(2000). Preventing school violence: The use of office discipline referrals to assess and monitor school-wide discipline interventions. Journal of Emotional and Behavioral Disorders, 8, 94–101. doi:10.1177/106342660000800205.

49.

von der Embse

N. P.

Kilgus

S. P.

Iaccarino

Levi-Nielsen

. (2017). Screening for student mental health risk: Diagnostic accuracy, measurement invariance, and predictive validity of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale (SAEBRS-SRS). School Mental Health, 9, 273–283. doi:10.1007/s12310-017-9214-7.

50.

von der Embse

N. P.

Laccarino

Mankin

Kilgus

S. P.

Magen

. (2017). Development and validation of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale. Assessment for Effective Intervention, 42, 186–192. doi:10.1177/1534508416679410.

51.

Walker

H. M.

Severson

H. H.

Feil

E.G.

(2014). Systematic screening for behavior disorders (SSBD) technical manual: Universal screening for preK–9 (2nd ed.). Eugene, OR: Pacific Northwest Publishing.

52.

Whiteside

S. P.

McCarthy

D. M.

Miller

J. D.

(2007). An examination of the factor structure of the social skills rating system parent elementary form. Assessment, 14, 246–254. doi:10.1177/1073191107302062.

53.

Wolf

E. J.

Harrington

K. M.

Clark

S. L.

Miller

M. W.

(2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. Educational and Psychological Measurement, 73, 913–934