An Investigation of the Psychometric Properties of the Early Identification System

Abstract

An identified barrier to the widespread adoption of universal socioemotional and behavioral screening in schools may be that existing instruments may be too burdensome or costly to administer. As a result, the County Schools Mental Health Coalition came together to develop a common assessment system, the Early Identification System (EIS). The purpose of this study was to evaluate the psychometric properties of the EIS-student report, using a sample of third- to eighth-grade students. A series of exploratory and confirmatory factor analytic models were used with one exploratory (n₁ = 450) and two holdout samples (n₂ = 450, n₃ = 940). A six correlated factor model was supported showing adequate model fit, reliability (range = .73-.90), and convergent validity. Results suggest that the EIS is a promising, cost-free, and quick-to-administer universal screening tool that may assist educators and school psychologists identify students at risk of future socioemotional and behavioral difficulties.

Keywords

universal screening prevention mental health behavior assessment factor analysis

A large number of students arrive in U.S. schools everyday with unaddressed mental health problems. On average, 13% to 20% of children living in the United States experience a mental health problem in any given year, and surveillance data between 1994 and 2011 suggest these prevalence rates are increasing (Centers for Disease Control and Prevention, 2013). When not effectively addressed, these early signs and symptoms culminate in a cascade of poor educational and life-course outcomes (Reid, Patterson, & Snyder, 2002). For example, early social, emotional, and behavioral challenges increase the likelihood of peer and teacher rejection and repeated removals from class (National Association of School Psychologists, 2010). Social isolation and repeated removals increase the odds of academic disengagement, course failures, and being referred for special education services (Hall-Lande, Eisenberg, Christenson, & Neumark-Sztainer, 2007), which increases the risk of retention and dropout (Nelson, Stage, Duppong-Hurley, Synhorst, & Epstein, 2007). Although preventive school-based interventions may mitigate this cascade of poor outcomes, an estimated 80% of youth struggling with the early signs and symptoms of oncoming social, emotional, and behavioral health problems never receive access to effective support services (Essau, 2005; Kataoka, Zhang, & Wells, 2002).

School, next to home, is the primary setting where children learn essential intra- and interpersonal skills. Schools also have an increasing role in implementing preventive programs (Burns & Rapee, 2016). To efficiently implement a range of effective school-based socioemotional supports that are necessary for academic achievement, U.S. schools have begun to adopt tiered prevention models. Tiered prevention models are a systematic, data-supported approach to providing increasingly intensive services/interventions to students who have a demonstrated need based on universal screening data (Lane, Carter, Jenkins, Dwiggins, & Germer, 2015). Tiered prevention models—similar to Response to Intervention (RtI) and Positive Behavior Interventions and Supports (PBIS)—rely on data to target intervention efforts at the universal or school level and at targeted or group/individual student levels (A. Thompson, Reinke, & Herman, 2014). Randomized studies reveal that when properly implemented, tiered prevention models may increase school safety, social and behavioral functioning, and academic achievement (d = 0.38; Horner et al., 2009).

Properly implemented tiered prevention models are dependent upon accurate and reliable data to inform decision making (Severson, Walker, Hope-Doolittle, Kratochwill, & Gresham, 2007). For example, school personnel should rely on screening data to identify at-risk students. School personnel also rely on screening data to select an appropriate range of scientifically supported programs and practices to mitigate identified risks. In addition, school personnel can compare universal screening scores alongside other sources of outcome data to appraise the impact of prevention efforts over time (A. Thompson et al., 2014). Although universal screening for social–emotional risk factors is foundational to developing a successful tiered prevention model, the practice has yet to become widely adopted in U.S. schools (Cook, Volpe, & Livanis, 2010; Volpe, Briesch, & Chafouleas, 2010).

Screening: Definition, Purpose, and Characteristics

Applied in schools, universal screening is conducted with the entire population of students in a school district or building (Severson & Walker, 2002). Scientifically based supports provided to students identified to be at risk have been shown to mitigate some of the risks experienced (Glover & Albers, 2007). To screen all students efficiently, a screening tool for mental health concerns are typically simple to administer, are completed by a reliable reporter, rely on a brief response format, focus on key risk factors, and yield sensitive results that are easy to interpret (Connolly, Bernstein, & Work Group on Quality Issues, 2007). In this way, screening differs from assessment, in that, screening identifies emergent concerns and the specific individuals who may endorse experiencing those concerns, whereas assessments are more comprehensive and may be multidomain investigations of specific identified problems. Identifying risk factors early permits comprehensive prevention and intervention supports to be designed and delivered before these conditions worsen (A. Thompson et al., 2014). The widespread adoption and implementation though of universal screening for social, emotional, and behavioral risk factors in schools remain limited for several reasons (Cook et al., 2010; Levitt, Saka, Romanelli, & Hoagwood, 2007).

Some Barriers to Screening Implementation

Common barriers to social, emotional, and behavioral screening in U.S. schools include time and cost, and overidentification of students (Volpe et al., 2010). Many currently available screening tools are extremely time-consuming (Dowdy, Ritchey, & Kamphaus, 2010). A possible reason for the lengthy administration is that school-based screening instruments often exceed the basic criteria of what is necessary to accurately screen an entire population of students (Burns & Rapee, 2016). Second, these screeners come at a significant cost; commercial screeners may cost about one dollar per protocol, not counting the start-up costs (e.g., purchasing manuals, scoring programs). As a recurring cost, schools are required to budget both the time and resources to administer the screeners on a regular basis. Third, school personnel express concern that too many children are overidentified by these tools (Volpe et al., 2010). Overidentification places a huge burden on schools to provide the supports to students when resources are limited. To address some of these concerns, the County Schools Mental Health Coalition (CSMHC)—a cooperative of six school districts, private schools, and University of Missouri researchers—developed the Early Identification System (EIS).

Targeted Risk Factors of the EIS

Risk indicators, which were predictive of negative social, emotional, and academic outcomes, were identified from the extant literature and guided by the developmental behavioral cascades theory (Cicchetti & Tucker, 1994; Patterson, Reid, & Dishion, 1992). Developmental behavioral cascades theory posits that symptoms of social, emotional, and behavioral problems can interfere with adaptive functioning. For instance, disruptive behavior in the classroom interferes with learning and alienates peers or teachers. Failures in adaptive functioning can lead to further symptoms (e.g., students who experience academic failure may develop internalizing problems; Herman, Lambert, Ialongo, & Ostrander, 2007). Thus, competence in age-salient developmental tasks plays a central role in the long-term risks for development associated with emotional and behavioral problems. These two substantive ways that competence and symptoms influence each other could occur simultaneously or in various combinations over time, producing phenomena described in terms of a cascade, or progressively cumulative and more negative outcomes (Patterson et al., 1992). As such, identifying early signs of risk and providing timely interventions can interrupt these negative developmental cascades.

Specifically, relational aggression, problems with academic competence, social skill deficits, problems with peer relationships, internalizing problems, externalizing problems, and problems with school engagement were seven constructs of interest given the relationship of these factors predicting poor behavioral and academic outcomes. For example, externalizing behavior, peer rejection, and social competence deficits in childhood are correlated with negative developmental outcomes in adolescence and young adulthood (Taylor, Davis-Kean, & Malanchuk, 2007; Tremblay et al., 2004). Early externalizing problems are one of the strongest predictors of later academic failure, delinquency, and drug use (Odgers et al., 2008; Temcheff et al., 2008). In addition, depression and anxiety symptoms (forms of internalizing behaviors) in youth often go unnoticed and most (up to 80%) do not receive any services for their symptoms (Lewinsohn, Rohde, & Seeley, 1998). Given the range of negative outcomes associated with internalizing problems such as suicide, poor academic performance, and substance use (Reid, Gonzalez, Nordness, Trout, & Epstein, 2004), there is an important need to identify and support youth experiencing internalizing problems.

In developing the EIS, the authors initially constructed a pool of items related to each area of risk reviewed. Discussion of the selected items occurred between researchers and practitioners working with the coalition to ensure the items captured symptoms of the key hypothesized risk constructs. The items selected (e.g., I have friends to eat lunch with) were related to well-known but malleable risk factors (e.g., peer rejection).

The Current Study

The risk indicators targeted have strong associations with significant negative academic, behavioral, emotional, and social outcomes. Early identification of children in need of further intervention is often considered a first step in a problem-solving process (Glover & Albers, 2007). To this end, the EIS-student report was developed and piloted in the partner schools. The purpose of this study was to investigate the factor structure, internal consistency, and convergent validity of the instrument among students in Grades 3 to 8.

Method

Participants

The current study was conducted across six schools (i.e., three elementary schools, two middle schools, and one private school with K-8 enrollment) in three school districts. Schools that participated in this study were recruited to pilot the EIS for the initial development of the measure. Students in Grades 3 to 8 who took the survey in the fall and/or winter of the school year were included in the study. A total of 1,592 participants were recruited. Two students had missing data and were excluded from the study (<0.01%).

In total, 1,590 students (47% = female, 45% = male, 8% = unknown) provided 1,840 complete screener responses (250 students provided responses from both fall and spring) within one school year (see Table 1). Participants were 86% White, 6% non-White, and 8% of unknown race/ethnicity. Approximately 61% were in elementary school (Grades 3-5) and 39% were in middle school (Grades 6-8).

Table 1.

Sample Composition by School and Screening Window (n = 1,840).

School	Fall	Winter	Total	%White	%FRPM
Private School	396	261	657	73^a	NA
Public Elementary A	307	0	307	90	30
Public Elementary B	0	352	352	91	19
Public Middle B	0	327	327	91	14
Public Elementary C	69	0	69	94	43
Public Middle C	128	0	128	89	42
Total	900	940	1,840

Note. Letters A, B, and C correspond to anonymized school districts. %White refers to percent of White students. %FRPM refers to the percent eligible for free or reduced price meals. NA = not applicable.

Another 18% were missing race/ethnicity information but were most likely White based on the school’s demographic profile.

Measures

The EIS used in this investigation was created through a two-step process. First, prior to item development, each of the authors met to discuss the purpose of the items and the hypothesized factors to which it should correspond. An initial item pool was reviewed by measurement experts in the field and by school personnel participating in the EIS survey. This review resulted in an item pool of 41 questions (see Figure 1 for a complete list of items and the distribution of responses). Response options were Likert-type scales (0 = never, 1 = sometimes, 2 = often, 3 = always). The items were hypothesized to correspond to factors of externalizing behavior, internalizing behavior, peer relations, school engagement, emotional regulation, academic competence, and social skills deficits. Four of the items were identified as risk factors on their own (i.e., “Other kids make fun of me at school,” “I am bullied by others,” “I get into fights with others,” and “I am late to school”) and were not included in the factor analyses.

Figure 1.

Distribution for all item responses (percentage never vs. sometimes, often, always) using the winter confirmatory sample (n = 940).

Screening Procedures

The EIS was administered in an online format to students in the participating schools. A note was sent home to parents explaining the purpose of the EIS and how the information would be used by schools with an option to opt out their child. No parent opted out their child. Students completed the EIS in the fall (October) and/or the winter (January) of the school year. Students completed the measure during an identified period of the school day. The items were read aloud to the students by a teacher or school mental health practitioner using a script that explained the purpose of the EIS, how the information will be used, and then provided a debriefing at the end. Contact information was also offered to students if they wanted to further discuss the information with an adult in the building. On average, the students completed the measure within 12 to 15 min.

Analytic Strategy

Items were hypothesized to measure the seven constructs of peer relations, internalizing behaviors, externalizing behaviors, emotional regulation, relational aggression, academic competence, and school engagement. The factor structure was investigated using a sequence of exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) models. Analyses were conducted over a series of four phases. Prior to analysis, the overall fall sample (n = 900) was randomly split in two to form an exploratory (n = 450) and a confirmatory (n = 450) sample. In the first phase, EFA was used to investigate the underlying factors using the items on the EIS using the exploratory sample. EFA is a commonly used technique for uncovering relationships between sets of variables with the goal of better understanding the underlying structure of these variables in terms of common themes or factors (Fabrigar & Wegener, 2012). In the factor analyses, a polychoric correlation matrix was used together with weighted least squares with mean and variance correction (WLSMV) estimation, which accounted for the ordinal nature of the data (Finney & DiStefano, 2006). An oblique (geomin) rotation was considered as factors are often correlated (Tabachnick & Fidell, 2013). Both EFA and CFA were conducted using R with the psych (Revelle, 2015) and the lavaan (Rosseel, 2012) packages, respectively.

For the EFA, the resulting factor structures were assessed using several criteria as recommended by B. Thompson and Daniel (1996). Factor solutions were evaluated based on the following criteria: (a) The number of factors retained should be consistent with parallel analysis (Horn, 1965). Parallel analysis has been referred to as one of the most accurate methods to determine the number of factors to retain (Fabrigar & Wegener, 2012). (b) Each rotated factor should have at least three appreciable factor coefficients (i.e., > .35). (c) Resulting dimensions should demonstrate acceptable reliability (i.e., omega > .70; Cicchetti, 1994). The EFA was performed in an iterative manner using the criteria specified where items that did not have appreciable factor loadings were excluded and the EFA procedures repeated. Items with loadings on more than one factor were revisited and were either excluded from the analysis or grouped with a particular factor based on our theoretical understanding of the factor and the strength and direction of the factor loadings.

The second phase focused on performing CFAs, allowing for correlated factors, using both the exploratory and two confirmatory samples. In addition to the fall confirmatory sample, a second confirmatory sample was used representing the winter administration of the screener. Using the confirmatory samples tested the model’s generalizability and stability.

As is common in CFA studies, several indices were consulted to assess model fit (Kline, 2011). As a stand-alone measure of model fit, chi-square was assessed but is known to reject reasonably specified models due to large sample sizes (Anderson & Gerbing, 1988). Three other indices were considered to evaluate model fit: the root mean square error of approximation (RMSEA), the Tucker–Lewis index (TLI), and the comparative fit index (CFI). For RMSEA, values <.08 are considered reasonable (Browne & Cudeck, 1992; Kline, 2011) and the latter two indices ranged from 0.0 to 1.0 with values at or greater than .90 indicating better model fit (Hu & Bentler, 1995).

In the third phase, internal scale consistencies were evaluated using categorical omega (Dunn, Baguley, & Brunsden, 2014; Gadermann, Guhn, & Zumbo, 2012). Although Cronbach’s (1951) alpha is the most common measure of reliability (Hogan, Benjamin, & Brezinski, 2000), several studies have highlighted the issues with alpha stemming from assumption violations (see McNeish, 2018). In the current study, omega is used to measure the reliability of a scale where each item is not assumed to contribute equally (i.e., congeneric) to the overall scale and is estimated using the MBESS package (Kelley & Pornprasertmanit, 2016).

Finally, we examined the convergent relationships among the latent variables through factor correlations (Westen & Rosenthal, 2003). The factor correlations were hypothesized to have no more than modest to moderate correlations with each other, given the comorbidity of risk factors (Willner, Gatzke-Kopp, & Bray, 2016).

Results

EFAs

Using the exploratory sample (n = 450), a parallel analysis was conducted with the fa.parallel function in the psych (Revelle, 2015) package using polychoric correlation matrices. Results indicated that six factors could be extracted. An initial EFA was specified to extract seven factors, based on our hypothesized constructs, but the resulting factor solution indicated that the seventh factor was comprised of only two items that were relatively similar to each other (“I have trouble finishing my work” and “I complete my schoolwork on time”). As a result, the latter item with the stronger loading was retained and six factors were then respecified for extraction, as suggested by the parallel analysis.

Based on the six-factor EFA, six items did not have appreciable factor loadings on any of the factors (i.e., <.35) and were subsequently removed one at a time in an iterative manner to assess changes in factor loadings (see Table 2 for final set of items). After the items with low factor loadings were excluded, one additional item showed strong double loadings on two factors and was excluded as well (see online appendix for factor correlations). The final exploratory factor solution consisted of six factors with 30 items (see Table 2). A parallel analysis was performed again using a reduced polychoric correlation matrix based on the 30 items. Results indicated six factors could be extracted and the six factors were related to the factors hypothesized with the exception of the academic readiness construct.

Table 2.

Final Six-Factor EFA (n = 450) and CFA (n = 940; in Parenthesis) Standardized Pattern Loadings.

Items	Peer relations	Internalizing behaviors	Externalizing behaviors	Relational aggression	Emotional regulation	School engagement
I have friends to talk to at school	.80 (.66)
I have friends to eat lunch with at school	.74 (.67)
I cooperate with others	.53 (.71)
I work well with my classmates	.37 (.72)
I have a hard time asking for help		.45 (.53)
I like myself		−.40 (−.67)
In the past month, I felt sad		.70 (.77)
In the past month, I felt fearful		.61 (.62)
In the past month, I felt lonely		.76 (.82)
In the past month, I felt worried		.62 (.60)
In the past month, I felt like I did not matter		.78 (.86)
In the past month, I felt hopeless		.86 (.83)
I feel left out by others		.63 (.76)
I get in trouble at school			.79 (.78)
I am sent out of class for bad behavior			.74 (.78)
I disrupt class			.62 (.75)
My friends get in trouble at school			.45 (.50)
I have trouble paying attention			.44 (.73)
I complete my schoolwork on time			−.48 (a)
I listen to my teachers			−.46 (−.85)
I am mean to others				.63 (.83)
I blame others for my mistakes				.54 (.64)
I talk about people behind their back				.78 (.66)
I make fun of others				.71 (.77)
I get mad easily					.95 (.87)
I have a hard time controlling my temper					.77 (.88)
I get crabby and irritated easily					.66 (.83)
I look forward to learning new things at school						.88 (.83)
I try hard to get good grades on my work						.47 (.82)
I think school is fun						.71 (.82)
Omega	.73	.90	.87	.82	.89	.82

Note. The following items were excluded: I am a good friend; I express my feelings well; I have trouble sitting still at school; I have trouble finishing my work; I miss school for reasons other than being sick; I can solve real-life problems; I need help with my emotions. Items shown have |loadings| > .35. a = item removed in the CFA analysis based on CFA analysis in the fall (n = 450). EFA = exploratory factor analysis; CFA = confirmatory factor analysis.

CFAs

Based on the extracted six-factor solution, a CFA was conducted using the exploratory sample to investigate whether further model changes were required. Model fit indices indicated the model had an acceptable fit with the data based on the majority of fit indices; χ²(390) = 1,026.46, RMSEA = .06, TLI = .89, CFI = .90. To improve model fit, we inspected the item loadings and their corresponding factors. One additional item (“I complete my schoolwork on time”) with the lowest factor loading in the entire model was excluded as it did not fit as well with the conceptually postulated factor of disruptive behaviors (the item was initially hypothesized to form on the academic readiness factor). The CFA was then rerun and model fit improved, χ²(362) = 875.76, RMSEA = .06, TLI = .91, CFI = .92.

However, to assess model generalizability, a separate confirmatory sample was used (n = 450). Model fit indices resulting from the CFA (see online appendix for factor loadings) using the confirmatory sample also indicated acceptable model fit; χ²(362) = 808.21, RMSEA = .05, TLI = .92, CFI = .93. To further assess model generalizability, a third, larger sample (n = 940) collected in winter was also used. Again, model fit indices using the winter sample were generally acceptable; χ²(362) = 1,830.33, RMSEA = .07, TLI = .90, CFI = .91 (see Table 2 for factor loadings).¹

Reliability

In addition, scale reliabilities using the confirmatory sample were investigated with ordinal coefficient omega (Dunn et al., 2014; see Table 2). Items with negative factor loadings were reverse coded. All scales demonstrated acceptable measures of internal consistency (range = .73-.90).

Convergent Validity

Correlations among the latent constructs are shown using both holdout samples (see Table 3). Generally, the absolute values of the correlation coefficients ranged from r = .25 to .72 (all ps < .001) and in the expected direction. For example, using the winter sample, students with higher externalizing behaviors were more likely to have poorer peer relationships (r = –.49), exhibit more relational aggression (r = .65), have more emotional regulation issues (r = .46), and report lower school engagement (r = –.57). The more modest correlations were between internalizing and externalizing behaviors (r = .26) and the strongest correlations were for relational aggression with externalizing behaviors (rs = .66-.72).

Table 3.

Factor Correlations Using Confirmatory Samples.

	Factor	1	2	3	4	5	6
1	Peer relations		−.63	−.49	−.38	−.55	.37
2	Internalizing behaviors	−.59		.26	.32	.66	−.25
3	Externalizing behaviors	−.49	.26		.72	.50	−.65
4	Relational aggression	−.48	.37	.65		.51	−.60
5	Emotional regulation	−.45	.52	.46	.49		−.38
6	School engagement	.48	−.26	−.57	−.47	−.32

Note. All correlations are statistically significant (ps < .001). Upper diagonal: using confirmatory sample in fall (n = 450). Lower diagonal: using confirmatory sample in winter (n = 940).

Discussion

Schools provide an influential social context for affecting both youth social and emotional development and also as a site where youth social and emotional health can be routinely monitored. Despite the availability of commercial socioemotional screening tools, most youth at risk of mental health concerns go undetected and few schools are able to efficiently screen all children regularly. The EIS was developed as a county-wide effort to address the issues of ease of use, cost, and availability, which schools, particularly schools in low-income areas, may experience when trying to conduct universal screening.

Six factors emerged in the original EFA and were replicated in CFAs with two separate samples: peer relations, internalizing behaviors, externalizing behaviors, relational aggression, emotional regulation, and school engagement. Although the factor of academic readiness did not clearly form as hypothesized, further work will be conducted to refine the items to be used. In addition, internal consistencies of the scale ranged from acceptable to excellent (ω = .73-.90).

Convergent validity, as shown by the correlations among the six factors, was supported and the correlations were as expected. The correlations supported the notion that the risk factors were similar but distinct constructs (i.e., no scale was correlated at r > .80, which may suggest that the same factor was being measured). The lowest correlation (r = .26) was found between internalizing and externalizing behaviors but was very similar to the correlation found in another study examining both constructs using different measures (e.g., r = .28; Willner et al., 2016). The strongest correlation between relational aggression and externalizing behaviors (r = .72) also made conceptual and theoretical sense (e.g., the students who most frequently get into trouble at school may also be more likely to be those who are mean to others).

Of importance to the coalition, EIS has been well received by school staff because of its ease of use and its limited burden on students and staff. With the support of technical assistance advisors from the university, school personnel have been trained to use screening data as part of school problem-solving model to guide team-based decision making at the school, grade, and student levels. The data are reported back to school teams via an online dashboard that provides information about students at risk and patterns of risk across grades, school, and time. School teams then use these data to make informed decisions about universal to intensive supports that are needed at each of these levels.

The current study provides an initial glimpse at the technical properties of this new measure. Although screeners that use multi-informants (e.g., students, teachers, parents) may be more discriminating (Kuhn et al., 2017), student self-reports are a valid source of data as some informants may also not be as sensitive to certain types of risk factors (e.g., teachers may not be highly sensitive to symptoms related to internalizing behaviors; Scott et al., 2009). Further planned studies will be conducted to examine the instrument’s sensitivity to change and treatment utility. In addition, we will compare student ratings with those of other informants, other measures (including commercially available measures), and other measurement methods (e.g., direct observation) to determine further properties of the EIS. In succeeding studies, we are targeting to have a larger and more diverse sample as well. Future studies will also incorporate validity screening items, which have shown to be important in improving the quality of surveys (Cornell, Klein, Konold, & Huang, 2012).

Implications

The EIS-student report represents the combined efforts of a partnership approach to instrument and intervention development. It arose from a common problem that was experienced across all school buildings in a single county. Working in unison with university faculty to develop a tool that could overcome these real-world concerns has produced a measure that has, based on informal feedback from coalition members, both social validity and acceptance and also, as demonstrated in this study, initial evidence of technical adequacy. The approach can serve as a model for future instrument development and assist schools in overcoming common barriers to adopting screening practices that are a key step in targeting concerns, allocating resources, and selecting effective actions to reduce the risks identified by the screening tool.

The EIS shows promise as a universal screening tool that may be used with students in elementary and middle schools. Through the use of the EIS, schools may identify students who are most likely to experience a variety of difficulties and, thus, allow schools to provide the necessary supports to students most at risk of future difficulties.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by a grant awarded by the Boone County Children Services Fund. The opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect those of the Boone County Children Services Fund.

Notes

ORCID iD

Francis L. Huang

Supplemental Material

Supplementary material is available for this article online.

References

Anderson

J. C.

Gerbing

D. W.

(1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103, 411-423.

Browne

M. W.

Cudeck

(1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258.

Burns

J. R.

Rapee

R. M.

(2016). Screening for mental health risk in high schools: The development of the Youth RADAR. Psychological Assessment, 28, 1220-1231.

Centers for Disease Control and Prevention. (2013). Mental health surveillance among children—United States, 2005-2011. Morbidity and Mortality Weekly Report, 62, 1-35.

Cicchetti

Tucker

(1994). Development and self-regulatory structures of the mind. Development and Psychopathology, 6, 533-549.

Cicchetti

D. V.

(1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284-290.

Connolly

S. D.

Bernstein

G. A, &

Work Group on Quality Issues (2007). Practice parameter for the assessment and treatment of children and adolescents with anxiety disorders. Journal of the American Academy of Child & Adolescent Psychiatry, 46, 267-283.

Cook

C. R.

Volpe

R. J.

Livanis

(2010). Constructing a roadmap for future universal screening research beyond academics. Assessment for Effective Intervention, 35, 197-205.

Cornell

Klein

Konold

Huang

(2012). Effects of validity screening items on adolescent survey data. Psychological Assessment, 24, 21-35.

10.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

11.

Dowdy

Ritchey

Kamphaus

R. W.

(2010). School-based screening: A population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166-176.

12.

Dunn

T. J.

Baguley

Brunsden

(2014). From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. British Journal of Psychology, 105, 399-412.

13.

Essau

C. A.

(2005). Frequency and patterns of mental health services utilization among adolescents with anxiety and depressive disorders. Depression and Anxiety, 22, 130-137.

14.

Fabrigar

L. R.

Wegener

D. T.

(2012). Exploratory factor analysis. New York, NY: Oxford University Press.

15.

Finney

DiStefano

(2006). Non-normal and categorical data in structural equation modeling. In Hancock

Mueller

(Eds.), Structural equation modeling: A second course (pp. 439-492). Greenwich, CT: Information Age Publishing.

16.

Gadermann

Guhn

Zumbo

(2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17. Retrieved from http://www.pareonline.net/getvn.asp?v=17&n=3

17.

Glover

T. A.

Albers

C. A.

(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117-135.

18.

Hall-Lande

J. A.

Eisenberg

M. E.

Christenson

S. L.

Neumark-Sztainer

(2007). Social isolation, psychological health, and protective factors in adolescence. Adolescence, 42, 265-286.

19.

Herman

K. C.

Lambert

S. F.

Ialongo

N. S.

Ostrander

(2007). Academic pathways between attention problems and depressive symptoms among urban African American children. Journal of Abnormal Child Psychology, 35, 265-274.

20.

Hogan

T. P.

Benjamin

Brezinski

K. L.

(2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523-531.

21.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.

22.

Horner

R. H.

Sugai

Smolkowski

Eber

Nakasato

Todd

A. W.

Esperanza

(2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11, 133-144.

23.

L. T.

Bentler

P. M.

(1995). Evaluating model fit. In Hoyle

R. H.

(Ed.) Structural equation modeling: Concepts, issues, and applications (pp. 76-99). Thousand Oaks, CA: SAGE.

24.

Huang

(2014). Analyzing group level effects with clustered data using Taylor series linearization. Practical Assessment, Research, and Evaluation, 19, 1-9.

25.

Huang

(2016). Alternatives to multilevel modeling for the analysis of clustered data. Journal of Experimental Education, 84, 175-196.

26.

Kataoka

S. H.

Zhang

Wells

K. B.

(2002). Unmet need for mental health care among US children: Variation by ethnicity and insurance status. American Journal of Psychiatry, 159, 1548-1555.

27.

Kelley

Pornprasertmanit

(2016). Confidence intervals for population reliability coefficients: Evaluation of methods, recommendations, and software for composite measures. Psychological Methods, 21, 69-92.

28.

Kline

(2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.

29.

Kuhn

Aebi

Jakobsen

Banaschewski

Poustka

Grimmer

. . . Steinhausen

H. C.

(2017). Effective mental health screening in adolescents: Should we collect data from youth, parents or both? Child Psychiatry and Human Development, 48, 385-392.

30.

Lane

K. L.

Carter

E. W.

Jenkins

Dwiggins

Germer

(2015). Supporting comprehensive, integrated, three-tiered models of prevention in schools: Administrators’ perspectives. Journal of Positive Behavior Interventions, 17, 209-222.

31.

Levitt

J. M.

Saka

Romanelli

L. H.

Hoagwood

(2007). Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology, 45, 163-191.

32.

Lewinsohn

P. M.

Rohde

Seeley

J. R.

(1998). Major depressive disorder in older adolescents: Prevalence, risk factors, and clinical implications. Clinical Psychology Review, 18, 765-794.

33.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23, 412-433.

34.

National Association of School Psychologists. (2010). Social skills: Promoting positive behavior, academic success, and school safety. Retrieved from http://www.nasponline.org/resources/factsheets/socialskills_fs.aspx

35.

Nelson

J. R.

Stage

Duppong-Hurley

Synhorst

Epstein

M. H.

(2007). Risk factors predictive of the problem behavior of children at risk for emotional and behavioral disorders. Exceptional Children, 73, 367-379.

36.

Odgers

C. L.

Caspi

Nagin

D. S.

Piquero

A. R.

Slutske

W. S.

Milne

B. J.

. . . Moffitt

T. E.

(2008). Is it important to prevent early exposure to drugs and alcohol among adolescents? Psychological Science, 19, 1037-1044.

37.

Patterson

G. R.

Reid

J. B.

Dishion

T. J.

(1992). Antisocial boys (Vol. 4). Eugene, OR: Castalia.

38.

Reid

J. B.

Patterson

G. R.

Snyder

J. E.

(2002). Antisocial behavior in children and adolescents: A developmental analysis and model for intervention. Washington, DC: American Psychological Association.

39.

Reid

Gonzalez

J. E.

Nordness

P. D.

Trout

Epstein

M. H.

(2004). A meta-analysis of the academic status of students with emotional/behavioral disturbance. The Journal of Special Education, 38, 130-143.

40.

Revelle

(2015). psych: Procedures for psychological, psychometric, and personality research. Evanston, IL: Northwestern University.

41.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36.

42.

Scott

M. A.

Wilcox

H. C.

Schonfeld

I. S.

Davies

Hicks

R. C.

Turner

J. B.

Shaffer

(2009). School-based screening to identify at-risk students not already known to school professionals: The Columbia suicide screen. American Journal of Public Health, 99, 334-339.

43.

Severson

H. H.

Walker

H. M.

(2002). Proactive approaches for identifying children at risk for sociobehavioral problems. In Lane

Gresham

O’Shaughnessy

(Eds.), Interventions for children with or at risk for emotional and behavioral disorders (pp. 33-53). Boston, MA: Allyn & Baco.

44.

Severson

H. H.

Walker

H. M.

Hope-Doolittle

Kratochwill

T. R.

Gresham

F. M.

(2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45, 193-223.

45.

Tabachnick

B. G.

Fidell

(2013). Using multivariate statistics (Vol. 6). Thousand Oaks, CA: SAGE.

46.

Taylor

L. D.

Davis-Kean

Malanchuk

(2007). Self-esteem, academic self-concept, and aggression at school. Aggressive Behavior, 33, 130-136.

47.

Temcheff

C. E.

Serbin

L. A.

Martin-Storey

Stack

D. M.

Hodgins

Ledingham

Schwartzman

A. E.

(2008). Continuity and pathways from aggression in childhood to family violence in adulthood: A 30-year longitudinal study. Journal of Family Violence, 23, 231-242.

48.

Thompson

Reinke

Herman

(2014). In Massat

Kelly

Constable

(Eds.), School social work: Practice, policy, and research (8th ed.). New York, NY: Oxford.

49.

Thompson

Daniel

L. G.

(1996). Factor analytic evidence for the construct validity of scores: A historical overview and some guidelines. Educational and Psychological Measurement, 56, 197-208.

50.

Tremblay

R. E.

Nagin

D. S.

Seguin

J. R.

Zoccolillo

Zelazo

P. D.

Boivin

. . . Japel

(2004). Physical aggression during early childhood: Trajectories and predictors. Pediatrics, 14, 3-9.

51.

Volpe

R. J.

Briesch

A. M.

Chafouleas

S. M.

(2010). Linking screening for emotional and behavioral problems to problem-solving efforts: An adaptive model of behavioral assessment. Assessment for Effective Intervention, 35, 240-244.

52.

Westen

Rosenthal

(2003). Quantifying construct validity: Two simple measures. Journal of Personality and Social Psychology, 84, 608-618.

53.

Willner

C. J.

Gatzke-Kopp

L. M.

Bray

B. C.

(2016). The dynamics of internalizing and externalizing comorbidity across the early school years. Development and Psychopathology, 28, 1033-1052.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

An Investigation of the Psychometric Properties of the Early Identification System–Student Report

Abstract

Keywords

Screening: Definition, Purpose, and Characteristics

Some Barriers to Screening Implementation

Targeted Risk Factors of the EIS

The Current Study

Method

Participants

Measures

Screening Procedures

Analytic Strategy

Results

EFAs

CFAs

Reliability

Convergent Validity

Discussion

Implications

Footnotes

Declaration of Conflicting Interests

Funding

Notes

ORCID iD

Supplemental Material

References

Supplementary Material