Assessment of the Risks and Needs of Middle School Students: Invariance Properties Related to Gender and Ethnicity

Abstract

The middle school version of the Washington Assessment of Risks and Needs of Students (msWARNS) is a self-report instrument designed for use by school personnel to identify barriers to school attendance and school success for sixth- to eighth-grade students. It measures six domains relevant to improving school outcomes that include aggression-defiance, depression-anxiety, substance use, peer deviance, home environment, and school engagement. In the present study, a bifactor S − 1 model, for which the aggression-defiance domain was the reference factor for the general factor and the other domains constituted the subfactors, had good fit and better fit than several other alternative models. Results of multigroup confirmatory factor analysis revealed invariance across different groups defined by gender and race/ethnicity (Native American, African American, Hispanic, and White), with a sample of referred middle school students (N = 2,356; ages 10–15 years). Reliability analyses support the use of the general factor to guide decision-making, the reliable use of the depression-anxiety factor for providing additional insights, and the remaining factors for guiding communication, as part of an assessment and intervention program for middle school students.

Keywords

middle school students truancy factorial invariance at risk

School truancy is associated with dropping out of school and many subsequent negative social and economic outcomes for youth (Belfield & Levin, 2007; Hammond et al., 2007; Henry & Huizinga, 2007; Rocque et al., 2017). For this reason, a cornerstone of intervention efforts to improve graduation outcomes and the long-term economic and social prospects for school-age children involves attempts to reduce absenteeism and truancy. Such efforts focus on not only high school but also on middle and elementary school, developmental epochs during which risks for truancy and school failure first appear (Baker et al., 2001; Johnson et al., 2012). Although interventions at earlier grades are common, resources utilized for middle school students are often identical to those used in high school settings (e.g., Dembo & Gulledge, 2009; Dennis et al., 2006; Grisso et al., 2001; National Institute on Drug Abuse, 2009). Given knowledge about differential social and psychological functioning across these age groups (Steinberg, 2014), this represents a potential problem for effective assessment and intervention. The risks and needs of middle school students may differ from those of high school students. For this reason, it is necessary to develop assessments for use with those younger populations and evaluate the comparability of scores (measurement invariance) across groups that differ in terms of the educational challenges they face.

This study evaluates key scoring inferences related to validity for an instrument designed to evaluate the constructs of risks and needs comprised of several facets (e.g., depression, anxiety) that interfere with school attendance and school success of middle school students. The instrument on which we focus is an adaptation of one previously developed for use with high school students—the Washington Assessment of Risks and Needs of Students (WARNS; George et al., 2015). The WARNS was developed with input from schools and juvenile courts to inform intervention efforts and facilitate cooperation across juvenile courts, schools, and social service agencies (George et al., 2015). One intended use of WARNS scores is to provide a basis for conversing with youth about the problems interfering with their school success and to facilitate communication among professionals. However, scores may be used in other ways, such as screening, referral for services, and intervention level needed, when combined with other sources of information in a student psychoeducational evaluation (George et al., 2015; Gotch & French, 2020; Strand & Lovrich, 2014).

The use of the WARNS with high school students has been examined from a few different psychometric angles. The factor structure has been supported, showing that a general risk and needs factor score may be best for making decisions about students, with sufficient internal consistency reliability (e.g., >.90; Strand et al., 2019). The WARNS structure and items do appear to function similarly for boys and girls and across African American, White American, and Hispanic student samples (Alpizar et al., 2020; French & Vo, 2020). WARNS scores are related to social and academic outcomes, whereby students with elevated scores are more likely to be arrested or suspended compared with their lower risk counterparts (George et al., 2015; Iverson et al., 2018). Although this evidence supports scoring, generalization, and extrapolation inferences in a validity framework (e.g., Kane, 2013), this evidence does not exist for the middle school adaptation of the WARNS, referred to as the msWARNS.

Assessment of Middle School Students

At multiple grade levels, absenteeism and truancy are associated with lower academic performance, lower test scores in reading and mathematics, grade retention, internalizing and externalizing behavior disorders, dropout, and juvenile justice system involvement (Belfield & Levin, 2007; Burton et al., 2014; Hammond et al., 2007). Subsequently, dropping out of school prior to graduation predicts risk in adulthood for economic, occupational, legal, and marital problems (Christenson & Thurlow, 2004; Farrington, 2005; Onifade et al., 2010; Rocque et al., 2017). In its more extreme manifestations, this developmental cascade is referred to as the school to prison pipeline (National Juvenile Justice Network, 2011).

Risk factors for school absenteeism include child, parent, family, peer, school, and community variables (Green et al., 2008; Johnson et al., 2012; Maynard et al., 2017; Skedgell & Kearney, 2018). Inspired by longitudinal studies and life-course criminology theory (Farrington, 2005; Sampson & Laub, 2005), the WARNS items are organized according to the following domains: aggression-defiance, depression-anxiety, substance use, peer deviance, family environment, and school engagement. Relative to WARNS items, some msWARNS items have been altered or deleted based on input from middle school counselors and principals regarding their appropriateness for younger students. For example, one item concerning use of hard drugs (e.g., cocaine, methamphetamine) was omitted, and another one altered to identify vaping in addition to smoking. In addition, to reduce complexity, the number of response categories for each item was reduced from four in the high school version to three in the middle school version. These changes to the instrument and a different target population (middle school students) necessitate an evaluation of the psychometric properties of msWARNS with that new population.

In this article, we assess the factor structure of the msWARNS and measurement invariance across gender and ethnicity. This investigation is guided by prior findings regarding the high school version of the instrument, which identified a bifactor model as providing the best fit and as illustrating measurement invariance across gender and ethnicity for two ethnic groups, Hispanic and European American (Strand et al., 2019). Similarly, in the present article, we explore model fit and conduct invariance testing to understand the psychometric profile of this middle school version of the instrument. Although invariance testing for ethnicity for the WARNS was possible with respect to only two groups, owing to sample limitations (European American and Hispanic), the present sample allows for invariance testing for these two ethnicities and for Native American and African American subsamples, as well. We hypothesize that invariance testing will reveal measurement invariance across gender and race/ethnicity, given invariance was supported in the high school WARNS. This information is necessary for making comparative score inferences across groups (Kane, 2013).

Given the previous work with the WARNS for high school students and that modifications did not focus on changing the content balance of domains measured, we hypothesized that a symmetrical bifactor model that included six first-order factors and one second-order factor would fit the data (see Figure 1). Recent work, however, has identified common problems with symmetrical bifactor models that include insignificant and negative factor loadings on first-order factors (Eid et al., 2017). In such cases, bifactor S − 1 models have performed well and are recommended for consideration if and when solutions indicate potential misfit (Burns et al., 2020). For the msWARNS, such a model would be comprised of five first-order correlated factors with a general factor on which load those five factors and also items of a sixth factor termed the reference factor (see Figure 2). Anticipating the usefulness of a bifactor S − 1 model, it is necessary to identify what might be an appropriate reference factor (Burns et al., 2020; Eid et al., 2017). The reference factor constitutes the set of items that theory would suggest are the primary contributors to a general factor. Given the significant contribution of aggression and defiance to negative school outcomes based on life-course criminology theory (Farrington, 2005), we identify the aggression-defiance factor as the reference factor. In addition to the two bifactor models just described, our analytic plan calls for reviewing the model fit of three other models for comparative purposes. These include a one-factor model with all items related to a single factor (i.e., general need/risk factor), a model with six correlated first-order factors (the six Needs Scales), and a higher-order model (i.e., the six first-order factors and one second-order factor). Each of these factor models can be consistent with the theorizing and aims of the msWARNS, although interpretation of scores would differ depending on which model demonstrated the best fit. A bifactor S − 1 model would call for interpretation of scores similar to a symmetrical bifactor model, albeit with the recognition that the variance of the reference factor is captured by the general factor. For both models, however, scores on the general factor are used for decisions regarding overall risk and domain scores are used for understanding specific areas of risk and need.

Figure 1.

Symmetrical Bifactor Model.

Figure 2.

Bifactor S − 1 Model.

Method

Participants

Study participants included 2,356 students (50% male) attending middle school (Grades 6–8) in Washington state. All completed the self-report assessment under the supervision of a school counselor or principal, usually in response to concerns about truancy or excessive absenteeism. However, data were not available to indicate the reason for completing the instrument (e.g., general school screening vs. truant concern). The sample included children from 63 school districts. The distribution of students by grade was as follows: 556 sixth graders, 859 seventh graders, and 941 eighth graders. Children ranged in age from 10 to 15 (M = 12.68, SD = 0.95) years. The sample included 323 students identified as receiving special education services, and 97 having a formal 504 Plan for the purpose of documenting individualized educational support services. Ethnic background of students included American Indian/Alaska Native (Native American, 211), African American (245), Hispanic (909), White (1,011), Asian (105), Pacific Islander (60), and Other (263). Multiple marking of ethnic background occurred on 448 occasions (e.g., marked both Native American and Hispanic). Subgroup analyses were not performed with the Asian or the Pacific Islander subgroups due to data sparseness in item responses and small sample size, nor were they performed with the ethnic/race category “Other.” To explore comparability of the ethnic groups regarding age and gender, analyses of variance (ANOVAs) were conducted and revealed no differences across the groups for student age, F(5, 2,087) = 1.91, ns, or gender, coded girls = 1, boys = 2, F(5, 2,087) = 1.30, ns.

Instrument

The msWARNS is a 39-item self-report measure designed to allow schools, courts, and youth service providers to assess risks and needs of students that may result in truancy and/or school failure and to inform interventions accordingly (George et al., 2015). During the assessment, students are asked to respond within the context of their experiences over the previous 2 months. Administration is online, requiring 10 to 30 minutes to complete (M_{completiontime} = 13 minutes). The msWARNS contains items used to generate scores on six subdomains and a cumulative total risk domain. The subdomains include aggression-defiance (seven items), depression-anxiety (seven items), substance abuse (five items), peer deviance (six items), family environment (seven items), and school engagement (seven items). The domains are derived from life-course criminology theory. These domains are linked to truancy, delinquency, and school dropout (Farrington, 2005; Howell, 2003; Maynard et al., 2017). Items are measured on a rating scale (0 = never or not much, 1 = sometimes, 2 = always or a lot) indicating frequency during the last 2 months. The Flesch–Kincaid grade level for the msWARNS is 1.7, with a Flesch reading ease value of 92.7. Both values are below a middle school reading level.

Analyses

Confirmatory Factor Analysis

Given existing theoretical and empirical evidence for the proposed factor structure of the instrument, confirmatory factor analytic procedures were utilized. Parameter estimation employed weighted least squares means and variance adjustment (WLSMV) with the Theta parametrization to account for ordinal data, using Mplus version 7.31 (Muthén & Muthén, 1998–2012). Model fit was evaluated via multiple criteria that include a chi-square significance test, the root mean square error of approximation (RMSEA), and comparative fit index (CFI). Given that the indices are based on maximum likelihood (ML) estimation, their applicability to robust estimation lacks empirical justification (e.g., Finney & DiStefano, 2013; Nye & Drasgow, 2011). Thus, criteria typically used (e.g., CFI > .95, Hu & Bentler, 1999) were relaxed to include CFI > .90 and RMSEA < .08. In addition, model parameters were inspected for out-of-range values as were residuals for localized strain (Brown, 2015). Pattern coefficients were evaluated against a threshold of 0.30 (Pett et al., 2003). Interpretation and use of the model were also considered in light of current use and theory.

Score Reliability

For the best fitting model for the full sample, internal consistency reliability estimates were to be calculated for the scores on the factors, to inform recommendations for score use. Given the analytical context of latent variables measured on an ordinal scale and the presence of a bifactor structure, score reliability was estimated through four indices—omega (ω), omega hierarchical (ω_H), omega subscale (ω_S), and omega hierarchical subscale (ω_HS). The ω coefficient is a latent variable analogue to Cronbach’s alpha coefficient, capturing all sources of common variance (Reise, 2012; Rodriguez et al., 2016). The ω_S coefficient reflects the proportion of combined common variance between the general and specific factor. The hierarchical coefficients reflected systematic score variance only in either the general factor (ω_H) or specific factor with general factor item variance removed (ω_HS). Coefficient values of at least 0.80 were considered sufficient for use of the instrument in research settings, whereas a criterion of 0.95 was set for high-stakes decisions about an individual (Nunnally & Bernstein, 1994). However, within the context of bifactor models, it has been recommended that omega hierarchical values of .50 for specific factors may support use of those scores independent of the general factor (Reise et al., 2013).

Factorial Invariance

Factorial invariance (FI) was examined across gender and race/ethnicity for the best fitting model using multigroup confirmatory factor analysis (MCFA). The students self-identified for these demographic categories, given options available on the assessment system. FI involves the examination of equivalence models by imposing model equality constraints (Brown, 2015) and assessing a decline in fit where a substantial decline signals a possible lack of invariance. Two sets of MCFA were generated in which the model was fit across each group.

The strategy entailed examining the fit of the proposed model in each group with no constraints. Next, the model was estimated for two groups simultaneously to obtain the baseline (configural) model. Finally, a model with all parameters set equal across the groups (i.e., a fully constrained, scalar model) was evaluated. A significant degradation of fit at this stage (i.e., decrease in CFI of more than .01 AND a decrease in RMSEA of .015) would signal a lack of invariance (Chen, 2007). We conducted the baseline-to-fully-constrained model approach because when WLSMV is employed with ordinal variables and theta parametrization, where indicators load on more than one factor, as is the case with a bifactor model, the metric model cannot be estimated. For additional specifications, see Mplus Version 7.1 Language Addendum (Muthén & Muthén, 1998–2012; cf., Millsap, 2011).

Missing Data

The total sample includes no instances of missing data because completion of the online version of the WARNS requires a response to all items and all administrations were completed using the online version of the instrument.

Results

Factor Model

Means and standard deviations for the total risk factor score and all domains, for the total sample, girls and boys, and the race/ethnic groups are presented in Table 1. Zero-order correlation coefficients between the six subdomains for the full sample are presented in Table 2, revealing primarily moderate associations across them. Model fit indices for the various factor structure models are presented in Table 3. The hypothesized best fitting model is the symmetrical bifactor model. Despite having good fit indices, however, an analysis of item loadings revealed two negative item loadings. By contrast, the results of a bifactor S − 1 model revealed good fit and no negative item loadings. In addition, it had better fit than the other three CFA models, which included a single factor model, a model with six correlated first-order factors, and a higher-order model. Therefore, the bifactor S − 1 model was identified as the best factor analytic solution for the msWARNS for this sample of referred middle school students.

Table 1.

Means and Standard Deviations for the Six WARNS Domains for the Total Sample and Sample Subgroups.

Group	Total risk score	Domain
Group	Total risk score	Aggression defiance	Depression anxiety	Substance abuse	Peer deviance	Family environment	School engagement
Total sample	19.06 (10.08)	2.00 (2.14)	5.68 (2.90)	0.41 (1.32)	2.23 (2.46)	3.57 (2.55)	5.17 (2.56)
Girls	19.58 (10.79)	1.95 (2.15)	6.26 (2.98)	0.46 (1.42)	2.13 (2.51)	3.74 (2.75)	5.04 (2.61)
Boys	18.41 (9.20)	2.02 (2.08)	5.08 (2.69)	0.35 (1.20)	2.30 (2.36)	3.37 (2.31)	5.29 (2.49)
Asian American	18.00 (10.47)	1.90 (2.40)	5.73 (3.00)	0.36 (1.18)	1.90 (2.29)	3.33 (2.70)	4.77 (2.75)
Pacific Islander	18.82 (10.00)	2.27 (2.26)	5.72 (3.05)	0.40 (1.53)	2.55 (2.50)	3.27 (3.19)	4.62 (2.31)
Native American	21.22 (10.98)	2.42 (2.46)	6.00 (2.93)	0.61 (1.75)	2.69 (2.48)	3.99 (2.63)	5.51 (2.63)
African American	20.31 (10.33)	2.60 (2.48)	5.34 (2.89)	0.45 (1.37)	2.90 (2.65)	3.62 (2.46)	5.40 (2.72)
Hispanic	19.51 (10.18)	2.07 (2.16)	5.69 (2.75)	0.48 (1.41)	2.35 (2.59)	3.68 (2.57)	5.24 (2.44)
White	19.44 (10.26)	2.02 (2.16)	5.93 (3.06)	0.41 (1.36)	2.24 (2.47)	3.64 (2.56)	5.20 (2.60)

Note. Table values are mean scores and (standard deviations). WARNS = Washington Assessment of Risks and Needs of Students.

Table 2.

Zero-Order Correlation Coefficients for the Domain Scores.

	Domain	1	2	3	4	5
1	Aggression-defiance	—	—	—	—	—
2	Depression-anxiety	.38	—
3	Substance abuse	.49	.20	—
4	Peer deviance	.59	.28	.44	—
5	Family environment	.59	.47	.38	.44	—
6	School engagement	.51	.32	.35	.42	.51

Note. All correlations were statistically significant, p < .01.

Table 3.

Confirmatory Factor Analysis Fit Indices by Model.

Model (N = 2,356)	χ²	df	p	RMSEA	90% CI	CFI
One-factor	13,302	702	<.001	.087	[.086, .089]	0.778
Six-factor	6,639	687	<.001	.061	[.059, .062]	0.895
Higher order	6,678	696	<.001	.060	[.059, .062]	0.895
Symmetrical bifactor^a	4,967	663	<.001	.052	[.051, .054]	0.924
Bifactor S − 1	5,034	660	<.001	.053	[.052, .054]	0.923

Note. df = degrees of freedom; RMSEA = root mean square error of approximation; CI = confidence interval; CFI = comparative fit index.

For aggression-defiance domain, negative loadings were obtained for Items MS16 and MS29.

The factor pattern coefficients based on the bifactor S − 1 model are presented in Table 4 for the full sample, and for boys and girls. For the full sample and for boys and girls, items demonstrated moderately high loadings (>.30) on both the reference factor and the specific factors in nearly all cases. Generally, a low factor loading on the general factor portended a relatively higher loading on the specific factor, and vice versa. All loadings were statistically significant. Not shown, correlations for the bifactor S − 1 factors ranged from −.05 to .52, with the highest correlations occurring for depression-anxiety with school engagement (.52) and for family environment with school engagement (.33).

Table 4.

Factor Pattern Coefficients for the Bifactor S − 1 Model Structure for Full Sample, Girls, and Boys.

Items	Total	Factor
Items	Total	Aggression defiance	Depression anxiety	Substance abuse	Peer deviance	Family environment	School engagement
37	.76/.76/.75	—
6	.65/.69/.61	—
2	.70/.73/.70	—
23	.59/.63/.55	—
29	.64/.65/.63	—
39	.80/.80/.80	—
16	.87/.88/.88	—
7	.28/.31/.25		.26/.29/.22
11	.42/.44/.43		.65/.61/.65
14	.21/.23/.21		.70/.73/.64
22	.53/.39/.33		.51/.53/.46
27	.33/.37/.30		.42/.42/.37
33	.19/.18/.23		.68/.70/.62
36	.32/.37/.28		.54/.57/.46
4	.70/.71/.70			.48/.49/.47
9	.76/.80/.71			.50/.45/.54
13	.79/.83/.75			.57/.53/.60
32	.79/.85/.72			.56/.48/.65
38	.75/.76/.74			.44/.42/.47
5	.72/.70/.74				.35/.44/.27
21	.63/.64/.61				.58/.60/.59
25	.64/.59/.69				.56/.63/.48
28	.61/.63/.58				.44/.49/.42
31	.72/.71/.70				.50/.52/.49
35	.61/.59/.62				.62/.66/.55
3	.59/.63/.57					.39/.34/.39
8	.55/.63/.47					.26/.21/.29
12	.37/.43/.31					.72/.79/.67
17	.63/.66/.62					.31/.25/.34
19	.63/.68/.58					.37/.29/.40
30	.49/.58/.40					.59/.52/.61
34	.66/.69/.64					.09/.14/.03
1	.46/.50/40						.62/.59/.65
10	.67/.72/.64						.12/.10/.16
15	.44/.52/.35						.69/.66/.72
18	.38/.44/.32						.66/.63/.70
20	.68/.71/.63						.12/.14/.10
24	.35/.42/.27						.73/.72/.75
26	.47/.46/.49						.33/.34/.33

Note. The order of values is as follows: full sample (comprised of both referred and nonreferred children)/girls/boys.

Reliability

Table 5 presents internal consistency reliability estimates for the full sample and by groups. The ω coefficient for the total risk score meets the .95 criteria for score use at the individual level. The ω_S coefficients for the subscales meet that criteria in some cases and fall somewhat short in others, with values ranging from .79 to .97. For the hierarchical coefficients, reliability estimates for the total risk factor (ω_H) exceed .80 for the total sample and for each of the gender and ethnicity subsamples, suggesting adequate reliability values for general use and research. However, subscale reliability values for items with general factor item variance removed (ω_HS) ranged widely from .09 to .63. Therefore, although the msWARNS reflects a multidimensional, bifactor structure, users should focus on the total risk factor score and use the subscale scores for providing information on areas of risk and need that may require additional assessment.

Table 5.

Internal Consistency Reliability Estimates for Middle School Washington Assessment of Risks and Needs of Students Scores by Demographic Groups for Bifactor S − 1 Model.

Scale	Overall	Female	Male	Native American	African American	Hispanic	White
Aggression-defiance	—/—	—/—	—/—	—/—	—/—	—/—	—/—
Depression-anxiety	.82/.56	.79/.55	.97/.52	.84/.51	.82/.62	.79/.55	.86/.58
Substance abuse	.96/.31	.96/.37	.84/.26	.98/.24	.96/.40	.96/.24	.97/.32
Peer deviance	.93/.40	.92/.39	.97/.41	.91/.52	.92/.45	.94/.37	.94/.46
Family environment	.87/.23	.84/.28	.89/.19	.86/.09	.86/.29	.88/.26	.87/.21
School engagement	.88/.38	.87/.46	.89/.31	.88/.36	.89/.33	.87/.37	.89/.37
Total risk	.97/.88	.96/.87	.97/.90	.97/.90	.97/.87	.97/.89	.97/.88

Note. Omega (ω) and omega subscale (ω_S) coefficients appear to the left of the slash (/); hierarchical omega (ω_H) and hierarchical omega subscale (ω_HS) values appear to the right of the slash.

Furthermore, reliability estimates for domain scores for boys and girls match each other closely and align with those obtained for the total sample. Internal consistency reliability estimates for both the total risk factor and all subdomains were similar across the four race/ethnic groups, and in relation to values obtained for the total sample (Table 5). Consistent with results for the full sample, they reveal sufficient reliability for the general factor and inadequate reliability of subfactors within the bifactor S − 1 model. This shows a reflection of the shared item covariance due to the general factor that is separate from the shared covariance among the groups of items of a given specific factor.

Factorial Invariance

Gender

The bifactor S − 1 model revealed good fit for boys and girls (Table 6). Moreover, invariance of the factor structure was supported. Finally, comparing across girls and boys, and to the full sample, the pattern coefficients are similar in magnitude and relative order, with a few exceptions (Table 4). Taken together, these results suggest that the bifactor S − 1 model is invariant between boys and girls.

Table 6.

Model Results for Invariance Testing for Gender and Ethnicity.

Model (N = 2,356)	χ²	df	p	RMSEA	ΔRMSEA	90% CI	CFI	ΔCFI
Gender
Boys	2,572	660	<.001	.050		[.048, .052]	.942
Girls	2,516	660	<.001	.049		[.047, .051]	.920
Configural	5,077	1,320	<.001	.049		[.048, .051]	.934
Scalar	4,746	1,418	<.001	.045	.004	[.043, .046]	.941	−.007
Race/ethnicity
Native American (n = 211)	1,033	660	<.001	.052		[.046, .058]	.933
Configural	4,505	1,320	<.001	.045		[.044, .047]	.933
Scalar	4,004	1,418	<.001	.039	.006	[.038, .041]	.946	−.013
African American (n = 245)	959	660	<.001	.043		[.037, .049]	.950
Configural	4,266	1,248	<.001	.045		[.044, .047]	.941
Scalar	3,798	1,343	<.001	.039	.006	[.038, .041]	.952	−.011
Hispanic (n = 909)	2,097	660	<.001	.049		[.047, .051]	.940
Configural	5,316	1,320	<.001	.051		[.049, .052]	.928
Scalar	4,895	1,418	<.001	.046	.005	[.044, .047]	.937	−.009
White (n = 1,011)	2,367	660	<.001	.051		[.048, .053]	.920
Configural	5,274	1,320	<.001	.050		[.049, .052]	.928
Scalar	4,854	1,418	<.001	.045	.005	[.044, .047]	.938	−.010

Note. RMSEA = root mean square error of approximation; CI = confidence interval; CFI = comparative fit index.

Race/Ethnicity

For the race/ethnic groups with suitable sample sizes, the bifactor S − 1 model revealed good fit (Table 6). Invariance of the factor structure for all race/ethnic groups, in that difference values for CFI and RMSEA, across configural and scalar models, met criteria. Factor pattern coefficients are significant and >.30 in nearly all cases for the general factor, across race/ethnic groups (Table 7). For all subgroups, one or more items in the family environment and school engagement subdomains were low, negative, and statistically insignificant (Table 7). This is especially evident for the family environment and school engagement subdomains, for which several items had insignificant and negative loadings across all race/ethnic groups. The Native American group had the most insignificant factor loadings for items in these subdomains. For all groups, however, items with low factor loadings on the subdomains had significant loadings on the general factor, suggesting the appropriateness and usefulness of the items across all race/ethnic groups. Therefore, these findings support measurement invariance for the bifactor S − 1 model with regard to race/ethnicity, and the interpretability of the general factor only with respect to decision-making for supporting a student through intervention plans or educational modification plans.

Table 7.

Factor Pattern Coefficients for the Bifactor S − 1 Model Structure for Four Race/Ethnic Groups.

Items	Total risk	Factor
Items	Total risk	Aggression defiance	Depression anxiety	Substance abuse	Peer deviance	Family environment	School engagement
37	.80/.81/.75/.79	—
6	.64/.68/.65/.67	—
2	.70/.79/.72/.72	—
23	.66/.48/.51/.63	—
29	.84/.77/.80/.84	—
39	.61/.72/.67/.62	—
16	.85/.90/.87/.87	—
7	.33/.23/.15/.37		.26/.29/.20/.34
11	.59/.40/.38/.41		.58/.72/.60/.71
14	.39/.23/.25/.15		.73/.73/.64/.76
22	.45/.27/.36/.35		.44/.56/.47/.54
27	.46/.26/.36/.33		.31/.37/.42/.45
33	.38/.05/.20/.16		.71/.62/.69/.71
36	.46/.19/.29/.33		−.07/.65/.49/.60
4	.91/.66/.71/.70			.36/.58/.42/.55
9	.73/.75/.80/.73			.62/.54/.41/.56
13	.80/.69/.82/.81			.56/.70/.53/.56
32	.84/.69/.80/.79			.52/.71/.57/.57
38	.80/.82/.79/.68			.41/.35/.40/.51
5	.57/.66/.79/.67				.46/.30/.30/.47
21	.52/.67/.66/.57				.58/.56/.54/.70
25	.50/.62/.66/.61				.67/.58/.58/.58
28	.55/.53/.68/.56				.54/.46/.39/.48
31	.59/.72/.76/.66				.60/.56/.47/.58
35	.50/.53/.61/.58				.65/.60/.64/.63
3	.63/.53/.56/.60					−.08/.41/.43/.34
8	.54/.62/.61/.52					.33/.17/.23/.27
12	.54/.26/.40/.39					.47/.81/.69/.73
17	.65/.66/.62/.65					−.02/.26/.33/.26
19	.66/.66/.63/.62					.02/.35/.40/.34
30	.60/.33/.49/.49					.55/.73/.62/.64
34	.78/.41/.66/.66					−.13/.27/.13/.09
1	.58/.55/.51/.41						.55/.65/.51/.67
10	.63/75/.65/.69						−.01/.14/.13/.09
15	.56/.57/.45/.44						.64/.66/.65/.69
18	.49/.43/.39/.41						.54/.49/.66/.67
20	.76/.67/.67/.68						.06/.11/.14/.16
24	.42/.46/.38/.36						.62/.63/.73/.73
26	.32/.33/.46/.47						.45/.44/.31/.35

Note. Values for ethnicities are ordered as follows: Native American/African American/Hispanic/White. Boldfaced items are nonsignificant (p > .05).

Discussion

The study investigated the factor structure of the middle school version of the WARNS (George et al., 2015). The sample comprised middle school students from 63 school districts in Washington state. It included students referred for assessment by school personnel in response to absenteeism or other threats to school success for the individual student, and also students administered the instrument as part of general school screening procedures. Given how the data were collected, we did not have access to reasons for referral. Therefore, we could not separate students into different groups based on different uses. However, the present sample is representative of the population for which the msWARNS was designed—middle school students for whom school personnel are seeking additional information for the purpose of enhancing school success, in terms of either individual-level or group-level planning and screening. The sample was diverse with respect to ethnicity and gender, allowing also for an investigation of factor structure invariance across these groups. Represented race/ethnic groups include Native Americans, African Americans, Hispanic Americans, and White Americans. Students of Asian American and Pacific Islander descent were included in the total sample but excluded from group-level analyses owing to small samples.

CFA results support a bifactor S − 1 model for the full sample of students and the gender and ethnic subgroups that were evaluated. In a bifactor S − 1 model, a single subdomain is identified as the reference factor and its items are indicators of the general factor only. In this way, the general factor of a bifactor S − 1 model reflects all instrument items, with all variance for the reference factor loading on the general factor. For all other subdomains, items load on both the general factor and the subfactor, where the shared covariance of items due to the general factor is separate from the shared covariance of a group of items for a given subdomain or specific factor. Such models overcome item-level technical problems that typically plague symmetrical bifactor models, and yet model a bifactor structure (Burns et al., 2020). The bifactor S − 1 model provided a good fit to the full sample and the subsamples, withstanding a few item-level issues for some ethnic group subsamples, discussed below.

These results align with theory informing WARNS development that postulates a general risk factor and specific subfactors (George et al., 2015; Strand et al., 2019). Results suggest a general factor comprised of all possible item-level variance of the aggression-defiance domain (the reference factor), combined with a portion of the item-level variance for items comprising the other domains, which is separate from the covariance of the items comprising a given specific factor. That is, for the items of the subdomains other than aggression-defiance (the reference factor), item-level variance is reflected in loadings on both the general factor and loadings on the domain-specific factor. For most items, high loadings are evident on the general factor and acceptable loadings on the domain-specific factors. That is, after controlling for the general factor, there remains variance to be explained by the domain-specific factors. In addition, invariance criteria were met for each subsample, supporting measurement invariance for the instrument with regard to gender and ethnicity. Given these results, it is possible to explore how domain factors differ among groups or offer predictive value while accounting for the general risk and needs factor (Chen et al., 2005).

A keen observation by a reviewer pointed to the lack of gender mean differences on domains such as aggression-defiance and peer deviance, which literature suggests might be expected (Farrington, 2005; Onifade et al., 2010; Shaw et al., 2012). At the item level, we observed that items assessing overt aggression had higher levels of endorsement by boys compared with girls, whereas girls endorsed items tapping emotional reactiveness at higher levels compared with boys. Given such items appear on the same scale, the mean level was not different. Future work should explore whether domains need further refinement to capture these differences and whether these differences are not as pronounced at younger age groups compared with older age groups.

Reliability coefficients were generated to investigate the interpretability of the factors comprising the bifactor model. A general rule of thumb is that factor reliability greater than .95 allows for high-stakes decision-making, whereas values below that cutoff are interpreted with greater caution (Nunnally & Bernstein, 1994). For the total risk score, omegas were above the .95 cutoff for the full sample and all subsamples (both genders and the five ethnicities), revealing the applied interpretability of the total risk score. Omega hierarchical values reveal that none of the subscales had adequate reliability to be viewed as stable measures of these specific domains, independent of the general factor. However, it has been recommended that a minimum omega hierarchical value of .50 for specific factors in a bifactor model may support the usefulness of those scores, independent of the general factor (Reise et al., 2013). One subscale domain, depression-anxiety, met this criterion for the full sample and all subsamples, suggesting that for this subfactor only, reliability within the context of a bifactor model is at an interpretable level. That is, one can cautiously interpret scores for that domain independently of its contribution to total risk factor score. For other subscale domains, reliability values are insufficient for their interpretation beyond what they contribute to the general factor. That said, the item responses or scores may be used by school counselors to begin conversations with students to try to understand issues the student may be experiencing with respect to that specific domain.

Results support an interpretive framework that recognizes a total risk factor score for which an externalizing symptom domain (aggression-defiance) constitutes a reference factor for a bifactor model. Within that model, internalizing symptoms (depression-anxiety) reveal a level of reliability that allow for their cautious use. The emergence of a framework in which reliable contributors are internalizing and externalizing domains is consistent with other investigations of the factor structure of child psychopathology (i.e., Forbes et al., 2016) and similar domains have emerged as latent profiles that were predictive of justice system involvement (Iverson et al., 2018).

Although the advantages of the bifactor S − 1 model as the model of choice were identified previously, it is the case that issues of fit appeared for some ethnic subgroups for some items. For all subgroups, negative, insignificant, or very low (<.20) loadings were observed for Item 20 (“I skipped or cut class”) with regard to its loading on the school engagement subfactor. It contributes little or nothing to that domain once variance for the general factor is accounted for. Therefore, it would be useful to evaluate that item with regard to its wording or appropriateness for that subfactor. In addition, for one ethnic subgroup, Native Americans, seven items yielded negative or nonsignificant item loadings at the subfactor level. This suggests that something may be different for this ethnic group compared with the others in terms of meaning. Four of those seven items belong to the family environment subscale, one to the depression-anxiety subscale, and two to the school engagement subscale. Future studies should investigate why this group of students may be interpreting the item content differently compared with other groups. Think-aloud studies with students may help understand this difference. While those items loaded weakly at the subscale level, they loaded strongly on the general factor, revealing that for this group, like the others, interpretation of the general factor is supported by the findings, whereas interpretation of subfactors other than depression-anxiety is not. The other issue not addressed here is one of social desirability in responses, which has been observed in similar measures (e.g., Latkan et al., 2017). We do not have the data on this behavioral tendency, but it is something that may need to be investigated in the future. Related to this type of responding is the possibility that certain groups show a consistent pattern of response behaviors across domains (e.g., Bais et al., 2020). Methods could be applied to examine this issue, and if it is related to the differences, we observed across groups.

Our results align with those for a sample of high school students who completed the WARNS (Strand et al., 2019). It appears, however, that subfactor reliability is generally lower for the middle school sample than was found for the high school sample. It is also the case that a symmetrical bifactor model fit well for the high school sample but not for the middle school. This leads to the question of whether the differences reflect developmental level (age or grade), or other sample differences such as level of risk. Therefore, it would be worthwhile for future research to investigate a combined middle and high school sample, similarly recruited, to examine factor structure and measurement invariance across the grade levels. Such an investigation would allow for determining whether psychometric differences reflect developmental level or risk level.

In summary, results reveal good fit for a bifactor S − 1 model for the msWARNS for a middle school sample of children, and factorial invariance for gender and ethnicity subgroups. In addition, there is evidence for similar item-level functioning as indicated by pattern coefficients across all subgroups. Reliability analyses reveal reliability adequate for the interpretation of the total risk factor score and the depression-anxiety domain score. Scores for the other domains do not demonstrate a level of reliability allowing for their interpretation within the context of also interpreting the total risk factor score. With these limits in mind, the msWARNS provides practitioners an instrument with measurement properties that support usefulness for assessing risks and needs of middle school students who are diverse with respect to gender and ethnicity.

Footnotes

Author note

Brian F. French is now affiliated to Washington State University, Pullman, WA, USA.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The present work was supported by a grant from the U.S. Department of Education’s Institute for Education Sciences (R305A210087).

Methodological Disclosure

We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.

ORCID iD

Paul S. Strand

References

Alpizar

French

B. F.

T. T.

(2020). Equivalence testing of a youth risk and needs assessment. Journal of Psychoeducational Assessment, 38(8), 1046–1051. https://doi.org/10.1177/0734282920930892

Bais

Schouten

Toepoel

(2020). Investigating response patterns across surveys: Do respondents show consistency in undesirable answer behaviour over multiple surveys? Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 147(1–2), 150–168. https://doi.org/10.1177/0759106320939891

Baker

M. L.

Sigmon

J. N.

Nugent

M. E.

(2001). Truancy reduction: Keeping students in school. U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention.

Belfield

Levin

H. M.

(Eds.). (2007). The price we pay: Economic and social consequences of inadequate education. Brookings Institution Press.

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (2nd ed). Guilford Press.

Burns

G. L.

Geiser

Mateu

Becker

S. P.

Beauchaine

T. P.

(2020). Application of the bifactor S – 1 model to multisource ratings of ADHD/ODD symptoms: An appropriate bifactor model for symptom ratings. Journal of Abnormal Child Psychology, 48(7), 881–894. https://doi.org/10.1007/s10802-019-00608-4

Burton

C. M.

Marshal

M. P.

Chisolm

D. J.

(2014). School absenteeism and mental health among sexual minority youth and heterosexual youth. Journal of School Psychology, 52(1), 37–47. https://doi.org/10.1016/j.jsp.2013.12.001

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

Chen

F. F.

Sousa

K. H.

West

S. G.

(2005). Testing measurement invariance of second-order factor models. Structural Equation Modeling, 12(3), 471–492. https://doi.org/10.1207/s15328007sem1203_7

10.

Christenson

S. L.

Thurlow

M. L.

(2004). School dropouts: Prevention considerations, interventions, and challenges. Current Directions in Psychological Science, 13(1), 36–39. https://doi.org/10.1111/j.0963-7214.2004.01301010.x

11.

Dembo

Gulledge

L. M.

(2009). Truancy intervention programs: Challenges and innovations to implementation. Criminal Justice Policy Review, 20, 437–456. https://doi.org/10.1177/0887403408327923

12.

Dennis

M. L.

Chan

Funk

R. R.

(2006). Development and validation of the GAIN short screener (GSS) for internalizing, externalizing and substance use disorders and Crime/Violence problems among adolescents and adults. American Journal on Addictions, 15, 80–91. https://doi.org/10.1080/10550490601006055

13.

Eid

Geiser

Koch

Heene

(2017). Anomalous results in G-factor models: Explanations and alternatives. Psychological Methods, 22(3), 541–562. https://doi.org/2099/10.1037/met0000083

14.

Farrington

D. P.

(Ed.). (2005). Integrated developmental & life course theories of offending. Advances in criminological theory (Vol. 14). Transaction.

15.

Finney

S. J.

DiStefano

(2013). Non-normal and categorical data in structural equation modeling. In Hancock

G. R.

Mueller

R. O

(Eds.), Structural equation modeling: A second course (2nd ed., pp. 439–492). Information Age.

16.

Forbes

M. K.

Tackett

J. L.

Markon

K. E.

Krueger

R. F.

(2016). Beyond comorbidity: Toward a dimensional and hierarchical approach to understanding psychopathology across the life span. Development and Psychopathology, 28, 971–986. https://doi.org/10.1017/S0954579416000651

17.

French

B. F.

T. T.

(2020). Differential item functioning of a truancy assessment. Journal of Psychoeducational Assessment, 38(5), 642–648. https://doi.org/10.1177/0734282919863215

18.

George

Coker

French

Strand

Gotch

McBride

McCurley

(2015). Washington Assessment of the Risks and Needs of Students (WARNS) user manual. Washington State Center for Court Research, Administrative Office of the Courts.

19.

Gotch

C. M.

French

B. F.

(2020). A validation trajectory for the Washington Assessment of Risks and Needs of Students. Educational Assessment, 25(Special Issue on Validity Studies), 65–82. https://doi.org/10.1080/10627197.2019.1702462

20.

Green

Gesten

Greenwald

Salcedo

(2008). Predicting delinquency in adolescence and young adulthood: A longitudinal analysis of early risk factors. Youth Violence and Juvenile Justice, 6, 323–342. https://doi.org/10/1177/1541204008320261

21.

Grisso

Barnum

Fletcher

K. E.

Cauffman

Peuschold

(2001). Massachusetts youth screening instrument for mental health needs of juvenile justice youths. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 541–548. https://doi.org/10.1097/00004583-200105000-00013

22.

Hammond

Linton

Smink

Drew

(2007). Dropout risk factors and exemplary programs: A technical report. Clemson, SC: National Dropout Prevention Center, Clemson University.

23.

Henry

K. L.

Huizinga

D. H.

(2007). School-related risk and protective factors associated with truancy among urban youth placed at risk. The Journal of Primary Prevention, 28(6), 505–519. https://doi.org/10.1007/s10935-007-0115-7

24.

Howell

J. C.

(2003). Preventing and reducing juvenile delinquency: A comprehensive framework. SAGE.

25.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

26.

Iverson

French

B. F.

Strand

P. S.

Gotch

C. M.

McCurley

(2018). Understanding school truancy: Risk–need latent profiles of adolescents. Assessment, 25(8), 978–987. https://doi.org/10.1177/1073191116672329

27.

Johnson

C. L.

Wright

K. A.

Strand

P. S.

(2012). Transitions of truants: Community truancy board as a turning point in the lives of adolescents. Journal of Juvenile Justice, 1(2), 34–51. https://www.ncjrs.gov/pdffiles/238336.pdf

28.

Kane

M. T.

(2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. https://doi.org/10.1111/jedm.12000

29.

Latkan

C. A.

Edwards

Davey-Rothwell

M. A.

Tobin

K. E.

(2017). The relationship between social desirability bias and self-reports of health, substance use, and social network factors among urban substance users in Baltimore, Maryland. Addictive Behaviors, 73, 133–136. https://doi.org/10.1016/j.addbeh.2017.05.005

30.

Maynard

Vaughn

Nelson

Salas-Wright

Heyne

Kremer

(2017). Truancy in the United States: Examining temporal trends and correlates by race, age, and gender. Children and Youth Services Review, 81, 188–196. https://doi.org/10.1016/j.childyouth.2017.08.008

31.

Millsap

R. E.

(2011). Statistical approaches to measurement invariance. Routledge.

32.

Muthén

L. K.

Muthén

B. O.

(1998–2012). Mplus user’s guide (7th ed.).

33.

National Institute on Drug Abuse. (2009). Problem oriented screening instrument for children.

34.

National Juvenile Justice Network. (2011). Telling it like it is: Youth speak out on the school to prison pipeline.

35.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). McGraw-Hill.

36.

Nye

C. D.

Drasgow

(2011). Assessing goodness of fit: Simple rules of thumb simply do not work. Organizational Research Methods, 14, 548–570. https://doi.org/10.1177/1094428110368562

37.

Onifade

Nyandro

Davidson

Campbell

(2010). Truancy and patterns of criminogenic risk in a young offender population. Youth Violence and Juvenile Justice, 8, 3–18. https://doi.org/10.1177/1541204009338251

38.

Pett

M. A.

Lackey

N. R.

Sullivan

J. J.

(2003). Making sense of factor analysis. SAGE.

39.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667–696. https://doi.org/10.1080/00273171.2012.715555

40.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. https://doi.org/10.1080/00223891.2012.725437

41.

Rocque

Jennings

Piquero

Ozkan

Farrington

(2017). The importance of school attendance: Findings from the Cambridge study in delinquent development on the life-course effects of truancy. Youth Violence and Juvenile Justice, 63, 592–612. https://doi.org/10.1177/0011128716660520

42.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98, 223–237. https://doi.org/10.1080/00223891.2015.1089249

43.

Sampson

R. J.

Laub

J. H.

(2005). A general age-graded theory of crime: Lessons learned and the future of life-course criminology. In Farrington

D. P.

(Ed.), Integrated developmental & life course theories of offending. Advances in criminological theory (Vol. 14, pp. 165–182). Transaction.

44.

Shaw

Hyde

Brennan

(2012). Early predictors of boys’ antisocial trajectories. Development and Psychopathology, 24, 871–888. https://doi.org/10.1017/S0954579412000429

45.

Skedgell

Kearney

C. A.

(2018). Predictors of school absenteeism severity at multiple levels: A classification and regression tree analysis. Children and Youth Services Review, 86, 236–245. https://doi.org/10.1016/j.childyouth.2018.01.043

46.

Steinberg

(2014). Age of opportunity: The new science of adolescence. Houghton, Mifflin, Harcourt.

47.

Strand

P. S.

Gotch

C. M.

French

B. F.

Beaver

J. L.

(2019). Factor structure and invariance of an adolescent risks and needs assessment. Assessment, 26, 1105–1116. https://doi.org/10.1177/1073191117706021

48.

Strand

P. S.

Lovrich

N. P.

(2014). Graduation outcomes for truant students: An evaluation of a school-based, court-engaged community truancy board with case management. Children and Youth Services Review, 43, 138–144. https://doi.org/10.1016/j.childyouth.2014.05.008