Evidence for the Technical Adequacy of the Risk Assessment and Distress Recovery: Examining Risks While Avoiding Profiling Youth With Mental Health Difficulties

Abstract

This study examined the psychometric properties of the Risk Assessment and Distress Recovery (RADR) Scale—a new self-report psychological screening tool for high school students that explores risk factors related to mental health characteristics of students who have engaged in school violence. The sample consisted of 1093 high school students from all four regions of the United States. A four-factor model consisting of coping skills, depression, suicidal ideation, and narcissism demonstrated good fit. Results of confirmatory factor analysis and measurement invariance, as well as internal consistency estimates, provide initial evidence for reliability and validity of the RADR. Exploratory analyses between the RADR and characteristics of past school shooters are also presented. Students with disabilities self-reported significantly greater risks for mental health concerns across all four constructs, with small (.15) to large (.80) effect sizes.

Keywords

bullying depression narcissism school shootings threat assessment

Students with internalizing mental health difficulties are less identifiable by school practitioners than those students who act out with externalizing aggressive or noncompliant behaviors (Weist et al., 2007). As a result, universal screening of students for mental health difficulties has become more common in schools across the United States (Dever et al., 2016; Kamphaus et al., 2014; Stiffler & Dever, 2015). The purpose of universal screening is not to diagnose any student with a disorder or disability; rather, the objective is to identify those students who exhibit characteristics putting them at elevated risk for disorders (Vannest, 2012). Still, long-standing concerns in universal screening include deciding specifically which types of mental health difficulties to screen for among school-age youth (Dowdy et al., 2010), as well as the risk of identifying more students for services than a school can reasonably be expected to serve (Dever et al., 2012; Moore et al., 2019). As school administrators and practitioners grapple with these concerns, there has been a growing call among multiple agencies to deliver more mental health services to at-risk youth in schools. Specifically, in the wake of school shootings across the United States, there is now a growing emphasis on youths’ mental health needs (Katsiyannis et al., 2018).

Although gun violence at schools is rare (Landrum et al., 2019), school-associated homicide and suicide of youths aged 14–19 are on the rise in the last decade (2010–2019) and currently stand at unprecedented levels (Katsiyannis et al., 2018). After the 2018 shooting at Marjory Stoneman Douglas (MSD) High School in Parkland, FL, the Interdisciplinary Group on Preventing School and Community Violence called for more preventative mental health services within K-12 schools (Astor et al., 2018). The relationship between mental health difficulties and gun violence is much more complicated than common public opinion suggests and the overwhelming majority of youth suffering with mental health difficulties will never attack anyone (Metzl & MacLeish, 2015), but data from the U.S. Department of Education, Secret Service, and the Federal Bureau of Investigation (FBI) indicate certain psychological problems have been common among previous school shooters (National Threat Assessment Center, 2019; Silver et al., 2018; Vossekuil et al., 2004). Especially concerning is that in most cases, these students were not receiving psychological services that may have helped remediate their struggles. If a screening tool were available to identify this constellation of mental health characteristics, it may help identify students struggling with certain difficulties that are often overlooked in schools. The purpose of such a tool would not be to label students as potential school shooters; rather, the goal would be to identify students who are struggling in silence and deliver the mental health services they need in an effort to reduce the possibility of future violent acts in schools.

Violence in Schools

More than 228,000 students have experienced gun violence at school since the 1999 Columbine High School shooting in Littleton, CO (Cox et al., 2019). Schools are supposed to be places where students feel safe and free to engage in a variety of learning opportunities (Katsiyannis et al., 2018; Klinger & Klinger, 2018). However, national data on crime victimization indicate students are 50% more likely to be the victims of violence at school than away from school (Musu-Gilette et al., 2017). According to data from the Centers for Disease Control and Prevention (2016), nearly 6% of students report missing school for fear of their own safety and 4% indicate they have brought a weapon to school in the last 30 days. Even more disheartening, the 21st century has already seen more deaths related to school shootings than all that occurred in the 20th century (dating back to the first known school shooting in 1940; Katsiyannis et al., 2018).

At the beginning of the 21st century, the Safe School Initiative—a collaborative effort between the U.S. Department of Education and the US Secret Service—identified 37 incidents of targeted school violence (including 41 participants) from 1974–2000 (Vossekuil et al., 2004). Although they ultimately concluded there was no useful profile of a would-be school shooter, there were several important findings about the attackers in their report. Seventy-six percent of attackers were white, and 85% were between the ages of 13–18. Most were suicidal (78%) or experiencing extreme depression (61%) and 83% exhibited outwardly an inability to cope with loss, but only 17% had been diagnosed with a psychological disorder prior to their attack. Seventy-one percent of students had experienced bullying. Despite these alarming statistics, nearly half of attackers were doing well academically, had social contacts with mainstream students, and were involved in organized social activities (Vossekuil et al., 2004).

Fifteen years after publication of the Safe School Initiative (Vossekuil et al., 2004), the US Secret Service published an analysis of 41 incidents of school violence in the United States from 2008 to 2017 (National Threat Assessment Center, 2019). Once again, a major conclusion from the report indicated there was no useful profile for a potential attacker. Nearly all (94%) attackers had undergone adverse childhood experiences (ACEs) in their homes (e.g., parental divorce, domestic abuse, drug use, or criminal charges among family members). Similar to the previous report, 80% of attackers had been bullied in school and the two most common psychological problems of shooters were depression (63%) and suicidal ideation (60%). The next most common psychological disorder was anxiety, but only in 29% of cases (National Threat Assessment Center, 2019). The report also examined personalities of previous school shooters and identified two prominent traits. The most common trait of the attackers was narcissism (e.g., inflated sense of self and lack of empathy). Narcissism is one of the three character traits known as the “Dark Triad” (Miller et al., 2012) and is listed by the FBI as a risk factor for school shooters (Bondü & Scheithauer, 2011). The second most prominent trait, which was also identified in the Safe School Initiative report (Vossekuil et al., 2004), was a need for better coping skills to deal with life stressors (National Threat Assessment Center, 2019). Despite these alarming statistics, only half of the attackers had received any kind of mental health counseling prior to the attack. Thus, the main conclusion by the report was to conduct more in-school mental health evaluations of all students in tandem with threat assessment procedures.

Threat Assessment and Mental Health Screening

Threat assessment is the most commonly recommended practice from the U.S. Department of Education, Secret Service, and the FBI for keeping schools safe (Katsiyannis et al., 2018). Threat assessment management (TAM) teams in schools enable practitioners to document and evaluate the seriousness of threatening behavior by students (Goodrum et al., 2018). They do so by identifying threats made by students (in person or online), assessing the validity of the threat by talking to the student and others who know them well, and managing the situation by seeking to understand the reasons for the threat and intervening accordingly (American Psychological Association, 2018). Although a worthy investment in schools, TAM teams are not focused on students’ mental health. Furthermore, despite TAM teams being present in more and more schools, incidents of school shootings have increased (Katsiyannis et al., 2018). This is not to suggest TAM teams are not necessary; indeed, they are crucial to helping prevent school violence. However, TAM teams alone are not sufficient to address the issue of school shootings, and they do not focus on students’ mental health. The utilization of TAM teams to prevent school shootings is recommended, but even the National Threat Assessment Center (2019) suggests there is a need to supplement TAM teams with targeted mental health services.

Some schools conduct additional systematic screenings of their student population for emotional and behavioral disorders. Although there are multiple screening tools available to assess students’ for mental health difficulties, teachers are the most common informants and many, due in part to the large numbers of students whom they serve, struggle to conduct screening in a regular and reliable fashion without significant training (Oakes et al., 2014). Furthermore, these tools most often emphasize problem behaviors related to inattention and aggression (Bruhn et al., 2014), rather than the mental health disorders that have most commonly appeared in past school shooters (e.g., depression, suicidality, narcissism, and struggles to cope with loss). Although there are well-known measures available for assessing depression among high school students, costs for purchasing these materials can be prohibitive for schools. For example, the Reynolds Adolescent Depression Scale, 2nd Edition (Reynolds, 2008), which is the most well-known measure, costs $217 for the manual and 25 test booklets. Each additional set of 25 costs $84. For districts with thousands of high school students, the annual cost can be hard to justify and schools may not be able to afford proper screening instruments. There are also several freely available instruments to assess depression (e.g., Patient Health Questionnaire-9; Spitzer et al., 1999), but these scales also do not assess constructs such as narcissism or coping skills. As such, schools need a comprehensive tool to address depression, suicidality, narcissism, and coping struggles.

Purpose

By creating the first reliable and feasible self-report screening instrument that systematically targets depression, suicidal ideation, narcissism, and poor coping skills, we provide the field with a useful tool that allows school practitioners to identify and implement targeted supports to students who are struggling with often unidentified struggles. This tool can be used in conjunction with threat assessment procedures in schools nationwide. Specifically, this new tool will help schools meet the charge of the Interdisciplinary Group on Preventing School and Community Violence (Astor et al., 2018), which called for greater attention to mental health in our schools.

The goals of the current study are to examine the psychometric properties of the newly developed Risk Assessment and Distress Recovery (RADR) scale and to explore relations between RADR constructs and the psychological characteristics and risk factors related to past school shooters in the United States. The purpose is most definitely not to identify students similar to past school shooters and then implement punitive measures. Rather, the purpose is to use the RADR as part of a universal screening process and to identify those students exhibiting a combination of mental health difficulties that have in the most extreme cases resulted in school violence when students were missed (i.e., slipped through the cracks) as being in need of mental health services.

Method

The sample in this study was drawn from a population of high school students across the United States. In collaboration with a marketing research firm, we targeted all four regions of the United States (Northeast, Midwest, South, and West) seeking to identify high school students in grades 9–12. The marketing firm maintained a national dataset of parents who had agreed to let their children consider completing online surveys. Our goal was to garner responses from at least 1000 students in a 1-week period. With direction from the lead author of the current study, the marketing firm sent email invitations to parents across the United States with information about the study and a hyperlink for their children to complete the online survey, if they chose. All procedures were approved by the first author’s Institutional Review Board at a major research university.

Participants

The final sample obtained by the end of the recruitment week was 1093 high school students from across the United States (see Table 1). Students from all but two states (New Mexico and North Dakota) responded to the survey, with the following regional breakdown: 39.61% (n = 433) South, 26.08% (n = 285) Midwest, 18.66% (n = 204) West, and 15.65% (n = 171) Northeast. The sample was 51.24% female and the average age of participants was 16.59 years (SD = 1.74, range = 13–19). The majority of the sample was white (73.11%). A total of 252 students (23.1%) identified as having a disability requiring special education services. Of this group, the most common disability was attention-deficit/hyperactivity disorder (ADHD; 33.73%, n = 85).

Table 1.

Demographic Characteristics of the Sample (N = 1093).

Characteristic		n	%
Gender
	Female	560	51.2%
	Male	533	48.8%
Race/ethnicity
	White	799	73.1%
	Black	131	12.0%
	Latinx	63	5.8%
	Asian	46	4.2%
	Multiracial	18	1.6%
	American Indian	13	1.2%
	Others	23	2.1%
Grade
	9	198	18.1%
	10	153	14.0%
	11	147	13.4%
	12	594	54.3%
Special education
	Not receiving services	841	76.9%
	Disability	252	23.1%

Note. Disability includes students receiving special education services under IDEA as well as ADHD under 504 plans. ADHD = Attention-Deficit/Hyperactivity Disorder.

Procedures

Consent to participate in the study was assumed on the part of the students given their willingness to complete an online survey after being delivered an electronic information sheet about the purpose of the project. Students were informed there were researchers interested in developing a new measure to assess the psychological wellbeing of high school students in the United States. Although students had the option to stop taking the survey at any time, all 1093 respondents provided complete data for analysis. We did not provide any financial incentive to participants for their involvement in the study. Students completed the survey in February of 2020.

Measure

The RADR scale was developed by the first author for the purposes of assessing four components of psychosocial wellbeing of high school students. Using REDCap, which is a secure web platform for building and managing online databases and surveys, we created the instrument. Prior to psychological wellbeing questions, students were asked six demographic questions (e.g., gender, race, age, grade level, disability status, and state of residence) and five questions regarding home and school experiences. Students were asked about (a) their involvement in bullying, (b) whether or not they had ever been suspended or expelled, (c) incidents of ACEs in their home (e.g., parental divorce, drug use or criminal charges against family, or domestic abuse), (d) whether or not a relationship with a significant other had ended in the last 6 months, and (e) whether there were guns in their home. Next, there were 32 items assessing students’ wellbeing across domains of depression, suicidality, narcissism, and coping struggles. As students navigated these questions, we included a note at the top of each section that provided students with the national suicide prevention lifeline phone number.

A three-stage process was undertaken to create the RADR. First, we reviewed commonly used measures of depression (e.g., Reynolds Adolescent Depression Scale, 2nd Edition; Reynolds, 2008), suicidal ideation (e.g., Suicidal Ideation Questionnaire; Reynolds, 1987), narcissism (e.g., Narcissistic Personality Inventory; Raskin & Terry, 1988), and coping skills (Adolescent Coping Orientation for Problem Experiences; Patterson & McCubbin, 1987) to determine appropriateness of items and constructs being assessed relative to our purposes. Second, we reviewed each scale in comparison to the Diagnostic and Statistical Manual of Mental Health Disorders—5th edition (DSM-5; American Psychiatric Association, 2018), which contains updated information regarding markers of these mental health disorders. These first two steps allowed us to create new questions assessing these four constructs. Finally, we shared the draft of the measure with first-year college students at a major research university to ascertain their opinions of the measure being used with high-school age students. Based on their feedback, we reworded several items throughout the measure. Coping skills, suicidal ideation, and depression were rated on a four-point Likert-type scale (0 = Not at all; 1 = Rarely; 2 = Sometimes; and 3 = Frequently). Coping skills items included questions such as When things in life get tough, how often do you focus on the positives in your life? Suicidal ideation items included questions such as In the past year, how often have you intentionally hurt yourself (e.g., cutting, pulling out hair)? Depression included questions such as In the past six months, how often have you felt so sad that you felt unable to get out of bed for no particular reason? Narcissism was rated on a four-point Likert-type scale (0 = Definitely disagree; 1 = Disagree somewhat; 2 = Agree somewhat; and 3 = Definitely agree) and asked students to indicate their level of agreement with various statements, such as I deserve attention because I am better than most other people I know.

Data Analysis

First, we examined the structure of RADR by conducting a confirmatory factor analysis (CFA) based on the theory-driven constructs used to design the measure. Specifically, using a structural equation modeling (SEM) approach, we created four latent factors: (a) coping skills, with nine items; (b) suicidal ideation, with four items; (c) depression, with nine items; and (d) narcissism, with ten items. We then examined factor loadings for each item and removed any items with standardized factor loadings less than 0.40 (Brown, 2006; Brown & Moore, 2012). Next, we evaluated model fit statistics. Specifically, we examined the chi-square statistic and Hu and Bentler’s (1999) joint criteria: Comparative Fit Index (CFI) and Tucker–Lewis Index (TLI) ≥ .95, the root mean square error of approximation (RMSEA) ≤ .06, and standardized root mean squared residual (SRMR) ≤ .08. If model fit criteria were not met, we examined modification indices to improve model fit. We used a conservative approach to modification indices by only adding correlations between observed items within factors to ensure all additional parameters fit within the theoretical model (Kline, 2011). The CFA was estimated using maximum likelihood in the Lavaan package (Rosseel, 2012) in (R Core Team, 2013).

Measurement invariance

Following the identification of an acceptable structural model, we examined invariance by student gender, disability, and race. We tested whether or not there was factorial invariance, or bias, by student characteristics following the procedures outlined by Milsap and Olivera-Aguilar (2012). First, we estimated a baseline model–based CFA model with four fixed factors for both groups (e.g., men and women). The baseline model represents configural invariance and says that the same number of factors holds for each group and the same variables define each factor across groups. Next, we restricted the factor loadings to invariance across the two groups. This model is said to represent metric invariance or weak factorial invariance (Hirschfeld & von Brachel, 2014). If metric invariance holds across the two groups, any population differences in the covariances between the measured variables are due to the common factors (Milsap & Olivera-Aguilar, 2012). Then, if metric invariance was found, we placed invariance constraints on the measurement intercepts, which is known as scalar invariance or strong factorial invariance (Hirschfeld & von Brachel, 2014). Metric invariance is required before scalar invariance is modeled because differences in factor loadings imply that the regressions of the measured variables on the factor scores are not parallel across groups, thus the intercepts are also not likely to be the same. We evaluated invariance for each group (i.e., gender, disability, and race) following Milsap and Olivera-Aguilar’s recommended (2012) sequence of models (i.e., configural, metric, and scalar) and compared each successive model using ΔCFI as prior research has noted concerns with likelihood ratio tests (e.g., Chen, 2007). Sellbom and Tellegen (2019) recommend ΔCFI ranging from .002 to .010 as a threshold for invariance; therefore, we used ΔCFI <0.010 as our indicator of invariance. All models were estimated in Lavaan (Rosseel, 2012).

Exploratory analyses

Next, we examined the relation between each of the four factors and related indicators associated with potential gun violence. Specifically, we regressed the four factors on students’ reports of bullying and self-report of suspension/expulsion, ACEs, ending a romantic relationship, and the presence of a gun in the home regressed on the four factors. The model also included the following demographic characteristics covariates: grade, gender, disability status, and race. We used an SEM approach in Lavaan (Rosseel, 2012) to estimate the relations among observed and latent variables.

Last, we conducted exploratory analyses to begin understanding the potential relation between RADR and characteristics of past school shooters. First, we explored the relation between the four latent factors and the presence of having a gun in the house. Research suggests that having a gun in the home significantly increases the likelihood of youth access to a firearm (Choi et al., 2017). Next, we examined profiles of students at or above two SDs on depression, suicidality, and narcissism, as well as those at or below two SDs on coping skills, which we considered to be an indicator of risk for each of the four RADR constructs. We created dichotomous indicators for risk for each of the factors. Then, we estimated logistic regression models and calculated odds ratios for student characteristics (i.e., age, gender, disability, and race) and associated predictors (bullying and self-report of suspension/expulsion, ACEs, ending a romantic relationship, and the presence of a gun in the home) predicting risk status for each RADR factor. Finally, we examined significant differences on mental health constructs and risk factors by disability status. Cohen’s (1992) d (M1–M2/σpooled) effects were defined as small (.2), medium (.5), and large (.8).

Results

Confirmatory Factor Analysis

First, we estimated a four-factor model with all original RADR items loading on their hypothesized factors. Upon review of the factor loadings, we removed six items due to low factor loadings (three from the coping skills and three from narcissism). Next, we examined the model fit statistics and found the reduced model did not meet the a priori criteria for acceptable fit (see Table 2). Therefore, we examined modification indices and added 11 residual correlations between items, all within factors (i.e., correlations must have been theoretically related). For example, we correlated the following two indicators of the depression factor: (a) In the last 6 months, how often have you felt like things would never get better? and (b) In the last 6 months, how often have you felt so sad that you thought it would last forever? The addition of the residual correlations improved model fit to meet our a priori criteria. The final model’s parameter estimates are presented in Table 3. Examination of Cronbach’s alpha for each of the four latent constructs indicated good to excellent internal consistency: coping skills (

α

= .82), suicidal ideation (

α

= .94), depression (

α

= .95), and narcissism (

α

= .90). See Figure 1 for the final RADR scale.

Table 2.

CFA Model Fit Statistics.

Parameter	Hypothesized Model	Reduced Model	Final Model
χ²	3257.1	1684.3	1199.2
Df	458	293	282
p-value	<.001	<.001	<.001
CFI	.52	.93	.95
TLI	.48	.92	.95
RMSEA	.08	.07	.05
SMRM	.25	.07	.07

Note. CFI = Comparative Fit Index; TLI = Tucker–Lewis Index; RMSEA = root mean square error of approximation; SRMR = standardized root mean squared residual.

Table 3.

RADR CFA Model Results.

Parameters			Std. est.	Est.	se	z	p-value
Factor correlations
Coping skills	∼∼	Suicidal ideation	−.04	−.01	.01	−1.06	.287
Coping skills	∼∼	Depression	−.06	−.02	.01	−1.72	0.086
Coping skills	∼∼	Narcissism	.14	.05	.01	3.92	<.001
Suicidal ideation	∼∼	Depression	.73	.51	.03	16.88	<.001
Suicidal ideation	∼∼	Narcissism	.51	.33	0.03	13.01	<.001
Depression	∼∼	Narcissism	.45	.30	.03	11.58	<.001
Factor loadings
Coping skills
Item 1			.45	1.00	.00
Item 2			.58	1.01	.08	12.32	<.001
Item 3			.76	1.30	.09	13.77	<.001
Item 4			.76	1.42	.10	13.77	<.001
Item 5			.77	1.45	.10	13.84	<.001
Item 6			.66	1.38	.11	13.10	<.001
Suicidal ideation
Item 1			.93	1.00	.00
Item 2			.89	.88	.02	39.76	<.001
Item 3			.84	.85	.02	35.19	<.001
Item 4			.80	.76	.03	29.81	<.001
Depression
Item 1			.80	1.00	.00
Item 2			.84	1.02	.03	37.11	<.001
Item 3			.85	.98	.03	32.76	<.001
Item 4			.81	.98	.03	30.95	<.001
Item 5			.83	.96	.03	31.70	<.001
Item 6			.85	1.00	.03	32.93	<.001
Item 7			.77	.88	.03	28.89	<.001
Item 8			.81	.95	.03	30.54	<.001
Item 9			.87	1.00	.03	33.82	<.001
Narcissism
Item 1			.78	1.00	.00
Item 2			.61	.74	.04	20.12	<.001
Item 3			.85	1.04	.04	29.43	<.001
Item 4			.76	.89	.03	26.12	<.001
Item 5			.68	.86	.04	22.72	<.001
Item 6			.64	.75	.04	21.10	<.001
Item 7			.78	.91	.03	26.66	<.001

Figure 1.

Risk Assessment and Distress Recovery Scale.

Measurement Invariance

We modeled configural, metric, and scalar invariance of the RADR by gender, disability, and race. Gender was coded as male and female, disability was coded as typically developing and disability, and race was coded white or not white. The CFI change was .003 for both the comparison between the configural and metric, and the metric and scalar comparisons. Review of the standardized coefficients suggested only minimal differences by indicators when comparing men and women. Overall, the results suggest measurement invariance for the RADR by gender. Next, we examined invariance by disability. The CFI change was 0.006 for configural and metric, and for metric and scalar. Given our a priori CFI threshold, evidence of measurement invariance was established by disability status. Last, we examined measurement invariance by race. Again, we found CFI change between configural and metric, and metric and scalar to be .006. Therefore, the results suggest support for measurement invariance by race (white and not white).

Exploratory Analyses

We examined the relations between the four latent factors of the RADR and students’ self-report of suspension/expulsion, ACEs, ending a romantic relationship, the presence of a gun in the home, and bullying. Approximately 20% of students reported that they had been suspended or expelled, 32% reported experiencing at least one of the ACEs, 18% reported ending a romantic relationship, and 31% had a gun at home. We modeled suspension/expulsion, ACEs, ending a romantic relationship, the presence of a gun in the home predicting the RADR factors, and the four factors of RADR predicting bullying involvement. Table 4 presents the results of the four variables predicting the RADR factors, controlling for gender, age, race, and disability status. Of the four predictors of interest, only having a gun in the home predicted coping skills. The coefficient was positive, suggesting that students with a gun in the home have more coping skills. Model results suggest that having a history of being suspended or expelled, breaking up with a romantic partner, and having a gun in the home were positively and significantly associated with suicidal ideation. All four predictors were positively and significantly associated with depression. Last, we found that a history of suspension or expulsion and breaking up with a romantic partner were both positively associated with narcissism.

Table 4.

RADR Predicting Suspensions/Explusions, ACEs, Breaking Up with a Romantic Partner, and Having Guns in the Home.

Parameters	Est.	Se	p-value	Std. est.
Coping skills
Susp/expul	−.07	.04	.095	−.06
ACEs	.05	.04	.164	.05
Romantic partner	−.05	.05	.270	−.04
Guns in home	.10	.04	.005	.10
Gender	.02	.03	.616	.02
Age	−.03	.01	.007	−.09
White	−.03	.04	.431	−.03
Disability	.01	.04	.758	.01
Suicidal ideation
Susp/expul	.19	.06	.002	.09
ACEs	.10	.05	.061	.06
Romantic partner	.66	.07	<.001	.31
Guns in home	.12	.05	.015	.07
Gender	.04	.05	.415	.02
Age	.02	.01	.109	.04
White	.04	.05	.440	.02
Disability	.44	0.06	<.001	0.23
Depression
Susp/expul	.24	.07	<.001	.11
ACEs	.21	.05	<.001	.12
Romantic partner	.58	.07	<.001	.26
Guns in home	.11	.05	.033	.06
Gender	−.07	.05	.140	−.04
Age	.01	.01	.591	.01
White	−.07	.05	.151	−.04
Disability	.43	.06	<.001	.22
Narcissism
Susp/expul	.25	.07	<.001	.12
ACEs	.02	.06	.749	.01
Romantic partner	.39	.07	<.001	.19
Guns in home	.08	.05	.143	.05
Gender	.27	.05	<.001	.17
Age	−.02	.01	.275	−.03
White	.07	.05	.171	.04
Disability	.20	.06	.001	.10

Note. Susp/expul = suspension or expulsion; ACES = adverse childhood experiences; RADR = Risk Assessment and Distress Recovery.

Table 5 presents the results for the association between RADR and bullying. For reference, 21% reported being a victim of bullying, 3% reported being a bully, and 5% reported being a bully victim (71% reported no bullying involvement). Students reporting bully victimization were more likely to report higher levels of depression. Although a small number of students self-reported being a bully, they were more likely to have higher suicidal ideation and narcissism. A similar result was found for self-reported bully victims, with higher suicidal ideation and narcissism.

Table 5.

Risk Assessment and Distress Recovery Predicting Bullying Involvement.

Parameter	Victim				Bully				Bully victim
Parameter	Est.	se	p-value	Std. est.	Est.	se	p-value	Std. est.	Est.	se	p-value	Std. est.
Gender	.00	.02	.914	.00	.01	.01	.480	.02	.00	.01	.990	.00
Age	−.04	.01	<.001	−.19	.00	.00	.258	.03	.00	.00	.498	.02
White	.02	.03	.501	.02	.00	.01	.744	−.01	−.04	.01	.003	−.09
Disability	.19	.03	<.001	.19	.03	.01	.023	.07	.03	.02	.076	.06
Coping skills	.01	.03	.816	.01	.01	.01	.345	.03	−.01	.01	.512	−.02
Suicidal ideation	−.03	.02	.045	−.06	.05	.01	<.001	.25	.03	.01	<.001	.12
Depression	.14	.02	<.001	.29	−.01	.01	.090	−.05	.01	.01	.326	.03
Narcissism	−.02	.02	.208	−.04	.03	.01	<.001	.13	.03	.01	.004	.10

Next, we examined characteristics of students that were significantly at-risk across the four latent factors. Specifically, we identified students that were two or more SDs below the mean for coping skills and two or more SDs above the mean for suicidal ideation, depression, and narcissism. We used a conservative approach to evaluating statistically significant odds ratios because of the large number of multiple comparisons. Specifically, we considered odds ratios significant if the p-value was below .01. The results are presented for all factors and predictors in Table 6. None of the odds ratios were significant for students in the coping skills risk group. Students with disabilities were significantly more likely to be in the suicidal ideation risk group (OR = 4.1). Students in the depression risk group were more likely to have a disability (OR = 6.8), report having been suspended or expelled (OR = 2.9), and ended a romantic relationship (OR = 4.8). Last, students in the narcissism risk group (i.e., very high levels of narcissism) were more likely to be older (OR = 1.5), have a disability (OR = 5.2), and ended a romantic relationship (OR = 3.7).

Table 6.

RADR Risk Profiles.

Factors predictors		95% CI
Factors predictors	OR	Lower	Upper	p-value
Coping skills
Age	1.22	1.03	1.49	.032
Gender	.88	.51	1.51	.643
White	1.43	.79	2.50	0.219
Disability	.87	.42	1.66	.686
Suspension/expulsion	1.87	.86	3.80	.096
ACEs	.44	.20	.91	.036
Romantic partner	.86	.32	1.99	0.743
Gun	.72	.36	1.33	.332
Victim	0.57	.03	3.10	.597
Bully	0.52	0.21	1.12	0.121
Bully victim	.76	0.12	2.80	.719
Suicidal ideation
Age	1.19	1.03	1.39	.022
Gender	1.68	1.05	2.73	.032
White	.94	.54	1.58	.821
Disability	4.08	2.56	6.54	<.001
Suspension/expulsion	1.32	.73	2.37	.096
ACEs	1.23	.69	2.15	0.036
Romantic partner	4.11	2.33	7.25	.743
Gun	1.14	.68	1.91	.313
Victim	6.65	2.68	16.30	.697
Bully	1.77	.96	3.21	.121
Bully victim	3.77	1.64	8.33	.719
Depression
Age	1.49	1.19	1.93	.022
Gender	.92	.49	1.71	.032
White	.92	0.43	1.81	.821
Disability	6.81	3.66	12.98	<.001
Suspension/expulsion	2.90	1.37	6.26	.006
ACEs	2.16	1.00	4.83	.053
Romantic partner	4.76	2.25	10.38	<.001
Gun	1.84	.94	3.63	0.076
Victim	2.02	.66	5.96	.208
Bully	1.21	.53	2.69	.642
Bully victim	1.33	.41	3.82	.609
Narcissism
Age	1.49	1.19	1.93	0.001
Gender	1.71	.92	3.28	.093
White	.94	.44	1.85	.861
Disability	5.16	2.80	9.67	<.001
Suspension/expulsion	1.92	.90	4.09	.089
ACEs	1.41	.67	2.98	.357
Romantic partner	3.66	1.74	7.74	0.001
Gun	1.99	1.03	3.86	.039
Victim	3.14	1.03	9.16	.390
Bully	1.38	.61	3.03	0.425
Bully victim	1.88	.60	5.31	.251

Note. ACEs = adverse childhood experiences; RADR = Risk Assessment and Distress Recovery

There were four students (three male, one female; two black, one white, and one multiracial) who met all of the following criteria: (a) scored two or more SDs below the mean for coping skills and two or more SDs above the mean for suicidal ideation, depression, and narcissism; (b) had experienced being bullied; and (c) answered Yes to all four risk factors (e.g., previous suspension/expulsion, experienced ACEs, romantic relationship ended in last 6 months, and guns in the home). These four students (ages 16–18) were all receiving special education services (3 for learning disability [LD] and 1 for ADHD). Finally, we examined group differences by disability status across risk factors and the four mental health characteristics. Results (see Table 7) indicated students with disabilities were significantly more likely to have been suspended or expelled (d = .67, p < .001), experience ACEs (d = .42, p < .001), recently breakup with a romantic partner (d = .54, p < .001), get bullied (d = .40, p < .001), and bully others students (d = .70, p < .001). Students with disabilities self-reported significantly lower scores on coping skills (d = .15, p = .043) and significantly higher scores on depression (d = .78, p < .001), suicidal ideation (d = .80, p < .001), and narcissism (d = .40, p < .001).

Table 7.

Differences in Responses by Disability Status.

	Students without disabilities (n = 841)		Students with disabilities (n = 252)		Effect size
Question/variable	M	SD	M	SD	Cohen’s d
Have you ever been suspended (in school or out) or expelled from school?	.14	.34	.39^***	.49	.67
Have any one of the following situations occurred in your home: (a) Parental divorce or separation, (b) drug use or criminal charges among family members, or (c) domestic abuse?	.27	.44	.46^***	.50	.42
In the last six months, has a relationship with a romantic partner ended or gotten worse?	.13	.34	.33^***	.47	.54
Do you get bullied by other students?	.02	.12	.08^***	.28	.40
Do you bully other students?	.15	.36	.42^***	.49	.70
Do you both bully and get bullied by other students?	.03	.18	.10^***	.29	.30
Coping skills	12.04	4.00	11.46^∗	4.01	.15
Depression	6.82	7.03	12.48^***	7.87	.78
Suicidal ideation	1.10	2.57	3.42^***	3.85	.80
Narcissism	7.33	5.02	9.45^***	5.92	.40

Note. *p < .05, ***p < .001; For the first six questions, options were Yes (1) or No (0); Total scores possible for coping skills (0–18), depression (0–27), suicidal ideation (0–12), and narcissism (0–21).

Discussion

The purpose of this study is to detail the psychometric properties of a new mental health screening tool for high school students. To be clear, the purpose is not to (a) profile students as would-be school shooters, (b) stigmatize students with mental health concerns or disabilities, or (c) replace current threat assessment procedures taking place in schools across the United States. Instead, our sole focus is to provide schools and practitioners with a user-friendly, psychometrically sound, and free, all-in-one screening tool to identify students suffering with mental health concerns not often or easily identified in schools. Identification, based on RADR screening, should lead to further mental health evaluation and, if necessary, intervention. The RADR was developed based on all recently available reports regarding characteristics of previous active shooter incidents from the U.S. Department of Education (Vossekuil et al., 2004), FBI (Silver et al., 2018), and Secret Service (National Threat Assessment Center, 2019). A common conclusion from all three reports is there is no useful profile of an active/school shooter. However, their conclusions also suggest more needs to be done to address mental health in the effort to diminish further school shootings incidents. We believe the RADR may provide useful information to help schools in their mission. For example, if students are found to rate high on narcissism (i.e., a trait not currently included in universal screening tools), intervention approaches for parents and teachers focused on affection and self-esteem rather than peer comparison and superiority may be needed (Brummelman et al., 2016).

Results of our analysis indicated measurement invariance across students based on gender, race, and disability status, indicating high utility of the measure. The four latent constructs of coping skills, suicidal ideation, depression, and narcissism also showed good to excellent internal consistency. We also found students who were more depressed and more suicidal were also more likely to have been suspended or expelled and to have a gun in their home. Given the dangers of access to firearms, especially for mentally fragile youth (Choi et al., 2017), these findings are particularly disconcerting. Similar to past research, our results confirm being bullied is related to higher levels of self-reported depression for school-age youth (Hunt et al., 2012; Klomek et al., 2011). Interestingly, we found students who bullied others (but did not experience bullying themselves) rated high on suicidal ideation and narcissism. The higher ratings on narcissism make sense, as recent research confirms narcissism positively predicts bullying perpetration (Farrell & Vallaincourt, 2019). However, the fact that bully perpetration positively predicts suicidal ideation could indicate these students are engaging in bullying behaviors as a cry for help. However, more research is needed to better understand the potential connection.

Students with Disabilities

One group of students from the current sample deserves particular attention. Students with disabilities rated higher on depression, suicidal ideation, and narcissism. Prior research has confirmed students with disabilities are more depressed than their peers (Cullinan & Sabornie, 2004; Garwood et al., 2017; Maag & Reid, 2006; Nelson & Harwood, 2011; Van Loan et al., 2019) and more at risk for suicide (McMillan & Jarvis, 2013). However, we are unaware of prior research indicating youth with disabilities rate higher on narcissism than their typically developing peers. Of particular concern is the finding that all four students who (a) scored in the high risk range (i.e., two or more SDs below the mean for coping skills and two or more SDs above the mean for suicidal ideation, depression, and narcissism), (b) had been bullied, (c) indicated they had experienced ACEs, suspension/expulsion, a recent breakup with a romantic partner, and (d) had a gun in the home, all had disabilities (ADHD or LD). Given data from the US Secret Service (National Threat Assessment Center, 2019) indicating 29% of previous attackers showed symptoms of or were diagnosed with ADHD and 20% of past shooters had neurological disorders (e.g., LD), our data indicate these four students from our sample are in need of intensive intervention. Granted, these students represent only 0.4% of our sample, but school shootings are typically conducted by just one student. Therefore, identifying those very few students most at-risk across all four factors and intervening as early as possible is a primary goal of the RADR. Interestingly, no students identified with emotional disturbance (ED) fit into this category. It could be the students with LD and ADHD may have qualified for services under ED but received a less stigmatizing label (Kauffman & Badar, 2013), or that those identified with ED were getting the proper psychological services to help remediate their internal struggles.

Limitations

All efforts were made to ensure our findings were accurate and generalizable. However, a number of limitations necessitate discussion. First, although we surveyed a large sample of high school students from across the United States, there were more white students and slightly more students with disabilities than in the general student population. Second, all demographic variables, including disability status, were student self-reports. We were unable to confirm the demographics from either the students’ parents or their schools. Third, for parsimony, we did not include a full measure of ACEs, but instead simply asked if students had experienced the ACEs of interest based on reports of school shooters (National Threat Assessment Center, 2019). As such, the findings do not directly align with prior ACEs research. Fourth, of the four factors, the coping skills factor had the lowest factor loadings and was the least correlated with most of the predictors in the exploratory analyses. It is unclear if the results are due to the items or the relation between coping skills and the hypothesized predictors of school shootings. Future research should continue to explore the relative value of the coping skills items and the factors’ relation with other observed constructs of interest. Finally, the use of modification indices to make adjustments to our measurement models slightly limits the generalizability of the RADR. Still, these indices are commonly used in measurement studies.

Conclusion

Mental health supports need to be used in conjunction with threat assessment procedures to prevent future school tragedies (National Threat Assessment Center, 2019). The RADR will help schools meet the charge of the Interdisciplinary Group on Preventing School and Community Violence, which called for greater attention to mental health in our schools (Astor et al., 2018), and the request in the House of Representatives Bill H.R. 4301 introduced to the Secretary of Education, Attorney General, and Secretary of Health and Human Services, to collect annual data from schools on safety and prevention measures absent or in place at the time of a shooting. Schools using the RADR alongside other universal screening tools, such as those targeting anxiety and externalizing disorders, could demonstrate their willingness to give proper attention to the mental health needs of the students.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was funded by the UVM CESS Innovation Fund Research Council and Jean S. Garvin Fellowship Fund.

ORCID iD

Justin D. Garwood

References

American Psychological Association (2018). Threat assessment in action. Retrieved from http://www.apa.org/monitor/2014/02/cover-threat.aspx.

Astor

R.A.

Bear

G.G.

Bradshaw

C.P.

Cornell

D.G.

Espelage

D. L.

Flannery

(2018). Call for action to prevent gun violence in the United States of America. University of Virginia.

Bondü

Scheithauer

(2011). Explaining and preventing school shootings: Chances and difficulties of control. In Heitmeyer

Haupt

H.-G.

Malthaner

Kirschner

(Eds.), Control of violence: Historical and international perspectives on violence in modern societies (pp. 295-314). Springer.

Brown

T. A.

(2006). Confirmatory factor analysis for applied research. Guilford Press.

Brown

T. A..

Moore

M. T.

(2012). Confirmatory factor analysis. In Hoyle

R. H.

(Ed.), The handbook of structural equation modeling (pp. 361-379). Guilford Press.

Bruhn

A. L.

Woods-Groves

Huddle

(2014). A preliminary investigation of emotional and behavioral screening practices in K-12 schools. Education & Treatment of Children, 37, 611-634.

Brummelman

Thomaes

Sedikides

(2016). Separating narcissism from self-esteem. Current Directions in Psychological Science, 25(1), 8-13.

Centers for Disease Control and Prevention (2016). Fact sheet: Understanding school violence. Retrieved from https://www.cdc.gov/violenceprevention/pdf/school_violence_fact_sheet-a.pdf.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464-504 doi:10.1080/10705510701301834

10.

Choi

N. G.

DiNitto

D. M.

Marti

C. N.

(2017). Youth firearm suicide: Precipitating/risk factors and gun access. Children and Youth Services Review, 83, 9-16.

11.

Cohen

(1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98-101.

12.

Cox

J. W.

Rich

Chiu

Muyskens

Ulmanu

(2019). More than 215,000 students have experienced gun violence at school since Columbine. Retrieved from www.washingtonpost.com/graphics/2018/local/school-shootings-database/.

13.

Cullinan

Sabornie

E. J.

(2004). Characteristics of emotional disturbance in middle and high school students. Journal of Emotional and Behavioral Disorders, 12, 157-167.

14.

Dever

B. V.

Raines

T. C.

Barclay

C. M.

(2012). Chasing the unicorn: Practical implementation of universal screening for behavioral and emotional risk. School Psychology Forum: Research in Practice, 6, 108-118.

15.

Dever

B. V.

Raines

T. C.

Dowdy

Hostutler

(2016). Addressing disproportionality in special education using a universal screening approach. The Journal of Negro Education, 85(1), 59-71.

16.

Dowdy

Furlong

Eklund

Saeki

Ritchey

(2010). Screening for mental health and wellness: Current school based practice and emerging possibilities. In Doll

Pfohl

Yoon

(Eds.), Handbook of youth prevention science (pp. 70-95). Routledge.

17.

Farrell

A. H.

Vallaincourt

(2019). Developmental pathways of perfectionism: Associations with bullying perpetration, peer victimization, and narcissism. Journal of Applied Developmental Psychology, 65, 1-10.

18.

Garwood

J. D.

Varghese

Vernon-Feagans

(2017). Internalizing behaviors and hyperactivity/inattention: Consequences for young struggling readers, and especially boys. Journal of Early Intervention, 39, 218-235.

19.

Goodrum

Thompson

A. J.

Ward

K. C.

Woodward

(2018). A case study on threat assessment: Learning critical lessons to prevent school violence. Journal of Threat Assessment and Management, 5, 121-136.

20.

Hirschfeld

von Brachel

(2014). Multiple-group confirmatory factor analysis in R-A tutorial in measurement invariance with continuous and ordinal indicators. Practical Assessment, Research & Evaluation, 19(7), 1-12.

21.

L.T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.

22.

Hunt

Peters

Rapee

R. M.

(2012). Development of a measure of the experience of being bullied in youth. Psychological Assessment, 24, 156-165.

23.

Kamphaus

R. W.

Reynolds

C. R.

Dever

B. V.

(2014). Behavioral and mental health screening. In Kettler

R. J.

Glover

T. A.

Albers

C. A.

Feeney-Kettler

K. A.

(Eds.), School psychology book series. Universal screening in educational settings: Evidence-based decision making for schools (pp. 249-273). American Psychological Association.

24.

Katsiyannis

Whitford

D. K.

Ennis

R. P.

(2018). Historical examination of United States intentional mass school shootings in the 20th and 21st centuries: Implications for students, schools, and society. Journal of Child and Family Studies, 27, 2562-2573.

25.

Kauffman

J. M.

Badar

(2013). How we might make special education for students with emotional or behavioral disorders less stigmatizing. Behavioral Disorders, 39(1), 16-27.

26.

Kline

R. B.

(2011). Principals and practice of structural equation modeling (3rd ed.). Guilford Press.

27.

Klinger

(2018). Keeping students safe every day. ASCD.

28.

Klomek

A. B.

Kleinman

Altschuler

Marrocco

Amakawa

Gould

M. S.

(2011). High school bullying as a risk for later depression and suicidality. Suicide and Life Threatening Behavior, 41, 501-516.

29.

Landrum

T. J.

Sweigart

C. A.

Collins

L. W.

(2019). School shootings: What we know, what we can do. Educational Leadership, 77, 36-41.

30.

Maag

J. W.

Reid

(2006). Depression among students with learning disabilities: Assessing the risk. Journal of Learning Disabilities, 39, 3-10.

31.

McMillan

J. M.

Jarvis

J. M.

(2013). Mental health and students with disabilities: A review of literature. Journal of Psychologists and School Counsellors, 23, 236-251.

32.

Metzl

J. M.

MacLeish

K. T.

(2015). Mental illness, mass shootings, and the politics of American firearms. American Journal of Public Health, 105, 240-249.

33.

Miller

J. D.

Few

L. R.

Seibert

Watts

Zeichner

Lynman

D. R.

(2012). An examination of the dirty dozen measure of psychopathy: A cautionary tale about the costs of brief measures. Psychological Assessment, 24, 1048-1053.

34.

Milsap

R. E.

Olivera-Aguilar

(2012). Investigating measurement invariance using confirmatory factor analysis. In Hoyle

R. H.

(Ed.), The handbook of structural equation modeling (pp. 380-392). Guilford Press.

35.

Moore

S. A.

Mayworm

A. M.

Sharkey

J. D.

Dowdy

(2019). Languishing students: Linking complete mental health screening in schools to tier 2 intervention. Journal of Applied School Psychology, 35(3), 257-289.

36.

Musu-Gillette

Zhang

Wang

Zhang

Oudekerk

B.A.

(2017). Indicators of school crime and safety: 2016 (NCES 2017- 064/NCJ 250650). U.S. Department of Justice.

37.

National Threat Assessment Center (2019). Protecting America’s schools: A U.S. Secret service analysis of targeted school violence. Department of Homeland Security.

38.

Nelson

J. M.

Harwood

H. R.

(2011). A meta‐analysis of parent and teacher reports of depression among students with learning disabilities: Evidence for the importance of multi‐informant assessment. Psychology in the Schools, 49, 371-384.

39.

Oakes

W. P.

Lane

K. L.

Germer

K. A.

(2014). Developing the capacity to implement tier 2 and tier 3 supports: How do we support our faculty and staff in preparing for sustainability. Preventing School Failure, 58, 183-190.

40.

Patterson

J. M.

McCubbin

H. I.

(1987). Adolescent coping style and behaviors: Conceptualization and measurement. Journal of Adolescence, 10, 163-186.

41.

R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrived from http://www.R-project.org/.

42.

Raskin

Terry

(1988). A principal-components analysis of the narcissistic personality inventory and further evidence of its construct validity. Journal of Personality and Social Psychology, 54, 890-902.

43.

Reynolds

(1987). Suicidal Ideation Questionnaire. Psychological Assessment Resources.

44.

Reynolds

W. M.

(2008). Reynolds adolescent depression scale (2nd ed). Psychological Assessment Resources.

45.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36 http://www.jstatsoft.org/v48/i02/.

46.

Sellbom

Tellegen

(2019). Factor analysis in psychological assessment research: Common pitfalls and recommendations. Psychological Assessment, 31(12), 1428-1441.

47.

Silver

Simons

Craun

(2018). A study of the pre-attack behaviors of active shooters in the United States between 2000-2013. Federal Bureau of Investigation, U.S. Department of Justice.

48.

Spitzer

R. L.

Kroenke

Williams

J. B. W.

, & the Patient Health Questionnaire Study Group (1999). Validity and utility of a self-report version of PRIME-MD: The PHQ primary care study. Journal of the American Medical Association, 282, 1737-1744.

49.

Stiffler

M. C.

Dever

B. V.

(2015). Mental health screening at school: Instrumentation, implementation, and critical issues. Springer.

50.

Van Loan

C. L.

Garwood

J. D.

Smith

S. W.

Daunic

A. P.

(2019). Take CHARGE! A randomized controlled trial of a social problem-solving curriculum to support students with emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 27, 143-153.

51.

Vannest

K. J.

(2012). Implementing interventions and progress monitoring subsequent to universal screening. School Psychology Forum: Research in Practice, 6(4), 119-136.

52.

Vossekuil

Fein

Reddy

Borum

Modzeleski

(2004). The final report an findings of the Safe School Initiative: Implications for the prevention of school attacks in the United States. U.S. Department of Education.

53.

Weist

M. D.

Rubin

Moore

ADelsheim

Wrobel

(2007). Mental health screening in schools. Journal of School Health, 77(2), 53-58.