Using the Depression,Anxiety,Stress Scales–21 With U.S. Adolescents: An Alternate Models Analysis

Abstract

As part of universal screening efforts in schools, validated measures that identify internalizing distress are needed. One promising available measure, the Depression, Anxiety, and Stress Scales–21 (DASS–21), has yet to be thoroughly investigated with adolescents in the United States. This study investigated the underlying factor structure of the DASS–21 in a sample of U.S. adolescents (N = 2,454) by using confirmatory factor analytic techniques to test several alternate models. A bifactor model specifying general Negative Affectivity and three specific factors of Depression, Anxiety, and Stress yielded the best fit. Results from this study suggest that (a) the DASS–21 scales reflect a common factor, indicating that a total score of the DASS–21 can be derived as a measure of general negative affectivity, and (b) the DASS–21 may not adequately differentiate between the experiences of negative affectivity, anxiety, and stress in U.S. adolescents.

Keywords

DASS–21 internalizing problems bifactor analysis factor analysis

A staggering proportion of adolescents are affected by mental health problems. For example, a recent U.S. national survey of adolescents found that 49.5% of the sample was affected by at least one mental health disorder over their lifetime, with 27.6% of these youth experiencing severe impairment (Merikangas et al., 2010). However, approximately 75% to 80% of youth in need of mental health services do not receive them (Kataoka, Zhang, & Wells, 2002), with particularly high rates of unmet needs among minority youth with internalizing distress (Gudiño, Lau, Yeh, McCabe, & Hough, 2009). Universal school-based screening is recommended as a first step in efforts to prevent, identify, and treat those currently facing, or at risk of, emotional and behavioral problems (Glover & Albers, 2007). Particularly important in universal screening efforts is the identification of youth experiencing internalizing problems. Given the range of negative outcomes associated with internalizing distress, including suicide, poor school performance, employment difficulties, and substance use (e.g., Reid, Gonzalez, Nordness, Trout, & Epstein, 2004), there is a critical need to identify and offer supports to students experiencing internalizing symptomatology. Although teachers are often the source of referrals for additional support in schools, teachers tend to under-identify students experiencing internalizing problems (e.g., Dowdy, Doane, Eklund, & Dever, 2013; Papandrea & Winefield, 2011). Rather, adolescents are increasingly recognized as ideal informants about their own internalizing states, with self-report measures being most useful for identifying internalizing problems in these youth (Smith, 2007). The goal of the current article is to examine the factor structure of one self-report measure of internalizing problems, the Depression, Anxiety, Stress Scales–21 (DASS–21), as a first step in assessing its potential usefulness as a universal screening tool for U.S adolescents.

Internalizing Problems in Adolescence

Internalizing problems are often grouped into the two primary subtypes of depression and anxiety (e.g., Zahn-Waxler, Klimes-Dougan, & Slattery, 2000). At any given time, between 2% and 18% of youth may meet criteria for an anxiety or depressive disorder, with the prevalence of internalizing problems increasing markedly during adolescence (Zahn-Waxler et al., 2000). Latino/a students, a population of students particularly relevant within the current study, may be at particular risk of internalizing distress given the higher prevalence rates and greater risk of depressive and anxiety symptoms when compared with other groups (Roberts, Roberts, & Chen, 1997; Varela & Hensley-Maloney, 2009; Zychinski & Polo, 2012). In addition, by adolescence, females are 2 times more likely than males to become anxious or depressed, with this trend continuing into adulthood (Zahn-Waxler et al., 2000). Latino male students, similarly, report fewer internalizing symptoms than their female peers (Sirin et al., 2015; Umaña-Taylor & Updegraff, 2007). In schools, internalizing problems have been linked to low academic achievement, difficulties with concentration and social relationships, and students’ perceptions of their school workload being too difficult (e.g., Fröjd et al., 2008).

Frequent comorbidity among internalizing problems (Zahn-Waxler et al., 2000) has led some scholars to question the distinction of the two internalizing subtypes (i.e., anxiety and depression; Clark & Watson, 1991). In their seminal review of various self-report measures of anxiety and depression, Clark and Watson (1991) observed poor discriminant validity across internalizing measures, leading to the conclusion that internalizing problems are best understood using a tripartite model that delineates a non-specific distress factor—negative affectivity—as a core component of each internalizing subtype. Negative affectivity has been described as a mood-dispositional dimension reflecting being upset or unpleasantly engaged and is characteristic of subjective feelings of nervousness, tension, and worry; negative view of self; and affective states such as anger, scorn, revulsion, guilt, and sadness (Watson & Clark, 1984). In addition to negative affectivity, Clark and Watson further delineated specific, and unique, factors of depression and anxiety, noting that the conditions are related through underlying negative affectivity, but are otherwise largely independent of each other. In particular, depression is distinguished from experiences of anxiety through its association with the absence of positive affect (i.e., zest for life, energy, delight, interest, enthusiasm, pride; Clark & Watson, 1991). In addition, anxiety, but not depression, is characterized by nervous tension and autonomic symptomatology, such as rapid breathing or heart rate (i.e., physiological hyperarousal; Clark & Watson, 1991). In research investigating the experiences of depression and anxiety among child and adolescent psychiatric patients, Joiner, Catanzaro, and Laurent (1996) found the tripartite model to be a good fit for their youth sample. Extending the tripartite model to assessment of internalizing problems posits that, to obtain an accurate representation of internalizing experiences, both the common (i.e., negative affectivity) and unique elements of internalizing problems need to be assessed: “general distress, the physiological tension and hyperarousal of anxiety, and the pervasive anhedonia of depression” (Clark & Watson, 1991, p. 331).

The DASS–21, an abbreviated version of Lovibond and Lovibond’s 42-item Depression, Anxiety, Stress Scales (DASS), is one measure that has been designed to assess the severity of internalizing symptoms associated with depression (e.g., hopelessness, anhedonia), anxiety (e.g., autonomic arousal, anxious affect), and stress (e.g., nervous arousal, ease to agitation; Lovibond & Lovibond, 1995). Research examining the DASS–21 has suggested that it is a viable tool for measuring both the general and specific features of internalizing problems (Brown, Chorpita, Korotisch, & Barlow, 1997; Tully, Zajac, & Venning, 2009; Willemsen, Markey, Declercq, & Vanheule, 2011). For example, Brown and colleagues (1997) have suggested that a strength of the DASS is its ability to assess the internalizing dimensions of depression, anxiety, and negative affectivity. Similarly, Tully and colleagues (2009) have suggested a conceptualization of the DASS–21 that supports Clark and Watson’s (1991) tripartite model of depression and anxiety. Others have argued for even broader conceptualizations of the usefulness of the DASS–21, for example, as a quadripartite measure of depression, anxiety, stress, and negative affectivity (Henry & Crawford, 2005; Szabo, 2010).

Universal Self-Report Screening for Internalizing Problems

The goal of universal screening that uses self-reports is to provide youths the opportunity to voice if they are experiencing distress or are at risk of later problems so that they may receive additional assessment and intervention services (Young, Sabbah, Young, Reiser, & Richardson, 2010). To achieve this goal, J. M. Levitt, Saka, Romanelli, and Hoagwood (2007) explain that cross-domain instruments, designed to provide more general information about individuals’ emotional or behavioral experiences, are best suited for universal screening. Although several such measures are currently used for universal screening of internalizing problems (see Jenkins et al., 2014, for a review of commonly used screeners), many of the currently available screening instruments are limited with regard to key psychometric properties for screening (e.g., sensitivity, specificity, positive predictive value; see J. M. Levitt et al., 2007, for a review of psychometric properties of many available screening tools). In particular, given that internalizing problems are often more difficult to detect than externalizing problems within a school setting (Cunningham & Suldo, 2014; Kamphaus, Reynolds, & Dever, 2014), low diagnostic accuracy estimates are often found when predicting internalizing problems (e.g., Kamphaus & Reynolds, 2007; Lane et al., 2009). For example, the self-report version of the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997) has been found to have high specificity (.96) and negative predictive value (.96), but low sensitivity (.29) and positive predictive value (.29; Goodman, 2001). Similarly, the Behavioral and Emotional Screening System (BESS; Kamphaus & Reynolds, 2007) has been found to have high specificity (.96) and negative predictive value (.93) as well as acceptable positive predictive value (.75), but has low sensitivity for detecting true, clinical-level problems (.64; Kamphaus & Reynolds, 2007). Given the limitations of existing and widely used screening tools, particularly with regard to accurately identifying internalizing distress, research is needed to determine whether alternative screening instruments designed to assess for internalizing distress are viable for use in U.S. schools. This study aims to provide initial psychometric information for one instrument, the DASS–21, that may be useful when screening for internalizing problems in schools.

The Depression, Anxiety, Stress Scales–21

The DASS–21 is widely used in Australia and the United Kingdom and has been proposed as an effective tool in identifying internalizing distress (Lovibond & Lovibond, 1995). However, much of the existing knowledge about the DASS–21 is limited to empirical investigations with non-U.S. adults (e.g., Henry & Crawford, 2005; Ng et al., 2007) and adolescents (e.g., Mellor et al., 2015; Szabo, 2010). Research conducted using the DASS–21 in the United States includes assessment of depression, anxiety, and stress in college students (e.g., Tull & Gratz, 2008) and pregnant women (e.g., Huang et al., 2014), as well as examinations of the psychometric properties of the scale with clinical and non-clinical adult populations (e.g., Antony, Beiling, Cox, Ens, & Swinson, 1998; Osman et al., 2012).

Although research has examined the use of the DASS–21 with adolescents in Western nations aside from the United States, caution is frequently called for when using assessments outside of the population with whom the tool was originally developed (e.g., Geisinger, 1994; Hui & Triandis, 1985). For example, cultural and linguistic differences among the original and target populations may influence individuals’ understanding of, and response to, the scale’s items and may also affect researchers’ and practitioners’ ability to compare results obtained within the target population with those obtained from the development population. Furthermore, research on the use of assessments across cultures emphasizes the need to validate the similarity of constructs measured in the new populations with whom the tool was not originally developed (Hui & Triandis, 1985). One method for assessing cross-cultural equivalence of a scale is to examine the internal structure (i.e., factor structure) of the construct being measured by the instrument; a construct that is equivalent across cultures should have the same internal structure (i.e., components and relations among components) across cultures and countries (Hui & Triandis, 1985). Analyzing the factor structure of the DASS–21with U.S. adolescents, therefore, represents an essential preliminary step in determining the usefulness of the scale with this population. However, research investigating the factor structure of the DASS–21 with U.S. adolescents is currently lacking.

Factor Analytic Studies of the DASS–21

Previous investigations into the factor structure of the DASS–21 have supported various internal structures, including a one-factor solution (e.g., Patrick, Dyck, & Bramston, 2010), a three-factor solution (e.g., Antony et al., 1998; Lovibond & Lovibond, 1995), or bifactor solutions including a general Negative Affectivity factor with either two (i.e., Depression/Anhedonic Depression and Anxiety/Physiological Hyperarousal; Tully et al., 2009; Willemsen et al., 2011) or three specific factors (i.e., Depression, Anxiety, Stress; Henry & Crawford, 2005). Many of these investigations into the factor structure of the DASS–21, however, made modifications to their models, such as allowing correlated errors, cross loadings, or altering the components of the factors to attain a well-fitting model. Such practices can be problematic in both interpretation of individuals’ results and in understanding the composition of the measure across samples (Osman et al., 2012). For example, when item errors are allowed to correlate, authors may be capitalizing on the characteristics of their specific sample(s), rather than representing patterns in the population as a whole (Brown, 2006). In addition, the implications for deriving or interpreting scores on the modified instrument have not been addressed (Osman et al., 2012).

The current study aims to address the noted gaps in the existing research of the factor structure of the DASS–21, particularly with U.S. adolescents. Lovibond and Lovibond (1995) proposed a three-factor structure of the DASS–21, wherein subscale scores of Depression, Anxiety, and Stress/Tension can be derived for adolescents. Research support for bifactor models, however, implies that a total score may also be derived from an individual’s DASS–21 responses (Brown, 2006). Moreover, because support has been found for bifactor models with both two and three specific factors, it is of theoretical interest to evaluate whether a general Negative Affectivity factor explains a significant proportion of the variance in DASS–21 items, and, if so, whether deriving subscale scores is practically meaningful. The present investigation uses confirmatory factor analytic (CFA) techniques to examine the factor structure of the DASS–21 among U.S. adolescents by testing several alternative models suggested by previous research with individuals in other Western nations. A secondary aim is to examine implications of the best fitting model for use of the DASS–21 in screening for internalizing distress among U.S. adolescents.

Method

Participants

Participants in the present study included 2,454 adolescents (52.4% male) attending one of two high schools in Southern California (N_{School 1} = 574, N_{School 2} = 1,880) and represent 84.2% and 90.2% of students enrolled at each school, respectively. The sample was comprised of 25.8% ninth-grade students, 29.7% 10th-grade students, 23.7% 11th-grade students, and 20.3% 12th-grade students. Grade-level information was unavailable for 15 students (0.6%). Data on ethnicity, free and reduced price lunch status, or socioeconomic circumstances were not collected for these individual participants. However, school-level data indicate students from School 1 were 87.6% Hispanic or Latino/a; 4.8% non-Hispanic White; 2.1% Asian, Filipino, or Pacific Islander; 4.8% African American; 0.3% American Indian or Alaska Native; and 0.3% of two or more races. Students at School 2 were 54.6% Hispanic or Latino/a; 37.6% non-Hispanic White; 3.6% Asian, Filipino, or Pacific Islander; 0.8% African American; 0.6% American Indian or Alaska Native; and 2.5% of two or more races. School-level data indicated that 82.1% of students attending School 1 and 48.2% of students attending School 2 were classified as socioeconomically disadvantaged (i.e., students who were eligible for free or reduced price school meals). All surveys were available in both English and Spanish. Less than 1% of students elected to use the Spanish form. Only surveys completed in English were used in the present study.

Measure

Depression Anxiety Stress Scales–21 (DASS–21)

The DASS–21 is a self-report measure designed to assess the negative emotional states of depression, anxiety, and stress (Lovibond & Lovibond, 1995; measure available from www.psy.unsw.edu.au/dass/). This 21-item measure is comprised of three subscales (Depression, Anxiety, Stress) that each contain seven items. The Depression subscale assesses dysphoria, hopelessness, devaluation of life, self-deprecation, lack of interest, anhedonia, and inertia. The Anxiety subscale assesses autonomic arousal, skeletal muscle effects, situational anxiety, and anxious affect. The Stress subscale measures levels of chronic non-specific arousal, assessing difficulty relaxing, nervous arousal, and ease to agitation, irritability, and impatience. Respondents use a 4-point response scale ranging from 0 (did not apply to me at all) to 3 (applied to me most of the time) to rate the extent to which each item applied to them over the past week. A sum score is created for each scale and then doubled to correspond to scores on the 42-item DASS, which aids in the interpretation of severity of scores. Across adult and adolescent samples, the internal consistency of each scale is high (Depression αs = .97 to .88, Anxiety αs = .92 to .79, Stress αs = .95 to .81; total α = .93), and high correlations are reported across the three scales (Antony et al., 1998; Tully et al., 2009). The DASS–21 has strong convergent validity with other measures assessing internalizing symptoms (rs from .68 to .79; Antony et al., 1998).

Consistent with the use of scales in different linguistic and cultural contexts, and given that the DASS–21 was developed 20 years ago for use with Australian adults, the clarity of items for use with U.S. youths was explored. Using a cognitive reviewing process (Willis, 1999), we asked undergraduate college students (n = 33) to complete the original-worded DASS–21 and to also provide item-by-item responses regarding the ease of understanding, commonality of words/phrases used by U.S. students, and relevance of items to students’ experiences. The students were also asked to offer wording modifications. The original wording was maintained for 10 items. The results of this cognitive review identified 11 items for possible minor revision to enhance interpretability. The researchers reviewed all student comments and made minor wording adjustments to enhance readability (see Table 1 for item wording for the items used in this study [right column]).

Table 1.

DASS–21 Item Wording Modifications for Use With U.S. Sample.

Item	Subscale	Original DASS–21 item wording	U.S. modified item wording
1	S	I found it hard to wind down.	I found it hard to relax and wind down.
2	A	I was aware of dryness of my mouth.	I was aware that my mouth was very dry.
3	D	I couldn’t seem to experience any positive feeling at all.	Same
4	A	I experienced breathing difficulty (e.g., excessively rapid breathing, breathlessness in the absence of physical exertion).	I experienced breathing difficulty (e.g., breathless or rapid breathing in the absence of physical exertion).
5	D	I found it difficult to work up the initiative to do things.	I didn’t have the initiative to do things.
6	S	I tended to over-react to situations.	Same
7	A	I experienced trembling (e.g., in the hands).	I experienced trembling or shaking.
8	S	I felt that I was using a lot of nervous energy.	I had a lot of nervous energy.
9	A	I was worried about situations in which I might panic and make a fool of myself.	Same
10	D	I felt that I had nothing to look forward to.	Same
11	S	I found myself getting agitated.	Same
12	S	I found it difficult to relax.	Same
13	D	I felt down-hearted and blue.	I felt sad, down, and blue.
14	S	I was intolerant of anything that kept me from getting on with what I was doing.	I was intolerant of anything that got in the way of what I was doing
15	A	I felt I was close to panic.	I felt overwhelmed and close to panic.
16	D	I was unable to become enthusiastic about anything.	Same
17	D	I felt I wasn’t worth much as a person.	Same
18	S	I felt that I was rather touchy.	I felt that I was easily annoyed and sensitive.
19	A	I was aware of the action of my heart in the absence of physical exertion (e.g., sense of heart rate increase, heart missing a beat).	I was aware of my heartbeat (e.g., rapid or irregular) in the absence of physical exertion.
20	A	I felt scared without any good reason.	Same
21	D	I felt that life was meaningless.	Same

Note. DASS–21 = Depression Anxiety Stress Scales–21; S = Stress; A = Anxiety, D = Depression.

Procedure

Data were collected in the fall semester of 2014 as part of universal mental health screening at two high schools. Prior to screening, consent forms were sent to the parents of all enrolled students at each school. Following the university’s human participants committee approval, passive parental consent and student assent to participate were attained prior to screening. A total of 117 parents and three students declined consent to participate. Five additional parents declined consent for their students’ data to be used in research.

During the first month of school, students with consent completed the screening survey, including the DASS–21, via paper-and-pencil format during one course period. If students were absent from class on the initial screening day, five additional attempts were made to allow students to participate in screening. On the day of screening, teachers were provided with a script for proctoring administration of the measures and explaining the purpose and use of screening results to students (i.e., to better support students at their school). Via the script, teachers informed students that their responses would be used by counselors and other school staff to inform support efforts for students at their school. In the event that survey results raised concerns, students were told that a follow-up meeting with their counselor would be held to plan for additional supports. Students identified as at-risk or in need of additional support were referred to school counselors and administrative staff. School personnel used these data, in conjunction with other existing school data (e.g., attendance, grades) to determine referrals for school-based prevention and intervention programs.

Statistical Analyses

Multiple CFA models were specified to assess which of several proposed models in the existing DASS–21 literature fit best with an adolescent U.S. sample. Data screening and descriptive analyses were performed using SPSS 22, and all models were specified using Mplus Version 7.1 (Muthén & Muthén, 1998-2013). Given that the DASS–21 uses a 4-point response scale, indicator variables were treated as ordinal categorical (Beauducel & Herzberg, 2006). Thus, weighted least squares with mean and variance adjustment (WLSMV) estimation was used on the input correlation matrix (e.g., Rhemtulla, Brosseau-Liard, & Savalei, 2012). To evaluate the fit of the tested models, the following fit indices were examined: chi-square test of model fit, comparative fit index (CFI), Tucker–Lewis index (TLI), and root mean square error of approximation (RMSEA). Model fit was assessed using criteria set forth by Hu and Bentler (1998) and Browne and Cudeck (1989): Good model fit was indicated by CFI and TLI values greater than .95, and RMSEA values less than .05; adequate fit was indicated by CFI and TLI values between .90 and .95, and RMSEA values between .05 and .08. As chi-square tests are heavily influenced by sample size (Fabrigar, Wegener, MacCallum, & Strahan, 1999), and given the large sample of the present study, chi-square values were expected to be significant for all models; models with chi-square significance values of p < .05 but with adequate or good fit according to each of the other fit indices were considered to have at least adequate fit.

Four models, informed by theory and previous investigations into the structure of the DASS–21, were tested. First, a one-factor model was specified, with each of the 21 items loading onto a single general factor, Negative Affectivity (e.g., Patrick et al., 2010). Second, the original three-factor model consisting of the correlated Depression, Anxiety, and Stress factors (Lovibond & Lovibond, 1995) was specified. Third, a bifactor model including a general Negative Affectivity factor, onto which all items were allowed to load, and orthogonal specific factors of Depression, Anxiety, and Stress, was specified (e.g., Henry & Crawford, 2005). Finally, a second bifactor model was tested, wherein a general Negative Affectivity factor and two orthogonal specific factors of Depression and Anxiety were specified (e.g., Tully et al., 2009). In this final model, all 21 of the DASS–21 items were included; the items originally hypothesized to load onto the Stress factor were allowed to load only onto the general Negative Affectivity factor. All models were tested without allowing correlated error terms or modifying the structure of the instrument.

The first and second models were specified using unit loading identification; the first item in each scale (i.e., Item 3—Depression, Item 2—Anxiety, and Item 1—Stress) was set as the reference variable and its unstandardized factor loading was constrained to 1. The third and fourth tested models utilized unit variance identification, whereby factor variances were constrained to 1, so that all item loadings on the general and specific factors could be directly compared. To determine whether significant improvements in fit were indicated with each model, adjusted chi-square difference tests accounting for WLSMV estimation were performed; statistically significant decreases in chi-square values indicate significant improvements in model misfit (Brown, 2006).

Results

Data Screening and Descriptive Statistics

Data screening conducted prior to CFA analyses indicated only one major violation of normality (Item 21, skew = 2.15; Curran, West, & Finch, 1996). However, because the WLSMV estimation method used does not assume normality, all DASS–21 items were retained for the present analyses. Table 2 shows the correlations, means, and standard deviations for each item. The factor that each item was designated to load on is indicated next to the item number. Correlations across all 21 items indicated weak to strong positive correlations among all variables, with all variables suspected of loading onto each hypothesized scale exhibiting moderate to strong positive relations with each other. Table 3 presents means and standard deviations for the total score and Depression, Anxiety, and Stress subscale scores overall and by gender. Independent samples t tests with equal variances not assumed indicated that for the total score and each subscale, female students reported significantly greater levels of internalizing distress than did male students.

Table 2.

Correlation Matrix for DASS–21 Items.

Item number/scale	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21
3. Depression	—
5. Depression	.56	—
10. Depression	.55	.54	—
13. Depression	.51	.45	0.57	—
16. Depression	.55	.55	.62	.61	—
17. Depression	.48	.44	.58	.64	.57	—
21. Depression	.48	.43	.55	.59	.54	.74	—
2. Anxiety	.38	.33	.29	.29	.33	.27	.26	—
4. Anxiety	.43	.42	.4	.37	.42	.39	.37	.38	—
7. Anxiety	.37	.35	.42	.45	.46	.44	.41	.32	.47	—
9. Anxiety	.34	.32	.43	.44	.41	.41	.36	.26	.33	.45	—
15. Anxiety	.44	.43	.5	.59	.57	.57	.51	.29	.46	.54	.5	—
19. Anxiety	.31	.28	.34	.34	.37	.37	.38	.26	.45	.42	.34	.44	—
20. Anxiety	.39	.39	.46	.51	.48	.51	.53	.27	.44	.51	.49	.59	.47	—
1. Stress	.38	.29	.24	.26	.27	.2	.19	.35	.27	.26	.27	.28	.22	.22	—
6. Stress	.34	.29	.33	.36	.34	.34	.28	.26	.32	.38	.41	.4	.26	.37	.27	—
8. Stress	.37	.36	.43	.46	.45	.41	.36	.31	.39	.55	.6	.54	.39	.52	.29	.41	—
11. Stress	.41	.4	.51	.51	.51	.44	.4	.32	.38	.5	.52	.57	.37	.48	.34	.39	.56	—
12. Stress	.44	.43	.46	.5	.52	.42	.39	.35	.37	.46	.45	.52	.34	.46	.5	.34	.51	.6	—
14. Stress	.43	.45	.48	.56	.55	.49	.45	.28	.35	.42	.44	.53	.32	.43	.26	.42	.43	.5	.49	—
18. Stress	.4	.4	.46	.55	.52	.5	.47	.24	.34	.41	.43	.52	.33	.47	.24	.42	.44	.51	.47	.62	—
M	0.78	0.83	0.58	0.64	0.56	0.45	0.35	0.75	0.43	0.55	0.92	0.43	0.56	0.39	1.28	0.86	0.87	0.68	0.8	0.92	0.79
SD	0.89	0.9	0.83	0.87	0.79	0.81	0.74	0.87	0.78	0.83	0.96	0.76	0.86	0.74	0.92	0.84	0.91	0.85	0.92	0.94	0.92

Note. All correlations are significant at p < .01. DASS–21 = Depression Anxiety Stress Scales–21.

Table 3.

Means and Standard Deviations for DASS–21 by Gender.

Scale	Female		Male		Overall		t	df
Scale	M	SD	M	SD	M	SD	t	df
Depression	8.87	9.32	7.82	8.80	8.35	9.07	−2.80**	2,301.71
Anxiety	8.97	8.63	7.10	7.57	8.02	8.70	−5.49***	2,203.53
Stress	10.94	13.94	10.94	8.45	12.41	9.01	−8.15***	2,290.34
Total	31.51	24.61	25.86	22.54	28.64	23.77	−5.87***	2,362.90

Note. t values represent comparisons between males and females.

p < .01. ***p < .001.

Given the dearth of research of the DASS–21 with U.S. adolescents, descriptive results of the current sample were also compared with the norming results obtained by Lovibond and Lovibond (1995). In the DASS–21 manual, Lovibond and Lovibond offer cut-score criteria that indicate levels of symptom severity for each scale but not for a total score. Higher scores are indicative of higher symptom levels, with individuals with scores above 21, 15, and 26 indicated as falling in the severe or extremely severe range of depression, anxiety, and stress symptoms, respectively (i.e., scores above the 95th percentile). Using these criteria, the mean depression and stress scores of the current sample were in the normal range and the mean anxiety score was in the mild range. In addition, 10.5%, 18.1%, and 10.7% of the adolescents in the current sample reported severe or extremely severe levels of depression, anxiety, and stress symptoms, respectively. However, it is important to note that that these values were determined based on criteria from an adult norming sample. The present adolescents’ means and standard deviations for the total score and subscales are similar to those found by Tully and colleagues (2009) with Australian adolescents.

Model Testing

First, a one-factor model, suggested by Patrick et al. (2010) was tested to investigate whether the DASS–21 is best understood as a general measure of negative affectivity, rather than differentiated states associated with depression, anxiety, or stress. As can be seen in Table 4, fit indices for this model indicated adequate to poor fit. Next, the original three-factor model supported with Australian individuals (Lovibond & Lovibond, 1995) was examined. This model yielded adequate fit among our adolescent U.S. sample. Table 5 provides chi-square difference test results, adjusted for WLSMV estimation, indicating whether significant improvements in fit are evident with each additional model specified. Chi-square difference testing results indicated significant improvements in model misfit in the three-factor model as compared with the one-factor model, p < .001, indicating that the three-factor model yields significantly better fit than the one-factor solution. Correlations among each factor were strong: Depression with Anxiety, r = .87, Depression with Stress, r = .86, Anxiety with Stress, r = .95. The large correlations between factors observed in the three-factor model and in previous investigations into the factor structure of the DASS–21 suggest that a general factor in addition to specific factors may be a better representation of the relations among DASS–21 items.

Table 4.

Fit Statistics for Each of the Tested Models.

Model	χ²	df	RMSEA [CI]	CFI	TLI
One-factor	4,163.402***	189	.093 [.090, .095]	0.927	0.919
Three-factor	2,941.534***	186	.078 [.075, .080]	0.950	0.943
Bifactor^a	2,276.729***	168	.072 [.069, .074]	0.961	0.952
Bifactor^b	2,720.896***	175	.077 [.074, .080]	0.953	0.944

Note. RMSEA = root mean square error of approximation; CI = confidence interval; CFI = comparative fit index; TLI = Tucker–Lewis index.

Bifactor model with general Negative Affectivity factor and three specific factors of Depression, Anxiety, and Stress.

Bifactor model with general Negative Affectivity factor and two specific factors of Depression and Anxiety.

***

p < .001.

Table 5.

Comparison of Models Tested Using Chi-Square Difference Tests.

Comparison	Δχ²	Δdf
One-factor versus three-factor	541.644***	3
Three-factor vs. bifactor^a	635.681***	18
Bifactor^b vs. bifactor^a	394.382***	7

Bifactor model with general Negative Affectivity factor and three specific factors of Depression, Anxiety, and Stress.

Bifactor model with general Negative Affectivity factor and two specific factors of Depression and Anxiety.

***

p < .001.

Consistent with research by Henry and Crawford (2005), results of model testing so far offered support for examination of the relation between a general negative affectivity construct with the constructs of depression, anxiety, and stress. To further examine these relations and the unidimensionality of the DASS–21, a bifactor model simultaneously estimating the loadings of the 21 items onto a general Negative Affectivity factor and the loadings of the 21 items onto the three specific factors of Depression, Anxiety, and Stress was specified. As bifactor modeling intends to assess unidimensionality of measures, examining whether specific factors explain variance above and beyond a general factor, factor correlations were constrained to zero. This model is depicted in Figure 1. As can be seen in Tables 4 and 5, this bifactor model yielded adequate fit and significant improvements in model fit over the three-factor model, p < .001.

Figure 1.

Path diagram of final DASS–21 bifactor model.

The final model specified included a general Negative Affectivity factor and two specific factors of Depression and Anxiety, with hypothesized Stress items only loading onto the Negative Affectivity factor. Again, each factor was constrained to be orthogonal. Fit statistics (see Table 4) indicated this model adequately fit the data. The adjusted chi-square difference test, however, revealed significant increases in model misfit in the second bifactor model with only two specific factors when compared with the initial bifactor model; thus, this model has significantly poorer fit than the bifactor model with three specific factors. Table 6 presents standardized loading estimates for all models tested.

Table 6.

Standardized Factor Loadings for Each of the Tested Models.

Item number/scale	One-factor	Three-factor			Bifactor^a		Bifactor^b
Item number/scale	One-factor	Depression	Anxiety	Stress	General	Specific	General	Specific
3. Depression	0.74	0.77			0.67	0.38	0.67	0.38
5. Depression	0.70	0.73			0.64	0.36	0.64	0.37
10. Depression	0.80	0.83			0.73	0.40	0.73	0.40
13. Depression	0.82	0.86			0.78	0.30	0.78	0.30
16. Depression	0.84	0.87			0.79	0.32	0.79	0.32
17. Depression	0.86	0.89			0.76	0.50	0.76	0.50
21. Depression	0.85	0.88			0.74	0.53	0.73	0.53
2. Anxiety	0.52		0.53		0.52	0.17	0.52	0.17
4. Anxiety	0.69		0.72		0.67	0.46	0.67	0.45
7. Anxiety	0.74		0.77		0.74	0.21	0.74	0.22
9. Anxiety	0.70		0.72		0.72	−0.05^†	0.72	−0.05^†
15. Anxiety	0.86		0.90		0.89	0.04^†	0.88	0.05^†
19. Anxiety	0.61		0.64		0.60	0.44	0.59	0.45
20. Anxiety	0.81		0.84		0.82	0.15	0.82	0.16
1. Stress	0.50			0.51	0.48	0.47	0.51
6. Stress	0.58			0.60	0.60	−0.03^†	0.59
8. Stress	0.75			0.77	0.76	0.08	0.76
11. Stress	0.79			0.82	0.80	0.16	0.81
12. Stress	0.76			0.79	0.76	0.45	0.78
14. Stress	0.76			0.79	0.79	−0.17	0.78
18. Stress	0.75			0.78	0.78	−0.17	0.77

Note. All p values < .01, except ^†p > .05.

Bifactor model with general Negative Affectivity factor and three specific factors of Depression, Anxiety, and Stress.

Bifactor model with general Negative Affectivity factor and two specific factors of Depression and Anxiety.

The bifactor model with three specific factors yielded the overall best fit within the present sample of adolescents. Examination of item loadings (see Table 6) revealed strong to very strong loadings of each item on the general factor. When compared with item loadings on the specific factor, the loadings for the general factor were stronger than those of the specific factor, with the exception of Item 1 on the Stress factor. In general, the Negative Affectivity factor explained more of the variance in the DASS–21 items than did the specific factors of Depression, Anxiety, and Stress. Most notably, the majority of item loadings onto the specific factors of Anxiety and Stress were weak or negligible, with only two items on each specific factor having acceptable loadings (i.e., >.30; Brown, 2006). All loadings on the general factor were significant at p < .05; however, two items on the Anxiety factor and one item on the Stress factor yielded nonsignificant loadings.

Omega coefficients and explained common variance (ECV) were calculated for the bifactor model with three specific factors to further assess the unidimensionality of the DASS–21 (Reise, 2012; Rodriguez, Reise, & Haviland, 2016). Coefficient omega values for the total score, Depression, Anxiety, and Stress subscales were ω = .97, ω = .99, ω = .99, and ω = .99, respectively, indicating strong reliability. The omega hierarchical coefficient (ω_h = .92) and ECV (.83) for the total score provided by the general factor each indicated that the general factor accounts for a significant proportion of the variance among DASS–21 items and supports the presence of a strong general DASS–21 factor. That is, if a composite were formed from a sum of the DASS–21 items, 92% of the variance of the total score would be attributed to variance in the general factor (Bottesi et al., 2015). Moreover, the omega subscale coefficients for the Depression (ω_s = .03), Anxiety (ω_s = .009), and Stress (ω_s = .003) subscales further indicated that the majority of the variance was explained by the general factor and that reliability of the subscale scores was substantially reduced when controlling for the general factor. Therefore, although the original three-factor model yielded adequate fit, the results of bifactor modeling demonstrate the presence of a strong general Negative Affectivity factor with minimal remaining indicator variance explained by the specific factors of Depression, Anxiety, and Stress.

Discussion

This study is the first to investigate the underlying factor structure of the DASS–21 among U.S. adolescents. The purpose of this study was to expand current knowledge regarding the factor structure of the DASS–21, as a first step in evaluating its potential use as a school-based screening instrument with U.S. adolescents. Several alternative models were examined using CFA techniques. A one-factor model assessing for unidimensionality of the scale was tested first, followed by a three-factor model, a bifactor model including a general Negative Affectivity factor and specific factors of Depression, Anxiety, and Stress, and a final bifactor model including general Negative Affectivity and specific factors of Depression and Anxiety.

Consistent with research conducted with adults in the United States (Osman et al., 2012) and the United Kingdom (Henry & Crawford, 2005), the bifactor model with three specific factors yielded the best fit in the present adolescent sample. Of the models tested, the one-factor model yielded the poorest fit, the three-factor model yielded adequate fit, and the bifactor model with two specific factors also yielded adequate fit. Correlations between each factor in the hypothesized three-factor model were very strong. The final, best fitting bifactor model (see Figure 1) yielded significantly better fit over all other models tested. Closer examination of this model, however, revealed that the general Negative Affectivity factor exhibited strong reliability and explained a significant portion of the variance in DASS–21 items. Moreover, loadings of the items with the general factor were all moderate to strong. Examination of the loadings of items onto the specific factors of Depression, Anxiety, or Stress, however, indicated that only the Depression factor exhibited strong relations with DASS–21 depression items above and beyond the variance explained by general Negative Affectivity. Specific-factor loadings of items on the Depression factor were all moderate to strong, while loadings of items on the Anxiety and Stress factors were inconsistent, with the majority of these items having weak or negligible loadings. Overall, the reliability of each subscale was greatly reduced when accounting for the general factor.

The item-level results obtained from this model suggest that (a) the DASS–21 subscales reflect a common factor (i.e., Negative Affectivity), indicating that a total score of the DASS–21 can be derived as a measure of general negative affectivity, and (b) the DASS–21 may not adequately differentiate between the experiences of negative affectivity, anxiety, and stress in U.S. adolescents. Therefore, deriving independent subscale scores of Anxiety and Stress may be more indicative of general negative affectivity than the unique states of anxiety or stress. Similarly, in an investigation of the DASS–21 with Australian adolescents, Szabo (2010) concluded that a bifactor model with specific factors of Depression, Anxiety, and Stress yielded the best fit. Extending Szabo’s conclusions to U.S. adolescents, the present findings suggest that the DASS–21 is appropriate for identifying symptoms of depression in adolescents, but that experiences of anxiety and stress in U.S. adolescents may differ from those assessed on the DASS–21 or that the DASS–21 is not adequately assessing these characteristics in U.S. adolescents. Therefore, findings of the current study support the use of the DASS–21 as a screening tool to identify general negative affectivity using a total score, but compelling evidence is lacking to support use of the Depression, Anxiety, and Stress subscales separately with U.S. adolescent populations to identify internalizing difficulties in each of these areas.

Consistent with recommendations that tools most appropriate for universal screening assess broad constructs, the general distress construct of negative affectivity observed in the present study offers preliminary support for the use of the DASS–21 to identify experiences of negative affectivity in adolescents as a first step in identifying internalizing distress. Considering the tripartite model, negative affectivity may be a core component of internalizing distress (Clark & Watson, 1991) and may provide indication that a student is experiencing internalizing symptomatology. Given the DASS–21’s ability to identify a general negative affectivity factor, it may function well as a first gate within a multiple-gating screening strategy. Within a multigating screening approach, students are provided with more comprehensive assessments following an initial identification of risk or distress; this practice is recommended as it is aligned with prevention science principles and due to its diagnostic accuracy and cost-effectiveness (Stiffler & Dever, 2015). Students identified as experiencing elevated negative affectivity via the DASS–21 when used as a first-gate screener may then be referred for follow-up assessment (i.e., second-gate assessment) using specialized or targeted instruments designed to offer more detailed information about the presence of internalizing pathology. For example, screening using the DASS–21 may allow practitioners to identify students experiencing elevated levels of negative affectivity, including states such as tension, worry, irritation, sadness, agitation, and apathy. Follow-up assessment would target specific internalizing problems, such as depression or anxiety, and offer more extensive evaluation of those traits specific to these internalizing states (i.e., anhedonia and physiological hyperarousal, respectively) to further inform intervention planning.

Similarly, use of the DASS–21 as a first gate within a multigating screening system is consistent with recommendations for improving access to, and delivery of, social-emotional interventions for all students, especially those who are at risk, within multi-tiered intervention frameworks (e.g., response to intervention [RTI]; V. H. Levitt & Merrell, 2009). Within an RTI framework, assessment and intervention are structured such that Tier 1 includes interventions designed to promote the well-being of all students and to prevent emergence of internalizing problems (e.g., schoolwide, classroom-based social-emotional curriculum). Tier 2 services provide targeted interventions to students who are exhibiting early symptoms of internalizing problems (e.g., elevated levels of sadness) and who are at risk of developing an internalizing disorder. Finally, Tier 3 services focus on individuals experiencing more severe symptomatology or pathology, and may include individualized counseling or wrap-around interventions coordinating services within the home, school, and community (V. H. Levitt & Merrell, 2009). As such, universal screening using the DASS–21 may be performed to identify students who may benefit from Tier 2 or Tier 3 intervention services. For example, students identified as having the highest levels of distress may be referred for additional, diagnostic assessment and individual counseling support services. In addition, students identified as having elevated levels of negative affectivity during schoolwide screening may be referred to participate in early intervention groups designed to ameliorate skill deficits (e.g., related to affect, cognition, or behavior; V. H. Levitt & Merrell, 2009) and reduce symptoms (e.g., teach coping skills, promote affect regulation, address cognitive biases). Considering the negative impact of internalizing problems on academic performance (e.g., Reid et al., 2004), use of the DASS–21 as part of a multiple-gate screening effort, situated within a multi-tiered system of intervention, may help direct students to services designed to ameliorate distress.

Limitations and Future Directions

Additional efforts to clarify the underlying factor structure of the DASS–21 among U.S. adolescents are warranted to replicate the current findings with additional, diverse samples. Measurement and structural invariance across younger and older adolescents, ethnicities, and genders should also be investigated. A limitation to generalizability of the current results is that the ethnicity of the adolescents’ participating in the current study was available on only a school, rather than individual, level; in addition, these school-level data suggest that the majority of youth in the present sample were of Latino/a descent and that many were likely to have experienced socioeconomic disadvantage. Considering the limitations of the current sample and the higher prevalence rates of internalizing problems in Latino/a youth, continued examination into the factor structure of the DASS–21 with additional samples that are most broadly representative of the U.S. population are needed. Given the significant differences in females’ and males’ reports of internalizing symptoms in the current study, and that females more often report higher rates of internalizing distress (e.g., Deković, Buist, & Reitz, 2004), additional research investigating the invariance of the supported bifactor model with males and females is also needed.

Given the weak and nonsignificant loadings of several items on the Stress and Anxiety subscales, it may be that these items are not functioning as intended with a U.S. sample. When considering individual items, given that item wording was slightly adapted for use with U.S. English-speaking adolescents, additional research is needed to evaluate how this might have affected this study’s findings and the DASS–21 psychometric characteristics with U.S. adolescents. We do note, however, that for this sample, the CFA factor loading patterns were unrelated to item modification status. All items had substantial loadings on their respective factors in the three-factor CFA. Similarly, when the bifactor model with general Negative Affectivity factor and three specific factors of Depression, Anxiety, and Stress is considered, the original and modified items performed comparably: (a) The two modified depression items had high loadings as did all five of the original items, (b) two of the five modified anxiety items had high loadings whereas the two original items had low loadings, and (c) one of four modified stress items had a high loading contrasted with one of three original items. Furthermore, future research may examine scale modifications, for example, eliminating items with weak or nonsignificant loadings, or brief versions using the highest loading items on each subscale. It may be that a briefer scale with only the items most strongly related to each factor would result in more reliable measurement of each internalizing domain in addition to general negative affectivity.

As the present study examined only the underlying factor structure of the DASS–21 in adolescents, additional research is needed to more definitively assess the relation of both the general Negative Affectivity factor and the specific factors of Depression, Anxiety, and Stress on youths’ experiences of internalizing distress in these areas. Research examining the concurrent and predictive validity of the DASS–21 total Negative Affectivity score with specific symptoms of depression, anxiety, and stress is needed to further inform the use of this tool as part of universal screening efforts to predict and identify internalizing problems in adolescents (Glover & Albers, 2007; J. M. Levitt et al., 2007). Moreover, as the present research supports the use of a total negative affectivity score, additional research is needed to evaluate cut scores for assessing severity of negative affectivity in adolescents for scale interpretation. Similarly, further psychometric validation of the scale should be conducted with U.S. adolescent samples, including examinations of convergent, divergent, and construct validity.

Overall, for U.S. adolescents, the current study provides a first step in understanding the factor structure of the DASS–21, a popular scale intended to measure internalizing problems. Results suggest that a general distress factor, negative affectivity, is associated with the majority of variation in DASS–21 items, with little remaining variance being associated with the depression, anxiety, or stress domains. Therefore, DASS–21 scales may more adequately capture variance common to anxiety, depression, and stress, that is, adolescents’ experiences of negative affectivity. Additional research, however, is needed to further understand the underlying structure of the DASS–21 with adolescents in the United States and its usefulness in prevention and early intervention efforts. Developing and validating screening tools that accurately predict and identify internalizing distress in youth are vital for preventing negative outcomes and encouraging the well-being of all youth.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Antony

M. M.

Beiling

P. J.

Cox

B. J.

Ens

M. W.

Swinson

R. P.

(1998). Psychometric properties of the 42-item and 21-item versions of the Depression Anxiety Stress Scales in clinical groups and a community sample. Psychological Assessment, 10, 176-181. doi:10.1037/1040-3590.10.2.176

Beauducel

Herzberg

P. Y.

(2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling: A Multidisciplinary Journal, 13, 186-203. doi:10.1207/s15328007sem1302_2

Bottesi

Ghisi

Altoè

Conforti

Melli

Sica

(2015). The Italian version of the Depression Anxiety Stress Scales-21: Factor structure and psychometric properties on community and clinical samples. Comprehensive Psychiatry, 60, 170-181. doi:10.1016/j.comppsych.2015.04.005

Brown

T. A.

(2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

Brown

T. A.

Chorpita

B. F.

Korotisch

Barlow

D. H.

(1997). Psychometric properties of the Depression Anxiety Stress Scales (DASS) in clinical samples. Behaviour Research and Therapy, 35, 79-89. doi:10.1016/S0005-7967(96)00068-X

Browne

M. W.

Cudeck

(1989). Single sample cross-validation indices for covariance structures. Multivariate Behavioral Research, 24, 445-455. doi:10.1207/s15327906mbr2404_4

Clark

L. A.

Watson

(1991). Tripartite model of anxiety and depression: Psychometric evidence and taxonomic implications. Journal of Abnormal Psychology, 100, 316-336. doi:10.1037/0021-843X.100.3.316

Cunningham

J. M.

Suldo

S. M.

(2014). Accuracy of teachers in identifying elementary school students who report at-risk levels of anxiety and depression. School Mental Health, 6, 237-250. doi:10.1007/s12310-014-9125-9

Curran

P. J.

West

S. G.

Finch

J. F.

(1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1, 16-29. doi:10.1037/1082-989X.1.1.16

10.

Deković

Buist

K. L.

Reitz

(2004). Stability and changes in problem behavior during adolescence: Latent growth analysis. Journal of Youth and Adolescence, 33, 1-12. doi:10.1023/A:1027305312204

11.

Dowdy

Doane

Eklund

Dever

(2013). A comparison of teacher nomination and screening to identify behavioral and emotional risk within a sample of underrepresented students. Journal of Emotional and Behavioral Disorders, 21, 127-137. doi:10.1177/1063426611417627

12.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi:10.1037/1082-989X.4.3.272

13.

Fröjd

S. A.

Nissinen

E. S.

Pelkonen

M. U. I.

Marttunen

M. J.

Koivisoto

A. M.

Kaltiala-Heino

(2008). Depression and school performance in middle adolescent boys and girls. Journal of Adolescence, 31, 485-498. doi:10.1016/j.adolescence.2007.08.006

14.

Geisinger

K. F.

(1994). Cross-cultural normative assessment: Translation and adaptation issues influencing the normative interpretation of assessment instruments. Psychological Assessment, 6, 304-312. doi:10.1037/1040-3590.6.4.304

15.

Glover

T. A.

Albers

C. A.

(2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117-135. doi:10.1016/j.jsp.2006.05.005

16.

Goodman

(1997). The strengths and difficulties questionnaire: A research note. Child Psychology & Psychiatry & Allied Disciplines, 38(5), 581–586. doi:10.1111/j.1469-7610.1997.tb01545.x

17.

Goodman

(2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1337-1345. doi:10.1097/00004583-200111000-00015

18.

Gudiño

O. G.

Lau

A. S.

Yeh

McCabe

K. M.

Hough

R. L.

(2009). Understanding racial/ethnic disparities in youth mental health services: Do disparities vary by problem type? Journal of Emotional and Behavioral Disorders, 17, 3-16. doi:10.1177/1063426608317710

19.

Henry

J. D.

Crawford

J. R.

(2005). The short-form version of the Depression Anxiety Stress Scales (DASS-21): Construct validity and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 44, 227-239. doi:10.1348/014466505X29657

20.

Bentler

P. M.

(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453. doi:10.1037/1082-989X.3.4.424

21.

Huang

J. Y.

Arnold

Qui

Miller

R. S.

Williams

M. A.

Enquobahrie

D. A.

(2014). Association of serum vitamin D with symptoms of depression and anxiety in early pregnancy. Journal of Women’s Health, 23, 588-595. doi:10.1089/jwh.2013.4598

22.

Hui

C. H.

Triandis

H. C.

(1985). Measurement in cross-cultural psychology: A review and comparison of strategies. Journal of Cross-Cultural Psychology, 16, 131-152. doi:10.1177/0022002185016002001

23.

Jenkins

L. N.

Damaray

M. K.

Wren

N. S.

Secord

S. M.

Lyell

K. M.

Magers

A. M.

. . . Tennant

(2014). A critical review of five commonly used social-emotional and behavioral screeners for elementary or secondary schools. Contemporary School Psychology, 18, 241-254. doi:10.1007/s40688-014-0026-6

24.

Joiner

T. E.

Catanzaro

S. J.

Laurent

(1996). Tripartite structure of positive and negative affect, depression, and anxiety in child and adolescent psychiatric inpatients. Journal of Abnormal Psychology, 105, 401-409. doi:10.1037/0021-843X.105.3.401

25.

Kamphaus

R. W.

Reynolds

C. R.

(2007). BASC-2 Behavioral and Emotional Screening System. Minneapolis, MN: Pearson Assessment.

26.

Kamphaus

R. W.

Reynolds

C. R.

Dever

B. V.

(2014). Behavioral and mental health screening. In Kettler

R. J.

Glover

T. A.

Albers

C. A.

Feeney-Kettler

K. A.

(Eds.), Universal screening in educational settings: Evidence-based decision making for schools (pp. 249-273). Washington, DC: American Psychological Association.

27.

Kataoka

S. H.

Zhang

Wells

K. B.

(2002). Unmet need for mental health care among U.S. children: Variation by ethnicity and insurance status. American Journal of Psychiatry, 159, 1548-1555. doi:10.1176/appi.ajp.159.9.1548

28.

Lane

K. L.

Little

M. A.

Casey

A. M.

Lambert

Wehby

Weisenbach

J. L.

Phillips

(2009). A comparison of systematic screening tools for emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 17, 93-105. doi:10.1177/1063426608326203

29.

Levitt

J. M.

Saka

Romanelli

L. H.

Hoagwood

(2007). Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology, 45, 163-191. doi:10.1016/u.usp.2006.11.005

30.

Levitt

V. H.

Merrell

K. W.

(2009). Linking assessment to intervention for internalizing problems of children and adolescents. School Psychology Forum: Research in Practice, 3(1), 13-26.

31.

Lovibond

S. H.

Lovibond

P. F.

(1995). Manual for the Depression Anxiety Stress Scales. Sydney, Australia: Psychology Foundation.

32.

Mellor

Vinet

E. V.

Bt Mamat

N. H.

Richardson

Roman

(2015). Factorial invariance of the DASS-21 among adolescents in four countries. European Journal of Psychological Assessment, 21, 138-142. doi:10.1027/1015-5759/a000218

33.

Merikangas

K. R.

Jian-ping

Burstein

Swanson

S. A.

Avenevoli

Lihong

. . . Swendsen

(2010). Lifetime prevalence of mental disorders in U.S. adolescents: Results from the National Comorbidity Study-Adolescent Supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 49, 980-989. doi:10.1016/j.jaac.2010.05.017

34.

Muthén

L. K.

Muthén

B. O.

(1998-2013). Mplus user’s guide (6th ed.). Los Angeles, CA: Author.

35.

Trauer

Dodd

Callaly

Campbell

Berk

(2007). The validity of the 21-item version of the Depression Anxiety Stress Scales as a routine clinical outcome measure. Acta Neuropsychiatrica, 19, 304-310. doi:10.1111/j.1601-5215.2007.00217.x

36.

Osman

Wong

J. L.

Bagge

C. L.

Freedenthal

Gutierrez

P. M.

Lozano

(2012). The Depression Anxiety Stress Scales-21 (DASS-21): Further examination of dimensions, scale reliability, and correlates. Journal of Clinical Psychology, 68, 1322-1338. doi:10.1002/jclp.21908

37.

Papandrea

Winefield

(2011). It’s not just the squeaky wheels that need the oil: Examining teachers’ views on the disparity between referral rates for students with internalizing versus externalizing problems. School Mental Health, 3, 222-235. doi:10.1007/s12310-011-9063-8

38.

Patrick

Dyck

Bramston

(2010). Depression Anxiety Stress Scale: Is it valid for children and adolescents? Journal of Clinical Psychology, 66, 996-1007. doi:10.1002/jclp.20696

39.

Reid

Gonzalez

J. E.

Nordness

P. D.

Trout

Epstein

M. H.

(2004). Meta-analysis of the academic status of students with emotional/behavioral disturbance. Journal of Special Education, 38, 130-143. doi:10.1177/00224669040380030101

40.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. doi:10.1080/00273171.2012.715555

41.

Rhemtulla

Brosseau-Liard

P. É.

Savalei

(2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. doi:10.1037/a0029315

42.

Roberts

R. E.

Roberts

C. R.

Chen

Y. R.

(1997). Ethnocultural differences in prevalence of adolescent depression. American Journal of Community Psychology, 25, 95-110. doi:10.1023/A:1024649925737

43.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137–150. doi:10.1037/met0000045

44.

Sirin

S. R.

Rogers-Sirin

Cressen

Gupta

Ahmed

S. F.

Novoa

A. D.

(2015). Discrimination-related stress effects on the development of internalizing symptoms among Latino adolescents. Child Development, 86, 709-725. doi:10.1111/cdev.12343

45.

Smith

S. R.

(2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Journal of Psychoeducational Assessment, 25, 139-149. doi:10.1177/0734282906296233

46.

Stiffler

M. C.

Dever

B. V.

(2015). Mental health screening at school: Instrumentation, implementation, and critical issues. New York, NY: Springer.

47.

Szabo

(2010). The short version of the Depression Anxiety Stress Scales (DASS-21): Factor structure in a young adolescent sample. Journal of Adolescence, 33, 1-8. doi:10.1016/j.adolescence.2009.05.014

48.

Tull

M. T.

Gratz

K. L.

(2008). Further examination of the relationship between anxiety sensitivity and depression: The mediating role of experiential avoidance and difficulties engaging in goal-directed behavior when distressed. Journal of Anxiety Disorders, 22, 199-210. doi:10.1016/j.janxdis.2007.03.005

49.

Tully

Zajac

Venning

(2009). The structure of anxiety and depression in a normative sample of younger and older Australian adolescents. Journal of Abnormal Child Psychology, 37, 717-726. doi:10.1007/s10802-009-9306-4

50.

Umaña-Taylor

Updegraff

K. A.

(2007). Latino adolescents’ mental health: Exploring the interrelations among discrimination, ethnic identity, cultural orientation, self-esteem, and depressive symptoms. Journal of Adolescence, 30, 549-567. doi:10.1016/j.adolescence.2006.08.002

51.

Varela

R. E.

Hensley-Maloney

(2009). The influence of culture on anxiety in Latino youth: A review. Clinical Child and Family Psychology Review, 12, 217-233. doi:10.1007/s10567-009-0044-5

52.

Watson

Clark

L. A.

(1984). Negative affectivity: The disposition to experience aversive emotional states. Psychological Bulletin, 96, 465-490. doi:10.1037/0033-2909.96.3.465

53.

Willemsen

Markey

Declercq

Vanheule

(2011). Negative emotionality in a large community sample of adolescents: The factor structure and measurement invariance of the short version of the Depression Anxiety Stress Scales (DASS-21). Stress & Health, 27, 120-128. doi:10.1002/smi.1342

54.

Willis

G. B.

(1999). Cognitive interviewing: A “how to” guide. Rockville, MD: Research Triangle Institute, Laboratory for Survey Methods and Measurement. Retrieved from www.hkr.se/pagefiles/35002/gordonwillis.pdf

55.

Young

E. L.

Sabbah

H. Y.

Young

B. J.

Reiser

M. L.

Richardson

M. J.

(2010). Gender differences and similarities in a screening process for emotional and behavioral risks in secondary schools. Journal of Emotional and Behavioral Disorders, 18, 225-235. doi:10.1177/1063426609338858

56.

Zahn-Waxler

Klimes-Dougan

Slattery

M. J.

(2000). Internalizing problems of childhood and adolescence: Prospects, pitfalls, and progress in understanding the development of anxiety and depression. Development and Psychopathology, 12, 443-466. doi:10.1017/S0954579400003102

57.

Zychinski

K. E.

Polo

A. J.

(2012). Academic achievement and depressive symptoms in low-income Latino youth. Journal of Child and Family Studies, 21, 565-577. doi:10.1007/s10826-011-9509-5