The German Version of the Perceived Stress Scale (PSS-10): Evaluation of Dimensionality,Validity,and Measurement Invariance With Exploratory and Confirmatory Bifactor Modeling

Abstract

The Perceived Stress Scale (PSS) is a popular instrument for measuring the degree to which individuals appraise situations in their lives as excessively uncontrollable and overloaded. Despite its widespread use (e.g., for evaluating intervention effects in stress management studies), there is still no agreement on its factor structure. Hence, the aim of the present study was to examine the dimensionality, measurement invariance (i.e., across gender, samples, and time), reliability, and validity of the PSS. Data from 11,939 German adults (73% women) were used to establish an exploratory bifactor model for the PSS with one general and two specific factors and to cross-validate this model in a confirmatory bifactor model. The model displayed strong measurement invariance across gender and was replicated in Study 2 in data derived from six randomized controlled trials investigating a web-based stress management training. In Study 2 (overall N = 1,862), we found strong temporal invariance. Also, our analyses of concurrent and predictive validity showed associations with depressive symptoms, anxiety, and insomnia severity for the three latent PSS factors. These results show the implications of the bifactor structure of the PSS that might be of consequence in empirical research.

Keywords

Perceived Stress Scale bifactor modeling measurement invariance

The Perceived Stress Scale (PSS; Cohen, Kamarck, & Mermelstein, 1983) is widely used to measure self-reported stress. It captures the degree to which individuals appraise situations in their lives as excessively uncontrollable and the extent to which they feel overloaded. The PSS has been applied widely in clinical and nonclinical samples, for example, to investigate how perceived stress is associated with cortisol levels (Pruessner, Hellhammer, & Kirschbaum, 1999), perceived health status (Leung, Lam, & Chan, 2010), and posttraumatic stress symptoms (Mitchell, Crane, & Kim, 2008), or to compare levels of perceived stress across gender (e.g., Barbosa-Leiker et al., 2013). However, although extensive effort has been devoted to examining the factor structure of the PSS-10 (mostly for the English version; for a review, see Lee, 2012), there is still no agreement on its factor structure. The main reasons for the disagreement may result from the use of different analytical approaches and differences in the treatment and interpretation of issues associated with the positively worded items. Most important, previous studies have disagreed with regard to the number and the nature (i.e., whether they are construct relevant or irrelevant) of the latent factors that have emerged in the analyses.

Recently, a confirmatory bifactor structure was proposed for the English and Spanish versions of the PSS (Perera et al., 2016). This modeling approach has the potential to improve our understanding of the covariance structure that underlies the PSS-10. Originally proposed by Holzinger and Swineford (1937), exploratory and confirmatory bifactor models have recently been “rediscovered” as an important and flexible alternative for modeling multidimensionality (Chen, West, & Sousa, 2006; Reise, 2012). In contrast to a unidimensional confirmatory factor analysis (CFA), a bifactor model reflects the idea that the true score variance of a particular instrument might be derived from different sources (Morin, Arens, & Marsh, 2015). In this article, we aim to improve our understanding of the covariance structure that underlies the German version of the PSS-10 by applying both exploratory and confirmatory bifactor modeling approaches.

To date, the psychometric properties of the German version of the PSS-10 have received only scant attention. Despite this lack of attention, this version of the PSS-10 has been used in a number of substantive studies (e.g., Boß et al., 2016; Ebert et al., 2016; Heber, Lehr, Ebert, Berking, & Riper, 2016). Hence, in this article, we examine the dimensionality of the German PSS-10 along with its measurement invariance (MI) across gender in Study 1. In Study 2, we replicate the findings in data obtained from the intervention and waitlist control groups of six randomized controlled trials in which we investigate the stress levels of employees. Using these data, we examine longitudinal MI of the PSS-10 and report on the concurrent and predictive validity of the bifactor model factor scores with measures of sleep, anxiety, and depressive symptoms.

Previous Research on the PSS Factor Structure

The first investigations of the psychometric quality of the PSS relied on principal component analyses (e.g., Cohen & Williamson, 1988). The original authors argued for the scale’s unidimensionality and dismissed the second factor comprising the positively worded items as “irrelevant” (p. 43). However, on the basis of diverse analytical approaches, the two-factor solution has repeatedly been reported for the English version (for an overview, see Lee, 2012) as well as for various translations (e.g., Andreou et al., 2011; González-Ramírez, Rodríguez-Ayán, & Hernández, 2013; Klein et al., 2016). The two dimensions have been computed as either orthogonal or oblique solutions (e.g., Barbosa-Leiker et al., 2013; Golden-Kreutz, Browne, Frierson, & Andersen, 2004; Hewitt, Flett, & Mosher, 1992; Roberti & Harrington, 2006). The factors may have been interpreted as orthogonal because the four positively worded items were found to cluster into a method factor. By contrast, studies with oblique factors have proposed two different, content-related factors: (a) perceived distress or perceived helplessness (negatively phrased items) and (b) perceived coping or perceived self-efficacy (positively phrased items; e.g., Roberti & Harrington, 2006). Because a common psychometric strategy for reducing acquiescence bias is to include both true and false answers for items scored in the direction of the criterion, such ambiguities due to oppositely worded items are frequently found in the literature for other scales as well (e.g., Tomas & Oliver, 1999). Moreover, for the PSS, several authors have argued that keying (i.e., reverse scoring) might also be confounded with content (Golden-Kreutz et al., 2004). The authors suggested that the positively worded PSS items tap positive emotions and feelings of confidence and that these items do not overlap in content with the negative items. Also, they proposed that this is consistent with models of stress that also consider positive psychological states (Folkman, 1997). In line with this reasoning, the two PSS factors have shown distinct predictive quality: For example, in one study, perceived self-efficacy predicted depression only in women but not in men (Hewitt et al., 1992).

To improve the usability of the scale for applied studies, Golden-Kreutz et al. (2004) proposed a hierarchical solution for the PSS with “stress” and “counterstress” as lower order factors and “perceived stress” as a higher order factor. Although this model is mathematically equivalent to a model with oblique first-order factors, it is conceptually different. A second-order model explicitly implements a higher order factor (“perceived stress”) that influences all observed variables indirectly through the first-order factors. However, hierarchical models imply stringent assumptions because the associations between the higher order factor and the observed items are represented as indirect effects (i.e., mediated by the first-order factor). In short, these proportionality constraints imply that the ratio of the higher order factor loadings to the first-order factor loadings are exactly the same for all indicators associated with the same first-order factor (cf. Morin et al., 2015) and are unlikely to hold in the population (Reise, 2012).

The Bifactor Approach

A more general alternative that can be used to model a multifactorial structure is a bifactor model. A bifactor model reflects the idea that items used in an instrument might be associated with more than one source of true score variance. Hence, a solution with f factors for a set of n items with one general factor and f − 1 specific factors might reflect both construct-irrelevant variance (i.e., pure method factors) and construct-relevant variance (i.e., different content areas). Most important, bifactor models do not impose the proportionality constraints inherent in models with higher order factors (Brunner, Nagy, & Wilhelm, 2012) but specify direct effects of all latent factors on the indicators. Also, in contrast to first-order factors in a typical second-order model, the specific factors in a bifactor model can be used to predict external criteria and can be tested for measurement variance. For the PSS, Perera et al. (2016) recently computed a confirmatory bifactor model in their English and Hispanic samples with one global factor and one specific factor that comprised the four positively worded items. Despite referring to the specific factor as a method factor, they proposed and confirmed that the specific factor had a pattern of correlations with external criteria that was similar to the pattern proposed for the general factor. Their findings indicated that the positively worded items might indeed reflect stress-related content not shared with the general factor. Moreover, Reise (2012) stated that exploratory approaches to bifactor modeling are greatly underused by applied researchers and emphasized the importance of exploring the data prior to proceeding to the testing of more restrictive, confirmatory models. Thus, in this article, we provide a more detailed account of the factor structure of the PSS by first investigating an exploratory bifactor model and then providing a cross-validation of this model with a confirmatory bifactor model.

Relevance for Applied Research

A detailed investigation of the underlying factor structure is highly relevant to applied researchers. A single, summed, unit-weighted PSS score is the most popular choice in clinical contexts (Perera et al., 2016) and is also used as a primary outcome in intervention studies (e.g., Ebert et al., 2016). However, given the multifactorial structure of the PSS-10, it remains unclear how to evaluate the total scale scores of the PSS-10 with regard to both group differences (e.g., gender) and changes in these total scores (e.g., following an intervention). Accordingly, the thus-far mixed results on gender differences in perceived stress levels might originate from modeling strategies (i.e., applying a unidimensional total score vs. modeling the method effects). In previous research, several studies have shown that women report higher levels of perceived stress (e.g., Barbosa-Leiker et al., 2013), but other researchers have argued that these results might be due to measurement bias with women tending to acquiesce to negatively worded items (Gitchel, Roessler, & Turner, 2011). In such situations, a bifactor model might allow researchers to more flexibly identify the source of true latent differences (i.e., to identify whether they are determined by the general factor or by the specific factors).

Also, when evaluating interventions, there is a need to clarify the extent to which a PSS total score is a good estimate of overall perceived stress. Although a demonstration of multidimensionality does not necessarily imply that the total score is an inadequate indicator of the intended construct (Rodriguez, Reise, & Haviland, 2016), a multifactorial structure requires greater investigation into psychometric properties to determine the consequences for scale scoring (Reise, Bonifay, & Haviland, 2013). This question is also interesting from an applied point of view: Under the premise that the subscales are psychometrically justified, they could show distinct correlations with external criteria and different change patterns. In order to evaluate interventions, researchers might be interested in, for example, whether their training is more likely to improve behavior patterns related to self-efficacy or instead to take effect by reducing perceived helplessness.

The Present Study

The overarching goal of this article was to gain a better understanding of the factor structure of the PSS in a rich data set and hence to provide guidance for applied researchers who wish to use the instrument in their studies. Accordingly, in a first step, we investigated a bifactor solution in a large sample of German adults. We evaluated the bifactor model in two random subsamples by applying an exploratory approach in the first subsample and by cross-validating the exploratory solution in the second subsample with a confirmatory bifactor model. Second, the confirmatory bifactor model was used to establish the MI (Meredith, 1993) of the PSS across gender. Third, we aimed to validate the bifactor CFA model in data obtained from six randomized controlled trials that were conducted to investigate a web-based stress management training. Due to its brevity, the PSS-10 is well suited for evaluating stress in eHealth or mHealth programs. Because the field of mental health promotion through Internet-based or mobile-based interventions is rapidly growing, it might be especially interesting to improve our understanding of the PSS structure in such samples. Fourth, we investigated longitudinal MI, which is a prerequisite for evaluating intervention effects over time. Finally, we examined the validity of the PSS with measures of depression, anxiety, and insomnia severity.

Study 1

Method

Participants and Measure

This study was based on an online survey presented on the website of a statutory health insurance company. The survey was intended to provide interested people with information on their current stress levels. Data were collected between January 2015 and April 2016. The sample consisted of a total of 11,939 participants (73% women) with no missing responses. The mean age of participants was 41 years (SD = 12.6, range: 18 to 80), and 78% had a partner. The 10 PSS items were answered on a 5-point Likert-type scale ranging from 0 = never to 4 = very often. The scores of the four positively worded items (Items 4, 5, 7, and 8) were reverse-coded to facilitate interpretation. Thus, higher scores indicated higher levels of perceived stress. The data and the code are available at the Open Science Framework: osf.io/dqnxa.

Models and Analyses

To evaluate the dimensionality of the PSS-10, we randomly divided the sample into two subsamples (N = 5,938 and N = 6,001). We performed exploratory bifactor analyses with bigeomin orthogonal rotation in the first subsample. We specified solutions with a single general factor and up to three group factors. Under the orthogonal rotation, all factors were uncorrelated with each other, which allowed us to apply an unequivocal decomposition of the item variance to specify each item’s reliability and specificity. We considered statistically significant factor loadings of ≥.30 to be relevant. The exploratory factor analysis (EFA) models were evaluated with a χ² statistic and its associated p value, a comparative fit index (CFI) ≥ .95, a root mean square error of approximation (RMSEA) ≤ .05 and its associated p value for tests of close fit, and a standardized root mean square residual (SRMR) ≤ .08 (Mueller & Hancock, 2010).

The EFA model that fit the data best was then validated in a bifactor CFA with the second subsample and assessed on the basis of the same fit indices. All models were estimated with the robust maximum likelihood (MLR) approach, which is less dependent on the assumption of multivariate normality. MLR has been suggested for approximately continuous data when the number of response categories is equal to or greater than five (e.g., Raykov, 2012). The MLR estimator implemented in Mplus (Version 7.2) is asymptotically equivalent to the corrected chi-square statistic T2* proposed by Yuan and Bentler (1998).

Finally, the bifactor CFA model was used to compare latent means across gender. To be able to interpret the differences in the latent means in a meaningful way, the loadings and the intercepts in both groups must be constrained to be equal (strong or scalar MI). Thus, strong MI implies that the meaning of the constructs and the levels of the underlying items are equal in the two groups (van de Schoot, Lugtig, & Hox, 2012). For a bifactor CFA, thus far, the only recommendations on the sensitivity of goodness-of-fit indices to detect MI exist with regard to the violation of weak (metric) invariance and for equal group sizes (Khojasteh & Lo, 2015). However, our study needed to incorporate mean structure tests of invariance to compare weak and strong MI, and because women comprised more than 70% of our sample, our group sizes violated the assumption of equality. Consequently, we applied Chen’s (2007) recommendations for the assessment of both weak and strong MI. Chen (2007) proposed that a change ≥−.010 in CFI supplemented by a change of ≥.015 in RMSEA or a change of ≥.030 in SRMR would indicate a violation of weak MI, whereas a change of ≥−.010 in CFI supplemented by a change of ≥−.015 in RMSEA or a change of ≥−.010 in SRMR would indicate a violation of strong MI.

Due to the large sample size, for all comparisons, we did not interpret the Δχ². Among others, Cheung and Rensvold (2002) have warned against the double standard of disregarding the chi-square statistic on the basis of its dependence on sample size but adopting it to test hypotheses about invariance. The reliabilities of the total score, the global factors, and the specific factors were computed with omega coefficients (Reise et al., 2013; Rodriguez et al., 2016). Omega (ω) represents the proportion of variance in the total score that can be attributed to all common factors, whereas omega subscale (ω_S) represents the proportion of variance in a subscale score that can be attributed to all common factors. For both coefficients, high values indicate high reliability in the multidimensional composite (Rodriguez et al., 2016).

In addition, the bifactor model offers a way to estimate omega coefficients that represent only the proportion of variance that can be attributed to a single common factor. The omega hierarchical coefficient (ω_H) represents the proportion of variance in the total score that can be attributed to the general factor. High values in ω_H indicate that the scale can be considered essentially unidimensional (Rodriguez et al., 2016). Finally, the omega hierarchical subscale (ω_HS) represents the proportion of reliable systematic variance in the subscale score that is unique to this subscale (i.e., after the general factor is controlled for).

Besides the omega coefficients, we computed the reliability index H (Hancock & Mueller, 2001). Whereas ω_H and ω_HS represent the correlation between a factor and its unit-weighted composite, H represents the correlation between a factor and an optimally weighted composite. In short, Hancock’s and Mueller’s construct reliability index H offers a way to evaluate “how well a set of items represents a latent variable, and, therefore, informs on whether the SEM measurement model is suitable and replicable across studies” (Rodriguez et al., 2016, p. 230). High H values above .70 indicate well-defined latent factors: Such factors are more likely to replicate across studies, whereas low H values indicate latent factors that are more likely to change across studies. Formulas for all indices are provided in the supplemental file.

Results

Bigeomin EFA Model Results

In a first step, we computed EFA models with bigeomin orthogonal rotation in the first subsample. Of these models, the model with one general factor and two specific factors yielded the best fit and a reasonable factor structure. This model had an excellent fit (CFI = .995; RMSEA = .032, 90% [confidence interval] CI [.026, .037]; SRMR = .009), whereas the fit of the model with only one specific factor was noticeably worse (CFI = .974; RMSEA = .060, 90% CI [.056, .065]; SRMR = .021). Moreover, although the model with three specific factors provided a better fit than the model with two specific factors, the factor loading for Item 5 (feeling that things were going one’s way) was >1, which indicated serious estimation problems, possibly due to overparameterization. Thus, the model with one general factor and two specific factors was retained. Table 1 presents the factor loadings for this best-fitting model.

Table 1.

Loadings in the Bifactor EFA and Bifactor CFA.

Item		Domain	Bifactor EFA			Bifactor CFA
Item		Domain	Stress	F1	F2	Stress	F1	F2
4	Feeling confident about one’s ability to handle one’s personal problems	POS	.50	.43	−.13	.43	.52
5	Feeling that things are going one’s way	POS	.54	.49	.05	.50	.52
7	Able to control irritations in one’s life	POS	.26	.38	−.05	.22	.39
8	Feeling on top of things	POS	.70	.37	−.00	.66	.45
1	Being upset because of something that happened unexpectedly	NEG	.49	−.00	.58	.58		.56
2	Feeling unable to control the important things in one’s life	NEG	.62	.05	.33	.69		.24
3	Feeling nervous and “stressed”	NEG	.65	−.04	.37	.69		.28
6	Not able to cope with all the things that one has to do	NEG	.79	−.18	−.00	.73
9	Being angered because of things that were outside of one’s control	NEG	.49	−.01	.51	.56		.34
10	Feeling difficulties were piling up so high that one could not overcome them	NEG	.75	−.00	.20	.81

Note. EFA = exploratory factor analysis; CFA = confirmatory factor analysis; POS = positive; NEG = negative. A bifactor EFA with orthogonal geomin rotation was computed with the first subsample (N = 5,938). A bifactor CFA was computed with the second subsample (N = 6,001). Values in bold indicate significant loadings (5% level) in the EFA. In the CFA, all loadings were significant.

All item loadings on the general factor were statistically significant at the 5% level, and all were substantial with a range of .49 to .79 except Item 7, which had a loading of .26. This item reflects the ability to control feelings of irritation and had a higher loading on the first specific factor (.38). To ensure that this low loading on the general factor was not an artifact of the bifactor modeling, we computed a unidimensional CFA. The CFA revealed a standardized loading of .27. Hence, to ensure that the proposed factor solution was comparable to previous studies, Item 7 was retained as an indicator of the general factor. The first specific factor F1 comprised the four positively worded and recoded items (Items 4, 5, 7, and 8). The second specific factor F2 also comprised four items (i.e., Items 1, 2, 3, and 9). These four items mostly reflect the experience negative affect (i.e., feeling upset, nervous, or angry). In contrast, Item 6 (not being able to cope with all the things one has to do) and Item 10 (feeling difficulties piling up too high to overcome) loaded substantially on the general factor only.

Bifactor CFA Results

In the next step, the best-fitting EFA solution was used to specify the bifactor CFA in order to validate the model in the second subsample. This model still showed an excellent fit (CFI = .988; RMSEA = .037, 90% CI [.033, .041]; SRMR = .017) despite the restrictions of zero cross-loadings required by the independent clusters model in a CFA. The loadings in this model are presented in Table 1, and the factor solution is presented in Figure 1. The ω reliability coefficient for the total score was .89, indicating a good reliability of the multidimensional composite.

Figure 1.

The confirmatory factor analysis bifactor model (N = 6,001).

In the next step, we computed ω_S for the two specific factors. The reliability coefficient ω_S for the specific factor composed of the positively worded items (F1) was .89. For F2, it was .88, again indicating good reliabilities for the multidimensional composites. To estimate the percentage of systematic variance in the total score that could be attributed exclusively to the general factor of perceived stress, we computed ω_H. For the general factor, ω_H was .77, suggesting that the general factor of perceived stress is the strongest determinant of the systematic variance underlying the total score of the PSS-10.

Next, to establish the proportion of unique reliable systematic variance for the two specific factors, for F1 (and equivalently for F2), we computed ω_HS. The coefficients were .08 and .05 for F1 and F2, respectively, suggesting that the two subscales provide only little unique and reliable variance.¹ Finally, we computed the factor’s reliability H (Hancock & Mueller, 2001) or replicability (Rodriguez et al., 2016). The H coefficients were .88 for the general factor, and .54 and .43 for the specific factors, respectively, indicating that only the general factor fulfilled Hancock and Mueller’s recommendations for a well-defined latent variable.

Table 1 presents the factor loadings for this confirmatory bifactor model, which we also used to test for MI across gender. We applied the cutoffs proposed by Chen (2007) to compare configural versus weak MI. All indices for the models (including χ² values) are shown in Table 2. The ΔCFI was .001, the ΔRMSEA was .004, and the ΔSRMR was .003. Hence, the indices showed that the factor loadings were equal across gender. For the comparison of weak versus strong MI, the ΔCFI was .003, the ΔRMSEA was .002, and the ΔSRMR was .002. Hence, again, the differences in the goodness-of-fit indices were small, indicating that strong MI held across gender.

Table 2.

Goodness-of-Fit Statistics for the Bifactor EFA and Bifactor CFA Models of the PSS-10 in Study 1.

Model		χ²(df)	p	RMSEA [90% CI]	SRMR	CFI	ΔRMSEA	ΔSRMR	ΔCFI	Δχ²(df)
M1	Bifactor EFA	124.21 (18)	<.001	.032 [.026, .037]	.009	.995
M2	Bifactor CFA	248.29 (27)	<.001	.037 [.033, .041]	.017	.988
M3	Bifactor CFA configural	289.61 (54)	<.001	.038 [.033, .042]	.018	.987
M4	Bifactor CFA weak	309.00 (69)	<.001	.034 [.030, .038]	.021	.986	.004	.003	–.001	19.39 (15)
M5	Bifactor CFA strong	372.18 (76)	<.001	.036 [.033, .040]	.023	.983	.002	.002	–.003	63.18 (7)
M6	Bifactor CFA strict	410.91 (86)	<.001	.035 [.032, .039]	.026	.981	.001	.003	–.002	38.73 (10)
M7	Bifactor CFA strict + equal latent variances	417.74 (89)	<.001	.035 [.032, .038]	.037	.981	.000	.013	.000	6.83 (3)

Note. PSS-10 = Perceived Stress Scale; EFA = exploratory factor analysis; CFA = confirmatory factor analysis; df = degrees of freedom; RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual; CFI = comparative fit index. Bifactor EFA N = 5,938. Bifactor CFA N = 6,001. The models M3 to M7 pertain to comparisons across gender.

On the basis of this model, we found that women had significantly higher scores on the general factor of perceived stress (z = 9.61, p < .001, d = 0.32) but not on the two specific factors (z = −0.34, p = .74, d = 0.01, and z = −0.67, p = .50, d = 0.03, respectively). In addition, we computed a model with strict MI and additionally constrained the variances of the three latent factors to equal across women and men. This model displayed an excellent fit (cf. Model 7 in Table 2), thus indicating that the reliabilities of the items were the same for women and men. Also, the restrictions in this model resulted in the same reliability coefficients for the overall model for women and men (i.e., the ω coefficient for the total score was .89, whereas the ω_H coefficient was .77).

Discussion

In Study 1, the results of the exploratory bifactor modeling in the first subsample indicated that one general and two specific factors should be extracted. Noticeably, the first specific factor comprised all four positively worded items, whereas the second specific factor comprised four of the six items worded in the direction of the criterion (i.e., negatively worded). Two items (Items 6 and 10) loaded on only the general factor. Thus, it seems that individuals differentiate between an experience of negative affect (i.e., the second factor) and a belief that they are not able to cope (Item 6) or to overcome (Item 10) difficulties. We were able to replicate this model in a bifactor CFA in the second subsample. Moreover, we found that the proposed model was invariant across gender and that women reported higher levels on the general factor.

Our results suggest that differences in the latent means between women and men pertain to true differences in general perceived stress and not to measurement bias due to women’s tendency to acquiesce to negatively worded items. The computed reliability and replicability coefficients indicated that the reliable variation in a PSS-10 total score is mostly attributable to a single, general latent variable and also that the replicability of the specific factors is lower than the replicability of the general factor. To extend these findings, the focus of the next study was to replicate the findings and to examine temporal invariance and the validity of the bifactor solution in data obtained from six randomized controlled trials in which we investigated the efficacy of a web-based stress management training.

Study 2

Method

Participants and Measure

The data for this study came from individuals who had signed up to participate in six different randomized controlled trials for evaluating the efficacy of a web-based stress management training (overall N = 1,862). The mean age of participants across all samples was 42 years (SD = 9.75, range: 18 to 65). The stress management training involved problem solving and emotion regulation. The training was set up in a secured web-based platform and included seven sessions consisting of general information, interactive exercises, quizzes, audio and video files, examples related to work, as well as downloadable worksheets and MP3 files. All six trials investigated the same stress management training but with varying levels of guidance (i.e., self-guided or with a low-level or more intensive guidance from an online coach; for more information, see Heber et al., 2016). Data were collected between January 2013 and April 2015, and the time lag between T1 and T2 was 8 weeks on average. In each sample, participants were randomly allocated to either the training group or the waitlist control group.

Besides the PSS-10, participants in Samples 1 to 3 provided data on depressive symptoms as measured by the Center for Epidemiological Studies Depression Scale (Hautzinger, Bailer, Hofmeister, & Keller, 2012) with 20 items rated on a scale ranging from 0 (rarely or none of the time) to 3 (most or almost all the time; e.g., “I felt that everything I did was an effort”). In Samples 4 to 6, participants reported their depressive symptoms on a 15-item version of the Center for Epidemiological Studies Depression Scale (Hautzinger et al., 2012). Furthermore, insomnia symptoms were measured with seven items from the insomnia severity index (Bastien, Vallieres, & Morin, 2001) in Samples 1 to 4. The Insomnia Severity Index items are answered on a 5-point Likert-type scale (e.g., “To what extent do you believe your sleep problems currently interfere with your daily functioning?”). In Samples 1 to 3, anxiety was measured with the subscale (seven items; range: 0 to 3) from the Hospital Anxiety and Depression Scales (Hermann-Lingen, Buss, & Snaith, 2011; e.g., “I get a sort of frightened feeling as if something awful is about to happen”).

In Table 3, we report demographic characteristics as well as the means, standard deviations, and internal consistencies of the instruments for all samples.

Table 3.

Demographic Characteristics, Means, Standard Deviations, and Reliabilities for all Scales at T1 and in the Six Samples in Study 2.

Sample	N (% women)	Age M (SD)	PSS			CES-D			ISI			HADS-A
Sample	N (% women)	Age M (SD)	M (SD)	Range	ω	M (SD)	Range	ω	M (SD)	Range	ω	M (SD)	Range	ω
1	264 (73)	43.3 (10.2)	2.55 (0.39)	0-4	.71	1.18 (0.40)	0-3	.85	2.18 (0.94)	0-4	.90	1.56 (0.48)	0-3	.75
2	263 (86)	42.9 (9.8)	2.53 (0.44)	0-4	.75	1.19 (0.44)	0-3	.88	1.89 (0.83)	0-4	.87	1.56 (0.47)	0-3	.73
3	263 (71)	41.6 (9.1)	2.59 (0.45)	0-4	.80	1.23 (0.44)	0-3	.87	1.84 (0.83)	0-4	.87	1.62 (0.50)	0-3	.76
4	408 (77)	41.9 (9.2)	2.56 (0.41)	0-4	.73	1.25 (0.46)	0-3	.83	1.96 (1.09)	0-4	.96	—	—	—
5	264 (69)	43.2 (9.8)	2.43 (0.51)	0-4	.81	1.17 (0.50)	0-3	.87	—	—	—	—	—	—
6	400 (75)	41.6 (10.3)	2.26 (0.56)	0-4	.83	1.08 (0.51)	0-3	.87	—	—	—	—	—	—

Note. PSS = Perceived Stress Scale; CES-D = Center for Epidemiological Studies Depression Scale; ISI = Insomnia Severity Index; HADS-A = Hospital Anxiety and Depression Scales–Anxiety Items. For the CES-D, the 20-item version was used in Samples 1 to 3, and a short version of 15 items was used in Samples 4 to 6.

Models and Analyses

In a first step, we tested whether the bifactor CFA model found in Study 1 could be replicated in the combined data in Study 2 at T1.² Full information maximum likelihood estimation was applied. To test model fit and make comparisons, we used the same cutoffs as in Study 1. Also, the model-based coefficients ω, ω_S, ω_H, ω_HS, and H were computed to evaluate the reliability and replicability of the bifactor solution. In addition, we examined whether the gender differences found in Study 1 could be replicated in Study 2.

In a second step, we investigated the longitudinal MI of the bifactor CFA model in individuals who were randomly allocated to the intervention groups (T1 before the training and T2 after the training, i.e., about 8 weeks later; overall N = 999) and to the waitlist control groups (overall N = 863). In a third step, to establish concurrent validity, the scores for depressive symptoms, anxiety, and insomnia severity assessed at T1 were regressed on the general and specific factors of the PSS at T1. For predictive validity, the mean scores for depressive symptoms, anxiety, and insomnia severity assessed at T2 were used. For this part of the analyses, we used only data from Trials 1 to 3 (overall N = 790; cf. Table 3) in which all three outcomes had been assessed. Also, for predictive validity, we differentiated between participants allocated to the respective control groups versus participants allocated to the intervention groups, under the assumption that the training might have changed the associations. For concurrent validity, we relied on randomization procedures and used the overall sample. Finally, building on the gender-related differences found in Study 1, we examined whether gender would be found to moderate the prediction of depressive symptoms at T2 by the stress scores at T1. All analyses in Study 2 were computed with the R package lavaan (Rosseel, 2012).

Results

Replication of the Proposed Model and Gender Differences

We tested whether the CFA bifactor model established in the second subsample in Study 1 would replicate in our data in Study 2. To do so, we examined cross-sectional MI at T1 across the subsample from Study 1 (N = 6,001) and the combined sample in Study 2 (N = 1,862) by computing configural, weak, and strong measurement models. The fit indices for all models are presented in Table 4. Overall, we found that for the bifactor CFA model, strong MI held across the data in Study 1 and Study 2, implying replication. The computed model-based omegas in Study 2 were .83 for ω, and the values for ω_S were .81 and .81 for F1 and F2, respectively. The omega coefficient ω_H was .69, and ω_HS was .08 for F1 and .06 for F2. Finally, the H coefficients were .80 for the general factor, .48 for F1, and .46 for F2.

Table 4.

Configural, Weak, and Strong Measurement Invariance Tests in Both Studies and Longitudinally in Study 2.

Cross-sectional invariance across Study 1 and Study 2
Model	χ²(df)	p	RMSEA [90% CI]	SRMR	CFI	ΔRMSEA	ΔSRMR	ΔCFI	Δχ²(df)
Configural	327.38 (54)	<.001	.036 [.032, .040]	.017	.987
Weak	467.85 (69)	<.001	.038 [.035, .042]	.028	.982	.002	.009	−.005	140.47 (15)
Strong	666.49 (76)	<.001	.044 [.042, .047]	.032	.973	.006	.004	−.009	198.64 (7)
Longitudinal invariance in Study 2
Configural	399.85 (270)	<.001	.023 [.018, .027]	.025	.987
Weak	544.61 (315)	<.001	.028 [.024, .032]	.040	.977	.005	.015	−.010	144,76 (45)
Strong	649.12 (335)	<.001	.032 [.028, .035]	.045	.969	.004	.005	−.008	104,51 (20)

Note. df = degrees of freedom; RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual; CFI = comparative fit index. Overall N in Subsample 2 of Study 1 = 6,001. Overall N in Study 2 = 1,862.

In addition, we examined whether the findings of higher latent stress scores on the general factor found in Study 1 would replicate in our data in Study 2. Using a model with strong MI (CFI = .980; RMSEA = .031, 90% CI [.023, .038]; SRMR = .029), we were able to confirm higher latent scores for perceived stress in women in Study 2. Again, women scored higher on the general factor z = 2.09, p = .036, d = 0.14, whereas the differences on the both specific factors were not significant (z = 0.20, p = .84, d = 0.02 for F1 and z = 0.04, p = .14, d = 0.14 for F2). Hence, in terms of effect sizes on the general factor, the difference between women and men was smaller in Study 2 compared with Study 1 (where d was 0.32). In addition, in Study 2, the effect sizes for the differences across gender were the same for the general factor and the second specific factor (i.e., negatively worded items).

Longitudinal Invariance of the Bifactor Solution

Next, we investigated the longitudinal invariance of the PSS across T1 and T2 for the individuals who were randomly assigned to the intervention groups (N = 999) and to the waitlist control groups (N = 863) separately. Before running the longitudinal analyses, we compared the measurement models across the groups for T1 and T2 separately. Due to the randomization procedures, differences at T1 were unlikely, but at T2, for individuals in the intervention groups, the difficulties in the items could have changed (cf. Fried et al., 2016). However, we found excellent fit for strong MI across the intervention and control groups at both measurement occasions.

Building on these results, we compared configural, weak, and strong MI longitudinally and across the groups simultaneously. The fit of the model with weak MI was excellent (CFI = .977; RMSEA = .028, 90% CI [.024, .032]; and SRMR = .040), and it was only slightly worse than the fit of the configural model (CFI = .987; RMSEA = .023, 90% CI [.018, .027]; and SRMR = .025). Additionally, constraining the intercepts to be equal across time and across groups led to a similar model fit (CFI = .969; RMSEA = .032; and SRMR = .045) indicating strong MI. In the intervention group, we found a substantial decrease in the latent means for the general factor (from 1.21 to 0.69) and the F2 (from 1.27 to 1.07) but not for F1 (from 0.83 to 0.79). As expected, in the control group, we found only a small and not significant decrease in the latent mean of the general factor (from 1.21 to 1.07), and no changes in F1 (0.84 vs. 0.82) and in F2 (1.29 vs. 1.22). Figure 2 shows density plots for the three latent factors across time in both the intervention and control groups, and Table 4 shows the fit indices for longitudinal MI across groups.

Figure 2.

Changes in the intervention group (IG) participants (N = 999) and the waitlist control group (CG) participants (N = 863) between baseline and posttest on the three latent factors of the Perceived Stress Scale (PSS-10).

Concurrent and Predictive Validity

For concurrent validity, we examined the point estimates for depressive symptoms, anxiety, and insomnia severity at T1 regressed on the general and specific factors of the PSS at T1. The model showed an excellent fit (CFI = .983; RMSEA = .03, 90% CI [.019, .041], SRMR = .024). Because the general and specific factors were orthogonal, the standardized partial regression coefficients for the respective outcomes represented bivariate correlations. We found that all three latent factors of the PSS were significantly related to depressive symptoms and anxiety, albeit with different effect sizes. The general factor was strongly related to depressivity (β = 0.63, p < .001) and anxiety (β = 0.47, p < .001). F1 (i.e., the factor comprising the positively worded items) and F2 showed moderate to small relations with depressivity (for F1: β = 0.24, p < .001; for F2: β = 0.12, p = .014) and with anxiety (for F1: β = 0.15, p = .003; for F2: β = 0.16, p = .002). Insomnia symptoms showed a moderate association (β = 0.22, p < .001) with the general factor and a small one (β = 0.11, p = .03) with F2.

For predictive validity, we examined the point estimates for depressive symptoms, anxiety, and insomnia severity at T2 regressed on the general and specific factors of the PSS at T1. The analyses were run in a multigroup model comparing participants allocated to the intervention groups (N = 395) and those allocated to the control groups (N = 395). The model showed an excellent fit (CFI = .983; RMSEA = .031, 90% CI [.016, .043], SRMR = .030). We found that in individuals allocated to the intervention groups, the general factor was the only relevant predictor of depressive symptoms (β = 0.31, p < .001), anxiety (β = 0.24, p = .001), and insomnia severity (β = 0.17, p = .015). For individuals who did not receive the training, the general factor was the only relevant predictor of depressive symptoms (β = 0.42, p = .001) and anxiety (β = 0.42, p < .001). However, the association between the general factor and insomnia severity was smaller and not significant (β = 0.11, p = .079). By contrast, the only relevant predictor of insomnia symptoms was the second specific factor (i.e., the factor comprising the items that were negatively worded; β = 0.17, p = .019).

Effects of Gender on the Association Between Stress and Depressive Symptoms

Building on the mean differences in perceived stress found between women and men, we examined whether gender would be found to moderate the association between latent stress scores at T1 and depressive symptoms at T2. Hence, we compared four groups (i.e., women in the intervention group, women in the control group, men in the intervention group, and men in the control group) in one multigroup analysis. The model with free estimation of the regression parameters in predicting depressive symptoms from stress showed an excellent fit (CFI = .977; RMSEA = .031, 90% CI [.022, .039]; SRMR = .042). In all four subgroups, only the general factor showed substantial effects on depressive symptoms (average β = 0.36).

In the next step, we constrained the regression parameters for the general factor to equality across women and men (but allowed for different parameters for the intervention vs. control groups). This model showed essentially the same fit (CFI = .977; RMSEA = .031, 90% CI [.022, .039]; SRMR = .043) as the unconstrained model, indicating no moderating effect of gender. The unstandardized regression parameter b was 0.53 (β = 0.32 and 0.34) for both men and women in the intervention groups, and, accordingly, 0.72 (β = 0.48 and 0.41) for both men and women in the control groups.

Discussion

The data in Study 2 were used to replicate the bifactor CFA proposed in Study 1 across the two studies. We found that the model replicated well. In addition, we replicated the gender differences in latent means, albeit with a smaller effect size. In addition, we showed strong longitudinal MI for both the intervention and control groups. Our examination of concurrent validity showed significant results for almost all associations between the three latent factors of the PSS and the criteria depressive symptoms, anxiety, and insomnia severity. Our examination of predictive validity showed that the general factor was the strongest determinant of depressive symptoms, anxiety, and insomnia severity in the intervention group, whereas in the control group, the second specific factor (i.e., the factor comprising the negatively worded items) was most relevant for sleep problems. These findings indicate that most likely, the specific factors in the bifactor model of the PSS are not pure method factors, but rather they comprise construct-relevant content.

General Discussion

With this research, we showed that a bifactor model with one general factor and two specific factors fits the data best. Hence, our results did not support the bifactor solution recently presented by Perera et al. (2016) with only one specific factor. Whereas we were able to replicate the specific factor that comprised the positively worded and subsequently reverse-keyed items, a second specific factor yielded significant variance and was needed to achieve an excellent fit. The bifactor solution that we found in the exploratory analyses was clearly supported in the second subsample and in the data obtained from six randomized controlled trials.

In both studies, we found that the bifactor CFA model was equivalent (i.e., displayed strong MI) for women and men with the former reporting significantly higher levels of perceived stress than the latter. This finding is consistent with previous studies (for an overview, see Lee, 2012). Still, the gender-related differences in the PSS scores had previously been inconclusive because some authors claimed that this result was an artifact of measurement bias, given that women are more likely to acquiesce to negatively worded items (Gitchel et al., 2011; Lee, 2012). By contrast, other authors have proposed true gender differences (e.g., Lavoie & Douglas, 2012). Our findings do not support the assumption of measurement bias, given that after establishing strong MI, we found no differences between women and men on the specific factors but only on the general factor. Moreover, the gender difference was more pronounced in Study 1, in which the sample is more likely to be representative of the general population in Germany. In Study 2, where we use data from individuals who had signed up to participate in randomized controlled trials the difference is smaller possibly indicating a more homogeneous sample.

Besides establishing a stable and replicable cross-sectional model of the PSS, we showed that the bifactor solution is temporally invariant, even in individuals taking part in a stress management training. This finding is relevant for the evaluation of interventions and the comparison of treatment approaches. If strong MI is violated, differences in the observed scores do not necessarily reflect changes in the means of the latent variables. Hence, computing and interpreting changes in the observed mean or sum score “may offer limited or even misleading insights into the structure and causes of the true progress patients make” (cf. Fried et al., 2016, p. 1355). However, we have to emphasize that strong MI over time can be achieved when computing a bifactor model and does not necessarily reflect the invariance of a unidimensional modeling strategy.

For all of the external criteria that we investigated except insomnia severity, the general factor was the strongest determinant. For insomnia severity, the second specific factor was the strongest predictor over time in the control group. For anxiety symptoms, the effect sizes of the two specific factors were similar. A further inspection of the content of the items that comprised the second specific factor showed that they mostly pertained to the experience of negative affect such as feeling upset (Item 1), nervous (Item 3), or angry (Item 9), and only Item 2 pertained to the feeling that one was not in control.

By contrast, the items forming the first specific factor rather reflected confidence and having control over irritations. The two items that loaded exclusively on the general factor (Items 6 and 10) referred to coping with an overload (difficulties piling up). Hence, it seems that the specific factors in the PSS-10 are not solely pure method effects such as wording effects. Rather, they reflect both method effects and clusters of items with similar content (cf. Golden-Kreutz et al., 2004) and should be considered construct relevant rather than irrelevant.

These findings extend the bifactor results reported by Perera et al. (2016). In their study, using a confirmatory approach, the authors proposed that one of the specific factors was a (nuisance) method factor. However, they also expected and found substantive associations between this specific factor and their criteria (depressive symptoms, anxiety, and anger), again implying that the specific factor might be construct relevant.

The PSS in Substantive Research

These findings raise the question of how to model PSS responses in substantive research. On the one hand, the ω_H reliability indices reported in this article did not reach the cutoffs for “essential unidimensionality” recommended by Rodriquez et al. (2016) and were also lower than the average indices across 50 bifactor solutions for other scales. On the other hand, in our data, the H indices suggested that the general factor was well-defined and could be replicated across studies, whereas the two specific factors were less stable.

Moreover, we argue that researchers should not base their evaluation on only a simple cutoff (i.e., ω_H ≥ .80), but rather it is the comparison between ω_H and ω that is critical. Stated differently, in scales that are less reliable overall (i.e., scales reflecting more heterogeneity in a construct), ω_H is less likely to reach the recommended cutoff. But an overall comparison between ω and ω_H revealed that 87% (Study 1) and 83% (Study 2) of the reliable variance in total scores could be attributed to the general factor. By contrast, only 12% (Study 1) or 14% (Study 2) of the reliable variance in total scores could be attributed to the specific factors, whereas 11% (Study 1) and 17% (Study 2) of the overall variance could be attributed to item uniqueness and random error. Moreover, in both studies, the correlations between the general factor and the observed total scores were high (.88 in Study 1 and .83 in Study 2).

On the basis of these percentages and correlations, one could argue that the recommendation of the original authors should be retained and that only the PSS-10 total score should be evaluated. This statement taps into a domain of intensive research and discussions on how to treat multidimensional measures, whether and in which cases the subscale scores are psychometrically justified, and whether they provide unique information above and beyond the total score. In substantive research, besides the most obvious cases when a total score is a better indicator of the latent construct represented by the subscale score than the subscale score itself, the decision of whether to use a total or a subscale score might not be clear.

For the PSS, we recommend that researchers make an informed decision that is based on the focus of interest. In cases when researchers are interested in evaluating training effects over time and need to establish MI, we recommend that a bifactor model be computed (the Mplus and lavaan code is provided at osf.io/dqnxa). In addition, researchers might be explicitly interested in modeling the specific factors. For example, in our study, we found that for some participants, the second specific factor predicted insomnia symptoms but the general factor did not. Also, in terms of intervention evaluation, we found small changes in the second factor between T1 and T2 in the intervention groups (i.e., the factor comprising more affect-related items, see Figure 2) but not in the first factor (i.e., the factor comprising the positively worded items that were subsequently reversed-keyed).

Researchers interested in using the PSS (e.g., as an inclusion criterion) may use the total score but should keep in mind that the total variance in this total score comprises not only true reliable variance in the general factor but also—besides measurement error and item uniqueness—unmodeled variance that should be attributed to the specific factors. Therefore, both the means of the PSS total score and covariances/correlations with other constructs should be interpreted carefully.

Limitations and Recommendations for Future Research

Related to this point, we need to mention that the reliability coefficients might be different in a sample that is representative of the (German) population. Our sample in Study 1 consisted of individuals who were interested in participating in an online “self-assessment of stress.” Hence, this sample might be biased by self-selection, and people interested in participating in this survey may possess certain characteristics, such as a higher stress level. Our samples in Study 2, however, should be representative of the kinds of individuals who are typically found in (online) stress management studies.

On the basis of these sample characteristics, the variance in the PSS-10 scores might be restricted in comparison with the general population, thus resulting in lower reliability estimates. Consequently, our findings require thorough replications in future studies and diverse samples. Although our first analyses were exploratory, most of our models were confirmed in the independent clusters model of CFA (ICM-CFA) framework.

Another way to model our data would have been to apply a bifactor exploratory structural equation modeling (ESEM; Morin et al., 2015) approach. However, because ESEM—in particular, with target rotation—is a more confirmatory approach than an EFA, and given our large sample, which we were able to divide into two random subsamples, we decided to compute an EFA first and then cross-validate the solution with a bifactor ICM-CFA. Also, the bifactor CFA is less demanding in terms of sample size in comparison with a bifactor ESEM. Because the PSS is often applied in intervention studies with smaller samples, we thought it was best to offer a modeling strategy that could also be applied in these applications. Furthermore, our data were self-reported, and with the external criteria that were available, we were not able to test for discriminant validity. Future studies might consider investigating the bifactor model of the PSS-10 with a multitrait–multimethod approach or within a trifactor framework (Bauer et al., 2013), for instance, with physiological measures or observer ratings and additional self-reports of general or affective well-being.

Conclusion

In this study, we showed how bifactor modeling and the derived model-based reliability indices might help us better understand the multidimensionality of the PSS-10. We argue that in addition to investigating the good fit of a model, the quality of a model should also be evaluated (e.g., through replicability indices). Given that the total scores of the PSS represent mostly—but not totally—individual differences on the general factor, the decision to rely on a bifactor model versus computing total scores should depend on researchers’ empirical questions.

Footnotes

Acknowledgements

We thank Prof. Dr. Arndt Büssing, University of Witten/Herdecke, for supporting the present study by providing the German Version of the PSS-10.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Data collection was part of the Online Health Training Project, Innovation Incubator, at Leuphana University. That project was funded by the European Union (project EFRE: CCI 2007DE161PR001) and the BARMER (German statutory health insurance company).

Supplementary Material

Supplementary material is available for this article online.

Notes

References

Andreou

Alexopoulos

Lionis

Varvogli

Gnardellis

Chrousos

Darviri

(2011). Perceived Stress Scale: Reliability and validity study in Greece. International Journal of Environmental Research and Public Health, 8, 3287-3298. doi:10.3390/ijerph8083287

Barbosa-Leiker

Kostick

Lei

McPherson

Roper

Hoekstra

Wright

(2013). Measurement invariance of the Perceived Stress Scale and latent mean differences across gender and time. Stress and Health, 29, 253-260. doi:10.1002/smi.2463

Bastien

C. H.

Vallieres

Morin

C. M.

(2001). Validation of the Insomnia Severity Index as an outcome measure for insomnia research. Sleep Medicine, 2, 297-307. doi:10.1016/S1389-9457(00)00065-4

Bauer

Howard

Baldasaro

Curran

Hussong

Chassin

Zucker

(2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18, 475-493. doi:10.1037/a0032475

Boß

Lehr

Reis

Vis

Riper

Berking

Ebert

(2016). Reliability and validity of assessing user satisfaction with web-based health interventions. Journal of Medical Internet Research, 18, e234. doi:10.2196/jmir.5952

Brunner

Nagy

Wilhelm

(2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80, 796-846. doi:10.1111/.1467-6494.2011.00749.x

Chen

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14, 464-504. doi:10.1080/10705510701301834

Chen

West

Sousa

(2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41, 189-225. doi:10.1207/s15327906mbr4102\_5

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233-255. doi: 10.1207/S15328007SEM0902_5

10.

Cohen

Kamarck

Mermelstein

(1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 385-396. doi:10.2307/2136404

11.

Cohen

Williamson

(1988). Perceived stress in a probability sample of the United States. In Spacapan

Oskamp

(Eds.), The social psychology of health (pp. 31-67). Newbury Park, CA: Sage.

12.

Ebert

Heber

Berking

Riper

Cuijpers

Funk

Lehr

(2016). Self-guided internet-based and mobile-based stress management for employees: Results of a randomised controlled trial. Occupational and Environmental Medicine, 73, 315-323. doi:10.1136/oemed-2015-103269

13.

Folkman

(1997). Positive psychological states and coping with severe stress. Social Science & Medicine, 45, 1207-1221. doi:10.1016/S0277-9536(97)00040-3

14.

Fried

van Borkulo

Epskamp

Schoevers

Tuerlinckx

Borsboom

(2016). Measuring depression over time . . . or not? Lack of unidimensionality and longitudinal measurement invariance in four common rating scales of depression. Psychological Assessment, 28, 1354-1367. doi:10.1037/pas0000275

15.

Gitchel

W. D.

Roessler

R. T.

Turner

R. C.

(2011). Gender effect according to item directionality on the Perceived Stress Scale for adults with multiple sclerosis. Rehabilitation Counseling Bulletin, 55, 20-28. doi:10.1177/0034355211404567

16.

Golden-Kreutz

Browne

Frierson

Andersen

(2004). Assessing stress in cancer patients: A second-order factor analysis model for the Perceived Stress Scale. Assessment, 11, 216-223. doi:10.1177/1073191104267398

17.

González-Ramírez

M. T.

Rodríguez-Ayán

M. N.

Hernández

R. L.

(2013). The Perceived Stress Scale (PSS): Normative data and factor structure for a large-scale sample in Mexico. Spanish Journal of Psychology, 16, E47. doi:10.1017/sjp.2013.35

18.

Hancock

Mueller

(2001). Rethinking construct reliability within latent variable systems. In Cudeck

du Toit

Sorbom

(Eds.), Structural equation modeling: Present and future—A Festschrift in honor of Karl Joreskog (pp. 195-216). Lincolnwood, IL: Scientific Software International.

19.

Hautzinger

Bailer

Hofmeister

Keller

(2012). Allgemeine Depressionsskala (ADS) Manual (Center for Epidemiological Studies Depression Scale [CES-D; Radloff LS, 1977]). Boston, MA: Hogrefe.

20.

Heber

Lehr

Ebert

Berking

Riper

(2016). Web-based and mobile stress management intervention for employees: A randomized controlled trial. Journal of Medical Internet Research, 18, e21. doi:10.2196/jmir.5112

21.

Hermann-Lingen

Buss

Snaith

(2011). Hospital Anxiety and Depression Scale: Deutsche version (3rd ed.). Bern, Switzerland: Hans Huber.

22.

Hewitt

Flett

Mosher

(1992). The Perceived Stress Scale: Factor structure and relation to depression symptoms in a psychiatric sample. Journal of Psychopathology and Behavioral Assessment, 14, 247-257.

23.

Holzinger

Swineford

(1937). The bi-factor method. Psychometrika, 2, 41-54.

24.

Khojasteh

W.-J.

(2015). Investigating the sensitivity of goodness-of-fit indices to detect measurement invariance in a bifactor model. Structural Equation Modeling: A Multidisciplinary Journal, 22, 531-541. doi:10.1080/10705511.2014.937791

25.

Klein

Brähler

Dreier

Reinecke

Müller

Schmutzer

. . . Beutel

(2016). The German version of the Perceived Stress Scale: Psychometric characteristics in a representative German community sample. BMC Psychiatry, 16, 159. doi:10.1186/s12888-016-0875-9

26.

Lavoie

J. A. A.

Douglas

K. S.

(2012). The Perceived Stress Scale: Evaluating configural, metric and scalar invariance across mental health status and gender. Journal of Psychopathology and Behavioral Assessment, 34, 48-57. doi:10.1007/s10862-011-9266-1

27.

Lee

(2012). Review of the psychometric evidence of the Perceived Stress Scale. Asian Nursing Research, 6, 121-127. doi:10.1016/j.anr.2012.08.004

28.

Leung

Lam

Chan

(2010). Three versions of Perceived Stress Scale: Validation in a sample of Chinese cardiac patients who smoke. BMC Public Health, 10, 513. doi:10.1186/1471-2458-10-513

29.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543. doi:10.1007/BF02294825

30.

Mitchell

Crane

Kim

(2008). Perceived stress in survivors of suicide: Psychometric properties of the Perceived Stress Scale. Research in Nursing & Health, 31, 576-485. doi:10.1002/nur.20284

31.

Morin

A. J. S.

Arens

A. K.

Marsh

H. W.

(2015). A bifactor exploratory structural equation modeling framework for the identification of distinct sources of construct-relevant psychometric multidimensionality. Structural Equation Modeling: A Multidisciplinary Journal, 23, 116-139. doi:10.1080/10705511.2014.961800

32.

Mueller

R. O.

Hancock

G. F.

(2010). Structural equation modeling. In Hancock

G. F.

Mueller

R. O.

(Eds.), The reviewer’s guide to quantitative methods in the social sciences (pp. 371-384). New York, NY: Routledge.

33.

Perera

Brintz

Birnbaum-Weitzman

Penedo

Gallo

Gonzalez

. . . Llabre

(2016). Factor structure of the Perceived Stress Scale-10 (PSS) across English and Spanish language responders in the HCHS/SOL Sociocultural Ancillary Study. Psychological Assessment, 29, 320-328. doi:10.1037/pas0000336

34.

Pruessner

Hellhammer

Kirschbaum

(1999). Burnout, perceived stress, and cortisol responses to awakening. Psychosomatic Medicine, 61, 197-204. doi:10.1097/00006842-199903000-00012

35.

Reise

(2012). Invited paper: The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. doi:10.1080/00273171.2012.715555

36.

Reise

Bonifay

Haviland

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129-140. doi:10.1080/00223891.2012.725437

37.

Raykov

(2012). Scale construction and development using structural equation modeling. In Hoyle

R. H.

(Ed.), Handbook of structural equation modeling (pp. 472-492). New York, NY: Guilford Press.

38.

Roberti

Harrington

(2006). Further psychometric support for the 10-item version of the Perceived Stress Scale. Journal of College Counseling, 9, 135-147. doi:10.1002/j.2161-1882.2006.tb00100.x

39.

Rodriguez

Reise

Haviland

(2016). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98, 223-237. doi:10.1080/00223891.2015.1089249

40.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36.

41.

Tomas

J. M.

Oliver

(1999). Rosenberg’s self-esteem scale: Two factors or method effects. Structural Equation Modeling: A Multidisciplinary Journal, 6, 84-98. doi:10.1080/10705519909540120

42.

van de Schoot

Lugtig

Hox

(2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology, 9, 486-492. doi:10.1080/17405629.2012.686740

43.

Yuan

Bentler

(1998). Normal theory based test statistics in structural equation modeling. British Journal of Mathematical and Statistical Psychology, 51, 289-309. doi: 10.1111/j.2044-8317.1998.tb00682.x

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB