Examining the Factor Structure and Measurement Invariance of the Difficulties in Emotion Regulation Scale Across Taiwanese and American University Students

Abstract

The Difficulties in Emotion Regulation Scale (DERS) is a widely used measure of emotion dysregulation. However, limited research has examined its factor structure and measurement invariance in cross-national samples. The present study tested competing measurement models and the measurement invariance of the DERS in university student samples from the United States (n = 324) and Taiwan (n = 399). Results indicated that the bifactor model with the Awareness subscale items removed demonstrated the best fit. The results of model-based indices provided evidence for the general emotion dysregulation factor of the DERS. Cross-national measurement invariance testing found partial strong invariance. These findings indicate that DERS would best be used as a measure of general emotion dysregulation among college students in the United States and Taiwan. These findings emphasize that future work is needed to examine cross-national differences in the construct and assessment of emotion dysregulation.

Keywords

difficulties in emotion regulation scale measurement invariance factor structure cross-national emotion regulation

In recent years, emotion regulation has sparked much interests in the field of psychology (see, e.g., Beauchaine, 2015). Emotion regulation can be viewed as the internal and external processes that individuals use to modify their emotional reactions in order to achieve their goals (Gross, 1998). Emotion regulation has been identified as a central feature underlying various forms of psychopathology (Aldao et al., 2016) and difficulties with emotion regulation have been demonstrated to be associated with various psychopathologies (e.g., Gratz et al., 2006; Nordgren et al., 2020). Given the significant role of emotion regulation and dysregulation in psychopathological development, a reliable and valid assessment of difficulties with emotion regulation is a pivotal task.

One of the most widely used measures of emotion regulation is the 36-item Difficulties in Emotion Regulation Scale (DERS; Gratz & Roemer, 2004). The DERS was primarily derived from Linehan’s (1993) theoretical framework on emotion dysregulation and was developed to measure multiple dimensions of emotion regulation difficulties. Emotion regulation difficulties can be conceptualized as deficiencies in the capacity to discern and accept emotions, and respond flexibly (Gratz & Roemer, 2004). According to Gratz and Roemer (2004), the underlying dimensions of the DERS include (1) lack of emotional awareness (Awareness), (2) lack of emotional clarity (Clarity), (3) difficulties engaging in goal-directed behaviors (Goals), (4) difficulties controlling impulsive behaviors (Impulse), (5) nonacceptance of emotional responses (Nonacceptance), and (6) limited access to emotion regulation strategies (Strategies). Extant studies have supported the psychometric properties of the DERS, and the DERS scores were found to have good reliability and construct validity (e.g., Fowler et al., 2014; Nordgren et al., 2020). The DERS has thus been recognized as a reliable and valid instrument and has been translated and validated into different languages, such as in Dutch (Neumann et al., 2010), and Hindi (Bhatnagar et al., 2020). Examples of the DERS items are shown in Table 1.

Table 1.

Examples of Items on the Difficulties of Emotion Regulation Scale by Subscale.

Item	Subscale
I pay attention to how I feel	Awareness
I am confused about how I feel	Clarity
When I am upset, I have difficulty concentrating	Goals
When I am upset, I lose control over my behaviors	Impulse
When I am upset, I become irritated with myself for feeling that way	Nonacceptance
When I am upset, I believe that I will remain that way for a long time	Strategies

Note. Gratz & Roemer (2004).

Although a large amount of research has tested the psychometric properties of the DERS, existing research has yielded mixed results regarding its factor structures. Indeed, most studies examining the original six-factor structure proposed by Gratz and Roemer (2004) can only find acceptable fit after making modifications to the models (e.g., Kokonyei et al., 2014; Neumann et al., 2010). Several studies also examined the second-order factor model with six first-order factors, and the results were mixed. For instance, Bardeen et al. (2012) found that the second-order factor model with six first-order factors exhibited adequate fit in a sample of American college students. However, Lee et al. (2016) found that the second-order factor model with six first-order factors did not result in adequate fit across three independent samples from the United States. One major reason lies in problems with the Awareness subscale, including it showing lower correlations with the other DERS factors (Neumann et al., 2010). These findings revealed potential problems with the Awareness subscale of the DERS. Bardeen et al. (2012) speculated that method effects (i.e., the tendency to respond in a way that is not related to item content) might exist since the Awareness subscale is the only subscale in which all items are reverse-scored.

To address problems with the Awareness subscale of the DERS, several studies explored a reduced version of the DERS in which the Awareness subscale items were removed. Two models have been tested: the five-factor model and the five-factor higher-order model. Several studies supported the five-factor model of a reduced version of the DERS (e.g., Bardeen et al., 2012; Lee et al., 2016). With respect to the five-factor higher-order model of a reduced version of the DERS, the results were not consistent however. For instance, Bardeen et al. (2012) found support for the five-factor higher-order model while Fowler et al. (2014) did not. Additionally, Lee et al. (2016) examined the four-factor model initially hypothesized by Gratz & Roemer (2004), but it has not received support.

Despite the inconsistent findings on the factor structures of the DERS, three recent studies examined the DERS with variations on the handling of the Awareness subscale. The results provided evidence for the bifactor model. Specifically, in a clinical sample in the United States, Osborne et al. (2017) found support for a bifactor model that excluded the Awareness subscale items from loading on the general factor. Nordgren et al. (2020) found evidence for a modified bifactor model in which the Awareness subscale was permitted to correlate with the Clarity subscale in a clinical sample in Sweden. Hallion et al. (2018) found support for a bifactor model with Awareness subscale items removed in a clinical sample in the United States. These three studies provided evidence for the bifactor model consisting of one general factor and five or six specific factors. However, whether the bifactor model can be applied to nonclinical populations or cross-national samples has not been evaluated.

Taken together, while numerous studies have examined the factor structures of the full measure of the DERS and the reduced version of the DERS, there is still a lack of consensus on its latent structure. Moreover, most of the research on the factor structures of the DERS has been conducted with European or European American samples (e.g., Bardeen et al., 2012; Lee et al., 2016). As pointed out by Henrich et al. (2010), research using predominantly WEIRD (Western, educated, industrialized, rich, and democratic) samples may not generalize to individuals from countries around the world. Thus, an examination of the factor structures of the DERS in cross-national samples is much needed in order to advance this line of research on the measurement of difficulties with emotion regulation.

Furthermore, scant research has tested the measurement invariance of the DERS across nations. This is problematic given that cultural variations regarding emotion regulation has been discussed extensively in theoretical and empirical literature. Specifically, the focus of Asian cultures tends to emphasize group harmony and promote cultural beliefs like subdued emotionality (Ruby et al., 2012) while the focus of Western cultures tends to be independence and encouraging open emotional expression (Markus & Kitayama, 1991). Cross-cultural research has demonstrated that expressive suppression, a particular form of emotion regulation, is higher in countries that emphasizes interdependence, compared to countries that emphasize independence (Matsumoto et al., 2008). Thus, given the fact that the DERS was developed to measure difficulties with emotion regulation, the important next step is to examine the measurement invariance of the DERS across nations.

Measurement invariance is often tested with multiple-group confirmatory factor analysis (CFA) and the fit of all four invariance models (configural, metric, scalar and strict invariance) allows for a full cross-national comparison (Chen, 2008). Since individuals from different nations or cultures may vary in how they manage their emotions (Matsumoto et al., 2008), these variations may further affect how they respond to the items designed to measure emotion dysregulation. Thus, if there is a lack of information on the measurement invariance of the DERS, whether the DERS measures the same construct across nations is called into question. To date, only one study has directly tested the measurement invariance of the DERS across nations (India and the United States). Snow et al. (2013) examined the five-factor model of a reduced version of the DERS in which the Awareness subscale items were removed, and no evidence of configural invariance was found. Such findings suggest that the DERS may contain different meaning and interpretations across nations and underscore the importance of further evaluating measurement equivalence of the DERS.

The Present Study

To our knowledge, no studies have examined the factor structures of the DERS in cross-national samples like the United States and Taiwan. To address this gap in the literature, the first aim of the current study was to examine the dimensionality of the DERS across the two countries. After choosing a theoretically plausible and psychometrically sound factor structure, the second goal of the present study was to test the measurement invariance of the DERS across these two countries in order to determine whether the DERS provides an equivalent and meaningful measurement across these groups.

Method

Participants and Procedure

Participants consisted of two samples from the United States and Taiwan. The American sample included 324 college students. Ages ranged from 18 to 35 (81.7% female; mean age = 20.08; standard deviation = 2.30). The ethnicity reported by the U.S. participants were Latino American (41.04%), European American (35.49%), Multiracial (7.71%), Asian American (7.09%), African American (4.32%), other (3.08%), and Native American (1.23%). The Taiwanese sample included 399 college students. Ages ranged from 18 to 25 (75.7% female; mean age = 20.35; standard deviation = 1.26).

For the American sample, data were collected from the psychology research pool via an online survey in English and, in exchange, students received credit for their psychology classes. For the Taiwanese sample, data were also collected online from the psychology research pool and other undergraduate classes with the permission of professors. Participants received extra credit for participation. The participants completed the survey in traditional Mandarin Chinese. The respective university’s Institutional Review Board approved the research.

Measures

The Difficulties in Emotion Regulation Scale (DERS; Gratz & Roemer, 2004) is a 36-item measure. Each item is measured on a 5-point Likert-type scale from 1 (almost never) to 5 (almost always) and higher scores indicate a higher level of emotion dysregulation. The DERS scores have been found to demonstrate strong reliability and adequate construct validity (Gratz & Roemer, 2004). The Chinese version of the DERS was previously translated from English into Mandarin and was pilot tested with 79 high school students (Lu, 2007). The Chinese version of the DERS was evaluated by two experts to be a suitable instrument for the participants in Taiwan. The two experts were psychology professors, and were fluent in Mandarin and English (see Supplementary Table S1 for the Chinese and English versions of the DERS items). The Chinese version of the DERS scores exhibited sound reliability and validity (Hsu, 2014; Lu, 2007).

Data Analysis

The data from the Taiwanese participants were complete (no missing values) and the data from the American participants were not complete (only one missing value). Given that the data had less than 1% missing cases (0.00,003%;), we used listwise deletion in the analyses.

In order to examine the measurement models of the DERS, confirmatory factor analysis (CFA) was conducted with R (R Core Team, 2020) using the Lavaan package. We chose CFA rather than exploratory factor analysis (EFA) because the factor structures of the DERS has been examined extensively in the empirical literature and CFA has more advantages (such as its ability to produce an unstandardized solution or to add constraints on the solution) than EFA (see Brown, 2015 for more discussions). Thus, we tested measurement models that were identified through literature review. We tested measurement models in two separate samples first and then tested measurement models in the whole sample. Through this analysis, we identified a best-fitting model.

Since the DERS items represented ordinal data, model parameters were estimated with robust maximum-likelihood estimation. Eleven models were tested (see Table 2). The first four models (Model 1–Model 4) examined the competing measurement models of the full 36-item DERS. The next seven models (Model 5–Model 11) examined competing models of the reduced version of the 30-item DERS (i.e., with the Awareness subscale removed). Model 1 was a unidimensional model. Model 2 was the original six-factor model (Graz & Roemer, 2004). Model 3 was a second-order factor model based on the work of Bardeen et al. (2012). Model 4 was a six-factor bifactor model. Model 5 was a unidimensional model. Model 6 was a five-factor model based on the work of Bardeen et al. (2012). Model 7 was a five-factor higher-order model based on the work of Bardeen et al. (2012). Model 8 was a five-factor bifactor model. Model 9 was a four-factor model, as initially hypothesized by Gratz and Roemer (2004). Model 10 was a four-factor higher-order model. Model 11 was a four-factor bifactor model.

Table 2.

DERS Measurement Models Examined.

Model	Model Description
Full measure
Model 1: Unidimensional model	All 36 items load onto a single latent factor
Model 2: Six-factor model	Items from each six subscales load onto their first-order factor and the six factors are correlated with each other
Model 4: Six-factor higher-order model	A second-order factor model with the six-factor model and one higher-order emotion dysregulation factor
Model 4: Six-factor bifactor model	A six-factor model and one general factor, wherein all seven factors (i.e., nonacceptance, strategies, goals, impulse, clarity, awareness, and the general factor) are uncorrelated with each other
Reduced measure (awareness subscale items removed)
Model 5: Unidimensional model	All 30 items load onto a single latent factor
Model 6: Five-factor model	Items from each five subscales load onto their first-order factor and the five factors are correlated with each other
Model 7: Five-factor higher-order model	A five-factor model and one higher-order emotion dysregulation factor
Model 8: Five-factor bifactor model	A five-factor model and one general emotion dysregulation factor, wherein all six factors (i.e., nonacceptance, strategies, goals, impulse, clarity, and the general factor) are uncorrelated with each other
Model 9: Four-factor model	Items from each four subscales (i.e., nonacceptance, strategies, goals/Impulse, clarity) load onto their first-order factor and the four factors are correlated with each other
Model 10: Four-factor higher-order model	A four-factor model and one higher-order emotion dysregulation factor
Model 11: Four-factor bifactor model	A four-factor model and one general emotion dysregulation factor, wherein all five factors (i.e., nonacceptance, strategies, goals/Impulse, clarity, and the general factor) are uncorrelated with each other

Model fit was assessed using the following fit indices: the root-mean square error of approximation (RMSEA), the comparative fit index (CFI), and the standardized root-mean square residual (SRMR). $χ^{2}$ statistics were also used to evaluate model fit though cautions need to be taken given that $χ^{2}$ values tend to be sensitive to sample size if the model is mis-specified (Meade et al., 2008). Good model fit was determined when the values of CFI were greater than .90 and the values of RMSEA and SRMR were less than .08 (Kline, 2016). Additionally, the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) were used to compare model fit of non-nested models.

For the best-fitting model, measurement invariance across nations was tested using multi-group CFAs, with robust maximum-likelihood estimation. Measurement invariance was examined using a series of increasingly restrictive model comparisons. First, a multi-group CFA with no equality constraints was tested across two groups (a configural invariance). Second, the restriction of equal factor loadings (weak invariance) was tested. Third, the restriction of equal item intercepts (strong invariance) was assessed. Finally, the restriction of equal residual variances (strict invariance) was evaluated.

Adding equality constraints on models tend to cause decreases in fit. Typically, the chi-square test is utilized to assess whether the decrease in fit is significant. However, as mentioned earlier, the use of the chi-square test has been criticized (Meade et al., 2008). Thus, the differences in model fit between the restricted model and the less restricted model were assessed as follows: measurement invariance should be rejected when change in CFI (ΔCFI) is less than −.01 (ΔCFI ≤ −.010) and change in RMSEA (ΔRMSEA) exceeds .015 (Cheung & Rensvold, 2002). In addition, the AIC and the BIC were used to compare competing models, with smaller values indicating better fit.

In instances where models assuming full weak or strong invariance were not supported, partial invariance was examined by identifying parameters that caused large modification indices. If strong invariance or partial strong invariance was supported, latent means between the U.S. and Taiwanese samples were compared (Byrne et al., 1989; Steenkamp & Baumgartner, 1998). Specifically, comparisons of factor means were examined by setting the factor mean of a group to zero, freely estimating the factor mean of the other group and testing for significant differences. If the bifactor model provided the best fit, model-based indices of reliability based on bifactor modeling were calculated, including coefficient omega, omega hierarchical, and omega hierarchical subscale (Rodriguez et al., 2016). These values were computed using the Bifactor Indices Calculator package (Dueber, 2017) in R. The minimum benchmark values of .50 recommended by Reise et al. (2013) were utilized to evaluate the OmegaH. In addition, the explained common variance (ECV) was calculated. A greater ECV value indicates that most of the explained variance is accounted for by the general factor and reflects the unidimensionality.

Results

Descriptive Statistics

Item mean (and standard deviation) values ranged from 1.78 to 3.63 (.79 – 1.34). Skewness values ranged between −.63 and 1.52; kurtosis values ranged between −1.16 and 1.48. Even though this is an ordinal data, skewness and kurtosis values were reported for reference.

Factor Structure of the DERS

The goodness-of-fit indices for the 11 measurement models of the DERS in the total sample are displayed in Supplementary Table S2 (the fit indices for the 11 measurement models of the DERS in the American and Taiwanese samples are displayed in Supplementary Table S3 and Table S4, respectively). The first set of models (Model 1–Model 4) was focused on the full 36-item DERS. None of the four measurement models evaluated with CFA showed good fit. The second set of models (Model 5–Model 11) was focused on the reduced version of the 30-item DERS. The unidimensional model (Model 5) showed poor fit. The five-factor (Model 6), five-factor higher-order (Model 7), four-factor (Model 9), four-factor higher-order (Model 10), and four-factor bifactor models (Model 11) showed slightly better fit than the unidimensional model (Model 5), though the CFIs of these five models fell below accepted thresholds. The five-factor bifactor model (Model 8; see Supplementary Figure S1) with one general factor and five orthogonal factors offered the best fit according to CFI, RMSEA, and SRMR ( $χ^{2} / d f$ = 1431/375 = 3.8, p < .05, RMSEA = .062, SRMR = .053, CFI = .907) and was interpretable. In addition, the AIC values and BIC values for the five-factor bifactor model (Model 8) were the smallest. Thus, the 30-item five-factor bifactor model (Model 8), with the highest CFIs and lowest RMSEAs, was chosen as the best-fitting model. Subsequently, measurement invariance for the 30-item and five-factor bifactor model (Model 8) was examined across nations. Supplementary Figure S1 exhibits the standardized factor loadings for the bifactor model. Standardized factor loadings for the general factor ranged from .16 to .80 and five items had loadings smaller than .40 (items 20, 24, 22, 1, and 7). In general, factor loadings tended to be stronger on the general factor and weaker on the specific factors, except for the Clarity factor.

Model-based Reliability and Dimensionality

Reliability results for the five-factor bifactor model (Model 8) are illustrated in Supplementary Table S5. The coefficient alpha and omega reliability estimates of the total and the subscales were high. The OmegaH for the total scale was high (.88) and the OmegaH results for all subscales were low, suggesting a strong general factor dimension. With respect to dimensionality, the ECV of the general factor was .66, which indicates that the general factor accounted for 66% of the common variance extracted. Thus, the DERS bifactor model provided evidence of a general factor.

Measurement Invariance of the DERS

Four levels of invariance for the 30-item and five-factor bifactor model (Model 8) were tested (the results are displayed in Supplementary Table S6). Configural invariance showed adequate model fit statistics. These findings provided support for a five-factor bifactor model across two nations. Weak invariance was then tested, and results indicated that the Δ $χ^{2}$ was statistically significant along with a small increase in AIC (180). In addition, the changes in CFI (ΔCFI) from configural to the weak invariance model exceeded the recommended cutoff for rejecting measurement invariance (ΔCFI = −.016), suggesting that there was substantial deterioration in model fit. However, the other fit indices supported an acceptable model fit including a trivial increase in RMSEA (ΔRMSEA = .004) and BIC decrease of 68. Together, these findings provide moderate support for weak invariance across nations. We then used modification indices to locate parameters that led to the largest increase in $χ^{2}$ when imposing equality constraints. These equality constraints were freed one at a time accumulatively until the partial weak invariance was established based on the drop in the CFI. In both groups, six-factor loadings were freely estimated: DERS 33 on Goals, DERS 29 on DERS, DERS 25 on DERS, DERS 3 on DERS, DERS 12 on Nonacceptance, and DERS 29 on Nonacceptance. The partial weak invariance was established (ΔCFI = −0.01). Next, in testing for strong measurement invariance, the changes in CFI exceeded the recommended cutoff compared to the baseline model (ΔCFI = −0.016). Modification indices suggested that the intercepts for Items 23, 9, 5, 30, and 36 were the source of the misfit. After releasing the equality constraints of these intercepts consecutively, partial strong measurement invariance was established. Finally, the test for strict measurement invariance showed substantial deterioration in model fit (ΔCFI = −.024, ΔRMSEA = .006).

Latent Mean Comparisons

Given the evidence of partial strong invariance of the DERS, we conducted latent mean comparisons to test whether the latent mean of the DERS general factor was invariant across nations. Results indicated that the Taiwanese sample reported significantly higher mean levels of the general factor on the DERS scores than the American sample (ΔM = −2.859, SE = .535, t = 5.346, p < .001). The effect size was measured by the standardized mean difference (i.e., Cohen’s d) and was 3.52, where the standardized mean difference was derived from the difference between estimated latent means with the latent variances fixed to one in both groups. However, the effect size should be interpreted with caution given that only partial strong invariance was established.

Discussion

The current study examined the factor structure and measurement invariance of the DERS across nations (the United States and Taiwan). The primary findings are that the bifactor model with the Awareness subscale items removed demonstrated better fit and that measurement invariance testing indicated that factor structures were equivalent across college students in the United States and Taiwan. Partial weak invariance and partial strong invariance were established across nations. Taiwanese students reported significantly higher emotion dysregulation than American students.

The present study extended existing research by cross-culturally testing whether the factor structure of the DERS was consistent with what has been found in other, predominantly European or European American samples. 11 CFA models were tested and evaluated. The results are consistent with Hallion et al.’s (2018) study, which indicated that the bifactor model with the Awareness subscale items removed demonstrated better fit. Our findings also add support for the clinical and measurement utility of the bifactor model of the DERS across the United States and Taiwan for college students.

The model-based reliability and dimensionality of the DERS based on the bifactor model were examined. The results are consistent with the study by Nordgren et al. (2020) and suggested that the general factor accounted for most of the explained variance in comparison to the specific factors. With respect to the subscales of the DERS, the OmegaH values varied and the results provided some support for the use of the Clarity subscale. The findings indicated that interpretations of the subscales of Goals, Impulse, Nonacceptance, and Strategies need to be made with caution in cross-national samples, give that most of the reliable variance was accounted for by the general factor than by these specific factors.

One of the controversies about the DERS pertains to the problems with the Awareness subscale. In line with previous studies (e.g., Bardeen et al., 2012; Osborne et al., 2017), our results provided evidence for the problems with the Awareness subscale in a cross-national sample. Thus, consistent with the recommendations made by previous studies (e.g., Hallion et al., 2018), the results of the present study suggested that the Awareness subscale be excluded when using the DERS in future studies. To address concerns with reverse-coded problems of the Awareness subscale, Bardeen et al. (2016) developed a modified version of the DERS (M-DERS) in which all of the reverse-coded items of the DERS were rephrased in a straightforward manner. The CFA results supported the factor structures of the M-DERS (Bardeen et al., 2016). Future research may consider using the M-DERS with culturally diverse samples. Another possible way to handle problems with the Awareness subscale is to use the DERS-16 (i.e., a 16-item brief version of the DERS proposed by Bjureberg et al., 2016) since the DERS-16 has excluded the Awareness subscale and exhibited sound reliability and validity (Bjureberg et al., 2016; Hallion et al., 2018). Given recent discussions on potential benefits of using shortened versions of the DERS (Skutch et al., 2019), it is suggested that future research use DERS-16 or other shortened versions of the DERS (e.g., DERS-SF developed by Kaufman et al., 2016 and the DERS-18 proposed by Victor & Klonsky, 2016) with culturally diverse samples and test the measurement invariance of the shortened versions of the DERS in cross-cultural samples.

Cross-national measurement invariance testing indicated that configural invariance was supported, revealing that the factor structures of the DERS are the same across the U.S. and Taiwanese samples. Partial weak invariance of the DERS was established. The following six-factor loadings were variant across samples: Item 33 on a specific factor (Goals), Item 29 on the general factor, Item 25 on the general factor, Item 3 on the general factor, Item 12 on a specific factor (Nonacceptance), and Item 29 on a specific factor (Nonacceptance). Variances of these six loadings across nations suggest that there are variations of how well these items can measure the latent construct of the DERS. Differences in loadings may be explained by reasons such as translation issues and cultural differences (Chen, 2008). Regarding translation issues, variant loadings may be the result of some items having different meanings in other languages. For instance, the direct translation of “out of control” in Item 3 (“I experience my emotions as overwhelming and out of control”) may make the Taiwanese participants to be less willing to report the frequency of this item than Americans since “out of control” in Chinese can be interpreted as an extreme expression of losing control. As pointed out by Geisinger (1994), using forward translation or back translation may not be enough in cross-cultural assessments. Thus, even though the Chinese version the DERS used in the present study followed the basic steps of translation (including translation and pilot tests), the translation procedures did not meet all of the requirements of the ITC Guidelines for Translating and Adapting Test (International Test Commission, 2018) and may affect the results of the study. It will be helpful to follow the ITC Guidelines for Translating and Adapting Test for cross-cultural assessments of the DERS for future research such as using reconciliation procedures by a group of experts and incorporating qualitative interviews after pilot tests of the instrument.

In terms of cultural differences, lack of loading equivalence may arise from the different meanings of emotions in different cultures. As pointed out by Markus and Kitayama (1991), how various emotions are experienced may depend on cultural values and can be broadly classified as ego-focused emotions versus other-focused emotions. Ego-focused emotions, such as anger, tend to use an individual’s self as the reference framework. In contrast, other-focused emotions, such as shame, tend to use another person as the referring source. This implicit distinction has important implications for individuals from different cultures. Specifically, individuals from individualistic cultures can be more skilled at experiencing these ego-focused emotions while individuals from interdependent cultures can be more skilled at experiencing other-focused emotions. Thus, for Americans, items that are connected to ego-focused emotions may be experienced more strongly, such as Item 29 (“When I’m upset, I become irritated with myself for feeling that way”). For Taiwanese students, however, items that are connected to other-focused emotions may be experienced more strongly, such as Item 25 (“When I’m upset, I feel guilty for feeling that way”) and Item 12 (“When I’m upset, I become embarrassed for feeling that way”). These six items may have different meanings across cultures, and differences at an item level can be referred to as item bias or differential item functioning (DIF) (e.g., Holland & Thayer, 1988). For these items, it should be noted that the response patterns caused by DIF may not be consistent with the raw scores. The raw scores are also affected by differences in item performance across groups, and this can be referred to as impact (e.g., Ackerman, 1992). Conceptually, raw scores can be a function of DIF and impact, and thus raw scores may be different from response patterns (Wu et al., 2017). For instance, Taiwanese students reported higher raw scores on Item 29 (Mean = 3.63) than American students (Mean = 2.43) though American students may be more likely to experience ego-focused emotions than Taiwanese students, and this discrepancy can be explained in the latent mean comparison in which Taiwanese participants reported higher factor means of the emotion dysregulation general factor than American participants.

Partial strong measurement invariance of the DERS was established. The following five intercepts showed noninvariance: Item 23 (“When I’m upset, I feel like I am weak”), Item 9 (“I am confused about how I feel”), Item 5 (“I have difficulty making sense out of my feelings”), Item 30 (“When I’m upset, I start to feel very bad about myself”), and Item 36 (“When I’m upset, my emotions feel overwhelming”). Specifically, American college students had a higher probability of agreeing with these items than Taiwanese college students. It should be noted that Taiwanese students reported higher raw scores than American students on Item 5 (Mean for Taiwanese = 2.43; Mean for American = 2.41) and Item 23 (Mean for Taiwanese = 3.20; Mean for American = 2.60). As mentioned earlier, the higher factor means of the emotion dysregulation reported by Taiwanese participants than American participants may account for this discrepancy. The lack of intercept invariance on these items can be attributed to a variety of factors. For instance, Taiwanese students may be less likely to endorse these items since agreeing with them may reflect vulnerability and may imply “losing face” (i.e., losing social standing due to being evaluated by others in a negative manner). Similarly, expressive suppression, a particular form of emotion regulation, is higher in countries that emphasizes collectivism (Matsumoto et al., 2008) and may further preclude Taiwanese students from agreeing with these items.

Since partial strong invariance was established, we were able to compare factor means. The latent mean comparison revealed that Taiwanese students reported a significantly higher latent mean on the emotion dysregulation general factor than American students, indicating that Taiwanese college students may be less capable of regulating their emotions. These findings seem to align with past research that suggested that there are cultural differences in emotion regulation (Matsumoto et al., 2008). One of the primary factors that contribute to cultural differences in emotion regulation is related to the distinctive cultural context, such as individualism and collectivism. For instance, in a collectivist culture like Taiwan, emotional control is emphasized. If conceptualizations of difficulties of emotion regulation involve emotional control, the findings that Taiwanese college students reported more emotion dysregulation is not surprising. However, since the DERS only exhibited partial strong invariance, some intercepts of the scale might still be different from one culture into another and have not been identified. Thus, latent mean differences of the DERS could be biased due to this inequality and interpretations need to be made with caution.

It should be noted that our results were not in line with previous research that examined measurement invariance of the DERS across nations (Snow et al., 2013). Specifically, Snow et al. (2013) did not find evidence to support the configural invariance across nations among college students from India and the United States. Despite the fact that different findings might be sample-specific, two reasons may account for these results. First, the measurement models that were used to test configural invariance were not the same. Snow et al. (2013) examined four measurement models and used the five-factor model of a reduced version of the DERS (excluding the Awareness subscale) as a baseline model, while the present study examined 11 models and used the bifactor model as a baseline model. Second, such differences might have arisen due to cultural differences between India and Taiwan. While Asian countries tend to be viewed as generally collectivist, the distinctive sociopolitical experiences of India and Taiwan may lead to different manifestations of collectivism.

The present findings should also be considered in light of study limitations. First, the data were only obtained from college students from the United States and Taiwan. More research is needed to examine the psychometric properties of the DERS in other countries that vary in the levels of individualism and collectivism. Moreover, the majority of the participants in both samples are female college students from a university in Taiwan and a university in the United States and may thus limit the generalizability of the findings. Indeed, the U.S. sample for this study was quite heterogenous and may not be representative of the typical ethnic distribution in the U.S. Second, while the current study provided support for the bifactor model of the DERS, controversies about bifactor models should be taken into consideration. As pointed out by Bonifay et al. (2017), bifactor models tend to exhibit better fit than other competing models due to problems such as overfitting rather than the appropriateness of the structure of the measured construct. Moreover, even though the bifactor model offered an adequate fit and was chosen as the final model, one of the fit indices was at the borderline cutoff value (CFI = .907) and the possibility of model mis-specification cannot be ruled out. Thus, without more research, it is difficult to know whether this is a valid representation of the factor structure in this population and the results need to be interpreted with caution. Third, although the 5-point scale used in this study is considered sufficient to be treated as continuous, it would be preferable if it could be specified as a categorical/ordinal scale (e.g., Li, 2016). Indeed, in the multi-group analysis, robust WLS estimators (e.g., DWLS estimator) have been applied to model estimation, but encountered convergence problems. Given the relatively larger model employed in the study, it is suggested that future research can obtain larger sample sizes and employ the estimator that is more suitable to ordinal data (e.g., DWLS) to validate the study (e.g., DiStefano & Morgan, 2014; Moshagen & Musch, 2014; Yang-Wallentin, Joreskog & Luo, 2010Yang-Wallentin et al., 2010). Fourth, the reasons for non-equivalence are not known. Different sources of bias such as construct bias (e.g., cultural differences in appropriate behaviors that are connected to the construct) and method bias (e.g., social desirability) may affect the results in cross-cultural studies (Van de Vijver & Tanzaer, 2004). More research will be needed in examining potential biases in the DERS in cross-cultural studies.

In conclusion, the current study examined the factor structure of the DERS in a cross-national sample. The results suggested that the bifactor model with the Awareness subscale items removed provided the best fit. Model-based indices suggested that the use of a general emotion dysregulation factor of the DERS is recommended with cross-national samples. The present study is also the first to test the measurement invariance across the United States and Taiwan. Results indicate that college students from the United States and Taiwan conceptualize the latent construct of emotion dysregulation measured by DERS similarly. The present study highlights the need for more research attending to the cultural factors in the dimensionality and measurement invariance of the construct of emotion dysregulation and DERS so that inaccurate conclusions in cross-cultural research can be avoided.

Supplemental Material

Supplemental Material—Examining the Factor Structure and Measurement Invariance of the Difficulties in Emotion Regulation Scale Across Taiwanese and American University Students

Supplemental Material for Examining the Factor Structure and Measurement Invariance of the Difficulties in Emotion Regulation Scale Across Taiwanese and American University Students by Yun-Jy Yeh, Jyun-Hong Chen, William Tsai, and Sasha Kimel in Journal of Psychoeducational Assessment

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the Ministry of Science and Technology in Taiwan (Grant No. MOST 108-2410-H-031-004).

ORCID iD

Yun-Jy Yeh

Supplemental Material

Supplemental material for this article is available online.

References

Ackerman

T. A.

(1992). A didactic explanation of item bias, item impact, and item validity from a multidimensional perspective. Journal of Educational Measurement, 29(1), 67–91. https://doi.org/10.1111/j.1745-3984.1992.tb00368.x

Aldao

Gee

De Los Reyes

Seager

(2016). Emotion regulation as a transdiagnostic factor in the development of internalizing and externalizing psychopathology: Current and future directions. Development and Psychopathology, 28(4pt1), 927–946. https://doi.org/10.1017/S0954579419000348

Bardeen

J. R.

Fergus

T. A.

Hannan

S. M.

Orcutt

H. K.

(2016). Addressing psychometric limitations of the difficulties in emotion regulation scale through item modification. Journal of Personality Assessment, 98(3), 298–309. https://doi.org/10.1007/s10862-012-9280-y

Bardeen

J. R.

Fergus

T. A.

Orcutt

H. K.

(2012). An examination of the latent structure of the difficulties in emotion regulation scale. Journal of Psychopathology and Behavioral Assessment, 34(3), 382–392. https://doi.org/10.1007/s10862-012-9280-y

Beauchaine

T. P.

(2015). Future directions in emotion dysregulation and youth psychopathology. Journal of Clinical Child & Adolescent Psychology, 44(5), 875–896. https://doi.org/10.1080/15374416.2015.1038827

Bhatnagar

Shukla

Pandey

(2020). Validating the factor structure of the Hindi version of the difficulties in emotion regulation scale. Journal of Psychopathology and Behavioral Assessment, 42(2), 377–396. https://doi.org/10.1007/s10862-020-09796-6

Bjureberg

Ljotsson

Tull

M. T.

Hedman

Sahlin

Lundh

Bjarehed

DiLillo

Messman-Moore

Gumpert

C. H.

Gratz

K. L.

(2016). Development and validation of a brief version of the difficulties in emotion regulation scale: The DERS-16. Journal of Psychopathology and Behavioral Assessment, 38(2), 284–296. https://doi.org/10.1007/s10862-015-9514-x

Bonifay

Lane

S. P.

Reise

S. P.

(2017). Three concerns with applying a bifactor model as a structure of psychopathology. Clinical Psychological Science, 5(1), 184–186. https://doi.org/10.1177/2167702616657069

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.

10.

Byrne

Shavelson

Muthen

(1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456–466. https://doi.org/10.1037/0033-2909.105.3.456

11.

Chen

F. F.

(2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95(5), 1005–1018. https://doi.org/10.1037/a0013193

12.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/s15328007sem0902_5

13.

DiStefano

Morgan

G. B.

(2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling, 21(3), 425–438. https://doi.org/10.1080/10705511.2014.915373

14.

Dueber

D. M.

(2017). Bifactor indices calculator: A microsoft excel-based tool to calculate various indices relevant to bifactor CFA models. https://doi.org/10.13023/edp.tool.01

15.

Fowler

J. C.

Charak

Elhai

J. D.

Allen

J. G.

Frueh

B. C.

Oldham

J. M.

(2014). Construct validity and factor structure of the difficulties in emotion regulation scale among adults with severe mental illness. Journal of Psychiatric Research, 58, 175–180. https://doi.org/10.1016/j.jpsychires.2014.07.029

16.

Geisinger

K. F.

(1994). Cross-cultural normative assessment: Translation and adaption issues influencing the normative interpretation of assessment instruments. Psychological Assessment, 6(4), 304–312. http://doi.org/10.1037/1040-3590.6.4.304

17.

Gratz

K. L.,

Roemer

(2004). Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the Difficulties in Emotion Regulation Scale. Journal of Psychopathology and Behavioral Assessment, 26(1), 41–54. https://doi.org/10.1023/B:JOBA.0000007455.08539.94

18.

Gratz

K. L.

Rosenthal

M. Z.

Tull

M. T.

Lejuez

C. W.

Gunderson

J. G.

(2006). An experimental investigation of emotion dysregulation in borderline personality disorder. Journal of Abnormal Psychology, 115(4), 850–855. https://doi.org/10.1037/0021-843x.115.4.850

19.

Gross

J. J

. (1998). The emerging field of emotion regulation: An integrative Review. Review of General Psychology, 2(3), 271–299. https://doi.org/10.1037/1089-2680.2.3.271

20.

Hallion

L. S.

Steinman

S. A.

Tolin

D. F.

Diefenbach

G. J.

(2018). Psychometric properties of the Difficulties in Emotion Regulation Scale (DERS) and its short forms in adults with emotional disorders. Frontiers in Psychology, 9, 539. https://doi.org/10.3389/fpsyg.2018.00539

21.

Henrich

Heine

S. J.

Norenzayan

(2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61–135. https://doi.org/10.1017/S0140525X0999152X

22.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129–145). Lawrence Erlbaum Associates Publishers.

23.

Hsu

Y. C.

(2014). Explore the relation of adolescents’ self-harm motivation, separation-individuation, emotion regulation and self-harm behaviors. (Master's thesis). National Taiwan Normal University.

24.

International Test Commission (2018). The ITC Guidelines for translating and adapting tests (second edition). International Journal of Testing, 18(2), 101–134. https://doi.org/10.1080/15305058.2017.1398166

25.

Kaufman

E. A.

Xia

Fosco

Yaptangco

Skidmore

C. R.

Crowell

S. E.

(2016). The difficulties in emotion regulation scale short form (DERS-SF): Validation and replication in adolescent and adult samples. Journal of Psychopathology and Behavioral Assessment, 38(3), 443–455, https://doi.org/10.1007/s10862-015-9529-3

26.

Kline

R. B.

(2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press.

27.

Kökönyei

Urbán

Reinhardt

Anna

Demetrovics

(2014). The difficulties in emotion regulation scale: Factor structure in chronic pain patients. Journal of Clinical Psychology, 70(6), 589–600. https://doi.org/10.1002/jclp.22036

28.

Lee

D. J.

Witte

T. K.

Bardeen

J. R.

Davis

M. T.

Weathers

F. W.

(2016). A factor analytic evaluation of the difficulties in emotion regulation scale. Journal of Clinical Psychology, 72(9), 933–946. https://doi.org/10.1002/jclp.22297

29.

C. H.

(2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093

30.

Linehan

M. M.

(1993). Cognitive-behavioral treatment of borderline personality. Guilford Press.

31.

M. C.

(2007). Exploring the relation of the motivation of deliberate self-harm, negative affect, experiential avoidance, emotion regulation and adolescents’ deliberate self-harm behaviors, (Master's thesis). National Chengchi University.

32.

Markus

H. R.

Kitayama

(1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 224–253. https://doi.org/10.1037/0033-295x.98.2.224

33.

Matsumoto

Yoo

S. H.

Nakagawa

Multinational Study of Cultural Display Rules . (2008). Culture, emotion regulation, and adjustment. Journal of Personality and Social Psychology, 94(6), 925–937. https://doi.org/10.1037/0022-3514.94.6.925

34.

Meade

A. W.

Johnson

E. C.

Braddy

P. W.

(2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93(3), 568–592. https://doi.org/10.1037/0021-9010.93.3.568

35.

Moshagen

Musch

(2014). Sample size requirements of the robust weighted least squares estimator. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 10(2), 60–70. https://doi.org/10.1027/1614-2241/a000068

36.

Neumann

van Lier

P. A. C.

Gratz

K. L.

Koot

H. M.

(2010). Multidimensional assessment of emotion regulation difficulties in adolescents using the Difficulties in Emotion Regulation Scale. Assessment, 17(1), 138–149. https://doi.org/10.1177/1073191109349579

37.

Nordgren

Monell

Birgegard

Bjureberg

Hesser

(2020). Factor structure of the difficulties in emotion regulation scale in treatment seeking adults with eating disorders. Journal of Psychopathology and Behavioral Assessment, 42(1), 111–126. https://doi.org/10.1007/s10862-019-09765-8

38.

Osborne

T. L.

Michonski

Sayrs

Welch

S. S.

Anderson

L. K.

(2017). Factor structure of the Difficulties in Emotion Regulation Scale (DERS) in adult outpatients receiving Dialectical Behavior Therapy (DBT). Journal of Psychopathology and Behavioral Assessment, 39(2), 355–371. https://doi.org/10.1007/s10862-017-9586-x

39.

R Core Team . (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, URL. https://www.R-project.org/

40.

Reise

S. P.

Scheines

Widaman

K. F.

Haviland

M. G.

(2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26. https://doi.org/10.1177/0013164412449831

41.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045

42.

Ruby

M. B.

Falk

C. F.

Heine

S. J.

Villa

Silberstein

(2012). Not all collectivisms are equal: Opposing preferences for ideal affect between East Asians and Mexicans. Emotion, 12, 1206–1209. https://doi.org/10.1037/a0029118

43.

Skutch

J. M.

Wang

S. B.

Buqo

Haynos

A. F.

Papa

(2019). Which brief is best? Clarifying the use of three brief versions of the difficulties in emotion regulation scale. Journal of Psychopathology and Behavioral Assessment, 41(3), 485–494. https://doi.org/10.1007/S10862-019-09736-Z

44.

Snow

N. L.

Ward

R. M.

Becker

S. P.

Raval

V. V.

(2013). Measurement invariance of the difficulties in emotion regulation scale in India and the United States. Journal of Educational and Developmental Psychology, 3(1), 147–215. https://doi.org/10.5539/jedp.v3n1p147

45.

Steenkamp

J. B. E.

Baumgartner

(1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25(1), 78–90. http://doi.org/10.1086/209528

46.

Van de Vijver

Tanzer

N. K.

(2004). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology, 54(2), 119–135. https://doi.org/10.1016/j.erap.2003.12.004

47.

Victor

S. E.

Klonsky

E. D.

(2016). Validation of a brief version of the difficulties in emotion regulation scale in five samples. Journal of Psychopathology and Behavioral Assessment, 38(4), 582–589. https://doi.org/10.1007/s10862-016-9547-9

48.

A. D.

Liu

Stone

J. E.

Zou

Zumbo

B. D.

(2017). Is difference in measurement outcome between groups differential responding, bias or disparity? A methodology for detecting bias and impact from an attributional stance. Frontiers in Education, 2, 39. https://doi.org/10.3389/feduc.2017.00039

49.

Yang-Wallentin

Jöreskog

K. G.

Luo

(2010). Confirmatory factor analysis of ordinal variables with misspecified models. Structural Equation Modeling, 17(3), 392–423. https://doi.org/10.1080/10705511.2010.489003

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.65 MB