An Examination of the Factor Structure of the Multidimensional Psychological Flexibility Inventory

Abstract

The Multidimensional Psychological Flexibility Inventory (MPFI), a 60-item self-report measure, assesses the Acceptance and Commitment Therapy Hexaflex. The factor structure of the MPFI was examined in this study. In a community sample of adults (N = 827), four models (correlated six-factor, one-factor, higher order, and bifactor) were tested for each of the constructs of interest (i.e., psychological flexibility and psychological inflexibility). All models, with the exception of the one-factor, provided adequate fit to the data. Differences between the three adequate fitting models were trivial in magnitude. Additional statistical indices from the bifactor models indicated that the general factors accounted for the large majority of reliable variance. The majority of the domain-specific factors evidenced redundancy with their respective general factors. Results from a series of structural regressions indicated that the domain-specific factors did not provide additional incremental utility above and beyond the general factors in predicting two relevant clinical constructs (i.e., health anxiety and depression). These results provide support for the use of the MPFI Flexibility and Inflexibility total scores, but not subscale scores. The MPFI may require further refinement to either greatly reduce the length of the measure, or to ensure that subscales have incremental utility.

Keywords

Multidimensional Psychological Flexibility Inventory bifactor analysis psychological flexibility psychometrics

Psychological flexibility, a construct that is particularly relevant to Acceptance and Commitment Therapy (ACT), is defined as “the ability to fully contact the present moment and the thoughts and feelings it contains without needless defense, and, depending on what the situation affords, persisting in or changing behavior in the pursuit of goals and values” (Hayes et al., 2011). Psychological flexibility is most often operationalized in terms of its component parts: acceptance, cognitive defusion, contact with the present moment, self as context, committed action, and values (Hayes et al., 1999). Each component of psychological flexibility, often times described as the ACT “Hexaflex,” represents a method of enhancing well-being by openly and flexibly responding to the present moment in spite of uncomfortable experiences (Hayes et al., 1999, 2011). While empirical evidence supports the use of ACT for treating individuals with a number of psychological disorders (e.g., Arch et al., 2012; Bai et al., 2020; Hayes et al., 2006), research related to the measurement of the domains of psychological flexibility has moved more slowly (Hayes et al., 2006).

Assessment of psychological flexibility has generally focused on experiential avoidance, or the unwillingness to remain in contact with aversive internal experiences (Hayes et al., 1999). The most common self-report measures of psychological flexibility are the Acceptance and Action Questionnaire–II (AAQ-II; Bond et al., 2011) and the Multidimensional Experiential Avoidance Scale (MEAQ; Gámez et al., 2011). The former and similar measures (e.g., Avoidance and Fusion Questionnaire; Fergus et al., 2012) serve as global measures of psychological inflexibility. These measures are unable to capture the unique variance that each dimension of the Hexaflex accounts for in predicting constructs of interest. The MEAQ and its shorter counterpart, the Brief Experiential Avoidance Questionnaire (Gámez et al., 2014), were developed to provide sufficient coverage of different manifestations of experiential avoidance and to address concerns regarding the construct validity of the AAQ-II (Rochefort et al., 2018; Wolgast, 2014). Specifically, some evidence indicates that the AAQ-II is more strongly related to measures of general distress than measures of acceptance or avoidance (Rochefort et al., 2018; Wolgast, 2014). The MEAQ has been successful in providing greater breadth of manifestations of experiential avoidance and providing better discriminant validity in comparison to the AAQ-II. However, the MEAQ was not developed with a specific theoretical model in mind and none of these measures captures all of the components of the ACT Hexaflex.

Recently, Rolffs et al. (2018) developed the Multidimensional Psychological Flexibility Inventory (MPFI) in an attempt to address this gap in the literature. Specifically, they designed a measure that would align with the underlying ACT theory by capturing the six proposed underlying dimensions of psychological flexibility (Hayes et al., 2011). Additionally, the authors propose six dimensions of psychological inflexibility (i.e., Experiential Avoidance, Lack of Contact With the Present Moment, Self as Content, Fusion, Lack of Contact With Values, Inaction), citing literature that these related but distinct constructs would contribute meaningful information in fully understanding relevant outcomes (Rolffs et al., 2018). Aligning with some previous research in this area (e.g., Kashdan & Rottenberg, 2010), the authors suggest that rather than distinct opposites, the six components of psychological flexibility are more strongly related to each other than to their inflexibility counterparts.

Rolffs et al. (2018) recruited adult participants from undergraduate and online research pools (e.g., Amazon’s Mechanical Turk). Three initial studies were conducted to refine an initial pool of 554 items to a final measure consisting of 60 items. The authors generated some of the items for the initial item pool themselves, but the large majority of items were taken from existing self-report measures used in studies from the ACT and mindfulness literature. Following initial exploratory and confirmatory factor analyses (EFA and CFA), item response theory (IRT) was used on a refined pool of 288 items in a second study to identify the five most effective items for each dimension of the ACT Hexaflex. EFA and CFA were conducted one final time to verify the proposed factor structure with the 60 items that were identified via IRT. As part of this verification process, all 60 items of the MPFI were examined simultaneously via EFA. Results of the EFA were indicative of a 12-factor solution. Each of the proposed hierarchal models (i.e., inflexibility and flexibility) provided adequate fit to the data in the follow-up CFA. Although the latent correlation between the global inflexibility and flexibility higher order factor in the CFA model was large in magnitude (i.e., r = −.74), the authors suggested that the two higher order factors provided valuable unique information, and thus, should be modeled independently. The MPFI represents an important step forward in this line of research because it is necessary to have psychometrically sound measures of psychological in/flexibility and the dimensions outlined in the Hexaflex model to evaluate ACT and its underlying theory. However, an independent examination of the MPFI’s factor structure has not yet been conducted.

As described, Rolffs et al. (2018) designed the MPFI to produce a global inflexibility and flexibility score, with each total score consisting of six distinct, but related, subscale scores. Use of total scores assumes that each domain-specific factor of the flexibility and inflexibility higher order factors represents the same overarching construct, while use of subscale scores assumes that the domain-specific factors provide unique information beyond the total scores. While Rolffs et al. (2018) has provided some initial evidence in support of two hierarchical models, hierarchical modeling does not directly test the assumption that items of the domain-specific factors provide unique information beyond a higher order or general factor (Reise, 2012). Because the MPFI was designed for use with total and subscale scores, it is important to use an analytic approach that can simultaneously address both of these assumptions (i.e., bifactor analysis; Chen et al., 2006).

Through the use of bifactor analysis, the unique contributions of both the general and domain-specific factors can be isolated (Reise, 2012), which allows for the simultaneous investigation of the general factors and the degree to which each domain-specific factor is meaningfully distinct from the general factors (Reise et al., 2010; Reise et al., 2013; Rodriguez et al., 2016). To our knowledge, the MPFI has not yet been examined using a bifactor modeling approach. Use of this approach may yield important new insights into the factor structure of the MPFI and help determine appropriate use of its total and subscale scores. For example, domain-specific factors that are redundant with their respective general factor might suggest that a total score should be used in lieu of subscale scores (Reise, 2012). Conversely, when domain-specific factors do not represent the same higher order construct, a total score may be deemed uninterpretable as an indicator of a single construct, while use of subscales scores may still be psychometrically justified.

Consistent with standard practice (Brown, 2015; Kline, 2016), we compared the fit of each bifactor model (i.e., inflexibility and flexibility) with competing models (i.e., a correlated six-factor model, a one-factor model, and a hierarchical model). The value of comparing bifactor models with alternative models has been questioned on the grounds that bifactor models may provide better fit to the data because of their inherent flexibility (Bonifay et al., 2017; Reise et al., 2016). As such, a number of additional statistical indices that were developed for use with the bifactor approach were examined in the present study (discussed in greater detail below; Rodriguez et al., 2016). These indices provide information that is consistent with the central aim of the present study (i.e., examining the incremental value of the domain-specific factors beyond the general factor, determining the replicability and stability of the factors).

A further benefit of utilizing bifactor modeling is the ability to determine whether domain-specific factors provide incremental value in predicting criterion variables above and beyond the contribution of the general factor (Brown, 2015). Thus, as a preliminary investigation, structural regression models were used to examine if the domain-specific factors of the MPFI accounted for unique variance in scores on two criterion measures (i.e., health anxiety and depression) after accounting for the general factors (i.e., inflexibility and flexibility). Per the recommendations of Brown (2015), the other models tested as part of this study (e.g., correlated six-factor, higher order) were not utilized for the structural regression models because they do not offer the same benefit.

Although health anxiety is just one of many variables that is theoretically relevant to both flexibility and inflexibility, it has been shown to be related to ACT-related constructs such as cognitive fusion and experiential avoidance in several studies (Fergus, 2015; Wheaton et al., 2010). Health anxiety has also recently been a focus of ACT outcome research, with several recently published studies examining moderators of treatment outcome with respect to health anxiety, especially within the context of the COVID pandemic (Landi et al., 2020; Leonidou et al., 2019). Similarly, a clear relationship between depression and psychological inflexibility has been established in a number of empirical studies (Fonseca et al., 2020; Kashdan & Rottenberg, 2010; Yasinski et al., 2020). Based on these previous findings, using health anxiety and depression as the criterion variables should allow for an adequate test of the incremental utility of the MPFI subscales beyond the total score.

Method

Participants and Procedure

Adult participants (N = 999) were recruited via Amazon’s Mechanical Turk (MTurk) to complete a battery of self-report measures as part of a larger study (Benfer, Rogers, & Bardeen, 2020). MTurk is an online platform that provides adults with the opportunity to participate in research studies for financial compensation. Evidence suggests that MTurk participants, compared with college student samples, are more attentive and more diverse (Behrend et al., 2011; Chandler & Shapiro, 2016; Hauser & Schwarz, 2016). Participants were required to be between 18 and 64 years of age and fluent in English. Quality control questions were embedded in the study to ensure that participants were attentive (Oppenheimer et al., 2009; Paolacci et al., 2010). Participants were excluded (n = 124) from analyses if they answered fewer than two quality control questions correctly (Bardeen et al., 2016; Bardeen & Michel, 2017; Rogers et al., 2018). As part of the larger study, participants were also asked to provide text responses to open-ended questions, thus allowing investigators to identify responses that were generated by “bots” (i.e., computer programs that complete online forms automatically; Yarrish et al., 2019). Forty-eight participants were identified as bots and excluded from further analysis. The average age of the final sample (N = 827) was 37.5 years (SD = 11.2), and the majority of participants were female (53.8%). The majority of the sample identified their race as White (75.8%), followed by Black (12.8%), Asian (5.7%), “other” (3.4%), and American Indian/Alaska Native (2.2%). Additionally, 8.9% of the sample identified their ethnicity as Hispanic or Latino.

Research procedures were approved by the local institutional review board prior to recruitment. Participants were able to complete the study from any computer with internet access. Data were collected via Qualtrics, a secure online survey platform. Participants were required to provide electronic consent, and there was no penalty for withdrawing from the study. Participants were debriefed and paid $1.75 on completing a battery of self-report measures (see below). This payment amount is consistent with compensation provided in studies of similar length (Buhrmester et al., 2011; Crump et al., 2013).

Self-Report Measures

The MPFI (Rolffs et al., 2018) is a 60-item measure of psychological in/flexibility. The scale is divided into 12 subscales with six representing flexibility (i.e., Acceptance, Present Moment Awareness, Self as Context, Defusion, Values, Committed Action) and six representing inflexibility (i.e., Experiential Avoidance, Lack of Contact with the Present Moment, Self as Content, Fusion, Lack of Contact with Values, Inaction). Participants were asked rate each item (e.g., “I tried to keep perspective even when life knocked me down” and “Negative feelings often trapped me in inaction”) based on how true the item was for them in the past two weeks on a 6-point scale (1 = never true to 6 = always true). Higher scores represent higher levels of the dimension being assessed. The scales and subscales of the MPFI have exhibited adequate internal consistency and evidence of convergent (e.g., AAQ-II, Avoidance and Fusion Questionnaire) and discriminant validity (e.g., emotional intelligence, curiosity; Rolffs et al., 2018). MPFI-Flexibility (α = .97) and MPFI-Inflexibility (α = .99) demonstrated excellent internal consistency in the current study. Internal consistency was also excellent for the subscales of both the flexibility (αs from .90 to .94) and inflexibility (αs from .93 to .96) composites.

The Whiteley Index–6 (WI-6; Asmundson et al., 2008) is a brief self-report measure of health anxiety derived from the original longer version of the measure (Pilowsky, 1967). Participants rate the degree to which each of the six statements (e.g., “Do you often worry about the possibility that you have got a serious illness”) is true for them on a 5-point scale (1 = not at all to 5 = a great deal). Several studies have suggested that the six-item version of the measure resolves concerns related to the factor structure of the original measure (Veddegjærde et al., 2014; Welch et al., 2009). Additionally, the WI-6 has demonstrated high internal consistency (Welch et al., 2009), convergence with other measures of health anxiety (Fergus, 2013), and invariance across racially diverse groups (Fergus et al., 2018). Internal consistency in the current sample was excellent (α = .92).

The Patient Health Questionnaire–9 (PHQ-9; Kroenke et al., 2001) is a brief self-report measure of depression. Participants rate the degree to which they have been bothered by symptoms of depression (e.g., “feeling, down, depressed, or hopeless”) over the past two weeks on a 4-point scale (0 = not at all to 3 = nearly every day). The PHQ-9 has demonstrated strong internal consistency and good convergent validity with other measures of depression as well as criterion-related validity with measures of quality of life (Kroenke et al., 2001). Internal consistency in the current sample was adequate (α = .81).

Data Analytic Strategy

Confirmatory Factor Analysis

Four models were examined for each of the higher order constructs (i.e., flexibility and inflexibility), thus resulting in a total of eight models being tested.¹ The first model was a correlated six-factor model. No secondary loadings were modeled, but the factors were allowed to intercorrelate. The second model was a one-factor model; 30 items loaded onto one factor (either Flexibility or Inflexibility). The third model was a higher order model in which the correlations from the six-factor solution in the first model were removed and direct pathways from the higher order factors to each first-order factor were modeled. The fourth model was a bifactor model in which 30 items were simultaneously loaded onto a general factor (either Flexibility or Inflexibility) and their respective domain-specific factors. All factor covariances were fixed to zero in the bifactor model (Brown, 2015).

Model Estimation and Comparison

All models were tested using robust maximum likelihood (MLR) estimation and Mplus 8.2 (L. K. Muthén & Muthén, 2015). The Mplus syntax used to test all of these models can be found in the online supplemental materials. Four commonly recommended fit statistics were used to evaluate the fit of each model: the comparative fit index (CFI), the Tucker–Lewis index (TLI), standardized root mean square residual (SRMR), and root mean square error of approximation (RMSEA; Brown, 2015; Kline, 2016). The following guidelines were used to evaluate model fit: CFI and TLI should be near .95, SRMR should be less than .08 (Hu & Bentler, 1999), RMSEA should be near .06, and the upper limit of the 90% RMSEA confidence interval (CI) should not exceed .10 (Kline, 2016). In addition to evaluating the fit statistics for each model, models were compared with χ² difference testing. The χ² difference test was used to evaluate model comparisons (Kline, 2016). However, χ² difference tests are strongly influenced by sample size, often indicating a significant difference when actual differences are trivial in magnitude (Cheung & Rensvold, 2002). As such, RMSEA 90% CIs were also used to compare models (Brown, 2015; Kline, 2016). Specifically, differences in model fit were considered nonsignificant when models had overlapping RMSEA 90% CIs (Wang & Russell, 2005).

Bifactor Model Evaluation

The bifactor model was further evaluated using the following statistical indices (Dueber, 2016; Rodriguez et al., 2016). OmegaH (ω_H) reflects the proportion of variance in the total score that can be attributed to the general factor. Omega HS (ω_HS) reflects the proportion of variance accounted for by each subscale factor after removing the variance due to the general factor. Whereas ωH is best understood as a measure of general factor reliability, explained common variance (ECV), calculated as the proportion of common variance that is accounted for by the general factor, is a better index of the unidimensionality of a measure (Dueber, 2016). Item-level explained common variance (I-ECV) provides a measure of the proportion of common variance for each item that can be explained by the general factor. I-ECV values greater than .80 to .85 are indicative of unidimensionality at the item level (Gorsuch, 1983; Stucky & Edelen, 2015). Percentage of uncontaminated correlations (PUC) is an indicator of the percentage of item correlations contaminated by variance attributed to the general and domain-specific factors. PUC is typically interpreted in combination with ECV. When both are greater than .70, common variance within a model is considered largely unidimensional. Average relative parameter bias (ARPB) represents the bias across parameters if items are forced into a unidimensional solution. ARPB values less than 0.10 or 0.15 suggest that multidimensionality within a measure is not substantial enough to preclude a unidimensional solution (B. Muthén et al., 1987; Rodriquez et al., 2016). The factor determinacy (FD) value represents the correlation between factors and factor scores and represents the degree to which factor scores are of practical value and should be used in measurement models (i.e., FD > .90 is suggested; Gorsuch, 1983). Construct replicability (H) indicates the degree to which a factor is well defined by its indicators. H values greater than .80 suggest that a latent variable will demonstrate sufficient stability across studies (Hancock & Mueller, 2001).²

Structural Regression Model

Structural regression models were used to examine whether the domain-specific factors of the MPFI relate to clinically relevant constructs (i.e., health anxiety and depression) after accounting for the general factors (i.e., either inflexibility or flexibility). The general and domain-specific factors from each bifactor model were simultaneously regressed onto the WI-6 and PHQ-9 in separate models. Domain-specific factors from each bifactor model that demonstrated redundancy with the general factor were removed from their respective models and the models were reassessed. Model fit was assessed using the guidelines outlined above. Additional criterion measures from the larger study were not used in the present study because they had already been used in published works and studies under review (e.g., Benfer et al., 2020).

Results

Model Estimation and Comparison for MPFI-Flexibility

Means, standard deviations, and bivariate correlations are presented in Table 1. Fit statistics for all models are presented in Table 2. Of the four models that were tested, the one-factor model was the only model that did not provide adequate fit to the data for MPFI-Flexibility. Relative to the six-factor model, the one-factor model provided significantly worse fit to the data, as evidenced by a significant change in χ² (χ²[15] = 2321.233, p < .001) and nonoverlapping RMSEA 90% CIs. The higher order, correlated six-factor, and bifactor models evidenced similar fit statistics. For the higher order model, all domain-specific factors exhibited significant factor loadings on the higher order factor (i.e., MPFI-Values = .92, MPFI-Self as Context = .89, MPFI-Commitment = .88, MPFI-Defusion = .87, MPFI-Awareness = .83, MPFI-Acceptance = .53). Although a significant χ² difference test suggested significantly worse fit of the higher order model to the six-factor model (χ²[9] = 490.600, p < .001), overlapping RMSEA 90% CIs suggested that the difference in fit between the higher order and correlated six-factor models was trivial in magnitude. Similarly, while a significant χ² difference test was observed between the higher order and bifactor model, χ²(24) = 568.747, p < .001, suggesting that the bifactor model provided better fit to the data, overlapping RMSEA 90% CIs suggested that the difference in fit between these models was trivial in magnitude. See Table 3 for standardized factor loadings from the bifactor model for MPFI-Flexibility. All items exhibited significant positive loadings onto the general factor (all ps < .001). Item loadings on the domain specific factors tended to be smaller than on the general factor. All variances of the domain-specific factors were significant at p < .001, with the exceptions of MPFI-Self as Context (p = .02) and MPFI-Values (p = .001). This, in addition to the large loadings of these factors on the higher order construct for the higher order model (.89 and .92, respectively), suggested the possibility that these factors were redundant with the general factor.

Table 1.

Descriptive Statistics and Bivariate Correlations.

Factors/subscales	M	SD	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
1. MPFI Flexibility	109.10	31.26	—
2. Present Moment Awareness	18.85	5.73	.84**	—
3. Self as Context	18.16	6.28	.90**	.72**	—
4. Acceptance	15.22	5.86	.74**	.62**	.63**	—
5. Contact with Values	19.95	6.17	.88**	.68**	.73**	.50**	—
6. Committed Action	20.07	6.42	.86**	.62**	.71**	.46**	.85**	—
7. Defusion	16.84	6.36	.87**	.64**	.76**	.62**	.71**	.70**	—
8. MPFI Inflexibility	72.44	30.04	−.27**	−.17**	−.19**	−.08*	−.29**	−.30**	−.34**	—
9. Lack of Contact with the Present Moment	10.82	5.90	−.23**	−.20**	−.16**	−.06	−.26**	−.26**	−.21**	.78**	—
10. Self as Content	11.22	5.94	−.23**	−.16**	−.14**	−.10*	−.23**	−.22**	−.31**	.84**	.60**	—
11. Experiential Avoidance	16.53	6.39	.03	.01	.10**	−.06	.08*	.04	−.04	.58**	.35**	.41**	—
12. Lack of Contact with Values	10.51	5.80	−.27**	−.16**	−.19**	−.03	−.35**	−.33**	−.29**	.87**	.63**	.67**	.33**	—
13. Inaction	11.00	6.47	−.31**	−.16**	−.25**	−.06	−.36**	−.39**	−.35**	.88**	.62**	.69**	.31**	.84**	—
14. Fusion	12.36	6.72	−.29**	−.15**	−.25**	−.08*	−.28**	−.29**	−.41**	.89**	.59**	.71**	.40**	.77**	.81	—
15. WI-6	13.80	6.00	−.06	−.03	−.06	.08*	−.10**	−.09*	−.11**	.54**	.42**	.47**	.28**	.48**	.51**	.52**	—
16. PHQ-9	6.10	6.01	−.17**	−.09**	−.14**	.06	−.22**	−.22**	−.21**	.71**	.56**	.60**	.32**	.65**	.68**	.68**	.52**	—

Note. MPFI = Multidimensional Psychological Flexibility Inventory; WI-6 = Whiteley Index–6; PHQ-9 = Patient Health Questionnaire–9.

p < .05, ^**p < .01, ^***p < .001.

Table 2.

Goodness of Fit Statistics for Tested Models.

Model	χ²	df	RMSEA	RMSEA 90% CI		CFI	TLI	SRMR
Model	χ²	df	RMSEA	LL	UL	CFI	TLI	SRMR
MPFI flexibility composite
One factor	3792.194***	405	.101	.098	.104	.745	.726	.080
Correlated six factor	812.564***	390	.036	.033	.040	.968	.965	.038
Higher order	1003.547***	399	.043	.040	.046	.955	.950	.054
Bifactor	861.774***	375	.040	.036	.043	.963	.958	.043
MPFI inflexibility composite
One factor	4502.053***	405	.111	.108	.114	.701	.679	.103
Correlated six factor	749.609***	390	.033	.030	.037	.974	.971	.027
Higher order	806.266***	399	.035	.032	.039	.970	.968	.038
Bifactor	720.220***	375	.033	.030	.037	.975	.971	.032

Note. MPFI = Multidimensional Psychological Flexibility Inventory; RMSEA = root mean square error of approximation; CFI = comparative fit index; TLI = Tucker–Lewis index; SRMR = standardized root mean square residual.

p < .05. ^**p < .01, ^***p < .001.

Table 3.

Standardized Factor Loadings for the Bifactor Model for MPFI Flexibility Composite.

Factors
Item	General	Acceptance	Present Moment Awareness	Self as Context	Defusion	Values	Committed Action
1	.496	.711
2	.554	.649
3	.440	.556
4	.619	.524
5	.658	.430
6	.689		.532
7	.667		.475
8	.706		.403
9	.693		.484
10	.734		.291
11	754			.446
12	.721			.488
13	.770			.281
14	.770			.363
15	.778			.319
16	.728				.423
17	.736				.485
18	.753				.418
19	.749				.409
20	.763				.393
21	.769					.292
22	.784					.436
23	.757					.375
24	.791					.362
25	.733					.289
26	.754						.437
27	.754						.482
28	.777						.403
29	.775						.417
30	.756						.400

Note. MPFI = Multidimensional Psychological Flexibility Inventory.

All factor loading were significant at p < .001.

Model Estimation and Comparison for MPFI-Inflexibility

Consistent with above, the one-factor model was the only model that did not provide adequate fit to the data for MPFI-Inflexibility. Relative to the six-factor model, the one-factor model provided significantly worse fit to the data, as evidenced by a significant change in χ² (χ²[15] = 2673.686, p < .001) and nonoverlapping RMSEA 90% CIs. The higher order, correlated six-factor, and bifactor models evidenced similar fit statistics. For the higher order model, all domain-specific factors exhibited significant factor loadings on the higher order factor (i.e., MPFI-Values/Lack = .94, MPFI-Inaction = .94, MPFI-Fusion = .92, MPFI-Self as Content = .85, MPFI-Present Moment/Lack = .76, MPFI-Experiential Avoidance = .44). Although a significant χ² difference test suggested significantly worse fit of the higher order model relative to the six-factor model, χ²(9) = 121.581, p < .001, overlapping RMSEA 90% CIs suggested that the difference in fit between these models was trivial in magnitude. Similarly, while a significant χ² difference test was observed between the higher order and bifactor model, χ²(24) = 135.608, p < .001, suggesting that the bifactor model provided better fit to the data, overlapping RMSEA 90% CIs suggested that the difference in fit between these models was trivial in magnitude. See Table 4 for standardized factor loadings from the bifactor model for MPFI-Inflexibility. All items exhibited significant positive loadings onto the general factor (all ps < .001). Item loadings on the domain-specific factors tended to be smaller than on the general factor, except in the case of the MPFI-Experiential Avoidance factor which exhibited larger loadings on the domain-specific factor. All variances of the domain-specific factors were significant at p < .001, with the exception of MPFI-Values/Lack (p = .02). This, in addition to the large loading of this factor on the higher order construct for the higher order model (.94), suggested the possibility that this factor was redundant with the general factor.

Table 4.

Standardized Factor Loadings for the Bifactor Model for MPFI Inflexibility Composite.

Factors
Item	General	Experiential Avoidance	Lack of Contact with the Present Moment	Self as Content	Fusion	Lack of Contact with Values	Inaction
1	.396	.741
2	.342	.829
3	.381	.752
4	.369	.784
5	.405	.732
6	.649		.552
7	.686		.605
8	.705		.555
9	.634		.616
10	.708		.510
11	.738			.354
12	.739			.414
13	.716			.554
14	.723			.550
15	.760			.381
16	.788				.320
17	.821				.336
18	.830				.354
19	.824				.355
20	.771				.445
21	.840					.261
22	.830					.357
23	.834					.340
24	.805					.235
25	.800					.222
26	.852						.330
27	.849						.323
28	.837						.403
29	.868						.287
30	.865						.254

Note. MPFI = Multidimensional Psychological Flexibility Inventory.

All factor loading were significant at p < .001.

Bifactor Model Evaluation for MPFI-Flexibility

Additional indices derived from the bifactor model are presented in Table 5. Results suggested acceptable reliability for the MPFI-Flexibility general (ω = .98) and domain-specific factors (ωs from .90 to .98). The majority of the reliable variance in MPFI-Flexibility was attributable to the general factor (ωH = .93). In contrast, the domain-specific factors accounted for substantially less variance in MPFI-Flexibility (ωHS scores from .16 to .47). The ECV of .73 indicates that the general factor accounts for 73% of common variance and 27% of common variance is spread across the domain-specific factors (i.e., Acceptance = 7.9%, Present Moment Awareness = 4.6%, Defusion = 4.3%, Committed Action = 4.3%, Self as Context = 3.5%, Values = 2.9%). Acceptance is the only domain-specific factor for which the majority of item-level variance was accounted for by this specific factor (ECV = .52). For the five-remaining domain-specific factors, the overwhelming majority of variance in the items associated with each domain-specific factor was accounted for by the general factor (ECV values from .18 to .29). Eight of 30 items exhibited I-ECV values greater than .80, which suggests that these items contribute more to the general factor than to their respective domain-specific factor (Gorsuch, 1983; Stucky & Edelen, 2015). Additionally, 14 of the remaining 22 items exhibited I-ECV values greater than .70, further calling into question the multidimensional nature of this measure. The PUC value for MPFI-Flexibility was .86. When interpreted in combination with the ECV value of .73 for the general factor, this indicates low relative bias and increased likelihood of unidimensionality. Model ARPB was extremely low (.02), which further supports a unidimensional solution (Rodriguez et al., 2016). The FD values for the general factor and the Acceptance factor (.97 and .90, respectively) suggest adequate FD. The FD values of the five remaining domain specific factors (.76 to 83) suggest that these factors may not be suitable for use as summed subscale scores or as latent variables in an structural equation modeling framework (Gorsuch, 1983). The general factor exhibited acceptable construct replicability (H = .97), while the construct replicability of the domain-specific factors was inadequate (Hs from .42 to .74).

Table 5.

Additional Bifactor Indices for the MPFI Inflexibility and Flexibility Composites.

	ω/ωS	ω_H/ω_HS	ECV	H	FD
MPFI Flexibility Composite
General Factor	.983	.925	.725	.972	.967
Acceptance	.901	.467	.520	.737	.899
Present Moment Awareness	.915	.258	.289	.561	.828
Self as Context	.929	.186	.207	.477	.787
Defusion	.934	.229	.247	.528	.814
Values	.926	.160	.176	.423	.756
Committed Action	.943	.225	.240	.531	.820
MPFI Inflexibility Composite
General Factor	.986	.915	.689	.978	.974
Experiential Avoidance	.932	.750	.804	.882	.954
Lack of Contact with the Present Moment	.947	.391	.414	.708	.911
Self as Content	.937	.256	.280	.583	.868
Fusion	.948	.159	.169	.437	.788
Lack of Contact with Values	.940	.100	.110	.315	.706
Inaction	.962	.118	.125	.372	.756

Note. ω/ωS = Omega/OmegaS; ω_H/ω_HS = OmegaH/OmegaHS; ECV = explained common variance; H = construct replicability; FD = factor determinacy.

Bifactor Model Evaluation for MPFI-Inflexibility

As seen in Table 5, the reliability of the MPFI-Inflexibility general (ω = .98) and domain-specific factors (ωs from .93 to .96) was acceptable. The majority of the reliable variance in MPFI-Inflexibility scores was attributable to the general factor (ωH = .92). In contrast, the domain-specific factors, with the exception of the Experiential Avoidance factor (ωHS = .75), accounted for substantially less variance in MPFI-Inflexibility (ωHS scores from .10 to .39). The ECV of .69 indicated that the general factor accounted for 69% of common variance and 31% of common variance was spread across the domain-specific factors (i.e., Experiential Avoidance = 12.7%, Lack of Contact with Present Moment = 7.0%, Self as Content = 4.5%, Fusion = 2.9%, Inaction = 2.2%, Lack of Contact with Values = 1.8%). Experiential Avoidance was the only domain-specific factor for which the majority of item-level variance was accounted for by this specific factor (ECV = .80). For the five-remaining domain-specific factors, the majority of variance in the items associated with each domain-specific factor was accounted for by the general factor (ECV values from .11 to .41). Sixteen of 30 items exhibited I-ECV values greater than .80, which suggests that these items contributed more to the general factor than to their respective domain-specific factor (Gorsuch, 1983; Stucky & Edelen, 2015). All of the items from Fusion, Lack of Contact with Values, and Inaction had I-ECV values greater than 80. The PUC value for MPFI-Inflexibility was .86. When interpreted in combination with the ECV value of .69 for the general factor, this indicates low relative bias and increased likelihood of unidimensionality. Model ARPB was extremely low (.03), which further supported a unidimensional solution (Rodriguez et al., 2016). The FD values for the general factor, Experiential Avoidance factor, and Lack of Contact with the Present Moment Factor (.97, .95, and .91, respectively) suggested adequate FD. The FD values of the four-remaining domain-specific factors (.76 to 87) suggested that these factors may not be suitable for use as summed subscale scores or as latent variables in a structural equation modeling framework (Gorsuch, 1983). The H values for the general factor and Experiential Avoidance factor (.98 and .88, respectively) suggested acceptable construct replicability. The construct replicability of the five-remaining domain-specific factors was inadequate (Hs from .37 to .71).

Structural Regression Models for MPFI-Flexibility

The structural regression model with health anxiety as the criterion variable provided adequate fit to the data, with all fit indices falling within the specified guidelines: χ²(557) = 1254.08, p < .001; RMSEA = .04 (90% CIs [.036, .042]); CFI = .96; TLI = .95; SRMR = .04. The general factor significantly predicted the WI-6 in the expected direction (β = −.16, p = .005). After accounting for the general factor, the Acceptance domain-specific factor also accounted for a significant portion of the variance, though not in the theoretically expected direction (β = .26, p < .001). In the initial iteration of the model, none of the other domain-specific factors accounted for additional significant variance (Present Moment Awareness: β = .16, p = .06, Self as Context: β = .14, p = .08, Defusion: β = .01, p = .93, Values: β = .10, p = .34, Committed Action: β = .14, p = .12).

A second iteration of the model was tested in which all redundant MPFI domain-specific factors were removed. In this version of the model, the general factor was a significant predictor of WI-6 scores in the expected direction (β = −.11, p < .01). After accounting for the general factor, the Acceptance domain-specific factor accounted for a significant amount of the variance in WI-6 scores, though the relationship was in a theoretically unexpected direction (β = .21, p <.001).

An additional structural regression was examined using depression as the criterion variable. This model provided adequate fit to the data, with all fit indices falling within the specified guidelines: χ²(628) = 1455.98, p < .001; RMSEA = .04 (90% CIs [.037, .043]); CFI = .95; TLI = .95; SRMR = .05. The general factor significantly predicted the PHQ-9 in the expected direction (β = −.27, p < .001). After accounting for the general factor, the Acceptance domain-specific factor accounted for a significant portion of the variance, though not in the theoretically expected direction (β = .30, p < .001). In the initial iteration of the model, none of the other domain-specific factors accounted for additional significant variance (Present Moment Awareness: β = .15, p = .13, Self as Context: β = .12, p = .27, Defusion: β = −.07, p = .53, Values: β = .06, p = .64, Committed Action: β = .02, p = .82).

A second iteration of the model was tested in which all redundant MPFI domain-specific factors were removed. In this version of the model, the general factor was a significant predictor of PHQ-9 scores in the expected direction (β = −.24, p < .001). After accounting for the general factor, the Acceptance domain-specific factor accounted for a significant amount of the variance in PHQ-9 scores, though the relationship was in a theoretically unexpected direction (β = .29, p <.001).

Based on the outcomes of these models, the Acceptance domain-specific factor may be redundant with the general factor in the Flexibility model. It should also be noted that while the Acceptance domain-specific factor was superior to the other subscales in the Flexibility composite, the bifactor indices generally did not provide strong support for the value of the subscale. For example, approximately half of the variance in the Acceptance domain-specific items were explained by the general factor. Furthermore, although the Acceptance domain-specific factor appears to be quite reliable (ω = .90), after accounting for variability related to the general factor, it only accounted for a smaller proportion of the reliable variance (ωH = .47) and construct replicability was inadequate.

Structural Regression Model for MPFI-Inflexibility

The structural regression model with health anxiety as the criterion variable provided adequate fit to the data, with all fit indices falling within the specified guidelines: χ²(557) = 1102.96, p < .001; RMSEA = .03 (90% CIs [.031, .037]); CFI = .97; TLI = .96; SRMR = .03. The general factor significantly predicted the WI-6 in the expected direction (β = .56, p < .001). After accounting for the general factor, none of the domain-specific factors accounted for a significant amount of variance in WI-6 scores (Experiential Avoidance: β = .03, p = .33, Lack of Contact with the Present Moment: β = .05, p = .37, Self as Content: β = .06, p = .41, Fusion: β = .11, p = .19, Lack of Contact with Values: β = .01, p = .91, Inaction: β = .08, p = .38).

An additional structural regression was examined using depression as the criterion variable. This model also provided adequate fit to the data, with all fit indices falling within the specified guidelines: χ²(628) = 1102.96, p < .001; RMSEA = .04 (90% CIs [.033, .039]); CFI = .96; TLI = .96; SRMR = .04. The general factor significantly predicted the PHQ-9 in the expected direction (β = .77, p < .001). After accounting for the general factor, none of the domain-specific factors accounted for a significant amount of variance in PHQ-9 scores (Experiential Avoidance: β = −.01, p = .71, Lack of Contact with the Present Moment: β = .02, p = .76, Self as Content: β = −.01, p = .84, Fusion: β = .07, p = .41, Lack of Contact with Values: β = −.06, p = .61, Inaction: β = .04, p = .67).

Discussion

The factor structure of the MPFI was examined in a large sample of community adults in the present study. Consistent with previous research, higher order models of MPFI-Flexibility and MPFI-Inflexibility demonstrated adequate fit to the data, while one-factor models did not (Rolffs et al., 2018). Additionally, this study is the first to test correlated six-factor and bifactor models of these two constructs. For both MPFI-Flexibility and MPFI-Inflexibility, the correlated six-factor model and the bifactor model demonstrated adequate fit to the data. While χ² difference testing suggested that both bifactor models provided better fit to the data than their corresponding higher order models, examination of RMSEA 90% CIs suggested that differences in fit were trivial. Importantly, examination of follow-up statistical indices from bifactor analysis allowed us to simultaneously test two assumptions that are central to measures for which total and subscale scores are calculated: (1) domain-specific factors represents the same overarching construct (either flexibility or inflexibility), and (2) the domain-specific factors provide unique information beyond the general factors (Dueber, 2016; Rodriguez et al., 2016).

Further examination of the bifactor model for MPFI-Flexibility suggests mixed evidence of multidimensionality. Generally, statistical indices suggested that the general factor was the most stable factor and accounted for the greatest proportion of variance. The pattern of results was similar for the inflexibility composite, with the majority of statistical indices suggesting unidimensionality. Both Acceptance and Experiential Avoidance (from the MPFI-Flexibility and MPFI-Inflexibility, respectively) exhibited some value as domain-specific factors, though the evidence was mixed. Across both general factors (inflexibility and flexibility), none of the domain-specific factors accounted for more than 20% of the unique variance beyond their respective general factors. The Acceptance and Experiential Avoidance domains did represent the largest portion of unique variance beyond the general factors, accounting for approximately 8% and 13%, respectively. Additionally, both of these domain-specific factors exhibited adequate FD, and the Experiential Avoidance factor met the specified cut off for construct replicability. However, for both bifactor models, the magnitude of factor loadings was larger for the total score than the domain-specific scores, suggesting the possibility that there may be considerable content redundancy between the domain-specific and general factors. Taken together, our results suggest that the subscale scores for both of these measures may not have significant value beyond their respective general factors, with the possible exception of the Acceptance and Experiential Avoidance subscales. It is also important to highlight that almost all of the domain-specific factors exhibited inadequate construct replicability and FD, which further calls into question the use of scores from domain-specific subscales.

It is important to note additional research will likely clarify the potential value of the MPFI subscales. While the Experiential Avoidance subscale provided the most convincing evidence of unique value, it is possible that other domain-specific factors may also provide unique predictive power when considered with other more diverse criterion variables. As the evidence currently stands, it seems inappropriate to utilize the MPFI subscales, and it is notable that the use of only total composite scores would yield a much blunter instrument than was intended by the authors of the measure. Furthermore, the use of the measure in this manner does not align with the theoretical basis of ACT and psychological flexibility as was originally intended. Additionally, these findings are in notable contrast to the existing literature which includes numerous studies utilizing measures of the psychological in/flexibility domain-specific factors (e.g., cognitive fusion as measured by the Cognitive Fusion Questionnaire) that have demonstrated meaningful relationships with related, but distinct forms of psychopathology such as anxiety, depression, and trauma-related disorders (e.g., Kelly et al., 2019; Kelso et al., 2020; Moroz & Dunkley, 2019; Thomas & Bardeen, 2020). The pattern of results from this study and the existing literature suggests that further refinement of the measurement of these constructs and the constructs themselves are worthy of further study.

The MPFI is a recently developed self-report measure that, according to its authors, aims to capture each of the domain-specific factors represented in the Hexaflex model (Rolffs et al., 2018). While it has not been used extensively in research, some studies have started to use the measure to capture the components of psychological flexibility (Rogge et al., 2019; Stabbe et al., 2019). Based on their study findings, the authors of these studies advocate for using the domain-specific factors of the flexibility and inflexibility factors to monitor treatment progress and make treatment decisions (Rogge et al., 2019; Stabbe et al., 2019). While the results of this study do indicate that the total scores of the flexibility and inflexibility scales are suitable for measuring these constructs, their respective subscales do not appear to have the same support. In sum, the results of this study suggest that it is most appropriate to interpret the MPFI based on composite rather than subscale scores.

It is also important to highlight that the results of this study provide some evidence that the Acceptance and Experiential Avoidance domains of the Flexibility and Inflexibility composites, respectively, may adequately capture distinct domains. This finding is particularly notable given recent research suggesting that existing measures of experiential avoidance may not make an adequate distinction between that construct and general distress (Wolgast, 2014; Rochefort et al., 2018). Further examination of how the MPFI total and subscale scores differentially relate to relevant outcomes could serve to provide further evidence of psychological flexibility and inflexibility as distinct. Additionally, a number of measures of the proposed In/Hexaflex components have been developed in recent years, which would allow for direct evaluation of the degree to which the MPFI provides adequate coverage of these constructs. Finally, additional research evaluating how the MPFI relates to clinical outcomes will be important in understanding the predictive value of psychological flexibility and its proposed components and counterparts, with this study providing an initial step toward this aim.

The predictive utility of the MPFI with regard to two relevant clinical constructs, health anxiety and depression, was evaluated. Results from a structural regression with Inflexibility predicting health anxiety showed that the general factor significantly predicted this clinical outcome. However, no domain-specific factor evidenced incremental utility in predicting health anxiety beyond this general factor. Results were similar with regard to the depression outcome. Taken together, these results suggest that domain-specific factors of the inflexibility factor may have relatively little unique value in relation to these specific outcomes. These results align with the results of the examination of the bifactor model and suggest that it may be inadvisable to model the domain-specific factors for the Inflexibility scale. However, the limited number of outcomes utilized in this investigation limits our ability to draw conclusions about the potential value of the domain-specific factors to other potentially relevant criterion variables.

Results of a structural regression with the MPFI Flexibility scale were less straight forward. Like the Inflexibility scale, the Flexibility general factor significantly predicted health anxiety, indicating that psychological flexibility does share a significant, negative relationship with this construct, as theory would suggest. After accounting for the Flexibility factor, the Acceptance factor of the MPFI also significantly predicted health anxiety, though the relationship was in a theoretically unexpected direction. Results from a second iteration of the model including only the general factor and the Acceptance subfactor, the two most promising scales, suggested that the Acceptance scale may also be redundant with the general factor. Additionally, this pattern of results suggests that the MPFI Flexibility composite and its respective domain-specific scales may have less predictive value overall in predicting clinical outcomes. Several studies have attempted to examine this hypothesis more closely by considering how different combinations of flexibility and inflexibility may differentially predict relevant outcomes. Stabbe et al. (2019) reported that even when individuals are high in both flexibility and inflexibility, as measured by the MPFI, inflexibility is a more powerful predictor of clinical outcomes. While flexibility may not predict negative mental health outcomes as well as inflexibility, it seems as though flexibility remains an important predictor of overall well-being and quality of life (Momeniarbat et al., 2017; Stabbe et al., 2019). Given the more general effects of higher flexibility, it also seems plausible that flexibility may buffer the negative effects of inflexibility, though this hypothesis has not been directly evaluated.

When considered in the context of additional research, the results of this study add to the understanding of how psychological flexibility is measured and how it may relate to clinical outcomes. Traditionally, measurement of ACT-related constructs has relied on the assumption that psychological flexibility and inflexibility are polar opposites, such that a measure of inflexibility can be used to extrapolate information about flexibility and vice versa. The approach used to construct the MPFI is in contrast to this approach in that flexibility and inflexibility scores are allowed to fluctuate independently. This independence allows for independent evaluation of these constructs, and, as some researchers have previously proposed, the two constructs do not appear to be polar opposites of a single construct (Kashdan & Rottenberg, 2010; McAndrews et al., 2019). Furthermore, the results of this study appear to suggest that psychological flexibility may have less value as a predictor of psychopathology. It is important to note that this study only considered two criterion variables and further research would be necessary to clarify the relationship between psychological flexibility and other pathological outcomes. It also seems equally important to examine the relationship of psychological flexibility with constructs related to psychological health, such as quality of life.

This study adds value to the existing literature in that it provides the first know bifactor analysis of the MPFI and also considers the fit of other relevant models, though it is important to recognize study limitations. The data for this study were collected via Amazon’s MTurk. While research suggests that MTurk is capable of producing high quality data (Chandler & Shapiro, 2016), MTurk samples may not be fully representative of the general population due to the tendency for these samples to be more highly educated and younger than the general population (Paolacci & Chandler, 2014). As such, replicating this study in a more representative sample will be important for ensuring generalizability. Additionally, a priori power analysis was not entirely feasible because the MPFI was a relatively new measure at the time that this study was conducted and there was a lack of existing studies examining the relationship between the domain-specific factors and criterion variables. However, even though sample size recommendations for confirmatory factor analysis vary greatly (Koran, 2016; MacCallum et al., 1999), MacCallum et al. (1996) suggest that the sample size used in the present study is substantially larger than that which is needed to assure power of at least 0.80 for rejecting the hypothesis of close fit if ε_a = 0.08 (see Table 4 in MacCallum et al., 1996).

It is also important to note that the sample for this study was unselected for pathology. Although evidence suggests dimensional, rather than categorical (presence vs. absence), conceptualizations of health anxiety and depression, and there was considerable variability in these symptoms in the present sample (e.g., 30% of the sample reported clinically significant health anxiety; Fergus et al., 2018), replication of the results of this study in a clinical population, with a thorough assessment of diagnostic history, may be important for ensuring generalizability of findings to those with more severe psychopathology.

To our knowledge, this study is the first to evaluate the factor structure of the MPFI since the original validation study (Rolffs et al., 2018). The results of this study provide clear evidence of strong general factors for each overarching domain (flexibility and inflexibility). Examination of a bifactor model of each of these domains suggests that the majority of the variance in MPFI scores is accounted for by the general factors. In contrast, the domain-specific factors accounted for relatively little variance in MPFI scores. Additionally, the majority of the domain-specific factors did not exhibit FD and construct replicability and appear to be largely redundant with their respective general factors. This pattern of results suggests that while continued use of the total scores is warranted, the domain-specific factors may be of little practical value for use as subscale scores and in measurement models. However, the Acceptance and Experiential Avoidance subscales may be exceptions to this larger pattern of results, as both exhibited some evidence of providing information above and beyond their respective general factors. Overall, these results suggest that the MPFI may require further refinement to clearly capture the proposed domain-specific constructs.

Supplemental Material

sj-pdf-1-asm-10.1177_10731911211024353 – Supplemental material for An Examination of the Factor Structure of the Multidimensional Psychological Flexibility Inventory

Supplemental material, sj-pdf-1-asm-10.1177_10731911211024353 for An Examination of the Factor Structure of the Multidimensional Psychological Flexibility Inventory by Kelsey N. Thomas, Joseph R. Bardeen, Tracy K. Witte, Travis A. Rogers, Natasha Benfer and Kate Clauss in Assessment

Footnotes

Author Note

The data set used in this study may be obtained from the corresponding author on reasonable request.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Joseph R. Bardeen

Natasha Benfer

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Arch

Eifert

G. H.

Davies

Plumb Vilardaga

J. C.

Rose

R. D.

Craske

M. G.

(2012). Randomized clinical trial of cognitive behavioral therapy (CBT) versus Acceptance and Commitment Therapy (ACT) for mixed anxiety disorders. Journal of Consulting and Clinical Psychology, 80(5), 760-765. https://doi.org/10.1037/a0028310

Asmundson

G. J. G.

Carleton

R. N.

Bovell

C. V.

Taylor

(2008). Comparison of unitary and multidimensional models of the Whiteley Index in a nonclinical sample: Implications for understanding and assessing health anxiety. Journal of Cognitive Psychotherapy: An International Quarterly, 22(2), 87-96. https://doi.org/10.1891/0889-8391.22.2.87

Bai

Luo

Zhang

Chi

(2020). Acceptance and Commitment Therapy (ACT) to reduce depression: A systematic review and meta-analysis. Journal of Affective Disorders, 260(January), 728-737. https://doi.org/10.1016/j.jad.2019.09.040

Bardeen

J. R.

Fergus

T. A.

Hannan

S. M.

Orcutt

H. K.

(2016). Addressing psychometric limitations of the Difficulties in Emotion Regulation Scale through item modification. Journal of Personality Assessment, 98(3), 298-309. https://doi.org/10.1080/00223891.2015.1091774

Bardeen

J. R.

Michel

J. S.

(2017). The buffering effect of religiosity on the relationship between intolerance of uncertainty and depressive symptoms. Psychology of Religion and Spirituality, 9(Suppl. 1), 90-95. https://doi.org/10.1037/rel0000123

Behrend

T. S.

Sharek

D. J.

Meade

A. W.

Wiebe

E. N.

(2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), Article 800. https://doi.org/10.3758/s13428-011-0081-0

Benfer

Rogers

T. A.

Bardeen

J. R.

(2020). Cognitive fusion potentiates the effect of maladaptive posttraumatic cognitions on posttraumatic stress symptoms. Journal of Contextual Behavioral Science, 17(July), 55-61. https://doi.org/10.1016/j.jcbs.2020.06.003

Bond

F. W.

Hayes

S. C.

Baer

R. A.

Carpenter

K. M.

Guenole

Orcutt

H. K.

Waltz

Zettle

R. D.

(2011). Preliminary psychometric properties of the Acceptance and Action Questionnaire–II: A revised measure of psychological inflexibility and experiential avoidance. Behavior Therapy, 42(4), 676-688. https://doi.org/10.1016/j.beth.2011.03.007

Bonifay

Lane

S. P.

Reise

S. P.

(2017). Three concerns with applying a bifactor model as a structure of psychopathology. Clinical Psychological Science, 5(1), 184-186. https://doi.org/10.1177/2167702616657069

10.

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Press.

11.

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6(1), 3-5. https://doi.org/10.1177/1745691610393980

12.

Chandler

Shapiro

(2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53-81. https://doi.org/10.1146/annurev-clinpsy-021815-093623

13.

Chen

F. F.

West

S. G.

Sousa

K. H.

(2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41(2), 189-225. https://doi.org/10.1207/s15327906mbr4102_5

14.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233-255. https://doi.org/10.1207/S15328007SEM0902_5

15.

Crump

M. J.

McDonnell

J. V.

Gureckis

T. M.

(2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLOS ONE, 8, e57410. https://doi.org/10.1371/journal.pone.0057410

16.

Dueber

D. M.

(2016). Bifactor Indices Calculator: A Microsoft Excel-based tool to calculate various indices relevant to bifactor CFA models. https://uknowledge.uky.edu/edp_tools/1/

17.

Fergus

T. A.

(2013). Repetitive thought and health anxiety: Tests of specificity. Journal of Psychopathology and Behavioral Assessment, 35(3), 366-374. https://doi.org/10.1007/s10862-013-9340-y

18.

Fergus

T. A.

(2015). I really believe I suffer from a health problem: Examining an association between cognitive fusion and healthy anxiety. Journal of Clinical Psychology, 71(9), 920-934. https://doi.org/10.1002/jclp.22194

19.

Fergus

T. A.

Kelley

L. P.

Griggs

J. O.

(2018). The Whiteley Index–6: An examination of measurement invariance among self-identifying Black, Latino, and White respondents in primary care. Assessment, 25(2), 247-258. https://doi.org/10.1177/1073191116645908

20.

Fergus

T. A.

Valentiner

D. P.

Gillen

M. J.

Hiraoka

Twohig

M. P.

Abramowitz

J. S.

McGrath

P. B.

(2012). Assessing psychological inflexibility: The psychometric properties of the Avoidance and Fusion Questionnaire for Youth in two adult sample. Psychological Assessment, 24(2), 402-408. https://doi.org/10.1037/a0025776

21.

Fonseca

Trindade

I. A.

Mendes

A. L.

Ferreira

(2020). The buffer role of psychological flexibility against the impact of major life events on depression symptoms. Clinical Psychologist, 24(1), 82-90. https://doi.org/10.1111/cp.12194

22.

Gámez

Chmielewski

Kotov

Ruggero

Suzuki

Watson

(2014). The Brief Experiential Avoidance Questionnaire: Development and initial validation. Psychological Assessment, 26(1), 35-45. https://doi.org/10.1037/a0034473

23.

Gámez

Chmielewski

Kotov

Ruggero

Watson

(2011). Development of a measure of experiential avoidance: The Multidimensional Experiential Avoidance Questionnaire. Psychological Assessment, 23(3), 692-713. https://doi.org/10.1037/a0023242

24.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Lawrence Erlbaum.

25.

Hancock

G. R.

Mueller

R. O.

(2001). Rethinking construct reliability within latent variable systems. In Cudeck

Du Toit

Sorbom

(Eds.), Structural equation modeling: Present and future (pp. 195-216). Scientific Software International.

26.

Hauser

D. J.

Schwarz

(2016). Attentive turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400-407. https://doi.org/10.3758/s13428-015-0578-z

27.

Hayes

S. C.

Luoma

J. B.

Bond

F. W.

Masuda

Lillis

(2006). Acceptance and commitment therapy: Model, processes, and outcomes. Behavior Research and Therapy, 44(1), 1-25. https://doi.org/10.1016/j.brat.2005.06.006

28.

Hayes

S. C.

Strosahl

K. D.

Wilson

K. G.

(1999). Acceptance and commitment therapy: An experiential approach to behavior change. Guilford Press.

29.

Hayes

S. C.

Strosahl

K. D.

Wilson

K. G.

(2011). Acceptance and commitment therapy: The process and practice of mindful change. Guilford Press.

30.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

31.

Kashdan

T. B.

Rottenberg

(2010). Psychological flexibility as a fundamental aspect of health. Clinical Psychology Review, 30(7), 865-878. https://doi.org/10.1016/j.cpr.2010.03.001

32.

Kelly

M. M.

DeBeer

B. B.

Meyer

E. C.

Kimbrel

N. A.

Gulliver

S. B.

Morissette

S. B.

(2019). Experiential avoidance as a mediator of the association between posttraumatic stress disorder symptoms and social support: A longitudinal analysis. Psychological Trauma: Theory, Research, Practice, and Policy, 11(3), 353-359. https://doi.org/10.1037/tra0000375

33.

Kelso

K. C.

Kashdan

T. B.

Imamoğlu

Ashraf

(2020). Meaning in life buffers the impact of experiential avoidance on anxiety. Journal of Contextual Behavioral Science, 16(April), 192-198. https://doi.org/10.1016/j.jcbs.2020.04.009

34.

Kline

R. B.

(2016). Principles and practice of structural equation modeling (4th ed.). Guilford Press.

35.

Koran

(2016). Preliminary proactive sample size determination for confirmatory factor analysis models. Measurement and Evaluation in Counseling and Development, 49(4), 296-308. https://doi.org/10.1177/0748175616664012

36.

Kroenke

Spitzer

R. L.

Williams

J. B. W.

(2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606-613.

37.

Landi

Pakenham

K. I.

Boccolini

Grandi

Tossani

(2020). Health anxiety and mental health outcome during COVID-19 lockdown in Italy: The mediating and moderating roles of psychological flexibility. Frontiers in Psychology, 11, Article 2195. https://doi.org/10.3389/fpsyg.2020.02195

38.

Leonidou

Panayiotou

Bati

Karekla

(2019). Coping with psychosomatic symptoms: The buffering role of psychological flexibility and impact on quality of life. Journal of Health Psychology, 24(2), 175–187. https://doi.org/10.1177/1359105316666657

39.

MacCallum

R. C.

Browne

M. W.

Sugawara

H. M.

(1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130-149. https://doi.org/10.1037/1082-989X.1.2.130

40.

MacCallum

R. C.

Widaman

K. F.

Zhang

Hong

(1999). Sample size in factor analysis. Psychological Methods, 4(1), 84-99. https://doi.org/10.1037/1082-989X.4.1.84

41.

McAndrews

Richardson

Stopa

(2019). Psychometric properties of acceptance measures: A systematic review. Journal of Contextual Behavioral Science, 12(April), 261-277. https://doi.org/10.1016/j.jcbs.2018.08.006

42.

Momeniarbat

Karimi

Erfani

Kiani

(2017). The role of neuroticism and psychological flexibility in chronic fatigue and quality of life in patients with type 2 diabetes. Romanian Journal of Diabetes Nutrition and Metabolic Diseases, 24(2), 137-148. https://doi.org/10.1515/rjdnmd-2017-0018

43.

Moroz

Dunkley

D. M.

(2019). Self-critical perfectionism, experiential avoidance, and depressive and anxious symptoms over two years: A three-wave longitudinal study. Behaviour Research and Therapy, 112(January), 18-27. https://doi.org/10.1016/j.brat.2018.11.006

44.

Muthén

Kaplan

Hollis

(1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52(3), 431-462. https://doi.org/10.1007/BF02294365

45.

Muthén

L. K.

Muthén

B. O.

(2015). Mplus (Version 7.4) [Computer software]. Muthén & Muthén.

46.

Oppenheimer

D. M.

Meyvis

Davidenko

(2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872. https://doi.org/10.1016/j.jesp.2009.03.009

47.

Paolacci

Chandler

(2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184-188. https://doi.org/10.1177/0963721414531598

48.

Paolacci

Chandler

Ipeirotis

P. G.

(2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411-419. http://sjdm.org/~baron/journal/10/10630a/jdm10630a.pdf

49.

Pilowsky

(1967). Dimensions of hypochondriasis. British Journal of Psychiatry, 113(494), 89-93. https://doi.org/10.1192/bjp.113.494.89

50.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667-696. https://doi.org/10.1080/00273171.2012.715555

51.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129-140. https://doi.org/10.1080/00223891.2012.725437

52.

Reise

S. P.

Kim

D. S.

Mansolf

Widaman

K. F.

(2016). Is the bifactor model a better model or is it just better at modeling implausible responses? Application of iteratively reweighted least squares to the Rosenberg Self-Esteem Scale. Multivariate Behavioral Research, 51(6), 818-838. https://doi.org/10.1080/00273171.2016.1243461

53.

Reise

S. P.

Moore

T. M.

Haviland

M. G.

(2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield unequivocal scores. Journal of Personality Assessment, 92(6), 544-559. https://doi.org/10.1080/00223891.2010.496477

54.

Rochefort

Baldwin

A. S.

Chmielewski

(2018). Experiential avoidance: An examination of the construct validity of the AAQ-II and the MEAQ. Behavior Therapy, 49(3), 435-449. https://doi.org/10.1016/j.beth.2017.08.008

55.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137-150. https://doi.org/10.1037/met0000045

56.

Rogers

T. A.

Bardeen

J. A.

Fergus

T. A.

Benfer

(2018). Factor structure and incremental utility of the Distress Tolerance Scale: A bifactor analysis. Assessment, 27(2), 297-308. https://doi.org/10.1177/1073191118789496

57.

Rogge

R. D.

Daks

J. S.

Dubler

B. A.

Saint

K. J.

(2019). It’s all about the process: Examining the convergent validity, conceptual coverage, unique predictive validity, and clinical utility of ACT process measures. Journal of Contextual Behavioral Science, 14(October), 90-102. https://doi.org/10.1016/j.jcbs.2019.10.001

58.

Rolffs

J. L.

Rogge

R. D.

Wilson

K. G.

(2018). Disentangling components of flexibility via the hexaflex model: Development and validation of the Multidimensional Psychological Flexibility Inventory (MPFI). Assessment, 25(4), 458-482. https://doi.org/10.1177/1073191116645905

59.

Stabbe

O. K.

Rolffs

J. L.

Rogge

R. D.

(2019). Flexibly and/or inflexibly embracing life: Identifying fundamental approaches to life with latent profile analyses on the dimensions of the Hexaflex model. Journal of Contextual Behavioral Science, 12(April), 106-118. https://doi.org/10.1016/j.jcbs.2019.03.003

60.

Stucky

B. D.

Edelen

M. O.

(2015). Using hierarchical IRT to create unidimensional measures from multidimensional data. In Reise

S. P.

Revicki

D. A.

(Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183-206). Routledge.

61.

Thomas

K. N.

Bardeen

J. R.

(2020). The buffering effect of attentional control on the relationship between cognitive fusion and anxiety. Behaviour Research and Therapy, 132(September), 103653. https://doi.org/10.1016/j.brat.2020.103653

62.

Veddegjærde

K. E. F.

Sivertsen

Wilhelmsen

Skogen

J. C.

(2014). Confirmatory factor analysis and item response theory analysis of the Whiteley Index: Results from a large population based study in Norway: The Hordaland Health Study (HUSK). Journal of Psychosomatic Research, 77(3), 213-218. https://doi.org/10.1016/j.jpsychores.2014.06.011

63.

Wang

Russell

S. S.

(2005). Measurement equivalence of the job descriptive index across Chinese and American workers: Results from confirmatory factor analysis and item response theory. Educational and Psychological Measurement, 65(4), 709-732. https://doi.org/10.1177/0013164404272494

64.

Welch

P. G.

Carleton

R. N.

Asmundson

G. J. G.

(2009). Measuring health anxiety: Moving past the dichotomous response option of the original Whiteley Index. Journal of Anxiety Disorders, 23(7), 1002-1007. https://doi.org/10.1016/j.janxdis.2009.05.006

65.

Wheaton

M. G.

Berman

N. C.

Abramowitz

J. S.

(2010). The contribution of experiential avoidance and anxiety sensitivity in the prediction of health anxiety. Journal of Cognitive Psychotherapy, 45, 229-239.

66.

Wolgast

(2014). What does the Acceptance and Action Questionnaire (AAQ-II) really measure? Behavior Therapy, 45(6), 831-839. https://doi.org/10.1016/j.beth.2014.07.002

67.

Yarrish

Groshon

Mitchell

J. D.

Appelbaum

Klock

Winternitz

Friedman-Wheeler

D. G.

(2019). Finding the signal in the noise: Minimizing responses from bots and inattentive humans in online research. The Behavior Therapist, 42, 235-242. https://www.researchgate.net/publication/336564144_Finding_the_Signal_in_the_Noise_Minimizing_Responses_From_Bots_and_Inattentive_Humans_in_Online_Research

68.

Yasinski

Hayes

A. M.

Ready

C. B.

Abel

Görg

Kuyken

(2020). Processes of change in cognitive behavioral therapy for treatment-resistant depression: Psychological flexibility, rumination, avoidance, and emotional processing. Psychotherapy Research, 30(8), 983-997. https://doi.org/10.1080/10503307.2019.1699972