More of a Good Thing Is Not Always Better: Validation of a Distress Overtolerance Measure

Abstract

Distress tolerance (DT) refers to the ability to tolerate aversive psychological states. Research has mainly focused on the link between low DT and psychopathology with little empirical work on individuals on the high end (i.e., distress overtolerance). Distress overtolerance has been conceptualized as a tendency to tolerate very high levels of distress despite the negative consequences to one’s well-being. Currently, no measures of distress overtolerance have been developed, and current measures for DT are not well-suited for measuring distress overtolerance. To establish distress overtolerance as a construct, an exploratory factor analysis (N = 251) of the distress overtolerance scale was conducted and revealed a two-factor structure (i.e., Capacity for Harm and Fear of Negative Evaluation). In Study 2 (N = 257), a confirmatory factor analysis revealed strong psychometric properties, the expected nomological network, good construct validity, and incremental criterion utility. Results showed that this scale can be used as a starting point for the theoretical framework behind distress overtolerance.

Keywords

distress tolerance distress overtolerance measurement assessment validation

Distress tolerance (DT) has been defined as the ability to persist in goal-directed behavior despite negative emotional states (e.g., uncomfortable bodily sensations, stress, negative affect). Low DT has been associated with a number of maladaptive outcomes and psychological disorders. Indeed, low DT has been tied to increased alcohol and marijuana coping motives (Bujarski, Norberg, & Copeland, 2012; Howell, Leyro, Hogan, Buckner, & Zvolensky, 2010; Leyro, Zvolensky, & Bernstein, 2010). Additionally, low DT has been linked with numerous psychological disorders, such as posttraumatic stress disorder, depression, borderline personality disorder, substance use disorders, and antisocial personality disorder (Ebner-Priemer et al., 2008; Gorka, Ali, & Daughters, 2012; Sargeant, Daughters, Curtin, Schuster, & Lejuez, 2011; Vujanovic, Bonn-Miller, Potter, Marshall, & Zvolensky, 2011). In light of this evidence, low DT has been a target in the treatment of psychological disorders (Bornovalova, Gratz, Daughters, Hunt, & Lejuez, 2012; Brown et al., 2008).

Despite its connection to numerous outcomes and its clinical utility, the measurement of DT is problematic for various reasons. Of relevance to the current study, researchers have consistently explored DT in linear terms, with higher DT being associated with lower maladaptive outcomes. By doing this, researchers are implicitly suggesting that individuals at the low end are prone to maladaptive outcomes, whereas individuals at the higher end are not. Although this type of assumption is common across psychological research, empirical work has suggested that linearity does not always hold up across personality constructs (e.g., Bunderson & Sutcliffe, 2003; Langer & Imber, 1979). For example, although high conscientiousness is normally adaptive, extreme levels of conscientiousness have been related to decreased job performance (e.g., higher stress reaction to negative feedback, lower performance ratings by supervisors; Cianci, Klein, & Seijts, 2010; Dunkley, Blankstein, Zuroff, Lecce, & Hui, 2006; Samuel & Widiger, 2011). Similarly, high levels of perfectionism have been related to increased psychopathology as well as a tendency to perseverate on unattainable goals (Bieling, Israeli, & Antony, 2004; Burns, 1980). Furthermore, empirical work using extant DT measures has shown similar results. In a clinical sample of patients with borderline personality disorder and posttraumatic stress disorder, higher DT was associated with increased number of suicide attempts (Anestis, Gratz, Bagge, & Tull, 2012; Anestis, Tull, Bagge, & Gratz, 2012). This suggests that high levels of DT can potentially be maladaptive.

Given this evidence, it is possible that extremely high levels of DT (i.e., distress overtolerance) may serve as a risk factor for psychopathology. Indeed, researchers have defined distress overtolerance as the tendency to persist through extremely high levels of distress despite harmful effects on well-being and long-term goals (Lynch & Mizon, 2011). Although empirical work is limited, the theoretical foundation of DT stems from the learned industriousness theory and can be extended to distress overtolerance (Eisenberger, 1992). This theory proposes that the amount of effort an individual displays is a function of their prior learning history. Specifically, individuals with a history of reinforcement for high effort and individuals with a history of low effort will experience high effort in the future as less aversive and more aversive, respectively. That is, the effort in itself may gain secondary reward characteristics that generalize across behaviors for individuals (Brandon et al., 2003; Lynch & Mizon, 2011; Quinn, Brandon, & Copeland, 1996). This model can naturally be extended to the continuum of DT—such that low DT individuals were previously reinforced for low levels of effort, resulting in low effort in itself being rewarding; whereas, distress overtolerant individuals were previously reinforced for extremely high levels of effort, resulting in extremely high effort in itself being rewarding. Given this, it could be posited that distress overtolerant individuals develop similar maladaptive outcomes (i.e., internalizing and externalizing disorders) as a low DT individual. However, the underlying mechanisms behind the personality traits will vary.

For example, a child in a household with parents who are strict and have high standards for achievement would likely not receive praise unless a task was performed at an exceptionally high level. Due to the infrequent reward, the child may learn to be perfectionistic and to persist through negative emotions no matter what, traits that would not be present in a low DT child. These traits may continue into adulthood, resulting in an individual that unrelentingly persists at the task at hand rather than disengaging and/or incorporating adaptive coping strategies (e.g., using stress management techniques). As a result, the distress overtolerant individual may resort to other ways to improve his or her mood (e.g., negative coping, alcohol use, restricted diet). In this sense, distress overtolerance can be conceptualized as (a) an inability to disengage from the goal at hand and (b) perfectionistic tendencies. Furthermore, the distress overtolerant individual would likely report a poor sense of well-being and a tendency to judge negative emotions as inconsequential feedback to their behaviors (Lynch & Mizon, 2011).

Although the theoretical framework provided above has yet to be studied empirically, work on a similar construct, overcontrolled personality (i.e., excessive self-control), provides a good parallel to distress overtolerance. Specifically, Lynch, Hempel, and Dunkley (2015) posit that individuals high in overcontrol are biologically predisposed to avoiding emotions, and as a result, produce a similar pattern of behavior to distress overtolerant individuals, such as emotional inhibition, rigid cognition, and extreme perfectionism. Although, to date, no direct measure of overcontrolled personality as defined above exists, empirical findings from clinical populations historically high in overcontrol (e.g., eating disorders, obsessive–compulsive disorder) corroborate the underlying traits specified by Lynch and colleagues (e.g., Asendorpf, Denissen, & van Aken, 2008; Chapman et al., 2007). Thus, work on overcontrol provides a strong framework for potential theoretical mechanisms underlying distress overtolerance.

Similar to overcontrol, work on perseveration and perfectionism can shed light on potential relations between distress overtolerance and psychopathology. Perseveration has been linked to depression (Pyszczynski & Greenberg, 1987) and cocaine use (Ersche, Roiser, Robbins, & Sahakian). Likewise, maladaptive perfectionism has been associated with alcohol use (Rice & Van Arsdale, 2010). Notably, although distress overtolerance overlaps with perseveration and perfectionism, distress overtolerance is hypothesized to be a discrete construct because it accounts for negative affect in its very definition. In contrast, perseveration and perfectionism are strictly task oriented, without any emphasis on negative affect. Thus, they provide a good foundation for the expected nomological network of distress overtolerance but are not considered to be redundant with the distress overtolerance construct.

Despite the conceptual work on distress overtolerance, there is still a lack of empirical research, in part due to the lack of measures assessing distress overtolerance. Although self-report scales like the Distress Tolerance Scale (DTS; Simons & Gaher, 2005) and behavioral measures like the Paced Auditory Serial Addition Task (Lejuez, Kahler, & Brown, 2003) can measure high levels of DT, it is not entirely valid to use them to study distress overtolerance per se. First, it is difficult to tease apart those with just high levels of DT from those with distress overtolerance (as both would mostly like cluster toward the high end). Second, current DT measures do not assess whether tolerating distress affects the individual’s well-being or future goals. Last, even when these constructs do not have the problem of conflation, quadratic relationships are unstable and inconsistent in personality research (Carter et al., 2013).

To further research in this area, the development of a distress overtolerance self-report measure is necessary. To date, it is unclear if distress overtolerance is an important predictor of psychopathology in the clinical setting. Yet, without the development of a distress overtolerance specific measure, the potential utility of this construct could not be properly examined. The development of such a measure will aid in our theoretical understanding of the construct and its nomological network. Moreover, it could potentially shed light on whether emphasis should be placed on distress overtolerance in clinical settings. Although treatments have been developed for low DT individuals (Bornovalova et al., 2012; Linehan, 1993), distress overtolerant behaviors are normally not a focal component of treatment. Additionally, the development of a distress overtolerance measure could potentially remedy measurement concerns, as distress overtolerant individuals are subsumed within existing DT questionnaires.

To open up research in this area, we conducted two studies using Amazon Mechanical Turk (MTurk). MTurk was chosen for a few reasons. Recent evidence suggested that MTurk is a sound method to gather samples for research in psychopathology, given its high anonymity and reliability (Shapiro, Chandler, & Mueller, 2013). Moreover, research has shown that the MTurk population has exhibited prevalence rates for psychopathology that are slightly higher than the general population, affording us the ability to examine samples that potentially have a greater range for psychological symptoms (Joinson, 2001; Shapiro et al., 2013). Last, given that the distress overtolerance construct is in its infancy, there was not a clear direction as to which clinical population would be most optimal for validation of the scale; therefore, we thought it was best to perform the initial validation on a community sample.

In Study 1, questions were developed targeting the hypothesized characteristics of distress overtolerance: poor well-being, perfectionistic tendencies, and an inability to disengage. When developing items, our primary goal was that the items were sensitive to differences between distress overtolerance and high, adaptive DT, given the problems of conflation in previously established DT measures. Next, an exploratory factor analysis (EFA) was conducted. In Study 2, a confirmatory factor analysis (CFA) was conducted on the resulting factor structure from Study 1. Additionally, the nomological network, construct validity, and incremental criterion utility of the scale were evaluated. Research on distress overtolerance is sparse, and thus, we did not hypothesize a factor structure or a comprehensive nomological network of distress overtolerance. With that said, we did have some expectations. Based on theory and clinical lore, we would expect distress overtolerance would be related to the following personality constructs: lower well-being, higher achievement, higher constraint, perseveration and perfectionism, and higher overall negative emotionality. In regard to discriminant and incremental criterion validity, we expected that distress overtolerance should predict relevant clinical constructs—namely, alcohol use quantity/frequency, alcohol dependence symptoms, drug use, depression, and anxiety (external correlates) above and beyond general trait negative emotionality, DTS, perseverance, perseveration, and perfectionism.

Study 1

Method

Participants

Participants were recruited from Amazon MTurk and received $0.05 on completion. Amazon MTurk is a crowd-sourcing website that recruits workers for a given study. In this platform, researchers are afforded the ability to place restrictions on the workers based on quality of work in the past and location. Previous research has found that data from this site are comparable to traditional survey methods and results in a broader demographic than college samples (Buhrmester, Kwang, & Gosling, 2011). Filters were placed on MTurk workers such that only the most qualified and reliable workers were allowed to take our survey. These filters limited workers to those in the United States, with a work approval rate greater than 90% and a history of completing work on MTurk (at least 50 jobs).

Participants (N = 7) who did not respond correctly to the attention check (i.e., “If you are paying attention to the survey right now, click ‘mostly true of me’”) were excluded from further analyses. The remaining 251 participants responded correctly to this item and were included in the analyses. The participants’ mean age was 36 years (SD = 9.51), with a range of 18 to 75 years. Gender was 30% male, 68% female, and 2% transgender. Race was 82% Caucasian, 7% African American, 4% Hispanic, 4% Asian, 2% Multiracial, and 1% Other. The sample was highly educated, with 55% earning a bachelor’s degree or higher, and one participant reporting less than a high school education.

Measures

Distress Overtolerance Scale

Items were created by targeting the hypothesized characteristics of distress overtolerance: poor well-being, perfectionistic tendencies, and an inability to disengage. Three goals were developed when considering item construction. First, items were created to avoid capturing individuals with low DT individuals or individuals with high, adaptive DT. Second, we wanted to be consistent with wording, and therefore, “distressed” or “stressed” were the only feelings that were referenced. The specificity of the terminology, helped in differentiating distress overtolerance from other similar constructs (e.g., perfectionism, perseveration). Third, we wanted to be parsimonious with our wording to lower possible item miscomprehension by participants. However, this was sacrificed, at times, to achieve our first goal, which we deemed most important. Once the items were generated, they were shown to an expert in DT work as well as an expert in personality and quantitative psychology. Both researchers believed the final item is set to be face and content valid. After this expert review, a finalized item set of 22 was generated. We also ensured that the items were interpretable through the qualitative information gathered at the end of the survey questioning the readability and interpretability of the items. The majority of participants stated that the items were clear.

Participants were given the following instructions, which were adapted from the DTS (Simons & Gaher, 2005), when answering the items: “Think of a time recently when you were experiencing a stressful or distressing event. Using the scale, please indicate how much it describes your beliefs about feeling stressed or distressed.” All items used a scale from 1 to 6 to allow for a wide range in responses (1 = completely untrue of me, 2 = mostly untrue of me, 3 = somewhat untrue of me, 4 = somewhat true of me, 5 = mostly true of me, 6 = completely true of me). Example statements included “I tolerate distress so much so that it starts to take a toll on my personal and/or professional life” and “I persist through negative tasks even when I am overly stressed, so others won’t notice how I am feeling.”

Analytical Procedure

To examine the factor structure and the underlying dimensions of the Distress Overtolerance Scale, we implemented a categorical EFA. This was chosen over CFA because we desired a more data-driven approach to creating a factor structure. The analysis used the robust diagonally weighted least squares estimation (weighted least squares means and variance in Mplus Version 7; Muthén & Muthén, 1998-2012), as this estimator has performed well with smaller sample sizes (Flora & Curran, 2004). Moreover, this estimator performs better with ordinal data than classic methods because it utilizes polychoric correlations instead of Pearson correlations. An oblique rotation was selected to allow the factors to correlate. The quartimin rotation was used for the analysis, as it is found to perform well under a variety of conditions by Browne (2001). According to Comrey and Lee (1992), loadings less than .32 are poor and should not be interpreted. Therefore, a cutoff of .32 was used.

Results

Preliminary Analyses

The 22 items exhibited some evidence of mild nonnormality, with skewness ranging from −0.77 to 0.32 and kurtosis ranging from −1.15 to 0.15. The skewness and kurtosis of the individuals that missed the attention checks (N = 7) also exhibited mild nonnormality, with skewness ranging from −0.82 to −0.88 and with kurtosis ranging from −1.18 to −1.24. Guides to determining nonnormality suggest that an acceptable range is from −2 to +2 for skewness and kurtosis (George & Mallery, 2010). Given that the skewness and kurtosis fell in the acceptable range, no transformations were made. Additionally, diagonally weighted least squares estimation can perform well with mild forms of nonnormality (Flora & Curran, 2004).

Exploratory Factor Analysis

A scree test suggested two factors, as evidenced by the elbow (see Floyd & Widaman, 1995). Researchers have suggested that more objective analyses, such as parallel analysis (PA), are ideal when conducting an EFA (Ledesma & Valero-Mora, 2007). A PA generates random data sets of equal sample size and compares the resulting eigenvalues with the current data’s eigenvalues. The number of factors to retain is the number of eigenvalues that are larger than the corresponding random eigenvalues (Ledesma & Valero-Mora, 2007). For the PA analysis, we followed standards in the literature by generating 100 random correlation matrices with the percentile of eigenvalues being 95. Our PA results suggested that a two-factor structure was appropriate. Therefore, we ultimately decided on a two-factor structure, given the scree test and PA analysis. Factor loadings higher than .32 were considered meaningful. Therefore, any items that produced cross-loadings higher than .32 on two factors were excluded from further analysis. Additionally, items that did not produce loadings higher than .32 on any factors were excluded. This process resulted in the removal of six items, leaving a total of 16 questions in the scale (see Table 1 for factor loadings). After examining the item loadings, the factors were clearly interpretable and were identified as Capacity for Harm (CH) and Fear of Negative Evaluation (FNE). The CH subscale consisted of 11 items pertaining to whether an individual tolerates distressing situations despite its effect on their well-being. The FNE subscale was made up of five items pertaining to whether individuals persist through stress due to their fear of negative evaluations from others if they were to quit. All items loaded strongly onto their respective factor (>.62) and poorly onto their nonrespective factor (<.20). Additionally, the two identified factors were moderately correlated and the total scale score and subscale scores demonstrated acceptable internal consistency (>.80; see Table 1). The final 16 items exhibited some evidence of mild nonnormality with skewness ranging from −0.77 to 0.16 and kurtosis ranging from −1.15 to 0.09. The full item questions, after item extraction, can be found in Table 1. In summary, psychometric evidence supported the two-factor model.

Table 1.

Item Content and Exploratory and Confirmatory Factory Analysis Loadings for the Distress Overtolerance Scale.

Item content	Exploratory		Confirmatory
Item content	CH	FNE	CH	FNE
I would be a happier person overall if I learned how to work through my negative emotions instead of just persisting through them at all costs.	.55	−.03	.67
I tolerate negative feelings or situations for so long that, afterward, I resort to other ways to boost my mood (e.g., drinking, using drugs, shopping, or eating excessively).	.63	.01	.70
After completing a distressing task, I often find that I have to pay a higher price than others for the same reward.	.83	−.14	.77
I think a lot of my problematic behavior would be reduced if I did not try to persist through so much stress.	.70	.07	.77
Negative things tend to happen when I push myself too hard.	.68	−.08	.71
People think that I should be more concerned with how my tolerance of distressing situations affects my mental and physical health.	.68	−.03	.75
Sometimes I get so wrapped up in fighting through my negative emotions that I lose sight of what is best for me.	.81	.00	.81
A lot of stress and negative feelings build up inside me that I do not let out.	.65	.11	.81
A lot of harmful consequences happen to me because I will not give up even during very stressful situations.	.69	.17	.80
I tolerate distress so much so that it starts to take a toll on my personal and/or professional life.	.74	.07	.87
I often think of times when I have given up during a distressing situation and feel very angry with myself.	.63	.08	.81
Even when the negative feelings become overwhelming, I feel pressure to continue with the difficult task or people might think I am “weak.”	.15	.67		.89
I care more about the completion of a stressful task than the negative consequences to my health and happiness.	.11	.65		.87
I persist through negative tasks even when I am overly stressed, so others won’t notice how I am feeling.	.11	.79		.86
I often push myself further than others do.	−.04	.67		.68
There is no excuse for not pushing through a stressful task.	−.19	.73		.68
α	.90	.81	.91	.87
M (SD)	3.67 (1.03)	4.03 (0.97)	3.45 (1.14)	3.94 (1.14)
Correlation with CH		.47***		.66***

Note. CH = Capacity for Harm; FNE = Fear of Negative Evaluation. Loadings are from the pattern matrix after oblique rotation. The bolded values reflect the items that comprise the respective factor.

p < .05. **p < .01. ***p < .001.

Study 2

Method

Purpose

Given the encouraging factor loadings in Study 1, we conducted a CFA to cross validate our measure in a separate sample. Additionally, we wanted to verify that the Distress Overtolerance Scale would demonstrate strong psychometric properties (e.g., convergent validity, discriminant validity, incremental criterion utility).

Participants

Participants were recruited from MTurk with the same requirements as Study 1 and received $4.00 on completion. The MTurk site prevented the same participant from completing the survey twice. To further ensure reliable responses, the same attention checks that were administered in Study 1 were administered in the current study at evenly spaced intervals (e.g., 20%, 40%, 60%, 80%, 100% completion) throughout the survey. Participants (N = 35) that missed more than 50% of these checks were excluded from analyses. The remaining 257 participants were included in the analyses. The participants’ mean age was 35 (SD = 10.96), with a range from 18 to 67 years. Gender was 30% male, 68% female, and 2% transgender. Race was 60% Caucasian, 7% African American, 8% Hispanic, 21% Asian, 3% Multiracial, and 1% Other. The sample was highly educated, with 75% indicating they completed a bachelor’s degree or higher.

Measures

Distress Overtolerance Scale

The remaining 16 items from the EFA sample were included to substantiate the two-factor structure of the Distress Overtolerance Scale. We used mean composite scores of the total scale score¹ and its subscales to determine the extent of its relationship with other scales. These scales are organized below by the type of validity they are supporting or rejecting.

Nomological Network

Personality Inventory

The Multidimensional Personality Questionnaire–Brief Form (MPQ-BF; Patrick, Curtin, & Tellegen, 2002) is a 155-item true–false scale composed of 11 subscales, including well-being (optimistic, enjoying activities), social potency (decisive, enjoy leadership), achievement (hardworking, ambitious), social closeness (sociable, warm, and affectionate), stress reaction (tense, nervous, easily upset), aggression (physically aggressive, victimizes others), alienation (feeling pushed around, feeling betrayed, and deceived), control (cautious, planful), harm avoidance (prefers safe activities and experiences), traditionalism (high moral standards, values a good reputation), and absorption (becomes immersed in own thoughts and feelings, responsive to evocative sensory experiences). The MPQ-BF also consists of three superfactors, including positive emotional temperament (joyful, engaged in rewarding activities), negative emotional temperament (anger, anxiety), and constraint (emotional inhibition, risk-taking). The MPQ-BF has shown strong reliability (α = .75-.84; Tellegen, 1982). In the current study, the internal consistency for the subscales ranged from .68 to .88.

Convergent Validity

Perfectionism and Perseveration

The Perfectionism, Persistence, and Perseveration Questionnaire (PPPQ-22; Serpell, Waller, Fearon, & Meyer, 2009) is a 22-item self-report measure of three psychological constructs: perfectionism, persistence, and perseveration. It asks on a 6-point scale how true the statement is (0 = not at all true of me to 5 = totally true of me). Perfectionism taps into high standards in all areas of functioning. Persistence taps into one’s ability to keep going with a behavior until a goal is met, even when the task is difficult. Perseveration is the tendency to continue with a particular behavior, even when it is no longer rewarding. Example items include “I hate making mistakes” (Perfectionism); “People describe me as someone who can stick at a task, even when it gets difficult” (Persistence); and “I keep trying to sort out problems in a relationship, even if I know it’s not going to survive” (Perseveration). This scale has shown predictive validity for psychological disorders and has demonstrated adequate internal consistency (α = .64-.76; Serpell et al., 2009). In the current study, the internal consistency for the subscales ranged from .50 to .83.

Distress Tolerance

The DTS (Simons & Gaher, 2005) is a 15-item scale used to assess an individual’s perceived capacity to withstand negative emotional events. Individuals are asked to respond on a scale from 1 to 5 (1 = strongly agree to 5 = strongly disagree) about how much the statement describes their beliefs about feeling distressed or upset. Higher scores on this scale indicate a higher level of DT. Sample items include “When I feel distressed or upset, I must do something about it immediately” and “When I feel distress or upset, I cannot help but concentrate on how bad the distress actually feels.” The DTS has demonstrated good validity and reliability over a 6-month interval (α = .61; Simons & Gaher, 2005). In the current study, the internal consistency was .88.

External Correlates

Alcohol and Drug Use

The Alcohol Use Disorder Identification Test–Core (Bohn, Babor, & Kranzler, 1995) is a 10-item questionnaire assess alcohol consumption, dependence symptoms, and personal/social difficulties from drinking over the past year. Sample items include “How often do you have a drink containing alcohol?” and “How often during the past year have you failed to do what was normally expected of you because of drinking?” Total score ranges from 0 to 40, where scores greater than 8 indicate the presence of alcohol use problems, and scores greater than 20 indicate severe alcohol use problems. This measure has been shown to be valid measure of alcohol use disorders (Bohn et al., 1995) with good internal consistency (α = .83; Hays & Merz, 1995). In the current study, the internal consistency was .92.

The Alcohol Dependence Scale (ADS; Horn, 1984) is a 25-item scale that assesses various alcohol occurrences over the past 12 months using a dichotomous (yes–no) scale, such as drinking patterns, emotional and physiological responses to alcohol, tolerance, and withdrawal side effects. Higher scores indicate higher alcohol dependence symptoms. Sample items include “Have you tried to cut down on your drinking and failed?” and “Do you gulp drinks (drink quickly)?” The scale has demonstrated good internal consistency (α = .92; Skinner & Allen, 1982). In the current study, the internal consistency was .88.

The Drug Use Disorder Identification Test (DUDIT; Berman, Bergman, Palmstierna, & Schlyter, 2005) is an 11-item scale that was adapted from the Alcohol Use Disorder Identification Test–Core (Bohn et al., 1995). It asks about frequency of drug consumption, dependency symptoms, and social problems due to drug use. Sample item include “How often do you use drugs other than alcohol?” and “Have you or anyone else been hurt (mentally or physically) because you used drugs?” The scale has demonstrated good internal consistency (α = .80; Berman et al., 2005). In the current study, the internal consistency was .91.

Depression and Anxiety

The Brief Symptom Inventory–53 (Derogatis & Melisaratos, 1983) is a 53-item questionnaire that measures psychological symptoms and distress. The items target the nine dimensions of psychological symptoms. For the purposes of the current investigation, we examined only the anxiety and depression subscales for two reasons: lack of expectation regarding the other subscales and to reduce the overall number of tests. The items are rated on a 5-point scale (0 = not at all to 4 = extremely). The Brief Symptom Inventory–53 has demonstrated good validity (α = .86-.88; Bingham, Harawa, & Williams, 2013). In the current study, the internal consistency for the depression subscale was .89 and for the anxiety subscale, it was .92.

Results

Preliminary Analyses

The distress overtolerance subscales did not exhibit evidence of nonnormality, with skewness ranging from −0.43 to −0.39 and kurtosis being 0.30 for both subscales. The skewness and kurtosis of the missing data due to the attention checks (N = 35) exhibited evidence of mild nonnormality, with skewness ranging from −0.71 to −0.72 and with kurtosis ranging from −0.75 to −0.76. We also examined evidence of nonnormality in our other variables of interest. The following scales/subscales exhibited nonnormality according to the standards by George and Mallery (2010) and to the normal quantile plots: MPQ (Control), DUDIT, and ADS. Due to this, natural log transformations were conducted on these variables. Following this, the DUDIT and ADS exhibited normal distributions. However, the MPQ (Control subscale) did not, and therefore, was analyzed in its original state. Past research has found age and gender to be significant correlates for both DT and externalizing/internalizing symptoms (Daughters et al., 2009). Therefore, analyses were conducted to determine whether any of our variables of interest were associated with age or gender (see Table 2). The demographic variables that were significantly correlated or that significantly different between groups were controlled for in all subsequent analyses.

Table 2.

Relationship of Covariates With Variables of Interest.

	Age (r)	Gender (F)
Main variables
DO_CH	−0.15*	0.01
DO_FNE	−0.05	4.87*
DO_Total	−0.13*	0.02
Nomological network
MPQ_Well-Being	−0.06	1.95
MPQ_Social Potency	−0.02	0.63
MPQ_Achievement	0.07	3.68
MPQ_Social Closeness	0.16*	0.17
MPQ_Aggression	−0.15*	1.27
MPQ_Stress Reaction	0.12	0.06
MPQ_Alientation	0.15*	25.13***
MPQ_Control	0.03	0.85
MPQ_Harm Avoidance	−0.01	0.02
MPQ_Traditionalism	0.02	0.53
MPQ_Absorption	0.01
MPQ_Positive Emotional Temperament	0.16	2.73
MPQ_Negative Emotional Temperament	0.09	11.01**
MPQ_Constraint	−0.01	1.07
MPQ_Absorption	0.01	1.25
Convergent validity
PPPQ_Perseveration	−0.16*	0.01
PPPQ_Perfectionism	0.09	1.30
PPPQ_Persistence	0.16*	0.19
DTS	0.13*	1.91
External validity
AUDIT	−0.33***	21.45***
ADS	−0.28***	3.77
DUDIT	−0.20**	26.96***
BSI_Depression	−0.27***	5.40*
BSI_Anxiety	−0.20**	0.90

Note. DO = Distress Overtolerance; CH = Capacity for Harm; FNE = Fear of Negative Evaluation; DO_Total = CH and FNE items; MPQ = Multidimensional Personality Questionnaire–Brief Form; PPPQ = Perfectionism, Persistence, and Perseveration Questionnaire; DTS = Distress Tolerance Scale; AUDIT = Alcohol Use Disorder Identification Test; ADS = Alcohol Dependency Scale; DUDIT = Drug Use Disorder Identification Test; BSI = Brief Symptom Inventory.

p < .05. **p < .01. ***p < .001.

Confirmatory Factor Analysis

To confirm the structure of the Distress Overtolerance Scale, we implemented a categorical CFA. We used the same analysis procedure as Study 1; however, we fixed variance of each factor to 1 to put the factors on a standard scale. Consistent with the EFA findings, the two subscales were highly correlated and the items exhibited good internal consistency (>.86; see Table 1). Moreover, the two-factor model showed strong factor loadings on their respective factors (see Table 1). The model fit was acceptable (comparative fit index = .96; Tucker–Lewis index = .95; root mean square error of approximation = .08; see Hu & Bentler, 1999).

Nomological Network

We conducted correlations between the distress overtolerance scale and the trait scales and super factors of the MPQ-BF, controlling for age and gender (see Table 3). Higher scores on the distress overtolerance total scale score were associated with lower well-being and higher achievement, constraint, and negative emotionality. Most notably, the pattern of correlations between the subscales were in line with our a priori theoretical framework. Specifically, the FNE subscale was correlated with achievement, and the CH subscale was correlated with well-being. Furthermore, the CH subscale was more predictive of the personality variables.

Table 3.

Construct Validity of the Distress Overtolerance Scale.

	CH	FNE	DO_Total	DTS
Nomological network
MPQ_Well-Being	−.27***	−.03	−.21**	.27***
MPQ_Social Potency	.20***	.19**	.21**	−.09
MPQ_Achievement	.11	.37***	.21**	−.02
MPQ_Social Closeness	.29***	.14*	.27***	−.24
MPQ_Aggression	.37***	.15*	.33***	−.22
MPQ_Stress Reaction	.63***	.33***	.59***	−.59***
MPQ_Alientation	.52***	.25***	.48***	−.48***
MPQ_Control	.15*	.13*	.16*	−.08
MPQ_Harm Avoidance	.04	.04	.05	.07
MPQ_Traditionalism	.12	.11	.13*	−.20**
MPQ_Absorption	.19**	.13*	.19**	−.20**
MPQ_Positive Emotional Temperament	.12	.24***	.18**	−.02
MPQ_Negative Emotional Temperament	.61***	.30***	.57***	−.53***
MPQ_Constraint	.14*	.14*	.16*	−.09
Convergent validity
PPPQ_Perseveration	.51***	.42***	.53***	−.39***
PPPQ_Perfectionism	.11	.36***	.21**	−.11
PPPQ_Persistence	−.12	.28***	.12	.12
DTS	−.61***	−.28***	−.56***	—
External validity
AUDIT	.21**	.09	.19**	−.09
ADS	.38***	.18*	.35***	−.24***
DUDIT	.11	.01	.08	−.09
BSI_Depression	.55***	.23***	.50***	−.47***
BSI_Anxiety	.55***	.24***	.50***	−.47***

p < .05. **p < .01. ***p < .001.

Convergent Validity

The distress overtolerance total scale score showed a significant positive association with perseveration and perfectionism. The DTS showed a negative correlation with perseveration and no relationship with perfectionism. This supports the notion that distress overtolerance and DT are, indeed, composed of different psychological components (see Table 3). Importantly, the two distress overtolerance subscales showed a slightly different pattern of relationships. The FNE subscale was correlated with perseveration and perfectionism; whereas, the CH subscale only showed a relation with perseveration. The significant, negative correlation between the DTS and distress overtolerance scale is further evidence of discriminant validity between the two constructs.

External Correlates

The distress overtolerance total scale score showed a significant positive association with the alcohol use, alcohol dependency, depression, and anxiety. Neither the distress overtolerance total scale score nor the individual subscales were associated with drug use (see Table 4). This was unexpected; however, as noted above, the DUDIT exhibited problematic kurtosis and positive skewness. Although the transformation improved this, there was still a floor effect. Of note, the pattern of correlations suggested that the CH subscale is more predictive of a wide range of psychopathology; whereas, the FNE subscale is primarily predictive of internalizing disorders.²

Table 4.

Incremental Criterion Utility of the DO Scale.

Predictor (β) (Above and beyond MPQ-NEM)
	NEM	CH	CH R²	FNE	FNE R²	DO_Tot	DO_Tot R²
AUDIT	.28***	.04	.00	.01	.001	.03	.001
ADS	.36***	.21***	.03*	.06	.003	.18**	.02*
DUDIT	.31***	.01	.00	−.08	.01	−.03	.001
BSI_Dep	.59***	.28***	.05***	.04	.002	.21***	.03***
BSI_Anx	.64***	.25***	.04***	.04	.001	.19**	.02**
Predictor (β) (Above and beyond DTS)
	DTS	CH	CH R²	FNE	FNE R²	DO_Tot	DO_Tot R²
AUDIT	−.09	.25***	.03**	.05	.003	.19*	.03**
ADS	−.24**	.31***	.06***	.11	.01	.26***	.04**
DUDIT	−.09	.08	.01	−.05	.001	.05	.002
BSI_Dep	−.46***	.42***	.11***	.09	.01	.34***	.08***
BSI_Anx	−.45***	.41***	.10***	.11*	.01*	.34***	.08***
Predictor (β) (Above and beyond PPPQ)
	Persev	Persis	Perf	CH	CH R²	FNE	FNE R²	DO_Tot	DO_Tot R²
AUDIT	.21***	−.17	−.09	.22**	.01	.07	.002	.13	.01
ADS	.36***	.09	.03	.16*	.02*	.05	.01	.15*	.02*
DUDIT	.17	.20	−.03	.13	.01	−.04	.003	.08	.01
BSI_Dep	.35***	−.12	.01	.48***	.14***	.16*	.02*	.41***	.11***
BSI_Anx	.43***	−.36***	.17*	.40***	.10***	.12*	.01*	.35***	.08***

Note. MPQ = Multidimensional Personality Questionnaire; DTS = Distress Tolerance Scale; PPPQ = Persistence, Perseveration, and Perfectionism Questionnaire; NEM = Negative Emotionality superfactor from MPQ; AUDIT = Alcohol Use Disorder Identification Test; ADS = Alcohol Dependency Scale; DUDIT = Drug Use Disorder Identification Test; BSI = Brief Symptom Inventory; Dep = Depression; Anx = Anxiety; Persev = Perseveration subscale; Persis = Persistence subscale; Perf = Perfectionism subscale; DO = Distress overtolerance; CH = Capacity for Harm; FNE = Fear of Negative Evaluation; DO_Tot = CH and FNE items. In all regressions, age and gender were entered into Step 1; MPQ-NEM/DTS/PPPQ subscales were entered into Step 2; the distress overtolerance scales entered in Step 3; the DTS and PPPQ subscale betas are shown for Step 2 only.

p < .05. **p < .01. ***p < .001.

Incremental Criterion Utility

We examined incremental criterion utility by conducting multiple regression analyses between the distress overtolerance scale and relevant outcomes, after accounting for the MPQ-Negative Emotionality superfactor in the first set of analyses; DTS in the second set of analyses and for the PPPQ subscales in the third set of analyses (see Table 4). Specifically, in Step 1 of the regression, age and gender were entered. In Step 2, either the DTS, MPQ-Negative Emotionality, or one of the PPPQ subscales was entered. In Step 3, either the CH, FNE, or total scale score was entered. The R² change from Step 2 to Step 3 for the distress overtolerance total scale score and subscale scores is presented in Table 4. The results for the distress overtolerance subscales/total scale score with external outcomes generally remained significant. Notably, after controlling for negative emotionality, the predictive utility of the FNE scale fell to nonsignificance; however, the CH scale continued to show predictive utility. Overall, this suggests that the distress overtolerance scale is contributing above and beyond related constructs, supporting its discriminant validity.

Discussion

The current studies were conducted to validate the Distress Overtolerance Scale in order to inform theory and further research in the area. In Study 1, an EFA revealed a two-factor structure. This consisted of a CH subscale (whether an individual tolerates extremely distressing situations despite its effect on their quality of life) and a FNE subscale (whether the individual persists through distress due to their fear of negative evaluation from others if they quit). This structure is in line with the theoretical framework provided by scholars in the area (e.g., Lynch et al., 2015; Lynch & Mizon, 2011). In Study 2, a CFA reinforced the findings from Study 1, showing acceptable model fit and strong factor loadings. Psychometric properties of the scale were also provided in Study 2. Evaluation of the nomological network, convergent, discriminant, external, and incremental utility all generally provided support for the construct validity of the Distress Overtolerance Scale.

The psychometric properties were promising for a few reasons. First, in line with our initial hypotheses, the nomological network showed that higher distress overtolerance is, indeed, related to lower well-being and higher achievement, negative emotionality, and constraint. Second, in line with our initial predictions, the pattern of relationships between the distress overtolerance scale and DTS scale with perseveration, persistence, and perfectionism suggests that they are tapping into different underlying processes. Specifically, the distress overtolerance total scale score was positively associated with perseveration and perfectionism; whereas, the DTS was negatively associated with perseveration and had no relation with perfectionism. Furthermore, the DTS and the distress overtolerance scale exhibited a significant negative correlation, further demonstrating discriminant validity between the two constructs. Third, the relationships with external outcomes were in line our predictions. Specifically, the distress overtolerance total scale score was related to alcohol dependency, general alcohol use symptoms, depression, and anxiety. However, it was not related to drug use, and as stated previously, could be due to floor effects in our sample. Last, the incremental criterion utility of both the CH and FNE subscales showed that they contribute unique relations to outcomes above and beyond previously developed scales. Importantly, the distress overtolerance total score showed significant R² change above and beyond these scales for the majority of analyses, suggesting that it accounts for unique variance in our outcome variables.

Some other findings are worth noting. First, the CH and FNE subscales appear to function in discrete ways, suggesting the two scales may tap into somewhat different underlying mechanisms. Across outcomes, the FNE subscale primarily predicted internalizing disorders; whereas, the CH subscale predicted both internalizing and externalizing disorders. Although contrary to our initial hypotheses, this finding supports the nature of these scales, given that the FNE subscale is tapping into rigidity and perfectionism more than the CH subscale. Indeed, research in perfectionism has mainly focused on its relation to higher internalizing symptoms (Cook & Kearney, 2009; Shafran & Mansell, 2001). Additionally, across outcomes, the CH subscale accounts for more variance than the FNE subscale, and provides higher evidence of incremental criterion validity. Thus, the question of whether the FNE subscale is conceptually or clinically useful remains unclear. In addition, we found some discrepancies between our findings and the theoretical framework of a similar construct, overcontrol (i.e., nonsignificant correlations between distress overtolerance with social closeness and positive temperament), and future research should consider if this will affect the construct in meaningful ways (Lynch et al., 2015).

In light of our findings, there are a few areas that need to be addressed in future research. First, the nomological network for the scale is in need of further development to understand how DT and distress overtolerance differ. Specifically, research determining whether there are specific disorders or mechanisms that are unique to one or the other is needed. Pinpointing specific mechanisms/disorders unique to distress overtolerance would also help determine whether the FNE subscale is worthwhile, given that, in some instances, it detracted from the overall predictive utility of the distress overtolerance total scale score. It may be that the FNE subscale is predictive of disorders that are characterized by perfectionistic tendencies, which was not a focus in the current validation. Second, both subscales should be cross-validated in a clinical population. For instance, cross-validation in populations hypothesized to have high levels of distress overtolerance (e.g., eating disorder populations) would be useful. Evidence suggesting that psychological symptoms fluctuate relative to changes in distress overtolerance would strengthen its potential clinical utility.

Third, recent conceptual articles have shown that bipolar measures may be more useful in psychological research, especially considering the movement for a dimensional model of psychopathology (see Samuel, 2011). Therefore, it may be useful for future research to consider the development of measure that considers the whole continuum of DT from distress intolerance to distress overtolerance. However, as noted above, caution should be taken to ensure items are able to properly discriminate high DT from distress overtolerance. Last, longitudinal studies addressing mean-level stability and other metrics that would assess whether distress overtolerance is a stable personality trait are necessary. This is especially important given the current study’s instructional set. We asked the participants to refer back to a time when they felt distressed, which may tap into a more state-like distress overtolerance construct. Although the instructional set has advantages in terms of ecological validity, the trait-like nature of the measure should nonetheless be explored.

While the current study is novel and provides many avenues for future research, there were some limitations. First, the EFA was not conducted on a clinical sample in addition to the MTurk sample. As a consequence, items that were removed may have been highly endorsed in a clinical sample. Therefore, we may have potentially removed items that could have been useful for identifying distress overtolerance in samples with elevated psychopathology. Additionally, the small number of items on the FNE subscale may have resulted in less variability in scores. Similarly, the finalized item set was small, and therefore, may have reduced the content validity of our scale. Future research could explore other potential items, encompassing behavioral, emotional, and cognitive distress overtolerance, which could increase the content validity of the scale. Moreover, there was no validity scale that assessed whether there were biases in responding. Last, we did not conduct family-wise alpha corrections in Study 2 given the arguments against them; including increased Type II error, reduced power, and ambiguity about what constitutes a family, the exploratory nature of the current study. As a result, we suggest increased attention toward effect sizes (i.e., R² change) rather than significance level. Nevertheless, the current study provides the first validation of a distress overtolerance measure. This study creates a new avenue of research that will allow for further study of distress overtolerance to a host of psychopathology and maladaptive behaviors.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Anestis

M. D.

Gratz

K. L.

Bagge

C. L.

Tull

M. T.

(2012). The interactive role of distress tolerance and borderline personality disorder in suicide attempts among substance users in residential treatment. Comprehensive Psychiatry, 53, 1208-1216.

Anestis

M. D.

Tull

M. T.

Bagge

C. L.

Gratz

K. L.

(2012). The moderating role of distress tolerance in the relationship between posttraumatic stress disorder symptom clusters and suicidal behavior among trauma exposed substance users in residential treatment. Archives of Suicide Research, 16, 198-211.

Asendorpf

J. B.

Denissen

J. J.

van Aken

M. A.

(2008). Inhibited and aggressive preschool children at 23 years of age: Personality and social transitions into adulthood. Developmental Psychology, 44, 997-1011.

Berman

A. H.

Bergman

Palmstierna

Schlyter

(2005). Evaluation of the Drug Use Disorders Identification Test (DUDIT) in criminal justice and detoxification settings and in a Swedish population sample. European Addiction Research, 11, 22-31.

Bieling

P. J.

Israeli

A. L.

Antony

M. M.

(2004). Is perfectionism good, bad, or both? Examining models of the perfectionism construct. Personality and Individual Differences, 36, 1373-1385.

Bingham

T. A.

Harawa

N. T.

Williams

J. K.

(2013). Gender role conflict among African American men who have sex with men and women: Associations with mental health and sexual risk and disclosure behaviors. American Journal of Public Health, 103, 127-133.

Bohn

M. J.

Babor

T. F.

Kranzler

H. R.

(1995). The Alcohol Use Disorders Identification Test (AUDIT): Validation of a screening instrument for use in medical settings. Journal of Studies on Alcohol, 56, 423-432.

Bornovalova

M. A.

Gratz

K. L.

Daughters

S. B.

Hunt

E. D.

Lejuez

C. W.

(2012). Initial RCT of a distress tolerance treatment for individuals with substance use disorders. Drug and Alcohol Dependence, 122, 70-76.

Brandon

T. H.

Herzog

T. A.

Juliano

L. M.

Irvin

J. E.

Lazev

A. B.

Simmons

V. N.

(2003). Pretreatment task persistence predicts smoking cessation outcome. Journal of Abnormal Psychology, 112, 448-456.

10.

Brown

R. A.

Palm

K. M.

Strong

D. R.

Lejuez

C. W.

Kahler

C. W.

Zvolensky

M. J.

. . . Gifford

E. V.

(2008). Distress tolerance treatment for early-lapse smokers: Rationale, program description, and preliminary findings. Behavior Modification, 32, 302-332.

11.

Browne

M. W.

(2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36, 111-150.

12.

Buhrmester

Kwang

Gosling

S. D.

(2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.

13.

Bujarski

S. J.

Norberg

M. M.

Copeland

(2012). The association between distress tolerance and cannabis use-related problems: The mediating and moderating roles of coping motives and gender. Addictive Behaviors, 37, 1181-1184.

14.

Bunderson

J. S.

Sutcliffe

K. M.

(2003). Management team learning orientation and business unit performance. Journal of Applied Psychology, 88, 552-560.

15.

Burns

U. D.

(1980, November). The perfectionist’s script for self-defeat. Psychology Today, pp. 34-51.

16.

Carter

N. T.

Dalal

D. K.

Boyce

A. S.

O’Connell

M. S.

Kung

M. C.

Delgado

K. M.

(2013). Uncovering curvilinear relationships between conscientiousness and job performance: How theoretically appropriate measurement makes an empirical difference. Journal of Applied Psychology, 99, 564-586.

17.

Chapman

A. L.

Lynch

T. R.

Rosenthal

M. Z.

Cheavens

J. S.

Smoski

M. J.

Krishnan

K. R. R.

(2007). Risk aversion among depressed older adults with obsessive compulsive personality disorder. Cognitive Therapy and Research, 31, 161-174.

18.

Cianci

A. M.

Klein

H. J.

Seijts

G. H.

(2010). The effect of negative feedback on tension and subsequent performance: The main and interactive effects of goal content and conscientiousness. Journal of Applied Psychology, 95, 618-630.

19.

Clark

L. A.

Watson

(1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309.

20.

Comrey

A. L.

Lee

H. B.

(1992). Interpretation and application of factor analytic results. In Comrey

A. L.

Lee

H. B.

(Eds.), A first course in factor analysis (pp. 240-263). Hillsdale, NJ: Lawrence Erlbaum.

21.

Cook

L. C.

Kearney

C. A.

(2009). Parent and youth perfectionism and internalizing psychopathology. Personality and Individual Differences, 46, 325-330.

22.

Daughters

S. B.

Reynolds

E. K.

MacPherson

Kahler

C. W.

Danielson

C. K.

Zvolensky

Lejuez

C. W.

(2009). Distress tolerance and early adolescent externalizing and internalizing symptoms: The moderating role of gender and ethnicity. Behaviour Research and Therapy, 47, 198-205.

23.

Derogatis

L. R.

Melisaratos

(1983). The brief symptom inventory: An introductory report. Psychological Medicine, 13, 595-605.

24.

Dunkley

D. M.

Blankstein

K. R.

Zuroff

D. C.

Lecce

Hui

(2006). Self-critical and personal standards factors of perfectionism located within the five-factor model of personality. Personality and Individual Differences, 40, 409-420.

25.

Ebner-Priemer

U. W.

Kuo

Schlotz

Kleindienst

Rosenthal

M. Z.

Detterer

. . . Bohus

(2008). Distress and affective dysregulation in patients with borderline personality disorder: A psychophysiological ambulatory monitoring study. Journal of Nervous and Mental Disease, 196, 314-320.

26.

Eisenberger

(1992). Learned industriousness. Psychological Review, 99, 248-267.

27.

Ersche

K. D.

Roiser

J. P.

Robbins

T. W.

Sahakian

B. J.

(2008). Chronic cocaine but not chronic amphetamine use is associated with perseverative responding in humans. Psychopharmacology, 197, 421-431.

28.

Flora

D. B.

Curran

P. J.

(2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491.

29.

Floyd

F. J.

Widaman

K. F.

(1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286-299.

30.

George

Mallery

(2010). SPSS for Windows step by step: A simple guide and reference, 17.0 update (10th ed.). Boston, MA: Pearson.

31.

Gorka

S. M.

Ali

Daughters

S. B.

(2012). The role of distress tolerance in the relationship between depressive symptoms and problematic alcohol use. Psychology of Addictive Behaviors, 26, 621-626.

32.

Hays

R. D.

Merz

J. F.

Nicholas

(1995). Response burden, reliability, and validity of the CAGE, Short MAST, and AUDIT alcohol screening measures. Behavior Research Methods, Instruments, & Computers, 27, 277-280.

33.

Horn

J. L.

(1984). Alcohol dependence scale (ADS) user’s guide. Toronto: Addiction Research Foundation 1984.

34.

Howell

A. N.

Leyro

T. M.

Hogan

Buckner

J. D.

Zvolensky

M. J.

(2010). Anxiety sensitivity, distress tolerance, and discomfort intolerance in relation to coping and conformity motives for alcohol use and alcohol use problems among young adult drinkers. Addictive Behaviors, 35, 1144-1147.

35.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

36.

Joinson

A. N.

(2001). Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European Journal of Social Psychology, 31, 177-192.

37.

Langer

E. J.

Imber

L. G.

(1979). When practice makes imperfect: Debilitating effects of overlearning. Journal of Personality and Social Psychology, 37, 2014-2024.

38.

Ledesma

R. D.

Valero-Mora

(2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research & Evaluation, 12(2), 1-11.

39.

Lejuez

C. W.

Kahler

C. W.

Brown

R. A.

(2003). A modified computer version of the Paced Auditory Serial Addition Task (PASAT) as a laboratory-based stressor. The Behavior Therapist, 26, 290-293.

40.

Leyro

T. M.

Zvolensky

M. J.

Bernstein

(2010). Distress tolerance and psychopathological symptoms and disorders: A review of the empirical literature among adults. Psychological Bulletin, 136, 576-600.

41.

Linehan

M. M.

(1993). Skills training manual for treating borderline personality disorder. New York: Guilford Press.

42.

Lynch

T. R.

Hempel

R. J.

Dunkley

(2015). Radically Open-Dialectical Behavior Therapy for disorders of over-control: Signaling matters. American Journal of Psychotherapy, 69, 141-162.

43.

Lynch

T. R.

Mizon

G. A.

(2011). Distress over-tolerance and distress intolerance: A behavioral perspective. In Zvolensky

M. J.

Bernstein

Vujanovic

A. A.

(Eds.), Distress tolerance: Theory, research, and clinical applications (pp. 52-79). New York, NY: Guilford.

44.

Muthén

L. K.

Muthén

B. O.

(1998-2012). Mplus user’s guide (Version 7). Los Angeles, CA: Author.

45.

Patrick

C. J.

Curtin

J. J.

Tellegen

(2002). Development and validation of a brief form of the Multidimensional Personality Questionnaire. Psychological Assessment, 14, 150-163.

46.

Pyszczynski

Greenberg

(1987). Self-regulatory perseveration and the depressive self-focusing style: A self-awareness theory of reactive depression. Psychological Bulletin, 102, 122-138.

47.

Quinn

E. P.

Brandon

T. H.

Copeland

A. L.

(1996). Is task persistence related to smoking and substance abuse? The application of learned industriousness theory to addictive behaviors. Experimental and Clinical Psychopharmacology, 4, 186-190.

48.

Rice

K. G.

Van Arsdale

A. C.

(2010). Perfectionism, perceived stress, drinking to cope, and alcohol-related problems among college students. Journal of Counseling Psychology, 57, 439-450.

49.

Samuel

D. B.

(2011). Assessing personality in the DSM-5: The utility of bipolar constructs. Journal of Personality Assessment, 93, 390-397.

50.

Samuel

D. B.

Widiger

T. A.

(2011). Conscientiousness and obsessive-compulsive personality disorder. Personality Disorders: Theory, Research, and Treatment, 2, 161-174.

51.

Sargeant

M. N.

Daughters

S. B.

Curtin

J. J.

Schuster

Lejuez

C. W.

(2011). Unique roles of antisocial personality disorder and psychopathic traits in DT. Journal of Abnormal Psychology, 120, 987-992.

52.

Serpell

Waller

Fearon

Meyer

(2009). The roles of persistence and perseveration in psychopathology. Behavior Therapy, 40, 260-271.

53.

Shafran

Mansell

(2001). Perfectionism and psychopathology: A review of research and treatment. Clinical Psychology Review, 21, 879-906.

54.

Shapiro

D. N.

Chandler

Mueller

P. A.

(2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213-220.

55.

Simons

J. S.

Gaher

R. M.

(2005). The Distress Tolerance Scale: Development and validation of a self-report measure. Motivation and Emotion, 29, 83-102.

56.

Skinner

H. A.

Allen

B. A.

(1982). Alcohol dependence syndrome: Measurement and validation. Journal of Abnormal Psychology, 91, 199-209.

57.

Tellegen

(1982). Brief manual for the multidimensional personality questionnaire. Unpublished manuscript, University of Minnesota, MN.

58.

Vujanovic

A. A.

Bonn-Miller

M. O.

Potter

C. M.

Marshall

E. C.

Zvolensky

M. J.

(2011). An evaluation of the relation between distress tolerance and posttraumatic stress within a trauma-exposed sample. Journal of Psychopathology and Behavioral Assessment, 33, 129-135.