Abstract
As part of a broader project to create a comprehensive self-report measure for the Hierarchical Taxonomy of Psychopathology consortium, we developed preliminary scales to assess internalizing symptoms. The item pool was created in four steps: (a) clarifying the range of content to be assessed, (b) identifying target constructs to guide item writing, (c) developing formal definitions for each construct, and (d) writing multiple items for each construct. This yielded 430 items assessing 57 target constructs. Responses from a heterogeneous scale development sample (N = 1,870) were subjected to item-level factor analyses based on polychoric correlations. This resulted in 39 scales representing a total of 213 items. The psychometric properties of these scales replicated well across the development sample and an independent validation sample (N = 496 adults). Internal consistency analyses established that most scales assess relatively narrow forms of psychopathology. Structural analyses demonstrated the presence of a strong general factor. Additional analyses of the 35 nonsexual dysfunction scales revealed a replicable four-factor structure with dimensions we labeled Distress, Fear, Body Dysmorphia, and Mania. A final set of analyses established that the internalizing scales varied widely—and consistently—in the strength of their associations with neuroticism and extraversion.
Keywords
The Hierarchical Taxonomy of Psychopathology (HiTOP) is an international consortium that is using data from structural studies to construct a quantitative nosological system (Kotov et al., 2017). Assessment is a crucial part of this effort. HiTOP-consistent measures are needed in the clinic to provide practitioners with a viable alternative to assessment methods based on traditional diagnostic systems (Ruggero et al., 2019). HiTOP-congruent measures also are needed in structural research to clarify the classification of conditions whose placement currently is inconsistent or unclear (e.g., mania, borderline pathology).
To facilitate future structural work, the group concluded that it was necessary to create a new, comprehensive self-report measure. Self-report scales have been developed to measure virtually all forms of psychopathology, and these instruments potentially could be combined to create a reasonably comprehensive assessment battery. However, they employ a diverse array of instructions, time frames (e.g., past 2 weeks, past month, past 12 months) and response formats (e.g., agree–disagree, frequency-based, intensity-based). These different methods complicate structural research by creating artifactual method effects (i.e., all other things being equal, variables assessed using the same format and time frame will be more strongly related than those based on different methods; see Johnson et al., 2011; Reio, 2010). Consequently, it is beneficial to assess all types of psychopathology using the same method.
Given the size and complexity of this task, the HiTOP Measurement Workgroup decided to construct this measure in multiple stages. In Phase 1 of the project, five different subgroups were created to develop preliminary sets of scales falling within the major spectra identified in structural research (Kotov et al., 2017); specifically, these five subgroups were charged with the assessment of (a) internalizing, (b) disinhibited and antagonistic externalizing, (c) thought disorder, (d) detachment, and (e) somatoform psychopathology. Each group was given substantial autonomy in creating preliminary Phase 1 scales, although efforts were made to minimize the overlap between them. In Phase 2 of the project, the preliminary scales from these five groups will be subjected to joint analyses to begin the process of creating a comprehensive measure.
This article documents the Phase 1 activities of the Internalizing Group. The term internalizing originated in the child psychopathology literature to describe forms of psychopathology that involve “problems within the self” (Achenbach, 1966, p. 10), as opposed to maladaptive interactions with the environment. Structural researchers subsequently adopted this term to describe a spectrum of symptoms and disorders characterized primarily by emotional dysfunction and/or related forms of behavioral avoidance (Watson et al., in press). The internalizing spectrum consistently emerges in structural studies of psychopathology (Watson et al., in press); it consists of several subfactors, including distress (e.g., major depression, dysthymic disorder, generalized anxiety disorder [GAD], posttraumatic stress disorder [PTSD]), fear (e.g., panic disorder, agoraphobia, social phobia, specific phobia), eating pathology (e.g., bulimia nervosa, anorexia nervosa, binge eating disorder), and sexual problems (e.g., low sexual desire, sex-related distress, difficulties with arousal). Finally, mania is interstitial, showing important connections to both internalizing and thought disorder (Watson et al., in press). This structural evidence forms the basis of the current research.
Development of the Item Pool
Establishing the Scope of the Domain
The item pool was developed in four stages. The initial stage (conducted in September/October 2017) focused on clarifying the scope of the assessment project. We used Figure 2 (which displays disorders) and Figure 3 (which lists symptom components and maladaptive traits) in Kotov et al. (2017) as the starting point for identifying the range of content to be subsumed within internalizing. Based on these sources, we established that the measure should assess content related to the depressive disorders, the anxiety disorders (including symptoms related to GAD, social phobia, panic disorder, agoraphobia, separation anxiety disorder, and specific phobia), the trauma-related disorders (especially PTSD), obsessive–compulsive disorder (OCD), sexual problems, irritability, and the maladaptive traits shown in Figure 3 of Kotov et al. (2017; e.g., hostility, perseveration, separation insecurity). As noted earlier, structural studies have indicated that eating pathology also clearly falls within the internalizing spectrum (Forbush et al., 2010; Forbush & Watson, 2013; Kotov et al., 2017). For purely practical reasons, however, eating pathology was assigned to the Somatoform Group due to its members having greater relevant expertise. Consequently, we will not be discussing eating pathology in this article. To ensure continuity of efforts between the two groups, the Chair of the Internalizing Group (DW) was included as a member of the Somatoform Group.
Three other considerations merit some discussion. First, as noted previously, mania is an interstitial construct between internalizing and thought disorder in the HiTOP model (Watson et al., in press). The Measurement Workgroup decided that the Internalizing Group would assume primary responsibility for assessing mania. However, some Thought Disorder Group members contributed to the writing of mania items. The Thought Disorder Group also collected its own data on these items and participated in analyses of this interstitial content.
Second, sleep-wake disorders also show important links to internalizing (e.g., Koffel et al., 2015, 2016; Koffel & Watson, 2009; Watson et al., 2012). The HiTOP model of internalizing already includes content related to insomnia and lassitude/fatigue, which help anchor two major dimensions of sleep pathology (Koffel, 2011). To expand our coverage of sleep pathology, we decided to assess nightmares as well. Nightmares have been linked to indicators of stress, anxiety, panic, depression, and suicidality; however, they exhibit particularly strong and consistent associations with PTSD (e.g., Koffel, 2011; Koffel et al., 2016; Levin & Nielsen, 2007; Watson et al., 2015).
Third, the obsessive–compulsive and related disorders (OCRDs) are a new diagnostic class in the Diagnostic and Statistical Manual of Mental Disorders–Fifth edition (DSM-5; American Psychiatric Association, 2013). OCD is the only OCRD currently included in the HiTOP model. The other OCRDs—hoarding disorder, body dysmorphic disorder, trichotillomania/hair-pulling disorder, and excoriation/skin-picking disorder—have not been examined in major structural studies of psychopathology, so their placement currently is unclear. Because OCD already was included in internalizing, we decided to assess content related to the other OCRDs as well. Based on recent factor analytic work (Watson et al., 2018), we included five specific OCRD constructs in our item pool: body dissatisfaction, body preoccupation, excoriation, hoarding, and trichotillomania.
Identifying the Target Constructs
In the second stage (conducted in October/November 2017), we identified the specific target constructs to be assessed in our measure. In this phase of the process, we opted for splitting (i.e., modeling narrowly defined symptom dimensions), as opposed to lumping (i.e., focusing on broader constructs). Consistent with recommended psychometric guidelines (Clark & Watson, 2019; Watson, 2012), our goal here was to include multiple markers to define all of the symptom dimensions that potentially could emerge in subsequent structural analyses.
Again, the figures provided in Kotov et al. (2017) provided a starting point for compiling these constructs. These HiTOP figures were supplemented by consulting the structural/assessment evidence establishing the existence of specific symptom dimensions within depression (e.g., Dornbach-Bender et al., 2017; Waszczuk et al., 2017; Watson et al., 2012), GAD (e.g., Mahoney et al., 2016), PTSD (e.g., Dornbach-Bender et al., 2017; Gootzeit et al., 2015; Waszczuk et al., 2017; Watson et al., 2012; Yufik & Simms, 2010), OCD (e.g., Dornbach-Bender et al., 2017; Foa et al., 2002; Waszczuk et al., 2017; Watson et al., 2012; Watson & Wu, 2005), social anxiety (e.g., Dornbach-Bender et al., 2017; Waszczuk et al., 2017; Watson et al., 2012), panic/agoraphobia (e.g., Dornbach-Bender et al., 2017; Waszczuk et al. 2017), specific phobia (e.g., Cutshall & Watson, 2004; Dornbach-Bender et al., 2017; Waszczuk et al., 2017; Watson et al., 2012), mania (e.g., Dornbach-Bender et al., 2017; Ruggero et al. 2014; Waszczuk et al., 2017; Watson et al., 2012), and sexual problems (e.g., Forbes et al., 2016a, 2016b). Finally, individual Group members were encouraged to suggest additional constructs in their areas of expertise.
This process eventually produced a set of 57 target constructs to be modeled in the initial item pool. These 57 constructs are shown in Table 1, along with a sample item for each one (the item writing process is described later). Table 1 also classifies these constructs into general areas of content (e.g., anxiety, sleep, depression). It should be emphasized that this classification scheme was introduced purely for the sake of convenience (most notably, to facilitate locating constructs within the model) and has no broader significance. For example, in addition to being a sleep-wake disorder in DSM-5 (American Psychiatric Association, 2013), insomnia also is a symptom of several other disorders—including major depression, GAD, and PTSD—and could have been placed in any one of several different categories.
Measurement Model: The 57 Target Constructs in the Item Pool.
Note. PTSD = posttraumatic stress disorder; OCD = obsessive–compulsive disorder; OCRDs = obsessive–compulsive and related disorders.
Writing Construct Definitions
The third stage (conducted in December 2017/January 2018) involved an iterative process of writing brief formal definitions for each target construct. During this process, Group members refined definitions falling within their areas of expertise. These construct definitions subsequently served as a guide for item writing. Appendix A (included in the online supplemental materials) provides the final definitions for all 57 constructs. Following the typical practice in internalizing symptom assessment (e.g., Watson et al. 2012), these constructs were conceptualized as unipolar in nature, such that all items were written in the keyed direction. In this regard, it should be noted that manifestations of low positive affect (“Anhedonia/low well-being”) and high positive affect (“Well-Being/high positive affect”) were modeled as two separate unipolar constructs.
Creating the Item Pool
The fourth stage was the formal creation of the item pool. This stage can be broken down into two main phases. Item writing was the focus of the first phase, which was conducted from January to May 2018. Group members were provided with the instructions and response format to be used in the measure, and were asked to “write as many nonredundant items as you can” for those constructs falling within their areas of expertise. Six Group members (MF, RK, HFLA, CR, MS, and DW) contributed items at this stage. This item writing process yielded a total of 1,110 items. Supplemental Table S1 (available online) presents the number of items written for each target construct in the preliminary item pool. The number of items varied considerably across constructs, ranging from 8 (Sexual Pain) to 38 (Irritability/Hostility), with a mean of 19.5 items. We will refer to these 57 sets of rationally organized items as homogenous item composites or HICs (see Clark & Watson, 2019; Hogan, 1983; Watson et al., 2012).
This preliminary pool was unwieldy and contained a large number of clearly redundant items. Consequently, the second stage (conducted in June/July 2018) involved selecting the best items for each HIC. Group members were asked to read through each HIC and to indicate the items that were the best indicators of the target construct, taking into account such issues as (a) ensuring comprehensiveness of coverage and (b) item redundancy/overlap. Raters were allowed to nominate as many—or as few—items as they saw fit, and could select a maximum of five items as representing the best overall indicators of the construct.
To facilitate this rating task, the preliminary item pool was divided into three spreadsheets, each containing roughly 300 to 400 items. The first set included the mania, maladaptive trait, and sexual dysfunction HICs (401 items overall); six Group members (MF [sex items only], RK, HFLA, CR, MS, DW) rated these items. The second set included the sleep, depression, and PTSD HICs (310 items overall); five Group members (MF, RK, HFLA, PP, and DW) evaluated these items. The final group included the anxiety, OCD, and OCRD HICs (399 items overall); five Group members (MF, RK, HFLA, MS, DW) rated these items.
For each target construct, we selected the highest rated items for inclusion in the final item pool. This yielded a final pool of 430 items. Supplemental Table S1 (available online) shows the number of items contained in each reduced HIC in the final item pool. The number of retained items ranged from 6 to 11, with a mean of 7.5 items per HIC. Appendix A (included in the online supplemental materials) presents all of the retained items for each HIC.
Participants and Procedure
Development Sample
Assessment Protocol
We collected data in the Development sample between September 2018 and May 2019. The assessment protocol was divided into four sections. The first section asked the respondents to provide basic demographic information, including gender, age, and race/ethnicity. To assess their clinical status, participants also were asked if they were “currently receiving psychological counseling/therapy for mental health issues” or had received “psychological counseling/therapy for mental health issues in the past.”
The second section contained the 30-item short form of the Big Five Inventory–2 (BFI-2-S; Soto & John, 2017). The BFI-2-S assesses the domains and facets of the five-factor model (FFM) of personality. Each BFI-2-S domain includes three two-item facet scales. The items are rated on a 5-point scale ranging from disagree strongly to agree strongly. We report results on the BFI-2-S Negative Emotionality and Extraversion domain scores.
The last two sections included the 430 items in the final item pool. Participants were asked whether there had been “significant times during the last 12 months during which the following statements applied to you”; they responded using a 4-point scale (not at all, a little, moderately, a lot). The internalizing items were split into two sections because of the skip-structure in the sexual dysfunction items; these items therefore were separated from the others. The third section of the protocol contained all 395 nonsexual dysfunction items, plus four “instructed response” validity checks (Meade & Craig, 2012) that were designed to make sure respondents were paying attention (e.g., “Please select ‘Moderately’ for this question”); the items were presented in a random order with items from the different HICs interspersed together.
The fourth section included the 35 sexual dysfunction items, plus three screening questions asking about sexual activity during the past year. All participants were presented with an initial set of 15 questions that assessed their level of interest in sex, as well as thoughts and emotions about sexual activity. The subsequent blocks of items were split up into experiences that required (a) sexual activity, including passionate kissing, foreplay, or masturbation (e.g., “I felt desire during sexual activity”; 12 items), (b) attempting to reach orgasm (e.g., “I was unable to reach orgasm”; 4 items), or (c) experiencing orgasm (e.g., “my orgasms were pleasurable”; 4 items). Each of these blocks began with a screening question asking whether or not the respondent had the relevant experience in the past 12 months. If they answered “No” to a screening question, they were instructed to skip to the end of the protocol.
Participants
We collected data from undergraduate students at the University of Notre Dame (n = 543), the University of North Texas (n = 782), and the University of Toronto Scarborough (n = 134). In addition, we used two different crowdsourcing platforms to recruit participants. First, we collected data from 328 adult participants via the recruiting service provided by Qualtrics. To be eligible for the study, interested participants completed an initial screening questionnaire indicating that they currently were receiving psychological treatment. Consequently, all of the Qualtrics participants currently were in psychological treatment (see Table 2). Second, we collected responses from 320 adult participants using the Prolific recruiting platform. To be eligible for the study, respondents indicated that they either were currently receiving psychological treatment or had received treatment in the past. Thus, the large majority of the Prolific participants had some history of psychological treatment. However, 54 respondents (16.9%) subsequently denied any history of psychological treatment when they completed the demographic items. Nevertheless, participants in both crowdsourcing subsamples reported higher levels of psychopathology than the college students.
Demographic Breakdown of the Development and Validation Samples.
Note. UNT = University of North Texas; UND = University of Notre Dame; UTS = University of Toronto–Scarborough; VALID = validation sample.
This question was not asked in this sample.
We collected a total of 2,107 observations across these five subsamples. We subsequently dropped (a) 146 participants who failed one or more of our four validity checks and (b) 81 participants who failed to complete at least 80% of the items in the main internalizing item pool. This yielded a final total of 1,870 participants in the Development sample (see Table 2 for the final number of observations in each subsample).
Table 2 provides demographic information for the final set of participants in each subsample. Compared with the students, respondents in the two crowdsourcing subsamples were substantially older (for Qualtrics, Mage = 42.4 years; for Prolific, Mage = 30.8 years) and had a higher proportion of men (44.8% in the Qualtrics subsample, 43.7% in the Prolific subsample).
Validation Sample
Assessment Protocol
We will test the generalizability of our findings using responses collected from a separate Validation sample (n = 502) that was recruited via Prolific; these data were collected in July 2020. These participants completed the 198 items contained in the 35 nonsexual dysfunction scales that were created during the Development phase (to be described subsequently); the sexual dysfunction items were not assessed in this sample. In addition, participants completed the Neuroticism (48 items) and Extraversion (41 items) domain scales from the Faceted Inventory of the Five-Factor Model (FI-FFM; Watson et al., 2019). The FI-FFM items are rated on a 5-point scale ranging from strongly disagree to strongly agree.
Participants
The Validation sample was recruited as part of a study examining the overlap between internalizing and somatoform psychopathology. It consisted of two subsamples. First, 252 participants endorsed a screening item indicating that they had visited a doctor’s office for a possible neurological problem at some point in their lifetime. The remaining 250 participants reported a lifetime history of psychological treatment. An inspection of their Prolific ID numbers revealed that 6 participants also were included in the Development sample; these participants were dropped, yielding a final sample of 496 nonoverlapping respondents.
Table 2 reports demographic information for this final set of participants. It is noteworthy that the two Prolific groups were very similar with regard to age (Mage = 30.8 and 31.7 years in the Development and Validation samples, respectively), gender (54.7% and 57.3% women, respectively), and treatment status (current treatment = 26.4% and 30.2%, respectively; past treatment history = 80.5% and 78.8%, respectively).
Scale Development Analyses
Overview
All scale-level analyses used Pearson correlations, whereas all item-level analyses employed polychoric correlations to account for the ordinal nature of the data. We begin this section by discussing the development of preliminary internalizing spectrum scales based on the 395 items included in the main section of the assessment protocol. As discussed earlier, the use of screening questions introduced systematic patterns of missingness into the sexual dysfunction items. Consequently, we describe the development of the sexual dysfunction scales separately.
Main Internalizing Analyses
Preliminary Analyses
As noted previously, we dropped participants in the Development sample who failed to complete at least 80% of the items in the main internalizing section. The large majority of the remaining participants had no missing data. For those with some missing data, we used Proc MI in SAS 9.4 (with 11 imputations) to complete the missing responses.
The main scale development analyses were conducted between June and September 2019. Our first major decision concerned how best to use data from the various Development subsamples. We began by aggregating the responses from the two crowdsourcing (n = 646) and three student subsamples (n = 1,224). Next, we created two random student subsamples using Proc Surveyselect in SAS 9.4. At this point, we had three subsamples of roughly equal size. We then conducted a series of analyses to determine whether these three subsamples produced similar findings. These analyses revealed that the subsamples produced results that were broadly similar to one another, while also demonstrating some nontrivial differences.
To illustrate this key point, we report principal factor analyses of the 52 nonsexual HICs in each subsample, using squared multiple correlations as the initial communality estimates. In each case, we extracted four factors and rotated them to oblique simple structure using Promax. The loadings from these solutions are presented in Supplemental Tables S2 through S4 (available online). Three factors—which we labeled Distress (e.g., Depressed Mood, Numbing, Nightmares), Fear (e.g., Situational Phobia, Public Spaces Anxiety), and Mania (e.g., Euphoric Mood, Grandiosity)—appeared to replicate well across all three subsamples, whereas the fourth factor did not.
We quantified factor similarity by computing comparability coefficients (Everett & Entrekin, 1980; Finn, 1986). Comparability coefficients involve deriving regression-based factor scoring weights for each solution (Everett & Entrekin, 1980; Finn, 1986; Gorsuch, 1983), which then can be applied to every data set. Everett (1983) suggested that comparability coefficients ≥.90 indicate that the same factors emerged across solutions. In our data, comparability coefficients ranged from .950 to .992 (median = .974) for the Distress factor, from .953 to .989 (median = .972) for the Fear factor, and from .916 to .992 (median = .958) for the Mania factor. Thus, these three factors replicated quite well across all three subsamples. In contrast, comparability coefficients for the fourth factor ranged from only .683 to .984 (median = .808), indicating poor replicability across solutions.
Based on these preliminary analyses, we concluded that the findings from the three subsamples were sufficiently similar to justify combining them into a single overall analysis. We therefore aggregated the responses into a single data set (N = 1,870) for all subsequent analyses.
Primary Analyses
The main scale development process proceeded in four basic steps. The overall pool of 395 items was too large to be subjected to item-level factor analyses. Accordingly, following the general data analytic plan developed by the HiTOP Measurement Workgroup, the first step was to conduct structural analyses based on the rational HICs. The goal here was to identify smaller subsets of closely related constructs that then could be subjected to more manageable item-level analyses. Once again, we conducted a principal factor analysis of the 52 nonsexual HICs, this time using the overall Development sample. Given that our goal was to divide our large item pool into more manageable subdomains, we sought to identify the maximum number of interpretable factors. This analysis revealed four meaningful factors.
The Promax loadings from this four-factor solution are presented in Table 3. Based on their core content, these four factors can be labeled Distress, Fear, Social/Somatic Anxiety, and Mania, respectively. Based on their loadings, we assigned the HICs to corresponding subdomains; note that because of the large size of the Distress factor, HICs with salient loadings on other factors (e.g., Well-Being) were assigned to other subdomains. At this point, the Distress subdomain was defined by 26 HICs representing 194 items; Fear included 11 HICs representing 86 items; Social/Somatic Anxiety was marked by 9 HICs representing 68 items; Mania was defined by 6 HICs representing 47 items.
Promax Factor Loadings of the 52 Rational HICs in the Development Sample.
Note. N = 1,870. Loadings with an absolute value ≥.40 are in bold.
Included in the Distress subdomain. bIncluded in the Fear subdomain. cIncluded in the Social/Somatic Anxiety subdomain. dIncluded in the Mania Subdomain. HIC = homogenous item composites.
Highest loading (absolute value within ± .01) in row.
The Distress subdomain still was larger than desirable for item-level analyses. We therefore subjected the 26 Distress HICs to an additional principal factor analysis. This analysis revealed two meaningful subfactors, which we labeled Core Distress and Panic/PTSD. The Promax loadings from this analysis are shown in Table 4. Core Distress consisted of 14 HICs representing 104 items; Panic/PTSD was defined by 12 HICs representing 90 items.
Promax Loadings of the 26 Distress HICs in the Development Sample.
Note. N = 1,870. Loadings ≥ .40 are in bold.
Included in the Core Distress subdomain. bIncluded in the Panic/PTSD subdomain. HIC = homogenous item composites; PTSD = posttraumatic stress disorder.
Higher loading (absolute value within ± .01) in row.
Based on these initial results, we had carved our item pool into five subdomains, ranging in size from 47 items (Mania) to 104 items (Core Distress). The second step in the scale development process was to conduct a series of item-level factor analyses within each of these subdomains. Consistent with the general data analytic plan developed by the HiTOP Measurement Workgroup, we sought to identify the maximum number of factors/scales that could be differentiated in our data. We began with an overall analysis of the items included within each subdomain. This initial analysis was supplemented by structural analyses of smaller subgroups of items to determine whether broader dimensions could be decomposed into more specific factors. These analyses produced a provisional set of 35 scales.
The third step focused on improving the discriminant validity of the scales by eliminating problematic items. This involved (a) identifying highly correlated scales [defined as correlating ≥.75] and (b) factoring their items in a single joint analysis; items with salient loadings on both factors were candidates for elimination at this stage. At the conclusion of this stage, the 35 provisional scales included a total of 258 items.
It should be noted that discriminant validity was a much greater concern in some areas than in others. This can be illustrated using the HICs that were included in each subdomain. The mean correlations (after r-to-z transformation) of the HICs within the Mania, Fear and Social/Somatic Anxiety subdomains were .50, .50, and .54, respectively. In contrast, the mean correlations within the two subdomains carved from the initial Distress factor were much higher: .65 for Panic/PTSD and .72 for Core Distress. The Core Distress subdomain clearly represented the most general problem vis-à-vis discriminant validity. In fact, seven of its constituent HICs—Anhedonia, Anxious Mood, Depressed Mood, Lassitude, Numbing, Worry, and Worthlessness/ Guilt—had correlations ranging from .71 to .89, with a mean value of .81.
The fourth step was to refine the scales further by eliminating redundant items wherever possible, while keeping a minimum of 3 items per scale. Item redundancy was operationally defined as polychoric correlations ≥.80. A total of 60 items were dropped due to redundancy, producing a final set of 198 items. After the completion of this final stage, no scale correlations were ≥.75. Only four item pairs (out of a total of 19,503 item correlations) had polychoric correlations ≥.80, all of them in the Excoriation and Trichotillomania scales.
Thus, the scale development process ultimately yielded 35 scales consisting of 198 items. The final scales are listed in Table 5 (along with the sexual dysfunction scales, which are discussed in the following section). Supplemental Table S5 (available online) displays the item composition of the scales vis-à-vis the original rational HICs. Most of the scales essentially represent reduced versions of the original HICs (see online Supplemental Table S5). However, nine scales are more complex. Five scales are blends of two HICs: Agoraphobia combines items from Enclosed Spaces Anxiety and Public Spaces Anxiety; Body Dissatisfaction includes content from Body Dissatisfaction and Body Preoccupation; Cognitive Problems contains items from Cognitive Problems and Hyperactive Cognition; Social Anxiety subsumes content from Interactive Anxiety and Performance Anxiety; and Traumatic Reactions combines items from Traumatic Avoidance and Traumatic Intrusions. The four remaining scales—Anhedonic Depression, Anxious Worry, Euphoric Energy, and Grandiosity—contain items from three or more HICs.
Average Interitem Correlations of the Preliminary Internalizing Spectrum Scales.
Note. N = 1,870 (Development sample), 496 (Validation Sample). Values shown are Pearson correlations. The number of items for each scale is in parentheses.
Sexual Dysfunction Analyses
Analyses of the sexual dysfunction items were conducted in September/October 2019. As noted earlier, these items contained systematic patterns of missingness. This patterned missingness was handled using full information maximum likelihood estimation for item-level analyses, and the mean of nonmissing responses for scale-level analyses. In addition to conducting analyses using the overall sample, we also analyzed men and women separately, given sex differences in each of the assessed domains of sexual function (e.g., Basson, 2000).
Preliminary analyses of the five rational HICs indicated a one-factor solution in the full sample (N = 1,862) and for the male (n = 585) and female (n = 1,256) subsamples separately, although low orgasmic function was a weak indicator of the factor (λs ≤ .21 vs. λs > .6, for all other indicators), consistent with its weak correlations (r < .30) with the other domains. Low orgasmic function was consequently treated as an orphan HIC and analyzed separately.
We conducted item-level analyses of the low sexual desire, difficulties with arousal, sexual pain, and distress related to sexual dysfunction items (n = 27) in an exploratory factor analysis framework, treating them as categorical indicators using weighted least square mean and variance adjusted estimation and Geomin rotation; parallel analysis (Horn, 1965) was based on treating the items as continuous indicators using maximum likelihood ratio estimation. A total of 1,862 participants were included in these analyses, as 8 respondents (0.4%) of the full sample were missing responses on all items.
Parallel analysis indicated four factors in the full sample, which also corresponded to the most interpretable solution. We named these factors Distress and Difficulties with Arousal (e.g., “I felt guilty about sexual difficulties”; “It was hard for me to become sexually aroused”), Low Sexual Desire (e.g., “I easily lost interest in sexual activity”), Solo Sexual Desire (e.g., “I wanted to engage in sexual activity by myself”), and Sexual Pain (e.g., “ I experienced unwanted pain during sex”). Highly similar factors emerged in the analyses of both women and men. After removing redundant items (again, defined as polychoric correlations ≥.80), we created four initial scales: Distress Related to Sexual Dysfunction (four items), Low Sexual Desire (four items), Solo Sexual Desire (three items), and Sexual Pain (four items). However, we subsequently dropped the Solo Sexual Desire scale, as it was weakly related to the other scales and essentially unrelated to BFI-2-S Negative Emotionality, suggesting that it was capturing neither psychopathology in general, nor difficulties with sexual functioning specifically.
The orgasmic function items (n = 8) were analyzed in the same way, based on the subset of participants who had responded to at least one of them (n = 1,538). Parallel analysis indicated two factors in the full sample, which corresponded with two interpretable factors that converged well in the male (n = 554) and female (n = 966) subsamples. We labeled these factors Low Orgasmic Function (e.g., “I found it difficult to reach orgasm”) and Pleasurable Orgasm (e.g., “My orgasms were pleasurable”). Each factor had three clear markers in all three solutions. However, paralleling the situation with Solo Sexual Desire, subsequent analyses indicated that Pleasurable Orgasm was nonpathological in nature, given that it was weakly (and sometimes negatively) correlated with the other scales and actually had a significant negative correlation with BFI-2-S Negative Emotionality. It therefore was dropped. Consequently, the scale development process produced four scales—Distress Related to Sexual Dysfunction, Low Sexual Desire, Sexual Pain, and Low Orgasmic Function—with a total of 15 items (see Table 5).
Scale Properties
Internal Consistency
In this section, we report results that help establish the nature and correlates of these preliminary internalizing scales. We begin by examining their internal consistency. Table 5 presents average interitem correlations (AICs) for each scale in the Development and Validation samples; these values are presented as Pearson correlations, which are most commonly reported in the assessment literature.
Assessment experts recommend that scale items should be moderately correlated with one another. Specifically, Clark and Watson (1995, 2019) recommend AICs in the .15 to .50 range. Clark and Watson (1995, 2019) further emphasize that the optimal AIC depends on the generality versus specificity of the assessed construct: Scales assessing very broad dimensions ideally should have AICs toward the bottom of this range, whereas measures of narrower constructs should have AICs toward the top of this range. Given that we sought to identify the maximum number of dimensions that clearly could be differentiated in our data (i.e., we opted for splitting rather than lumping), most of our scales should tap narrow constructs and have AICs closer to the top of this range.
In the Development sample, 15 of 39 scales (38.5%) had AICs within this optimal range (values ranged from .377 to .492), and an additional eight scales (20.5%) had slightly higher AICs ranging from .507 to .549. Overall, the AICs ranged from .377 to .687, with a median value of .533. In the Validation sample, 15 of 35 scales (42.9%) had AICs within the optimal range (values ranged from .279 to .487), and an additional eight scales (22.9%) had slightly higher AICs ranging from .502 to .549. Overall, the AICs ranged from .279 to .728 in this sample, with a median value of .507. As expected, most of these preliminary scales assess narrow dimensions reflecting specific forms of psychopathology.
Finally, it is worth noting that some scales consistently had much higher AICs than others. We quantified this assertion by computing a column-vector correlation for the 35 scales assessed in both samples. This vector correlation was .71, indicating a strong level of consistency across the two samples.
Internal Structure
General Factor Solution
Next, we examined the internal structure defined by the scales. These structural analyses were designed to answer two questions. First, do the scales define a single general factor? If so, this would provide some preliminary evidence that they all are indicators of the same overarching domain (viz., internalizing). We examined this issue by conducting a separate principal factor analysis of the scales included in each sample.
Table 6 presents the loadings from these general factor solutions. These results confirm the existence of a strong general factor in both samples. Thirty-three scales (84.6%) and 34 scales (97.1%) had loadings ≥.50 in the Development and Validation samples, respectively. Indicators of distress and negative emotionality tended to be particularly strong markers of this general dimension. For example, Anhedonic Depression, Shame/Guilt, Anxious Worry, Angry Hostility, and Separation Insecurity all had loadings ranging from .77 to .80 in the Development sample, and from .72 to .79 in the Validation sample. Panic, agoraphobia, PTSD, and OCD symptoms also were strong markers of this factor.
Loadings of the Internalizing Scales on a General Factor.
Note. N = 1,475 (Development sample), 496 (Validation sample). Loadings with an absolute value ≥.40 are in bold.
We examined the consistency of this general factor by computing Tucker’s (1951) congruence coefficient (K), based on the loadings for the 35 scales that were common to both solutions. Lorenzo-Seva and ten Berge (2006) suggest that K values ≥.95 indicate good factor similarity, whereas values between .85 and .94 represent fair similarity. In our data, K = .994, establishing that the same general factor emerged in both samples.
Four-Factor Solution
Second, do these scales define meaningful subdimensions? If so, this would help explicate the nature of the basic types of content subsumed within this domain. We examined this issue using the 35 scales that were common to both samples. We conducted separate principal factor analyses in each solution. Four interpretable and well-defined factors emerged in each sample and were rotated using Promax.
Table 7 presents the loadings from these four-factor solutions (the factor correlations are reported in online Supplemental Table S6). It is noteworthy that these factors broadly resemble those that were obtained earlier from the structural analyses of the rational HICs (see Table 3). The largest factor can be labeled Distress and it clearly resembles the broad first factor shown in Table 3. This factor is defined by scales reflecting subjective distress and negative emotionality—such as Anhedonic Depression, Anxious Worry, Traumatic Reactions, Suicidality, Shame/Guilt, Angry Hostility, and Physiological Panic—that also tended to be the strongest markers of the general factor (see Table 6). We labeled the second factor Body Dysmorphia. It is similar to the third factor in Table 3 and had four consistent markers—Body Dissatisfaction, Body Focus, Perfectionism, and Appetite Gain—with loadings >.40 in both solutions. The third factor represents Fear (similar to the second factor in Table 3); it had six consistent markers—Situational Phobia, Animal Phobia, Cleaning, Blood-Injection Phobia, Agoraphobia, and Compulsive Rituals—across samples. The fourth factor is Mania, and essentially represents the same dimension as the last factor in Table 3. It had five consistent markers—Euphoric Energy, Grandiosity, Decreased Need for Sleep, Recklessness, and Well-Being—across solutions. 1
Promax Loadings of the 35 Nonsexual Dysfunction Scales in the Four-Factor Solution.
Note. N = 1,870 (Development sample), 496 (Validation Sample). Loadings with an absolute value ≥.40 are in bold. DEV = Development sample; VAL = Validation sample.
We computed comparability coefficients to quantify the level of factor similarity across samples. The Distress, Fear, and Mania factors had comparability coefficients ranging from .956 to .978 (median = .974), thereby demonstrating an impressive level of structural stability. In contrast, coefficients for the Body Dysmorphia factor were only .865 (Development sample) and .877 (Validation sample), representing a more modest level of replicability. These lower coefficients reflect the fact that this factor was broader in the Development sample (nine scales had loadings ≥.40) than in the Validation sample (only four scales had loadings ≥.40).
Correlations With Personality
Finally, we examine relations with neuroticism/negative emotionality (N/NE) and extraversion. At the diagnostic level, previous research has established that internalizing disorders are related to multiple FFM traits, but show the strongest and most consistent links to N/NE (Goldberg et al., 2009; Kotov et al., 2017; Watson et al., in press; Widiger et al., 2019). For example, Kotov et al. (2010) reported meta-analytic associations between FFM traits and 10 depressive and anxiety disorder diagnoses. These diagnoses were most strongly related to N/NE (mean r = .39), but also showed moderate negative associations with both conscientiousness (mean r = −.28) and extraversion (mean r = −.25); they were largely unrelated to openness (mean r = −.09) and agreeableness (mean r = .01). Based on these results, we would expect indicators of internalizing to be positively correlated with N/NE and—to a lesser extent—negatively related to extraversion.
At the same time, however, it is clear that some internalizing symptoms are more strongly related to N/NE than others. For instance, Watson and Naragon-Gainey (2014) examined associations between FFM domain scores and both self-reported and interview-based symptoms of anxiety, depression, and mania. They found that N/NE correlated strongly with symptoms of general distress/negative emotionality (e.g., depressed mood, anxious mood, worry); more moderately with symptoms of social phobia, panic, PTSD, and OCD; and only modestly with other types of symptoms, including agoraphobia and specific phobia.
Table 8 presents correlations between the internalizing scales and N/NE and extraversion domain scores based on the BFI-2-S in the Development sample and the FI-FFM in the Validation sample. We will begin by noting that the results were highly consistent across samples and measures. We quantified this point by computing column-vector correlations for the 35 scales common to both samples. These vector correlations were .97 for N/NE and .93 for extraversion, establishing a very strong level of replicability.
Correlations of the Preliminary Internalizing Spectrum Scales With Neuroticism/Negative Emotionality and Extraversion Domain Scores.
Note. N = 1,534 to 1,868 (development sample), 496 (validation sample). Correlations with an absolute value ≥.40 are in bold. DEVEL = Development sample; VALID = Validation sample.
Table 8 indicates that the internalizing scales varied widely in their associations with N/NE. Consistent with previous research, distress-based scales that are strong markers of the general Internalizing factor tended to have particularly substantial links to this trait. Seven scales had correlations ≥.50 with N/NE in both samples, with coefficients ranging from .52 to .75: these included Anxious Worry, Anhedonic Depression, Separation Insecurity, Shame/Guilt, Social Anxiety, Angry Hostility, and Cognitive Problems. In marked contrast, five scales—Decreased Need for Sleep, Trichotillomania, Blood-Injection Phobia, Euphoric Energy, and Grandiosity—consistently were weakly correlated with N/NE, with coefficients ranging from only −.12 to .27 across samples. More generally, indicators of mania, specific phobia, the OCRDs, and sexual dysfunction tended have relatively weak associations with N/NE.
Most scales had relatively modest associations with extraversion. Consistent with the findings of Watson and Naragon-Gainey (2014), extraversion was most strongly negatively related to Social Anxiety, with correlations of −.62 and −.45 in the Development and Validation samples, respectively. It also was moderately negatively correlated with Anhedonic Depression (rs = −.48 and −.25, respectively). Furthermore, as would be expected based on previous research (Watson et al., 2019), some scales consistently displayed positive associations with extraversion. These included Well-Being (rs = .48 and .59, respectively) and some indicators of mania (for Grandiosity, rs = .18 and .44, respectively; for Euphoric Energy, rs = .15 and .34, respectively).
Future Directions
We have described the development of 39 preliminary internalizing scales, which were created during Phase 1 of the HiTOP measure development project. As we discussed at the beginning of this paper, the development of these preliminary scales is part of a larger effort by the HiTOP consortium to create a comprehensive self-report measure of psychopathology. In Phase 2, our 213 retained items will be combined with those from the other spectra (viz., somatoform, thought disorder, detachment, and disinhibited and antagonistic externalizing) to further the development of this comprehensive measure.
Many of our preliminary scales will undergo significant modification in response to these Phase 2 analyses, and some may disappear altogether. As one example, the Somatoform Group—which, as discussed previously, was tasked with the assessment of eating pathology—has created a preliminary Body Image and Weight Concerns scale. This scale contains content (e.g., “I was dissatisfied with the shape of my body,” “I thought other people’s bodies looked better than mine”) that appears quite similar to that contained in our Body Dissatisfaction scale (e.g., “I was dissatisfied with my weight,” “I spent a lot of time comparing my appearance to that of others”). It seems very likely that these items will be collapsed into a single scale in Phase 2. As another example, the Detachment and Thought Disorder Groups both created preliminary Anhedonia scales that contain content that clearly overlaps with many of the items in our Anhedonic Depression scale. All of these items may be collapsed into a single scale as a result of the Phase 2 analyses.
The Phase 2 analyses also will help clarify where our items/scales ultimately fall within the hierarchical structure of psychopathology. To be sure, many of our preliminary scales clearly represent core manifestations of internalizing. Scales such as Anxious Worry, Anhedonic Depression, Separation Insecurity, Shame/Guilt, and Cognitive Problems are (a) strongly correlated with one another, (b) substantial markers of the general Internalizing factor (see Table 6), and (c) strongly correlated with N/NE (see Table 8). Generally speaking, these types of scales—which also are strong indicators of the Distress subdimension (see Table 7)—are likely to define internalizing in subsequent phases of the HiTOP assessment project.
In contrast, other scales performed more equivocally, such that their ultimate placement remains uncertain. These scales include indicators of mania, specific phobia, sexual dysfunction, and the OCRDs. It will be interesting to see where these scales/items are located within a comprehensive symptom-based structure of psychopathology.
Supplemental Material
sj-pdf-1-asm-10.1177_10731911211003976 – Supplemental material for The Development of Preliminary HiTOP Internalizing Spectrum Scales
Supplemental material, sj-pdf-1-asm-10.1177_10731911211003976 for The Development of Preliminary HiTOP Internalizing Spectrum Scales by David Watson, Miriam K. Forbes, Holly F. Levin-Aspenson, Camilo J. Ruggero, Yuliya Kotelnikova, Shereen Khoo, R. Michael Bagby, Matthew Sunderland, Praveetha Patalay and Roman Kotov in Assessment
Footnotes
Authors’ Note
The reported research was approved by the Institutional Review Boards at the University of Notre Dame, the University of North Texas, and the University of Toronto Scarborough.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
