Abstract
This study aimed to replicate a study by Caspi and colleagues, which proposed that the structure of psychopathology is characterized by a general psychopathology factor, in addition to smaller internalizing and externalizing factors. Our study expanded the approach of the original by using continuous adolescent data and testing additional models, including both self- and parent-reported data, to bolster the robustness of the findings. Our findings indicate that the structure of psychopathology is best characterized by a model including a general factor, in addition to smaller internalizing and externalizing factors. These results emphasize the importance of this model for understanding the structure of psychopathology. Given the increasing emphasis on the importance of, and need for, replication, the overall evidence of a general factor seems rather robust.
Traditional research on psychopathology has usually suggested a two-dimensional structure of mental disorders, with regard to both adult diagnostic data on mental disorders (Krueger, Caspi, Moffitt, & Silva, 1998; Krueger, Johnson, & Kling, 2006) and factor analytic studies of continuous measures of psychopathology in children, adolescents, and adults (Achenbach, 1966; Achenbach & Edelbrock, 1978). Over the decades, these findings have created the widely accepted and demonstrated distinction between two different, though positively correlated, factors underlying psychopathology: internalizing and externalizing problems/disorders (Forbush & Watson, 2013; Kendler, Prescott, Myers, & Neale, 2003; Slade & Watson, 2006; Vollebergh et al., 2001). More recent research has provided evidence for a third factor, labeled thought problems or dysregulation, which includes psychotic symptoms, for example. These symptoms have usually been excluded in studies on common mental disorders due to methodological and conceptual issues (Kotov, Gamez, Schmidt, & Watson, 2010; Wigman et al., 2011).
Recently, this two- (or three-) dimensional structure of mental disorders has been debated, with increasing emphasis placed on the existence of a general psychopathology factor instead of, or in addition to, the more specific dimensions. Specifically, a recent series of confirmatory factor analyses (CFAs) of longitudinal psychopathology data from the Dunedin study has challenged traditional models by providing strong support for a latent general overall severity factor, with high loadings in all individual symptom dimensions (Caspi et al., 2013). It is interesting that when taking into account this general psychopathology factor, the internalizing and externalizing factors remained, but loadings of symptoms on the internalizing and externalizing factors decreased substantially, whereas the usually positive correlation between internalizing and externalizing factors became a negative correlation. In addition, the thought-problem symptoms did not form a third latent factor in the general psychopathology model, but rather, due to their very high loadings on the general factor, appeared to be core symptoms of the general psychopathology factor instead. This general dimension was ingeniously coined the p-factor because it conceptually parallels a familiar dimension in psychology: the g-factor of general intelligence.
Caspi and colleagues are not the first to suggest an underlying general psychopathology factor. The notion of a general psychopathology factor is similar to the “index of definition” (ID; Wing, Mann, Leff, & Nixon, 1978). Also, the work of Gibbons and Hedeker and, more recently, Lahey and colleagues, for example, proposed a general psychopathology factor in addition to more domain-specific latent factors (Gibbons & Hedeker, 1992; Lahey et al., 2012; Lahey, Van Hulle, Singh, Waldman, & Rathouz, 2011). Similar to Caspi’s, these studies did not model the general psychopathology factor as a higher-order factor, but rather at the same level as the domain-specific latent factors, with all specific symptoms loading directly on the general psychopathology factor (i.e., a “bifactor” model). Nonetheless, although the various models may not be incompatible, the proposed models differ from each other in various ways. For example, Caspi and colleagues allowed the internalizing and externalizing factors to correlate, whereas the study by Gibbons and Hedeker only included internalizing symptoms, and the model proposed by Lahey did not allow for correlations between the internalizing and externalizing latent factors in the general factor model. By allowing the factors to correlate, Caspi revealed that the positive correlation between the internalizing and externalizing factors turned negative in the general factor model, which suggests that internalizing and externalizing problems are clearly distinct after controlling for general vulnerability for psychopathology.
In the current article, we attempt to replicate and expand the findings by Caspi and colleagues. Caspi’s model seems most elaborate with regard to the symptom dimensions included and matches best with our longitudinal cohort data, which maximizes comparability. Aiming at providing a close replication of Caspi’s model, we compared the same models, including the same paths and using the same model specifications. Nonetheless, our study design differs from Caspi’s approach in various ways. Whereas Caspi et al. focused on (young) adult samples, the current study uses data from a large sample of Dutch adolescents. Given that many mental disorders have their onset during early adolescence (Bernstein, Borchardt, & Perwien, 1996; Hankin et al., 1998), attempting to replicate Caspi’s findings in an adolescent sample may provide insight into whether the model as proposed by Caspi also reflects the structure of psychopathology in adolescents. In addition, we added symptoms that are known to onset before the adolescent years (e.g., attention problems) or that are highly adolescence-specific (i.e., delinquency), and we used continuous questionnaire data instead of (semi)structured diagnostic interview data. Finally, we aimed to check the robustness of the best-fitting model by adding all parent-reported symptom data corresponding to the self-report measures included in the main analyses.
Method
Sample
Our study used data from the first, second, third, and fourth wave of the TRacking Adolescents’ Individual Lives Survey (TRAILS; Ormel et al., 2012). TRAILS is a large prospective cohort study of 2,230 Dutch adolescents, followed bi- or triennially from age 11. Sample selection has been extensively described by de Winter and colleagues (2005).
Mean age was 10.5 years (SD = 0.58) at the first wave, 13.6 (SD = 0.59) at the second wave, 16.1 (SD = 0.59) at the third wave, and 19.1 (SD = 0.60) at the fourth wave. About 10% of our sample was identified as having a non-Western ethnicity. Having a non-Western ethnicity was defined as having at least one parent born in a non-Western country (N = 230 at Wave 1), of whom 20% had a parent or parents born in Surinam, 16% in the Dutch Antilles, 16% in Indonesia, 6.5% in Morocco, 5.5% in Turkey, and 36% in other non-Western countries such as Iraq, Iran, or Somalia (see also Vollebergh et al., 2005). Attrition during follow-ups was somewhat higher in males and in adolescents of non-Western ethnicity, and in those with divorced parents, low socioeconomic status (SES), low IQ and academic achievement, or poor physical health or with behavior and substance use problems (Nederhof et al., 2012). Each assessment wave was approved by the national ethical committee (CCMO, www.ccmo.nl).
Measures
Self-report
TRAILS assessed problem behaviors, as rated by adolescent subjects, using the following instruments: the Youth Self Report (YSR; T1, T2, T3), the Adult Self Report (ASR; T4), the Revised Child Anxiety and Depression Scale (RCADS; T1, T2, T3), and the Community Assessment of Psychic Experiences (CAPE; T3). The YSR has been widely used to assess self-report symptom dimensions (Achenbach, 1991). Symptom dimensions covered by the YSR, and included in the current study, are anxious-depressed, withdrawn-depressed, aggressive behavior, delinquent behavior, attention-hyperactivity problems, and thought problems. At T4, when adolescents were 19 years old on average, the ASR replaced the YSR. Although the ASR contains slightly different items (more appropriate for older adolescents) the symptoms covered were identical to the symptoms covered by the YSR. Items were scored on a 3-point Likert-type scale (0 = not true to 2 = very or often true).
The RCADS consists of a list of items measuring anxiety, obsessive-compulsive, and depression symptoms (Chorpita, Yim, Moffitt, Umemoto, & Francis, 2000; Mathyssek et al., 2013). We focused on the anxiety and obsessive-compulsive scales because these were not fully covered by the YSR/ASR. The symptom dimensions covered were generalized anxiety disorder, social anxiety, separation anxiety, panic disorder, and obsessive-compulsive disorder. Items were scored on a 4-point Likert scale (0 = never to 3 = always).
TRAILS used the CAPE positive experiences scale (20 items) to assess psychotic experiences (Wigman et al., 2011). Each item assesses (a) frequency and (b) distress associated with an experience, both scored on a 4-point Likert-type scale (0 = never/not distressed to 3 = nearly always/very distressed). For the current study, we used both the frequency and the distress variables.
Parent report
To test the robustness of our models, we added all parent-reported data that corresponded to the self-report measures of our final models. The study assessed problem behaviors as rated by the parent using the Child Behavior Checklist (CBCL; T1, T2, T3), which is one of the most commonly used parent-report questionnaires in child and adolescent psychiatric research (Achenbach, 1991; Verhulst & Achenbach, 1995). The symptom dimensions covered by the CBCL are anxious-depressed, withdrawn-depressed, aggressive behavior, delinquent behavior, attention-hyperactivity problems, and thought problems. Items were scored on a 3-point Likert-type scale (0 = not true to 2 = very or often true).
Statistical analysis
Replicating Caspi’s models in adolescent data
In line with Caspi and colleagues, we tested four models using CFA: a three-correlated-factors model including internalizing, externalizing, and thought problems (Model A; see Fig. 1); a bifactor model with internalizing, externalizing, and thought problems at the first level and a general psychopathology factor at the second level (Model B; see Fig. 1); a revised-bifactor model with internalizing and externalizing problems at the first level and the general psychopathology factor at the second level with the thought-problem factor removed from the model (Model B′; see Fig. 1); and a one-factor model (Model C; see Fig. 1). Notably, we statistically specified all models to be identical to Caspi’s models to maximize comparability. Also identical to Caspi’s models, we included an additional wave-factor into each model to expose age- and assessment-related variance.
The structure of psychopathology. Three models were tested using confirmatory factor analysis: a correlated-factors model (Model A), a bifactor model (Model B; i.e., Model A plus all symptom dimensions loading on a general factor), and a one-factor model (Model C). Model B′ shows the revised-bifactor model (i.e., Model B but without a separate thought-problem factor).
All CFA analyses were performed using robust maximum likelihood estimation in Mplus version 7.1 (Muthén & Muthén, 2007). We assessed how well each model fit the data using the chi-square value, the comparative fit index (CFI), the Tucker–Lewis index (TLI), and the root mean square error of approximation (RMSEA). Nonsignificant chi-square values normally reflect good model fit. However, when using large sample sizes this test is strongly overpowered and the fit statistics are significant. CFI and TLI values greater than .95 indicate a good fit, as well as RMSEA scores less than .05 (Bollen & Curran, 2006). The critical CFI/TLI value for an acceptable model is .90, and for RMSEA the critical value for acceptable model fit is .08.
Checking the robustness of the best-fitting models by adding parent-reported symptoms
For our best fitting models we added all parent-reported symptom data that corresponded to the self-report measures included in the main analyses (Fig. 2). For these models, we also included a reporter factor to pull out informant-related variance.
Two additional models were tested: Model ASelf+parent report (Model A, but with all available parent-report data added to the model) and Model B′self+parent report (Model B′, but with all available parent-report data added to the model).
Results
The structure of psychopathology
Replicating Caspi’s models in adolescent data
We aimed at replicating the four models tested by Caspi and colleagues in as much detail as possible. In the first model (A), we tested the hypothesis that the observed symptom dimensions are characterized by three correlated factors: internalizing (including anxious-depressed, withdrawn-depressed, GAD, social anxiety, separation anxiety, and panic disorder), externalizing (including delinquency, aggression, and attention problems), and thought problems (including thought problem symptoms, OCD, and psychotic experiences; see Fig. 1). Model fit statistics showed that the data did not fit the model adequately: χ2(723) = 5148.82, CFI = .890, TLI = .875, RMSEA = .052, 90% confidence interval (CI) = .051–.054. In addition, the standardized factor loading of aggression on the latent externalizing factor was larger than 1 and the residual variance of aggression was negative. This indicated that the three-factor model was not optimally identified and thus did not explain the structure of the symptom dimensions across adolescence well. Table 1 shows this model with standardized factor loadings and the correlations between the three latent variables: internalizing, externalizing, and thought problems. Note that caution is needed in interpreting these loadings and correlations because of the model identification problems.
Standardized Factor Loadings and Correlations of the Three-Factor Model (A) and the Revised-General-Factor Model (B′)
Next, we tested the bifactor model (B), hypothesizing that symptom dimensions reflect the three latent factors— internalizing, externalizing, and thought problems—as well as a latent general psychopathology factor (Fig. 1). The model assumed this general factor to directly influence all symptom dimensions. In line with Caspi and colleagues, we constrained the general factor to be uncorrelated with the three (correlated) first-level latent variables. We could not identify this model, which seems to be in line with Caspi and colleagues who also reported that this model was not valid for their data. Subsequently we tested a revised-bifactor model as suggested by Caspi and colleagues (Model B′; see Fig. 1 and Table 1). That is, we removed the latent thought-problem factor from the model, assuming that our identification problems with Model B were mostly due to the variance of the thought-problem factor (i.e., thought-problem symptoms, OCD, and psychotic experiences) being subsumed into the general psychopathology factor. Model fit statistics suggested that the data fit the revised model acceptably: χ2(716) = 4665.65, CFI = .902, TLI = .887, RMSEA = .050, 90% CI = .048–.051. As in Caspi’s study, we could not directly compare the three-factor and revised-bifactor models because the models are not fully nested. But based on the CFI, TLI, and RMSEA, we can conclude that the revised-bifactor model offers a slightly better and more parsimonious solution. It is important that revising the model solved the model identification problems we encountered when fitting the three-factor model to the data (i.e., negative error variance and factor loadings > 1). This suggests that the general-factor model could explain the structure of symptom dimensions across adolescence better than the three-factor model.
Comparing the factor loadings in the three-correlated-factors model versus the revised-bifactor model revealed similar information to that reported by Caspi and colleagues. We found that some symptom dimensions were more indicative of general psychopathology (the general factor) than other symptom dimensions. Overall, internalizing symptom dimensions were more indicative of the general factor than externalizing symptom dimensions. Of the externalizing symptom dimensions, attention problems (not included by Caspi) were more indicative of the general factor than aggression and delinquency; the factor loading of attention problems on the latent externalizing factor reduced by about half after taking the general factor into account. Seemingly, much of the variance initially ascribed to the latent externalizing factor was now subsumed by the general factor. In the revised-bifactor model, all factor loadings of internalizing symptom dimensions on the latent internalizing factor were substantially reduced or even disappeared compared with the three-correlated-factors model, with the largest changes found for withdrawn-depressed, and panic disorder. In addition, and identical to Caspi, we found a significant positive correlation (r = .440) between externalizing and internalizing factors in the correlated-factors model, but a negative correlation (r = −.438) between externalizing and internalizing factors in the revised-bifactor model. Finally, and like Caspi, we tested the one-factor model in which all symptom dimensions were assigned only to the general factor (Model C; see Fig. 1). This model could not be identified, which is in line with Caspi and colleagues who also reported that this model was not valid for their data.
Checking the robustness of the best-fitting models by adding parent-reported symptoms
Based on the findings presented earlier, we consider Caspi’s revised-bifactor model as the final model. For both this model and the correlated-factors model, we added all parent-reported symptom data that corresponded to the self-report measures included in the main analyses (Fig. 2). The model fitting results were as follows: results for Model Aself+parent report: χ2(1521) = 10935.87, CFI = .848, TLI = .829, RMSEA = .053, 90% CI = .052–.054; results for Model B′self+parent report: χ2(1514) = 10348.30, CFI = .857, TLI = .838, RMSEA = .051, 90% CI = .050–.052. Thus, testing and comparing Models Aself+parent report and B′self+parent report revealed similar information as for the self-report-only models (overall model fit was even somewhat better than for the self-report-only models). Path estimates are available on request.
Discussion
In the current study we replicated the findings of Caspi and colleagues, who proposed that the structure of psychopathology is reflected by a general psychopathology factor in addition to a smaller internalizing and externalizing factor. Overall, despite methodological differences (i.e., adolescent sample, continuous measures of psychopathology, and additional symptom dimensions not included by Caspi), our findings clearly and consistently replicated their findings. Our results confirmed the existence of a general psychopathology factor underlying all symptoms of psychopathology. Taking into account this general factor, the internalizing factors in particular, but also the externalizing factors, were clearly attenuated. The robustness of these findings was confirmed by additional model testing in which we added all available parent-report data to the model, thus revealing that the model remains valid when multiple informants are used to measure psychopathology.
The extent to which the specific dimensions loaded on the general factor differed both between and within the broader domains of internalizing, externalizing, and thought problems. In line with Caspi, thought problem symptoms did not form a latent thought problem factor independent from the general factor. This suggests the relative severity of thought problems, and may indicate that the general psychopathology factor should be interpreted as characterizing the overall severity of psychopathology, for which thought problems appear to be core symptoms. In contrast, with the addition of the general factor, enough variance was left to form the internalizing and externalizing factors, indicating that these factors are necessary to provide a full picture of the underlying structure of psychopathology, although the symptom loadings were very strongly attenuated. This raises the question of how the two dimensions should be interpreted once the more general psychopathological characteristics are taken into account (Caspi et al., 2013; Martel, 2013; Oldehinkel, Hartman, de Winter, Veenstra, & Ormel, 2004). Our finding that the usual positive correlation found between these two dimensions turned negative when adding the general factor seems to further complicate interpretation.
We found that, when adding the general factor, the decreases in factor loadings on the latent internalizing/externalizing factors were substantially larger for the internalizing symptom dimensions than for the externalizing. This may either reflect that internalizing problems are more pathological in general or it may just reflect that we included some adolescent specific externalizing symptoms that may naturally decrease when adolescents get older and therefore may not characterize an individual’s vulnerability for severe psychopathology. Indeed, when looking at within-domain differences, it seemed that adolescent-specific problems, such as delinquency, were not very indicative of the general factor. Adding more chronic or life-course-persistent externalizing problems (e.g., antisocial personality disorder) might further clarify whether externalizing symptom dimensions are indeed less indicative of psychopathology.
Taken together, our findings clearly and consistently confirmed that the structure of psychopathology is best reflected by a model that includes a general factor of psychopathology. According to Caspi and colleagues, higher scores on the general factor may not only indicate an increased risk for developing a mental disorder, but also that the general factor correlates more strongly with personality, life impairment, developmental history, and brain integrity, than do the internalizing and externalizing factors. In addition to specific consequences, the general factor may also have specific shared determinants such as self-control, neuroticism, and low SES, although conclusive experimental data is still lacking (Brunner, 1997; Moffitt et al., 2011; Neeleman, Bijl, & Ormel, 2004; Neeleman, Ormel, & Bijl, 2001).
Also, as extensively described by Lahey and colleagues (2011), the general factor may have a substantial genetic component. That is, genetic factors may set individuals at risk for all symptom dimensions, albeit in different degrees. This notion seems to fit well with recent genetic-evolutionary ideas about the relationship between mutational load and mental disorders as elaborated especially by Keller and Miller (2006). They argued that mental disorders reflect the inevitable mutational load on the thousands of genes underlying human behavior. The mutations involved may influence characteristics that affect the risk for any and all mental disorders. This would explain why the genes involved in the etiology of psychopathology seem to operate in a pleiotropic fashion, enhancing risk for any and all disorders rather than breeding true to any single disorder (Smoller, 2013). Keller and Miller’s (2006) watershed model of the pathways connecting upstream genes to downstream phenotypes provides an excellent model of how genes may act in a pleiotropic fashion and “contribute” to a general factor.
It is important not to reify the general factor. The general factor represents the shared variance among all psychopathology measures, and this can have different causes. One possibility is that all forms of psychopathology, to some extent, share a common etiology. Another alternative is that the factor represents dynamic processes in which one dimension raises the risk of developing another disorder or dimension, rather than a single unitary cause. Current network approaches to psychopathology assume such dynamic processes (Cramer, Waldorp, van der Maas, & Borsboom, 2010). Another alternative is that the general factor reflects distress, which is a concomitant/feature of virtually all mental disorders. Nonetheless, the findings showing a rather strong general factor suggest that there may be multiple characteristics that account for meaningful covariance amongst major forms of psychopathology. In their article, Caspi and colleagues put forward the question of whether the general factor is merely a statistical reductio ad absurdum, or rather is real and meaningful. In response, we like to answer: The general factor certainly seems useful, but how it is best interpreted remains unclear given the many possible interpretations (mutation load, early persistent adversity, dynamic processes, distress, etc.). The same is true for interpreting the internalizing and externalizing factors that remain after the general factor is taken into account. Do these factors truly represent different underlying dimensions of other forms of psychopathology, or do they reflect only different behavioral problems in coping with environmental stressors? In the absence of an answer to these questions, the current findings are best interpreted as a step toward a more parsimonious view of the structure of psychopathology than that implied by traditional differentiations between (a) categorically distinct disorders and (b) spectra or domains of disorders that include internalizing, externalizing, and (in some recent studies) thought problems.
Footnotes
Acknowledgements
We are grateful to all adolescents, their parents, and their teachers who participated in this research and to everyone who worked on this project and made it possible.
Declaration of Conflicting Interests
The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.
Funding
This research is part of the TRacking Adolescents’ Individual Lives Survey (TRAILS). Participating centers of TRAILS include various departments of the University Medical Center and University of Groningen, the Erasmus University Medical Center Rotterdam, the University of Utrecht, the Radboud Medical Center Nijmegen, and the Parnassia Bavo group, all in the Netherlands. TRAILS has been financially supported by various grants from the Netherlands Organization for Scientific Research NWO (Medical Research Council Program Grant GB-MW 940-38-011; ZonMW Brainpower Grant 100-001-004; ZonMw Risk Behavior and Dependence Grants 60-60600-98-018 and 60-60600-97-118; ZonMw Culture and Health Grant 261-98-710; Social Sciences Council Medium-Sized Investment Grants GB-MaGW 480-01-006 and GBMaGW 480-07-001; Social Sciences Council Project Grants GB-MaGW 457-03-018, GB-MaGW 452-04-314, and GB-MaGW 452-06-004; NWO Large-Sized Investment Grant 175.010.2003.005); the Sophia Foundation for Medical Research (Projects 301 and 393), the Dutch Ministry of Justice (WODC), the European Science Foundation EuroSTRESS Project FP-006), and the participating universities.