Construct Validity of Adolescents’ Self-Reported Big Five Personality Traits

Short Personality Trait Measures and Conceptual Breadth

Initial evidence for the existence of the Big Five originated from early factor analytic studies of hundreds of personality trait descriptors or adjectives (Goldberg, 1990, 1993; John et al., 2008). Each of these five broad traits thus encompasses several primary personality traits. In other words, the Big Five traits are constructs with large conceptual breadth or bandwidth (Cattell, 1966). A number of integrative instruments measuring these broad traits and some of their corresponding primary traits have been developed (see chapters in Boyle, Matthews, & Saklofske, 2008). However, because users of personality measures are often faced with limited testing time, particularly in research situations, complete integrative personality trait measures are often considered too lengthy. For example, in large community-based longitudinal studies, in which personality traits are only one of several constructs being assessed, using shorter measures becomes practically inevitable. Not surprisingly, short measures of the Big Five traits are currently quite popular. A number of such measures exists, most of them having satisfactory basic psychometric properties (e.g., Big Five Questionnaire–Children Version [BFQ-C]—Barbaranelli, Caprara, Rabasca, & Pastorelli, 2003; Mini-International Personality Item Pool Big Five Measure [Mini-IPIP]—Donnellan, Oswald, Baird, & Lucas, 2006; International Personality Item Pool Big Five Measure [IPIP50]—Goldberg, 1999; Ten-Item Personality Inventory [TIPI]—Gosling, Rentfrow, & Swann, 2003; Big Five Inventory [BFI]—John, Donahue, & Kentle, 1991; John et al., 2008; NEO Five-Factor Inventory [NEO-FFI-3]—McCrae & Costa, 2010; Five Factor Model Rating Form [FFMRF]—Mullins-Sweatt, Jamerson, Samuel, Olson, & Widiger, 2006; Big Five Inventory–10 [BFI-10]—Rammstedt & John, 2007; Big Five Mini Markers [BFMM]—Saucier, 1994; Mini Modular Markers [3M40]—Saucier, 2002).

Various authors have reviewed issues pertaining to the validity and usefulness of short measures in general (e.g., Silverstein, 1990; Smith, McCarthy, & Anderson, 2000), and conceptual and statistical strategies have been proposed to shorten long versions (Marsh, Ellis, Parada, Richards, & Heubeck, 2005; Stanton, Sinar, Balzer, & Smith, 2002). Most of these authors have pointed to some nontrivial shortcomings of short measures of conceptually complex or broad constructs. One central shortcoming is the limited conceptual breadth. Indeed, conceptually broad constructs typically consist of several primary traits. In principle, for a short measure to be considered a valid alternative to the full-length one, each of its primary traits should be represented. A short measure of a broad construct has limited conceptual breadth when some primary traits are proportionally less represented, or worse, not represented at all (Loevinger, 1957; Smith, Fischer, & Fister, 2003). In other words, such a measure does not have adequate content validity (Haynes, Richards, & Kubany, 1995).

As argued by different scholars, the main reason why short personality measures have reduced conceptual breadth has to do with the common ways items are retained (Marsh et al., 2005; Saucier & Goldberg, 2002; Smith et al., 2000; Stanton et al., 2002). These authors point to three common empirical methods to develop short personality trait scales: factor saturation (selecting items with the highest factor loadings), factor discrimination (selecting univocal items without cross-loadings), and internal consistency maximization (selecting items contributing the most to internal consistency). These methods are useful, but they all tend to lead to the same shortcoming: scales with limited conceptual breadth in which items tapping some fundamental primary personality traits are absent (see Saucier & Goldberg, 2002). However, it is critical to preserve an adequate conceptual breadth in a personality measure for two related reasons. First, when representative sampling of all the primary traits embedded in a broad trait is not maintained, the measure’s content validity is compromised (Haynes et al., 1995; Smith et al., 2000; Smith et al., 2003). When content validity is not adequate, it affects almost all other psychometrics properties (Haynes et al., 1995). Second, preserving conceptual breadth is important when personality trait measures are used to predict external criteria (i.e., criterion validity). As pointed out by different scholars, all things being equal, a conceptually broader scale will tend to be related to a broader array of criteria (Loevinger, 1957; Saucier & Goldberg, 2002). This is particularly important for short Big Five measures because they are often used in community and longitudinal studies to predict a multitude of criterion variables. Of course, measures of conceptually broad predictors generally tend to provide optimal prediction of equally complex and heterogeneous criteria, whereas measures of narrower predictors are most efficient in predicting specific criteria (Cronbach & Gleser, 1965; Loevinger, 1957). As noted by Hogan and Roberts (1996), psychologists and other social scientists generally study conceptually complex or broad constructs or syndromes. In this context, short measures of broad constructs like Big Five personality traits preserving conceptual breadth are important.

An examination of the content of most existing Big Five short measures indicates that they suffer from a limited conceptual breadth. Indeed, a number of theoretically important primary traits are usually absent, which ipso facto limits content validity, but may also impact other psychometric properties. The point is not to argue that existing short personality trait measures are inappropriate or not useful, but merely to notice that most, if not all, of them have limited conceptual breadth. Yet evidence is still lacking regarding the impact of this limitation on other psychometric properties, such as convergent validity with independent measures of related constructs or criterion validity with consequential outcomes.

Rationale for Items Modification and Addition

In this study, the impact of adding conceptual breadth in a short Big Five personality trait measure was evaluated. While existing studies typically aimed at reducing the number of items, the reverse strategy was used in this study. Indeed, starting with an existing measure, a number of items tapping missing fundamental primary traits were added to the scales.

The BFI (John et al., 1991; John et al., 2008) was selected as an appropriate measure to provide an initial item pool because of its adequate basic psychometric properties and, more specifically, its satisfactory content validity as compared with other noncommercial short personality trait measures. Moreover, this measure contains short verbal statement items, which are preferable to single-trait adjective items because the former are more likely to be easily and unequivocally understood by early adolescents (Barbaranelli et al., 2003). Also, single-trait adjectives are often at a higher level of abstraction than short verbal statements, making them potentially harder to translate precisely in different languages (Saucier & Goldberg, 2002). The items measuring the BFI traits posted on the public domain International Personality Item Pool website were used as the initial items pool (http://ipip.ori.org/newBFITable.htm). The items are also available in John et al. (2008). As will be explained below, because of all the modifications made to the original item pool, the resulting new measure was named the Big Five Personality Trait Short Questionnaire (BFPTSQ), to avoid confusion with the original questionnaire. The BFPTSQ items are presented in the appendix. Back translation from the original French version (with all the modifications and additions described next) was used to make this English version.

Because this study was conducted with French Canadian early adolescents, it was important to ensure that all items would be well understood by the participants. A number of researchers have reported that adolescents have difficulty reading or understanding many items in personality questionnaires (e.g., Allik et al., 2004; De Fruyt, Mervielde, Hoekstra, & Rolland, 2000; McCrae, Costa, & Martin, 2005), arguably because these measures were constructed and validated for use with adults. Therefore, the first step was to translate and revise the items. Three researchers and three graduate students familiar with personality theory (and the Big Five model more specifically), experienced in psychometric test construction, as well as in research with adolescents first translated the items into French. Modifications were made until a consensus was reached. Afterward, the items were reviewed for language level. The criterion was that adolescents starting high school (at ages 12 or 13 in the province of Quebec) be able to understand the items. This led to minor modifications to most of the items. A number of items were slightly simplified, and for a few of them an additional descriptor or stem connected by a comma was added. Even if it is generally not recommended to have multiple stems in an item, the objective was merely to provide a synonym or to elaborate.

Before adding new items, one Openness item from the original item pool was deleted because it was judged less relevant for adolescents and not central to the target construct (“prefer work that is routine”). Moreover, an Extraversion item that was judged equivocal was removed (“generates a lot of enthusiasm”) and replaced with an item tapping social dominance or leadership (17).

The second step was to add new items representing important primary personality traits. To keep the measure short, with a testing time of 10 to 15 minutes for adolescents (5 to 10 minutes for adults), a maximum of 50 items overall, with 10 items per scale, was decided on. The goal was not to add items for all missing primary traits in the original item pool, but to add only a few of the most fundamental.

Scholarly reviews on trait structure and empirical research with adolescents guided the choice of which primary traits were candidates for adding items. For Openness, the original BFI scale has good content coverage for the primary traits of Intellectual Inquisitiveness, Creativity, and Aesthetic and Artistic Appreciation. It was decided that only one item tapping Openness to Cultural Diversity (31r) would be added. This primary trait is part of some comprehensive Openness models (McCrae & Sutin, 2009). For instance, it would be represented by the facet “Openness to Values” in the NEO-PI-3 (McCrae & Costa, 2010). As discussed by McCrae and Sutin (2009), this social attitude can be considered part of Openness because open persons are generally lower in ethnocentrism and thus, more open to different cultures and lifestyles. Assessing individual differences in this social attitude is important because it is associated with behavioral and emotional adjustment of both adolescents and teachers attending multicultural high schools in large metropolitan areas (García-Coll & Magnuson, 1997). Among existing short Big Five measures, there are no items tapping Openness to Cultural Diversity in the BFI, the IPIP50, the NEO-FFI-3, the BFMM, or the 3M40.

For Extraversion, the original BFI scale has good content coverage for the primary traits of Sociability, Expressiveness, Assertiveness, and Activity. It was decided that an item tapping Sensation Seeking (42) and one tapping Joyfulness or Positive Emotions (47) would be added. These two primary traits are part of various comprehensive models of Extraversion (Wilt & Revelle, 2009). For instance, they would correspond to Excitement Seeking and Positive Emotions in the NEO-PI-3 (McCrae & Costa, 2010). Sensation Seeking is certainly a fundamental personality trait that has proven useful for understanding the development of negative life outcomes (Zuckerman, 2009). For instance, it is related to antisocial behavior (Jones et al., 2011). Assessing individual differences in Sensation Seeking in adolescents can be useful for predicting substance use and to decide which adolescents should participate in a preventive intervention (Sargant, Tanski, Stoolmiller, & Hanewinkel, 2010). Joyfulness is also an important primary trait, as some scholars consider it to be the core aspect of Extraversion (Tellegen & Waller, 2008; Watson & Clark, 1997). Assessing individual differences in Joyfulness is important because it is related to various life outcomes, such as attachment styles (Shiota, Keltner, & John, 2006), stress recovery, and resilience (Ong, Bergeman, Bisconti, & Wallace, 2006). Among existing short Big Five measures, there are no items tapping Sensation Seeking in the BFI, the BFQ-C, the IPIP50, the BFMM, or the 3M40, while there are no items tapping Joyfulness in the BFI, the IPIP50, the BFMM, or the 3M40.

For Agreeableness, the original BFI scale has good content coverage for the primary traits of Altruism, Compassion, Forgiveness, and Cooperation. It was decided that an item tapping Machiavellianism (48r) would be added. This constitutes a ubiquitous primary personality trait in the human experience (Jones & Paulhus, 2009; Wilson, Near, & Miller, 1996) and is arguably a central construct for studying antisocial behavior development and psychopathy (Barry, Kerig, Stellwagen, & Barry, 2010). This primary trait is part of various comprehensive Agreeableness models and would be represented by the facet Straightforwardness and, to a lesser extent, Altruism in the NEO-PI-3 (McCrae & Costa, 2010). ¹ Assessing individual differences in this construct is important because it is clearly associated with serious conduct problems and antisocial behavior in children, adolescents, and adults (Frick et al., 2003; Jones et al., 2011). Among existing short Big Five measures, there are no items tapping Machiavellianism in the BFI, the BFQ-C, the IPIP50, the BFMM, or the 3M40.

For Conscientiousness, the original BFI scale has good content coverage for the primary traits of Dutifulness, Self-Discipline, Planfulness, and Order. It was decided that an item tapping Impulsiveness (49r) would be added. Impulsiveness constitutes without a doubt a fundamental primary personality trait (Madden & Bickel, 2010) and is part of most comprehensive models of Conscientiousness (Roberts, Jackson, Fayard, Edmonds, & Meints, 2008). In the NEO-PI-3, it would be represented by the facet Deliberation (McCrae & Costa, 2010). ² Assessing individual differences in impulse control is important because it is related to executive functions and plays a key role in various psychopathologies, such as ADHD and antisocial behavior (Crews & Boettiger, 2009; Jones et al., 2011; Newman & Wallace, 1993; Nigg, 2006). Among existing short Big Five measures, there are no items tapping Impulsiveness in the BFI, the BFQ-C, the IPIP50, the NEO-FFI-3, the BFMM, or the 3M40.

Finally, for Emotional Stability, the original BFI scale has good content coverage for the primary traits of Anxiousness, Fearfulness, Worry, and Depressed Mood. It was decided that an item tapping Low Self-Worth (45r) and another one tapping Irritability (50r) would be added. It is clear that Self-Worth (or Self-Esteem) is a crucial trait of the human experience (Bosson & Swann, 2009). Irritability (or Anger) is also a central construct in the human personality (Barefoot & Boyle, 2009; Snaith & Taylor, 1985). Both of these primary traits are part of various comprehensive models of Neuroticism (Clark & Watson, 2008; Tellegen & Waller, 2008). For instance, in the NEO-PI-3 (McCrae & Costa, 2010), low Self-Worth would be represented by the facet Vulnerability and, to a lesser extent, by Depression, while Irritability would be represented by the facet Angry Hostility. Assessing individual differences in Self-Worth is important because it is related to psychopathology development, particularly internalizing psychopathologies (Mann, Hosman, Schaalma, & de Vries, 2004) and aggression and antisocial behavior (Donnellan, Trzesniewski, Robins, Moffitt, & Caspi, 2005). Assessing individual differences in Irritability is also important because it is related to the development of internalizing psychopathologies (Stringaris, Cohen, Pine, & Leibenluft, 2009), ADHD (Leibenluft, Cohen, Gorrindo, Brook, & Pine, 2006), reactive aggression (Scarpa & Raine, 1997), antisocial behavior (Jones et al., 2011), and substance use (Tarter, Blackson, Brigham, Moss, & Caprara, 1995). Among existing short Big Five measures, there are no items tapping Irritability in the BFI and the NEO-FFI-3, while there are no items tapping Low Self-Worth in the BFI, the BFQ-C, the IPIP50, the BFMM, or the 3M40.

Evaluation of Construct Validity and Impact of Added Conceptual Breadth

To ensure that the modified items as well as the newly added items were valid, a group of experts in personality trait theory and test construction were first asked to rate the relevance or representativeness of the items with regard to their target Big Five trait. Factor validity was then assessed in two ways. First, to ensure that all items were associated with their target broad trait, factor analyses were conducted. Second, given the well-documented gender differences in personality trait means (Else-Quest, Hyde, Goldsmith, & Van Hulle, 2006; Schmitt, Realo, Voracek, & Allik, 2008), measurement invariance across genders was also evaluated. Indeed, before testing mean differences, there is a need to test whether the constructs of interest are the same in both groups, and few studies have fully tested measurement invariance of personality traits across genders (Marsh et al., 2010). Reliability was also estimated. Given the secondary objective of this study, reliability estimates from the scales with original content (i.e., without the new items) and those with added items were compared statistically. It was hypothesized that given their small number, the added items would have no or marginal impact on reliability estimates. Indeed, the conceptual breadth of the scales is increased, which could reduce their reliability; but they also become longer, which should slightly increase reliability. Convergent validity was also assessed using the NEO-PI-3 scales (McCrae & Costa, 2010). This questionnaire can arguably be considered a gold standard for convergent validity given its widespread use in personality research. Again, to determine whether the BFPTSQ scales with added items provided enhanced convergent validity, the correlations derived from the original-content scales and those derived from the added-item scales were compared using statistical tests. It was hypothesized that the scales with added items would have significantly higher correlations with each of the broad Big Five traits, as well as with primary traits from the NEO-PI-3 corresponding to specific content of the newly added items. Finally, criterion validity was assessed by correlating the personality scales with consequential outcomes. Specifically, because several studies showed that they are related to personality traits, measures of externalizing and internalizing psychopathology (e.g., De Pauw & Mervielde, 2010; Klein et al., 2012; Tackett, 2006; Tackett, Martel et al., 2012; Widiger & Smith, 2008) as well as academic achievement (e.g., Noftle & Robins, 2007; Poropat, 2009) were used. Based on these studies, it was hypothesized that the externalizing psychopathology and substance use scales would be negatively related to Agreeableness and Conscientiousness and positively related to Extraversion. For internalizing psychopathology scales, it was expected that they would be negatively related to Emotional Stability and Extraversion. It was also hypothesized that academic achievement would be negatively related to Conscientiousness and Openness. Finally, because of their larger conceptual breadth, it was hypothesized that the added-item scales would provide significantly higher correlations to each relevant outcome.

Procedure and Participants

The data come from the first assessment of an ongoing prospective longitudinal study on personality, antisocial behavior, and psychopathology development during adolescence. Eight French-language high schools from four different school boards of the larger Montreal metropolitan and Quebec City areas were initially targeted to participate (Quebec, Canada). These schools were a priori targeted to sample adolescents from various socioeconomic statuses and ethnic backgrounds. After the project was presented to all school administrations, one school declined to participate. In total, 41 classes participated in the first assessment: 29 regular classes, 8 international curriculum classes (i.e., enriched curriculum), and 4 specialized classes for special-needs students (i.e., with behavior problems and/or learning difficulties).

Before the data collection, the project was sanctioned by the University of Montreal’s ethical review board as well as all the school boards ethical committees. All the adolescents signed a consent form, completed a form in which they provided their contact information (email, home address, etc.), and were given an envelope to bring to their parents. The envelop contained a letter for the parents explaining the study, a consent form, a questionnaire about their child, a questionnaire about themselves, and a preaddressed and prepaid envelop to return all the material. All parents who did not return the signed consent form were contacted by phone. Only six parents refused to allow their child to participate in the study.

For the data collection, two trained research assistants visited each group between January and June 2009. Because the initial questionnaire in this longitudinal study was extensive, the adolescents were given two regular class periods (75 minutes each) to fill it out. All those who were absent from school the day of the assessment were contacted by phone to set a new date for the data collection. Moreover, to minimize missing data, all adolescents who were unable to fully complete the questionnaire during the two periods were contacted and asked to return the missing responses by email. Two $20 gift certificates were randomly drawn in each group among the participating adolescents.

The questionnaires of 1,036 adolescents were gathered in the first assessment of this study. Personality data were available for 1,028 of them (only a subsample of 598 filed the NEO-PI-3). Their mean age was 12.71 (SD = 0.59), with most aged 12 (35.5%) or 13 (59.1%) and only a few aged 14 (4.4%) or 15 (1.0%). The sample was equally proportioned by gender (females, n = 520, 50.2%). Consistent with the province of Quebec population, most of them were Caucasian (74%), while Blacks (4.9%), Hispanics (3.3%), Asians (3.2%), Arabs (7.4%), Canadian First Nations (2.8%), and multiethnic persons (4.4%) made up the rest of the sample. The large majority were born in Canada (90.8%). Most of them lived with their two biological parents (68.1%), while 16.2% lived in a shared custody situation with their biological mother and father, 5.0% with their biological mother and her new partner, 1.7% with their biological father and his new partner, 6.1% with their biological mother alone, 1.2% with their biological father alone, and the rest with another family member or an adoptive family or in foster care (1.7%). With regard to siblings, 65.5% (M = 0.98, SD = 0.98) of these adolescents lived with at least one brother, while 61.6% lived with at least one sister (M = 0.90, SD = 0.93).

Measures

Statistical Analyses

All analyses were conducted using Mplus Version 6.12 (Muthén & Muthén, 2010). Unless otherwise noted, all analyses were conducted using the robust maximum likelihood estimator (MLR), which provides adjusted standard errors and statistical fit tests that are robust to nonnormality in the data. Confidence intervals (95%) were calculated and reported to provide information regarding the precision of all relevant parameter estimates (Cumming, 2012).

Content Validity

In general, the six experts’ ratings all point to adequate content validity. No item received a rating of 0 (“not relevant”). For individual items, 76% (38 out of 50) have a perfect agreement with an I-CVI = 1. All the other items have an I-CVI above the recommended criterion of .78, except three items, which have an I-CVI of .666: Items 42 (Sensation Seeking), 45 (Low Self-Worth), and 48 (Machiavellianism). Nonetheless, four out of six raters identified these three items, which are all newly added ones, as “very representative.” Regarding content validity of the scales, all received highly favorable ratings from the experts, with all estimates above the recommended criterion. The S-CVI is .966 for Conscientiousness, .949 for Openness, Extraversion, and Emotional Stability, and .933 for Agreeableness.

Factor Validity

The goodness-of-fit statistics from the different factor analytic models are presented in Table 1. All indices suggest that ICM-CFA clearly does not fit the data (M1). Adding a priori CUs (M1b) significantly improved the fit, but it was still a poor-fitting model. Fitting an ESEM model (M2) largely improved fit over the ICM-CFA model as suggested by the large Δχ², ΔCFI, and ΔRMSEA. After the results from all the tested rotations had been examined, the target loading rotation was selected because it provided a somewhat clearer factor loading pattern (i.e., the target loadings tended to be slightly larger, while the cross-loadings tended to be smaller than with other rotation methods) and the factor correlations were only slightly higher. ⁵ The fit of this model, however, remains unacceptable because the CFI and TLI values were below the acceptable criterion. A model adding a priori CUs (M2b) again significantly improved the fit to the data. In contrast to the preceding models, this ESEM with CUs shows fit indices all in the satisfactory range, with CFI and TLI above .90, as well as RMSEA and SRMR below .06.

OPEN IN VIEWER

Table 1.

Goodness-of-Fit Statistics From the Confirmatory Factor Analytic and Exploratory Structural Equation Models.

Model	χ2 (df)	CFI	TLI	RMSEA	90% CI	SRMR	Ref	ΔSχ2 (df)	ΔCFI	ΔRMSEA
M1: ICM-CFA	5178.65* (1165)	.680	.663	.058	[.056, .060]	.081	–	–	–	–
M1b: ICM-CFA with CUs	4285.04* (1144)	.750	.732	.052	[.050, .053]	.079	M1	782.43* (21)	+.070	−.006
M2: ESEM	2604.68* (985)	.871	.839	.040	[.038, .042]	.034	M1	2419.94* (180)	+.191	−.018
M2b: ESEM with CUs	1811.17* (964)	.932	.914	.029	[.027, .031]	.028	M2	780.65* (21)	+.061	−.011
Gender invariance
MG1: Configural invariance	2895.64* (1928)	.922	.901	.031	[.029, .034]	.033	–	–	–	–
MG2: λ invariant	3141.31* (2153)	.921	.910	.030	[.028, .032]	.042	MG1	255.03* (225)	−.001	−.001
MG3: λ, τ invariant	3431.88* (2198)	.901	.890	.033	[.031, .035]	.045	MG2	301.80* (45)	−.020	+.003
MG3b: λ, τ partially invariant	3307.64* (2194)	.911	.900	.031	[.029, .034]	.044	MG2	138.13* (41)	−.010	+.001
MG4: λ, τ, δ invariant	3481.55* (2244)	.901	.892	.033	[.031, .035]	.051	MG3b	171.17* (50)	−.010	+.002
MG5: λ, τ, δ, CUs invariant	3518.22* (2265)	.899	.891	.033	[.031, .035]	.051	MG4	35.77 (21)	−.002	+.000
MG6: λ, τ, δ, CUs, ξ/φ invariant	3555.29* (2280)	.898	.890	.033	[.031, .035]	.054	MG5	36.63 (15)	−.001	+.000
MG7: λ, τ, δ, CUs, ξ/φ, η invariant	3757.23* (2285)	.882	.873	.035	[.033, .037]	.064	MG6	220.36* (5)	−.016	+.002

Note. ICM-CFA = independent clusters model confirmatory factor analysis; ESEM = exploratory structural equation modeling; χ² = chi square; df = degrees of freedom; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; 90% CI = 90% confidence interval of the RMSEA; SRMR = standardized root mean square residual; Ref = reference model; ΔSχ² = Satorra–Bentler scaled chi-square difference test; Δdf = change in degrees of freedom; ΔCFI = change in CFI; ΔRMSEA = change in RMSEA; λ = factor loadings; τ = intercepts; δ = uniquenesses; ξ = factor variances; φ = factor covariances; η = factor means.

*

p < .01.

Table 2 presents the standardized factor loadings from the ESEM model with CUs (M2b). Most target loading items are substantial and are clearly statistically related to their expected factor. Still, among the target loadings, 10 out of 50 have a value below .30, though they are statistically related to their expected factor. The standard errors tend to be small for all items, including those with low loadings. Examination of the confidence intervals suggests that all target loadings are relevant as none includes a value of 0. Apart from the 50 target loadings, there is a total of 105 statistically significant cross-loadings, with 58 clearly significant at .001 and 47 of lesser significance at .05.

OPEN IN VIEWER

Table 2.

Standardized Factor Loadings From the Exploratory Structural Equation Model of the BFPTSQ Items.

	Openness		Extraversion		Agreeableness		Conscientiousness		Emotional Stability
Item	λ	95% CI	λ	95% CI	λ	95% CI	λ	95% CI	λ	95% CI	δ
1	.504**	[.373, .634]	.219**	[.146, .292]	.133**	[.068, .198]	−.047	[−.129, .035]	.051	[−.009, .112]	.640
6	.304**	[.215, .393]	.255**	[.178, .332]	.065	[−.034, .163]	−.061	[−.144, .022]	−.135**	[−.207, −.064]	.792
11	.471**	[.365, .576]	−.074	[−.154, .006]	−.044	[−.114, .027]	.242**	[.155, .329]	.089*	[.027, .151]	.687
16	.534**	[.427, .640]	.067	[−.003, .137]	.107*	.038, .176]	−.156**	[−.235, −.077]	.005	[−.054, .064]	.711
21	.577**	[.446, .708]	.115*	[.039, .191]	.103*	[.040, .166]	−.032	[−.117, .053]	.080*	[.021, .140]	.622
26	.306**	[.205, .407]	.049	[−.030, .129]	.137*	[.060, .215]	.063	[−.020, .146]	−.105*	[−.179, −.030]	.850
31r	.269**	[.144, .388]	−.077	[−.161, .008]	.218**	[.133, .303]	.044	[−.046, .134]	.009	[−.068, .087]	.898
36	.408**	[.325, .490]	−.091*	[−.160, −.023]	−.041	[−.120, .039]	.196**	[.122, .270]	.063	[−.002, .128]	.778
41	.214**	[.107, .321]	−.070	[−.154, .014]	.076	[−.010, .162]	.033	[−.055, .122]	.007	[−.068, .083]	.947
46	.363**	[.271, .456]	−.131*	[−.206, −.056]	.066	[−.014, .147]	.011	[−.072, .095]	−.003	[−.072, .065]	.871
2	.180**	[.105, .255]	.525**	[.454, .595]	.036	[−.035, .107]	.028	[−.035, .091]	−.078*	[−.139, −.016]	.642
7r	−.239**	[−.307, −.170]	.642**	[.581, .704]	.005	[−.059, .069]	−.011	[−.068, .046]	.192**	[.124, .260]	.521
12	.079	[−.007, .164]	.440**	[.369, .512]	.110*	[.029, .192]	.107*	[.032, .181]	.007	[−.056, .071]	.740
17	.240**	[.167, .313]	.468**	[.401, .535]	−.142**	[−.211, −.074]	.101*	[.036, .166]	.093*	[.034, .153]	.601
22r	−.193**	[−.270, −.115]	.726**	[.656, .796]	−.021	[−.091, .050]	−.109*	[−.173, −.044]	−.021	[−.094, .053]	.504
27	.209**	[.138, .280]	.448**	[.382, .513]	−.167**	[−.232, −.102]	.228**	[.166, .291]	.191**	[.130, .252]	.539
32r	−.151*	[−.240, −.062]	.668**	[.594, .742]	−.021	[−.093, .050]	−.002	[−.067, .064]	.234**	[.156, .313]	.467
37	.053	[−.035, −.142]	.628**	[.553, .703]	.144**	[.066, .221]	.011	[−.057, .078]	−.043	[−.094, .008]	.576
42	.116*	[.018, .215]	.289**	[.205, .373]	−.068	[−.155, .019]	−.051	[−.135, .033]	.039	[−.034, .112]	.882
47	.104*	[.001, .206]	.296**	[.214, .379]	.149**	[.061, .237]	−.091*	[−.175, −.007]	−.226**	[−.300, −.152]	.846
3r	−.022	[−.101, .057]	−.156**	[−.229, −.083]	.387**	[.308, .466]	.050	[−.025, .126]	.070*	[.004, .136]	.795
8	.201**	[.116, .386]	.053	[−.018, .124]	.264**	[.164, .365]	.180**	[.094, .266]	−.064	[−.137, .009]	.770
13r	.021	[−.057, .098]	−.078*	[−.138, −.017]	.420**	[.350, .489]	.206**	[.137, .274]	.112*	[.048, .175]	.686
18	.228**	[.144, .313]	−.083*	[−.157, −.008]	.223**	[.120, .326]	−.053	[−.140, .035]	.051	[−.029, .132]	.873
23	.192**	[.082, .302]	.161**	[.068, .253]	.269**	[.166, .373]	.031	[−.059, .120]	−.034	[−.115, .048]	.845
28r	−.078	[−.180, .025]	.114**	[.050, .177]	.550**	[.478, .621]	−.047	[−.116, .022]	.110*	[.046, .173]	.665
33	.208**	[.105, .312]	.144**	[.064, .224]	.391**	[.280, .502]	.188**	[.099, .278]	−.105*	[−.178, −.033]	.620
38r	−.004	[−.125, .116]	−.153**	[−.222, −.085]	.542**	[.466, .617]	.167**	[.097, .238]	.088*	[.018, .158]	.570
43	.219**	[.116, .322]	.306**	[.221, .392]	.326**	[.223, .429]	.029	[−.059, .117]	−.039	[−.114, .037]	.718
48r	−.060	[−.143, .024]	−.143**	[−.205, −.081]	.504**	[.435, .573]	.045	[−.027, .118]	.044	[−.018, .106]	.690
4	.057	[−.030, .145]	−.018	[−.071, .034]	.009	[−.057, .075]	.716**	[.649, .792]	−.089*	[−.143, −.036]	.474
9r	−.155**	[−.239, −.070]	.048	[−.023, .118]	.308**	[.228, .387]	.213**	[.135, .291]	.024	[−.046, .095]	.747
14	.045	[−.042, .133]	.052	[−.003, .108]	−.031	[−.095, .033]	.681**	[.611, .750]	−.088*	[−.142, −.034]	.536
19r	−.244**	[−.317, −.171]	.002	[−.061, .065]	.017	[−.067, .102]	.587**	[.520, .655]	.035	[−.031, .101]	.643
24r	−.123*	[−.211, −.034]	.062	[−.008, .133]	.194**	[.115, .274]	.416**	[.337, .494]	.032	[−.040, .105]	.728
29	.093*	[.022, .163]	.058	[.003, .113]	.009	[−.058, .077]	.618**	[.557, .679]	−.048	[−.105, .009]	.574
34	.100*	[.021, .179]	.074*	[.019, .129]	−.089*	[−.152, −.025]	.713**	[.654, .773]	−.029	[−.084, .025]	.472
39	.141*	[.046, .237]	.047	[−.019, .112]	−.012	[−.092, .069]	.441**	[.359, .522]	−.096*	[−.164, −.028]	.754
44r	−.096*	[−.187, −.005]	−.088*	[−.158, −.017]	.005	[−.082, .092]	.472**	[.393, .552]	.236**	[.164, .307]	.692
49r	−.125*	[−.220, −.030]	−.223**	[−.288, −.158]	.317**	[.240, .395]	.278**	[.206, .351]	.165**	[.098, .232]	.607
5r	−.195**	[−.276, −.113]	.217**	[.147, .287]	.152**	[.073, .230]	.164**	[.090, .237]	.259**	[.183, .335]	.756
10	.211**	[.119, .303]	.072	[−.007, .150]	−.086*	[−.170, −.003]	−.032	[−.107, .043]	.589**	[.508, .669]	.622
15r	−.001	[−.063, .061]	.053	[−.003, .110]	−.011	[−.070, .048]	−.118**	[−.174, −.062]	.678**	[.623, .739]	.538
20r	−.154**	[−.224, −.084]	.033	[−.029, .095]	.022	[−.049, .094]	−.012	[−.079, .056]	.537**	[.475, .598]	.661
25	.121*	[.030, .211]	−.024	[−.099, .051]	−.125*	[−.207, −.043]	.020	[−.059, .099]	.468**	[.394, .543]	.782
30r	−.115*	[−.213, −.017]	.003	[−.065, .072]	.250**	[.169, .331]	−.014	[−.089, .061]	.379**	[.306, .452]	.744
35	.260**	[.164, .357]	−.030	[−.103, .044]	−.093*	[−.180, −.006]	.003	[−.070, .077]	.614**	[.535, .693]	.608
40r	.057	[−.007, .120]	.065*	[.012, .118]	−.020	[−.080, .040]	−.096*	[−.151, −.041]	.773**	[.728, .819]	.404
45r	−.081	[−.162, .001]	.210**	[.144, .276]	−.035	[−.113, .043]	.144**	[.072, .216]	.459**	[.387, .532]	.670
50r	−.076	[−.177, .025]	−.085*	[−.152, −.017]	.354**	[.279, .430]	.080*	[.008, .151]	.350**	[.279, .421]	.644

Note. Shaded entries are the target loading items. Item numbers with an r are reverse scored. λ = factor loadings; δ = uniquenesses; 95% CI = 95% confidence interval.

*

p < .05. **p < .001.

Examination of each factor shows that for Openness, the anchor item (21) taps ingenuity and creativity. The new Openness to Cultural Diversity item (31r), although it is clearly statistically related to Openness, has a somewhat low standardized loading. As expected, this item has a significant cross-loading on Agreeableness. For Extraversion, the anchor item (22r) taps talkativeness and liveliness. The new Sensation Seeking (42) and Joyfulness (47) items are clearly statistically related to Extraversion, but also tend to have small loadings. As expected, Item 47 has a significant cross-loading on Emotional Stability. For Agreeableness, the anchor item (28r) taps warmth and kindness. The new Machiavellianism item (48r) is strongly related to Agreeableness and, in fact, has one of the highest loadings on that factor. For Conscientiousness, the anchor item (4) taps thoroughness or diligence. The new Impulsiveness item (49r) is statistically related to Conscientiousness, but has a somewhat low loading. Moreover, this is a complex item because it has significant cross-loadings on all other factors, particularly on Agreeableness. Finally, for Emotional Stability, the anchor item (40r) taps nervousness. The new Low Self-Worth (45r) and Irritability (50r) items are clearly related to Emotional Stability and are in fact items with some of the highest loadings on that factor. As expected, Item 45r has a significant cross-loading on Extraversion, as does Item 50r on Agreeableness.

Table 3 presents the latent factor correlations and their 95% confidence intervals from the ICM-CFA and ESEM models. As expected, the factor correlations from ESEM are substantially smaller than those from ICM-CFA. While the average absolute factor correlation for ICM-CFA is .27 (SD = .17), it is .13 (SD = .08) for ESEM. For example, in both models, the largest correlation is between Agreeableness and Conscientious; it is .605 in ICM-CFA, but decreases to .348 in ESEM. The second largest correlation in both models is between Openness and Extraversion; it is .447 in ICM-CFA, but decreases to .269 in ESEM. Interestingly, the lowest and only nonsignificant correlation in ICM-CFA is between Openness and Emotional Stability, but it becomes negative and significant in ESEM.

OPEN IN VIEWER

Table 3.

Point and Interval Estimate of Factor Correlations of the BFPTSQ.

	1. Openness		2. Extraversion		3. Agreeableness		4. Conscientiousness		5. Emotional Stability
	φ	95% CI	φ	95% CI	φ	95% CI	φ	95% CI	φ	95% CI
1.	–	–	.269**	[.211, .327]	−.032	[−.092, .028]	.227**	[.130, .324]	−.119**	[−.181, −.058]
2.	.447**	[.349, .545]	–	–	−.025	[−.100, .050]	.078*	[.009, .146]	.179**	[.102, .256]
3.	.259**	[.144, .373]	.059	[−.064, .181]	–	–	.348**	[.279, .418]	.141**	[.081, .201]
4.	.318**	[.221, .416]	.187**	[.095, .278]	.605**	[.527, .682]	–	–	.155**	[.087, .223]
5.	.041	[−.045, .127]	.367**	[.281, .453]	.235**	[.141, .328]	.137**	[.048, .226]	–	–

Note. Latent factor correlations from the final exploratory structural equation model (ESEM) are presented above the diagonal, while latent correlations from the independent clusters model confirmatory factor analysis (ICM-CFA) are presented below the diagonal. φ = factor covariance/correlation; 95% CI = 95% confidence interval.

*

p < .05. **p < .001.

The goodness-of-fit statistics from the gender invariance tests are presented in Table 1. Fitting the ESEM model with all freely estimated parameters for males and females separately provides acceptable fit to the data (MG1). Constraining all factor loadings to equality did not significantly worsen the fit (MG2). This result is noteworthy, because such a test involves many more parameters (i.e., all target loadings and cross-loadings are constrained to equality) than in ICM-CFA. Constraining intercepts to equality across genders, however, did seem to suggest the possibility of non-invariance (MG3). Indeed, the scaled chi-square difference test is significant and the change in CFI is at the upper limit of the cutoff criterion suggesting the possibility of non-invariance. Therefore, a model with partial invariance of intercepts (MG3b) was estimated (Byrne, Shavelson, & Muthén, 1989). Based on modification indices, four items were freed across groups: females had higher item intercepts for Items 26, 41, and 47 and a lower item intercept for Item 40. This model provided a better fit than the model with fully invariant intercepts. Constraining item uniquenesses across genders did not lead to a significant worsening in model fit (MG4). Indeed, even though the scaled chi-square test is significant, the change in CFI is not significant and the change in RMSEA is again trivial. Another model constraining the CUs to equality across genders did not lead to a significant worsening of fit (MG5), suggesting that these parameters are not gender-specific. Overall, even if the fit indices tended to gradually decrease with the increases in equality constraints (CFI and TLI), the RMSEA remained in the acceptable range for all models, even considering its confidence intervals. All these results reasonably suggest that the BFPTSQ factor structure show full measurement invariance across genders.

Constraining the variances/covariances across genders did not lead to a significant worsening of fit (MG6). As expected, however, constraining the latent factor means to equality across genders did lead to a significant worsening in fit (MG7). Examination of the modification indices and the results of the preceding model (MG6) revealed a number of significant mean differences between males and females. With the males’ means fixed to 0 and the variances fixed to 1 in both groups—which allows interpretation of these differences in terms of traditional effect size—the females’ latent means were significantly lower in Emotional Stability (−.755, p < .0001), significantly higher in Agreeableness (.578, p < .0001) and Conscientiousness (.344, p < .001), and marginally higher in Openness (.158, p < .10). There was no significant gender difference in Extraversion.

Reliability

The point and interval estimates of reliability and the comparison tests between reliability estimates (Δρ) are presented in Table 4. In general, all estimates suggest that the BFPTSQ scales have adequate reliability. The traditional α and ρ without CUs tend to be similar across all scales. As expected, however, ρ conditional on CUs are all noticeably lower, decreasing below .70 for Openness and Agreeableness. The estimates tend to be somewhat higher for Extraversion and Emotional Stability, while they tend to be the lowest for Openness and Agreeableness.

OPEN IN VIEWER

Table 4.

Point and Interval Reliability Estimates of the BFPTSQ Scales.

	Original-Content Scale			Added-Items Scale
	α	ρ	95% CI	α	ρ	95% CI	Δρ	95% CI
	Openness
No CU	.716	.712**	[.685, .740]	.712	.702**	[.672, .732]	−.011*	[−.020, −.002]
With CUs		.648**	[.613, .683]		.646**	[.609, .682]	−.002	[−.013, .009]
	Extraversion
No CU	.819	.828**	[.811, .846]	.802	.813**	[.795, .832]	−.015**	[−.022, −.008]
With CUs		.783**	[.758, .808]		.777**	[.754, .801]	−.006	[−.014, .003]
	Agreeableness
No CU	.701	.705**	[.677, .734]	.722	.723**	[.694, .752]	.018*	[.007, .029]
With CUs		.655**	[.618, .693]		.690**	[.658, .723]	.035**	[.024, .047]
	Conscientiousness
No CU	.795	.796**	[.777, .815]	.796	.792**	[.772, .812]	−.004	[−.009, .002]
With CUs		.778**	[.756, .800]		.776**	[.753, .799]	−.002	[−.008, .004]
	Emotional Stability
No CU	.792	.802**	[.782, .821]	.808	.812**	[.793, .830]	.010*	[.002, .018]
With CUs		.764**	[.741, .788]		.786**	[.765, .808]	.022	[.013, .031]

Note. α = Cronbach’s alpha internal consistency coefficient; ρ = latent variable model composite reliability estimate; 95% CI = 95% confidence interval; Δρ = delta parameter (composite reliability difference estimate); CU = correlated uniqueness.

*

p < .05. **p < .001.

Overall, the delta parameters (Δρ) resulting from the comparison tests between the original-content scales and the added-item scales tend to suggest that both scales provide similar reliability estimates. This is supported by the fact that the confidence intervals of all composite reliability estimates are overlapping. This demonstrates that the newly added items are not detrimental to scale reliability, despite the fact that a number of them show low factor loadings. In fact, when considering ρ without CUs, the results showed that the added items significantly decrease reliability for Openness and Extraversion, while they significantly increase reliability for Agreeableness and Emotional Stability. However, when the ρ conditional on CUs are considered—which are arguably closer to the true models—all differences between reliability estimates are nonsignificant, except for Agreeableness, which continues to suggest that the added item increases reliability.

Convergent Validity

The point and interval correlation estimates between the BFPTSQ and NEO-PI-3 scales as well as the comparison tests between correlations (Δr) are presented in Table 5. The overall pattern of correlations suggests adequate convergent validity. All correlations between the broad-trait scales are high, mostly around .70. For the scales with added items, they range from .629 for Openness to .804 for Conscientiousness. The correlations between the BFPTSQ scales and the corresponding NEO-PI-3 primary-trait scales are generally lower, but all are clearly significant. The only exception is the correlation between BFPTSQ Agreeableness and Modesty, which is quite low and even nonsignificant for the original-content scale.

OPEN IN VIEWER

Table 5.

Point and Interval Correlation Estimates Between BFPTSQ and NEO-PI-3 Scales.

	Original-Content Scale		Added-Items Scale
NEO-PI-3 Scales	r	95% CI	r	95% CI	Δr	95% CI
	Openness
Openness	.591**	[.540, .642]	.629**	[.583, .675]	.038**	[.020, .056]
Fantasy	.377**	[.308, .446]	.385**	[.315, .455]	.008	[−.014, .031]
Aesthetics	.459**	[.394, .524]	.509**	[.448, .570]	.050**	[.030, .071]
Feelings	.317**	[.244, .389]	.331**	[.260, .401]	.014	[−.010, .038]
Actions	.186**	[.108, .265]	.171**	[.092, .249]	−.016	[−.039, .008]
Ideas	.465**	[.406, .524]	.512**	[.459, .566]	.047**	[.026, .068]
Values	.196**	[.116, .276]	.204**	[.128, .280]	.008	[−.015, .031]
	Extraversion
Extraversion	.650**	[.602, .698]	.694**	[.650, .737]	.043**	[.030, .057]
Warmth	.476**	[.407, .545]	.494**	[.427, .562]	.019*	[.002, .036]
Gregariousness	.453**	[.387, .518]	.477**	[.413, .542]	.025**	[.010, .040]
Assertiveness	.662**	[.617, .708]	.653**	[.607, .699]	−.010	[−.022, .003]
Activity	.481**	[.417, .544]	.490**	[.427, .553]	.010	[−.006, .025]
Excitement Seeking	.347**	[.282, .412]	.435**	[.375, .495]	.088**	[.072, .104]
Positive Emotions	.340**	[.262, .418]	.390**	[.314, .465]	.050**	[.032, .067]
	Agreeableness
Agreeableness	.612**	[.565, .659]	.671**	[.628, .713]	.059**	[.047, .070]
Trust	.472**	[.408, .536]	.477**	[.413, .541]	.005	[−.010, .020]
Straightforwardness	.432**	[.373, .492]	.532**	[.479, .586]	.100**	[.086, .114]
Altruism	.593**	[.545, .640]	.598**	[.549, .646]	.005	[−.008, .018]
Compliance	.436**	[.376, .497]	.471**	[.410, .531]	.034**	[.020, .048]
Modesty	.076	[.000, .162]	.114*	[.024, .204]	.039**	[.023, .054]
Tender-Mindedness	.322**	[.248, .396]	.337**	[.264, .409]	.015*	[−.001, .030]
	Conscientiousness
Conscientiousness	.789**	[.760, .818]	.804**	[.776, .832]	.015**	[.006, .024]
Competence	.580**	[.528, .632]	.598**	[.547, .649]	.018*	[.006, .029]
Order	.616**	[.568, .664]	.616**	[.568, .664]	.000	[−.011, .012]
Dutifulness	.674**	[.629, .719]	.681**	[.635, .726]	.007	[−.005, .018]
Achievement Striving	.614**	[.566, .662]	.604**	[.555, .652]	−.010	[−.022, .001]
Self-Discipline	.723**	[.685, .761]	.727**	[.689, .764]	.004	[−.006, .014]
Deliberation	.467**	[.408, .527]	.519**	[.462, .576]	.052**	[.039, .064]
	Emotional Stability
Neuroticism a	.733**	[.695, .771]	.777**	[.745, .809]	.044**	[.030, .058]
Anxiety	.692**	[.650, .734]	.680**	[.638, .722]	−.012	[−.027, .004]
Angry Hostility	.469**	[.407, .532]	.510**	[.451, .569]	.040**	[.024, .057]
Depression	.560**	[.504, .616]	.619**	[.569, .670]	.059**	[.044, .075]
Self-Conscientiousness	.486**	[.427, .545]	.534**	[.480, .588]	.048**	[.032, .065]
Impulsiveness	.285**	[.205, .366]	.319**	[.242, .396]	.034**	[.015, .053]
Vulnerability	.639**	[.590, .687]	.662**	[.617, .706]	.023*	[.009, .037]

Note. In this table, each Big Five personality trait is correlated with its corresponding higher-order and primary-trait scales from the NEO-PI-3. 95% CI = 95% confidence interval; Δr = delta parameter (correlation difference estimate).

a.

Because Emotional Stability is simply the opposite pole, all correlations with Neuroticism and its primary traits from the NEO-PI-3 are actually negative and are presented in absolute value to simplify the table.

*

p < .05. **p < .001.

Overall, the delta parameters (Δr) resulting from the comparison tests between the original-content scales and those with added items suggest that the new scales significantly improved convergent validity. Indeed, all correlations between the broad Big Five trait scales are significantly higher for the added-item scales. However, the confidence intervals of all correlation pairs overlap somewhat. With regard to the primary-trait scales, the expected results are observed. Indeed, for Extraversion, the increase in correlation is largest for Excitement Seeking and Positive Emotions. For Agreeableness, the increase is largest for Straightforwardness, while for Conscientiousness, it is largest for Deliberation. For Emotional Stability, the increase in correlation is largest for Depression, Self-Consciousness, and Angry Hostility. In addition to these expected improved correlations, several other correlations with primary-trait scales also significantly increased, which is to be expected since all primary traits have a common variance associated with their corresponding broad Big Five trait. Again, the confidence intervals of all correlation pairs overlap somewhat.

Criterion Validity

The point and interval correlation estimates between the BFPTSQ and outcome scales as well as the comparison tests between correlations (Δr) are presented in Table 6. The pattern of correlations between the Big Five and outcome scales generally suggests adequate criterion validity. Openness is related to lower scores in CD and to higher scores in BD and GAD. As expected, Openness is most strongly related to higher scores in GPA. Extraversion is related to higher scores in ADHD, CD, BD, and substance use, but to lower scores in MDD and SOP. As anticipated, Extraversion is most strongly related to lower scores in SOP and to higher scores in ADHD. Agreeableness is related to lower scores on all psychopathology scales and substance use, but to higher scores in GPA. As projected, Agreeableness is most strongly related to the three externalizing psychopathology scales, particularly CD and ODD. Conscientiousness is also related to lower scores on all psychopathology scales and substance use, but to higher scores in GPA. As hypothesized, Conscientiousness is most strongly related to higher scores in GPA and to lower scores in ADHD. Emotional Stability is related to lower scores on all psychopathology scales, except CD and substance use. Emotional Stability is not related to GPA. As expected, Emotional Stability is most strongly related to the four internalizing psychopathology scales, particularly MDD, GAD, and SOP. ⁶

OPEN IN VIEWER

Table 6.

Point and Interval Estimates of Correlations Between BFPTSQ and Outcomes Scales.

	Original-Content Scale		Added-Items Scale
Outcome Scales	r	95% CI	r	95% CI	Δr	95% CI
	Openness
ADHD	.057 †	[−.006, .120]	.000	[−.064, .064]	−.057**	[−.075, −.038]
CD	−.094*	[−.161, −.027]	−.134**	[−.198, −.071]	−.040**	[−.060, −.020]
ODD	−.003	[−.065, .059]	−.042	[−.104, .020]	−.039**	[−.059, −.019]
MDD	.063*	[−.001, .127]	.051	[−.012, .114]	−.012	[−.031, .007]
BD	.090*	[.027, .153]	.071*	[.007, .134]	−.019*	[−.038, .000]
GAD	.093*	[.029, .156]	.095*	[.032, .157]	.002	[−.018, .021]
SOP	−.030	[−.095, .035]	−.016	[−.080, .048]	.014	[−.006, .033]
SUBS	−.017	[−.094, .061]	−.037	[−.105, .031]	−.020	[−.046, .006]
GPA	.262**	[.192, .333]	.279**	[−.210, .348]	.017	[−.003, .037]
	Extraversion
ADHD	.178**	[.111, .244]	.219**	[.155, .283]	.041**	[.026, .057]
CD	.105**	[.041, .168]	.094*	[.028, .160]	−.011	[−.028, .007]
ODD	.056	[−.011, .123]	.074*	[.007, .141]	.018*	[.003, .033]
MDD	−.140**	[−.208, −.072]	−.107**	[−.176, −.038]	.034**	[.020, .048]
BD	.153**	[.087, .220]	.194**	[.129, .259]	.040**	[.027, .054]
GAD	−.088*	[−.157, −.020]	−.057	[−.126, .011]	.031**	[.017, .045]
SOP	−.341**	[−.399, −.282]	−.324**	[−.383, −.265]	.016*	[.002, .031]
SUBS	.121**	[.064, .179]	.112**	[.056, .168]	−.009	[−.038, .020]
GPA	.061	[−.015, .137]	.075*	[−.002, .153]	.014	[−.003, .032]
	Agreeableness
ADHD	−.308**	[−.371, −.244]	−.338**	[−.399, −.277]	−.030**	[−.044, −.017]
CD	−.393**	[−.458, −.328]	−.429**	[−.490, −.369]	−.036**	[−.052, −.021]
ODD	−.442**	[−.497, −.387]	−.480**	[−.532, −.427]	−.037**	[−.050, −.024]
MDD	−.200**	[−.261, −.138]	−.207**	[−.270, −.145]	−.007	[−.020, .005]
BD	−.169**	[−.233, −.106]	−.210**	[−.272, −.147]	−.040**	[−.053, −.027]
GAD	−.136**	[−.200, −.072]	−.143**	[−.208, −.079]	−.007	[−.020, .006]
SOP	−.164**	[−.228, −.101]	−.157**	[−.222, −.093]	.007	[−.007, .020]
SUBS	−.124**	[−.197, −.052]	−.140**	[−.214, −.066]	−.016 †	[−.003, .001]
GPA	.323**	[.255, .391]	.314**	[.246, .382]	−.009	[−.024, .005]
	Conscientiousness
ADHD	−.512**	[−.564, −.461]	−.560**	[−.607, −.512]	−.047**	[−.058, −.037]
CD	−.361**	[−.425, −.298]	−.401**	[−.460, −.341]	−.039**	[−.052, −.027]
ODD	−.384**	[−.442, −.327]	−.439**	[−.492, −.386]	−.055**	[−.066, −.043]
MDD	−.264**	[−.324, −.204]	−.297**	[−.356, −.237]	−.033**	[−.044, −.021]
BD	−.252**	[−.316, −.188]	−.299**	[−.360, −.237]	−.047**	[−.058, −.035]
GAD	−.133**	[−.198, −.068]	−.171**	[−.235, −.106]	−.038**	[−.050, −.027]
SOP	−.126**	[−.191, −.061]	−.141**	[−.207, −.076]	−.015*	[−.027, −.004]
SUBS	−.155**	[−.249, −.062]	−.178**	[−.268, −.087]	−.022**	[−.035, −.010]
GPA	.404**	[.340, .467]	.401**	[.338, .463]	−.003	[−.015, .010]
	Emotional Stability
ADHD	−.223**	[−.291, −.154]	−.264**	[−.332, −.196]	−.041**	[−.057, −.026]
CD	.001	[−.069, .071]	−.036	[−.107, .036]	−.037**	[−.051, −.022]
ODD	−.282**	[−.346, −.217]	−.327**	[−.390, −.264]	−.045**	[−.061, −.030]
MDD	−.527**	[−.576, −.479]	−.574**	[−.619, −.530]	−.047**	[−.060, −.034]
BD	−.236**	[−.301, −.171]	−.274**	[−.336, −.211]	−.037**	[−.053, −.022]
GAD	−.606**	[−.651, −.560]	−.616**	[−.661, −.570]	−.010	[−.022, .002]
SOP	−.428**	[−.483, −.373]	−.467**	[−.520, −.414]	−.040**	[−.053, −.026]
SUBS	.013	[−.065, .090]	−.010	[−.087, .067]	−.023**	[−.036, −.010]
GPA	.036	[−.039, .111]	.069 †	[−.005, .144]	.033**	[.016, .051]

Note. All psychopathology scales are dimensional scores representing the sum of the frequency for all items. ADHD = Attention Deficit Hyperactivity Disorder; CD = Conduct Disorder; ODD = Oppositional Defiant Disorder; MDD = Major Depression Disorder; BP = Bipolar Disorder; GAD = Generalized Anxiety Disorder; SOP = Social Phobia; SUBS = Substance Use; GPA = Grade Point Average; 95% CI = 95% confidence interval; Δr = delta parameter (correlation difference estimate).

†

p < .10. *p < .05. **p < .001.

Overall, the delta parameters (Δr) resulting from the comparison tests between the original-content scales and those with added items suggest that that the new scales significantly improved criterion validity. For Openness, the added-item scale increases the negative relations with CD (i.e., it becomes more negative), but reduces the positive relation with BD. For Extraversion, the added-item scale increases the positive correlations with ADHD, ODD, and BD, but reduces the negative relations with MDD and SOP. For Agreeableness, the added-item scale increases the negative relation with ADHD, CD, ODD, and BD, but does not change the relations with internalizing psychopathology scales or GPA. For Conscientiousness, the added-item scale increases the negative relations with all psychopathology scales, but does not change the relations with GPA. Finally, for Emotional Stability, the added-item scale increases the negative relations with ADHD, ODD, MDD, BD, and SOP. However, the confidence intervals of all correlation pairs overlap somewhat.

Construct Validity of the BFPTSQ

For the evaluation of content validity, experts in personality theory and questionnaire construction were asked to rate the adequacy or representativeness of the items. All items were identified as valid indicators of their target trait according to the usual criteria (Polit & Beck, 2006), which suggests that the BFPTSQ scales have adequate content validity. However, three items, all newly added ones, were identified as somewhat less representative, namely Items 42 (Sensation Seeking), 45 (Low Self-Worth), and 48 (Machiavellianism). This suggests that even though most scholars would agree that these items represent important personality traits, there appears to be no clear consensus among experts as to their place within the Big Five taxonomy.

The tests of the BFPTSQ factor validity revealed five notable results. First, the Big Five factor structure was well recovered in a sample of French Canadian adolescents. There are still few published item-level factor analyses of the Big Five structure with adolescents self-reports, but the results of this study tend to replicate those from other studies with adolescent samples (e.g., Allik et al., 2004; McCrae et al., 2002; W.D. Parker & Stumpf, 1998; Soto et al., 2008). Second, all the newly added items were significantly associated with their target Big Five trait. However, some of these added items tended to show low factor loadings. This is perhaps not so surprising because, in accordance with the classic bandwidth-fidelity dilemma (Cronbach & Gleser, 1965), if there is an increase in the conceptual breadth of a scale composed of repeated items measuring only a few primary traits, these new items will inevitably have somewhat lower loadings. Even though factor loadings are commonly considered meaningful when they exceed .30 or .40, this popular rule of thumb is generally not recommended for deciding whether an item is part of a factor. Indeed, as pointed out by Preacher and MacCallum (2003), now that some statistical programs calculate the standard error associated with each factor loading, it is advisable to use statistical significance and confidence intervals. Also be taken into account is the fact that including CUs in an ESEM model typically results in lower loading estimates for some items. In factor analyses of the NEO-FFI items, Marsh et al. (2010) also observed that a number of items show standardized loadings below the common recommendation of .30. Apart from the low loadings, another characteristic of the factor solution observed in this study is that there were a number of complex items (i.e., items with significant loadings on two or more factors). For instance, the Impulsiveness item (48r) added to the Conscientiousness scale is one such item. Even though it would be conceptually preferable to have no such complex items (Donnellan et al., 2006), given that Impulsiveness is such a fundamental personality trait, researchers must deal with the trade-off between a full representation of primary traits within a Big Five measure and the increased complexity of interpreting scale scores with complex items.

A third notable result from factor validity tests is that it is difficult—if even possible—to achieve good model fit to the data by using ICM-CFA for a complex structure like the Big Five when it is measured by several items. Indeed, as recently demonstrated by a number of researchers, it is more appropriate to use ESEM to model all possible factor loadings because it is closer than ICM-CFA to the true model of the Big Five structure (e.g., Furnham, Guenole, Levine, & Chamorro-Premuzic, 2013; Marsh et al., 2010; Rosellini & Brown, 2011). In this study, out of a total of 250 possible factor loadings (number of items times the number of factors), there was a total of 105 (42%) significant cross-loadings. It should be noted, however, that the fit of the ESEM model is acceptable, but far from excellent according to typical criteria suggested for practical fit indices (i.e., Hu & Bentler, 1999). This is perhaps not surprising because as Marsh, Hau, Balla, and Grayson (1998) noted, as the number of indicators in a factor model increases, there tends to be a decrease in fit, even for properly specified models.

A fourth interesting result from the factor analyses is that factor correlations are considerably lower with ESEM than with ICM-CFA. As shown by Marsh et al. (2010), when researchers use ICM-CFA, thereby fixing to 0 the numerous significant cross-loadings that are actually expected in a Big Five personality trait structure, factor correlations are vastly inflated because this is the way these cross-loadings can be represented (see also Asparouhov & Muthén, 2009). Using ESEM provides factor correlations that are probably closer to the true population parameters and supports the discriminant validity among the Big Five traits as measured by the BFPTSQ.

A fifth interesting finding from the factor analyses is the measurement invariance across genders. This is important because to make valid factor mean comparisons, researchers must demonstrate at least that the loadings and intercepts are invariant across groups (Millsap & Olivera-Aguilar, 2012). Unfortunately, these verifications are still rarely done in personality research. The results of this study suggest that it is reasonable to assume that the BFPTSQ factor structure shows measurement invariance across genders. That is, factor loadings, item uniquenesses, CUs, and factor variances/covariances (or correlations) were all fairly invariant across males and females. However, the item intercepts were not fully invariant and four of them were freed across groups. Two of these items were from Openness and were significantly higher for females, and, interestingly, both tap artistic interests (26 and 41r). This suggests that females are significantly more disposed than males to have artistic interests during adolescence. Another intercept tapping joyfulness (47) was significantly higher for females, which suggests they are more prone than males to experience positive emotions during adolescence. Finally, the intercept of a reversed item tapping nervousness (40r) was significantly lower for females, which suggests that females are more predisposed than males to experience anxiety. Non-invariant item intercepts suggest that the mean differences observed across groups in the corresponding factors are not uniquely due to latent factor mean differences, but also in part to differences in these invariant item intercepts (Millsap & Olivera-Aguilar, 2012). However, even though researchers would want all intercepts to be invariant across groups, partial invariance is certainly acceptable for making valid mean comparisons when there are only a few invariant items (Byrne et al., 1989).

Backed up with this measurement invariance, the results showed that there are sizable mean gender differences in adolescents’ personality traits. In order of magnitude, females have significantly lower levels of Emotional Stability, significantly higher levels of Agreeableness and Conscientiousness, and marginally higher levels of Openness. There was no significant gender difference in Extraversion in this sample. These results are consistent with those from a recent meta-analysis with adult samples, except for the absence of differences in Extraversion (Schmitt et al., 2008). However, when gender differences in Extraversion are observed in adolescents, they are typically rather small (Allik et al., 2004). A study by Costa, Terracciano, and McCrae (2001) could help in interpreting these results. These authors observed that while mean gender differences tended to be observed across all primary traits subsumed by Neuroticism and Agreeableness, gender differences were not consistent across primary traits for Openness and Extraversion. For these two broad traits, some primary traits favored women, while others favored men. Differences within broad trait thus explained that mean gender differences were rather small for Openness and Extraversion. It is thus possible that the particular items included in the BFPTSQ scales partly explain the absence of mean difference in Extraversion.

With regard to the reliability of the BFPTSQ scales, overall, the results suggest that the estimates are all acceptable. Of importance, the comparison tests (i.e., delta parameters) between the original-content scales and the BFPTSQ ones with added items revealed that the new items did not improve reliability, nor did they have a detrimental impact. Of course, only one or two items were added per scale, so the absence of significant influence on reliability estimates is not unexpected. Still, it is interesting that reliability estimates were not significantly decreased, considering that some of these new items tended to have somewhat low factor loadings. Another noteworthy observation concerning reliability is that the latent variable model composite reliability (ρ) estimates were systematically lower when models including CUs were considered. It is known that the traditional Cronbach’s alpha coefficient will be accurate only when the items are clearly unidimensional and essentially tau-equivalent and the uniquenesses are uncorrelated (Raykov, 2001, 2012). The results of this study confirmed that CUs that are not accounted for tend to inflate reliability estimates of personality trait scales.

Concerning convergent validity, overall, the correlations with the NEO-PI-3 (McCrae & Costa, 2010) scales suggest adequate validity of the BFPTSQ scales. All the correlations between the broad-trait scales are high, ranging from .629 for Openness to .804 for Conscientiousness. The comparison tests confirmed that, compared with those with original content, the BFPTSQ scales with added items significantly increased these correlations for all Big Five traits. Moreover, the correlations between the BFPTSQ scales and their target NEO-PI-3 primary-trait scales were generally moderate to high, and all were significant. Because nearly 43% (13 out of 30) of the correlations with NEO-PI-3 primary-trait scales did not significantly increase with the added items, this suggests that the correlation increases are not merely a matter of increased general variance. Interestingly, the comparison tests confirmed that the BFPTSQ scales with new items showed significantly higher correlations with a number of NEO-PI-3 primary-trait scales. Critically, the comparison tests generally confirmed expectations regarding the BFPTSQ scales as compared with those with the original content: for Extraversion, the increase in correlation was larger for Excitement Seeking and Positive Emotions; for Agreeableness, it was for Straightforwardness; for Conscientiousness, it was for Deliberation (which could also be called Impulsiveness based on its content); and for Emotional Stability, it was for Depression and Self-Consciousness. Contrary to expectations, however, the BFPTSQ Openness scale with a new item tapping Openness to Cultural Diversity did not significantly increase the relation with the NEO-PI-3 Openness to Values scale, which is surprising since it contains similar items. It should be noted that for comparison tests, when the confidence intervals are considered, all overlap somewhat. Even though nonoverlapping 95% confidence intervals are rather stringent tests of the difference between two estimates (Cumming, 2012), this suggest that the correlation increase flagged as statistically significant as per the delta parameters tends to be small.

As for criterion validity, overall the correlations with the outcome measures suggest adequate concurrent validity of the BFPTSQ scales. As expected based on scholarly reviews (Klein et al., 2012; Tackett, Martel, et al., 2012; Widiger & Smith, 2008) and meta-analytic findings derived with adult samples (Malouff et al., 2005), externalizing psychopathology scales tended to be related to low levels of Agreeableness and Conscientiousness and, to a lesser extent, to low levels of Emotional Stability and high levels of Extraversion. However, the BFPTSQ Extraversion scale did not show a significantly higher correlation for substance use. A higher correlation was expected because of the added Sensation Seeking item, which is typically related to substance use (Sargant et al., 2010; Zuckerman, 2009). Internalizing psychopathology scales, meanwhile, tended to be related to low levels of Emotional Stability, Agreeableness, and Conscientiousness and, to a lesser extent, to low levels of Extraversion. An exception is bipolar disorder, which was related to high levels of Extraversion, even though it is considered a mood disorder. As expected based on Malouff et al.’s (2005) meta-analytic findings, Openness showed small but significant relations to some psychopathology scales. Indeed, this meta-analysis found that for self-reports, Openness is positively related to psychopathology, while for observer ratings, there is a negative relation. In the present study, Openness was negatively related to CD and positively related to MDD, BD and GAD. However, compared with the other correlations, they tended to be small. Moreover, as expected based on meta-analytic findings (Poropat, 2009), Conscientiousness and Openness were both positively related to GPA. In this sample of adolescents, Agreeableness was also clearly related to GPA, much more strongly than is typically observed in adult samples (Noftle & Robins, 2007). The comparison tests generally confirmed that, compared with those with the original content, the BFPTSQ scales with added items provide several significantly higher correlations with outcome scales. Because 33% (15 out of 45) of the correlations with outcome scales did not increase significantly with the added items, this again suggests that the correlation increases are not simply a matter of increased general variance. When the 95% confidence intervals are considered for comparison tests, all overlap somewhat, which suggests that the correlation increase flagged as statistically significant as per the delta parameters tends to be small.

Impact of Added Conceptual Breadth

This study showed that by adding only a few items tapping important primary personality traits, factor validity is adequate, reliability is not affected, but convergent and criterion validities are somewhat improved. Even if the BFPTSQ is far from being optimal in terms of content coverage and psychometric properties, the results of this study underscore the significance of its added conceptual breadth.

Because of the limited time available in many assessment situations, short personality measures are popular. In principle, to be valid, short measures should have items tapping all of the most important primary traits subsumed in a given Big Five trait (Haynes et al., 1995; Smith et al., 2003). Therefore, the minimal number of items in a short scale should arguably be the number of primary traits encompassed in a given Big Five trait. If some items tapping fundamental primary traits do not load strongly (but significantly) on their target factor, they should not be excluded a priori for this reason. In other words, results of factor or reliability analyses should not supersede issues of conceptual breadth and validity in questionnaire construction (Cizek, Rosenberg, & Koos, 2008; Loevinger, 1957). This should rather be taken as a challenge for personality researchers to find the place of these primary personality traits in the complex multivariate space of the Big Five taxonomy. This challenge is difficult, however, because even though personality psychologists have reached a reasonable consensus on the hierarchical nature and number of broad personality traits (Markon et al., 2005), they are not even close to a consensus concerning the nature and number of fundamental primary traits.

Limitations and Future Research

The BFPTSQ psychometric properties evaluated in this study are generally satisfactory, but a few limitations should be mentioned. First, all the psychometric properties of the BFPTSQ were evaluated for one sample of adolescents. Even though efforts were made to gather a large, very roughly representative sample of adolescents, the results need to be replicated. Second, because studying personality trait development is important, a cross-validation should be carried out with an adult sample. Third, the French-language version of the BFPTSQ was validated, so it would be important to replicate these results using the English- or other-language versions. Fourth, it would be interesting to evaluate whether the factor structure and measurement invariance across genders can be replicated with reports from informants, such as parents and teachers. Fifth, there is a potential shared method effect in the criterion validity tests because self-report measures were used for the assessment of both personality traits and outcomes. It would be important to replicate these results using different informants or methods. Sixth, the evaluation of criterion validity was conducted with a limited number of outcomes (i.e., psychopathology and achievement). Clearly, personality traits have been shown to be associated with a wide range of other positive and negative life outcomes, so additional predictive studies using these new scales are needed. Seventh, this study provided only a preliminary evaluation of incremental validity. Indeed, before making any strong claims about the incremental validity of the BFPTSQ, it would be important to demonstrate that its scales significantly add to the prediction of consequential outcomes beyond what can be predicted by scales from other known short Big Five measures. Despite these limitations, the results of this study suggest that the BFPTSQ appears to be a potentially useful alternative to existing Big Five short measures.