An Updated Guideline for Assessing Discriminant Validity

Origin of the Concept of Discriminant Validity

The term “discriminant validity” was coined by Campbell and Fiske (1959), who presented a validation technique based on the long-standing idea that tests can be invalidated by too high correlations with unrelated tests (Campbell, 1960; Thorndike, 1920). However, the term was introduced without a clear definition of the concept (Reichardt & Coleman, 1995); instead, the article focuses on discriminant validation or how discriminant validity can be shown empirically using multitrait-multimethod (MTMM) matrices. The original criteria, illustrated in Table 3, were as follows: (a) two variables that measure the same trait (T1) with two different methods (M1, M2) should correlate more highly than any two variables that measure two different traits (T1, T2) with different methods (M1, M2); (b) two variables that measure the same trait (T1) with two different methods (M1, M2) should correlate more highly than any two variables that measure two different traits (T1, T2) but use the same method (M1); and (c) the pattern of correlations between variables that measure different traits (T1, T2) should be very similar across different methods (M1, M2) (Campbell & Fiske, 1959).

OPEN IN VIEWER

Table 3.

Multitrait-Multimethod Correlation Matrix and Original Criteria for Discriminant Validity.

		Method M1			Method M2
	Traits	T1	T2	T3	T1	T2	T3
M1	T1	1						Discriminant Validity:All MTHM > HTHMAll MTHM > HTMMHTMM11 ≈ HTMM31 HTMM12 ≈ HTMM32 HTMM13 ≈ HTMM33
	T2	HTMM11	1
	T3	HTMM12	HTMM13	1
M2	T1	MTHM21	HTHM24	HTHM27	1
	T2	HTHM22	MTHM25	HTHM28	HTMM31	1
	T3	HTHM23	HTHM26	MTHM29	HTMM32	HTMM33	1

Note: HTMM = same method and different traits (heterotrait-monomethod); MTHM = different methods and same trait (monotrait-heteromethod); HTHM = different methods and different traits (heterotrait-heteromethod).

Generalizing the concept of discriminant validity outside MTMM matrices is not straightforward. Indeed, the definitions shown in Table 2 show little connections to the original MTMM matrices. A notable exception is Reichardt and Coleman (1995), who criticized the MTMM-based criteria for being dichotomous and declared that a natural (dichotomous) definition of discriminant validity outside the MTMM context would be that two measures x1 and x2 have discriminant validity if and only if x1 measures construct T1 but not T2, x2 measures T2 but not T1, and the two constructs are not perfectly correlated. They then concluded that a preferable continuous definition would be “the degree to which the absolute value of the correlation between the two constructs differs from one” (Reichardt & Coleman, 1995, p. 516). A similar interpretation was reached by McDonald (1985), who noted that two tests have discriminant validity if “the common factors are correlated, but the correlations are low enough for the factors to be regarded as distinct ‘constructs’” (p. 220).

Constructs and Measures

Discriminant validity is sometimes presented as the property of a construct (Reichardt & Coleman, 1995) and other times as the property of its measures or empirical representations constructed from those measures (McDonald, 1985). This ambiguity may stem from the broader confusion over common factors and constructs: The term “construct” refers to the concept or trait being measured, whereas a common factor is part of a statistical model estimated from data (Maraun & Gabriel, 2013). Indeed, Campbell and Fiske (1959) define validity as a feature of a test or measure, not as a property of the trait or construct being measured. In fact, if one takes the realist perspective that constructs exist independently of measurement and can be measured in multiple different ways (Chang & Cartwright, 2008), ¹ it becomes clear that we cannot use an empirical procedure to define a property of a construct.

The Role of the Factor Model

Factor analysis has played a central role in articles on discriminant validation (e.g., McDonald, 1985), but it cannot serve as a basis for a definition of discriminant validity for two reasons. First, validity is a feature of a test or a measure or its interpretation (Campbell & Fiske, 1959), not of any particular statistical analysis. Moreover, discriminant validity is often presented as a property of “an item” (Table 2), implying that the concept should also be applicable in the single-item case, where factor analysis would not be applicable. Second, a factor model where each item loads on only one factor may be too constraining for applied research (Asparouhov et al., 2015; Marsh et al., 2014; Morin et al., 2017; Rodriguez et al., 2016). For example, Marsh et al. (2014) note that in psychology research, the symptoms or characteristics of different disorders commonly overlap, producing nonnegligible cross-loadings in the population. Constraining these cross-loadings to be zero can inflate the estimated factor correlations, which is problematic, particularly for discriminant validity assessment (Marsh et al., 2014). Moreover, a linear model where factors, error terms, and observed variables are all continuous (Bartholomew, 2007) is not always realistic. Indeed, a number of item response theory models have been introduced (Foster et al., 2017; Reise & Revicki, 2014) to address these scenarios and have been applied to assess discriminant validity using MTMM data (Jeon & Rijmen, 2014).

Generalized Definition of Discriminant Validity

We present a definition that does not depend on a particular model and makes it explicit that discriminant validity is a feature of a measure instead of a construct: ² Two measures intended to measure distinct constructs have discriminant validity if the absolute value of the correlation between the measures after correcting for measurement error is low enough for the measures to be regarded as measuring distinct constructs.

This definition encompasses the early idea that even moderately high correlations between distinct measures can invalidate those measures if measurement error is present (Thorndike, 1920), which serves as the basis of discriminant validity (Campbell & Fiske, 1959). The definition can also be applied on both the scale level and the scale-item level. Consider the proposed definition in the context of the common factor model:

Σ = Λ Φ Λ ′ + Θ

1

where Σ is the interitem correlation matrix; Φ is the factor correlation matrix, where all correlations are assumed to be positive for simplicity; Λ is a factor pattern (loading) matrix; and Θ is the item error covariance matrix. Within this context, our definition can be understood in two equivalent ways:

Σ i , j ≪ ΛJ Λ i,j ′

2

( Λ ′ Λ ) -1 Λ ′ ( Σ − Θ ) Λ ( Λ ′ Λ ) k,l −1 ≪ 1

3

where J is a unit matrix (a matrix of ones) and ≪ denotes much less than. Equation 2 is an item-level comparison (category 2 in Table 2), where the correlation between items i and j, which are designed to measure different constructs, is compared against the implied correlation when the items depend on perfectly correlated factors but are not perfectly correlated because of measurement error. Equation 3 shows an equivalent scale-level comparison (part of category 1 in Table 2) focusing on two distinct scales k and l. The factor correlations are solved from the interitem correlations by multiplying with left and right inverses of the factor pattern matrix to correct for measurement error and are then compared against a perfect correlation. Generalizing beyond the linear common factor model, Equation 3 can be understood to mean that two scales intended to measure distinct constructs have discriminant validity if the absolute value of the correlation between two latent variables estimated from the scales is low enough for the latent variables to be regarded as representing distinct constructs.

Our definition has several advantages over previous definitions shown in Table 2. First, it clearly states that discriminant validity is a feature of measures and not constructs and that it is not tied to any particular statistical test or cutoff (Schmitt, 1978; Schmitt & Stults, 1986). Second, the definition is compatible with both continuous and dichotomous interpretations, as it suggests the existence of a threshold, that is, a correlation below a certain level has no problem with discriminant validity but does not dictate a specific cutoff, thus also allowing the value of the correlation to be interpreted instead of simply tested. Third, the definition is not tied to any particular measurement process (e.g., single administration) but considers measurement error generally, thus supporting rater, transient, and other errors (Le et al., 2009; Schmidt et al., 2003). Fourth, the definition is not tied to either the individual item level or the multiple item scale level but works across both, thus unifying the category 1 and category 2 definitions of Table 2. Fifth, the definition does not confound the conceptually different questions of whether two measures measure different things (discriminant validity) and whether the items measure what they are supposed to measure and not something else (i.e., lack of cross-loadings in Λ , factorial validity), ³ which some of the earlier definitions (categories 3 and 4 in Table 2) do.

This definition also supports a broad range of empirical practice: If considered on the scale level, the definition is compatible with the current tests, including the original MTMM approach (Campbell & Fiske, 1959). However, it is not limited to simple linear common factor models where each indicator loads on just one factor but rather supports any statistical technique including more complex factor structures (Asparouhov et al., 2015; Marsh et al., 2014; Morin et al., 2017; Rodriguez et al., 2016) and nonlinear models (Foster et al., 2017; Reise & Revicki, 2014) as long as these techniques can estimate correlations that are properly corrected for measurement error and supports scale-item level evaluations.

Techniques That Assess Correlations

There are three main ways to calculate a correlation for discriminant validity assessment: a factor analysis, a scale score correlation, and the disattenuated version of the scale score correlation. While item-level correlations or their disattenuated versions could also be applied in principle, we have seen this practice neither recommended nor used. Thus, in practice, the correlation techniques always correspond to the empirical test shown as Equation 3. Regardless of how the correlations are calculated, they are used in just two ways: either by comparing the point estimates against a cutoff (i.e., the square root of the AVE) or by checking whether the absolute value of the CI contains either zero or one. Using the cutoff of zero is clearly inappropriate as requiring that two factors be uncorrelated is not implied by the definition of discriminant validity and would limit discriminant validity assessment to the extremely rare scenario where two constructs are assumed to be (linearly) independent. We will next address the various techniques in more detail.

Factor Analysis Techniques

Factor correlations can be estimated directly either by exploratory factor analysis (EFA) or CFA, but because none of the reviewed guidelines or empirical applications reported EFA correlations, we focus on CFA. The estimation of factor correlations in a CFA is complicated by the fact by default latent variables are scaled by fixing the first indicator loadings, which produces covariances that are not correlations. Correlations (denoted ρ CFA ) can be estimated by either freeing the factor loadings and scaling the factors by fixing their variances to 1 (i.e., A in Figure 2) or standardizing the factor covariance matrix (i.e., B), for example, by requesting standardized estimates that all SEM software provides. Both techniques produce the same estimate, although the standard errors (and CIs) can be different (Gonzalez & Griffin, 2001).

OPEN IN VIEWER

Figure 2.

Factor correlation estimation. (A) Fixing the variances of factors to unity (i.e., not using the default option). (B) Fixing one of the loadings to unity (i.e., using the default option).

Of the correlation estimation techniques, CFA is the most flexible because it is not tied to a particular model but requires only that the model be correctly specified. Thus, cross-loadings, nonlinear factor loadings or nonnormal error terms can be included because a CFA model can also be used in the context of item response models (Foster et al., 2017), bifactor models (Rodriguez et al., 2016), exploratory SEMs (Marsh et al., 2014) or other more advanced techniques. However, these techniques tend to require larger sample sizes and advanced software and are consequently less commonly used.

Scale Score Correlations and Disattenuated Correlations

The simplest and most common way to estimate a correlation between two scales is by summing or averaging the scale items as scale scores and then taking the correlation (denoted ρ SS ). ⁴ The problem with this approach is that the scores contain measurement errors, which attenuate the correlation and may cause discriminant validity issues to go undetected. ⁵ To address this issue, the use of disattenuated or error-corrected correlations where the effect of unreliability is removed is often recommended (Edwards, 2003; J. A. Shaffer et al., 2016):

ρ 12 = ρ X Y ρ X ρ Y

4

where the disattenuated correlation ( ρ 12 ) is a function of the scale score correlation ( ρ X Y ) and the scale score reliabilities ( ρ X , ρ Y ). That is, a disattenuated correlation is the scale score correlation from which the effect of unreliability is removed. ⁶

Reliability can be estimated in different ways, including test-retest reliability, interrater reliability and single-administration reliability, ⁷ which each provide information on different sources of measurement error (Le et al., 2009; Schmidt et al., 2003). Given our focus on single-method and one-time measurements, we address only single-administration reliability, where measurement errors are operationalized by uniqueness estimates, ignoring time and rater effects that are incalculable in these designs. The two most commonly used single-administration reliability coefficients are tau-equivalent reliability, ⁸ often referred to as Cronbach’s alpha, and congeneric reliability, usually called composite reliability by organizational researchers and McDonald’s omega or ω by methodologists. ⁹ As the names indicate, the key difference is whether we assume that the items share the same true score (tau-equivalent, B in Figure 3) or make the less constraining assumption that the items simply depend on the same latent variable but may do so to different extents (congeneric, C in Figure 3). Recent research suggests that the congeneric reliability coefficient is a safer choice because of the less stringent assumption (Cho, 2016; Cho & Kim, 2015; McNeish, 2017).

OPEN IN VIEWER

Figure 3.

The assumptions of parallel, tau-equivalent, and congeneric reliability. (A) parallel, (B) tau-equivalent, (C) congeneric.

The reliability coefficients presented above make a unidimensionality assumption, which may not be realistic in all empirical research. While the disattenuation formula (Equation 4) is often claimed to assumes that the only source of measurement error is random noise or unreliability, the assumption is in fact more general: All variance components in the scale scores that are not due to the construct of interest are independent of the construct and measurement errors of other scale scores. This more general formulation seems to open the option of using hierarchical omega (Cho, 2016; Zinbarg et al., 2005), which assumes that the scale measures one main construct (main factor) but may also contain a number of minor factors that are assumed to be uncorrelated with the main factor. However, using hierarchical omega for disattenuation is problematic because it introduces an additional assumption that the minor factors (e.g., disturbances in the second-order factor model and group factors in the bifactor model) are also uncorrelated between two scales, which is neither applied nor tested when reliability estimates are calculated separately for both scales, as is typically the case. While the basic disattenuation formula has been extended to cases where its assumptions are violated in known ways (Wetcher-Hendricks, 2006; Zimmerman, 2007), the complexities of modeling the same set of violations in both the reliability estimates and the disattenuation equation do not seem appealing given that the factor correlation can be estimated more straightforwardly with a CFA instead.

While simple to use, the disattenuation correction is not without problems. A common criticism is that the correction can produce inadmissible correlations (i.e., greater than 1 or less than –1) (Charles, 2005; Nimon et al., 2012), but this issue is by no means a unique problem because the same can occur with a CFA. However, a CFA has three advantages over the disattenuation equation. First, CFA correlations estimated with maximum likelihood (ML) can be expected to be more efficient than multistep techniques that rely on corrections for attenuation (Charles, 2005; Muchinsky, 1996). Second, the disattenuation equation assumes that the scales are unidimensional and that all measurement errors are uncorrelated, whereas a CFA simply assumes that the model is correctly specified and identified. Third, calculating the CIs for a disattenuated correlation is complicated (Oberski & Satorra, 2013). However, this final concern can be alleviated to some extent through the use of bootstrap CIs (Henseler et al., 2015); in particular, the bias-corrected and accelerated (BCa) technique has been shown to work well for this particular problem (Padilla & Veprinsky, 2012, 2014).

AVE/SV or Fornell-Larcker Criterion

While commonly used, the AVE statistic has been rarely discussed by methodological research and, consequently, is poorly understood. ¹⁰ One source of confusion is the similarity between the formula for AVE and that of congeneric reliability (Fornell & Larcker, 1981a):

AVE = ∑ i = 1 k λ i 2 ∑ i = 1 k λ i 2 + ∑ i = 1 k σ e i 2

5

CR = ( ∑ i = 1 k λ i ) 2 ( ∑ i = 1 k λ i ) 2 + ∑ i = 1 k σ e i 2

6

The meaning of AVE becomes more apparent if we rewrite the original equation as:

AVE = ∑ i = 1 k σ x i 2 ρ i ∑ i = 1 k σ x i 2

7

where ρ i = λ i 2 σ x i 2 is the reliability and σ x i 2 = λ i 2 + σ e i 2 is the variance of item i. This alternative form shows that AVE is actually an item-variance weighted average of item reliabilities. With standardized estimates, AVE reduces to an average of item reliabilities. Thus, the term “average indicator reliability” might be more informative than “average variance extracted.”

Fornell and Larcker (1981a) presented a decision rule that discriminant validity holds for two scales if the AVEs for both are higher than the squared factor correlation between the scales. We refer to this rule as AVE/SV because the squared correlation quantifies shared variance (SV; Henseler et al., 2015). Because a factor correlation corrects for measurement error, the AVE/SV comparison is similar to comparing the left-hand side of Equation 3 against the right-hand side of Equation 2. Therefore, AVE/SV has a high false positive rate, indicating a discriminant validity problem under conditions where most researchers would not consider one to exist, as indicated by A in Figure 4. This tendency has been taken as evidence that AVE/SV is “a very conservative test” (Voorhees et al., 2016, p. 124), whereas the test is simply severely biased.

OPEN IN VIEWER

Figure 4.

The inappropriateness of the AVE as an index of discriminant validity. (A) despite high discriminant validity, the AVE/SV criterion fails, (B) despite low discriminant validity, the AVE/SV criterion passes.

In applied research, the AVE/SV criterion rarely shows a discriminant validity problem because it is commonly misapplied. The most common misapplication is to compare the AVE values with the square of the scale score correlation, not the square of the factor correlation (Voorhees et al., 2016). Another misuse is to compare AVEs against the .5 rule-of-thumb cutoff, which Fornell and Larcker (1981a) presented as a convergent validity criterion. Other misuses are that only one of the two AVE values or the average of the two AVE values should be greater than the SV (Farrell, 2010; Henseler et al., 2015). The original criterion is that both AVE values must be greater than the SV. Finally, the AVE statistic is sometimes calculated from a partial least squares analysis (AVE_PLS), which overestimates indicator reliabilities and thus cannot detect even the most serious problems (Rönkkö & Evermann, 2013).

Heterotrait-Monotrait Ratio (HTMT)

The heterotrait-monotrait (HTMT) ratio was recently introduced in marketing (Henseler et al., 2015) and is being adopted in other disciplines as well (Kuppelwieser et al., 2019). While Henseler et al. (2015) motivate HTMT based on the original MTMM approach (Campbell & Fiske, 1959), this index is actually neither new nor directly based on the MTMM approach. The original version of the HTMT equation is fairly complex, but to make its meaning more apparent, it can be simplified as follows:

HTMT i j = σ i j ¯ σ i ¯ σ j ¯

8

where σ i ¯ and σ j ¯ denote the average within scale item correlation and σ i j ¯ denotes the average between scale item correlation for two scales i and j. This simpler form makes it clear that HTMT is related to the disattenuation formula (Equation 4). In fact, HTMT is equivalent to a disattenuated correlation of unit-weighted composites using parallel reliability (i.e., the standardized alpha, proof in the appendix). Thus, while marketed as a new technique, the HTMT index has actually been used for decades; parallel reliability is the oldest reliability coefficient (Brown, 1910), and disattenuated correlations have been used to assess discriminant validity for decades (Schmitt, 1996). Thus, the term “HTMT” is misleading, giving the false impression that HTMT is related to MTMM and obscuring the fact that it is simply a variant of disattenuated correlation. Thus, we favor a more transparent term “disattenuated correlation using parallel reliability” (denoted ρ D P R ) because this more systematic name tells what the coefficient is (i.e., correlation), how it is obtained (i.e., disattenuated), and under what conditions it can be used (i.e., parallel reliability).

Because the ρ D P R statistic is a disattenuated correlation, it shares all the interpretations, assumptions, and limitations of the techniques that were explained earlier. Compared to the tau-equivalence assumption, this technique makes an even more constraining parallel measurement assumption that the error variances between items are the same (A in Figure 3). Most real-world data deviate from these assumptions, in which case ρ D P R yields inaccurate estimates (Cho, 2016; McNeish, 2017), making this technique an inferior choice.

Model Fit Comparison Techniques

Model comparison techniques involve comparing the original model against a model where a factor correlation is fixed to a value high enough to be considered a discriminant validity problem. Their general idea is that if the two models fit equally well, the model with a discriminant validity problem is plausible, and thus, there is a problem. The most common constraints are that (a) two factors are fixed to be correlated at 1 (i.e., A in Figure 5) or (b) two factors are merged into one factor (i.e., C in Figure 5), thus reducing their number by one.

OPEN IN VIEWER

Figure 5.

(A) constrained model for χ²(1), (B) common misuse of χ²(1), (C) constrained model for χ²(merge), (D) model equivalent to C.

The key advantage of these techniques is that they provide a test statistic and a p-value. However, this also has the disadvantage that it steers a researcher toward making yes/no decisions instead of assessing the degree to which discriminant validity holds in the data. Compared to correlation-based techniques, where a single CFA model provides all the estimates required for discriminant validity assessment, model comparison techniques require more work because a potentially large number of comparisons must be managed. ¹¹ We next assess the various model comparison techniques presented and used in the literature.

CFI(1)

To address the issue that the χ 2 ( 1 ) test can flag correlations that differ from 1 by trivial amounts as significant, some recent articles (Le et al., 2010; J. A. Shaffer et al., 2016) have suggested comparing models by calculating the difference between the comparative fit indices (CFIs) of two models (ΔCFI), which is compared against the .002 cutoff (CFI(1)). This idea is adapted from Cheung and Rensvold’s (2002) proposal in the measurement invariance literature, and the .002 cutoff is based on the simulation by Meade et al. (2008).

The idea behind using ΔCFI in measurement invariance assessment is that the degrees of freedom of the invariance hypothesis depend on the model complexity, and the CFI index and consequently ΔCFI are less affected by this than the χ 2 (Meade et al., 2008). This idea is reasonable in the original context, but it does not apply in the context of CFI(1) comparison where the difference in degrees of freedom is always one, leaving this test without its main justification. Indeed, CFI(1) can be proved (see the appendix) to be equivalent to calculating the Δ χ 2 and comparing this statistic against a cutoff defined based on the fit of the null model ( χ B 2 ) and its degrees of freedom ( d f B ) ,

Δ χ 2 > 1 + .002 ( χ B 2 − d f B )

9

instead of using the .95 percentile from the χ ( 1 ) 2 distribution, or 3.84, as a cutoff. Because the expected value of χ B 2 increases with sample size in a way similar to Δ χ 2 , this comparison can be regarded as less sensitive to sample size. However, it is unclear whether this alternative cutoff has more or less power (i.e., whether 1 + .002 ( χ B 2 − d f B ) is greater or less than 3.84) because the effectiveness of CFI(1) has not been studied.

Changing the test statistic—or equivalently the cutoff value—is an ultimately illogical solution because the problem with the χ 2 ( 1 ) test is not that its power increases with sample size but that a researcher is ultimately not interested in whether the correlation between two variables differs from exactly 1; rather, a researcher is interested in whether the correlation is sufficiently different from 1. Thus, a more logical approach is to change the null hypothesis instead of adjusting the tests to be less powerful.

χ 2 ( c u t ) and CFI ( c u t )

The lack of a perfect correlation between two latent variables is ultimately rarely of interest, and thus, it is more logical to use a null hypothesis that covers an interval (e.g., ϕ 12 >.9). This test can be implemented in any SEM software by first fitting a model where ϕ 12 is freely estimated. If the estimate falls outside the interval (e.g., less than .9), then the correlation is constrained to be at the endpoint of the interval, and the model is re-estimated. A significant result from a nested model comparison means that the original interval hypothesis can be rejected. We use the labels χ 2 ( cut ) and CFI ( cut ) to denote tests that depend on whether the comparison is performed based on the χ 2 test or by comparing Δ CFI against the .002 cutoff. While the idea that a number less than 1 can be used as a cutoff was briefly mentioned in John and Benet-Martínez (2000) and J. A. Shaffer et al. (2016), we are unaware of any studies that have applied interval hypothesis tests or tested their effectiveness.

χ 2 ( m e r g e ) and Other Comparisons Against Models with Fewer Factors

Many studies assess discriminant validity by comparing the hypothesized model with a model with fewer factors. This is typically done by merging two factors (i.e., C in Figure 5), and we refer to the associated nested model comparison as χ 2 ( merge ) . As mentioned above, this test is a more constrained version of χ 2 ( 1 ) where all pairs of correlations with either of the factors and a third factor are constrained to be the same (i.e., D in Figure 5). Because this test imposes more constraints than χ 2 ( 1 ) does, it has more statistical power. It also avoids the inadmissible solution issue of χ 2 ( 1 ) . Merging two factors will always produce the same χ 2 regardless of how the latent variables are scaled, and thus, this test is less likely to be incorrectly applied. However, the test also has a major weakness that, in contrast to χ 2 ( 1 ) , this test cannot be extended to other cutoffs (i.e., χ 2 ( cut ) ). Thus, while the test can be applied for testing perfect overlap between two latent variables, it cannot answer the question of whether the latent variables are sufficiently distinct.

As with the other techniques, various misconceptions and misuses are found among empirical studies. The most common misuse is to include unnecessary comparisons, for example, by testing alternative models with two or more factors less than the hypothesized model. Another common misuse is to omit some necessary comparisons, for example, by comparing only some alternative models, instead of comparing all possible alternative models with one factor less than the original model.

Single Model Fit Techniques

Discriminant validity has also been assessed by inspecting the fit of a single model without comparing against another model. These techniques fall into two classes: those that inspect the factor loadings and those that assess the overall model fit.

Cross-Loadings

Cross-loadings indicate a relationship between an indicator and a factor other than the main factor on which the indicator loads. Beyond this definition, the term can refer to two distinct concepts, factor pattern coefficients or factor structure coefficients (see Table 5), and has been confusingly used with both meanings in the discriminant validity literature. ¹⁵ Structure coefficients are correlations between items and factors, so their values are constrained to be between –1 and 1. Pattern coefficients, on the other hand, are analogous to (standardized) coefficients in regression analysis and are directional (Thompson & Daniel, 1996). Structure coefficients are not directly estimated as part of a factor analysis; instead, they are calculated based on the pattern coefficients and factor correlations. If the factors are rotated orthogonally (e.g., Varimax) or are otherwise constrained to be uncorrelated, the pattern coefficients and structure coefficients are identical (Henson & Roberts, 2006). However, the use of uncorrelated factors can rarely be justified (Fabrigar et al., 1999), which means that, in most cases, pattern and structure coefficients are not equal.

OPEN IN VIEWER

Table 5.

Factor Pattern Coefficients, Correlations, and Structure Coefficients.

	Definitions				Numerical Example
Items	Factor Pattern Coefficients		Factor Structure Coefficients		Factor Pattern Coefficients		Factor Structure Coefficients
	Factor1	Factor2	Factor1	Factor2	Factor1	Factor2	Factor1	Factor2
x1	λ 11	0	λ 11	λ 11 ρ	.70	0	.70	.35
x2	λ 21	0	λ 21	λ 21 ρ	.60	0	.60	.30
x3	λ 31	0	λ 31	λ 31 ρ	.80	0	.80	.40
x4	0	λ 42	λ 42 ρ	λ 42	0	.90	.45	.90
x5	0	λ 52	λ 52 ρ	λ 52	0	.70	.35	.70
x6	0	λ 62	λ 62 ρ	λ 62	0	.80	.40	.80

Note: Factor correlation ρ = .5 . The factor structure coefficients are the matrix multiplication of the factor pattern coefficients and factor correlations.

While both the empirical criteria shown in Equation 2 and Equation 3 contain pattern coefficients, assessing discriminant validity based on loadings is problematic. First, these comparisons involve assessing a single item or scale at a time, which is incompatible with the idea that discriminant validity is a feature of a measure pair. Second, pattern coefficients do not provide any information on the correlation between two scales, and structure coefficients are an indirect measure of the correlation at best. ¹⁶ Third, while the various guidelines differ in how loadings should be interpreted (Henseler et al., 2015; Straub et al., 2004; Thompson, 1997), they all share the features of relying mostly on authors’ intuition instead of theoretical reasoning or empirical evidence. In summary, these techniques fall into the rules of thumb category and cannot be recommended.

Summary: Under What Conditions Is Each Technique Useful?

Our review of the literature provides several conclusions. Overall, χ 2 ( cut ) and CI_CFA (cut) can be recommended as general solutions because they meet the definition of discriminant validity, have the flexibility to adapt to various levels of cutoffs, and can be extended to more complex scenarios such as nonlinear measurement models (Foster et al., 2017), scales with minor dimensions (Rodriguez et al., 2016), or cases in which factorial validity is violated because of cross-loadings. Some of the other techniques can be useful for specific purposes. χ 2 ( merge ) , χ 2 ( 1 ) , and CI_CFA(1) can be used if theory suggests nearly perfect but not absolutely perfect correlations. They can also be useful as a first step in discriminant validity assessment; if any of them indicates a problem, then so will any variant of the techniques that use a cutoff of less than 1. Disattenuated correlations are useful in single-item scenarios, where reliability estimates could come from test-retest or interrater reliability checks or from prior studies. These techniques could also be used in multiple item scenarios, if a researcher does not have access to SEM software, or in some small sample scenarios (Rosseel, 2020). If the effect of measurement error can be assumed to be negligible, even using scale score correlations can be useful as a rough check. The techniques that assess the lack of cross-loadings (pattern coefficients) and model fit provide (factorial) validity information, which is important in establishing the assumptions of the other techniques, but these techniques are of limited use in providing actual discriminant validity evidence.

Simulation Design

For simplicity, we followed the design used by Voorhees et al. (2016) and generated data from a three-factor model. We assessed the discriminant validity of the first two factors, varying their correlation as an experimental condition. Voorhees et al. (2016) used only two factor correlation levels, .75 and .9. We wanted a broader range from low levels where discriminant validity is unlikely to be a problem up to perfect correlation, so we used six levels: .5, .6, .7, .8, .9, and 1. The third factor was always correlated at .5 with the first two factors. The number of items was varied at 3, 6, and 9, and factor loadings were set to [0.9,0.8,0.8], [0.8,0.7,0.7], [0.5,0.4,0.4], [0.9,0.6,0.3], and [0.8, 0.8, 0.8]. For the six- and nine-item scenarios, each factor loading value was used multiple times. All factors had unit variances in the population, and we scaled the error variances so that the population variances of the items were one. Cross-loadings (in the pattern coefficients) were either 0, 1, or 2. This condition was implemented following the approach by Voorhees et al. (2016); in the cross-loading condition, either the first or first two indicators of the second latent variable also loaded on the first latent variable. The same value was used for both loadings, and the values were scaled down from their original values so that the factors always explained the same amount of variance in the indicators. In the six- and nine-item conditions, the number of cross-loaded items was scaled up accordingly. Sample size was the final design factor and varied at 50, 100, 250, and 1,000. The full factorial (6 × 3 × 5 × 3 × 4) simulation was implemented with the R statistical programming environment using 1,000 replications for each cell. We estimated the factor models with the lavaan package (Rosseel, 2012) and used semTools to calculate the reliability indices (Jorgensen et al., 2020). The full simulation code is available in Online Supplement 2, and the full set of simulation results at the design level can be found in Online Supplement 3.

Correlation Estimates and Their Confidence Intervals

We first focus on the scenarios where the factor model was correctly specified (i.e., there were no cross-loadings). Because ρ DPR and HTMT were proven equivalent and always produced identical results, we report only the former. We also omit the two low correlation conditions (i.e., .5, .6) because the false positive rates are already clear in the .7 condition.

Because the pattern of results is very similar for correlations and their confidence intervals, we present both sets of results together. Table 6 shows the correlation estimates by sample size, number of items, and factor loading conditions. Table 7 shows the coverage and the balance of the CIs by sample size and selected values of loading condition, omitting ρ SS because of its generally poor performance in the correlation results. Ideally, the coverage of a 95% CI should be .95, and the balance should be close to zero. The CIs for ρ CFA were obtained from the CFAs, and for ρ DPR , we used bootsrap percentile CIs, following Henseler et al. (2015). For comparison, we also calculated the bootstrap percentile CIs for ρ DTR and ρ DCR . All bootstrap analyses were calculated with 1,000 replications.

OPEN IN VIEWER

Table 6.

Correlation Estimates by Sample Size, Number of Items, and Loadings.

Sample Size, Indicators, Loadingsa	Technique	Factor Correlation
		.7		.8		.9		1.0
		M	SD	M	SD	M	SD	M	SD
50	ρ CFA	.694	.111	.801	.088	.897	.070	1.000	.049
	ρ DPR	.697	.108	.802	.086	.901	.066	1.003	.047
	ρ DTR	.698	.108	.804	.086	.903	.066	1.005	.048
	ρ DCR	.694	.108	.800	.087	.898	.066	1.000	.047
	ρ SS	.583	.095	.672	.080	.753	.065	.838	.045
100	ρ CFA	.700	.076	.801	.061	.899	.045	1.000	.032
	ρ DPR	.700	.075	.802	.059	.900	.044	1.002	.031
	ρ DTR	.701	.075	.803	.060	.901	.044	1.003	.031
	ρ DCR	.699	.076	.801	.060	.899	.044	1.000	.031
	ρ SS	.588	.068	.673	.057	.755	.042	.841	.030
250	ρ CFA	.699	.043	.799	.038	.899	.028	1.001	.019
	ρ DPR	.699	.043	.799	.037	.900	.027	1.001	.019
	ρ DTR	.700	.043	.800	.037	.900	.027	1.002	.019
	ρ DCR	.699	.043	.799	.037	.899	.027	1.001	.019
	ρ SS	.588	.039	.672	.035	.757	.027	.842	.018
1,000	ρ CFA	.699	.023	.800	.018	.899	.014	1.000	.009
	ρ DPR	.699	.023	.800	.018	.899	.014	1.000	.009
	ρ DTR	.699	.023	.800	.018	.899	.014	1.000	.009
	ρ DCR	.699	.023	.800	.018	.899	.014	1.000	.009
	ρ SS	.589	.021	.674	.017	.756	.013	.842	.009
3	ρ CFA	.699	.043	.799	.038	.899	.028	1.001	.019
	ρ DPR	.699	.043	.799	.037	.900	.027	1.001	.019
	ρ DTR	.700	.043	.800	.037	.900	.027	1.002	.019
	ρ DCR	.699	.043	.799	.037	.899	.027	1.001	.019
	ρ SS	.588	.039	.672	.035	.757	.027	.842	.018
6	ρ CFA	.699	.039	.799	.030	.900	.019	1.000	.009
	ρ DPR	.699	.039	.799	.030	.900	.019	1.000	.009
	ρ DTR	.699	.039	.800	.030	.900	.019	1.000	.009
	ρ DCR	.699	.039	.799	.030	.900	.019	1.000	.009
	ρ SS	.639	.038	.731	.030	.823	.021	.914	.010
9	ρ CFA	.699	.038	.798	.028	.899	.016	1.000	.006
	ρ DPR	.699	.038	.798	.028	.899	.016	1.000	.006
	ρ DTR	.699	.038	.798	.028	.899	.016	1.001	.006
	ρ DCR	.699	.038	.798	.028	.898	.016	1.000	.006
	ρ SS	.657	.037	.751	.029	.845	.018	.941	.007
0.5, 0.4, 0.4	ρ CFA	.709	.156	.802	.161	.915	.157	1.012	.170
	ρ DPR	.727	.153	.819	.158	.934	.157	1.028	.168
	ρ DTR	.728	.153	.819	.158	.934	.157	1.029	.168
	ρ DCR	.711	.152	.802	.156	.917	.155	1.012	.169
	ρ SS	.288	.058	.325	.058	.369	.053	.407	.053
0.8, 0.7, 0.7	ρ CFA	.699	.051	.803	.042	.900	.035	1.001	.028
	ρ DPR	.701	.052	.805	.043	.903	.036	1.004	.029
	ρ DTR	.702	.052	.805	.043	.904	.036	1.004	.029
	ρ DCR	.699	.051	.802	.043	.900	.036	1.001	.028
	ρ SS	.544	.043	.624	.037	.699	.033	.778	.026
0.8, 0.8, 0.8	ρ CFA	.699	.043	.799	.038	.899	.028	1.001	.019
	ρ DPR	.699	.043	.799	.037	.900	.027	1.001	.019
	ρ DTR	.700	.043	.800	.037	.900	.027	1.002	.019
	ρ DCR	.699	.043	.799	.037	.899	.027	1.001	.019
	ρ SS	.588	.039	.672	.035	.757	.027	.842	.018
0.9, 0.6, 0.3	ρ CFA	.697	.062	.800	.052	.901	.047	1.000	.036
	ρ DPR	.767	.083	.879	.073	.989	.071	1.101	.065
	ρ DTR	.768	.082	.879	.073	.990	.071	1.102	.066
	ρ DCR	.698	.072	.799	.062	.901	.058	1.003	.049
	ρ SS	.453	.049	.519	.044	.586	.040	.651	.034
0.9, 0.8, 0.8	ρ CFA	.700	.043	.797	.032	.902	.023	1.000	.014
	ρ DPR	.701	.043	.798	.033	.904	.025	1.002	.015
	ρ DTR	.701	.043	.798	.033	.904	.024	1.002	.015
	ρ DCR	.700	.043	.796	.033	.902	.024	1.000	.014
	ρ SS	.610	.040	.695	.032	.787	.024	.872	.015

Note: M = mean; SD = standard deviation; Techniques: CFA = confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; SS = summed scale without disattenuation.

a. When the sample size was varied, the number of indicators was 3, and all loadings were .8. When the number of indicators was varied, the sample size was 250, and all loadings were .8. When the loadings were varied, the sample size was 250, and the number of indicators was 3.

OPEN IN VIEWER

Table 7.

Coverage and Balance of 95% Confidence Intervals by Loadings and Sample Size.

Loadings	Sample Size	Technique	Factor Correlation
			.7		.8		.9		1.0
			Coverage	Balance	Coverage	Balance	Coverage	Balance	Coverage	Balance
0.8, 0.8, 0.8	50	ρ CFA	.905	–.057	.899	–.085	.912	–.068	.955	–.023
		ρ DPR	.921	–.037	.922	–.048	.908	–.056	.927	–.045
		ρ DTR	.922	–.040	.915	–.059	.908	–.068	.909	–.067
		ρ DCR	.927	–.027	.919	–.049	.916	–.046	.933	–.033
	100	ρ CFA	.915	–.051	.925	–.055	.936	–.046	.955	–.025
		ρ DPR	.917	–.021	.938	–.022	.944	–.030	.940	–.030
		ρ DTR	.922	–.030	.932	–.028	.940	–.042	.938	–.038
		ρ DCR	.925	–.019	.932	–.016	.943	–.027	.941	–.021
	250	ρ CFA	.966	–.016	.938	–.022	.947	–.025	.949	–.015
		ρ DPR	.967	–.011	.947	–.011	.949	–.015	.946	–.024
		ρ DTR	.968	–.008	.949	–.013	.944	–.022	.942	–.026
		ρ DCR	.967	–.005	.948	–.008	.948	–.012	.945	–.017
	1,000	ρ CFA	.949	–.001	.955	–.017	.954	–.008	.950	–.002
		ρ DPR	.938	.004	.956	–.012	.947	–.007	.947	–.007
		ρ DTR	.947	.003	.956	–.010	.952	–.004	.953	–.009
		ρ DCR	.946	.006	.956	–.008	.951	–.003	.951	–.005
0.9, 0.6, 0.3	50	ρ CFA	.911	–.034	.919	–.030	.923	–.036	.954	.015
		ρ DPR	.851	–.143	.859	–.141	.822	–.176	.806	–.194
		ρ DTR	.852	–.142	.848	–.150	.810	–.188	.803	–.197
		ρ DCR	.964	–.017	.957	–.023	.942	–.035	.952	–.023
	100	ρ CFA	.942	–.016	.939	–.005	.949	–.019	.944	.002
		ρ DPR	.859	–.135	.843	–.157	.810	–.190	.756	–.244
		ρ DTR	.866	–.128	.842	–.158	.803	–.195	.747	–.253
		ρ DCR	.956	–.010	.943	–.007	.945	–.017	.940	–.020
	250	ρ CFA	.940	–.006	.943	–.017	.937	–.031	.964	–.002
		ρ DPR	.832	–.164	.786	–.214	.694	–.306	.581	–.417
		ρ DTR	.826	–.170	.781	–.219	.682	–.318	.568	–.430
		ρ DCR	.943	–.007	.949	–.005	.937	–.023	.942	–.012
	1,000	ρ CFA	.958	.006	.947	–.023	.950	–.008	.947	–.027
		ρ DPR	.621	–.379	.453	–.545	.279	–.721	.117	–.883
		ρ DTR	.613	–.387	.446	–.552	.264	–.736	.108	–.892
		ρ DCR	.955	–.005	.947	–.003	.951	–.017	.945	–.017

Note: Balance is the difference between frequencies for the population value above and below the confidence interval. The number of indicators was always 3. Techniques: CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability.

The first set of rows in Table 6 shows the effects of sample size. Scale score correlation ρ SS was always negatively biased due to the well-known attenuation effect. All disattenuation techniques and CFA performed better, and in large samples (250, 1,000), their performance was indistinguishable. This result was expected because all these approaches are consistent and their assumptions hold in this set of conditions. In the smallest sample size (50), CFA was slightly biased to be less efficient than the disattenuation-based techniques, but the differences were in the third digit and thus were inconsequential. The number of indicators, shown in the second set of rows in Table 6, affects the bias of the scale score correlation ρ SS because increasing the number of indicators increases reliability and, consequently, reduces the attenuation effect. The effect for other techniques was an increase in precision, which was expected because more indicators provide more information from which to estimate the correlation.

The third set of rows in Table 6 demonstrates the effects of varying the factor loadings. When the factor loadings were equal (all at .8), the performance of CFA and all disattenuation techniques was identical, which was expected, as explained above. Table 7 shows that in this condition, the confidence intervals of all techniques performed reasonably well. The balance statistics were all negative, indicating that when the population value was outside the CI, it was generally more frequently below the lower limit of the interval than above the upper limit. In other words, all CIs were slightly positively biased. This effect and the general undercoverage of the CIs were most pronounced in small samples. C I CFA performed slightly worse than disattenuated correlations when the sample size was very small and the correlation between the factors was weaker. However, outside the smallest sample sizes, the differences were negligible in the third decimals.

When the loadings varied, ρ DTR and ρ DPR became positively biased. The reason for this result is that these estimation techniques assume tau-equivalence or equal reliability of the indicators; when this assumption does not hold, the techniques have been shown to produce negatively biased reliability estimates, thus leading to overcorrection for attenuation and producing positive bias. The same results are mirrored in the second set of rows in Table 7; both C I DPR and C I DTR produced positively biased CIs with poor coverage and balance.

In summary, Table 6 and Table 7 show that the best-performing correlation estimate was ρ CFA , followed by ρ DCR and that the corresponding confidence intervals C I CFA and C I DCR outperform the others. When the condition of being tau-equivalent was violated (e.g., the loadings were .9, .6, and .3), ρ CFA and ρ DCR produced estimates that were more accurate than those produced by other methods, but ρ CFA was slightly more precise, having smaller standard deviations. Similarly, C I CFA and C I DCR were largely unaffected and retained their performance from the tau-equivalent condition.

Inference Against a Cutoff

Our main results concern inference against a cutoff and are relevant when a researcher wants to make a yes/no decision about discriminant validity. We start by assessing the performance of the techniques that can be thought of as tests of a perfect correlation or as rules of thumb. Because the differences between ρ DPR (i.e., HTMT) and ρ DTR were negligible, only the former is reported. Similarly, C I DTR is omitted due to nearly identical performance with C I DPR .

Table 8 clearly shows that some of the techniques have either unacceptably low power or a false positive rate that is too high to be considered useful. Techniques that directly compare the point estimates of correlations with a cutoff (i.e., ρ CFA ( 1 ) , ρ DPR ( 1 ) , and ρ CR ( 1 ) ) have very high false negative rates because an unbiased and normal correlation estimate can be expected to be below the population value (here, 1) exactly half the time. In contrast, the AVE/SV technique that uses factor correlation following Fornell and Larcker’s (1981a) original proposal has a very high false positive rate. As expected based on our analysis, the misused version using scale score correlation, AVE / SV SS has a smaller false positive rate because the attenuation bias in the scale score correlation worked to offset the high false positive rate of the AVE comparison. Clearly, none of these techniques can be recommended.

The various model comparisons and CIs performed better. Among the three methods of model comparison (CFI(1), χ 2 (1), and χ 2 ( merge ) ), χ 2 (1) was generally the best in terms of both the false positive rate and false negative rate. While the difference was small, it is surprising that χ 2 (1) was strictly superior to χ 2 ( merge ) , having both more power and a smaller false positive rate. Our explanation for this finding is that although χ 2 ( merge ) imposes more constraints on the model, these constraints work differently when the factors are perfectly correlated and when they are not. When the factors are perfectly correlated, imposing more constraints means that the model can be declared to misfit in more ways, thus leading to lower power. In contrast, when the correlation between the factors is less than 1, the additional constraints are somewhat redundant because constraining the focal correlation to 1 will also bias all other correlations involving the focal variables. Thus, the amount of misfit produced by the first constraint is greater than the other constraints that χ 2 ( merge ) contributes. This phenomenon leads to a higher false positive rate because while the additional constraints contribute degrees of freedom, they contribute less misfit.

Another interesting finding is that although CFI ( 1 ) was proposed as an alternative to χ 2 ( 1 ) based on the assumption that it had a smaller false positive rate, this assumption does not appear to be true: the false positive rates of these techniques were comparable, and in larger samples (250, 1,000), the false positive rate of CFI ( 1 ) even exceeded that of χ 2 ( 1 ) . Generalizing this finding to larger models requires caution because the CFI comparison depends on the fit of the null model, which depends on model size. Nevertheless, it is clear that the CFI comparison does not generally have a smaller false positive rate than χ 2 ( 1 ) .

The performance of the CIs ( CI CFA ( 1 ) , CI DPR ( 1 ) , and CI DCR ( 1 ) ) was nearly identical in the tau-equivalent condition (i.e., all loadings at .8), but in the congeneric condition (i.e., the loadings at .3, .6, and .9), CI DPR ( 1 ) had an excessive false positive rate due to the positive bias explained earlier. CI DCR ( 1 ) had a larger false positive rate than CI CFA ( 1 ) , particularly in small samples, possibly due to violating the large sample assumption of bootstrapping. In summary, Table 8 supports the use of CI CFA ( 1 ) and χ 2 ( 1 ) . The former had slightly more power but a larger false positive rate than the latter. The performance of these two techniques converged in large samples.

OPEN IN VIEWER

Table 8.

Detection Rates of the Discriminant Validity Problem as a Perfect Correlation by Technique.

Sample Size	Technique	Loadings and Factor Correlation
		0.8, 0.8, 0.8				0.9, 0.6, 0.3
		.7*	.8*	.9*	1.0†	.7*	.8*	.9*	1.0†
50	ρCFA(1)	.000	.001	.016	.513	.007	.016	.063	.503
	ρDPR(1)	.000	.001	.016	.530	.051	.110	.258	.770
	ρDCR(1)	.000	.000	.016	.511	.014	.031	.107	.529
	CICFA(1)	.013	.075	.293	.992	.108	.194	.388	.984
	CIDPR(1)	.016	.072	.273	.986	.410	.616	.898	1.000
	CIDCR(1)	.014	.066	.256	.983	.244	.397	.697	.988
	AVE / SV CFA	.161	.582	.968	1.000	.733	.941	.992	1.000
	AVE / SV SS	.014	.141	.692	.975	.178	.457	.729	.871
	CFI(1)	.008	.037	.160	.935	.085	.140	.249	.916
	χ 2 (1)	.006	.043	.199	.967	.110	.185	.332	.968
	χ 2 ( merge )	.009	.056	.243	.967	.130	.210	.358	.968
100	ρCFA(1)	.000	.000	.002	.520	.000	.001	.028	.510
	ρDPR(1)	.000	.000	.002	.528	.016	.059	.204	.827
	ρDCR(1)	.000	.000	.002	.517	.001	.006	.051	.525
	CICFA(1)	.000	.004	.106	.990	.022	.088	.241	.985
	CIDPR(1)	.000	.003	.105	.984	.241	.386	.773	.999
	CIDCR(1)	.000	.003	.093	.980	.076	.182	.483	.984
	AVE / SV CFA	.063	.603	.993	1.000	.822	.980	.999	1.000
	AVE / SV SS	.000	.071	.690	.990	.097	.438	.728	.878
	CFI(1)	.000	.006	.096	.973	.028	.083	.211	.955
	χ 2 (1)	.000	.002	.081	.973	.024	.086	.234	.972
	χ 2 ( merge )	.000	.005	.105	.970	.038	.114	.263	.977
250	ρCFA(1)	.000	.000	.000	.519	.000	.000	.006	.505
	ρ DPR ( 1 )	.000	.000	.000	.527	.001	.016	.148	.920
	ρDCR(1)	.000	.000	.000	.515	.000	.001	.017	.509
	CICFA(1)	.000	.000	.005	.983	.000	.005	.124	.984
	CIDPR(1)	.000	.000	.005	.984	.074	.225	.516	.999
	CIDCR(1)	.000	.000	.005	.982	.002	.030	.204	.980
	AVE / SV CFA	.009	.584	1.000	1.000	.911	.999	1.000	1.000
	AVE / SV SS	.000	.010	.671	.999	.028	.426	.697	.886
	CFI(1)	.000	.000	.019	.994	.001	.020	.133	.987
	χ 2 (1)	.000	.000	.004	.974	.000	.006	.123	.976
	χ 2 ( merge )	.000	.000	.008	.972	.000	.014	.152	.976
1,000	ρCFA(1)	.000	.000	.000	.491	.000	.000	.000	.510
	ρDPR(1)	.000	.000	.000	.495	.000	.001	.106	.997
	ρDCR(1)	.000	.000	.000	.488	.000	.000	.000	.518
	CI CFA ( 1 )	.000	.000	.000	.983	.000	.000	.001	.985
	CIDPR(1)	.000	.000	.000	.981	.000	.028	.320	1.000
	CIDCR(1)	.000	.000	.000	.980	.000	.000	.015	.979
	AVE / SV CFA	.000	.583	1.000	1.000	.988	1.000	1.000	1.000
	AVE / SV SS	.000	.000	.666	1.000	.001	.396	.667	.900
	CFI(1)	.000	.000	.000	1.000	.000	.000	.017	1.000
	χ 2 (1)	.000	.000	.000	.979	.000	.000	.001	.981
	χ 2 ( merge )	.000	.000	.000	.978	.000	.000	.003	.979

Note: Averages over all indicator numbers. Techniques: ρ = comparing the correlations against a cutoff; CI = comparing if a cutoff is included in the 95% confidence interval; AVE / SV = comparing the AVE statistics against the squared correlation; CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; CFI = nested model comparison using the CFI rule; χ 2 = nested model test; (1) = two factors are constrained to be perfectly correlated; (merge) = two factors are merged into one.

* False positive rate. ^†true positive rate = (1 – false negative rate).

Table 9 considers cutoffs other than 1, using values of .85, .90, and .95 that are sometimes recommended in the literature, showing results that are consistent with those of the previous tables. The techniques that compared the CI against a cutoff (i.e., CI CFA ( cut ) , CI DPR ( cut ) and CI DCR ( 1 ) ) had more power than those that compared an estimate against a cutoff (i.e., ρ CFA ( cut ) , ρ DPR ( cut ) , and ρ DCR ( cut ) ), especially in small samples, and between the three, CI CFA ( cut ) showed a lower false positive rate while having power similar to that of CI DPR ( cut ) and to a smaller extent CI DCR ( cut ) . Among the techniques that compared model fit (i.e., CFI ( cut ) and χ 2 ( cut ) ), CFI ( cut ) had slightly more power but also a higher false positive rate in small samples than χ 2 ( cut ) . In larger samples, the power of the two techniques was similar, but χ 2 ( cut ) generally had the lowest false positive rate. This finding and the sensitivity of the CFI tests to model size, explained earlier, make χ 2 ( cut ) the preferred alternative of the two. In summary, CI CFA ( cut ) and χ 2 ( cut ) are generally the best techniques. They have different strengths: CI CFA ( cut ) has slightly more power, but χ 2 ( cut ) enjoys a considerably lower false positive rate.

OPEN IN VIEWER

Table 9.

Detection Rates by Technique Using Alternative Cutoffs.

Sample Size	Technique	Cutoff and Factor Correlation
		.85				.9				.95
		.7*	.8*	.9†	1.0†	.7*	.8*	.9†	1.0†	.7*	.8*	.9*	1.0†
50	ρ CFA ( cut )	.085	.289	.784	.950	.049	.126	.517	.925	.033	.064	.193	.866
	ρ DPR ( cut )	.121	.348	.818	.968	.075	.182	.576	.946	.051	.101	.270	.892
	ρ DCR ( cut )	.095	.301	.785	.958	.056	.140	.524	.930	.036	.073	.212	.857
	CI CFA ( cut )	.694	.945	.992	.996	.413	.788	.982	.995	.216	.439	.895	.992
	CI DPR ( cut )	.695	.940	.999	1.000	.477	.777	.990	1.000	.331	.522	.884	.999
	CI DCR ( cut )	.669	.929	.999	1.000	.441	.746	.988	1.000	.305	.476	.858	.999
	CFI ( cut )	.416	.781	.971	.992	.227	.509	.915	.987	.146	.259	.649	.977
	χ 2 ( cut )	.541	.884	.992	.998	.301	.629	.967	.997	.192	.333	.760	.995
100	ρ CFA ( cut )	.041	.194	.850	.973	.023	.068	.510	.953	.015	.034	.126	.907
	ρ DPR ( cut )	.066	.264	.878	.980	.037	.110	.582	.963	.022	.061	.206	.925
	ρ DCR ( cut )	.044	.211	.846	.975	.022	.072	.518	.952	.014	.036	.141	.897
	CI CFA ( cut )	.396	.891	.996	.999	.189	.525	.987	.998	.115	.230	.734	.996
	CI DPR ( cut )	.455	.888	.999	1.000	.279	.577	.989	1.000	.198	.337	.760	1.000
	CI DCR ( cut )	.410	.869	.999	1.000	.236	.528	.986	1.000	.166	.280	.721	.999
	CFI ( cut )	.317	.793	.987	.998	.156	.417	.952	.996	.096	.188	.617	.990
	χ 2 ( cut )	.314	.818	.997	1.000	.169	.419	.974	.999	.114	.209	.614	.997
250	ρ CFA ( cut )	.016	.102	.927	.989	.008	.029	.513	.977	.004	.014	.071	.945
	ρ DPR ( cut )	.027	.169	.942	.991	.013	.060	.589	.980	.006	.025	.143	.956
	ρ DCR ( cut )	.015	.110	.919	.989	.008	.030	.508	.977	.004	.014	.077	.939
	CI CFA ( cut )	.145	.704	.998	1.000	.079	.233	.985	1.000	.049	.114	.426	.998
	CI DPR ( cut )	.224	.743	.999	1.000	.139	.336	.987	1.000	.095	.192	.540	.999
	CI DCR ( cut )	.168	.698	.999	1.000	.097	.262	.981	1.000	.064	.137	.468	.999
	CFI ( cut )	.208	.819	.996	.999	.093	.321	.981	.998	.063	.117	.571	.996
	χ 2 ( cut )	.134	.635	.998	1.000	.078	.210	.975	1.000	.053	.111	.381	.999
1,000	ρ CFA ( cut )	.002	.030	.976	.999	.000	.006	.505	.995	.000	.002	.025	.980
	ρ DPR ( cut )	.003	.095	.980	.999	.000	.025	.622	.997	.000	.003	.093	.983
	ρ DCR ( cut )	.002	.033	.975	.999	.000	.006	.507	.995	.000	.002	.026	.979
	CI CFA ( cut )	.032	.253	1.000	1.000	.016	.071	.980	1.000	.006	.033	.174	1.000
	CI DPR ( cut )	.069	.378	1.000	1.000	.028	.146	.984	1.000	.011	.074	.309	1.000
	CI DCR ( cut )	.036	.279	1.000	1.000	.020	.081	.979	1.000	.008	.038	.198	1.000
	CFI ( cut )	.063	.846	.999	1.000	.027	.132	.995	1.000	.013	.060	.400	1.000
	χ 2 ( cut )	.033	.238	1.000	1.000	.018	.071	.975	1.000	.007	.035	.170	1.000

Note: Averages over all indicator numbers and loading. Techniques: ρ = comparing correlation against a cutoff; CI = comparing if a cutoff is included in the 95% confidence interval, CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; CFI = nested model comparison using the CFI rule; χ 2 = nested model test.

* False positive rate. ^†true positive rate = (1 – false negative rate).

Effects of Model Misspecification

We now turn to the cross-loading conditions to assess the robustness of the techniques when the assumption of no cross-loadings is violated. The results shown in Table 10 show that all estimates become biased toward 1. ρ DCR was slightly most robust to these misspecifications, but the differences between the techniques were not large. Table 11 presented the detection rates of different techniques using alterative cutoffs and over the cross-loading conditions and showed similar results. All techniques were again affected, and both the power and false positive rates increased across the board when the correlation between the factors was less than one. As before, the effect was stronger for smaller population correlations.

OPEN IN VIEWER

Table 10.

Mean Correlation Estimate Under Model Misspecification.

Cross-Loading Estimation	Technique	Population Cross-Loadings and Factor Correlation
		No Cross-Loadings				1 Cross-Loading				2 Cross-Loadings
		.7	.8	.9	1.0	.7	.8	.9	1.0	.7	.8	.9	1.0
Assumed zero	ρ CFA	.699	.800	.901	1.001	.819	.878	.938	1.002	.885	.921	.957	1.000
	ρ DPR	.707	.808	.911	1.012	.808	.876	.943	1.011	.879	.922	.962	1.008
	ρ DTR	.708	.809	.911	1.013	.808	.876	.943	1.012	.879	.922	.963	1.008
	ρ DCR	.700	.800	.901	1.002	.801	.868	.935	1.002	.871	.914	.955	1.000
	ρ SS	.560	.640	.720	.801	.633	.692	.749	.801	.700	.740	.777	.805
Estimated	ρ CFA					.700	.801	.903	1.009	.694	.790	.894	1.021

Note: Averages the correlations over the conditions shown in Table 6. Techniques: CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; SS = summed scale without disattenuation.

OPEN IN VIEWER

Table 11.

Detection Rates by Technique Using Alternative Cutoffs Under Model Misspecification.

Cross-Loadings and Whether Estimated	Technique	Cutoff and Factor Correlation
		.85				.9				.95
		.7*	.8*	.9†	1.0†	.7*	.8*	.9†	1.0†	.7*	.8*	.9*	1.0†
No cross-loadingsAssumed zero	ρ CFA ( cut )	.034	.152	.886	.980	.018	.055	.511	.965	.011	.026	.101	.927
	ρ DPR ( cut )	.052	.216	.906	.986	.029	.091	.592	.973	.018	.045	.175	.941
	ρ DCR ( cut )	.037	.161	.883	.982	.019	.060	.514	.966	.012	.029	.111	.920
	CI CFA ( cut )	.313	.697	.997	.999	.169	.400	.984	.998	.091	.199	.555	.997
	CI DPR ( cut )	.356	.735	.999	1.000	.225	.455	.988	1.000	.152	.276	.620	1.000
	CI DCR ( cut )	.315	.691	.999	1.000	.192	.400	.983	1.000	.129	.227	.558	.999
	CFI ( cut )	.248	.810	.989	.998	.121	.341	.961	.996	.075	.151	.557	.991
	χ 2 ( cut )	.251	.641	.997	.999	.137	.328	.973	.999	.086	.167	.478	.998
1 cross-loadingAssumed zero	ρ CFA ( cut )	.445	.837	.958	.980	.168	.442	.895	.965	.038	.098	.450	.925
	ρ DPR ( cut )	.253	.757	.962	.986	.109	.297	.883	.974	.051	.115	.368	.938
	ρ DCR ( cut )	.194	.739	.954	.982	.067	.229	.863	.965	.029	.067	.290	.919
	CI CFA ( cut )	.900	.996	.999	.999	.621	.917	.997	.999	.311	.541	.940	.997
	CI DPR ( cut )	.732	.996	1.000	1.000	.469	.787	.999	1.000	.288	.462	.867	.999
	CI DCR ( cut )	.727	.996	1.000	1.000	.425	.781	.999	1.000	.237	.408	.859	.999
	CFI ( cut )	.912	.985	.995	.998	.596	.930	.989	.996	.214	.482	.946	.991
	χ 2 ( cut )	.880	.995	.999	1.000	.563	.895	.997	.999	.254	.469	.918	.998
1 cross-loadingEstimated	ρ CFA ( cut )	.058	.191	.848	.962	.037	.088	.518	.938	.025	.051	.160	.883
	CI CFA ( cut )	.385	.750	.994	.998	.245	.488	.978	.996	.155	.287	.641	.994
	CFI ( cut )	.367	.871	.986	.996	.216	.498	.964	.993	.147	.267	.745	.987
	χ 2 ( cut )	.342	.711	.996	.999	.221	.433	.972	.998	.157	.264	.591	.995
2 cross-loadingsAssumed zero	ρ CFA ( cut )	.855	.943	.970	.977	.411	.819	.936	.961	.067	.203	.740	.918
	ρ DPR ( cut )	.779	.940	.975	.983	.307	.761	.941	.969	.110	.230	.709	.932
	ρ DCR ( cut )	.760	.930	.970	.979	.235	.733	.929	.961	.057	.143	.663	.914
	CI CFA ( cut )	.996	.998	.999	.999	.938	.996	.998	.998	.492	.773	.992	.996
	CI DPR ( cut )	.997	1.000	1.000	1.000	.803	.996	.999	1.000	.459	.693	.995	.999
	CI DCR ( cut )	.996	1.000	1.000	1.000	.793	.995	1.000	1.000	.397	.660	.993	.999
	CFI ( cut )	.986	.994	.996	.997	.941	.984	.992	.995	.434	.858	.976	.990
	χ 2 ( cut )	.996	.998	.999	.999	.916	.994	.998	.998	.415	.720	.989	.996
2 cross-loadingsEstimated	ρ CFA ( cut )	.105	.257	.791	.922	.071	.150	.530	.895	.045	.091	.239	.837
	CI CFA ( cut )	.510	.814	.987	.993	.384	.614	.973	.990	.291	.444	.752	.987
	CFI ( cut )	.565	.921	.983	.992	.408	.704	.968	.988	.303	.466	.885	.980
	χ 2 ( cut )	.482	.790	.992	.997	.361	.573	.971	.995	.276	.408	.709	.990

Note: Averages over all conditions. Techniques: ρ = comparing the correlations against a cutoff; CI = comparing if a cutoff is included in the 95% confidence interval; CFA= confirmatory factor analysis; Dxx = disattenuated correction using PR = parallel reliability, TR = tau-equivalent reliability, and CR = congeneric reliability; CFI = nested model comparison using the CFI rule; χ 2 = nested model test.

* False positive rate. ^†true positive rate = (1 – false negative rate).

In the cross-loading conditions, we also estimated a correctly specified CFA model in which the cross-loadings were estimated. Table 10 shows that the mean estimate was largely unaffected, but the variance of the estimates (not reported in the table) increased because of the increased model complexity. This effect is seen in Table 11, where the pattern of results for CFA models was largely similar between the cross-loading conditions, but the presence of cross-loadings increased the false positive rate. This result is easiest to understand in the context of CI CFA ( cut ) ; when estimates became less precise, this also widened the confidence intervals and, consequently, increased the frequency of results where the cutoff fell within the interval.

The cross-loading results underline the importance of observing the assumptions of the techniques and that not doing so may lead to incorrect inference. However, two conclusions that are new to discriminant validity literature can be drawn: First, the lack of cross-loadings in the population (i.e., factorial validity) is not a strict prerequisite for discriminant validity assessment as long as the cross-loadings are modeled appropriately. Second, while the lack of factorial validity can lead scale-item pairs to have complete lack of discriminant validity (see Equation 2), this does not always invalidate scale-level discriminant validity (see Equation 3) as long as this is properly modeled.

Contradictions With Prior Studies

Our simulation results clearly contradict two important conclusions drawn in the recent discriminant validity literature, and these contradictions warrant explanations. First, Henseler et al. (2015) and Voorhees et al. (2016) strongly recommend ρ DPR (HTMT) for discriminant validity assessment. We prove that the HTMT index is simply a scale score correlation disattenuated with parallel reliability (i.e., the standardized alpha) and thus should not be expected to outperform modern CFA techniques, which our simulation demonstrates. The different conclusions are due to the limitations of these prior studies. While Henseler et al. (2015) explain that ρ DPR is a factor correlation estimate, they do not compare it against other factor correlation estimation techniques. Thus, their results do not indicate the superiority of ρ DPR but simply indicate that AVE/SV, which was their main comparison, performs very poorly. The follow-up study by Voorhees et al. (2016) considered a broader set of techniques, including CI CFA ( 1 ) and χ 2 ( 1 ) . The problem in their study was that the different techniques were applied using different cutoffs: ρ DPR was used with cutoffs of .80, .85, and .90, whereas the other techniques always used the cutoff of 1 and were thus predestined to fail in a study where a correlation of .90 was used as a discriminant validity problem condition. A more complete study would have used the same cutoffs (.80, .85, and .90) that were applied to ρ DPR with χ 2 and CI CFA as well, as we did.

Second, our results also challenge J. A. Shaffer et al.’s (2016) recommendation that discriminant validity should be tested by a CFI comparison between two nested models (CFI(1)). The reason for this contradiction is that their recommendation is based on the assumption that if the CFI difference works well for measurement invariance assessment, it should also work well when discriminant validity assessment. The assumption appears to be invalid as it ignores an important difference between these uses: Whereas the degrees of freedom of an invariance test scale roughly linearly with the number of indicators, the degrees of freedom in CFI(1) are always one.

Magnitude of the Discriminant Validity Correlations

In the discriminant validity literature, high correlations between scales or scale items are considered problematic. However, the literature generally has not addressed what is high enough beyond giving rule of thumb cutoffs (e.g., 85). Our definition of discriminant validity suggests that the magnitude of the estimated correlation depends on the correlation between the constructs, the measurement process, and the particular sample, each of which has different implications on what level should be considered high. To warn against mechanical use, we present a scenario where high correlation does not invalidate measurement and a scenario where low correlation between measures does not mean that they measure distinct constructs.

A large correlation does not always mean a discriminant validity problem if one is expected based on theory or prior empirical observations. For example, the correlation between biological sex and gender identity can exceed .99 in the population. ¹⁷ However, both variables are clearly distinct: sex is a biological property with clear observable markers, whereas gender identity is a psychological construct. These two variables also have different causes and consequences (American Psychological Association, 2015), so studies that attempt to measure both can lead to useful policy implications. In cases such as this where the constructs are well defined, large correlations should be tolerated when expected based on theory and prior empirical results. Of course, large samples and precise measurement would be required to ensure that the constructs can be distinguished empirically (i.e., are empirically distinct).

A small or moderate correlation (after correcting for measurement error) does not always mean that two measures measure concepts that are distinct. For example, consider two thermometers that measure the same temperature, yet one is limited to measuring only temperatures above freezing, whereas the other can measure only temperatures below freezing. While both measure the same quantity, they are correlated only by approximately .45 because the temperature would always be out of the range of one of the thermometers that would consequently display zero centigrade. ¹⁸ In the social sciences, a well-known example is the measurement of happiness and sadness, two constructs that can be thought of as opposite poles of mood (D. P. Green et al., 1993; Tay & Jebb, 2018). Consequently, any evaluation of the discriminant validity of scales measuring two related constructs must precede the theoretical consideration of the existence of a common continuum. If this is the case, the typical discriminant validity assessment techniques that are the focus of our article are not directly applicable, but other techniques are needed (Tay & Jebb, 2018).

As the two examples show, a moderately small correlation between measures does not always imply that two constructs are distinct, and a high correlation does not imply that they are not. Like any validity assessment, discriminant validity assessment requires consideration of context, possibly relevant theory, and empirical results and cannot be reduced to a simple statistical test and a cutoff no matter how sophisticated. These considerations highlight the usefulness of the continuous interpretation of discriminant validity evidence.

On Choosing a Technique and a Cutoff

While a general set of statistics and cutoffs that is applicable to all research scenarios cannot exist, we believe that establishing some standards is useful. Based on our study, CI CFA ( cut ) and χ 2 ( cut ) appear to be the leading techniques, but recommending one over another solely on a statistical basis is difficult due to the similar performance of the techniques. However, we recommend CI CFA ( cut ) for practical reasons. First, CI CFA ( cut ) is less likely to be misused than χ 2 ( cut ) . While we found some evidence of misapplication of χ 2 ( cut ) due to incorrect factor scaling, we did not see any evidence of the same when factor correlations were evaluated; these can be obtained postestimation simply by requesting standardized estimates from the software. Second, CI CFA ( cut ) makes it easier to transition from testing of discriminant validity to its evaluation because the focal statistic is a correlation, which organizational researchers routinely interpret in other contexts. In sum, CI CFA ( cut ) is simpler to implement, easier to understand, and less likely to be misapplied. χ 2 ( cut ) is slightly more accurate, but considering that even the simpler χ 2 ( 1 ) is often misapplied, we do not think that the potential precision gained by using χ 2 ( cut ) is worth the cost of risking misapplication.

Equally important to choosing a technique is the choice of a cutoff if the technique requires one. Of the recent simulation studies, Henseler et al. (2015) suggested cutoffs of .85 and .9 based on prior literature (e.g., Kline, 2011). Voorhees et al. (2016) considered the false positive and false negative rates of the techniques used in their study and concluded that the cutoff of .85 had the best balance of high power and an acceptable Type I error rate. However, as explained in Online Supplement 5, such conclusions are to a large part simply artifacts of the simulation design. In sum, it seems that deriving an ideal cutoff through simulation results is meaningless and must be established by consensus among the field.

Empirical studies seem to agree that correlations greater than .9 indicate a problem and that correlations less than .8 indicate the lack of a problem. For example, Le et al. (2010) diagnosed a discriminant validity problem between job satisfaction and organizational commitment based on a correlation of .91, and Mathieu and Farr (1991) declared no problem of discriminant validity between the same variables on the basis of a correlation of .78. However, mixed judgments were made about the correlation values between .8 and .9. Lucas et al. (1996) acknowledged discriminant validity between self-esteem and optimism based on ρ D T R of .83 ( ρ S S = .72 ), but Credé et al. (2017) criticized the conceptual redundancy between grit and conscientiousness based on a disattenuated correlation of .84 ( ρ S S = .66 ). Thus, if there is a threshold, it is likely to fall between .8 and .9.

Sources of Error Other Than the Random Error and Item-Specific Factor Error

Our focus on the common scenario of single-method and one-time measurements limits the researcher to techniques that operationalize measurement error as item uniqueness either directly (e.g., CFA) or indirectly (e.g., ρ DTR ). However, several articles point out that this may be a simplistic view of measurement error that only considers random and item-specific factor errors, ignoring time-specific transient errors (Le et al., 2009; Woehr et al., 2012). Because transient error can correlate between items and scales, they can either inflate or attenuate correlation estimates calculated using single-administration reliability estimates. While there is little that can be done about this issue if one-time measures are used, researchers should be aware of this limitation. Of course, if multiple measurement occasions are possible, the CFA-based techniques can also be used to model these other sources of error (Le et al., 2009; Woehr et al., 2012), and these more complex models can then be applied with the guideline that we present next.

From a Cutoff to a Classification System

While many articles about discriminant validity consider it as a matter of degree (e.g., “the extent to…”) instead of a yes/no issue (e.g., “whether…”), most guidelines on evaluation techniques, including Campbell and Fiske’s (1959) original proposal, focus on making a dichotomous judgment as to whether a study has a discriminant validity problem (B in Figure 6). This inconsistency might be an outcome of researchers favoring cutoffs for their simplicity, or it may reflect the fact that after calculating a discriminant validity statistic, researchers must decide whether further analysis and interpretation is required. This practice also fits the dictionary meaning of evaluation, which is not simply calculating and reporting a number but rather dividing the object into several qualitative categories based on the number, thus requiring the use of cutoffs, although these do not need to be the same in every study.

To move the field toward discriminant validity evaluation, we propose a system consisting of several cutoffs instead of a single one. Discriminant validity is a continuous function of correlation values (C in Figure 6), but because of practical needs, correlations are classified into discrete categories indicating different degrees of a problem (D in Figure 6). Similar classifications are used in other fields to characterize essentially continuous phenomena: Consider a doctor’s diagnosis of hypertension. In the past, everyone was divided into two categories of normal and patient, but now hypertension is classified into several levels. That is, patients with hypertension are further subdivided into three stages according to their blood pressure level, and each level is associated with different treatments.

OPEN IN VIEWER

Figure 6.

Relationship between correlation values and the problem of discriminant validity. (A) Linear model (implied by existing definitions), (B) Dichotomous model (existing techniques), (C) Threshold model (implied by the definition of this study), (D) Step model (proposed evaluation technique).

Table 12 shows the classification system we propose. We emphasize that these are guideline that can be adjusted case-by-case if warranted by theoretical understanding of the two constructs and measures, not strict rules that should always be followed. Based on our review, correlations below .8 were seldom considered problematic, and this is thus used as the cutoff for the first class, “No problem,” which strictly speaking is not a proof of no problem, just no evidence of a problem. When correlations fall into this class, researchers can simply declare that they did not find any evidence of a discriminant validity problem. The next three steps are referred to as Marginal, Moderate, and Severe problems, respectively. The Severe problem is the most straightforward: two items or scales cannot be distinguished empirically, and researchers should rethink their concept definitions, measurement, or both. In empirical applications, the correlation level of .9 was nearly universally interpreted as a problem, and we therefore use this level as a cutoff between the Marginal and Moderate cases. In both cases, the high correlation should be acknowledged, and its possible cause should be discussed. In the Marginal case, the interpretation of the scales as representations of distinct constructs is probably safe. In the Moderate case, additional evidence from prior studies using the same constructs and/or measures should be checked before interpretation of the results to ensure that the high correlation is not a systematic problem with the constructs or scales.

OPEN IN VIEWER

Table 12.

Proposed Classification and Cutoffs.

Classification	CICFA (sys)	χ 2 (sys)
Severe problem	1 ≤ UL	χ 1 2 − χ o r g 2 < 3.84
Moderate problem	.9 ≤ UL < 1	Not “Marginal problem” AND χ 1 2 − χ o r g 2 > 3.84
Marginal problem	.8 ≤ UL < .9	Not “No problem” AND χ .9 2 − χ o r g 2 > 3.84
No problem	UL < .8	ρ CFA < .8 AND χ .8 2 − χ o r g 2 > 3.84

Note: ρ CFA is the correlation obtained using CFA, UL is the 95% upper limit of ρ CFA when ρ CFA > 0 , and the absolute value of the 95% lower limit of ρ CFA when ρ CFA < 0 , χ o r g 2 is the chi-square value of the original model, and χ c 2 is the chi-square value of the comparison model where the focal correlation is fixed to c when ρ CFA > 0 and − c when ρ CFA < 0 .

How to Implement the Proposed Techniques

The proposed classification system should be applied with CI_CFA(cut) and χ 2 ( cut ) , and we propose that these workflows be referred to as CI_CFA(sys) and χ 2 ( sys ) , respectively. Both workflows start by estimating a CFA model that includes all scales that are evaluated for discriminant validity. Instead of using the default scale setting option to fix the first factor loadings to 1, scale the latent variables by fixing their variances to 1 (A in Figure 2); this should be explicitly reported in the article. The covariances between factors obtained in the latter way equal the correlations; alternatively, when using CI_CFA(sys), the standardized factor solution can be inspected. Next, inspect the upper limits (lower limits for negative correlations) of the 95% CIs of the estimated factor correlations and compare their values against the cutoffs in Table 12. ¹⁹