A Reformulated Correlated Trait–Correlated Method Model for Multitrait–Multimethod Data Effectively Increases Convergence and Admissibility Rates

Abstract

The correlated trait–correlated method (CTCM) model for the analysis of multitrait–multimethod (MTMM) data is known to suffer convergence and admissibility (C&A) problems. We describe a little known and seldom applied reparameterized version of this model (CTCM-R) based on Rindskopf’s reparameterization of the simpler confirmatory factor analysis model. In a Monte Carlo study, we compare the CTCM, CTCM-R, and the correlated trait–correlated uniqueness (CTCU) models in terms of C&A, model fit, and parameter estimation bias. The CTCM-R model largely avoided C&A problems associated with the more traditional CTCM model, producing C&A solutions nearly as often as the CTCU model, but also avoiding parameter estimation biases known to plague the CTCU model. As such, the CTCM-R model is an attractive alternative for the analysis of MTMM data.

Keywords

MTMM multitrait–multimethod confirmatory factor analysis improper solutions convergence

The choice of a particular confirmatory factor analytic (CFA) model for the analysis of multitrait–multimethod (MTMM) data remains controversial. Two of the most popularly adopted models, namely, the correlated trait–correlated method model (CTCM) and the correlated trait–correlated uniqueness model (CTCU), have been shown to suffer from empirical estimation (Conway, Lievens, Scullen, & Lance, 2004; Marsh & Bailey, 1991) and theoretical problems (Lance, Noble, & Scullen, 2002), respectively. More than 30 years ago, Rindskopf (1983) presented a reparameterization of the CFA model that effectively implements nonnegative restrictions on uniqueness estimates and thus avoids Heywood cases, but this model has received very little attention in the literature.

In the current study, we (a) show how Rindskopf’s reparameterized CFA model is extended to MTMM data and (b) compare the CTCM-R model to the other two frequently used CFA models (i.e., the CTCM and CTCU models) in a comprehensive Monte Carlo simulation study. The three models were empirically evaluated based on model convergence and admissibility (C&A), model goodness of fit, and accuracy of the parameter estimates. In the following sections, after a brief review of relevant literature on the strength and limitations of the CTCM and CTCU models, we describe the CTCM-R model in more detail, and then present descriptions of the simulation study design, our results, and a brief discussion of the importance of our findings.

Background

One of the main sources of controversy in the MTMM literature has concerned the optimal analytic approach for MTMM data, and consensus has evolved over time that the most appropriate analytic approach is some form of structural equation modeling (SEM; Eid, Lischetzke, & Nussbeck, 2006; Kenny & Kashy, 1992; Schmitt & Stults, 1986). Two of the most frequently adopted SEMs are the CTCM (or one or more of its special cases, see Widaman, 1985) and the CTCU models (Kenny, 1976; Marsh, 1989).

The CTCM model can be written as

MTMM = [\begin{matrix} Λ_{T} & Λ_{M} \end{matrix}] [\begin{matrix} Φ_{TT'} & sym \\ 0 & Φ_{MM'} \end{matrix}] [\begin{matrix} Λ_{T}' \\ Λ_{M}' \end{matrix}] + Θ

where MTMM is the p×p MTMM correlation matrix where, typically (but not necessarily), p = T*M (where T refers to the number of Traits and M refers to the number of Methods). Λ_T (p×T) and Λ_M (p×M) contain a priori specified fixed and freely estimated factor loadings connecting the observed variables to their respective Trait and Method factors, Φ_TT′ (T×T) and Φ_MM′ (M×M) are both symmetric matrices that contain freely estimated correlations among the Trait and Method factors, respectively, and Θ is a diagonal matrix containing estimated uniquenesses (i.e., residuals from the variables’ regressions on Trait and Method factors). The CTCM model is faithful to Campbell and Fiske’s (1959) original statement of the MTMM methodology and permits the estimation of the full range of parameters contained within the Trait and Method factor space. However, it has long been known to suffer C&A problems associated with empirical underidentification issues (Brannick & Spector, 1990) associated with implied equality constraints in the model when population factor loadings are (nearly) equal (Kenny & Kashy, 1992).

Marsh (1989) and Marsh and Bailey (1991) elaborated on the CTCU model developed by Kenny (1976) as an alternative to the CTCM model that largely avoids empirical estimation problems associated with it. The CTCU model can be written as

MTMM = Λ_{T} Φ_{TT'} Λ_{T}' + Θ

where again Λ_T (p×T) and Φ_TT′ (T×T) contain the Trait factor loadings and correlations, respectively. The CTCU model parameterizes Method effects not as Method factors’ effects on variables (as in the CTCM model) but as unique variances and covariances. Thus, $u_{ij}^{2} = e_{ij}^{2} + s_{ij}^{2} + σ_{Mj}^{2}$ : uniquenesses ( $u_{ij}^{2}$ ) contain nonsystematic variance ( $e_{ij}^{2}$ ) specific variance ( $s_{ij}^{2}$ ) and Method variance ( $σ_{Mj}^{2}$ ), respectively. Also, $E (δ_{ij}, δ_{i' j'}) = σ_{δ_{ij} δ_{i' j'}}$ for all j = j′. That is, the CTCU model estimates covariances among residuals for variables that share a common method. As such, Θ is a symmetric matrix containing unique (confounded with Method) variances along the diagonal and covariances among uniquenesses on the off-diagonal for variables that share a common method. Off-diagonal elements will form M subcovariance matrices for MTMM data in which Traits are nested within Methods and M− 1 nonzero subdiagonals when Methods are nested within Traits for an M-Method MTMM matrix.

Several studies have compared the CTCM model and the CTCU model with simulated data (Conway et al., 2004; Marsh & Bailey, 1991; Zhang, Jin, Leite, & Algina, 2014) or previous published data (Lance et al., 2002). This literature indicates that while the CTCU model largely solves C&A problems associated with the CTCM model, it suffers from a number of conceptual limitations (Lance et al., 2002) and yields upwardly biased estimates of Trait factor loadings and correlations when Method factors are correlated (Conway et al., 2004), and Lance, Dawson, Birklebach, and Hoffman’s (2010) and Lance, Fan, Siminovsky, Morgan, and Shaikh’s (2014) reviews show that this is routinely the case. The sparse literature on the CTCM-R model (Dillon, Kumar, & Mulani, 1987; Lance & Fan, 2016; Marsh, 1989) suggests that it may retain the positive features of the CTCM model and yet yield C&A rates that are comparable to the CTCU model.

CTCM-R Model

In the traditional CFA model ( $\hat{Σ} = Λ Φ Λ' + Θ$ ) estimates of uniquenesses in Θ can be negative and it has been speculated that these may be due to small population uniquenesses, sampling error, model underidentification, skewed data, overfitting, and/or multiplicative relationships (Cote, 1995; Wothke, 1993). Rindskopf (1983) presented a reparameterization of the CFA model (CFA-R) that effectively implements nonnegative restrictions on uniquenesses estimates. This model can be written as

\hat{Σ} = [\begin{matrix} Λ & Θ_{δ}^{1 / 2} \end{matrix}] [\begin{matrix} Φ & sym \\ 0 & I \end{matrix}] [\begin{matrix} Λ' \\ Φ_{δ}^{1 / 2} \end{matrix}]

Here, Λ and Φ contain the fixed and free factor loadings and factor correlations, respectively, just as in the traditional CFA model, $Θ_{δ}^{1 / 2}$ is a diagonal matrix containing the square roots of the uniquenesses and I is the identity matrix. Thus, irrespective of whether the estimates in $Θ_{δ}^{1 / 2}$ are positive or negative, their squares are positive.

The CFA-R model has been implemented successfully in only a few CFA (La Du & Tanaka, 1989; Nagy, Trautwein, & Lüdtke, 2010) and SEM studies (Hashoul-Andary et al., 2016; Walker & Weber, 1987) where it effectively resolved inadmissibly issues. However, it appears that the CFA-R model has been studied systematically only once before using simulated data. Dillon et al. (1987) compared (a) the CFA-R model, (b) an alternative parameterization presented by Bentler (1976), and (c) simply fixing offending estimates (i.e., Heywood cases) to zero on an ad hoc basis in a simple CFA model containing five manifest and two latent variables. Based on their findings, Dillon et al. (1987) had little to recommend for the first two approaches, saying “though these two approaches represent theoretically elegant ways of handling this problem, they have limited practical usefulness compared with the simpler approach in which the offending parameter estimate is fixed at zero” (p. 134). We disagree with respect to the CFA-R model: fixing offending estimates to zero is an ad hoc symptom-based approach, while the CFA-R model represents a more general and preemptive model-based strategy.

The CFA-R model may be extended to the CTCM-R model, written as

MTMM = [\begin{matrix} Λ_{T} & Λ_{M} & Θ^{1 / 2} \end{matrix}] [\begin{matrix} Φ_{TT'} & sym \\ 0 & Φ_{MM'} \\ 0 & 0 & I \end{matrix}] [\begin{matrix} Λ_{T}' \\ Λ_{M}' \\ Θ^{1 / 2} \end{matrix}]

where Λ_T (p×T) and Λ_M (p×M) contain the Trait and Method factor loadings, respectively; Φ_TT′ (T×T) and Φ_MM′ (M×M) contain Trait and Method correlations, respectively; Θ^1/2 (p×p) is a diagonal matrix containing the square roots of the uniquenesses; and I (p×p) is the identity matrix. As with the CFA-R model, the CTCM-R model avoids Heywood cases that are frequently encountered using the CTCM model to analyze MTMM data because uniquenesses are calculated as the squared elements in Θ^1/2 (i.e., regardless of whether individual elements in Θ^1/2are positive or negative, their squares are positive by definition). However, inadmissible solutions can still occur because estimated Trait and/or Method correlations can still exceed |1.0|.

The CTCM-R model has also been applied in only a few MTMM studies (e.g., Kinicki, McKee-Ryan, Schriesheim, & Carson, 2002; Lance et al., 2010; Marsh, 1989), and Lance and Fan (2016) and Lance et al. (2014) found that it performed effectively in the reanalysis of previously published MTMM data. However, we know of no systematic attempt to study the performance of the CTCM-R model relative to the CTCM and CTCU models in the analysis of simulated data with known population parameters.

Study Purpose

The goal of this study was to systematically compare the CTCM-R model relative to the CTCM and CTCU models using Monte Carlo simulations under a variety of study conditions that are representative of characteristics of MTMM matrices present in the literature. We chose the CTCM and CTCU models as comparison models because (a) of their popularity in the literature and (b) their relative advantages and disadvantages are already well known. Our general expectation was that the CTCM-R model would demonstrate superior C&A rates as compared to the CTCM model and simultaneously avoid parameter estimation bias that is known to plague the CTCU model. That is, we expected that the CTCM-R model would preserve the strengths and avoid the weaknesses of both the CTCM and CTCU models.

Method

This simulation study had four major steps: (a) generating model-implied population covariance matrices based on predefined population values; (b) analyzing sample data with the CTCM, CTCM-R, and CTCU models; (c) obtaining model convergence and admissibility rates, goodness-of-fit indices, and parameter estimates; and (d) examining and comparing the three models’ performance in terms of model C&A rates, model–data fit, and bias in parameter estimates (i.e., deviation of model estimates from true population values).

Population Characteristics

Population data were simulated to represent realistic MTMM data toward maximizing the generalizability of its findings to actual MTMM data found in the literature. As such, population values for the current simulation study came from the latest large-scale review of MTMM studies to date (Lance et al., 2014) and represented high, medium, and low values on the various characteristics of MTMM data that were manipulated (see below). Table 1 shows a summary of characteristics of MTMM matrices reviewed by Lance et al. (2014) and the population parameters and their levels chosen for the present study. Values chosen for this study represented a wide variety of subject areas (e.g., assessment centers, personality, attitudes, job performance, self-concept) and measurement methods (e.g., response formats, test forms, rater sources) and included values that others have used in MTMM simulation studies (Conway et al., 2004; Lance, Woehr, & Meade, 2007). As such, the simulation was a completely crossed factorial design with 4 (Sample Size: 100, 250, 500, 1,000) × 3 (Trait Factor Loading Size: .31, .50, .69) × 3 (Trait Correlation Size: .07, .36, .61) × 3 (Method Factor Loading Size: .17, .28, .48) × 3 (Method Correlation Size: .02, .29, .58) × 2 (Number of Traits Factors: 3, 5) × 2 (Number of Method Factors: 3, 5), = 1,296 unique conditions.

Table 1.

Population Values of Simulation Study.

	Characteristics of MTMM studies reviewed by Lance et al. (2014)			Current study
	Median	20th percentile	80th percentile	Population values
Number of methods	3	2	5	3, 5
Number of traits	4	3	6	3, 5
Sample size	183	90	421	100, 250, 500, 1,000
Method loading	.28	.17	.48	.17, .28, .48
Trait loading	.50	.31	.69	.31, .50, .69
Method correlation	.29	.02	.58	.02, .29, .58
Trait correlation	.36	.07	.61	.07, .36, .61

Note. MTMM = multitrait–multimethod. k = 258 studies.

Procedure

We generated population correlation matrices using parameter values shown in Table 1. Specifically, population MTMM matrices were created using combinations of predefined population values shown in Table 1 with the CTCM model as the generating population model. Note that within a specified condition, all factor loadings and correlations were varied ±.10 so that their averages were equal to the specified population values. Table 2 shows one example of the Λ and Φ matrices used to generate one model-implied matrix by calculating $\hat{Σ} = Λ Φ Λ' .$ Uniquenesses were calculated as $Θ_{δ} = I - diag (\hat{Σ}) .$ Then the ith population matrix was calculated as $Σ = Λ Φ Λ' + Θ_{δ} .$ Once all model-implied population matrices were generated, they were transferred into triangular matrices by Cholesky decomposition. We then entered these decomposed triangular matrices into PRELIS (Jöreskog & Sörbom, 1996) for adding random error components. For every sample size, we created 100 sample data sets for each population matrices. Thus, a total of 129,600 sample covariance matrices was generated (1,296 population conditions × 100 replications).

Table 2.

Example of Lambda and Phi Matrices for Generating a 5T3M Population Matrix.

		Population design matrix
Population matrix		M1	M2	M3	T1	T2	T3	T4	T5
Lambda matrix	M1T1	0.18			0.4
	M1T2	0.28				0.5
	M1T3	0.38					0.6
	M1T4	0.18						0.4
	M1T5	0.28							0.6
	M2T1		0.28		0.5
	M2T2		0.38			0.4
	M2T3		0.18				0.6
	M2T4		0.28					0.5
	M2T5		0.38						0.4
	M3T1			0.38	0.6
	M3T2			0.18		0.5
	M3T3			0.28			0.4
	M3T4			0.38				0.6
	M3T5			0.18					0.5
Phi matrix	M1	1
	M2	0.29	1
	M3	0.19	0.39	1
	T1				1
	T2				0.36	1
	T3				0.26	0.46	1
	T4				0.46	0.26	0.36	1
	T5				0.36	0.36	0.46	0.26	1

Note. MTMM = multitrait–multimethod. The pattern matrix for 5T3M population MTMM matrix with averaged Method Loading at .28, Trait loading at .5, Method Correlation at .29, Trait correlation at .36.

We used LISREL 8 (Jöreskog & Sörbom, 1993) to analyze the generated sample data with each of the three targeted CFA-MTMM models (i.e., CTCM, CTCM-R, and CTCU). We harvested data from LISREL output files that contain model convergence and admissibility indicators, a number of model-fit indices (e.g., comparative fit index, Tucker–Lewis index, χ², root mean square error of approximation), and parameter estimates that were all saved externally and analyzed for model performance comparisons.

Results

We conducted six 4 × 3 × 3 × 3 × 3 × 2 × 2 analyses of variance using the following dependent variables:

A binary variable coded 1 = the solution was convergent within 1,000 iterations and admissible (contained no standardized factor loadings or factor correlations >1.0 and no negative uniquenesses, where applicable) = 1, or 0 = the solution was nonconvergent and/or inadmissible (the solution did not converge within 1,000 iterations and/or contained at least one standardized factor loading or factor correlation >1.0 or at least one negative uniqueness)

A binary indicator of model goodness-of-fit coded 1 = the solution met Hu and Bentler’s (1999) cutoff criteria for Bentler’s (1990) comparative fit index and the Tucker–Lewis index ≥.95, the root mean square error of approximation ≤.06, and the standardized root mean square residual ≤.08, or 0 = the solution failed to satisfy one or more of these cutoff criteria

Bias in estimation of Trait and Method factor loadings and correlations measured as the deviation between the estimates and the true population values

Due to the extremely large sample size, every effect was statistically significant, even using an alpha rate as stringent as .0001. As such, we chose to report results associated with a minimally practically significant effect size indicated by η²≥ .02.¹ Using this criterion, no higher order effects beyond two-way interactions were indicated as being practically significant.

As Table 3 shows, and as expected, the CTCM model suffered C&A problems with an overall lower C&R rate (29.2%) than the other two models (η² = .237), despite the fact that it was the generating model. The CTCM-R model, even though it is mathematically equivalent to the CTCM model, returned C&A solutions much more often (62.5%), approaching that of the CTCU model (87.5%). As such, even though the CTCM and CTCM-R are mathematically equivalent, the CTCM-R parameterization returned C&A solutions more than twice as often as the more traditional CTCM model.

Table 3.

Results: Model Convergence and Admissibility (C&A).

Effect	% C&A solutions	η²
Analytic model
CTCM	29.2%	.237
CTCM-R	62.5%
CTCU	87.5%
Sample size
100	39.7%	.077
250	55.9%
500	66.9%
1,000	76.5%
Population trait loading
.31	45.7%	.043
.50	64.3%
.69	69.2%
Population method loading
.17	45.5%	.050
.28	61.4%
.48	72.2%

Note. C&A = model convergence and admissibility; CTCM = correlated trait–correlated method; CTCU = correlated trait–correlated uniqueness.

Table 3 also shows an effect of sample size on C&A (η² = .077): not surprisingly, sample estimates became more stable with larger sample sizes. C&A rates also increased with larger population Trait (η² = .043) and Method (η² = .050) factor loadings. Since both Trait and Method factors represent sources of systematic variance, this suggests that, in general, C&A solutions are achieved more often with more reliable data.

Table 4 shows results for two small effects for analytic model and Trait loading level. Here, the best model fit is for the CTCM model and in one sense this comes as no surprise because it is the generating model. However, even though it is the generating model the CTCM model reached C&A solutions relatively infrequently. Thus, when a C&A solution is achieved for the CTCM model it is usually a very well fitting solution. The CTCM-R model is more forgiving in the sense that it achieves C&A solutions far more often than the CTCM model but sometimes at the cost of obtaining a reproduced matrix whose fit to the data does not satisfy all of the Hu and Bentler (1999) cutoff criteria for the goodness-of-fit indices we analyzed. Of course, the CTCU model was the most forgiving of all in terms of achieving C&A but had the lowest percentage of solutions yielding good model fit because, of course, it is the wrong model (i.e., it was not the generating model). Still, model fit was good for the CTCU model almost 92% of the time. This is consistent with Lance et al.’s (2007) findings that good model fit is not necessarily informative as to the correctness of the model fit to the data. Table 4 also shows that higher population Trait loadings also led to more frequent C&A solutions and better model fit, again likely due to variables’ higher reliabilities in these conditions.

Table 4.

Results: Good Model Fit.

Effect	# C&A solutions	% good fit	η²
Analytic model
CTCM	37,812	99.4%	.023
CTCM-R	80,997	97.9%
CTCU	113,418	91.8%
Population trait loading
.31	59,224	90.8%	.020
.50	83,280	94.9%
.69	89,723	98.2%

Note. C&A = model convergence and admissibility; CTCM = correlated trait–correlated method; CTCU = correlated trait-correlated uniqueness.

Table 5 shows results for parameter estimation bias. Results for the CTCM and CTCM-R models indicated negligible bias for Trait factor loadings and slightly negative (conservative) bias for Trait correlations. Conversely, estimates for both Trait factor loadings and correlations were significantly positively biased for the CTCU model and this is consistent with previous research (Conway et al., 2004). We also found two interaction effects on Trait factor loading bias and these are shown in Figures 1 and 2. Figure 1 shows that Trait factor loading bias remains negligible across levels of Method factor loadings for the CTCM and CTCM-R models, but as Method loadings increase, positive bias in the estimation of Trait factor loadings increases dramatically for the CTCU model (η² = .021). Similarly, Figure 2 shows that bias in Trait loading estimates for the CTCM and CTCM-R models remains negligible across variations in Method correlations but as Methods correlate more highly, substantial bias in Trait factor loading estimates also incurs for the CTCU model (η² = .024). These findings are consistent with Conway et al.’s (2004) results: When Method factor loadings or correlations are low, minimal bias incurs for estimates of Trait factor loadings and correlations for the CTCU model, but when Method factor loadings and correlations are high, substantial bias incurs. This would not be a problem were it not for the fact that several large-scale reviews of MTMM-related research have estimated mean Method correlations at .34 (Doty & Glick, 1998), .37 (Buckley, Cote, & Comstock, 1990), .48 (Cote & Buckley, 1987), .52 (Lance et al., 2010), and .63 (Williams, Cote, & Buckley, 1989)—values that Figures 1 and 2 show introduce significant bias into CTCU estimates of Trait factor loadings and correlations.

Table 5.

Results: Parameter Estimation Bias.

Dependent variable	M	SD	η²
Trait loading
CTCM	−.001	.051	.055
CTCM-R	.016	.076
CTCU	.045	.050
Trait correlation
CTCM	−.025	.149	.125
CTCM-R	−.033	.165
CTCU	.082	.131

Note. CTCM = correlated trait–correlated method; CTCU = correlated trait-correlated uniqueness.

Figure 1.

Interaction effect of method loading level and analytic model on trait factor loading bias.

Figure 2.

Interaction effect of method correlation level and analytic model on trait factor correlation bias.

Discussion

This study presented a comprehensive examination of the CTCM-R model in relation to two other widely applied CFA-MTMM models, the CTCM and CTCU models, in terms of C&A, model fit, and accuracy of parameter estimates. In general terms, the most important finding from this study was that the CTCM-R model at the same time preserved positive features of both the CTCM and CTCU models and avoided their shortcomings. The CTCM-R model is just as faithful to Campbell and Fiske’s (1959) original theoretical presentation of the MTMM methodology as is its mathematically identical CTCM model but largely avoids its C&A problems. On the other hand, the CTCM-R model maintained C&A rates that approached those of the CTCU model without suffering its conceptual limitations and inherent estimation biases.

The CTCM and CTCM-R models yielded slightly upwardly biased estimates for Method factor loadings (mean bias = .027 and .055, respectively) and Method correlations (mean bias = .051 and .055, respectively) and this was unexpected as the CTCM model was the generating model. But these biases pale in comparison to the large negative biases incurred by the CTCU model for Method loadings (mean bias = −.313) and correlations (mean bias = −.304) by virtue of the fact that these parameters are not estimated under the CTCU model. In effect, the omission of Method factors under the CTCU model creates an unmeasured variables problem (James, 1980) that, under routine application of the CTCU model, (a) renders Method variance unquantified in an amalgam of Method, unique and nonsystematic effort variance, and (b) leads to the biases in Trait factor loadings and correlations shown in Table 5. We do note, however, that Conway (1998) and Scullen (1999) have proposed two-step procedures by which Method variance can be estimated from uniquenesses’ covariances. However, to our knowledge, these procedures have been reported being used only four times previously (Li, Hughes, Kwok, & Hsu, 2012; Lievens & Conway, 2001; Siers & Christiansen, 2013; Van Iddekinge, Raymark, Eidson, & Attenweiler, 2004), and the accuracy of these procedures has not been investigated. This is an obvious direction for future work.

Cautions and Limitations

First, the CTCM-R model retains the positive features of the CTCM model and largely avoids its C&A problems, but it does not eliminate all possible sources of nonconvergent and inadmissible solutions. Model misspecifications can still lead to Trait and/or Method factor correlation estimates >|1.0|. Note that the CTCU model also does not solve this problem for Trait factors and skirts the issue entirely with respect to Method factors because they are not modeled.

Second, it could be argued that the CTCM-R model is unnecessary as most SEM software allows the user to intervene when faced with inadmissible estimates. For example, EQS automatically fences parameter estimates to boundary values (Bentler, 2006), and inequality constraints can be invoked in LISREL (Jöreskog & Sörbom, 1993) and Mplus (Muthén & Muthén, 1998-2012). However, these “fixes” may obscure important sources of model misspecification and are often undertaken in a post hoc, atheoretical fashion (e.g., Bowler & Woehr, 2006). Savalei and Kolenikov (2008) provide a discussion of the trade-offs between diagnosticity of unconstrained but improper solutions versus constrained and interpretable solutions. As a model-based (vs. symptom-based) solution, the CTCM-R model seeks a pre-emptive solution to inadmissible estimates and uninterpretable solutions.

Third, we note that there are many other mathematical models for MTMM data that we did not consider here. For example, Widaman (1985) presented a taxonomy of CFA models for MTMM data, the majority of which were special cases of the CTCM model. One of these, a correlated trait–uncorrelated method (CTUM) model is similar to the CTCU model in many respects² and has been shown to produce similarly biased model parameter estimates (Lance & Fan, 2016). Like the CTCU model, Eid’s (2000) CTC(M-1) model exhibits better C&A rates than the CTCM model but provides an incomplete representation of the MTMM factor space and for this reason we chose not to study it here. Also, we chose not to study Campbell and O’Connell’s (1967) direct product model as it is concerned with the estimation of interactive Trait × Method relationships whereas we investigated only linear effects. Finally, it should be noted that there is an emerging literature on generalizability theory (GT) approaches to the analysis of MTMM-like data (e.g., Jackson, Michaelides, Dewberry, & Kim, 2016; Putka & Hoffman, 2013; Woehr, Putka, & Bowler, 2012). GT offers certain advantages over CFA approaches to the analysis of MTMM data. For example, GT is typically capable of modeling (many) more data facets and can much more easily accommodate ill-structured data (Putka, Lance, Le, & McCloy, 2011) compared with CFA, but GT necessarily constrains measurement facets and elements within facets to be orthogonal and can be computationally intensive compared to CFA (Woehr et al., 2012).

Conclusion

The present study shows that the CTCM-R model simultaneously retains the attractive features of both the CTCM and CTCU models and avoids their limitations. Combined with evidence that the CTCM-R model performs well in the analysis of actual MTMM data (e.g., Lance et al., 2014) the present results with simulated data indicate that the CTCM-R model is a viable candidate analytic model for MTMM data.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Bandalos

D. L.

(2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9, 78-102. doi:10.1207/S15328007SEM0901_5

Bentler

P. M.

(1976). Multistructure statistical model applied to factor analysis. Multivariate Behavioral Research, 11, 3-25.

Bentler

P. M.

(1990). Comparative fit indices in structural models. Psychological Bulletin, 107, 238-246.

Bentler

P. M.

(2006). EQS 6 structural equations program manual. Encino, CA: Multivariate Software.

Bowler

M. C.

Woehr

D. J.

(2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114-1124.

Bowler

M. C.

Woehr

D. J.

(2009). Assessment center construct-related validity: Stepping beyond the MTMM matrix. Journal of Vocational Behavior, 91, 1114-1124. doi:10.1016/j.jvb.2009.03.008

Brannick

M. T.

Spector

P. E.

(1990). Estimation problems in the block-diagonal model of the multitrait–multimethod matrix. Applied Psychological Measurement, 14, 325-339.

Buckley

M. R.

Cote

J. A.

Comstock

S. M.

(1990). Measurement errors in the behavioral sciences: The case of personality/attitude research. Educational and Psychological Measurement, 50, 447-474.

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

10.

Campbell

D. T.

O’Connell

E. J.

(1967). Method factor in multitrait-multimethod matrices: Multiplicative rather than additive?Multivariate Behavioral Research, 2, 409-426.

11.

Conway

J. M.

(1998). Estimation and uses of the proportion of method variance for multitrait-multimethod data. Organizational Research Methods, 1, 209-222.

12.

Conway

J. M.

Lievens

Scullen

S. E.

Lance

C. E.

(2004). Bias in the correlated uniqueness model for MTMM data. Structural Equation Modeling, 11, 535-559.

13.

Cote

J. A.

(1995). What causes estimation problems when analyzing MTMM data?Advances in Consumer Research, 22, 345-353.

14.

Cote

J. A.

Buckley

M. R.

(1987). Estimating trait, method and error variance: Generalizing across 70 construct validation studies. Journal of Marketing Research, 26, 315-318.

15.

Dillon

W. R.

Kumar

Mulani

(1987). Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101, 126-135.

16.

Doty

D. H.

Glick

W. H.

(1998). Common methods bias: Does common methods variance really bias results?Organizational Research Methods, 1, 374-406. doi:10.1177/109442 819814002

17.

Eid

(2000). A multitrait–multimethod model with minimal assumptions. Psychometrika, 65, 241-261.

18.

Eid

Lischetzke

Nussbeck

F. W.

(2006). Structural equation models for multitrait-multimethod data. In Eid

Diener

(Eds.), Handbook of multimethod measurement in psychology (pp. 283-299). Washington, DC: American Psychological Association.

19.

Hashoul-Andary

Assayag-Nitzan

Yuval

Aderka

I. M.

Litz

Bernstein

(2016). A longitudinal study of emotional distress intolerance and psychopathology following exposure to a potentially traumatic event in a community sample. Cognitive Therapy Research, 40, 1-13. doi:10.1007/s10608-015-9730-4

20.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118

21.

Jackson

D. J. R.

Michaelides

Dewberry

Kim

(2016). Everything that you have ever been told about assessment center ratings is confounded. Journal of Applied Psychology, 101, 976-994. doi:10.1037/apl0000102

22.

James

J. R.

(1980). The unmeasured variables problem in path analysis. Joural of Applied Psychology, 65, 415-421.

23.

Jöreskog

K. G.

Sörbom

(1993). LISREL 8 user’s reference guide. Chicago, IL: Scientific Software.

24.

Jöreskog

K. G.

Sörbom

(1996). PRELIS 2: User’s reference guide. Scientific Software.

25.

Kenny

D. A.

(1976). An empirical application of confirmatory factor analysis to the multitrait-multimethod matrix. Journal of Experimental Social Psychology, 12, 247-252.

26.

Kenny

D. A.

Kashy

D. A.

(1992). Analysis of the multitrait–multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.

27.

Kinicki

A. J.

McKee-Ryan

F. M.

Schriesheim

C. A.

Carson

K. P.

(2002). Assessing the construct validity of the Job Descriptive Index: A review and meta-analysis. Journal of Applied Psychology, 87, 14-32.

28.

La Du

T. J.

Tanaka

J. S.

(1989). Influence of sample size, estimation method, and model specification on goodness-of-fit assessments in structural equation models. Journal of Applied Psychology, 74, 625-635.

29.

Lance

C. E.

Dawson

Birklebach

Hoffman

B. J.

(2010). Method effects, measurement error, and substantive conclusions. Organizational Research Methods, 13, 435-455. doi:10.1177/1094428109352528

30.

Lance

C. E.

Fan

(2016). Convergence, admissibility and fit for alternative confirmatory factor analysis models for multitrait-multimethod (MTMM) data. Educational and Psychological Measurement, 76, 487-507. doi:10.1177/0013164415601884

31.

Lance

C. E.

Fan

Siminovsky

Morgan

Shaikh

(2014, May). A rose is a rose: Is a “method” a method? Paper presented at the meeting of the Society for Industrial and Organizational Psychology, Honolulu, HI.

32.

Lance

C. E.

Noble

C. L.

Scullen

S. E.

(2002). A critique of the correlated trait–correlated method and correlated uniqueness models for multitrait–multimethod data. Psychological Methods, 7, 228-244. doi:10.1037//1082-989X.7.2.228

33.

Lance

C. E.

Woehr

D. J.

Meade

A. W.

(2007). Case study: A Monte Carlo investigation of assessment center construct validity models. Organizational Research Methods, 10, 430-448. doi:10.1177/1094428106289395

34.

Hughes

J. N.

Kwok

Hsu

(2012). Evidence of convergent and discriminant validity of child, teacher, and peer reports of teacher-student support. Psychological Assessment, 24, 54-65. doi:10/1037/a0024481

35.

Lievens

Conway

J. M.

(2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202-1222. doi:10.1037/0021-9010.86.6.1202

36.

Marsh

H. W.

(1989). Confirmatory factor analysis of multitrait–multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361.

37.

Marsh

H. W.

Bailey

(1991). Confirmatory factor analyses of multitrait-multimethod data: A comparison of alternative models. Applied Psychological Measurement, 15, 47-70.

38.

Muthén

L. K.

Muthén

B. O.

(1998-2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

39.

Nagy

Trautwein

Lüdtke

(2010). The structure of vocational interests in Germany: Different methodologies, different conclusions. Journal of Vocational Behavior, 76, 153-169. doi:10.1016/j.jvb.2007.07.002

40.

Olsson

U. H.

Foss

Troye

S. V.

Howell

R. D.

(2000). The performance of ML, GLS and WLS estimation in structural equation modeling under conditions of misspecification and nonnormality. Structural Equation Modeling, 7, 557-595. doi:10.1207/S15328 007SEM0704_3

41.

Putka

D. J.

Hoffman

B. J.

(2013). Clarifying the contribution of assessee-, dimension-, exercise-, and assessor-related effects to reliable and unreliable variance in assessment center ratings. Journal of Applied Psychology, 98, 114-133. doi:10.1037/a0030887

42.

Putka

D. J.

Lance

C. E.

McCloy

R. A.

(2011). A cautionary note on modeling multitrait-multirater data arising from ill-structured measurement designs. Organizational Research Methods, 14, 503-529. doi:10.1177/1094428110362107

43.

Rindskopf

(1983). Parameterizing inequality constraints on unique variances in linear structural models. Psychometrika, 48, 73-83.

44.

Savalei

Kolenikov

(2008). Constrained versus unconstrained estimation in structural equation modeling. Psychological Methods, 13, 150-170. doi:10.1037/1082-989X.13.2.150

45.

Schmitt

Stults

D. M.

(1986). Methodology review: Analysis of multitrait–multimethod matrices. Applied Psychological Measurement, 10, 1-22.

46.

Scullen

S. E.

(1999). Using confirmatory factor analysis of correlated uniquenesses to estimate method variance in multitrait-multimethod matrices. Organizational Research Methods, 2, 275-292.

47.

Siers

B. P.

Christiansen

N. D.

(2013). On the validity of implicit association measures of personality traits. Personality and Individual Differences, 54, 361-366. doi:10.1016/j.paid .2012.10.004

48.

Van Iddekinge

C. H.

Raymark

P. H.

Eidson

C. E.

Jr. Attenweiler

W. J.

(2004). What do structured interviews really measure? The construct validity of behavioral description interviews. Human Performance, 17, 71-93.

49.

Walker

Weber

(1987). Supplier competition, uncertainty and make-or-buy decisions. Academy of Management Journal, 30, 589-596.

50.

Widaman

K. F.

(1985). Hierarchically nested covariance structures models for multitrait–multimethod data. Applied Psychological Measurement, 9, 1-26.

51.

Williams

L. J.

Cote

J. A.

Buckley

M. R.

(1989). Lack of method variance in self-reported affect and perceptions at work: Reality or artifact?Journal of Applied Psychology, 74, 462-446.

52.

Woehr

D. J.

Putka

D. J.

Bowler

M. C.

(2012). An examination of G-theory methods for modeling multitrait–multimethod data: Clarifying links to construct validity and confirmatory factor analysis. Organizational Research Methods, 15, 134-161. doi:10.1177/1094428111408616

53.

Wothke

(1993). Nonpositive definite matrices in structural modeling. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 256-293). Newbury Park, CA: Sage.

54.

Zhang

Jin

Leite

W. L.

Algina

(2014). Additive models for multitrait-multimethod data with a multiplicative trait-method relationship: A simulation study. Structural Equation Modeling, 21, 68-80. doi:10.1080/10705511.2014.856698