Alternative Techniques for Assessing Common Method Variance

Statistical Procedures for Detecting, Estimating, and Correcting for CMV

There are two broad classes of statistical techniques proposed in extant literature to detect and correct for CMV. One class of statistical techniques includes those based on the traditional multitrait-multimethod (traditional MTMM) analysis (Campbell & Fiske, 1959) and confirmatory factor analysis complemented by MTMM (referred to as CFA-based MTMM henceforth). Another well-documented set of statistical remedies for CMV is classified as partial correlation techniques (Podsakoff et al., 2003). Since Podsakoff et al. (2003, 2012) provide excellent detailed reviews of most of these procedures, along with discussing the advantages and disadvantages of each, we refer the reader to these articles and provide only a brief explanation of the partial correlation techniques that are of primary focus in this study.

One particular partial correlation method, namely, Lindell and Whitney’s (2001) implementation of the marker variable technique (referred to as L&W-MVT henceforth) is particularly relevant for the remainder of this article. It is noteworthy that given its simplicity and ease of implementation, L&W-MVT is becoming increasingly popular among researchers for estimating the level of CMV afflicting data in self-report survey-based studies (e.g., Alge, Ballinger, Tangirala, & Oakley, 2006; Becker, Greve, & Albers, 2009; Frazier, Maltz, Antia, & Rindfleisch, 2009; Mathwick, Wiertz, & De Ruyter, 2008). When applying L&W-MVT in an a priori fashion, researchers need to implement one or more prechosen marker variable(s) along with the substantive variables into their survey instrument. By definition, the marker variable(s) should be theoretically unrelated to at least one of the substantive variables included on the questionnaire, but nonetheless, it is implied that it/they also load/s on the presumed common method variable (Lindell & Whitney, 2001). The added marker variable should also have published evidence of high validity and reliability. Being unrelated to substantive variables in the study, the marker variable should theoretically have a zero correlation with the substantive variables. Given this, any covariation that exists between this marker variable and a theoretically unrelated substantive variable(s) is due almost entirely to the method-associated measurement features that this marker variable shares with such substantive variables. ²

In an a priori application of L&W-MVT, the smallest of all correlations between the marker variable and the other substantive variables in a study (referred to as r_M henceforth) is assumed to be the study-specific estimate of CMV (Lindell & Whitney, 2001; Richardson, Simmering, & Sturman, 2009; Williams et al., 2010). To compute a correlation corrected for the effects of CMV, L&W-MVT uses the following formula:

r Y i . S = r Y i − r S 1 − r S ,

1

where r_Yi·S is the correlation corrected for CMV and r_Yi is the original observed uncorrected correlation between two substantive variables. Further, r_S is either an a priori estimate of CMV (e.g., r_M in situations when the researcher is able to introduce the marker variable into the survey instrument a priori to data collection) or a post hoc proxy for CMV (e.g., r_{S 1} or r_{S 2}, as will be discussed later in this article). The extent of CMV bias can be assessed by determining whether an originally significant correlation (r_Yi ) post CMV correction (r_Yi.S ) becomes statistically nonsignificant.

The CMV-corrected partial correlation r_Yi.S is also subject to sampling error, and its significance can be tested by a t statistic calculated using the following equation:

t α / 2 , N − 3 = r Y i . S ( 1 − r Y i . S 2 ) ( 1 − r Y i . S 2 ) ( N − 3 ) ( N − 3 ) ,

2

where t _{α/2, N − 3} is the test statistic used to assess the significance of the CMV-corrected correlation r_Yi·S and N represents the study-specific sample size. It is relevant to note that L&W-MVT makes an equal weight assumption of CMV in that it assumes that CMV equally affects all the relationships between substantive variables in a particular research context. It is also noteworthy that L&W-MVT assumes that CMV can only inflate, but never deflate, trait correlations.

Importantly, Williams et al. (2010) recently presented a very flexible yet rigorous implementation of the marker variable technique (referred to as Williams-MVT henceforth). Williams-MVT requires the researcher to execute several structural equation models on the collected data and then compare these models by undertaking chi-square difference tests. Specifically, in addition to the traditional CFA-based measurement model with the marker variable, Williams-MVT requires researchers to execute the baseline model (i.e., substantive variables correlated with one another but not with marker variable, and substantive items also do not load on marker variable), the method-C model (i.e., substantive variables correlated with one another but not with marker variable, and items of substantive variables load on marker variable with equal magnitudes), the method-U model (i.e., substantive variables correlated with one another but not with marker variable, and items of substantive variables load on marker variable with unconstrained-unequal magnitudes), and the method-R model (i.e., model similar to the method-C or method-U models, but correlations across substantive variables are constrained to the values seen in the baseline model) and then perform appropriate model comparisons. Williams-MVT also requires researchers to undertake decomposition of the reliabilities of substantive variables into method-associated and trait-associated reliability. Finally, this method urges researchers to execute the method-S model so as to implement a sensitivity analysis compensating for the sampling error on estimates of CMV. The reader is referred to Williams et al. (2010) for further details.

Williams-MVT has several unique strengths that compensate for some of the concerns that have been raised among researchers about L&W-MVT (Podsakoff et al., 2012). First, Williams-MVT attempts to test and control for CMV at an item level by allowing the items of substantive variables in the study to load on the marker variable in the study. Second, Williams-MVT can also test the validity of the equal weights assumption that L&W-MVT makes universally. Finally, unlike L&W-MVT, Williams-MVT does not assume that CMV can only lead to inflation in the trait correlations. Overall, Williams et al. (2010) present a comprehensive implementation of MVT. However, Williams-MVT also places heavier demands on researchers as its implementation is significantly more detailed and complex compared to an application of L&W-MVT.

Because of the concerns raised regarding the assumptions made by L&W-MVT (e.g., Podsakoff et al., 2003), we believe that it is important to establish validity of L&W-MVT (vis-à-vis more comprehensive techniques such as Williams-MVT and MTMM). Hence, in the next subsection, we undertake an empirical study to compare and assess several of these competing techniques in the TPB domain. If in the TPB domain the L&W-MVT produces results that are very similar to those from the other techniques, we will have more confidence in using L&W-MVT to reassess the results of published TPB studies, as the L&W-MVT is uniquely suited to account for the effects of CMV.

An Empirical Comparison of Techniques for Detecting and Estimating CMV

This section empirically compares four competing techniques to detect whether CMV substantially influences TPB study results in terms of both magnitude and significance and estimate the level of CMV afflicting the data using multiple techniques. These techniques include L&W-MVT, Williams-MVT, traditional MTMM, and CFA-based MTMM. This empirical study is unique in that it tests the efficacy of the L&W-MVT by comparing it with two rigorous statistical techniques, namely, Williams-MVT and CFA with MTTM.

Methodology

In this empirical study, we implement these four techniques in two distinct and validated TPB research contexts. The first research context is the implementation of TPB to predict fast food (FF) patronage intentions based on the model proposed by Bagozzi, Wong, Abe, and Bergami (2000). Bagozzi et al.’s model suggests that three TPB variables, namely, attitudes (ATT-FF), subjective norms (SN-FF), and past behavior (PB-FF) toward fast food, together determine both intentions (INTENT-FF) of consuming fast food and behavioral expectations (EXPECT-FF) of consuming fast food. The second research context is the implementation of TPB to study compliance with speed limits (SL) when driving, based on the model proposed by Elliott, Armitage, and Baughan (2003). Elliott et al.’s model suggests that three TPB variables, namely, attitudes (ATT-SL), subjective norms (SN-SL), and perceived behavioral control (PBC-SL) toward complying with speed limits, determine intentions (INTENT-SL) of complying with speed limits, which in turn predict expected future behavior (FB-SL) of complying with speed limits.

One hundred and forty-one undergraduate students consented to participate in the study in small batches in a laboratory setting. All of those who consented completed the study in its entirety. In addition, all of the participants indicated that they had driven in the past four months, thus making them eligible for answering questions related to speed limit compliance. The same participants responded to both the fast food patronage and speed limit compliance questions. For both studies, we balanced the order in which participants responded to the Web-based questionnaire and the paper-and-pencil (P&P) questionnaire. Further, an unrelated activity that lasted 45 to 60 minutes temporally separated administration of the Web and the P&P questionnaires. The median age of the participants was 20 years and 62% were female. Details of the survey instruments used for both research contexts, including specific measurement items, are provided in Appendix A.

In designing the study, we carefully chose marker variables that were theoretically unrelated to the phenomena under consideration. As such, the marker variable used for the fast food research context was Raju’s (1980) seven-item Exploratory Shopper scale. Likewise, Moorman and Matulich’s (1993) three-item Work-Life Balance scale represented the marker variable for the speed limit research context. As is recommended by Lindell and Whitney (2001), we ensured that these marker variables shared all of the method-associated measurement features with the respective substantive variables in each research context. Essentially, we operationalized the marker variable items in a way that was similar in content, structure, and format to the substantive items.

To implement the MTMM and CFA-based MTMM analyses, two versions of the survey questionnaire, representing two methods, were prepared: one was Web based and the other was paper-and-pencil. We used the Web version for the fast food research context and the P&P version for the speed limit research context to test the efficacy of the single-method techniques, which include L&W-MVT and Williams-MVT. We included all of the items associated with the fast food research context in the Web questionnaire, while all of the items associated with the speed limit research context were included in the P&P questionnaire. Each questionnaire also included items used to measure the respective marker variables for each research context.

Meanwhile, the P&P questionnaire for the fast food research context and the Web questionnaire for the speed limit research context were designed exclusively for applying the multiple-method techniques (i.e., traditional MTMM and CFA-based MTMM) and therefore did not contain the marker variables. As such, these specific questionnaires included a randomly selected subset of items as indicated in Appendix A. For the fast food research context, the items specifically measured using the P&P version were expected to be combined with the remaining items measured using the Web version within the context of the traditional MTMM and CFA-based MTMM techniques, thereby ensuring that all items for each construct were utilized. Likewise, for the speed limit research context, the items specifically measured using the Web version were expected to be combined with the remaining items measured using the P&P version within the context of the MTMM and CFA-based MTMM techniques, again ensuring that all items for each construct were utilized.

Data Analysis

Results Without Correcting for CMV

For both research contexts, we began the analysis by applying a traditional CFA on the substantive variables without consideration of method bias. We implemented CFA on the data collected using the Web questionnaire for the fast food research context and on the data collected using the P&P questionnaire for the speed limit research context. The measurement models for both showed satisfactory fit (Hu & Bentler, 1999). For fast food: χ²(125) = 268.13; comparative fit index (CFI) = .97; standardized root mean square residual (SRMR) = .058; root mean square error of approximation (RMSEA) = .09. For speed limit: χ²(142) = 264.64; CFI = .97; SRMR = .064; RMSEA = .079. Overall, we found that the psychometric properties of the scale were satisfactory (Bagozzi & Yi, 1988; Fornell & Larcker, 1981) for both research contexts. The estimated correlations across constructs from the measurement model, as well as estimated path coefficients and explained variance from the structural model, are displayed in the “Uncorrected Estimates” column in Table 1.

OPEN IN VIEWER

Table 1.

Correlations, Structural Paths, and SMC Estimates for Different Methods.

	Uncorrected Estimates	CMV-Corrected Estimates
	CFA Model Without Marker Variable	Lindell and Whitney (2001) Implementation of MVT	Williams et al. (2010) Implementation of MVT–Method C	CFA-Based MTMM Model
A. Fast food patronage (FF) study (rS = rM = .08)
Factor correlations
r(ATT-FF,SN-FF)	.60*	.56*	.60*	.60*
r(ATT-FF,PB-FF)	.62*	.58*	.62*	.55*
r(SN-FF,PB-FF)	.57*	.53*	.57*	.54*
r(INTENT-FF,EXPECT-FF)	.83*	.81*	.84*	.82*
r(ATT-FF,INTENT-FF)	.69*	.66*	.69*	.62*
r(SN-FF,INTENT-FF)	.58*	.54*	.58*	.59*
r(PB-FF,INTENT-FF)	.79*	.77*	.79*	.81*
r(ATT-FF,EXPECT-FF)	.66*	.63*	.66*	.54*
r(SN-FF,EXPECT-FF)	.63*	.59*	.63*	.56*
r(PB-FF,EXPECT-FF)	.82*	.80*	.83*	.83*
Structural paths
β(ATT-FF→INTENT-FF)	.29*	.29*	.25*	.19*
β(SN-FF→INTENT-FF)	.09	.08	.04	.13
β(PB-FF→INTENT-FF)	.56*	.56*	.67*	.63*
β(ATT-FF→EXPECT-FF)	.18*	.19*	.20*	.06
β(SN-FF→EXPECT-FF)	.18*	.17*	.19*	.13*
β(PB-FF→EXPECT-FF)	.61*	.60*	.56*	.72*
Predictive power
SMC (INTENT-FF)	.69	.68	.77	.71
SMC (EXPECT-FF)	.73	.71	.69	.71
B. Speed limit compliance (SL) study (rS = rM = .01)
Factor correlations
r(ATT-SL,SN-SL)	.61*	.60*	.61*	.49*
r(ATT-SL,FB-SL)	.29*	.28*	.28*	.28*
r(ATT-SL,PBC-SL)	.38*	.37*	.38*	.37*
r(SN-SL,FB-SL)	.36*	.35*	.36*	.32*
r(SN-SL,PBC-SL)	.33*	.32*	.32*	.35*
r(FB-SL,PBC-SL)	.62*	.61*	.62*	.61*
r(ATT-SL,INTENT-SL)	.54*	.53*	.53*	.51*
r(SN-SL,INTENT-SL)	.44*	.43*	.44*	.43*
r(PBC-SL,INTENT-SL)	.81*	.80*	.83*	.83*
r(INTENT-SL,FB-SL)	.69*	.68*	.69*	.71*
Structural paths
β(ATT-SL→INTENT-SL)	.23*	.23*	.20*	.20*
β(SN-SL→INTENT-SL)	.07	.07	.08	.08
β(PBC-SL→INTENT-SL)	.70*	.69*	.74*	.73*
β(INTENT-SL→FB-SL)	.69*	.68*	.69*	.71*
Predictive power
SMC (INTENT-SL)	.72	.71	.76	.74
SMC (FB-SL)	.48	.46	.48	.50

Note: ATT = attitude; SN = subjective norm; PBC = perceived behavioral control; INTENT = intentions; EXPECT = behavioral expectations; FB = future behavior; CMV = common method variance; MVT = marker variable technique; SMC = squared multiple correlations; MTMM = multitrait-multimethod. All tests of significance are based on p < .05.

*p < .05.

Results of the Partial Correlation Techniques

To apply L&W-MVT, we executed a model similar to that undertaken for the CFA discussed earlier. The only difference is that in this case, the marker variable (i.e., Exploratory Shopper for the fast food context and Work-Life Balance for the speed limit context) is added into the respective CFA measurement models. The measurement models showed good overall fit. For fast food: χ²(260) = 483.40; CFI = .96; SRMR = .063; RMSEA = .078. For speed limit: χ²(194) = 326.67; CFI = .97; SRMR = .063; RMSEA = .07. To assess CMV, we studied the correlations between the marker variable and the remaining substantive variables in the study. Per L&W-MVT, the smallest of the absolute values of the correlations between the marker variable and all of the other substantive constructs is the estimate of CMV (namely, r_M ) in each research context. We found r_M to be equal to .08 and .01 for the fast food and speed limit research contexts, respectively. Both of these estimates of r_M are not significantly different from zero, thus indicating that the level of CMV that afflicts the data is unlikely to influence the results of the study based on correlation significance. Further, based on these r_M estimates, CMV-corrected correlations were calculated using Equation 1 wherein r_S = r_M in this case for both research contexts. The CMV-corrected correlations are displayed in the column labeled “Lindell and Whitney (2001) Implementation of MVT” in Table 1. To formally compare the original and CMV-corrected correlations, we conducted a series of chi-square difference tests (Bollen, 1989) where each original correlation was replaced with its CMV-corrected correlation value to determine if the substitution significantly deteriorated fit (Δχ²[1] > 3.84, p < .05). Through this analysis, in both research contexts, none of the original uncorrected correlations was significantly different from the CMV-corrected correlations, suggesting that CMV bias estimated by L&W-MVT is not harmful in the context of these studies.

We extended the aforementioned analysis by also estimating CMV-corrected structural path coefficients and explained variance of endogenous variables for both the research contexts. The results are presented in Table 1. The CMV-corrected path estimates and levels of explained variance are close to the original estimates. We used chi-square difference tests to formally compare the path coefficients pre and post CMV correction. These tests showed that none of the CMV-corrected path estimates was significantly different from the original path estimates in both research contexts. Based on the results of the L&W-MVT analyses, we get an initial sense that the level of CMV afflicting our data is minimal.

Next, we implemented Williams-MVT. Again, the marker variables were included in each of the models. The process involved the execution of several different types of measurement models, including the classical CFA model with the marker variable incorporated, the baseline model, the method-C model, the method-U model, and the method-R model. The model fit details and relevant chi-square difference tests are specified in Appendix B for the fast food research context (see Table B.1) and for the speed limit research context (see Table B.2). For both research contexts, the comparison of the baseline model with the method-C model indicated insignificant differences. Specifically, for fast food: Δχ²(1) = .67; for speed limit: Δχ²(1) = .34. Both of these differences are smaller than the .05 chi-square critical value for 1 degree of freedom of 3.84. This indicates that equal method effects due to the marker variable are likely not present in the data for both research contexts. Further, in both contexts, the comparison of the method-C model with the method-U models also did not indicate significant differences. Specifically, for fast food: Δχ²(17) = 17.69 < critical value of 27.58; for speed limit: Δχ² (18) = 15.76 < critical value of 28.86. This indicates that in both research contexts, there is no statistical difference between Lindell and Whitney’s (2001) equal method effects model (i.e., the CMV model) and the unequal method effects model (i.e., the UMV model). ³ As such, we infer that the equal method effects CMV model (as presumed by L&W-MVT) provides a reasonable fit to our data for both research contexts. For both research contexts, the comparison of the method-U model with the method-R model also did not indicate significant differences. Specifically, for fast food: Δχ²(10) = .75 < critical value of 18.30; for speed limit: Δχ²(10) = .33 < critical value of 18.30. This indicates that the effects of the marker variables in the respective research contexts did not significantly bias correlations among the substantive variables. The method-C estimates of variable correlations along with the estimated path coefficients and explained variance are presented in the “Williams et al. (2010) Implementation of MVT-Method C Estimates” column in Table 1. ⁴

Results of the Multiple Method Techniques

In this section, we present the results of the multiple-method techniques for controlling and assessing CMV. We began by implementing the traditional MTMM technique where the extent of CMV is estimated by the difference between the monomethod-heterotrait (MH) correlations and the heteromethod-heterotrait (HH) correlations. For the fast food research context, we find that the average MH correlation was .636, while the average HH correlation was .633. For the speed limit research context, we find that the average MH correlation was .424, while the average HH correlation was .427. Since the magnitudes of the average MH and HH correlations are very close in both research contexts, we can conclude that per the traditional MTMM technique, CMV is not a concern.

In the CFA-based MTMM model, the trait factors were allowed to correlate with one another. Though the items within the same construct across the Web and P&P methods within MTMM were different, the two latent method factors were nevertheless allowed to be correlated with each other (Williams et al., 1989). The results of the CFA-based MTMM model suggest a good fit with the data for both research contexts. For fast food: χ²(106) = 185.16; CFI = .99; SRMR = .044; RMSEA = .072. For speed limit: χ²(122) = 162.72; CFI = .98; SRMR = .065; RMSEA = .055. The CFA-based MTMM model includes two method variables in the model, one for the Web method and the other for the P&P method. Including the method factors in the model corrects for the effects of CMV that might be afflicting the relationships between substantive variables in the model. As such, the correlations measured between substantive constructs in the CFA-based MTMM model are essentially the CMV-corrected correlations. Subsequently, we used these CMV-corrected correlations to implement path analyses across the substantive constructs. The results for correlations and structural paths between substantive variables are presented in the “CFA-Based MTMM Model Estimates” column of Table 1.

The factor relationships and path estimates based on the CFA-based MTMM method are very similar to those of the original uncorrected estimates. We executed chi-square difference tests to examine the differences between each of the original uncorrected correlations and CMV-corrected correlations and paths. The results reveal that in both research contexts, none of the CMV-corrected correlations and path estimates statistically differed from the original uncorrected correlations. Overall, the CFA-based MTMM model results suggest that CMV does not afflict the data in either of our research contexts. In the context of CFA-based MTMM as applied in this empirical study, the average proportions of trait, method, and error variance, respectively, were 62.73%, 8.84%, and 28.42% (for the fast food context) and 68.02%, 6.59%, and 25.37% (for the speed limit context).

Conclusions

Overall, our results based on applying each of the four statistical techniques are in agreement and consistently suggest that CMV does not have a statistically significant influence on our data in both the fast food and speed limit research contexts. ⁵ Furthermore, the application of Williams-MVT did not indicate significant differences between the baseline and method-C models for both research contexts, thus suggesting that the equal effects CMV model should not be a concern in our data. Even the difference between the method-C and method-U models was not significant (for both research contexts), indicating that Lindell and Whitney’s (2001) assumption of an equal method effects CMV model is a reasonable one for our data.

In sum, this empirical study provides evidence that L&W-MVT is a suitable technique to estimate and to correct for the ill effects of CMV on correlations and path estimates between substantive variables in the TPB domain. Malhotra et al. (2006) have also presented similar encouraging results on the efficacy of L&W-MVT, as applied in the narrower technology acceptance domain of MIS research. However, the empirical study in this article was able to present three important implications beyond the ones presented by Malhotra et al. (2006).

First, because application of the L&W-MVT returned very similar results compared to the other methods, our study is the first to present evidence that L&W-MVT’s assumption regarding equal CMV weights across substantive variables (i.e., the CMV model assumption made by Lindell and Whitney 2001) is not a limiting aspect of L&W-MVT (at least in the TPB domain). This is relevant because several CMV researchers have expressed certain reservations in the efficacy of L&W-MVT because of the equal weights assumptions made by L&W-MVT. Second, since Williams-MVT does not assume that CMV can only lead to inflation in trait correlations, our study provides evidence that in the TPB domain, L&W-MVT’s assumption that CMV can only cause inflation also does not seem to be a limiting aspect. Third, we find that assessing CMV at the construct level (as done by applying the L&W-MVT) compared to item level (allowed by Williams-MVT) produces very similar results, which gives us more confidence in the efficacy of the L&W-MVT. Because of undertaking this laboratory study, we now have initial evidence that supports the validity of L&W-MVT compared to that of other CMV measurement techniques, in the TPB domain. Thus, we infer that L&W-MVT is suitable for assessing the effects of CMV in the TPB domain. As such, next, we go on to apply L&W-MVT in a post hoc manner to reassess the results presented in a large sample of published TPB studies.

An Application of the Marker Variable Technique to Published TPB Research

Compared with other methods for assessing CMV, a significant strength of L&W-MVT is that it can be applied in a post hoc fashion and without access to raw data sets or the item-level correlations. Essentially, in this section, we rely on L&W-MVT to assess CMV in published extant research by applying it in a post hoc manner on the substantive construct-level correlations presented in published studies. We report the results of a post hoc analysis of CMV-corrected correlations and path estimates in a large sample of previously published TPB studies. The goal of this reanalysis is to assess the measured impact of CMV on study results to determine if, or the extent to which, CMV could invalidate published findings in the TPB domain.

Lindell and Whitney (2001) propose that a reasonable proxy for CMV is the smallest observed correlation among all of the substantive variables included on a questionnaire, referred to as r_{S 1} henceforth. Lindell and Whitney also propose that the two substantive constructs exhibiting the lowest correlation are the most theoretically unrelated among all pairs of substantive constructs. The use of the smallest correlation as the proxy value for CMV also constrains the analysis such that the lower limit of the CMV-corrected correlations is equal to zero. If the proxy value were larger than the smallest observed correlation, then the sign of at least one of the CMV-corrected correlations would become negative. This violates the assumption that CMV should not affect the sign of the relationship between substantive constructs (Lindell & Whitney, 2001). However, it should also be noted that the r_{S 1} proxy for CMV also includes true score variance between the two theoretically related substantive variables in a study, in addition to CMV. Hence, in interpreting any results based on the r_{S 1} proxy for CMV, we should keep in mind that r_{S 1} serves as an aggressive proxy for CMV since it also represents some true score variance in it. It is noteworthy that r_{S 1} is not an estimate of CMV. Unlike r_M , which is a study specific estimate of CMV ⁶ , r_{S 1} is a post hoc study-specific proxy for CMV. Essentially, r_{S 1} is an aggressive proxy for CMV in that it presents a proxy of the magnitude of the upper limit for the CMV afflicting the substantive variables in the study.

Collection of Published Study Results

Individual studies included in the reanalysis were selected from the following set of journals published from January 1990 to December 2008: Journal of Applied Psychology, Journal of Applied Social Psychology, Journal of Social Psychology, Personality and Social Psychology Bulletin, Journal of Personality and Social Psychology, Journal of the Academy of Marketing Science, Journal of Public Policy and Marketing, Journal of Retailing, Journal of Business Research, Journal of Consumer Psychology, Journal of Consumer Research, Journal of Marketing, Journal of Marketing Research, and Psychology and Marketing. We included studies that tested the original theory of reasoned action (TRA; Fishbein & Ajzen, 1975) or TPB (Ajzen, 1991) frameworks or extensions of them, including the multi-attribute approach to attitude formation (Cohen, Fishbein, & Ahtola, 1972), the theory of trying (Bagozzi & Warshaw, 1990), and the technology acceptance model (Davis, Bagozzi, & Warshaw, 1989). The published studies represent research conducted in a wide variety of behavioral contexts, including product purchases, exercise, employee turnover, and blood donation, among others (Ajzen, 2001; Sheppard, Hartwick, & Warshaw, 1988).

To remain consistent with the assumptions of the marker variable technique, we selected studies if they met the following criteria. First, data for all predictor and criterion variables of interest (i.e., TPB variables) had to be collected from the same respondent in a single administration using a cross-sectional survey. A critical assumption of the marker variable technique is that it should be applied only to cross-sectional data (Lindell & Whitney, 2001). Thus, it should not be applied to studies that include procedural remedies to address CMV, such as longitudinal separation between predictor and criterion variable measures and the use of multiple sources whereby the data for predictor and criterion variables are provided by two separate respondents. In a small number of the studies we examined, data used to measure the key TPB variables (e.g., belief → attitude → intention) were assessed using a single cross-sectional survey, which was followed by the collection of data to measure a single variable (e.g., follow-up measure of actual behavior) at a separate time. In such cases, only those variables measured in the initial cross-sectional survey were included in the reanalysis. In other words, the variables must share a common measurement method (i.e., aspects such as data collected at same time, in same location, and through a single medium). In addition, any variable that was measured using categorical response options (e.g., male/female; yes/no) was not included in the reanalysis and not considered as a potential determinant of the proxy for CMV.

Through a process of individually examining each published article in each journal, we identified 253 studies that met the aforementioned criteria. However, for each study, we also had to consider whether the published article included the complete construct-level correlation matrix. Although post hoc implementation of L&W-MVT does not require having access to the raw data set or item-level correlations, it does require a complete construct-level correlation matrix in order to identify the proxy r_S value (i.e., the smallest correlation or r_{S 1}). Out of the 253 studies that met our sample selection criteria, 104 (41%) did not include a complete published correlation matrix. We then individually contacted each of the corresponding authors to request the correlation matrices. This process led to the receipt of an additional 25 correlation matrices and left us with a final sample of 174 separate studies published in 111 articles. ⁷ Appendix C presents a quantitative summary of the individual study characteristics as well as a breakdown by research discipline, theoretical context, and survey administration technique.

The next step involved selecting the correlations to be included in the reanalysis from each of the studies. We included correlations that represent hypothesized relationships between predictor and criterion variables contained within the TPB framework including attitudes, subjective norms, perceived behavioral control, intentions, and actual behaviors. Over time, TPB models have been expanded to include additional behavioral predictors (e.g., affective responses, self-efficacy, past behaviors), which were considered if modeled as direct predictors of attitudes or intentions (Conner & Armitage, 1998; Sheppard et al., 1988). We identified 755 correlations that represented relationships between TPB variables. Of these 755 correlations, 709 (93.9%) were significant (p < .05), making up the final sample to be included in the reanalysis. We focused only on statistically significant correlations to meet our parallel goals of examining effects on correlation magnitude and reduction to nonsignificance of the originally significant correlation, when applying L&W-MVT in a post hoc fashion.

Reanalysis of Correlations

In the following analysis, we use Equation 1 to calculate a CMV-corrected correlation r_Yi·S corresponding to the original correlation r_Yi after correcting for the study-specific r_{S 1} proxy for CMV. Using Equation 2 we are able to determine if CMV correction affects the significance of the observed relationships between substantive variables by testing whether the CMV-corrected correlations remain significant. In addition to reassessing correlations by partialling out the smallest observed correlation (r_{S 1}), we also partialled out the second smallest observed correlation, referred to as r_{S 2}, as an even more aggressive study-specific proxy for CMV than r_{S 1}. We also conducted a series of sensitivity analyses by varying the proxy values of CMV in order to determine the effects of a range of hypothetical CMV levels on published study results. Table 2 presents a summary of the results of the reanalysis.

OPEN IN VIEWER

Table 2.

Summary of the Reanalysis of Correlation Size and Significance.

Proxy Value of CMV	Average CMV- Corrected Correlation	Percentage of Significant Correlationsa	Average Size Differenceb	Percentage of Size Reductionc
No CMV (rS = 0)	.49	100.0	—	—
Smallest correlation (rS = rS 1 = .09)d	.43	94.9	.05	13.2
Second smallest correlation (rS = rS 2 = .13)	.40	90.0	.08	19.9
rS = .05e	.46	95.3	.02	7.6
rS = .10	.43	91.4	.05	16.0
rS = .15	.40	86.0	.09	25.1
rS = .20	.36	81.8	.12	34.7
rS = .25	.32	75.3	.16	43.8

Note: Summary results of the reanalysis are based on mean values (N = 709). All tests of significance are based on p < .05. Values assume that censoring occurs at 100% reduction. CMV = common method variance.

*p < .05.

^aPercentage of correlations that remained significant post CMV correction.

^bDifference in magnitude between the original and CMV-corrected correlations.

^cPercentage of size reduction in the original correlations post CMV correction.

^dThe average value of the CMV proxy across all studies at the r_{S 1} level was .09 and at the r_{S 2} level was .13.

^eSensitivity analysis values of r_S were held constant across all correlations for each level of the sensitivity tests.

Each study-specific correlation matrix was examined to select both the smallest (r_{S 1}) and second smallest (r_{S 2}) correlations to be used as the proxies for CMV. Across all studies included in the reanalysis, the average size of r_{S 1} was equal to .09 and the average size of r_{S 2} was equal to .13. Along parallel lines, the average size of r_{S 1} was equal to .08 and the average size of r_{S 2} was equal to .11 as reported by Malhotra et al. (2006) in the narrower technology acceptance model (TAM) domain. Essentially, it is relevant to note that the average values for r_{S 1} and r_{S 2} that we find in the TPB domain are close to the average values for r_{S 1} and r_{S 2} found in the TAM domain.

As displayed in Table 2, the average size of the original, uncorrected correlations (r_Yi ) included in the sample was equal to .49. Using the r_{S 1} proxy for CMV, the average size of the original correlations decreased from .49 to .43 following CMV correction, and 5.1% of the correlations became nonsignificant (p ≥ .05). Using the r_{S 2} proxy for CMV, the average correlation magnitude decreased to .40, and 10% of original correlations became nonsignificant. Based on these summary results, we can conclude that in the TPB domain, CMV has minimal influence on variable relationships. This is because only about 5% of the original correlations became nonsignificant post CMV correction when using the aggressive r_{S 1} proxy for CMV, an occurrence that can be attributed to chance alone. When using the more aggressive r_{S 2} proxy for CMV, roughly 10% of original correlations became nonsignificant post CMV correction. Given this, we argue that CMV continues to exert a minimal influence on published study results.

In most survey-based research, once the significance of a correlation is established and used to confirm any relevant hypotheses, regrettably little attention is given to whether that correlation is significant but weak or significant and strong (Wilkinson & the APA Task Force on Statistical Inference, 1999). However, even when considering the impact of CMV on the strength of variable relationships, our results show that the average size of the difference in magnitude between the original and CMV-corrected correlations, using the r_{S 1} proxy for CMV, is roughly equal to .05. This represents a 13.2% reduction in the magnitude of the original correlation. Using the r_{S 2} proxy, the average difference in size between the original and CMV-corrected correlations is roughly equal to .08, which represents a 19.9% reduction.

Although it might appear that CMV could have a slightly higher influence on the strength of relationships, it is important to note that even these aggressive CMV corrections only reduce the magnitudes of the correlations by 13% to 20% rather than the 80% to 100% that some critics have alleged in the past. As highlighted earlier, r_{S 1} and r_{S 2} are aggressive proxies (not estimates) of CMV, as they both include true score variance between two substantive variables in addition to CMV. Overall, the results of the reanalysis of published study results show only minimal effects of CMV on the continued significance of correlations between substantive variables and reduction in correlation magnitude in the TPB domain.

Sensitivity Analysis of the Summary Results

As indicated earlier, we decided to undertake a CMV sensitivity analysis as Lindell and Whitney (2001) have recommended. For our r_{S 1} proxy for CMV of .09, we calculated the 90th percentile value, which turned out to be .25. Hence, we decided to undertake sensitivity analyses from an r_S value of .05 until r_S reached the highest level of .25 at intervals of .05. In summary, we carried out a detailed sensitivity analysis of the summary results by varying the r_S proxy from .05 to .25 in increments of .05 and holding these estimates constant across all studies included in the sample. Note that the results of CMV-corrected correlations were censored such that the size of the CMV correction is limited to 100%; that is, the CMV correction cannot cause the correlation to change direction, which is aligned with the marker variable methodology (Lindell & Whitney, 2001). This is specifically relevant where the more extreme r_S values used in the sensitivity analysis exceeded the magnitudes of the original observed correlations.

Using Equations 1 and 2, we are able to examine the effects of multiple hypothetical values for CMV on our sample of correlations. The detailed results of the sensitivity analysis, including average change in correlation magnitude, are shown in Table 2. When r_S was held constant at the very highly aggressive level of .20 across all the studies, the average size of the CMV-corrected correlations decreased to .36, and 18.2% of the correlations became nonsignificant. Further, when r_S was held constant at the extremely aggressive level of .25 across all the studies, the average size of the CMV-corrected correlations decreased to .32, and 24.7% of the correlations became nonsignificant. Using this extremely aggressive r_S proxy that is very unlikely to be encountered in actual studies, the effects of CMV became more pronounced. Despite that, more than 75% of the original correlations remained significant even using this extremely aggressive proxy for CMV. This outcome should increase our confidence in the validity of findings in published TPB research, in light of the potential effects of CMV.

Modeling the Effects of Study and Correlation Characteristics on CMV

In the following set of analyses, we determine whether the impact of CMV on study results across studies is influenced by the types of relationships assessed, the survey administration technique used, and specific survey design characteristics. To conduct these analyses, we first coded for a variety of characteristics corresponding to the individual studies and correlations included in the sample, many of which are presented in Appendix C. Additionally, we also coded for the relationship type that is represented by each observed correlation because extant literature suggests that CMV effects may vary across types of variable relationships (Cote & Buckley, 1987; Crampton & Wagner, 1994; Podsakoff et al., 2003). This will assist in determining whether specific relationship types may be differentially vulnerable to the effects of CMV.

We used logistic regression to examine the effects of study and correlation characteristics on the likelihood that an originally significant correlation becomes nonsignificant post CMV correction. The binary dependent variable specifies whether the CMV-corrected correlation remains significant or becomes nonsignificant. In the model, we implemented the influence of different relationship types and survey methods as predictors using dummy variable coding. Specifically, the model included six categories corresponding to relationship type including belief → attitude, attitude → intention, subjective norm → intention, perceived behavioral control → intention, affective responses → intention, and intention → behavior. An “others” category, used to capture relationships that could not be classified in any of the aforementioned categories, served as the base category in the coding scheme. To test for the effects of using a specific survey type on the continued significance of CMV-corrected correlations, we also included three categories corresponding to the mode of survey administration: mail, telephone, and electronic (personal survey served as the base category).

In this analysis, we also examined whether the influence of CMV varies on the basis of survey design characteristics that are reflected in the attributes of each study-specific correlation matrix. Pertaining to this, the average correlation size, total number of correlations (reflecting the number of variables measured), and number of significant correlations were considered as predictors in the model. Additionally, we investigated if the effects of CMV would vary based on whether the predictor and criterion variables that determined each correlation matched in terms of being measured using the same number of response options (e.g., 5 points for both; 7 points for both), scaling technique (e.g., bipolar for both; unipolar for both), and labels for scale endpoints (e.g., strongly disagree–strongly agree for both; completely disagree–completely agree for both). Researchers rely on various rating scale formats differing in terms of numbers of response options and labels (Weijters, Cabooter, & Schillewaert, 2010), which could influence the effects of CMV. These three independent variables were implemented as dummy variables based on whether there was a match or not.

Overall, we found that most of the predictor-criterion variable pairs matched in terms of using the same number of response options (86.3%), scaling techniques (78.9%), and labels for scale endpoints (78.5%). However, we are interested in investigating whether, in the cases where there are differences, if this has an impact on the continued significance of the correlation. The logit model included the three key parameters from Equations 1 and 2 as control variables, specifically, r_Yi , r_S , and N. In addition, the amount of variance explained by the original correlation (i.e., r_Yi ²) was included in the model to test for any nonlinear effects of r_Yi on the probability of a correlation becoming nonsignificant. We conducted seven logit analyses corresponding to two distinct proxies for CMV, specifically r_{S 1} and r_{S 2}, and five hypothetical proxy levels used in the sensitivity analysis (.05 to .25 at intervals of .05). The results of these analyses are presented in Table 3.

OPEN IN VIEWER

Table 3.

Logit Analysis of the Likelihood of a CMV-Corrected Correlation Becoming Nonsignificant.

Predictors	Study-Specific CMV Proxy Values		Value of CMV Used for Sensitivity Tests
	rS = rS 1	rS = rS 2	rS = .05	rS = .10	rS = .15	rS = .20	rS = .25
Intercept	4.87	8.27*	25.84*	25.79*	30.25*	62.27*	71.16*
Original correlation (rYi )	−5.67*	−63.63*	−124.14*	−128.49*	−134.98*	−238.35*	−279.31*
CMV proxy (rS )a	37.67*	47.94*	—	—	—	—	—
Sample size (N)	−.01*	−.00*	−.02*	−.01*	−.01*	−.01*	−.01*
Percentage variance explained (rYi 2)	.08	.08	1.09*	1.10*	1.04*	1.90*	2.14*
Matrix characteristicsb
Mean correlation size	3.41	6.30	3.37	1.97	2.10	−6.15	5.24
Number of correlations	.04	.04*	.01	.01	.00	.01	.00
Number of significant  correlations	−.17*	−.17*	−.05*	−.02	−.02	−.04	−.01
Relationship type
Belief → attitude	6.84*	5.86*	2.08	2.74*	1.86	1.97	4.52*
Attitude → intention	3.35	2.15	1.40	.36	1.17	.19	2.01
SN → intention	4.00	2.42	−.13	.85	1.25	–1.25	2.19
PBC → intention	3.41	2.89*	−1.33	.42	1.97	1.55	1.89
Intention → behavior	6.88	5.44	2.31	−1.11	−3.64	1.62	−3.42
Affect → intention	Dc	3.96	–1.35	2.57	.65	−.75	3.66
Survey method
Mail	.20	.13	−3.29*	−.98	−2.13	−3.42	−1.78*
Telephone	−2.53	−4.02	−1.89	−.45	.85	−1.51	−.17
Electronic	D	1.58	D	D	2.74	−3.05	−5.76
Scale characteristics
Scale types match	1.62	.18	.05	.47	–.99	–1.33	−.40
Scale labels match	–.60	.33	.01	.64	.28	–.68	.00
Scale points match	–.79	–1.35	–1.12	–.03	–.47	.36	.97
Model fit
AIC	111.54	127.65	97.56	104.26	103.11	87.97	104.89
BIC	192.13	218.93	178.79	185.49	189.83	173.78	191.61
Log likelihood	−37.77	−453.82	−3.78	–34.13	−32.56	−24.53	−33.44
Number of observations	650	709	674	674	709	709	709

Note: Base category for relationship type coding is “others.” Base category for survey method coding is “personal.” CMV = common method variance; SN = subjective norms; PBC = perceived behavioral control; AIC = Akaike Information Criterion; BIC = Bayesian information criterion.

^aBecause they are held constant across correlations, CMV proxy values are not used as predictors in the sensitivity tests.

^bMatrix characteristics are based on the mean value corresponding to each study-specific correlation matrix.

^cD indicates perfect prediction; corresponding observations were removed before estimation of the likelihood function.

*p < .05.

As expected, across the seven logit analyses, the coefficients corresponding to the size of the original correlation (i.e., r_Yi ) and sample size were negative and significant. This suggests that as the size of the original correlation and the sample size increased, the probability that the original correlation would become nonsignificant post CMV correction decreased. Also as expected, the coefficients for the size of the r_S were positive and significant for both the r_{S 1} and r_{S 2} proxies for CMV. In the sensitivity analysis, the r_S proxy for CMV was held constant across all correlations and therefore was not included in the logit model. However, in each of the sensitivity analyses, we find that the slack left by not including the r_S in the model was filled by the variance explained (i.e., r_Yi ²) term. The coefficients corresponding to the variance explained term (r_Yi ²) were significant across all five sensitivity analyses, which along with the continued significance of r_Yi suggests that the size of the original correlation (i.e., r_Yi ) might have a nonlinear influence on the probability of a correlation becoming nonsignificant.

The coefficients corresponding to the correlation matrix characteristics were not significant across most of the analyses, suggesting that these predictors do not reliably influence the probability of a CMV-corrected correlation becoming nonsignificant. Exceptions included the negative and significant coefficients corresponding to the number of significant correlations in each study-specific correlation matrix when CMV was modeled at the r_{S 1,} r_{S 2}, and .05 levels. This suggests that at lower levels of CMV, as the number of significant correlations reported in a correlation matrix increases, there is a corresponding decrease in the probability that an originally significant correlation becomes nonsignificant after correcting for CMV. Moreover, of the 18 coefficients corresponding to survey administration technique (i.e., three survey methods × seven CMV levels, dropping three due to perfect prediction), only one (5.55%) was significant in the sensitivity analysis, a finding that can be attributed to chance. This indicated that the survey method did not impact the continued significance of correlations post CMV correction. Further, none of the coefficients corresponding to the three measurement scale characteristics were significant.

Finally, of the 41 coefficient estimates corresponding to relationship type (i.e., six relationship types × seven CMV levels, dropping one due to perfect prediction), only five (12.1%) were significant, which can also mostly be attributed to chance. One notable finding was that four out of these five significant coefficients were associated with the belief → attitude relationship type category. However, it is relevant to note that the actual percentage of belief → attitude type correlations that became nonsignificant was consistently lower than corresponding percentages for many of the other relationship type correlations at each of the seven levels (i.e., r_{S 1,} r_{S 2}, and each of the five sensitivity analysis levels). As such, we argue that the significance of the belief → attitude relationships in this analysis can be attributed to chance as well.

Overall, the majority of predictors employed in the logistic regression analyses were found to be nonsignificant in the prediction of an originally significant correlation becoming nonsignificant after adjusting for multiple levels of CMV. As expected though, we did find that sample size was a significant predictor in each of the models. ⁸

Accounting for the Effects of CMV Using Structural Modeling

To further examine the influence of CMV on published study results, we applied L&W-MVT to a reanalysis of parameters used to test structural models based on the TPB framework. We first determined the percentage of originally significant parameters representing paths in a structural model that continue to be significant following CMV correction. We also considered the effects of CMV on relationship strength by examining the average reduction in the magnitudes of path coefficients and squared multiple correlations (SMCs). To begin, we summarize the process used to select the studies that were included in the reanalysis of structural modeling results.

Identification of Published Study Results Based on Structural Modeling

The first step in this reanalysis involved examining each of the 174 studies that were included in the reanalysis of correlation coefficients to determine if the original study employed structural modeling. We selected studies if the authors had tested structural relationships among variables using either traditional path analysis or structural equation modeling techniques (Jöreskog & Sörbom, 1996). In order to examine the influence of CMV on published study results, we first replicated the results as reported in the published studies. To conduct the replications, we used LISREL to test structural models where each of the substantive latent constructs was treated as a single-item variable without measurement error. We were restricted to using single-item measurements because we had access to correlation matrices only at the construct level and not at the item level. Using this approach, we replicated all of the published results corresponding to each study. With only minor exceptions, likely due to the use of different estimation techniques, the replicated path estimates were consistent with the published parameters based on the full measurement model.

Based on the aforementioned criteria, we identified 45 studies, from which we collected 321 path estimates; 213 of these 321 path estimates were significant and were included in the reanalysis. On average, approximately seven individual path estimates were drawn from each study. Across all studies, we chose to focus on how many of the significant path coefficients representing TPB relationships became nonsignificant post CMV correction. We also collected estimates of the SMCs corresponding to each of the endogenous variables contained within the structural models. We identified 149 SMCs, which were used to determine the effects of CMV on explained variance.

Reanalysis of Structural Path Estimates and Explained Variance

To begin the reanalysis of path estimates, each study-specific correlation matrix was examined to select both the smallest (i.e., r_{S 1}) and second smallest (i.e., r_{S 2}) correlations to be, respectively, used as an aggressive and more aggressive proxy for CMV. Across all studies, the average value of r_{S 1} was equal to .08 and the average value of r_{S 2} was equal to .14. Next, we used Equation 1 to correct all of the observed correlations in each study-specific correlation matrix for CMV using the r_{S 1} and r_{S 2} proxies. These full CMV-corrected correlation matrices were used to reproduce the structural modeling results for each study. Specifically, we calculated two CMV-corrected correlation matrices for each study, one using the r_{S 1} CMV proxy and the other using the r_{S 2} CMV proxy. The summary results of the reanalysis are presented in Table 4.

OPEN IN VIEWER

Table 4.

Summary of the Reanalysis of Path Estimates and Explained Variance.

Proxy Value of CMV	Path Estimates (N = 213)			SMCs (N = 149)
	CMV-Corrected Path Coefficient	Percent of Significant Parametersa	Size Differenceb	Percentage Variance Explained
No CMV (rS = 0)	.32	100.0	—	37.4
Study-specific proxy
Smallest correlation  (rS = rS 1 = .08)c	.31	96.7	.02	35.0
Second smallest  correlation (rS = rS 2 = .14)	.29	93.2	.03	33.0
Sensitivity estimates
rS = .10d	.30	92.5	.03	34.2
rS = .20	.27	79.8	.06	30.6

Note: Summary results of the reanalysis are based on mean values. All tests of significance are based on p < .05. CMV = common method variance; SMC = squared multiple correlation.

^aPercentage of path estimates that remain significant post CMV correction.

^bDifference in magnitude between original and CMV-corrected path estimates.

^cThe average values of the r_{S 1} and r_{S 2} proxies for CMV across the 45 studies included in this analysis are .08 and .14, respectively.

^dSensitivity values for r_S are held constant across all coefficients in the sensitivity tests.

*p < .05.

Using r_{S 1} as the proxy for CMV, the average difference in magnitude between the original and CMV-corrected path estimates was equal to .02 and approximately 97% of the 213 published path estimates remained significant (p < .05). The average explained variance percentage across endogenous variables decreased from approximately 37.4% to 35%. All of these decreases are minimal. Using r_{S 2} as the proxy for CMV, the average difference in magnitude between the original and CMV-corrected path estimates was equal to .03 and approximately 93.2% of the published path estimates remained significant (p < .05). The average explained variance percentage decreased from 37.4% to 33%. When the r_{S 2} proxy for CMV was used, we find that less than 7% of the CMV-corrected path estimates were reduced to statistical nonsignificance (refer to Table 4) compared with 10% of the CMV-corrected correlation coefficients (refer to Table 2). These results are particularly relevant for researchers who tend to focus more on the significance of hypothesized structural paths and less on the correlations from which these estimates are derived. After comparing the findings from the reanalysis of path estimates to the findings from the reanalysis of correlations, we note that path estimates are even more resistant to the effects of CMV compared with the correlation coefficients from which they are derived. This finding is consistent with past research that arrives at a similar conclusion (Siemsen, Roth, & Oliveira, 2010).

Sensitivity Analysis of the Summary Results

As in the reanalysis of correlations, the results are constrained such that the size of the CMV correction is limited to 100% (Lindell & Whitney, 2001). Given the large number of structural equation models involved, we limited our sensitivity analysis of the effects of CMV on structural modeling results by fixing the r_S value at levels of .10 and .20 across studies. Approximately 92.5% of the 213 originally significant path estimates remained significant (p < .05) when the r_S estimate was equal to .10, while 79.8% remained significant (p < .05) when r_S was held at an extremely aggressive level of .20 across all studies. Concerning the effects of CMV on percentage of explained variance, we found on average a decrease from the original percentage value of 37.4% to 34.2% when r_S = .10 and to 30.6% when r_S = .20.

Researchers tend to base their substantive conclusions more often on the magnitude and significance of path coefficients and not on individual correlations. Hence, we argue that our conclusions on the impact (or lack thereof) of CMV on path coefficients and their continued significance are even more telling than our conclusions based on the sustained significance of correlations. Given this, we infer that CMV has a minimal impact on the validity of findings presented in extant TPB research and should not be a concern.

Discussion

Researchers do not have a clear idea on whether, or to what extent, published research in the broad TPB domain is affected by CMV. To that end, this research has two main goals. The first is to empirically compare several different techniques for estimating and correcting for CMV. The results of this study provide support that the assumptions made by L&W-MVT do not limit the efficacy of this technique as applied to TPB research. The second goal of this research is to quantitatively examine the influence of CMV in published studies based on the TPB domain. After having shown that the assumptions made by L&W-MVT are not a limitation within the TPB domain, we rely on L&W-MVT in a post hoc manner in the form of a reanalysis of published study results. Specifically, we examine the effects of CMV on the reduction of originally significant correlations and path estimates to nonsignificance post CMV correction. We also examine the effects of CMV on the percentage reduction in the magnitude of correlations. For each set of analyses, we test the sensitivity of the results using a wide range of aggressive estimates of the level of CMV. It should be noted that even the smallest (i.e., r_{S 1}) and the second smallest (i.e., r_{S 2}) correlations are aggressive proxies for CMV due to their inclusion of true score variance between substantive variables in a study in addition to CMV.

Overall, the results of our reanalysis suggest that in the TPB domain, published results are resistant to CMV bias in the context of continued significance of correlations between substantive variables and percentage reduction in correlation due to CMV. Applying the same caveat of the aggressive nature of the r_{S 1} and r_{S 2} proxies used, the reanalysis of structural modeling results suggests that structural path estimates in the TPB domain appear even more resistant to the effects of CMV in comparison to the correlations from which they are derived. This finding is particularly relevant since researchers generally base their substantive conclusions on the magnitude and significance of path coefficients and not on individual correlations. Thus, the comparative impact (or lack thereof) of CMV on structural path coefficients is an important finding and provides further support for the conclusion that CMV is not a concern in TPB research.

We acknowledge that our findings, which support that CMV is not a concern in the TPB domain, are likely at odds with the findings reported by Cote and Buckley (1987) and Williams et al. (1989). We sense that there are several underlying reasons for the difference between the inferences drawn from our reanalysis and those presented in previous reanalyses. We speculate that the first reason for these differences is that unlike Cote and Buckley and Williams et al., our reanalysis focused on comprehensively assessing relationships among constructs implemented within studies (in the TPB domain) published in a large number of quality journals over a fairly large interval of time. In contrast, of the 70 studies used in Cote and Buckley’s reanalysis, only 11 were focused specifically on the TPB/attitudes domain. Further, the studies used in the Williams et al. reanalysis were broadly based on Spector’s (1987) sample of studies. Spector’s study focused on a very specific research context, specifically, affect and perceptions at work in organizations, and not on the TPB domain in a broad sense. Second, the set of studies included in our reanalysis are different from those reanalyzed by Cote and Buckley, Spector, and Williams et al., given that they were published in different time periods. The studies included in our reanalysis are substantially more recently published than those included in the past reanalyses. Third, the criteria for selection of studies in the various reanalyses are different. For example, the past reanalyses, undertaken by Cote and Buckley and Williams et al., specifically selected only those studies that had originally implemented the MTMM method. In contrast, the criteria for selection of studies in our reanalysis were substantially more inclusive in the sense that we used all TPB studies published in 14 journals over 18 years, as long as the study had presented the correlation matrix between the substantive variables (or we were able to obtain one from the authors).

We would also like to highlight that even the MTMM procedure, which was heavily relied on in some of the earlier works that assessed CMV, is not without its problems. First, when same-source correlations are compared to multisource correlations, one cannot assume that the multisource correlations are the correct ones and the same-source are inflated. Multisource (compared to same-source) correlations included in these analyses are known to be quite inaccurate and thus cannot be assumed to be correct (Frese & Zapf, 1988). Second, the CFA analyses of the block diagonal model from MTMMs have certain computational problems. These analyses have been shown to have in many cases severe estimation problems that involve estimates of correlations much greater than 1.0 and variances well below zero. ⁹ Extant research (e.g., Brannick & Spector, 1990) highlights many such issues.

Given these issues with the different methods used to assess CMV in these other reanalyses versus in our reanalysis, the differences in our results vis-à-vis their results are not entirely surprising. Future research might focus on understanding these differences and analyzing if, and to what extent, they are attributable to each of the underlying reasons.

As with any research, there are limitations that should be considered. One concern with the marker variable technique has to do with the susceptibility of correlations to measurement error based on scale reliability. When using L&W-MVT, correlations disattenuated for measurement error (Spearman, 1904) can be substituted for the original correlation (i.e., r_Yi ) and the CMV estimate (i.e., r_M ) in Equations 1 and 2 (Lindell & Whitney, 2001). When the marker variable technique is used in a post hoc manner for reanalysis of published study results, as is the case here, calculating a disattenuated correlation requires that scale reliabilities be reported for all variables used to determine the original uncorrected correlations (i.e., r_Yi ) as well as the proxy marker variable correlations (i.e., r_{S 1} and r_{S 2}). Unfortunately, in many of the studies that were included in our reanalysis, scale reliabilities were not reported for all variables. In fact, of the 709 originally significant correlations that were adjusted for CMV, 339 (47.8%) were determined by either predictor or criterion variables (or both) that did not have a reported reliability level. In the interest of preserving our large sample size, we did not apply the disattenuation correction as it would have reduced the number of correlations by half. However, we encourage all those researchers who are applying the L&W-MVT in an a priori manner to use disattenuated correlations in Equations 1 and 2. Further, we strongly suggest that researchers report scale reliabilities as well as correlation matrices including all substantive variables in their published studies, as this will allow for more complete meta-analytic investigations of past research. Accordingly, we also urge researchers to routinely include CMV corrections along with corrections for unreliability and variance attenuation when undertaking meta-analyses. ¹⁰

It has been argued in extant literature that the equal weight assumption (i.e., equal effects of CMV across all measured variable relationships) made by L&W-MVT limits its ability to provide an accurate estimation of CMV (Podsakoff et al., 2003; Richardson et al., 2009). However, as shown earlier in our empirical study, the equal weights assumption made by L&W-MVT did not affect the comparability of the L&W-MVT results with the results of other CMV tests, thus suggesting that this assumption might not be unreasonable. This finding is consistent with the substantial amount of theoretical and empirical evidence that is accumulating in support of the predictive validity of the equal weights model (e.g., Bobko, Roth, & Buster, 2007). The same is also true with the L&W-MVT assumption that CMV can only inflate trait correlations. We acknowledge, however, that these inferences may not be tenable in the context of other research domains. Despite the high efficacy of L&W-MVT in the TPB domain as evidenced in our empirical comparison study, we nevertheless urge researchers who are methodologically oriented to also implement Williams-MVT in their analyses to gain further insights that are uniquely provided by Williams-MVT alone.

Overall, this research makes multiple contributions, including (1) empirically comparing multiple methods to illustrate the similarity of conclusions made about the occurrence of CMV across methods, (2) organizing the substantial body of TPB research, (3) aggressively testing the susceptibility of published TPB findings to the effects of CMV, (4) modeling the effects of CMV using regression analyses to test for potential effects of study characteristics, and (5) extending L&W-MVT to an analysis of CMV on structural paths to provide further evidence that the effects of CMV do not invalidate published TPB research. Although researchers are always encouraged to approach study results with healthy skepticism, we hope that this research takes the nascent steps toward putting to rest concerns about the adverse influence of CMV in the TPB domain.