A Monte Carlo Study of the Effects of Common Method Variance on Significance Testing and Parameter Bias in Hierarchical Linear Modeling

Abstract

Despite that common method variance (CMV) is widely regarded as a serious threat to the validity of findings based on self-reports, there is insufficient research on its confounding influence. We extend Evans’s (1985) pioneering work, and the more recent works by Ostroff, Kinicki, and Clark (2002) and Siemsen, Roth, and Oliveira (2010), to delineate the influence of CMV in a two-level hierarchical linear model based on self-report data. Our simulation results clearly show that in the absence of true effects, it is extremely unlikely for CMV to generate significant cross-level interactions. In fact, if a true cross-level interaction exists, CMV tends to lower the likelihood of its identification and erroneously underestimate the regression coefficient. Our simulation results also show that CMV may lead to a false significant cross-level main effect and overestimate the regression coefficient when no true effect exists. To reduce the probability of Type I errors, we show that raising the significance level to .01, the split sample strategy, and the addition of more CMV contaminated variables are effective in the vast majority of real-life situations and are more effective than increasing the number of groups or persons in each group. Both the split sample strategy and the addition of more CMV contaminated variables are also effective in reducing parameter bias when no true cross-level main effect exists. Trade-offs associated with different strategies are discussed.

Keywords

common method variance self-report data hierarchical linear modeling cross-level relationships Monte Carlo approach

The confounding influence of common method variance (CMV) has attracted a great deal of attention in organizational research (e.g., Brannick, Chan, Conway, Lance, & Spector, 2010; Conway, 2002; Podsakoff, MacKenzie, Lee, & Podsakoff, 2003; Spector, 2006; Williams & Anderson, 1994). Defined as the shared variance among variables due to the use of a common measurement method (Spector & Brannick, 2009), CMV may inflate the observed relationships between variables (Spector, 2006). Pace (2010) concluded that “CMV may be the most common and dangerous threat to correct interpretation of research results or the most indiscriminately used reviewer critique leading to rejection of good manuscripts” (p. 421). Podsakoff et al. (2003) identified four potential sources of CMV, namely, self-report data, item characteristics, item context, and measurement context, but concerns about CMV are “almost exclusively limited to the use of self-report surveys” (Spector & Brannick, 2010, p. 403). There is consensus that the use of self-report measures threatens the validity of findings, and many journal reviewers indiscriminately reject papers that are based on self-report data. For instance, if respondents rate their leaders on their leadership style and leader effectiveness, they may try to maintain consistency between these two types of ratings, thus artificially inflating the correlation involved (Antonakis, Bendahan, Jacquart, & Lalive, 2010).

Two common strategies to circumvent the problem of self-report data include the collection of data from multiple sources and temporal spacing of variables (see a review by Podsakoff, MacKenzie, & Podsakoff, 2012). However, the use of multiple sources is not always applicable because many important variables, such as personality, attitudes, cognitions, and values, are primarily measured by self-reports and hard to be observed by others (Podsakoff et al., 2012). These variables often occupy key roles in theoretical models, functioning as independent variables, mediators, moderators, and dependent variables. If a moratorium is put on investigating percept–percept relationships, many conceptual models will be significantly impoverished because of the exclusion of intra-individual processes. The remedy of temporal spacing of measures may not be feasible in some research contexts, and the time gap involved may introduce other types of biases, such as the nonequivalence of contextual characteristics across the two waves of measurement. A more productive approach is to identify conditions in which CMV is problematic and develop strategies to minimize its influence (see reviews by Podsakoff et al., 2003, 2012; Richardson, Simmering, & Sturman, 2009). Despite the fact that CMV has received attention for several decades, empirical investigation on the extent to which CMV is a problem is limited (for exceptions, see Cote & Buckley, 1988; Evans, 1985; Siemsen, Roth, & Oliveira, 2010; Williams & Brown, 1994). Pioneering this line of research, a Monte Carlo study by Evans (1985) and a statistical proof by Siemsen et al. (2010) showed that an interaction effect cannot be produced by CMV, but bivariate linear relationships could be either inflated or deflated by CMV.

Most empirical work concerning the influence of CMV is confined to single-level designs, typically at the individual level (for an exception, see Ostroff, Kinicki, & Clark, 2002). However, multilevel perspectives on organizational phenomena are now commonplace (e.g., Herold, Fedor, Caldwell, & Liu, 2008; Hofmann, Griffin, & Gavin, 2000; Kirkman, Chen, Farh, Chen, & Lowe, 2009; Klein & Kozlowski, 2000). The use of self-report measures is very common in multilevel research, namely, measures at different levels are collected from the same respondents (Ostroff et al., 2002). For this type of design, there is the concern about the confounding influence of CMV on cross-level interactions, namely, the moderating effects of higher level variables on the relationships between lower lever variables (e.g., Erdogan & Enders, 2007; Henderson, Wayne, Shore, Bommer, & Tetrick, 2008; Hofmann, Morgeson, & Gerras, 2003; Kirkman et al., 2009; Scott & Barnes, 2011; Walumbwa, Hartnell, & Oke, 2010), and on cross-level main effects, namely, the influence of higher level variables on lower level variables (e.g., Herold et al., 2008; Liao & Rupp, 2005). However, there is limited research that explicitly investigates the extent to which CMV is a problem in multilevel designs.

When higher level variables are formed by aggregating individual-level responses, it is intuitive that their effects may be less susceptible to the influence of CMV (e.g., Kirkman et al., 2009; Liao & Rupp, 2005). Some researchers doubt that CMV is likely to give rise to significant cross-level moderation effects (e.g., Henderson et al., 2008; Hofmann et al., 2003; Kirkman et al., 2009; Scott & Barnes, 2011; Walumbwa et al., 2010). The single-level results of Evans (1985) are sometimes used to defend the conclusion that cross-level interactions cannot be generated by CMV. For example, Erdogan and Enders (2007) defended their cross-level interaction effects based on several self-report variables by citing Evans (1985). Nonetheless, to the best of our knowledge, there is no compelling evidence to support the relevance of Evans’s (1985) single-level conclusions for cross-level interaction effects. Hence, the first objective of the current research is to extend previous single-level findings to the multilevel context by addressing whether and when CMV inflates cross-level interaction effects.

Equally important, prior research has demonstrated the confounding influence of CMV on cross-level main effects. Schulte, Ostroff, and Kinicki (2006) discussed the possibility of the confounding influence of CMV with regard to their cross-level results. More specifically, in a study of the relationship between climate and satisfaction, Ostroff et al. (2002) compared cross-level correlations with cross-level split correlations, with half of the sample randomly selected to provide the overall climate score and the other half, the overall satisfaction score. There was clear evidence for the inflationary effect of common method bias, as the cross-level correlations were larger than the cross-level split correlations, leading to the conclusion that “researchers need to be concerned about the presence of method variance when conducting cross-level and multilevel studies” (Ostroff et al., 2002, p. 366).

Note that there is a methodological concern in the method employed by Ostroff et al. (2002), namely, the assignment of group-level scores to individuals for computing the cross-level correlations violated the independence assumption. Individuals in the same group “are exposed to similar stimuli within the group” (Hofmann, 1997, p. 725), which should be taken into account for drawing valid conclusions. Hierarchical linear modeling (HLM), the multilevel analogue of ordinary regression, is a more appropriate analytic technique to evaluate whether CMV plays a role in multilevel contexts. HLM is perhaps the most popular analytical tool in multilevel organizational research (e.g., Klein, Dansereau, & Hall, 1994; Klein & Kozlowski, 2000; Morgeson & Hofmann, 1999) because of its ability to “simultaneously investigate relationships within a particular hierarchical level, as well as relationships between or across hierarchical levels” (Hofmann, 1997, p. 726). Given that there is no systematic research on how CMV influences cross-level main effects under different circumstances, the second objective of this research is to augment the works of Evans (1985), Siemsen et al. (2010), and Ostroff et al. (2002) by evaluating the spurious influence of CMV on cross-level main effects with an HLM analysis as well as investigating the conditions in which CMV is problematic. Finally, to deal with the confounding influence of CMV in multilevel designs, the third purpose is to identify feasible statistical remedies to mitigate the effects of CMV and avoid erroneous inferences.

In sum, this study extends Evans’s (1985) pioneering work and the more recent works of Ostroff et al. (2002) and Siemsen et al. (2010) to evaluate the extent to which CMV associated with self-report data is a problem as well as identify feasible statistical remedies to control for CMV in a two-level hierarchical linear model based on the Monte Carlo approach. We begin by introducing CMV in the multilevel context and specifying the parameters and their values for the simulation study. We then present simulation results derived from a systematic assessment of the influence of CMV on cross-level interaction and main effects across different parameter values. Finally, we offer conclusions about the effects of CMV on cross-level relationships and provide recommendations to mitigate these effects to avoid erroneous inferences.

Hierarchical Linear Model and CMV

To test the influence of CMV, we propose a two-level hierarchical linear model with an individual-level predictor $X$ , a group-level predictor $Z$ , and an individual-level outcome $Y$ . In this two-level framework, we define CMV as the error components of individual- and group-level predictors that are correlated with an individual-level outcome due to a common method bias, namely, the use of a self-report measurement method. In our multilevel context, the CMV source is assumed at the individual level. We define common method bias in the multilevel context as the extent that cross-level relationships become higher or lower than their true values due to the influence of CMV. Based on the results of significance testing, a Type I error occurs when one erroneously concludes that a cross-level effect exists, and a Type II error occurs when one erroneously concludes that no cross-level effect exists.

We assume that self-report data at the two levels are collected from the same respondents or raters. The group-level predictor is based on the aggregation of an individual-level variable ( $Z_{i n d i v i d u a l}$ ), which is different from the individual-level predictor $X$ . This two-level model resembles the single-level model examined by Evans (1985), and the major difference is that in our model one predictor is at the group level, whereas in his model, the two predictors are at the same level. It is outside the scope of the article to address the situation where there is an individual-level predictor and a group-level predictor, which is formed by aggregating the individual-level predictor.

Following Bryk and Raudenbush (1992), the two-level hierarchical linear model examined is as follows:

\begin{aligned} L e v e l 1 : Y_{i j} = β_{0 j} + β_{1 j} X_{i j} + e_{i j} \\ L e v e l 2 : β_{0 j} = γ_{00} + γ_{01} Z_{j} + u_{0 j}, \\ β_{1 j} = γ_{10} + γ_{11} Z_{j} + u_{1 j} \end{aligned}

where $i$ and $j$ denote the individual index and group index, respectively; $β_{0 j}$ and $β_{1 j}$ denote the individual-level intercepts and slopes in group $j$ , respectively; $γ_{00}$ and $γ_{10}$ are the group-level intercepts; $γ_{01}$ and $γ_{11}$ are the group-level slopes relating $Z_{j}$ to the individual-level intercept and slope terms; $e_{i j}$ is the Level 1 residual; $u_{0}$ and $u_{1}$ represent residuals of the group-level intercept and slope, which are assumed to follow the normal distribution $N (0, τ_{00})$ and $N (0, τ_{11})$ , respectively. The model can be rewritten as:

Y_{i j} = γ_{00} + γ_{01} Z_{j} + γ_{10} X_{i j} + γ_{11} Z_{j} X_{i j} + u_{0 j} + u_{1 j} X_{i j} + e_{i j} .

This model provides a direct test for a cross-level interaction effect by regressing the individual-level slopes on the group-level variable $Z$ . For the case of a cross-level main effect, which is evaluated by regressing the individual-level intercepts on $Z$ , we remove the cross-level interaction in Equation 1, which results in the following main effect model:

\begin{aligned} Y_{i j} = β_{0 j} + β_{1 j} X_{i j} + e_{i j} \\ β_{0 j} = γ_{00} + γ_{01} Z_{j} + u_{0 j} \\ β_{1 j} = γ_{10} + u_{1 j} \end{aligned}

The aforementioned model can be rewritten as:

Y_{i j} = γ_{00} + γ_{01} Z_{j} + γ_{10} X_{i j} + u_{0 j} + u_{1 j} X_{i j} + e_{i j}

HLM provides t tests to determine whether the parameter estimates (i.e., the individual- and group-level regression parameters, $γ_{00}$ , $γ_{01}$ , $γ_{10}$ , and $γ_{11}$ ) are significantly different from zero (Hofmann, 1997). In Equation 2, a t test for the $γ_{11}$ parameter is used to examine whether a group-level predictor $Z_{}$ moderates the relationship between $X$ and $Y$ (a cross-level moderating effect). In Equation 4, a t test for the $γ_{01}$ parameter is used to assess whether the relationship between $Z_{}$ and $Y$ is significantly different from zero (a cross-level main effect). In addition, χ² tests are used to evaluate whether the group-level residual variance (e.g., variance in $u_{0 j}$ and $u_{1 j}$ : $τ_{00}$ and $τ_{11}$ ) significantly departs from zero. As described by Hofmann and Gavin (1998), a major feature that distinguishes HLM from an ordinary regression model is that HLM not only allows the individual-level parameters to vary across groups but also estimates the group-level variance components.

According to Evans (1985), the observed scores on each variable are consisted of true and error components, and the error term can be partitioned into a correlated component and a random component (Campbell, 1976):

\begin{aligned} Y = Y_{t} + e_{Y C} + e_{Y R} \\ X = X_{t} + e_{X C} + e_{X R} \\ Z = Z_{t} + e_{Z C} + e_{Z R}, \end{aligned}

where the subscript $t$ denotes the true score, which is often unobservable, and $e_{. C}$ and $e_{. R}$ represent the correlated error and random error, respectively. Substituting the partitioned scores specified in Equation 5 into Equation 2, we obtain:

\begin{aligned} Y = Y_{t} + e_{Y C} + e_{Y R} = γ_{00} + γ_{01} (Z_{t} + e_{Z C} + e_{Z R}) + γ_{10} (X_{t} + e_{X C} + e_{X R}) \\ + γ_{11} (Z_{t} + e_{Z C} + e_{Z R}) (X_{t} + e_{X C} + e_{X R}) + u_{0} + u_{1} (X_{t} + e_{X C} + e_{X R}) + e \\ = γ_{00} + γ_{01} (Z_{t} + e_{Z C} + e_{Z R}) + γ_{10} (X_{t} + e_{X C} + e_{X R}) + γ_{11} (Z_{t} X_{t} + Z_{t} e_{X C} + Z_{t} e_{X R} + X_{t} e_{Z C} \\ + e_{Z C} e_{X C} + e_{Z C} e_{X R} + X_{t} e_{Z R} + e_{Z R} e_{X C} + e_{Z R} e_{X R}) + u_{0} + u_{1} (X_{t} + e_{X C} + e_{X R}) + e . \end{aligned}

The terms involving $e_{Y R}$ , $e_{Z R}$ , $e_{X R}$ , and $e$ have a mean of zero, and due to the independent nature of these error terms, all of the terms involving random errors can be combined and denoted by a single random error $e^{*}$ . The model can be rewritten as:

\begin{aligned} Y_{t} + e_{Y C} = γ_{00} + γ_{01} (Z_{t} + e_{Z C}) + γ_{10} (X_{t} + e_{X C}) + γ_{11} (Z_{t} X_{t} + Z_{t} e_{X C} + X_{t} e_{Z C} + e_{Z C} e_{X C}) \\ + u_{0} + u_{1} (X_{t} + e_{X C}) + e^{*} \end{aligned}

where

e^{*} = (γ_{10} + γ_{11} Z_{t} + γ_{11} e_{Z C} + u_{1}) e_{X R} + (γ_{01} + γ_{11} X_{t} + γ_{11} e_{X C}) e_{Z R} + γ_{11} e_{Z R} e_{X R} - e_{Y R} + e .

To assess the impact of CMV on a cross-level main effect, we introduce the observed scores as given in Equation 5 into Equation 4 and simplify the equation by combining the random error terms $e_{. R}$ with $e$ and obtain the following equation:

Y_{t} + e_{Y C} = γ_{00} + γ_{01} (Z_{t} + e_{Z C}) + γ_{10} (X_{t} + e_{X C}) + u_{0} + u_{1} (X_{t} + e_{X C}) + e^{*},

where

e^{*} = γ_{01} e_{Z R} + γ_{10} e_{X R} + u_{1} e_{X R} - e_{Y R} + e .

It can be seen from Equation 7 and Equation 8 that the random error $e^{*}$ is a function of $e_{. R}$ and that $e^{*}$ can be used to describe the effects of $e_{. R}$ . Even though the $e^{*}$ term may have some impact on the estimation, the evaluation of the influence of CMV is our major focus. Following Evans’s (1985) approach, we excluded $e_{. R}$ in the data generation step, and only the true scores and the correlated errors $e_{. C}$ were taken into account.

Choice of Parameters and Setting of Values

In analyzing the influence of CMV, Evans (1985) included four parameters in his simulation study: the size of the relationships between the outcome and the two predictors, whether the interaction effect truly exists, the proportion of CMV to true variance, and the size of the relationship between CMV in the outcome and that in the two predictors (see p. 309). Hence, we adapt these four parameters for our two-level model: (1) the size of the true relationship between $X$ and $Y$ ; (2) the size of the true relationship between $X Z$ (the product term of $X$ and $Z$ ) and $Y$ , which represents the cross-level interaction; (3) the size of the true relationship between $Z$ and $Y$ , which represents the cross-level main effect; (4) the ratio of true variance to CMV; and (5) the size of the relationship between CMV in $Y$ and that in the two independent variables ( $r_{e_{Y C} e_{X C}}$ and $r_{e_{Y C} e_{Z C}}$ ).

We derive two equations to simultaneously estimate all the parameters in our model (see Equation 7 for cross-level interaction and Equation 8 for cross-level main effect). Both Equation 7 and Equation 8 contain a product term $u_{1}$ $e_{X C}$ , where $u_{1}$ is the residual of Level 1 slopes, indicating that the left side of both equations ( $Y_{t} + e_{Y C}$ ) is a function of $u_{1}$ . Hence, the magnitude of $u_{1}$ is related to the error in $Y$ ( $e_{Y C}$ ) and may influence the parameter estimates. In HLM, the magnitude of $u_{1}$ is often depicted by the variance of the Level 1 slopes ( $τ_{11}$ ), which reflects the variation of the relationship between $X$ and $Y$ across groups (Hofmann, 1997). As $u_{1} e_{X C}$ is included in both Equation 7 and Equation 8, $τ_{11}$ can affect the results of t tests by influencing the estimate of the standard error in the denominator. Ignoring the variance of Level 1 slopes may result in erroneous inferences, and we include it as the sixth parameter. Finally, because the variance of the group-level predictor $Z$ would affect the statistical power for detecting significant effects (cf. Berk, 2004), it is added as the seventh parameter.

With regard to setting the values of the parameters, we follow Evans (1985) to set the true relationships of the predictors ( $X$ , $Z$ , and $X Z$ ) with the outcome ( $Y$ ) at two levels (.3 and .7) to represent weak and strong relationships, respectively. In line with Evans (1985) and Richardson et al. (2009), we include six ratios of true variance to CMV: 1:0, 10:1, 4:1, 2:1, 1:1, and 1:1.5, with 1:0 representing no CMV. As implied in Evans (1985) and Siemsen et al. (2010), the method effect for a given variable is assumed to be the same for each individual. In addition, we add the 1.53:1 ratio to represent approximately the common threshold for acceptable reliability, namely, .70, resulting in seven ratios.¹ Note that the ratios of 1:1 and 1:1.5 represent extreme and infrequent situations in which CMV is as much as or more than true variance.

With regard to the correlation of the CMV of $X$ with that of $Y$ ( $r_{e_{Y C} e_{X C}}$ ), we set two levels, .3 and .7, to represent low and high levels, respectively. We also include these two levels for the correlation between the CMV of $Z_{i n d i v i d u a l}$ and that of $Y$ . Because the group-level predictor is based on the aggregation of individual-level responses $Z_{i n d i v i d u a l}$ , we use the two levels of .7 and .3 divided by the square root of the number of persons per group ( $.7 / \sqrt{\begin{matrix} g r o u p & s i z e \end{matrix}}$ and $.3 / \sqrt{\begin{matrix} g r o u p & s i z e \end{matrix}}$ ) to represent the high and low correlation between the CMV of $Z$ and that of the outcome ( $r_{e_{Y C} e_{Z C}}$ ), respectively.² We believe that the situations represented by the value of .7 should be infrequent as it involves a very large overlap between the CMV of the predictors and that of the outcome.

In our review of the HLM studies in the organizational literature, only a handful of studies published in leading journals report the size of the variance of slopes ( $τ_{11}$ ) (e.g., Hofmann et al., 2003; Liao & Subramony, 2008). This literature suggests that the estimate for residual variance in slopes generally falls into three intervals: <.1, .1 to .3, and >.5. We thus set three levels to denote low, moderate, and high variance, namely, .1, .3, and .5, respectively.

With regard to the variance of group-level predictors ( $V a r (Z)$ ),³ we identify 23 values in articles published in leading journals, such as Journal of Applied Psychology and Academy of Management Journal, between 2003 and 2010. Based on these values, we set .5 and 1.5 as the low and upper bounds of the 95% confidence interval, denoting a small and large variance of a group-level predictor, respectively. For a schematic representation of the parameter values, see Table 1.

Table 1.

Parameters and Their Values for Different Models.

	Parameter and Setting of Value
Model	Correlation Between $X$ and Y and $Z$ and Y	Correlation Between XZ and Y	Ratio of True Variance to CMV	$τ_{11}$	( $r_{e_{Y C} e_{X C}}$ , $r_{e_{Y C} e_{Z C}}$ )	$V a r (Z)$
No cross-level interaction	(.3, .3) and (.7, .7)	0
Existence of a cross-level interaction	(.3, .3) and (.7, .7)	.3 and .7	1:0, 10:1, 4:1, 2:1, 1.53:1, 1:1, and 1:1.5	.1, .3, and .5	(.7, $.7 / \sqrt{\begin{matrix} g r o u p s i z e \end{matrix}}$ ) and (.3, $.3 / \sqrt{\begin{matrix} g r o u p s i z e \end{matrix}}$ )	.5 and 1.5
No cross-level main effect	(.3, 0) and (.7, 0)	0
Existence of a cross-level main effect	(.3, .3), (.7, .3), (.3, .7), and (.7, .7)	0

Note: For ratio of true variance to CMV, $τ_{11}$ , ( $r_{e_{Y C} e_{X C}}$ , $r_{e_{Y C} e_{Z C}}$ ), and $V a r (Z)$ , the same set of values were used for different models. CMV = common method variance.

In summary, the current research investigates whether and when CMV inflates cross-level interaction and main effects across systematic variations of the seven parameters in a simulation study. We also offer suggestions to mitigate the confounding influence of CMV and avoid erroneous inferences based on the simulation results.

Method

We used a simulation program, S-Plus version 6.2, Professional Edition, for constructing multiple data sets to evaluate the impact of CMV in the HLM context. In the simulation, there were two levels of the relationship between a predictor and the outcome, resulting in six scenarios (two levels for $X$ and $Y$ , two for $Z$ and $Y$ , and two for $X Z$ and $Y$ ). For each scenario, there were seven levels of the ratio of true variance to CMV, three levels of the variance of Level 1 slopes ( $τ_{11}$ ), two levels of the correlation between CMV in the outcome and that in the two predictors ( $r_{e_{Y C} e_{X C}}$ and $r_{e_{Y C} e_{Z C}}$ ), and two levels of the variance of the group-level predictor ( $V a r (Z)$ ), resulting in 504 possible scenarios (6 × 7 × 3 × 2 × 2). These 504 scenarios were assessed for cross-level interaction and main effects separately, leading to 1,008 scenarios. To reduce the concern about sampling errors, we followed the approach of Richardson et al. (2009) to replicate each of the scenarios 100 times, resulting in a total of 100,800 data sets. Following Evans (1985), we focused on the simulation results for four different types of models, namely, without/with a true cross-level interaction and without/with a true cross-level main effect.

Although it is difficult to determine the optimal sample size required for conducting HLM (Hofmann, 1997), several researchers have made some general recommendations. For instance, both Bassiri (1988) and Van Der Leeden and Busing (1994) showed that a sample of 30 groups with 30 persons in each group or a sample of 150 groups with 5 persons in each group was needed to obtain sufficient power to detect cross-level interactions. However, the use of much smaller samples has become the standard practice, typically involving less than 40 groups with around 5 persons in each group (e.g., Erdogan & Enders, 2007; Henderson et al., 2008; Herold et al., 2008; Herold, Fedor, & Caldwell, 2007; Hirst, Van Knippenberg, & Zhou, 2009; Marrone, Tesluk, & Carson, 2007). Due to practical difficulties, it is not surprising to observe such a gap in the sample size recommended and the sample size actually adopted in practice. To make our simulations realistic, we adopted 40 groups with 5 persons in each group as the baseline design for the simulations, denoted as the 40 × 5 design. However, when we observed a significant influence of CMV, we included larger sample sizes to provide a comprehensive account of its effects.

Procedure for Data Generation

To generate data sets by adapting the approach of Evans (1985, p. 309) to the two-level hierarchical linear model, we assumed two independence relationships:

the independence between the true scores of the individual-level variable and of the group-level variable ( $X_{t}$ and $Z_{t}$ );

the independence between the true scores and correlated errors (i.e., $X_{t}$ and $e_{X C}$ , $Z_{t}$ and $e_{Z C}$ ).

The first step was to generate the true scores of the two independent variables

X

and

Z_{i n d i v i d u a l}

. The true score of

X

was generated from the standard normal distribution

N (0, 1)

,⁴ and the true score of

Z_{i n d i v i d u a l}

was generated from the normal distribution with a mean of 0 and two levels of variance (high variance: 1.5 × number of persons per group; low variance: .5 × number of persons per group). Accordingly, the variance of the group-level variable

Z

based on the aggregation of

Z_{i n d i v i d u a l}

had two levels of variance, .5 and 1.5. The variable

X Z

was calculated by multiplying

X

and

Z

. The scores of the dependent variable

Y

were constructed by Equation 7, where the parameters varied according to the values specified in Table 1. For example,

Y

in the strong cross-level interaction and weak cross-level main effect model was obtained by the following equation:

Y = .3 X + .3 Z + .7 X Z + u_{0} + u_{1} X + e,

where both $u_{0}$ and $e$ were independently generated from the normal distribution $N (0, .5)$ , $u_{1}$ was generated from the distribution $N (0, τ_{11})$ , and $τ_{11}$ was set as .1, .3, or .5. Similar procedures were used to generate correlated error scores. For example, in the 40 × 5 design, if $V a r (Z) = 1.5$ , $r_{e_{Y C} e_{X C}}$ = $.7$ , and $r_{e_{Y C} e_{Z_{i n d i v i d u a l} C}}$ = .7, the following equations would be used:

\begin{aligned} e_{Y C} = N o r m a l (0, 1) \\ e_{X C} = .7 e_{Y C} + N o r m a l (0, .51) \\ e_{Z_{i n d i v i d u a l} C} = 1.917 e_{Y C} + N o r m a l (0, 3.825) \\ e_{Z C} = t h e g r o u p m e a n o f e_{Z_{i n d i v i d u a l} C} \end{aligned} .

Therefore, the correlation between

e_{Y C}

and

e_{Z C}

was equal to

.7 / \sqrt{5}

. Following the approach of Evans (1985), after true and error scores were obtained, we assigned weights to true scores and then added error scores. For instance, if we expected the true variance to CMV ratio to be 10:1, the observed scores would be calculated as follows:

\begin{aligned} Y = 10 Y_{t} + e_{Y C} \\ X = 10 X_{t} + e_{X C} \\ Z = 10 Z_{t} + e_{Z C} \end{aligned} .

For comparison purposes, the data sets with no correlated error terms were generated by setting the ratio as 1:0.

The S-Plus program provides t tests that indicate whether a parameter estimate (a cross-level interaction or main effect term) in each replication significantly departs from zero (Hofmann, 1997), and the conventional p value of .05 was used as a cut-off point for hypothesis testing. To investigate the effects of CMV, we used Type I error rate, defined as the probability of making wrong inferences when there is no true effect, and true positive rate, defined as the probability of making correct inferences when there is a true effect (absence of Type II errors). Type I error rates and true positive rates shown in Tables 2 through 6 represent the proportion of incorrect and correct inferences among the 100 replications for each scenario, respectively. We note that low statistical power, which is a function of small sample size and a small effect, may also result in inaccurate assessment of a model, and this point is considered in interpreting the results.

Table 2.

Type I Error Rates for No True Cross-Level Interaction.

				Ratio of True Variance to CMV
				No CMV		10:1		4:1		2:1		1.53:1		1:1		1:1.5
				Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)
Model	$r_{e_{Y C} e_{X C}}$	$r_{e_{Y C} e_{Z C}}$	$τ_{11}$	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5
A1 .3X+.3Z	.7	$.7 / \sqrt{5}$	.1	.06	.06	.08	.05	.07	.04	.06	.05	.05	.05	.00	.05	.05	.02
A2			.3	.08	.02	.07	.02	.03	.03	.03	.03	.02	.03	.02	.02	.02	.05
A3			.5	.05	.07	.06	.07	.06	.07	.06	.05	.07	.04	.07	.03	.03	.06
A4	.3	$.3 / \sqrt{5}$	.1	.03	.07	.03	.05	.05	.05	.07	.03	.09	.02	.08	.03	.03	.04
A5			.3	.05	.07	.04	.09	.05	.06	.04	.08	.04	.07	.05	.06	.06	.04
A6			.5	.05	.05	.05	.04	.06	.07	.06	.06	.08	.06	.08	.05	.05	.04
B1 .7X+.7Z	.7	$.7 / \sqrt{5}$	.1	.03	.08	.03	.08	.05	.06	.05	.04	.06	.04	.03	.05	.05	.04
B2			.3	.04	.05	.06	.06	.06	.05	.06	.06	.08	.05	.08	.03	.03	.04
B3			.5	.06	.07	.06	.07	.05	.05	.05	.05	.04	.06	.04	.05	.05	.05
B4	.3	$.3 / \sqrt{5}$	.1	.05	.01	.05	.01	.08	.04	.05	.06	.05	.06	.04	.07	.07	.05
B5			.3	.04	.07	.04	.07	.03	.07	.04	.09	.06	.09	.03	.06	.06	.05
B6			.5	.06	.08	.05	.06	.09	.06	.08	.05	.09	.04	.05	.02	.02	.05

Note: CMV = common method variance.

To assess the extent of the bias in parameter estimates for cross-level interactions and main effects, we systematically selected some scenarios typical of real-life situations: no CMV, high and low ratios of true variance to CMV (4:1 vs. 1:1). The value for parameter bias in each scenario was obtained by subtracting the true value from the average parameter estimate based on 100 replications. A positive value represents an overestimation of a regression coefficient, and a negative value denotes an underestimation. The farther a value deviates from 0, the more severe the parameter bias. To conserve space, the results are only briefly described because the levels of Type I error rates and true positive rates tend to correspond to the extent of parameter bias.

Results

No True Cross-Level Interaction

The results computed from the models without cross-level interactions were organized in two blocks, as shown in Table 2. As our focus was on the cross-level interaction effect ( $X Z$ ), we fixed the correlations of $X$ and $Z$ with $Y$ together at .3 and .7 to represent a weak and strong correlation, respectively, to simplify our simulations. Column 1 depicts Blocks A and B, which were derived from models where both $X$ and $Z$ had low and high correlations with $Y$ , respectively. The second column indicates the correlation of the CMV of $X$ with that of $Y$ , the third column indicates the correlation of the CMV of $Z$ with that of $Y$ , and the fourth column indicates the variance of Level 1 slopes. The subsequent columns indicate the ratio of true variance to CMV and the variance of $Z$ . Type I error rate shown in each scenario indicates the probability of erroneously detecting a cross-level interaction effect among 100 replications based on t tests.

Table 2 clearly shows that regardless of the variation of the parameter values, an increase in CMV in relation to true variance (from left to right) has little impact on the Type I error rates. In other words, the vast majority of the 100 replications in each scenario detected no significant cross-level interaction given that there was no such interaction in the data. Regarding the bias in parameter estimates, we found that for both large and small ratios of true variance to CMV (4:1 and 1:1), the values for parameter bias did not deviate largely from 0. CMV did not have a large influence on parameter bias for a cross-level interaction effect when there is no true effect in the data. This finding is consistent with the conclusion of Evans (1985) and Siemsen et al. (2010) for single-level results, suggesting that a significant cross-level interaction effect is extremely unlikely to be produced by CMV.

The Presence of a True Cross-Level Interaction

Table 3 presents the results of true positive rates estimated from models in which a true cross-level interaction exists. The format of this table is similar to that of Table 2, with two levels of correlation between $X Z$ and $Y$ in column 1.

Table 3.

True Positive Rates for a True Cross-Level Interaction.

				Ratio of True Variance to CMV
				No CMV		10:1		4:1		2:1		1.53:1		1:1		1:1.5
				Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)
Model	$r_{e_{Y C} e_{X C}}$	$r_{e_{Y C} e_{Z C}}$	$τ_{11}$	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5
A1 .3X + .3Z + .3XZ	.7	$.7 / \sqrt{5}$	.1	.75	.97	.78	.97	.73	.96	.58	.86	.41	.72	.22	.44	.17	.16
A2			.3	.57	.94	.54	.94	.49	.88	.37	.75	.28	.62	.18	.36	.07	.18
A3			.5	.36	.74	.36	.69	.32	.66	.27	.56	.25	.49	.17	.25	.09	.09
A4	.3	$.3 / \sqrt{5}$	.1	.85	.99	.84	.99	.75	.99	.49	.94	.40	.81	.16	.46	.08	.14
A5			.3	.50	.95	.45	.94	.40	.90	.33	.75	.30	.66	.20	.40	.15	.11
A6			.5	.32	.79	.34	.79	.35	.74	.29	.60	.26	.51	.21	.31	.14	.11
B1 .7X + .7Z + .3XZ	.7	$.7 / \sqrt{5}$	.1	.72	.97	.72	.97	.68	.96	.53	.87	.43	.74	.27	.43	.12	.15
B2			.3	.46	.88	.44	.86	.39	.85	.32	.71	.29	.64	.23	.40	.07	.16
B3			.5	.35	.76	.35	.75	.31	.72	.26	.66	.23	.54	.12	.37	.10	.24
B4	.3	$.3 / \sqrt{5}$	.1	.79	.97	.80	.97	.71	.96	.56	.86	.42	.71	.14	.35	.07	.15
B5			.3	.58	.87	.58	.84	.54	.79	.39	.65	.34	.53	.15	.30	.06	.12
B6			.5	.41	.72	.42	.73	.40	.70	.33	.54	.28	.42	.20	.30	.14	.10
C1 .3X + .3Z + .7XZ	.7	$.7 / \sqrt{5}$	.1	1	1	1	1	1	1	.98	1	.91	.99	.64	.92	.24	.45
C2			.3	.99	1	.98	1	.96	1	.91	1	.80	.97	.61	.86	.31	.43
C3			.5	.87	1	.87	1	.86	1	.79	1	.72	.97	.45	.88	.22	.33
C4	.3	$.3 / \sqrt{5}$	.1	1	1	1	1	1	1	.99	1	.88	1	.59	.84	.19	.34
C5			.3	.99	1	.99	1	.99	1	.94	1	.83	.98	.45	.81	.17	.32
C6			.5	.93	1	.93	1	.91	1	.83	.99	.63	.99	.40	.72	.13	.34
D1 .7X + .7Z + .7XZ	.7	$.7 / \sqrt{5}$	.1	1	1	1	1	1	1	.98	1	.93	1	.64	.87	.27	.40
D2			.3	.97	1	.97	1	.97	1	.89	1	.78	.99	.52	.83	.16	.42
D3			.5	.93	1	.94	1	.91	1	.81	.99	.57	.97	.44	.80	.21	.43
D4	.3	$.3 / \sqrt{5}$	.1	1	1	1	1	1	1	.99	1	.94	1	.59	.83	.12	.33
D5			.3	1	1	.99	1	.95	1	.86	1	.75	1	.51	.82	.18	.27
D6			.5	.94	1	.93	1	.92	1	.80	.99	.70	.96	.51	.70	.26	.34

Note: CMV = common method variance.

It is clear that CMV can decrease the probability of detecting a statistically significant cross-level interaction effect, especially when both the relative amount of true variance and the interaction effect were low. We also note some similar results across the four blocks. The true positive rates were generally reduced by an increase in the variance of Level 1 slopes ( $τ_{11}$ ), but raised by an increase in the variance of $Z$ , showing that a larger variance of $Z$ can enhance the statistical power for detecting significant effects (cf. Berk, 2004). With regard to the bias in parameter estimates, we found that when there was a true cross-level interaction, the presence of CMV led to serious parameter bias in the direction of underestimation. For the 4:1 ratio of true variance to CMV, the parameter bias ranged from –.23 to –.55 with a mean level of –.39, and for the 1:1 ratio, from –.21 to –.54, with a mean level of –.37. For both the 4:1 and 1:1 ratios of true variance to CMV, parameter bias was similar for both weak and strong interaction effects and was large, namely, over 70%, which is given by the ratio of parameter bias (average estimated value – true value) to the true value.

In sum, consistent with the single-level results of Evans (1985) and Siemsen et al. (2010), CMV generally reduces the statistical power to detect true cross-level interactions as well as leads to an underestimation of the regression coefficient. Type II errors are especially serious when the ratio of true variance to CMV is low, when the interaction effect is low, when the variance of Level 1 slopes is high, and when the variance of the group-level predictor is low. In our simulations, the values of the variance of Level 1 slopes and of the group-level predictor were realistic as they were based on published articles. Thus, a sizeable number of papers employing HLM may underreport significant cross-level interactions because of high variance of Level 1 slopes and low variance of the group-level predictor. Our simulation results also reveal that the parameter for a true cross-level interaction effect can be severely underestimated due to CMV, as in a single-level model (Siemsen et al., 2010).

No True Cross-Level Main Effect

The results for the models without a true cross-level main effect are presented in Table 4. The format of this table is similar to that of Table 2, except that the two effect sizes of the individual-level predictor $X$ are presented in column 1 instead.

Table 4.

Type I Error Rates for No True Cross-Level Main Effect.

				Ratio of True Variance to CMV
				No CMV		10:1		4:1		2:1		1.53:1		1:1		1:1.5
				Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)
Model	$r_{e_{Y C} e_{X C}}$	$r_{e_{Y C} e_{Z C}}$	$τ_{11}$	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5
A1 .3X	.7	$.7 / \sqrt{5}$	.1	.07	.04	.08	.05	.10	.08	.11	.09	.12	.10	.19	.20	.32	.35
A2			.3	.02	.09	.03	.06	.01	.07	.06	.07	.11	.07	.22	.20	.40	.31
A3			.5	.06	.05	.05	.03	.06	.04	.08	.05	.12	.08	.19	.21	.36	.34
A4	.3	$.3 / \sqrt{5}$	.1	.07	.05	.07	.03	.06	.05	.07	.04	.09	.05	.09	.05	.21	.12
A5			.3	.02	.07	.02	.06	.03	.06	.04	.07	.07	.06	.09	.09	.13	.17
A6			.5	.04	.07	.04	.06	.06	.05	.09	.01	.10	.04	.14	.11	.21	.19
B1 .7X	.7	$.7 / \sqrt{5}$	.1	.07	.08	.08	.09	.08	.07	.09	.07	.07	.06	.11	.14	.25	.21
B2			.3	.06	.07	.05	.06	.06	.04	.08	.07	.12	.09	.19	.14	.30	.27
B3			.5	.08	.03	.08	.04	.06	.03	.04	.05	.05	.06	.11	.09	.17	.27
B4	.3	$.3 / \sqrt{5}$	.1	.06	.06	.04	.06	.05	.07	.07	.06	.07	.07	.11	.07	.15	.16
B5			.3	.03	.07	.03	.07	.03	.06	.03	.05	.05	.07	.10	.06	.16	.11
B6			.5	.07	.05	.04	.04	.03	.03	.06	.03	.08	.05	.09	.09	.15	.11

Note: CMV = common method variance.

Blocks A and B show the results generated from a model where $X$ has either a low or high correlation with $Y$ . When the ratio of true variance to CMV was 10:1, 4:1, or 2:1, there were relatively low Type I error rates in erroneously detecting a cross-level main effect. Noticeable Type I error rates occurred for the ratio of 1.53:1 and became more drastic as the ratio worsened to 1:1 and 1:1.5. Regarding parameter bias, noticeable overestimation occurred for the 1:1 ratio of true variance to CMV as compared with the 4:1 ratio. For the 1:1 ratio, the parameter bias ranged from .05 to .15, whereas for the 4:1 ratio, from –.02 to .03, which were small. Parameter bias was not serious unless the ratio of true variance to CMV was unfavorable (1:1). In addition, for the 1:1 ratio, parameter bias was more serious when the correlation between the CMV of the predictors and that of the outcome was high ( $r_{e_{Y C} e_{X C}}$ = $.7$ and $r_{e_{Y C} e_{Z C}}$ = $.7 / \sqrt{5}$ ) and the variance of the group-level predictor was small ( $V a r (Z) = .5$ ).

Consistent with Ostroff et al. (2002), who showed the inflationary influence of CMV on cross-level correlations, our findings provide support for the widespread worry that CMV may result in Type I errors, namely, the false identification of a significant cross-level main effect. Note that Ostroff et al. only provide a general test of the effect of CMV without exploring the conditions under which its effects are more serious. Because HLM takes into account the interdependence of individuals in a nested data structure and simultaneously estimates individual- and group-level parameters (Bryk & Raudenbush, 1992), our results are based on a more appropriate analytic technique and refine Ostroff et al.’s findings by showing that Type I errors are noticeable only when the proportion of CMV is relatively large (true variance to CMV ratios: 1.53:1, 1:1, and 1:1.5). Our results also revealed that for the cases of no true cross-level main effect, the higher the proportion of CMV, the more serious the parameter bias, which is consistent with the conclusion implied in Equation 7 in Siemsen et al. (2010, p. 461) for a single-level regression model. In a nutshell, the parameter estimation of cross-level main effects could be severely confounded by the magnitude of CMV. To sum up, a larger proportion of CMV may lead to a significant cross-level main effect that does not truly exist as well as an overestimation of its regression coefficient.

Solutions for Controlling for Type I Errors and Parameter Bias in the Presence of CMV

To control for Type I errors caused by CMV in assessing cross-level main effects, we investigated the effectiveness of two common strategies: increasing the significance level (Underwood, 1997) and the use of a larger sample. We first adopted the .01 instead of the .05 significance level in the t tests for significance testing. As expected, the first section of Table 5 (the 40 × 5 design) shows that the Type I error rates decrease except for the situations where there is more CMV than true variance (see Table 4) and at the same time the CMV of $X$ and $Z_{i n d i v i d u a l}$ correlates with that of $Y$ at a level as high as .7. We believe that this pattern is extremely unlikely in a real-life situation, and we therefore conclude that the use of the .01 significance level is generally effective in reducing the probability of erroneous cross-level main effects due to the presence of CMV (Type I errors). However, it should be noted that this strategy cannot correct the parameter bias caused by CMV.

Table 5.

Type I Error Rates for Effects of Significance Level and Sample Size for No True Cross-Level Main Effect.

				Ratio of True Variance to CMV
				No CMV		1.53:1		1:1		1:1.5
				Var (Z)		Var (Z)		Var (Z)		Var (Z)
Model	$r_{e_{Y C} e_{X C}}$	$r_{e_{Y C} e_{Z C}}$	$τ_{11}$	.5	1.5	.5	1.5	.5	1.5	.5	1.5
40 groups × 5 individuals in each group (p < .01)
.3X	.7	$.7 / \sqrt{5}$	.1	.02	.01	.02	.04	.06	.06	.17	.14
			.3	.01	.04	.04	.02	.08	.05	.21	.09
			.5	.00	.05	.01	.05	.05	.08	.17	.13
	.3	$.3 / \sqrt{5}$	.1	.00	.01	.02	.01	.05	.05	.07	.10
			.3	.00	.01	.01	.02	.01	.02	.01	.02
			.5	.01	.02	.03	.01	.04	.01	.08	.05
.7X	.7	$.7 / \sqrt{5}$	.1	.01	.00	.03	.03	.03	.04	.09	.11
			.3	.02	.00	.02	.01	.04	.06	.11	.17
			.5	.01	.00	.03	.00	.04	.04	.07	.11
	.3	$.3 / \sqrt{5}$	.1	.02	.00	.01	.01	.03	.01	.03	.04
			.3	.05	.01	.06	.01	.05	.02	.06	.07
			.5	.00	.01	.02	.04	.04	.04	.09	.08
40 groups × 12 individuals in each group (p < .01)
.3X	.7	$.7 / \sqrt{12}$	.1	.01	.00	.03	.02	.05	.03	.06	.05
			.3	.04	.02	.06	.01	.06	.01	.07	.05
			.5	.01	.00	.01	.01	.01	.03	.02	.04
	.3	$.3 / \sqrt{12}$	.1	.00	.01	.00	.03	.00	.04	.03	.04
			.3	.02	.02	.01	.01	.02	.01	.02	.02
			.5	.00	.00	.01	.02	.02	.03	.02	.05
.7X	.7	$.7 / \sqrt{12}$	.1	.01	.01	.02	.02	.03	.02	.05	.04
			.3	.01	.01	.01	.01	.02	.02	.04	.05
			.5	.01	.02	.00	.03	.01	.02	.04	.05
	.3	$.3 / \sqrt{12}$	.1	.00	.01	.00	.01	.00	.01	.00	.03
			.3	.02	.00	.04	.01	.03	.02	.03	.02
			.5	.02	.01	.02	.02	.02	.03	.01	.02
40 groups × 30 individuals in each group (p < .05)
.3X	.7	$.7 / \sqrt{30}$	.1	.04	.04	.05	.03	.08	.04	.10	.10
			.3	.06	.03	.05	.07	.08	.09	.09	.12
			.5	.04	.06	.04	.05	.06	.07	.10	.11
	.3	$.3 / \sqrt{30}$	.1	.09	.10	.04	.07	.08	.06	.09	.09
			.3	.03	.07	.04	.07	.05	.10	.07	.11
			.5	.07	.04	.10	.06	.08	.06	.10	.07
.7X	.7	$.7 / \sqrt{30}$	.1	.01	.03	.04	.02	.05	.03	.07	.05
			.3	.04	.08	.03	.08	.03	.09	.08	.12
			.5	.06	.04	.08	.05	.09	.06	.12	.07
	.3	$.3 / \sqrt{30}$	.1	.03	.04	.09	.02	.15	.05	.14	.11
			.3	.08	.09	.10	.11	.09	.08	.10	.11
			.5	.07	.05	.05	.04	.10	.06	.10	.07

Note: CMV = common method variance.

We increased the number of persons in each group while maintaining the significance level at .01. As expected, the 40 × 12 design presented in the second section showed negligible Type I errors, and if there is serious doubt about the threat of CMV, this more conservative design could be adopted.

We also explored the effectiveness of setting the significance level at .05, but increasing the sample size, both in terms of the number of groups and persons in each group. Contrary to our expectation, increasing the number of groups to 60 and 150 (the 60 × 5 and 150 × 5 designs) increased the Type I error rates with no alleviation of parameter bias. Parameter bias in both the 60 × 5 and 150 × 5 designs was similar to that of the 40 × 5 design. We then explored the effect of increasing the number of persons per group (30 and 10) while maintaining the number of groups at 40 (the 40 × 30 and 40 × 10 designs). As expected, the Type I error rates decreased with the number of persons in each group, with the lowest rates in the 40 × 30 design (see the third section of Table 5), but the level was still on average slightly higher than that of the 40 × 5 design at p < .01. With regard to parameter bias for the cases of no true cross-level main effect, parameter bias was lower for the 40 × 10 design and lowest for the 40 × 30 design in comparison with the 40 × 5 design.

We therefore conclude that increasing the number of groups or persons in each group is not as effective as raising the significance level to .01 in reducing the probability of Type I errors, but increasing the number of persons in each group is more effective in reducing parameter bias than increasing the number of groups when no true cross-level main effect exists. The 40 × 30 design seems effective in controlling for Type I errors in real-life situations, in which CMV is unlikely to exceed true variance. As noted before, using the p value of .01 involves a significant trade-off of reducing the power to detect a true cross-level main effect (the inflation of Type II errors) with no reduction of parameter bias.

Exploratory Analysis of Two Other Strategies for Mitigating CMV

In the extant literature, two other methods are proposed to control for CMV in multilevel designs. The split sample strategy (Ostroff et al., 2002) involves the use of half of a group to calculate the group-level predictor and the other half of the group for evaluating a model, thus avoiding the common source problem. The addition of more CMV contaminated variables is shown to be effective for single-level designs (Siemsen et al., 2010), and this strategy may be applied in a multilevel context.⁵ To explore the effectiveness of the split sample strategy, we used a relatively larger sample size (40 groups with 10 persons in each group) and randomly split each group into two halves. One half (40 × 5) was used to obtain the aggregate measure of the group-level variable $Z$ , and the other half, along with $Z$ estimated from the first half, was used in the HLM analysis. With regard to the addition of more CMV contaminated predictors, according to Siemsen et al.’s (2010) mathematical formula (Equation 18, p. 466), when there is no true effect, the more CMV contaminated predictors, the better the parameter estimates. As the first attempt to explore the effectiveness of this approach in an HLM context, we added up to 6 more group-level predictors, all with no true correlation with $Y$ , into the models without a true cross-level main effect.

The results clearly showed that for the cases of no cross-level main effect, both strategies were as effective as raising the significance level to .01 in reducing Type I error rates and led to less serious parameter bias as compared to the corresponding values for the 40 × 5 design at p < .05, especially when there was a larger proportion of CMV (1:1 ratio of true variance to CMV). These two strategies seem more desirable than raising the significance level to .01 due to their ability in reducing parameter bias in addition to Type I errors.

However, for the cases of true cross-level main effects, the true positive rates for both strategies were generally smaller than the corresponding values for the 40 × 5 design at p < .05, suggesting a higher level of Type II errors. In addition, both strategies also resulted in more serious parameter bias as compared to the corresponding values for the 40 × 5 design at p < .05. The split sample strategy may lead to more Type II errors and parameter bias, probably because a very large sample size is required by this approach. It is unclear why the addition of more CMV contaminated variables leads to more Type II errors and parameter bias. There are important unresolved issues associated with these two strategies, which deserve careful attention in future research. In sum, it turns out that these two strategies, like the use of p value of .01, also involve a trade-off between Type I and Type II errors and result in parameter bias, and they should be employed with caution.

Existence of a True Cross-Level Main Effect

In this section, we evaluated the cases in which a true cross-level main effect exists. To provide a detailed picture, we varied the correlations between $X$ and $Y$ and between $Z$ and $Y$ independently rather than fixed them together as in previous analyses. The results for the true positive rates of a true cross-level main effect are presented in Table 6, which is organized into four blocks based on the magnitude of the correlation of $X$ and $Z$ with $Y$ .

Table 6.

True Positive Rates for a True Cross-Level Main Effect.

				Ratio of True Variance to CMV
				No CMV		10:1		4:1		2:1		1.53:1		1:1		1:1.5
				Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)		Var (Z)
Model	$r_{e_{Y C} e_{X C}}$	$r_{e_{Y C} e_{Z C}}$	$τ_{11}$	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5	.5	1.5
A1 .3X + .3Z	.7	$.7 / \sqrt{5}$	.1	.42	.78	.43	.80	.48	.81	.55	.84	.62	.84	.68	.81	.76	.80
A2			.3	.44	.82	.43	.85	.43	.82	.52	.86	.53	.84	.62	.82	.67	.81
A3			.5	.41	.67	.40	.66	.43	.68	.45	.68	.49	.66	.56	.74	.70	.76
A4	.3	$.3 / \sqrt{5}$	.1	.41	.79	.40	.80	.37	.76	.41	.77	.40	.75	.45	.70	.47	.64
A5			.3	.31	.79	.31	.80	.33	.80	.29	.78	.31	.76	.33	.67	.32	.57
A6			.5	.33	.78	.35	.78	.36	.77	.40	.77	.39	.72	.41	.63	.39	.55
B1 .7X + .3Z	.7	$.7 / \sqrt{5}$	.1	.40	.78	.41	.80	.39	.80	.41	.81	.44	.80	.47	.78	.55	.80
B2			.3	.31	.79	.30	79	.33	80	.33	.79	.38	.80	.38	.79	.48	.78
B3			.5	.39	.83	.39	.84	.38	.82	.39	.81	.40	.77	.43	.76	.46	.78
B4	.3	$.3 / \sqrt{5}$	.1	.41	.77	.43	.77	.41	.75	.47	.72	.45	.66	.37	.60	.42	.54
B5			.3	.36	.81	.31	.81	.32	.78	.31	.72	.30	.67	.30	.60	.29	.49
B6			.5	.46	.79	.44	.78	.40	.75	.39	.75	.36	.72	.35	.63	.37	.50
C1 .3X + .7Z	.7	$.7 / \sqrt{5}$	.1	.97	1	.96	1	.95	1	.94	.99	.94	.99	.95	.98	.92	.97
C2			.3	.94	1	.93	1	.93	1	.94	1	.91	1	.89	.99	.91	.95
C3			.5	.95	1	.95	1	.94	1	.92	1	.93	1	.93	.99	.87	.95
C4	.3	$.3 / \sqrt{5}$	.1	.95	1	.94	1	.93	1	.88	1	.85	.99	.79	.96	.69	.88
C5			.3	.95	1	.95	1	.95	1	.90	1	.89	.99	.81	.99	.65	.91
C6			.5	.98	1	.98	1	.98	1	.97	1	.96	1	.91	.99	.82	.93
D1 .7X + .7Z	.7	$.7 / \sqrt{5}$	.1	.97	1	.96	1	.96	1	.94	1	.93	1	.89	1	.87	.99
D2			.3	.95	1	.95	1	.94	1	.94	1	.94	1	.91	.99	.90	.96
D3			.5	.96	1	.97	1	.95	1	.90	1	.89	.99	.87	.99	.84	.98
D4	.3	$.3 / \sqrt{5}$	.1	.91	1	.92	1	.91	1	.88	1	.84	.99	.75	.95	.68	.86
D5			.3	.94	1	.94	1	.94	1	.92	1	.88	.99	.79	.98	.70	.88
D6			.5	.97	1	.97	1	.98	1	.93	1	.86	1	.77	.98	.64	.93

Note: CMV = common method variance.

Our results show that CMV either lowered or increased the true positive rates depending on different combinations of parameter values. When $r_{e_{Y C} e_{X C}}$ = $.3$ and $r_{e_{Y C} e_{Z C}}$ = $.3 / \sqrt{5}$ , it is interesting to note that a decrease in the true variance to CMV ratio tended to decrease rather than increase the true positive rates. That is, CMV tends to reduce the chance of detecting a true effect.

With regard to parameter bias for a strong cross-level main effect, a trend of underestimation was noted, with the magnitude ranging from –.30 to –.01, with a mean level of –.09. More noticeable parameter bias occurred for the 1:1 ratio of true variance to CMV (ranging from –.30 to –.13) than for the model with a ratio of 4:1 (ranging from –.04 to –.01). The parameter bias for the 1:1 ratio was on average eight times larger than that for the 4:1 ratio.

When there was a weak cross-level main effect ( $.3 Z$ —Blocks A and B), we note that the statistical power of the design was relatively low, as true positive rates were well below .95. Nonetheless, true positive rates were higher for $V a r (Z) = 1.5$ than for $V a r (Z) = .5$ , providing evidence that the variance of group-level predictors can help enhance statistical power (Berk, 2004). With regard to the influence of CMV, when $V a r (Z) = 1.5$ , $r_{e_{Y C} e_{X C}}$ = $.3$ , and $r_{e_{Y C} e_{Z C}}$ = $.3 / \sqrt{5}$ , the low true positive rates caused by CMV occurred only when the ratios of true variance to CMV were 1.53:1, 1:1, and 1:1.5. In contrast, when $V a r (Z) = .5$ , $r_{e_{Y C} e_{X C}}$ = $.3$ , and $r_{e_{Y C} e_{Z C}}$ = $.3 / \sqrt{5}$ , CMV showed no obvious effect on true positive rates.

With regard to the bias in parameter estimates for a weak cross-level main effect, a general trend of underestimation was noted, with the magnitude ranging from –.11 to .03 and a mean level of –.02. More noticeable parameter bias occurred for the 1:1 ratio of true variance to CMV (ranging from –.11 to .01, with a mean level of –.06) than for the ratio of 4:1 (ranging from –.03 to .03, with a mean level of –.01). The parameter bias for the 1:1 ratio was on average four times larger than for the 4:1 ratio.

To sum up, when there is a strong true cross-level main effect, CMV tends to lower the probability of detecting the effect. When there is a weak true cross-level main effect, the statistical power of the 40 × 5 design in detecting the effect is generally low. CMV may increase or decrease true positive rates, depending on the true variance to CMV ratio, the correlation between the CMV of the predictors and that of the outcome, and the variance of $Z$ . Consistent with the cases of no true cross-level main effect, our result revealed that the extent of parameter bias was influenced by the ratio of true variance to CMV. For typical real-life situations, which involve moderate values of these several parameters, the effects of CMV on both Type II errors and parameter estimation seem mild.

Supplementary Analysis With a Large Sample

To explore if the previous results generalize to a larger sample size, we considered a relatively large sample size (60 groups with 12 persons in each group) for the four different types of models (without/with a true cross-level interaction and without/with a true cross-level main effect). For the sake of parsimony, we systematically selected several variations of the seven parameters in the simulation.

For models without a true cross-level interaction or main effect, CMV can create an artificial cross-level main effect, but not a cross-level interaction effect as in the 40 × 5 design. With regard to parameter bias, CMV does not result in large parameter bias for a cross-level interaction when no true effect exists, but leads to noticeable overestimation of the parameter for a cross-level main effect when there is no true effect and large CMV, which is similar to the results for the 40 × 5 design. For models with a true cross-level interaction or main effect, the true positive rates for the 60 × 12 design were on average higher than the corresponding values for the 40 × 5 design, which may be due to the increased power associated with a larger sample size. Regarding parameter bias, a general trend of underestimation was found when the proportion of CMV or the correlation between CMV of the predictors and that of the outcome variable increased. Parameter bias became severe for the 1:1 ratio of CMV to true variance. To sum up, the pattern for the Type I error rates, true positive rates, and parameter bias was generally consistent with those reported for the 40 × 5 design, thus supporting the generalizability of our findings to a larger sample size.

Discussion

There is a dearth of research on whether and when CMV inflates cross-level relationships in hierarchical linear modeling in organizational research and whether the role of CMV in single levels of analysis generalizes to multilevel designs. Based on a Monte Carlo approach, we address this important gap by delineating the influence of CMV on cross-level interaction and main effects and identifying strategies to reduce the probability of Type I errors and parameter bias by mitigating the confounding role of CMV.

The Role of CMV in Cross-Level Interactions

Our results clearly show that CMV cannot result in significant cross-level interaction effects that do not exist in the data nor any bias in parameter estimates. In contrast, CMV tends to suppress the identification of a true cross-level interaction and lead to an underestimation of the regression coefficient. This conclusion is in line with the single-level results of Evans (1985) and Siemsen et al. (2010). Evans concluded that “virtually no interactions were created by adding correlated error … interaction effects are strongly attenuated by the addition of correlated error” (pp. 316-317). In a similar vein, Siemsen et al. concluded based on statistical derivations that “CMV cannot create an artificial interaction effect. CMV can only deflate existing interactions” (p. 469) and emphasized that “researchers should not be criticized for CMV if the main purpose of their study is to establish interaction effects” (p. 470).

Our results strongly support the view that when there is a good theoretical reason to examine whether a group-level variable based on self-report moderates a percept-percept relationship, reviewers should not categorically raise the issue of CMV as a major methodological criticism. In fact, researchers have to be vigilant about the possibility that CMV may reduce the likelihood of confirming a well-grounded prediction of a cross-level interaction effect based on self-report measures. Interestingly, our simulation reveals a general trend that the higher the variance of the group-level predictor, the higher the true positive rates for detecting cross-level interactions. Thus, a potentially useful strategy is to collect data from geographically dispersed groups or diverse organizations, which may increase the variance of a group-level predictor. Strategies that help alleviate the underestimation problem caused by CMV in detecting a true cross-level interaction effect deserve attention in future research.

To account for the underidentification of true cross-level interactions due to CMV, we speculate that the reduction in covariation of CMV across levels may be a possible explanation.⁶ Specifically, individual-level method variance tends to be reduced in the aggregated score $Z$ because it may be “averaged out” if the number of groups and persons in each group are reasonably large. The CMV component of the cross-level interaction term $X Z$ may be further reduced in the formation of the product term, and it follows that the method variance of $X Z$ and $Y$ may covary at a lower level. In other words, the correlation between the CMV of $Y$ and that of the cross-level interaction term, $r_{e_{Y C} e_{X C} e_{Z C}}$ , may become small and exert little influence when there is no true effect. In contrast, when there is a true cross-level interaction ( $γ_{11} \neq 0$ ), CMV with a relatively small $r_{e_{Y C} e_{X C} e_{Z C}}$ may “pull down” the observed correlation between Y and XZ, because a low correlation due to CMV may lead to an observed correlation lower than the correlation based on true scores, thus suppressing the detection of a true effect and leading to the underestimation of the regression coefficient.

Because of the consistency between single-level results and our multilevel results, it is possible that the explanation for the impact of CMV on single-level interactions may generalize to the multilevel context.⁷ For single-level interactions, as implied in Siemsen et al.’s (2010, p. 469) Equation 24 and Equation 25, when there is no true effect, CMV does not contribute to the true covariance between the outcome variable and the product term, which is the numerator in the parameter estimation formula. When there is a true interaction effect, however, the denominator becomes larger because of the variance contributed by the CMV contamination, thus attenuating the true relationship between the product term and the outcome variable. In other words, we speculate that as CMV is modeled as linear, additive variance (see Equation 5), which is different from the multiplicative nature of an interaction term (Cortina, 1993), CMV may not affect the true covariance between the outcome variable and the cross-level interaction term, but rather add noise to the denominator and lead to underestimation. To sum up, the mechanisms underlying the underestimation of true cross-level interaction effects due to CMV deserve careful scrutiny in future research.

The Role of CMV in Cross-Level Main Effects

Some researchers may believe intuitively that the influence of CMV is less severe in a multilevel than in a single-level data set (e.g., Liao & Rupp, 2005). Consistent with this view, Ostroff et al. (2002) reported that the influence of CMV appears to be more serious for aggregate correlations as compared to cross-level correlations. However, our results do not support the optimistic expectation that CMV can be generally ignored for cross-level main effects, because the CMV correlation between the individual-level outcome ( $e_{Y C}$ ) and the group-level variable ( $e_{Z C}$ ) is not zero after aggregation.

Going beyond the general conclusion that method variance is present at multiple levels of analysis (Ostroff et al., 2002), our research delineates when CMV can affect a cross-level main effect. That is, when the ratios of true variance to CMV are relatively low (1.53:1, 1:1, and 1:1.5), Type I errors may occur. Large CMV may lead researchers to erroneously report a significant cross-level main effect that does not truly exist and result in an overestimation of the regression coefficient. Following the logic for explaining the role of CMV in cross-level interactions, one speculation is that because the reduction in the covariation of the method variance of $Z$ and $Y$ may be less in the case of cross-level main effects than in the case of cross-level interaction effects, the influence of CMV on cross-level main effects may be stronger. Another speculation is that CMV modeled as additive variance can inflate main effects, which are also based on additive variance.⁸ Again, future research is needed to probe the mechanisms underlying the effect of CMV on cross-level main effects.

Strategies for Reducing Type I Errors and Parameter Bias in Assessing Cross-Level Main Effects

Our results have important implications for mitigating the confounding influence of CMV in an HLM context. First, our results suggest that the use of the .01 significance level in the 40 groups × 5 persons design is effective in reducing the probability of Type I errors in the presence of CMV. This strategy is problematic only under very extreme conditions, in which CMV is larger than true variance, and at the same time the CMV of the predictors correlates highly with that of the outcome. In other words, unless CMV is extremely strong, which is unlikely in real-life situations, it will not generate a cross-level correlation that is significant at the .01 level. However, we note that this strategy involves a very significant trade-off of reducing the power to detect a true effect (the inflation of Type II errors) and that it cannot alleviate the problem of parameter bias. The decision to adopt this strategy depends on the cost of Type I errors vis-à-vis that of Type II errors.

Second, an increase in the number of groups or the number of persons per group, while maintaining the significance level of .05, turns out to be ineffective. Following Hofmann’s (1997) recommendation, we analyzed the 60 or 150 groups × 5 persons design and the 40 groups × 30 persons design. Surprisingly, Type I errors are more likely for the 60 × 5 and 150 × 5 designs than for the 40 × 5 design, and parameter bias is not alleviated by increasing the number of groups. We speculate that when the number of persons per group (5 persons in a group) does not change, the CMV correlation remains the same. But for a larger number of groups (i.e., 60 or 150), more random effects representing the group effect need to be estimated. Beal (1991) noted that when “there are too many random effects in the model, some of the variances and covariances will be poorly estimated” (p. 219). In this situation, the potentially poor estimation of standard deviation may result in more Type I errors, a speculation that awaits future verification. In addition, although the 40 groups × 30 persons design seems inadequate for controlling for Type I errors in the simulation results, it is on average better than the 40 × 5 design in this regard. Regarding parameter bias, for the cases of no cross-level main effect, the increase in the number of persons per group can alleviate parameter bias, whereas increasing the number of groups does not reduce parameter bias.

We speculate that increasing the number of persons in each group can directly lower the correlation between the CMV of $Z$ and that of the outcome variable ( $r_{e_{Y C} e_{Z C}}$ ), which is given by the correlation at the individual level ( $r_{e_{Y C} e_{Z_{i n d i v i d u a l} C}}$ ) divided by square root of the number of persons per group. However, the correlation is not affected by increasing the number of groups, and we conclude that in the case of no cross-level main effect, the use of more groups is clearly not effective in reducing the probability of Type I errors and parameter bias, but the use of more people per group, such as the 40 groups × 30 persons design, may be somewhat useful, a topic that deserves exploration in future research.

Third, we conducted exploratory analysis to investigate the effectiveness of the split sample procedure and the addition of more CMV contaminated variables to control for the influence of CMV. Our findings provide support for Ostroff et al.’s (2002) split sample strategy in reducing Type I errors and parameter bias in an HLM context. Regarding the addition of more CMV contaminated variables, our research extends Siemsen et al.’s (2010) conclusion for the cases of no true effect for single-level models to a multilevel context such that when there is no true cross-level main effect, adding more CMV contaminated variables is effective in reducing Type I error rates and parameter bias.

Interestingly, our results show that both strategies are similar to the use of p = .01 in that Type II error is elevated. For the split sample approach, a future research direction is to systematically explore its effectiveness as a function of sample size, especially the number of persons per group. A larger sample size gives rise to more statistical power, which should result in more accurate estimates (cf. Cohen, 1988). With regard to the addition of more CMV contaminated variables, our results show that Siemsen et al.’s (2010) analytical approach for single-level designs, and the mathematical formula (Equation 18, p. 466) proposed for deciding the number of such variables needed in reducing parameter bias, are unlikely to be directly applicable in a multilevel context. Deriving a mathematical equation that can appropriately describe the effects of CMV on parameter bias in multilevel models is very complex, but it is an important topic for future research.

CMV in Assessing Cross-Level Main Effects That Truly Exist

For the case in which a true cross-level main effect exists, CMV may reduce the likelihood of identifying the effect when the effect size is large. CMV may increase or decrease true positive rates when the effect size is small. Similar to cross-level interactions, a high between-group variance may increase true positive rates for cross-level main effects. To enhance the probability of correctly detecting a true main effect, one strategy is to gather data from diverse sources (e.g., units in different locations) to increase between group variance as in the case of cross-level interactions. Contrary to the popular belief that CMV tends to inflate observed relationships, it may sometimes suppress the likelihood of identifying a true cross-level relationship. Equally important, our results also suggest that the extent of parameter bias is affected by the ratio of CMV to true variance. A large proportion of CMV results in noticeable parameter bias with a trend of underestimation for both weak and strong cross-level main effects.

Researchers have to be careful about the effect of CMV when the data provide no support for their theoretical predictions of cross-level main effects. With regard to the mixed effects of CMV for models where true cross-level main effects exist, we speculate that the impact of CMV may depend on the relative size of the correlation between the CMV of $Y$ and that of $Z$ ( $r_{e_{Y C} e_{Z C}}$ ) to the true correlation between $Y_{t}$ and $Z_{t}$ ( $r_{Y_{t} Z_{t}}$ ).⁹ It can be shown that the observed correlation $r_{Y Z}$ assumes a value between $r_{Y_{t} Z_{t}}$ and $r_{e_{Y C} e_{Z C}}$ . Specifically, when $r_{e_{Y C} e_{Z C}}$ is close to $r_{Y_{t} Z_{t}}$ , CMV may not play a strong role. When $r_{e_{Y C} e_{Z C}}$ is much higher than $r_{Y_{t} Z_{t}}$ , CMV may “pull up” the observed correlation between $Y$ and $Z$ , thus increasing true positive rates. When $r_{e_{Y C} e_{Z C}}$ is much lower than $r_{Y_{t} Z_{t}}$ , CMV may “pull down” the observed correlation, which is analogous to the pattern observed in models where true cross-level interactions exist, but are underestimated.

It is also possible that the additive nature of CMV may account for its impact on cross-level main effects, which are also additive in nature (Cortina, 1993). Indeed, as indicated in both Equation 7 and Equation 8 in Siemsen et al. (2010, p. 461), CMV tends to exert additive impact on single-level main effects. How the additive nature of CMV causes mixed effects on cross-level main effects deserves attention in future research.

Overview of Strategies to Control for CMV in Multilevel Modeling

Previous research has identified some statistical remedies to address the problem of CMV, such as the omitted variables model (Kim & Frees, 2006, 2007), Lewbel’s (1997) internal instrumental variable method (Ebbes, Bockenholt, & Wedel, 2004), and two-stage least squares (see a review by Antonakis et al., 2010; Ebbes et al., 2004). These strategies are effective under some specific circumstances and are reviewed together with the strategies evaluated in the present research in Table 7.

Table 7.

Summary of Strategies to Control for Common Method Variance (CMV) in Multilevel Modeling.

Remedy	Reference	Source of CMV	Assumption or Requirement	Application and Trade-Off
Omitted variables method	Kim and Frees (2006, 2007)	Raters as a higher level source of CMV	CMV is at a higher level than predictors; at each level, there is at least one predictor not contaminated by CMV at the same level	Panel or longitudinal data, such as ecological momentary assessment data
Lewbel’s (1997) internal instrumental approach	Ebbes, Bockenholt, and Wedel (2004)	All sources	Predictors follow a strongly skewed distribution	Any multilevel context as long as predictors show a skewed distribution; the greater the skewness, the less the parameter bias
Two-stage least squares	Ebbes et al. (2004); Antonakis, Bendahan, Jacquart, and Lalive (2010)	All sources	The existence of an external instrumental variable that is not correlated with CMV but has a strong correlation with the predictor	Any multilevel context as long as an external instrumental variable can be identified
Addition of more CMV contaminated variables	Siemsen, Roth, and Oliveira (2010)	All sources	The more CMV contaminated variables, the better the parameter estimates when there is no true effect	Reduce Type I errors along with parameter bias but increase Type II errors
Split sample	Ostroff, Kinicki, and Clark (2002)	Self-report	CMV due to self-report data (source of CMV is at the same level as predictors)	Second-level predictor based on aggregation; large sample size; reduce Type I errors along with parameter bias but increase Type II errors
Temporal gap	Ostroff et al. (2002)	Self-report	CMV due to self-report data	Second-level predictor based on aggregation; magnitude of effectiveness is not clear
p < .01	Present study	Self-report	CMV due to self-report data	Second-level predictor based on aggregation; reduce Type I errors with no alleviation of parameter bias, but increase Type II errors

First, researchers in education and economics have developed some techniques to correct for the influence of CMV in a multilevel context, known as “omitted variables” (e.g., Ebbes et al., 2004; Kim & Frees, 2006, 2007). An important assumption made by Kim and Frees (2006, 2007) is that CMV is at a higher level than the predictors in a model. Under this situation, the authors recommended the use of an “orthogonal” transformation method to address the problem of parameter bias caused by an omitted variable, which is CMV in this context. A type of data amenable to this approach is panel data, because respondents are observed over multiple time points and they can be viewed as a higher level source of common method variance. For instance, this approach is suitable for analyzing ecological momentary assessment data, the collection of which requires raters to report their typical experiences and behavioral responses at many time points (Beal & Weiss, 2003). A different approach to eliminating the influence of an omitted effect is Lewbel’s (1997) internal instrumental variable method. For this approach to work, the skewness of a predictor needs to be strong (Ebbes et al., 2004), but the source of CMV can be at a higher level than or within the same level as the predictors.

Second, the work of Ebbes et al. (2004) and a review by Antonakis et al. (2010) suggested the use of a two-stage least squares procedure to address the problem of CMV if researchers can identify an external instrumental variable that does not correlate with $e$ , an omitted common cause, but has a strong correlation with the predictor. Although Antonakis et al. noted that “one of the biggest challenges researchers face when attempting to estimate instrumental variable models has to do with where to find instruments” (p. 1103), the use of external instrumental variables seems an effective way to mitigate the influence of CMV if such variables can be identified.

Third, our study focuses on self-report as the source of common method bias, in which case the association between CMV in the outcome variable and predictors is assumed to be within the same level. Our results show that to reduce the probability of Type I errors, raising the significance level to .01 is more effective than increasing the number of groups or persons in each group, although at the risk of heightened Type II errors and with no reduction of parameter bias. In addition, both the split sample procedure (Ostroff et al., 2002) and the addition of more CMV contaminated variables (Siemsen et al., 2010) are effective in reducing Type I errors and parameter bias in the cases of no true cross-level main effect, but also at the expense of increasing Type II errors. These two strategies tend to increase parameter bias in the cases of a true cross-level main effect.

Fourth, if self-report measures are assumed to be the source of CMV, the use of nonaggregated group-level variables from a different source, such as supervisory ratings, should be effective because the group-level predictor and the outcome variable come from different sources. However, Podsakoff et al. (2003) noted that other sources of common method bias, including item characteristics, item context, and measurement context, may bias cross-level relationships despite of the collection of data from multiple sources. For instance, Feldman and Lynch (1988) noted that the use of a similar response format may give rise to a bias in which “cognitions generated in answering one question will be retrieved to answer subsequent questions” (p. 427). This type of CMV can bias the cross-level results obtained.

To illustrate this point, we assessed a model where the Level 2 variable was not based on the aggregation of a Level 1 variable but was provided by a different source, such as supervisors. We investigated some typical scenarios (no CMV, 4:1, and 1:1 ratios of true variance to CMV) and generated the CMV of $Z$ based on the within-group mean of the CMV of the dependent variable $Y$ . The results revealed that the Type I error rates, true positive rates, and parameter bias were generally consistent with those for the model where the Level 2 predictor was aggregated from individual-level responses. We conclude that regardless of the source of CMV, its influence on cross-level relationships is primarily determined by the correlation between the CMV of $Z$ and that of the outcome variable ( $r_{e_{Y C} e_{Z C}}$ ). We note that a nonaggregated variable tends to be less reliable as compared to an aggregated variable (Scullen, 1997). In addition to the influence of CMV, the effect of aggregation on significance testing and parameter estimation needs to be taken into account in future research.

In a similar vein, the split sample design proposed by Ostroff et al. (2002) can address CMV due to the use of self-report data, but may not be able to address other sources of CMV. Note that the split sample design has one more constraint, namely, “the inherent difficulty of obtaining large enough samples to conduct cross-level spilt” (Ostroff et al., 2002, p. 366).

Finally, temporal spacing of measures may help minimize the impact of CMV because Ostroff et al.’s (2002) results clearly suggest that incorporating time delays between independent and dependent variables can help reduce response bias associated with CMV. However, this strategy may involve other types of biases, such as the nonequivalence of the context in which the different waves of data are collected.

To conclude, despite the fact that CMV has been regarded as a serious threat to the validity of findings based on self-report measures for decades, very little empirical research has been undertaken to investigate how serious the threat is. Our study adds to this important literature by an extension to the HLM context. Our results are in line with the single-level results of Evans (1985) and Siemsen et al. (2010) in that CMV does not pose a threat to the validity of significant cross-level interaction effects. With regard to cross-level main effects, we extend Ostroff et al.’s (2002) study by documenting the possible influence of CMV in an HLM context and provide researchers with a range of possible strategies to mitigate such confounding influence.

Footnotes

Acknowledgments

We would like to thank Associate Editor Adam W. Meade and three anonymous reviewers for their constructive comments and suggestions.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research was supported in part by an MOE (Ministry of Education in China) Youth Foundation Project of Humanities and Social Sciences (grant number 12YJC630087) and by National Natural Science Foundation of China (NSFC) (grant number 81201817).

Notes

References

Antonakis

Bendahan

Jacquart

Lalive

(2010). On making causal claims: A review and recommendations. The Leadership Quarterly, 21, 1086–1120.

Bassiri

(1988). Large and small sample properties of maximum likelihood estimates for the hierarchical linear model (Unpublished doctoral dissertation). Michigan State University, East Lansing.

Beal

S. L.

(1991). Computing initial estimates with mixed effects models: A general method of moments. Biometrika, 78, 217–220.

Beal

D. J.

Weiss

H. M.

(2003). Methods of ecological momentary assessment in organizational research. Organizational Research Methods, 6, 440–464.

Berk

R. A.

(2004). Regression analysis: A constructive critique. Thousand Oaks, CA: Sage.

Bliese

P. D.

(2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analyses. In Klein

K. J.

Kozlowski

S. W. J.

(Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco, CA: Jossey-Bass.

Brannick

M. T.

Chan

Conway

J. M.

Lance

C. E.

Spector

P. E.

(2010). What is method variance and how can we cope with it? A panel discussion. Organizational Research Methods, 13, 407–420.

Bryk

A. S.

Raudenbush

S. W.

(1992). Hierarchical linear models. Newbury Park, CA: Sage.

Campbell

J. P.

(1976). Psychometric theory. In Dunnette

M. D.

(Ed.), Handbook of industrial and organizational psychology (pp. 85–122). Chicago, IL: Rand McNally.

10.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ: Erlbaum.

11.

Conway

J. M.

(2002). Method variance and method bias in I/O psychology. In Rogelberg

S. G.

(Ed.), Handbook of research methods in industrial and organizational psychology (pp. 344–365). Oxford, UK: Blackwell Publishers.

12.

Cortina

J. M.

(1993). Interaction, nonlinearity, and multicollinearity: Implications for multiple regression. Journal of Management, 19, 915–922.

13.

Cote

J. A.

Buckley

(1988). Measurement error and theory testing in consumer research: An illustration of the importance of construct validation. Journal of Consumer Research, 14, 579–582.

14.

Ebbes

Bockenholt

Wedel

(2004). Regressor and random-effects dependencies in multilevel models. Statistica Neerlandica, 58, 161–178.

15.

Erdogan

Enders

(2007). Support from the top: Supervisors’ perceived organizational support as a moderator of leader-member exchange to satisfaction and performance relationships. Journal of Applied Psychology, 92, 321–330.

16.

Evans

M. G.

(1985). A Monte Carlo study of the effects of correlated method variance in moderated multiple regression analysis. Organizational Behavior and Human Decision Processes, 36, 305–323.

17.

Feldman

J. M.

Lynch

J. G.

(1988). Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. Journal of Applied Psychology, 73, 421–435.

18.

Henderson

D. J.

Wayne

S. J.

Shore

L. M.

Bommer

W. H.

Tetrick

L. E.

(2008). Leader-member exchange, differentiation, and psychological contract fulfillment: A multilevel examination. Journal of Applied Psychology, 93, 1208–1219.

19.

Herold

D. M.

Fedor

D. B.

Caldwell

S. D.

(2007). Beyond change management: A multilevel investigation of contextual and personal influences on employees’ commitment to change. Journal of Applied Psychology, 92, 942–951.

20.

Herold

D. M.

Fedor

D. B.

Caldwell

Liu

(2008). The effects of transformational and change leadership on employees’ commitment to a change: A multilevel study. Journal of Applied Psychology, 93, 346–357.

21.

Hirst

Van Knippenberg

Zhou

(2009). A cross-level perspective on employee creativity: Goal orientation, team learning behavior, and individual creativity. Academy of Management Journal, 52, 280–293.

22.

Hofmann

D. A.

(1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23, 723–744.

23.

Hofmann

D. A.

Gavin

M. B.

(1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24, 623–641.

24.

Hofmann

D. A.

Griffin

M. A.

Gavin

M. B.

(2000). The application of hierarchical linear modeling to organizational research. In Klein

K. J.

Kozlowski

S. W. J.

(Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 467–511). San Francisco, CA: Jossey-Bass.

25.

Hofmann

D. A.

Morgeson

F. P.

Gerras

S. J.

(2003). Climate as a moderator of the relationship between leader-member exchange and content specific citizenship: Safety climate as an exemplar. Journal of Applied Psychology, 88, 170–178.

26.

James

L. R.

Demaree

R. G.

Wolf

(1993). r_wg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.

27.

Kim

J. S.

Frees

E. W.

(2006). Omitted variables in multilevel models. Psychometrika, 71, 659–690.

28.

Kim

J. S.

Frees

E. W.

(2007). Multilevel modeling with correlated effects. Psychometrika, 72, 505–533.

29.

Kirkman

B. L.

Chen

Farh

J.-L

Chen

Z. X.

Lowe

K. B.

(2009). Individual power distance orientation and follower reactions to transformational leaders: A cross-level, cross-cultural examination. Academy of Management Journal, 52, 744–764.

30.

Klein

K. J.

Dansereau

Hall

R. J.

(1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195–229.

31.

Klein

K. J.

Kozlowski

S. W. J.

(2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3, 211–236.

32.

Lewbel

(1997). Constructing instruments for regressions with measurement error when no additional data are available, with an application to patents and R&D. Econometrica, 65, 1210–1213.

33.

Liao

Rupp

D. E.

(2005). The impact of justice climate and justice orientation on work outcomes: A cross-level multifoci framework. Journal of Applied Psychology, 90, 242–256.

34.

Liao

Subramony

(2008). Employee customer orientation in manufacturing organizations: Joint influences of customer proximity and the senior leadership team. Journal of Applied Psychology, 93, 317–328.

35.

Marrone

J. A.

Tesluk

P. E.

Carson

J. B.

(2007). A multilevel investigation of antecedents and consequences of team member boundary-spanning behavior. Academy of Management Journal, 50, 1423–1439.

36.

Miller

(1975). Experimental design and statistics. London: Methuen.

37.

Morgeson

F. P.

Hofmann

D. A.

(1999). The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of Management Review, 24, 249–265.

38.

Ostroff

Kinicki

A. J.

Clark

M. A.

(2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfaction relationship. Journal of Applied Psychology, 87, 355–368.

39.

Pace

V. L.

(2010). Method variance from the perspectives of reviewers: Poorly understood problem or overemphasized complaint? Organizational Research Methods, 13, 421–434.

40.

Pagano

R. R.

(1981). Understanding statistics in the behavioral sciences. St. Paul, MN: West.

41.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J. Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.

42.

Podsakoff

P. M.

MacKenzie

S. B.

Podsakoff

N. P.

(2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569.

43.

Richardson

H. A.

Simmering

M. J.

Sturman

M. C.

(2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12, 762–800.

44.

Schulte

Ostroff

Kinicki

A. J.

(2006). Organizational climate systems and psychological climate perceptions: A cross-level study of climate-satisfaction relationships. Journal of Occupational and Organizational Psychology, 79, 645–671.

45.

Scott

B. A.

Barnes

C. M.

(2011). A multilevel field investigation of emotional labor, affect, work withdrawal, and gender. Academy of Management Journal, 54, 116–136.

46.

Scullen

S. E.

(1997). When ratings from one source have been averaged, but ratings from another source have not: Problems and solutions. Journal of Applied Psychology, 82, 880–888.

47.

Siemsen

Roth

Oliveira

(2010). Common method bias in regression models with linear, quadratic, and interaction effects. Organizational Research Methods, 13, 456–476.

48.

Spector

P. E.

(2006). Method variance in organizational research: Truth or urban legend? Organizational Research Methods, 9, 221–232.

49.

Spector

P. E.

Brannick

M. T.

(2009). Common method variance or measurement bias? The problem and possible solutions. In Buchanan

Bryman

(Eds.), Handbook of organizational research methods (Vol. 10, pp. 346–362). London: Sage.

50.

Spector

P. E.

Brannick

M. T.

(2010). Common method issues: An introduction to the feature topic in organizational research methods. Organizational Research Methods, 13, 403–406.

51.

Underwood

A. J.

(1997). Experiments in ecology: Their logical design and interpretation using analysis of variance. Cambridge, UK: Cambridge University Press.

52.

Van Der Leeden

Busing

F. M. T. A.

(1994). First iteration versus igls/rigls estimates in two-level models: A Monte Carlo study with ML3. Preprint PRM 94-03, Psychometrics and Research Methodology, Leiden, Netherlands.

53.

Walumbwa

F. O.

Hartnell

C. A.

Oke

(2010). Servant leadership, procedural justice climate, service climate, employee attitudes, and organizational citizenship behavior: A cross-level investigation. Journal of Applied Psychology, 95, 517–529.

54.

Williams

L. J.

Anderson

S. E.

(1994). An alternative approach to method effects by using latent-variable models: Applications in organizational behavior research. Journal of Applied Psychology, 79, 323–331.

55.

Williams

L. J.

Brown

B. K.

(1994). Method variance in organizational behavior and human resources research: Effects on correlations, path coefficients, and hypothesis testing. Organizational Behavior and Human Decision Processes, 57, 185–209.