Are We Correcting Correctly?

Typical Correction for Artifacts

Consistent with recommendations by Hunter and Schmidt (2004), Raju, Burke, Normand, and Langlois (1991), and others for conducting meta-analysis, the artifacts that are most commonly corrected for are the reliability of the predictor (r _xx), the reliability of the criterion (r _yy), and range restriction. Cortina (2003) showed that 78.6% of the 1,647 meta-analyses that were published in the Journal of Applied Psychology through 1997 corrected for one or more of these artifacts. Out of these 1,294 meta-analyses, 275 (approximately 21%) corrected the sample size-weighted mean correlation for more than one of these artifacts. Aguinis, Dalton, et al. (2011) showed that 51.2% of 5,560 meta-analyzed effect sizes included in their review corrected for measurement error in the dependent variable and 48.3% of 5,346 meta-analyzed effect sizes corrected for measurement error in the independent variable. Only 9.5% of 5,541 meta-analyzed effect sizes were corrected for range restriction on the dependent variable and 11.6% of 5,568 meta-analyzed effect sizes were corrected for range restriction on the independent variable.

Traditional correction formulae allow one to correct for multiple artifacts simultaneously. Earlier formulae assumed statistical independence among range restriction estimates, estimates of predictor reliability, and estimates of criterion reliability (Mendoza & Mumford, 1987). Later work by Mendoza and Mumford (1987), Theron (1998), and others identified the mathematical connections between range restriction and reliability, thus leading to the development of correction formulae that take this interdependence into account. However, no such formulae exist for reliabilities, and given the panoply of sources of overlap among reliabilities, it may be that such formulae are not possible.

The assumption of independent reliabilities is based on one of the axioms of classical test theory that states that errors associated with one measure are uncorrelated with errors associated with another measure (e.g., Lord & Novick, 1968; Novick, 1966; Zimmerman & Williams, 1977). This independence assumption is a within-study phenomenon, meaning that errors associated with the measurement of the true score of variable X in a given study, should be uncorrelated with errors associated with the measurement of the true score of variable Y in that same study. While this assumption has been contested by previous research (e.g., Cronbach, 1947; Novick, 1966; Rozeboom, 1966; Viswesvaran, Ones, & Schmidt, 1996), attempts to estimate it empirically are bound to fail. To estimate the correlation of errors within studies, one would have to compute the measurement error of variables X and Y as the difference between the true score and the observed score for each of the variables. Then one could correlate the error terms for the measurement of variables X and Y. However, we do not know the true score in the first place. Zimmerman and Williams (1977) have developed an adjusted formula for attenuation correction that takes correlations of measurement errors into account. But the fact that we cannot obtain empirical estimates for the errors reduces the utility of this work.

When considering the formulae used for the correction of attenuation in meta-analysis, however, we face a slightly different issue. While being conceptually based on the same rationale as the correction for attenuation in a single study case, the formulae in meta-analysis typically use a mean attenuation correction based on artifact distributions (e.g., Hunter & Schmidt, 2004; Pearlman, Schmidt, & Hunter, 1980). The question of independence then becomes a cross-study issue, and as such assumptions of independence based in classical test theory may be even more complicated.

In the next section, we review the correction for unreliability that is commonly used in the psychometric tradition of meta-analysis represented by the Hunter and Schmidt (1990, 2004) approach. As we pointed out earlier, according to Aguinis, Dalton, et al. (2011), this is the most commonly used approach in published meta-analyses in top management and organizational studies journals. We then discuss arguments for and against the assumption of independence at the cross-study level and derive research questions for the current article.

Correction for Unreliability

Variables are never measured without error. Measurement error masks the relationship between the true scores of the variables, usually resulting in an observed correlation that is weaker than the true score correlation. To better estimate the relationship between the true scores of two variables, researchers can either improve their measures or use formulae to correct for attenuation (Hunter & Schmidt, 2004; Pearlman et al., 1980).

Formula 1 expresses the general correction for attenuation for a single relationship between two variables X and Y.

ρ x y = r x y r x x r y y

1

The numerator represents the correlation between variables X and Y that is observed in a given study. The denominator contains the product of the two correction factors, that is, the square roots of the reliabilities of the measures of X and Y. With this formula, researchers “correct” for unreliability of the measures they used to assess X and Y, with the result being an estimate of the true score correlation between X and Y, that is, the correlation if they could be measured without error.

Just as authors of primary studies wish to estimate correlations between variables measured without error, so too do meta-analysts wish to correct mean effect sizes for flaws such as unreliability. To correct for attenuation in meta-analysis estimates, mean attenuation factors for predictor and criterion unreliability are typically computed (Hunter & Schmidt, 2004). These mean attenuation factors are based on reliability information from the studies in the meta-analysis that report it. The sample-size-weighted mean correlation in meta-analysis is then corrected for predictor and criterion unreliability by dividing by these mean attenuation factors (see Formula 2).

ρ = r ˉ r x x r y y

2

By aggregating reliabilities across the studies included in the meta-analysis and representing reliability as a single aggregate measure, we should no longer be concerned solely with interdependence issues at the within-study level. We also need to be concerned with interdependence at the between-study level. As outlined above, unfortunately, we cannot empirically assess the correlation of errors at the within-study level. We can, however, empirically examine the interdependence of reliabilities at the between-study level to determine the severity of this issue. In the following section, we provide a discussion of the arguments that previous research has raised with regard to interdependence among reliability estimates across studies.

The Assumption of Independence of Reliabilities Across Studies

James, Demaree, and Mulaik (1986) explain that the validity generalization procedures underlying the psychometric approach to meta-analysis are built on the following structural equation: r_i = ρ_i + e_i. In this equation, r_i is the observed score for the subject/sample, ρ_i is the true score for the subject/sample, and e_i is the random measurement or sampling error. As James et al. note, given that both the true score and the error are unknown, this equation is underidentified. So, to make use of this equation, validity generalization (VG) procedures need to make a set of assumptions to facilitate estimation of the true and error values. As James et al. summarize,

Given the basic structural equation r_i = ρ_i + e_i, it is assumed that (a) the mean error is zero within each study (population, situation), (b) ρ_i and e_i are unrelated across studies, and (c) σ_r ² = σ_ρ ² + σ_e ² (cf. Callender & Osburn, 1980, 1982; Hunter, Schmidt, & Jackson, 1982). Furthermore, implicit in the use of several VG estimating equations is the assumption that the errors are normally distributed and/or the assumption that the within-study error variances are homogeneous. (p. 446)

Apart from these widely known and discussed statistical assumptions, however, Hunter and Schmidt’s (2004) approach to meta-analysis is based on further assumptions that are less well-known and that speak directly to the issue of independence among statistical artifacts. These assumptions are not of a statistical nature, but they nevertheless are at the heart of the artifact correction formulae that most organizational researchers use. In their book, Hunter and Schmidt (2004) wrote,

The algebraic formulas used in this meta-analysis make certain assumptions, namely, that artifact parameters are (1) independent of the actual correlation and (2) independent of each other. Examination of the substantive nature of artifacts suggests that the independence assumptions are reasonable; most artifact values are less a matter of scientific choice and more a function of situational and resource constraints. The constraints for different artifacts usually have little to do with each other. (p. 139)

While the data for the Pearlman et al. meta-analysis consisted of studies in the area of personnel selection, in which commercially available tests were widely available, the same cannot be said for many other meta-analyses conducted on research topics that would not often use a commercially available test (e.g., organizational climate, psychological safety, organizational citizenship behaviors). It is reasonable to suppose that when researchers choose scales for their studies, they apply similar kinds of criteria for the selection of the measures for the predictor and the outcome. For example, how many items do the measures have (i.e., how long will it take participants to complete), has each measure been shown to be reliable in previous studies, or can the measures be administered simultaneously? Thus, while factors that lead to nonindependence might not stem from the construction of the scales per se, they may stem from the ways in which the tests are administered, the characteristics of the sample, or other situational constraints. So, in the end, whether predictor and criterion reliabilities are correlated is really an empirical question that should be answered by collecting the appropriate data.

Interestingly, the new meta-analytic approaches that were created to account for the fact that true scores and errors are correlated also include variance components to account for the correlation between statistical artifacts in their breakdown of total variance (see Mendoza & Reinhardt, 1991; Raju et al., 1991; Raju et al., 1998). However, none of the studies explains what the assumption of interdependence is based on. Furthermore, none of these studies has examined empirically the effect of correlated reliabilities across studies on the corrected correlations. Rather, the main purpose of these studies was to explore the utility of the new approaches as compared to other established meta-analytic techniques. The Monte Carlo simulations used in these articles focus on other interdependencies, mostly between the true score and the error score. There is a clear gap in our knowledge here of the magnitude of reliability correlations and the effect that correlated reliabilities have on our estimates of the corrected mean correlation. In the next section, we explore potential statistical and situational factors that can create interdependencies between predictor and criterion reliabilities across studies before examining empirically the question of correlated predictor and criterion reliabilities across studies.

Factors Contributing to Interdependence Between Predictor and Criterion Reliabilities

There are likely to be many factors that contribute to a correlation between predictor and criterion reliabilities across studies in a meta-analysis. We focus on two categories of factors, statistical and situational. To illustrate our argument, we use as an example a hypothetical meta-analysis of the relationship between the constructs conscientiousness and overall job performance. The meta-analysis consists of a number of original studies with different sample sizes and different measures of the focal constructs. Based on values taken from these studies, the meta-analyst computed the sample-size-weighted mean correlation and variance, correcting for sampling error in both estimates. In the next step, the researcher set out to correct for statistical artifacts as per Hunter and Schmidt (2004). Hence, our example meta-analysis is quite similar in design to many published meta-analyses.

We want to first discuss two statistical factors that are likely to create a correlation between the predictor and criterion reliabilities, true score variance and range restriction. Overlap between true score variances creates an underlying connection between reliability estimates.¹ An observed correlation between two variables, in our example conscientiousness and job performance, is a function of the underlying relationship between the true scores of these two variables. At the same time, the respective reliabilities of these two variables are also affected by the true scores in that they represent the amount of true score variance in the observed variance for the variables. Thus, if there is an underlying relationship between the true score of conscientiousness and the true score of job performance, there should also be a correlation between their reliabilities given that the true scores in one variable covary with the true scores of the other.

A related, but slightly distinct, influence factor is range restriction, which could differentially affect reliabilities in primary studies (as it would also affect the true score variance and the observed correlation between the two variables). In our example, this could mean that one study included in our meta-analysis collected data on the relationship between conscientiousness and job performance from senior job incumbents. Given that these incumbents likely have progressed in the organization in part due to their conscientiousness and prior job performance, this data set would likely suffer from direct range restriction, thus suppressing reliabilities among other things. At the same time, our meta-analysis might include a second study in which data for conscientiousness and job performance were collected from a sample of interns or newly hired employees. This sample likely suffers much less from direct range restriction, and hence the reliabilities for both the predictor and the criterion would be higher. The facts that both reliabilities would be low in studies suffering from range restriction and that both reliabilities would be higher in studies not suffering from range restriction would create covariation of predictor and criterion reliabilities across studies in our meta-analysis.²

A different line of argument in support of interdependencies between predictor and criterion reliabilities can be built based on work by James and colleagues (e.g., James, Demaree, Mulaik, & Ladd, 1992; James, Demaree, Mulaik, & Mumford, 1988; James et al., 1986), which focused on an examination of the influence of situational factors on VG procedures. James et al. (1992) suggest that traditional VG models establish the existence of situational moderators only indirectly by assessing whether substantial residual variance is left over after accounting for variance attributable to statistical artifacts. These models do not in fact model or test potential moderating effects. The residualization approach assumes that situational moderators affecting true validities are independent of statistical artifacts. However, James et al. assume that situational factors (such as organizational settings or biases; James et al., 1986; 1988) affect both the true variance and the artifacts, which makes the “artifacts” not actually artifacts but an expression of an omitted spurious effect of situational factors. These situational factors affecting true validities and statistical artifacts generate correlations between the artifacts and the true validity. This correlation violates the assumption of errors being uncorrelated with true scores and then renders the residualization approach (as well as the formulae for corrected correlations and variance) flawed as it cannot account for the interdependence.

Going one step further, we argue that the same situational moderators can affect both predictor and criterion reliabilities and create interdependencies among the statistical artifacts. Above and beyond modeling the correlation between true validity and statistical artifacts, the residualization approach would also need to account for correlations between the reliabilities and between other statistical artifacts to avoid overcorrection. Failure to account for the correlation between reliabilities would result in double correction for the same measurement error.

Applying this thinking to our example of a meta-analysis between conscientiousness and job performance, the type of organization could impact both the true correlation between these two variables and their respective reliabilities. For example, employees in organizations in which level of detail is important (e.g., accounting firms) might need to be more conscientious than employees in other organizations in which level of detail is not as important. This would mean that over time only the more conscientious employees remain and ascend within the organization, which leads to range restriction in our independent variable. This situational factor would then be associated with weaker validities for the conscientiousness/performance relationship in detail-oriented organizations than in other, nonrestricted, organizations. This restriction of range would also lead to smaller reliabilities for both the predictor and the criterion.

Alternatively, given that employees might be more detail-oriented and conscientious in detail-oriented organizations, they might devote more attention to completing the questionnaires provided to them. This in turn might generate more reliable measures of the two constructs than could be expected in other organizations. This would contribute to a correlation between the predictor and criterion reliabilities. In both cases, we would expect to see stronger correlations between the predictor and criterion reliabilities across primary-level studies, which in turn would lead the meta-analyst to overcorrect the sample-weighted mean correlation.

Present Study

In our discussion of the issue of interdependence of predictor and criterion reliabilities, we have argued that previous research has largely neglected the issue. Previous research has examined in detail other statistical assumptions of meta-analysis, such as the assumption of uncorrelated errors and true validities, and has largely concluded that meta-analytic approaches should be adapted to take these interdependencies into account. Previous research has also examined the interdependencies of reliabilities and range restriction, in particular criterion reliability and range restriction, and has offered altered formulae to take these interdependencies into account such as to not overcorrect the sample-size-weighted mean correlation. As, for example, research by LeBreton, Burgess, Kaiser, Atchley, and James (2003) shows the effects of range restriction on reliability coefficients can be quite extensive and hence, can greatly affect corrected correlations and subsequently the conclusions we draw based on these correlations.

However, no such examination exists for the interdependencies between predictor and criterion reliabilities. Without an empirical examination of this issue, we do not know the magnitude of the problem. Hence, the purpose of the current article is twofold: (a) to empirically examine whether predictor and criterion reliabilities are correlated across studies and (b) to identify the extent of their interdependency.

To accomplish these goals, we conducted two empirical examinations. In Study 1, we examined the correlation between predictor and criterion reliabilities across all relevant studies published in the Academy of Management Journal (AMJ) and the Journal of Applied Psychology (JAP) between January 2004 and October 2011. This resulted in a mix of studies across different research topics, variables, measures, samples, and conditions. This type of meta-analysis is very atypical. Usually, meta-analysts would focus on one research topic, such as the relationship between two specific variables, rather than collecting estimates across studies with different research topics. In the current study, however, we wanted to explore how much reliabilities would correlate when there was no consistent underlying relationship between the independent variable (IV) and the dependent variable (DV) across studies, that is, when the IV-DV relationships across studies should not correlate. This gives us a “baseline” condition, in which we can exclude the underlying effect that the true score correlation between the predictor and criterion constructs has on the correlation between the reliabilities across studies. In addition, given that there is no substantive, meaningful connection between the variables or research conditions across the original studies included in Study 1, we would not expect to see any relationships between the reliabilities, unless they represent situational factors or larger external influence factors, such as publication requirements or institutionalized norms for the conduct of research. Any correlations between the predictor and criterion reliabilities we find here represent the minimum correlation between reliabilities researchers should expect to have in their meta-analyses, even if the relationship between their IV and DV is nonexistent.

In Study 2, we collected original studies involving perceived organizational support (POS) and some of its antecedents and outcomes. In comparison to Study 1, Study 2 provides us with a more typical meta-analysis setting, in which the researcher assesses a substantive, meaningful underlying IV-DV relationship in a specific research topic area. We collected predictor and criterion reliability information from relevant original studies and examined the reliability correlations for different IV/DV combinations to examine the strength of their relationship. In this study, the effect of the true score correlation on the correlation between reliabilities is captured. Statistical and research design influence factors as well as situational factors are also likely to contribute to the correlation between the predictor and criterion reliabilities. Comparing results from the two studies allows us to compare the typical meta-analysis situation with a “baseline” situation and to obtain comprehensive estimates of relationships among reliabilities.

Study 1

Method

Compilation of Studies

The data collection process included an extensive manual search for published studies that included information about predictor and criterion reliability. To limit the vast amount of studies available to a manageable pool of studies, we chose to focus our search on two of the most renowned journals in the field of organization studies that predominantly publish empirical research: the Academy of Management Journal and the Journal of Applied Psychology. Our data set includes all applicable studies published in these journals between January 2004 and October 2011.

A study was included if it had reliability information for both the predictor measure and the criterion measure of interest in any given study. The particular research topic of a study did not matter at all, given that our concern is about the correlation between measurement errors of the predictor and the criterion measure that are supposedly unrelated to the theoretical constructs under examination. Part of the goal of this study was to include studies from as many different topics in organizational studies as possible to ensure that the findings are in fact not dependent on a particular research topic, specific theoretical constructs, or certain fields of research or discipline areas.

From each study we collected the following information: The type of reliability coefficient the study reported for the predictor and the criterion (i.e., internal consistency, test–retest, split-half, alternate forms, intrarater, or interrater), the magnitude of each reliability coefficient, information about the predictor and criterion constructs (i.e., which constructs were actually measured, for example, transformational leadership, job performance, emotional exhaustion), sample size, information about the type of sample (i.e., whether the sample consisted of students, employees, MBA students, or firms/organizations), the number of items used to measure the predictor and the criterion, whether the predictor and criterion information was provided by the same source or by different sources (i.e., the same person or different people), and how many raters provided information (for intra- and interrater reliability coefficients). Internal consistency reliability was indexed with Cronbach’s alpha. These values were coded as internal consistency reliability when respondents completed a measure about themselves to distinguish them from ratings of others. Intrarater reliability was also usually indexed with Cronbach’s alpha; however, the target of the questions was different from the person who completed the questionnaire (e.g., peer reports). Whereas these last two indices referred to consistency among items, interrater reliability refers to consistency among raters. We also coded for alternate forms reliability (correlation between two alternate forms of a questionnaire), split-half reliability (correlation between two halves of a test), and test–retest reliability (correlation between an original testing occasion and a retesting occasion with the same questionnaire after a certain amount of time had elapsed). However, because there were not enough studies that had used alternate forms, split-half, and test–retest reliability indices, we did not include them in our analyses (see below).

Four coders went through the journal issues to collect all the information from each study. The first author of the study provided detailed instructions to the three other coders and trained them on how to collect the information from the studies. All the information that needed to be collected from the studies was readily available from the methods sections. No subjective coding of any information was necessary. In the rare case that one of the three coders was uncertain about collecting the appropriate information from a study, the coder approached the first author of this study and they collected the necessary information from the article together. The first author conducted frequent checks of the studies coded by the other coders. Coding mistakes were rare, but when they were uncovered, the first author worked with the coder to remedy the mistakes and to provide further training to avoid these mistakes in the future.

We succeeded in locating 518 studies for a variety of predictors. From these 518 studies, we obtained 628 pairs of reliability estimates (see Table 1). We chose to focus on combinations of predictor and criterion reliability coefficients that had a sufficient number of studies available. As a result, our analysis focuses on the following five combinations of predictor and criterion reliabilities:

OPEN IN VIEWER

Table 1.

Number of Independent Reliability Estimates Obtained for All Predictor/Criterion Reliability Combinations.

		Criterion
		Alpha	Interrater	Split-Half	Intrarater	Test–Retest
Predictor	Alpha	399	23		128	4
		(316)	(21)		(110)	(4)
	Interrater	7	10		4
		(6)	(10)		(4)
	Split-half
	Test–retest	2			2
		(2)			(2)
	Intrarater	28	1		19
		(24)	(1)		(17)
	Parallel form	1 (1)

Note: Number of studies the estimates were obtained from in brackets.

Criterion internal consistency reliability/predictor internal consistency reliability

Criterion internal consistency reliability/predictor intrarater reliability

Criterion intrarater reliability/predictor internal consistency reliability

Criterion intrarater reliability/predictor intrarater reliability

Criterion interrater reliability/predictor internal consistency reliability

Data Analysis

Outlier analysis

In our study, we considered cases as outliers when their reliability coefficients for the predictor or the criterion were unusually large or small when compared to all the other values in the data set. We followed the approach for identifying outliers advocated by Aguinis, Gottfredson, and Joo (2013). They recommend a three-step approach that consists of the identification of error outliers, interesting outliers, and finally, influential outliers. Error outliers are deemed to have occurred due to inaccuracies, for example, due to coding errors, errors in computation, or errors in data manipulation. In the case of our meta-analysis, the only relevant error type would be due to coding errors, given that we obtained effect sizes from primary studies. So, if computation errors or errors in data manipulation happened in the primary studies, we would not be able to identify them. Aguinis et al. recommend two steps, identification of the potential error outlier and determination of whether the potential error outlier is an actual error outlier. For error identification, Aguinis et al. suggest using both single and multiple construct techniques by employing visual and quantitative tools.

In the current meta-analysis, we used the following visual tools: (a) We examined plots of the distributions of the reliability indices to identify cases that were in the tails of the distribution and thus very different from the rest of the values in the data set (i.e., single construct technique) and (b) we examined scatter plots of the bivariate relationships between predictor and criterion reliability to examine whether certain cases were situated far away from the main cloud of cases (i.e., multiple construct technique). Regarding quantitative tools, we chose to employ the following three-pronged approach for the quantitative identification of outliers: First, we checked for univariate outliers for each of the two reliability estimates (i.e., single construct technique). We conducted a standard deviation analysis and examined in more detail studies with reliability estimates that were more than 3 standard deviations above or below the mean. We chose 3 standard deviation units as a more conservative measure, given that our data are not normally distributed. Second, we checked for bivariate outliers using the difference-in-fit standardized (DFFITS) statistic. Third, we employed the approach suggested by Huffcutt and Arthur (1995), using the sample-adjusted meta-analytic deviancy (SAMD) statistic. The SAMD statistic uses a similar approach as the DFFITS statistic, but it takes the sample size on which each reliability estimate is based into account. We computed the SAMD statistic for each reliability estimate. For both the SAMD statistic and the DFFITS statistic we used scree plots to determine the values that are furthest away from the rest of the values (i.e., visual tool).

Based on these analyses, we identified eight potential error outliers. To determine whether these were actual error outliers, we went back to the original articles to check for coding errors. Amongst these cases, we found one coding error, which we corrected.

In the next step, we explored the characteristics of these cases a bit further. We observed that several of the high SAMD statistics were found for cases with large sample sizes. Given that this statistic takes sample size into account, studies with large samples were particularly prone to receiving a high SAMD statistic, even when their effect sizes were well within the limits of 3 standard deviations from the mean. We also found several effect sizes that were above or below 3 standard deviation units from the mean. None of these outliers were “interesting,” that is, outliers that are not due to inaccuracy and that could further theoretical knowledge by being representations of unique cases. Consequently, we moved on to determining whether these outliers are influential outliers, that is, outliers that have an effect on our parameter estimates. Following Aguinis et al.’s (2013) recommendations, we ran the main statistical analyses first using all data and then using the data set from which outliers were excluded to determine the effect that the identified outliers had on our data analysis.

For the data sets containing the effect sizes for the predictor and criterion internal consistency combination, there was virtually no difference in the results when running the analyses with outlier scores included or excluded, so we report results based on the full data set. The same was the case for the data set containing the effect sizes for the predictor intrarater/criterion internal consistency reliability combination.

For the data sets containing the predictor internal consistency/criterion interrater reliability combination and for the predictor and criterion intrarater reliability combination, there was one study that was very influential due to its sample size of N = 9,627. Despite the fact that the reported reliability coefficients were well within the range of the reliability coefficients reported by other studies in the same data sets, the inclusion of this study changed the correlation coefficients substantially. With the study included in the data set, the correlation between predictor internal consistency and criterion interrater reliability was r = –.45. Without the study, the correlation was r = –.06 (correlation with Fisher’s z-transformed scores: r = .08). For the predictor and criterion intrarater reliability combination, the inclusion of this study resulted in a correlation of r = .12, while its exclusion resulted in a correlation of r = .16 (correlation with Fisher’s z-transformed scores: r = .16). The main influence here seems to stem from the large sample size. Given the lower likelihood of sampling error in a large data set, we should put more rather than less confidence in a large study like this. As such, we decided to keep the study in our data set.

For the data set containing the predictor internal consistency/criterion intrarater reliability combination, we found one influential outlier where the reported criterion reliability was very low in comparison to the other criterion reliabilities in the data set. When the study was included the correlation was r = .01 (correlation with Fisher’s z-transformed scores: r = –.05), when it was excluded the correlation was r = .03 (correlation with Fisher’s z-transformed scores: r = –.04). The sample size for this study also was rather small (n = 84). Given that sampling error is likely to be larger in small samples than in big ones, we decided to eliminate this study from the data set. The conclusions were not significantly altered by the inclusion or exclusion of this study.

Main analysis

The procedure in the current meta-analysis is slightly different from other meta-analyses. Rather than determining the mean corrected correlation between two constructs across studies, the current meta-analysis involves computation of the correlation between reliability estimates obtained from the studies.

Being aware that nonnormal distributions of the data can affect correlation coefficients, we first assessed the shape of the predictor and criterion reliability distributions in our five data sets. The results are reported in Table 2. All of the underlying distributions were nonnormal as assessed by a Kolmogorov–Smirnov test for single samples, although most of the distributions did not show high values for skew or kurtosis. Most distributions were slightly skewed toward higher reliability scores, which can be expected given that studies with low reliabilities would have difficulties being published. This selective publication effect could also be seen in the fact that two of the five data sets did not show reliability scores below .70 (i.e., the predictor intrarater/criterion internal consistency and the predictor and criterion intrarater reliability combinations). However, the other three reliability combinations showed distributions of reliability scores that were much less affected by truncation.

OPEN IN VIEWER

Table 2.

Correlation Coefficients, Means, and Standard Deviations for the Five Combinations of Predictor and Criterion Reliability in the “Baseline” Meta-Analysis in Study 1.

	Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Intrarater		Criterion Intrarater/Predictor Alpha		Criterion Intrarater/Predictor Intrarater		Criterion Interrater/Predictor Alpha
	Pred.	Crit.	Pred.	Crit.	Pred.	Crit.	Pred.	Crit.	Pred.	Crit.
M	.85	.85	.91	.83	.86	.88	.92	.88	.86	.61
SD	.07	.08	.04	.06	.06	.05	.03	.03	.05	.23
Min	.53	.59	.74	.75	.60	.68	.74	.77	.65	.42
Max	.98	.98	.98	.97	.98	.97	.96	.98	.94	.97
Mdn	.86	.86	.93	.78	.88	.88	.93	.88	.88	.42
Skew	–0.78	–0.77	–1.82	0.47	–1.18	–0.80	–3.6	–1.5	–1.39	0.46
Kurtosis	0.57	0.05	3.28	–1.38	1.35	0.62	13.6	5.1	1.21	–1.67
Sample size	193,256		19,787		53,985		13,267		17,366
Number of effect sizes	399		28		127		19		23
Pearson’s r	.16**		–.12**		.03**		.12**		–.45**
Pearson’s r (on Fisher’s z scores)	.21**		–.14**		–.04**		.04**		–.29**

**p < .01.

Recent work by Bishara and Hittner (2012) has examined the impact of violations to the assumption of normally distributed data on the significance testing of different correlation coefficients and on alternative approaches to normalize the data (e.g., through transformation and ranking). In their simulations, they found that Pearson’s r was quite robust with respect to Type I error rates, unless the sample size in the data was small or the underlying distributions of both the predictor and the criterion were highly leptokurtic. Small sample sizes were not an issue in our data sets. In addition, four out of five of our data sets did not show high positive levels of kurtosis (i.e., were leptokurtic) for both the predictor and the criterion. The exception was the predictor and criterion intrarater reliability combination, which showed strongly elevated kurtosis levels for the predictor and elevated levels for the criterion. This is due to the inclusion of one large sample. As we mentioned earlier, we also ran the analyses with this large sample excluded. When excluding this study from the analysis, the kurtosis values were not elevated and the correlation between the reliability coefficients still reached significance. So, slightly elevated Type I error rates (i.e., around .07 as reported in Bishara & Hittner’s study) should not be a major concern here. Overall, Pearson’s r should provide an accurate estimate of the correlation between predictor and criterion reliability. Nevertheless, to compensate for the skew in the data, we transformed the obtained reliability coefficients with Fisher’s r-to-z transformations and also ran our analyses using these transformed scores. We report the obtained Pearson correlations for the observed and the transformed reliability coefficients in our tables.

We used the following variant of the sample-size-weighted mean correlation formula to compute the sample-size-weighted correlation between the reliabilities in our study,

r w e i g h t e d = ( ∑ n i ) x i − x ˉ ) ( y i − y ˉ ) ) / ( N − 1 ) s d ( x ) w e i g h t e d s d ( y ) w e i g h t e d

where x_i represents the predictor reliability coefficient, y_i represents the criterion reliability coefficient, and n_i represents the sample size of the study from which the reliability estimates were obtained.

Similarly, we adapted the formula to compute the standard deviation for this correlation coefficient to obtain a weighted estimate of the standard deviation.

s d ( x ) w e i g h t e d = ∑ ( n i ( x i − x ˉ ) 2 ) ∑ n i

An analogous formula was used to compute sd(y)_weighted.

Using these formulae, we computed separate sample-size-weighted correlation coefficients for each of the five different combinations of predictor and criterion reliability listed above.

Study 2

In our second study, we aimed to examine the issue of correlated reliabilities in a more typical meta-analysis situation, in which the researcher focuses on a specific research topic and examines a particular IV-DV relationship. The purpose of this examination was two-fold. On one hand, we wanted to see whether or not we would also find a correlation between predictor and criterion reliabilities across studies when looking at a set of original studies from the same topic area, that is, we wanted to see if we could replicate our results from Study 1. On the other hand, we wanted to explore whether the extent to which the reliabilities would correlate differed in this more typical scenario. As we argued earlier in the article, true score correlations in the underlying IV-DV relationship should contribute to the correlation between reliabilities. If we examine the same IV-DV relationship with (arguably) the same underlying true score correlation across studies, then the correlation of reliabilities across studies should be even higher than in Study 1.

Results

Tables 3 through 6 summarize the correlations, means, and standard deviations for the relationship between predictor and criterion reliability coefficients across the different POS meta-analysis data sets. Across all combinations between POS and its antecedents and outcomes, the means of the reported reliabilities were all very high. Only the mean for the criterion reliability coefficient for self-rated OCB-O was lower at .74. Two mean reliabilities were above .90 (both for POS measures in the self- and other-rated OCB-O data sets), nine reliabilities were between .85 and .90, and another two reliabilities were at least .80. Thus, 79% (i.e., 11 out of the 14) of the observed mean reliabilities were above .85. This prevalence of high reliability indices mirrors our findings from Study 1. And as was the case with Study 1, restriction of range due to the screening out of articles with low reliabilities likely suppresses these correlations.

OPEN IN VIEWER

Table 3.

Correlation Coefficients, Means, and Standard Deviations for the Relationship Between Predictor and Criterion Internal Consistency Reliability in the POS Meta-Analysis in Study 2 (All Studies Included).

	Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha
	Predictor POS	Criterion Affective Commitment	Predictor POS	Criterion Job Satisfaction	Predictor POS	Criterion Self- Rated Performance	Predictor POS	Criterion Self- Rated OCB-O	Predictor Procedural Justice	Criterion POS
M	.882	.846	.898	.848	.886	.805	.917	.743	.882	.854
SD	.065	.060	.058	.063	.062	.085	.039	.123	.060	.078
Min	.63	.52	.63	.57	.72	.49	.83	.55	.75	.62
Max	.97	.96	.97	.957	.96	.93	.96	.93	.98	.98
Mdn	.90	.86	.91	.85	.90	.82	.94	.66	.90	.88
Skew	–1.32	–1.17	–1.45	–0.708	–1.11	–0.350	–0.655	0.343	–0.827	–0.944
Kurtosis	1.49	1.59	2.04	1.86	0.714	–0.212	–1.03	–1.63	–0.424	0.697
Sample size	77,958	77,958	61,037	61,037	9,900	9,900	14,854	14,854	24,527	24,527
Number of effect sizes	230	230	140	140	31	31	24	24	69	69
Pearson’s r (reliabilities)	.302**		.069**		–.083**		–.797**		.339**
Pearson’s r (reliabilities, Fisher’s z scores)	.287**		.040**		–.036**		–.784**		.371**

Note: OCB-O = organizational citizenship behavior toward the organization; POS = perceived organizational support.

OPEN IN VIEWER

Table 4.

Correlation Coefficients, Means, and Standard Deviations for Predictor and Criterion Reliability (Self-Rated and Other-Rated) in the POS Meta-Analysis in Study 2 (All Studies Included).

	Criterion Alpha/Predictor Alpha		Criterion Intra/ Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Intra/Predictor Alpha
	Predictor POS	Criterion Self- Rated Performance	Predictor POS	Criterion Other- Rated Performance	Predictor POS	Criterion Self- Rated OCB-O	Predictor POS	Criterion Other- Rated OCB-O
M	.886	.805	.892	.889	.917	.743	.914	.824
SD	.062	.085	.055	.059	.039	.123	.030	.086
Min	.72	.49	.70	.69	.83	.55	.9	.4
Max	.96	.93	.97	.99	.96	.93	1.0	.9
Mdn	.90	.82	.91	.89	.94	.66	.91	.84
Skew	–1.11	–0.350	–1.908	–0.773	–0.655	0.343	0.113	–0.692
Kurtosis	0.714	–0.212	3.983	0.742	–1.03	–1.63	–0.565	1.427
Sample size	9,900	9,900	11,668	11,668	14,854	14,854	3,908	3,908
Number of effect sizes	31	31	50	50	24	24	20	20
Pearson’s r (reliabilities)	–.083**		.118**		–.797**		.083**
Pearson’s r (reliabilities, Fisher’s z scores)	–.036**		.011		–.784**		.074**

Note: OCB-O = organizational citizenship behavior toward the organization; POS = perceived organizational support.

OPEN IN VIEWER

Table 5.

Correlation Coefficients, Means, and Standard Deviations for the Relationship Between Predictor and Criterion Internal Consistency Reliability in the POS Meta-analysis in Study 2 (Outliers Excluded).

	Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Alpha
	Predictor POS	Criterion Affective Commitment	Predictor POS	Criterion Job Satisfaction	Predictor POS	Criterion Self- Rated Performance	Predictor POS	Criterion Self- Rated OCB-O	Predictor Procedural Justice	Criterion POS
M	.895	.849	.899	.851	.895	.81	.892	.843	.897	.88
SD	.0488	.0538	.055	.057	.049	.082	.028	.089	.045	.052
Min	.70	.61	.73	.66	.77	.64	.85	.55	.75	.77
Max	.97	.96	.97	.957	.96	.93	.96	.93	.98	.98
Mdn	.90	.86	.91	.85	.90	.82	.88	.91	.91	.885
Skew	–1.05	–1.13	–1.213	0.697	–0.758	–0.118	0.69	–1.11	–0.719	–0.435
Kurtosis	1.38	2.02	–0.074	–0.615	–0.089	–1.37	–0.73	0.345	–0.145	–0.295
Sample size	72,353	72,353	60,173	60,173	9,307	9,307	7,250	7,250	20,353	20,353
Number of effect sizes	226	226	137	137	29	29	22	22	65	65
Pearson’s r (reliabilities)	.344**		.070**		–.196**		–.447**		.139**
Pearson’s r (reliabilities, Fisher’s z scores)	.31**		.040**		–.118**		–.408**		.218**

Note: OCB-O = organizational citizenship behavior toward the organization; POS = perceived organizational support.

OPEN IN VIEWER

Table 6.

Correlation Coefficients, Means, and Standard Deviations for Predictor and Criterion Reliability (Self-Rated and Other-Rated) in the POS Meta-Analysis in Study 2 (Outliers Excluded).

	Criterion Alpha/Predictor Alpha		Criterion Intra/Predictor Alpha		Criterion Alpha/Predictor Alpha		Criterion Intra/Predictor Alpha
	Predictor POS	Criterion Self- Rated Performance	Predictor POS	Criterion Other- Rated Performance	Predictor POS	Criterion Self- Rated OCB-O	Predictor POS	Criterion Other- Rated OCB-O
M	.895	.81	.908	.897	.892	.843	.913	.827
SD	.049	.082	.029	.048	.028	.089	.029	.08
Min	.77	.64	.85	.80	.85	.55	.9	.7
Max	.96	.93	.97	.99	.96	.93	1.0	.9
Mdn	.90	.82	.91	.89	.88	.91	.91	.84
Skew	–0.758	–0.118	–0.280	–0.114	0.69	–1.11	0.125	–0.122
Kurtosis	–0.089	–1.37	–0.379	–0.753	–0.73	0.345	–0.545	–1.396
Sample size	9,307	9,307	10,263	10,263	7,250	7,250	3,885	3,885
Number of effect sizes	29	29	45	45	22	22	19	19
Pearson’s r (reliabilities)	–.196**		–.134**		–.447**		.128**
Pearson’s r (reliabilities, Fisher’s z scores)	–.118**		–.198**		–.408**		.095**

Note: OCB-O = organizational citizenship behavior toward the organization; POS = perceived organizational support.

The main finding of the current study is that the correlations between the predictor and criterion reliability estimates were significantly different from the previously assumed zero correlation. In this second study, we analyzed the correlation between predictor and criterion reliabilities when there is a substantive underlying relationship between the IV and the DV, in our case between POS and some of its antecedents and outcomes. In this context, which represents a typical meta-analysis of organizational phenomena, we generally find much larger correlations between the predictor and criterion reliabilities.

For the analyses that included all data (see Tables 3 and 4), the largest correlation between predictor and criterion reliability of r = –.78 (p < .01; correlation with Fisher’s z-transformed scores: r = –.78) was observed for the POS/self-rated OCB-O relationship (i.e., a predictor and criterion internal consistency reliability combination). Also quite substantial were the correlations between predictor and criterion internal consistency reliability found for the POS/affective commitment relationship at r = .30 (p < .01; correlation with Fisher’s z-transformed scores: r = .29) and for the procedural justice/POS relationship at r = .34 (p < .01; correlation with Fisher’s z-transformed scores: r = .37). While still significant at the p < .01 level, the reliability correlations for the other POS data sets were smaller. The correlation between predictor and criterion internal consistency reliabilities for the POS/job satisfaction relationship was r = .07 (correlation with Fisher’s z-transformed scores: r = .04). For the POS/self-rated performance relationship it was r = –.08 (correlation with Fisher’s z-transformed scores: r = –.04) (i.e., a predictor and criterion internal consistency reliability combination). Examining the two predictor internal consistency and criterion intrarater reliability combinations, the relationship between the reliabilities for the POS/other-rated performance relationship it was r = .12 (correlation with Fisher’s z-transformed scores: r = .01) and for the POS/other-rated OCB-O relationship it was r = .08 (correlation with Fisher’s z-transformed scores: r = .07).

When excluding outliers (see Tables 5 and 6), the correlation between predictor and criterion internal consistency reliabilities for the POS/self-rated OCB-O relationship decreased, but remained very substantial at r = –.45 (p < .01; correlation with Fisher’s z-transformed scores: r = –.41). The correlation between reliabilities for the POS/affective commitment relationship slightly increased to r = .34 (p < .01; correlation with Fisher’s z-transformed scores: r = .31) and decreased for the procedural justice/POS relationship to r = .14 (p < .01; correlation with Fisher’s z-transformed scores: r = .22). There was no change in the correlation between predictor and criterion internal consistency reliabilities for the POS/job satisfaction relationship. The correlation between the reliabilities for the POS/self-rated performance relationship increased to r = –.20 (correlation with Fisher’s z-transformed scores: r = –.12). For the POS/other-rated performance relationship (i.e., a predictor internal consistency and criterion intrarater reliability combination) the correlation between reliabilities changed direction (i.e., we now observed a negative correlation between the reliabilities) and increased in strength to r = –.13 (correlation with Fisher’s z-transformed scores: r = –.20). This is the only combination between POS and a related variable, for which the directionality of the correlation between reliabilities changed when excluding outliers. Finally, when excluding outliers, the correlation between reliabilities for the POS/other-rated OCB-O relationship (i.e., the second predictor internal consistency and criterion intrarater reliability combination) increased to r = .13 (correlation with Fisher’s z-transformed scores: r = .10).

Discussion

In the current study, we have provided empirical evidence that predictor and criterion reliabilities are correlated across studies. While previous research has provided evidence for interdependencies between reliabilities and range restriction (e.g., LeBreton et al., 2003; Mendoza & Mumford, 1987; Theron, 1998), to our knowledge no prior study has examined whether interdependencies also exist between predictor and criterion reliabilities. This is an important question, given that the underlying correction formulae in the Hunter and Schmidt (2004) approach assume independence between all study artifacts.

The current study also provides an estimate of the extent to which predictor and criterion reliabilities are correlated. In Study 1, we computed the correlation between predictor and criterion reliability estimates across all applicable studies published in two of the most highly regarded journals in the field of organization sciences, AMJ and JAP. This was our baseline condition. In Study 2, we reported findings from a meta-analysis on POS and its antecedents and outcomes. We computed the correlation between predictor and criterion reliability estimates for seven different data sets that correlated POS and another variable. In both studies, we found significant correlations between predictor and criterion reliabilities. In Study 1, correlations between predictor and criterion reliabilities in AMJ and JAP ranged from small (i.e., r = .03) to moderate (i.e., r = .16, r = –.12, and r = .12) to large (r = –.45). In Study 2, reliability correlations ranged from small (i.e., r = .07, r = .08, and r = –.08) to moderate (i.e., r = .12) to large (r = .30, r = .34, and r = –.80).

Noteworthy for both studies is that the medians of the reported reliabilities for the predictor and the criterion measures were very high. In Study 1, 2 of the 10 reported median reliability estimates are in the .90s. Another 6 out of the 10 are in the upper .80s, and 1 out of the 10 is in the upper .70s. Only one of the median reliability estimates is below .50. In Study 2, 7 out of the 14 reported median reliability estimates are in the .90s. Another 4 are in the upper .80s, and 2 are in the lower .80s. Only one median reliability coefficient is below .70. This indicates severe truncation in the reliability distributions, which leads to range restriction and subsequently a suppression of the correlations among reliabilities. In other words, we would assume that in studies that include a wider range of reliability indices, we would find even stronger correlations between the reliability coefficients.

What do these correlations mean for the meta-analyses that exist in our field? Related to our discussion of Formula 2 earlier, the Hunter and Schmidt (2004) correction formulae meta-analysts currently use correct the mean observed correlation for predictor and criterion reliability independently. The facts that these two reliability estimates are correlated and that the formulae do not take this correlation into account mean that we correct for their shared portion twice. Thus, we are currently overcorrecting for unreliability. As pointed out above, this is a threat to the accuracy of findings reported by meta-analyses. This is a crucial issue because meta-analysis is seen as a mechanism for the generation of unbiased and accurate estimates of relationships among variables.

However, at this point, we do not know whether the correlation between reliabilities and our failure to account for their interdependence actually leads to a substantial bias in meta-analytically derived corrected mean correlation coefficients. Previous research has provided arguments for two divergent views on whether the difference in estimates might be small or large. For example, Aguinis, Pierce, et al. (2011) argue that technical refinements in meta-analysis usually lead to very small substantive changes in the results and in our subsequent conclusions. Simulation research by Raju et al. (1998) has further shown that even under violations of the assumption of independence between the validity coefficient and study artifacts, the traditional VG models provided corrected mean correlations with reasonable accuracy. So, it could be that taking correlations between predictor and criterion reliabilities into account does not substantially change our findings or conclusions.

At the same time, research into the interdependence between reliabilities and range restriction demonstrates that correlations between study artifacts can have substantial impact on findings and the conclusions we draw from them. Work by LeBreton et al. (2003), for example, shows that range restriction has a large effect on interrater reliability estimates, which are typically rather low (e.g., Viswesvaran et al., 1996). When using interrater reliability indices that are not affected by range restriction, interrater reliability estimates showed levels of between-source agreement similar to within-source agreement. As a consequence, LeBreton et al. suggested that conclusions about the meaning of low interrater reliability estimates for substantive research topics (in their case 360-degree feedback and its interpretation) should be revisited. Correlations between reliabilities could have similarly strong effects onto conclusions drawn from meta-analysis.

In the current article, we find that some of the correlations between reliabilities are quite substantial, that is, in the .30s and .80s. To explore the implications, let us look at the specific example of POS/self-rated OCB-O from our meta-analysis in Study 2. The mean observed correlation between these variables (uncorrected) is .36. The mean reliability for the predictor is .92, the mean reliability for the criterion is .74. These reliabilities are correlated .80, so they share 64% of their variance. So, correcting for predictor unreliability only, the partially corrected correlation between POS and self-rated OCB-O would be .375. Correcting for criterion unreliability in a second step, we would correct the partially corrected correlation to .44, if we were not taking the correlation between the reliabilities into account. However, 64% of the variance between them, we have already accounted for. So, 64% of that additional increase should be too much, that is, .042.

In a second example, let us assume that a meta-analyst was interested in the relationship between learning goal orientation and training performance (assessed by raters). Let us assume that she found a mean observed correlation (uncorrected) between these two variables of .40. In this meta-analysis, the mean predictor reliability was found to be .80. Given that the meta-analyst did not consistently find information about the criterion reliabilities in the original studies, she decided to correct for criterion unreliability by using the estimate for interrater reliability from Viswesvaran et al.’s (1996) study, that is, r = .52. The results of our meta-analysis in Study 1 show that the correlation between predictor internal consistency and criterion interrater reliability is –.45. So, they share about 20% of their variance. Correcting for predictor unreliability, the partially corrected correlation between learning goal orientation and training performance would be .447. Correcting for criterion unreliability in a second step as if the predictor and criterion reliabilities were uncorrelated results in a corrected correlation coefficient of .62. However, 20% of the variance was already accounted for in the predictor unreliability correction. So, that means that we are overcorrecting by .035.

These are two grossly simplified examples. The actual effect of correlated reliabilities on the mean corrected correlation is unlikely to be this simple. However, due to the fact that an actual correction formula that can take the correlations between reliabilities into account has not yet been derived, this is the closest estimate of its actual effect that we can provide.

Arguably, these inflations of the corrected correlation do not seem to be very big. However, given that the core purpose of meta-analysis is to provide the most reliable, accurate, and stable estimates of relationships between variables, any technical refinement that leads to an improvement in our estimation of corrected correlations should be worthwhile. Even if the increase in accuracy is small, it gets us closer to the core purpose of meta-analysis. In addition, one can argue that small effect sizes can sometimes make a large difference. Cortina and Landis (2009) provide a detailed discussion of situations in which the existence of small effects can actually have large implications. They suggest, for example, that in situations in which we expect no effect, evidence for a small effect can be impressive. In addition, in situations in which we expect small effects to accumulate over time (e.g., a few more lives saved from more healthy living practices, or less energy consumed at the end of the year from consciously turning off lights when one leaves the room) small effects can have important outcomes.

In the case of meta-analyses, we might be able to extend these arguments to similar conditions. For example, if a meta-analyst found an effect, even a small one, in situations where research does not expect one, this should be a noteworthy finding. A case in point might be the current study that finds nonzero correlations between predictor and criterion reliabilities when decades of research have assumed there is none. On the flip side, correcting for the correlation between study artifacts might also mean that effect sizes previously reported by meta-analyses, might decrease in size. This would, of course, also be important to know, given that many organizational decisions (such as the design of selection systems or trainings) are informed by meta-analytic evidence. Either way, enabling meta-analysts to make more accurate predictions seems to be a worthwhile cause.

Sources of Interdependence Between Reliability Estimates

James and colleagues (e.g., James et al., 1986; James et al., 1988) have argued that situational factors, such as the organizational settings in which data were collected or assessment biases, could have an influence on the interdependence between study artifacts and validity. We extended the argument to suggest that situational factors could also be the root of interdependencies between predictor and criterion reliabilities. For example, if a large number of primary studies in a meta-analysis were collected from similar samples, with a similar type of assessment, or in very similar settings, this could create an interdependence between the reliabilities across studies. For example, one of the potential influence factors mentioned frequently by prior research is the sample size of the primary study (e.g., James et al., 1986; Raju et al., 1998). Prior research has lamented that, while we correct the observed correlation for sampling error, we do not do the same for the study artifacts, when they are influenced just as much by sampling errors (e.g., Raju et al., 1991). In fact, the influence of sample size would be the same for the predictor and the criterion validity in a given study. To assess this issue in the current study, we ran a post hoc analysis on the data from Study 1, using sample size as a moderator of the predictor/criterion reliability relationship. We found that all relationships between predictor and criterion reliabilities were moderated by sample size (see Table 7). In some cases, while significant, the interaction effect was smaller (e.g., for the predictor internal consistency/criterion interrater reliability combination, the predictor intrarater reliability/criterion internal consistency combination, and for the predictor and criterion intrarater combination). In some cases, however, the effect of sample size on the relationship between predictor and criterion reliability was quite substantive (e.g., for the predictor and criterion internal consistency combination and the predictor internal consistency/criterion intrarater reliability combination). All in all, the role of sampling error is something that future research should investigate further.

OPEN IN VIEWER

Table 7.

Moderated Regressions With Sample Size as the Moderator for the Five Combinations of Predictor and Criterion Reliability.

	Criterion Alpha/Predictor Alpha		Criterion Alpha/Predictor Intrarater		Criterion Intrarater/Predictor Alpha		Criterion Intrarater/Predictor Intrarater		Criterion Interrater/Predictor Alpha
	b	sR 2	b	sR 2	b	sR 2	b	sR 2	b	sR 2
Step 1
Predictor reliability	.23**	.036**	.397**	.057**	.037**	.002**	.17**	.025**	–.374**	.007**
Sample size	–.000003**	.017**	–.000012**	.59**	–.000001**	.007**	–.000001**	.011**	–.00005**	.622**
	R 2	.04**	R 2	.59**	R 2	.007**	R 2	.025**	R 2	.838**
	F(2, 193253)	4190.21	F(2, 19784)	14253.00	F(2, 53982)	202.05	F(2, 13264)	172.11	F(2, 17363)	44832.68
Step 2
Predictor reliability × sample size	.000156**	.112**	–.000387**	.063**	–.000518**	.214**	.002**	.055**	–.00096**	.049**
	R 2	.15**	R 2	.653**	R 2	.222**	R 2	.08**	R 2	.887**
	F(3, 193252)	11686.97	F(3, 19783)	12402.51	F(3, 53981)	5135.48	F(3, 13263)	384.93	F(3, 17362)	45393.82
	▵R 2	.11**	▵R 2	.063**	▵R 2	.215**	▵R 2	.055**	▵R 2	.049**
	▵F(1, 193252)	25571.61	▵F(1, 19783)	3565.53	▵F(1, 53981)	14893.85	▵F(1, 13263)	790.11	▵F(1, 17362)	7547.06

Note: All variables were centered prior to analysis.

**p < .01.

Further research design characteristics that might create correlations between predictor and criterion reliabilities across studies could include the number of items on the predictor and criterion tests, the number of raters to assess interrater or intrarater reliability, the type of sample (e.g., student vs. organizational), or whether information for the predictor and the criterion measure were collected from the same source or from different sources. These are just a few examples for how research design characteristics might be at the root of interdependencies between predictor and criterion reliabilities across studies. Future research will need to assess the impact of each of these factors.

Conclusion

Many researchers look to meta-analysis to answer questions that cannot be answered in one single study. Because meta-analysis results frequently have lasting implications for research and practice, it is imperative that the results are as accurate as can be. Therefore, our findings that study artifacts are not independent across studies, as previously assumed, have important implications for researchers conducting meta-analyses. Taking the correlation between the reliabilities into account is important to not overcorrect the observed mean effect size. In this article, we have provided an estimate of the extent to which predictor and criterion reliabilities are correlated. In addition, we have provided implications for future research to generate a solution to the problem and suggested guidelines for the reporting of artifact correlations in future meta-analyses to provide an assessment of the robustness of underlying assumptions of study artifact interdependence.