Correlation Attenuation Due to Measurement Error

Abstract

Issues with correlation attenuation due to measurement error are well documented. More than a century ago, Spearman proposed a correction for attenuation. However, this correction has seen very little use since it can potentially inflate the true correlation beyond one. In addition, very little confidence interval (CI) research has been done for correction for attenuation. In the present study, the authors propose a bootstrap procedure for estimating the deattenuated correlation and corresponding CIs. The authors use Monte Carlo simulations to generate data under certain conditions and assess the performance of the bootstrapped deattenuated correlation. The authors investigate for bias and 95% CI coverage. Results indicate that the bootstrap deattenuated correlation provided adequate percentile CI coverage in all but three conditions. The bias-corrected and accelerated CI, however, provided adequate coverage under all simulation conditions.

Keywords

bootstrap confidence interval deattenuated correlation attenuation measurement error reliability

Social/behavioral research ranges in scope and specificity, but it shares an underlying characteristic: In one way or another, researchers must work with variables measured with error. Such variables come in the form of unobservable variables such as anxiety, depression, IQ, and so on, which cannot be directly measured. Thus, each of these variables must be indirectly measured through items at which point measurement error is introduced. These types of variables are typically known as factors or latent variables. Furthermore, working with these variables frequently involves estimating relationships between them. The fundamental statistic for estimating relationships between continuous variables is the Pearson product–moment correlation coefficient (or correlation coefficient). In fact, it is the driving force behind general linear models (GLMs; e.g., regression, analysis of variance [ANOVA]) and is a benchmark measure of effect size. In general, researchers desire a strong correlation because it is a clear demonstration of a relationship. However, all statistical models assume that the variables in the model are measured without error, and none are robust to this assumption. As such, the correlation is not immune to this assumption and the issues it creates can propagate to other methodologies that are based on the correlation. Although the impact of measurement error on the correlation has been known for more than a century, to date very little work has been done to address and minimize its impact. In light of this we propose a bootstrap methodology with accompanying confidence intervals (CIs) as a viable solution to address the effect of measurement error on the correlation and investigate its properties under several simulation conditions.

General Overview

Researchers in the social/behavioral sciences are often unable to find strong empirical support for theories that otherwise appear plausible (Buckley, Cote, & Comstock, 1990). Failure to find empirical support may be explained from a theoretical view. Specifically, the model may not account for all factors influencing the variability on the outcome of interest. Other researchers propose that method effects and measurement error are more plausible explanations for discrepancies across research studies (Buckley et al., 1990; Kassarjian, 1971; Lance, Dawson, Birkelbach, & Hoffman, 2010). These researchers emphasize a relationship between poor validity of a measure that may be due to method effects (Fiske, 1987), and measurement error that is rarely accounted for in the literature (Schmidt & Hunter, 1996). For example, a measure may be developed for a specific population but is then used with a general population resulting in poor correlation among variables (Buckley et al., 1990; Kassarjian, 1971). Such studies generally conclude a poor relationship between variables instead of investigating the validity of the measure with the population investigated and/or correcting for measurement error.

Unfortunately, measurement error can cause a number of serious problems in statistical models all of which lead to inaccurate inferences or conclusions. For example, when the ubiquitous correlation coefficient is calculated, measurement error in either variable will attenuate the correlation, making the relationship appear weaker than it would be without measurement error (Allen & Yen, 1979; Carroll, Ruppert, Stefanski, & Crainiceanu, 2006). In regression models, measurement error in the independent variables can result in coefficients that are smaller, larger, or even have a different sign than the coefficients that would occur if there were no measurement error. See Hobfoll, Shoham, and Ritter (1991) and Thomas and Williams (1991) for examples. In addition, measurement error in the dependent variable can increase the residual variance, reduce power, and decrease R² (Carroll et al., 2006).

The central statistic to GLMs is the correlation coefficient. When interest is in estimating relationships between predictor(s) and a criterion, a regression model is estimated. When interest is in the effect of an independent variable on the dependent variable, an ANOVA model is estimated. Each of these models is a form of the GLM, and an underlying correlation between the variables is assumed in each case. As such, not only does the correlation determine the magnitude of the GLM parameter estimates, it can also drive a researcher’s decision on what variables to include in the model. As previously noted, however, measurement error in the variables negatively affects GLM parameter estimates and corresponding inferences.

An inaccurate correlation coefficient with corresponding CIs can also affect research that synthesizes other research findings such as a meta-analysis. In this type of analysis, the correlation coefficient is a benchmark measure of the effect size and is used as a data point. Even if the correlation for every study is based on a reliability of .90 for the variables, it still indicates that 10% of the variance is due to measurement error. When all these studies are synthesized through a meta-analysis, the 10% error for each correlation quickly propagates through the analysis which may result in a situation where empirical results do not support plausible theories (Buckley et al., 1990). In fact, a recent meta-analytic article indicated that “for the PTSD–psychological aggression association, 98% of the variance was accounted for by methodological artifacts such as sampling and measurement error” (Taft, Watkins, Stafford, Street, & Monson, 2011, p. 22).

Thus, getting an accurate estimate of even a fundamental statistic as the correlation coefficient is essential because it is the building block to more advanced statistical models and is commonly used to synthesize research results through a meta-analysis. If the correlation coefficient with corresponding CIs is inaccurate because of measurement error, it will negatively affect the parameter estimates and inferences of statistical models based on the correlation coefficient and meta-analytic research in which it is used as a data point.

Correlation Attenuation

The correlation coefficient is biased as a result of measurement error (Charles, 2005; Fan, 2003; Mutch & Tisak, 2005; Schmidt & Hunter, 1996; Tarkkonen & Vehkalahti, 2005; You & Chen, 2006). The magnitude of the bias depends on the reliabilities of the measurement instruments used to represent the variables: Poor reliability estimates attenuate the correlation and, consequently, the effect size (Fan, 2003).

The lack of consistency in measuring variables (i.e., low reliability) results in obtaining a weaker (or attenuated) relationship between the variables than may be true in the population (Fan, 2003; Fleiss & Shrout, 1977; H. G. Johnson, 1944; Kirkpatrick, 1997; Murphy, 2008; Mutch & Tisak, 2005; Pedhazur, 1997; Winne & Belfry, 1982; Zimmerman & Williams, 1997). The relationship between the correlation coefficient and reliability can be seen by the equation for the deattenuated correlation coefficient (Spearman, 1904):

ρ_{TyTx} = \frac{ρ_{yx}}{\sqrt{ρ_{yy'}} \sqrt{ρ_{xx'}}},

where ρ_TxTy is the true correlation, ρ_xy is the observed correlation, and ρ_yy′ and ρ_xx′ are the reliabilities for y and x, respectively. To estimate the deattenuated correlation one just replaces the parameters with the parameter estimates as follows:

{\hat{ρ}}_{TyTx} = \frac{{\hat{ρ}}_{xy}}{\sqrt{{\hat{ρ}}_{yy'}} \sqrt{{\hat{ρ}}_{xx'}}} .

Note that Equations (1) and (2) are valid for continuous and ranked (Spearman, 1904) variables.

According to Equations (1) and (2), the true correlation equals the observed correlation when there is no measurement error in x and y, that is, the reliabilities of both variables equal one (ρ_yy = ρ_xx = 1). When this does not hold, any measurement error variance in the variables results in correlation attenuation (Charles, 2005; Zimmerman & Williams, 1997). To correct for attenuation, accurate reliability estimates are required. According to Winne and Belfry (1982), the most common reliability coefficients are the Spearman–Brown split-half correlation and coefficient alpha (or Cronbach’s alpha). Assuming that the set of items in the measurement instrument are at least tau-equivalent or essentially tau-equivalent, coefficient alpha is equal to reliability (Graham, 2006; Lord, Novick, & Birnbaum, 1968; McDonald, 1999; Zinbarg, Revelle, Yovel, & Li, 2005). If the items are congeneric, coefficient alpha underestimates reliability, but coefficient omega is equal to reliability of the measurement instrument (McDonald, 1999; Zinbarg et al., 2005). Furthermore, both coefficients alpha and omega are equal to the reliability of the set of items when they are at least tau-equivalent. As noted by Winne and Belfry, using an underestimated reliability in Equations (1) and (2) results in overestimation of the correlation.

The two major psychometric components of a measurement instrument are validity and reliability (Allen & Yen, 1979; Crocker & Algina, 1986; Tarkkonen & Vehkalahti, 2005). Validity is concerned with whether the instrument measures what it purports to measure. Reliability, on the other hand, refers to the consistency or precision of the measure and is derived from the classical true score model as

x = τ + u,

where x is an observed score, τ is a true score, and u is a measurement error (Allen & Yen, 1979; Crocker & Algina, 1986; Tarkkonen & Vehkalahti, 2005). The true score model further assumes no correlation between the true score and the error, which allows for the separation of true score variance from the error variance,

σ_{x}^{2} = σ_{τ}^{2} + σ_{u}^{2},

where $σ_{x}^{2}$ is the observed variance, $σ_{τ}^{2}$ is the true variance, and $σ_{u}^{2}$ is the measurement error variance. By definition, reliability is the ratio of the true variance over the observed variance of the measurement instrument:

ρ_{xx'} = \frac{σ_{τ}^{2}}{σ_{x}^{2}} = \frac{σ_{τ}^{2}}{σ_{τ}^{2} + σ_{ε}^{2}} .

Tarkkonen and Vehkalahti (2005) indicate that the majority of traditional studies focus on sampling error, often ignoring measurement error and, according to Fan (2003) and Schmidt and Hunter (1996), also underreport obtained reliability estimates. Both sampling and measurement error are mutually exclusive but both have a unique effect on the outcome. To see further derivations and an explanation of the measurement error variance, the interested reader is referred to Tarkkonen and Vehkalahti (2005).

The issue with bias due to measurement error carries over to path models. Recognizing the effect of correlation attenuation on their proposed teaching and learning path model, Stayrook and Corno (1979) corrected the path coefficients for attenuation. Unlike other discussed studies that applied the classical true score model, Stayrook and Corno followed generalizability theory, a more general extension of the classical true score model, where the true (or universe) score is “the average of all possible representative observations” (p. 229). In other words, the true score is the mean of the sampling distribution composed of representative observations. After correcting for attenuation, these researchers noted changes in their path coefficients that were more congruent with their hypotheses. Although a couple of path coefficients decreased in value, the magnitude of this change was not large and the model fit improved.

A clear interpretation from the discussion thus far can be made: Measurement error affects statistical results and corresponding conclusions. Despite the serious implications of measurement error, however, some studies still fail to report reliability estimates or refute the possibility of measurement error (Schmidt & Hunter, 1996). According to Schmidt and Hunter, there is yet to be a psychological variable that could derive an absolute value. Most psychological measures thus far developed do not account for 100% of the variability, leaving room for measurement error. Schmidt and Hunter further point out that even when reliability estimates are reported, researchers do not correct their results for measurement error, or if corrections are applied this is often done incorrectly. The researchers believe this confusion is due to lack of concrete examples as to when and how to use correction for attenuation.

To further illustrate their point, Schmidt and Hunter (1996) present a number of scenarios depicting common misconceptions and errors in judgment. For example, in one scenario the researcher in question assumes that measurement error and sampling error are one and the same. As previously noted the two are independent and refer to different research issues. Measurement error refers to an inconsistency in assessing a psychological construct; the bias is minimized when items generated to assess one factor are highly correlated, which implies that items measure the same construct. On the other hand, sampling error refers to the difference between a sample statistic and a population parameter. Sampling bias is a possible source of sampling error: Specifically, having overrepresentation or underrepresentation of some members of the population can bias a statistic.

Another common mistake noted by Schmidt and Hunter (1996) is the misconception associated with an adequate reliability estimate of .70 (Nunnally, 1978). Even if both variables have reliability of .70, “the bias factor is $\sqrt{. 70 (. 70)} = . 70$ ” (Schmidt & Hunter, 1996, p. 204). This indicates that 30% of the variance between the two variables is measurement error. Overall, the researchers point out that there is no measurement instrument free of error, at least not in the social/behavioral sciences. They also point out that in order for theory and cumulative knowledge to progress, measurement error in research must be considered and handled carefully.

Methodological Studies

Recent methodological research has investigated correcting measurement error in the correlation coefficient. In one such study, Zimmerman and Williams (1997) conducted a Monte Carlo simulation to analyze the properties of Spearman’s deattenuated correlation. The conditions of the simulation were as follows: distribution shape of the data, sample size, reliability coefficient, and population correlation. The researchers noted that the deattenuated correlation was unbiased, but only when the reliability of the variables was high (e.g., ≥.70). In contrast, if the reliability was low the deattenuated correlation greatly exceeded the true correlation. This was exacerbated with small samples (e.g., n = 20) in which case a deattenuated correlation greater than one became frequent. In addition, Zimmerman and Williams noted that the variability of corrected correlations increased as a function of poor reliability estimates, which may result in the overestimation of the association between the two variables. Zimmerman and Williams did not investigate CI coverage.

In another investigation, Charles (2005) addressed the criticisms associated with the deattenuated correlation. As was previously discussed, correcting for attenuation when using poor reliability estimates results in overestimation of the correlation, which can result in a corrected correlation coefficient that goes beyond ±1. Instead of abandoning the attenuation correction method, Charles proposed the use of confidence sets. Unlike CIs, the idea behind confidence sets is that interest lies in finding population bounds likely to have generated the obtained sample as opposed to CIs which reflect sampled values likely to have been generated by a given population. According to Charles, corrected correlations consist of true correlations and nuisance correlations (see Charles, 2005, for details).

Charles (2005) investigated confidence sets for correction for attenuation due to measurement error. The confidence sets were derived from regressing the sampling distribution standard deviation from the deattenuated correlation on the sample size, geometric mean of reliability, square of true-score correlation, and their two- and three-way interactions. The regression equation accounted for 99.6% of the deattenuated correlation variability. The method performed reasonably well under the investigated conditions. However, it has seen little use in the applied literature. One reason may be that the method is counterintuitive to what researchers are used to in that statistics are used to estimate population parameters as opposed to finding a range of populations that generate the sample data.

The purpose of the current study is to assess the performance of the bootstrapped deattenuated correlation and corresponding CIs based on Spearman’s original correlation correction. In particular, interest lies in simulation conditions similar across earlier studies. The bootstrapped deattenuated correlation was assessed for bias and interval estimation. We start with the bootstrapped deattenuated correlation.

Bootstrapped Deattenuated Correlation

The bootstrapped deattenuated correlation for a pair of variables x and y can be summarized in three steps. Suppose X = (x₁, x₂, . . ., x_n) is the observed sample where each x_i = (y_i, x_i) is a 1 × 2 vector. First, obtain a bootstrap sample $X^{(b)} = (x_{1}^{(b)}, x_{2}^{(b)}, \dots, x_{n}^{(b)})$ , which is a bth random resample from X with replacement. Note that X and X^(b) have the same sample size. Second, compute the bth bootstrap estimate of the deattenuated correlation as

{\hat{ρ}}_{TyTx}^{(b)} = \frac{{\hat{ρ}}_{yx}^{(b)}}{\sqrt{{\hat{ρ}}_{yy'}} \sqrt{{\hat{ρ}}_{xx'}}}

from X^(b). Last, ${\hat{ρ}}_{TyTx}^{(1)}, {\hat{ρ}}_{TyTx}^{(2)}, \dots, {\hat{ρ}}_{TyTx}^{(B)}$ represents the empirical sampling distribution (ESD) for ${\hat{ρ}}_{TyTx}$ for b = 1, 2, . . ., B bootstrap samples. The ESD can then be summarized to obtain statistical quantities associated with ${\hat{ρ}}_{TyTx}$ for statistical inference.

The bootstrap estimate of standard error (SE) for the deattenuated correlation is

SE ({\hat{ρ}}_{TyTx}) = {[\frac{1}{B - 1} \sum_{b = 1}^{B} ({\hat{ρ}}_{TyTx}^{(b)} - {\bar{ρ}}_{TyTx})]}^{1 / 2},

where b is the bootstrap replicate, ${\hat{ρ}}_{TyTx}^{(b)}$ is the estimated deattenuated correlation for the bth bootstrap replicate, and

{\bar{ρ}}_{TyTx} = \frac{1}{B} \sum_{b = 1}^{B} {\hat{ρ}}_{TyTx}^{(b)} .

There are several types of bootstrap CIs that can be constructed, but here we will focus on the percentile bootstrap (PB) and the bias-corrected and accelerated bootstrap (BCa) CIs because the sampling distribution of the correlation coefficient is not normal (N. C. Johnson, Kotz, & Balakrishnan, 1995). The PB CI can be constructed by computing the α/2 and 1 −α/2 percentiles from the ${\hat{ρ}}_{TyTx}$ ESD where α is the significance level. For further technical and theoretical details about the BCa CI or the bootstrap in general, see Efron and Tibshirani (1993).

In the current study, these methods were used to estimate 100(1 –α)% CIs for the deattenuated correlation under several simulation conditions, where α = .05. In addition, all corresponding bootstrap quantities were estimated from a total of 1,000 bootstrap samples.

Method

Simulation

A Monte Carlo simulation was used to investigate the properties of the bootstrapped deattenuated correlation CIs. Specifically, the impact of three simulation factors was investigated in a 5 (correlation magnitude) × 5 (reliability) × 6 (sample size) simulation design. All data were simulated assuming multivariate normality. To increase the generality of the results, data were simulated under similar scenarios investigated in earlier studies (Charles, 2005; Zimmerman & Williams, 1997). In this study, 1,000 replications were obtained for each combination of the Monte Carlo simulation.

Data were simulated as follows. First, generate an n×k multivariate true variable data T = (τ_i1,τ_i2) such that T~N(0, P), where i = 1, 2, . . ., n, j = 1, 2, . . ., k, and P is the k×k true correlation matrix. Note that k = 2. Second, the observed variables were generated as

x_{ij} = τ_{ij} + u_{ij},

where $u_{ij} ~ N (0, f_{i} (ρ_{jj}))$ . The variance of u_ij is defined as

f_{i} (ρ_{jj}) = \frac{var (τ_{ij}) (1 - ρ_{jj})}{ρ_{jj}},

where ρ_ij is the reliability of jth variable. The following simulation conditions were investigated.

Correlation (ρ)

Small, medium, and large correlations, as defined by (Cohen, 1988), were investigated. The Pearson product–moment correlation (ρ) measures the magnitude of a relationship between the two continuous variables and can be interpreted as an effect size (ρ²). Cohen recommends the following interpretation of the correlation coefficient: ρ = .10, .30, .50 represents a small, moderate, and strong correlation, respectively. To broaden the investigation, we focused on the correlation coefficients ranging from .10 to .50 in increments of .10.

Reliability (ρ_ij)

The reliability of both variables in the correlation was manipulated. The rule of thumb in the behavioral sciences is to consider a reliability coefficient of .70 as an adequate reliability estimate (Nunnally, 1978). Nonetheless, we investigated reliability estimates ranging from .50 to .90 in increments of .10, which mimic scenarios similar to what is often seen in applied research.

Sample size (n)

The following sample sizes were investigated: n = 50, 100, 150, 200, 250, 300. These are typical sample sizes found in social/behavioral research.

Procedure

In each replication, the deattenuated correlation was computed along with the corresponding bootstrap 95% CIs. Relative bias for the deattenuated correlation was computed as

{\hat{ρ}}_{bias} = \frac{{\hat{ρ}}_{TyTx} - ρ}{ρ},

where ρ is the true population correlation. The average of the estimated SE was computed as

\bar{SE} = \frac{\sum_{i = 1}^{rep} SE ({\hat{ρ}}_{T_{y} T_{x}, i})}{rep},

where rep is the number of replications and $SE ({\hat{ρ}}_{T_{y} T_{x}, i})$ is the bootstrap SE for the bootstrapped deattenuated correlation for each replication.

CI coverage was assessed using Bradley’s (1978) liberal criterion, which is defined as 1 − 1.5α≤ 1 −ρ≤ 1 − 0.5α (Romano, Kromrey, & Hibbard, 2010). Coverage is defined as the proportion of estimated CIs that contain the true population correlation. Thus, with α = .05 acceptable coverage falls within the interval [.925, .975].

Results

Point Estimate Bias

The focus of the study is the bootstrapped deattenuated correlation CIs; however, parameter estimate bias was also investigated since it can have an effect on the CIs. Tables 1 through 5 display the results and there is no observable bias. In fact, the magnitude of the bias was .00 ± .07. However, the quantities for bias behave in the anticipated manner: Bias decreases as sample size and/or reliability increases.

Table 1.

95% CI Coverage for Reliability by Sample Size at .10 Correlation

Reliability	n	Relative bias	RMSE	Percentile CI	BCa CI
.50	50	−.0260 (.2865)	.2865	.9300	.9330
	100	.0739 (.1984)	.1986	.9450	.9470
	150	.0059 (.1626)	.1626	.9460	.9430
	200	−.0003 (.1378)	.1378	.9530	.9560
	250	.0365 (.1301)	.1301	.9490	.9460
	300	.0004 (.1132)	.1132	.9490	.9480
.60	50	−.0032 (.2430)	.2430	.9210	.9250
	100	−.0751 (.1677)	.1679	.9410	.9430
	150	.0403 (.1363)	.1364	.9330	.9350
	200	−.0061 (.1194)	.1194	.9490	.9440
	250	.0095 (.1027)	.1027	.9470	.9480
	300	−.0026 (.0963)	.0963	.9460	.9480
.70	50	−.0303 (.2018)	.2018	.9380	.9430
	100	−.0237 (.1421)	.1421	.9410	.9450
	150	.0140 (.1231)	.1231	.9310	.9370
	200	−.0221 (.0980)	.0980	.9490	.9490
	250	−.0016 (.0883)	.0883	.9590	.9610
	300	−.0499 (.0827)	.0829	.9460	.9470
.80	50	.0216 (.1809)	.1810	.9340	.9450
	100	.0189 (.1291)	.1291	.9370	.9360
	150	.0398 (.1050)	.1051	.9420	.9400
	200	.0250 (.0878)	.0878	.9450	.9410
	250	.0340 (.0777)	.0778	.9440	.9440
	300	−.0185 (.0714)	.0714	.9520	.9550
.90	50	−.0713 (.1527)	.1529	.9430	.9390
	100	.0092 (.1152)	.1152	.9280	.9340
	150	−.0311 (.0891)	.0892	.9480	.9470
	200	−.0343 (.0766)	.0767	.9530	.9500
	250	−.0287 (.0693)	.0694	.9510	.9520
	300	.0033 (.0628)	.0628	.9460	.9480

Note: CI = confidence interval; RMSE = root mean square error; BCa = bias-corrected and accelerated bootstrap. Numbers in parentheses are standard errors; CIs in boldface fall outside the interval [.925, .975]. All CIs are based on 1,000 bootstrap samples.

Table 2.

95% CI Coverage for Reliability by Sample Size at .20 Correlation

Reliability	n	Relative bias	RMSE	Percentile CI	BCa CI
.50	50	−.0037 (.2824)	.2824	.9350	.9460
	100	.0183 (.1943)	.1943	.9490	.9480
	150	.0075 (.1674)	.1674	.9290	.9320
	200	−.0059 (.1388)	.1388	.9490	.9490
	250	.0178 (.1286)	.1286	.9410	.9390
	300	.0034 (.1207)	.1207	.9350	.9380
.60	50	−.0479 (.2404)	.2406	.9280	.9350
	100	.0074 (.1660)	.1660	.9380	.9380
	150	.0190 (.1323)	.1324	.9460	.9440
	200	−.0224 (.1176)	.1177	.9440	.9460
	250	−.0083 (.1057)	.1057	.9470	.9450
	300	.0137 (.0934)	.0935	.9490	.9520
.70	50	−.0599 (.2055)	.2059	.9280	.9310
	100	−.0108 (.1386)	.1386	.9430	.9500
	150	−.0031 (.1123)	.1123	.9480	.9470
	200	−.0223 (.0969)	.0970	.9510	.9490
	250	−.0304 (.0901)	.0903	.9440	.9480
	300	−.0099 (.0832)	.0832	.9410	.9410
.80	50	.0198 (.1710)	.1710	.9340	.9400
	100	.0022 (.1220)	.1220	.9400	.9410
	150	−.0064 (.0982)	.0982	.9540	.9530
	200	−.0116 (.0831)	.0831	.9480	.9480
	250	−.0031 (.0755)	.0755	.9470	.9510
	300	−.0042 (.0683)	.0683	.9460	.9500
.90	50	.0164 (.1534)	.1534	.9380	.9440
	100	.0179 (.1104)	.1105	.9300	.9330
	150	−.0150 (.0879)	.0879	.9470	.9500
	200	.0092 (.0770)	.0770	.9430	.9450
	250	−.0026 (.0678)	.0678	.9440	.9470
	300	.0123 (.0631)	.0631	.9470	.9420

Table 3.

95% CI Coverage for Reliability by Sample Size at .30 Correlation

Reliability	n	Relative bias	RMSE	Percentile CI	BCa CI
.50	50	−.0411 (.2861)	.2863	.9240	.9270
	100	.0139 (.1938)	.1939	.9460	.9470
	150	−.0121 (.1578)	.1578	.9480	.9480
	200	−.0021 (.1391)	.1391	.9440	.9400
	250	−.0152 (.1221)	.1222	.9520	.9530
	300	.0030 (.1146)	.1146	.9510	.9510
.60	50	.0150 (.2276)	.2277	.9350	.9400
	100	−.0041 (.1641)	.1641	.9430	.9440
	150	−.0219 (.1272)	.1274	.9460	.9480
	200	−.0054 (.1116)	.1117	.9540	.9560
	250	−.0025 (.1015)	.1015	.9440	.9460
	300	−.0026 (.0963)	.0963	.9450	.9420
.70	50	−.0236 (.2040)	.2041	.9240	.9290
	100	−.0009 (.1387)	.1387	.9360	.9680
	150	.0025 (.1124)	.1124	.9480	.9530
	200	.0090 (.0954)	.0954	.9470	.9410
	250	−.0058 (.0856)	.0857	.9480	.9490
	300	−.0062 (.0754)	.0754	.9610	.9620
.80	50	−.0151 (.1690)	.1691	.9470	.9440
	100	−.0193 (.1125)	.1126	.9580	.9610
	150	−.0126 (.0979)	.0980	.9430	.9480
	200	.0008 (.0835)	.0835	.9380	.9400
	250	−.0103 (.0734)	.0735	.9500	.9530
	300	−.0023 (.0687)	.0687	.9490	.9480
.90	50	−.0249 (.1449)	.1451	.9470	.9510
	100	−.0343 (.1052)	.1057	.9400	.9400
	150	−.0073 (.0844)	.0844	.9450	.9440
	200	.0030 (.0717)	.0717	.9600	.9580
	250	.0023 (.0664)	.0664	.9460	.9430
	300	.0065 (.0592)	.0593	.9520	.9500

Table 4.

95% CI Coverage for Reliability by Sample Size at .40 Correlation

Reliability	n	Relative bias	RMSE	Percentile CI	BCa CI
.50	50	−.0270 (.2812)	.2814	.9330	.9390
	100	.0009 (.1946)	.1946	.9410	.9390
	150	−.0097 (.1533)	.1533	.9520	.9530
	200	.0148 (.1326)	.1327	.9550	.9520
	250	−.0025 (.1154)	.1154	.9570	.9580
	300	−.0043 (.1116)	.1116	.9520	.9510
.60	50	−.0130 (.2181)	.2181	.9380	.9400
	100	.0102 (.1554)	.1555	.9430	.9450
	150	.0036 (.1260)	.1260	.9520	.9520
	200	−.0067 (.1131)	.1132	.9410	.9460
	250	.0128 (.1010)	.1012	.9360	.9350
	300	−.0005 (.0884)	.0884	.9480	.9530
.70	50	−.0091 (.1960)	.1961	.9270	.9260
	100	−.0127 (.1293)	.1264	.9560	.9550
	150	.0087 (.1085)	.1085	.9440	.9450
	200	.0021 (.0917)	.0917	.9480	.9490
	250	−.0049 (.0841)	.0841	.9500	.9510
	300	.0135 (.0747)	.0749	.9620	.9610
.80	50	−.0099 (.1636)	.1637	.9320	.9410
	100	−.0034 (.1124)	.1124	.9450	.9490
	150	−.0067 (.0923)	.0923	.9480	.9460
	200	−.0151 (.0773)	.0776	.9510	.9510
	250	−.0088 (.0712)	.0713	.9440	.9450
	300	−.0033 (.0651)	.0651	.9440	.9450
.90	50	−.0161 (.1407)	.1407	.9280	.9340
	100	−.0132 (.0971)	.0972	.9400	.9420
	150	.0107 (.0774)	.0775	.9480	.9450
	200	−.0074 (.0691)	.0691	.9450	.9460
	250	−.0033 (.0619)	.0619	.9400	.9410
	300	−.0042 (.0573)	.0573	.9460	.9470

Table 5.

95% CI Coverage for Reliability by Sample Size at .50 Correlation

Reliability	n	Relative bias	RMSE	Percentile CI	BCa CI
.50	50	−.0028 (.2637)	.2637	.9350	.9370
	100	.0010 (.1787)	.1787	.9600	.9580
	150	−.0112 (.1529)	.1530	.9420	.9440
	200	−.0067 (.1288)	.1289	.9530	.9580
	250	.0005 (.1227)	.1227	.9340	.9380
	300	−.0103 (.1085)	.1086	.9550	.9530
.60	50	−.0162 (.2182)	.2184	.9360	.9390
	100	−.0192 (.1505)	.1508	.9460	.9500
	150	.0057 (.1266)	.1266	.9400	.9400
	200	−.0018 (.1066)	.1067	.9380	.9420
	250	.0013 (.0957)	.0957	.9530	.9560
	300	.0029 (.0866)	.0866	.9450	.9440
.70	50	−.0226 (.1806)	.1810	.9330	.9380
	100	−.0019 (.1252)	.1252	.9430	.9430
	150	−.0031 (.1025)	.1025	.9370	.9430
	200	−.0028 (.0878)	.0878	.9450	.9450
	250	−.0069 (.0813)	.0814	.9400	.9410
	300	−.0043 (.0723)	.0724	.9550	.9540
.80	50	−.0090 (.1479)	.1480	.9390	.9430
	100	−.0062 (.1032)	.1032	.9420	.9480
	150	.0016 (.0837)	.0837	.9410	.9440
	200	−.0080 (.0742)	.0743	.9430	.9460
	250	.0051 (.0669)	.0667	.9440	.9440
	300	−.0015 (.0613)	.0613	.9430	.9440
.90	50	−.0197 (.1304)	.1308	.9290	.9320
	100	−.0014 (.0873)	.0873	.9460	.9480
	150	−.0005 (.0746)	.0746	.9350	.9380
	200	−.0035 (.0641)	.0641	.9400	.9420
	250	−.0000 (.0563)	.0563	.9500	.9480
	300	−.0006 (.0529)	.0529	.9490	.9500

Standard Error and Root Mean Squared Error

Although the SE is not technically required for the PB or BCa CI, it gives an indication of the precision of the deattenuated correlation. The results in Tables 1 through 5 indicate that the deattenuated correlation has very little variability. In addition, the SE becomes smaller as the sample size and reliability increase, that is, the deattenuated correlation becomes more precise. The root mean square error (RMSE) is similar to the SE in magnitude and behaves in a similar manner, that is, the RMSE becomes smaller as the sample size and reliability increase.

Confidence Intervals

For the majority of simulation conditions, both the PB and BCa CIs met Bradley’s criteria (Bradley, 1978). Results are shown in Tables 1 through 5. In only the following three instances with a sample size of 50, the PB CIs fell slightly below the lower bound of Bradley’s criteria: (a) when the correlation was .10 and reliability was .60 and (b) when the correlation was .30 and the reliability was .50 or .70. However, the BCa CIs met Bradley’s criteria in all conditions. The slight underestimation of the PB CIs in the three conditions indicated above could be due to the PB CI requiring a sample size larger than 50 and/or could be due to simulation (or sampling) error.

Figures 1 and 2 display the 95% CI coverage for each method over sample size. As can be seen, both the PB and BCa CIs had similar coverage probabilities. However, the BCa CIs had more consistent estimation in that they had less variability and thus were narrower.

Figure 1.

Percentile bootstrap 95% confidence interval coverage over sample size

Figure 2.

Bias-corrected and accelerated 95% confidence interval coverage over sample size

Discussion

Measurement error in the social/behavioral sciences is common because of the nature of the latent variables of interest in these fields such as anxiety, depression, IQ, and so on. These types of variables exist in potential but cannot be directly measured. Thus, they must be indirectly measured through items in a measurement instrument at which point the measurement error is introduced. However, measurement error in these types of variables can hinder the correlation between them which can propagate to other statistics and methodologies that are based on the correlation such as GLMs and meta-analysis (Buckley et al., 1990; Carroll et al., 2006; Kassarjian, 1971; Lance et al., 2010; Schmidt & Hunter, 1996; Taft et al., 2011). Thus, it is important to get an accurate correlation estimate and accompanying CIs when there is measurement error in the variables.

Although the effects of measurement error on the correlation coefficient, and potential solution by way of the deattenuated correlation, have been known for over a century (Spearman, 1904), to date very little of it is used in applied work (Charles, 2005; Schmidt & Hunter, 1996). The main issues hindering its use are that CIs for it have been difficult to develop and that the deattenuated correlation can potentially be greater than one. Several recent studies have been conducted to investigate these issues further (Charles, 2005; Zimmerman & Williams, 1997). Although the first study had promising results when reliability and sample size were .70 and 100 or greater, respectively, it did not investigate CIs. The second study investigated confidence sets as opposed to CIs for the deattenuated correlation. This method also showed promise; however, the lower bound had accuracy issues with high correlations (e.g., r≥ .70) and small sample sizes (e.g., n≤ 100). Even so, the methodology has seen little use in applied research.

Here we introduce a bootstrapped deattenuated correlation coefficient with corresponding CIs and assessed its statistical properties through simulation. Simulation conditions investigated included true population correlation, sample size, and reliability of both variables. According to the results, the BCa CIs were acceptable under all 150 conditions of the simulation. The PB CIs had unacceptable coverage in three simulation conditions: The conditions included a sample size of 50, reliability of .70 or less, and correlation between .10 and .30. For the conditions investigated, the BCa method showed excellent promise whereas some caution is warranted for the PB method with sample size of 50 or less and reliability of .70 or less. Thus, the bootstrapped deattenuated correlation is an adequate method for correcting the attenuated correlation due to measurement error and provides good estimates of corresponding CIs.

Bootstrap methodology is a flexible technique that can be used to solve a variety of statistical problems, but to our knowledge it has not been used to address measurement error in the correlation coefficient. In this capacity, the bootstrap holds much promise with three advantages. First, within this investigation, the bootstrap remained relatively unbiased and did not exceed the correlation bounds (±1), which was the primary criticism in earlier studies. Second, the bootstrap provided the ESD for the deattenuated correlation, which was recommended by Charles (2005) and “. . . difficult, if not impossible . . .” according to Hakstian, Schroeder, and Rogers (1988, p. 27). As such, the bootstrapped deattenuated correlation did not have any lower and/or upper bound accuracy issues. Third, the bootstrap deattenuated correlation can be extended to GLMs. The parameter estimates of any GLM can be obtained through the covariance (correlation) matrix and mean vector for a set of variables (for details, see Timm, 2002). In this respect, one replaces the covariance matrix with the deattenuated covariance matrix at each bth step from which the GLM parameter estimates are obtained. From here, the ESD for each GLM parameter estimate is generated and can be used to make inferences.

A limitation here is that the bootstrapped deattenuated correlation’s performance was assessed using normally distributed data. Although the normality assumption dominates research in social/behavioral sciences, there is no indication that any proposed CI for the deattenuated correlation, including the one here, is robust to the normality assumption (Charles, 2005; Zimmerman & Williams, 1997). Zimmerman and Williams did not investigate the CI for the deattenuated correlation; however, they indicated that the deattenuated estimate of the correlation is unbiased under a variety of distributions when the reliability is high (e.g., .70). This suggests that the bootstrapped deattenuated correlation may hold promise under nonnormality given that the procedure only requires an unbiased estimate to generate the correct CIs. Even so, future research should investigate the performance of the bootstrapped deattenuated correlation under nonnormality.

While other studies about the deattenuated correlation investigated a larger range for reliability, here we only focused on reliability that ranged from .50 to .90. This decision was made as a matter of practicality. It is difficult to imagine a researcher using a measurement instrument with reliability less than .70. A measurement instrument with a reliability estimate of .70 indicates that 30% of the instrument’s variance is error variance; a reliability estimate of .50 indicates that 50% of the instrument’s variance is error variance. Thus, we did not investigate as wide a range of reliability as previous studies; however, the range that was investigated is sufficient for applied research. In addition, the reality is that in applied research a measurement instrument with more than 30% of its variance being error variance should probably not be used (Nunnally, 1978). In this case, the researcher should find or create a more adequate measurement instrument.

As previously mentioned, the deattenuated correlation has seen very little use in the social/behavioral sciences since its inception more than 100 years ago (Charles, 2005). Although several reasons have been discussed, the most likely reason is the vast popularity of coefficient alpha (Cortina, 1993; Hogan, Benjamin, & Brezinski, 2000). It is widely known that coefficient alpha is a lower bound estimate of reliability, and therefore it is used to provide a conservative estimate of reliability. Given that the deattenuated correlation, including the one proposed here, assumes that the reliability estimate is accurate, researchers most likely worry that using coefficient alpha will inflate the estimated deattenuated correlation and possibly make it larger than one. This is a valid concern. However, researchers should be reminded that coefficient alpha is not the only measure of reliability.

As discussed earlier, an alternative to coefficient alpha is coefficient omega (McDonald, 1999; Zinbarg et al., 2005). Coefficient omega provides a comparable form of reliability to coefficient alpha but does not have the strict assumption that the items have to be tau-equivalent or essentially tau-equivalent. When this assumption is not met, coefficient alpha will underestimate the reliability whereas coefficient omega will not. Hence, there is an option to using coefficient alpha. See McDonald (1999) and Zinbarg et al. (2005) for further details.

As noted by Charles (2005), arguing that the deattenuated correlation is imperfect indicates that the attenuated correlation is even more imperfect. If attenuation due to measurement error continues to be ignored, researchers will continue to have a difficult time finding a true relationship (or effect) that may exist. Furthermore, it automatically implies that variables are measured without error, which is not the case in the social/behavioral sciences (Schmidt & Hunter, 1996). Although attenuated correlations continue to be used, it is mostly out of a defensive position in that it is better to be conservative than overoptimistic. In addition, Schmidt and Hunter suggest that such a position is emotional as opposed to rational, and that failure to correct for attenuation is the second worst methodological error in the literature—second only to declaring significance when a p value is less than .05.

In conclusion, this preliminary study has illustrated the increase in accuracy when combining the deattenuated correlation with the bootstrap procedure. According to the results, the bootstrapped deattenuated correlation is an adequate method for correcting the attenuated correlation due to measurement error and provides equally adequate CIs. Given the controversy over the deattenuated correlation, it is hoped that the presentation here, at the very least, is a good start at revisiting this century-old problem.

Interested researchers can obtain an easy-to-use R function for the bootstrapped deattenuated correlation with example data free of charge by contacting the corresponding author.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Allen

M. J.

Yen

W. M.

(1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole.

Bradley

J. V.

(1978). Robustness? British Journal of Mathematics and Statistical Psychology, 31, 144-152.

Buckley

M. R.

Cote

J. A.

Comstock

S. M.

(1990). Measurement errors in the behavioral sciences: The case of personality/attitude research. Educational and Psychological Measurement, 50, 447-474. doi:10.1177/001316449050300110.1177/0013164490503001

Carroll

R. J.

Ruppert

Stefanski

L. A.

Crainiceanu

C. M.

(2006). Measurement error in nonlinear models: A modern perspective (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC.

Charles

E. P.

(2005). The correction for attenuation due to measurement error: Clarifying concepts and creating confidence sets. Psychological Methods, 10, 206-226. doi:10.1037/1082-989X.10.2.20610.1037/1082-989X.10.2.206

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Cortina

J. M.

(1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98-104. doi:10.1037/0021-9010.78.1.9810.1037/0021-9010.78.1.98

Crocker

L. M.

Algina

(1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart & Winston.

Efron

Tibshirani

(1993). An introduction to the bootstrap. New York, NY: Chapman & Hall.

10.

Fan

(2003). Two approaches for correcting correlation attenuation caused by measurement error: Implications for research practice. Educational and Psychological Measurement, 63, 915-930. doi:10.1177/001316440325131910.1177/0013164403251319

11.

Fiske

D. W.

(1987). Construct invalidity comes from method effects. Educational and Psychological Measurement, 47, 285-307. doi:10.1177/001316448747200110.1177/0013164487472001

12.

Fleiss

J. L.

Shrout

P. E.

(1977). The effects of measurement errors on some multivariate procedures. American Journal of Public Health, 67, 1188-1191. doi:10.2105/ajph.67.12.118810.2105/ajph.67.12.1188

13.

Graham

J. M.

(2006). Congeneric and (essentially) tau-equivalent estimates of score reliability. Educational and Psychological Measurement, 66, 930-944. doi:10.1177/001316440628816510.1177/0013164406288165

14.

Hakstian

A. R.

Schroeder

M. L.

Rogers

W. T.

(1988). Inferential procedures for correlation coefficients corrected for attenuation. Psychometrika, 53, 27-43.

15.

Hobfoll

S. E.

Shoham

S. B.

Ritter

(1991). Women’s satisfaction with social support and their receipt of aid. Journal of Personality and Social Psychology, 61, 332-341. doi:10.1037/0022-3514.61.2.33210.1037/0022-3514.61.2.332

16.

Hogan

T. P.

Benjamin

Brezinski

K. L.

(2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523-531. doi:10.1177/0013164002197069110.1177/00131640021970691

17.

Johnson

H. G.

(1944). An empirical study of the influence of errors of measurement upon correlation. American Journal of Psychology, 57, 521-536. doi:10.2307/141724710.2307/1417247

18.

Johnson

N. C.

Kotz

Balakrishnan

(1995). Continuous univariate distributions (Vol. 2, 2nd ed.). New York, NY: Wiley.

19.

Kassarjian

H. H.

(1971). Personality and consumer behavior: A review. Journal of Marketing Research, 8, 409-418.

20.

Kirkpatrick

L. A.

(1997). Effects of multiple determinacy and measurement error on trait-behavior and behavior-behavior relations: An integrated conceptual model. Personality and Social Psychology Bulletin, 23, 199-209. doi:10.1177/014616729723200810.1177/0146167297232008

21.

Lance

C. E.

Dawson

Birkelbach

Hoffman

B. J.

(2010). Method effects, measurement error, and substantive conclusions. Organizational Research Methods, 13, 435-455. doi:10.1177/109442810935252810.1177/1094428109352528

22.

Lord

Novick

Birnbaum

(1968). Statistical theories of mental test scores. Oxford, England: Addison-Wesley.

23.

McDonald

R. P.

(1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

24.

Murphy

K. R.

(2008). Explaining the weak relationship between job performance and ratings of job performance. Industrial and Organizational Psychology, 1, 148-160. doi:10.1111/j.1754-9434.2008.00030.x10.1111/j.1754-9434.2008.00030.x

25.

Mutch

Tisak

(2005). Measurement error and the correlation between positive and negative affect: Spearman (1904, 1907) revisited. Psychological Reports, 96(1), 43-46. doi:10.2466/pr0.96.1.43-4610.2466/pr0.96.1.43-46

26.

Nunnally

J. C.

(1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.

27.

Pedhazur

E. J.

(1997). Multiple regression in behavioral research: Explanation and prediction (3rd ed.). Orlando, FL: Harcourt Brace.

28.

Romano

J. L.

Kromrey

J. D.

Hibbard

S. T.

(2010). A Monte Carlo study of eight confidence interval methods for coefficient alpha. Educational and Psychological Measurement, 70, 376-393. doi:10.1177/001316440935569010.1177/0013164409355690

29.

Schmidt

F. L.

Hunter

J. E.

(1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223. doi:10.1037/1082-989X.1.2.19910.1037/1082-989X.1.2.199

30.

Spearman

(1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72-101.

31.

Stayrook

Corno

L. Y. N.

(1979). An application of generalizability theory in disattenuating a path model of teaching and learning. Journal of Educational Measurement, 16, 227-237. doi:10.1111/j.1745-3984.1979.tb00104.x10.1111/j.1745-3984.1979.tb00104.x

32.

Taft

C. T.

Watkins

L. E.

Stafford

Street

A. E.

Monson

C. M.

(2011). Posttraumatic stress disorder and intimate relationship problems: A meta-analysis. Journal of Consulting and Clinical Psychology, 79, 22-33. doi:10.1037/a002219610.1037/a0022196

33.

Tarkkonen

Vehkalahti

(2005). Measurement errors in multivariate measurement scales. Journal of Multivariate Analysis, 96, 172-189. doi:10.1016/j.jmva.2004.09.00710.1016/j.jmva.2004.09.007

34.

Thomas

S. P.

Williams

R. L.

(1991). Perceived stress, trait anger, modes of anger expression, and health status of college men and women. Nursing Research, 40, 303-307.

35.

Timm

N. H.

(2002). Applied multivariate analysis. New York, NY: Springer.

36.

Winne

P. H.

Belfry

M. J.

(1982). Interpretive problems when correcting for attenuation. Journal of Educational Measurement, 19, 125-134. doi:10.1111/j.1745-3984.1982.tb00121.x10.1111/j.1745-3984.1982.tb00121.x

37.

You

Chen

(2006). Estimation of a semiparametric varying-coefficient partially linear errors-in-variables model. Journal of Multivariate Analysis, 97, 324-341. doi:10.1016/j.jmva.2005.03.00210.1016/j.jmva.2005.03.002

38.

Zimmerman