On the Unlikely Case of an Error-Free Principal Component From a Set of Fallible Measures

Background, Notation, and Assumptions

We assume that a set of observed measures is given, whereby at least two of them contain measurement error (with positive variance), as it will often be the case in the majority of contemporary studies in the educational and behavioral disciplines (see Raykov et al., 2016). We denote these measures by X₁, . . . , X_k (k > 1) and their collection by Ξ, that is, Ξ = {X₁, . . . , X_k}. Using the classical test theory decomposition, which effectively always exists in empirical educational, behavioral, and social research (e.g., Raykov & Marcoulides, 2011), X_j = T_j+e_j holds, where T_j and e_j are the true and error score of the jth measure, respectively (j = 1, . . . , k). In addition, we assume in the remainder that at least two of these k error scores, e₁ through e_k, are correlated. Last but not least, we assume that any observed variable used below, including all PCs, has positive variance—a condition that can be hardly considered restrictive in the overwhelming majority of contemporary behavioral and social studies.

In PCA, as is well known, particular linear combinations Y₁, . . . , Y_k of the original measures in Ξ are found, which possess the property that the first one, that is, Y₁, maximizes observed variance, and in addition each of the following PCs Y₂, . . . , Y_k maximize the “remaining” variance in the set of X₁, . . . , X_k under the condition that any subsequent PC is uncorrelated with all preceding PCs extracted before it (e.g., Timm, 2002). As usual, we assume in the rest of this article that the covariance matrix Σ of X₁, . . . , X_k is positive definite and subjected to PCA. (Analyzing instead the correlation matrix does not alter the following developments, findings, and interpretations.) We stress that the remaining discussion evolves exclusively at the population level, like that in Raykov et al. (2016).

As elaborated in detail in the literature, the weights in the linear combinations of X₁, . . . , X_k that render each of the PCs are the elements of the normalized eigenvectors pertaining to the successive eigenvalues of Σ (e.g., Johnson & Wichern, 2002),

Σ = P Λ P ′ ,

with the orthogonal matrix P containing these eigenvectors as columns and Λ being the diagonal matrix with these eigenvalues along its main diagonal (priming denotes transposition in this note).

How Probable Is It for a Principal Component to Be Error-Free?

If interested specifically in the jth PC, Y_j, denoting by p_sj its pertinent weights for the observed variables, the following equation holds:

Y j = p 1 j X 1 + p 2 j X 2 + ⋯ + p k j X k

(j = 1, . . . , k). Using the classical test theory decomposition for the manifest measures (note that if the sth of them, X_s, is error-free, then e_s = 0 is valid for all cases; 1≤s≤k), evidently the corresponding decomposition of Y_j is as follows (e.g., Raykov et al., 2016):

Y j = p 1 j X 1 + p 2 j X 2 + ⋯ + p k j X k = ( p 1 j T 1 + p 2 j T 2 + ⋯ + p k j T k ) + ( p 1 j e 1 + p 2 j e 2 + ⋯ + p k j e k ) .

From Equation (3) it follows, therefore, that the jth PC will be error-free if and only if the variance of its error score is 0, that is, if and only if

Var ( p 1 j e 1 + p 2 j e 2 + ⋯ + p k j e k ) = 0 ,

where Var(.) denotes variance (j = 1, . . . , k). However, in the presently considered setting Equation (4) will hold if and only if

p 1 j 2 Var ( e 1 ) + p 2 j 2 Var ( e 2 ) + ⋯ + p k j 2 Var ( e k ) + 2 ∑ q < r p q j p r j Cov ( e q , e r ) = 0 ,

where Cov(. , .) denotes covariance and the sum on the left-hand side (LHS) of (5) runs over all pairs of error terms that are correlated (i.e., over all error term pairs with nonzero covariance that are included only once in it).

From Equation (5) it is seen that a PC will not contain any measurement error if and only if the nonzero error covariances are such that their linear combination with weights being the products of the corresponding PC weights equals (half of) the negative of the linear combination of the error variances with weights being the squares of the pertinent PC weights.

Given that the jth PC is fixed in the discussion in this section, the variances and covariances in the LHS of Equation (5) span a proper subspace S of R^k+t, where S could be viewed as the parameter space for the presently considered setting concerned only with the error variance in Y_j, t is the number of (nonduplicated) nonzero error covariances in (5), and R^k+t denotes the set of all real vectors of dimension k+t. We note that the space S has also dimensionality k + t. The reason is the fact that there is no relationship between the variances and covariances in the LHS of (5) (apart from that resulting from the error variance of Y_j being nonnegative), which could make the dimensionality of S lower than k + t. However, the set of all (k + t)-dimensional vectors consisting of possible values for the variances and covariances in the LHS of (5), which satisfy the condition (equality) stated in Equation (5), is a proper subspace S* of S. This is because the LHS of (5) is a linear combination of all the variances and covariances in it that generated the space S to begin with. For this reason, the dimensionality of S* is lower than k + t, that is, dim(S*) < k + t, where dim(.) denotes dimensionality. Therefore, S* has mass (probability) of 0 in the space S of interest here (e.g., Apostol, 2013). ¹

Hence, from the preceding discussion it follows that the probability of the condition in Equation (5) holding for a particular PC, say the jth, is 0 (j = 1, . . . , k). That is, in practice it is (in effect) impossible that Equation (5) will hold in a given empirical study (even if conducted at the population level). Therefore, it is practically impossible for a PC, which is extracted from a set of observed measures where at least two are fallible, to have perfect reliability regardless whether the error terms of these fallible measures correlate (for the case of uncorrelated errors, see Raykov et al., 2016; see also Note 1).

Conclusion

The impressive popularity of PCA in the empirical sciences may have contributed to the circulation of some myths about PCs among certain quarters of applied researchers. One of them is that the first (or subsequent) PC extracts “all reliable variance” in the analyzed set of observed variables that may—and usually will—contain imperfect measures in educational and psychological studies, with the potential implication that at least the first PC may be perfectly reliable. The present note extended the earlier discussion in Raykov et al. (2016) dealing with the uncorrelated error case, to the setting where at least two errors in fallible observed variables possess nonzero covariance. The note demonstrated that while it is possible for the first (or later) PC to have perfect reliability in case of correlated errors—when a set of manifest measures is subjected to PCA and at least two of them are fallible—the probability is 0 of this occurring in a given empirical study (even if conducted at the population level; see also Note 1).

Therefore, it is practically impossible for a PC to be error-free if extracted from a set of observed measures with at least two of them containing measurement errors that correlate. Together with the findings in Raykov et al. (2016) this note leads thus to the conclusion that in practice no PC can be free of measurement error for a set of observed variables that are not all perfectly reliable, irrespective of whether the error terms of the individual measures correlate, and hence no PC can practically be free of measurement error then. With this conclusion in mind, the present note’s message is therefore that while PCs have highly desirable, observed variance maximization features, in practical terms this does not imply that a PC—even the first—will or could be error-free if not all measures in the PC analyzed set are perfectly reliable.