Proportion of Indicator Common Variance Due to a Factor as an Effect Size Statistic in Revised Parallel Analysis

Abstract

Past research suggests revised parallel analysis (R-PA) tends to yield relatively accurate results in determining the number of factors in exploratory factor analysis. R-PA can be interpreted as a series of hypothesis tests. At each step in the series, a null hypothesis is tested that an additional factor accounts for zero common variance among measures in the population. Integration of an effect size statistic—the proportion of common variance (PCV)—into this testing process should allow for a more nuanced interpretation of R-PA results. In this article, we initially assessed the psychometric qualities of three PCV statistics that can be used in conjunction with principal axis factor analysis: the standard PCV statistic and two modifications of it. Based on analyses of generated data, the modification that considered only positive eigenvalues ( ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ ) overall yielded the best results. Next, we examined PCV using minimum rank factor analysis, a method that avoids the extraction of negative eigenvalues. PCV with minimum rank factor analysis generally did not perform as well as ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ , even with a relatively large sample size of 5,000. Finally, we investigated the use of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ in combination with R-PA and concluded that practitioners can gain additional information from ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ and make more nuanced decision about the number of factors when R-PA fails to retain the correct number of factors.

Keywords

parallel analysis effect size exploratory factor analysis

A number of factor analytic experts (e.g., Fabrigar, Wegener, MacCallum, & Strahan, 1999; Schmitt, 2011) have recommended the use of parallel analysis (PA) to assess the number of factors underlying measures in conducting exploratory factor analysis (EFA). Traditional PA (T-PA) compares the eigenvalues for a sample correlation matrix with the mean eigenvalues for correlation matrices for M (e.g., M = 100) parallel datasets (PDs) generated such that the variables are independent. Despite the relative accuracy of PA, Harshman and Reddon (1983) and Turner (1998) believed T-PA to be flawed. They argued against the use of a reference distribution of eigenvalues for PDs with uncorrelated variables. They contended that the proper reference distribution of eigenvalues to reach a conclusion about the kth factor should be based on PDs with k−1 underlying factors.

To address these concerns, Green, Levy, Thompson, Lu, and Lo (2012) suggested revised PA (R-PA). When assessing whether at least k factors underlie a set of measures with R-PA, PDs are generated assuming measures are a function of k−1 factors rather than 0 factors. Ideally, PDs should be generated based on the population loadings of these k−1 factors. In practice, the population factor loadings are unknown, and factor loadings based on the sample dataset are used.

R-PA differs from T-PA in two other ways. First, R-PA uses principal axis factoring (PAF) rather than principal components analysis (PCA) because unlike PCA, PAF allows for measurement error and thus is more appropriate for educational/psychological data (e.g., Ford, MacCallum, & Tait, 1986). A second difference is that the eigenvalues for the sample data are compared with the 95th percentile rather than the mean of eigenvalues for the referent distribution (Buja & Eyuboglu, 1992; Glorfeld, 1995), such that each step in R-PA is similar to the hypothesis testing with a nominal alpha of .05.

Results of Monte Carlo studies offer support for R-PA (Green et al., 2012; Green, Redell, Thompson, & Levy, 2016; Green, Thompson, Levy, & Lo, 2015; Green, Xu, & Thompson, 2018) relative to other PA methods. Ruscio and Roche (2012) also offered a PA method that uses the proper referent distribution, but Green et al. (2017) conducted a Monte Carlo study suggesting R-PA generally was preferable across the examined conditions.

R-PA, Hypothesis Testing, and Effect Size

R-PA can be viewed as a sequential, hypothesis-testing process. At each step in the process, the null hypothesis is assessed that k−1 factors are sufficient to reproduce the population correlation matrix. Alpha is set at .05 given the 95th percentile eigenvalue rule is applied. Rejection of the null hypothesis implies at least k factors underlie the population correlation matrix. Nonrejection is interpreted as k−1 factors underlie this matrix.

Because R-PA is a series of hypothesis tests, it is open to the misinterpretations that commonly occur with hypothesis testing. First, nonrejection of the null hypothesis in R-PA should not necessarily imply acceptance of the null hypothesis. Nonrejection can be due to a lack of power. In previous Monte Carlo studies (Green et al., 2012; Green et al., 2015), R-PA underestimated the number of factors in conditions that have low power (e.g., small sample size, lower factor loadings, and strong factor correlations). Second, rejection of the null hypothesis conducting R-PA does not necessarily imply that an additional factor is nontrivial. A number of psychometricians have argued that a very large number of factors is likely to underlie any set of measures (e.g., Cudeck & Browne, 1992; Tucker, Koopman, & Linn, 1969). Within this context, the researcher who is conducting EFA is attempting to determine the number of major factors that comes “close” to reproducing the correlations among them, and to ignore the trivial factors (Fabrigar et al., 1999).

By including an effect size statistic in the R-PA process, researchers must not only consider the results of the hypothesis tests, but also address whether each additional factor is weak and should be ignored or sufficiently strong to have psychometric meaning. In so doing, researchers must make more nuanced decisions about the number of factors.

Purpose of Article

A number of researchers have suggested the proportion of common variance of indicators explained by a factor (PCV) as an effect size index for that factor (e.g., Reise, 2012; Ten Berge & Sočan, 2004). The purpose of this article was to investigate the choice of PCV statistics for use with R-PA. The results are presented in three studies. In Study 1, we assessed the psychometric qualities of a frequently applied PCV index using PAF. Due to problems with this statistic, we also evaluated two modifications to this index. In Study 2, we assessed the quality of a PCV index that uses a lesser-known factor extraction method, minimum rank factor analysis (MRFA; Lorenzo-Seva, 2013; Shapiro & Ten Berge, 2002; Sočan, 2003; Ten Berge & Kiers 1991; Ten Berge & Sočan, 2004). MRFA avoids the problem in the computation of PCV using PAF, such that all factors have positive eigenvalues. In Study 3, we examined the use of these effect size statistics in the application of R-PA.

Study 1

Definition of PCV as a Factor Effect Size Index

We begin by considering the PCV of a factor in the population. With factor analysis, communalities for indicators ( $γ_{p}$ ) are substituted along the diagonal of a correlation matrix to yield a reduced correlation matrix, where a communality gives the proportion of variance of an indicator explained by the underlying factors. Factors are then extracted from this reduced correlation matrix. Each eigenvalue for an extracted factor gives the variance of the indicators accounted for by that factor and, thus, should be greater than or equal to zero. If we knew the underlying structure of a set of indicators, we could compute correctly PCV for factor k′ in the population $(π_{k'})$ :

π_{k^{'}} = \frac{Λ_{k^{'}}}{\sum_{k = 1}^{K} Λ_{k}},

where Λ_k′ is the eigenvalue for the k^′th factor, and $\sum_{k = 1}^{K} Λ_{k}$ is the sum of the eigenvalues for the reduced correlation matrix across the K indicators. Given the communalities are correct and the number of factors is correctly specified (N_F:Correct) and less than K, the last K−N_F:Correct eigenvalues must be zeros, and the denominator for Equation 1 can be reexpressed as $\sum_{k = 1}^{N_{F : Correct}} Λ_{k}$ .

There is an alternative computation of PCV if the correct model is unknown in calculating communalities for the reduced correlation matrix. Squared multiple correlations (SMC, denoted as $ρ_{p}^{2}$ for any indicator p) between indicators and all other indicators can be used as estimates of the correct communalities. We will focus on SMCs in this article in order to be consistent with the choice of communality estimates for R-PA, although corrected SMC (Cureton & D’Agostino, 2013) or quantities other than SMCs have been suggested in the literature (Mulaik, 2009). It should be noted that a squared multiple correlation gives the proportion of indicator variance attributable to other indicators rather than the proportion of indictor variance attributable to the underlying factors. Thus, although we are interested in $π_{k'}$ , we may have to focus on the population PCV with population squared multiple correlations ( $π_{SMC : k'}$ ) as rough estimates of the communalities. Accordingly, $π_{SMC : k'}$ is based on the eigenvalues $(Λ_{SMC : k'})$ of a reduced correlation matrix with squared multiple correlations along the diagonal:

π_{SMC : k^{'}} = \frac{Λ_{SMC : k^{'}}}{\sum_{k = 1}^{K} Λ_{SMC : k}} .

$π_{SMC : k'}$ is a negatively biased estimator of $π_{k'}$ at the population level (i.e., $π_{SMC : k'} - π_{k'} < 0$ ) because $ρ_{p}^{2}$ is generally less than $γ_{p}^{2}$ for any indicator p.

$π_{k'}$ and $π_{SMC : k'}$ are PCV parameters in the population. At the sample level, we do not know the correct model and thus cannot estimate $π_{k'}$ , but can estimate $π_{SMC : k'}$ . To obtain an estimate of $π_{SMC : k'}$ , we substitute sample estimates for parameters on the right side of Equation 2 to obtain

{\hat{π}}_{SMC : k'} = \frac{{\hat{Λ}}_{SMC : k'}}{\sum_{k = 1}^{K} {\hat{Λ}}_{SMC : k}} .

A Problem With ${\hat{π}}_{SMC : k'}$

In computing PCV in a sample (i.e., Equation 3), we factor analyze a reduced correlation matrix with SMCs along the diagonal to obtain eigenvalues. The numerator of Equation 3 contains the eigenvalue for the k^′th factor, an estimate of the common variance of the indicators explained by the k^′th factor. The denominator is the sum of eigenvalues across all factors, a rough estimate of the common variance of the indicators explained by all possible factors. A problem with this approach becomes apparent in conducting a common factor analysis: the first number of eigenvalues generally are positive, whereas the remaining eigenvalues are negative. Conceptually, the results are nonsensical in that an eigenvalue is the variance of the indicators due to any one factor, and a variance cannot have negative values.

It is crucial to assess the statistical properties of ${\hat{π}}_{SMC : k'}$ in that this index is reported by popular statistical packages, including SAS and Stata. In the SAS User’s Guide (SAS Institute Inc., 2009), an example (labeled Example 33.2 Principal Factor Analysis) is presented based on the factor analysis of five variables. Eigenvalues of the reduced correlations (with SMCs along the diagonal) as well as proportions of common variance are presented; the first three eigenvalues and proportions are positive in value, whereas the last two are negative. As reported in the text in the SAS User’s Guide, the reported results are perplexing in that the first two eigenvalues accounted for 101.31% of the common variance. Appropriately, the SAS manual indicates that this out-of-bound estimate occurred because the reduced correlation matrix was not positive definite and accordingly yielded negative eigenvalues. A similar example is presented in the Stata User’s Guide Release 13 (StataCorp, 2013) in discussing the output generated by the factor analysis procedure.

Alternative Estimators of PCVs

Given the problems with ${\hat{π}}_{SMC : k'}$ , we considered two adaptations of ${\hat{π}}_{SMC : k'}$ to assess PCV. The first adaptation was to alter the denominator. We took a rather simple approach to this adaptation: Negative eigenvalues are problematic so let us get rid of them. In other words, rather than summing across all eigenvalues in the denominator, we sum across only the positive eigenvalues. Thus, at the population level, we can define an alternative PCV ( $π_{SMC : k'}^{+ Λ}$ ),

π_{SMC : k'}^{+ Λ} = \frac{Λ_{SMC : k'}}{\sum_{k = 1}^{N_{+ Λ}} Λ_{SMC : k}},

where the denominator is the sum of the $N_{+ Λ}$ positive eigenvalues.¹ At the sample level, we substitute the estimates on the right side of the equation:

{\hat{π}}_{SMC : k'}^{+ \hat{Λ}} = \frac{{\hat{Λ}}_{SMC : k'}}{\sum_{k = 1}^{N_{+ \hat{Λ}}} {\hat{Λ}}_{SMC : k}} .

It should be noted that $\sum_{k = 1}^{N_{+ \hat{Λ}}} {\hat{Λ}}_{SMC : k}$ must be greater than or equal to $\sum_{k = 1}^{K = p} {\hat{Λ}}_{SMC : k}$ , and thus ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ must be less than or equal to ${\hat{π}}_{SMC : k'}$ .

A second adaptation is a corrected ${\hat{π}}_{SMC : k'}$ (denoted ${\hat{π}}_{SMC : k'}^{Corrected}$ ) that takes into account our expectation that ${\hat{π}}_{SMC : k'}$ is an overestimate of $π_{k'}$ . ${\hat{π}}_{SMC : k'}$ is corrected by the mean eigenvalues for parallel samples. More specifically, for the kth factor, the corrected effect size statistic involves (a) computing ${\hat{Λ}}_{SMC : k'}$ based on the reduced correlation matrix; (b) calculating comparable eigenvalues for the parallel samples generated assuming k−1 factors, denoted as ${\hat{Λ}}_{SMC : k'}^{m}$ (m represents the mth parallel dataset and M represents the total number of parallel datasets); (c) computing a mean eigenvalue across the parallel datasets; (d) subtracting the quantity computed in Step c_K from the quantity determined in Step a; and (e) dividing the result of Step d by $\sum_{k = 1}^{K} {\hat{Λ}}_{SMC : k}$ . The equation for ${\hat{π}}_{SMC : k'}^{Corrected}$ is thus

{\hat{π}}_{SMC : k'}^{Corrected} = \frac{({\hat{Λ}}_{SMC : k'} - \sum_{m = 1}^{M} {\hat{Λ}}_{SMC : k'}^{m} / M)}{\sum_{k = 1}^{K} {\hat{Λ}}_{SMC : k}} .

The subtraction in Step (c) provides a downward correction of ${\hat{π}}_{SMC : k'}$ . In addition, Step (c) ensures that when the number of factors is zero in the population (e.g., in a null model with no common factor), ${\hat{π}}_{SMC : k'}^{Corrected}$ is 0 for all k under finite samples and thus nonzero eigenvalues are avoided.

Purpose of Study 1

The objective of Study 1 was to assess the psychometric quality of the presented PCV indices. Initially, we examined the bias of $π_{SMC : k'}$ and $π_{SMC : k'}^{+ Λ}$ at the population level. Bias is defined as the difference between either of these parameters and $π_{k'}$ . Note we did not assess ${\hat{π}}_{SMC : k'}^{Corrected}$ in that it is undefined in the population.

At the sample level, biases of these PCV indices were assessed by calculating $E ({\hat{π}}_{SMC : k'}) - π_{k'}$ , $E ({\hat{π}}_{SMC : k'}^{+ \hat{Λ}}) - π_{k'}$ , and $E ({\hat{π}}_{SMC : k'}^{Corrected}) - π_{k'}$ , where E() is an expected value. As a byproduct of assessing the biases of these PCV indices, we assessed whether the statistical properties of ${\hat{π}}_{SMC : k'}$ warrant its use in popular statistical packages or whether one of the alternative investigated indices (i.e., ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}} and {\hat{π}}_{SMC : k'}^{Corrected}$ ) has better statistical properties.

Method

Design

We manipulated three dimensions to evaluate the bias of $π_{SMC : k'}$ and $π_{SMC : k'}^{+ Λ}$ in estimating $π_{k'}$ at the population: factor-model type, magnitude of factor loadings, and correlations between factors, where appropriate. At the sample level, we manipulated the same three dimensions plus sample size to evaluate the psychometric qualities of ${\hat{π}}_{SMC : k'}, {\hat{π}}_{SMC : k'}^{+ \hat{Λ}},$ and ${\hat{π}}_{SMC : k'}^{Corrected}$ . We described these design dimensions below:

Factor-model types used to generate sample data are presented in Figure 1: (a) an unidimensional model with all eight indicators loading on a single factor; (b) a two-factor, perfect-cluster model with four indicators loading on each of the two factors; (c) a three-factor, perfect-cluster model with four indicators loading on each of the three factors; (d) a two-factor, bifactor model with all 8 indicators loading on a general factor and 4 indicators also loading on a group factor; and (e) a three-factor, bifactor model with all 12 indicators loading on a general factor and 4 indicators loading on each group factor.

Factor loadings on unidimensional models were .5s or .7s for indicators that were a function of a single factor. For the two-factor or three-factor perfect-cluster models, the nonzero loadings on the factors were .5s or .7s. For the bifactor models, the indicators on the general factor had loadings of .5s or .7s, and the 4 indicators on the group factor(s) had loadings of .5s.

Correlations between factors were .0, .4, or .8 for any two-factor or three-factor perfect-cluster model.

Number of observations was set at 200 or 400 for the sample-level simulation.

Figure 1.

Models for data generation in Study 1.

Data Generation and Analyses

At both the population and sample levels, we restricted our presentation to the PCAs for the first three factors, given that the maximum correct number of factors manipulated in the population was three. PCAs for the rest of the factors were in general close to 0 (or negative for $π_{SMC : k'}$ ).

At the population level, we computed a reproduced correlation matrix based on the parameters of the models for each combination of the manipulated dimensions. Reduced correlation matrices were then created by substituting the correct $γ_{p}$ or SMCs along the diagonal of the generated correlation matrices. Note that the correct $γ_{p}$ was obtained by specifying the correct number of factors. In reality, the correct number of factors is unknown, and thus $γ_{p}$ is replaced by SMCs. Each of the reduced correlation matrices was analyzed using PAF to obtain eigenvalues for the unrotated factors. $π_{SMC : k'}$ and $π_{SMC : k'}^{+ Λ}$ were computed according to Equations 2 and 4, respectively, as well as $π_{k'}$ using Equation 1.

At the sample level, 1,000 sample datasets were created for each combination of the manipulated dimensions. The factors and errors in the model all follow N(0, 1). Correlation matrices were computed for each sample dataset and analyzed using PAF to obtain eigenvalues to compute ${\hat{π}}_{SMC : k'}$ and ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ . In addition, to compute ${\hat{π}}_{SMC : k'}^{Corrected}$ , we generated 100 parallel datasets based on the estimated loadings given 0, 1, and 2 factors for each sample dataset to assess the PCVs for 1, 2, and 3 factors, respectively.

All data generation and analyses were implemented in R (R Core Team, 2017).

Results

We considered the results of the PCV indices at the population level and then at the sample level.

Bias at the Population Level

We present the results of $π_{SMC : k'}$ , $π_{SMC : k'}^{+ Λ}$ , and $π_{k'}$ for the first three factors in Table 1. Bias at the population level was assessed by $π_{SMC : k'} - π_{k'}$ and $π_{SMC : k'}^{+ Λ} - π_{k'}$ . We considered a bias that was greater than or equal to .10 as substantial and bolded these values in Table 1, as well as all Tables 2 through 5. Overall, $π_{SMC : k'}$ performed poorly. $π_{SMC : k'}$ was positively biased across all conditions for the first factor. $π_{SMC : k'}$ for the first factor was greater than its upper bound of 1.0 in 10 of the 18 conditions and was higher than 1.20 in four of these 10 conditions. In the remaining 8 conditions, $π_{SMC : k'}$ was .08 or greater relative to $π_{k'}$ . For the second factor, substantial positive bias for $π_{SMC : k'}$ was observed for perfect-cluster models with zero correlation among factor. The bias for $π_{SMC : k'}$ decreased rapidly with an increase in the correlation among factors. Minimal bias (not exceeding .05) was observed for the remaining conditions. For the third factor, $π_{SMC : k'}$ evidenced negative values when $π_{k'} = 0$ . Considerable positive bias occurred with the three-factor perfect-cluster model when the correlation among factors was 0, but this bias was minimal when correlations among factors were large.

Table 1.

PCVs at the Population Level for One-, Two-, and Three-Factor Models.

		PCV for first factor			PCV for second factor			PCV for third eigenvalue
ρ_FF′	λ	$π_{k'}$	$π_{SMC : k'}$	$π_{SMC : k'}^{+ \hat{Λ}}$	$π_{k'}$	$π_{SMC : k'}$	$π_{SMC : k'}^{+ \hat{Λ}}$	$π_{k'}$	$π_{SMC : k'}$	$π_{SMC : k'}^{+ \hat{Λ}}$
One-factor underlying all indicators
—	.5	1.00	1.38	1.00	0	−.05	−.04	0	−.05	−.04
—	.7	1.00	1.13	1.00	0	−.02	−.02	0	−.02	−.02
Two-factor, perfect-cluster model
0	.5	.50	.88	.50	.50	.88	.50	0	−.13	−.07
	.7	.50	.63	.50	.50	.63	.50	0	−.04	−.03
.4	.5	.70	1.22	.73	.30	.46	.27	0	−.11	−.07
	.7	.70	.89	.71	.30	.36	.29	0	−.04	−.03
.8	.5	.90	1.39	.94	.10	.08	.06	0	−.08	−.05
	.7	.90	1.10	.92	.10	.09	.08	0	−.03	−.03
Three-factor, perfect-cluster model
0	.5	.33	.58	.33	.33	.58	.33	.33	.58	.33
	.7	.33	.42	.33	.33	.42	.33	.33	.42	.33
.4	.5	.60	1.04	.63	.20	.30	.18	.20	.30	.18
	.7	.60	.77	.62	.20	.24	.19	.20	.24	.19
.8	.5	.87	1.26	.92	.07	.06	.04	.07	.06	.04
	.7	.87	1.04	.89	.07	.06	.05	.07	.06	.05
Bifactor model with one group factor
0	.5	.87	1.16	.90	.13	.13	.10	0	−.04	−.03
	.7	.91	1.03	.93	.09	.08	.07	0	−.02	−.02
Bifactor model with two group factors
0	.5	.75	.95	.77	.20	.23	.19	.05	.05	.04
	.7	.84	.92	.85	.13	.13	.12	.04	.03	.03

Note. PCV = proportion of common variance. Values that yielded ≥.10 bias were bolded.

Table 2.

PCVs at the Population Level and Mean Effect Sizes at the Sample Level for the First Factor.

		Population			Sample with N = 200			Sample with N = 400
ρ_FF′	λ	$π_{k'}$	$π_{SMC : k'}$	$π_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}$	${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}^{Corrected}$	${\bar{\hat{π}}}_{SMC : k'}$	${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}^{Corrected}$
One-factor underlying all indicators
—	.5	1.00	1.38	1.00	1.22	.89	1.00	1.29	.95	1.14
—	.7	1.00	1.13	1.00	1.09	.97	.99	1.11	.99	1.04
Two-factor, perfect-cluster model
0	.5	.50	.88	.50	.88	.55	.59	.90	.55	.69
	.7	.50	.63	.50	.67	.54	.55	.66	.53	.58
.4	.5	.70	1.22	.73	1.06	.68	.78	1.13	.71	.93
	.7	.70	.89	.71	.86	.71	.74	.87	.71	.79
.8	.5	.90	1.39	.94	1.22	.85	.97	1.30	.90	1.12
	.7	.90	1.10	.92	1.05	.90	.95	1.08	.91	1.00
Three-factor, perfect-cluster model
0	.5	.33	.58	.33	.59	.38	.59	.61	.38	.43
	.7	.33	.42	.33	.46	.38	.36	.46	.37	.38
.4	.5	.60	1.04	.63	.82	.56	.60	.92	.60	.74
	.7	.60	.77	.62	.71	.60	.61	.74	.61	.67
.8	.5	.87	1.26	.92	1.02	.75	.83	1.13	.83	.98
	.7	.87	1.04	.89	.97	.85	.88	1.01	.87	.94
Bifactor model with one group factor
0	.5	.87	1.16	.90	1.08	.86	.88	1.12	.88	1.01
	.7	.91	1.03	.93	1.01	.91	.93	1.02	.92	.96
Bifactor model with two group factors
0	.5	.75	.95	.77	.87	.72	.76	.91	.75	.83
	.7	.84	.92	.85	.90	.83	.83	.91	.84	.86

Note. PCV = proportion of common variance. Values that yielded ≥.10 bias were bolded.

In comparison with $π_{SMC : k'}$ , $π_{SMC : k'}^{+ Λ}$ showed much less bias in estimating $π_{k'} .$ We considered separately results when $π_{k'} > 0$ and results when $π_{k'} = 0$ , aggregating across the first, second, and third factors. In the conditions in which $π_{k'} > 0$ , $π_{SMC : k'}^{+ Λ}$ was equal to $π_{k'}$ for all 12 estimates with a single factor model and perfect-cluster models with uncorrelated factors. For the remaining estimates in which $π_{k'} > 0$ , $π_{SMC : k'}^{+ Λ}$ was within .03 of $π_{k'}$ 28 times and between .04 and .05 of $π_{k'}$ 2 times. When $π_{k'} = 0$ , $π_{SMC : k'}^{+ Λ}$ was less than or equal to 0 in all 12 conditions; $π_{SMC : k'}^{+ Λ}$ was between .00 and −.05 for 10 of the estimates and was equal to −.07 for the remaining 2 estimates.

To explain the differences in results between $π_{SMC : k'}$ and $π_{SMC : k'}^{+ Λ}$ , we examined the eigenvalues for the extracted factors. In Figure 2, we present a graph of eigenvalues for the condition in which the reduced correlation matrix was based on a bifactor model with one group factor having loadings of .5s. The pattern of eigenvalues for this condition was similar to the patterns of eigenvalues for the other conditions. It should be noted that (a) the eigenvalues based on SMCs as communality estimates were positive when the eigenvalues based on correct communalities were positive, (b) the eigenvalues based on SMCs as communality estimates were negative when the eigenvalues based on correct communalities were zero, and (c) the eigenvalues were uniformly lower when the communality estimates were SMCs versus when they were correct values.

Figure 2.

Eigenvalues of reduced correlation matrices based on a bifactor model with one group factor having factor loadings of .5s.

Because $π_{SMC : k'}$ and $π_{SMC : k'}^{+ Λ}$ have the same numerator, differences in these estimates are due to differences in their denominators: the sum of all eigenvalues for $π_{SMC : k'}^{+ Λ}$ versus the sum of only the positive eigenvalues for $π_{SMC : k'}^{+ Λ}$ .

Bias at the Sample Level

At the sample level, we considered not only PCV estimates based on all eigenvalues and based on all positive eigenvalues, but also estimates that are corrected using parallel datasets. The results of these analyses for the first, second, and third factors are presented in Tables 2, 3, and 4, respectively. The population values for three of the PCVs also are included in these tables to assess bias.

Table 3.

PCVs at the Population Level and Mean Effect Sizes at the Sample Level for the Second Factor.

		Population			Sample with N = 200			Sample with N = 400
ρ_FF′	λ	$π_{k'}$	$π_{SMC : k'}$	$π_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}$	${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}^{Corrected}$	${\bar{\hat{π}}}_{SMC : k'}$	${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}^{Corrected}$
One-factor underlying all indicators
—	.5	0	−.05	−.04	.10	.07	.00	.06	.04	.00
—	.7	0	−.02	−.02	.03	.02	.00	.01	.01	.00
Two-factor, perfect-cluster model
0	.5	.50	.88	.50	.64	.40	.46	.72	.44	.59
	.7	.50	.63	.50	.55	.45	.49	.58	.47	.54
.4	.5	.30	.46	.27	.42	.27	.26	.44	.28	.34
	.7	.30	.36	.29	.35	.29	.30	.35	.29	.33
.8	.5	.10	.08	.06	.16	.11	.03	.12	.08	.05
	.7	.10	.09	.08	.10	.09	.07	.10	.08	.08
Three-factor, perfect-cluster model
0	.5	.33	.58	.33	.47	.30	.47	.52	.32	.38
	.7	.33	.42	.33	.40	.33	.32	.41	.33	.36
.4	.5	.20	.30	.18	.32	.22	.16	.32	.21	.21
	.7	.20	.24	.19	.26	.22	.20	.26	.21	.22
.8	.5	.07	.06	.04	.14	.10	.02	.11	.08	.03
	.7	.07	.06	.05	.09	.08	.04	.08	.07	.05
Bifactor model with one group factor
0	.5	.13	.13	.10	.15	.12	.04	.14	.11	.10
	.7	.09	.08	.07	.09	.08	.07	.09	.08	.07
Bifactor model with two group factors
0	.5	.20	.23	.19	.23	.19	.16	.23	.19	.19
	.7	.13	.13	.12	.13	.12	.11	.13	.12	.12

Note. PCV = proportion of common variance. Values that yielded ≥.10 bias were bolded.

Table 4.

PCVs at the Population Level and Mean Effect Sizes at the Sample Level for the Third Factor.

		Population			Sample with N = 200			Sample with N = 400
ρ_FF′	λ	$π_{k'}$	$π_{SMC : k'}$	$π_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}$	${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}^{Corrected}$	${\bar{\hat{π}}}_{SMC : k'}$	${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$	${\bar{\hat{π}}}_{SMC : k'}^{Corrected}$
One-factor underlying all indicators
—	.5	0	−.05	−.04	.04	.03	−.01	.02	.01	.00
—	.7	0	−.02	−.02	.01	.01	.00	.00	.00	.00
Two-factor, perfect-cluster model
0	.5	0	−.13	−.07	.07	.04	.01	.03	.02	.01
	.7	0	−.04	−.03	.01	.01	.00	.00	.00	.01
.4	.5	0	−.11	−.07	.07	.04	−.01	.03	.02	.00
	.7	0	−.04	−.03	.01	.01	.00	.00	.00	.01
.8	.5	0	−.08	−.05	.06	.04	.00	.03	.02	.00
	.7	0	−.03	−.03	.01	.01	.00	.00	.00	.00
Three-factor, perfect-cluster model
0	.5	.33	.58	.33	.36	.24	.36	.42	.26	.33
	.7	.33	.42	.33	.33	.28	.29	.36	.29	.34
.4	.5	.20	.30	.18	.22	.15	.10	.24	.16	.16
	.7	.20	.24	.19	.20	.17	.16	.21	.18	.19
.8	.5	.07	.06	.04	.01	.07	.00	.07	.05	.01
	.7	.07	.06	.05	.06	.05	.03	.06	.05	.04
Bifactor model with one group factor
0	.5	0	−.04	−.03	.03	.02	−.03	.01	.01	.00
	.7	0	−.02	−.02	.01	.01	.00	.09	.00	.00
Bifactor model with two group factors
0	.5	.05	.05	.04	.07	.06	.02	.06	.05	.03
	.7	.04	.03	.03	.04	.04	.02	.04	.03	.02

Note. PCV = proportion of common variance. Values that yielded ≥.10 bias were bolded.

We begin by describing the results in Table 2 for the first factor. ${\hat{π}}_{SMC : k'}$ was positively biased in all conditions, largely attributed to the bias at the population level. Disturbingly, the mean of ${\hat{π}}_{SMC : k'}$ yielded out-of-bound values (i.e., greater than 1) in 17 of 36 conditions, and the bias was greater for the larger sample size. For $π_{SMC : k'}^{+ \hat{Λ}}$ , the bias was minimal (within .03) in 22 conditions, moderate (between .04 and .07) in 12 conditions, and large (>.07) in the remaining 2 conditions. $π_{SMC : k'}^{+ \hat{Λ}}$ demonstrated less bias than either of the other two estimates in 10 of the 18 conditions when N was equal to 200, and ${\hat{π}}_{SMC : k'}^{Corrected}$ showed the least amount of bias in 7 of these conditions (with 1 tie). $π_{SMC : k'}^{+ \hat{Λ}}$ had the least bias in all 18 conditions when N was equal to 400.

The results for the second factor are shown in Table 3. ${\hat{π}}_{SMC : k'}$ yielded the least bias in 7 conditions; $π_{SMC : k'}^{+ \hat{Λ}}$ yielded the least biased in 18 conditions; and ${\hat{π}}_{SMC : k'}^{Corrected}$ showed the least bias in 8 conditions (with ties between at least two of the three estimators in the other 3 conditions). It should be noted that for the conditions in which ${\hat{π}}_{SMC : k'}$ was least biased, the other estimates were generally only slightly more biased. Also ${\hat{π}}_{SMC : k'}^{Corrected}$ performed best for models with one underlying factor.

The results for the third factor are shown in Table 4. For the models with $π_{k'} = 0$ , ${\hat{π}}_{SMC : k'}^{Corrected}$ yielded the least biased estimates of $π_{k'}$ for 15 out of the 20 conditions, but the degree of bias was small in general. For the models with $π_{k'} > 0$ , ${\hat{π}}_{SMC : k'}$ was the least biased estimate of $π_{k'}$ for 7 conditions; $π_{SMC : k'}^{+ \hat{Λ}}$ for 3 conditions; and ${\hat{π}}_{SMC : k'}^{Corrected}$ for 1 condition (with ties between two estimates for the remaining 5 conditions). However, the degree of bias was relatively similar across conditions relative to the results for the first and second factors.

In summary, no single estimator performed consistently better than others across all 54 conditions for the first, second, and third factors. However, $π_{SMC : k'}^{+ \hat{Λ}}$ overall tended to demonstrate less bias than the other two estimators. In addition, the standard deviations of the three estimates across replications (i.e., the empirical standard errors) were very similar to each other, although they displayed a consistent pattern across all conditions: $π_{SMC : k'}^{+ \hat{Λ}}$ yielded the smallest standard deviation, followed by ${\hat{π}}_{SMC : k'}^{Corrected}$ and then ${\hat{π}}_{SMC : k'}$ .

Study 2

A major difficulty with the considered PCVs is that they are based on negative eigenvalues for later factors. These results are nonsensical in that eigenvalues represent the variance accounted for by a factor, which should be greater than or equal to zero. ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ tended to produce less biased estimates of $π_{k'}$ in comparison with ${\hat{π}}_{SMC : k'}$ and ${\hat{π}}_{SMC : k'}^{Corrected}$ ; however, ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ is an ad hoc modification to ${\hat{π}}_{SMC : k'}$ .

A more elegant mathematical alternative is to use an EFA method that does not allow for negative eigenvalues. There is such a method, minimum rank factor analysis (MRFA; Lorenzo-Seva, 2013; Shapiro & Ten Berge, 2002; Sočan, 2003; Ten Berge & Kiers 1991; Ten Berge & Sočan, 2004). MRFA yields optimal communalities for an observed correlation matrix in the sense that the reduced correlation matrix is positive semidefinite; that is, MRFA does not allow for negative eigenvalues. MRFA is not as well known as other EFA methods (e.g., PAF and maximum likelihood) and is not part of major statistical packages. However, it is available as Windows (Lorenzo-Seva & Ferrando, 2006) and R programs (Navarro-Gonzalez & Lorenzo-Seva, 2017).

MRFA eliminates the problem with negative variances for factors and thus may yield more accurate estimates at the population and sample levels (denoted $π_{MRFA : k'}$ and ${\hat{π}}_{MRFA : k'}$ , respectively). Previous research indicates that MRFA yields positively biased estimates for unexplained common variance after extracting a fixed number of factors, particularly as a function of sample size (Shapiro & Ten Berge, 2002; Sočan, 2003). The implication is that ${\hat{π}}_{MRFA : k'}$ is likely to be biased, although the degree and type of bias (i.e., negatively or positively biased) are likely to differ across the extracted factors. Shapiro and Ten Berge (2002) offered a method to compute the asymptotic bias of the unexplained variance, which is appropriate for the analysis of covariance matrices, but not for correlation matrices.

A study by Timmerman and Lorenzo-Seva (2011) proposed a parallel analysis approach that incorporates MRFA as the factor extraction method, and judgments about the number of factors are made based on proportions of explained common variance. The results of their Monte Carlo study indicated that, under a number of conditions, the proposed method can yield relatively accurate conclusions in the assessment of the number of factors for ordered polytomous items. Their study offers some support for MRFA; however, they did not investigate the accuracy of MRFA in estimating PCV, the focus of the current study. Thus, it is unknown based on their results whether MRFA is a useful effect size statistic for parallel analysis.

Purpose of Study 2

The purpose of Study 2 is to explore the psychometric properties of $π_{MRFA : k'}$ and ${\hat{π}}_{MRFA : k'}$ and to compare their properties to those for $π_{SMC : k'}$ and ${\hat{π}}_{SMC : k'}$ as well as $π_{SMC : k'}^{+ Λ} and {\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ . We did not include ${\hat{π}}_{SMC : k'}^{Corrected}$ because the index is undefined in the population, is not a standard estimate of PCV, and is less accurate than ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ .

Method

We used the same design in Study 2 as we employed in Study 1 with one exception. We included sample sizes of 1,000, 2,000, and 5,000 besides 200 and 400 to explore large-sample properties. We generated data and conducted analyses of these data using comparable methods as used in Study 1.

Results

Bias at the Population Level

At the population level, $π_{MRFA : k'}$ yielded the same values as $π_{k'}$ ; that is, $π_{MRFA : k'}$ perfectly reproduced the proportion of common variance in the population. Thus, $π_{MRFA : k'}$ was superior to $π_{SMC : k'}$ and $π_{SMC : k'}^{+ Λ}$ at the population level.

Bias at the Sample Level

At the sample level, we present in Table 5 the results of ${\hat{π}}_{MRFA : k'}$ for the first, second, and third extracted factor for sample sizes of 200 and 400. For the first extracted factor, ${\hat{π}}_{MRFA : k'}$ was consistently negatively biased. ${\hat{π}}_{MRFA : k'}$ , on average, was .14 less than $π_{k'}$ when N = 200 and .11 less than $π_{k'}$ when N = 400. ${\hat{π}}_{MRFA : k'}$ was a much less accurate estimator of $π_{k'}$ in comparison with $π_{SMC : k'}^{+ \hat{Λ}}$ . The mean absolute differences between $π_{SMC : k'}^{+ \hat{Λ}}$ and $π_{k'}$ were .10 and .08 for sample sizes of 200 and 400, respectively. In comparison, the mean absolute differences between $π_{SMC : k'}^{+ \hat{Λ}}$ and $π_{k'}$ were .04 and .02 for sample sizes of 200 and 400, respectively.

Table 5.

PCV at the Population Level and Mean Estimate of PCV Using MRFA at the Sample Level for the First Three Factors.

		Factor 1			Factor 2			Factor 3
		$π_{k' : correct}$	${\bar{\hat{π}}}_{MRFA : k'}$		$π_{k' : correct}$	${\bar{\hat{π}}}_{MRFA : k'}$		$π_{k' : correct}$	${\bar{\hat{π}}}_{MRFA : k'}$
ρ_FF′	λ	$π_{k' : correct}$	N = 200	N = 400	$π_{k' : correct}$	N = 200	N = 400	$π_{k' : correct}$	N = 200	N = 400
One-factor underlying all indicators
—	.5	1.00	.68	.74	0	.12	.10	0	.09	.07
—	.7	1.00	.85	.89	0	.06	.04	0	.04	.03
Two-factor, perfect-cluster model
0	.5	.50	.43	.44	.50	.34	.37	0	.11	.09
	.7	.50	.48	.49	.50	.41	.43	0	.05	.04
.4	.5	.70	.51	.55	.30	.25	.26	0	.11	.09
	.7	.70	.62	.64	.30	.28	.28	0	.05	.04
.8	.5	.90	.63	.69	.10	.15	.13	0	.10	.08
	.7	.90	.78	.81	.10	.11	.11	0	.05	.04
Three-factor, perfect-cluster model
0	.5	.33	.28	.29	.33	.24	.25	.33	.19	.22
	.7	.33	.33	.33	.33	.29	.30	.33	.24	.27
.4	.5	.60	.39	.43	.20	.19	.19	.20	.14	.15
	.7	.60	.50	.52	.20	.20	.20	.20	.16	.17
.8	.5	.87	.53	.59	.07	.12	.10	.07	.09	.08
	.7	.87	.71	.75	.07	.09	.08	.07	.07	.07
Bifactor model with one group factor
0	.5	.87	.71	.75	.13	.14	.14	0	.07	.05
	.7	.91	.84	.86	.09	.10	.09	0	.03	.02
Bifactor model with two group factors
0	.5	.75	.58	.62	.20	.17	.18	.05	.08	.07
	.7	.84	.75	.77	.13	.12	.12	.04	.05	.04

Note. PCV = proportion of common variance; MRFA = minimum rank factor analysis. Values that yielded ≥.10 bias were bolded.

For the second extracted factor, ${\hat{π}}_{MRFA : k'}$ was on average relatively unbiased. Similar to the first extracted factor, ${\hat{π}}_{MRFA : k'}$ was a somewhat less accurate estimator of $π_{k'}$ than ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ . The mean absolute differences between ${\hat{π}}_{MRFA : k'}$ and $π_{k'}$ were .05 and .04 for sample sizes of 200 and 400, respectively. In comparison, the mean absolute differences between ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ and $π_{k'}$ were .03 and .02 for sample sizes of 200 and 400, respectively.

For the third extracted factor, ${\hat{π}}_{MRFA : k'}$ tended to be positively biased. ${\hat{π}}_{MRFA : k'}$ , on average, was .06 greater than $π_{k'}$ when N = 200 and .05 greater than $π_{k'}$ when N = 400. ${\hat{π}}_{MRFA : k'}$ was a somewhat less accurate estimator of $π_{k'}$ than ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ . The mean absolute differences between ${\hat{π}}_{MRFA : k'}$ and $π_{k'}$ were .06 and .05 for sample sizes of 200 and 400, respectively. In comparison, the mean absolute differences between ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ and $π_{k'}$ were .03 and .02 for sample sizes of 200 and 400, respectively.

Based on these results, ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ appeared to be a more accurate estimator of $π_{k'}$ than ${\hat{π}}_{MRFA : k'} .$ Given that $π_{MRFA : k'} = π_{k'}$ and $π_{SMC : k'}^{+ Λ} \neq π_{k'}$ at the population level, ${\hat{π}}_{MRFA : k'}$ should become a more accurate estimator of $π_{k'}$ relative to ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ as sample size increases. To assess whether ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ continues to demonstrate more accuracy for large sample sizes, we included conditions with sample sizes of 1,000, 2,000, and 5,000.

For the first extracted factor, ${\hat{π}}_{MRFA : k'}$ was generally negatively biased when N = 1,000, 2,000, and 5,000. ${\hat{π}}_{MRFA : k'}$ , on average, was .08 less than $π_{k'}$ when N = 1,000, .06 less than $π_{k'}$ when N = 2,000, and .04 less than $π_{k'}$ when N = 5,000. However, ${\hat{π}}_{MRFA : k'}$ was still a much less accurate estimator of $π_{k'}$ than ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ . In comparison, ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ tended to be positively biased; on average, it was .02 greater than $π_{k'}$ for all three sample sizes. The mean absolute differences between ${\hat{π}}_{MRFA : k'}$ and $π_{k'}$ were .08, .06, and .04 for sample sizes of 1,000, 2,000, and 5,000, respectively. In comparison, the mean absolute differences between ${\hat{π}}_{MRFA : k'}$ and $π_{k'}$ were .02 for all three sample sizes.

For the second extracted factor, both ${\hat{π}}_{MRFA : k'}$ and ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ were on average slightly negatively biased; on average, the biases for both statistics were −.01 across different sample sizes. ${\hat{π}}_{MRFA : k'}$ and ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ also displayed similar accuracies across sample size; on average, the mean absolute differences between the two alternative PCVs and $π_{k'}$ were between .01 and .02 across sample sizes.

For the third extracted factor, ${\hat{π}}_{MRFA : k'}$ tended to be slightly positively biased, with mean differences of +.01 across sample sizes. In contrast, ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ demonstrated a negative bias, with mean differences of −.02 across sample sizes. ${\hat{π}}_{MRFA : k'}$ and ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ displayed similar accuracies across sample size; on average, the mean absolute differences between the two alternative PCVs and $π_{k'}$ were between .02 and .03 across sample sizes.

In addition, the standard deviations of ${\hat{π}}_{MRFA : k'}$ across replications were very similar to those of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ . However, ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ resulted in slightly smaller standard deviations (.01 or .02 smaller) across all conditions.

Study 3

Based on the analyses of the generated data in the two previous studies, we found ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ to be the preferred PCV index. In Study 3, we focus on the usefulness of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ in combination with R-PA.

Purpose of Study 3

The objective of Study 3 was to demonstrate how ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ can be used to yield a more nuanced interpretation of R-PA. We concentrate on two situations in which researchers may choose to make decisions that go counter to the standard interpretation of R-PA after taking into account ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ : (a) inclusion of a factor that was not significant but has a nontrivial ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ and (b) exclusion of a factor that was significant but has a trivial ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ .

Method

Design

In Study 3, we generated and analyzed data for two-factor and three-factor perfect-cluster models as well as two-factor bifactor models. Because the interpretation of results for the perfect-cluster models is essentially the same as those for the two-factor bifactor models, we present the results for only the latter models. The bifactor models consisted of 8 indicators, with all indicators loading on the general factor and 4 indicators loading on the group factor.

In Study 3, we manipulated the factor loadings and sample size. Factor loadings on the general factor were .5s or .7s, whereas factor loadings on the group factor were .3s, .4s, .5s, or .6s. The number of observations was set at 200, 500, or 800.

Data Generation and Analyses

As with the previous studies, the factor and error scores for the model were generated to be normally distributed. One-thousand-sample datasets were created for each combination of the manipulated dimensions.

We conducted a series of hypothesis tests required in performing a revised parallel analysis. Following the steps of R-PA, we initially evaluated the null hypothesis that 0 factors underlie a correlation matrix. For all 1,000 replications in each condition, this hypothesis was rejected, implying that more than 0 factors were required. Next, we tested the null hypothesis that 1 factor explains the correlation matrix. The number of replications in which this hypothesis was not rejected and was rejected was recorded for each condition. Based on the standard use of R-PA, we reached one of two conclusions: nonrejection of this null hypothesis suggests 1 factor is sufficient to explain the correlation matrix, whereas rejection implies that 2 or more factors are required. For replications in which this hypothesis was rejected, an additional hypothesis test was conducted to evaluate the null hypothesis that 2 factors are necessary to explain the correlation matrix. The number of replications in which this hypothesis was not rejected and was rejected was recorded for each condition. Nonrejection of this null hypothesis suggests 2 factors are sufficient to explain the correlation matrix, whereas rejection implies that 3 or more factors are required. Given the bifactor model used to generate the data included 2 factors, we stopped testing and concluded that either the correct number of factors was determined if the hypothesis was nonsignificant or the number of factors was overestimated if the hypothesis was significant.

To assess whether the effect size statistic ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ augments the interpretation of R-PA, we report the mean of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ ( ${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$ ) separately for nonsignificant and significant results at each step in the R-PA process. The reported ${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$ depended on the hypothesis that was tested. More specifically, when the null hypothesis was assessing $k' - 1$ underlying factors, the effect size was computed for the $k'$ factor $({\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}})$ .

In interpreting the results, it is important to keep in mind two issues. First, hypothesis test results are a function of sample size and effect size. Thus, for any one condition, the mean effect size value will be greater for significant versus nonsignificant hypothesis tests. Second, the number of observations decreases as one proceeds through the sequence of tests with R-PA in that R-PA does not proceed when a hypothesis test is nonsignificant.

Results

In Table 6, we present ${\bar{\hat{π}}}_{SMC : k'}^{+ \hat{Λ}}$ within the sequence of steps of R-PA. In this table, we focus on the types of outcomes with effect sizes that suggest an alternative estimate of the number of factors relative to R-PA.

Table 6.

Percent Correct with R-PA as well as ${\bar{\hat{π}}}_{k' :}^{+ \hat{Λ}}$ (Number of Replications Out of 1,000) for Nonsignificant (NS) and Significant (Sig.) Tests of Null of Hypotheses of One or Two Factors for Data Generated Using a Bifactor Model With One General Factor and One Group Factor

		N = 200					N = 500					N = 800
			${\bar{\hat{π}}}_{SMC : k' = 2}^{+ \hat{Λ}}$ (# of reps)		${\hat{π}}_{SMC : k' = 3}^{+ \hat{Λ}}$ (# of reps)			${\bar{\hat{π}}}_{SMC : k' = 2}^{+ \hat{Λ}}$ (# of reps)		${\hat{π}}_{SMC : k' = 3}^{+ \hat{Λ}}$ (# of reps)			${\bar{\hat{π}}}_{SMC : k' = 2}^{+ \hat{Λ}}$ (# of reps)		${\hat{π}}_{SMC : k' = 3}^{+ \hat{Λ}}$ (# of reps)
			H₀: 1 factor; H₁: ≥2 factors		H₀: 2 factors; H₁≥3 factors			H₀: 1 factor; H₁: ≥2 factors		H₀: 2 factors; H₁≥3 factors			H₀: 1 factor; H₁: ≥2 factors		H₀: 2 factors; H₁≥3 factors
λ_Gen	Λ_Grp	% Cor.	NS	Sig.	NS	Sig.	% Cor.	NS	Sig.	NS	Sig.	% Cor.	NS	Sig.	NS	Sig.
.5	.3	15.2	.071 (833)	.118 (167)	.029 (152)	.068 (15)	47.8	.038 (495)	.067 (505)	.010 (478)	.031 (27)	69.5	.025 (256)	.051 (744)	.005 (695)	.016 (49)
	.4	50.4	.075 (471)	.116 (529)	.025 (504)	.060 (25)	90.4	.041 (62)	.085 (938)	.008 (904)	.028 (34)	97.1	.028 (6)	.077 (994)	.004 (971)	.013 (23)
	.5	79.6	.072 (176)	.124 (824)	.020 (796)	.053 (28)	97.4	− (0)	.108 (1000)	.007 (974)	.021 (26)	98.8	− (0)	.106 (1000)	.003 (988)	.012 (12)
	.6	93.2	.072 (39)	.133 (931)	.016 (932)	.047 (29)	97.4	− (0)	.125 (1000)	.005 (974)	.016 (26)	98.0	− (0)	.123 (1000)	.002 (980)	.006 (20)
.7	.3	41.9	.028 (563)	.048 (437)	.009 (419)	.024 (18)	88.2	.013 (87)	.030 (913)	.002 (882)	.009 (31)	96.9	.007 (6)	.026 (994)	.001 (969)	.003 (25)
	.4	89.5	.032 (74)	.059 (926)	.008 (895)	.022 (31)	97.7	− (0)	.052 (1000)	.002 (977)	.006 (23)	98.0	− (0)	.050 (1000)	.001 (980)	.002 (20)
	.5	97.3	.024 (1)	.080 (999)	.006 (973)	.018 (26)	97.8	− (0)	.077 (1000)	.001 (978)	.006 (22)	99.2	− (0)	.076 (1000)	.001 (992)	.001 (8)
	.6	97.1	− (0)	.102 (1000)	.004 (971)	.014 (29)	97.0	− (0)	.101 (1000)	.001 (970)	.004 (30)	100.0	− (0)	.099 (1000)	− (0)	− (0)

Nonsignificant Test and Nontrivial Effect Size

We first considered conditions when R-PA failed to reach the correct number of factors due to small sample size and thus a lack of power of the R-PA significance tests. This occurred most frequently with nonsignificant tests assessing the null hypothesis of a single underlying factor, a sample size of 200, and loadings on the general factor of .5s (e.g., 83.3% and 47.1% of the replications when the loadings on the group factors were .3s and .4s, respectively). In these conditions, ${\bar{\hat{π}}}_{SMC : k' = 2}^{+ \hat{Λ}}$ were .07 or greater, indicating that approximately 7% of the common variance is accounted for by the second factor. Given this effect size, researchers might consider that a second factor underlies the variables rather than using the stopping rule of R-PA and reaching the conclusion that a single factor is sufficient.

Although it may be tempting to suggest a cutoff criterion for ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ (e.g., a factor is defined as relevant if ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}} > . 05$ ), we believe that setting such a cutoff is counterproductive. Rather, we believe researchers should consider the results of R-PA, ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ , and the rotated factor solutions in the context of the variables that are being analyzed and what they are purported to measure. It is interesting to note that in the same conditions ${\bar{\hat{π}}}_{SMC : k' = 3}^{+ \hat{Λ}}$ (i.e., the effect size for the third factor) ranged in value from .016 to .029 for nonsignificant results when evaluating the null hypothesis that 2 factors are sufficient. Thus, the mean effect sizes for the second factor were twice the size, or greater, than the mean effect sizes for the third factor when data were generated with two underlying factors.

Significant Test and Trivial Effect Size

We next examined conditions when R-PA reached nominally the correct number of factors, but one of the factors was sufficiently weak that it might be evaluated as psychometrically inconsequential. This result can occur if sample size is large, and thus the R-PA significance tests have high power. To address this possibility, we key on the results for a sample size of 800, factor loadings on the general factor of .7s, and factor loadings on the group factor of .3s. For this condition, 99.4% of the replications yielded significant tests; however, the ${\bar{\hat{π}}}_{SMC : k' = 2}^{+ \hat{Λ}}$ for the significant results was only .026. If researchers found similar findings for their data, they might decide that a single factor is adequate to explain the correlation among the variables. Before making a final decision, however, researchers should examine the one-factor solution as well as rotated factor solutions (and in particular two-factor solutions) in the context of the analyzed variables and their purported meaning.

For our Monte Carlo study, we increased the loadings on the group factor from .3 to .6, in essence defining a stronger group factor. Appropriately the ${\bar{\hat{π}}}_{SMC : k' = 2}^{+ \hat{Λ}}$ increased by .025 for each increase of .1 on the group factor loadings. Thus, researchers are less likely to call the second factor inconsequential as the group factor increases in strength.

Conclusion

Initially, researchers who are conducting an exploratory factor analysis first must determine the number of factors underlying the reduced correlation matrix among variables. Methods like R-PA examine the eigenvalues associated with the extracted factors; these eigenvalues give the common variance accounted for by a factor. It is convenient to examine the eigenvalues relative to each other, more explicitly the PCV accounted by a factor. Some major statistical packages compute ${\hat{π}}_{SMC : k'}$ by dividing the eigenvalue for a factor by the sum of the eigenvalues. The problem with this approach is that some eigenvalues are negative and thus nonsensical. We suggested alternatives to this index, including ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ , which is computed by excluding the negative eigenvalues. Based on Study 1, this index overall outperformed the alternatives.

An extraction method, minimum rank factor analysis, obviates problems with ${\hat{π}}_{SMC : k'}$ in that it does not allow for negative eigenvalues. The resulting index of ${\hat{π}}_{MRFA : k'}$ worked optimally in Study 2 at the population level, but was outperformed by ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ for small to moderately large samples (i.e., sample sizes of 200 to 5,000).

Given that ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ overall tended to produce better estimates than the investigated alternatives, we explored in Study 3 the use of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ in combination with R-PA. As described in this study, researchers are likely to seek a binary decision using a cutoff criterion: unacceptable or acceptable. Accordingly, researchers may want a cutoff criterion for ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ to aid their decisions, such that when ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ is above the cutoff, they accept the next factor, and otherwise, reject the next factor. However, we argue that researchers should resist the use of cutoffs. In making an interpretation of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ , it is crucial to take into account that it is a “partialled” statistic; that is, it examines the proportion of common variance accounted for by a factor after partialling out previously extracted factors. Thus, the magnitude of ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ can be quite small for later extracted factors.

The decision about the number of factors is a complex one and should involve multiple methods (Henson & Roberts, 2006; Velicer, Eaton, & Fava, 2000). In making this decision, it is important not to dismiss the context of the study. Fabrigar et al. (1999) made this point quite clearly,

Furthermore, it is important to remember that the decision of how many factors to include in a model is a substantive issue as well as a statistical issue. A model that fails to produce a rotated solution that is interpretable and theoretically sensible has little value. Therefore, a researcher should always consider relevant theory and previous research when determining the appropriate number of factors to retain. (p. 281)

R-PA, as well as any other method to determine the number of factors, can be inaccurate, particularly under conditions with small samples, measurement items that have poor quality, and factors that are highly correlated. In such cases, researchers should be aware of the fact that a single answer for the number of factors obtained from R-PA can be misleading. An effect size statistic, in combination with substantive considerations, can thus help researchers determine the number of factors in a more nuanced way. For our Monte Carlo study, we illustrated how ${\hat{π}}_{SMC : k'}^{+ \hat{Λ}}$ can augment the results of R-PA.

The simulation design in the present study can be extended in a couple of directions. First, we employed models that have simple or bifactor structures with less than four factors. Psychology and educational research can involve many factors (e.g., eight factors) with complicated cross-loading structures. It would be useful to investigate effects size statistics for parallel analyses with more complex structures. Second, we assumed that factor, error, and observed scores all follow multivariate normal distributions. When nonnormality exists or when data are collected based on Likert-type scales, it remains a question as to whether our conclusions hold. Future studies are needed to address these concerns.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Buja

Eyuboglu

(1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27, 509-540.

Cudeck

Browne

M. W.

(1992). Constructing a covariance matrix that yields a specified minimizer and a specified minimum discrepancy function value. Psychometrika, 57, 357-369.

Cureton

E. E.

D’Agostino

R. B.

(2013). Factor analysis: An applied approach. London, England: Psychology Press.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.

Ford

J. K.

MacCallum

R. C.

Tait

(1986). The applications of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291-314.

Glorfeld

L. W.

(1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377-393.

Green

S. B.

Levy

Thompson

M. S.

W.-J.

(2012). A proposed solution to the problem with using completely random data to assess the number of factors with parallel analysis. Educational and Psychological Measurement, 72, 357-374.

Green

S. B.

Redell

Thompson

M. S.

Levy

(2016). Accuracy of revised and traditional parallel analyses for assessing dimensionality with binary data. Educational and Psychological Measurement, 76, 5-21.

Green

S. B.

Thompson

M. S.

Levy

W.-J.

(2015). Type I and II error rates and overall accuracy of the revised parallel analysis method for determining the number of factors. Educational and Psychological Measurement, 75, 428-457.

10.

Green

S. B.

Thompson

(2018). Relative accuracy of two parallel analysis methods that use the proper reference distribution. Educational and Psychological Measurement, 78, 589-604.

11.

Harshman

R. A.

Reddon

J. R.

(1983). Determining the number of factors by comparing real with random data: A serious flaw and some possible corrections. Proceedings of the Classification Society of North America at Philadelphia, pp. 14-15.

12.

Henson

R. K.

Roberts

J. K.

(2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66, 393-416.

13.

Lorenzo-Seva

(2013). How to report the percentage of explained common variance in exploratory factor analysis. Unpublished manuscript.

14.

Lorenzo-Seva

Ferrando

P. J.

(2006). FACTOR: A computer program to fit the exploratory factor analysis model. Behavior Research Methods, 38(1), 88-91.

15.

Mulaik

S. A.

(2009). Foundations of factor analysis. Boca Raton, FL: CRC Press.

16.

Navarro-Gonzalez

Lorenzo-Seva

(2017). Dimensionality assessment using Minimum Rank Factor Analysis. Retrieved from https://cran.r-project.org/web/packages/DA.MRFA/index.html

17.

R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/

18.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696.

19.

Ruscio

Roche

(2012). Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. Psychological Assessment, 24(2), 282-292.

20.

SAS Institute Inc. (2009). SAS/STAT® 9.2 user’s guide (2nd ed.). Cary, NC: Author.

21.

Schmitt

T. A.

(2011). Current methodological considerations in exploratory and confirmatory factor analysis. Journal of Psychoeducational Assessment, 29, 304-321.

22.

Shapiro

Ten Berge

J. M.

(2002). Statistical inference of minimum rank factor analysis. Psychometrika, 67, 79-94.

23.

Sočan

(2003). The incremental value of minimum rank factor analysis (Unpublished doctoral dissertation). University of Groningen, Netherlands.

24.

StataCorp. (2013). Stata: Release 13 [Statistical software]. College Station, TX: Author.

25.

Ten Berge

J. M.

Kiers

H. A

. (1991). A numerical approach to the approximate and the exact minimum rank of a covariance matrix. Psychometrika, 56, 309-315.

26.

Ten Berge

J. M.

Sočan

. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69, 613-625.

27.

Timmerman

M. E.

Lorenzo-Seva

(2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209-220.

28.

Tucker

L. R.

Koopman

R. F.

Linn

R. L.

(1969). Evaluation of factor analytic research procedures by means of simulated correlation matrices. Psychometrika, 34, 421-459.

29.

Turner

N. E.

(1998). The effect of common variance and structure pattern on random data eigenvalues: Implications for the accuracy of parallel analysis. Educational and Psychological Measurement, 58, 541-556.

30.

Velicer

W. F.

Eaton

C. A.

Fava

J. L.

(2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In Goffin

R. D.

Helmes

(Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at seventy (pp. 41-71). New York, NY: Springer.

Proportion of Indicator Common Variance Due to a Factor as an Effect Size Statistic in Revised Parallel Analysis

Abstract

Keywords

R-PA, Hypothesis Testing, and Effect Size

Purpose of Article

Study 1

Definition of PCV as a Factor Effect Size Index

A Problem With π ^ SMC : k ′

Alternative Estimators of PCVs

Purpose of Study 1

Method

Design

Data Generation and Analyses

Results

Bias at the Population Level

Bias at the Sample Level

Study 2

Purpose of Study 2

Method

Results

Bias at the Population Level

Bias at the Sample Level

Study 3

Purpose of Study 3

Method

Design

Data Generation and Analyses

Results

Nonsignificant Test and Nontrivial Effect Size

Significant Test and Trivial Effect Size

Conclusion

Footnotes

Declaration of Conflicting Interests

Funding

Notes

References

A Problem With ${\hat{π}}_{SMC : k'}$