Exploratory Bifactor Analysis of the WJ-III Cognitive in Adulthood via the Schmid

Abstract

The Woodcock–Johnson-III cognitive in the adult time period (age 20 to 90 plus) was analyzed using exploratory bifactor analysis via the Schmid–Leiman orthogonalization procedure. The results of this study suggested possible overfactoring, a different factor structure from that posited in the Technical Manual and a lack of invariance across both age ranges under study. Even when forcing the seven-factor fit, the structure was problematic. The results from the 20 to 39 age group displayed patterns of convergence with and divergence from the Technical Manual’s structure. The results from the 40 and above age group were generally consistent with the Technical Manual’s structure except for retrieval fluency. This study is consistent with the body of exploratory factor analysis structural validity evidence suggesting that contemporary tests of cognitive ability, particularly those based on Cattell–Horn–Carroll theory, are overfactored and lack alignment with their respective Technical Manual’s presented structure.

Keywords

exploratory bifactor analysis higher order factor analysis Cattell–Horn–Carroll theory Schmid–Leiman orthogonalization general intelligence

The Woodcock–Johnson-III (WJ-III) is a widely used instrument that is often cited for its utility in clinical diagnosis and educational classification. Containing an element of circularity, the WJ-III served as the initial evidentiary basis for Cattell–Horn–Carroll (CHC) theory, while CHC theory influenced the development of the WJ-III. The test authors have promoted CHC taxonomy as “ . . . the most comprehensive and empirically supported framework available for understanding the structure of human cognitive abilities” (p. 9; McGrew & Woodcock, 2001). With such a bold claim and because the WJ-III is potentially used in decisions about thousands of individuals’ lives, the stakes are high for the instrument’s structure.

The WJ-III structure has primarily been investigated using confirmatory factor analysis (CFA) methodology. These investigations have largely supported its structure (e.g., Keith & Reynolds, 2010; Taub & McGrew, 2004). There are two notable recent exceptions. Dombrowski (2013) and Dombrowski and Watkins (2013) subjected the two school-age correlations matrices (ages 9 to 13 and 14 to 19) in the WJ-III full battery and the WJ-III Cognitive to exploratory bifactor analysis via a Schmid–Leiman (SL) orthogonalization procedure. In both studies, these researchers demonstrated not only possible overfactoring but also a lack of structural invariance across the two age ranges. The lack of factor invariance is not restricted solely to the WJ-III (e.g., Stanford Binet–Fifth edition [SB5] via Canivez, 2008; Cognitive Assessment System [CAS] via Canivez, 2011; SB5 via DiStefano & Dombrowski, 2006; Reynolds Intellectual Assessment Scales [RIAS] via Dombrowski, Watkins, & Brogan, 2009), but it poses a serious problem. Tests of cognitive ability require factorial invariance across developmental periods so that results from one age period may be compared with results in another (i.e., age 9 to age 16; Horn, McArdel, & Mason, 1983; Labouvie & Ruetsch, 1995; Reise, Widaman, & Pugh, 1993). If structure is found to be inconsistent across groups, then this considerably degrades capacity to compare scores across developmental periods (Reise et al., 1993) and limits the instrument’s clinical use.

The purpose of this study is to conduct an exploratory and hierarchical factor analysis of the WJ-III Cognitive on two normative sample (e.g., ages 20 to 39; 40 plus) correlation matrices that span the adult time period. The WJ-III Cognitive in the adult time period has never been subjected to exploratory and higher order factor analysis. Instead, an understanding of the structure of the WJ-III Cognitive in adulthood rests primarily on CFA.

However, several structural validity studies are available in the adult literature. The Wechsler Adult Intelligence Scale (WAIS) has been analyzed using the SL procedure as well as bifactor CFA analysis. These studies found that the instruments are primarily dominated by the g factor (e.g., the French WAIS-III [Golay & Lecerf, 2011] and the WAIS-IV [Canivez & Watkins, 2010a, 2010b; Niileksela, Reynolds, & Kaufman, 2012]) and that primary interpretive emphasis should reside at that level.

There are several choices for exploratory factor analysis, but one with a well-established research base is the SL orthogonalization procedure (Schmid & Leiman, 1957). The SL procedure is a type of bifactor model that has been widely used by psychometric researchers in cognitive ability (Canivez, 2013). In fact, John Carroll used this procedure, and literally insisted on its use, when he created his Three Stratum Theory of Cognitive Abilities (Carroll, 1993). It is perplexing that the SL was overlooked by the WJ-III test authors when Carroll’s theory was considerably influential in the development of the WJ-III. Because the WJ-III Cognitive in adulthood has never been subjected to exploratory factor analysis, both within its Technical Manual and in the outside literature, this study seeks to fill this critical empirical gap.

Method

Participants

The WJ-III authors collected and reported information relative to seven age groups: 2 to 3 years, 4 to 5 years, 6 to 8 years, 9 to 13 years, 14 to 19 years, 20 to 39 years, and 40 years and older. The data for the WJ-III Cognitive norms were collected from a nationally representative sample of 8,818 participants from age 2 through 90 plus. The Technical Manual reports that the normative data were matched to the 2000 U.S. Census for geographic region, community size, sex, race, educational level, and occupation. Demographic characteristics are provided in the WJ-III Technical Manual. For this study, the two adult (20 to 39 years and 40 plus years) subtest correlation matrices (20 by 20) were obtained from the Technical Manual. The 20 to 39 age range contained an average of 1,100 participants, while the 40 plus age range contained an average of 807 participants.

Instrument

The WJ-III Tests of Cognitive Abilities (WJ-III Cognitive; Woodcock, McGrew, & Mather, 2001) contains 20 cognitive tests that are purported to measure g and seven CHC factors: visual-spatial thinking (Gv), fluid reasoning (Gf), processing speed (Gs), long-term retrieval (Glr), auditory processing (Ga), short-term memory (Gsm), and comprehension-knowledge (Gc). The WJ-III also yields a general intellectual-ability score reflective of g.

Procedure

The correlation matrices within this study were analyzed using Bartlett’s Test of Sphericity (Bartlett, 1954) and the Kaiser–Meyer–Olkin (KMO; Kaiser, 1974) statistic. The intercorrelation matrices were subjected to principal axis factoring (Cudeck, 2000; Fabrigar, Wegener, MacCallum, & Strahan, 1999; Tabachnick & Fidell, 2007) with promax rotation (k = 4). The minimum average partial (MAP; Velicer, 1976) test and parallel analysis (Horn, 1965) were used to determine the number of factors to extract. Scree plots (Cattell, 1966) were also inspected (see Figures 1 and 2). In addition, the seven-factor solution presented in the Technical Manual was forced. Finally, a bifactor analysis using the SL (Schmid & Leiman, 1957) procedure was applied to the oblique first-order factors to elucidate the structure of the WJ-III across the two adult correlation matrices.

Figure 1.

Scree plots for HPA for the WJ-III Cognitive age 20 to 39.

Figure 2.

Scree plots for HPA for the WJ-III Cognitive age 40 Plus.

Results

Exploratory (First-Order) Analyses

Results from Bartlett’s Test of Sphericity (Bartlett, 1950) for both analyses’ age ranges indicated that the correlation matrices were not random (20 to 39 age range χ² = 12,863.01, df = 190, p < .000; 40 plus age range χ² = 11,361.11, df = 190, p < .000). For the 20 to 29 and 40 plus age ranges, the KMO (Kaiser, 1974) statistic was .905 and .934, respectively, well above the minimum standard for conducting a factor analysis suggested by Kline (1994). Measures of sampling adequacy for each variable were also within reasonable limits. Thus, the correlation matrices were appropriate for factor analysis.

Factor-extraction criteria

Parallel analysis (Horn, 1965) suggested that three factors be retained for the 20 to 39 age range, while two factors were indicated for the 40 plus age range. The MAP (Velicer, 1976) criterion recommended retention of two factors for the 20 to 39 age range and one factor for the 40 plus age range (see Figures 1 and 2). A visual scree test indicated evidence for one strong factor at both age ranges with the possibility of two additional factors at the 20 to 39 age range and one additional factor at the 40 plus age range. Because MAP recommended the retention of two factors and parallel analysis recommended the retention of three factors, both a two- and a three-factor solutions were extracted and analyzed at the 20 to 39 age range. Across both age ranges, seven factors were extracted in accord with the theoretical structure indicated in the Technical Manual. At age 20 to 39, Factors 5 through 7 were essentially trivial factors with nominal root sizes. At age 40 plus, Factors 4 through 7 were trivial factors. Psychometrically, the extraction of two factors at age 20 to 39 and 40 plus was plausible, but interpretability and linkage to theory were problematic when attempting to interpret beyond the general factor. Extraction of three factors at age 20 to 39 was also plausible but offered only a two subtest long-term-retrieval (Glr) factor and a four subtest processing-speed (Gs) factor apart from a first factor that represented an agglomeration of the remaining subtests.

Principal axis factoring with promax rotation

The adult-aged correlation matrices were separately subjected to principal axis factoring (PAF) with an oblique (promax) rotation. The age 20 to 39 and 40 plus analyses suggested that the first factor accounted for 41.35% and 50.01% of the variance, respectively. This dwarfed the variance accounted for by the second factor at the 20 to 39 and 40 plus age range (7.79% and 6.49%, respectively). Correlations between the extracted factors were as follows: The 20 to 39 age range yielded correlations between the seven factors ranging from .25 to .69 (Mdn = .41). Similarly, factor analysis of the 40 plus age range yielded correlations between the seven factors ranging from .24 to .69 (Mdn = .60). High correlation among factors suggests the possible presence of a higher order factor which needs to be extracted and examined (Gorsuch, 1983; Thompson, 2004).

Hierarchical Factor Analysis (SL Orthogonalization)

Forced seven-factor solution

Results from the SL (Schmid & Leiman, 1957) procedure on the seven-factor solution across both age ranges are presented in Tables 1 and 2. In the age 20 to 39 SL analysis, the higher order factor accounted for 35.06% of the total variance and 57.95% of the common variance. In the age 40 plus SL analysis, the higher order factor accounted for 43.92% of the total variance and 67.64% of the common variance. The general factor also accounted for between 11% and 65% (Mdn = 34.5%) of individual subtest variance in the 20 to 39 analysis. The g factor accounted for between 21% and 66% (Mdn = 42%) of individual subtest variability in the 40 plus analysis. For the 20 to 39 analyses, the seven first-order factors accounted for a small proportion of the total variance (2.37% to 6.36%) and common variance (3.92% to 10.51%). The first-and second-order factors combined to measure 60.5% of the variance in the WJ-III Cognitive, reflecting 39.5% unique variance. For the 40 plus analyses, the seven, first-order factors accounted for 2.06% to 7.49% of the total variance and 1.34% to 7.49% of the common variance. The first- and second-order factors of the 40 plus analysis combined to measure 64.9% of the variance in the WJ-III, reflecting 35.1% unique variance. The results of both analyses demonstrate a robust manifestation of general intelligence where the combined influence of general intelligence and uniqueness exceeded the contributions made by the first-order factors. The reliability of WJ-III Cognitive was also estimated with ωh, which ranged from .158 to .462 across all analyses. Low ωh coefficients suggest that interpretation of the factor indices beyond the general factor is inappropriate as little variance exists beyond the general factor (Reise, 2012).

Table 1.

WJ-III Sources of Variance According to a Schmid–Leiman Orthogonalization (Seven Factor) Ages 20 to 39.

	Second-order factor		First-order factors
Subtest	G	S²	F1	S²	F2	S²	F3	S²	F4	S²	F5	S²	F6	S²	F7	S²	h²	u²
Visual auditory (Glr)	.69	47	.72	.51	−.05	.00	.01	.00	−.01	.00	.01	.00	.04	.00	.01	.00	.99	.01
Visual auditory delayed (Glr)	.65	43	.70	.49	−.01	.00	.01	.00	.01	.00	−.01	.00	.01	.00	−.01	.00	.92	.08
Picture recognition (Gv)	.39	15	.20	.04	.22	.05	.10	.01	.04	.00	−.06	.00	−.05	.00	.06	.00	.26	.74
Pair cancellation (Gs)	.40	16	.01	.00	.65	.42	.12	.01	−.09	.01	−.01	.00	.05	.00	−.05	.00	.61	.39
Decision speed (Gs)	.47	22	−.03	.00	.61	.38	.07	.00	.09	.01	−.07	.00	−.00	.00	.03	.00	.62	.38
Visual matching (Gs)	.54	29	−.03	.00	.49	.24	.05	.00	.03	.00	.14	.02	.07	.01	.05	.00	.56	.44
Rapid naming (Gs)	.42	17	.01	.00	.35	.12	−.17	.03	.15	.02	.02	.00	.24	.06	−.08	.01	.41	.59
Spatial relations (Gv)	.58	33	.01	.00	.14	.02	.50	.25	−.14	.02	−.03	.00	.05	.00	.02	.00	.63	.37
Concept form (Gf)	.71	50	.11	.01	.05	.00	.30	.09	.06	.00	.11	.01	−.03	.00	−.09	.01	.63	.37
Analysis- synthesis (Gf)	.69	47	.09	.01	−.01	.00	.23	.05	.15	.02	.10	.01	−.08	.01	.00	.00	.57	.43
Planning (Gv)	.33	11	−.01	.00	.08	.01	.19	.03	−.06	.00	.04	.00	.05	.00	.05	.00	.16	.84
Retriev fluency (Glr)	.54	29	.05	.00	.12	.01	−.21	.04	.48	.23	.00	.00	−.03	.00	.02	.00	.58	.42
Verbal comprehen (Gc)	.80	65	−.03	.00	−.11	.01	.22	.05	.38	.14	−.02	.00	.08	.01	−.04	.00	.86	.14
General info (Gc)	.77	59	−.07	.00	−.05	.00	.20	.04	.34	.12	−.02	.00	.10	.01	.03	.00	.76	.24
Numbers reversed (Gsm)	.63	39	.01	.00	.03	.00	.06	.00	.00	.00	.39	.15	−.07	.01	−.03	.00	.55	.45
Auditory work memory (Gsm)	.70	49	.00	.00	−.02	.00	−.03	.00	.04	.00	.38	.14	.04	.00	.12	.01	.65	.35
Memory for Wrds (Gsm)	.60	36	−.03	.00	−.04	.00	.00	.00	−.06	.00	.36	.13	.24	.06	−.07	.00	.55	.45
Sound blending (Ga)	.65	42	.05	.00	.02	.00	.10	.01	−.02	.00	.03	.00	.46	.21	.04	.00	.64	.36
Incomplete words (Ga)	.60	32	.00	.00	−.01	.00	.04	.00	.07	.00	.02	.00	.39	.16	.05	.00	.49	.51
Auditory attention (Ga)	.44	19	.00	.00	−.01	.00	.01	.00	.01	.00	.00	.00	.04	.00	.68	.47	.66	.34
Common variance (%)	57.95		8.93		10.51		5.26		4.82		3.92		4.35		4.28
Total Variance (%)	35.06		5.40		6.36		3.18		2.91		2.37		2.63		2.59		60.5	39.5
Eigenvalue			8.27		1.56		1.35		1.01		0.91		0.86		0.77
%Variance			41.35		7.79		6.77		5.07		4.57		4.31		3.83
ω	.944		.862		.796		.761		.857		.772		.728		.656
ωh	.866		.403		.452		.166		.209		.166		.230		.462

Note: Loadings ≥.30 are bolded, italicized . Eigenvalues derived from PAF/Promax. The eigenvalue of the eighth, unretained factor was .759. h² = communality coefficient; u² = uniqueness; ω = omega; ωh = omega hierarchical.

Table 2.

WJ-III Sources of Variance According to a Schmid–Leiman Orthogonalization (Seven Factor) Ages 40 Plus.

	Second-order factor		First-order factors
Subtest	G	S²	F1	S²	F2	S²	F3	S²	F4	S²	F5	S²	F6	S²	F7	S²	h²	u²
Pair cancellation (Gs)	.65	42	.50	.25	−.12	.01	−.06	.00	−.01	.00	.04	.00	.17	.03	.01	.00	.72	.28
Visual matching (Gs)	.74	54	.44	.20	−.01	.00	−.05	.00	−.05	.00	.10	.01	.05	.00	.05	.00	.76	.24
Decision speed (Gs)	.65	42	.43	.19	.03	.00	.02	.00	.03	.00	−.05	.00	.01	.00	.02	.00	.61	.39
Rapid naming (Gs)	.54	29	.42	.17	.09	.01	.14	.02	.04	.00	−.02	.00	−.04	.00	−.13	.02	.51	.49
Retrieval fluency (Glr)	.64	41	.31	.10	.27	.07	.03	.00	−.01	.00	−.06	.00	−.18	.03	.05	.00	.62	.38
Picture recognition (Gv)	.51	26	.15	.02	−.04	.00	.02	.00	.13	.02	.07	.00	.14	.02	−.02	.00	.33	.67
Verbal comprehen (Gc)	.80	64	−.05	.00	.55	.30	−.03	.00	.03	.00	.04	.00	.09	.01	.00	.00	.95	.05
General information (Gc)	.76	57	−.03	.00	.52	.27	.05	.00	−.03	.00	.04	.00	.13	.02	−.03	.00	.87	.13
Incomplete words (Ga)	.57	32	−.00	.00	.03	.00	.55	.30	−.04	.00	.01	.00	.00	.00	.09	.01	.63	.37
Sound blending (Ga)	.60	36	.00	.00	−.01	.00	.50	.25	.03	.00	.11	.01	.07	.01	−.01	.00	.62	.38
Auditory attention (Ga)	.47	22	.16	.02	−.02	.00	.27	.07	.00	.00	−.02	.00	.11	.01	.01	.00	.33	.67
Visual auditory delay (Glr)	.77	59	.04	.00	−.01	.00	−.03	.00	.57	.32	−.01	.00	−.01	.00	−.00	.00	.91	.09
Visual auditory (Glr)	.81	66	−.02	.00	.02	.00	.02	.00	.53	.28	.01	.00	.01	.00	.04	.00	.94	.06
Memory for words (Gsm)	.56	31	−.04	.00	−.03	.00	.25	.06	.00	.00	.43	.18	−.02	.00	−.04	.00	.56	.44
Auditory work memory (Gsm)	.74	55	.06	.00	.07	.01	.09	.01	.00	.00	.39	.16	−.11	.01	.04	.00	.74	.26
Numb reversed (Gsm)	.70	50	.09	.01	.13	.02	−.11	.01	.02	.00	.30	.09	.03	.00	.04	.00	.62	.38
Planning (Gv)	.45	21	.06	.00	.07	.00	.10	.01	.00	.00	−.04	.00	.35	.13	−.01	.00	.35	.65
Spatial relations (Gv)	.60	36	.06	.00	.17	.03	−.03	.00	.02	.00	−.07	.01	.31	.10	.07	.00	.51	.49
Analysis-synthesis (Gf)	.77	60	−.02	.00	.01	.00	.04	.00	.05	.00	.05	.00	−.01	.00	.35	.12	.72	.28
Concept formation (Gf)	.74	55	.01	.00	.04	.00	.08	.01	.01	.00	−.03	.00	.06	.00	.33	.11	.67	.33
Common variance (%)	67.64		7.49		5.65		5.83		4.86		3.66		2.80		2.06
Total variance (%)	43.92		4.86		3.67		3.79		3.15		2.37		1.82		1.34		64.9	35.1
Eigenvalue			10.00		1.30		1.05		0.92		0.83		0.72		0.65
% Variance			50.01		6.49		5.24		4.59		4.16		3.59		3.27
ω	.960		.875		.945		.751		.962		.811		.558		.814
ωh	.900		.233		.302		.295		.314		.194		.158		.137

Note: Loadings ≥.30 are bolded, italicized . Eigenvalues derived from PAF/Promax. The eigenvalue of the eighth, unretained factor was .607. h² = communality coefficient; u² = uniqueness; ω = omega; ωh= omega hierarchical.

Hierarchical Factor Analysis Using Psychometrically Sound Factor-Extraction Rules

Age 20 to 39

In the age 20 to 39 SL analysis (see Table 3), the higher order factor accounted for 32.4% of the total variance and 65.0% of the common variance. The g factor accounted for between 11% and 54% (Mdn = 33%) of individual subtest variability. The three lower order factors accounted for 4.8% to 6.8% of the total variance and 9.7% to 13.7% of the common variance. The first- and second-order factors combined to measure 49.8% of the variance in the WJ-III Cognitive, reflecting 50.2% unique variance.

Table 3.

WJ-III Sources of Variance According to a Schmid–Leiman Orthogonalization (Three Factor) Ages 20 to 39.

	Second-order factor
Subtest	G	S²	F1	S²	F2	S²	F3	S²	h²	u²
Verbal comprehension (Gc)	0.73	53	0.50	0.25	0.05	0.00	−0.15	0.02	0.80	0.20
General information (Gc)	0.69	48	0.50	0.25	−0.01	0.00	−0.09	0.01	0.74	0.26
Incomplete words (Ga)	0.52	27	0.38	0.14	−0.05	0.00	0.00	0.00	0.42	0.58
Sound blending (Ga)	0.61	37	0.36	0.13	0.02	0.00	0.03	0.00	0.50	0.50
Auditory work memory (Gsm)	0.63	39	0.33	0.11	0.03	0.00	0.09	0.01	0.51	0.49
Retrieval fluency (Glr)	0.48	23	0.31	0.10	−0.05	0.00	0.08	0.01	0.34	0.66
Memory for words (Gsm)	0.53	29	0.31	0.09	0.01	0.00	0.06	0.00	0.38	0.62
Analysis-synthesis (Gf)	0.65	43	0.26	0.07	0.20	0.04	−0.01	0.00	0.53	0.47
Auditory attention (Ga)	0.38	14	0.24	0.06	−0.04	0.00	0.07	0.00	0.21	0.79
Numbers reversed (Gsm)	0.57	33	0.22	0.05	0.11	0.01	0.12	0.01	0.40	0.60
Planning (Gv)	0.33	11	0.11	0.01	0.07	0.00	0.09	0.01	0.13	0.87
Visual auditory delay (Glr)	0.71	50	−0.05	0.00	0.60	0.36	−0.02	0.00	0.86	0.14
Visual auditory (Glr)	0.73	54	−0.02	0.00	0.60	0.36	−0.04	0.00	0.90	0.10
Concept formation (Gf)	0.69	48	0.20	0.04	0.26	0.07	0.04	0.00	0.59	0.41
Spatial relations (Gv)	0.57	33	0.13	0.02	0.21	0.05	0.11	0.01	0.40	0.60
Picture recognition (Gv)	0.42	17	−0.01	0.00	0.21	0.05	0.19	0.03	0.26	0.74
Pair cancellation (Gs)	0.45	20	−0.09	0.01	0.06	0.00	0.62	0.39	0.60	0.40
Decision speed (Gs)	0.49	24	0.04	0.00	−0.02	0.00	0.56	0.31	0.55	0.45
Visual matching (Gs)	0.54	29	0.07	0.01	0.02	0.00	0.50	0.25	0.55	0.45
Rapid naming (Gs)	0.40	16	0.18	0.03	−0.09	0.01	0.31	0.09	0.29	0.71
Common variance (%)	65.0		13.7		9.7		11.7
Total variance (%)	32.4		6.8		4.8		5.8		49.8	50.2
ω	.933		.890		.863		.784
ωh	.804		.221		.230		.414

Note. Loadings ≥.30 are bolded, italicized . h² = communality coefficient; u² = uniqueness; ω = omega; ωh = omega hierarchical.

Age 40 plus

In the age 40 plus SL analysis (see Table 4), the higher order factor accounted for 31.2% of the total variance and 61.9% of the common variance. The g factor accounted for between 15% and 45% (Mdn = 31.5%) of individual subtest variability. For the age 40 plus analysis, the first-order factors accounted for 9.4% to 9.8% of the total variance and 18.6% to 19.5% of the common variance. The first- and second-order factors combined to measure 50.3% of the variance in the WJ-III Cognitive, reflecting 49.7% unique variance.

Table 4.

WJ-III Sources of Variance According to a Schmid–Leiman Orthogonalization (Two Factor) Ages 40 Plus.

	Second-order factor
Subtest	G	S²	F1	S²	F2	S²	h²	u²
Pair cancellation (Gs)	0.56	31	0.67	0.45	−0.19	0.04	0.80	0.20
Visual matching (Gs)	0.63	40	0.56	0.32	−0.02	0.00	0.72	0.28
Decision speed (Gs)	0.57	32	0.51	0.26	−0.02	0.00	0.58	0.42
Visual auditory delayed Glr)	0.62	38	0.37	0.13	0.17	0.03	0.54	0.46
Picture recognition (Gv)	0.43	19	0.33	0.11	0.04	0.00	0.30	0.70
Rapid naming (Gs)	0.49	24	0.33	0.11	0.10	0.01	0.36	0.64
Visual auditory (Glr)	0.66	43	0.32	0.10	0.24	0.06	0.60	0.40
Spatial relations (Gv)	0.50	25	0.32	0.10	0.12	0.01	0.37	0.63
Retrieval fluency (Glr)	0.56	31	0.26	0.07	0.22	0.05	0.43	0.57
Numbers reversed (Gsm)	0.58	34	0.25	0.06	0.25	0.06	0.47	0.53
Planning (Gv)	0.39	15	0.23	0.05	0.10	0.01	0.21	0.79
Incomplete words (Ga)	0.50	25	−0.11	0.01	0.53	0.29	0.55	0.45
Sound blending (Ga)	0.52	27	−0.06	0.00	0.51	0.26	0.54	0.46
General information (Gc)	0.65	42	0.07	0.01	0.49	0.24	0.66	0.34
Verbal comprehension (Gc)	0.67	45	0.11	0.01	0.47	0.22	0.68	0.32
Memory for words (Gsm)	0.47	22	−0.05	0.00	0.46	0.21	0.43	0.57
Auditory work memory (Gsm)	0.62	39	0.10	0.01	0.44	0.19	0.59	0.41
Analysis-synthesis (Gf)	0.62	38	0.21	0.06	0.30	0.09	0.52	0.48
Concept formation (Gf)	0.60	36	0.22	0.06	0.27	0.07	0.49	0.51
Auditory attention (Ga)	0.42	17	0.16	0.03	0.20	0.04	0.24	0.76
Common variance (%)	61.9		19.5		18.6
Total variance (%)	31.2		9.8		9.4		50.3	49.7
ω	.937		.900		.898
ωh	.749		.292		.309

Note. Loadings ≥ .30 are bolded, italicized . h² = communality coefficient; u² = uniqueness. ω = omega, ωh = omega hierarchical.

The reliability of WJ-III Cognitive for these last two analyses was also estimated with ωh, which ranged from .22 to .44 across both analyses. Again, low ωh coefficients suggest that interpretation of the factor indices beyond the general factor is inappropriate as little variance exists beyond the general factor (Reise, 2012).

Discussion

The WJ-III Cognitive test authors overlooked exploratory factor analysis (EFA) structural analyses in favor of singular reliance on CFA. This approach to cognitive ability scale development over the past decade has led to criticism of internal structure (e.g., Canivez, 2008; DiStefano & Dombrowski, 2006; Dombrowski, 2013; Dombrowski & Watkins, 2013), concern about confirmation bias (Greenwald, Pratkanis, Leippe, & Baumgardner, 1986), possible overfactoring (Canivez, 2013; Fraizer & Youngstrom, 2007), and a lack of measurement invariance (e.g., DiStefano & Dombrowski, 2006; Dombrowski, 2013). Independent research on the WJ-III has also been based primarily on CFA methodology. This body of research has largely been supportive of the factor structure of the instrument and its relationship with CHC theory (e.g., Floyd, McGrew, Barry, Rafael, & Rogers, 2009; Keith & Reynolds, 2010; Taub & McGrew, 2004). Two recent exceptions include studies using exploratory bifactor analysis (e.g., Dombrowski, 2013; Dombrowski & Watkins, 2013).

Although the WJ-III Cognitive test authors invoked Carroll’s theory as a guide to the development of the instrument, they overlooked the EFA procedures (e.g., PAF followed by an SL orthogonalization) explicitly recommended by Carroll and other experts in factor analysis (e.g., Canivez, 2013; Carroll, 1993; Gorsuch, 1983; Gustafsson & Snow, 1997; McClain, 1996; Thompson, 2004). Because of these important omissions, the two adult aged (20 to 39 years; 40 plus years) correlation matrices were subjected to EFA and higher order factor analysis.

The results of this study suggest concerns about the theoretical structure of the WJ-III Cognitive across the adult age range. Similar concerns about the structure of the WJ-III full battery and the WJ-III Cognitive were raised when the school-age period (9 to 19) was analyzed (Dombrowski, 2013; Dombrowski & Watkins, 2013). The results of the present study add to these concerns, extending the time period under question (9 to 90 plus).

Use of EFA factor-extraction procedures (e.g., parallel analysis and MAP as supplemented by a visual scree) that are considered to be the most psychometrically robust suggests that the WJ-III Cognitive is a three-factor test at ages 20 to 39 and a two-factor instrument at ages 40 plus. However, extracting this number of factors generally renders the instrument less available for interpretation because the subtest alignment lacks full linkage to theory and the structure posited in the WJ-III Technical Manual. At the 20 to 39 age range (Table 3), the subtests that load Gc (Verbal Comprehension and General Information) combine with two of the three Gsm subtests, two of the Ga subtests, and retrieval fluency (Glr) to form the first factor. The second factor loads two Glr subtests (Visual-Auditory Learning and Visual-Auditory Learning Delayed), while the third factor loads all four Gs subtests. At age 40 plus (Table 4), two factors emerged. The first factor represents a combination of the Gs, Gv, and Glr (two of three) subtests, while the second factor combines the Gc, Gsm, and Ga (two of three subtests). Structural analyses using psychometrically sound EFA procedures do not support the seven-factor model presented in the Technical Manual. They do support the WJ-III Cognitive as a solid measure of general intelligence across the 20 to 90 plus time period.

When casting aside the above mentioned factor-extraction decision rules and extracting seven factors in accord with the structure posited in the Technical Manual, there were areas of convergence with and divergence from the Technical Manual. At age 40 plus (Table 4), the forced seven-factor solution was generally consistent with the Technical Manual’s posited structure except for Glr, which instead loaded Gs. At ages 20 to 39 (Table 3), there were areas of convergence with and divergence from the Technical Manual. The Gs, Gc, and Gsm subtests alignment was consistent with the Technical Manual’s structure. Ga and Glr were partially aligned, but Glr paired with the Gc subtests. Auditory attention (Ga) formed a distinct, single subtest seventh factor. The subtests that comprise Gf and Gv paired together but had poor loadings on a third factor. Thus, forcing the seven-factor fit generally holds at 40 plus (with the exception of Glr) but does not hold at ages 20 to 39. Providing further support for exclusive interpretation of the second-order factor, the reliability of the broad factor had strong estimates (ω = .93 to .96, ωh = .75 to .90). Estimates for primary factors were low (ωh = .158 to .462) and not necessarily sufficiently high for measuring unique constructs (Reise, 2012) and for individual interpretation.

Conclusion and Implications for Practitioners

The field has moved away from its psychometric roots and cast aside EFA analyses when developing recent versions of IQ scales. Prior cognitive-ability instruments (e.g., WJ-R; SB-IV) used EFA and CFA to elucidate internal structure. This omission could pose serious problems. First, research suggests that recent intelligence tests may be overfactored (see Frazier & Youngstrom, 2007). Reasons for overfactoring have included lenient factor-extraction decision-making rules, increasingly complex models of intelligence, and commercial pressure on test publishers for an ever-increasing array of constructs to interpret. Second, the derived factor structure of an instrument helps determine how it should be interpreted with interpretation predicated on convergence among the sources of structural validity.

The results of the present study add to the body of EFA structural validity literature that cautions against overlooking interpretation of the higher order factor (g) in favor of interpretation of lower order factors (e.g., Canivez, 2013; Canivez & Watkins, 2010b; DiStefano & Dombrowski, 2006, 2013; Dombrowski & Watkins, 2013; Dombrowski et al., 2009; Glutting, Watkins, Konold, & McDermott, 2006; Nelson & Canivez, 2012; Oh, Glutting, Watkins, Youngstrom, & McDermott, 2004; Parkin & Beaujean, 2012; Watkins, 2010). It is concerning that subsequent EFA studies on instruments linked to CHC theory, including this one, have generally failed to produce evidence supportive of interpretation of lower order factors (e.g., Canivez, 2008; DiStefano & Dombrowski, 2006; Dombrowski, 2013; Dombrowski & Watkins, 2013).

Given the considerable divergence in structural results between EFA and CFA, the small amount of variance accounted for by lower order factors, and low omega hierarchical estimates, the most conservative practical implication would be to interpret the instrument where there is greatest convergent evidence (i.e., at the level of g). A more lenient approach—but one that might be asking the practitioner to skate on thin psychometric ice—would be to view lower order factors as potential screeners of a particular ability (e.g., Ga or Gsm) so long as there is convergent evidence regarding subtest alignment between independent studies and the Technical Manual. When a more thorough understanding or assessment of a particular cognitive ability is necessary, then the clinician or researcher could follow up with a full-scale assessment of that particular ability (e.g., a full-scale memory test). Interpretation of lower order factors in a test such as the WJ-III Cognitive could lead to erroneous diagnostic decision making by relying on factors that lack adequate psychometric support. The evidence is accumulating to suggest that caution should be heeded when interpreting the seven lower order, CHC factors on the WJ-III Cognitive.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bartlett

M. S

. (1950). Tests of significance in factor analysis. British Journal of Psychology, 3, 71-85.

Bartlett

M. S.

(1954). A further note on the multiplying factors for various X2 approximations in factor analysis. Journal of the Royal Statistical Society, 16, 296-298.

Canivez

G. L.

(2008). Orthogonal higher-order factor structure of the Stanford-Binet Intelligence Scales for children and adolescents. School Psychology Quarterly, 23, 533-541.

Canivez

G. L.

(2011). Hierarchical factor structure of the Cognitive Assessment System: Variance partitions from the Schmid–Leiman (1957) procedure. School Psychology Quarterly, 26, 305-317. doi:10.1037/a0025973

Canivez

G. L.

(2013). Psychometric versus actuarial interpretation of intelligence and related aptitude batteries. In Saklofske

D. H.

Reynolds

C. R.

Schwean

V. L.

(Eds.), The Oxford handbook of child psychological assessment (pp. 84-112). New York, NY: Oxford University Press.

Canivez

G. L.

Watkins

M. W.

(2010a). Exploratory and higher-order factor analyses of the Wechsler Adult Intelligence Scale-Fourth edition (WAIS-IV) adolescent subsample. School Psychology Quarterly, 25, 223-235. doi:10.1037/a0022046

Canivez

G. L.

Watkins

M. W.

(2010b). Investigation of the factor structure of the Wechsler Adult Intelligence Scale-Fourth edition (WAIS-IV): Exploratory and higher-order factor analyses. Psychological Assessment, 22, 827-836. doi:10.1037/a0020429

Carroll

J. B.

(1993). Human cognitive abilities. Cambridge, UK: Cambridge University Press.

Cattell

R. B.

(1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276.

10.

Cudeck

(2000). Exploratory factor analysis. In Tinsley

H. E. A.

Brown

S. D.

(Eds.), Handbook of multivariate statistics and mathematical modeling (pp. 265-296). New York, NY: Academic Press.

11.

DiStefano

Dombrowski

S. C.

(2006). Investigating the theoretical structure of the Stanford Binet-Fifth edition. Journal of Psychoeducational Assessment, 24, 123-136. doi:10.1177/0734282905285244

12.

Dombrowski

S. C.

(2013). Investigating the Structure of the WJ-III Cognitive at School Age. School Psychology Quarterly, 28, 154-169. doi:10.1037/spq0000010

13.

Dombrowski

S. C.

Watkins

M. W.

(2013). Exploratory and higher-order factor analysis of the WJ-III full test battery: A school age analysis. Psychological Assessment, 25, 442-455. doi:10.1037/a0031335

14.

Dombrowski

S. C.

Watkins

M. W.

Brogan

M. J.

(2009). An exploratory investigation of the factor structure of the Reynolds Intellectual Assessment Scales (RIAS). Journal of Psychoeducational Assessment, 27, 494-507. doi:10.1177/0734282909333179

15.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi:10.1037/1082-989X.4.3.272

16.

Floyd

R. G.

McGrew

K. S.

Barry

Rafael

Rogers

(2009). General and specific effects on Cattell-Horn-Carroll broad ability composites: Analysis of the Woodcock–Johnson-III Normative Update Cattell–Horn–Carroll factor clusters across development. School Psychology Review, 38, 249-265.

17.

Frazier

T. W.

Youngstrom

E. A.

(2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? Intelligence, 35, 169-182. doi:10.1016/j.intell.2006.07.002

18.

Glutting

J. J.

Watkins

M. W.

Konold

T. R.

McDermott

P. A.

(2006). Distinctions without a difference: The utility of observed versus latent factors from the WISC-IV in estimating reading and math achievement on the WIAT-II. Journal of Special Education, 40, 103-114.

19.

Golay

Lecerf

(2011). Orthogonal higher-order structure and confirmatory factor analysis of the French Wechsler Adult Intelligence Scale (WAIS-III). Psychological Assessment, 23, 143-152. doi:10.1037/a0021230

20.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

21.

Greenwald

A. G.

Pratkanis

A. R.

Leippe

M. R.

Baumgardner

M. H.

(1986). Under what conditions does theory obstruct research progress? Psychological Review, 93, 216-229.

22.

Gustafsson

J. E.

Snow

R. E.

(1997). Ability profiles. In Dillon

R. F.

(Ed.), Handbook on testing (pp. 107-135). Westport, CT: Greenwood Press.

23.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. doi:10.1007/BF02289447

24.

Horn

J. L.

McArdle

J. J.

Mason

(1983). When is invariance not invariant: A practical Scientist’s look at the ethereal concept of factor invariance. The Southern Psychologist, 1, 179-188.

25.

Kaiser

H. F.

(1974). An index of factorial simplicity. Psychometrika, 39, 31-36.

26.

Keith

T. Z.

Reynolds

(2010). Cattell-Horn-Carroll abilities and cognitive tests: What we’ve learned from 20 years of research. Psychology in the Schools, 47, 635-650. doi:10.1002/pits.20496

27.

Kline

(1994). An easy guide to factor analysis. New York, NY: Routledge.

28.

Labouvie

Ruetsch

(1995). Testing for equivalence of measurement scales: Simple structure and metric invariance reconsidered. Multivariate Behavioral Research, 30, 63-76.

29.

McClain

A. L.

(1996). Hierarchical analytic methods that yield different perspectives on dynamics: Aids to interpretation. Advances in Social Science Methodology, 4, 229-240.

30.

McGrew

K. S.

Woodcock

R. W.

(2001). Technical manual: Woodcock-Johnson III. Itasca, IL: Riverside Publishing.

31.

Nelson

J. M.

Canivez

G. L.

(2012). Examination of the structural, convergent, and incremental validity of the Reynolds Intellectual Assessment Scales (RIAS) with a clinical sample. Psychological Assessment, 24, 129-140. doi:10.1037/a0024878

32.

Niileksela

C. R.

Reynolds

M. R.

Kaufman

A. S.

(2012). An alternative Cattell–Horn–Carroll (CHC) factor structure of the WAIS-IV: Age invariance of an alternative model for ages 70-90. Psychological Assessment. Advance online publication. doi:10.1037/a0031175

33.

H. J.

Glutting

J. J.

Watkins

M. W.

Youngstrom

E. A.

McDermott

P. A.

(2004). Correct interpretation of latent versus observed abilities: Implications from structural equation modeling applied to the WISC-III and WIAT linking sample. Journal of Special Education, 38, 159-173.

34.

Parkin

J. R.

Beaujean

A. A.

(2012). The effects of Wechsler Intelligence Scale for Children—Fourth Edition cognitive abilities on math achievement. Journal of School Psychology, 50, 113-128. doi:10.1016/j.jsp.2011.08.003

35.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. doi:10.1080/00273171.2012.715555

36.

Reise

S. P.

Widaman

K. F.

Pugh

R. H.

(1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114, 552-566.

37.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF02289209

38.

Tabachnick

B. G.

Fidell

L. S.

(2007). Using multivariate statistics (5th ed.). Boston, MA: Pearson Education.

39.

Taub

G. E.

McGrew

K. S.

(2004). A confirmatory factor analysis of Cattell-Horn-Carroll theory and cross-age invariance of the Woodcock-Johnson Tests of Cognitive Abilities III. School Psychology Quarterly, 19, 72-87.

40.

Thompson

(2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.

41.

Velicer

W. F.

(1976). Determining the number of components form the matrix of partial correlations. Psychometrika, 31, 321-327. doi:10.1007/BF02293557

42.

Watkins

M. W.

(2010). Structure of the Wechsler Intelligence Scale for Children—Fourth edition among a national sample of referred students. Psychological Assessment, 22, 782-787. doi:10.1037/a0020043

43.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001). Woodcock-Johnson III Tests of Cognitive Abilities. Itasca, IL: Riverside Publishing.

Exploratory Bifactor Analysis of the WJ-III Cognitive in Adulthood via the Schmid–Leiman Procedure

Abstract

Keywords

Method

Participants

Instrument

Procedure

Results

Exploratory (First-Order) Analyses

Factor-extraction criteria

Principal axis factoring with promax rotation

Hierarchical Factor Analysis (SL Orthogonalization)

Forced seven-factor solution

Hierarchical Factor Analysis Using Psychometrically Sound Factor-Extraction Rules

Age 20 to 39

Age 40 plus

Discussion

Conclusion and Implications for Practitioners

Footnotes

Declaration of Conflicting Interests

Funding

References