Investigating the Structure of the WJ-III Cognitive in Early School Age Through Two Exploratory Bifactor Analysis Procedures

Abstract

Two exploratory bifactor methods (e.g., Schmid–Leiman [SL] and exploratory bifactor analysis [EBFA]) were used to investigate the structure of the Woodcock–Johnson III (WJ-III) Cognitive in early school age (age 6-8). The SL procedure is recognized by factor analysts as a preferred method for EBFA. Jennrich and Bentler recently developed an alternative EBFA procedure. They claim that EBFA more readily produces independent cluster structure and overcomes the proportionality constraint experienced by the SL. The results of both analyses support the preeminence of the g factor at age 6 to 8. Examination of omega coefficients, the divergent factor structure, and the small amount of variance accounted for by the lower order factors suggests caution when interpreting beyond the higher order factor. Implications for interpretation of the WJ-III Cognitive at age 6 to 8 are discussed.

Keywords

exploratory factor analysis higher order factor analysis bifactor analysis Schmid–Leiman Orthogonalization general intelligence

Since its development, the factor structure of the Woodcock–Johnson III (WJ-III) has been primarily investigated and subsequently validated using confirmatory factor analysis (CFA; see Keith et al., 2010; Taub & McGrew, 2004). This is concerning because the WJ-III was substantially revised and predicated on a new theory of cognitive ability. Generally, when a new instrument is significantly revised and reformulated based on a new theory of cognitive ability, then both exploratory and confirmatory methods are necessary to elucidate structure (Gorsuch, 1983). Exploratory factor analysis (EFA) is also appropriate when initial CFA results are unclear or inadequate (Gorsuch, 1997). In a set of independent analyses on the WJ-III, Dombrowski and colleagues (e.g., Dombrowski, 2013a, 2013b; Dombrowski & Watkins, 2013) subjected the correlations matrices (age 9 to 90 plus) to exploratory bifactor analysis (EBFA) via a Schmid–Leiman (SL) Orthogonalization. Dombrowski et al. found not only possible overfactoring but also a factor structure that diverges from that presented in the Technical Manual. Dombrowski et al. used the same exploratory bifactor procedure (i.e., SL) that was used by Carroll (1993) to create and validate his Three Stratum Theory of Cognitive Abilities. Carroll’s theory was influential in the development of Cattell–Horn–Carroll (CHC) theory and the WJ-III. It is unknown why the authors of the WJ-III chose to rely solely on CFA procedures to validate the WJ-III when the instrument was substantially revised and predicated on a new theory of cognitive ability (e.g., CHC Theory).

The purpose of this article is to investigate the structure of the WJ-III Cognitive in early school age (6-8) using two methods of EBFA (i.e., SL and EBFA). Exploratory bifactor modeling is recommended when seeking to understand new or substantially revised cognitive ability instruments because of the frequency of higher order factors and correlated traits within those domains. The SL procedure is an elegant exploratory bifactor model with a history of use in the cognitive ability literature (see Canivez, 2013; Carroll, 1995). It helps to uncover simple structure by partialing out the influence of higher order factors (Carretta & Ree, 2001; Carroll, 1993, 1995, 2003; Gustafsson & Snow, 1997). Carroll (1995) argued that orthogonal factors are appropriate only when produced in the context of a SL solution: “I insist, however, that the orthogonal factors should be those produced by the Schmid–Leiman (1957) orthogonalization procedure” (p. 437).

Concerns have been raised about a possible limitation of the SL procedure (Brunner, Nagy, & Wilhelm, 2012; Dombrowski, 2013b; Reise, 2012; Reise, Moore, & Haviland, 2010; Yung, Thissen, & McLeod, 1999). Loadings obtained via the SL may contain proportionality constraints (Yung et al., 1999) that could produce a biased estimate of corresponding population values (Reise, 2012). Jennrich and Bentler (2011) recently developed a type of EBFA that they claim overcomes the SL’s proportionality constraint. Jennrich and Bentler (2011) contended that because their analytic bifactor model is more general than the two-stage model on which the SL is based, there may be cases where EBFA produces loadings that approximate bifactor structure more readily than SL. However, an application on real-world data has not been presented in the literature. Reise et al. (2010) predicted that that the impact of proportionality on real-world data will be negligible if a researcher is interested in identifying patterns of salient and nonsalient loadings.

Because the WJ-III has been recently criticized as being overfactored and having a divergent factor structure from that posited in the Technical Manual (e.g., Dombrowski, 2013a, 2013b; Dombrowski & Watkins, 2013) and within CFA studies (Keith et al., 2010; Taub & McGrew, 2004), an analysis using two methods of exploratory bifactor modeling may be worthwhile. A basic premise of factor analysis posits that when multiple methods of factor analysis converge, then confidence in a derived factor structure may be accomplished (Gorsuch, 1983).

Method

Participants

The data for the WJ-III Cognitive norms were collected from a nationally representative sample of 8,818 participants from age 2 through 90 plus. The WJ-III Technical Manual reports that the normative data were matched to the 2000 U.S. Census for geographic region, community size, sex, race, educational level, and occupation. Demographic characteristics are provided in the WJ-III Technical Manual. For this study, the early school aged period (6-8 years) subtest correlation matrices (20 by 20) were obtained from the Technical Manual. The 6 to 8 age range contained an average of 870 participants.

Instrument

The WJ-III Tests of Cognitive Abilities (WJ-III Cognitive; Woodcock, McGrew, & Mather, 2001c) contains 20 cognitive tests that are purported to measure g and seven CHC factors: visual–spatial thinking (Gv), fluid reasoning (Gf), processing speed (Gs), long-term retrieval (Glr), auditory processing (Ga), short-term memory (Gsm), and comprehension–knowledge (Gc). The WJ-III also yields a general intellectual ability score reflective of g.

Procedure

The correlation matrices within this study were analyzed using Bartlett’s (1950, 1954) Test of Sphericity and the Kaiser–Meyer–Olkin (KMO; Kaiser, 1974) statistic to determine appropriateness for factor analysis. Next, minimum average partials (MAP; Velicer, 1976) and parallel analysis (Horn, 1965) were used to determine the number of factors to extract. These procedures were conducted using O’Conner’s (2000) SPSS program. A Horn Parallel Analysis (HPA) scree plot (Cattell, 1966; Horn, 1965) was also inspected (Figure 1) as a supplemental means to determine the number of factors to retain for rotation (Thompson, 2004; Velicer, Eaton, & Fava, 2000; Zwick & Velicer, 1986). The intercorrelation matrix was subject to EBFA (Jennrich & Bentler, 2011) using R code. The SL (1957) procedure was also applied to the oblique first-order factors following a principal axis factoring with promax rotation. The SPSS code developed by Wolff and Preising (2005) was used for this analysis. Both SL and EBFA forced the seven-factor fit, permitting a comparison of exploratory bifactor fit with the structure posited in the Technical Manual. Finally, omega coefficients were determined using a program developed by Watkins (2013).

Figure 1.

Scree plots for HPA for the WJ-III Cognitive Age 6 to 8.

Results

Exploratory (First Order) Analyses

Results from Bartlett’s (1950) Test of Sphericity indicated that the correlation matrix was not random, χ² = 6,955.81, df = 190, p < .000. The KMO (Kaiser, 1974) statistic was .875, well above the minimum standard for conducting a factor analysis suggested by Kline (1994). Measures of sampling adequacy for each variable were also within reasonable limits. Thus, the correlation matrices were appropriate for factor analysis.

Factor extraction criteria

Parallel analysis (Horn, 1965) suggested the retention of three factors whereas the MAP (Velicer, 1976) criterion recommended retention of four factors. An HPA scree test (Figure 1) indicated evidence for one strong factor with the possibility of one to two additional factors. Because MAP recommended the retention of four factors and parallel analysis recommended the retention of three factors, both a three- and a four-factor solution were extracted and analyzed. The four-factor solution was neither psychometrically nor theoretically viable as it produced a trivial fourth factor with no salient (≥.30) loadings in the SL analysis and a single subtest loading in the EBFA analysis. Therefore, the three-factor solution was considered to be the most psychometrically plausible. Because the three-factor solution diverged significantly from the seven-factor fit presented in the Technical Manual, seven factors were also extracted.

Hierarchical Factor Analysis (SL Orthogonalization and EBFA)

Forced seven-factor solution

Results from the Schmid and Leiman (1957) procedure and Jennrich & Bentler’s (2011) EBFA procedure on the forced seven-factor solution are found in Tables 1 and 2. With EBFA, the higher order factor accounted for 29.26% of the total variance and 56.27% of the common variance. The SL analysis accounted for 29.65% of the total and 60.92% of the common variance. The general factor accounted for between 7% and 55% (Mdn = 28.5%) of individual subtest variance in the EBFA. The g factor accounted for between 8% and 67% (Mdn = 29.5%) of individual subtest variability in the SL analysis. For EBFA, the seven first-order factors accounted for a small proportion of the total variance (1.857%-5.39%) and common variance (3.55%-10.36%). The first- and second-order factors combined to measure 52.0% of the variance in the WJ-III Cognitive, reflecting 48.0% unique variance. For the SL analysis, the seven, first-order factors accounted for 1.52% to 1.71% of the total variance and 3.13% to 10.01% of the common variance. The first- and second-order factors of the SL analysis combined to measure 49.0% of the variance in the WJ-III, reflecting 51.0% unique variance. The results of both analyses demonstrate a robust manifestation of general intelligence where the combined influence of general intelligence and uniqueness exceeded the contributions made by the first-order factors.

Table 1.

Schmid–Leiman Orthogonalization Procedure.

	Second-order factor		First-order factors														h²	u²
Subtest	G	S²	F1	S²	F2	S²	F3	S²	F4	S²	F5	S²	F6	S²	F7	S²	h²	u²
Visual Auditory Delay (Glr)	.62	38	.72	52	.00	0	0	0	−.03	0	−.01	0	.03	0	−.03	0	.90	.10
Visual Auditory (Glr)	.70	49	.67	44	.00	0	.01	0	.03	0	.01	0	−.03	0	.04	0	.94	.06
Pair Cancellation (Gs)	.41	17	.03	0	.63	40	.03	0	.07	1	−.12	1	−.03	0	.02	0	.59	.41
Decision Speed (Gs)	.45	20	−.02	0	.57	32	.01	0	−.08	1	.05	0	.03	0	.10	1	.54	.46
Visual Matching (Gs)	.55	30	−.03	0	.46	21	−.06	0	.03	0	.12	1	.08	1	.04	0	.54	.47
Picture Recognition (Gv)	.28	8	.08	1	.10	1	−.01	0	.03	0	.05	0	−.01	0	.06	0	.10	.90
General Information (Gc)	.73	54	.01	0	.05	0	.36	13	−.01	0	.01	0	.04	0	−.04	0	.67	.33
Verbal Comprehension (Gc)	.82	67	.03	0	.02	0	.31	9	.01	0	.07	1	.05	0	−.04	0	.77	.23
Sound Blending (Ga)	.55	30	−.02	0	−.01	0	.19	4	.13	2	−.02	0	−.06	0	.19	4	.39	.61
Incomplete Words (Ga)	.45	20	−.02	0	−.09	1	.18	3	−.01	0	.01	0	.09	1	.22	5	.30	.70
Planning (Gv)	.35	12	−.01	0	.06	0	.02	0	.32	10	−.11	1	−.02	0	.00	0	.24	.76
Memory for Words (Gsm)	.54	29	−.03	0	−.13	2	.03	0	.28	8	.03	0	.08	1	.00	0	.39	.61
Numbers Reversed (Gsm)	.61	37	.00	0	.12	1	−.02	0	.27	8	.09	1	.01	0	−.10	1	.48	.52
Spatial Relations (Gv)	.35	12	.05	0	.01	0	.01	0	.16	2	.07	0	−.10	1	−.02	0	.16	.84
Auditory Work Mem (Gsm)	.58	34	.04	0	−.07	1	−.05	0	.15	2	.13	2	.15	2	.15	2	.43	.57
Analysis-Synthesis (Gf)	.58	33	.01	0	−.02	0	.02	0	−.07	1	.36	13	−.03	0	.04	0	.47	.53
Concept Formation (Gf)	.70	49	.00	0	.01	0	.10	1	.03	0	.30	9	−.10	1	−.04	0	.60	.40
Rapid Naming (Gs)	.40	16	.01	0	.00	0	.03	0	−.01	0	−.08	1	.46	21	.10	1	.38	.62
Retrieval Fluency (Glr)	.50	25	.00	0	.09	1	.04	0	−.01	0	.01	0	.44	19	−.15	2	.47	.53
Auditory Attention (Ga)	.37	13	.00	0	.14	2	.02	0	−.02	0	.02	0	−.02	0	0.46	21	.36	.64
Common Variance (%)	60.92		10.01		10.45		3.19		3.51		3.13%		4.89%		3.90%
Total Variance (%)	29.65		4.87		5.09		1.55		1.71		1.52%		2.38%		1.90%		.49	.51
	ω_h = .841		ω_s = .455		ω_s = .460		ω_s = .088		ω_s = .131		ω_s = .149		ω_s=.092		ω_s =.144

Note. Loadings ≥.30 are italicized. Note that alignment of subtests with respective seven broad Cattell–Horn–Carroll (CHC) first-order factors posited in the Woodcock–Johnson III (WJ-III) Technical Manual is indicted following each subtest name (Gc, Gs, Ga, Glr, etc.). h² = Communality coefficient; u² = Uniqueness; ωh = Omega Hierarchical; ωs = Omega Subscale.

Table 2.

Exploratory Bifactor Analysis (Jennrich & Bentler, 2011).

	Second-order factor		First-order factors														h²	u²
Subtest	G	S²	F1	S²	F2	S²	F3	S²	F4	S²	F5	S²	F6	S²	F7	S²	h²	u²
Visual Auditory (Glr)	.68	46	.71	50	−.02	0	.03	0	−.04	0	.03	0	.05	0	.03	0	.97	.03
Visual Auditory Delay (Glr)	.61	37	.72	52	.01	0	.02	0	.00	0	−.04	0	−.05	0	−.02	0	.90	.10
Pair Cancellation(Gs)	.45	20	.02	0	.65	42	.01	0	.02	0	.01	0	.08	1	−.06	0	.64	.36
Decision Speed (Gs)	.50	25	−.03	0	.48	23	−.06	0	.09	1	.09	1	−.11	1	.08	0	.52	.48
Visual Matching (Gs)	.66	44	.10	1	.38	14	−.26	7	.02	0	−.05	0	−.12	1	.03	0	.68	.32
Picture Recognition (Gv)	.27	07	.10	1	.08	1	.01	0	.08	1	.09	1	.11	1	.15	0	.14	.02
General Information (Gc)	.63	40	.08	1	−.04	0	.53	28	.05	0	.00	0	.00	0	.02	1	.69	.31
Verbal Comprehension (Gc)	.74	55	.08	1	−.08	1	.39	15	−.01	0	−.04	0	−.01	0	.01	0	.72	.28
Sound Blending (Ga)	.50	25	.00	0	−.12	1	.23	5	−.10	1	.12	1	.10	1	−.06	1	.36	.64
Incomplete Words (Ga)	.43	18	−.03	0	−.18	3	.20	4	−.05	0	.14	2	−.07	0	−.16	0	.31	.69
Retrieval Fluency (Glr)	.51	26	−.04	0	.07	0	.02	0	.66	44	−.07	0	.01	0	.00	0	.70	.49
Rapid Naming (Gs)	.43	18	−.07	0	−.02	0	−.03	0	.32	10	.13	2	−.08	1	−.25	0	.38	.62
Auditory Attention (Ga)	.39	15	−.02	0	.05	0	.00	0	−.08	1	.60	36	.01	0	−.01	0	.52	.48
Planning (Gv)	.33	11	−.01	0	.04	0	.04	0	.01	0	.04	0	.42	18	−.01	0	.29	.71
Spatial Relations (Gv)	.34	12	.07	0	−.04	0	.00	0	−.10	1	−.06	0	.13	2	.08	0	.16	.84
Analysis-Synthesis (Gf)	.58	34	.03	0	−.10	1	−.03	0	−.03	0	.02	0	−.07	0	.35	1	.48	.52
Concept Formation (Gf)	.67	45	.05	0	−.09	1	.10	1	−.09	1	−.09	1	.04	0	.30	0	.58	.42
Numbers Reversed (Gsm)	.63	40	−.01	0	.05	0	−.10	1	−.03	0	−.12	1	.21	4	.08	0	.48	.52
Auditory Work Mem (Gsm)	.60	36	.00	0	−.12	1	−.11	1	.07	0	.14	2	.16	3	.07	0	.44	.64
Memory for Words (Gsm)	.56	31	−.07	0	−.21	4	−.04	0	−.07	0	−.08	1	.21	4	−.11	47	.44	.64
Common Variance (%)	56.27		10.36		9.13		6.12		5.83		4.74%		3.70%		3.55%
Total Variance (%)	29.26		5.39		4.75		3.18		3.03		2.46%		1.93%		1.85%		.52	.48
	ω_h = .837		ω_s = .471		ω_s = .351		ω_s = .253		ω_s = .223		ω_s =.130		ω_s =.141		ω_s =.052

Note. Loadings ≥.30 are italicized. Note that alignment of subtests with respective seven broad CHC first-order factors posited in the Woodcock–Johnson III (WJ-III) Technical Manual is indicted following each subtest name (Gc, Gs, Ga, Glr, etc.). h² = Communality coefficient; u² = Uniqueness; ω_h = Omega Hierarchical; ω_s = Omega Subscale.

The reliability of WJ-III Cognitive was also estimated across both analyses with ω_h and ω_s. The ω_h coefficient for the general factor (.841 for SL, .837 for EBFA) was high and sufficient for interpretation. Omega subscale (ω_s) coefficients for the seven lower order factors ranged from .052 to .471 across both analyses. Low ω_s coefficients suggest that interpretation of the factor indices beyond the general factor is inappropriate as little variance exists beyond the general factor (Reise, 2012).

Hierarchical factor analysis based on factor extraction rules

Tables 3 and 4 present the results of a higher order factor analysis based on the extraction of three factors. As noted, the results of both analyses suggest the prominence of the general factor and generally struggled to find subtest alignment reminiscent of that posited in the Technical Manual. With the SL analysis, the general factor accounted for 60.6% of the common variance and 24.4% of the total variance, exceeding that accounted for by the lower order factors (12.5%-15.5% common variance; 5.0%-6.2% total variance). The first- and second-order factors within the SL analysis combine to measure 40.2% of the variance in the WJ-III Cognitive, reflecting 59.8% unique variance. With EBFA, the general factor accounted for 69.7% of the common variance and 30.8% of the total variance, dwarfing that apportioned to lower order factors (5.8%-12.7% common variance; 2.6%-5.6% total variance). The first- and second-order factors in the EBFA combine to measure 44.1% of the variance in the WJ-III Cognitive, reflecting 55.9% unique variance. The results of both analyses demonstrate a robust manifestation of general intelligence in the WJ-III where the combined influence of general intelligence and uniqueness exceeded the contributions made by the first-order factors.

Table 3.

WJ-III Sources of Variance According to a Schmid–Leiman Orthogonalization (Three Factor).

	Second-order factor		F1	S²	F2	S²	F3	S²	h²	u²
Subtest	G	S²	F1	S²	F2	S²	F3	S²	h²	u²
Verbal Comprehension (Gc)	.71	50	.54	29	−.02	00	−.12	01	.64	.41
General Information (Gc)	.62	39	.32	.10	−.04	00	−.10	01	.50	.25
Sound Blending (Ga)	.49	24	.30	09	−.08	01	−.01	00	.34	.11
Memory for Words (Gsm)	.49	24	.30	09	−.02	00	.02	00	.34	.11
Incomplete Words (Ga)	.40	16	.28	08	−.10	01	.02	00	.33	.11
Concept Formation (Gf)	.61	38	.27	08	.05	00	−.12	02	.47	.22
Auditory Work Mem (Gsm)	.54	29	.26	07	.11	01	−.03	00	.37	.14
Analysis-Synthesis (Gf)	.51	26	.21	04	.09	01	−.11	01	.32	.10
Numbers Reversed (Gsm)	.55	30	.20	04	.22	05	−.06	00	.39	.15
Rapid Naming (Gs)	.36	13	.16	03	.19	03	.06	00	.19	.04
Planning (Gv)	.32	10	.16	03	.07	01	.01	00	.14	.02
Auditory Attention (Ga)	.35	12	.15	02	.12	02	.01	00	.16	.02
Spatial Relations (Gv)	.32	10	.13	.02	.02	00	−.10	.01	.13	.02
Decision Speed (Gs)	.41	17	−.01	00	.59	35	−.04	00	.52	.27
Visual Matching (Gs)	.51	26	.05	00	.59	35	−.02	00	.61	.37
Pair Cancellation (Gs)	.38	14	−.04	00	.57	32	−.07	00	.47	.22
Retrieval Fluency (Glr)	.44	19	.15	02	.28	08	.01	00	.29	.09
Picture Recognition (Gv)	.27	07	.06	00	.12	02	−.10	01	.10	.01
Visual Auditory (Glr)	.69	47	.05	00	.01	00	−.72	51	.99	.97
Visual Auditory Delay (Glr)	.60	36	.01	00	.03	00	−.69	47	.84	.70
Common Variance (%)	60.6		12.5		15.5		13.3
Total Variance (%)	24.4		5.0		6.2		5.4		40.2	59.8
	ω_h = .756		ω_s = .184		ω_s = .394		ω_s = .517

Note. Loadings ≥.30 are italicized. WJ-III = Woodcock–Johnson III; h² = Communality coefficient; u² = Uniqueness; ω_h = Omega Hierarchical; ω_s = Omega Subscale.

Table 4.

WJ-III Sources of Variance via Exploratory Bifactor Analysis (Three Factor).

	Second-order factor		F1	S²	F2	S²	F3	S²	h²	u²
Subtest	G	S²	F1	S²	F2	S²	F3	S²	h²	u²
Decision Speed (Gs)	.48	23	.54	29	−.06	00	.00	00	.52	.27
Pair Cancellation (Gs)	.44	19	.53	28	−.05	00	.03	00	.47	.22
Visual Matching (Gs)	.58	34	.52	27	.11	01	−.04	00	.62	.38
Incomplete Words (Ga)	.46	21	−.22	05	−.17	03	−.06	00	.29	.08
Sound Blending (Ga)	.54	29	−.20	04	−.03	00	−.03	00	.34	.12
General Information (Gc)	.70	49	−.19	04	−.16	03	.06	00	.55	.30
Retrieval Fluency (Glr)	.51	26	.19	04	−.14	02	−.06	00	.32	.10
Verbal Comprehension (Gc)	.79	62	−.18	03	−.08	01	.07	00	.67	.45
Picture Recognition (Gv)	.54	29	−.15	02	.17	03	−.07	00	.35	.12
Numbers Reversed (Gsm)	.61	37	.12	01	.30	09	.00	00	.48	.23
Rapid Naming (Gs)	.44	19	.10	01	−.26	07	−.11	01	.28	.08
Concept Formation (Gf)	.67	45	−.07	00	.26	07	.07	00	.53	.28
Spatial Relations (Gv)	.34	12	−.04	00	.21	04	.07	00	.16	.03
Analysis-Synthesis (Gf)	.56	31	−.01	00	.21	04	.06	00	.36	.13
Memory for Words (Gsm)	.54	29	−.15	02	.17	03	−.07	00	.35	.12
Auditory Work Mem (Gsm)	.60	36	−.01	00	.13	02	−.03	00	.38	.14
Planning (Gv)	.36	13	.00	00	.13	.02	−.04	00	.15	.02
Visual Auditory (Glr)	.69	48	−.03	00	.04	00	.68	46	.94	.88
Visual Auditory Delay (Glr)	.60	36	.02	00	−.03	00	.73	53	.89	.79
Auditory Attention (Ga)	.40	16	.05	00	−.07	00	−.04	00	.17	.03
Common Variance (%)	69.7		12.7		5.8		11.9
Total Variance (%)	30.8		5.6		2.6		5.3		44.1	55.9
	ω_h = .881		ω_s = .038		ω_s = .059		ω_s = .318

Note. Loadings ≥.30 are italicized. WJ-III = Woodcock–Johnson III; h² = Communality coefficient; u² = Uniqueness; ω_h = Omega Hierarchical; ω_s = Omega Subscale.

The reliability of WJ-III Cognitive was also estimated across both analyses with ω_h and ω_s. The ω_h coefficient for the general factor (.756 for SL, .881 for EBFA) was high and sufficient for interpretation. ω_s coefficients for the seven lower order factors ranged from .038 to .517 across both analyses. Low ω_s coefficients suggest that interpretation of the factor indices beyond the general factor is inappropriate as little variance exists beyond the general factor (Reise, 2012).

Discussion

This article investigates the theoretical structure of the WJ-III Cognitive at age 6 to 8 through two EBFA procedures. The first (e.g., SL) procedure is well established with a long-standing research history; the second (e.g., EBFA; Jennrich & Bentler, 2011) is a newly developed, promising technique with scant applied research coverage. This study is one of the first to use EBFA on applied real-world data. EBFA may be a useful additional form of factor analysis for exploring constructs with highly correlated factors or traits.

When examining the forced seven-factor solution, both procedures accounted for a similar percentage of the total variance (29.26% for EBFA and 29.65% for SL) although the SL encompassed slightly higher proportion of common variance (60.92% vs. 56.27%). When comparing specific subtest’s g loadings, the SL produced higher g loadings in 12 of the 20 subtests whereas EBFA was higher in 7 cases. EBFA produced slightly higher, first (i.e., lower) order factor subtest loadings in nine subtests although the SL had seven subtests that produced higher loadings. EBFA produced a communality of .52 reflecting .48 unique variance. The SL produced a communality of .49 reflecting .51 unique variance.

Reviewing specific subtest g loadings, 12 of the subtests in the SL analysis accounted for a greater proportion of the common variance compared with EBFA. However, the pattern of salient and nonsalient loadings was nearly identical with both procedures. This is consistent with Reise et al.’s prediction regarding impact of proportionality and could suggest a negligible influence of proportionality on results when using applied data. Additional research on the impact of proportionality is necessary. Further research on EBFA using real-world data along with Monte Carlo simulation will also be worthwhile.

The results of this study raise concerns about the theoretical structure of the WJ-III Cognitive across the early school aged time range (6-8). Both the SL and EBFA suggest the preeminence of the higher order factor and indicate that the instrument may have a different factor structure from that posited in the Technical Manual. Use of factor extraction procedures (e.g., parallel analysis and MAP as supplemented by an HPA scree test) that are considered to be psychometrically robust suggests that three factors should be extracted from the WJ-III Cognitive at 6 to 8 years of age. However, extracting this number of factors generally renders the instrument less available for interpretation because the subtest alignment lacks full linkage to theory and the structure posited in the WJ-III Technical Manual. The SL three-factor results (Table 3) suggest the location of a verbal factor (Verbal Comprehension [Gc], General Information [Gc], and Sound Blending [Ga]), a processing speed factor (Decision Speed [Gs], Visual Matching [Gs], and Pair Cancellation [Gs]), and a long-term retrieval factor (Visual Auditory Learning [Glr] and Visual Auditory Learning Delayed [Glr]). The EBFA three-factor results (Table 4) indicate the possibility of a processing speed factor (Decision Speed [Gs], Visual Matching [Gs], and Pair Cancellation [Gs]) and a long-term retrieval factor (Visual Auditory Learning [Glr] and Visual Auditory Learning Delayed [Glr]). A third factor contained only Numbers Reversed (Gsm). Thus, structural analyses using psychometrically sound EFA procedures do not support the seven-factor model presented in the Technical Manual. They do support the WJ-III Cognitive as a solid measure of general intelligence across the six to eight time periods.

When casting aside the above mentioned factor extraction decision rules and extracting seven factors in accord with the structure posited in the Technical Manual, there were areas of convergence with and divergence from the Technical Manual. Both the SL and EBFA analyses (Tables 1 and 2) found that the Gc and Gf subtests alignment was consistent with the Technical Manual’s structure. Both analyses also found that Visual Auditory Learning and Visual Auditory Learning Delayed paired together to form a distinct Glr factor whereas Retrieval Fluency (Glr) paired with Rapid Naming (Gs) to form a second processing speed factor. Auditory attention (Ga) and Planning (Gv) singularly loaded additional, separate factors (sixth and seventh). Thus, forcing the seven-factor fit does not hold across both analyses with the exception of Gc and Gf.

The forced seven-factor solution in both analyses supports the WJ-III cognitive as a solid measure of general intelligence across the six to eight time period. Moving beyond this level of interpretation is questionable. This is consistent with the conclusion of several recent studies that investigated the WJ-III (e.g., Dombrowski, 2013a, 2013b; Dombrowski & Watkins, 2013). Providing further support for sole interpretation of the higher order factor, the reliability of the g factor for both analyses had strong estimates (ω_h = .75-.88). Estimates for primary factors were low (ω_s = .038-.517) and insufficiently high for measuring unique constructs (Reise et al., 2012) and for individual interpretation.

Conclusion and Implications for Practitioners

The results of this study suggest that the WJ-III Cognitive at the 6 to 8 year time period is a solid measure of general intelligence. Consistent with the body of exploratory structural validity evidence (e.g., Canivez, 2013; Canivez & Watkins, 2010; DiStefano & Dombrowski, 2006; Dombrowski, 2013a, 2013b; Dombrowski & Watkins, 2013; Dombrowski, Watkins, & Brogan, 2009; Glutting, Watkins, Konold, & McDermott, 2006; Nelson & Canivez, 2012; Oh, Glutting, Watkins, Youngstrom, & McDermott, 2004; Parkin & Beaujean, 2012; Watkins, 2010; Watkins, Glutting, & Lei, 2007), this study suggests caution about overlooking interpretation of the higher order factor (g) in favor of interpretation of lower order factors. More specifically, this study’s results are also consistent with EFA structural validity studies on instruments linked to CHC theory that have generally failed to produce evidence that supports interpretation of lower order factors (e.g., Canivez, 2008; DiStefano & Dombrowski, 2006; Dombrowski, 2013a, 2013b; Dombrowski & Watkins, 2013).

Given the divergence in structural results between EFA and CFA analyses, the small amount of variance accounted for by lower order factors, and low Omega subscale estimates, the most empirically grounded practical implication would be to interpret the WJ-III Cognitive where there is the greatest convergent evidence (i.e., at the level of g). A less empirically defensible position would be to view lower order factors as screeners of a particular ability. However, the need for additional convergent evidence regarding subtest alignment between independent studies and the Technical Manual should not be overlooked when moving to this level of interpretation. When a more psychometrically grounded understanding of a particular cognitive ability is necessary, then the clinician or researcher could follow-up with a well-validated full scale assessment of that particular ability (e.g., a full scale test of auditory processing or memory). Interpretation of lower order factors in a test such as the WJ-III Cognitive should be approached tentatively as this practice may lead to inaccurate decision making through interpretation of factors that lack adequate psychometric support. The evidence is accumulating to suggest that caution should be headed when interpreting the seven lower order CHC factors on the WJ-III Cognitive. The present study adds to this body of research by extending this admonition downward to the 6 to 8 age range.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bartlett

M. S.

(1950). Tests of significance in factor analysis. British Journal of Psychology, 3, 71-85.

Bartlett

M. S.

(1954). A further note on the multiplying factors for various X2 approximations in factor analysis. Journal of the Royal Statistical Society, 16, 296-298.

Brunner

Nagy

Wilhelm

(2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80, 796-846.

Canivez

G. L.

(2008). Orthogonal higher-order factor structure of the Stanford-Binet Intelligence Scales for children and adolescents. School Psychology Quarterly, 23, 533-541.

Canivez

G. L.

(2013). Psychometric versus actuarial interpretation of intelligence and related aptitude batteries. In Saklofske

D. H.

Reynolds

C. R.

Schwean

V. L.

(Eds.), The Oxford handbook of child psychological assessment (pp. 84-112). New York, NY: Oxford University Press.

Canivez

G. L.

Watkins

M. W.

(2010). Investigation of the factor structure of the Wechsler Adult Intelligence Scale–Fourth Edition (WAIS–IV): Exploratory and higher order factor analyses. PsychologicalAssessment, 22, 827–836. doi:10.1037/a0020429

Caretta

T. R.

Ree

J. J.

(2001). Pitfalls of ability research. International Journal of Selection and Assessment, 9, 325–335. doi:10.1111/1468-2389.00184

Cattell

R. B.

(1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276.

Carroll

J. B.

(1993). Human cognitive abilities. Cambridge, UK: Cambridge University Press.

10.

Carroll

J. B.

(1995). On methodology in the study of cognitive abilities. Multivariate Behavioral Research, 30, 429–452. doi:10.1207/s15327906mbr3003_6

11.

DiStefano

Dombrowski

S. C.

(2006). Investigating the theoretical structure of the Stanford Binet–Fifth edition. Journal of Psychoeducational Assessment, 24, 123-136. doi:10.1177/0734282905285244

12.

Dombrowski

S. C.

(2013a). Exploratory bifactor analysis of the WJ-III Cognitive in adulthood via the Schmid–Leiman procedure. Journal of Psychoeducational Assessment. Advance online publication. doi:10.1177/0734282913508243

13.

Dombrowski

S. C.

(2013b). Investigating the structure of the WJ-III Cognitive at school age. School Psychology Quarterly, 28, 154-169. doi:10.1037/spq0000010

14.

Dombrowski

S. C.

Watkins

M. W.

(2013). Exploratory and higher order factor analysis of the WJ-III full test battery: A school-aged analysis. Psychological Assessment, 25, 442-455. doi:10.1037/a0031335

15.

Dombrowski

S. C.

Watkins

M. W.

Brogan

M. J.

(2009). An exploratory investigation of the factor structure of the Reynolds Intellectual Assessment Scales (RIAS). Journal of Psychoeducational Assessment, 27, 494-507. doi:10.1177/0734282909333179

16.

Glutting

J. J.

Watkins

M. W.

Konold

T. R.

McDermott

P. A.

(2006). Distinctions without a difference: The utility of observed versus latent factors from the WISC-IV in estimating reading and math achievement on the WIAT-II. The Journal of Special Education, 40, 103–114. doi:10.1177/00224669060400020101

17.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

18.

Gorsuch

R. L.

(1997). Exploratory factor analysis: Its role in item analysis. Journal of Personality Assessment, 68, 532–560. doi:10.1207/s15327752jpa6803_5

19.

Gustafsson

J. E.

Snow

R. E.

(1997). Ability profiles. In Dillon

R. F.

(Ed.), Handbook on testing (pp. 107-135). Westport, CT: Greenwood Press.

20.

Holzinger

K. J.

Swineford

(1937). The bi-factor method. Psychometrika, 2, 41-54.

21.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. doi:10.1007/BF02289447

22.

Jennrich

R. I.

Bentler

P. M.

(2011). Exploratory bi-factor analysis. Psychometrika, 6, 537-549. doi: 10.1007/s11336-011-9218-4

23.

Kaiser

H. F.

(1974). An index of factorial simplicity. Psychometrika, 39, 31-36.

24.

Keith

T. Z.

Reynolds

(2010). Cattell-Horn-Carroll abilities and cognitive tests: What we’ve learned from 20 years of research. Psychology in the Schools, 47, 635-650. doi:10.1002/pits.20496

25.

Kline

(1994). An easy guide to factor analysis. New York, NY: Routledge.

26.

Nelson

J. M.

Canivez

G. L.

(2012). Examination of the structural, convergent, and incremental validity of the Reynolds Intellectual Assessment Scales (RIAS) with a clinical sample. Psychological Assessment,24, 129–140. doi:10.1037/a0024878

27.

O’Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instruments & Computers, 32, 396–402.doi:10.3758/BF03200807

28.

H. J.

Glutting

J. J.

Watkins

M. W.

Youngstrom

E. A.

McDermott

P. A.

(2004). Correct interpretation of latent versus observed abilities: Implications from structural equation modeling applied to the WISC-III and WIAT linking sample. The Journal of Special Education, 38, 159–173. doi:10.1177/00224669040380030301

29.

Parkin

J. R.

Beaujean

A. A.

(2012). The effects of Wechsler Intelligence Scale for Children—Fourth Edition cognitive abilities on math achievement. Journal of School Psychology, 50, 113–128. doi:10.1016/j.jsp.2011.08.003

30.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696.

31.

Reise

S. P.

Moore

T. M.

Haviland

M. G.

(2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scores. Journal of Personality Assessment, 92, 544-559.

32.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF02289209

33.

Taub

G. E.

McGrew

K. S.

(2004). A confirmatory factor analysis of Cattell-Horn-Carroll theory and cross-age invariance of the Woodcock-Johnson Tests of Cognitive Abilities III. School PsychologyQuarterly, 19, 72-87.

34.

Thompson

(2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association. doi:10.1037/10694-000

35.

Velicer

W. F.

(1976). Determining the number of components form the matrix of partial correlations. Psychometrika, 31, 321-327. doi:10.1007/BF02293557

36.

Velicer

W. F.

Eaton

C. A.

Fava

J. L.

(2000). Construct explication through factor or component analysis: A view and evaluation of alternative procedures for determining the number of factors or components. In Goffin

R. D.

Helmes

(Eds.), Problems and solutions in humanassessment: A festschrift to Douglas Jackson at seventy (pp. 41–71). Norwell, MA: Kluwer Academic.

37.

Watkins

M. W.

(2010). Structure of the Wechsler Intelligence Scale for Children—Fourth Edition among a national sample of referred students. Psychological Assessment, 22, 782–787. doi:10.1037/a0020043

38.

Watkins

M. W.

(2013). Omega. [Computer software]. Phoenix, AZ: Ed & Psych Associates.

39.

Watkins

M. W.

Glutting

J. J.

Lei

P.-W.

(2007). Validity of the full-scale IQ when there is significant variability among WISC-III and WISC-IV factor scores. Applied Neuropsychology, 14, 13–20. doi:10.1080/09084280701280353

40.

Wolff

H.-G.

Preising

(2005). Exploring item and higher order factor structure with the Schmid-Leiman solution: Syntax codes for SPSS and SAS. Behavior Research Methods, 37, 48–58. doi:10.3758/BF03206397

41.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001c). Woodcock–Johnson III tests of cognitive abilities. Itasca, IL: Riverside.

42.

Yung

Y. F.

Thissen

McLeod

L. D.

(1999). On the relationship between the higher-order factor model and the hierarchical factor model. Psychometrika, 64, 113-128.

43.

Zwick

W. R.

Velicer

W. F.

(1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.