Exploratory Bifactor Analysis of the WJ-III Achievement at School Age via the Schmid

Abstract

The structure of academic achievement measures has been rarely investigated in the literature apart from that which appears in the instruments’ technical manuals. This is concerning, given the widespread use of academic achievement instruments when making educational decisions about children. The Woodcock–Johnson III (WJ-III) Achievement for school-aged children (age 9-19) was investigated using exploratory bifactor analysis via the Schmid–Leiman (SL) orthogonalization procedure. This is the first time the SL has been applied to an academic achievement measure. The results revealed a unidimensional model of academic achievement across the standard 11 subtest battery, a two- or three-factor model at age 9 to 13 in the extended battery, and a two-factor model at age 14 to 19 across the extended battery. Forcing the four-factor fit in the standard battery required extracting eigenvalues as low as 0.67 and yielded areas of both convergence with and divergence from the structure posited in the Technical Manual. Forcing the six-factor fit across the extended battery yielded Heywood Cases, a lack of convergence of the factor solution, and the need to truncate iterations at 2 to force the fit. The results of this study indicate that the WJ-III Achievement is a solid model of general achievement across the 9 to 19 age range. Examination of omega coefficients, the divergent factor structure, and the small amount of variance accounted for by the lower order factors suggest caution when interpreting beyond this level (i.e., the academic clusters). Implications for interpretation of the WJ-III Achievement at age 9 to 19 are discussed.

Keywords

exploratory factor analysis exploratory bifactor analysis Schmid–Leiman orthogonalization academic achievement Woodcock–Johnson III

The structure of academic achievement measures has rarely been independently investigated outside of the various instruments’ respective technical manuals. This is surprising considering the widespread use and daily influence that measures of academic achievement have on children’s lives when making educational programming decisions. The Woodcock–Johnson III (WJ-III) Achievement (Woodcock, McGrew, & Mather, 2001) is one of the major individual tests of academic achievement and has been pervasively used to understand children’s learning strengths and weaknesses and for the evaluation of learning disabilities. However, its structure has never been investigated using exploratory factor analytic (EFA) procedures.

The interpretive manual indicates that the WJ-III Achievement offers a total achievement score reminiscent of a higher order factor and is organized into four lower order factors (i.e., reading, mathematics, written language, oral language clusters) in the standard battery and six lower order factors (i.e., reading, mathematics, written language, academic knowledge, and a supplemental cluster) in the extended battery (see McGrew & Woodcock, 2001, Technical Manual).

The existing literature regarding the structure of tests of academic achievement is scant, so this literature review is limited in scope. Reynolds (1979) investigated the Peabody Individual Achievement Test (PIAT) and found that its structure had two factors consistent with Cattell’s concept of fluid and crystallized intelligence. Williams and Eaves (2001) investigated the structure of the Woodcock Reading Mastery Test, a narrow band measure of reading, and found that the structure of this instrument was unidimensional (total reading) and not multidimensional as specified in the Technical Manual. Similarly, Williams, Fall, Eaves, Darch, and Woods-Groves (2007) explored the structure of the KeyMath Normative Update and located a single math factor rather than the three-factor-correlated solution posited by the Technical Manual. All three studies revealed a different factor structure from that presented in the instruments’ respective technical manuals. Beyond these three studies, little research has been conducted on the structure of academic achievement measures.

The purpose of this article is to investigate the factor structure of the WJ-III Achievement (standard and extended battery) during school age (9-19) using exploratory bifactor analysis via the Schmid–Leiman (SL) orthogonalization procedure. Exploratory bifactor modeling is appropriate when seeking to understand instruments with presumed higher order factors (i.e., total achievement) and correlated traits (i.e., reading, writing, and oral language). The SL (Schmid & Leiman, 1957) procedure is an elegant exploratory bifactor procedure with a history of use in the cognitive ability and personality literature (see Canivez, 2013; Carroll, 1993; Dombrowski, Watkins & Brogan, 2009). It has never been applied to the construct of academic achievement but is very appropriate for such purpose as academic achievement tests have correlated factors and a total achievement score reminiscent of a higher order factor. The SL helps uncover simple structure by partialling out the influence of hierarchical factors on first-order factors; this provides a clearer sense of an instrument’s structure by more directly furnishing the variance apportioned to higher and lower order factors (Carretta & Ree, 2001; Carroll, 1993, 1995, 2003; Gustafsson & Snow, 1997). Carroll (1995) discussed the importance of the SL procedure when analyzing instruments with correlated traits and used the SL procedure when he created his Three Stratum Theory of Cognitive Abilities. His theory was influential in the development of not only Cattell–Horn–Carroll (CHC) theory but also the WJ-III Achievement.

Recent concerns regarding the factor structure of the WJ-III (e.g., Dombrowski, 2013, 2014a, 2014b; Dombrowski & Watkins, 2013) also suggest that an analysis of the WJ-III Achievement using the SL procedure appears worthwhile. A basic premise of factor analysis posits that when multiple methods of factor analysis converge, then confidence in a derived factor structure may be accomplished (Gorsuch, 1983). The investigation of academic achievement measures has been overlooked within the empirical literature, despite the widespread use of academic achievement tests in school-aged populations.

Method

Participants

The data for the WJ-III Achievement norms were collected from a nationally representative sample of 8,818 participants from age 2 through 90 plus. Demographic characteristics are provided in the WJ-III Technical Manual. For this study, the two school-aged (9-13 and 14-19 years) subtest correlation matrices (22 by 22) were obtained from the Technical Manual. The 9 to 13 age range contained an average of 1,574 participants, whereas the 14 to 19 age range contained an average of 1,298 participants.

Instrument

The WJ-III Tests of Achievement (Woodcock et al., 2001) contains 22 achievement tests. The WJ-III Achievement is organized into a standard battery and an extended battery. The standard battery comprises 11 subtests and measures four academic cluster areas: reading, writing, mathematics, and oral language. The extended battery comprises 22 subtests and measures six academic cluster areas: reading, writing, mathematics, oral language, academic knowledge, and a supplemental cluster. Both batteries yield a total achievement score reflective of a higher order factor.

Procedure

Minimum average partials (MAP; Velicer, 1976) and parallel analysis (Horn, 1965) were used to determine the number of factors to extract. These procedures were conducted using O’Connor’s (2000) SPSS program. A Horn Parallel Analysis (HPA) scree plot (Cattell, 1966; Horn, 1965) was also inspected (Figures 1 & 2) as a supplemental means to determine the number of factors to retain for rotation. The SL procedure (Schmid & Leiman, 1957) was applied to the oblique first-order factors following a principal axis factoring with promax rotation (Wolff & Preising, 2005). To permit a comparison of the fit offered in the Technical Manual for both the extended (22 subtest) and the standard (11 subtest) batteries across the 9 to 13 and the 14 to 19 age ranges, the Technical Manual’s six- and four-factor respective fits were forced. Finally, omega coefficients were determined using a program developed by Watkins (2013).

Figure 1.

Scree plots for HPA for the WJ-III Achievement age 9 to 13.

Figure 2.

Scree plots for HPA for the WJ-III Achievement age 14 to 19.

Results

Exploratory (First-Order) Analyses

Factor retention criteria

For the extended battery, parallel analysis (Horn, 1965) suggested the retention of two factors at ages 9 to 13 and 14 to 19. The MAP (Velicer, 1976) criterion recommended retention of three factors at age 9 to 13 and two factors at age 14 to 19. A HPA scree test on the 9 to 13 and the 14 to 19 correlation matrices (Figures 1 and 2) indicated evidence for one strong factor with the possibility of one additional factor. Because MAP recommended the retention of three factors and parallel analysis recommended the retention of two factors across the 9 to 13 age range, both a two- and a three-factor solution were extracted and analyzed for the age 9 to 13 extended battery analysis.

For the 11 subtest standard batteries across both age ranges, parallel analysis and MAP recommended the extraction of a single factor. Because the recommended factor solutions across both age ranges and batteries (standard and extended) indicated a very different factor structure from that posited in the Technical Manual, the respective six- and four-factor structures were forced. The six-factor structure in the age 9 to 13 extended battery failed to converge because of Heywood cases (i.e., communalities > 1.0), which often occurs when too many factors are attempted to be extracted. Accordingly, the iteration process was truncated at two to force the six-factor solution in this age range.

Exploratory Bifactor Analysis (SL Orthogonalization)

Forced four-factor solution (standard 11 subtest battery)

Results from the Schmid and Leiman (1957) procedure on the four-factor standard battery forced analysis across both age ranges are presented in Tables 1 and 2. In the age 9 to 13 SL analysis, the higher order factor accounted for 47.2% of the total variance and 72.8% of the common variance. In the age 14 to 19 SL analysis, the higher order factor accounted for 46.5% of the total variance and 72.3% of the common variance. The general factor also accounted for between 27% and 67% (Mdn = 48%) of individual subtest variance in the 9 to 13 analysis. The higher order factor accounted for between 40% and 58% (Mdn = 45%) of individual subtest variability in the 14 to 19 analysis. For the 9 to 13 age range, the four first-order factors accounted for a small proportion of the total variance (2.7%-5.6%) and common variance (4.2%-7.3%). The first- and second-order factors combined to measure 65% of the variance in the WJ-III Achievement standard battery, reflecting 35% unique variance. For the 14 to 19 age range, the four first-order factors accounted for 4.3% to 9.7% of the total variance and 2.8% to 6.3% of the common variance. The first- and second-order factors of the 14 to 19 analysis combined to measure 64% of the variance in the WJ-III, reflecting 36% unique variance. The results of both analyses demonstrate a robust manifestation of a higher order factor in the WJ-III Achievement standard battery where the combined influence of the second-order factor and uniqueness exceeded the contributions made by the first-order factors.

Table 1.

WJ-III Achievement Ages 9 to 13 Forced Four-Factor Fit Standard Battery (Schmid–Leiman Orthogonalization).

	Second-order factor
Subtest	G	S ²	F1	S ²	F2	S ²	F3	S ²	F4	S ²	h ²	u ²
Letter Word Identification	.82	67	.35	13							.80	.20
Spelling	.81	65	.29	09							.76	.24
Passage Comprehension	.75	57	.20	04			.24	06			.66	.34
Writing Samples	.68	47	.19	04							.52	.48
Reading Fluency	.69	48			.49	24					.76	.24
Math Fluency	.60	36			.45	20			.28	08	.65	.35
Writing Fluency	.68	46			.39	15					.61	.39
Story Recall	.52	27					.44	19			.46	.54
Understanding Directions	.53	29					.42	18			.48	.52
Calculation	.69	47							.47	22	.70	.30
Applied Problems	.72	52							.44	19	.75	.25
Common variance (%)	72.8		4.2		8.6		7.2		7.3
Total variance (%)	47.2		2.7		5.6		4.7		4.7		.65	.35
	ω_h = .868		ω_s = .090		ω_s = .262		ω_s = .253		ω_s = .243

Note. Loadings ≥.30 are bolded, italicized , and considered salient. Loadings ≥.20 are italicized and considered aligned with their respective factors. h² = communality coefficient; u² = uniqueness; ω_h = omega hierarchical; ω_s = omega subscale.

Table 2.

WJ-III Achievement Standard Battery Ages 14 to 19 Forced Four-Factor Fit (Schmid–Leiman Orthogonalization).

	Second-order factor
Subtest	G	S ²	F1	S ²	F2	S ²	F3	S ²	F4	S ²	h ²	u ²
Story Recall	.64	41	.29	08							.50	.50
Passage Comprehension	.71	50	.26	07							.59	.41
Understanding Directions	.66	44	.26	07							.52	.48
Writing Samples	.65	42							.22	05	.51	.49
Reading Fluency	.76	57			.45	20					.79	.21
Math Fluency	.67	45			.39	15	.25	06			.68	.32
Writing Fluency	.66	43			.30	09					.54	.46
Calculation	.69	47					.52	27			.75	.25
Applied Problems	.76	58					.38	14			.75	.25
Spelling	.63	40							.57	32	.74	.26
Letter Word ID	.65	43							.53	28	.71	.29
Common variance (%)	72.3		4.3		6.4		7.2		9.7
Total variance (%)	46.5		2.8		4.1		4.6		6.3		.64	.36
	ω_h = .869		ω_s = .096		ω_s = .192		ω_s = .235		ω_s = .353

The reliability of WJ-III Achievement was also estimated across both age ranges with ω_h and ω_s. The ω_h coefficient for the general factor (.87) across both age ranges was high and sufficient for interpretation. Omega subscale (ω_s) coefficients for the four lower order factors ranged from .092 to .353 across both age range’s analyses. Low ω_s coefficients suggest that interpretation of the factor indices beyond the general factor is inappropriate as little variance exists beyond the general factor (Reise, 2012).

Forced six-factor solution (22 subtest extended battery)

Results from the Schmid and Leiman (1957) procedure on the six-factor extended battery forced solution across both age ranges are presented in Tables 3 and 4. Both tables furnish the proportion of variance apportioned to the higher order factor and lower order factors. With the age 9 to 13 SL analysis, the higher order factor accounted for 41.9% of the total variance and 73.1% of the common variance. With the age 14 to 19 SL analysis, the higher order factor accounted for 46.2% of the total variance and 66.5% of the common variance. In the 9 to 13 analysis, the general factor accounted for a substantial proportion of the variance, but G-loadings and communality estimates on two subtests were well in excess of 1.0 suggesting severe overfactoring.

Table 3.

WJ-III Achievement Standard Battery Ages 9 to 13 Forced Six-Factor Fit (Schmid–Leiman Orthogonalization).

Subtest	Second-order factor		First-order factors
Subtest	G	Var	F1	Var	F2	Var	F3	Var	F4	Var	F5	Var	F6	Var	h ²	u ²
Letter Word ID	.45	20	.42	17											.41	.59
Spelling	.46	21	.41	17											.42	.58
Passage Comp	.70	49	.40	16											.65	.35
Applied Problems	.72	52	.39	15					−.21	05					.74	.26
Reading Vocabulary	.71	50	.39	15											.67	.33
Academic Knowledge	.67	44	.38	15											.65	.35
Quant Concepts	.49	24	.38	15					−.20	04					.44	.56
Editing	.53	28	.38	15											.46	.54
Sound Awareness	.70	48	.37	14											.69	.31
Reading Fluency	.52	27	.36	13							−.21	04			.48	.52
Word Attack	.39	15	.35	12											.34	.66
Writing Sample	.46	21	.35	12											.34	.66
Calculation	.65	43	.34	12											.61	.39
Writing Fluency	.51	26	.34	12											.45	.55
Spelling of Sounds	.34	12	.34	11											.28	.72
Oral Communication	.61	38	.32	10											.52	.48
Picture Vocabulary	.38	15	.32	10											.33	.67
Story Recall	1.32	174	.31	10	.24	06									1.94	—
Understand Direct	.84	70	.31	09									.31	10	.91	.09
Math Fluency	.40	16	.29	08											.33	.67
Story Recall Delayed	1.48	219	.28	08	.21	04			.20	04					2.41	—
Handwriting	.22	05													.09	.91
%Total variance		46.2	12.2		1.5		1.1		1.1		1.2		1.2		64.3	35.7
%Common variance		66.5	6.60		5.30		8.50		5.60		3.60		4.00
	ω_h = .76		ω_s = .316

Table 4.

WJ-III Achievement Standard Battery Ages 14 to 19 Forced Six-Factor Fit (Schmid–Leiman Orthogonalization).

Subtest	Second-order factor		First-order factors
Subtest	G	Var	F1	Var	F2	Var	F3	Var	F4	Var	F5	Var	F6	Var	h ²	u ²
Picture Vocabulary	.67	45	.46	21											.68	.32
Academic Knowledge	.80	63	.39	15											.81	.19
Oral Communication	.67	45	.33	11											.59	.41
Passage Comprehension	.71	50	.28	08											.59	.41
Reading Vocabulary	.75	56	.28	08											.68	.32
Understand Direct	.63	40											.31	10	.57	.43
Spelling	.74	54			.37	14									.71	.29
Editing	.70	49			.36	13									.64	.36
Word Attack	.63	40			.34	12									.57	.43
Letter Word ID	.78	61	.20	04	.31	10									.75	.25
Handwriting	.20	04			.25	06									.14	.86
Spelling of Sounds	.59	35			.23	05							.20	04	.46	.54
Writing Samples	.68	46													.52	.48
Calculation	.66	43					.49	24							.67	.33
Applied Problems	.78	60					.42	17							.80	.20
Quantitative Concepts	.79	62					.39	15							.80	.20
Story Recall Delayed	.60	36							.59	35					.72	.28
Story Recall	.67	44							.57	33					.78	.22
Reading Fluency	.62	38									.57	32			.73	.27
Math Fluency	.54	29					.27	07			.44	19			.60	.40
Writing Fluency	.58	34									.42	18			.55	.45
Sound Awareness	.67	45											.39	15	.65	.35
%Total variance		44.6	3.9		3.3		3.0		3.4		3.6		1.7		63.6	36.4
%Common variance		70.2	6.1		5.3		4.8		5.3		5.7		2.7
	ω_h = .91		ω_s = .16		ω_s = .16		ω_s = .23		ω_s = .39		ω_s = .25		ω_s = .15

For the 14 to 19 analyses, the six, first-order factors accounted for 1.7% to 3.9% of the total variance and 2.7% to 6.1% of the common variance. The first- and second-order factors of the 14 to 19 analysis combined to measure 63.6% of the variance in the WJ-III, reflecting 36.4% unique variance. The results of both analyses demonstrate a robust manifestation of a higher order factor in the WJ-III Achievement standard battery where the combined influence of the second-order factor and uniqueness exceeded the contributions made by the first-order factors.

The reliability of WJ-III Achievement was also estimated across both extended battery analyses with ω_h and ω_s. The ω_h coefficient for the general factor across both age ranges was high and sufficient for interpretation (.76 for 9-13; .91 for 14-19). Omega subscale (ω_s) coefficients for the six lower order factors from the 14 to 19 group ranged from .15 to .39. Omega subscale could not be completed on the 9 to 13 analysis as a result of five trivial factors. Low ω_s (or nonexistent) coefficients suggest that interpretation of the factor indices beyond the general factor is inappropriate as little variance exists beyond the general factor (Reise, 2012).

Hierarchical Factor Analysis Based on Psychometrically Sound Factor Retention Rules

Parallel analysis and the MAP test across both age ranges in the standard 11 subtest battery suggested a single-factor solution. For the extended battery, Tables 5 through 7 present the results of a higher order factor analysis based upon the extraction of two (ages 9-13 and 14-19) and three factors (age 9-13). As noted, the results of both analyses suggest the prominence of a higher order factor and struggled to find subtest alignment reminiscent of that posited in the Technical Manual. At age 9 to 13 (three-factor extraction; Table 3), the hierarchical factor accounted for 73.1% of the common variance and 41.9% of the total variance, exceeding that accounted for by the lower order factors (6.1%-12.03% common variance; 3.5%-6.9% total variance). The first- and second-order factors at age 9 to 13 combine to measure 57.3% of the variance in the WJ-III Achievement (extended battery), reflecting 42.7% unique variance. The age 9 to 13, two-factor solution (Table 4) indicated that the hierarchical factor accounted for 59.6% of the common variance and 31.7% of the total variance, exceeding that accounted for by the lower order factors (18.1%-22.3% common variance; 9.6%-11.8% total variance). The first- and second-order factors at age 9 to 13 (two-factor solution) combined to measure 53.2% of the variance in the WJ-III Achievement (standard battery), reflecting 46.9% unique variance. At age 14 to 19 (Table 5), the general factor accounted for 63.9% of the common variance and 33.6% of the total variance, exceeding that apportioned to lower order factors (16.0%-20.0% common variance; 8.4%-10.5% total variance). The first- and second-order factors at age 14 to 19 combine to measure 52.5% of the variance in the WJ-III Achievement (extended battery), reflecting 47.5% unique variance. The results of all three analyses demonstrate a robust manifestation of a general factor in the WJ-III Achievement where the combined influence of general achievement and uniqueness exceeded the contributions made by the first-order factors (i.e., academic clusters).

Table 5.

WJ-III Achievement Extended Battery Ages 9 to 13 according to a Schmid–Leiman Orthogonalization (Three Factor).

	Second-order factor
Subtest	G	S ²	F1	S ²	F2	S ²	F3	S ²	h ²	u ²
Story Recall	.66	43	.40	16					.61	.39
Story Recall Delayed	.60	36	.35	12					.51	.49
Academic Knowledge	.75	57	.33	11					.71	.29
Oral Comprehension	.64	41	.28	08					.50	.50
Understanding Directions	.61	37	.25	06					.43	.57
Picture Vocabulary	.61	37	.24	06	.26	07	−.21	04	.54	.46
Reading Vocabulary	.74	54	.23	05	.22	05			.65	.35
Applied Problems	.75	56	.20	04			.30	09	.69	.31
Word Attack	.59	35			.50	25			.61	.39
Spelling of Sounds	.57	33			.48	23			.56	.44
Letter Word ID	.73	53			.48	23			.76	.24
Spelling	.71	50			.42	17	.21	04	.73	.27
Sound Awareness	.66	44			.33	11			.56	.44
Editing	.68	47			.29	09			.58	.42
Writing Samples	.64	41			.27	07			.50	.50
Passage Comprehension	.74	55			.25	06			.64	.36
Math Fluency	.52	27					.59	35	.62	.38
Calculation	.66	43					.45	20	.64	.36
Writing Fluency	.60	36			.21	04	.33	11	.51	.49
Quantitative Concepts	.72	51					.27	07	.62	.38
Reading Fluency	.64	41			.20	04	.26	07	.52	.48
Handwriting	.22	05					.21	04	.10	.90
Common variance (%)	73.1		6.1		12.0		8.8
Total variance (%)	41.9		3.5		6.9		5.1		57.3	42.7
	ω_h = .873		ω_s = .138		ω_s = .224		ω_s = .236

Table 6.

WJ-III Achievement Extended Battery Ages 9 to 13 according to a Schmid–Leiman Orthogonalization (Two Factor).

	Second-order factor
Subtest	G	S ²	F1	S ²	F2	S ²	h ²	u ²
Spelling	.66	43	.57	32			.76	.24
Math Fluency	.45	21	.56	31			.54	.46
Writing Fluency	.54	29	.51	26			.56	.44
Letter Word ID	.67	45	.46	21			.67	.33
Word Attack	.56	31	.45	20			.52	.48
Reading Fluency	.57	32	.43	18			.51	.49
Spelling of Sounds	.54	29	.43	18			.48	.52
Editing	.61	38	.41	17			.56	.44
Calculation	.55	30	.39	15			.46	.54
Writing Samples	.57	32	.37	14			.48	.52
Quantitative Concepts	.61	37	.33	11	.20	04	.52	.48
Handwriting	.20	04	.31	10			.16	.84
Sound Awareness	.59	35	.28	08	.23	05	.48	.52
Applied Problems	.63	39	.28	08	.26	07	.54	.46
Story Recall	.51	26			.60	36	.64	.36
Academic Knowledge	.63	40			.59	35	.75	.25
Oral Communication	.53	28			.49	24	.53	.47
Story Recall Delayed	.46	21			.49	24	.46	.54
Picture Vocabulary	.53	28			.48	23	.51	.49
Reading Vocabulary	.63	40			.44	19	.61	.39
Understanding Directions	.50	25			.39	15	.41	.59
Passage Comprehension	.65	42	.25	06	.31	10	.58	.42
Common variance (%)	59.6		22.3		18.1
Total variance (%)	31.7		11.8		9.6		53.2	46.9
	ω_h = .718		ω_s = .332		ω_s = .381

Table 7.

WJ-III Achievement Extended Battery Ages 14 to 19 according to a Schmid–Leiman Orthogonalization (Two Factor).

	Second-order factor
Subtest	G	S ²	F1	S ²	F2	S ²	h ²	u ²
Story Recall	.54	29	.57	33			.64	.36
Academic Knowledge	.68	46	.54	30			.76	.24
Picture Vocabulary	.58	33	.54	30			.64	.36
Story Recall Delayed	.48	23	.52	27			.52	.48
Oral Communication	.59	34	.45	21			.55	.45
Reading Vocabulary	.64	41	.45	20			.62	.38
Understanding Directions	.55	30	.38	14			.45	.55
Applied Problems	.65	43	.36	13			.58	.42
Passage Comprehension	.62	39	.34	11			.53	.47
Quantitative Concepts	.67	45	.33	11	.21	04	.53	.47
Sound Awareness	.58	33	.24	06	.22	05	.44	.56
Math Fluency	.50	25			.51	26	.52	.48
Spelling	.66	43			.50	25	.68	.32
Word Attack	.56	32			.45	20	.52	.48
Reading Fluency	.57	33			.43	19	.51	.49
Writing Fluency	.54	29			.39	15	.45	.55
Spelling of Sounds	.53	28			.37	13	.42	.58
Letter Word ID	.69	47			.36	13	.64	.36
Editing	.60	36			.36	13	.51	.49
Writing Samples	.60	36	.21	04	.27	07	.47	.53
Handwriting	.19	03			.26	07	.12	.88
Calculation	.55	30	.20	04	.24	06	.40	.60
Common variance (%)	63.9		20.0		16.0
Total variance (%)	33.6		10.5		8.4		52.5	47.5
	ω_h = .759		ω_s = .316		ω_s = .291

The reliability of WJ-III Achievement was also estimated across both analyses with ω_h and ω_s. The ω_h coefficient for the general factor (.873 for age 9-13 three factor; .718 for age 9-13 two factor; .759 for age 14-19) was high and sufficient for interpretation. Omega subscale (ω_s) coefficients for the lower order factors ranged from .138 to .381 across all analyses. Lower ω_s coefficients suggest that interpretation of the factor indices beyond the general factor may not be appropriate as little variance exists beyond the general factor (Reise, 2012).

Discussion

This study investigated the WJ-III Achievement during school age by analyzing the age 9 to 13 and 14 to 19 correlation matrices across both the standard (11 subtest) and extended (22 subtest) batteries. Use of EFA factor extraction procedures (e.g., parallel analysis and MAP as supplemented by a HPA visual scree) that are considered to be the most psychometrically robust suggests that the WJ-III Achievement extended battery is a two- or three-factor test at age 9 to 13 and a two-factor instrument at age 14 to 19 (see Tables 3 -5). However, extracting this number of factors generally renders the instrument less available for interpretation because the subtest alignment and factor structure lack full linkage to the theoretical structure posited in the WJ-III Technical Manual. Factor extraction decision-making results indicated that the structure of the WJ-III Achievement is unidimensional across both age ranges of the 11 subtest standard battery.

Because the results based upon psychometrically sound factor extraction decision-making rules did not comport with the structure posited in the Technical Manual (i.e., six-factor solution for extended battery and four-factor solution for standard battery), the Technical Manual’s theoretical structure was forced. This required casting aside even the most lenient factor extraction decision-making rules (i.e., eigenvalue >1) and extracting factors with eigenvalues as low as 0.67 within the standard battery.

As noted in Tables 1 and 2, the higher order factor captures an inordinate amount of the variance (72%) across both age range’s analyses of the standard battery. Omega coefficients also lend support for the prominence of the higher order factor with omega hierarchical at .87 across both standard battery’s forced analyses. Omega subscale across both analyses ranged from .09 to .35 again suggesting preeminence of the hierarchical factor (presumably total achievement).

When disregarding the above-mentioned factor extraction decision rules and forcing the extraction of factors in accord with the structure posited in the Technical Manual, the standard battery’s (age 9-13) forced four-factor solution experienced areas of convergence with and divergence from the Technical Manual. Letter Word Identification, Spelling, Writing Samples, and Passage Comprehension paired together to form the first factor (presumably a reading and writing factor). Reading Fluency, Math Fluency, and Writing Fluency paired together to form an academic fluency second factor. Story Recall and Understanding Directions formed the third factor (presumably Oral Language) along with Passage Comprehension, which cross-loaded this factor. Calculation and Applied Problems paired together to form the fourth factor (Mathematics) along with Math Fluency, which cross-loaded this factor. At age 14 to 19, Story Recall, Passage Comprehension, and Understanding Directions paired together to form the first factor (presumably an oral language and reading comprehension factor). Reading Fluency, Math Fluency, and Writing Fluency formed a second factor (academic fluency). Calculation and Applied Problems formed a third factor (mathematics) along with Math Fluency, which cross-loaded this factor. Spelling and Letter Word Identification along with Writing Samples formed a fourth factor (presumably a writing and reading factor). Thus, forcing the four-factor fit across the standard battery at ages 9 to 19 did not fully align with the academic cluster structure posited in the Technical Manual.

Forcing the six-factor fit across both age ranges was even more problematic. At age 9 to 13 (Table 3), the structure is vastly different and generally uninterpretable. At age 14 to 19 (Table 4), the forced, six-factor fit does not generally align with the Technical Manual’s posited structure. The first factor appears to be an oral language and reading factor; the second is a written language and phonemic awareness factor; the third is a mathematics factor; the fourth appears to resemble a memory factor; the fifth is an academic fluency factor; and the sixth appears to be an auditory processing factor. The only alignment between the Technical Manual’s posited structure and the age 14 to 19 results are with the mathematics factor.

Conclusion and Implications for Practice

There are several implications for practitioners. First, the WJ-III Achievement may not offer a clear picture of a child’s specific academic achievement in the clusters it purports to measure (reading, writing, oral language, knowledge, supplemental) with the exception of mathematics and what emerged as an academic fluency factor. In other words, the Technical Manual claims to measure reading, writing, mathematics, and oral language across both the extended and the standard batteries. However, the results of this study suggest a different factor structure and therefore caution when moving to interpretation of the academic achievement clusters. Neither the results based upon factor extraction decision making nor the results of a forced fit analysis completely aligned with the Technical Manual’s posited structure.

Unfortunately, the field has not cast its gaze upon the structure of measures of academic achievement in the same way it has upon measures of cognitive ability. Given the significant role that standardized measures of academic achievement such as the WJ-III Achievement play in the lives of school-aged children, it is due time for increased investigation of this topic.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Author Biography

Stefan C. Dombrowski is professor and director of the school psychology program at Rider University. He has written 4 books, 2 assessment instruments and more than 50 articles/chapters on topics such as structural validity, prenatal exposures, children’s mental health, and psychoeducational assessment.

References

Canivez

G. L.

(2013). Psychometric versus actuarial interpretation of intelligence and related aptitude batteries. In Saklofske

D. H.

Reynolds

C. R.

Schwean

V. L.

(Eds.), The Oxford handbook of child psychological assessments (pp. 84-112). New York, NY: Oxford University Press.

Carretta

T. R.

Ree

J. J.

(2001). Pitfalls of ability research. International Journal of Selection and Assessment, 9, 325-335. doi:10.1111/1468-2389.00184

Carroll

J. B.

(1993). Human cognitive abilities. Cambridge, UK: Cambridge University Press.

Carroll

J. B.

(1995). On methodology in the study of cognitive abilities. Multivariate Behavioral Research, 30, 429-452. doi:10.1207/s15327906mbr3003_6

Carroll

J. B.

(2003). The higher-stratum structure of cognitive abilities: Current evidence supports g and about ten broad factors. In Nyborg

(Ed.), The scientific study of general intelligence: Tribute to Arthur R. Jensen (pp. 5-21). New York, NY: Pergamon Press.

Cattell

R. B.

(1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276.

Dombrowski

S. C.

(2013). Investigating the structure of the WJ-III cognitive at school age. School Psychology Quarterly, 28, 154-169. doi:10.1037/spq0000010

Dombrowski

S. C.

(2014a). Exploratory bifactor analysis of the WJ-III cognitive in adulthood via the Schmid–Leiman procedure. Journal of Psychoeducational Assessment. 32, 330–341 Advance online publication. doi:10.1177/0734282913508243

Dombrowski

S. C.

(2014b). Investigating the structure of the WJ-III Cognitive in early school age through two exploratory bifactor analysis procedures. Journal of Psychoeducational Assessment, 32, 483-494. doi:10.1177/0734282914530838

10.

Dombrowski

S. C.

Watkins

M. W.

(2013). Exploratory and higher order factor analysis of the WJ-III Full Test Battery: A school age analysis. Psychological Assessment, 25, 442-455. doi:10.1037/a0031335

11.

Dombrowski

S. C.

Watkins

M. W.

Brogan

M. J.

(2009). An exploratory investigation of the factor structure of the Reynolds Intellectual Assessment Scales (RIAS). Journal of Psychoeducational Assessment, 27, 494-507. doi:10.1177/0734282909333179

12.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

13.

Gustafsson

J.-E.

Snow

R. E.

(1997). Ability profiles. In Dillon

R. F.

(Ed.), Handbook on testing (pp. 107-135). Westport, CT: Greenwood Press.

14.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185. doi:10.1007/BF02289447

15.

McGrew

K. S.

Woodcock

R. W.

(2001). Technical manual: Woodcock-Johnson III. Itasca, IL: Riverside Publishing.

16.

O’Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using Parallel Analysis and Velicer’s MAP Test. Behavior Research Methods, Instruments, & Computers, 32, 396-402. doi:10.3758/BF03200807

17.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696.

18.

Reynolds

C. R.

(1979). Factor structure of the Peabody Individual Achievement Test at five grade levels between grades one and 12. Journal of School Psychology, 17, 270-274.

19.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF02289209

20.

Velicer

W. F.

(1976). Determining the number of components form the matrix of partial correlations. Psychometrika, 31, 321-327. doi:10.1007/BF02293557

21.

Watkins

M. W.

(2013). Omega [Computer program]. Phoenix, AZ: Ed. & Psych Associates.

22.

Williams

T. O.

Eaves

R. C.

(2001). Exploratory and confirmatory factor analyses of the Woodcock reading mastery tests–revised with special education students. Psychology in the Schools, 38, 561-567.

23.

Williams

T. O.

Fall

Eaves

R. C.

Darch

Woods-Groves

(2007). Factor analysis of the KeyMath—Revised Normative Update Form A. Assessment for Effective Intervention, 32, 113-120. doi:10.1177/15345084070320020201

24.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001). Woodcock-Johnson III Tests of Achievement. Itasca, IL: Riverside Publishing.

25.

Wolff

H.-G.

Preising

(2005). Exploring item and higher order factor structure with the Schmid-Leiman solution: Syntax codes for SPSS and SAS. Behavior Research Methods, 37, 48-58. doi:10.3758/BF03206397

Exploratory Bifactor Analysis of the WJ-III Achievement at School Age via the Schmid–Leiman Orthogonalization Procedure

Abstract

Keywords

Method

Participants

Instrument

Procedure

Results

Exploratory (First-Order) Analyses

Factor retention criteria

Exploratory Bifactor Analysis (SL Orthogonalization)

Forced four-factor solution (standard 11 subtest battery)

Forced six-factor solution (22 subtest extended battery)

Hierarchical Factor Analysis Based on Psychometrically Sound Factor Retention Rules

Discussion

Conclusion and Implications for Practice

Footnotes

Declaration of Conflicting Interests

Funding

Author Biography

References