Broad and Narrow CHC Abilities Measured and Not Measured by the Wechsler Scales

Abstract

In this commentary, we reviewed two clinical validation studies on the Wechsler Scales conducted by Weiss and colleagues. These researchers used a rigorous within-battery model-fitting approach that demonstrated the factorial invariance of the Wechsler Intelligence Scale for Children–Fourth Edition (WISC-IV) and Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV) across populations. Beyond the important finding of invariance across populations, however, these studies did not provide much additional clarification about exactly what is and what is not measured by the WISC-IV and WAIS-IV vis-à-vis an overarching empirically validated theory of the structure of cognitive abilities. To that end, we argued that a better understanding of the cognitive constructs (both broad and narrow) measured by the Wechsler scales or invariance across selection of variables is necessary and will require theory-driven joint or cross-battery confirmatory factor analysis. Recommendations for conducting this type of research were offered within the context of the Cattell-Horn-Carroll theory of cognitive abilities.

Keywords

Wechsler Scales CHC Theory Cross-Battery Factor Analysis

We reviewed two studies that examined four- and five-factor Wechsler Intelligence Scale for Children–Fourth Edition (WISC-IV) and Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV) measurement models and their invariance across normal and clinical samples (Weiss, Keith, Zhu, & Chen, 2013a, 2013b). In our opinion, the major contribution of these well-designed and executed studies is the finding that the four- and five-factor models were invariant across samples. The findings are of particular import because these measures are so widely used, and often with examinees who may experience symptoms similar to those encountered in the clinical samples. The rigorous model-fitting approach used to test for measurement invariance by the researchers is welcome (Meredith, 1993). In all, the authors should be congratulated for putting together two fine studies. In fact, beyond the typical nitpicking, there is not much for us to critique with regard to the studies themselves or to the interpretations drawn from them. The studies, however, stimulated some thoughts among us, and our intention is to share some of these thoughts in hopes of opening up some dialogue about how to continue to improve the understanding and assessment of human cognitive abilities.

In this commentary, we argue two primary points. First, an examination of the structural validity of the Wechsler Scales (and similar batteries) via within-battery factor analysis is limited by the subtests contained within the battery, which in turn limits further understanding of the constructs measured by these batteries (although we were glad to see that the authors included the supplemental tests in their analysis). Theory-driven joint or cross-battery confirmatory factor analysis (CB-CFA) will be necessary to understand more clearly the constructs measured by cognitive batteries. Second, CB-CFAs should more routinely model the narrow abilities that are measured by the Wechsler scales, particularly in light of their importance in explaining variance in academic skills above and beyond the variance accounted for by g (see Flanagan, Ortiz, & Alfonso, 2013). Throughout this commentary, we opine with regard to what cognitive constructs are and are not measured by the Wechsler Scales and recommend that these instruments be supplemented with measures of other cognitive constructs to assess more completely the breadth of abilities that are important in understanding learning and achievement.

Clinical Validation of the Four- and Five-Factor Interpretive Approaches for the Wechsler Scales

According to the WISC-IV Technical and Interpretive Manual (Wechsler, 2003), four factors underlie the WISC-IV, namely Verbal Comprehension (VC), Perceptual Reasoning (PR), Working Memory (WM), and Processing Speed (PS). In independent factor analyses of the WISC-IV standardization sample, five factors were modeled (e.g., Keith, Fine, Taub, Reynolds, & Kranzler, 2006). For example, Keith and colleagues split the PR factor into separate Fluid Reasoning (Gf; Matrix Reasoning and Picture Concepts) and Visual Processing (Gv; Block Design and Picture Completion) factors (see also Chen, Keith, Chen, & Chang, 2009; Reynolds, Keith, Flanagan & Alfonso, 2013). Although Keith and colleagues found both the WISC-IV four-factor model and the Cattell-Horn-Carroll (CHC) five-factor model equally plausible, they recommended the five-factor model because it was based on a well-supported theory. The purpose of the WISC-IV study conducted by Weiss and colleagues was to determine whether the four- and five-factor models were invariant across normal and clinical samples. They found that both models fit the data well, reported one model was not superior to the other, and concluded both models were invariant across samples.

Similarly, the WAIS-IV Technical and Interpretive Manual (Wechsler, 2008) states that four factors underlie the WAIS-IV, namely VC, PR, WM, and PS. In independent factor analyses of the WAIS-IV standardization data, Benson, Hulac, and Kranzler (2010) and Ward, Bergman, and Hebert (2012) found that a CHC five-factor model fit the data better than a four-factor model whereas Canivez and Watkins’ (2010) research supported the four-factor model. Like the WISC-IV study, the purpose of the WAIS-IV study conducted by Weiss and colleagues was to determine whether the four- and five-factor models were invariant across normal and clinical samples. Similar to Benson et al. and Ward et al., they found that the five-factor model fit the data better, providing support for separating the PR factor into separate Fluid Reasoning (FR; Matrix Reasoning, Arithmetic, and Figure Weights) and Perceptual Organization (PO; Block Design, Visual Puzzles, and Picture Completion) factors (see also Niileksela, Reynolds, & Kaufman, 2012). Moreover, because both the four- and five-factor models provided a good fit to the clinical data, Weiss and colleagues concluded that the WAIS-IV subtests have the same meaning regardless of clinical status.

Overall, it is important to know that differences in scores on the Wechsler scales across normative and combined clinical samples are due to differences in the common factors, and hence the test is not biased. The current studies provide an important contribution in this area, as measurement invariance, which requires both mean and covariance structures, should be applied within a model-fitting framework across groups as performed in these two studies (Meredith, 1993); it is simply not sufficient to exclude means or to factor analyze data within groups. It is not only insufficient to use exploratory factor analysis (EFA) to analyze data separately within clinical groups as is sometimes done, but EFA may produce misleading findings. For example, if subtest scores are used in part to determine clinical group status, the correlational structure among those subtests will be altered. Because EFA is most often based on correlations and not covariances, the findings from EFAs have to be interpreted with great caution because they may be due to statistical artifacts (Nesselroade & Thompson, 1995).

To reiterate, the Weiss et al. (2013a, 2013b) studies were important because they demonstrated a critical issue related to factorial invariance, invariance across populations. Beyond the findings of invariance across populations, however, the Weiss et al. studies did not provide much additional clarification about exactly what is and what is not measured by the WISC-IV and WAIS-IV vis á vis an overarching theory of the structure of cognitive abilities. In other words, the within-battery factor analyses conducted by Weiss and his colleagues could not fully resolve the controversies surrounding these batteries, such as how the Arithmetic subtest should be interpreted or issues related to the potential bifurcation of the Perceptional Reasoning Index. What could not be fully addressed, although it has been addressed to some extent across Wechsler revisions that include new subtests, is invariance related to selection of variables (Mulaik, 2010; Thurstone, 1947). Would the constructs measured by Arithmetic be the same as reported in these studies if additional measures of quantitative reasoning and applied mathematical thinking were included in the analysis? Would the same four or five Wechsler common factors arise if subtests from different batteries were factor analyzed with the Wechsler subtests? Would a PR factor split be more apparent if additional measures of fluid reasoning and visual-spatial thinking from other batteries were included in a factor analysis? Weiss and colleagues were constrained in their analysis and understanding of the WISC-IV and WAIS-IV because they had to rely solely on the subtests that comprise these batteries. A better understanding of the constructs measured by the Wechsler scales may never be fully accomplished via within-battery factor analysis.

Within-Battery Factor Analyses of the Wechsler Scales: Are They Still Necessary?

The primary structural validity tool of the Wechsler scales has been factor analysis. The earlier versions of the Wechsler Scales were evaluated using EFA (e.g., Blaha & Wallbrown, 1996; Cohen, 1959; Donders, 1993)—a method that identifies the factor structure of a cognitive battery by allowing the data to “speak for themselves,” particularly when there is no a priori theory specified (Carroll, 1993; Flanagan, McGrew, & Ortiz, 2000). By contrast, CB-CFA typically examines the extent to which an a priori hypothesized factor structure or model fits the data, as well as how the fit of the model compares to alternative models, and when used in a somewhat exploratory manner, how the model might be modified to fit the data better (Keith & Reynolds, 2012). About 15 years ago, Keith (1997) demonstrated the prominence of theory-based test development and research activities, and he and his colleagues (and other researchers) have used CHC theory-driven CFA to analyze the standardization data of the Wechsler scales (e.g., Keith et al., 2006; Keith & Witta, 1997; Weiss et al., 2013a, 2013b). Much has been learned about the Wechsler scales based on these analyses, and all for the better.

For the most part, the CHC theory-driven confirmatory factor analyses of the Wechsler scales can be categorized as within-battery. Within-battery factor analysis is confined to the tests from a single cognitive battery (e.g., the 15 subtests of the WISC-IV). Joint or cross-battery factor analysis (CB-FA) includes tests from more than one cognitive battery (e.g., the WISC-IV and the Woodcock-Johnson III, Tests of Cognitive Abilities [WJ III; Woodcock, McGrew, & Mather, 2001]). In CB-FA, the 15 WISC-IV subtests (along with WJ III tests, for example) are allowed, or are specified, to load on the various cognitive factors included in the theory.

The preponderance of structural within-battery validity studies summarized by Flanagan and colleagues (2000) and highlighted in Table 1, support the internal validity of the Wechsler scales. The movement away from the original Verbal and Performance scales in the direction of the four-factor structure (VC, PR, WM, PS) is supported by a substantial body of systematic within-battery factor analysis research conducted over the past 20 to 25 years (see Appendix A in Flanagan et al., 2000; see also Flanagan & Kaufman, 2009; Lichtenberger & Kaufman, 2013). Nevertheless, the ever changing interpretation of the original Freedom from Distractibility (FFD) tests (viz., arithmetic) and the current PR tests (e.g., do they measure Gv or Gf?) is concerning (Flanagan et al., 2000; McGrew, 1997). For example, within-battery factor analyses of the various Wechsler scales have shown that Arithmetic has changed its factor allegiance from the original Verbal factor to that of FFD, WM, and more recently Fluid Reasoning (Gf). Given that Arithmetic migrates to various factors, perhaps it should be a supplemental subtest on the WAIS-IV, as it is on the WISC-IV.

Table 1.

Within-Battery and Cross-Battery Factor Analyses of the Child and Adult Versions of the Wechsler Scales.

Battery	Factors Measured Based on Test Manual	Factors Measured Based on Selected Independent Within-Battery Research	Factors Measured Based on Independent Cross-Battery Research
WISC-R	VIQ, PIQ	Cohen (1959): VC, PO, FD Donders (1993): VC, PO, FD Anderson & Dixon (1995): VC, PO, FD Valencia, Rankin, & Oakland (1997); VC, PO, FD	Woodcock (1990): Gc, Gv, Gsm, Gs, Gq Stone (1992): Verbal Conceptual (Gc), Visual-Spatial (Gv), Attention-Quantitative (Gsm, Gf-RQ), Processing Speed (Gs)
WISC-III	VCI (Gc), POI (Gv), FDI (Gq, Gsm), PSI (Gs)	Tupa, O’Dougherty, Wright, & Fristad (1997): VCI (Gc), POI (Gv), FDI (Gq, Gsm), PSI (Gs) Keith & Witta (1997): VC (Gc), PO (Gv), PS (Gs) Burton, Sepehri, Hecht, VandenBrock, Ryan, & Drabman (2001): VCI (Gc), Constructional Praxis (Gv1), Visual Reasoning (Gv2), Working Memory (Gsm; FDI), PSI	Phelps, McGrew, Knopik, & Ford (2005): Gc, Gv, Gsm, Gs, Gq Reynolds, Keith, Flanagan, & Alfonso (in press): Gc, Gf, Gv, Gsm (Gs and Gq not represented in this research)
WISC-IV	VCI (Gc), PRI (Gf, Gv), WMI (Gsm), PSI (Gs)	Keith, Fine, Taub, Reynolds, & Kranzler (2006): Gf, Gc, Gv, Gsm, Gs Sattler (2008): VCI (Gc), PRI (Gf, Gv), WMI (Gsm), PSI (Gs) Watkins, Wilson, Kotz, Carbone, & Babula (2006): VCI (Gc), PRI (Gf, Gv), WMI (Gsm), PSI (Gs) Watkins (2006, 2010): VCI (Gc), PRI (Gf, Gv), WMI (Gsm), PSI (Gs) Golay, Reverte, Rossier, Favez & Lecerf (2012): Gf, Gc, Gv, Gsm, Gs Weiss, Keith, Zhu, & Chen (2012b): Gf, Gc, Gv, Gsm, Gs	Reynolds et al. (in press): Gc, Gf, Gv, Gsm (Gs and Gq not represented in this research)
WAIS-R	VIQ, PIQ	Silverstein (1982): VC, PO O’Grady (1983): VC, PO, FD Atkinson & Cyr (1984): VC, PO, FD Waller & Waldman (1990): VC, PO, FD	Woodcock (1990): Gc, Gv, Gsm, Gs, Gq
WAIS-III	VCI, POI, WMI, PSI	Ward, Ryan, & Axelrod (2000): VC, PO, WM, PS Taub, McGrew, & Witta (2004): VC, PO, WM, PS
WAIS-IV	VCI, PRI, WMI, PSI	Benson, Hulac, & Kranzler (2010): Gc, Gf, Gv, Gsm, Gs Canivez & Watkins (2010): VC, PR, WM, PS Niileksela, Reynolds, & Kaufman (2012): Gf, Gc, Gv, Gsm, Gs Ward, Bergman, & Hebert (2012): Verbal Comprehension (Gc), Spatial Visualization (Gv), Perceptual Reasoning (Gf), Auditory Working Memory (Gsm), Processing Speed (Gs) Weiss, Keith, Zhu, & Chen (2012a): Gf, Gc, Gv, Gsm, Gs	Holdnack, Zhou, Larabee, Millis, & Salthouse (2011): VC, PR, Processing Speed, Auditory Working Memory, Visual Working Memory, Auditory Memory, Visual Memory

Note: VIQ = Verbal Intelligence Quotient; PIQ = Performance Intelligence Quotient; VC(I) = Verbal Comprehension (Index); PO(I) = Perceptual Organization (Index); FD(I) = Freedom from Distractibility (Index); PSI = Processing Speed Index; WM(I) = Working Memory (Index); Gc = Crystallized Intelligence; Gf = Fluid Reasoning; Gsm = Short-term Memory; Gs = Processing Speed; Gv = Visual Processing; Gq = Quantitative Knowledge; RQ = Quantitative Reasoning.

As demonstrated by Woodcock (1990), it is clear that the structural validity of the Wechsler scales can be improved on through the application of theory-based cross-battery factor analysis research (see also Flanagan et al., 2000; Keith, 2005; Keith & Reynolds, 2010, 2012; Reynolds et al., in press). Furthermore, the CHC-based CB-CFAs of the Wechsler scales conducted by Phelps, McGrew, Knopik, and Ford (2005) shed light on the constructs measured and not measured by these scales (see Table 1). For example, in the CHC-driven CB-CFAs, markers for at least seven or eight broad CHC abilities were included, representing the breadth of cognitive abilities specified by CHC theory (Woodcock, 1990). In addition to Gc, Gv, Gf, Gsm, and Gs, markers for Long-term Storage and Retrieval (Glr), Auditory Processing (Ga), and Quantitative Knowledge (Gq) were specified. As expected, Arithmetic loaded on Gq (not Gf or Gsm) in both Woodcock’s and Phelps and colleagues’ studies (see also Golay, Reverte, Rossier, Favez, & Lecerf, 2012).

Nevertheless, as a result of the within-battery WISC-IV study conducted by Weiss et al. in this issue (2013b), questions about what the Arithmetic subtest measures remain. To date, the WISC-IV has only been included in one CB-CFA (Reynolds et al., in press). Unfortunately, the study conducted by Reynolds and colleagues did not include Gq markers. CB-CFAs with sufficient markers of Gq (i.e., tests of math knowledge and tests of math achievement), Quantitative Reasoning (i.e., tests involving reasoning both inductively and deductively with numbers), Gf (i.e., tests of induction and tests of general sequential reasoning not involving numbers), and Short-term Memory (Gsm; tests of memory span and tests of working memory) will therefore be needed to best understand how the Arithmetic subtest should be interpreted. Even better, maybe it is time to take Arithmetic into the laboratory. Examinees might be asked what type of strategies or thought processes they used while performing the Arithmetic subtest. It might be that Arithmetic loads on different factors for different people because people use different types of strategies while performing this multifaceted task (Keith & Reynolds, 2010).

Noteworthy is the fact that the extant within-battery and cross-battery factor analysis research and the results of content validity (e.g., expert consensus) studies were used by Flanagan and colleagues (2013) to inform how the Wechsler subtests should be classified according to the broad and narrow abilities they measure. These CHC classifications are found in Table 2. The results of Weiss and colleagues (2013a, 2013b) provide additional validity support for these Wechsler CHC classifications and did not suggest the need to alter the classifications in any way.

Table 2.

Broad and Narrow CHC Ability Representation on the Wechsler Scales Based on Within- and Cross-Battery Factor Analyses and Expert Consensus.

	Gf	Gc	Gv	Gsm	Glr	Ga	GS
WISC-IV	Matrix reasoning (I) Picture concepts (I) Underrepresented	Vocabulary (VL) Information (K0) Similarities (VL, Gf:I) Comprehension (K0) Word reasoning (VL, Gf:I)	Block design (Vz) Picture completion (CF, Gc:K0)	Digit span (MS, MW) Letter-number Sequencing (MW) Arithmetic (MW; Gf:RQ)	Not measured	Not measured	Symbol search (P) Coding (R9) Cancelation (P)
WAIS-IV	Matrix reasoning (I) Figure weights (RQ)	Vocabulary (VL) Information (K0) Similarities (VL, Gf:I) Comprehension (K0)	Block design (Vz) Picture completion (CF, Gc:K0) Visual puzzles (Vz)	Digit span (MS, MW) Letter-number Sequencing (MW) Arithmetic (MW; Gf:RQ)	Not measured	Not measured	Symbol search (P) Coding (R9) Cancelation (P)

Note: CHC classifications are from Flanagan, Ortiz, and Alfonso (2013) based on their review of the literature and expert consensus studies. Gf = Fluid Reasoning; Gc = Crystallized Intelligence; Gv = Visual Processing; Gsm = Short- term Memory; Glr = Long-term Storage and Retrieval; Ga = Auditory Processing; Gs = Processing Speed; RQ = Quantitative Reasoning; I = Induction; VL = Lexical Knowledge; K0 = General (verbal) Knowledge; Vz = Visualization; CF = Flexibility of Closure; MW = Working Memory; MS = Memory Span; P = Perceptual Speed; R9 = Rate-of-Test-Taking.

Although replication of findings is necessary, it is our opinion that within-battery factor analysis of the WISC-IV and WAIS-IV is no longer necessary. Improvements with regard to how the Wechsler subtests should be interpreted will require theory-driven CB-CFAs (or something more novel), not within-battery factor analysis. In this regard, we agree with Dr. Richard Woodcock who, at the inaugural meeting of the Richard Woodcock Institute for Advancement of Contemporary Cognitive Assessment, stated that test authors and publishers have a responsibility to provide the results of CB-FAs in their test manuals (Woodcock, 2012). Although we also believe that independent researchers will ultimately provide the peer reviewed research that is necessary to support or refute validity findings published in test manuals and to further inform interpretation of cognitive ability tests. In addition, when cross-battery data are gathered, there is a greater likelihood of obtaining information about the CHC narrow abilities that are measured by cognitive batteries. This information is important because there is a mounting body of research demonstrating the critical role of narrow cognitive abilities in the prediction of specific academic skills (see Flanagan et al., 2013; Flanagan, Ortiz, Alfonso, & Mascolo, 2006; and McGrew & Wendling, 2010 for summaries).

Do the Wechsler Scales Measure CHC Narrow Abilities Important for Learning and Achievement?

Yes, but not as many as other batteries. A considerable amount of research has demonstrated an empirical relationship among cognitive abilities, neuropsychological processes, and specific academic skills (see Flanagan et al., 2006; Fletcher, Lyon, Fuchs, & Barnes, 2007; Hale & Fiorello, 2004; and McGrew & Wendling, 2010 for summaries). Much of the recent research on cognitive-academic relationships has been interpreted within the context of CHC theory (e.g., Flanagan et al., 2013) and with specific instruments developed from CHC theory (e.g., McGrew & Wendling, 2010). In addition, statistical analyses, such as structural equation modeling, have been used to understand the extent to which specific cognitive abilities (both broad and narrow) explain variance in academic skills above and beyond the variance accounted for by g (e.g., Floyd, McGrew, & Evans, 2008; Juarez, 2012; McGrew, Flanagan, Keith, & Vanderwood, 1997; Vanderwood, McGrew, Flanagan, & Keith, 2002).

For example, in the area of reading, narrow abilities in seven broad CHC domains appear to be important to assess in individuals who struggle to read (McGrew & Wendling, 2010). Specifically, narrow abilities subsumed by Gc (language development, lexical knowledge, listening ability, general information), Gsm (memory span, working memory), Ga (phonetic coding), Glr (associative memory, naming facility, meaningful memory), and Gs (perceptual speed) are related significantly to reading achievement. Furthermore, developmental results suggest that the Ga, Gs, and Glr relations with reading are strongest during the early elementary school years, after which they systematically decrease in strength (e.g., Flanagan et al., 2006; McGrew, 1993). In contrast, the strength of the relations between Gc abilities and reading achievement increases with age (McGrew & Wendling). The Gv abilities of orthographic processing and visual memory are also related to reading achievement (e.g., Berninger, 1990). Narrow Fluid Reasoning (Gf) abilities appear related primarily to reading comprehension from childhood to young adulthood. Finally, related to Gf, the role of executive functions in reading achievement (particularly reading comprehension) has been demonstrated in the neuropsychology literature (e.g., McCloskey, Whitaker, Murphy, & Rogers, 2012), and is consistent with neuropsychological evidence (Decker, Hill, & Dean, 2007).

In a school setting, in particular, cognitive batteries are administered to students who experience difficulties in the learning process (e.g., students who are suspected of having a specific learning disability). In light of the cognitive-achievement relations research, practitioners seek to measure the cognitive abilities and processes that are most closely associated with the academic areas in which a student has difficulty. Table 3 shows the narrow abilities purportedly measured by the WISC-IV (and Wechsler Individual Achievement Test, Third Edition, Pearson, 2009; WIAT-III) that are related significantly to reading achievement. In addition, this table shows the narrow abilities related to reading that are measured by another popular cognitive battery—the WJ III (cognitive and achievement)—as well as a popular neuropsychological battery—the NEPSY-II (Korkman, Kirk, & Kemp, 2007). As may be seen in this table, the WJ III and NEPSY-II measure about one and one half as many narrow abilities important for reading achievement than the WISC-IV/WIAT-III. Three main conclusions may be drawn from this observation. First, modeling of CHC narrow abilities should be part of a systematic program of Wechsler-based CHC CB-CFA research. Second, future editions of the WISC-IV (and WIAT-III) should measure a greater breadth of narrow abilities. Third, the Wechsler scales should be conormed with other tests to allow for a greater breadth of narrow abilities to be measured.

Table 3.

Abilities and Processes Related to Reading Achievement Measured by Subtests on Popular Batteries.

Relevant Broad CHC Ability and Neuropsychological Domain	Relevant Narrow CHC Ability and Neuropsychological Process	Most Relevant WISC-IV and WIAT-III Subtests	Most Relevant WJ III COG and ACH Subtests	Most Relevant NEPSY-II Subtests
Gf—fluid reasoning	I–Induction	Matrix reasoning	Concept formation	Animal sorting
	RG–gen seq reasoning (deduction)	− −	Analysis-synthesis	− −
Gc— Crystallized Intelligence	LS–listening ability	WIAT-III listening comprehension	WJ III NU ACH oral compreshension	Comprehension of instructions
	K0–general information	Information	General information	Body part naming and identification
	VL–lexical knowledge	Vocabulary	Verbal comprehension	− −
Gsm—short-term memory	MS–memory span	Digit span–forward	Memory for words	List memory
	MW–working memory capacity	Letter-number sequencing	Auditory working memory	Auditory attention and response set
Gv—visual processing	MV–visual memory	− −	Picture recognition	Memory for designs
	Orthographic processing	− −	− −	− −
Ga—auditory processing	PC–phonetic coding	WIAT-III early reading skills	Incomplete words	Phonological processing
	US–speech-sound discrimination	− −	− −	− −
Glr—long-term storage and retrieval	NA–naming facility (rapid naming)	− −	Rapid picture naming	Speeded naming
	MA–associative memory	− −	Visual auditory learning	Memory for names
	M6–free recall memory	− −	− −	List memory delayed
	MM–meaningful memory	− −	WJ III NU ACH story recall	Narrative memory
Gs—processing speed	RS–reading speed (with full comp)	WIAT-III oral reading fluency	WJ III NU ACH reading fluency	− −
	P–perceptual speed	Symbol search	Visual matching	− −
Attention	Selective; focused; sustained	Cancellation	Consider broad attention clinical cluster	Auditory attention and response set
Executive function	Consider cascading production Decrements/increments model (McCloskey, Perkins, & VanDivner, 2009)	− −	Consider executive processes clinical cluster	Word generation

Note: Dashes indicate that there are no direct measures of the ability or process in the corresponding battery. All Gc narrow abilities involve LD or Language Development.

Conclusions and Recommendations

The Weiss et al. (2013a, 2013b) studies of the WISC-IV and WAIS-IV provided important information about the factorial invariance of these instruments, specifically invariance across populations. In this commentary we argued, however, that a better understanding of the constructs measured by the Wechsler scales, or invariance across selection of variables will require CB-FA. Interestingly, we were only able to locate 14 CB-FAs (mostly confirmatory) in the peer-reviewed literature. Nevertheless, these studies should be used as a guide for future CB-CFA research, as there is much to learn from what has already been accomplished.

Table 4 provides a summary of the 14 studies that used CB-CFA with ability subtests since 1990. This table reports the major findings of each study, as well as a “Comments” column that highlights the unique contribution each study offered to the literature. The majority of these studies used Fluid-Crystallized (Gf-Gc) or CHC-driven CB-CFA to evaluate the structural validity of intelligence and cognitive ability batteries. Several conclusions can be drawn from the information presented in this table.

Table 4.

A Description of Cross-Battery Factor Analysis Studies.

Study	Description of Sample	Cognitive Batteries Included	Major Findings	Comments
Woodcock (1990)	Nine data sets with sample sizes ranging from N = 53 to N = 4,261	WJ, WJ-R, KABC, SB:IV, WAIS, WAIS-R, WISC-R	There were several major findings of this set of exploratory and confirmatory factor analyses. First, the WJ-R measured eight broad Gf-Gc cognitive abilities, while the other intelligence tests measured between three and five. As such, when not using the WJ-R, it was suggested that clinicians “cross” batteries to obtain the information necessary for a particular evaluation. Second, intelligence tests are underfactored because the breadth of cognitive abilities is not represented in most within- battery factor analyses. Third, it was suggested that clinicians base interpretations of specific cognitive abilities on at least two clean measures of any factor assessed (i.e., tests with a strong loading on one factor and insignificant loading on all others). Fourth, caution should be exercised in the use of mixed measures of ability (i.e., subtests that measure more than one factor to a substantial degree), as these measures are psychometrically ambiguous and therefore difficult to interpret.	Following Woodcock’s cross-battery factor analyses, it became clear that the most popular tests of intelligence were both limited in breadth and depth of abilities measured and were underfactored. The findings reported in this article provided the impetus for the development of the cross-battery approach to measurement and interpretation (McGrew & Flanagan, 1998). Woodcock’s classification of tests as either strong, moderate, or mixed measures of broad Gf- Gc abilities were used in test classifications for cross-battery assessment (e.g., McGrew & Flanagan, 1998; Flanagan, Ortiz, & Alfonso, 2007). In short, this study laid the foundation for (a) subsequent joint or cross-battery factor analyses of cognitive ability tests; (b) the criteria for classifying subtests as measures of the Gf-Gc (now CHC) broad abilities; (c) the CHC cross-battery approach; (d) the notion that there are multiple cognitive abilities that ought to be measured and their predictive validity studied.
Stone (1992)	N = 115 School aged children	DAS, WISC-R	Four-factor solution: Verbal-conceptual (VC), Visual-spatial (VS), Attention-Quantitative (AQ), Processing Speed (PS). Underfactoring on the AQ factor was a problem due to a limited number of indicators to measure all factors underlying the two batteries.	The VC, VS, and PS factors were supported by multiple indicators of Gc, Gv, and Gs, respectively. The AQ factor appeared to tap at least two factors, namely Gsm (memory span) and Gf (Quantitative Reasoning). This study was the first to include the DAS in a cross-battery factor analysis.
McGhee (1993)	N = 100, ages 8 to 12 years, 11 months.	WJ-R, DTLA-3, DAS	This study was conducted to explore Gf-Gc theory and gain information about the abilities measured by three intelligence batteries. Six models were tested. An eight-factor Gf-Gc model emerged as the best fit to the data. The eight factors included Glr, Gsm-a (auditory), Gs, Ga, Gv, Gc, Gf, Gsm-v (visual).	This was the first study to include the DTLA-3 in a theory-driven, cross-battery confirmatory factor analysis. The WJ-R was found to measure the same abilities as suggested by Woodcock (1990). Of the five DAS subtests that were included in this study, the classifications of two (i.e., Recall of Designs and Matrices) differed from those reported in Stone (1992). Stone found the Recall of Designs subtest measured Gv, whereas McGhee reported that it measured Gsm-v. It is likely that both labels are appropriate because Gv subsumes a narrow ability (i.e., visual memory) similar to McGhee’s Gsm-v ability. Stone found that the DAS Matrices subtest had significant loadings on three factors (Gc/Gv/Gs), whereas McGhee found that this subtest measures Gf. Stone did not specify a Gf factor in his analyses. McGhee’s finding that DAS Matrices measures Gf is consistent with current research. Overall, McGhee provided further support for the factor structure of the WJ-R and clarification of the abilities underlying the DAS subtests.
Flanagan & McGrew (1998)	N = 114, grades 5 to 8, ages 10 years, 11 months to 15 years, 11 months	WJ-R, KAIT	The KAIT measures Visual Memory (a narrow Gv ability), Memory Span (a narrow Gsm ability), Gf, Gc, and Glr (Associative Memory). A Gf-Gc nine-factor model was the most plausible a priori model fit of the WJ-R/KAIT data, a finding that extended the replicability of the Gf-Gc model to a non-White sample. The factorial structure of the KAIT put forward by its authors (i.e., a two-factor Gf-Gc model) was not supported.	Findings contributed to the theoretical literature by supporting the invariance of a contemporary Gf-Gc model in a non-White sample. In addition, this study showed that when a sufficiently diverse array of cognitive tests are used, older, more simple (e.g., dichotomous), models are not supported. This was the first study to demonstrate that most of the Gf-Gc factors that are represented on the WJ-R and KAIT warrant a narrow stratum I (and not a broad stratum II) interpretation. In other words, this study pointed out that certain composites intended to provide estimates of broad cognitive abilities actually provided estimates of narrow abilities. Therefore, it was suggested that practitioners recognize the breadth and depth of coverage of cognitive abilities on intelligence batteries and cross them when necessary to ensure that a sufficient range of stratum I indicators are assessed before generalizing about functioning in broad Gf-Gc domains.
Keith, Kranzler, & Flanagan (2001)	N = 155, ages 8 to 11 years	CAS, WJ III	The PASS model underlying the CAS was not supported. Results support the alternative (CHC) explanation of the factor structure of the CAS in which the Planning/Attention scales are combined and interpreted as measures of Processing Speed (Gs). The CAS Successive Scale was found to be a measure of Short-term Memory (Gsm—Memory Span). The CAS Simultaneous Scale was found to be a measure of broad Visualization (Gv).	This study did not test the validity of the WJ III. The WJ III tests were used to provide markers for the CHC abilities. This study was the first to use confirmatory factor analysis to test the viability of the PASS theory.
Roid (2003)	N = 145, ages 4 to 11 years	SB5, WJ III	Five models were contrasted, including a one-factor (g) model, a two-factor (Gf-Gc) model, a three-factor model (abstract-visual, quantitative/knowledge, and memory), a four-factor model (Gf, Gc, quantitative, and working memory), and a five-factor model (Gf, Gc, Gq- RQ, Gv, and Gsm-MW). Results demonstrated that the five-factor CHC model fit the data best.	The findings of this study are consistent with extant SB5 CHC classifications. The five-factor model included two narrow abilities, namely Quantitative Reasoning (RQ) and Working Memory (MW). Interestingly, although RQ is a narrow ability subsumed by Gf, it emerged as a factor separate from Gf, the latter of which consisted of tests that measure the narrow abilities of inductive and general sequential reasoning.
Tusing & Ford (2004)	N = 158 4 and 5 year olds	DAS, WJ-R	In this study, five broad ability factors were reliably identified, namely Gc, Glr, Gsm, Ga, and nonverbal (or a Gv-like factor).	This was the first study to use cross-battery confirmatory factor analysis in a preschool sample. The results of these analyses demonstrated that the structure of cognitive abilities is more differentiated in young children than what had been demonstrated consistently in previous within-battery factor analyses.
Phelps, McGrew, Knopik, & Ford (2005)	N = 148 ages 8 to 12 years	WISC-III, WJ III COG, WJ III ACH, WJ III DS	Data provide consistent validity evidence for the previously identified WISC-III and WJ III broad CHC test classifications (from Flanagan, McGrew, & Ortiz, 2000). The specification and evaluation of the g+broad+narrow CHC model provided empirical information for evaluating logically based narrow ability WISC-III/WJ III test classifications.	This study was among the first to model CHC narrow abilities. The logical classifications reported in Flanagan and Ortiz (2001) and Flanagan, Ortiz and Alfonso (2007) based on their expert consensus analyses were supported by this research.
Hunt (2007)	N = 200 ages 4 to 5 years, 11 months	WJ III, KABC-II	Three models were contrasted, including a one-factor (g) model, a two-stratum CHC model and a three-stratum CHC model. The latter two models included Gc, Gv, Glr, and Gsm. The two-stratum CHC model fit the data best, although the fit was only slightly better than the three-stratum CHC model.	Ga, Gf, and Gs were not represented in the models for two primary reasons. First, Gf did not appear to represent a distinct ability when the WJ III was factor analyzed independently. Second, the KABC-II does not include measures of Gf at the preschool age range. Furthermore, Ga and Gs are not measured by the KABC-II. The overall conclusions of the study were that the KABC-II is more developmentally appropriate for use with preschoolers and that crossing selected tests from the KABC-II and the WJ III would provide the most robust measurement of Gc, Gv, Glr, and Gsm in preschool children.
Sanders, McIntosh, Rothlisberg, & Finch (2007)	N = 131 Grades 3-5 (ages 8 years, 3 months - 12 years, 3 months)	WJ III, DAS	A synthesized three-stratum model provided the most parsimonious representation of the data, as compared to a one factor g model and a two- stratum CHC model. The study demonstrated that Gc, Glr, Gv, Gf, Gs, and Gsm were measured by both the WJ III and DAS. Note that the WJ III Ga tests were not administered. Data provide consistent validity evidence for the previously identified WJ III and DAS broad CHC test classifications (Flanagan & Ortiz, 2001; Woodcock, McGrew, & Mather, 2001, 2007).	WJ III Rapid Picture Naming loaded on the Gs factor, which is consistent with the results reported in the WJ III manual (Woodcock et al., 2001, 2007). However, despite its high correlation with Gs tests, Rapid Picture Naming has been classified as a measure of Glr because its task requirements are consistent with the CHC narrow ability theoretical construct of Naming Facility (Flanagan et al., 2007; Woodcock et al., 2001, 2007).
Floyd, Bergeron, Hamilton, Parra, & McGrew (2010)	N = 100 Ages 8 to 18 years	WJ III, D-KEFS	Based on the best-fitting CFA model tested, the following factors explain the cross-battery data set: Gc, Executive Function (EF), Gsm, Gv, Glr, and Gs. Although the Gf construct did not emerge in this study, well-established tests of Gf loaded on Gc (e.g., WJ III Analysis-Synthesis) along with D-KEFS tests that require reasoning with prior verbal knowledge, and EF (e.g., WJ III Concept Formation) along with D- KEFS tests that require mental flexibility (e.g., Verbal Fluency Category Switching). The CHC and EF classifications of WJ III and D- KEFS are highly consistent with those provided by Flanagan, Alfonso, Ortiz, and Dynda (2010) based on content analysis.	The findings of Floyd et al. provide validity evidence for Flanagan et al.’s CHC and neuropsychological classifications of the D- KEFS. All D-KEFS tests that loaded on Gc, for example, were classified by Flanagan et al. as measures of Gc. However, Flanagan et al. also indicated that each of these tests measured Gf, a finding consistent with Floyd et al.’s observation that each of the D-KEFS Gc tests also require reasoning with prior verbal knowledge. Floyd et al.’s classifications of D-KEFS tests of Glr, Gs, and EF were also consistent with Flanagan et al.’s classifications with few and only minor exceptions. For example, both sets of analyses classified Design Fluency Switching as EF and Gs. However, Flanagan et al., also stated that Gv is likely involved in Design Fluency tasks.
Keith & Reynolds (2010) ^a	N = 544, ages 6 to 12 years 6 months	WISC-R KABC	The WISC-R and KABC measure overlapping constructs that are explainable from CHC theory, including Gc, Gv, Gsm, Gq, and Grw. Given a lack of sufficient clear measures, it was not possible to specify Gf and Gs factors. When a higher order g factor was added to the model, the two arithmetic tests had the highest loading on the g factor.	Matrix Analogies was the only Gf indicator and Coding was the only Gs indicator. In the current analysis, these two tests loaded on Gv because there were not a sufficient number of indicators to specify separate Gf and Gs factors. There were no indicators of Ga and Glr in the WISC- R/KABC data set.
Holdnack et al. (2011)		WAIS-IV WMS-IV	Analyses demonstrated that two models fit the data well and that both models were equally plausible. The seven factor model included: Verbal Comprehension, Perceptual Reasoning, Processing Speed, Auditory Working Memory, Visual Working Memory, Auditory Memory, and Visual Memory. The five-factor model consisted of Verbal Comprehension, Perceptual Reasoning, Processing Speed, Working Memory and Memory with a hierarchical g.	This study included only the core subtests of the WAIS-IV and six subtests from the WMS-IV. It is necessary to include relevant markers of Glr fully to appreciate the abilities underlying this battery of subtests.
Reynolds, Keith, Flanagan, & Alfonso (in press)	N = 535, ages 6 to 16 years	KABC-II, WJ III, WISC-III, WISC-IV, PIAT- R/NU	CHC broad abilities were found to be invariant across populations and selection of variables. Gf was found to be statistically indistinguishable from g. There was some evidence of a Flynn Effect across the WISC-III and WISC-IV standardization samples. The effect was not entirely due to g, however, as some of this effect was attributed to a specific factor mean related to the Similarities subtest. A reference variable approach allowed for a greater breadth of broad ability indicators to be used while studying CHC broad abilities	Arithmetic from the WISC-III was found to measure Gsm, or more specifically working memory. Additional tests of quantitative reasoning or applied math, however, were not included, which precluded tests of alternative hypotheses. Picture Completion was not a good measure of Gv, and appeared to measure Gc. WISC-IV subtests loaded on the expected factors, including the Matrix Reasoning and Picture Concepts tests, which have been identified previously as indicators of Gf.

The authors reanalyzed data from Keith and Novak (1987).

First, the CB-CFAs conducted since 1990 included a total of 20 cognitive batteries. However, 11 of the 14 studies included either the WJ-R or WJ III. Only one study included the WISC-IV (Reynolds et al., in press) and only one study included the WAIS-IV (Holdnack, Zhou, Larrabee, Mills, & Salthouse, 2011). Because the WJ-R and WJ III batteries were developed from Gf-Gc and CHC theory, respectively, the structure of these batteries guided nearly all CB-CFAs included in Table 4 and few models tested alternative explanations of the WJ factor structure. These studies were necessary to derive a better understanding of the broad cognitive abilities measured by commonly used cognitive batteries, including the Wechsler scales, but the CB-CFAs conducted to date are too WJ “heavy.”

Second, nine CHC broad cognitive abilities were represented among the CB-CFA studies, including Gf, Gc, Gv, Gsm, Glr, Ga, Gs, Gq, and Grw. The number of CHC broad ability factors representing the best fitting models in these studies ranged from five to nine, with Gf, Gc, Gsm, Gv, and Gs among the most frequently represented. The broad abilities of Gq and Grw were not included in most CB-CFAs because subtests that assess these abilities are typically measured by achievement batteries. The broad abilities of Ga and Glr were represented by only a few subtests in CB-CFAs because these abilities are not typically measured by a majority of intelligence batteries, including the Wechsler scales. Therefore, it is recommended that future Wechsler-based CB-CFAs include tests from batteries on which Ga and Glr are well represented.

Third, with few exceptions (e.g., Phelps et al., 2005), nearly all of the CB-FAs included in Table 4 modeled broad cognitive abilities. There needs to be more modeling of narrow abilities in light of their importance in understanding learning and in explaining performance in specific academic skills. Greater focus on narrow cognitive abilities may result in better diagnostic assessments (see Table 3).

Fourth, the sample sizes included in the analyses ranged from 53 to 4,261 (median sample size = 145), with the majority of samples ranging from 100 to 200 participants. Overall, the sample size to variable ratio is somewhat low for the type of analyses that were conducted. It is very likely that sample sizes are generally small because cross-battery studies are quite time consuming. That is, cross-battery studies require data from at least two batteries. Giving two complete (or even partial) batteries to a sample of individuals typically requires administration of about 20 subtests at a minimum. Conducting this type of study, therefore, is onerous for researchers and participants alike. Also, in many instances, it may be more desirable to administer subtests from multiple batteries to the same subjects, particularly to test models that include narrow abilities. In order to circumvent the limitations of CB-FAs, Keith and Reynolds (2010) suggested the use of reference variable methodology, which takes advantage of planned missingness (McArdle, 1994). This methodology combines data from different studies, given an overlapping series of tests, referred to as the reference variables (Keith & Reynolds; Reynolds et al., in press). To assist in designing future CB-CFAs, including those using the reference variable methodology, we compiled a table that includes all of the subtests (from Table 4) that have been included in CB-FAs and the number of times each subtest was included in an analysis (e.g., once, twice, three, or more times). This information (found in Table 5) can be used to identify the subtests from various batteries that have the most consistent construct validity support and therefore, can be used confidently as reference variables.

Table 5.

CHC Broad Ability Classifications of Cognitive Ability Subtests Based on Cross-Battery Factor Analyses.

	Gf	Gc	Gv	Gsm	Glr	Ga	Gs	Gq
WISC-IV WAIS-IV WPPSI-III WMS-IV WNV¹	Matrix Reasoning Picture Concepts Figure Weights	VOCABULARY INFORMATION SIMILARITIES COMPREHENSION Word Reasoning Receptive Vocabulary Picture Naming	BLOCK DESIGN PICTURE COMPLETION Visual Puzzles OBJECT ASSEMBLY PICTURE ARRANGEMENT Mazes	DIGIT SPAN Letter-Number Sequencing Logical Memory Verbal Paired Associates Designs Visual Reproduction Spatial Addition Symbol Span			Symbol Search CODING Cancellation	Arithmetic
KABC-II	Pattern Reasoning Story Completion	Expressive Vocabulary Verbal Knowledge RIDDLES	Face Recognition TRIANGLES Gestalt Closure Rover Block Counting Conceptual Thinking	NUMBER RECALL WORD ORDER Hand Movements	Atlantis Rebus Atlantis Delayed Rebus Delayed
WJ III NU COG and DS	CONCEPT FORMATION ANALYSIS-SYNTHESIS	VERBAL COMPREHENSION GENERAL INFORMATION Story Recall Oral Comprehension PICTURE VOCABULARY Academic Knowledge ORAL VOCABULARY Listening Comprehension Verbal Analogies	SPATIAL RELATIONS PICTURE RECOGNITION Planning VISUAL CLOSURE	MEMORY FOR WORDS NUMBERS REVERSED AUDITORY WORKING MEMORY MEMORY FOR SENTENCES	VISUAL-AUDITORY LEARNING Visual-Auditory Learning –Delayed Retrieval Fluency RAPID PICTURE NAMING MEMORY FOR NAMES Memory for Names– Delayed	SOUND BLENDING Auditory Attention INCOMPLETE WORDS Work Attack Sound Patterns	VISUAL MATCHING DECISION SPEED Pair Cancelation Math Fluency Writing Fluency CROSS OUT	Calculation APPLIED PROBLEMS Quantitative Concepts
SB5	Nonverbal Fluid Reasoning Verbal Fluid Reasoning Nonverbal Quantitative Reasoning Verbal Quantitative Reasoning	Nonverbal Knowledge Verbal Knowledge	Nonverbal Visual- Spatial Processing Verbal Visual-Spatial Processing	Nonverbal Working Memory Verbal Working Memory
DAS-II	MATRICES Picture Similarities Sequential & Quantitative Reasoning	Early Number Concepts Naming Vocabulary WORD DEFINITIONS Verbal Comprehension VERBAL SIMILARITIES	PATTERN CONSTRUCTION RECALL OF DESIGNS Recognition of Pictures Copying Matching Letter-Like Forms	RECALL OF DIGITS- FORWARD RECALL OF DIGITS- BACKWARD Recall of Sequential Order	Rapid Naming Recall of Objects- Immediate Recall of Objects- Delayed	Phonological Processing	Speed of Information Processing
D-KEFS	Design Fluency: Switching Verbal Fluency: Category Switching	Sorting: Free Sorting Word Context Sorting: Sort Recognition Twenty Questions		Trail Making: Number-Letter Switching	Verbal Fluency: Category Verbal Fluency: Letter		Color-Word Interference: Inhibition
CAS			Nonverbal Matrices Verbal Spatial Relations Figure Memory		Word Series Sentence Repetition Sentence Questions		Matching Numbers Planned Codes Planned Connections Expressive Attention Number Detection Receptive Attention
KAIT	Logical Steps Mystery Codes	Definitions Auditory Comprehension Double Meanings Auditory Delayed Recall Famous Faces	Memory for Block Designs		Rebus Learning Rebus Delayed Recall
DTLA-4²	Symbolic Relations	Word Opposites Basic Information	Design Sequences Reversed Letters Story Construction Design Reproduction	Sentence Imitation Word Sequences Story Sequences

Note: Subtests in bold, uppercase type indicate that they were included in at least three cross-battery factor analyses that yielded consistent results. Thus, there is little doubt regarding the CHC broad abilities they measure. Subtests in bold, lowercase type indicate that they were included in two cross-battery factor analyses that yielded consistent results. Thus, it is likely that they are measures of the CHC broad abilities listed. Subtests in lowercase type indicate that they were included in only one cross-battery factor analysis or results of multiple analyses were inconsistent. Subtests in italics indicate that they were not included in any cross-battery factor analysis to date and thus, CHC broad ability classifications are based on expert consensus and within-battery factor analyses. WISC-IV = Wechsler Intelligence Scale for Children-Fourth Edition (Wechsler, 2003); WAIS-IV = Wechsler Adult Intelligence Scale-Fourth Edition (Wechsler, 2008); WPPSI-III = Wechsler Preschool and Primary Scale of Intelligence-Third Edition (Wechsler, 2002); WNV = Wechsler Nonverbal Scale of Ability (Naglieri & Wechsler, 2006); KABC-II = Kaufman Assessment Battery for Children-Second Edition (Kaufman & Kaufman, 2004); WJ III NU COG = Woodcock-Johnson III, Normative Update, Tests of Cognitive Abilities (Woodcock, McGrew, & Mather, 2001, 2007); WJ III NU DS = Woodcock-Johnson III, Normative Update, Disagnostic Supplement (Woodcock, Schrank, & McGrew, 2004, 2007); WMS-IV = Wechsler Memory Scale–Fourth Edition (Pearson, 2009); SB5 = Stanford-Binet Intelligence Scales-Fifth Edition (Roid, 2003); DAS-II = Differential Ability Scales-Second Edition (Elliott, 2007). KAIT = Kaufman Adolescent and Adult Intelligence Test (Kaufman & Kaufman, 1993); CAS = Cognitive Assessment System (Naglieri & Das, 1997); D-KEFS = Delis-Kaplan Executive Function System (Delis, Kaplan, & Kramer, 2001); DTLA-4 = Detroit Test of Learning Aptitude-Fourth Edition (Hammill, 1998). Gf = Fluid Intelligence; Gc = Crystallized Intelligence; Gv = Visual Processing; Gsm = Short-term Memory; Glr = Long-term Storage and Retrieval; Ga = Auditory Processing; Gs = Processing Speed; Gq = Quantitative Knowledge.

The WNV has not been included in any cross-battery factor analyses to date. However, tests from this battery (e.g., Object Assembly, Picture Arrangement) were included in this type of analysis when they were on the WISC-III.

The DTLA-4 has not been included in any cross-battery factor analyses to date. However, the tests listed in the DTLA-4 row are the ones from the DTLA-3 that were included in this type of analysis.

Table 5 lists the current editions of the batteries that were included in CB-FAs. Subtests of the latest editions of cognitive batteries are included in Table 5 and are listed in the column of the broad ability presumed to be measured by the subtest. For example, the Wechsler Vocabulary subtest is listed in the Gc column, as this subtest measures a narrow Gc ability, namely lexical knowledge. Subtests printed in bold, uppercase type indicate that they were included in at least three CB-FAs that yielded consistent results. In general, the more times a subtest was included in CB-FA and the more consistently the subtest was found to measure a particular broad ability, the more confidence one can have in that classification. For example, the Wechsler Block Design subtest has consistently loaded on a Gv factor in more than three CB-FAs and therefore, we can be confident that the primary broad ability measured by this subtest is Gv.

Subtests printed in bold, lowercase type in Table 5 were included in two CB-FAs that yielded consistent results. KABC-II Atlantis and Rebus (listed under Glr) are in this category. It is very likely that these subtests (bold, lowercase) are measures of the CHC broad abilities listed. Subtests printed in lowercase type were included in only one CB-FA or results of multiple analyses were inconsistent. The Wechsler arithmetic subtest falls into this category for the latter reason. Subtests printed in italics in Table 5 were not included in any CB-FA analysis to date and thus, CHC broad ability classifications are based on expert consensus and within-battery factor analyses. Several Wechsler subtests fall into this category, such as Figure Weights and Word Reasoning. Although these cognitive tests have been classified according to the CHC broad abilities they measure via within-battery FA and content analysis using multiple experts as raters (e.g., Flanagan et al., 2013), the construct validity of these subtests will be enhanced when they are included in CHC designed CB-CFAs.

The results of CB-FAs summarized in Table 5 provide a strand of validity evidence for the CHC broad ability constructs underlying currently used cognitive ability tests. The subtests printed in bold uppercase letters should be used as markers of their respective broad CHC abilities in reference variable studies. Overall, Table 5 may be used to understand the CHC broad abilities measured by cognitive ability subtests and as a blueprint for designing future Wechsler-based (and other) CB-CFAs. The narrow ability classifications provided in Table 2 and in other sources (e.g., Flanagan et al., 2013; Miller, 2013) should be used to guide the design of Wechsler-based (and other) CB-CFAs that model narrow abilities.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Anderson

Dixon

W. E.

Jr. (1995). Confirmatory factor analysis of the Wechsler Intelligence Scale for Children–revised with normal and psychiatric adolescents. Journal of Research on Adolescence, 5, 319-332.

Atkinson

Cyr

J. J.

(1984). Factor analysis of the WAIS-R: Psychiatric and standardization samples. Journal of Consulting and Clinical Psychology, 52, 714-716.

Benson

Hulac

D. M.

Kranzler

J. H.

(2010). Independent examination of Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV): What does the WIAS-IV measure? Psychological Assessment, 22, 121-130.

Berninger

V. W.

(1990). Multiple orthographic codes: Key to alternative instructional methodologies for developing the orthographic-phonological connections underlying word identification. School Psychology Review, 19, 518-533.

Blaha

Wallbrown

F. H.

(1996). Hierarchical factor structure of the Wechsler Intelligence Scale-III. Psychological Assessment, 8, 214-218.

Burton

D. B.

Sepehri

Hecht

VandenBrock

Ryan

J. J.

Drabman

(2001). Confirmatory factor analysis of the WISC-III in a clinical sample with cross-validation in the standardization sample. Child Neuropsychology, 7, 104-116.

Carroll

J. B.

(1993). Human cognitive abilities: A survey of factor-analytic studies. New York, NY: Cambridge University Press.

Canivez

G. L.

Watkins

M. W.

(2010). Investigation of the factor structure of the Wechsler Adult Intelligence Scale–Fourth Edition (WAIS-IV): Exploratory and higher order factor analyses. Psychological Assessment, 22, 827-836.

Chen

Keith

Chen

Chang

(2009). What does the WISC-IV measure?: Validation of the scoring and CHC-based interpretive approaches. Journal of Research in Education Sciences, 54, 85-108.

10.

Cohen

(1959). The factorial structure of the WISC at ages 7-6, 10-6, and 13-16. Journal of Consulting Psychology, 23, 285-299.

11.

Decker

S. L.

Hill

S. K.

Dean

R. S.

(2007). Evidence of the construct similarity in executive functions and fluid reasoning abilities. International Journal of Neuroscience, 117, 735-748.

12.

Delis

D. C.

Kaplan

Kramer

J. H.

(2001). Delis Kaplan Executive Function System. San Antonio, TX: The Psychological Corporation.

13.

Donders

(1993). Factor structure of the WISC-R in children with traumatic brain injury. Journal of Clinical Psychology, 49, 256-260.

14.

Elliott

C. D.

(2007). Differential Ability Scales–second edition. San Antonio, TX: Harcourt Assessment.

15.

Flanagan

D. P.

Alfonso

V. C.

Ortiz

S. O.

Dynda

A. M.

(2010). Best practices in cognitive assessment for school neuropsychological evaluations. In Miller

D. C.

(Ed.), Best practices in school neuropsychology: Guidelines for effective practice, assessment, and evidence-based intervention (pp. 101-140). New York, NY: John Wiley and Sons.

16.

Flanagan

D. P.

Kaufman

A. S.

(2009). Essentials of WISC-IV assessment (2nd ed.). New York, NY: John Wiley.

17.

Flanagan

D. P.

McGrew

K. S.

(1998). Interpreting intelligence tests from contemporary Gf-Gc theory: Joint confirmatory factor analyses of the WJ-R and KAIT in a non-White sample. Journal of School Psychology, 36, 151-182.

18.

Flanagan

D. P.

McGrew

K. S.

Ortiz

S. O.

(2000). The Wechsler intelligence scales and Gf- Gc theory: A contemporary approach to interpretation. Needham Heights, MA: Allyn and Bacon.

19.

Flanagan

D. P.

Ortiz

S. O.

(2001). Essentials of cross-battery assessment. New York, NY: Wiley.

20.

Flanagan

D. P.

Ortiz

S. O.

Alfonso

V. C.

(2007). Essentials of cross-battery assessment, second edition. Hoboken, NJ: Wiley.

21.

Flanagan

D. P.

Ortiz

S. O.

Alfonso

V. C.

(2013). Essentials of cross-battery assessment, Third Edition. Hoboken, NJ: Wiley.

22.

Flanagan

D. P.

Ortiz

S. O.

Alfonso

V. C.

Mascolo

(2006). The achievement test desk reference (ATDR)—second edition: A guide to learning disability identification. New York, NY: Wiley.

23.

Fletcher

J. M.

Lyon

G. R.

Fuchs

L. S.

Barnes

M. A.

(2007). Learning disabilities: From identification to intervention. New York, NY: Guilford Press.

24.

Floyd

R. G.

McGrew

K. S.

Evans

J. J.

(2008). The relative contributions of the Cattell-Horn-Carroll cognitive abilities in explaining writing achievement during childhood and adolescence. Psychology in the Schools, 45, 132-144.

25.

Floyd

R. G.

Bergeron

Hamilton

Parra

G. R.

McGrew

K. S.

(2010). How do executive functions fit with the Cattell - Horn - Carroll model? Some evidence from a joint factor analysis of the Delis - Kaplan Executive Function System and the Woodcock - Johnson III Tests of Cognitive Abilities. Psychology in the Schools, 47, 721-738.

26.

Golay

Reverte

Rossier

Favez

Lecerf

(2012). Further insights on the French WISC-IV factor structure through Bayesian structural equation modeling. Psychological Assessment, 20, 1-13.

27.

Hale

J. B.

Fiorello

C. A.

(2004). School neuropsychology: A practitioner’s handbook. New York, NY: Guilford Press.

28.

Hammill

D. D.

(1998). Detroit Tests of Learning Aptitude - Fourth Edition. Austin, TX: PRO-ED.

29.

Holdnack

J. A.

Zhou

Larrabee

G. J.

Millis

S. R.

Salthouse

T. A.

(2011). Confirmatory factor analysis of the WAIS-IV/WMS-IV. Assessment, 18, 178-191.

30.

Hunt

M. S.

(2007). A joint confirmatory factor analysis of the Kaufman Assessment Battery for Children, Second Edition, and the Woodcock-Johnson Tests of Cognitive Abilities, Third Edition, with preschool children. Unpublished Dissertation, Ball State University, Muncie, Indiana.

31.

Juarez

B.M.

(2012). Effects of general and specific cognitive abilities on reading achievement in a referred sample (Dissertation abstracts). St. John’s University, New York.

32.

Kaufman

A. S.

Kaufman

N. L.

(1993). Kaufman Adolescent and Adult Intelligence Test. Circle Pines, MN: American Guidance Service.

33.

Kaufman

A. S.

Kaufman

N. L.

(2004). Kaufman Assessment Battery for Children–second edition. Circle Pines, MN: American Guidance Service.

34.

Keith

T. Z.

(1997). Using confirmatory factor analyis to aid in understanding the constructs measured by intelligence tests. In Flanagan

D. P.

Genshaft

J. L.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 373-402). New York: Guilford.

35.

Keith

T. Z.

(2005). Using confirmatory factor analysis to aid in understanding the constructs measured by intelligence tests. In Flanagan

D. P.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (2nd ed., pp. 581-614). New York, NY: Guilford.

36.

Keith

T. Z.

Fine

J. G.

Taub

Reynolds

M. R.

Kranzler

J. H.

(2006). Higher order, multisample, confirmatory factor analysis of the Wechsler Intelligence Scale for Children–Fourth Edition: What does it measure? School Psychology Review, 35, 108-127.

37.

Keith

T. Z.

Kranzler

J. H.

Flanagan

D. P.

(2001). What does the cognitive assessment system (CAS) measure? Conjoint confirmatory factor analysis of the CAS and the Woodcock-Johnson tests of cognitive ability (3rd ed.). School Psychology Review, 30, 89-119.

38.

Keith

T. Z.

Novak

C. G.

(1987). Joint facture structure of the WISC-R and K-ABC for referred school children. Journal of Psychoeducational Assessment, 5, 370-386.

39.

Keith

T. Z.

Reynolds

M. R.

(2010). CHC and cognitive abilities: What we’ve learned from 20 years of research. Psychology in the Schools, 47, 635-650.

40.

Keith

T. Z.

Reynolds

M. R.

(2012). Using confirmatory factor analysis to aid in understanding the constructs measured by intelligence tests. In Flanagan

D. P.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (3rd ed.) (pp. 758-799). New York, NY: Guilford.

41.

Keith

T. Z.

Witta

E. L.

(1997). Hierarchical and cross-age confirmatory factor analysis of the WISC-III: What does it measure? School Psychology Quarterly, 12, 89-107.

42.

Korkman

Kirk

Kemp

S. L.

(2007). NEPSY II. San Antonio, TX: PsychCorp.

43.

Lichtenberger

E. O.

Kaufman

A. S.

(2013). Essentials of WAIS-IV assessment, second edition. Hoboken, NJ: Wiley.

44.

McArdle

J. J.

(1994). Structural factor analysis experiments with incomplete data. Multivariate Behavioral Research, 29, 409-454.

45.

McCloskey

Perkins

L. A.

Van Divner

(2009). Assessment and intervention for executive function difficulties. New York, NY: Routledge.

46.

McCloskey

Whitaker

Murphy

Rogers

(2012) Intellectual, cognitive, and neuro-psychological assessment in three-tier service delivery system in schools. In Flanagan

D. P.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (3rd ed., pp. 852-882). New York, NY: Guilford Press.

47.

McGhee

(1993). Fluid and crystallized intelligence: Confirmatory factor analysis of the Differential Abilities Scale, Detroit Tests of Learning Aptitudes 3, and Woodcock - Johnson Psycho-Educational battery - Revised. Journal of Psychoeducational Assessment [Monograph Series: Woodcock Johnson PsychoEducational Bettery - Revised.], 20-38.

48.

McGrew

K. S.

(1993). The relationship between the WJ-R Gf-Gc cognitive clusters and reading achievement across the lifespan. Journal of Psychoeducational Assessment, Monograph Series: WJ-R Monograph, 39-53.

49.

McGrew

K. S.

(1997). Analysis of the major intelligence batteries according to a proposed comprehensive Gf-Gc framework. In Flanagan

D. P.

Genshaft

J. L.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 151-179). New York: Guilford.

50.

McGrew

K. S.

Flanagan

D. P.

(1998). The intelligence test desk reference (ITDR): Gf-Gc cross-battery assessment. Boston, MA: Allyn & Bacon.

51.

McGrew

K. S.

Flanagan

D. P.

Keith

T. Z.

Vanderwood

(1997). Beyond g: The impact of Gf-Gc specific cognitive abilities research on the future use and interpretation of intelligence tests in the schools. School Psychology Review, 26, 177-189.

52.

McGrew

K. S.

Wendling

B. J.

(2010). Cattell-Horn-Carroll cognitive-achievement relations: What we have learned from the past 20 years of research. Psychology in the Schools, 47, 651-675.

53.

Meredith

(1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525-543.

54.

Miller

D. C.

(2013). Essentials of school neuropsychological assessment (2nd ed.). Hoboken, NJ: John Wiley.

55.

Mulaik

S. A.

(2010). Foundations of factor analysis—second edition. Boca Raton, FL: Taylor & Francis.

56.

Naglieri

J. A.

Das

J .P.

(1997). Cognitive Assessment System. Itasca, IL: Riverside Publishing.

57.

Naglieri

J. A.

Wechsler

(2009). Wechsler Nonverbal Scale of Ability. San Antonio, TX: Pearson.

58.

Nesselroade

J. R.

Thompson

W. W.

(1995). Selection and related threats to group comparisons: An example comparing factorial structures of higher and lower ability groups. Psychological Bulletin, 117, 271-284.

59.

Niileksela

C. R.

Reynolds

M. R.

Kaufman

A. S.

(2012). An alternative Cattell-Horn-Carroll (CHC) factor structure of the WAIS-IV: Age invariance of an alternative model for ages 70-90. Manuscript submitted for publication.

60.

O’Grady

K. E.

(1983). A confirmatory maximum likelihood factor analysis of the WAIS-R. Journal of Consulting and Clinical Psychology, 51, 826-831.

61.

Pearson

N. C. S.

(2009). Wechsler Individual Achievement Test–Third Edition. San Antonio, TX: Author.

62.

Pearson

N. C. S.

(2009). Wechsler Memory Scale–Fourth Edition. San Antonio, TX: Author.

63.

Phelps

McGrew

K. S.

Knopik

S. N.

Ford

(2005). The general (g), broad, and narrow CHC stratum characteristics of the WJ III and WISC-IV tests: A confirmatory cross-battery investigation. School Psychology Quarterly, 20, 66-88.

64.

Reynolds

M. R.

Keith

T. Z.

Flanagan

D. P.

Alfonso

V. C.

(in press). A cross-battery, reference variable, confirmatory factor analytic investigation of the CHC taxonomy. Journal of School Psychology.

65.

Roid

G. H.

(2003). Stanford Binet Intelligence Scales–fifth edition. Itasca, IL: Riverside.

66.

Sanders

McIntosh

D. E.

Dunham

Rothlisberg

B. A.

Finch

(2007). Joint confirmatory factor analysis of the Differential Ability Scales and the Woodcock-Johnson Tests of Cognitive Abilities - Third Edition. Psychology in the Schools, 44, 119-138.

67.

Sattler

(2008). Assessment of children: Cognitive foundations (5th ed.). San Diego, CA: Author.

68.

Silverstein

A. B.

(1982). Factor structure of the Wechsler Adult Intelligence Scale–revised. Journal of Consulting and Clinical Psychology, 50, 661-664.

69.

Stone

B. J.

(1992). Joint confirmatory analyses of the DAS and WISC-R. Journal of School Psychology, 30, 185-195.

70.

Taub

G. E.

McGrew

K. S.

Witta

E. L.

(2004). A confirmatory analysis of the factor structure and cross-age invariance of the Wechsler Adult Intelligence Scale–Third Edition. Psychological Assessment, 16, 85-89.

71.

Thurstone

L. L.

(1947). Multiple-factor analysis: A development and expansion of the vectors of the mind. Chicago, IL: University of Chicago Press.

72.

Tupa

D. J.

O’Dougherty Wright

Fristad

M. A.

(1997). Confirmatory factor analysis of the WISC-III with child psychiatric inpatients. Psychological Assessment, 9, 302-306.

73.

Tusing

M. E.

Ford

(2004). Examining preschool cognitive abilities using a CHC framework. International Journal of Testing, 4, 91-114.

74.

Valencia

R. R.

Rankin

R. J.

Oakland

(1997). WISC-R factor structures among White, Mexican American, and African American children: A research note. Psychology in the Schools, 34, 11-16.

75.

Vanderwood

M. L.

McGrew

K. S.

Flanagan

D. P.

Kieth

T. Z.

(2002). The contribution of general and specific cognitive abilities to reading achievement. Le arning and Individual Differences, 13, 159-188.

76.

Waller

N. G.

Waldman

I. D.

(1990). A reexamination of the WAIS-R factor structure. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 2, 139-144.

77.

Ward

L. C.

Ryan

J. J.

Axelrod

B. N.

(2000). Confirmatory factor analyses of the WAIS-III standardization data. Psychological Assessment, 12, 341-345.

78.

Ward

L. C.

Bergman

M. A.

Hebert

K. R.

(2012). WAIS-IV subtest covariance structure: Conceptual and statistical considerations. Psychological Assessment, 24, 328-340.

79.

Watkins

M. W.

(2006). Orthogonal higher order structure of the Wechsler Intelligence Scale for children–Fourth Edition. Psychological Assessment, 18, 123-125.

80.

Watkins

M. W.

(2010). Structure of the Wechsler Intelligence Scale for children–Fourth Edition among a national sample of referred students. Psychological Assessment, 22, 782-787.

81.

Watkins

M. W.

Wilson

S. M.

Kotz

K. M.

Carbone

M. C.

Babula

(2006). Factor structure of the Wechsler Intelligence Scale for children–Fourth Edition among referred students. Educational and Psychological Measurement, 66, 975-983.

82.

Wechsler

(1981). Wechsler Adult Intelligence Scale–revised. San Antontio, TX: The Psychological Corporation.

83.

Wechsler

(1991). Wechsler Intelligence Scale for Children–Third Edition. San Antonio, TX: The Psychological Corporation.

84.

Wechsler

(1997). Wechsler Adult Intelligence Scale–Third Edition. San Antonio, TX: The Psychological Corporation.

85.

Wechsler

(2002). Wechsler Preschool and Primary Scale of Intelligence–Third Edition. San Antonio, TX: The Psychological Corporation.

86.

Wechsler

(2003). Wechsler Intelligence Scale for Children–Fourth Edition. San Antonio, TX: The Psychological Corporation.

87.

Wechsler

(2008). Wechsler Adult Intelligence Scale–Fourth Edition. San Antonio, TX: Pearson.

88.

Wechsler

(2003). Wechsler Intelligence Scale for Children–fourth edition technical and interpretive manual. San Antonio, TX: Psychological Corporation.

89.

Wechsler

(2008). Wechsler Adult Intelligence Scale–fourth edition technical and interpretive manual. San Antonio, TX: Pearson.

90.

Wechsler

Naglieri

J. A.

(2006). Wechsler Nonverbal Scale of Ability. San Antonio, TX: Harcourt Assessment.

91.

Weiss

L. G.

Keith

Zhu

Chen

(2013a). WAIS-IV and Clinical Validation of the Four- and Five-Factor Interpretative Approaches. Journal of Psychoeducational Assessment, 31, 94-113.

92.

Weiss

L. G.

Keith

Zhu

Chen

(2013b). WISC-IV and Clinical Validation of the Four- and Five-Factor Interpretative Approaches. Journal of Psychoeducational Assessment, 31, 114-131.

93.

Woodcock

R. W.

Johnson

M. B.

(1989). Woodcock-Johnson psycho-educational battery–revised. Allen, TX: DLM Teaching Resources.

94.

Woodcock

R. W.

(1990). Theoretical Foundations of the WJ-R Measures of Cognitive Ability. Journal of Psychoeducational Assessment, 8, 231-258.

95.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001, 2007). Woodcock-Johnson III normative update tests of achievement. Rolling Meadows, IL: Riverside Publishing.

96.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001, 2007). Woodcock-Johnson III Normative Updatetests of Cognitive Abilities. Itasca, IL: Riverside.

97.

Woodcock

R. W.

(2012, September). Cognitive/Educational Assessment: Past, Present, Future. Paper presented at the Richard Woodcock Institute for Advancement of Contemporary Cognitive Assessment, Tufts University, Medford, MA.