Investigation of the Factor Structure of the Comprehensive Test of Nonverbal Intelligence–Second Edition (CTONI-2) Using Exploratory Factor Analysis

Abstract

The present study examined the structure of the Comprehensive Test of Nonverbal Intelligence–Second Edition (CTONI-2) normative sample using exploratory factor analysis, multiple factor extraction criteria, and higher-order exploratory factor analytic techniques that were not reported in the in the CTONI-2 Examiner’s Manual. Results indicated that retention of the hypothesized CTONI-2 measurement model resulted in several theoretically inconsistent factor loadings, low factor loadings, and the retention of impermissible factors (e.g., factors with no salient loadings). Hierarchical exploratory analyses with the Schmid and Leiman procedure found that the second-order g factor accounted for large portions of total and common variance, in contrast to the two first-order factors that accounted for small portions of dimensional variance. Results suggest that the CTONI-2 provides a strong measure of general intelligence and clinical interpretation should be limited to that level of the measurement instrument.

Keywords

CTONI-2 exploratory factor analysis intelligence structural validity

The Comprehensive Test of Nonverbal Intelligence–Second Edition (CTONI-2; Hammill, Pearson, & Wiederholt, 2009a), a revision of the popular Comprehensive Test of Nonverbal Intelligence (CTONI; Hammill, Pearson, & Wiederholt, 1997), measures nonverbal reasoning abilities across the life span (ages 6-89). The CTONI-2 is composed of six subtests (Geometric Analogies, Geometric Categories, Geometric Sequences, Pictorial Analogies, Pictorial Categories, and Pictorial Sequences) that combine to form two subscale quotients (Geometric and Pictorial) as well as a higher-order full-scale composite which is thought to estimate Spearman’s (1904) general intelligence factor or g. All CTONI-2 test items are presented via matrix arrays that require examinees to point to the correct response rather than respond verbally. As a result, it is suggested that the CTONI-2 may be a more appropriate measure of intelligence for individuals with known or suspected language or motor difficulties (Hammill & Pearson, 2009). It should also be noted that despite the suggestion in the title, “nonverbal” intelligence is not a viable entity as the CTONI-2, and other related tests more accurately measure general intelligence through nonverbal means (Ortiz, Ochoa, & Dynda, 2012).

Despite the suggestion of a multifactored hierarchical assessment structure, the psychometric evidence provided in the Examiner’s Manual (Hammill, Pearson, & Wiederholt, 2009b) did not examine the actual hypothesized CTONI-2 assessment model. Examination of the internal structure of the measurement instrument was limited to subjecting the subtest intercorrelation matrix for the entire CTONI-2 normative sample (N = 2,287) to a principal components analysis (PCA) with a varimax rotation. Results suggested the retention of a single component, interpreted by the test authors as reflecting a general intelligence factor (g) that captured 98% of the variance in the manifest variables. These same procedures were also utilized to validate the original CTONI (Pearson, 2003). Despite the implication that the CTONI-2 is a one-factor test, multiple first-order factors were specified in the measurement instrument and users are encouraged to interpret individual performance along these dimensions. For instance, the Manual suggests that discrepant performance across the Pictorial and Geometric scales may be clinically noteworthy (p. 24) and base rates for observed differences are reported.

Although the aforementioned validation procedures have been referred to as a factor analysis in both the Manual and professional literature (e.g., McCallum, Bracken, & Wasserman, 2001; Pearson, 2003; Rossen, Shearer, Penfield, & Kranzler, 2005), PCA is not considered to be a factor analytic procedure due to the fact that its algorithm differs mathematically from the assumptions of the common factor model (Fabrigar & Wegener, 2012, Gorsuch, 1983). Although it has been argued that there are negligible differences between most principal components and factor analysis solutions (e.g., Fabrigar, Wegener, MacCallum, & Strahan, 1999; Velicer & Jackson, 1990), components analysis is computed without regard for the influence of latent variables and does not discriminate between different dimensions of variance in the manifest variables (e.g., shared and unique variance). As a result, the components derived from PCA should not be interpreted as reflecting latent dimensions such as the general factor (Bentler & Kano, 1990; Jensen, 1998; Preacher & MacCallum, 2003; Widaman, 1993).

Furthermore, the strength of the “general factor” loadings reported in the Manual (p. 76) may be an artifact of method variance as the authors utilized a varimax rotation to further orthogonalize the components derived from their PCA analysis. Components analysis is a data reduction procedure that results in components that are orthogonal to each other thus, an additional orthogonal rotation of the derived components can arbitrarily constrain the ability of subsequent components to account for meaningful variance that can result in a loss of valuable data (Costello & Osborne, 2005). In addition, when components are uncorrelated, it can inflate the values of variance accounted for by the retained components (Gorsuch, 1997; McArdle, 1990).

Also missing from the Manual were proportions of variance accounted for by the higher-order factor (g) and the two first-order factors, higher-order subtest loadings, and subtest specificity estimates. This information is vital given the fact that all cognitive measures are composed of reliable variance that is attributable to a higher-order general factor, reliable variance that is attributable to first-order group factors, and error variance (Carroll, 1993, 1995). In the absence of the variance decomposition procedures described above, clinicians do not have the information necessary for determining the relative importance of the CTONI-2 factor and subtest scores relative to the full-scale IQ. To date, the CTONI-2 has yet to be examined using higher-order variance partitioning procedures such as these.

Despite these limitations, additional examinations of the structural validity of the measurement instrument within the technical literature have been scarce. In the intervening period, since the publication of the original CTONI, published validity studies have largely been limited to examining concurrent and predictive relationships between CTONI/CTONI-2 variables and external measures (e.g., Beaujean et al., 2011; Lassiter, Harrison, Matthews, & Bell, 2001; Lassiter, Matthews, & Feeback, 2007; Rossen et al., 2005). Despite the information provided by these researchers, these procedures are inadequate for validating the internal structure of a measurement instrument (Clark & Watson, 1995; Cronbach & Meehl, 1955). As a result, the underlying factor structure of the CTONI-2 is presently unknown.

Purpose of Current Study

To address these gaps in the literature, the present study examined the latent structure of the CTONI-2 using exploratory factor analysis (EFA) to examine the tenability of the proposed two-factor measurement model as well as the proportions of CTONI-2 subtest variance attributed to the higher-order general dimension and to the two first-order dimensions using the hierarchical exploratory factor analytic techniques described by Carroll (1993, 1995). Given the fact that the CTONI-2 measurement model was not designed to comport with any particular theory of cognitive abilities (Hammill et al., 2009b, p. 2), these procedures are appropriate as a first step in validating and generating hypotheses about the structure of the measurement instrument (Gorsuch, 2003; Kellow, 2005). Although these higher-order procedures have been utilized extensively to evaluate the technical properties of many traditional cognitive assessment measures (see Canivez, 2013, for a review), their application to nonverbal measures such as the CTONI-2 have been less frequent. It is believed that the results from the current study will provide important information for the dissemination of empirically supported interpretive strategies for the CTONI-2.

Method

Participants and Measurement Instrument

The CTONI-2 was standardized on a nationally representative sample of 2,827 participant’s ages 6 to 89 years closely approximating the 2005 U.S. Census estimates for geographic region, gender, race, and parent education level. The CTONI-2 has six subtests (M = 10, SD = 3) that form two first-order factor scales (M = 100, SD = 15): Pictorial Quotient and Geometric Quotient. In addition, the sum of scores from the CTONI-2 subtests combine to form a higher-order full-scale nonverbal IQ (NIQ) composite (M = 100, SD = 15). Average internal consistency estimates range from .83 to .87 for the subtests and from .90 to .95 for the composite scores. Additional psychometric information is provided for the CTONI-2 in the Manual (Hammill et al., 2009b) and an independent test review is available (Delan, Kaya, & Ritter, 2012).

Data Analyses

The intercorrelation matrix of the six CTONI-2 subtests for the entire normative sample (Hammill et al., 2009b, p. 71) was subjected to principal axis EFA) using higher-order validation procedures endorsed by Carroll (1995) using SPSS version 21 for Windows. As recommended by Gorsuch (1983), multiple criteria for determining the number of factors to retain were examined. These procedures included eigenvalues >1 (Guttman, 1954), the visual scree test (Cattell, 1966), standard error of the scree ( $S E_{S c r e e}$ ; Zoski & Jurs, 1996), Horn’s parallel analysis (HPA; Horn, 1965), and minimum average partials (MAP; Velicer, 1976). While the scree test was used to visually determine the optimum number of factors to retain, it is a subjective methodology. As a consequence, it has been recommended (e.g., Ho, 2014; Zwick & Velicer, 1986) that researchers utilize more robust quantitative methods as criteria for factor extraction in EFA. In a recent simulation study, Raiche, Walls, Magis, Riopel, and Blais (2012) found the $S E_{S c r e e}$ to be the most accurate scree method and was used as programmed by Watkins (2007). As recommended by Frazier and Youngstrom (2007), HPA and MAP were also included as they potentially protect against the threat of overfactoring in EFA. HPA indicated factors were meaningful when eigenvalues from the CTONI-2 norming sample data were larger than eigenvalues produced by random data simulations specified to contain the same number of participants and factors produced from the norming data. Random data for HPA analyses were generated using the Monte Carlo PCA for Parallel Analysis program (Watkins, 2000) with 100 replications to produce stable estimates. MAP procedures were conducted using O’Connor’s (2000) SPSS syntax program.

For higher-order exploratory analyses, the current study limited iterations in first-order principal axis factor extraction to two in estimating final communality estimates. According to Gorsuch (2003), limiting iterations to two provides an optimal balance between sampling and measurement error in estimating communality, providing stable parameter estimates while protecting against the problem of capitalizing on chance estimation of parameters (MacCallum, Roznowski, & Necowitz, 1992). The correlation matrix for the CTONI-2 normative sample was subjected to a first-run EFA (principal axis extraction of two factors), followed by a promax (oblique) rotation (k = 4; Gorsuch, 2003). Because an indirect hierarchical measurement model was implied for the CTONI-2, higher-order factor analysis using the Schmid–Leiman (1957) procedure was applied to oblique first-order factors to elucidate the structure of the CTONI-2 using the MacOrtho program by Watkins (2004). This procedure allows for the extraction of a second-order factor from a first-order factor correlation matrix. According to Schmid and Leiman (1957), this transforms “an oblique solution containing a hierarchy of higher-order factors into an orthogonal solution which not only preserves the desired interpretation characteristics of the oblique solution, but also discloses the hierarchical structuring of the variables” (p. 53).

Also missing from the Manual were estimates for model-based reliabilities that more accurately reflect the multidimensional nature of the first-order factor scores specified on the measurement instrument (i.e., Carroll, 1993, 1995). In psychometrics, it is common for reliability to be estimated by conventional coefficients such as alpha. However, Chen, Hayes, Carver, Laurenceau, and Zhang (2012) noted that “for multidimensional constructs, the alpha coefficient is complexly determined, and McDonald’s (1999) omega–hierarchical (ω_h) provides a better estimate for the composite score and thus should be used” (p. 228). ω_h is the model-based reliability estimate for the hierarchical general intelligence factor independent of the variance of group factors. Omega subscale (ω_s) is the model-based reliability estimate of a group factor with all other group and general factors removed (Reise, 2012). Omega estimates (ω_h and ω_s) may be obtained from EFA Schmid–Leiman solutions and were produced using the Omega program (Watkins, 2013). For dimensional variance to be considered sufficiently reliable, Omega coefficients should exceed .50 (Reise, 2012; Reise, Bonifay, & Haviland, 2013).

Results

Factor-Extraction Criteria

Parallel analysis (Horn, 1965) and the MAP (Velicer, 1976) criterion suggested that one factor be retained for the CTONI-2 normative sample. In addition, a visual scree test (Figure 1) indicated evidence for one strong general factor. In sum, none of the extraction tests supported the retention of two factors, as suggested by the CTONI-2 measurement model with the exception of publisher theory/conjecture. Despite factor extraction criteria suggesting extraction of only one factor, two factors were extracted to comport with the proposed test structure indicated in the Manual. Similar procedures have been utilized in higher-order examinations of other cognitive measures (e.g., Dombrowski, Watkins, & Brogan, 2009; Nelson, Canivez, Lindstrom, & Hatt, 2007) in which extraction tests did not support the retention of additional first-order factors. It is believed that these methods provide for a direct examination of publisher theory with results that are potentially relevant for clinicians who utilize the CTONI-2 in practice.

Figure 1.

Scree plot of parallel analysis for Comprehensive Test of Nonverbal Intelligence–Second Edition (CTONI-2) subtests (ages 6-89).

First-Order EFA: Two-Factor Extraction

First-run EFA results produced a Kaiser–Meyer–Olkin Measure of Sampling Adequacy coefficient of .854 and Bartlett’s Test of Sphericity χ²(15) = 4,480.58, p < .001. Communality estimates ranged from .312 (Pictorial Categories, Geometric Categories) to .529 (Geometric Sequences). Based on these obtained values, it was determined that EFA procedures were appropriate (Tabachnick & Fiddell, 2007). Obtained pattern and structural coefficients are provided in Table 1. Consistent with previous research (e.g., Gorsuch, 1997; McArdle, 1990), the first eigenvalue accounted for significantly less variance in the CTONI-2 variables (50.61%) when compared with the explained variance attributed to the first component reported in the CTONI-2 Manual. Whereas the subtest loading patterns permitted the identification of first-order Geometric and Pictorial factors, Geometric Analogies cross-loaded on both latent dimensions and Pictorial Sequences migrated from the Pictorial factor and was associated with a theoretically inconsistent factor. Correlations between the first-order factors was .83 based on the promax rotation (k = 4), and indicated the presence of a higher-order dimension.

Table 1.

Factor Pattern and Structure Coefficients From Principal Axis Extraction of Two CTONI-2 Factors With Promax Rotation (k = 4) and Factor Correlations.

Subtest	g ^a	Factor pattern coefficients factor structure coefficients
Subtest	g ^a	Geometric	Pictorial	Geometric	Pictorial
Geometric sequences	.709	.682	.054	.727	.621
Geometric categories	.548	.499	.069	.557	.485
Geometric analogies	.696	.308	.420	.658	.677
Pictorial sequences	.677	.556	.147	.678	.610
Pictorial categories	.545	.118	.456	.498	.554
Pictorial analogies	.635	.035	.638	.566	.667
Eigenvalue		3.04	0.73
Variance (percent)		50.61	12.20
Factor correlation matrix
Geometric		1.00
Pictorial		.83	1.00

Note. Salient factor pattern loadings (e.g., ≥.30) are denoted in bold. Factor pattern coefficients in bold italics denote salient loading on theoretically inconsistent factor. g = general intelligence; CTONI-2 = Comprehensive Test of Nonverbal Intelligence–Second Edition.

Factor structure coefficients from first unrotated factor (g-loadings) are correlations between subtest and general factor (Jensen, 1980).

Schmid–Leiman Higher-Order Analysis: Two-Factor Solution

Higher-order analyses using the Schmid–Leiman (1957) procedure are presented in Table 2. The second-order g factor accounted for 37% of the total variance and 87.1% of the common variance. The general factor also accounted for 27% to 45% (Mdn = 39%) of the individual subtest variability. At the first-order level, Geometric Ability accounted for an additional 3.2% of the total variance and 7.5% of the common variance, and Pictorial Ability accounted for an additional 2.3% of the total variance and 5.4% of the common variance. The first- and second-order factors combined to measure 44.3% of the variance in CTONI-2 scores resulting in 55.7% unique variance (combination of specific and error variance). Subtest specificity (reliable variance unique to the individual measures) ranged from .33 to .56.

Table 2.

Sources of Variance in the Comprehensive Test of Nonverbal Intelligence–Second Edition (CTONI-2) for Ages 6 to 89 (N = 2,287) According to an Orthogonalized (Schmid & Leiman, 1957) Higher-Order Factor Model.

Subtest	General		Geometric^a		Pictorial^a		h²	u²
Subtest	b	Variance	b	Variance	b	Variance	h²	u²
Geometric sequences	.671	.450	.280	.078	.022	.000	.529	.471
Geometric categories	.518	.268	.205	.042	.028	.001	.311	.689
Geometric analogies	.664	.441	.126	.016	.172	.030	.486	.514
Pictorial sequences	.641	.411	.228	.052	.060	.004	.466	.534
Pictorial analogies	.614	.377	.014	.000	.262	.069	.446	.554
Pictorial categories	.523	.274	.048	.002	.187	.035	.311	.689
Percent total variance	37.0		3.2		2.3		44.3	55.7
Percent common variance	87.1		7.5		5.4		100.0
	ω_h = .796		ω_s = .068		ω_s = .049

Note. b = standardized loading of subtest on factor; Var = variance (b²) explained in the subtest; h² = communality; u² = uniqueness. Bold denotes theoretically consistent first-order factor loadings. Bold italics denote nontrivial loading on theoretically inconsistent factor. ω_h = Omega hierarchical; ω_s = Omega subscale.

Factor is impermissible due to nonsalient subtest loadings.

Model-Based Reliability Estimates

ω_h and ω_s coefficients were estimated based on the Schmid–Leiman results in Table 2. To examine latent factor reliability of the theoretically based scores, Picture Sequences was included in the Pictorial factor for estimation of ω_s. The ω_h coefficient for general ability (.796) was high and sufficient for scale interpretation. However, the ω_s coefficients for the two implied CTONI-2 group factors (Geometric and Pictorial) were considerably lower (.068 and .049, respectively), each possessing too little true score variance for confidant clinical interpretation (Reise, 2012; Reise et al., 2013).

Discussion

The recently revised joint standards for educational and psychological testing (American Educational Research Association [AERA], American Psychological Association [APA], & The National Council on Measurement in Education [NCME], 2014) stipulate that the dimensions purported to be measured by a psychological test must be supported with appropriate psychometric evidence (e.g., reliability and validity studies). Although the CTONI-2 is designed to measure two first-order factors, its hypothesized multilevel structure was not examined in the Manual. Given the absence of relevant construct validity information available in the CTONI-2 Manual and in the available technical literature, additional information is needed to support the clinical interpretation procedures advocated for the CTONI-2 in applied practice (e.g., Hammill & Pearson, 2009). Accordingly, the purpose of the present study was to examine the structural validity of the CTONI-2 using exploratory factor analytic procedures that take into account the higher-order structure of the measurement instrument. The present study sought to clarify the structural validity of the CTONI-2 scores and to apportion subtest variance correctly according to higher-order and lower-order dimensions (e.g., Carroll, 1995). It is believed that the results from these analyses will better inform interpretive practice for the measurement instrument.

Inspection of oblique pattern coefficients for the first-order factors (Table 1) indicated that retention of two factors resulted in subtest migration (subtest that loads higher on a theoretically inconsistent factor) and cross-loading (subtest with salient loadings on both latent dimensions). In considering these results, it is worth noting that in higher-order studies of the Reynolds Intellectual Assessment Scale (RIAS; Dombrowski et al., 2009; Nelson et al., 2007), where researchers proceeded to examine publisher theory despite lack of empirical support from extraction tests, no theoretically inconsistent factor loadings were observed. That is, those constructs were at least supported by oblique subtest loadings that aligned with theoretically consistent first-order factors. This was not observed for the CTONI-2, indicating that evidence for the viability of the Pictorial and Geometric factors was questionable prior to commencing with appropriate higher-order analysis (e.g., Carroll, 1995).

When applied to the CTONI-2 normative sample, the Schmid–Leiman (1957) transformation demonstrated that the reliable variance for each CTONI-2 subtest can be sourced to multiple dimensions. The most important of these dimensions was general intelligence which accounted for substantially more variance in the six CTONI-2 subtests than any of the orthogonal first-order factors. In addition, the unique component (a mixture of specific and error variance) was extensive in all subtests ranging from 47% (Geometric Sequences) to 69% (Geometric Categories, Pictorial Categories) of explained variance. The combined effects of general intelligence and uniqueness overshadowed the influence of the two hypothesized CTONI-2 first-order factors. The first-order factors accounted for 5.4% (Pictorial) to 7.5% (Geometric) of the common variance and 2.3% (Pictorial) to 3.2% (Geometric) of the total variance. Whereas the ω_h coefficient for general intelligence (.796) was high and sufficient for scale interpretation, ω_s coefficients for the Geometric (.068) and Pictorial (.049) factors were extraordinarily small, indicating that interpretation beyond the NIQ is hopelessly flawed (Reise, 2012; Reise et al., 2013).

On balance, the CTONI-2 appears to provide a relatively strong estimate of general cognitive ability (g). Nevertheless, predictive relationships between the NIQ and criterion achievement measures in the Manual were noticeably weaker when compared with similar estimates from more diverse intelligence tests such as the Wechsler Scales. DeThorne and Schaefer (2004), suggest that this attenuation is not unique to the CTONI-2 and is the product of the fact that traditional verbal tasks (e.g., Vocabulary), that contain large portions of g variance, are omitted from nonverbal measures.

In addition, it appears that the CTONI-2, similar to many tests of cognitive abilities, may overestimate the number of factors when disregarding traditional validation procedures such as EFA and more restrictive extraction criteria such as HPA and MAP (Canivez, 2013; Frazier & Youngstrom, 2007). Inspection of the Schmid–Leiman results in Table 2 reveal that although Pictorial Analogies was aligned (b ≥ .20; Carroll, 1993) with the Pictorial factor, all Pictorial subtests failed to load saliently (e.g., ≥.30) on that latent dimension once the reliable variance attributable to the general factor was partialed out. Although it may be argued that the results from the current study do nothing but confirm the PCA analysis that was reported in the CTONI-2 Manual, these results did not appear to guide the development of the measurement instrument as additional first-order factors were specified by the test authors. As a result, the present analysis is the first direct examination of the specified structure of the CTONI-2 and is one of the few studies to decompose the subtest variance of a nontraditional cognitive assessment measure using recommended higher-order procedures (e.g., Carroll, 1995). These results suggest that if assessment and interpretation of first-order dimensions is of critical importance, the CTONI-2 test authors will likely need to increase the number of subtests estimating those dimensions to increase the amount of variance at that level of measurement (Canivez, 2011).

Limitations and Future Directions

The present study is the first to examine the internal structure of the CTONI series using appropriate factor analytic techniques (e.g., Bentler & Kano, 1990; Costello & Osborne, 2005; Fabrigar & Wegener, 2012). Nevertheless, the present study is not without limitations that should be taken into consideration when interpreting these results. Most notably, the results are derived from a single intercorrelation matrix representing the entire CTONI-2 normative sample spanning an expansive age range. Although this is the only matrix provided in the CTONI-2 Manual, additional examination of the construct validity of the measurement instrument at different points of the age span would be beneficial.¹ In addition, the CTONI-2 Manual refers to the Geometric and Pictorial dimensions as “contexts in which the abilities are measured” (Hammill et al., 2009b, p. 3), and unique aspects of the CTONI-2 subtests (e.g., analogies, categories, and sequences) as reflecting higher-order cognitive abilities. Therefore, it may be useful to examine the viability of competing measurement models along these lines using confirmatory factor analytic (CFA) techniques. Such an examination would permit clinicians to be able to consider the convergence or divergence of CFA and EFA results for the measurement instrument.

Although EFA and CFA are considered to be complimentary procedures, Gorsuch (1983) noted that they provide answers to different empirical questions and that when the results from these procedures are in agreement, greater confidence can be placed in the internal structure of a test. Related investigations (e.g., Lakin, 2012; Lakin & Gambrell, 2012) of other pictorial-based cognitive measures suggest that these latent dimensions may be plausible alternatives to those specified in the CTONI-2. Similarly, CFA procedures may be instructive in uncovering the latent dimensions sampled by CTONI-2 measures.

Although the present results suggest that a strong higher-order dimension akin to g dominates CTONI-2 measures, a question regarding what this dimension actually represents remains unanswered. Whereas the Manual reports moderate to strong correlations between the NIQ and several IQ scores from more comprehensive measures of intelligence, results from applied studies have been less consistent. Whereas Lassiter et al. (2001) found that the NIQ for the CTONI correlated most strongly with a measure of Fluid Reasoning (Gf) on the Kaufman Adolescent and Adult Intelligence Test, a similar investigation (Lassiter et al., 2007) found equivalent correlations between the NIQ and measures of g and Gf on the Woodcock Johnson III. Similar research has yet to be conducted on the CTONI-2 and would be instructive for determining whether or not the NIQ is best interpreted as a measure of g or Gf. While distinguishing the unique effects of Gf from g may be difficult as isomorphism between these two latent dimensions has commonly been reported within the psychometric literature (e.g., Weiss, Keith, Zhu, & Chen, 2013a, 2013b), Golay, Reverte, Rossier, Favez, and Lecerf (2013) were able to disentangle these two constructs on the French version of the Wechsler Scale using Bayesian structural equation modeling.

Conclusion

Results from the current study do not support the test structure reported in the CTONI-2 Manual. Whereas strong support was found for the structural validity of the NIQ, retention of additional first-order factors was not supported. As a consequence, general intelligence accounts for the vast majority of common variance in the CTONI-2; thus, interpretation beyond the NIQ is not advised. Although the CTONI-2 Manual encourages users to interpret individual performance across the Geometric and Pictorial dimensions, these results suggest that interpretation of those constructs may result in overinterpetation of the measurement instrument (Glutting, Watkins, Konold, & McDermott, 2006) and that these measures should be more appropriately conceptualized as “pseudo-factors” and should not be used for diagnostic decision making at the present time.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

American Educational Research Association, American Psychological Association, & National Council on Measurement on Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Beaujean

A. A.

Firmin

M. W.

Attai

Johnson

C. B.

Firmin

R. L.

Mena

K. E.

(2011). Using personality and cognitive ability to predict academic achievement in a young adult sample. Personality and Individual Differences, 51, 709-714. doi:10.1016/j.paid.2011.06.023

Bentler

P. M.

Kano

(1990). On the equivalence of factors and components. Multivariate Behavioral Research, 25, 67-74. doi:10.1207/s15327906mbr2501_8

Canivez

G. L.

(2011). Hierarchical factor structure of the cognitive assessment system: Variance partitions from the Schmid-Leiman (1957) procedure. School Psychology Quarterly, 26, 305-317. doi:10.1037/a0025973

Canivez

G. L.

(2013). Psychometric versus actuarial interpretation of intelligence and related aptitude batteries. In Saklofske

D. H.

Reynolds

C. R.

Schwean

V. L.

(Eds.), The oxford handbook of child psychological assessment (pp. 84-112). New York, NY: Oxford University Press.

Carroll

J. B.

(1993). Human cognitive abilities: A survey of factor-analytic studies. New York, NY: Cambridge University Press.

Carroll

J. B.

(1995). On methodology in the study of cognitive abilities. Multivariate Behavioral Research, 30, 429-452. doi:10.1207/s15327906mbr3003_6

Cattell

R. B.

(1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245-276. doi:10.1207/s15327906mbr0102_10

Chen

F. F.

Hayes

Carver

C. S.

Laurenceau

J.-P.

Zhang

(2012). Modeling general and specific variance in multifaceted constructs: A comparison of the bifactor model to other approaches. Journal of Personality, 80, 219-251. doi:10.1111/j.1467–6494.2011.00739.x

10.

Clark

L. A.

Watson

(1995). Constructing validity: Basic issues in Objective Scale Development. Psychological Assessment, 7, 309-319. doi:10.1037/1040-3590.7.3.309

11.

Costello

A. B.

Osborne

J. W.

(2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1-9. Retrieved from http://pareonline.net/pdf/v10n7.pdf

12.

Cronbach

L. J.

Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. doi:10.1037/h0040957

13.

Delan

Kaya

Ritter

N. L.

(2012). Test review: Comprehensive Test of Nonverbal Intelligence-Second Edition (CTONI-2). Journal of Psychoeducational Assessment, 30, 209-213. doi:10.1177/0734282911415614

14.

DeThorne

L. S.

Schaefer

B. A.

(2004). A guide to child nonverbal IQ measures. American Journal of Speech-Language Pathology, 13, 275-290. doi:10.1044/1058-0360(2004/029)

15.

Dombrowski

S. C.

Watkins

M. W.

Brogan

M. J.

(2009). An exploratory investigation of the factor structure of the Reynolds Intellectual Assessment Scales (RIAS). Journal of Psychoeducational Assessment, 27, 497-507. doi:10.1177/0734282909333179

16.

Fabrigar

L. R.

Wegener

D. T.

(2012). Exploratory factor analysis. New York, NY: Oxford University Press.

17.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. doi:10.1037/1082-989X.4.3.272

18.

Frazier

T. W.

Youngstrom

E. A.

(2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring? Intelligence, 35, 169-182. doi:10.1016/j.intell.2006.07.002

19.

Glutting

J. J.

Watkins

M. W.

Konold

T. R.

McDermott

P. A.

(2006). Distinctions without a difference: The utility of observed versus latent factors from the WISC-IV in estimating reading and math achievement on the WIAT-II. Journal of Special Education, 40, 103-114. doi:10.1177/00224669060400020101

20.

Golay

Reverte

Rossier

Favez

Lecerf

(2013). Further insights on the French WISC-IV factor structure through Bayesian structural equation modeling. Psychological Assessment, 25, 496-508. doi:10.1037/a0030676

21.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

22.

Gorsuch

R. L.

(1997). Exploratory factor analysis: Its role in item analysis. Journal of Personality Assessment, 68, 532-560. doi:10.1207/s15327752jpa6803_5

23.

Gorsuch

R. L.

(2003). Factor analysis. In Schinka

J. A.

Velicer

F. F.

(Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (pp. 143-164). Hoboken, NJ: John Wiley.

24.

Guttman

(1954). Some necessary and sufficient conditions for common factor analysis. Psychometrika, 19, 149-161. doi:10.1007/BF02289162

25.

Hammill

D. D.

Pearson

N. A.

(2009). Comprehensive Test of Nonverbal Intelligence-Second Edition. In Naglieri

J. A.

Goldstein

(Eds.), Practitioner’s guide to assessing intelligence and achievement (pp. 233-264). Hoboken, NJ: John Wiley.

26.

Hammill

D. D.

Pearson

N. A.

Wiederholt

J. L.

(1997). Comprehensive Test of Nonverbal Intelligence. Austin, TX: Pro-Ed.

27.

Hammill

D. D.

Pearson

N. A.

Wiederholt

J. L.

(2009a). Comprehensive Test of Nonverbal Intelligence (2nd ed.). Austin, TX: Pro-Ed.

28.

Hammill

D. D.

Pearson

N. A.

Wiederholt

J. L.

(2009b). Comprehensive Test of Nonverbal Intelligence examiner’s manual (2nd ed.). Austin, TX: Pro-Ed.

29.

(2014). Handbook of univariate and multivariate data analysis with IBM SPSS (2nd ed.). Boca Raton, FL: CRC Press.

30.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychomerika, 30, 179-185. doi:10.1007/BF02289447

31.

Jensen

A. R.

(1980). Bias in mental testing. New York, NY: Free Press.

32.

Jensen

A. R.

(1998). The g factor: The science of mental ability. Westport, CT: Praeger.

33.

Kellow

J. T.

(2005). Exploratory factor analysis in two prominent measurement journals: Hegemony by default. Journal of Modern Applied Statistical Methods, 4, 283-287. Retrieved from http://digitalcommons.wayne.edu/cgi/viewcontent.cgi?article=1178&;context=jmasm

34.

Lakin

J. M.

(2012). Multidimensional ability tests and culturally and linguistically diverse students: Evidence of measurement invariance. Learning and Individual Differences, 22, 397-403. doi:10.1016/j.lindif.2011.12.003

35.

Lakin

J. M.

Gambrell

J. L.

(2012). Distinguishing verbal, quantitative, and figural facets of fluid intelligence n young students. Intelligence, 40, 560-570. doi:10.1016/j.intell.2012.07.005

36.

Lassiter

K. S.

Harrison

T. K.

Matthews

T. D.

Bell

N. L.

(2001). The validity of the Comprehensive Test of Nonverbal Intelligence as a measure of fluid intelligence. Assessment, 8, 95-103. doi:10.1177/107319110100800109

37.

Lassiter

K. S.

Matthews

T. D.

Feeback

(2007). An examination of the CTONI utilizing Gc-Gf theory: A comparison of the CTONI and WJ-III. Psychology in the Schools, 44, 567-577. doi:10.1002/pits.20248

38.

MacCallum

R. C.

Roznowski

Necowitz

L. B.

(1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490-504. doi:10.1037/0033-2909.111.3.490

39.

McArdle

J. J.

(1990). Principles versus principals of structural factor analysis. Multivariate Behavioral Research, 25, 81-87. doi:10.1207/s15327906mbr2501_10

40.

McCallum

Bracken

Wasserman

(2001). Essentials of nonverbal assessment. New York, NY: John Wiley.

41.

McDonald

R. P.

(1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

42.

Nelson

J. M.

Canivez

G. L.

Lindstrom

Hatt

C. V.

(2007). Higher-order exploratory analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439-456. doi:10.1016/j.jsp.2007.03.003

43.

O’Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavioral Research Methods, 32, 396-402. doi:10.3758/BF03200807

44.

Ortiz

S. O.

Ochoa

S. H.

Dynda

A. M.

(2012). Testing with culturally and linguistically diverse populations: Moving beyond the verbal-performance dichotomy into evidence-based practice. In Flangan

D. P.

Harrison

P. L.

(Eds.), Contemporary intellectual assessment: Theories, tests, and issues (3rd ed., pp. 526-552). New York, NY: Guilford Press.

45.

Pearson

(2003). Comprehensive Test of Nonverbal Intelligence. In McCallum

R. S.

(Ed.), Handbook of nonverbal assessment (pp. 141-162). New York, NY: Plenum.

46.

Preacher

K. J.

MacCallum

R. C.

(2003). Repairing Tom Swift’s electric factor analysis machine. Understanding Statistics, 2, 13-43. doi:10.1207/S15328031US0201_02

47.

Raiche

Walls

T. A.

Magis

Riopel

Blais

(2012). Non-graphical solutions for Cattell’s scree test. Methodology, 9, 23-29. doi:10.1027/1614-2241/a000051

48.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. doi:10.1080/00273171.2012.715555

49.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95, 129-140. doi:10.1080/00223891.2012.725437

50.

Rossen

E. A.

Shearer

D. K.

Penfield

R. D.

Kranzler

J. H.

(2005). Validity of the Comprehensive Test of Nonverbal Intelligence (CTONI). Journal of Psychoeducational Assessment, 23, 161-172. doi:10.1177/073428290502300205

51.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF02289209

52.

Spearman

(1904). “General intelligence,” objectively determined and measured. American Journal of Psychology, 15, 201-293. Retrieved from http://psychclassics.yorku.ca/Spearman

53.

Tabachnick

B. G.

Fiddell

L. S.

(2007). Using multivariate statistics (5th ed.). Boston, MA: Pearson.

54.

Velicer

W. F.

(1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 31, 321-327. doi:10.1007/BF02293557

55.

Velicer

W. F.

Jackson

D. N.

(1990). Component analysis versus common factor analysis: Some further observations. Multivariate Behavioral Research, 25, 89-95. doi:10.1207/s15327906mbr2501_12

56.

Watkins

M. W.

(2000). Monte Carlo PCA for parallel analysis [Computer software]. State College, PA: Ed & Psych Associates.

57.

Watkins

M. W.

(2004). MacOrtho [Computer software]. State College, PA: Ed & Psych Associates.

58.

Watkins

M. W.

(2007). SEscree [Computer software]. Phoenix, AZ: Ed & Psych Associates.

59.

Watkins

M. W.

(2013). Omega [Computer software]. Phoenix, AZ: Ed & Psych Associates.

60.

Weiss

L. G.

Keith

T. Z.

Zhu

Chen

(2013a). WAIS-IV and clinical validation of the four- and five-factor interpretative approaches. Journal of Psychoeducational Assessment, 31, 93-113. doi:10.1177/0734282913478030

61.

Weiss

L. G.

Keith

T. Z.

Zhu

Chen

(2013b). WISC-IV and clinical validation of the four- and five-factor interpretative approaches. Journal of Psychoeducational Assessment, 31, 114-131. doi:10.1177/0734282913478032

62.

Widaman

K. F.

(1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263-311. doi:10.1207/s15327906mbr2803_1

63.

Zoski

K. W.

Jurs

(1996). An objective counterpart to the visual scree test for factor analysis: The standard error scree. Educational and Psychological Measurement, 56, 443-451. doi:10.1177/0013164496056003006

64.

Zwick

W. R.

Velicer

W. F.

(1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442. doi:10.1037/0033-2909.99.3.432