The Reynolds Intellectual Assessment Scales: Measurement Invariance Across Four Language Groups

Abstract

The Reynolds Intellectual Assessment Scales (RIAS) measures general intelligence and its two main components, verbal and nonverbal intelligence, each comprising of two subtests. The RIAS has been recently standardized in Denmark, Germany, Switzerland, and Spain. Using the standardization samples of the U.S. (n = 2,438), Danish (n = 983), German (n = 2,103), and Spanish (n = 1,933) versions of the RIAS, this study examined measurement invariance across these four language groups for a single-factor structure, an oblique two-factor structure with a verbal and nonverbal factor, and a bifactor structure with a general, a verbal, and a nonverbal factor. Single-group confirmatory factor analysis (CFA) supported the oblique two-factor and bifactor structure for each language group but not the single-factor structure. The bifactor analysis revealed that the general factor accounted for the largest proportion of common variance in each language group, while the amount of variance accounted for by the two specific factors was small and their reliabilities low. Multiple-group CFA supported scalar invariance in both, the oblique two-factor and bifactor structure.

Keywords

intelligence Reynolds Intellectual Assessment Scales (RIAS)cross-cultural measurement invariance

Intelligence is one of the most frequently studied psychological constructs (e.g., Goldstein, Princiotta, & Naglieri, 2015). A more recently introduced intelligence test is the Reynolds Intellectual Assessment Scales (RIAS) developed in the United States by Reynolds and Kamphaus (2003) and normed for individuals between the ages of 3 and 90 years. Over the last decade, several studies have repeatedly demonstrated the convergent, discriminant, and predictive validity of the RIAS version (Beaujean, Firmin, Michonski, Berry, & Johnson, 2010; Edwards & Paulin, 2007; Krach, Loe, Jones, & Farrally, 2009; Nelson & Canivez, 2012). Moreover, the RIAS has been evaluated to be independent of motor coordination, visual-motor speed, and reading skills, and has been found to be user friendly in terms of its administration, scoring, and interpretation (Andrews, 2007; Dombrowski & Mrazik, 2008; Elliot, 2004). Recently, the RIAS was successfully adapted to three other languages: The Danish version (Hartmann & Andresen, 2011) standardized in Denmark, the German version (Hagmann-von Arx & Grob, 2014) standardized in Switzerland and Germany, and the Spanish version (Santamaría & Fernández Pinto, 2009) standardized in Spain.

The theoretically proposed factor structure of the RIAS is based on Carroll’s (1993) three-stratum theory. The RIAS consists of four intelligence subtests at the first Stratum and two factors—verbal intelligence index (VIX) and nonverbal intelligence index (NIX)—at the second Stratum. These factors serve as indicators of crystallized and fluid intelligence, which, in turn, are combined to a composite intelligence index (CIX) at the third Stratum reflecting general intelligence (g). There is also a composite memory index (CMX) at the second Stratum, which is composed of scores from two supplemental subtests and not integrated into the measure of general intelligence (Dombrowski, Watkins, & Brogan, 2009; Nelson & Canivez, 2012; Nelson, Canivez, Lindstrom, & Hatt, 2007). Based on the theoretical assumptions, Reynolds and Kamphaus (2003) evaluated a series of factor structures for the RIAS and posit an oblique two-factor structure with four subtests measuring two correlated factors, verbal and nonverbal intelligence. Two alternative structures may be of special interest. One is a single-factor solution with general intelligence. The other alternative is a bifactor structure with one general factor measuring general intelligence and two specific factors measuring verbal and nonverbal intelligence. These three structures are shown in Figure 1. The two memory subtests have been found to be separate from the other RIAS subtests (e.g., Nelson et al., 2007).

Figure 1.

The single-factor structure (a), two-factor structure (b), and bifactor structure (c) of the RIAS.

Assessing the factor structure of psychological tests has been found to be important in ensuring that test scores can be interpreted according to the posited test structure (e.g., Widaman & Reise, 1997). The RIAS factor structure proposed by Reynolds and Kamphaus (2003; Figure 1b) has been repeatedly supported in the United States and the three other language groups, in which the RIAS is available (i.e., Danish, German, and Spanish), using standardization samples (Hagmann-von Arx & Grob, 2014; Hartmann & Andresen, 2011; Reynolds & Kamphaus, 2003; Santamaría & Fernández Pinto, 2009). The factor structure was further supported in an independent study with a referred U.S. student sample (Beaujean, McGlaughlin, & Margulies, 2009). Studies have also found evidence for a single-factor structure (Figure 1a) for the U.S. RIAS standardization sample (Dombrowski et al., 2009) and for referred samples (Beaujean & McGlaughlin, 2014; Nelson & Canivez, 2012; Nelson et al., 2007). Combining the RIAS subtest with other tests of crystallized and fluid intelligence, Nelson and Canivez (2012) found in a clinical sample that the oblique two-factor structure was superior compared with the single-factor structure. Using the Schmid and Leiman (1957) procedure, which contains proportionality constraints, Dombrowski et al. (2009), Nelson and Canivez (2012), and Nelson et al. (2007) found also support for a higher order factor structure with general intelligence as a second-order factor, which accounted for the largest proportion of common variance.

Although the proposed RIAS structure with two main components was supported in each individual language group providing evidence for configural invariance of the RIAS (Meredith, 1993), it is unknown whether higher levels of factorial (measurement) invariance hold across the four language groups. The higher levels of measurement invariance are (e.g., Meredith, 1993; Widaman & Reise, 1997) metric invariance (invariant factor loadings across groups), scalar invariance (invariant factor loadings and intercepts across groups), and invariance of residual variances (invariant factor loadings, intercepts, and residual variances across groups). Configural invariance indicates that the factor structure is the same across groups but does not warrant that the observed variables measure the latent construct in the same way. Metric invariance implies that the observed variables are related to the latent variable equivalently across groups. This level of invariance allows a comparison of the groups in terms of path coefficients and covariances between observed and latent variables (Chen, Sousa, & West, 2005) and predictive or convergent validity from one group can be transferred to other groups (Widaman & Reise, 1997). Scalar invariance implies that group differences in observed means are due to a difference in latent means and, so, permits a comparison of the groups in terms of factor and observed means and variances (Widaman & Reise, 1997). This level of invariance allows practitioners to compare results of different language versions and assess individuals with a migration background in their native language when they have insufficient language skills in the country of resettlement (Sattler, 2001), which has become increasingly important as migration has increased worldwide in recent decades (Eurostat, 2014; U.S. Department of Homeland Security, 2013). Finally, invariance of residual variances allows a comparison of residual variances across groups (Chen et al., 2005).

The goal of the present study was to assess measurement invariance of the U.S., Danish, German, and Spanish versions of the RIAS for a single-factor structure, an oblique two-factor structure with a verbal and nonverbal factor, and a bifactor structure with a general, a verbal, and a nonverbal factor using standardized data of the standardization samples. The analysis of standardized data is desirable in situations where not identical tests are administered (e.g., Cudeck, 1989; Kim & Ferree, 1981) and is most relevant for practice, where very often standardized test scores are calculated and interpreted. Because invariance of residual variances holds rather rarely (e.g., Van De Schoot, Schmidt, De Beuckelaer, Lek, & Zondervan-Zwijnenburg, 2015), results were expected to provide evidence for scalar invariance but not invariance of residual variances.

Method

Participants

The study included a total of 7,457 individuals from the RIAS standardization samples. The U.S. standardization sample includes 2,438 individuals from the United States, the Danish standardization sample includes 983 individuals from Denmark, the German standardization sample includes 2,103 individuals from Switzerland and Germany, and the Spanish standardization sample includes 1,933 individuals from Spain. All standardization samples are equally distributed within each age group with respect to participants’ gender (50% female) and are meant to match the Census data on educational attainment (years of education completed in the United States and level of education in the other samples) of the respective countries. The U.S. sample considered also ethnicity and geographic region of the participants. The U.S. RIAS version is standardized for individuals aged 3 to 94 years (M = 24.07 years, SD = 22.42). The Danish (M = 20.83 years, SD = 19.53 years) and German (M = 19.56 years, SD = 20.33 years) versions are standardized for individuals aged 3 to 99 years (M = 19.56 years, SD = 20.33 for the German version, M = 20.83 years, SD = 19.53 for the Danish version). The Spanish RIAS version is standardized for individuals aged 3 to 94 years (M = 21.75 years, SD = 20.51). Demographic characteristics are provided in the Technical Manuals (Hagmann-von Arx & Grob, 2014; Hartmann & Andresen, 2011; Reynolds & Kamphaus, 2003; Santamaría & Fernández Pinto, 2009).

Measures

The adaptations of the RIAS for Denmark, Germany and Switzerland, and Spain were done in several steps. First, items were translated by experienced translators and individuals with psychological education. Second, all items were examined whether they are suitable for the respective language group. Items referring specifically to the American culture were modified so that they were appropriate for the culture of the respective language group (e.g., the American Newspaper machine was replaced by a machine used in European countries). In addition, the items were empirically pretested and the order of the items was adjusted to ensure an order of ascending difficulty (e.g., questions regarding baseball were easier for Americans while questions regarding soccer were easier for Europeans). Finally, the adaptations were standardized using a representative sample of the respective language group.

The RIAS yields three factors: CIX, VIX, and NIX. VIX and NIX comprise two subtests each. The verbal subtests are Guess What (GWH; identifying an object or a concept through the use of verbally presented clues) and Verbal Reasoning (VRZ; completing verbal analogies). The nonverbal subtests are Odd-Item Out (OIO; identifying a picture that does not go with the others) and What’s Missing (WHM; identifying the missing element of a presented picture). The CIX is calculated from the sum of the T scores (M = 50; SD = 10) of the four intelligence subtests. The RIAS also includes a conormed, supplemental CMX comprising two subtests, Verbal Memory (VRM; reproducing verbally presented sentences and short stories), and Nonverbal Memory (NVM; recognizing visually presented objects). In line with suggestions of others (e.g., Nelson & Canivez, 2012), the memory subtests were not included in the analysis.

Data Analysis

The software package lavaan (Rosseel, 2012) in R and maximum likelihood estimation method were used to assess measurement invariance of the RIAS across the four language groups. Correlation matrices of the subtest T scores (M = 50; SD = 10) with means and standard deviations were used as data input (see Online Supplemental Table 1), because the publisher of the U.S. RIAS version refused to make the raw data available. First, as suggested by Meade, Johnson, and Braddy (2008), a single-group confirmatory factor analysis (CFA) was conducted to test the three models presented in Figure 1 for each language group separately. It is worth noting that these models are scale-invariant allowing an appropriate analysis of standardized data (Cudeck, 1989). For the bifactor model, the explained common variance (ECV; Ten Berge & Sočan, 2004), coefficient omega hierarchical (ω_H; McDonald, 1999), which has been found to be a more appropriate measure of reliability than coefficient alpha (e.g., Watkins, 2017), and construct replicability H (Hancock & Mueller, 2001) were calculated for the general factor and the two specific factors using the software Omega¹ (Watkins, 2013). Second, a multiple-group CFA was conducted across the four language groups. Finally, the original U.S. version was compared with each of the other three language versions.

The different levels of measurement invariance were tested using procedures proposed by Meredith (1993), Milfont and Fischer (2010), Steenkamp and Baumgartner (1998), and Widaman and Reise (1997). First, configural invariance (Model 1) was tested. In all three models, the factor loading of GWH was set to 1. In the oblique two-factor model, the factor loading of OIO was also set to 1 as well. In the bifactor model, the loadings on the verbal and nonverbal factors were all set to 1. Because a bifactor model with only two indicators per factor is not identified, the variances of the verbal factor and nonverbal factor were constrained to be equal. In doing so, this model parallels the model estimated by Beaujean and McGlaughlin (2014). Finally, the mean of the general factor, verbal factor, and nonverbal factor were fixed to zero. Next, metric invariance (Model 2), scalar invariance with free latent factor means for the Danish, German, and Spanish versions (Model 3), and invariance of residual variances (Model 4) were assessed.

To assess the model fit, the following goodness-of-fit indices and criteria were used: comparative fit index (CFI) of ≥.95, McDonald’s noncentrality index (Mc) of ≥.90 and root mean square error of approximation (RMSEA) of ≤.06 (Hu & Bentler, 1999). Because of the large sample size, the chi-square test statistic was not considered (Meade et al., 2008), instead the ΔCFI and ΔMc were calculated, with values of ≤.002 for ΔCFI and ≤.005 for ΔMc indicating insignificant reduction in model fit (Meade et al., 2008).

Results

Single-Group CFA

For each language group, each factor model was tested with the four intelligence subtests as indicators. It is critical to note that the oblique two-factor model and the bifactor model are statistically equivalent (Canivez, 2016; Canivez & Watkins, 2016; Lee & Hershberger, 1990; Reise, 2012) and statistically not distinguishable from one another using single-group CFA. The fit estimates of these models are presented in Table 1. As can be seen, all language groups yielded a good fit for the oblique two-factor model and the bifactor model. For the single-factor model, the RMSEA indicated a poor fit for all four language groups.

Table 1.

Fit Indices for Single-Group Confirmatory Factor Analyses for the Four Language Groups.

Structure	df	χ²	CFI	Mc	RMSEA	90% CI RMSEA
Single-factor model
United States	2	77.113	.979	.985	.124	[.101, .149]
Danish	2	148.658	.972	.928	.273	[.237, .311]
German	2	259.697	.905	.941	.248	[.223, .273]
Spanish	2	63.336	.968	.984	.126	[.100, .154]
Two-factor/bifactor model
United States	1	0.526	1.000	1.000	<.001	[.000, .048]
Danish	1	1.048	1.000	1.000	.007	[.000, .085]
German	1	0.245	1.000	1.000	<.001	[.000, .046]
Spanish	1	2.868	.999	1.000	.031	[.000, .076]

Note. n_{United States} = 2,438, n_Danish = 983, n_German = 2,103, n_Spanish = 1,933. df = degrees of freedom; CFI = comparative fit index; Mc = McDonald’s noncentrality index; RMSEA = root mean square error of approximation; CI = confidence interval.

Using bifactor analysis, ECV, ω_H, and replicability H for the general factor and the two specific factors are given in Table 2. The estimates of the ECV show that more than 70% of the common variance is attributable to the general factor in each language group. For the two specific factors, the amount of variance accounted was small and their reliabilities low. The standardized factor loadings for the oblique two-factor and bifactor model are provided in Online Supplemental Table 2.

Table 2.

ECV, ω_H, and Replicability H.

Language group	ECV	ω_H	Replicability H
United States
g	.863	.779	.822
v	.069	.096	.150
nv	.069	.110	.150
Danish
g	.950	.950	.966
v	.025	.047	.087
nv	.025	.048	.087
German
g	.725	.708	.762
v	.139	.204	.288
nv	.136	.212	.284
Spanish
g	.810	.704	.758
v	.095	.120	.177
nv	.095	.139	.177

Note. ECV = explained common variance; ω_H = omega hierarchical; g = general intelligence; v = verbal intelligence; nv = nonverbal intelligence.

Multiple-Group CFA

Next, measurement invariance across all groups using multiple-group CFA was assessed for the oblique two-factor model and the bifactor model but not for the single-factor model due to its poor fit. Four models were tested implying configural invariance, metric invariance, scalar invariance with free latent factor means for the Danish, German, and Spanish versions, and invariance of residual variances. Fit estimates for the oblique two-factor model (Figure 1b) are provided in Table 3. As expected, Model 1 (configural invariance), Model 2 (metric invariance), and Model 3 (scalar invariance) showed a good fit. Comparing Model 1 with Model 2 and Model 2 with Model 3, the ΔCFI and ΔMc revealed no significant reduction in model fit, indicating that scalar invariance holds across all four groups. That is, the factor structure, the loadings, and the intercepts did not differ significantly across the four groups. Model 4 implying invariance of residual variances yielded a poor fit, meaning that the variances of the error terms varied significantly across groups.

Table 3.

Fit Indices for Multiple-Group Confirmatory Factor Analyses Evaluating Measurement Invariance of the Oblique Two-Factor Structure.

Model	df	χ²	CFI	Mc	RMSEA	90% CI	Δdf	Δχ²	ΔCFI	ΔMc
1. Configural invariance	4	4.687	1.000	1.000	.010	[.000, .037]	—	—	—	—
2. Metric invariance	10	22.986	.999	.999	.026	[.012, .041]	—	—	—	—
2 versus 1	—	—	—	—	—	—	6	18.299	.001	.001
3. Scalar invariance	16	23.362	.999	1.000	.016	[.000, .029]	—	—	—	—
3 versus 2	—	—	—	—	—	—	6	0.377	<.001	<.001
4. Residual invariance	28	1,601.708	.883	.900	.174	[.166, .181]	—	—	—	—
4 versus 3	—	—	—	—	—	—	12	1,579.238	.117	.100

Note. N = 7,457, n_{United States} = 2,438, n_Danish = 983, n_German = 2,103, n_Spanish = 1,933. df = degrees of freedom; CFI = comparative fit index; Mc = McDonald’s noncentrality index; RMSEA = root mean square error of approximation; CI = confidence interval.

Results of the bifactor model (Figure 1c) are given in Table 4. In contrast to single-group analysis, the fit statistics for the oblique two-factor model and the bifactor model were not identical using multiple-group analysis. The configural model, metric model, and scalar model showed a good fit, but the ΔCFI indicated a significant reduction in fit of the metric model compared with the configural model. The model implying invariance of residual variances model yielded a poor fit. Comparing the fit of the bifactor model with the fit of the oblique two-factor model, the values of the CFI, Mc, and RMSEA were all in favor of the oblique two-factor model.

Table 4.

Fit Indices for Multiple-Group Confirmatory Factor Analyses Evaluating Measurement Invariance of the Bifactor Structure.

Model	df	χ²	CFI	Mc	RMSEA	90% CI	Δdf	Δχ²	ΔCFI	ΔMc
1. Configural invariance	4	4.687	1.000	1.00	.010	[.000, .037]	—	—	—	—
2. Metric invariance	13	80.318	.995	.995	.053	[.042, .064]	—	—	—	—
2 versus 1	—	—	—	—	—	—	9	75.631	.005	.004
3. Scalar invariance	16	80.373	.995	.996	.046	[.037, .057]	—	—	—	—
3 versus 2	—	—	—	—	—	—	3	0.056	<.001	<.001
4. Residual invariance	28	1,633.823	.880	.898	.175	[.168, .183]	—	—	—	—
4 versus 3	—	—	—	—	—	—	12	1,579.238	.115	.098

Note. N = 7,457, n_{United States} = 2,438, n_Danish = 983, n_German = 2,103, n_Spanish = 1,933. df = degrees of freedom; CFI = comparative fit index; Mc = McDonald’s noncentrality index; RMSEA = root mean square error of approximation; CI = confidence interval.

The results of the comparison of the original U.S. version with each of the other language versions are shown in Table 5 for the oblique two-factor solution and in Table 6 for the bifactor solution. For the oblique two-factor structure, invariance of residual variances could be supported for the German version, indicating that the factor structure, the loadings, intercepts, and residual variances were not significantly different across the two groups. Scalar invariance was supported for the Danish and Spanish versions. For the bifactor structure, invariance of residual variances held for the German and Spanish versions, but the ΔCFI indicated a significant reduction in fit for the Spanish version between the scalar model and the model implying residual variance invariance. For the Danish version, only configural invariance hold, meaning that the U.S. version and the Danish version differed in the factor loadings, intercepts, and residual variances.

Table 5.

Fit Indices for Multiple-Group Confirmatory Factor Analyses Evaluating Measurement Invariance of the Oblique Two-Factor Structure Comparing the U.S. Group to the Danish, German, and Spanish Groups.

Model	df	χ²	CFI	Mc	RMSEA	90% CI	Δdf	Δχ²	ΔCFI	ΔMc
1. Configural invariance
U.S.-Danish	2	1.574	1.000	1.000	<.001	[.000, .045]	—	—	—	—
U.S.-German	2	0.771	1.000	1.000	<.001	[.000, .031]	—	—	—	—
U.S.-Spanish	2	3.394	1.000	1.000	.018	[.000, .050]	—	—	—	—
2. Metric invariance
U.S.-Danish	4	13.869	.999	.999	.038	[.017, .061]	—	—	—	—
U.S.-German	4	2.560	1.000	1.000	<.001	[.000, .026]	—	—	—	—
U.S.-Spanish	4	5.502	1.000	1.000	.013	[.000, .037]	—	—	—	—
2 versus 1
U.S.-Danish	—	—	—		—	—	2	12.295	.001	.001
U.S.-German	—	—	—		—	—	2	1.789	<.001	<.001
U.S.-Spanish	—	—	—		—	—	2	2.108	<.001	<.001
3. Scalar invariance
U.S.-Danish	6	13.870	.999	.999	.028	[.008, .047]	—	—	—	—
U.S.-German	6	2.856	1.000	1.000	<.001	[.000, .016]	—	—	—	—
U.S.-Spanish	6	5.502	1.000	1.000	<.001	[.000, .026]	—	—	—	—
3 versus 2
U.S.-Danish	—	—	—		—	—	2	0.001	<.001	<.001
U.S.-German	—	—	—		—	—	2	0.296	<.001	<.001
U.S.-Spanish	—	—	—		—	—	2	<0.001	<.001	<.001
4. Residual invariance
U.S.-Danish	10	1,311.661	.852	.827	.276	[.263, .289]	—	—	—	—
U.S.-German	10	27.500	.997	.998	.028	[.016, .040]	—	—	—	—
U.S.-Spanish	10	1,854.681	.737	.810	.195	[.187, .202]	—	—	—	—
4 versus 3
U.S.-Danish	—	—	—		—	—	4	1,297.791	.147	.083
U.S.-German	—	—	—		—	—	4	24.644	.003	.001
U.S.-Spanish	—	—	—		—	—	4	42.825	.007	.003

Table 6.

Fit Indices for Multiple-Group Confirmatory Factor Analyses Evaluating Measurement Invariance of the Bifactor Structure Comparing the U.S. Group to the Danish, German, and Spanish Groups.

Model	df	χ²	CFI	Mc	RMSEA	90% CI	Δdf	Δχ²	ΔCFI	ΔMc
1. Configural invariance
U.S.-Danish	2	1.574	1.000	1.000	.000	[.000, .045]	—	—	—	—
U.S.-German	2	0.771	1.000	1.000	.000	[.000, .031]	—	—	—	—
U.S.-Spanish	2	3.394	1.000	1.000	.018	[.000, .050]	—	—	—	—
2. Metric invariance
U.S.-Danish	5	49.127	.995	.994	.072	[.054, .091]	—	—	—	—
U.S.-German	5	0.828	.999	.999	.021	[.000, .040]	—	—	—	—
U.S.-Spanish	5	6.792	1.000	1.000	.013	[.000, .034]	—	—	—	—
2 versus 1
U.S.-Danish	—	—	—	—	—	—	3	47.553	.005	.003
U.S.-German	—	—	—	—	—	—	3	9.057	.001	<.001
U.S.-Spanish	—	—	—	—	—	—	3	3.398	<.001	<.001
3. Scalar invariance
U.S.-Danish	6	49.127	.995	.994	.065	[.049, .082]	—	—	—	—
U.S.-German	6	10.119	.999	1.000	.017	[.000, .035]	—	—	—	—
U.S.-Spanish	6	6.492	1.000	1.000	.008	[.000, .030]	—	—	—	—
3 versus 2
U.S.-Danish	—	—	—	—	—	—	1	<0.001	<.001	<.001
U.S.-German	—	—	—	—	—	—	1	0.291	<.001	<.001
U.S.-Spanish	—	—	—	—	—	—	1	<0.001	<.001	<.001
4. Residual invariance
U.S.-Danish	10	1,332.570	.849	.842	.278	[.266, .291]	—	—	—	—
U.S.-German	10	29.372	.997	.998	.029	[.017, .042]	—	—	—	—
U.S.-Spanish	10	49.981	.993	.005	.043	[.031, .055]	—	—	—	—
4 versus 3
U.S.-Danish	—	—	—	—	—	—	4	1,283.442	.146	.082
U.S.-German	—	—	—	—	—	—	4	19.253	.002	.001
U.S.-Spanish	—	—	—	—	—	—	4	43.189	.007	.003

Discussion

This study assessed measurement invariance of the RIAS across the U.S., Danish, German, and Spanish versions for a single-factor structure, an oblique two-factor structure, and a bifactor structure using the standardization samples. Employing CFA techniques, results supported the oblique two-factor structure and the bifactor structure for all four versions. The bifactor structure tested in this article is statistically equivalent to the two-factor structure using single-group analysis and has been recognized as the best representation of Carroll’s (1993) three-stratum theory (Beaujean, 2015; Carroll, 2003; Cucina & Howardson, 2016). The results of the bifactor analysis support previous findings that the general factor accounted for the largest proportion of common variance in each language group (e.g., Dombrowski et al., 2009; Nelson & Canivez, 2012; Nelson et al., 2007). The finding that the amount of variance accounted for by the two specific factors was small and that the reliabilities tended to be low in each language group impeded a confident interpretation of these two factors. This last finding is not surprising because with only two indicators per specific factor it is difficult to capture substantial unique variance in a bifactor structure. No evidence was found supporting the single-factor structure with one general factor, indicating that this structure is too parsimonious for the RIAS.

Using multiple-group CFA, the findings of the current study provide evidence for scalar invariance in both, the oblique two-factor model and the bifactor model. However, the oblique two-factor model fitted the data slightly better than the bifactor model, a finding that corroborates the notion that the oblique two-factor structure is theoretically more accepted for the RIAS (e.g., Reynolds & Kamphaus, 2003). This is further in line with previous studies providing evidence for the oblique two-factor structure (Beaujean et al., 2009; Hagmann-von Arx & Grob, 2014; Hartmann & Andresen, 2011; Reynolds & Kamphaus, 2003; Santamaría & Fernández Pinto, 2009). However, the results of the bifactor analysis revealed a strong general factor and a good model fit, although there are only two indicators per specific factor, which requires that the loadings of the specific factors are set to 1 and the variances of the specific factors are constrained to equality to identify the model. Invariance of residual variances did not hold across the four language groups (i.e., residual variances varied across groups), which may be due to varying reliabilities of the scales across groups (DeShon, 2004). Invariance of residual variances was only found between the U.S. version and the German version and, to a lower degree, between the U.S. version and the Spanish version.

This study has several strengths and some limitations. One strength is the evaluation of the RIAS factor structure across all language groups in which the RIAS currently is available. The large samples and the employment of confirmatory methods are further strengths. It should be noted, though, that analyses were conducted using standardized data of the standardization samples because raw data were not made available by all publishers of the RIAS. The use of standardized data, which express the extent to which a person deviates from the norm, is a common practice to test psychometric properties of intelligence tests (e.g., Beaujean, Freeman, Youngstrom, & Carlson, 2012; Beaujean & McGlaughlin, 2014; Benson, 2007; Benson, Beaujean, & Taub, 2015; Benson, Hulac, & Kranzler, 2010; Bialer, 1974; Kranzler, Benson, & Floyd, 2015; Kranzler, Floyd, Benson, Zaboski, & Thibodaux, 2016; Naglieri, Taddei, & Williams, 2013; Nelson & Canivez, 2012; Taub & Benson, 2013). However, future studies may use raw scores (i.e., number of questions answered correctly), which would enable researchers to address additional questions, such as whether general populations differ in their factor means. Due to the wide age range of the RIAS, the raw scores are typically highly correlated with age (Reynolds & Kamphaus, 2003). Therefore, analyses using raw scores would require the assessment of age-homogeneous groups (cf. Bialer, 1974). A second limitation of the current study is the use of samples representing the general populations, and, thus, it remains unknown whether measurement invariance holds for specific subpopulations. There is one study focusing on referred students, which found evidence that the U.S. RIAS factor structure holds across samples of referred students and the norming data of the RIAS (Beaujean et al., 2009). Furthermore, the brevity of the RIAS is a clear strength as it allows for efficient testing (Andrews, 2007; Dombrowski & Mrazik, 2008; Elliot, 2004) but limits the statistical evaluation of alternative factor structures. For example, additional measures of verbal and nonverbal abilities could be added that would allow the estimation of a joint bifactor model with free factor variances (see, for example, Canivez, Konold, Collins, & Wilson, 2009; Nelson & Canivez, 2012). Finally, invariance was examined at the level of subtests assuming similar loadings of the items across the language groups. Future research may analyze measurement invariance at the item-level using raw scores.

In conclusion, the current study strongly supports the oblique two-factor structure and the bifactor structure with a strong general factor. Both factor structures showed scalar measurement invariance across all four language groups, meaning that the RIAS test structure is comparable across individuals from the United States, Denmark, Switzerland and Germany, and Spain. No evidence was found for the single-factor structure, which seems to be too simple for the RIAS.

Supplemental Material

Supplement_Material_RIAS – Supplemental material for The Reynolds Intellectual Assessment Scales: Measurement Invariance Across Four Language Groups

Supplemental material, Supplement_Material_RIAS for The Reynolds Intellectual Assessment Scales: Measurement Invariance Across Four Language Groups by Jasmin T. Gygi, Thomas Ledermann, Alexander Grob, Myriam Rudaz and Priska Hagmann-von Arx in Journal of Psychoeducational Assessment

Footnotes

Acknowledgements

We are grateful to Hogrefe Psykologisk Forlag A/S, Virum, Denmark, for providing the data on the Danish RIAS standardization sample, and TEA Ediciones, Madrid for providing the data on the Spanish RIAS standardization sample. We thank Cecil R. Reynolds for helpful discussions on this topic.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Thomas Ledermann

Supplemental Material

Supplemental material is available for this article online.

Notes

References

Andrews

J. J. W.

(2007). Review of RIAS: Reynolds Intellectual Assessment Scales. Journal of Psychoeducational Assessment, 25, 402-408.

Beaujean

A. A.

(2015). John Carroll’s views on intelligence: Bi-factor vs. higher-order models. Journal of Intelligence, 3, 121-136. doi:10.3390/jintelligence3040121

Beaujean

A. A.

Firmin

M. W.

Michonski

J. D.

Berry

Johnson

(2010). A multitrait—Multimethod examination of the Reynolds Intellectual Assessment Scales in a college sample. Assessment, 17, 347-360. doi:10.1177/1073191109356865

Beaujean

A. A.

Freeman

M. J.

Youngstrom

Carlson

(2012). The structure of cognitive abilities in youths with manic symptoms: A factorial invariance study. Assessment, 19, 462-471. doi:10.1177/1073191111399037

Beaujean

A. A.

McGlaughlin

S. M.

(2014). Invariance in the Reynolds Intellectual Assessment Scales for Black and White referred students. Psychological Assessment, 26, 1394-1399. doi:10.1037/pas0000029

Beaujean

A. A.

McGlaughlin

S. M.

Margulies

A. S.

(2009). Factorial validity of the Reynolds Intellectual Assessment Scales for referred students. Psychology in the Schools, 46, 932-950. doi:10.1002/pits.20435

Benson

(2007). Cattell—Horn—Carroll cognitive abilities and reading achievement. Journal of Psychoeducational Assessment, 26, 27-41.

Benson

Beaujean

A. A.

Taub

G. E.

(2015). Using score equating and measurement invariance to examine the Flynn effect in the Wechsler Adult Intelligence Scale. Multivariate Behavioral Research, 50, 398-415. doi:10.1080/00273171.2015.1022642

Benson

Hulac

Kranzler

(2010). Independent examination of the Wechsler Adult Intelligence Scale—Fourth Edition (WAIS-IV): What does the WAIS-IV measure? Psychological Assessment, 22, 121-130. doi:10.1037/a0017767

10.

Bialer

I. R. V.

(1974). Raw versus standardized intelligence test scores: I—IS Throne’s position valid? The British Journal of Mental Subnormality, 20, 93-96. doi:10.1179/bjms.1974.017

11.

Canivez

G. L.

(2016). Bifactor modeling in construct validation of multifactored tests: Implications for understanding multidimensional constructs and test interpretation. In Schweizer

DiStefano

(Eds.), Principles and methods of test construction: Standards and recent advancements (pp. 247-271). Göttingen, Germany: Hogrefe Publishers.

12.

Canivez

G. L.

Konold

T. R.

Collins

J. M.

Wilson

(2009). Construct validity of the Wechsler Abbreviated Scale of Intelligence and Wide Range Intelligence Test: Convergent and structural validity. School Psychology Quarterly, 24, 252-265. doi:10.1037/a0018030

13.

Canivez

G. L.

Watkins

M. W.

(2016). Review of the Wechsler Intelligence Scale for Children–Fifth Edition: Critique, commentary, and independent analyses. In Kaufman

A. S.

Raiford

S. E.

Coalson

D. L.

(Eds.), Intelligent testing with the WISC-V (pp. 683-702). Hoboken, NJ: John Wiley.

14.

Carroll

J. B.

(1993). Human cognitive abilities: A survey of factor-analytic studies. New York, NY: Cambridge University Press.

15.

Carroll

J. B.

(2003). The higher-stratum structure of cognitive abilities: Current evidence supports g and about ten broad factors. In Nyborg

(Ed.), The scientific study of general intelligence: Tribute to Arthur R. Jensen (pp. 5-21). New York, NY: Pergamon Press.

16.

Chen

F. F.

Sousa

K. H.

West

S. G.

(2005). Testing measurement invariance of second-order factor models. Structural Equation Modeling, 12, 471-492.

17.

Cucina

J. M.

Howardson

G. N.

(2016). Woodcock-Johnson-III, Kaufman Adolescent and Adult Intelligence Test (KAIT), Kaufman Assessment Battery for Children (KABC), and Differential Ability Scales (DAS) support Carroll but not Cattell-Horn. Psychological Assessment, 29, 1001-1015. doi:10.1037/pas0000389

18.

Cudeck

(1989). Analysis of correlation matrices using covariance structure models. Psychological Bulletin, 105, 317-327.

19.

DeShon

R. P.

(2004). Measures are not invariant across groups without error variance homogeneity. Psychology Science, 46, 137-149.

20.

Dombrowski

S. C.

Mrazik

(2008). Test review: Reynolds Intellectual Assessment Scales. Canadian Journal of School Psychology, 23, 223-230.

21.

Dombrowski

S. C.

Watkins

M. W.

Brogan

M. J.

(2009). An exploratory investigation of the factor structure of the Reynolds Intellectual Assessment Scales (RIAS). Journal of Psychoeducational Assessment, 27, 494-507. doi:10.1177/0734282909333179

22.

Edwards

O. W.

Paulin

R. V.

(2007). Referred students’ performance on the Reynolds Intellectual Assessment Scales and the Wechsler Intelligence Scale for Children. Journal of Psychoeducational Assessment, 25, 334-340. doi:10.1177/0734282907300453

23.

Elliot

R. W.

(2004). Test review: Reynolds Intellectual Assessment Scales. Archives of Clinical Neuropsychology, 19, 325-328.

24.

Eurostat. (2014). The EU in the world—Population. Retrieved from http://ec.europa.eu/eurostat/statistics-explained/index.php/The_EU_in_the_world_-_population#Migration_and_asylum

25.

Goldstein

Princiotta

Naglieri

J. A.

(Eds.). (2015). Handbook of intelligence: Evolutionary theory, historical perspective, and current concepts. doi:10.1007/978-1-4939-1562-0

26.

Hagmann-von Arx

Grob

. (2014). Reynolds Intellectual Assessment Scales (RIAS). Bern, Switzerland: Hans Huber.

27.

Hancock

G. R.

Mueller

R. O.

(2001). Rethinking construct reliability within latent variable systems. In Cudeck

du Toit

Sorbom

(Eds.), Structural equation modeling: Present and future (pp. 195-216). Lincolnwood, IL: Scientific Software International.

28.

Hartmann

Andresen

(2011). Dokumentation af den danske standardisering af RIAS [Documentation for the Danish standardization of RIAS]. Virum, Denmark: Hogrefe Psykologisk.

29.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55. doi:10.1080/10705519909540118

30.

Kim

J.-O.

Ferree

G. D.

(1981). Standardization in causal analysis. Sociological Methods & Research, 10, 187-210.

31.

Krach

S. K.

Loe

S. A.

Jones

W. P.

Farrally

(2009). Convergent validity of the Reynolds Intellectual Assessment Scales (RIAS) using the Woodcock—Johnson tests of cognitive ability, third edition (WJ-III) with university students. Journal of Psychoeducational Assessment, 27, 355-365. doi:10.1177/0734282909331749

32.

Kranzler

J. H.

Benson

Floyd

R. G.

(2015). Using estimated factor scores from a bifactor analysis to examine the unique effects of the latent variables measured by the WAIS-IV on academic achievement. Psychological Assessment, 27, 1402-1416. doi:10.1037/pas0000119

33.

Kranzler

J. H.

Floyd

R. G.

Benson

Zaboski

Thibodaux

(2016). Classification agreement analysis of Cross-Battery assessment in the identification of specific learning disorders in children and youth. International Journal of School & Educational Psychology, 43, 124-136.

34.

Lee

Hershberger

(1990). A simple rule for generating equivalent models in covariance structure modeling. Multivariate Behavioral Research, 25, 313-334. doi:10.1207/s15327906mbr2503_4

35.

McDonald

R. P.

(1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

36.

Meade

A. W.

Johnson

E. C.

Braddy

P. W.

(2008). Power and sensitivity of alternative fit indices in tests of measurement invariance. Journal of Applied Psychology, 93, 568-592. doi:10.1037/0021-9010.93.3.568

37.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 48, 525-543. doi:10.1007/BF02294825

38.

Milfont

T. L.

Fischer

(2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3, 111-130.

39.

Naglieri

J. A.

Taddei

Williams

K. M.

(2013). Multigroup confirmatory factor analysis of U.S. and Italian children’s performance on the PASS theory of intelligence as measured by the cognitive assessment system. Psychological Assessment, 25, 157-166. doi:10.1037/a0029828

40.

Nelson

J. M.

Canivez

G. L.

(2012). Examination of the structural, convergent, and incremental validity of the Reynolds Intellectual Assessment Scales (RIAS) with a clinical sample. Psychological Assessment, 24, 129-140. doi:10.1037/a0024878

41.

Nelson

J. M.

Canivez

G. L.

Lindstrom

Hatt

C. V.

(2007). Higher-order exploratory factor analysis of the Reynolds Intellectual Assessment Scales with a referred sample. Journal of School Psychology, 45, 439-456. doi:10.1016/j.jsp.2007.03.003

42.

Reise

S. P.

(2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47, 667-696. doi:10.1080/00273171.2012.715555

43.

Reynolds

C. R.

Kamphaus

R. W.

(2003). Reynolds Intellectual Assessment Scales: Professional manual. Lutz, FL: Psychological Assessment Resources.

44.

Rosseel

(2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1-36.

45.

Santamaría

Fernández Pinto

(2009). RIAS Escalas de Inteligencia de Reynolds y RIST Test de Inteligencia Breve de Reynolds Manual: Adaptación Española [RIAS Reynolds Intelligence Scales and RIST Reynolds Brief Intelligence Test Manual: Spanish adaptation]. Madrid, Spain: TEA Ediciones.

46.

Sattler

J. M.

(2001). Assessment of children: Cognitive applications. La Mesa, CA: Author.

47.

Schmid

Leiman

J. M.

(1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61. doi:10.1007/BF02289209

48.

Steenkamp

J. B. E.

Baumgartner

(1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25, 78-107. doi:10.1086/209528

49.

Taub

G. E.

Benson

(2013). Matters of consequence: An empirical investigation of the WAIS-III and WAIS-IV and implications for addressing the Atkins intelligence criterion. Journal of Forensic Psychology Practice, 13, 27-48.

50.

Ten Berge

J. M. F.

Sočan

. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69, 613-625. doi:10.1007/BF02289858

51.

U.S. Department of Homeland Security. (2013). Yearbook of immigration statistics. Retrieved from http://www.dhs.gov/yearbook-immigration-statistics

52.

Van De Schoot

Schmidt

De Beuckelaer

Lek

Zondervan-Zwijnenburg

(2015). Editorial: Measurement invariance. Frontiers in Psychology, 6, Article 1064. doi:10.3389/fpsyg.2015.01064

53.

Watkins

M. W.

(2013). Omega [Computer software]. Phoenix, AZ: Ed & Psych Associates.

54.

Watkins

M. W.

(2017). The reliability of multidimensional neuropsychological measures: From alpha to omega. The Clinical Neuropsychologist, 31, 1113-1126. doi:10.1080/13854046.2017.1317364

55.

Widaman

K. F.

Reise

S. P.

(1997). Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In Bryant

Windle

West

(Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research (pp. 281-324). Washington, DC: American Psychological Association.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB