This study investigated the relationship between examinee proficiency and the
structure of the Test of English as a Foreign Language (TOEFL) and the Speaking
proficiency in English Assessment Kit (SPEAK). Specifically, using multi-group
structural equation modeling, this study tested two competing hypotheses about the
relationship: whether or not the dimensions of language ability become more or less
differentiated as a function of increasing examinee proficiency. Among four
alternative models (a second-order model, a correlated specific factor model, a
single general factor model, and a totally divisible model), a second-order factor
model was established as a baseline model for three ability groups and then
estimated simultaneously with cross-group equality imposed. The results showed that
the tests were partially equivalent, thereby supporting neither of the hypotheses.
Finally, the multiple indicators and multiple causes modeling analysis suggested
that the language backgrounds of test-takers were related to measurement non-invariance.
Get full access to this article
View all access options for this article.
References
1.
Alderson, C.1991: Language testing in the 1990s: how far have we come? how
much further have we to go? In
Anivan, S.
, editor, Current developments in language testing.
Singapore: SEAMEO Regional Language Centre
, 1-26.
2.
Bachman, L. F.
and
Palmer, A. S.1981:
The construct validation of the FSI oral interview
. Language Learning31,
67-86
.
3.
Bachman, L. F.
and
Palmer, A. S.1982:
The construct validation of some components of communicative proficiency
. TESOL Quarterly16,
449-465
.
4.
Bachman, L. F.
,
Davidson, F. G.
and
Foulkes, J.1990:
A comparison of the abilities measured by the Cambridge and Educational
Testing Service EFL Test Batteries
. Issues in Applied Linguistics1,
30-55
.
5.
Bachman, L. F.
,
Davidson, F. G.
,
Ryan, K.
and
Choi, I-C.1995: An investigation into the comparability of two tests of
English as a foreign language: the Cambridge TOEFL comparability study.
Cambridge: UCLES
.
6.
Bentler, P. M.1990:
Comparative fit indexes in structural models
. Psychological Bulletin107,
238-246
.
7.
Bentler, P. M.
and
Wu, E.1995: EQS for Windows: user’s guide.
Encino, CA: Multivariate Software, Inc.
8.
Bollen, K. A.1989: Structural equations with latent variables.
New York: Wiley
.
9.
Browne, M. W.
and
Cudeck, R.1992:
Alternative ways of assessing model fit
. Sociological Methods and Research21,
230-258
.
10.
Byrne, B. M.1994: Structural equation modeling with EQS and EQS/Windows: basic
concepts, applications, and programming.
Thousand Oaks, CA: Sage Publications
.
11.
Byrne, B. M.
,
Shavelson, R. J.
and
Muthén, B.1989:
Testing for the equivalence of factor covariance and mean structures: the
issue of partial measurement invariance
. Psychological Bulletin105,
456-466
.
12.
Carroll, J. B.1983: Psychometric theory and language testing. In
Oller, J. W. Jr.
, editor, Issues in language testing research.
Rowley, MA: Newbury House
, 80-107.
13.
Cheung, G. W.
and
Rensvold, R. B.1999:
Testing factorial invariance across groups: a reconceptualization and
proposed new method
. Journal of Management25,
1-27
.
14.
Cliff, N.1983:
Some cautions concerning the application of causal modeling methods
. Multivariate Behavioral Research18,
115-126
.
15.
Cudeck, R.
and
Browne, M. W.1983:
Cross-validation of covariance structures
. Multivariate Behavioral Research18,
147-167
.
16.
Davidson, F.
and
Liu, A.1997:
Creating normally distributed ability subgroups
. Language Testing Update22,
58-66
.
17.
Educational Testing Service. (ETS)1985: A guide to SPEAK.
Princeton, NJ: Educational Testing Service
.
18.
Educational Testing Service. (ETS)1987: TOEFL test and score manual.
Princeton, NJ: Educational Testing Service
.
19.
Elder, C.1996:
The effect of language background on foreign language test performance: the
case of Chinese, Italian, and Modern Greek
. Language Learning46,
233-282
.
20.
Fouly, K. A.
,
Bachman, L. F.
and
Cziko, G. A.1990:
The divisibility of language competence: a confirmatory approach
. Language Learning40,
1-21
.
21.
Gass, S. M.
and
Selinker, L.2001: Second language acquisition: an introductory course.
Mahwah, NJ: Lawrence Erlbaum Associates
.
22.
Ginther, A.
and
Stevens, J.1998: Language background, ethnicity, and the internal construct
validity of the advanced placement Spanish language examination. In
Kunnan, A. J.
, editor, Validation in language assessment.
Mahwah, NJ: Lawrence Erlbaum Associates
, 169-194.
23.
Hale, G. A.
,
Rock, D. A.
and
Jirele, T.1989: Confirmatory factor analysis of the Test of English as a
Foreign Language. TOEFL research report 32.
Princeton, NJ: Educational Testing Service
.
24.
Harley, B.
,
Cummins, J.
,
Swain, M.
and
Allen, P.1990: The nature of language proficiency. In
Harley, B.
,
Allen, P.
,
Cummins, J.
and
Swain, M.
, editors, The development of second language proficiency.
New York: Cambridge University Press
, 7-25.
25.
Kim, M.2001:
Detecting DIF across the different language groups in a speaking test
. Language Testing18,
89-114
.
26.
Kline, R. B.1998: Principles and practice of structural equation modeling.
London: The Guilford Press
.
27.
Kunnan, A. J.1992:
An investigation of a criterion-referenced test using G-theory, factor and
cluster analyses
. Language Testing9,
30-49
.
28.
Kunnan, A. J.1995: Test taker characteristics and test performance: a structural
modeling approach.
Cambridge: Cambridge University Press
.
29.
Larsen-Freeman, D.
and
Long, M.1991: An introduction to second language acquisition research.
New York: Longman
.
30.
Lynch, B.
and
McNamara, T. F.1998:
Using G-theory and Many-facet Rasch measurement in the development of
performance assessments of the ESL speaking skills of immigrants
. Language Testing15,
158-180
.
31.
Meiron, B.
and
Schick, L.
(2000). Ratings, raters and test performance: an
exploratory study. In
Kunnan, A. J.
, editor, Fairness and validation in language assessment.
Cambridge: Cambridge University Press
.
32.
Mulaik, S. A.
and
Quartetti, D. A.1997:
First order or higher order general factor?Structural Equation Modeling4,
193-211
.
33.
Muthén, B.1988: Some uses of structural equation modeling in validity
studies: extending IRT to external variables. In
Wainer, H.
and
Braun, H. I.
, editors, Test validity.
Hillsdale, NJ: Lawrence Erlbaum Associates
, 213-238.
34.
Muthén, B.2001: Lecture notes for Course ED M231E: Statistical analysis with
latent variables.
School of Education, University of California,
Los Angeles
.
35.
Muthén, B.
and
Christoffersson, A.1981:
Simultaneous factor analysis of dichotomous variables in several groups
. Psychometrika46,
407-419
.
36.
Oller, J. W. Jr.1979: Language tests at school.
London: Longman
.
37.
Oltman, P. K.
,
Stricker, L. J.
and
Barrows, T.1988: Native language, English proficiency, and the structure of the
TOEFL. TOEFL research report 27.
Princeton, NJ: Educational Testing Service
.
38.
Purpura, J. E.1998:
Investigating the effects of strategy use and second language test
performance with high- and low-ability test takers: a structural equation
modeling approach
. Language Testing15,
333-379
.
39.
Raykov, T.
and
Marcoulides, G. A.2000: A first course in structural equation modeling.
Mahwah, NJ: Lawrence Erlbaum Associates
.
40.
Rindskopf, D.
and
Rose, T.1988:
Some theory and applications of confirmatory second-order factor analysis
. Multivariate Behavioral Research23,
51-67
.
41.
Ryan, C.
and
Bachman, L.1992:
Differential item functioning on two tests of EFL proficiency
. Language Testing9,
12-29
.
42.
Sasaki, M.1996: Second language proficiency, foreign language aptitude, and
intelligence: quantitative and qualitative analyses.
New York: Peter Lang
.
43.
SPSS Inc. 2001: SPSS for Windows release 10.0 standard version.
Chicago, IL: SPSS Inc.
44.
Steenkamp, J.-B., E. M.
and
Baumgartner, H.1998:
Assessing measurement invariance in cross-national consumer research
. Journal of Consumer Research25,
78-90
.
45.
Swinton, S. S.
and
Powers, D. E.1980: Factor analysis of the TOEFL. TOEFL Research Reports 6.
Princeton, NJ: Educational Testing Service
.
46.
Thomas, M.1994:
Assessment of L2 proficiency in second language acquisition research
. Language Learning44,
307-336
.
47.
Vollmer, H. J.1983: The structure of foreign language competence. In
Hughes, A.
and
Porter, D.
, editors, Current developments in language testing.
London: Academic Press
, 3-29.
48.
Vollmer, H. J.
and
Sang, F.1983: Competing hypotheses about second language ability: a plea
for caution. In
Oller, J. W.
, editor, Issues in language testing.
Rowley, MA: Newbury House Publishers
, 29-79.
49.
Wheaton, B.1987:
Assessment of fit in over-identified models with latent variables
. Sociological Methods and Research16,
118-154
.
