This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score. Confirmatory factor analysis (CFA) and multivariate generalizability theory (G theory) were combined to analyze the responses of 214 admits to a study-abroad program to two role-play speaking tasks in a Spanish speaking assessment designed for student placement and diagnosis. The CFA and G theory approaches provided complementary information, which generally confirmed the key features of the assessment design: (1) the multicomponential and yet highly correlated nature of the five analytic rating scales: Pronunciation, Vocabulary, Cohesion, Organization and Grammar, (2) the high dependability of the ratings and the resulting placement decisions appropriate for the high-stakes decision-making context based on these analytic rating scales and the composite score, and (3) the largest contribution of Grammar to the composite score variance, which was consistent with the intention of program faculty members to reflect in the test design the relative importance of knowledge of grammar for students' academic success in the study-abroad program.
Get full access to this article
View all access options for this article.
References
1.
Alderson, J.C.2005a: The challenge of (diagnostic) testing: Do we know what we are measuring?Plenary paper presented at the 27th Annual Language Testing Research Colloquium. Ottawa , Canada, 20—22 July.
2.
—— 2005b: Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum.
3.
Alderson, J.C. and Huhta, A.2005: The development of a suite of computer-based diagnostic tests based on the Common European Framework. Language Testing22, 301—20.
4.
Bachman, L.F.2005: Statistical analyses for language assessment. Cambridge: Cambridge University Press.
5.
Bachman, L.F., Davidson, F., Ryan, K. and Choi, I.-C.1995: An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study. Cambridge: Cambridge University Press.
6.
Bachman, L., Lynch, B. and Mason, M.1995: Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing12, 238—57.
7.
Bachman, L., Lynch, B., Mason, M. and Egbert, M.1992: EAP Language Ability Assessment System: Interim report . Los Angeles: University of California , Los Angeles, Department of TESL and Applied Linguistics.
8.
Bachman, L.-F. and Palmer, A.1981: The construct validation of the FSI oral interview. Language Learning31, 67—86.
9.
—— 1982: The construct validation of some components of communicative proficiency. TESOL Quarterly16, 449—65.
10.
Bachman, L.F. and Savignon, S.J.1986: The evaluation of communicative language proficiency: A critique of the ACTFL Oral Interview. The Modern Language Journal70, 380—97.
11.
Bae, J. and Bachman, L.F.1998: A latent variable approach to listening and reading: Testing factorial invariance across two groups of children in the Korean/English two-way immersion program. Language Testing15, 380—414.
12.
Bagozzi, R.-P.1991: Further thoughts on the validity of measures of elation, gladness and joy. Journal of Personality and Social Psychology61, 98—104.
13.
Bentler, P.-M.1985—2002: EQS for Windows 6.0 Beta Build 94 Version. [Computer software] . Encino, CA: Multivariate Software, Inc.
14.
Brennan, R.L.1999: mGENOVA, Version 2.0. [Computer software].
15.
—— 2001: Generalizability theory. New York: Springer-Verlag.
16.
Brennan, R.L. and Kane, M.T.1977a: An index of dependability for mastery tests. Journal of Educational Measurement14, 277—89.
17.
—— 1977b: Signal/noise ratios for domain-referenced tests. Psychometrika42, 609—25.
18.
Brown, A.1995: The effect of rater variables in the development of an occupation-specific language performance test. Language Testing12, 1—15.
19.
Brown, J.-D. and Bailey, K.M.1984: A categorical instrument for scoring second language writing skills. Language Learning34, 21—42.
20.
Campbell, D.T. and Fiske, D.W.1959: Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin56, 81—105.
21.
Carroll, J.B.1983: Psychometric theory and language testing. In Oller, J.W., Jr., editor, Issues in language testing research, Rowley, MA: Newbury House, 80—107.
22.
Cronbach, L.J., Gleser, G.C., Nanda, H. and Rajaratnam, N.1972: The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
23.
Elder, C.1993: How do subject specialists construe classroom language proficiency?Language Testing10, 235—54.
24.
Hu, L. and Bentler, P.M.1995: Evaluating model fit. In Hoyle, R.H., editor, Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage Publications, 76—99.
Jarjoura, D. and Brennan, R.1982: A variance components model for measurement procedures associated with tables of specifications. Applied Psychological Measurement6, 161—71.
27.
—— 1983: Multivariate generalizability models for tests developed from tables of specification. In Fyans, L.J., Jr., editor, Generalizability theory: Inferences and practical applications, San Francisco, CA: Jossey-Bass, 83—101.
28.
Jöreskog, K.G.1974: Analyzing psychological data by structural analysis of covariance matrices. In Atkinson, R.C., Krantz, D.H., Luce, R.D. and Suppes, P., editors, Contemporary developments in mathematical psychology: Measurement, psycho-physics, and neural information processing (Vol. 2), San Francisco: Freeman, 1—56.
29.
Kane, M. and Case, S.M.2004: The reliability and validity of weighted composite scores . Applied Measurement in Education17, 221—40.
30.
Kline, R.B.1998: Principles and practice of structural equation modeling . New York: Guilford Press.
31.
Kondo-Brown, K.2002: A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing19, 3—31.
32.
Kunnan, A.J.1995: Test taker characteristics and test performance: A structural modeling approach. Cambridge: Cambridge University Press.
33.
Lee, Y.-W.2005: Dependability of scores for a new ESL speaking test: Evaluating prototype tasks. (TOEFL Monograph Series, MS-28). Princeton, NJ: Educational Testing Service.
34.
Lee. Y.-W. and Kantor, R.2005: Dependability of new ESL writing test scores: Evaluating prototype tasks and alternative rating schemes. (TOEFL Monograph Series, MS-31) . Princeton, NJ: Educational Testing Service.
35.
Li, W.1990: Multivariate generalizability of hierarchical measurement. Unpublished doctoral dissertation, University of California , Los Angeles.
36.
Linacre, J.M.1989: Many-facet Rasch measurement. Chicago : MESA Press.
37.
Linn, R.L. and Werts, C.E.1979: Covariance structures and their analysis. In Traub, R., editor, New directions for testing and measurement: Methodological development (No. 4). San Francisco: Jossey-Bass53—73.
38.
Llosa, L.2005: Building and supporting a validity argument for a standards-based classroom assessment of English proficiency. Unpublished PhD dissertation , University of California, Los Angeles.
39.
Lumley, T. and McNamara, T.1995: Rater characteristics and rater bias: Implications for training. Language Testing12, 54—71.
40.
Lynch, B. and McNamara, T.1998: Using G theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants . Language Testing15, 158—80.
41.
Marcoulides, G.A.1994: Selecting weighting schemes in multivariate generalizability studies. Educational and Psychological Measurement54, 3—7.
42.
—— 1996: Estimating variance components in generalizability theory: The covariance structure analysis approach. Structural Equation Modeling3, 290—99.
43.
—— 2000: Advances and new developments in G-theory: Applications of structural equation modeling. Workshop presented at the 22nd Language Testing Research Colloquium, Vancouver , Canada, March 8—11.
44.
Marsh, H.W.1988: Multitrait-multimethod analysis. In Keeves, J.P., editor, Educational research methodology, measurement, and evaluation: An international handbook, Oxford: Pergamon, 570—80.
45.
—— 1989: Confirmatory factor analysis of multitrait-multimethod data: Many problems and a few solutions. Applied Psychological Measurement13, 335—61.
46.
Marsh, H.W. and Grayson, D.1995: Latent variable models of multitrait-multimethod data . In Hoyle, R.H., editor, Structural equation modeling: Concepts, issues and applications, Thousand Oaks, CA: Sage, 177—98.
47.
McNamara, T.1990: Item response theory and the validation of an ESP test for health professionals. Language Testing7, 52—75.
48.
—— 1996: Measuring second language proficiency. London: Longman.
49.
Petersen, N.S., Kolen, M.J. and Hoover, H.D.1989: Scaling, norming, and equating. In Linn, R.L., editor, Educational Measurement, third edition. New York: American Council on Education and Macmillan, 221—62.
50.
Pollitt, A. and Hutchinson, C.1987: Calibrating graded assessments: Rasch partial credit analysis of performance in writing. Language Testing4, 72—92.
51.
Raykov, T. and Marcoulides, G.A.2006: Estimation of generalizability coefficients via a structural equation modeling approach to scale reliability evaluation. International Journal of Testing6(1), 81—95.
52.
Rindskopf, D. and Rose, T.1988: Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research23, 51—67.
53.
Rosenfeld, M., Leung, S. and Oltman, P.K.2001: The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels. (TOEFL Monograph Series, MS-21). Princeton, NJ: Educational Testing Service.
54.
Satorra, A.1990: Robustness issues in structural equation modeling: A review of recent developments. Quality & Quantity24, 367—86.
55.
Satorra, A. and Bentler, P.1999:. A scaled difference chi-square test statistic for moment structure analysis. UCLA Statistics Series, No. 260, University of California, Los Angeles.
56.
Savignon, S.J.1985: Evaluation of communicative competence: The ACTFL Provisional Proficiency Guidelines. The Modern Language Journal69, 129—42.
57.
Sawaki, Y.2003: A comparison of summarization and free recall as reading comprehension tasks in web-based assessment of Japanese as a foreign language. Unpublished doctoral dissertation, University of California , Los Angeles.
58.
—— 2005: The generalizability of summarization and free recall ratings in L2 reading assessment. JLTA Journal7, 21—44.
59.
Schoonen, R.2005: Generalizability of writing scores: an application of structural equation modeling. Language Testing22, 1—30.
60.
Shavelson, R.-J. and Webb, N.-M.1991: Generalizability theory: A primer. Newbury Park, CA: Sage Publications.
61.
Shin, S.-K.2005: Did we take the same test? Examinee language proficiency and the structure of language tests. Language Testing22, 31—57.
62.
Wang, M.W. and Stanley, J.C.1970: Differential weighting: A review of methods and empirical studies. Review of Educational Research4, 663—705.
63.
Webb, N.-M., Shavelson, R.-J. and Maddahian, E.1983: Multivariate generalizability theory. In Fyans, L.J., Jr., editor, Generalizability theory: Inferences and practical applications, San Francisco, CA: Jossey-Bass, 67—81.
64.
Weigle, S.-C.1998: Using FACETS to model rater training effects. Language Testing15, 263—87.
65.
Werts, C.E., Linn, R.L. and Jöreskog, K.G.1974: Intraclass reliability estimates: Testing structural assumptions . Educational and Psychological Measurement, 34, 25—33.
Xi, X.2003: Investigating language performance on the graph description task in a semi-direct oral test. Unpublished doctoral dissertation, University of California, Los Angeles.
