The fuzziness of assessing second language speaking performance raises two difficulties in scoring speaking performance: indistinction between adjacent levels and overlap between scales. To address these two problems, this article proposes a new approach, confidence scoring, to deal with such fuzziness, leading to confidence scores between two adjacent levels applied to three scales. Since confidence scores have to be transformed to an exact score for test interpretation and use, membership functions and rule bases are applied and a confidence scoring algorithm is developed. Confidence scoring is demonstrated in the paper by an example to facilitate easy understanding. The paper then describes a pilot study that was conducted to try out the confidence scoring design. Initial results reveal that: first, confidence scoring is as feasible as traditional scoring; second, confidence scoring performs better in scoring dependability and in correlations with established benchmarks. At the end of the article, further studies are called for in order to build a validity argument and make further revisions to the confidence scoring method described here.
AldersonJ. C. (1991). Bands and scores. In AldersonJ. C.NorthB. (Eds.), Language testing in the 1990s: The communicative legacy (pp. 71–86). London: Macmillan.
2.
BachmanL. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
3.
BachmanL. F.PalmerA. S. (1996). Language testing in practice. Oxford: Oxford University Press.
4.
BachmanL. F.PalmerA. S. (2010). Language assessment in practice. Oxford: Oxford University Press.
5.
BaiS.-M.ChenS.-M. (2008). Evaluating students’ learning achievement using fuzzy membership functions and fuzzy rules. Expert Systems with Applications, 34(1), 399–410.
6.
BrennanR. L. (2001). Generalizability theory. New York: Springer-Verlag.
7.
BrownA. (2005). Interviewer variability in oral proficiency interviews. Frankfurt: Peter Lang.
8.
BrownA. (2006a). An examination of the rating process in the revised IELTS Speaking Test. In McGovernP.WalshS. (Eds.), IELTS research reports 2006 (pp. 41–70). Canberra & Manchester: IELTS Australia and British Council.
9.
BrownA. (2006b). Candidate discourse in the revised IELTS Speaking Test. In McGovernP.WalshS. (Eds.), IELTS research reports 2006 (pp. 71–89). Canberra & Manchester: IELTS Australia and British Council.
10.
BrownA.IwashitaN.McNamaraT. (2005). An examination of rater orientations and test taker performance on English-for-Academic-Purposes speaking tasks (TOEFL Monograph No. 29). Princeton, NJ: Educational Testing Service.
11.
BrownA.TaylorL. (2006). A world survey of examiners’ views and experience of the revised IELTS Speaking Test. Cambridge ESOL: Research Notes, 26, 14–18.
12.
CanaleM. (1983). On some dimensions of language proficiency. In OllerJ. (Ed.), Issues in language testing research (pp. 333–342). Rowley, MA: Newbury House.
13.
CanaleM.SwainM. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1–47.
14.
Chalhoub-DevilleM. (2003). Second language interaction: Current perspectives and future trends. Language Testing, 20(4), 269–383.
15.
ChapelleC. A. (1998). Construct definition and validity inquiry in SLA research. In BachmanL.CohenA. (Eds.), Interfaces between second language acquisition and language testing research (pp. 32–70). Cambridge: Cambridge University Press.
16.
ChapelleC.EnrightK.JamiesonM. (Eds.). (2008). Building a validity argument for the Test of English as a Foreign Language. New York: Routledge.
17.
China National Office for Teaching Chinese as a Foreign Language. (2002). Teaching syllabus of college Chinese majors for oversea students. Beijing: Beijing Language and Culture University Press.
18.
DaviesA. (1990). Principles of language testing. Oxford: Basil Blackwell.
19.
DaviesA. (2008). Assessing academic English: Testing English proficiency, 1950–1989 – the IELTS solution. Cambridge: Cambridge University Press.
20.
DouglasD. (1994). Quantity and quality in speaking test performance. Language Testing, 11(2), 125–144.
21.
DouglasD. (2000). Assessing languages for specific purposes. Cambridge: Cambridge University Press.
22.
DouglasD.SelinkerL. (1992). Analyzing oral proficiency test performance in general and specific purpose contexts. System, 20(3), 317–328.
23.
DouglasD.SelinkerL. (1993). Performance on a general versus a field-specific test of speaking proficiency by international teaching assistants. In DouglasD.ChapelleC. (Eds.), A new decade of language testing research: Selected papers from the 1990 Language Testing Research Colloquium (pp. 235–256). Alexandria, VA: TESOL.
24.
Educational Testing Service. (2005). TOEFL iBT tips: How to prepare for the next generation TOEFL test and communicate with confidence. Princeton, NJ: Educational Testing Service.
25.
EnrightM. K.BridgemanB.EignorD.LeeY.-W.PowersD. E. (2008). Prototyping measures of listening, reading, speaking, and writing. In ChapelleC. A.EnrightM. K.JamiesonJ. M. (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 145–186). New York: Routledge.
26.
FulcherG. (1996). Does thick description lead to smart tests? A data-based approach to rating scale construction. Language Testing, 13(2): 208–238.
27.
FulcherG. (2003). Testing second language speaking. London: Pearson Education.
IngramD. E.WylieE. (1993). Assessing speaking proficiency in the International English Language Testing System. In DouglasD.ChapelleC. (Eds.), A new decade of language testing research: Selected papers from the 1990 Language Testing Research Colloquium (pp. 220–234). Alexandria, VA: TESOL.
32.
IwashitaN.BrownA.McNamaraT.O’HaganS. (2008). Assessed levels of second language speaking proficiency: How distinct?Applied Linguistics, 29(1), 24–49.
33.
JinT. (2009). An investigation into fuzzy scoring methods for speaking tests. Master’s degree thesis. Shanghai: Shanghai Jiao Tong University, Beijing: China Master Theses Full-text Database.
34.
JinT.WangY.SongC.GuoS. (2008). An empirical study of fuzzy scoring methods for speaking tests. Modern Foreign Languages, 31(2): 157–164.
35.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
36.
LawC. K. (1996). Using fuzzy numbers in educational grading system. Fuzzy Sets and Systems, 83(3), 311–323.
37.
LeeY.-W. (2005). Dependability of scores for a new ESL speaking test: Evaluating prototype tasks. TOEFL Monograph no. 28. Princeton, NJ: Educational Testing Service.
38.
LeeJ.-W.SchallertD. (1997). The relative contribution of L2 language proficiency and L1 reading ability to L2 reading performance: A test of the threshold hypothesis in an EFL context. TESOL Quarterly, 31(4), 713–739.
39.
McNamaraT. F. (1996). Measuring second language performance. London: Longman.
40.
NorthB. (2000). The development of a common framework scale of language proficiency. New York: Peter Lang.
41.
PearlmanM. (2008). Finalizing the test blueprint. In ChapelleC. A.EnrightM. K.JamiesonJ. M. (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 227–258). New York: Routledge.
42.
SalehI.KimS. (2009). A fuzzy system for evaluating students’ learning achievement. Expert Systems with Applications, 36(3), 6236–6243.
43.
SawakiY. (2007). Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite. Language Testing. 24(3): 355–390.
44.
SmithsonM.VerkuilenJ. (2006). Fuzzy set theory: Applications in the social sciences. Thousand Oaks, CA: Sage.
45.
TurnerC. E.UpshurJ. (2002). Rating scales derived from student samples: Effects of the scale marker and the student sample on scale content and student scores. TESOL Quarterly, 36(1), 49–70.
46.
Van LeekwijckW.KerreE. E. (1999). Defuzzification: Criteria and classification. Fuzzy Sets and Systems, 108(2), 159–178.
47.
WangH.-Y.ChenS.-M. (2008). Evaluating students’ answerscripts using fuzzy numbers associated with degrees of confidence. IEEE Transactions on Fuzzy Systems, 16(2), 403–415.
48.
WangL.EignorD.EnrightM. K. (2008). A final analysis. In ChapelleC. A.EnrightM. K.JamiesonJ. M. (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 259–318). New York: Routledge.
49.
XiX. (2007). Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use. Language Testing, 24(2), 251–286.
50.
XiX.MollaunP. (2006). Investigating the utility of analytic scoring for the TOEFL Academic Speaking Test (TAST) (TOEFL iBT Research Report No. TOEFLiBT-01). Princeton, NJ: Educational Testing Service.
51.
ZadehL. A. (1965). Fuzzy sets. Information and Control, 8(3), 338–353.
52.
ZimmermannH.-J. (2001). Fuzzy set theory and its applications (4th ed.). Boston, MA: Kluwer Academic.
