The Language Policy of Placement Tests for Newcomer English Learners

Language Policy and Tests

Language testing is a central mechanism of both language policy and education policy (Shohamy, 2001; Shohamy & McNamara, 2009). Researchers have documented not only how tests serve as a powerful means of implementing language and education policy, but often as the policy itself. Menken (2008), for instance, documented how New York City English Learner (EL) high school students experienced high-stakes graduation tests. She found that the graduation test shaped what content was taught in schools, how it was taught, by whom, and in what language, with the graduation test essentially becoming the language policy in New York high schools. In this New York case and in many other contexts worldwide, researchers of language policy have examined how the testing mechanism not only enacts but often becomes the default or de facto language policy and education policy, driving curricular decisions and a wide range of educational practices (Shohamy, 2014; Spotti, 2013). The field of language policy (and its subfield, language education policy) increasingly has focused on how particular polices are taken up or enacted by participants in everyday situations (King & De Fina, 2010); this work often requires researchers to engage in discourse analysis or other qualitative research approaches (see Mortimer & Wortham, 2015, for an overview).

As the field of language policy has taken an ethnographic and discursive turn (McCarty, 2014) and increasingly turned its gaze to how language policies are experienced on the ground, language tests have received greater critical scrutiny (Maryns, 2004). McNamara, Khan, and Frost (2015), for instance, documented how language tests for migrant adults were legitimized as valid gatekeepers in the immigration process in Australia, with test takers viewing their score as a marker of “outsider” status, despite their successful professional and social integration. Within K-12 contexts, researchers have documented how language proficiency tests potentially mislead us about students’ abilities. MacSwan and Rolstad (2006), for instance, demonstrated how widely used measures of ELs’ native language skills (e.g., Language Assessment Scales-Oral, Español) dramatically underestimated children’s language capacities, leading to ELs’ disproportionate representation in special education. Findings from this line of work can be linked to researchers’ calls for greater attention to revising existing approaches to testing ELs given that present research and practice assessment paradigms overlook the complex nature of language and bilingualism (e.g., Solano-Flores & Trumbull, 2003; Valdés & Figueroa, 1994). As Hakuta and McLaughlin (1996) asserted more than two decades ago (but is still often the case today), EL testing practices are often driven by policies rather than theory and research on language development.

Although both EL testing practices and language proficiency assessments have received growing attention in the United States and beyond (Cumming, 2009; Davies, 2009; Jones & Saville, 2009; Leung & Scott, 2009; Menken, 2009), none to date has examined the most widespread assessment used to identify, and often to place, K-12 students. This is particularly remarkable in the United States given that the Lau decision is perhaps the most important federal language education policy to date, mandating that all U.S. schools take “affirmative steps” to ensure equal access to the curriculum, with identification of non-English-proficient students arguably the most crucial component. As detailed below, for many U.S. school districts, the W-APT serves as both a central mechanism to ensure compliance with the Lau ruling and policy guidelines and, moreover, an important tool for placing students in EL programs.

W-APT

The 2002 No Child Left Behind (NCLB) Act focused new attention on EL students and required U.S. states to annually measure EL growth. In response to state demands for an assessment tool for ELs, WIDA ² partnered with Center for Applied Linguistics (CAL) to develop, refine, and expand the annual Assessing Comprehension and Communication in English State-to-State (ACCESS) test. In 2013-2014, 33 states administered the ACCESS test to 1,372,611 EL students (WIDA, n.d.-a). When state agencies join the WIDA Consortium, they gain the right to administer ACCESS tests to ELs, but also to professional development and WIDA curricula and standards. In addition, member states also receive access to the W-APT, often referred to as “the screener,” at no additional cost. ³ Because the W-APT is a pen-and-paper test that is locally administered and scored, no data exist on how many students take this test each year. Given that most newcomers to the United States take this test and most EL transfer students (from a different U.S. district or charter school) also take this test, it is likely administered to hundreds of thousands of students each year. The latest 2015 federal education law, the Every Student Succeeds Act (ESSA), makes accountability for ELs a priority, for instance, requiring that each state develop and adopt a uniform, standardized system for identifying and exiting ELs, policies that suggest states will continue to partner with WIDA (and to use a WIDA screening tool).

The W-APT was developed concurrently with the ACCESS test in 2005 and re-released with updates since then (2007 and 2013). The W-APT format and scoring differ from the ACCESS test, and the W-APT is not articulated with the ACCESS test in such a way that growth across W-APT and ACCESS scores can be measured. (This was never the plan or recommendation by WIDA.) The W-APT, according to WIDA, was designed to be used as one of several criteria to determine eligibility for EL support services; an indicator of a student’s language proficiency level (one through six) on the WIDA continuum; an aid to determining appropriate levels and amounts of instructional services; and a guide for tier placement on the ACCESS test (WIDA, n.d-b).

The W-APT thus serves as language education policy in multiple ways. First and more importantly, it is the primary mechanism for the identification of eligibility for EL services, thus ensuring compliance and enacting two federal policies: Lau and NCLB (now ESSA). The W-APT also works as de facto policy by informing school districts’ placement decisions concerning programs and schools. In some large districts (such as the one studied here), certain high schools are designated as newcomer schools, and students’ W-APT scores determine their high school assignment and trajectory. For instance, if they earn a low W-APT score, their English credits from overseas might not be accepted, they might take more English as a second language (ESL) classes (rather than academic content classes), or they might be placed in lower-level content courses. The ramifications of these practices can be serious for students, undermining high school completion and limiting access to college prep courses (e.g., Callahan, 2013; Kanno & Kangas, 2014). And third, as discussed below, from an ethnography of language policy perspective, the highly scripted nature of the WIDA test works as language policy as it regulates the speech and interactional modes of participants.

Scripted Talk

The W-APT, in contrast to other language tests (e.g., the American Council on the Teaching of Foreign Languages Oral Proficiency Interview; Swender, 2003), is fully scripted; test administrators are to read all instructions and test items exactly as printed in the handbook in English. Only the initial section of the test (assessing student-speaking skills) allows for flexibility, with test administrators given several options as follow-up questions (e.g., “Tell me more.” or “Can you be more specific?”). The potential limitations of partially scripted or unscripted oral proficiency interviews have been examined in some detail. Brown (2005), for instance, found that International English Language Testing System (IELTS) interviewers were far from neutral, and that “which interviewer an oral interview candidate encounters can significantly affect their chances of achieving a desired score” (p. 243). Moreover, variation in scores across interviewers/raters has been shown to be driven by differences in interviewer discourse, including their elicitation patterns (e.g., complex vs. simple, direct vs. indirect), rapport work (tone, expressions of interest), and speech style (rate, repairs), all of which shaped interaction, including test takers’ performance and ultimately test administrators’ score. Brown’s research, and that of others (e.g., Brooks, 2009; Delieza, 2012), supports the broader claim that test performance is socially constructed rather than an independent cognitive trait (e.g., McNamara, 1997).

Although the nature of unscripted (or partially scripted) oral proficiency interviews has been widely examined (e.g., Lazaraton, 1992), much less studied is the discourse of scripted interview talk. Nearly all work to date on scripted talk examines how scripts work as curricular materials. For example, scripted curricula (e.g., Direct Instruction, Open Court, Reading Mastery, and Success for All) have been adopted by thousands of elementary schools, generating a growing interest in how scripts shape and constrain classroom interaction and outcomes (e.g., Milner, 2013). In general, scripted curricula tend to focus on instruction of explicit, direct, systematic literacy skills, often advertised as a means to improve standardized test scores (Ede, 2006). Findings on effectiveness are mixed, however. For instance, one study of para-educators working with first graders who were identified as “at risk” for reading failure found that use of scripted reading instruction led to higher rates of on-task instructional opportunities for student practice (relative to non-scripted instruction; Cooke, Galloway, Kretlow, & Helf, 2011). Others have expressed concern about rushed pacing and superficial treatment of complex concepts, and the negative ways in which these materials limit teacher creativity, improvisation, and responsiveness to student needs (e.g., Dresser, 2012; Sawyer, 2004). A related line of research has focused on the impact of scripts that are often part of commercial texts in language classrooms. These scripted texts (often audio performances with accompanying transcripts) tend to differ from authentic, unplanned interactions. Generally, unplanned spoken discourse contains more pauses, fillers, and false starts; is less grammatically complex; and has “less careful” phonological characteristics (Wagner, 2014). Experimental work suggests that, at least for intermediate Spanish second language learners, scripted texts were easier to understand than unscripted texts (Wagner & Toth, 2014).

Scripted spoken tests such as the W-APT are designed to constrain test administrators’ speech with the student. Although this potentially strengthens interrater reliability, an important benefit in contexts where little or limited training is provided, little is known about how scripted tests such as the W-APT work in practice. In light of this gap, two research questions drive this study:

Abdi: Assessing Emergent Literacy

Nineteen-year-old Abdi walked through the doors of the Center 4 days after the 2015-2016 school year had started. Somali is his first language, and although he had been in the United States since June, this would be his first formal school experience. He arrived alone by taxi from his temporary housing, where he is staying with his parents and siblings. He is quiet and nervous as he walks with the test administrator to the back room. The administrator does not speak Somali; there is not much small talk.

They get seated side-by-side at the table, and Jane, an experienced tester and former ESL teacher, begins with the script:

Good morning. We’re going to talk about some pictures in English. When I ask you a question, just answer the best you can. Are you ready? First, we’re going to talk about things high school students do. Here are some pictures of activities high school students do. I’m going to ask you some questions about these pictures. [pointing to a photo of soccer players] What are these students doing?

Abdi is silent. The test administrator goes on to the next questions: “Do you like to play soccer?” “What is he doing?” “Do you like to swim?” Abdi does not respond; Jane turns the page and moves on to the listening section:

Now you are going to do a listening test. I’ll ask you a question and you will point to the correct answer on the page. [points to pictures with her finger]. This says “practice.” Look at the big picture. It will help you understand what I say. The big picture shows people enjoying a day in the park. Now, when I say, “letter A,” find letter A in your text booklet. Letter A. Can you point to the letter A?

Abdi looks down at his test booklet, motionless. Jane moves on to the reading portion of the test. Abdi is not able to complete practice exercises and does not go on to do the test items for either of these sections. Abdi receives a 0 point and a score of 1 for all three sections.

Jane advances to the fourth and final section of the test, which focuses on writing. Rather than follow the script:

Now you are going to take a writing text. It is important that you do the best you can. On this page you will read about a special event that Emily planned. On this page you will write about a special event that you can plan. Look at the directions in the black box at the top of this page . . .

Jane says, “Now let me see if you can write something. Can you write your name?” She hands him the (party planning) answer sheet and points to the first line. He slowly, carefully prints his first name, using capital and lowercase letters appropriately, but with some letters above and others below the line. This takes him a full minute. Jane thanks him and takes the sheet. He is also given a native language assessment (in Somali), which asks him to answer short biographical questions in Somali. He is unable to complete this, writing a large letter M shape as an answer to the first question (his name). Because of his lack of transcripts, overall W-APT score (1), and age (19), Abdi is placed as a ninth grader in the city’s all newcomer high school.

What is most salient in this interaction is how the formal testing apparatus of the W-APT was long and unwieldy given the student’s low English proficiency levels. In response, the experienced tester adapted (and violated) formal guidelines, for instance, by not going through with the listening and reading sections based on practice performance, and not following the script for the writing component of the exam, and instead asking Abdi to write his name. As she explained afterward, she views the invitation writing task as “weird, inappropriate and contrived,” noting that her son with a U.S. MA degree never wrote letters, or addressed envelops, so how was someone who just arrived expected to write out a formal invitation? (Field notes, September 2, 2015). More broadly, this interaction suggests the ways that experienced testers negotiate and adapt the W-APT. Although these practices contradict the testing instructions in the technical manual, in some cases, they result in a more humane and probably equally accurate evaluation.

Bakalcha: Off Script Adjustments

Bakalcha, also 19, recently arrived to the United States. He is a native speaker of Oromo and comes to the front desk of the New Student Center with no appointment, but a white plastic bag from the United Nations (UN) International Organization for Migration, jammed full of documents. School started in the district about 2 weeks ago. Bakalcha is sharply dressed with new black athletic shoes with still very white stripes and a new-looking black leather jacket (although it is summerlike outside). It takes nearly an hour to complete intake paperwork, in part because he has a complicated medical history. A bilingual woman, a few years older than Bakalcha, accompanies him and helps to answer questions about his background: “Has he lived in a refugee camp?” (yes), “Has he missed more than a month of school?” (yes), “What other languages does he speak?” (Amharic). After more waiting and signatures, Bakalcha (B) is led to the back testing room by a newly hired staffer and test administrator, Mohammad (M), a Somali–English bilingual man in his mid-20s. The administrator has a casual, relaxed tone as they settle in.

Although there are many notable features of this interaction, here, we focus on the ways that Mohammad simultaneously makes the test (and text) both more and less challenging (See Table 1). For instance, the administrator (M), perhaps in attempt to be less formal, to minimize social distance, and to make test interaction less stressful, repeatedly goes off script. In doing so, however, his language becomes more complex. For instance, he uses colloquial expressions such as “these guys” (turns 11 and 37) and discourse connectors such as “so” (turns 3, 5, 22, 39). He makes questions more indirect by embedding them (e.g., “Can you tell me what else do you see in these pictures?” [31] “Can you tell me what other things you see in these pictures?” [35] “Can you tell me what these things are?” [22]). This lengthens and grammatically complicates his requests. He also makes his language more complex by turning simple information questions from the script (e.g., “What is this?”) into embedded commands (e.g., “And tell me what this is?” [24]).

OPEN IN VIEWER

Table 1.

Testing Excerpt 1 (Bakalcha).

Note. Text in italics corresponds to test script. See Appendix B for transcription guidelines.

		Participant utterances [and actions]	Scripted text for test administration [and notes on actions / test scoring]
1	M	All right [passing the student the test booklet]. How do you say your first name?
2	B	Bakalcha
3	M	This is just an assessment. It’s not uh a big test, just to see how your score. So uh just do the best you can. Take your time and complete every question to the best of your ability. [points to student test booklet] So, you want to open that up? Just answer the best you can. Are you ready?	The script is: Good morning (student’s name). We’re going to talk about some pictures in English. When I ask you a question, just answer the best you can. Are you ready?
4	B	[student flips through pages and M helps him find initial page with picture of activities]
5	M	All right, even if you find some of these questions pretty easy I want you to answer->[orients student’s book so that pictures are vertical rather than horizontal]All right? So first we are going to talk about things high school students do. Here are some pictures of activities high school students like to do. I’m going to ask you some questions about these pictures. [points to soccer players] So what are these students doing?	First we are going to talk about things high school students do. Here are some pictures of activities high school students like to do. I’m going to ask you some questions about these pictures. [points to soccer players] So what are these students doing?
6	B	xxx football
7	M	[points to football at bottom of drawing] What is that?	Do you like to play soccer?
8	B	ball?
9	M	yep. And what’s he doing here? [points to drawing of swimmer]	What is he doing?
10	B	swimming [points to same picture]	Scripted follow-up is Do you like to swim?
11	M	What about these guys . . . what are they doing? [points to sketch of people sitting in theatre seats eating popcorn looking at picture/screen of dinosaur and sky scrapers]	What are they doing?
12	B	Watching TV.
13	M	What about this? [points to picture of 5 people standing around a table with cookies and soda and balloons in background]	What other things do you see in these pictures? [or] What else do you see in the pictures?
14	B	[pause] it’s birthday	[M notes a score of “meets” for Part A Speaking Test 1]
15	M	good, good. [pause] Bakalcha, what is your favorite activity? What do you like to do most? What do you like to do?	What’s your favorite activity? [or] What activity do you most like to do? [or] What activity do you like best?
16	B	[brief pause] [no response]
17	M	It’s OK. [writes with pencil in his book]	[M notes a score of “?” for Part A Speaking Test 2]
18	M	Ok, now I want you to tell me which activities you like better: activities you do by yourself, or activities you do with other people?	Now I want you to tell me which activities you like better: activities you do by yourself, or activities you do with other people?
19	B	By myself	Why do you say that? Tell me more. Can you be more specific?[M notes a score of “meets” for Part A Speaking Test 3, and moves on to Part B.]
20	M	By yourself? OK. Alright, now we are going to talk about this picture, ok? [points to the picture of people standing in a group, some with tall hats and some with short hats]
21	B	[Lifts folder up to look at picture at 45 degree angle]
22	M	So now we are going to talk about an event in the history of a made-up place. Look at this picture. It shows what happened when people from one country, the tall-hat people, first arrived in the country of the short-hat people. Now let me ask you some questions about this picture. Can you tell me what these things are? What’s that? [points to area around two boats in the picture]	So now we are going to talk about an event in the history of a made-up place. Look at this picture. It shows what happened when people from one country, the tall-hat people, first arrived in the country of the short-hat people. Now let me ask you some questions about this picture. What is this?
23	B	Water [moves hand between pictures of two boats]
24	M	Good, good And tell me what this is? [points to boat at top of page]
25	B	[licks lips and smiles]
26	M	Ok, it’s OK. And what about, what about this? Is this a tall hat or a short hat? [pointing to tall hat] Is this a tall hat or a short hat? [pointing to short hat]	Is this a tall hat or a short hat? [pointing to tall hat] Is this a tall hat or a short hat? [pointing to short hat]
27	B	[moves hand up] xxx
28	M	Tall hat?
29	M	What about this one?
30	B	[holds thumb and finger together]
31	M	short hat? Ok, good. Can you tell me what else do you see in these pictures?	What else do you see in these pictures?
32	B	Boats
33	M	Say that again?
34	B	Boats
35	M	What else . . . Can you tell me what other things you see in these pictures?
36	B	Men (not v. clear/audible)	[M notes a score of “?” for Part B Speaking Test 1, and moves on Test 2.]
37	M	OK. What are these guys doing? What’s happening here?	What are they doing?
38	B	[smiles and shakes head]
39	M	After some time, things began to change in the country of the short-hat people. This picture shows what happened a few months later. So, what do you think is happening here?	After some time, things began to change in the country of the short-hat people. This picture shows what happened a few months later. So, what do you think is happening here?
40	B	Welcome [mumbled]
41	M	Ok. Good. What do you think is happening here?	What do you think is happening here?
42	B	Promise [mumbled]	[M notes a score of “meets” for Part B Speaking Test 2, and moves on Test 3, about fact versus opinion, which B is unable to respond to]

Mohammad’s modifications are exacerbated by his very generous interpretation of the scoring system, awarding him two “meets” (expectations) and three “?” (a temporary placeholder if tester is unsure how to score), and only two “approaches” in the initial speaking section. According to W-APT guidelines, “meets” (p. 30) should be awarded when a student meets the expectations of the task in quality (i.e., the type of discourse produced) and quality in terms of vocabulary, usage, and control of the language. In general, “meets” should be awarded if “the student is able to give a performance that leaves little doubt that he or she would be able to understand and attempt a response to the task at the next higher level” (p. 30). “Approaches,” in contrast, corresponds to responses that are characterized by “giving brief answers when more extended responses are expected, groping for vocabulary and structure, [or] excessive hesitating” (p. 30). Given that this student cannot produce more than one-word answers to any of these questions, “approaches” would be a more accurate score. However, Bakalcha continued to move on in the test because he was awarded three “meets” by Mohammad, and a speaking level (raw score) of 2, supposedly characterized by “phrases, short oral sentences; general language related to content area; when using simple discourse, language is generally comprehensible and fluent” (p. 27). There is little correspondence between these scoring criteria descriptions and the student language here.

For the final writing portion of the test, Mohammad hands him the party invite worksheet and says:

All right, here you go. So now, think about some special events in your life or at school. Choose to write about one. This picture shows some examples, so you can write about a school event, a sport event, uh some kind of entertainment event, plan your special event here, so name the event, name your event, so this is your event, you’re going to name your event, the event, name the event, name the occasion. So people to invite, write the people’s names to invite, when is the event, where is it located, and any other information, and the second part, write a note to invite a friend. If you can, make at least four sentences . . . Good? So let me know when you’re done, OK? [silence] Do you have any questions? All right!

He leaves him alone with the paper and returns 2.5 min later. Bakalcha has written nothing on the paper, and the test is stopped. Mohammad notes on the scoring sheet that student is “unable to write,” and he receives an overall score of 1.8.

As a final step, we (the researchers) ask the student to write a little in Oromo about his family. Bakalcha quickly and fluently writes in English: “Oromo. My father name Budri My mother name Weyio My brother name Abdiso My sister name Bicuu.” After a few minutes, we ask him to write again in Oromo; he writes the same basic phrases immediately below the English. His work indicates that he mastered the basic mechanics of writing, with fluent penmanship, clear word boundaries, and sentence boundaries as indicated by vertical line spacing. The structure and set-up of the writing task in the W-APT did not allow this student to demonstrate his writing competencies. Because of his lack of transcripts, overall W-APT score (1.8), and age (19), Bakalcha is placed as a ninth grader in the city’s newcomer high school.

Paolo: Code-Switching Through a Monolingual Test

Paolo (P), who will be 17 when school starts in September, arrived to the United States in March 2015, crossing the border by foot and then private car to reunite with his mother. This is his second visit to the New Student Center in August 2015; he has completed all paperwork and is here to take the W-APT. He is a bit nervous as he enters the testing room, and the bilingual test administrator, Silvia (S), chats with him in Spanish to put him more at ease and to clarify his educational history. He reports that he attended school in Mexico since age 4, but changed schools frequently in recent years as he moved from Puebla to Veracruz with his grandparents, and can remember the names of two of the three secondary schools he attended. The last time he attended school was in Veracruz in January 2015. This initial interaction takes about 5 min, and is relaxed and friendly, punctuated by frequent laughter. Silvia’s tone then becomes more serious as she explains the purpose of the test (See Table 2).

OPEN IN VIEWER

Table 2.

Testing Excerpt 2 (Paolo).

Note. Text in italics corresponds to test script; underlined text is translated from Spanish to English. See Appendix B for transcription guidelines.

		Original	Translation and/or scripted text
1	S	Entonces. Ahora. Te explico un poco pero todo tiene que ser en inglés, ¿sí?	Ok then Now. I’ll explain you a little but everything has to be in English, OK?
2	P	[nods]
3	S	Si lo entiendes esta bien, y si no, no->	If you understand, that’s OK, and if not, then not->
4	P	[nods] OK.
5	S	Continuamos. OK?	We continue. OK?
6	P	[nods]
7	S	No más español.	No more Spanish.
8	P	Mhm [smiles nervously]
9	S	Ok, Paolo. OK? My name is Silvia . . . [starts reading the script] I’m going to tell you about some pictures in English, OK?When I ask you a question . . .Just answer the best you can. First we’re going to talk about some things high school students do. Here are some pictures of activities high school students like to do. I’m going to ask you some questions about these pictures. OK? . . . What are these students doing? [pointing]	First we’re going to talk about some things high school students do. Here are some pictures of activities high school students like to do. I’m going to ask you some questions about these pictures. OK? . . . What are these students doing? [pointing]
10	P	Soccer? Xxx play soccer? Están jugando fútbol [laughs].	Soccer? Xxx play soccer? They’re playing soccer [laughs].
11	S	OK: And [pause] do you like to play soccer?	Do you like to play soccer?
12	P	Yes [nods]
13	S	OK. So what is he’s doing?	What is he doing?
14	P	Mhm I don’t know [laughs] Cómo se dice nadar?	Mhm I don’t know [laughs] How do you say “swim”?
15	S	OK, swimming.
16	P	Swimming
17	S	Mhm. Do you like to swim?	Do you like to swim?
18	P	Mhmm, poquito [eye contact, small, and moves hand so-so]	Mhmm, a little [eye contact, small, and moves hand so-so]
19	S	OK OK so what are they doing? [pointing]	What are they doing?
20	P	OK [looks to her booklet] Cuál?	OK [looks to her booklet] Which one?
21	S	In this picture [points]
22	P	Oh este. [pause] Cine	Oh, this one [pause] Movie theater
23	S	OK	OK
24	P	Mhm [looks to Silvia’s question book and reads question] Que qué es lo que veo en la imagen?	Mhm [looks to Silvia’s question book and reads question] What is it that I see in the image?
25	S	Oh, well. Just concentrate on the pictures
26	P	Mhm
27	S	And if you understand what I say. OK Um:: So the question is what are they doing in this picture? [looks at picture, making circular motions with her pen over it]
28	P	[uses fingers to point to eyes] Viendo TV? Si? [looks at Silvia for approval, but she doesn’t show it] No [laughs].	[uses fingers to point to eyes] Watching TV? Yes? [looks at Silvia for approval, but she doesn’t show it] No [laughs].
29	S	That’s fine. [both laugh] . . . What else do you see in this picture [looks at picture, making circular motions with her pen over it]? . . . On THESE pictures.	What else do you see in this picture [looks at picture, making circular motions with her pen over it]? . . . On THESE pictures.
30	P	Todas? [moves hands over all pictures]	All of them? [moves hands over all pictures]
31	S	Mhm [nods]
32	P	Um:: No, no le entiendo.	Um:: No, I don’t understand.
33	S	Ok, . . . So let’s move and do the next part, OK? [motions with hand]

This excerpt highlights how resourceful and communicative Paolo is as a test taker. This is particularly notable in light of the bilingual environment created by the test administrator (S). After nearly 6 min of chatting in Spanish, she “flips the switch” to English, and follows the monolingual English W-APT script. Paolo, however, continues to operate in a bilingual frame, answering some questions in both English and Spanish and seeking clarification in Spanish, for instance, “Mhm I don’t know. Cómo se dice nadar?” (turn 14), “Viendo TV? Si?” (turn 28), and “Todas?” (turn 30).

This presents a challenge to Silvia, as she clearly understands but is constrained by test protocol. She negotiates this compromise by responding to his Spanish questions in English (e.g., offering English word for “swim,” and clarifying which picture in English and gestures [turns 15, 21, 23, 29]). Although later in the transcript, she uses small amounts of Spanish, mostly for praise and re-direction (e.g., No te preocupes [translation: Don’t worry.]). Although only English-language responses are counted within the W-APT scoring rubric, Paolo’s use of Spanish indicates not only his comprehension of some of the materials but also his strong overall communicative skills.

Also notable here is Paolo’s literacy and general school skills. Although he received the lowest possible score for his writing and reading in the W-APT, there were many indications of his strong literacy skills. Most obvious were his attempts to read the test administrator’s script and questions. In Turn 24 here, and again later in the test, Silvia re-directs his gaze away from her printed script toward the student test materials. By Turn 221 in the exam, Silvia moves her chair and holds the paper at an angle that he cannot see.

Paolo struggled for nearly 10 min with the party invitation task, only partially completing this; however, he was able to write quickly and fluently in Spanish about his family, with complete sentences and mostly standard punctuation:

Mi padre se llama Paolo [last name]. Mi madre es Rosa [last name]. Tengo un hermano mas pequeño que yo llamado David tiene 12 años de edad. el esta estudiando La Secondaria y por circuntencias Familiares mis padres no viven Juntos pero si tenemos una buena communication.

He wrote this rapidly and without careful editing. Although there were non-standard features (e.g., uneven punctuation and spelling), he clearly had a strong command of basic literacy skills in Spanish. Yet, because of his lack of transcripts, overall W-APT score (1), and age (17), Paolo is placed as a ninth grader in the city’s newcomer high school.

Hamse: Testing Transfers

Hamse is an 18-year-old who arrived in the United States in January of 2014. He enrolled that same month in one of the city’s international charter schools that serves mostly recent arrivals, and attended that school for a year and a half. He now wishes to transfer to one of the city’s comprehensive high schools, where he says he has more friends. He reports that he attended school in Kenya regularly prior to coming to the United States. Because he is a transfer student with Somali as his home language, he is required to take the W-APT. His test administrator is a highly experienced former ESL teacher who works on a contract basis, Betsy. They chat in English as they walk back to the testing room.

Hamse is able to talk about concrete items (swimming, soccer, boats) with ease, but struggles to identify and explain what is a fact and what is an opinion, and is given a “3” on the speaking portion of the text. They move onto the listening portion; the first section involves matching the test administrator’s spoken text to one of three line drawings. The test administrator reads the script quickly:

Look at the big picture. It shows a fire truck in front of a school building. I am going to say what a teacher might tell you to do for a fire drill. Look at the top of the next page. Number 1. When you hear the alarm, line up by the door so we can go outside together. Which picture shows what you should do?

Hamse must then choose between three line drawings depicting (A) a boy with a backpack walking down hall, (B) a group of students by a door, and (C) an adult pulling a fire alarm. He correctly chooses B.

Betsy moves on: “Number 2. Now we will line up on the lawn. After you hear the principal say ‘All clear,’ we will return to our classroom. Which picture shows what you should do after you hear ‘all clear?’” Hamse must choose between pictures of (A) five students sitting at a table; (B) three students lining up by what might be a wall, trailer, or bus; (C) three students lining up by a door; and (D) five students standing around teacher by classroom door. Hamse, like many of the students we observed, does not choose the correct answer (C). He does, however, earn enough points in this section to move on to Part B, where he also only misses one of the three (synonym for “gargantuan”). This is sufficient to move on to Part C. Here, he also only misses one of the three (“What is a similar function for flowers and cones?”), and for Part D, he also only misses one of the three (“What is the outcome for Cuba and for Puerto Rico at the end of the Spanish-American war?”). At the highest level of the listening test (Part E: Babysitting), students are asked to solve problems around feeding a baby, including converting gallons to ounces, adding and subtracting fractions, and algebraic equations. These questions, like some in the reading portion, demand both English listening and academic math skills. Hamse is unable to answer any of these correctly, and gets eight of the 15 questions correct on this section overall, earning at raw score of 2 for listening.

For the reading portion of the test, Hamse successfully answers questions demanding interpretation of a web address, pie chart, and table of contents. However, he misses questions that ask him to interpret a line graph, to extrapolate from an advertisement, and to calculate travel speeds. Hamse gets seven of the 15 questions correct on this section overall, earning at raw score of 1 for listening. He completes his writing assessment fluently and quickly, writing, “Dear: Risalli, Mali, I have a happy birthday part on friday at 8.30 AM. Have you can come to my part and have a fun with me. Please let me know if you want a ride. Thanks. Sincerely, Risalli, Mali.” This receives a score of 3. His writing on his native language assessment is also fluent, with neat script, clear word boundaries, and mostly regular use of capitalization, writing additional text below the lines on form when describing his family. Hamse’s overall adjusted W-APT score is 2.35. Because of his age (18), he is assigned to the newcomer high school that serves older teens.

Hamse’s W-APT performance and score illustrate how challenging the test is for beginning- to intermediate-level students. There are several dimensions here. First, certain items seem problematic. For instance, the “return to our classroom” question measures students’ ability to make inferences from ambiguous line drawings as much as their aural listening skills. Second, many aspects of the language test actually assess content knowledge in academic areas such as math. Students’ English assessment is thus dependent upon their formal math skills. Moreover, the scoring is set up in such a way that missing even a few questions means students receive a 1 or 2. Because students who get anywhere from one to eight correct, all receive a score of 1, there is little to no differentiation at the beginning, entry, or emerging levels. For instance, all five of the students profiled here, including long-term English language learner Cristina (below), earned fewer than nine correct on the reading portion and thus received a raw score of 1 on the reading test, despite having widely variable literacy skills.

Cristina: Long-Term English Language Learner

Cristina is 14 years old and nervous. This is her second time this week at the Center. There were not enough test administrators on-site for her to complete the W-APT on Monday, the first day of school, when her family came for the first time to the Center to register. Cristina moved from Morelos, Mexico to Brooklyn, New York when she was 6. She attended school for 1 year in Mexico, and 9 years in the United States. Because one of her home languages is Spanish, and she is new to the district, she is required to take the W-APT. She is assigned an experienced contract tester and former ESL teacher, Karen, who walks her to the back testing room and chats with her in English along the way.

Cristina is one of the very few students we observed who scored a 6 on the speaking portion of the test. She was able to provide meaningful and fluent answers to questions such as “How do you tell the difference between a fact and an opinion?” Cristina explained, “Um, an opinion is uh, it’s your, like something you believe? Something you say? And a fact is something real life and everybody knows it’s true.” For the listening component, Karen skipped the practice portion of the test and moved directly to the “fire drill” section, which requires students to match spoken text to pictures (Part A), as described above. Like Hamse, Cristina misidentified the “return to our classroom” drawing, but had answered enough items correctly to continue to next sections. In Part D, on the Spanish American War, she gets two out of three incorrect. Here, students need to match the administrator’s extended spoken text with sentence-long statements and a map. Cristina correctly answers the map question but misses the two questions that require reading, earning a total of 8 points on this section and raw score of 2 for listening. This sort of large disparity between speaking and listening skills was common, according to test administrators, and is perhaps due to the fact that the listening component of the test demands substantial reading.

In line with this explanation, Cristina found the reading section more challenging. Although she was able to answer all of Part A correctly (e.g., reading and making inferences about a website name, pie chart, and graph), the next section (Part B), analyzing the tables of contents of biographies of Michelangelo, Van Gogh, and Monet, is much harder for her. She gets only one of three correct, giving her a raw score of 1 for reading.

For the final writing section, planning a special event, she wrote, “Dear Kathy: Hey there, my little cousin is having his 1st year birthday party. I would love if you can come to the party on July 15 at the park at 2:00pm. I hope you can make it. Love, Cristina.” She scored a 3, which according to WIDA characterizes writing that “shows an attempt at organization of sentences into paragraphs. The student’s intent is evident and generally successful. Although the sentences may be more complex, they may be unrelated or not cohesive” (p. 33). She wrote out this invitation quickly and fluently, in marked contrast to her writing on her native language literacy assessment, which she struggled to comprehend, complete, and answered mostly in English. For instance, for the last question of the native language assessment asking her to write about her family, she printed, “I really don’t know what to say about them. Mi familia es (scratched out). Tengo 2 hermans 1 hermanas.” Because of her age (14) and W-APT score (2.6), she is placed in a mainstream high school as a 10th grader and eligible for EL services.

Cristina’s overall W-APT score of 2.6 was between “emerging” and “developing” on the WIDA Scale. Her testing experience raises several issues concerning the idiosyncrasies of the test. For instance, some of the materials, and drawings in particular, are low-quality and poorly designed (e.g., hard-to-decipher line-drawn visuals; paper copies in binders, which do not lie flat). These features potentially affect both validity and reliability. Furthermore, the nature of the scoring system collapses a very wide range of performances into Categories 1 and 2. For instance, Cristina, who could understand and explain a range of academic topics, only scored a 2 on listening, in part because the band for Category 1 (0-7 correct) is so wide. This score is also a reflection of the fact that the listening portion of the test demands substantial reading. Moreover, Cristina, who could read a graph, interpret a web address, and answer questions about pie chart, scored a 1 on reading, the same score as students profiled here who never participated formal schooling. As we discuss below, this suggests limitations in what the W-APT can (and cannot) tell us about ELs.