Enhancing the Interpretive Reading and Analytical Writing of Mainstreamed English Learners in Secondary School

Abstract

In this study, 72 secondary English teachers from the Santa Ana Unified School District were randomly assigned to participate in the Pathway Project, a cognitive strategies approach to teaching interpretive reading and analytical writing, or to a control condition involving typical district training focusing on teaching content from the textbook. Pathway teachers learned how to use an on-demand writing assessment to help mainstreamed English learners understand, interpret, and write analytical essays. In Year 2, treatment effects were replicated on an on-demand writing assessment (d = .67) and showed evidence of transfer to improved performance on a standardized writing test (d = .10). The results underscore the efficacy of a cognitive strategies reading/writing intervention for mainstreamed English learners (ELs) in the secondary grades.

Keywords

reading and writing instruction English learners secondary education

In a darkened auditorium at Godinez High School, a small group of 11th and 12th graders in the Pathway Project are waiting to be called up on stage to take their places in the spotlight. Some are nervously rehearsing their lines; others are all smiles and counting the minutes to show time. As Mr. Patrick, the emcee for Parent Night and principal of Carr Intermediate School, calls out the numbers of selected raffle tickets in English and Spanish and excited siblings rush forward to receive door prizes—UCI T shirts, bookstore gift cards, and the like—students carry a large table and six chairs onto the stage. As the last door prize is retrieved, Mr. Patrick announces, “Ok. Our next presentation is from students at Segerstrom High School. So, Segerstrom, you’re up.”

Six students in medieval garb assume their positions at the table as several others stand to the side with placards and two step forward to the microphone:

“Good evening,” one begins. “The Pathway students at Segerstrom would like to act out what goes on in the mind of a reader who has to make sense of a work of literature.”

“So tonight,” the other student adds, “we will perform Act III: Scene 4 of Macbeth and demonstrate the strategies we used to understand Shakespeare’s play. The setting is a banquet hall in the palace where Macbeth is hosting a feast for his Lords.”

Macbeth:

You know your own degrees; sit down: at first and last the hearty welcome.

Lords:

Thanks to your majesty.

Macbeth:

We will mingle,

And play the humble host

Our hostess keeps her seat, but in time

We will require her welcome.

First Murderer appears at the door.

Macbeth:

There’s blood on thy face.

First Murderer:

’Tis Banquo’s

Macbeth:

’Tis better thee without than he within.

Is he dispatch’d?

As Macbeth (aka, a student) pauses, two students dressed as pages step forward carrying placards.

Page #1: (Holding up the Tapping Prior Knowledge placard):

I already know that “’Tis” means “it is” and “thee” means you, but I’ve got to translate the rest of this…(Thinking, hesitating) It is better to have blood on your face than that Banquo have blood flowing through his body.

Page #2: (Holding up the Summarizing and Clarifying Understanding placards):

So Macbeth is saying he’s glad Banquo is dead. The next line, “Is he dispatch’d” must mean “Is he dead?”

These students from Segerstrom High School were not simply dramatizing Shakespeare’s play. Collaborating with their teachers, they wrote this skit explicitly to demonstrate the cognitive strategies they had been taught to access on a daily basis in their classroom as readers and writers.

What Is the Pathway Project?

The Pathway Project is a long-term partnership between the UC Irvine site of the National Writing Project (UCIWP) and the Santa Ana Unified School District (SAUSD), the fifth largest school district in California with one of the largest Latino student populations in the state (98.5%), where 75% of the students are classified as being at the poverty level, and 88% are English learners (ELs). Teachers in the Pathway Project learn how to integrate cognitive strategy instruction and process writing to develop students’ text-based interpretive reading and analytical writing abilities. Results from our earlier study (Olson & Land, 2007; Olson, Land, AuBuchon, & Anselmi, 2010) suggest that integrating strategy instruction within a text-based approach to analytical writing can also enhance ELs’ writing ability.

The Pathway Project is an intensive 46-hour professional development intervention in which secondary English teachers are trained to improve mainstreamed ELs’ interpretive reading and text-based analytical writing abilities by: (1) using a cognitive strategies approach to reading and writing instruction, (2) instructing students to revise a pretest on-demand writing assessment into multiple draft essays, and (3) receiving ongoing support from experienced Pathway teachers who serve as coaches to the teachers in the experimental condition. We conducted a randomized controlled trial in SAUSD to test the efficacy of a cognitive strategies-based intervention designed to enhance students’ interpretive reading and analytical writing abilities. The first year of the field trial took place during the 2007–2008 school year and the second year took place during the 2008–2009 school year. Our first year results (Kim et al., 2011) indicated that the Pathway Project intervention improved students’ analytical writing ability (d = .35) and scores on the California Standards Test in English language arts (d = .07). In this article, we examine whether Pathway teachers who were trained for a second consecutive year were able to replicate positive impacts with a new cohort of approximately 2,000 Grade 6 to 12 students who entered the study in Year 2.

Why Teach Interpretive Reading and Analytical Writing to Mainstreamed Secondary English Learners?

Currently, ELs are the fastest growing segment of the K–12 student population, with the largest increases occurring in Grades 7 through 12 where ELs grew by 70% between 1992 and 2002 (Hoffman & Sable, 2006; Kindler, 2002). At its most general level, the term English learners refers to students whose limited proficiency in English result in their having difficulty learning content in English. Recent estimates indicate that more than 10% (5 million) of all school age children are ELs (Fox, 2007). Although ELs in the United States speak more than 350 languages, 77% speak Spanish as their first language (Hopstock & Stephenson, 2003), 40% have origins in Mexico (Hernandez, Denton, & Macartney, 2008), and 60% of ELs in Grades 6 through 12 come from low-income families (Batalova, Fix, & Murray, 2005; Capps et al., 2005). At the same time that EL enrollments have increased in U.S. public schools, researchers and policymakers have highlighted large literacy gaps based on students’ English language proficiency. Results from the National Assessment of Educational Progress (NAEP) indicate that the poor reading and writing performance of ELs in the middle grades persists through high school. On its most recent administration of the NAEP, ELs scored over 1 standard deviation below native English-speaking students on the NAEP reading and writing test in Grades 8 and 12 (National Center for Education Statistics, 2007).

Many content teachers of ELs avoid teaching their students to write analytical essays because they think the skills required are too sophisticated for the population they serve. Yet, 26 states have established high-stakes graduation exams that assess high-level reading and writing abilities (Horwitz et al., 2009). A study of prototype test items for high school exit exams across the nation (Wong Fillmore & Snow, 2003) reveals the degree of academic literacy expected of all secondary students, including ELs, who are assessed on their performance on a range of complex tasks, including: summarizing texts, using linguistic cues to interpret and infer the writer’s intentions and messages, assessing the writer’s use of language for rhetorical and aesthetic purposes, evaluating evidence and arguments presented in texts, and composing and writing extended, reasoned texts that are supported with evidence. For example, the California High School Exit Examination (CAHSEE) includes an on-demand essay in which students must demonstrate “a thoughtful, thorough, comprehensive grasp of text” in compositions that accurately and coherently provide specific textual details to support the thesis, use precise language, employ sentence variety, and contain few errors in the conventions of written English (California Department of Education, 2008a). On the 2010 administration of the CAHSEE, 81% of all students tested passed the exam, compared to 73% of Hispanic/Latino students and 42% of ELs (California Department of Education, n.d.).

Cognitive Strategies for Mainstreamed English Learners

Numerous reports from policy centers and blue-ribbon panels “implicate poor understandings of cognitive strategies as the primary reason why adolescents struggle with reading and writing” (Deshler, Palinscar, Biancarosa, & Nair, as cited in Conley, 2008, p. 84; Graham, 2006; Snow & Biancarosa, 2003). In fact, researchers have noted a “growing inequality” in classroom instruction where students designated as “honors students” are exposed to rigorous academic work designed to promote higher literacy, whereas low achievers, children of the poor, and second language learners often receive instruction that places a premium on the “transmission of information, providing very little room for the exploration of ideas, which is necessary for the development of deeper understanding” (Applebee, Langer, Nystrand, & Gamoran, 2003, p. 689). According to a recent Carnegie Corporation report, inadequate educator capacity and the limited use of research-based instructional practices prevent adolescent ELs from learning academic English and meeting content standards in English language arts (Short & Fitzsimmons, 2007). To address the needs of adolescent ELs, the report encouraged teachers to help ELs use cognitive strategies to understand, interpret, and write essays about complex text.

A number of instructional frameworks and recommendations support approaches that incorporate strategy instruction to advance ELs’ development of English (Francis, Rivera, Lesaux, Keiffer, & Rivera, 2006; Goldenberg, 2008; Schleppegrel, 2009). These frameworks extend the work of a wide body of research on what experienced native English-speaking student readers and writers do when they construct meaning from and with texts to ELs. They are built on the premise that both ELs and native English speakers benefit from the same types of high-quality teaching, including strategy instruction (August & Hakuta, 1997; August & Shanahan, 2006; Genesee, Lindhom-Leary, Saunders, & Christian, 2006; Goldenberg, 2008). Grounded in studies that demonstrate the efficacy of cognitive strategy use in reading (Block & Pressley, 2002; Pearson & Duke, 2002; National Institute of Child Health and Development, 2000; Paris, Wasik, & Turner, 1991; Tierney & Pearson, 1983; Tierney & Shanahan, 1991), the frameworks stress the importance of including modeling, scaffolding, guided practice, and independent use of strategies so that students develop the ability to select and implement appropriate strategies independently and to monitor and regulate their use (Block & Pressley, 2002).

In addition, the frameworks are grounded in recent research that indicates the efficacy of strategy instruction in writing (Graham, 2006; Graham & Perin, 2007; Troia & Graham, 2002). Like the frameworks designed for native English speakers, the EL frameworks suggest that reading and writing should be taught together because this engages students in a greater use and variety of cognitive strategies than do reading and writing taught separately (Tierney & Shanahan, 1991, p. 272). This exposure to and practice in an array of cognitive strategies promotes and enhances critical thinking (Tierney, Soter, O’Flahavan, & McGinley, 1989), enabling many learners who lack the linguistic resources required to access challenging text not only to comprehend the text but also to be able to respond to it critically. Research suggests that students need to purposefully select and orchestrate cognitive strategies that are appropriate for the literacy task at hand (see e.g., Flower & Hayes, 1981; Paris et al., 1991; Pressley, 2000). This means that learners must master a wide variety of strategies and understand when to select and implement a strategy as well as how to regulate its use.

Short and Fitzsimmons (2007) hypothesize that strategy instruction develops ELs’ English by providing them with an explicit focus on language, increasing their exposure to academic texts, making the texts they read comprehensible, giving ELs multiple opportunities to affirm or correct their understanding and use of language, assisting them in retrieving new language features and in using these features for academic purposes, and providing them with the means of learning language on their own, outside of class. They further hypothesize that adolescent ELs of an intermediate level of English proficiency and above have sufficient proficiency to benefit from strategy instruction (Echevarria, Short, & Vogt, 2008; Short & Fitzsimmons, 2007). These students have automatized lower-level reading and writing skills. They possess the language proficiency required to use cognitive strategies that will provide them access to the higher order cognitive reading and writing tasks that they will encounter when they have mainstreamed into regular content instruction.

Because mainstreamed ELs are learning academic content while they are learning the language in which the content is taught, most ELs generally need more instruction than their native English-speaking peers in order to perform well on high-stakes exams (e.g., Schleppegrel, 2009; Short & Fitzsimmons, 2007). In particular, they require guided practice in reading and forming interpretations about complex texts, conveying those interpretations in well-reasoned essays, and mastering writing conventions in English (August & Shanahan, 2006; Francis et al., 2006; Goldenberg, 2008). Explicitly teaching cognitive strategies to ELs can help them obtain this practice. However, many factors such as previous schooling, knowledge of a first language, access and exposure to text, opportunities to write for a variety of audiences and purposes, and motivation also contribute to ELs’ development of reading and writing (Meltzer & Hamann, 2005; Valdés, 2002) and should be considered along with the instruction of cognitive strategies when designing instructional programs for ELs.

To summarize, then, research suggests that ELs might require more explicit instruction in strategy use than that of their native-speaking peers in learning such features as academic vocabulary, literary themes, symbolism, and writing conventions (August & Shanahan, 2006; Francis et al., 2006; Gersten, Baker, Haager, & Graves, 2005; Goldenberg, 2008). What is needed are carefully designed studies of the efficacy of cognitive strategies approaches, particularly in secondary, urban settings.

Theory of Change and Study Goals

The theory of change linking the Pathway Project professional development activities to student outcomes flows from existing research on effective teacher professional development programs designed to improve adolescent literacy outcomes. Among the 15 elements of effective adolescent literacy programs (Biancarosa & Snow, 2004), 3 are most critical to improving student outcomes: (1) ongoing and sustained professional development to improve teacher practice, (2) the use of pretest student data to inform instructional activities, and (3) the use of summative outcomes to evaluate efficacy. Figure 1 describes the logic model guiding the Pathway Project intervention (Rossi, Lipsey, & Freeman, 2004).

Figure 1.

Logic model for the Pathway Project.

Our study was designed to address the following research questions: (1) To what extent will teachers’ participation in the Pathway Project professional development intervention over 2 years improve academic outcomes for mainstreamed Latino ELs on an on-demand writing assessment and the California Standards Test in English language arts? We hypothesized that Pathway teachers who remained in the study for a second year would have a positive impact on students’ text-based analytical writing ability and the California Standards Test (CST) in English language arts of a second cohort of students. (2) Do students in all classrooms taught by Pathway teachers improve their performance on the California Standards Test (English language arts) compared to students in all classrooms taught by control teachers? We hypothesized that treatment effects on the California Standards Test (CST) in English language arts would generalize to all classrooms taught by Pathway and control teachers, not only the one classroom that was included for the Year 2 impact evaluation.

Method

Participants

District context

The study was conducted in 9 middle schools and 6 high schools in the Santa Ana Unified School District. In 2007–2008, the SAUSD enrolled 57,061 students in Grades K to 12, was the fifth largest school district in California (Education Data Partnership, 2010), and had the greatest percentage of non-White students (96%) and low-income students (78%) in Orange County. A total of 9 of the 15 secondary schools had failed to meet state and federal accountability goals and were identified for program improvement.

Study design

In our multisite cluster randomized field trial, secondary schools were the sites, teachers were clusters randomly assigned to the Pathway Project, and students were randomly assigned to classrooms. Our study was designed to improve power by creating school by grade randomization blocks and then randomly assigning classroom teachers within each block to experimental conditions.¹ Because of resource constraints, we did not collect student data and observe lessons for all classrooms taught by each teacher in our study. Thus, students in our study were sampled through a two-step process. First, SAUSD employs a software program to randomly assign eligible students (i.e., students meeting California English Language Development Test [CELDT] criteria for being mainstreamed) into English classes where lessons are designed for native English speakers. Second, we selected one English language arts classroom to include in the study. Most English teachers, however, are responsible for teaching multiple sections, including classes for mainstreamed ELs and remedial and English language development classes for students who score below grade level and have very limited English proficiency (e.g., CELDT scores of 1 or 2). The CELDT is a measure of English proficiency and was used to help identify classrooms with mostly intermediate or above students. In the one classroom selected for the study, teachers administered an on-demand writing assessment at pretest and posttest, coaches provided support in helping teachers integrate Pathway activities into the curriculum, and raters observed two lessons during the school year. To reduce potential threats to internal validity, teachers in the control group were given resources (e.g., classroom library books) and received the Pathway professional development in the third year of the study (2009–2010).

Teacher participants in the experimental sample

In summer 2007, teachers in the 15 secondary schools were recruited to participate in the study and randomly assigned to the Pathway Project or control condition. Each participating teacher was paid a $1,000 stipend to complete all research activities. Both Pathway and control teachers received a portion of their stipend upon the completion of teacher surveys, the administration of the on-demand writing assessment, and the completion of a classroom observation. We administered a baseline survey to obtain background information on participating teachers. On average, teachers in our study had 14.34 years of total teaching experience, and 64% had earned a master’s degree. Although teachers received their baccalaureate degrees from over 25 different undergraduate institutions, a plurality (42%) graduated from a California State University. There was no statistically significant difference between Pathway teachers and control teachers in the total years of teaching experience (p = .54), the percentage who earned a master’s degree (p = .15), and the percentage who graduated from a California State University (p = .82). These findings indicate that participants in both conditions were similar on observed teacher characteristics measured at baseline in fall 2007.

During the 2007–2008 school year, California experienced a severe budget crisis, causing the district to lay off teachers included in this study. As a result, we lost 18 treatment and 13 control teachers in 2008–2009 to attrition. Chi-square analyses revealed no significant difference in attrition rates by condition, χ²(103) = 1.30, p = .26. Furthermore, we found no difference between Pathway teachers (M = 7.00, SD = .89) and control teachers (M = 7.21, SD = .92) on a baseline measure of teacher self-efficacy (p = .29) and no difference in self-efficacy between teachers who remained in the study (M = 7.07, SD = .95) and teachers lost to attrition (M = 7.13, SD = .89). In addition, there was no evidence of differential attrition of teachers with particularly high or low self-efficacy scores by condition. Among Pathway teachers, there was no difference in teacher efficacy between teachers who remained and left the study, and the same was true for control teachers. In sum, although 30% of the teacher sample was lost to attrition from Year 1 to Year 2, there was no systematic difference in attrition rates between conditions and no evidence that teachers who remained in the study were different from those who remained on a baseline measure of teacher self-efficacy.

Student participants’ English language proficiency

In California, the California English Language Development Test was used to determine the English language proficiency of children whose primary home language was not English. The CELDT measures each student’s ability in reading, writing, speaking, listening, and comprehension. Performance on each subtest is aggregated to an overall CELDT score, which yields five performance levels: beginning (1), early intermediate (2), intermediate (3), early advanced (4), and advanced (5). In SAUSD, students scoring early advanced or advanced on the CELDT as well as students scoring intermediate and at the mid-basic level on the California Standards Test in English language arts are mainstreamed into regular English language arts classrooms. In our sample, approximately 88% of the students met criteria for being mainstreamed into regular ELA classrooms.

Comparison of Pathway and control classrooms at baseline

Shown in Table 1 are descriptive statistics for Pathway and control classrooms at baseline in Year 1 of the study (fall 2007). There was no statistically significant difference between the two groups on the California Standards Test or in the proportion of students who were Latino/a, English learners whose primary home language was Spanish, and eligible for free- or reduced-price lunch. In the Year 2 sample (2008–2009), there was also no difference between conditions in the percentage of Latino students (Pathway = 94%, control = 95%), the percentage of English learners whose primary home language was Spanish (Pathway = 86%, control = 87%), and the percentage of students eligible for free- or reduced-price lunch (Pathway = 80%, control = 84%). In addition, there was no statistically significant difference on the CST between students in Pathway classrooms (M = 324.22, SD = 18.10) and control classrooms (M = 322.86, SD = 22.52), t = 0.27, ns.

Table 1

Baseline Comparison of Pathway and Control Classrooms on Demographic and Achievement Variables

	Control (n = 51)		Pathway (n = 52)
Variable	M	SD	M	SD	t	p
California Standards Test (English language arts)	321.70	22.21	320.95	19.61	0.180	0.858
Male (%)	0.50	0.11	0.48	0.08	0.816	0.417
Latino/a (%)	0.95	0.08	0.95	0.06	−0.257	0.798
English language learner whose primary home language is Spanish (%)	0.88	0.11	0.87	0.11	0.770	0.443
Eligible for free- or reduced- price lunch (%)	0.79	0.12	0.78	0.13	0.676	0.501

Note. California Standards Test (CST) scores are scaled scores from the English language arts CST from spring 2007.

Final Year 2 sample for student achievement analysis on posttest measures

The final Year 2 sample in 2008–2009 (n = 72) was smaller than the baseline Year 1 sample in 2007–2008 Year 1 (n = 103) due to teacher attrition. As a result, the final sample size for the Assessment of Literary Analysis (ALA), our on-demand writing assessment, was 72: 34 Pathway classrooms and 38 control classrooms. To adhere to budget constraints, we scored posttest on-demand essays for a random sample of approximately 50% of the students within each class. In addition, because 12th-grade students do not take the CST, our analyses of the CST did not include the 12th-grade classrooms that were included in the randomization at the beginning of the study. In the Year 2 experimental sample, all impact analyses of posttest ALA and CST scores were based on 66 classrooms, including 31 Pathway classrooms and 35 control classrooms.

Procedures

Description of typical professional development activities

Control teachers received a total of 26 hours (3 days, 6 hours prior to the first day of the school year and 8 days, 1 hour per day of professional development during the school year) that focused broadly on interpreting test data, using test data to improve schools’ CST scores, helping students improve their summarizing strategies during reading activities, forming professional learning communities (DuFour & Eaker, 1998; Marzano, Pickering, & Pollock, 2001), and understanding the teacher’s guide for the Holt Elements of Literature textbook series, Grades 6–12. The curriculum for all English teachers is aligned with the district pacing guide and is focused on the teaching of literature and informational texts, vocabulary, reading strategies, grammar, and writing strategies. Students take quarterly multiple choice reading comprehension tests related to their textbook but only one district writing proficiency test per year. All teachers are expected to assign one extended piece of writing every 6 weeks. The textbook contains instruction in cognitive reading strategies and embeds questions during and after reading selections to facilitate students’ comprehension. The textbook also includes some limited writing process lessons. Because the Holt textbook focused more on cognitive strategies in reading than writing, the strongest contrast between the Pathway professional development activities and the business-as-usual professional development was in the amount of additional time that Pathway teachers spent learning how to implement writing strategies. Finally, there were no coaches to support control group teachers in curriculum and instruction.

Description of the Pathway Project intervention

In addition to 26 hours of typical professional development that all district secondary English teachers received, experimental teachers in the Pathway Project participated in 46 hours of training (via six 6-hour released days during the school year and five 2-hour afterschool sessions) focused on methods for helping mainstreamed ELs to develop the academic literacy necessary to meet state English language arts and English language development content standards, with special emphasis on interpretive reading and analytical writing. These included: literary response and analysis, comprehension and analysis of informational nonfiction texts, and development of clear, coherent, focused essays. Training was led by the developers of the Pathway Project from the UC Irvine site of the National Writing Project and supported by literacy coaches who participated as treatment teachers in a previous quasi-experimental research study (Olson & Land, 2007). There are three core components of the Pathway Project: (1) training in the use of the cognitive strategies tool kit and curriculum materials, (2) intervention activities focused on the revision of the pretest on-demand writing assessment into a multiple draft essay, and (3) coaching from a more experienced, veteran teacher previously trained in the Pathway Project on how to integrate a cognitive strategies approach into the existing English language arts curriculum.

Cognitive strategies tool kit and curriculum materials

Strategy instruction in this study occurred within the context of teaching reading and writing as a process and involved pre-reading, during reading, and post-reading activities as well as prewriting, planning, drafting, sharing, revising, and editing activities. First, teachers were introduced to a model of the cognitive strategies that make up a readers’ and writers’ tool kit shown in Figure 2. Teachers used the following analogy to explain the tool kit to students:

When we read, we have thinking-tools or cognitive strategies inside our heads that we access to construct meaning. Researchers say that when we read, we’re composing, just as when we write. What they mean is that while we read, we’re creating our own draft of the story inside our heads and as we keep reading and come across something we didn’t expect to happen or suddenly make a big discovery about what something means, we start on a second draft of our understanding. So, when you think of yourself as a reader or writer, think of yourself as a craftsman, skilled in making things with your hands, but instead of reaching into a metal tool kit for a hammer or a screwdriver to construct or build tangible or real objects you can actually see, you’re reaching into your mental tool kit to construct meaning from or with words.

Figure 2.

Cognitive strategies: Reader’s and writer’s tool kit.

Students also received bookmarks as well as 8 ½ × 11 copies of cognitive strategies sentence starters in English and in Spanish (to take home to their parents) that illustrate what goes on in the mind of a reader or writer in the act of meaning construction. For example, a sentence starter for revising meaning is “At first I thought—but now I…,” and a starter for reflecting and relating is, “So, the big idea is….” To build students’ declarative knowledge of what cognitive strategies are, teachers presented scaffolded lessons we called “tutorials” (Bruner, 1978) in which they introduced each of the tools in the tool kit to students within the context of reading and writing about high interest literary and nonfiction texts. To enhance their procedural knowledge of how to implement the strategies, students received instruction on how to make marginal annotations and keep dialectical journals in which they not only used their bookmarks to guide them in responding to texts but indicated which cognitive strategies they were accessing in their written responses. Finally, to foster conditional knowledge of when to use a cognitive strategy, which strategy to use, and why, students were taught to think aloud in response to complex texts while a partner recorded their responses and then to label their strategy use as well as to write metacognitive reflections describing the cognitive strategies they used in order to form interpretations about texts and write analytical essays. The skit the students from Segerstrom High School wrote and performed at Godinez High School to demonstrate the cognitive strategies they used to interpret Macbeth demonstrates their strong command of declarative, procedural, and conditional knowledge (Paris, Lipson, & Wixson, 1995) as they applied what they learned independently.

Formative assessment and revision of pretest

Second, teachers learned how to use results from a pretest on-demand writing assessment to provide instruction in text-based analytical writing. To that end, professional development focused on preparing students to read, make inferences, and form interpretations about complex literary texts and to convey interpretations in thoughtful, well-organized essays that present a clear thesis supported with appropriate textual evidence. The centerpiece of the interpretive reading and analytical writing intervention was an extensive set of materials focused on the revision of students’ pretest writing assessment (a literature-based analytical essay) into a multiple draft essay. Student performance on this timed, on-demand pretest essay was used to inform the intervention as teachers engaged in analyzing students’ work and identifying students’ strengths and areas for growth (Biancarosa & Snow, 2004; Black & Wiliam, 1998).

Based on the teachers’ analysis of students’ pretest essays on theme in 2008–2009, lessons were designed to address students’ needs. We did not want to overly influence students’ own deeper thinking about the two works of literature we used for our pre/post Assessment of Literary Analysis, “The Scarlet Ibis” or “The Medicine Bag,” as they revised their pretest essays on these short stories. Therefore, we selected “What Do Fish Have to Do With Anything?” by Avi (1997), as the “training” text for the lesson sequence on theme. This coming-of-age story about the interactions between a young boy living with his mother, who is suffering from depression, and a homeless man camped out in front of their apartment complex lends itself well to an exploration of theme because of the differing perspectives of all three characters.

To help students distinguish between a topic and a theme, teachers first provided the detailed explanation described in the following:

A story’s theme is different from its topic or subject. The topic is simply what it’s about. The theme is the author’s point about a topic. Think of a topic as the What of the story and the theme as the So what? To identify a theme, sometimes it helps to brainstorm a list of topics or big ideas in a story. Common topics for themes that you’ll find in stories are usually abstract nouns that deal with human relationships and include terms like belonging, courage, family, friendship, hope, identity, prejudice, respect, revenge, and trust. A theme statement must be a complete sentence that states the author’s message about life or about human relationships. A good theme statement applies to people in general, not just to the specific characters in the story. Here are some examples of theme statements:

Growing up means taking responsibility for yourself.

It is important to accept people for what they are on the inside and not the outside.

The distinction between the topic as the What of the story and the theme as the So what? seemed to turn on a light bulb for many students. Students were asked to reread the text and note any topic words that they felt captured the big ideas of “What Do Fish Have to Do With Anything?” and developed theme statements for the training text such as “If you close your eyes, you close your heart” or “All you need is love; don’t hold back on it.” Then, they returned to their pretest literature and developed sample theme statements to focus on in their revised essays.

Many struggling readers and writers, especially ELs who have had little practice, think that the point of writing a literary response-based essay is to demonstrate that they understood what they read by retelling the story—and yet this type of response will only merit a 1 on the 4-point scale on the CAHSEE. To model effective and ineffective analytical writing on theme, we provided students with the two essays that are available at www.gse.uci.edu/uciwp. Teachers designated colors for three types of assertions that make up an analytical essay. For example, they might say,

Plot summary reiterates what is obvious and known in a text. Reiterate means to repeat in order to make something very clear. Plot summary is yellow because it’s like the sun. It makes things as plain as day. We need some plot summary to orient our reader to the facts, but we don’t need to retell the entire story. Commentary is blue like the ocean because the writer goes beneath the surface of things to look at the deeper meaning and to offer opinions, interpretations, insights, and “AH-HAs.” Supporting detail is green because it’s what glues together plot summary and commentary. It’s your evidence to support your claims.

Starting with the weaker paper, students color-coded each sentence almost entirely in yellow, whereas they coded a balance of yellow, blue, and green sentences throughout the stronger paper. Students then applied this color-coding strategy to their pretest essays to visibly see whether they had simple summarized or included interpretation and evidence and then revised their essays.

Coaching

The third core component of the Pathway Project involves coaching. Throughout their participation in the intervention, teachers received ongoing support from a veteran teacher at their school site who had previous exposure to the project during an earlier 8-year quasi-experimental study in SAUSD (Olson & Land, 2007). Pathway coaches attended professional development trainings along with their school team, convened five school site meetings, conducted informal nonevaluative classroom observations upon request, and assisted teachers in integrating interpretive reading and analytical writing instruction using the cognitive strategies approach into the lessons in their Holt textbook. Research indicates that when coaching is combined with professional development, teachers are more likely to implement innovations in their classroom (Buly, Coskie, Robinson, & Egawa, 2006; Joyce & Showers, 2002; Olson & Land, 2008).

Measures

Student demographic characteristics

The SAUSD research and evaluation office provided student-level data on student gender, ethnicity, eligibility for free- or reduced-price lunch, and English learner status.

California English Language Development Test

The CELDT is used to determine a student’s proficiency of English language skills in reading, writing, speaking, listening, and comprehension. Scores on each of the five subsections of the CELDT are aggregated to create a total scaled score. The scaled score is used to create five performance levels. Across grades, reliability coefficients on the CELDT ranged from .75 to .92 (California Department of Education, 2008c).

Assessment of Literary Analysis, Grades 6 to 12

The Assessment of Literary Analysis is an on-demand, writing assessment. In our Year 2 study, the ALA pretest was administered in October 2008 and the posttest was administered in May 2009. Students were prompted to explore the interactions between the characters and interpret the symbolism in either “The Scarlet Ibis” or “The Medicine Bag” and to write an analytical essay in which they developed and supported a theme statement about what the narrator learned by the end of the story. Essays were organized by classrooms and then randomly assigned to raters. Order effects were controlled by counterbalancing the administration of the two timed writing assessments across classrooms. Each rater scored essays holistically on a 6-point scale to assess the quality and depth of interpretation, the clarity of the thesis, the organization of ideas, the appropriateness and adequacy of textual evidence, sentence variety, and the correct use of English language conventions. Our 6-point scale was based on rubrics used to evaluate the essay portion of the California High School Exit Examination (California Department of Education, 2008a), the California STAR 7 Direct Writing Assessment (California Department of Education, 2008b), and the National Assessment of Educational Progress (ACT, Inc., 2007).

Our first-year results indicated that the ALA can be scored reliably on a 6-point scale, with 57% exact agreement between raters and 84% within 1-point agreement. In both Year 1 and Year 2, a third rater scored essays in which raters disagreed by more than 1 point. In particular, discrepancies were resolved by taking the average of the first two raters and then summing this number with the third rater’s score. For example, if the first two raters assigned a score of 2 and 4, and the third rater assigned a score of 3, the final score was 6.

California Standards Test, English language arts, Grades 6 to 11

The California Standards Test is a summative assessment designed to measure student mastery of the English language arts content standards. In Grades 6 to 11, each CST is timed (170 minutes) and includes 75 multiple-choice items. Reported internal reliability coefficients on the CST in English language arts in Grades 6 to 11 exceeded .90 (Educational Testing Service, 2009, p. 500). We also created two additional scores based on the reading and writing portions of the CST. The CST reports performance (i.e., number correct) for five subtests. The CST reading score included performance on three subtests: (1) word analysis, fluency, and systematic vocabulary development; (2) reading comprehension with a focus on informational text; and (3) literary response and analysis. The CST writing score included performance on two subtests: (4) written and oral English language conventions and (5) writing strategies. Reported reliabilities for each of the reading subtests ranged from .61 to .84; reliabilities for each of the writing subtests ranged from .74 to .85 (Educational Testing Service, 2009, pp. 506–507).

Fidelity of implementation measures

We used two measures to assess the fidelity of implementation: the Pathway Quality Checklist, an instrument specifically aligned with our intervention, and the Pathway Observation Measure, an adapted form of the Center for the Improvement of Early Literacy Achievement (CIERA) that was developed by Taylor, Pearson, Peterson, and Rodriguez (2005). In 2007–2008, we observed 86 of the participating teachers (treatment and control) for one class period (46 minutes). Due to budget constraints, we observed 28 Pathway classrooms and 26 control classrooms, which was a subsample of the 86 classrooms that were observed once in the Year 1 study. Each of these classrooms was observed twice in Year 2 (winter 2009 and spring 2009). We used the same five trained raters both years who were blinded to teachers’ status as treatment or control and double-observed one third of the observations for the purposes of reliability. Rater agreement (number of agreements divided by the total agreements and disagreements) was .97 and interrater correlations exceeded .86. In addition, the mean correlation between pairs of raters across observations was .87 for the reading codes and .86 for the writing codes.

Pathway Quality Checklist

In order to document teachers’ implementation of what intervention was used and how well they were delivered, we developed a Pathway Quality Checklist. This measure relies on the research base on effective literacy instruction, including principles that appear to enhance student learning across various models and curricula. In their study of primary grade reading instruction, Foorman and Schatschneider (in press) have noted wide variance in the quality with which teachers implement curricula but also found that scores on a Likert rating form were linked to literacy outcomes. The teacher-focused measure includes a 6-point Likert scale (1 = little or weak evidence to 6 = impressive evidence) on four items (alpha = .90) that the observer uses to rate the degree to which the teacher: (1) demonstrates knowledge of subject matter, (2) delivers a lesson that is appropriate for the needs of students, (3) engages students, and (4) uses language arts strategies that are consistent with the intervention. The student-focused measure includes the same 6-point Likert scale on three items (alpha = .87) that the observer uses to rate the degree to which the students: (1) exhibit command of reading and writing strategies, (2) are on task and engaged in the lesson, and (3) use strategies consistent with the Pathway Project intervention.

In our Year 1 impact study (2007–2008), Pathway teachers (M = 4.48, SD = 1.33) were rated as using more cognitive strategies consistent with Pathway professional activities than control teachers (M = 3.35, SD = 1.50), t = 4.01, p < .01. For the Year 2 study, Pathway teachers (M = 4.12, SD = 0.94) were more likely than control teachers (M = 3.5, SD = 0.79) to be observed delivering lessons appropriate to students (t = 2.73, p < .05). In addition, raters indicated that Pathway teachers (M = 4.03, SD = 1.12) were more likely than control teachers (M = 2.90, SD = 0.96) to use strategies consistent with Pathway professional activities relative to control teachers (t = 4.11, p < .01). Finally, raters indicated that students of Pathway teachers (M = 3.17, SD = 0.89) were more likely to demonstrate effective use of cognitive strategies compared with students of control teachers (M = 2.51, SD = 1.01), t = 2.58, p < .05.

Pathway observation measure

To address how much of the intervention was delivered (quantity) and how the intervention was delivered (process of implementation), we adapted a measure developed by Taylor et al. (2005) in consultation with Russell Gersten, Instructional Resources Group, who has designed a reliable measure of teachers’ literacy instruction for elementary reading instruction for ELs (Gersten et al., 2005) and with P. David Pearson, one of the developers of the CIERA. The CIERA measure has been used in a study of reading comprehension instruction and writing instruction Grades 1 through 5 to determine the influence of teacher practices that encourage cognitive engagement in literacy in high-poverty classrooms. The system provides five levels of information involving (1) major literacy focus, (2) specific literacy activity, (3) material, (4) teacher response, and (5) student response. Interobserver agreement was calculated on each level of coding. Reliabilities ranged from 82% to 95% with a median of 86%. The measure has demonstrated strong criterion-related validity in that specific codes (e.g., use of higher order questions, teaching of comprehension strategies) correlate with reading growth. To map the CIERA onto Pathway, we created two aggregated subsets of codes to identity both (a) Pathway-specific reading activities and (b) Pathway-specific writing activities.

Table 2 displays means and standard deviations on the Pathway observation measure across three time points for a subsample of teachers. The first observation, conducted in winter 2009, revealed no significant differences on the reading and writing observation measure. However, on the final observation, there was suggestive evidence (d = .38) that Pathway teachers (M = 1.68, SD = 2.18) devoted more time implementing Pathway-specific writing activities than control classrooms (M = .97, SD = 1.54), t(51) = 1.39, p = .173.

Table 2

Average Number of Intervals in Which a Pathway Specific Reading or Writing Activity Was Observed by Raters Among a Subsample of Year 2 Teachers

	Pathway			Control
Variable	M	SD	n	M	SD	n	t	p	d
Observation 1 (spring 2008)
Pathway writing	2.33	2.28	30	2.00	2.41	24	0.517	0.608	0.14
Pathway reading	2.60	2.37	30	2.75	2.13	24	−0.244	0.808	−0.07
Observation 2 (winter 2009)
Pathway writing	2.04	2.10	28	2.04	2.31	26	0.005	0.996	0.00
Pathway reading	2.66	2.22	28	2.50	2.02	26	0.284	0.777	0.08
Observation 3 (spring 2009)
Pathway writing	1.68	2.18	28	.968	1.54	25	1.39	0.173	0.38
Pathway reading	2.93	2.18	28	2.65	2.10	25	0.47	0.640	0.13

Data Analytic Strategy

In our multisite cluster randomized field trial, students were nested within classrooms, which were nested within grade by school randomization blocks (i.e., blocks). Since each teacher was responsible for teaching multiple sections of English, we selected one classroom to participate in the study and administered ALA posttests to students in this class. Thus, there was a one-to-one correspondence between teachers and classrooms. We emphasize that randomization occurred at the teacher/classroom level: Each teacher was randomly assigned to treatment or control. As such, we estimated treatment effects at the teacher/classroom level.

To address our first research question, we used a three-level hierarchical linear model to estimate the impact of the Pathway intervention on observed teaching practices and academic outcomes for mainstream ELs (i.e., ALA, CST English Language Arts), included a pretest classroom-level covariate (i.e., CST pretest mean) to improve the precision of the estimated treatment effect, and examined whether the treatment effect varies across the grade by school randomization blocks. To facilitate interpretation of the impact estimates, both the ALA and CST scores were normed within grade level to yield a standardized mean difference between Pathway and control classrooms. The fully specified level 1 equation is written as

Y_{i j k l} = μ_{j k l} + \in_{i j k l},

where Y_ijk is the outcome (e.g., posttest score) for student i in classroom j in block k, which has mean µ_jk. The student-level error terms, ∈_ijk, represent the deviations of the students’ scores from their classroom mean and are assumed to be independent normal variables with mean zero and variance σ². Model 1 describes Level 1 for a continuous posttest score (i.e., ALA or CST).

At Level 2, the mean of Model 1 is written

μ_{j k} = (β_{0} + α_{0 k}) + (β_{1} + α_{1 k}) β_{1} {(Pathway)}_{j k} + β_{2} {(Pretest Covariate)}_{j k},

where µ_jk, the posttest mean for classroom j in block k, is the sum of the block effect, β₀ + α_0k, a treatment effect for Pathway classrooms, β₁ + α_1k, and the pretest covariate with the effect, β₂. Here fixed effects are represented by β and classroom-level random effects by α. We included a classroom-level covariate and its effect, β₂, in Equation 2 to improve the precision of the estimated treatment effect on the two posttest outcomes (i.e., ALA and CST). Finally, at Level 3, we examined potential heterogeneity in the block and treatment effects. Thus at level 3, we formulate α_0k and α_1k as random effects,

(\begin{matrix} α_{0 k} \\ α_{1 k} \end{matrix}) ~ N_{2} ((\begin{matrix} 0 \\ 0 \end{matrix}), (\begin{matrix} τ_{0}^{2} & τ_{01} \\ τ_{01} & τ_{1}^{2} \end{matrix})),

where $τ_{0}^{2}$ and $τ_{1}^{2}$ are the variances of the block and treatment random effects, respectively, and τ₀₁ is their covariance. In our three-level hierarchical linear model, the key parameters of interest are β₁, the estimated fixed effect for the Pathway treatment on each respective student outcome, and the variance of its random effect indicating whether the treatment effect varies significantly across the randomization blocks. Our goal was to estimate treatment effects on our measure of text-based analytical writing (ALA) and interpretive reading (CST).

To address our second research question, we also explored the effect of the Pathway intervention on all of the students taught by a Pathway teacher (beyond those who are in the experimental classrooms). In this analysis, there was an additional level of hierarchy in the hierarchical linear modeling (HLM) or generalized linear mixed model (GLMM). In particular, students were nested within classrooms, classroom sections were nested within teachers, and teachers were nested within grade by school randomization blocks. Because the unit of randomization remained at the teacher level, we estimated the treatment effect at the teacher level. Thus, the extra level of hierarchy was added between Levels 1 and 2 of the models outlined previously. Level 1 described the variability of students within each classroom, and the added level described the variability of classrooms within each teacher. As in Model 1, we use σ² to denote the variance of individuals within classrooms and ω² to denote the variance of classroom section means corresponding to a particular teacher. We then used Model 2 to describe the variability of teachers within block as well as the treatment effects and finally Model 3 to describe potential heterogeneity in the block and treatment effects.

To formalize the model when we explore the effect of the Pathway intervention on all sections, we replace Equation 1 with

Y_{i j k l} = µ_{j k l} + \in_{i j k l},

where Y_ijkl is the posttest score for student i in classroom section l taught by teacher j in randomization block k, µ_jkl is the mean of students in this classroom section, and ∈_ijkl is the independent normal deviation of this student from the classroom section mean. The classroom section mean can then be written,

µ_{j k l} = (β_{0} + α_{0 k} + ξ_{j k l}) + (β_{1} + α_{1 k}) + {(Pathway)}_{j k} + β_{2} {(Pretest Covariate)}_{j k l} .

The difference between this model and the model for the classrooms in our study is that it includes a random effect, ξ_jkl, for classroom section l taught by teacher j in randomization block k. This additional term accounts for the fact that we have teachers teaching multiple classrooms within the randomization block. To assess the significance of this additional random effect, we fit the model with and without the random effect and calculated −2logLR = −2log(LR), where LR is the likelihood ratio statistic. We then simulated data under the reduced model

µ_{j k l} = (β_{0} + α_{0 k}) + (β_{1} + α_{1 k}) {(Pathway)}_{j k} + β_{2} {(Pretest Covariate)}_{j k l},

with the variance parameters and fixed effects set to their estimates, and calculated the proportion of times that the resulting statistic, −2logLR, was as large or larger than what was observed in our data set.

When testing the significance of the variability of the treatment effect across randomization blocks, α_1k, the reduced model was

µ_{j k l} = (β_{0} + α_{0 k} + ξ_{j k l}) + β_{1} {(Pathway)}_{j k} + β_{2} {(Pretest Covariate)}_{j k l} .

Finally, when testing for the significance of the randomization block random effect, α_0k, we calculated −2logLR comparing the model with a single random effect, α_0k,

µ_{j k l} = (β_{0} + α_{0 k}) + β_{1} {(Pathway)}_{j k} + β_{2} {(Pretest Covariate)}_{j k l},

to a model with no random effects,

µ_{j k l} = β_{0} + β_{1} {(Pathway)}_{j k} + β_{2} {(Pretest Covariate)}_{j k l} .

This method for testing the randomization block random effect is consistent with the method that we used for analyzing the classrooms involved in the study.

Results

(1) To what extent will teachers’ participation in the Pathway Project professional development intervention over 2 years improve academic outcomes for mainstreamed Latino ELs on an on-demand writing assessment and the California Standards Test in English language arts? For our Year 2 study (2008–2009), we conducted an impact analysis on student outcomes to assess whether the Year 1 results could be replicated with a new sample of students. We fit three statistical models that included random effects for teachers and the randomization block (i.e., the school by grade randomization pool). First, we fit a multilevel model to estimate the impact on the ALA. As shown in Table 3, students in classrooms taught by Year 2 Pathway teachers scored .67 standard deviations higher on the ALA than students in classrooms taught by Year 2 control teachers. The magnitude of the Year 2 effect size indicates that the average student who was taught by a Pathway teacher improved from the 50th to the 75th percentile on the ALA posttest. Thus, the Year 2 impact on the ALA on-demand writing assessment replicated the .35 effect size observed in Year 1 (Kim et al., 2011).

Table 3

Hierarchical Linear Models Used to Predict Student-, Classroom-, and Randomization Block–Level Achievement Outcomes

	Assessment of Literary Analysis			California Standards Test (CST) Total			CST Reading Subtest			CST Writing Subtest
Fixed Effect	Coefficient	SE	t	Coefficient	SE	t	Coefficient	SE	t	Coefficient	SE	t
Intercept (β₀)	−0.307	0.073	−4.226**	−0.023	0.051	−0.452	−0.020	0.050	−0.409	−0.024	0.050	−0.475
Pathway treatment effect (β₁)	0.674	0.128	5.247**	0.066	0.051	1.280	0.066	0.052	1.258	0.065	0.052	1.247
Classroom pretest score (β₂)	0.479	0.083	5.786**	0.846	0.056	14.991**	0.781	0.056	13.919**	0.749	0.056	13.253**
Random Effect	Estimate	−2logLR	p	Estimate	−2logLR		Estimate	−2logLR		Estimate	−2logLR
Block intercept ( $τ_{0}^{2}$ )	0.145	57.448**	0.000	0.059	42.683**	0.000	0.054	32.783**	0.000	0.053	35.839**	0.000**
Pathway treatment effect ( $τ_{1}^{2}$ )	0.515	43.837**	0.000	0.005	1.754	0.226	0.006	2.021	0.193	0.005	1.679	0.239
Students (σ²)	0.730			0.750			0.782			0.786

Note. LR = likelihood ratio.

~p < .10. *p < .05. **p < .01. ***p < .001.

Second, we fit a multilevel model to estimate impacts on the English language arts portion of the CST total score and the CST scores on the reading and writing subtest. There was a positive but nonsignificant impact on the CST total language arts score (d = .066), the CST reading subtest (d = .066), and the CST writing subtest (d = .065). The magnitude of each treatment effect is similar to the magnitude of the Year 1 impact on the CST total. Given the magnitude of the Year 2 effect size for the CST posttest scores, the results indicate that the average student who was taught by a Pathway teacher improved from the 50th to the 53rd percentile on each of the three CST posttest scores.

(2) Do students in all classrooms taught by Pathway teachers improve their performance on the California Standards Test in English language arts compared to students in all classrooms taught by control teachers? Our third multilevel model was used to estimate impacts on the CST English language arts test using data from all classrooms taught by Pathway and control classrooms. In this analysis, our goal was to examine the extent to which students in all classrooms taught by Pathway teachers performed better than students in all classrooms taught by control teachers. As shown in Table 4, the analyses based on the Year 2 experimental teacher sample indicates that Pathway students scored approximately .094 standard deviations higher than control students on the CST total English language arts score and .10 standard deviations higher than controls on the CST writing subtest. There was a positive, nonsignificant impact on the CST reading subtest (d = .083). Given the previous results on the ALA, the results suggest that improvements in teachers’ ability to implement cognitive strategies in their English classrooms transferred to improvements on the CST writing scores of all students taught by Pathway teachers.

Table 4

Hierarchical Linear Models Used to Predict Student-, Classroom-, and Randomization Block–Level Achievement Outcomes (All Classrooms)

	California Standards Test (CST)Total			CST Reading Subtest			CST Writing Subtest
Fixed Effect	Coefficient	SE	t	Coefficient	SE	t	Coefficient	SE	t
Intercept (β₀)	−0.247	0.047	−5.234**	−0.220	0.047	−4.718**	−0.221	0.047	−4.734**
Pathway treatment effect (β₁)	0.094	0.043	2.178*	0.083	0.043	1.906~	0.100	0.044	2.238*
Classroom pretest score (β₂)	0.941	0.026	36.378**	0.860	0.027	32.457**	0.829	0.027	31.131**
Random Effect	Estimate	−2logLR	p	Estimate	−2logLR	p	Estimate	−2logLR	p
Block Intercept ( $τ_{0}^{2}$ )	0.066	323.610**	0.000	0.063	253.727**	0.000	0.063	247.143**	0.000
Pathway treatment effect ( $τ_{1}^{2}$ )	0.020	5.910*	0.026	0.021	5.655*	0.029	0.023	6.446*	0.010
Sections within teacher (ω²)	0.000	0.000	1.000	0.000	0.000	1.000	0.000	0.000	1.000
Students (σ²)	0.613			0.653			0.658

Note. LR = likelihood ratio.

~p < .10. *p < .05. **p < .01. ***p < .001.

Discussion

In the conclusion of their meta-analysis of writing instruction, Graham and Perin (2007) point out that there is a “serious gap” in the research literature pertaining to secondary adolescents from low-income families, inner-city settings, and/or with low English language proficiency. In addition, the meta-analytic findings revealed comparatively small effects on writing outcomes for studies involving English learners. Shanahan and Beck (2006) concur. In fact, they note that there are so few experimental studies designed to improve literacy outcomes for ELs that it is currently not possible to conclude that any instructional approaches “consistently confer advantages to the learner” (p. 446).

Given the dearth of research regarding effective literacy interventions for secondary mainstreamed ELs, we believe our work has the potential to contribute to the scientific knowledge base regarding strategies for enhancing the academic literacy of secondary ELs both in California, which serves 25% of the nation’s ELs enrolled in K–12 schools, and nationally (California Legislative Analysts Office, 2007). Our findings highlight the efficacy of implementing a cognitive strategies approach for ELs using a range of pedagogical strategies to make visible for ELs the thinking tools accessed by experienced readers and writers during the process of meaning construction. Additionally, our study results are consistent with Taylor et al.’s (2005) findings on the influence of teachers’ practices that encourage cognitive engagement at the elementary level and confirm Langer’s (2000) findings at the secondary level. In particular, our results indicate that teachers can learn to engage mainstreamed English learners in higher level interpretive reading and analytical writing about texts through direct strategy instruction, modeling of strategy use, and creating opportunities for students to practice and apply these skills through teacher coaching and feedback.

Our main finding is that the Year 2 results largely replicated the Year 1 results. Our impact estimates suggest larger, positive effects on the ALA, which is the on-demand writing assessment tied mostly directly to the Pathway intervention, than on CST. The magnitude of the Year 2 effect size on the ALA (d = .67) is nearly twice as large as the Year 1 effect size (d = .35). As Pressley (2002) has observed, cognitive strategies interventions are complex and take time for teachers to internalize and implement with confidence and competence. Our finding of higher Year 2 gains reinforces Taylor et al. (2005), who noted that the difference in impact across 1 versus 2 years highlights the importance of “sustained school improvement efforts” (p. 64).

When reviewing their students’ pretests and posttests at our end of-year inservice, Year 2 teachers’ informal observations revealed the following indicators of growth in the posttest essays: clearer essay structure (introduction, main body, conclusion); the presence of a claim/thesis statement in the introduction in response to the prompt; less reliance on retelling; more and deeper analysis/commentary; ample support, including quotes from the text; sentence variety; more academic expressions; and some improvement in the conventions of written English. Figure 3 includes an introduction to a pretest and posttest by a ninth-grade student in a Year 2 experimental teacher’s class. Note the shift in the student’s reliance on plot summary in the pretest to the inclusion of interpretation and commentary in the posttest as well as the articulation of a clear theme statement.

Figure 3.

Writing sample at pretest and posttest for a ninth-grade student taught by a Pathway teacher.

Our Year 2 findings also suggest evidence of transfer to students’ writing ability on other measures that are not tied directly to the Pathway intervention. In assessing the external validity of the findings, we noted that students in all classrooms taught by Pathway teachers outperformed students in classrooms taught by control teachers on the CST writing subtest (d = .10) but not on the CST reading subtest (d = .083). In both the Year 1 and Year 2 results, there is evidence that improved performance on the ALA on-demand writing test generalized to improvements in the CST writing subtest. Nonetheless, the magnitude of the effect size on the CST writing subtest is clearly smaller than the ALA, and there is also no evidence of improvement on the CST reading subtest. Although Graham and Hebert (2010) found that writing instruction generalized to improved reading achievement, there was no evidence of transfer to improved student reading outcomes in our study.

How can we explain the disparity in the magnitude of the treatment effects on the ALA and CST writing subtest? First, research suggests that effects are larger on assessments that are similar in content and format to those used in the original intervention (Hernstein, Nickerson, de Sanchez, & Swets, 1986; Hill, Bloom, Black, & Lipsey, 2008). As a result, we would expect to see larger effects on the ALA, which was used as a formative assessment to inform writing instruction during regular classroom instruction. Second, our implementation measures indicated relatively larger differences on measures of writing instruction than reading instruction. On the Pathway observation measure, there was suggestive evidence indicating that Pathway teachers were more likely to implement writing instruction consistent with Pathway professional development. On the third and final classroom observation in spring 2009, the effect size on the Pathway observation for writing instruction (d = .38) was nearly three times as large as the effect size for reading instruction (d = .13). These implementation data suggest that Pathway teachers were more likely to use writing activities consistent with the intervention, thereby producing the largest improvement in students’ writing ability.

Limitations and Next Steps

There are several limitations that should inform future research. First, the external validity of our findings should be replicated in a future experiment outside Santa Ana, CA. Indeed, a central limitation of our study is that it was implemented in a school district where the UCI Writing Project has had a long history of providing professional development. Through this project’s previous teacher training workshops, teachers in both the control group and the experimental group may have learned approaches to teaching the writing process and specific aspects of composition that affected student outcomes on the ALA. One next step is to conduct a randomized field trial in a district where the intervention developer and the UCI Writing Project has had no prior relationship with teachers or administrators. SAUSD is also somewhat unique in its homogenous student population (Latino mainstreamed ELs). We seek to explore the impacts of the intervention in another large urban district whose cultural diversity more closely mirrors the demographics of California. In so doing, we hope to compare the impact of the intervention on native English speakers as well as on ELs. In addition, our analyses for the ALA test included students from Grades 6 to 12, whereas our analyses for the CST excluded Grade 12 students. Thus, our impact analyses are based on different samples of students.

Second, a longitudinal study to track student progress through secondary school and into postsecondary education is necessary to document whether the intervention contributes to students’ academic success in the long term. Finally, pedagogical strategies designed to develop EL students’ command and use of academic language would enhance the current intervention. It might be the case that students could access the strategies better if they had greater knowledge of the academic language in the texts that they were reading and responding to in their writing. Cognitive strategies may play a critical role in ELs’ improvement of writing, as evidenced by improved ALA scores. However, other factors, including academic vocabulary, grammar, and a range of discourse features, may also merit instruction in English language arts classes. In addition, many learner characteristics might also affect students’ development and use of strategies, including the learners’ first and second language proficiency, background knowledge, experience, previous education, and gender. Further research is needed to address the limitations of the current study and to replicate these promising findings in other school districts.

Footnotes

We gratefully acknowledge funding from the Institute of Education Sciences to support our study, “The Pathway Project: A Cognitive Strategies Approach to Reading and Writing Instruction for Teachers of Secondary English Language Learners (Grant No. R305W06016).” The views expressed in this article reflect the opinions of the authors and not the funding agency. We are grateful to the students, teachers, principals, and administrators in the Santa Ana Unified School District for supporting the implementation of the study. Finally, we thank Harold Himmelfarb, Judith Langer, David Pearson, Steve Graham, and Paul LeMahieu for providing support and feedback during the design and implementation of this study.

1

We estimated the minimum detectable effect size (Bloom, 2005), which is the smallest true impact that can be detected with 80% power using a two-tailed test with alpha set at .05. In our multisite cluster randomized field trial in which teachers were placed into school by grade blocks and then randomly assigned to conditions, we used Optimal Design (Raudenbush, Liu, Spybrook, Martinez, & Congdon, 2006) to estimate the minimum detectable effect size based on the following design parameters: the number of schools (K = 15), the anticipated number of teacher clusters per site (J = 8), two different estimates of the intraclass correlation (ρ = .05 and .10), the percentage of the variance in the student posttest scores explained by the pretest covariate (R² = .50), and the power of the blocking variable (B = .05). We used the district’s average class size of 30 students to estimate the number of students per cluster (i.e., classroom). Based on the parameters of our study design, there was sufficient power (80%) to detect a standardized mean difference of .12 on the student outcome measures, which is typical of effect sizes generated by randomized experiments of cognitive strategies instruction in the secondary grades (Slavin et al., 2008).

References

ACT Inc. (2007). Writing framework for the 2011 national assessment of educational progress. Iowa City, IA: Author, The National Assessment Governing Board.

Analysis of the 2007-08 budget bill: education. (2007, February 21). Retrieved from http://www.lao.ca.gov/analysis_2007/education/ed_11_anl07.aspx

Applebee

Langer

Nystrand

Gamoran

(2003). Discussion-based approaches to developing understanding: Classroom instruction and student performance in middle and high school English. American Education Research Journal, 40, 685–730.

August

Hakuta

(Eds.). (1997). Improving schooling for language-minority children: A research agenda. Washington, DC: National Academy Press

August

Shanahan

(Eds.). (2006). Developing literacy in second-language learners: Report of the national literacy panel on language-minority children and youth. Mahwah, NJ: Lawrence Erlbaum Inc.

Avi

T. M.

(1997). What do fish have to do with anything?: And other stories. Cambridge, MA: Candlewick Press.

Batalova

Fix

Murray

(2005). English language learner adolescents: Demographics and literacy achievements (Report to the Center for Applied Linguistics). Washington, DC: Migration Policy Institute.

Biancarosa

Snow

C. E.

(2004). Reading next—A vision for action and research in middle and high school literacy: A report to Carnegie Corporation of New York (2nd ed.). Washington, DC: Alliance for Excellent Education.

Black

Wiliam

(1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139–148.

10.

Block

C. C.

Pressley

(Eds.). (2002). Comprehension instruction: Research-based best practices. New York, NY: Guilford.

11.

Bloom

H. S.

(2005). Learning more from social experiments: Evolving analytics approaches. New York, NY: Russell Sage Foundation.

12.

Bruner

(1978). The role of dialogue in language acquisition. In Sinclair

Jarvella

R. J.

Levelt

W. J. M.

(Eds.), The child’s conception of language (pp. 241–256). New York, NY: Springer.

13.

Buly

M. R.

Coskie

Robinson

Egawa

(2006). Literacy coaching: Coming out of the corner. Voices in the Middle, 13(4), 24–28.

14.

California Department of Education. (2008a). English language arts teacher guide, California High School Exit Examination. Sacramento, CA: Author.

15.

California Department of Education. (2008b). Teacher guide for the 2008 California writing standards test in Grade 7. Sacramento, CA: Author.

16.

California Department of Education. (2008c). Technical report for the California English Language Development Test (2007–08 ed.). Sacramento, CA: Author

17.

California Department of Education. (n.d.). Data quest. Retrieved from http://dq.cde.ca.gov/dataquest/

18.

Capps

Fix

Murray

Ost

Passel

Herwantoro

(2005). The new demography of America’ schools: Immigration and the No Child Left Behind Act. Washington, DC: Urban Institute.

19.

Conley

M. W.

(2008). Cognitive strategy instruction for adolescents: What we know about the promise, what we don’t know about the potential. Harvard Educational Review, 78(1), 84–106.

20.

DuFour

Eaker

(1998). Professional learning communities at work: Best practices for enhancing student achievement. Alexandra, VA: Association for Supervision and Curriculum Development.

21.

Echevarria

Vogt

M. E.

Short

(2008). Making content comprehensible to English learners: The SIOP model (3rd ed.). Boston: Pearson/Allyn & Bacon.

22.

Education Data Partnership. (2010). Fiscal, demographic, and performance data on California’s k-12 schools. Retrieved from www.ed-data.k12.ca.us

23.

Educational Testing Service. (2009). California standards tests technical report, spring 2008 administration. Princeton, NJ: Author.

24.

Flower

Hayes

J.R.

(1981). A cognitive process theory of writing. College Composition and Communication, 32, 365–387.

25.

Foorman

B. R.

Schatschneider

Eakin

M. N.

Fletcher

J. M.

Moats

L. C.

Francis

D. J.

(in press). The impact of instructional practices in Grades 1 and 2 on reading and spelling achievement in high poverty schools. Contemporary Educational Psychology, 31, 1–29.

26.

Foorman

B. R.

Schatschneider

Fletcher

J. M.

Francis

D. J.

Moats

L. C.

(2003). The impact of instructional practices in grades 1 and 2 on reading and spelling achievement in high poverty schools. Manuscript submitted for publication.

27.

Fox

(2007). Securing the future: US immigrant integration policy, a reader. Washington, DC: Migration Policy Institute: National Center on Immigrant Integration Policy.

28.

Francis

D. J.

Rivera

Lesaux

Keiffer

Rivera

(2006). Practical guidelines for the education of English language learners: Research-based recommendations for instruction and academic interventions. Portsmouth, NH: Center on Instruction. Retrieved from http://www.centeroninstruction.org/files/ELL1-Interventions.pdf

29.

Genesee

Lindholm-Leary

Saunders

Christian

(2006). Educating English language learners: A synthesis of research evidence. New York, NY: Cambridge University Press.

30.

Gersten

Baker

Haager

Graves

(2005). Exploring the role of teacher quality in predicting reading outcomes for first grade English learners: An observational study. Remedial and Special Education, 26, 197–206.

31.

Goldenberg

(2008). Teaching English language learners: What the research does—and does not—say. American Educator, 32, 7–23, 42–44.

32.

Graham

(2006). Strategy instruction and the teaching of writing. In MacArthur

Graham

Fitzgerald

(Eds.), Handbook of writing research (pp. 187–207). New York, NY: Guilford.

33.

Graham

Hebert

M. A.

(2010). Writing to read: Evidence for how writing can improve reading. A Carnegie Corporation Time to Act Report. Washington, DC: Alliance for Excellent Education.

34.

Graham

Perin

(2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445–476.

35.

Hernandez

D. J.

Denton

N. A.

Macartney

S. E.

(2008). Children in immigrant families: Looking to America’s future. Social Policy Report, 22(3), 3–22.

36.

Hernstein

R. J.

Nickerson

R. S.

de Sanchez

Swets

J. A.

(1986). Teaching thinking skills. American Psychologist, 41, 1279–1289.

37.

Hill

C. J.

Bloom

H. S.

Black

A. B.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2, 172–177.

38.

Hoffman

Sable

(2006). Public elementary and secondary students, staff, schools, and districts: School year 2003-2004. Washington, DC: National Center for Educational Statistics.

39.

Hopstock

P. J.

Stephenson

T. G.

(2003). Native languages of LEP students (Descriptive study of services to LEP students and LEP students with disabilities, Special Topic Report No. 1). Washington, DC: U.S. Department of Education.

40.

Horwitz

Uro

Price-Baugh

Simon

Uzzell

Lewis

Casserly

(2009). Succeeding with English language learners: Lessons learned from the Great City schools. Retrieved from http://www.cgcs.org/cms/lib/DC00001581/Centricity/Domain/4/ELL_Report09.pdf

41.

Joyce

Showers

(2002). Student achievement through staff development (3rd ed.). Alexandria, VA: ASCD.

42.

Kim

Olson

C. B.

Scarcella

Kramer

Pearson

van Dyk

…Land

(2011). Can a cognitive strategies approach to reading and writing instruction improve literacy outcomes for low income English language learners in the middle and high school grades? Results from a multi-site cluster, randomized controlled trial of the Pathway Project. Journal of Research on Educational Effectiveness, 4, 231–263.

43.

Kindler

(2002). Survey of states’ limited English proficient students and available educational programs and services, 2001-2002 (Summary report). Washington, DC: National Clearinghouse for English Language Acquisition.

44.

Langer

J. A.

(2000). Beating the odds: Teaching middle and high school students to read and write well (Report No. 12014). Albany, NY: National Research Center on English Learning & Achievement.

45.

Marzano

R. J.

Pickering

D. J.

Pollock

J. E.

(2001). Classroom instruction that works: Research-based strategies for increasing student achievement. Alexandra, VA: Association for Supervision and Curriculum Development.

46.

Meltzer

Hamann

E.T.

(2005). Meeting the literacy development needs of adolescent English language learners through content area learning part two: Focus on classroom teaching and learning strategies. Retrieved from http://www.lab.brown.edu/pubs/adlit/adell_litdv2.pdf

47.

National Center for Education Statistics. (2007). The nation’s report card: Reading 2007. Washington, DC: U.S. Department of Education, Institute of Education Sciences. Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007496

48.

National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No. 00-4769). Washington, DC: U.S. Government Printing Office.

49.

Olson

C. B.

(2011). The reading/writing connection: Strategies for teaching and learning in the secondary classroom (3rd ed.). Boston, MA: Allyn and Bacon/Pearson.

50.

Olson

C. B.

Land

(2007). A cognitive strategies approach to reading and writing instruction for English language learners in secondary school. Research in the Teaching of English, 41, 269–303.

51.

Olson

C. B.

Land

(2008). Taking a reading/writing intervention on the road: Lessons learned from the PAthway Project. Research in the Teaching of English, 42, 3, 259–269.

52.

Olson

C. B.

Land

AuBuchon

Anselmi

(2010). Teaching secondary English learners to understand, analyze, and write interpretive essays about theme. Journal of Adolescent and Adult Literacy, 54, 245–256.

53.

Paris

S. G.

Lipson

M. Y.

Wixson

K. K.

(1995). Becoming a strategic reader. In Ruddell

R. B.

Ruddell

M. R.

Singer

(Eds.), Theoretical models and processes of reading (pp. 788–810). Newark, NJ: International Reading Association.

54.

Paris

S. G.

Wasik

B. A.

Turner

J. C.

(1991). The development of strategic readers. In Barr

Kamil

M. L.

Mosenthal

Pearson

P. D.

(Eds.), Handbook of reading research (pp. 609–640). New York, NY: Longman.

55.

Pearson

P. D.

Duke

N. K.

(2002). Comprehension instruction in the primary grades. In Block

C. C.

Pressley

(Eds.), Comprehension instruction: Research based best practices (pp. 247–258). New York: Guilford.

56.

Pressley

(2000). What should comprehension instruction be the instruction of? In Kamil

M. L.

Mosenthal

P. B.

Pearson

P. D.

Barr

(Eds.), Handbook of reading research (pp. 545–561). Mahwah, NJ: Lawrence Erlbaum.

57.

Pressley

(2002). Reading instruction that works: The case for balanced teaching (2nd ed.). New York, NY: Guilford.

58.

Raudenbush

S.W.

Liu

X.F.

Spybrook

Martinez

Congdon

(2006). Optimal design software for multi-level and longitudinal research (Version 1.77) [Computer software]. Retrieved June 18, 2010, from http://sitemaker.umich.edu/group-based

59.

Rossi

P. H.

Lipsey

M. W.

Freeman

H. W.

(2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

60.

Schleppegrell

M. J.

(2009, October). Language in academic subject areas and classroom instruction: what is academic language and how can we teach it? Paper presented at workshop on The Role of Language in School Learning sponsored by the National Academy of Sciences, Menlo Park, CA. Retrieved from http://www7.nationalacademies.org/cfe/Paper_Mary_Schleppegrell.pdf

61.

Shanahan

Beck

I. L.

(2006). Effective literacy teaching for English-language learners. In Shanahan

August

(Eds.), Developing literacy in second-language learners (pp. 415–488). Mahwah, NJ: Lawrence Erlbaum.

62.

Short

Fitzsimmons

(2007). Double the work: Challenges and solutions to acquiring language and academic literacy for adolescent English language learners—A report to Carnegie Corporation of New York. Washington, DC: Alliance for Excellent Education.

63.

Slavin

R. E.

Cheung

Groff

Lake

(2008). Effective reading programs for middle and high schools: A best evidence synthesis. Reading Research Quarterly, 43(3), 290–322.

64.

Snow

Biancarosa

(2003). Adolescent literacy and the achievement gap: What do we know and where do we go from here? New York, NY: Carnegie Corporation.

65.

Taylor

Pearson

Peterson

Rodriguez

(2005). The CIERA School Change Framework: An evidence-based approach to professional development and school reading improvement. Reading Research Quarterly, 40(1), 40–69.

66.

Tierney

R. J.

Pearson

P. D.

(1983). Toward a composing model of reading. Language Arts, 60, 568–580.

67.

Tierney

R. J.

Shanahan

(1991). Research on the reading-writing relationship: Interactions, transactions, and outcomes. In Barr

Kamil

Mosenthal

Pearson

P. D.

(Eds.), Handbook of reading research (Vol. 2, pp. 246–280). New York, NY: Longman.

68.

Tierney

R. J.

Soter

O’Flahavan

J. O.

McGinley

(1989). The effects of reading and writing upon thinking critically. Reading Research Quarterly, 24, 134–173.

69.

Tompkins

G. E.

(1997). Literacy for the 21st century: A balanced approach. Upper Saddle River, NJ: Prentice Hall.

70.

Troia

Graham

(2002). The effectiveness of a highly explicit, teacher-directed strategy instruction routine: Changing the writing performance of students with learning disabilities. Journal of Learning Disabilities, 35, 290–305.

71.

Valdés

(2002). Learning and not learning English: Latino students in American schools. New York, NY: Teachers College Press.

72.

Wong Fillmore

Snow

(2003). What teachers need to know about language. In Adger

C. T.

Snow

C. E.

Christian

(Eds.), What teachers need to know about language (pp. 10–46). McHenry, IL: The Center for Applied Linguistics.