Abstract
This design research project developed and evaluated curriculum for developmental writing classes in community colleges. The core of the curriculum was self-regulated strategy instruction, which has been shown to be effective with adolescents who are struggling as writers. In the curriculum, students learned strategies for planning, drafting, and revising compositions with an emphasis on using knowledge of text organization to guide planning and self-evaluation. In addition to specific writing strategies, students learned strategies for self-regulation. The study is part of a project that developed two levels of developmental writing courses, but only the lower level course is addressed in this article. This article reports findings from the first two cycles of implementation and revision. Over two semesters, the curriculum was implemented in eight classes taught by three instructors and revised after an analysis of quantitative and qualitative data. Substantial gains in writing achievement and motivation were found, especially in the second cycle. In addition to successes, the article discusses design challenges for the curriculum and professional development.
Community colleges are a critical component of the postsecondary education system in the United States. In 2010, 40% of students who entered postsecondary programs immediately after high school attended community colleges (National Center for Education Statistics, 2012). Open-admission policies and low tuition make college possible for many students who could otherwise not attend, especially low-income and minority students (Perin, 2006). But open access also means that many students are underprepared for college work. From 40% to 60% of new community college students take developmental courses in reading, writing, or mathematics, and only a minority of students complete the required sequence of developmental courses and go on to pass the following credit-bearing course (Attewell, Lavin, Domina, & Levey, 2006; Bailey, Jeong, & Cho, 2008; for a review see Bremer et al., 2013).
Despite the prevalence of developmental classes and the large costs to institutions and students, little systematic research has addressed the effects of developmental education (Levin & Calcagno, 2008). A review of research on developmental literacy classes (Perin, 2013) found only 13 experimental or quasi-experimental studies, none of which focused on approaches to writing instruction. Basic writing is an active area of scholarship, represented by several journals, including the Journal of Basic Writing, founded 25 years ago by Mina Shaughnessy. However, the field of basic writing is part of the discipline of English with traditions of scholarship that have moved away from empirical research, particularly research on instructional methods, over the past 25 years (Haswell, 2005). Research on effective writing instruction for this population is much needed.
The purpose of this article is to report results from a program of design research undertaken with the goal of developing and evaluating curricula for developmental writing classes in community colleges. The core of the curriculum is strategy instruction in planning and revising processes combined with self-regulation strategies that are appropriate for older students. Strategy instruction with self-regulation components has been studied extensively with struggling writers at the elementary and secondary school levels and has been shown to have substantial effects on quality of writing (Graham, 2006; Graham & Perin, 2007). Graham and Perin’s (2007) meta-analysis of writing instruction research with adolescents found positive effects of strategy instruction in all 20 studies examined with a large overall effect size (0.82); effects were larger for struggling than for average writers, and effects were larger for studies that included explicit self-regulation components. Two studies have found positive effects of strategy instruction in adult education programs (Berry & Mason, 2012; MacArthur & Lembo, 2009), and general self-regulation strategies have been found effective with college students who have learning disabilities (Butler, 2003). Although writing strategies are commonly taught in developmental writing classes, no research has investigated pedagogical approaches that include the key elements of self-regulated strategy instruction (Graham & Harris, 2005; MacArthur, 2011). In the curriculum designed by this project, students learn strategies for planning, drafting, and revising compositions with an emphasis on using knowledge of text organization to guide planning and self-evaluation. In addition, the curriculum includes an overall self-regulation strategy that includes task analysis and goal setting, strategy selection (including task management strategies), monitoring of progress, and reflection.
Theoretical and Empirical Rationale
Our design research project was guided by a theory of change (see Table 1), which articulated expected links among context, intervention, and desired outcomes. These outcomes include the desired writing achievement and the learning outcomes that are necessary to support those improvements in writing, including discourse knowledge, strategies, self-regulation, motivation, and basic skills. The components of the intervention are also specified in the theory, and the connections between the intervention and outcomes are explained below. Finally, the theory of change includes considerations of the professional development (PD) and adaptation to the college setting that are needed to make the intervention feasible and acceptable to instructors and writing-program administrators who are responsible for adoption.
Theory of Change.
Writing Achievement
The overall pedagogical goal was for students to develop sufficient proficiency in academic writing to be successful in a 1st-year composition course. Following standards for high school achievement in preparation for college (Common Core State Standards, 2010), we included narrative, explanatory, and persuasive writing. We are aware of controversies about the traditional forms of writing taught in schools. The characteristics of written discourse vary widely depending on the social context and purpose of particular texts (Bazerman & Rogers, 2008; Prior, 2006). In particular, written discourse varies substantially across academic disciplines (MacArthur, in press; Shanahan & Shanahan, 2008). However, the focus of this project was on improving pedagogical methods for achieving writing outcomes traditional in 1st-year college composition and basic writing courses. We aimed to develop curricula that would be acceptable and feasible to implement in existing settings. The project developed curricula for two levels of developmental writing courses. The writing outcomes for the lower level course focused on writing from personal experience and general knowledge. The outcomes for the higher level developmental course included writing from sources, including summarization and an introduction to research. Only the lower level course is addressed in this article.
Learning Outcomes
Writing is a complex cognitive and social process that draws on writers’ knowledge, skills, and strategies, as well as self-regulation and motivation (Hayes, 1996; Hidi & Boscolo, 2006). First, proficient writers have considerable discourse, or rhetorical knowledge about the forms of text that are appropriate for various purposes, including knowledge about genre, organization, and characteristics of good writing; writers use this knowledge in planning and evaluating their work. Second, they use a range of strategies to manage the complex processes of planning and revising. Proficient writers engage in task analysis and goal setting during planning and use their discourse knowledge strategically to generate and organize content. Proficient and struggling writers differ dramatically in their strategies for planning and revising (MacArthur, 2011). Third, proficient writing involves self-regulation processes, such as goal setting, self-monitoring and self-evaluation, and task management (Harris, Graham, MacArthur, Reid, & Mason, 2011; Schunk & Zimmerman, 2007). Fourth, high levels of motivation and a sense of self-efficacy are important for success. Students who believe that writing is a meaningful process of communication and idea development, and who have confidence in their skills, are more interested and more likely to persist in the face of difficulty than students who view writing as a school exercise or who lack confidence (Hidi & Boscolo, 2006). Finally, basic writing and language skills and reading comprehension skills are important for effective writing. Sentence production, including grammatical errors and control over complex sentence structures, has been studied extensively with basic college writers (Smith, Cheville, & Hillock, 2006).Reading comprehension is correlated with writing achievement (Fitzgerald & Shanahan, 2000), and writing instruction affects reading comprehension (Graham & Hebert, 2010).
Intervention Components
Specific Writing Strategies
The central idea of strategy instruction is that it is possible to understand the conscious cognitive processes used by proficient writers and to teach those processes in some form to less proficient writers. Most research on writing strategies has used task-specific strategies that integrate discourse knowledge (e.g., knowledge about text structure) with procedural knowledge about using that knowledge strategically (Englert, Raphael, Anderson, Anthony, & Stevens, 1991; Graham & Harris, 2005). For example, when good writers plan, they set goals and subgoals based on audience and purpose and use their knowledge of genre requirements, or text structure, to generate and organize content (Hayes & Flower, 1980). Consequently, our planning strategies ask students to analyze writing tasks for audience and purpose, choose an appropriate text structure, and use the elements of that structure to brainstorm and to organize their ideas. Similarly, in our revising strategies, students learn evaluation criteria based on the text structure to guide revision (e.g., For a persuasive text, does the introductory paragraph clearly state the author’s position ?).
Self-Regulation Strategies
Explicit attention to the development of self-regulation is a key component of strategy instruction (Harris & Graham, 2009; MacArthur, 2011). Writing strategies support self-regulation by giving students a systematic way to approach complex tasks. In addition, research has found support for instruction in self-regulation strategies, including self-monitoring, self-evaluation, self-instructions, goal setting, self-reinforcement, and management of time and environment (Harris et al., 2011). The curriculum includes an overall self-regulation strategy that includes task analysis and goal setting, strategy selection, monitoring of progress, and reflection.(Note that summarization and writing from sources was included only in the higher level curriculum.)
Pedagogical Approach
The literature includes well-developed instructional processes for teaching self-regulated strategies (Harris & Graham, 2009; MacArthur, 2011). Teachers explain strategies explicitly, model them using think-aloud methods, and provide extended guided practice as students apply the strategies to their own writing. The following core principles guided development of the curricula. First, all instruction uses meaningful writing tasks. Teachers discuss the purposes of various genres and the reasons for the organizational elements using models of good papers. Without this meaningful context, strategies may fail to make sense to students and are unlikely to generalize to other tasks and settings. Second, teachers explain the strategies and model them while thinking aloud. Explanations address the purpose and value of the strategies as well as how to carry them out. Think-aloud modeling is critical for demonstrating cognitive processes that are otherwise invisible. Self-regulation strategies are modeled as part of the process and discussed. Third, teachers scaffold student use of the strategies through collaborative and guided practice. In this process, teachers evaluate student understanding and provide feedback both on their use of the strategy and on their writing performance. Teachers gradually release control as students develop independence. Fourth, self-evaluation is emphasized. Students learn evaluation criteria based on genre elements, which are introduced along with each genre and practiced through peer review and self-evaluation. Finally, motivation and self-regulation are emphasized. Self-regulation strategies for setting goals, managing tasks, monitoring strategy use, evaluating performance, and coping with difficulties all help students to be more successful on tasks and to feel more in control of their learning. Motivation is important for all writers, but it is especially important for struggling writers (Cox, 2009). For low-achieving students, success in using strategies can change their attributions for success and failure from personal ability to effective use of strategies (Butler, 2003).
Contextual Factors
A critical goal of the project was to develop curriculum and PD approaches adapted to the context of community colleges. In general, community college instructors responsible for basic writing courses come from backgrounds in English and composition and are familiar with many writing strategies. They also understand the context. Thus, our design research approach was collaborative in intent. The project staff brought years of research on and experience in self-regulated strategy instruction with adolescents, as well as knowledge about composition and rhetoric; the community college faculty brought their contextual knowledge and experience. The goal was to improve existing approaches to teaching writing strategies and skills by applying research-based knowledge about self-regulated strategy instruction. An additional goal was to develop a plan for PD for new instructors.
Method
Research Design
Design research, or formative experiment, is a systematic approach to the development and improvement of instructional methods in natural classroom settings through cycles of development, implementation, evaluation, and revision (Anderson & Shattuck, 2012; Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003; Reinking & Bradley, 2008). The design process begins with a clear pedagogical goal and with theories of learning and instruction, represented here by the theory of change (Table 1), but recognizes that successful use of instructional methods in natural settings requires extensive adaptation to the social and instructional context. It is a collaborative method that values the contributions of teachers and students as well as researchers. It draws on multiple methods of data collection and analysis, both quantitative and qualitative, to analyze how teachers and students understand the instruction, what aspects of the instruction enhance and interfere with learning, and the effects of the instruction on student learning. Design research is iterative at micro and macro levels (Gravemeijer & Cobb, 2006). Minicycles are based on frequent discussion with teachers about the instruction and relatively immediate adjustments based on the observation data. For example, misunderstandings of instruction by the teacher or students might be remedied quickly, or an idea generated from observation and discussion might be added for the next class session. Macrocycles are the periodic thorough reviews and revisions of the intervention before a subsequent round of implementation.
Participants
The study was conducted at a community college on the east coast of the United States and included teachers and students from developmental writing courses. This article reports on the first two rounds of implementation in the spring and fall of 2011. The study is part of a project that developed curricula for two levels of developmental writing, but for reasons of space, only the lower level course, Developmental Writing I, is addressed in this article. In Round one (spring 2011), one male teacher with two classes participated. He had a master’s degree in education and 13 years experience teaching developmental reading and 2 years with developmental writing in the same community college. Prior to the project, the lower level basic writing course had focused primarily on grammar and sentence writing practice with some paragraph writing; the instructor indicated eagerness to try a curriculum focused on more extensive writing.
In Round two (fall 2011), two additional instructors were added with a total of six classes. Both new instructors were female, part-time instructors. One had a master’s degree in elementary education and 4 years experience teaching middle school English as well as 3 years experience teaching developmental writing part time at the community college. The other had retired after a career in higher education that included teaching and administration and had been teaching developmental reading and writing courses as well as psychology courses at the community college for several years. (Note that four additional teachers were included in the larger project working on Developmental Writing II.)
In Round one, complete pretest and posttest data and interviews were available for 10 students from two sections of Developmental Writing I; an additional four students completed the pretest but dropped the course. In Round two, pretest and posttest data were available for 36 students in Developmental Writing I. The full sample of 36 students in Round two was 70% male, 35% White, 35% African American, 14% Hispanic, 8% Asian American, and 5% multiracial; 28% were not born in the United States, and 26% spoke a language other than English at home. The mean age of the students in the second round was 24 years. The composition of the sample in Round two was similar for the smaller group in Round one. Students were placed in Developmental Writing I if they received a score below 53 on the Accuplacer writing test (about 1 SD below the mean of beginning college students);the mean score for the sample in both rounds was 48 (SD = 13).The mean score on the Accuplacer reading test was also 48 (SD = 16). Reading and writing scores correlated moderately (r = 0.51).
Measures of Student Outcomes
Essays
In the first and last weeks of class, students wrote essays on persuasive topics. Although students learned to write in several genres, it was not feasible to administer multiple posttest writing tasks and still give students time to write and revise. Persuasion was chosen because of its fundamental importance to academic writing (Nussbaum & Kardash, 2005) and because it draws on the other genres. For example, causes and effects are common reasons for a position, and personal experience can be used for support. Students had a choice of three topics at each occasion; topics were selected based on a survey of students and consultation with instructors. Essays were scored independently by two raters for quality (ideas, content, and organization) and for conventions (grammar, usage, and mechanics), each on a 7-point scale. Interrater reliability was adequate with correlations of 0.78 for quality and 0.75 for conventions.
Motivation
A questionnaire was developed with four scales: self-efficacy for writing, achievement goal orientation for writing, beliefs about writing, and affect toward writing. The scales were developed on the basis of prior research (Kauffman et al., 2010; Pajares & Valiante, 2006; White & Bruning, 2005) with items adapted for the college population. The scales were pilot tested and revised in Round one and administered at pretest and posttest in Round two of this study. The self-efficacy items assessed writers’ confidence in their ability for a range of writing tasks, skills, and processes. The goal orientation items assessed the degree to which goals were focused on mastery, performance, or avoidance. The beliefs items assessed beliefs that emphasized the substance of writing or mechanics. The affect scale tapped feelings about writing. In Round two, the questionnaire was administered to 133 students in the two levels of developmental writing classes (50% male, mean age 21; 50% minorities), and principal components factor analysis was applied. For goals, factors were found for avoidance, performance, and mastery goals, explaining 27%, 23%, and 9% of the variance, respectively; reliabilities (Cronbach α) ranged from 0.63 to 0.80. For belief, factors were found for substance and conventions, explaining 30% and 19% of the variance; reliability for substance was 0.82 and for mechanics 0.70. Analysis of the self-efficacy scale found only one reliable factor, explaining 55% of the variance (α = 0.95), and the affect scale had a single factor explaining 71% of the variance (α = 0.90).
Placement Test
Students took writing and reading subtests of the computer-administered Accuplacer test (College Board, 2003) as part of registration and placement. The writing tests were multiple-choice tests that tapped ability to edit and rewrite sentences. The tests have adequate reliability and validity (College Board, 2003).
Student Interviews
A sample of students was interviewed at pretest and posttest. At pretest, the interview included questions about students’ reasons for taking the course and their goals regarding pursuing their education. The posttest interview included questions about their opinions concerning the curriculum and instruction as well as about their knowledge of key concepts. The interview asked about their overall opinion of the class and specific components, such as peer review; their strengths and weaknesses as writers; and how much they thought they had learned. Knowledge questions tapped the learning outcomes in the theory of change. Students were asked to explain the specific planning and revising strategies, the elements of the genres and the evaluation criteria, and the strategies for goal setting and self-regulation. The interviews were coded by two of the researchers who met to discuss and resolve their coding.
Measures of Instruction
Participant Observation
The primary source of data during implementation was participant observation. Research staff with expertise in writing instruction observed each instructor at least weekly. Participant observation was chosen as the best way to gain access to the classroom interactions and the thinking of students and teachers. In addition to taking notes during whole class instruction, the observers informally talked and worked with students during individual and small-group work time. The observers also talked informally with teachers. The observations provided data about student understanding of the strategies and genres, as well as information about implementation.
Teacher Interviews
Teachers were interviewed individually before and after the semester. The pretest interview included questions about the college curriculum, their own curriculum and instruction, and their perceptions of their students. It also included questions about their previous training and PD and background information. The posttest interview focused on their perceptions about the curriculum, including strengths and weaknesses, student responses, and suggestions for revision.
Round One
Curriculum Development
Prior to design of the first version of the curriculum, we had two days of PD, including teachers from both levels of the course. We explained the core principles of strategy instruction and demonstrated the methods, and we discussed the instructors’ current approaches to the courses and how strategy instruction would fit. We also interviewed teachers and observed in their classes over one semester to understand their students and instructional methods.
The curriculum included a consistent set of strategies for planning, drafting, and revising that would be used throughout the course. The planning strategy included setting goals by analyzing the writing task for topic, audience, purpose, and form (genre); brainstorming; and organizing through the use of a graphic organizer that differed by genre. Drafting included using the plan, writing main idea sentences, and providing supporting details. Revising included evaluation based on the elements of the form and editing. The goal setting and self-evaluation components of the writing strategy were intended to support self-regulation. In addition, the teacher was encouraged to support self-regulation by emphasizing goal setting, task management, and reflection on how the strategies were working.
The curriculum was organized into units on writing in five traditional genres: narrative, procedural, comparison, cause-effect, and argument. Students wrote two papers in each genre, starting with paragraph-length papers at the start of the semester, but switching to multiparagraph essays by the final unit.
Each unit included the following consistent components: First, the instructor and class discussed the purpose and elements of the genre. For example, for persuasive writing, they discussed when and where they and others use persuasion and what is needed to persuade someone, which led to a discussion of the text elements (e.g., position, reasons, evidence, and counterarguments). Second, the teacher presented and discussed a model paper and a weak paper. The model paper was used to identify the genre elements and illustrate how they contributed to overall quality; the weak paper was used to show how to evaluate a paper by looking for the elements and evaluating their quality. These same genre-specific evaluation criteria were used throughout the unit (e.g., Are the reasons clear and connected to the position?). Third, the teacher explained and modeled the writing strategy, using think-aloud procedures to plan, draft, and revise a paper, and demonstrating the entire process live. Fourth, teacher modeling was followed by collaborative practice, in which the teacher guided use of the strategy while students generated all the content. Next, students applied the strategy to write their own papers with teacher guidance as needed; students usually started writing in class and continued for homework. At the next session, to prepare for peer review, the class practiced applying the evaluation criteria and making revisions to papers written by unknown peers, guided by the instructor. Students then worked in pairs with multiple partners for peer review. Following peer review, students edited for conventions with teacher assistance. The cycle of student planning and drafting with teacher support as needed, peer review, editing, and teacher feedback was followed for a second paper with greater student independence. Throughout the process, self-regulation was encouraged. Grammar instruction was embedded as part of editing in response to individual student difficulties. No overall grammar curriculum or exercises were used.
Implementation
The curriculum was implemented by one instructor in two sections of the course. In addition to the initial workshop for all instructors from both levels of the course, individual PD was provided, including demonstration teaching, coteaching, and general discussions to address specific concerns. Demonstration lessons were usually provided for the section of the class that met first in the week, allowing the instructor to be observed during the second section. Also, the instructor and research staff met weekly to discuss student progress, plan for future lessons, and clarify concepts and instructional procedures. Finally, the research staff held discussions with the instructor about the format of specific materials and asked his feedback about specific changes. No formal measure of fidelity of implementation was used during this development stage, but field notes indicate that all lesson components for each unit were taught. The instructor’s skill at modeling and supporting strategies developed over the semester.
Interviews and discussions with the instructor indicated that he thought the curriculum was effective in supporting students as writers. In his feedback, he suggested that the students needed more repetition and more opportunities to review the writing strategy and elements of each genre. He also explained that he felt pressed for time, and that his students needed more time to truly understand specific concepts, such as the opposing position in the last unit on persuasion. Furthermore, he suggested that the peer review procedure was not as effective as we had anticipated because the students were either not able to give specific feedback or because they were not taking the task seriously. Finally, he explained that he would like to be able to better understand how the students’ progressed, how their understanding changed, and what they thought about their progress. In addition, the observations revealed one apparent limitation: Self-regulation strategies were not taught systematically.
Student Outcomes
Students made significant gains in writing quality and conventions. Writing quality increased significantly (t(9) = 4.4, p < .01) from pretest (M = 1.4, SD = 0.21) to posttest (M = 2.45, SD = 0.76); nine of 10 students increased their scores while one stayed the same. Similarly, overall ratings of conventions showed significant gains (t(9) = 2.7, p = .026) from pretest (M = 1.95, SD = 0.60) to posttest (M = 2.75, SD = 0.86). Of course, a major limitation of a design study is the lack of a control group, so it is not possible to know whether gains were due to the curriculum, to writing instruction in general, or to other factors. Nonetheless, positive gains are a first step, and the posttest scores were similar to the pretest scores of students in the higher level of developmental writing, suggesting that they might be prepared for that class.
Analysis of the student interviews found that 11 of 12 students gave positive overall evaluations of the course, while one had a mixed review; the same students gave positive responses to the overall question about how much they learned. When asked about their strengths, nearly half of the students (five) mentioned attitudinal factors like confidence or liking writing, and nearly half (five) mentioned substantive factors like having good ideas. The weaknesses mentioned balanced these strengths; five students said they needed to work harder, and six mentioned problems with conventions. All of the students thought they had learned a lot about planning but just over half (seven) thought they had learned a lot about revising. Most students (eight) had positive opinions about peer review, but two responded negatively and three gave mixed reviews, both based on the idea that their peers did not know enough to help them.
On the questions tapping discourse knowledge of the genres and knowledge of the strategies taught, students were not very specific. All students remembered some of the names of the genres, but only three of 12 could give the elements or evaluation criteria for more than one genre. They all remembered the major steps of the writing strategy (i.e., planning, drafting, revising), but most of them could not give details about the planning and revising strategies.
Round Two
Curriculum Revisions
Based on the analysis from the first round, the instructor’s feedback, and our observations, several changes were made to the curriculum in the second round. First, to improve learning of the elements of the genres and steps in the strategy, we added additional review at the beginning of lessons, posters of genre elements, and quizzes to be used at the instructor’s discretion. Further, we emphasized the connections among the genre elements, the graphic organizer, and the evaluation criteria used for revision. Research on self-regulated strategy instruction with struggling writers discusses the importance and challenge of mastering the text structures and strategy procedures (Harris & Graham, 2009); we had included a variety of supports, but more seemed needed.
Second, to enhance learning about self-regulation and to support students’ growth as self-regulated learners, an overall self-regulation strategy called the goals strategy was added. This goals strategy included goal setting, strategy selection, monitoring of progress, and reflection. It was based on theoretical considerations (Schunk & Zimmerman, 2007) and research on goal setting and self-regulation with college students who have learning disabilities (Butler, 2003). Strategy selection included strategies for task management and maintenance of motivation as well as writing strategies. Students wrote journal entries focused on these goals strategies and discussed their journal reflections in class. At the end of each genre unit, the students were asked to critically evaluate their progress by examining the final product and the strategies they employed, and to decide what strategies they would continue to use or what other strategies they would consider.
Third, we expanded the lessons on persuasive writing to permit more time to learn about opposing positions, which is one of the primary difficulties of argumentative writing from middle school (Ferretti, MacArthur, & Dowdy, 2000) through college (Nussbaum & Kardash, 2005). The additional lessons included a class debate and sentence frames to assist students with stating the opposing position and the rebuttal.
Fourth, we enhanced the procedures for peer review and editing in several ways. We emphasized the value of learning to give feedback as a way of improving one’s own writing (Cho & MacArthur, 2011), and we included specific procedures for training in peer review with instructor modeling, collaborative practice, and student practice on papers written by unknown students. In addition, we revised the peer review procedures to require the reviewer to provide written suggestions and the writer to make notes about planned changes. Finally, we developed an instructional frame for editing lessons that began with collaborative practice editing a sample paper selected to illustrate a common problem, followed by application to the students’ own papers. Instructors then conferenced with students about errors and helped them set and record individual editing goals. In this way, instruction in grammar and writing conventions was still embedded in the writing instruction, but instructors had a clearer format for addressing common problems.
Professional Development
In addition to curriculum changes, a primary focus of Round two was the design of PD. We recruited two additional instructors and provided PD in the summer and immediately prior to the start of the fall semester. (At this point, PD was separate for instructors of the two levels of the course.) The sessions were designed to apply our model of strategy instruction to PD. The key instructional principles and activities were explained as a strategy for teaching strategies. We explained and discussed the key principles of strategy instruction, modeled the strategy for teaching strategies via demonstration teaching with the instructors as students, provided guided practice in which the instructors taught the lessons with feedback, discussed application across the genres, and addressed questions and concerns. Additional modeling was provided by giving instructors videos of specific lessons for one of the units conducted by the research staff that they could watch at home. In addition, the first instructor shared his experiences in Round one and explained his perspective on the gains made by his students. In the week before the semester started, we met with the instructors again and repeated the process in which project staff modeled teaching the lessons, and instructors practiced teaching with feedback.
PD for the new instructors continued during the semester. First, research staff met individually with the instructors to discuss the instructional procedures in advance of teaching key lessons. In addition, the research staff observed instructors weekly and provided specific feedback guided by questions about the key components of instruction. Finally, project staff modeled lessons upon request by the instructors.
Implementation
Based on the observations and interviews with the instructors, we found that all instructors taught all five units with acceptable fidelity. They reported that the PD was effective, citing the modeling and practice in the summer sessions, the frequent observations with feedback, and the demonstration teaching in their classes. Modeling of the strategies was challenging for the teachers, especially modeling of the drafting process. Despite their concerns, the instructors did model the strategies in ways that observers rated as effective overall. Learning to model strategies and provide guided practice will continue to be a key issue for PD.
The curriculum revisions were mostly helpful. The expanded lessons on persuasion seemed to help students address opposing positions in their writing more effectively. The peer review worked better with the additional practice evaluating papers by unknown peers and establishing the expectation that peer reviewers provide written suggestions. Some transfer from peer review to self-evaluation was observed, but students still struggled with evaluating their own writing.
However, several issues need to be considered in future implementation. First, the pace of the curriculum was demanding. Though all units were taught, each instructor found it necessary to eliminate one of the writing assignments because of time constraints. Also, they felt that more time for in-class writing would be beneficial to students. Second, it was difficult to integrate the goals strategy with the writing strategy. Often students did not complete the reflective journal entries on goals with serious attention.
Student Outcomes
Writing Achievement
Pretest and posttest compositions were available from 34 students (see Table 2). Gains in quality from pretest to posttest were statistically significant (t(33) = 10.5, p < .001) with a large effect size of 1.95 (calculated as gains based on posttest standard deviations). Put in another way, all but two of the 34 students made some gain in quality over the semester, and 62% made gains of two or more. Gains in writing conventions were also significant (t(33) = 6.8, p < .001) with a large effect size of 1.18.
Writing Outcomes for Round Two (n = 34).
Note. Quality and conventions were scored on 7-point scales.
p < .001.
Motivation
We anticipated that students would make gains in self-efficacy and affect. We also anticipated that students’ goals would move from performance to mastery goals, and that their beliefs would move from an emphasis on mechanics to a greater emphasis on the substance of writing. The preliminary analysis confirmed all of our expectations but one (see Table 3). Students made significant gains in self-efficacy (t(35) = 5.4, p < .001) and affect (t(35) = 4.6, p < .001). On goals, students increased in mastery motivation (t(35) = 2.5, p < .05) and decreased in performance motivation (t(35) = 2.2, p < .05). On beliefs, students increased their beliefs about the importance of substantive factors (t(35) = 3.7, p < .001). The only anticipated changes that were not realized were that there were no significant decreases in avoidance motivation or in beliefs in the importance of mechanics.
Writing Motivation for Round Two (n = 36).
Note. All measures are on a scale of 1 to 5, except for self-efficacy, which was measured on a scale of 0 to 100.
p < .05. ***p < .001.
The student interviews were consistent with the quantitative findings about improved self-efficacy and affect. Postinterviews were conducted with 16 students (six females, five for whom English was a second language, and five to six students from each instructor). All students but one reported an overall positive experience in the class and that they had learned a lot about writing. In addition, all but three spontaneously (in response to the general question about the course) commented that learning how to plan and organize a paper contributed to the effectiveness of the course. Further, many of these students reported that they had learned much more about planning than they had in previous courses, stating that planning helped them to organize their thoughts and focus their papers. Not all of the students liked writing after taking the course; however, nearly half (seven) of those interviewed reported more confidence in writing, with others mentioning that their attitude toward writing has now changed for the better.
Knowledge of Genres and Strategies
Based on the interview questions, knowledge of the writing strategy and genre elements was improved from Round one, perhaps as a result of the enhanced review in the curriculum. Although only half of the 16 students had memorized all steps of the strategy, most remembered specific parts of the planning strategy, including goal setting (14), brainstorming (10), and the graphic organizer (eight). In regard to revising, the majority of the students (12) felt that they had learned quite a bit, although only one mentioned it spontaneously. About half of the students (six to seven) specifically mentioned peer review, self-evaluation, and the use of the genre elements for the evaluative process. Most students (13) valued the peer review procedure, both as a process for helping another student and as a process for learning from another student’s mistakes.
Discussion
Design research is a method for developing and evaluating instructional methods that are feasible and effective in real educational settings. Though self-regulated strategy instruction has been extensively validated with younger students, the approach required substantial adaption for developmental writing classes. Intense involvement in a few classrooms allowed us to adapt the curriculum and PD.
Based on qualitative and quantitative data from Round one, we made significant changes in the curriculum from Round one to two. We included more regular review of the strategy and genre elements, and students demonstrated more knowledge in their interviews. We enhanced the lessons on counterarguments in the unit on persuasive writing and found that students were very engaged in debates and managed counterarguments more effectively in their writing. We provided more practice with evaluation criteria for peer review and self-evaluation; observations showed that students gave more helpful comments, and students’ views of peer review improved.
Important curriculum challenges remain. First, we designed a goals strategy as a framework for discussing self-regulation strategies with students and asked students to write about this strategy in journals. Improvements in student motivation, self-efficacy, beliefs, and goals suggest that the goals strategy had some effects. However, observations and interviews also revealed that class discussions of the goals strategy were often not productive, and students did not always take the journals seriously. In addition, the writing strategy and goals strategy overlapped; the goals strategy mentioned selecting strategies, and the writing strategy mentioned setting goals. For future work, we made modest changes in the goals strategy to remove the overlaps, focused it more on task management, and changed the title to Strategies for Academic Success. We will continue to develop ways to support class discussion of these self-regulation strategies.
A second important issue is pacing and the number of writing assignments. We planned the curriculum with two writing assignments in each of five genres to provide students with the opportunity to write more independently on the second paper. This pace proved to be too intensive even though the assignments were relatively short. There is an inherent tension between the importance of revising and polishing each assignment and getting adequate repetition of the writing process to master the planning and revising strategies. A related issue is the importance of time for writing in class. Instructors thought that more in-class writing would allow them to observe their students as they used the strategy and provide more guidance; it also reduces the probability of plagiarism. Issues of time will vary by site as institutions have differing schedules. Thus, the curriculum is designed to permit instructors to select units of instruction that fit their needs.
Equally important as curriculum design is the design of PD. Our approach to PD was informed by our own experiences with strategy instruction and by observations and discussions with instructors. In general, our approach was to model strategy instruction in our PD. Thus, we explained the key principles of strategy instruction and the strategy for teaching strategies, and we modeled these strategies via demonstration teaching with the instructors as students. Then the instructors taught the lessons and we provided guidance and feedback. In general, college writing instructors know many task-specific writing strategies, and the instructors in this study had taught common strategies in the past. Thus, we emphasized how to teach strategies rather than what strategies to teach. Our PD focused on the pedagogical methods drawn from the research on self-regulated strategy instruction (Graham & Harris, 2005; MacArthur, 2011). In particular, we focused on think-aloud modeling, collaborative practice, and gradual release of responsibility.
One key challenge for instructors is learning to model strategies effectively. Particular challenges in modeling included thinking aloud while problem solving, modeling using language and topics at the level of the students, and modeling of the drafting process. Think-aloud modeling puts teachers in the role of performing as writers, and they all had some level of discomfort at first. All instructors did gradually improve and learn to clearly model the strategies for planning, drafting, and evaluating. In addition, instructors were generally able to model using language appropriate for their students and an appropriate level of complexity of organization. Finally, the process of drafting in front of their students was particularly demanding. Drafting inevitably involves some breaks and discontinuities, which made instructors uncomfortable until they realized that those problems were exactly what the students needed to see to understand that writing is challenging even for skilled writers.
The design research study helped to develop a curriculum that produced good outcomes for writing quality and motivation. Writing outcomes from the second round were considerably more positive than those from the first, both for quality and for conventions. In the second round, students made average gains for quality of two points on a 7-point scale, with all but two students making some gain, and 62% gaining two points. Although the curriculum did not include any grammar instruction other than editing support embedded with writing, students also made large gains in conventions in their writing.
Perhaps, equally important were the gains in motivation. Students who have greater confidence in their writing skill, find writing more satisfying, and have goals focused on learning to write rather than on grades or avoiding embarrassment are more likely to be successful in later classes.
It is important to note a major limitation of the current findings. The results represent gains from participation in a developmental writing course. However, without a control group that received typical writing instruction, it is not possible to conclude that the self-regulated strategy instruction curriculum was responsible for the gains. Demonstrating pretest-posttest gains is just one step in evaluating the effectiveness of the curriculum, although a crucial step. Another important limitation is that all the design work was done at a single institution. Both of these limitations will be addressed in future work. A quasi-experimental study including new institutions is planned for the final year of the project.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported in this article was supported by the Institute for Education Science, U.S. Department of Education, grant R305A100614. The opinions expressed are those of the authors.