Abstract
A critical challenge for computer-based writing instruction is providing appropriate and adaptive practice. The current study examined three modes of computer-based writing practice with the goal of identifying those with the greatest learning and motivational value. High school students learned about writing strategies by studying lessons within the Writing Pal tutoring system and then practiced relevant strategies via essay-based practice, strategy practice, or game-based strategy practice. Students acquired strategy knowledge regarding their assigned topics, but there were no main effects of practice format. Similar findings were observed for students’ beliefs about the value of writing practice. However, the effects of practice format depended on prior literacy ability in subtle ways. Essay-based practice appeared to be more effective for skilled readers, whereas less skilled readers benefitted more from game-based practice. Overall, multiple forms of practice opportunities may optimize benefits, although non-game forms of strategy practice are less preferable to students than game-based formats.
Writing instruction can be approached through diverse methods, including strategy instruction, scaffolding for prewriting and goal-setting, and peer support (Graham, MacArthur, & Fitzgerald, 2013; Graham & Perin, 2007). Underlying all of these pedagogical practices is the fundamental role of practice—developing writers require time and multiple opportunities to hone their declarative and procedural knowledge of writing (Kellogg & Whiteford, 2009). Complex processes such as writing can also be decomposed into constituent tasks (see Hayes, 2012), which might then be productively practiced separately before (re)integrating them within the complete process (i.e., part-task vs. whole-task training; Wickens, Hutchins, Carolan, & Cumming, 2012, 2013). A long-standing plea has been for instructors to increase the number and depth of writing assignments that incorporate authentic writing tasks, processes, and practice (Applebee & Langer, 2009).
In response, educators and researchers are increasingly using computer-based tools to support writing instruction, such as automated writing evaluation (AWE) systems that provide automated scoring and feedback on student writing (e.g., Shermis & Burstein, 2013; Wilson & Czik, 2016) and intelligent tutoring systems (ITSs) that both provide feedback and teach writing strategies (e.g., Calvo, O’Rourke, Jones, Yacef, & Reimann, 2011; Roscoe & McNamara, 2013). However, enabling authentic writing practice presents a critical challenge for educational technology developers. Classroom teachers, although limited by time and resources, can nonetheless adapt to the needs of their students (e.g., Kiuhara, Graham, & Hawken, 2009). For example, students are often asked to compose prompt-based, argument essays. On reviewing students’ writing, teachers might realize that students struggle to use transitional phrases (e.g., “on the other hand”) as cohesive cues. Adaptive teachers can then create supplemental opportunities for practicing cohesion-building strategies before asking students to revise their work. Moreover, teachers could implement multiple forms of practice geared toward individual student needs or preferences (e.g., a mix of examples, games, or extra lectures).
This degree of flexible practice can be difficult to implement in educational technologies because personalization and adaptivity must be “built in” to the system. Developers must decide how and whether to expend limited resources to craft diverse instructional elements, along with the underlying algorithms that coordinate these resources. There is currently little empirically supported guidance about which modes of practice are more or less effective across varying circumstances. Should computer-based tools focus on a single “best” practice approach or strive to provide multiple forms of practice? Should any types of practice be discontinued, and if so, when? For whom?
One relevant issue is student engagement. Practice is not necessarily fun and requires significant, sustained effort (Kellogg & Whiteford, 2009). Writing, in particular, evokes student attitudes ranging from enjoyment to extreme anxiety (e.g., Troia, Shankland, & Wolbers, 2012), and developers of educational technologies for writing must take students’ motivation into account. Additionally, personalized writing strategy instruction and practice may depend on learners’ prior literacy ability. Reading and writing—as activities that depend on similar underlying lexical, syntactic, semantic, and rhetorical knowledge and skills—share a close connection (Fitzgerald & Shanahan, 2000; Shanahan, 2015; Stotsky, 1983). Students’ ability (or lack thereof) to access existing literacy skills and knowledge potentially influences the benefits of various forms of writing practice.
To address these issues, this article contrasts three forms of practice that could be implemented in computer-supported writing strategy instruction: essay-based practice, strategy practice, and game-based strategy practice. Each format possesses different strengths and weaknesses related to demands on prior literacy abilities or potential for engagement. This work is conducted within a population of high school adolescent writers, who represent a crucial target for writing instruction.
Computer-Supported Writing Practice
Writing is a complex activity that entails nonlinear and iterative processes of idea generation, organization, translation, and refinement (e.g., Hayes, 2012), and learning how to coordinate these activities is critical for writing development. To support students’ cognitive and motivational navigation of these tasks—whether through “offline” or computer-based support—educators must provide instruction that emphasizes concrete writing strategies and provides the background knowledge needed to employ the strategies (Graham, Harris, &, Chambers, 2016; Graham & Perin, 2007). Students must then develop their understanding and proficiency with these strategies via iterative practice opportunities (Kellogg & Whiteford, 2009) in some form (e.g., essay-based practice, strategy practice, and game-based practice).
Essay-based practice
Essay-based practice—writing and revising complete essays—is perhaps the canonical and most traditional format. Within an AWE or ITS environment, students might be assigned a writing prompt and given a virtual “scratch pad” to take notes or make an outline. Subsequently, they can put these plans into action by authoring essays, which are then submitted for automated scoring and formative feedback to guide revisions. Across multiple cycles, mindful students can reflect on whether their essays (or specific sections) are improving, whether newly acquired strategies are working, and whether the overall writing task seems more enjoyable. In other words, learners have the opportunity to integrate and coordinate multiple strategies and skills, and this type of authentic “whole task” practice can support strategy acquisition (Lim, Reiser, & Olin, 2009; van Merriënboer & Kirschner, 2017). A number of commercially available AWE systems enable such practice, allowing students to write, receive feedback, and revise: Educational Testing Service’s e-rater/Criterion (Burstein, Tetrault, & Madnani, 2013), Pearson’s WriteToLearn (Foltz, Streeter, Lochbaum, & Landauer, 2013), and Measurement Incorporated’s Project Essay Grade (Wilson, Olinghouse, & Andrada, 2014).
The richness of essay-based practice also presents cognitive and motivational challenges. As with any skill that entails multiple stages or components, practicing the whole task can be overwhelming to novice learners (see Salden, Paas, & van Merriënboer, 2006; Wickens et al., 2013). In learning to write, students often struggle to coordinate the multiple, requisite cognitive processes of essay composition (e.g., planning, drafting, and revising) simultaneously (Braaksma, Rijlaarsdam, van den Bergh, & van Hout-Wolters, 2004; Olive, 2014). As such, whole-task writing practice could be particularly challenging for students with lower reading and writing skills. Indeed, students’ reading skills correlate strongly with writing skills and have been found to meaningfully influence the effects of writing strategy acquisition (Allen, Snow, Crossley, Jackson, & McNamara, 2014; Roscoe, Jacovina, Harry, Russell, & McNamara, 2015; Shanahan, 2015). Skilled readers have stronger vocabulary knowledge and language fluency to draw upon, and they already possess a better understanding of the features of good text. These writers might benefit from whole-task, essay-based practice that encourages (or requires) integrating prior knowledge and newly learned strategies in a complete composition. Less skilled readers, by contrast, may struggle with internalizing new strategies and encounter difficulties when attempting to enact these strategies while planning, drafting, and revising an entire essay.
Strategy practice
An alternative approach, targeted strategy practice, decomposes the writing process into tractable components that can be practiced and mastered individually before tackling a complete essay. Instead of practicing many strategies simultaneously, students benefit from honing in on just one or two explicit and actionable strategies. Such practice likely helps learners of all reading ability levels, with particular benefits for less skilled readers (Gillespie & Graham, 2014) by enabling practice of key concepts and procedures with fewer initial cognitive demands and distractions. Research on part-task training of complex cognitive tasks suggests that this approach can lead to more efficient and robust skill acquisition (Wickens et al., 2012, 2013).
Research has indeed strongly supported the value of strategy instruction for students’ writing development (Graham & Perin, 2007; Graham et al., 2016; Newell et al., 2011). Studies have demonstrated that teaching students about concrete writing strategies (e.g., using precise language, elaborating, citing verifiable rather than hypothetical evidence, and considering counterarguments), providing them with the underlying background and genre knowledge, and enabling practice opportunities, support writers of diverse ages and ability levels. Benefits of strategy instruction have been documented for learners in college (e.g., Butler & Britt, 2011; MacArthur, Philippakos, & Ianetta, 2015), middle and high school (e.g., De La Paz & Felton, 2010; Graham & Perin, 2007), and elementary school (e.g., De La Paz & Sherman, 2013; Ferretti, Lewis, & Andrews-Weckerly, 2009) and seem particularly beneficial for struggling writers with weaker reading skills or learning disabilities.
Game-based practice
Educational games and game-based practice offer another route for providing computer-based writing strategy practice (Habgood & Ainsworth, 2011; Jackson & McNamara, 2013; Proske, Roscoe, & McNamara, 2014). Although far from a motivational panacea (see Wouters, van Nimwegen, van Oostendorp, & van der Spek, 2014), educational games can leverage learners’ intrinsic enjoyment of play to inspire improved interest in associated learning tasks (Ryan, Rigby, & Przybylski, 2006). In terms of strategy-based practice, typical practice activities (e.g., analyzing examples of text using an assigned strategy) can be embedded within game features and narratives to potentially make them more engaging. Moreover, as students gradually add to their repertoire of writing skills, they feel more empowered or motivated to write (Troia et al., 2012). Thus, game-based strategy practice may synthesize the benefits of both part-task training and increased motivational engagement.
Researchers and educators have explored the use of games in writing instruction in a variety of ways (Colby, 2017), including games that link writing to game mechanics (i.e., players must write well to succeed in the game; e.g., Barab, Pettyjohn, Gresalfi, Volk, & Solomou, 2012; Dickey, 2011; Liao, Chang, & Chan, 2018). For example, Dickey (2011) observed that a narrative game environment for argument writing, Murder on Grimm Isle, was effective in engaging college students’ interest and curiosity, and students were able to use lessons learned from the game to improve prewriting strategies (e.g., evidence generation). Similarly, Barab et al. (2012) studied middle school students’ argumentative writing and engagement with the Plague: Modern Prometheus segment of Quest Atlantis. In the game, the alignment of students’ written thesis statements, evidence, and their audience influenced their interactions with game characters (i.e., arguments had narrative consequences). In comparison to a non-game comparison lesson, students who wrote and practiced in the game environment exhibited higher learning gains (i.e., improved persuasive writing quality and genre knowledge) and motivation. Finally, Warren and colleagues (Warren, Dondlinger, & Barab, 2009; Warren, Dondlinger, Stein, & Barab, 2008) have investigated benefits of the Anytown multiuser virtual environment for supporting elementary school students’ writing. In this game-like environment, students take on the role of reporters to write about a series of mysterious events. Researchers found that the game-like curriculum encouraged more voluntary writing practice and increased writing achievement in comparison to a traditional, non-game comparison curriculum.
Contrasting formats with Writing Pal
Although prior research has provided empirical evidence for the plausibility and effectiveness of various practice formats, the relative merits of these formats have rarely been tested directly (see Proske et al., 2014). Another resource for research on writing strategy acquisition and practice—the Writing Pal (W-Pal) ITS—has a modular design that affords such comparisons. In W-Pal, high school adolescents are taught writing strategies via a series of animated, 5-minute multimedia lessons. The strategies developed for W-Pal modules were informed by English education experts and prior literature on argumentative writing and revising strategies (e.g., Butler & Britt, 2011; De La Paz & Graham, 2002; Graham & Perin, 2007; Roscoe, Allen, Weston, Crossley, & McNamara, 2014). Students have the opportunity to practice those strategies by playing games or writing practice essays with automated formative feedback (Allen, Crossley, & Snow, 2014; Roscoe, Brandon, Snow, & McNamara, 2013; and see Method section).
The modular and multipronged design of W-Pal makes it ideal for studying the role of distinct components of writing instruction in combination or isolation. For instance, in a module on freewriting (see Elbow, 1989; Li, 2007), students are taught a FAST mnemonic for generating ideas (i.e., Figure out the prompt, Ask and answer questions, Support ideas with evidence, and Think about the other side). After studying lessons on each step of the mnemonic, students can play Freewrite Flash to practice. In that game, students freewrite on a given prompt for about 2 minutes and earn points by generating a diverse pool of ideas. The game rewards students for typing quickly and continuously, and for including a large variety of key words and concepts. In the game, students are not asked to organize their freewrite or translate it into an essay; idea generation strategies are practiced outside of the context of other writing tasks and demands. Alternatively, or additionally, students can practice freewriting in the context of composing essays. The essay authoring interface includes a “scratch pad” for students to freewrite (or write down other notes and outlines). Thus, multiple forms of practice are available.
W-Pal researchers have contrasted the benefits of using the full W-Pal system (i.e., lessons, games, and essay practice) versus intensive essay-based practice (i.e., using only the essay writing and feedback tools of W-Pal) over multiple sessions. Both groups made modest gains in writing quality and writing strategy knowledge. However, students who had access to game-based practice expressed more positive writing self-efficacy and motivation and were able to articulate more new strategy concepts at posttest (Allen et al., 2014; Roscoe et al., 2013). For students who played the mini-games, game scores predicted strategy knowledge acquisition and game enjoyment was associated with whether students perceived the games as helpful for learning. Thus, the cognitive and motivational benefits of game-based writing practice are linked.
Proske et al. (2014) specifically examined motivational and learning outcomes for four different types of practice: game-based, question-based, model-based, and writing-based. German undergraduate students enrolled in English language classes first learned strategies for writing the introduction section of an essay via W-Pal lesson videos. Students then practiced these strategies by playing a science-fiction themed game (i.e., rescuing spaceships) in which they identified thesis statements and stylistic techniques in example paragraphs (i.e., game-based practice condition). For question-based practice, students completed the same strategy identification tasks but all game elements were removed—the task consisted of a series of example paragraphs and probes about which strategies were used. In model-based practice, students studied examples of complete introduction paragraphs (i.e., included a thesis statement, preview of arguments, and an attention-grabbing hook) and were prompted to attend to and analyze these elements. Finally, in writing-based practice, students were instructed to write introductory paragraphs in response to three different prompts.
Students made equivalent gains in strategy knowledge across all four practice formats (i.e., no effect of condition). On measures of students’ ability to strategically revise prewritten paragraphs, writing-based practice was somewhat less effective than other conditions, whereas model-based practice was slightly more effective. Model-based practice may have been beneficial because it allowed students to hone in on and analyze important features of introductions without the additional cognitive burden of generating text or navigating distracting game mechanics. In terms of motivation, game-based practice was perceived as more interesting and engaging than question-based practice. Thus, for tasks that were highly isomorphic, game-based formats were more motivating. Interestingly, game-based practice did not differ significantly from model-based or writing-based practice in terms of motivation. Studying complete examples or authoring examples might have been perceived as more authentic, which translated into similar levels of engagement as an amusing game-based format.
The study conducted by Proske et al. (2014) revealed positive aspects of model-based practice and uniquely contrasted multiple writing practice formats simultaneously, but it had a few potential limitations. First, participants were nonnative English speakers using learning materials written in English. The challenge or novelty of using W-Pal in that context likely influenced learning and motivational outcomes. In addition, W-Pal was ostensibly developed for high school-aged adolescents, and participants in the study were university students. It is unclear whether observed results would generalize to younger learners. Finally, the study assessed learning for only one content area and did not examine students’ prior reading ability.
In sum, practicing relatively novel writing strategies can occur in a variety of formats, including traditional essay-based methods or strategy-based approaches that focus upon key strategies in isolation. The latter form of part-task practice could facilitate learning and motivation, particularly for less skilled learners, by helping to break down and emphasize fewer strategies and procedures at any one time. In addition, strategy practice can be embedded within games by adding game features (e.g., points, levels, and competition) and narratives (e.g., piloting a spaceship) to encourage more positive engagement.
Research Questions
The current study is guided by three research questions informed by prior research on writing instruction, practice, and related individual differences. Importantly, this work continues the uncommon practice of contrasting multiple formats rather than focusing only a single format. First, we consider whether the format of writing strategy practice in a computer-based system influences strategy knowledge acquisition (RQ1). Second, we consider whether practice format influences changes in attitudes toward writing and practice (RQ2). Finally, we assess whether the effects of practice format are differentially beneficial for more and less skilled readers (RQ3). We hypothesize that strategy practice and game-based strategy practice will be more beneficial to novice writers due to focusing practice efforts on specific strategies and that game-based practice will be the most motivating. Strategy-based and game-based practice effects may be stronger for less skilled readers, due to lower demands on their literacy skills.
Method
W-Pal and Learning Topics
The current study was conducted using learning materials, activities, and tools obtained from the W-Pal tutoring system. In this experiment, students studied one of two writing strategy modules: Body Building or Conclusion Building.
Body Building
In the Body Building module, students learn strategies for authoring the paragraphs that form the middle or “body” of an essay. In a series of four lessons (i.e., Overview, Topic Sentences, Evidence Sentences, and Strengthening Your Evidence), students are taught to organize body paragraphs around concise arguments that are supported with factual and verifiable evidence. Students also learn how to recognize problematic evidence that lacks detail or relies too heavily on hypotheticals or opinions. These strategies are summarized by the C.A.S.E. mnemonic: Concise Argument and Supporting Evidence.
To practice relevant strategies, students can play two games: Fix It: Bodies and RoBoCo. In Fix It: Bodies (see Figure 1), students are presented with four example body paragraphs exhibiting errors in one or more critical elements (e.g., missing topic sentence or inappropriate evidence). Students earn points for identifying the problem and recommending solutions (e.g., replacing off-topic evidence with relevant evidence). Correct responses earn points along with “golden circuits.” In the final stage, students play a Sudoku-like puzzle where the golden circuits can be spent to obtain solution hints. Thus, better performance during the “learning” portion of the game translates into helpful resources in the “playing” portion. Game duration naturally varies across students, but a single iteration of this game requires about 5 minutes.
Fix It: Bodies game from the Writing Pal Body Building module.
In RoBoCo (see Figure 2), students take on the role of a robot builder in the Robot Body Company. Students earn robot parts by authoring two body paragraphs (i.e., topic sentence and supporting evidence) in response to a prompt. Throughout this process, students can request strategy hints about good versus poor body paragraphs. For example, hints encourage students to provide sufficient elaboration, avoid overly subjective language when presenting evidence (e.g., “I believe” and “I feel”), avoid purely hypothetical examples (e.g., “if-then”), and employ causal language when making claims (e.g., “because” and “leads to”). Natural language processing (NLP) metrics (see McNamara, Crossley, & Roscoe, 2013) assess student contributions based on these linguistic properties, and higher scoring topic and evidence sentences earn more parts to be assembled into robot models. At the end of the game, students’ robots are displayed at the “Annual Robot Show” and the game score is expressed as company profits. Thus, generating better paragraphs results in more engaging visual displays (more robots) and score rewards. A single iteration of the game requires about 10 minutes depending on the student.
RoBoCo game from the Writing Pal Body Building module.
Conclusion Building
In the Conclusion Building module, students learn strategies for authoring the final summary of an essay. In a series of four lessons (i.e., Overview, Summarize the Essay, Close the Essay, and Hold the Reader’s Interest), students learn to remind the reader of main ideas and to bring the essay to a clear, engaging end. For example, students are taught to avoid introducing new evidence and to link themes from the essay to broader issues or calls to action. These strategies are summarized via the R.E.C.A.P. mnemonic: Restate the thesis, Explain how thesis was supported, Close the essay, Avoid new arguments and evidence, and Present ideas in an interesting way.
Students can practice these strategies by playing several games, including Fix It: Conclusions and LockDown. Fix It: Conclusions has identical design, duration, and game mechanics as the Body Building version of the game but uses example conclusion paragraphs. In LockDown (see Figure 3), students take on the role of an agent working for the Writing Intelligence Agency. To prevent computer hackers from attacking the system, students must “lock down” a series of three essays by generating conclusion paragraphs. Students receive an essay prompt and an outline of main arguments and evidence, and have a maximum of 5 minutes to author a relevant conclusion paragraph. Throughout the task, students can access reminders regarding the R.E.C.A.P. mnemonic and strategies. Simple, underlying NLP-based metrics (e.g., keyword comparisons) assess linguistic indicators that students have restated the thesis and summarized the main ideas, while avoiding repeating the specific points of evidence or adding new evidence.
LockDown game from the Writing Pal Conclusion Building module.
After each round, students receive feedback on potential weaknesses in their paragraphs, such as reminders to “Summarize the thesis and arguments, but do not discuss specific evidence or examples.” Finally, at the end of the game, students receive an overall score, and are either successful or unsuccessful in obstructing the hackers. A single iteration of the game requires about 10 minutes depending on the student.
Participants
Participants were 163 high school students (aged 15 to 17 years), recruited from local schools in the southwestern United States, who completed all components of the study. High school adolescents represent a critical target for innovative writing instruction for many reasons. For example, national samples of over 24,000 Grade 8 and 28,000 Grade 12 students were examined in 2012 (National Center for Education Statistics, 2012). At both grade levels, only one quarter of adolescents wrote at a proficient (24%) or advanced (3%) level. About half of all adolescents wrote at just a basic level (52% to 54%) and one fifth wrote at a below basic level (20% to 21%). These worrying patterns mirror NAEP reports from 2007 (Salahu-Din, Persky, & Miller, 2008) and 2002 (Persky, Daane, & Jin, 2003)—over time, adolescents’ writing deficits have remained steady rather than improving. In addition, surveys of high school teachers suggest that they are underprepared to enact best practices for writing instruction, and few assignments involve substantive research, analysis, or argument (Kiuhara et al., 2009). This is a concern because many high school students are preparing for their professional and academic futures, and writing challenges are a significant hurdle that impacts students’ college and job prospects.
Students were recruited via flyers distributed by their classroom teachers and received US$30.00 for their participation. Most students completed the study in an after-school setting in their own schools, although researchers accommodated several students’ schedules by offering evening and weekend sessions in a university lab setting. Students were separated by one or more desk spaces, or with tall dividers between them, to prevent viewing of others’ work or computer screens. Students were also monitored by researchers at all times, and researchers enforced a quiet environment of individual work.
Demographic data were obtained via a background survey completed at the beginning of the study. A slight majority of students self-identified as female (57.7%), and students were identified as primarily Hispanic (42.3%) or Caucasian (29.4%), although the sample also included African American (14.1%), Asian (2.5%), or other backgrounds (11.7%). Most students reported English as their first language (80.4%). Finally, slightly less than half of students self-reported a GPA greater than 3.0 (42.9%). There were no differences in background across students who studied either topic and no differences across the three experimental conditions.
Students were asked to estimate their daily computer use and overall discomfort with using computers. Students expressed a range of responses: little to no use (6.7%), less than 1 hour (22.1%), 1 to 2 hours (30.7%), 3 to 4 hours (25.2%), and 5 or more hours (15.3%), but most students used computers at least 1 hour per day (71.3%). Computer discomfort was rated on a 6-point scale from “1” (very comfortable) to “6” (very uncomfortable). Students reported a high level of comfort (M = 1.7, SD = 0.8). Computer use and comfort did not differ across learning domain or experimental condition.
Procedure and Conditions
The study occurred over three phases in a single 2-hour session per student. In the pretest phase, participants completed an assessment of their prior writing strategy knowledge pertaining to body and conclusion paragraphs. Students also completed a survey measure regarding their attitudes toward writing strategies and practice. Finally, as a measure of prior reading ability, students completed the Gates-MacGinitie Vocabulary Test.
In the learning and practice phase, all students interacted with strategy lessons and practice activities for either Body Building or Conclusion Building topics. Students were randomly assigned to one of three practice activities within their topic: essay practice, strategy practice, or game-based strategy practice.
Finally, in the posttest phase, students completed a second assessment of their strategy knowledge. Because each student only studied one of the two topics, questions on the “studied” topic assessed learning while questions on the “unstudied” topic served as a control. Students also completed a posttest survey of writing attitudes.
Essay practice condition
Essay-based practice was implemented by assigning students to interact with W-Pal AWE tools. Students were given 25 minutes to compose a prompt-based argument essay, after which the essays were automatically submitted to W-Pal for scoring and formative feedback. Students were then allotted 10 additional minutes to revise. Overall, students spent about 40 minutes in this writing practice activity.
Students were randomly assigned to one of two practice prompts (see Appendix A). They were instructed to take a stance on the assigned issue and then to develop and defend their point of view on the topic using relevant evidence and examples. Two prompts were chosen to ensure that any observed results were not simply due to a single idiosyncratic prompt selection. Analyses (not reported for brevity) confirmed that both prompts performed similarly. There were no differences at pretest, posttest, or in strategy gains.
Game-based strategy practice condition
Students who interacted with the Body Building module played two games: RoBoCo and Fix It: Body Building. Students who interacted with the Conclusion Building module also played two games: Lockdown and Fix It: Conclusion Building. Both games were played for about 20 minutes. Thus, game-based practice duration closely matched the duration of essay-based practice.
Strategy practice condition
We created “non-game” versions of all four practice games. Specifically, RoBoCo was replaced with the Topic and Evidence Writer practice task; Fix It: Body Building was replaced with the Body Paragraph Editor task; Lockdown was replaced with the Conclusion Writer task; and Fix It: Conclusion Building was replaced with the Conclusion Editor task. All game narratives, mechanics, character graphics, and similar elements were removed, but the writing and strategy identification tasks remained unchanged. Students continued to receive feedback and other indicators of right and wrong answers. For example, in the Body Paragraph Editor task (i.e., the non-game alternative to Fix It: Body Building), students read a series of paragraphs, identified the primary flaw exhibited, and attempted to recommend a relevant solution. However, the fictional narrative of collecting “golden circuits” was removed, and students did not complete a Sudoku-like puzzle or earn a “game score.”
Students practiced their two assigned activities for 20 minutes each, thus equating time-on-task across all three conditions.
Measures
Strategy knowledge assessments
Students’ knowledge of Body Building and Conclusion Building were assessed via two quizzes (see Appendix B). Each quiz comprised eight multiple-choice questions related to writing strategies taught in W-Pal modules. For example, students were asked about the functions of body paragraphs or conclusion paragraphs, to identify strategies for improving each type of paragraph, and to recognize features of “good” and “bad” example paragraphs. Incorrect foils were crafted to seem plausible and to align with typical misconceptions. The order of questions and answer foils was randomized across pretest and posttest versions to ensure that questions and answers never appeared in the same position.
An example Body Building question asked, “Why is it important to include a body paragraph in an essay?” The correct answer stated that “Body paragraphs support the arguments in the essay.” Incorrect foils suggested alternative functions including making an essay longer, stating the purpose of the essay, and providing a summary of the arguments. An example Conclusion Building question asked, “Which of the following is a good way to summarize your main ideas in the conclusion paragraph?” The correct choice stated, “Describe how your ideas connect to the main argument.” Incorrect foils suggested strategies such as simply making a bulleted list, adding in new dates and facts, or ignoring the issue because main ideas were already summarized in the body of the essay.
Knowledge assessments were scored by tallying the number of correct responses. Thus, each topic quiz could range in score from “0” to “8.”
Writing attitudes survey
Students’ attitudes toward writing, strategies, and practice were examined via 12 probes constructed by the researchers (see Appendix C). Students rated their agreement with a series of straightforward statements about writing and practice using a 6-point Likert scale.
Because this measure was not based on a specific, preexisting scale, an exploratory factor analysis using principal axis factoring and oblique rotation examined whether all pretest attitude items loaded on a single factor or were perhaps separable. In fact, three factors emerged with eigenvalues greater than 1.00 and which cumulatively accounted for 74.82% of the variance. Six items loaded on the first factor (loadings > .62), which comprised attitudes regarding enjoyment and interest in writing practice and strategies (e.g., “Writing practice is entertaining” and “I am excited about writing strategies”). Two items loaded on the second factor (loadings > .72), which related to perceptions of importance and value of strategies and practice (i.e., “It is important for me to know writing strategies” and “I learn valuable things through writing practice”). Finally, three items loaded on a third item (loadings > .81) and appeared to relate to general attitudes about writing (e.g., “I think writing is interesting” and “Writing fascinates me”). One item failed to load on any factor (“During writing practice I need something to grab my attention”) and was excluded. Similar analyses were conducted for posttest attitude items—three factors again emerged that comprised the exact same items.
Based on the aforementioned patterns, three composite pretest and posttest attitude scores were created by averaging the ratings for a given factor: interest in strategies and practice, value of strategies and practice, and interest in writing.
Prior reading ability
Prior reading ability and vocabulary knowledge were assessed via the Gates-MacGinitie Vocabulary Test (4th ed., Form S) level 10/12 (MacGinitie & MacGinitie, 1989). The vocabulary section contains 45 simple sentences with each sentence containing an underlined word. Students were asked to select the word that was most closely related to the underlined word from a list of five choices. Students were given the standard instructions, two practice questions, and 10 minutes to finish the test. Previous research has shown positive correlations between scores on the Gates-MacGinitie Vocabulary Test and reading comprehension ability (e.g., Allen et al., 2014; r = .79).
Results
Whereas prior research focused on a single topic (i.e., Introduction Building), the current study investigated two topics (i.e., Body Building or Conclusion Building), although each student studied only one topic. We first analyze outcomes for the two topics separately to determine whether there are meaningful topic-specific effects. If patterns are overall similar for both topics, they can then be collapsed in subsequent analyses of the effects of prior reading ability.
Body Building Topic
Prior ability and pretest equivalence
Mean Prior Ability, Knowledge Assessment Scores, and Writing Attitudes by Condition for Students Who Studied the Body Building Topic.
Note. GMVT = Gates-MacGinitie Vocabulary Test.
Main effects of practice
Gains in strategy knowledge from pretest to posttest were assessed via repeated measures ANOVA with practice format entered as a between-subjects variable. After studying lessons on Body Building and engaging in assigned practice activities, students made moderate gains in Body Building strategy knowledge at the posttest, and exhibited no gains in Conclusion Building strategy knowledge (i.e., unstudied topic). Thus, interacting with instructional materials from W-Pal appeared to facilitate acquisition of strategy knowledge.
Students also demonstrated changes, albeit mixed, in their attitudes toward writing and writing practice. Students reported a significant decrease in writing interest from pretest to posttest. This result perhaps underscores the effort associated with practice. Learning new skills and engaging in practice can generate fatigue, which may translate into transitory decreases in general motivation for writing. However, we also observed a small increase in students’ interest in writing practice. Thus, interacting with some of the W-Pal practice activities might have made the act of practicing more pleasant (despite also leading to fatigue). There were no changes in students’ attitudes about the value of practice, which tended to be positive at pretest and remain so throughout the study (i.e., students endorsed the value of practice).
Interaction with practice format
There were no significant interactions between practice format and learning gains for either the Body Building topic (i.e., studied topic), F(2,78) < 1.00, p = .555, partial η2 = .015, or Conclusion Building topic (i.e., unstudied topic), F(2,78) < 1.00, p = .839, partial η2 = .004. Likewise, there were no significant interactions between practice format and changes in interest in writing practice, F(2,78) < 1.00, p = .809, partial η2 = .005, value of writing practice, F(2,78) < 1.00, p = .893, partial η2 = .003, or general attitudes toward writing, F(2,78) < 1.00, p = .710, partial η2 = .009.
Conclusion Building Topic
Prior ability and pretest
Mean Prior Ability, Knowledge Assessment Scores, and Writing Attitudes by Condition for Students Who Studied the Conclusion Building Topic.
Note. GMVT = Gates-MacGinitie Vocabulary Test.
Main effects of practice
Students exhibited significant gains in Conclusion Building strategy knowledge. Students also demonstrated a significant decrease in scores on the Body Building strategy knowledge quiz (i.e., the unstudied topic). As stated earlier, studying instructional materials provided by W-Pal seemed to support strategy knowledge acquisition for the studied topic. Students’ scores for the unstudied topic perhaps decreased because those questions were not salient during study and practice, and thus they made no effort to learn those topics. Another possibility is that negative transfer (see Schwartz, Chase, & Bransford, 2012) occurred between topics. For example, the properties of good conclusions (i.e., avoiding offering new evidence) somewhat conflict with properties of good body paragraphs (i.e., providing elaborated evidence), perhaps leading to confusion. After studying Conclusion Building strategies (and without the opportunity to study Body Building strategies), students may have misapplied the new strategies.
No significant positive or negative changes were observed for writing attitudes and interests. Except for a small gain in perceptions of the importance of writing practice, students’ attitudes were stable across pretest and posttest administrations of the survey.
Effects of practice format
There were no significant interactions between practice format and learning gains for either the Conclusion Building topic (i.e., studied topic), F(2,78) < 1.00, p = .865, partial η2 = .004, or Body Building topic (i.e., unstudied topic), F(2,78) < 1.34, p = .267, partial η2 = .033. Similarly, there were no significant interactions between practice format and changes in interest in writing practice, F(2,78) = 1.92, p = .154, partial η2 = .046, value of writing practice, F(2,78) = 1.04, p = .360, partial η2 = .026, or general attitudes toward writing, F(2,78) < 1.00, p = .707, partial η2 = .009.
Effects of Prior Reading Ability
The preceding analyses suggest that students learned about writing strategies via W-Pal although attitudinal outcomes were mixed. More importantly, these results also suggested that the main effects of practice format were largely equivalent; no differences in learning or attitude were observed depending on whether students practiced by writing an essay, engaging in targeted strategy practice, or playing strategy-driven games. In our final analyses, we evaluate how the effects of practice format potentially depend on students’ reading skills. As described previously, less skilled readers could find certain kinds of practice (e.g., games) more enjoyable or less taxing, whereas skilled readers benefit from all forms of practice and particularly more integrative formats (e.g., essays).
For simplicity, and due to overall similar patterns for students who studied Body Building or Conclusion Building, data were aggregated across topics for this analysis. Pretest and posttest strategy knowledge scores were extracted for each students’ studied topic only.
Exploratory linear models investigated how pretest scores and attitudes, practice format, and prior reading ability interacted to predict respective posttest scores and attitudes. We examined (a) main effects of pretest, practice format, and reading ability; (b) two-way interactions between pretest and format, pretest and reading ability, and reading ability and format; and (c) three-way interactions. For brevity, we summarize these trends here but do not report the full models or statistics.
Overall, across most measures (i.e., topic knowledge, practice interest, practice value, and writing interest), we observed no evidence of a main effect of practice format—knowledge and motivation gains were not significantly influenced by practice format alone. However, pretest measures and prior reading ability were often significant predictors of posttest measures and demonstrated significant two-way interactions. For example, students who possessed more strategy knowledge and better reading ability at pretest also demonstrated higher strategy knowledge at posttest. Likewise, students who were skilled readers and who believed (at pretest) that writing practice is more valuable were more likely to view writing practice as valuable at posttest. In such cases, two-way interactions between pretest measures and reading ability also suggested additive effects. Stronger knowledge or more positive attitudes from the outset combined productively with reading skill to elicit better outcomes. Three-way interactions were difficult to interpret in these models, however.
Mean Strategy Knowledge and Attitude Scores by Practice Format and Prior Reading Ability.
Overall, collapsing across studied topics, students demonstrated significant gains in strategy knowledge, F(1,157) = 15.41, p < .001, partial η2 = .089. We also observed an overall increase in interest in practice, F(1,157) = 4.64, p = .033, partial η2 = .029; and a possible increase in perceived value of practice, F(1,157) = 3.20, p = .075, partial η2 = .02. General interest in writing decreased from pretest to posttest, F(1,157) = 5.85, p = .017, partial η2 = .036. Students learned about writing strategies from studying and practicing with W-Pal modules, and seemed to slightly improve their attitudes toward practice. However, they likely experienced some fatigue regarding writing and writing-related practice.
There were no significant two-way interactions between studying and practice format for either strategy knowledge or attitudes. In addition, we observed no significant two-way interactions between studying and prior reading ability. However, we did observe main effects of prior reading ability (i.e., collapsed across pretest and posttest). Skilled readers knew more about writing strategies than less skilled readers, F(1,157) = 23.48, p < .001, partial η2 = .130, and reported a slightly greater interest in writing than less skilled readers, F(1,157) = 3.39, p = .068, partial η2 = .021. There was no difference in attitudes toward writing practice as a function of prior reading ability.
Tests of three-way interactions between studying, practice format, and prior reading ability revealed several tentative relationships. Most notably, a three-way interaction was found for strategy knowledge gains (see Table 3). Skilled readers appeared to benefit most from essay-based practice (d = .88), but less skilled readers benefited most from game-based practice (d = .62). Three-way interactions related to motivation were only marginally significant or nonsignificant. Nonetheless, it is worth noting that a fairly consistent pattern emerged that warrants future investigation. Skilled readers appeared to find essay-based practice and game-based practice more engaging than targeted strategy practice without games. That is, skilled readers increased somewhat in their interest and value in writing practice in essay and game contexts but exhibited a slight decrease in attitudes in strategy-only practice. In contrast, less skilled readers appeared to be more engaged by game-based and targeted strategy practice (i.e., small gains in practice-related attitudes) than essay-based practice (i.e., slight decrement in practice-related attitudes).
Finally, overall writing interest decreased somewhat for both more and less skilled writers. Again, we attribute this effect to fatigue associated with most forms of practice and skill acquisition. This decrement seemed to be stronger for skilled readers in all contexts, perhaps because they least needed writing instruction and practice and thus were the least motivationally engaged by the tasks in the study.
Discussion
This study considered the relative merits of different forms of practice for writing strategy acquisition (RQ1) and writing attitudes (RQ2), and considered whether students’ prior literacy ability led to differential benefits of different practice formats (RQ3). Specifically, using the W-Pal system, the study assessed how essay-based practice, strategy practice, and game-based strategy practice contributed to high school adolescents’ learning about writing strategies and attitudes toward writing and practice.
Aligning Practice Format, Instructional Goals, and Student Characteristics
Current results corroborated prior research (e.g., Proske et al., 2014) suggesting that a variety of practice formats are viable for writing strategy instruction in educational technologies. Students gained in strategy knowledge regardless of whether they practiced writing essays, completed strategy-based practice tasks, or played educational games. Similarly, we observed somewhat positive changes in attitudes toward practice across formats. Thus, no singular approach was best, but neither did the data offer strong injunctions against any practice method. An important implication for educational technology developers and educators is that optimal practice design must transcend narrow metrics of learning efficacy (i.e., identifying the one “most effective” format) or engagement (i.e., identifying the one “most fun” format), despite the appeal of reducing costs or streamlining the software. As is the case in traditional writing instruction, there are multiple pathways to writing development (Graham et al., 2013; Graham & Perin, 2007).
Logistical constraints such as ease of implementation, cost, and time remain nonnegligible; yet, they are not sufficient justification to deny students diverse practice opportunities. For example, many AWE developers have already invested in robust essay-based practice and formative feedback functions. And, within that space, there remain many unanswered questions about how to design valid and valued automated writing feedback that will be trusted and used by students (Dikli & Bleyle, 2014; Roscoe, Wilson, Johnson, & Mayra, 2017; Stevenson & Phakiti, 2014). Undoubtedly, there are ample opportunities to improve automated supports for writing instruction through innovations that provide better feedback. To what extent is it desirable to iteratively improve existing tools rather than to “shoehorn in” other practice formats such as educational games? Game design is difficult and expensive.
Indeed, one possible limitation of W-Pal (and the current study) is that its development team did not include professional game designers. Games such as Fix It, RoBoCo, and LockDown, although functional and instructionally viable, might not have met students’ standards for contemporary digital games. Although not manipulated in this study, it is plausible that subtle issues of game mechanics or aesthetics influence the impact of games on learning. Nonetheless, multiple prior studies using W-Pal have reported that students do find the games helpful and engaging, and sometimes visually pleasing, despite their lack of professional polish (e.g., Allen et al., 2014; Roscoe et al., 2013; Roscoe & McNamara, 2013). And, when games do incorporate more sophisticated graphics and narratives, the benefits are stronger (e.g., Barab et al., 2012; Dickey, 2011).
Gaming and motivation are also complex, and there is not a simple or straightforward connection between game features and student engagement (Garris, Ahlers, & Driskell, 2002; Wouters & van Oostendoorp, 2016). In this study, we probed attitudes toward practice and writing via straightforward, researcher-constructed items (e.g., agreement with statements such as “I learn valuable things through writing practice”). Results were mixed, showing both improvements and decrements in attitudes, and did not differ based on practice format. However, students’ feelings toward writing are multidimensional (Troia et al., 2012). In addition to interest, prior work has revealed the importance of self-efficacy (Bruning, Dempsey, Kauffman, McKim, & Zumbrunn, 2013; Pajares, Johnson, & Usher, 2007), epistemic beliefs about writing (Sanders-Reio, Alexander, Reio, & Newman, 2014; White & Bruning, 2005), and more. Practice format may interact with each (or none) of these factors, and such interactions might be mediated or moderated by students’ prior ability or other individual differences. Thus, in future work, it will be important to include additional motivational constructs and predictors.
Most importantly, the current study observed interactions between prior reading ability and practice format. Skilled readers appeared to gain the most from essay-based practice whereas less skilled readers gained most from game-based practice, which aligns to prior work on whole- versus part-task training methods (Lim et al., 2009; Salden et al., 2006; Wickens et al., 2012, 2013) and connections between reading and writing skill development (Fitzgerald & Shanahan, 2000; Shanahan, 2015). We argue that essay-based practice allowed skilled readers to apply their prior knowledge and newly acquired strategies in the context of a complete composition. Skilled readers knew more about writing and strategies, and gained more from the lessons and practice activities. This is unsurprising given that materials were presented in primarily verbal and textual format. Skilled readers were likely better at comprehending the presented information and could link the new information to existing knowledge or mental models. As a result, these students were ready to capitalize on and benefit from the more complex and integrative nature of whole-task, essay-based practice. Skilled readers may have been able to leverage their vocabulary and text comprehension abilities to better reflect on the meaning of their new strategies in relation to planning, drafting, and revising.
In contrast, less skilled readers benefitted more from practice that decomposed and targeted specific strategies and skills, and which embedded such practice in a game setting. These readers likely lacked a strong repertoire of literacy knowledge and skills to draw upon, and the structured format of game-based practice provided necessary scaffolding for their strategy acquisition. Less skilled readers may have struggled with simultaneously internalizing new strategies and coordinating the full writing process, whereas game-based strategy practice better aligned with their needs. For developers and educators, such findings imply that technologies need to assess students’ existing knowledge and skills (perhaps through stealth assessments conducted as students use the system; Shute & Ventura, 2013), and then offer or recommend practice activities tailored to those abilities.
Finally, other aspects of writing development go beyond strategy knowledge or enjoyment to encompass communication and interpersonal interactions, understanding of audience, and even understanding of oneself. For instance, giving, receiving, analyzing, and implementing feedback are essential to writing (e.g., Parr & Timperley, 2010; Patchan, Schunn, & Correnti, 2016). Students need to learn not only how to compose texts but also how to interact with an audience and respond to comments. Iterative essay-based practice with automated feedback could nurture these skills by helping students appreciate the effects of responding to feedback across essay drafts. Students can reflect on how their essay composition was evaluated holistically or with respect to select features. Alternatively, games and narratives have been used to explore how students view themselves, their roles, and potential careers (i.e., identity formation; Barab, Gresalfi, & Ingram-Goble, 2010; Siyahhan, Ingram-Goble, Barab, & Solomou, 2017), and games for writing could serve a similar aim (Warren et al., 2009). Rather than “writing for the sake of writing,” games may introduce students to new roles and goals that lend an additional sense of purpose. Through gaming, for example, students might explore the tasks and responsibilities of investigative journalists or scientists. In these roles, students might not only learn about the importance of verifiable sources, data, clarity, or cohesion, but they could also become inspired to pursue writing-based careers. In short, the design of activities for computer-supported writing practice may need to consider a broader array of instructional goals and alignments with learner, educator, and curricular needs.
Conclusion
In the classroom, skilled teachers can readily adapt their lessons and practice assignments to the needs of individual learners. However, similar levels of flexibility and personalization can be difficult to achieve in computer-based learning environments. Each minor or major variation in potential instructional tasks must be built in to the system, often requiring significant time or money. Consequently, educational technology developers strive to carefully assess how and whether to expend limited resources to create or incorporate different materials, practice opportunities, and feedback along with the artificial intelligence and natural language algorithms that guide delivery of these features. When an optimal balance is struck, educational technologies can provide powerful tools to supplement writing instruction. Our findings suggest that developers, despite costs or logistics, should not sidestep the importance of multiple and diverse practice opportunities. The “ideal” form of writing practice might not be driven by the format itself, but how well the system (like a classroom teacher) can align instructional tasks, goals, and student characteristics. Given that most educational technologies must serve diverse populations of learners, an important implication is that technology developers must aim for flexibility—offering multiple forms and types of practice whenever feasible.
Footnotes
Appendix A: Writing Prompts
Appendix B: Strategy Knowledge Questions
Appendix C: Writing Attitudes Survey
Acknowledgments
The authors thank Russell Brandon and Devin Russell for their contributions to pilot testing and data collection.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Institute of Education Sciences, US Department of Education, through Grant R305A120707. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the IES.