Engagement and Games for Learning

Abstract

Aim. In this article, we argue for an expanded definition and treatment of engagement when studying games for learning.

Background. It is common for researchers in the field to collapse behavioral, cognitive, and affective engagement into one shared category. We assert that educators and games for learning researchers should examine all types of engagement using multiple methodologies as a means to better understand precisely what students are learning from game play.

Method. We present findings from a small (N = 58) qualitative study that highlights the importance of accounting for behavioral, cognitive, and affective engagement when evaluating learning from games. We also discuss methodologies that are successful for this type of evaluation.

Results. Finally, we discuss the implications of our findings for those creating games for learning and for measuring learning gains.

Keywords

affective behavioral cognitive definitions engagement games for learning methodology treatments video game play

In educational research, engagement is often cited as being viewed along three axes: behavioral, cognitive, and affective (Fredericks, Blumenfeld, & Paris, 2004). More specifically, behavioral engagement is typically defined as school attendance, involvement in academic tasks, and the level of compliance with school-related norms such as rule-following and conflict avoidance (Karweit, 1989; Peterson, Swing, Stark, & Wass, 1984). In this type of engagement, behaviors are observable and easily measurable. Cognitive engagement typically refers to a psychological investment in academic activities, competence in subject matter, and metacognition (Boekarts, Pintrich, & Zeidner, 2000; Zimmerman, 1990). Affective engagement¹ is the broadest of the three categories and encompasses emotional reactions to schooling, relationships with other actors in, and related to, school, and feelings about the value of schooling (Eccles et al., 1993).

Engagement as discussed in the games for learning literature generally appears to fall along the behavioral axis of engagement as defined by education research. It is typical to find games studies that use individual or collapsed engagement constructs to measure behavioral and cognitive engagement such as persistence, time-on-task, flow, immersion, and commitment (Baker, D’Mello, Mercedes, Rodrigo, & Graesser, 2010; Csikszentmihalyi, 1990; Ermi & Mäyrä, 2005; Grimshaw, Charlton, & Jagger, 2011). Although these studies do not ignore affective engagement, we assert that affective and emotional engagement should be considered with equal importance as behavioral and cognitive engagement. We suggest that without an understanding of all three types of engagement, including players’ emotions and attitudes, we are limiting our understanding of engagement in various contexts.

In this article, we present findings from a small (N = 58) qualitative study that highlights the importance of accounting for behavioral, cognitive, and affective engagement during game play. In addition, we discuss qualitative methodologies, including think-alouds, interviews, and observations that may help researchers examine the three types of engagement during game play more deeply, leading to a much richer understanding of what influences learning via game play.

The overarching research questions that guided this study were as follows:

Research Question 1: How can we characterize students’ behavioral, cognitive, and affective engagement during game play as it relates to learning?

Research Question 2: How does this information influence our interpretation of data from game analytics?

Research Question 3: How does the use of different methodologies serve to highlight different types of engagement?

Framework

We draw on theories of school engagement and sociocultural and ecological theory as it relates to learning in video games to better understand how students engage with games. We understand this engagement as occurring at the intersection of the individual and the setting. The context here includes both the school and the game settings. In this section, we elaborate on the ways that we view engagement, both as something individuals do and as something game contexts make available or unavailable.

Engagement

The positive relationship between school engagement and school achievement has been a robust finding in the research literature. Students who are connected and engaged with school are more likely to attend, succeed academically, graduate, and avoid engaging in delinquent behaviors (Connell, Halpem-Felsher, Clifford, Crichlow, & Usinger, 1995; Schapps, 2003; Wentzel, 1998; Wilson & Elliott, 2003). Therefore, it is important to deeply study the ways in which students engage with educational tasks and content. For our purposes, this involves how students engage with educational games. We draw on the three categories of engagement: behavioral, cognitive, and affective (as previously defined), as we consider the types of ways in which students are engaged with games for learning.

However, in the video game studies that we report, it is helpful to frame all three of these categories a bit differently so as to translate between educational and video game communities, thereby discussing engagement with educational games rather than just engagement with educational tasks. Behavioral engagement can be correlated to behavioral theory, as games can be viewed as systems of rewards and reinforcements, while acknowledging that this characterization is not adequate in explaining the complexity of play (Salen & Zimmerman, 2003). Behavioral engagement may be one aspect in which a player engages in game play, but to classify game play only as behavioral oversimplifies the player experience. For example, when training someone to learn using behavioral methods, systems of rewards are typically established to reinforce the learning of the desired behavior. In video games, a desired behavior is often long-term game play, and attributes exist within a game system that may function to reward this behavior; however, multiple attributes usually keep players engaged.

Cognitive engagement can be likened to immersion in game play. Immersion can be thought of as absorbing involvement in game play—when players are immersed, they may lose their sense of time and their awareness of their surroundings. Immersion may relate to how serious a player is about playing the game, as well as the contextual factors that shape player activity (Consalvo, 2009; Rodriguez, 2006). However, although we equate cognitive engagement with immersion, immersion does not satisfy our definition of cognitive engagement. In the education community, we also add concepts such as thinking, learning, and metacognition to round out this definition.

Emotional or affective engagement with video games has been identified as one of the motivating components of game play (Gee, 2008). A few of the types of things that have been cited as being emotionally engaging are risks and challenges in high-stakes game play, emotional investment in player characters, the formation and maintenance of social groups (e.g., guilds), and powerful game narratives (e.g., FINAL FANTASY VII, 1997). Although we do not disagree with this definition of engagement, we have sometimes found that affective engagement is not treated as its own entity. Rather, the concept of “flow” is used to measure this type of engagement (Csikszentmihalyi, 1990). In using “flow,” a system that collapses behavioral, cognitive, and affective engagement into one larger category, the importance of affect and emotion are minimized rather than highlighted. In addition, we question how researchers can adequately measure flow, given that it is a state of total immersion.

In games for learning, in particular, motivation and engagement are often assumed to carry the potential to increase learning gains. One way of framing the relationship between motivation, engagement, and learning is as persistent re-engagement; where learners are characterized as those who typically opt to participate, choose challenging tasks, stay focused, and are committed to completing the task. These characteristics are also typical of successful video game play (Garris, Ahlers, & Driskell, 2002). However, the success of creating a game that supports persistent re-engagement in play and in learning is contingent upon a number of different conditions, including the quality of the game, the context in which the game is played, the learning supports around the game, and students’ feelings about or attitudes toward a game.

In our work, we highlight context in relation to affect and emotion. Student feelings and emotions are particularly important here, as these factors can determine whether a student will engage with a game. If a student has positive affect toward a game, he or she will continue to play and persist. However, if a learner’s affect is negative, he or she is likely to disengage from the game. The context in which a student plays will often influence these emotions. For example, if a student is part of a classroom environment in which the teacher encourages and lauds the use of technology, a student might be more likely to feel positively toward games for learning. Context, which can include communities of practice, guided participation and the socio-material is supported by sociocultural theories of learning and development (Lave & Wenger, 1991; Rogoff, 2003; Vygotsky, 1978), which highlight how individual learning and developmental processes are fundamentally connected to their social and cultural contexts.

In Toward an Ecology of Gaming, Salen (2008) describes an “overall ‘ecology’ of gaming, game design, and play, in the sense of how all of the various elements—from code to rhetoric to social practices and aesthetics—cohabit and populate the game world” (p. 2). Looking at the learning ecologies, hidden agendas, and gaming literacies as conceptual categories allows us to think of gaming beyond just the game and play state to also consider the game and its context as a complex system. It is also important for us to think about this ecology beyond the usual measures of persistence, time-on-task, flow, immersion, and commitment.

This study took behavioral engagement, cognitive engagement, and affective engagement as part of the overall ecology of the game and learning. Behavioral and cognitive engagement were examined from both educational and gaming paradigms—emotional or affective engagement was examined largely from an educational perspective, as the field of learning and education have focused on this type of engagement more deeply overall. In addition, multiple methodologies were used in examining these types of engagement.

Method

Setting and Participants

Setting

A mid-sized elementary school in an urban area that serves as a lab school for a large university in the Southern/Central part of the United States. As of the 2011-2012 school year, the school serves 260 students, and the demographic makeup is 72% Hispanic, 16% African American, 11% White, and 1% Asian. Sixty-eight percent of the students qualify for free or reduced lunch.

Participants

Participants in the first part of the study were 28 fourth and 30 fifth graders (ages 8-10) from the elementary school. During interviews, students were asked about their experiences with games. Most of the students reported playing video games and board games. We found a wide range of responses with respect to how much time students spent playing games outside of school, from almost no game play to those who played daily. The majority seemed to fall somewhere in the middle. All of the students had access to computers and the Internet outside of school. We found no bias toward platform (e.g., console, mobile, or PC) for the games they played. The students mentioned a variety of game types: card games, trading games, board games, fighting games, sports, and so on. Student frequently mentioned playing educational games, but we do not know if this has to do with parental control over the content of gaming or if the students opted to play these games themselves.

Procedures

Students were taken out of their classrooms and they worked one-to-one with one of four researchers for a 20-minute session of game play and simultaneous think-alouds.² The students played a fraction puzzle game, representative of the idea of partitioning or splitting, named REFRACTION (2010; http://games.cs.washington.edu/refraction/; see Figures 1 and 2). The purpose of the game is to direct a laser using a sequential series of “benders” and “splitters” toward a spaceship in order to power it. The splitters allow the player to split the laser into different sizes represented by fractions. The power and size of the laser varies by level as well as navigating the laser around fixed obstacles.

Figure 1.

Refraction level with one-third ships.

Figure 2.

Refraction level solved for one-third.

Each student played REFRACTION and answered fraction assessment items embedded in the game. During game play, students were videotaped using the software iShowU. This software uses the webcam on the computer to record the students face in parallel with recording all audio on-screen activity. This allowed researchers to watch game play synced with player video and audio recordings. During the session, students were asked how they were thinking and feeling throughout game play, as well as about the choices that they were making while playing the game (“think-alouds”). A subset of students (9 fourth graders and 15 fifth graders) were interviewed for approximately 10 minutes following game play. Only one researcher in the study conducted post–game play interviews; therefore, the students who were randomly selected to work with that researcher were interviewed. Students who were interviewed were also asked questions about their interest in math, gaming, and other topics related to math and games.

Debriefing

As part of this study, no formal debriefing was undertaken. Students who only took part in the think-alouds and game play did not have the opportunity to debrief as time with those students was limited and we chose to devote that time to game play. For students who took part in the post–game play interview process, those interviews may be considered a type of debriefing, although this was not necessarily our intention in interviewing the students. Many of our interview questions asked students to reflect upon their thoughts and feelings about the game, which may also be considered as part of a debriefing process.

Game progression

As mentioned, each student spent approximately 20 minutes with each researcher. During this time, the students were introduced to the game, completed math-related assessments, and had the opportunity to engage in non-assessment game play. When students first logged into the game, they completed four tutorial levels that explained the purpose of the game and how to play the game. Both of the case study students had played the game in school the previous year so the tutorial was a review.

Immediately following the tutorial is what we refer to as the embedded assessment level (see Figure 3).

Figure 3.

Marcus playing the embedded assessment level.

This level asks students to solve a more complicated laser splitting problem in which they have to “satisfy” ships by feeding one-sixth of a laser into one ship and one-ninth of a laser into another ship. This level is considered more advanced than other levels because it is not only difficult spatially to navigate the laser toward the spaceships, but it also requires the combination of multiple benders and splitters in a specific sequence (i.e., a three-splitter has to be used before a two-splitter to satisfy ships that require one-sixth and one-ninth lasers) in order to create the appropriate sized laser for each ship. This level was also the only timed level in the game—students had 2 minutes to complete the level before they were automatically forwarded to a set of school-like assessment questions. We created this embedded assessment level for a few reasons. First, we thought it would be diagnostic in terms of learning because we assumed that if a student understood how to make one-sixth and one-ninth, then they were grasping the splitting or partitioning math concepts. In addition, we also assumed that students had mastered some of the spatial problem solving required of being successful in this game. As the case studies suggest, the latter is true, while the former is not.

After the embedded assessment level, students immediately took a standard school-like assessment consisting of four screens. For these assessments, students had to demonstrate knowledge of fraction comparisons, conceptual understanding of fraction splitting, as well as fraction addition not embedded in game play. These assessments were based on previously validated math assessment items and used as a diagnostic tool for helping us determine students’ understanding of fractions. Following these assessments, students were returned to regular game play and used the rest of the session to progress through the game.

Analysis

For analysis purposes, we first analyzed post–game play interview data for general understanding related to students’ emotional and affective engagement with the game. We then constructed case studies to extend those findings. In addition, we have access to extensive game analytics, including time played, number of clicks, board states, and so on. The think-aloud data helped us assess cognitive and affective engagement during game play. Game analytics and assessment data helped illuminate other elements of cognitive engagement as well as behavioral engagement. Post–game play interview data deepened our understanding of students’ emotional engagement with the game.

Interview data

Interview data were transcribed into individual Word documents and imported to the qualitative research program HyperResearch. Data were then sorted into the following categories based on interview questions: Affect, Game Experience, Hints, Learning, and Perceived Purpose of Game. The sorted data were reviewed and it was decided that for present purposes, the sections related to Affect, Games Experience, and Learning would be presented. These categories were chosen as they were richer and more coherent than the others.

Following this sorting, the data were read by two researchers and subjected to several rounds of open and axial coding. Discussion and refinement of codes occurred between each round of coding. Once a final set of codes and their definitions were created (see the appendix), two researchers completed the coding of the transcripts. Ten percent of the interview transcripts were double coded for reliability and inter-rater reliability was 90%.

Case study data

Based on performance during game play, with a particular focus on game analytics, students who appeared to have performed similarly during game play and the think-alouds were chosen for further examination (discussed in more detail below). Then, two researchers examined the video of the students’ game play experience closely, field-noting from the videos to examine cognitive and affective engagement during the sessions. These field-notes, or running records, were then constructed into case studies, which we used to compare the overall experiences of the student.

Game analytics data

For all game play studies, we collect detailed game analytics such as number and types of player clicks, types of answers, number of moves per level, and time played per level and game. Detailed log data provide quantitative information about how the player was engaged with the game that can be further explained by video data.

Findings

In this section, two sets of findings will be discussed. First, we share findings from the post–game play interviews and highlight those findings in particular that could not be captured from solely examining game analytics or in-game assessment data. Then, we drill down further into the case study data to further illuminate marked differences in student experiences with the games, even when the associated analytics are similar. All of these responses to the game help us understand different facets of student engagement with the game, particularly in relation to cognitive and affective engagement, the measures that are difficult to discern from game analytic data alone.

Interviews

Students reported a variety of thoughts and feelings in relation to the game. In terms of affective engagement, they discussed likes and dislikes, emotional responses to the game, and feelings about learning math in this way. They also discussed their focus during game play, which further helped illuminate how they were engaged.

Likes and dislikes

Most students reported liking the game; however, they were divided concerning the reasons why and mostly fell into two categories in this respect. One group of students (n = 17) enjoyed non-math aspects of the game, such as the characters and the act of saving characters. The other group (n = 8) reported liking the game because it helped them learn math. Other students (n = 15) mentioned non-math aspects of the game and math at the same time. Surprisingly, few students disliked the embedded assessment items, which were the most “school like” tasks in the game, and said that they would do them at home if they were part of the game. Although measures such as time-on-task or persistence from game analytics could lead us to infer that students like a game, without this interview information, we would not understand why they like the game. Revealing a student’s reasons for liking or disliking an educational game in a school context helps us identify which aspects of the game contribute to engagement and learning.

Emotional responses to the game

Most of the students reported feeling positive emotions in response to the game (n = 15), and at the same time, many reported feeling aggravated or frustrated while playing the game (n = 13). Eleven students reported these emotions together. In addition, many students reported that this frustration made them feel determined to finish and persist, even when facing particularly difficult problems. One said, “I got frustrated, but it was like a good kind of frustrated, like when you—you’re like, oh, come on! When you’re trying to do something.” Understanding the types of feelings associated with game play, and justifications for types of feelings associated with game play helps us understand the analytics we collect from students better. It also helps us make sense of different kinds of patterns we find in those analytics, giving us the ability to make a game appropriately adaptive for students depending on their emotional states.

Learning via games

Overall, students expressed positive opinions about learning via REFRACTION. Thirteen reported that they think a game similar to this would help a friend like math more. Most of them reported that playing the game helped them like math more. Interestingly, although these students all reported liking math, many of them reported that their friends do not like math.

Most thought that if they continued to learn in this way they would like math even more than they already do. However, most of the students, in response to the question, “Would you like to learn more about math in a game like this, more like in school with your teacher, or maybe a little bit of both?” the majority said that they would like to do a little bit of both rather than the expected answer of, “in games like this.” However, a few students did say that they prefer to learn via games. When asked why, one student said, “And it teaches you about ... it teaches you about learning when you don’t even know that you’re learning.” This shows that some students position school learning and game learning differently, further driving the point that when evaluating games for learning we have to take into account the theoretical paradigms from both learning and games research. A game presented in a formal educational setting primes the player with the expectation that learning is involved in the game and not just play that will likely be tied to curriculum.

Discussion of interview data related to affective engagement

The emotions and responses mentioned in this section are typical of a game play experience, which signals that the students are often in a game play mode. Part of the goal of using games for learning is the assumption that game play often seems to lead to increased engagement—however, what engages a person in a game may not be the same as what engages that same person in school. As success ebbs and flows in the game, resultant emotions sometimes include frustration and confusion. These emotions, which are often viewed negatively in school, are essential to game play. Therefore, students’ overall affect and emotion must be understood within the parameters of a typical game-playing framework when examining games for learning.

However, with that in mind, it is difficult to assess through interview questions alone whether or not the emotions reported here are within the expected parameters. In addition, without attention to cognition as well as emotion, we risk losing valuable data that relate to a student’s learning and understanding during game play. However, including interview data with further analysis of video data for evidence of math understanding through game play, as reported via case study, deepens our findings. As shown in the next section, the details exposed in the case studies present the need to expand methodologies further to include qualitative analysis of game play, using a variety of methods, for understanding engagement in games for learning. We used think-alouds and video data, in addition to the post–game play interviews, to provide us with a fuller picture of engagement with the game.

Case Studies

For the case studies, we examine two students’ play sessions in depth. We discuss our criteria for choosing our cases and we highlight the similarities and differences across our two cases.

Case study criteria

Our criteria for choosing the students fell into three categories. First, we wanted to make sure that the students were of similar ability in terms of math, and therefore we picked two students who were average in terms of math ability as determined by state-administered standardized assessments. Second, we looked at some aspects of game analytics that demonstrated similar performance. For example, when examining speed of progression through the game, we wanted to make sure that our cases were progressing at relatively the same rate. Third, we chose to focus on two students who were particularly comfortable with a “think-aloud” protocol. We did not want to compare one student who was particularly vocal about his thinking and feeling to another student who was not vocal for fear of drawing incorrect conclusions about the less vocal student. It is important to note that we consider each of these categories to comprise surface features, as they give us information about each student, but not much insight into their personalities or identities as learners. Ultimately, we chose two students from the same fifth-grade class, Alicia and Marcus, to be the focus of our cases.

In the following two sections, we describe our cases. First, we give a qualitative overview of each student that includes a focus on his or her vocalized math understanding and general engagement and affect during the session. Then, we focus in on his or her treatment of the embedded assessment level. Finally, we discuss the overall implications of using qualitative data to strengthen our understanding of more surface and quantitative measures that we collect as part of game analytics. Throughout this section, we use the word diagnostic to mean how well the in-game assessments or the game analytics themselves are measuring student understanding and learning.

Alicia

Overall, it was clear early on that Alicia understood the math presented to her in the game. However, it did not appear that she was as adept with the spatial reasoning required of the game as she was with the math. In other words, her math ability was not matched to her spatial problem-solving ability, limiting her progression in the game. It was not clear initially that Alicia had a solid grasp on the math, but over the course of the game, it became clear that mathematically she had no problems with the ideas presented to her in REFRACTION.

Alicia was able to articulate her understanding of math concepts thoroughly while playing. Alicia’s general affect through this session was concentrated and subdued. She focused and paid attention to game play with minimal reminders to “think-aloud.” Although concentrated and somewhat reserved, she smiled regularly, showing a few moments of delight or surprise during game play. However, she was not overly animated or dramatic in those moments. Overall, Alicia was highly engaged with the game throughout the session.

In the embedded assessment level, the student needs to direct a laser to a one-sixth ship and a one-ninth ship. Upon starting this level, Alicia picked a three-way splitter to start, so the researcher asked her why she chose that splitter first and she quickly responded by saying,

Alicia:

Because then ... Because I needed to go down there (pointing to the one-ninth ship). Then, you go up there (pointing to the one-sixth ship).

Researcher:

OK.

Alicia:

And since it’s one-third, one-third is half of one-sixth.

This exchange demonstrates that Alicia was considering both the mathematical and spatial criteria needed for solving the level. In addition, the math talk and speed of response shows that she was considering the relationship between how one-third is related to one-sixth. Although mathematically one-sixth is a half of one-third, Alicia’s response reveals the mathematical thinking she used to reach her understanding.

However, Alicia did not seem to have the same grasp of how to make one-ninth. When discussing how to satisfy the one-ninth ship, she took longer to think aloud when asked and said, “I think one-thirds ... I think it’s ... it’s one ... no, it’s one-sixth of nine, I think.” The difference in time to respond and less articulate math talk around the question of how to solve for one-ninth indicates that she understands the relationship between one-third and one-sixth, but that she does not necessarily understand the relationship between one-third and one-ninth. Interestingly, however, when examining Alicia’s game play over time, it was revealed that she did understand how to make one-ninth, although her initial verbal explanation of the problem indicated otherwise.

Although Alicia demonstrated she understood the math, many other students who did not understand any of this math were able to solve this level because of their spatial reasoning ability via trial-and-error and a bit of luck. However, Alicia was not able to successfully complete the timed embedded assessment level before the 2 minutes mark. Had we only examined game analytics and not captured Alicia’s reasoning through the think-alouds, we would have made the assumption that Alicia did not understand the math in the level because she did not complete the level before the time was up. However, through think-aloud protocol, we know she does understand the math, but struggled with the spatial component of game play. Having captured Alicia’s reasoning, we were able to determine how she was making sense of the game play and the math. Consequently, if game progression and adaptivity were based only on game analytics, Alicia would have subsequently encountered math problems that were far below her math level. In addition, having revealed Alicia’s ability to demonstrate her understanding of math through game play was hindered not by the difficulty of the math problem, but by the difficulty of the spatial reasoning required to complete the puzzle. Although it is true that a different assessment might assist us in not making such errors, without first knowing her understanding, we would not even be able to assess the diagnostic value of our assessments. In the future, this will help to avoid making poor choices related to designing game adaptivity, but it also leads us to question what we deem diagnostic in game play in general.

Marcus

In general, Marcus demonstrated an understanding about the math involved in the game to some degree, but not as much as Alicia. Unlike Alicia, Marcus grasped the spatial reasoning and problem solving needed for the game immediately. This was evident as he moved quickly using a lot of trial-and-error in his game play. Marcus articulated his understanding frequently by declaring what he thought almost constantly as he played. When he seemed less certain in his understanding he usually posed his “think-aloud” in the form of a question to the researcher (the researcher responded to such questions with another question). His overall affect engagement was focused and lively, showing more excitement and interest than we saw from Alicia. He moved around a bit in his chair while playing, he typically had his face very close to the screen, and he reported non-critical actions as well as explained his problem solving and play strategy. Like Alicia, Marcus was highly immersed with the game throughout the session.

Recall that the embedded assessment level is a timed level that requires the student to direct the laser into a one-sixth ship and a one-ninth ship. In the embedded assessment level, Marcus demonstrated his thinking through trial and error.

Researcher:

All right. Now what? What do you think about this? (referring to the new level with a one-sixth ship and a one-ninth ship)

Marcus:

[whistles] They each need one-sixth. So, what I’ll do ... [selects three-splitter] ... is give them one-third ... [uses it] ... and then ... [selects another three-splitter; uses it] (placement of these splitters near the one-sixth ship indicating he is trying to solve for one-sixth first)

Researcher:

OK. So, why’d you pick that one?

Marcus:

Because ... Wait. Is that one-third? (referring to the three-way splitter)

Researcher:

I don’t know. You tell me.

Marcus:

Let’s see. Wait. What is it giving me? Oh, one-ninth! Ooh! (Realization that his combination of two three-way splitters makes one-ninth and solves for the other ship)

Researcher:

So, what are you gonna do?

Marcus:

Got to put it into half, I think. [moves three-splitter out of play] Wait. One-third. I need to split it in half, so I need this, I think ... [selects two-splitter] (He is still solving for the one-sixth ship)

Researcher:

OK. And why ... ?

Marcus:

If you split—

Researcher:

Yeah.

Marcus:

—it in half, I think that’s what happens.

Researcher:

OK.

Marcus:

[uses it] Is it there now?

Researcher:

And why’d you make that? How did you know that?

Marcus:

Um, how—how did I know that? I just ... well, it’s hard. It just came to me like that. [laughs]

Through trial and error, Marcus not only successfully completed the level, but also gained valuable knowledge for future levels. In a later level, the first to introduce one-ninth as a non-assessment level, Marcus explained how he solved for one-ninth:

Marcus:

[sighs] One-ninth.

Researcher:

All right. Now, how do you make one-ninth?

Marcus:

Now, I’m, uh. You split two of these one-thirds.

Researcher:

OK. How do you know that?

Marcus:

[selects two-splitter; uses it] Because I’ve made mistakes where I split 2 one-thirds. [chuckles]

Researcher:

Oh, OK. So, you learned from the last level?

Marcus:

Yeah. So, I’m just going to ...

We learn that Marcus’ ability to solve for one-ninth at the higher level is rooted in the previously played embedded assessment level. His learning about how to create one-ninth occurred from trial and error rather than from the deliberate and successful completion of the one-ninth ship in the embedded assessment level. This shows that although he now knows how to split into one-ninth, we also know that this understanding is not necessarily conceptual, but rather, procedural. We would have not gained this information from our analytics or our interviews—we needed the think-alouds, which showed Marcus’ thinking, to help us understand what learning still needed to happen.

Comparing cases

As our in-depth study of Alicia and Marcus shows, combining understanding of student engagement and thinking with game play analytics is vital to creating high-quality games for learning. Without an analysis of the behavioral, cognitive, and affective engagement through think-alouds and video analysis, not only could we misconstrue when learning does or does not happen via game play, but also we miss out on the opportunity to understand why learning happens and at what points during game play. Marcus’ trial-and-error play technique could have been interpreted as evidence of learning of introductory math concepts. Even though he did successfully complete the timed embedded assessment level, he did so by playing around and accidentally discovered that using 2 three-way splitters creates a laser that is one-ninth. We would not have known this without having an understanding of some of his thinking. We would not have known his thinking without first examining the different ways in which he was engaged with the games.

However, this accident provided him with a learning opportunity. When presented again with a one-ninth ship, he remembered his mistake and applied it as a solution in a future level. It could be argued that although he did not receive formal instruction on the mathematical relationship between one-third (as represented by the three-way splitter) and one-ninth, through dragging and dropping laser splitters a relationship was made visible (2 three-way lasers, creates one-ninth). However, his play strategy is different from Alicia’s. To contrast, she also did not complete the embedded assessment and did not have an accidental discovery that led to later game play success even though both students possess approximately the same level of math understanding. This difference in play strategy suggests that the design of the level could be improved to encourage the type of trial-and-error Marcus demonstrated. This could be achieved by eliminating the timer or embedding hints in earlier levels encouraging players to try multiple combinations if they are not sure how to achieve the amount.

It is clear from these case studies that while Marcus and Alicia were selected because some of their more descriptive game analytics appeared to be very similar (i.e., time played, levels completed), as were the other surface features we discussed when selecting our cases, their experiences were quite different in relation to the game. Marcus learned from the game, while Alicia arguably did not. Marcus was highly engaged and expressed more excitement in general about the game. Although Alicia was engaged, she was more subdued overall.

Discussion and Implications

Our findings, from both the interviews and the case studies, demonstrate the paucity of understanding provided to us by some game play analytics and limited measures for cognition, affect, and general engagement. The depth and variety of behavioral, cognitive, and affective engagement was only evident after considering the game play analytics in conjunction with player interviews and case studies of think-alouds. Post–game play interviews, or debriefing, with students after game play illustrated the need to better understand the player’s prior experiences, feelings, and beliefs about math, games, and learning, which helped in understanding their dispositions to playing games for learning and their in-game choices and preferences. In addition, as demonstrated in the case studies, in-game assessments were not found to provide sufficient information to fully appreciate how much the player was understanding or not understanding the math through game analytics alone. The case studies revealed that students’ understandings of math were better than demonstrated through game play. This further drives the point that researchers should consider using multiple qualitative methodologies and analysis of game play experiences, rather than just game play data, because student experiences were qualitatively different even when quantitatively similar.

Qualitative analysis of player engagement is especially important when designing games for learning that are intended to be adaptive to the learner. Games that are proposed to support learning based solely on game analytics may not provide a complete enough picture to explain the “why” behind the analytics. In other words, the resultant data may exhibit strong patterns that suggest learning, but without a deeper understanding of the player’s reasoning and affect of what happened during game play sessions, we cannot be certain that the desired learning outcomes are achieved.

Using game analytics alone as the guideposts for designing games for learning may produce game play experiences that are detrimental to learning. For example, if we had used our in-game assessments to judge Alicia’s ability, we may have had the game adapt educationally so her subsequent levels were easier. However, Alicia understood the math and those problems might have been too easy, causing her to disengage with the game and thereby losing out on all opportunities for learning. On the other hand, someone like Marcus who solved through trial-and-error, but possibly did not know the math, could subsequently receive more difficult and unsolvable math problems, causing him to disengage due to the level of difficulty being too high. In revealing how players are cognitively engaging with games for learning, we may better avoid potential negative affective engagement not just by revealing the “why” behind their successes and failures of game play, but by understanding their predispositions toward the learning content as well.

Emotional experiences in games, such as boredom, frustration, enjoyment, and so on, can be the catalyst for further engagement and ultimately learning. Without understanding the nuances of engagement, we cannot differentiate between instruction, learning, and game play, and what elements and combination of each achieve desired levels of engagement and contribute to learning.

We urge researchers to note that engagement in video games is a complicated construct that has a large impact on learning, and by expanding definitions of engagement in games for learning and using a variety of qualitative methodologies for deeper understanding of this engagement, we can maximize our ability to create great learning opportunities for all students.

Footnotes

Appendix

List of Codes.

Code name	Definition	Example
Math identity	Feelings about math separate from the game.	Interviewer: OK. [0:08:26] Do you like math?
		Student 42: Yes, I do.
		Interviewer: Why do you like math?
		Student 42: Well, I like it because, um, well, math all adds up. Well, it’s kind of a combination of science too, because in science you need math and reading and writing, so ...
Thoughts on the game	Thoughts and feelings about the game. This may have to do with likes and dislikes of the game; it may be about mechanics or math, including justification for those thoughts and feelings.	[0:03:03] What did you like about this game?
		Student 43: I like about this game how they have it in space—
		Interviewer: Mm-hmm.
		Student 43: —and how it’s dark and the stars and stuff.
		Interviewer: OK.
		Student 43: The cool lasers, the blockers, the, um, the, uh, turners.
		Interviewer: OK.
		Student 43: It’s pretty fun.
Emotional response	This relates to how the game made them feel, the types of feeling associated with game play, and justifications for types of feelings associated with game play, that is, I was having fun, it was frustrating.	Student 43: It’s pretty fun.
		Interviewer: All right. And why is that stuff fun?
		Student 43: It’s fun because it gives you a feeling of joy.
Game type	Types of games played (entertainment/educational, including board games).	Interviewer: OK. Let’s see. [0:10:11] Do you play games outside of school?
		Student 515: Only a little bit.
		Interviewer: Only a little bit. And what kind of games do you play?
		Student 515: I like video games.
		Interviewer: Yeah. Which ones?Student 515: On my DSi, I have one that is like it, um, is called, um ... like Day Challenges. And, um, it helps you like with reading, math, and science. And I have a game on Moshi Monsters, and that one, you kind of get to play around. And on there, you can play people on there with math, reading, counting, um, and like learning about the states and stuff.
		Interviewer: No? Do you play board games?Student 515: Yes.Interviewer: What board games are your favorites?Student 515: Um, like, um, Candy Land, Monopoly.
Game recommendations	Any changes to the current games that participants recommend, including recommendations for the assessment items.	OK. [0:04:32] If you could change this game, what would you change and why?Student 514: I wouldn’t change anything. The only thing I would change is probably taking off the magnifier.Interviewer: [laughs] Yeah.Student 514: I would—and have like different levels hard, easy, medium, and that’s basically all.
Time	Amount of reported frequency of game play per week.	Interviewer: Cool. All right. How often do you play games? Like, how often do you play Moshi Monster or board games or ... ?
Time	Amount of reported frequency of game play per week.	Student 517: Um, it’s usually like for, um, sometimes 2 or 3 times a week.
Learned	This is reported learning via game play and includes mention of both math and/or the game mechanic for those who said they learned.	Interviewer: OK. Do you think you learned anything from the game? From playing?
		Student 41: Yes.
		Interviewer: What did you learn?
		Student 41: I learned how to make one-eighth using just three one-half’s
Opinions on learning	This includes school learning. Any mention of learning related to affect, and any feelings about learning in this way.	Interviewer: OK. Do you think you learned anything from the game? From playing?
		Student 41: Yes.
		Interviewer: What did you learn?
		Student 41: I learned how to make one-eighth using just three one-half’s.
Affordances and constraints	What about the game helped them learn, and what about the game made it harder for them to learn. This includes assessment items.	Interviewer: That’s a—that’s a great thing to learn. [0:05:57] What do you think was most helpful for your learning in the game?
		Student 42: I think the thing that was most helpful was that the benders all had a different, um, uh, you know, fraction, like the half or like the one—
		Interviewer: OK.
		Student 42: —that split in two was a half, so, yeah.

Acknowledgements

We would like to thank the creators of REFRACTION and the Center for Game Science at the University of Washington for making this work possible. We would also like to thank Taylor Martin from the University of Utah and Carmen Petrick Smith from the University of Vermont for helping to make this work possible by finding the location for data collection as well as for helping with data collection.

Author Contributions

All authors contributed equally to this article. RP and TH completed all of the analysis and writing. RP wrote the first full draft with TH contributing sections of that first full draft. NV and JB guided the analysis and both reviewed and edited drafts of the article throughout the writing process.

Authors’ Note

The opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of the Bill and Melinda Gates Foundation or the Office of Naval Research.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Bill and Melinda Gates Foundation Grant OPP1031488 and the Office of Naval Research Grant N00014-12-C-0158.

Notes

Author Biographies

Rachel S. Phillips received her PhD in learning sciences from the University of Washington and is a research associate at the University of Washington. Her current research interests include games for learning with a particular focus on underserved youth.

Contact: rachelsp@uw.edu.

Theresa Horstman received her PhD in learning sciences from the University of Washington and is a research associate at the University of Washington Bothell. Her current research interests include analyzing how the design of digital games supports learning. Her research experience includes math and science games, integrating new technologies in curriculum design, and online induction support for teachers.

Contact: thorst@uw.edu.

Nancy Vye, PhD, is a principal research scientist in the College of Education at the University of Washington. Her research focuses on challenge-based learning, formative assessment, and educational technology that enhances teaching and learning in classroom settings.

Contact: nancyvye@uw.edu.

John Bransford, PhD, was the Shauna C. Larson professor of the learning sciences at the University of Washington in Seattle, Washington, and is now a Professor Emeritus. He has authored and co-authored a number of books and articles on learning, assessment, and technology for the National Academy of Science, National Academy of Education, The James S. MacArthur Foundation, and other groups, and is currently working with several different research groups, including the Games for Learning Team at the UW.

Contact: bransj@uw.edu.

References

Baker

D’Mello

S. K. D.

Mercedes

Rodrigo

Graesser

A. C.

(2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68, 223-241.

Boekarts

Pintrich

P. R.

Zeidner

(Eds.). (2000). Handbook of self-regulation: Theory, research and applications. San Diego, CA: Academic Press.

Connell

J. P.

Halpem-Felsher

B. L.

Clifford

Crichlow

Usinger

(1995). Hanging in there: Behavioral, psychological, and contextual factors affecting whether African American adolescents stay in school. Journal of Adolescent Research, 10, 41-63.

Consalvo

(2009). There is no magic circle. Games and Culture, 4, 408-417.

Csikszentmihalyi

(1990). Flow. New York, NY: Harper Perennial.

Eccles

J. S.

Midgley

Wigfield

Buchanan

C. M.

Reuman

Flanagan

Iver

D. M.

(1993). Development during adolescence: The impact of stage-environment fit on young adolescents’ experience in school and in families. American Psychologist, 48, 90-101.

Ermi

Mäyrä

(2005). Fundamental components of the gameplay experience: Analysing immersion. In de Castell

Jenson

(Eds.), Digital Games Research Association International Conference (pp. 15-27). Vancouver, British Columbia, Canada: Digital Games Research Association (DiGRA).

FINAL FANTASY VII. (1997). Square Enix.

Fredericks

Blumenfeld

Paris

(2004). School engagement: Potential of the concept, and state of the evidence. Review of Educational Research, 74, 59-109.

10.

Garris

Ahlers

Driskell

J. E.

(2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33, 441-467.

11.

Gee

J. P.

(2008). Learning and games. In Salen

(Ed.), The ecology of games: Connecting youth, games, and learning (pp. 21-40). Cambridge, MA: The MIT Press.

12.

Grimshaw

Charlton

J. P.

Jagger

(2011). First-person shooters: Immersion and attention. Culture, 5, 29-44.

13.

Karweit

(1989). Time and learning: A review. In Slavin

R. E.

(Ed.), School and classroom organization (pp. 69-95). Hillsdale, NJ: Lawrence Erlbaum.

14.

Lave

Wenger

(1991). Situated learning and legitimate peripheral participation. Cambridge, UK: Cambridge University Press.

15.

Peterson

Swing

Stark

Wass

(1984). Students’ cognitions and time on task during mathematics instruction. American Educational Research Journal, 21, 487-515.

16.

REFRACTION. (2010). Center for Games Science, Computer Science and Engineering, University of Washington. Retrieved from http://games.cs.washington.edu/refraction/

17.

Rodriguez

(2006). The playful and the serious: An approximation to Huizinga’s Homo Ludens. Game Studies, 6(1). Retrieved from http://gamestudies.org/0601/articles/rodriges

18.

Rogoff

(2003). The cultural nature of human development. Oxford, UK: Oxford University Press.

19.

Salen

(2008). Toward an ecology of gaming. In Salen

(Ed.), The ecology of games: Connecting youth, games, and learning (pp. 1-20). Cambridge, MA: The MIT Press.

20.

Salen

Zimmerman

(2003). Rules of play: Game design fundamentals. Cambridge, MA: The MIT Press.

21.

Schapps

(2003, April). The role of supportive school environments in promoting academic success. Sacramento: California Department of Education Press.

22.

Vygotsky

L. S.

(1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

23.

Wentzel

K. R.

(1998). Social relationships and motivation in middle school. Journal of Educational Psychology, 90, 202-209.

24.

Wilson

Elliott

(2003, June). The interface of school climate and school connectedness: An exploratory review and study. Paper presented at the Wingspread Conference on School Connectedness: Strengthening Health and Educational Outcomes for Teens, Racine, WI.

25.

Zimmerman

B. J.

(1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 21, 3-17.