Engagement and Games for Learning

Setting

A mid-sized elementary school in an urban area that serves as a lab school for a large university in the Southern/Central part of the United States. As of the 2011-2012 school year, the school serves 260 students, and the demographic makeup is 72% Hispanic, 16% African American, 11% White, and 1% Asian. Sixty-eight percent of the students qualify for free or reduced lunch.

Participants

Participants in the first part of the study were 28 fourth and 30 fifth graders (ages 8-10) from the elementary school. During interviews, students were asked about their experiences with games. Most of the students reported playing video games and board games. We found a wide range of responses with respect to how much time students spent playing games outside of school, from almost no game play to those who played daily. The majority seemed to fall somewhere in the middle. All of the students had access to computers and the Internet outside of school. We found no bias toward platform (e.g., console, mobile, or PC) for the games they played. The students mentioned a variety of game types: card games, trading games, board games, fighting games, sports, and so on. Student frequently mentioned playing educational games, but we do not know if this has to do with parental control over the content of gaming or if the students opted to play these games themselves.

Debriefing

As part of this study, no formal debriefing was undertaken. Students who only took part in the think-alouds and game play did not have the opportunity to debrief as time with those students was limited and we chose to devote that time to game play. For students who took part in the post–game play interview process, those interviews may be considered a type of debriefing, although this was not necessarily our intention in interviewing the students. Many of our interview questions asked students to reflect upon their thoughts and feelings about the game, which may also be considered as part of a debriefing process.

Game progression

As mentioned, each student spent approximately 20 minutes with each researcher. During this time, the students were introduced to the game, completed math-related assessments, and had the opportunity to engage in non-assessment game play. When students first logged into the game, they completed four tutorial levels that explained the purpose of the game and how to play the game. Both of the case study students had played the game in school the previous year so the tutorial was a review.

Immediately following the tutorial is what we refer to as the embedded assessment level (see Figure 3).

OPEN IN VIEWER

Figure 3.

Marcus playing the embedded assessment level.

This level asks students to solve a more complicated laser splitting problem in which they have to “satisfy” ships by feeding one-sixth of a laser into one ship and one-ninth of a laser into another ship. This level is considered more advanced than other levels because it is not only difficult spatially to navigate the laser toward the spaceships, but it also requires the combination of multiple benders and splitters in a specific sequence (i.e., a three-splitter has to be used before a two-splitter to satisfy ships that require one-sixth and one-ninth lasers) in order to create the appropriate sized laser for each ship. This level was also the only timed level in the game—students had 2 minutes to complete the level before they were automatically forwarded to a set of school-like assessment questions. We created this embedded assessment level for a few reasons. First, we thought it would be diagnostic in terms of learning because we assumed that if a student understood how to make one-sixth and one-ninth, then they were grasping the splitting or partitioning math concepts. In addition, we also assumed that students had mastered some of the spatial problem solving required of being successful in this game. As the case studies suggest, the latter is true, while the former is not.

After the embedded assessment level, students immediately took a standard school-like assessment consisting of four screens. For these assessments, students had to demonstrate knowledge of fraction comparisons, conceptual understanding of fraction splitting, as well as fraction addition not embedded in game play. These assessments were based on previously validated math assessment items and used as a diagnostic tool for helping us determine students’ understanding of fractions. Following these assessments, students were returned to regular game play and used the rest of the session to progress through the game.

Interview data

Interview data were transcribed into individual Word documents and imported to the qualitative research program HyperResearch. Data were then sorted into the following categories based on interview questions: Affect, Game Experience, Hints, Learning, and Perceived Purpose of Game. The sorted data were reviewed and it was decided that for present purposes, the sections related to Affect, Games Experience, and Learning would be presented. These categories were chosen as they were richer and more coherent than the others.

Following this sorting, the data were read by two researchers and subjected to several rounds of open and axial coding. Discussion and refinement of codes occurred between each round of coding. Once a final set of codes and their definitions were created (see the appendix), two researchers completed the coding of the transcripts. Ten percent of the interview transcripts were double coded for reliability and inter-rater reliability was 90%.

Case study data

Based on performance during game play, with a particular focus on game analytics, students who appeared to have performed similarly during game play and the think-alouds were chosen for further examination (discussed in more detail below). Then, two researchers examined the video of the students’ game play experience closely, field-noting from the videos to examine cognitive and affective engagement during the sessions. These field-notes, or running records, were then constructed into case studies, which we used to compare the overall experiences of the student.

Game analytics data

For all game play studies, we collect detailed game analytics such as number and types of player clicks, types of answers, number of moves per level, and time played per level and game. Detailed log data provide quantitative information about how the player was engaged with the game that can be further explained by video data.

Likes and dislikes

Most students reported liking the game; however, they were divided concerning the reasons why and mostly fell into two categories in this respect. One group of students (n = 17) enjoyed non-math aspects of the game, such as the characters and the act of saving characters. The other group (n = 8) reported liking the game because it helped them learn math. Other students (n = 15) mentioned non-math aspects of the game and math at the same time. Surprisingly, few students disliked the embedded assessment items, which were the most “school like” tasks in the game, and said that they would do them at home if they were part of the game. Although measures such as time-on-task or persistence from game analytics could lead us to infer that students like a game, without this interview information, we would not understand why they like the game. Revealing a student’s reasons for liking or disliking an educational game in a school context helps us identify which aspects of the game contribute to engagement and learning.

Emotional responses to the game

Most of the students reported feeling positive emotions in response to the game (n = 15), and at the same time, many reported feeling aggravated or frustrated while playing the game (n = 13). Eleven students reported these emotions together. In addition, many students reported that this frustration made them feel determined to finish and persist, even when facing particularly difficult problems. One said, “I got frustrated, but it was like a good kind of frustrated, like when you—you’re like, oh, come on! When you’re trying to do something.” Understanding the types of feelings associated with game play, and justifications for types of feelings associated with game play helps us understand the analytics we collect from students better. It also helps us make sense of different kinds of patterns we find in those analytics, giving us the ability to make a game appropriately adaptive for students depending on their emotional states.

Learning via games

Overall, students expressed positive opinions about learning via REFRACTION. Thirteen reported that they think a game similar to this would help a friend like math more. Most of them reported that playing the game helped them like math more. Interestingly, although these students all reported liking math, many of them reported that their friends do not like math.

Most thought that if they continued to learn in this way they would like math even more than they already do. However, most of the students, in response to the question, “Would you like to learn more about math in a game like this, more like in school with your teacher, or maybe a little bit of both?” the majority said that they would like to do a little bit of both rather than the expected answer of, “in games like this.” However, a few students did say that they prefer to learn via games. When asked why, one student said, “And it teaches you about ... it teaches you about learning when you don’t even know that you’re learning.” This shows that some students position school learning and game learning differently, further driving the point that when evaluating games for learning we have to take into account the theoretical paradigms from both learning and games research. A game presented in a formal educational setting primes the player with the expectation that learning is involved in the game and not just play that will likely be tied to curriculum.

Discussion of interview data related to affective engagement

The emotions and responses mentioned in this section are typical of a game play experience, which signals that the students are often in a game play mode. Part of the goal of using games for learning is the assumption that game play often seems to lead to increased engagement—however, what engages a person in a game may not be the same as what engages that same person in school. As success ebbs and flows in the game, resultant emotions sometimes include frustration and confusion. These emotions, which are often viewed negatively in school, are essential to game play. Therefore, students’ overall affect and emotion must be understood within the parameters of a typical game-playing framework when examining games for learning.

However, with that in mind, it is difficult to assess through interview questions alone whether or not the emotions reported here are within the expected parameters. In addition, without attention to cognition as well as emotion, we risk losing valuable data that relate to a student’s learning and understanding during game play. However, including interview data with further analysis of video data for evidence of math understanding through game play, as reported via case study, deepens our findings. As shown in the next section, the details exposed in the case studies present the need to expand methodologies further to include qualitative analysis of game play, using a variety of methods, for understanding engagement in games for learning. We used think-alouds and video data, in addition to the post–game play interviews, to provide us with a fuller picture of engagement with the game.

Case study criteria

Our criteria for choosing the students fell into three categories. First, we wanted to make sure that the students were of similar ability in terms of math, and therefore we picked two students who were average in terms of math ability as determined by state-administered standardized assessments. Second, we looked at some aspects of game analytics that demonstrated similar performance. For example, when examining speed of progression through the game, we wanted to make sure that our cases were progressing at relatively the same rate. Third, we chose to focus on two students who were particularly comfortable with a “think-aloud” protocol. We did not want to compare one student who was particularly vocal about his thinking and feeling to another student who was not vocal for fear of drawing incorrect conclusions about the less vocal student. It is important to note that we consider each of these categories to comprise surface features, as they give us information about each student, but not much insight into their personalities or identities as learners. Ultimately, we chose two students from the same fifth-grade class, Alicia and Marcus, to be the focus of our cases.

In the following two sections, we describe our cases. First, we give a qualitative overview of each student that includes a focus on his or her vocalized math understanding and general engagement and affect during the session. Then, we focus in on his or her treatment of the embedded assessment level. Finally, we discuss the overall implications of using qualitative data to strengthen our understanding of more surface and quantitative measures that we collect as part of game analytics. Throughout this section, we use the word diagnostic to mean how well the in-game assessments or the game analytics themselves are measuring student understanding and learning.

Alicia

Overall, it was clear early on that Alicia understood the math presented to her in the game. However, it did not appear that she was as adept with the spatial reasoning required of the game as she was with the math. In other words, her math ability was not matched to her spatial problem-solving ability, limiting her progression in the game. It was not clear initially that Alicia had a solid grasp on the math, but over the course of the game, it became clear that mathematically she had no problems with the ideas presented to her in REFRACTION.

Alicia was able to articulate her understanding of math concepts thoroughly while playing. Alicia’s general affect through this session was concentrated and subdued. She focused and paid attention to game play with minimal reminders to “think-aloud.” Although concentrated and somewhat reserved, she smiled regularly, showing a few moments of delight or surprise during game play. However, she was not overly animated or dramatic in those moments. Overall, Alicia was highly engaged with the game throughout the session.

In the embedded assessment level, the student needs to direct a laser to a one-sixth ship and a one-ninth ship. Upon starting this level, Alicia picked a three-way splitter to start, so the researcher asked her why she chose that splitter first and she quickly responded by saying,

This exchange demonstrates that Alicia was considering both the mathematical and spatial criteria needed for solving the level. In addition, the math talk and speed of response shows that she was considering the relationship between how one-third is related to one-sixth. Although mathematically one-sixth is a half of one-third, Alicia’s response reveals the mathematical thinking she used to reach her understanding.

However, Alicia did not seem to have the same grasp of how to make one-ninth. When discussing how to satisfy the one-ninth ship, she took longer to think aloud when asked and said, “I think one-thirds ... I think it’s ... it’s one ... no, it’s one-sixth of nine, I think.” The difference in time to respond and less articulate math talk around the question of how to solve for one-ninth indicates that she understands the relationship between one-third and one-sixth, but that she does not necessarily understand the relationship between one-third and one-ninth. Interestingly, however, when examining Alicia’s game play over time, it was revealed that she did understand how to make one-ninth, although her initial verbal explanation of the problem indicated otherwise.

Although Alicia demonstrated she understood the math, many other students who did not understand any of this math were able to solve this level because of their spatial reasoning ability via trial-and-error and a bit of luck. However, Alicia was not able to successfully complete the timed embedded assessment level before the 2 minutes mark. Had we only examined game analytics and not captured Alicia’s reasoning through the think-alouds, we would have made the assumption that Alicia did not understand the math in the level because she did not complete the level before the time was up. However, through think-aloud protocol, we know she does understand the math, but struggled with the spatial component of game play. Having captured Alicia’s reasoning, we were able to determine how she was making sense of the game play and the math. Consequently, if game progression and adaptivity were based only on game analytics, Alicia would have subsequently encountered math problems that were far below her math level. In addition, having revealed Alicia’s ability to demonstrate her understanding of math through game play was hindered not by the difficulty of the math problem, but by the difficulty of the spatial reasoning required to complete the puzzle. Although it is true that a different assessment might assist us in not making such errors, without first knowing her understanding, we would not even be able to assess the diagnostic value of our assessments. In the future, this will help to avoid making poor choices related to designing game adaptivity, but it also leads us to question what we deem diagnostic in game play in general.

Marcus

In general, Marcus demonstrated an understanding about the math involved in the game to some degree, but not as much as Alicia. Unlike Alicia, Marcus grasped the spatial reasoning and problem solving needed for the game immediately. This was evident as he moved quickly using a lot of trial-and-error in his game play. Marcus articulated his understanding frequently by declaring what he thought almost constantly as he played. When he seemed less certain in his understanding he usually posed his “think-aloud” in the form of a question to the researcher (the researcher responded to such questions with another question). His overall affect engagement was focused and lively, showing more excitement and interest than we saw from Alicia. He moved around a bit in his chair while playing, he typically had his face very close to the screen, and he reported non-critical actions as well as explained his problem solving and play strategy. Like Alicia, Marcus was highly immersed with the game throughout the session.

Recall that the embedded assessment level is a timed level that requires the student to direct the laser into a one-sixth ship and a one-ninth ship. In the embedded assessment level, Marcus demonstrated his thinking through trial and error.

Through trial and error, Marcus not only successfully completed the level, but also gained valuable knowledge for future levels. In a later level, the first to introduce one-ninth as a non-assessment level, Marcus explained how he solved for one-ninth:

We learn that Marcus’ ability to solve for one-ninth at the higher level is rooted in the previously played embedded assessment level. His learning about how to create one-ninth occurred from trial and error rather than from the deliberate and successful completion of the one-ninth ship in the embedded assessment level. This shows that although he now knows how to split into one-ninth, we also know that this understanding is not necessarily conceptual, but rather, procedural. We would have not gained this information from our analytics or our interviews—we needed the think-alouds, which showed Marcus’ thinking, to help us understand what learning still needed to happen.

Comparing cases

As our in-depth study of Alicia and Marcus shows, combining understanding of student engagement and thinking with game play analytics is vital to creating high-quality games for learning. Without an analysis of the behavioral, cognitive, and affective engagement through think-alouds and video analysis, not only could we misconstrue when learning does or does not happen via game play, but also we miss out on the opportunity to understand why learning happens and at what points during game play. Marcus’ trial-and-error play technique could have been interpreted as evidence of learning of introductory math concepts. Even though he did successfully complete the timed embedded assessment level, he did so by playing around and accidentally discovered that using 2 three-way splitters creates a laser that is one-ninth. We would not have known this without having an understanding of some of his thinking. We would not have known his thinking without first examining the different ways in which he was engaged with the games.

However, this accident provided him with a learning opportunity. When presented again with a one-ninth ship, he remembered his mistake and applied it as a solution in a future level. It could be argued that although he did not receive formal instruction on the mathematical relationship between one-third (as represented by the three-way splitter) and one-ninth, through dragging and dropping laser splitters a relationship was made visible (2 three-way lasers, creates one-ninth). However, his play strategy is different from Alicia’s. To contrast, she also did not complete the embedded assessment and did not have an accidental discovery that led to later game play success even though both students possess approximately the same level of math understanding. This difference in play strategy suggests that the design of the level could be improved to encourage the type of trial-and-error Marcus demonstrated. This could be achieved by eliminating the timer or embedding hints in earlier levels encouraging players to try multiple combinations if they are not sure how to achieve the amount.

It is clear from these case studies that while Marcus and Alicia were selected because some of their more descriptive game analytics appeared to be very similar (i.e., time played, levels completed), as were the other surface features we discussed when selecting our cases, their experiences were quite different in relation to the game. Marcus learned from the game, while Alicia arguably did not. Marcus was highly engaged and expressed more excitement in general about the game. Although Alicia was engaged, she was more subdued overall.