Comparison Between Two Types of Educational Computer Games

Introduction

Digital games, whether played via computer, console, or portable device, have become a major part of society in industrialized nations, and to no surprise, have found their way into educational contexts. A study sponsored by the Joan Ganz Cooney Center in 2013 revealed that “Nearly three-quarters (74%) of K-8 (elementary and middle) teachers report using digital games for instruction,” with 80% of game-using teachers primarily using dedicated educational video games (Takeuchi & Vaala, 2014, p. 5). Educational video games are available for mobile devices (Bendak, 2018), as browser games, and as installed software for personal computers in areas such as math (Novak & Tassell, 2015), finance (Carenys, Moya, & Perramonc, 2017), history (Chapman, 2016), and science (Barab et al., 2009).

A great deal of research has established the effectiveness of digital game-based learning in generating gains in both content knowledge and motivation (Annetta, Minogue, Holmes, & Cheng, 2009; Ebner & Holzinger, 2007; Ke, 2008; Squire, DeVane, & Durga, 2008). Surprisingly though, little research has been done comparing the effectiveness and engagement of different types of digital games. Malone (1984) started from a basic learning activity and added elements one at a time by recording participants’ reactions to create his theory of challenge, curiosity, and fantasy. Ke’s (2008) study found that participants preferred games with an endogenous fantasy that integrated educational content into the gameplay compared to games with exogenous fantasy, in which the educational content interrupted the narrative of the game. Further, existing theories such as Papert’s (1998) theory of constructionism, Kiili’s (2005) flow-based theory of educational game design, and Gee’s (2005) components of good educational digital game creation primarily examined in-depth, long-playing adventure games and simulations that ignored or even condemned shorter, casual games that could be easily integrated into teaching environments.

Although there has been significant experimental testing of the overall effectiveness of educational digital games as compared to traditional teaching methods, experimental comparison of games as compared to each other, specifically regarding educational game theory, remains largely untested. Further, little has been found that takes into account players’ minority status and socio-economic status when comparing content knowledge gains and video game engagement. Theories of effective digital game design should be tested so that educators and game designers can be guided by proven principles in selecting and designing educational games. As a result, the purpose of this study is to explore differences between two types of computer games with regards to both engagement and change in content knowledge, while considering gender and minority and socio-economic status.

Background

Educational video games are an undeniable part of the cultural landscape, and as such, have become a part of the educational landscape. The question is not whether video and computer games are used for educational purposes, but how to maximize the benefits of their use.

Students of different ages in various countries have demonstrated an increase in student engagement when using games. Students have also had an equal or greater gain in content knowledge when using computer and video game-based learning methods as opposed to using traditional methods of instruction. Hwang, Chiu, and Chen (2015) found significant gains in learning, motivation, and satisfaction among students learning through gameplay among 87 Taiwanese sixth grade students. Ku, Chen, Wu, Lao, and Chan (2014) found significant differences in confidence and mathematical ability gains between game-playing fourth-grade students and those undertaking traditional instruction, and found these differences to be most pronounced among students with the lowest starting ability levels. Another study from Chang et al. (2016) documented an increase in learning engagement of fifth grade students in mathematics classrooms across five schools in eastern United States (U.S.), compared to classrooms that implemented paper and pencil learning. At the college level, Almeida (2012) discovered that undergraduate students at an eastern U.S. university acquired factual knowledge concerning information processing using computer-based games. Because games were personalized to the students’ learning level, Almeida showed that games helped improve student motivation as they advanced toward higher levels of learning. In Scotland, Barr (2017) used commercial video games such as Minecraft and Laura Croft to determine whether playing these games can help improve graduate-level student skills as critical thinking and reflection, social consciousness, effective communication, adaptability, and being resourceful. Barr found that communication ability, adaptability, and resourcefulness improved in a relatively short period of time among the undergraduates examined in the study. Thus, video and computer games have had effect upon learning and engagement of students in various levels and content areas.

Video games have also had an impact on foreign language acquisition. deHaan’s (2005) conducted a case study of a single intermediate-level student of the Japanese language who sought to enhance his understanding of the language through playing a Japanese baseball game, Jiikyoo Pawafuru Puro Yakkyu 6, on the Nintendo 64 console. Repetition in the game enhanced both the ability to play the game and understanding of the spoken and written Japanese in the game. The user’s sense of control and the fantasy role of controlling a baseball team align with Gee’s (2005) traits customize and identity, respectively. Shirazi, Ahmadi, and Mehrdad (2016) performed an experimental study that examined whether video games helped 40 Iranian EFL students learn speech acts of apology and request. The difference between the pre- and post-tests for the experimental and control groups was significant in that learning of languages can be affected by video games. Another study conducted in Thailand examined whether 16 undergraduate students playing an online MMORPG game could improve the quality and quantity of second language interaction and willingness to communicate (Reinders & Wattana, 2011). The researchers discovered that students communicated more frequently and was less opposed to speaking in English, as compared to face-to-face courses. Thus, digital games have value in assisting with language learning.

Extensive studies have had investigated whether educational video games are effective and engaging compared to non-game-based teaching methods (Chen, 2017) and investigated students’ preferences with regard to genres of digital games (Vahlo, Kaakinen, Holm, & Koponen, 2017). But, researchers have not compared different video games to one another as much with regard to engagement and content knowledge gain. While Malone (1984) tested his theory with regard to engagement, no studies showed whether his principles of challenge, curiosity, and fantasy led to a greater content knowledge gain.

Based upon the problem identified concerning games and their influence upon engagement and learning of content, the following research questions are presented in this study. In addition, five hypotheses have been created in relation to the research questions as explained in the findings.

Theoretical Foundation

Malone (1984) published the results of an experimental research into the traits possessed by educational digital games that created intrinsic motivation in players. Originally presented in 1982, a year when the most popular games included Tron, Pac Man, and Dig Dug, Malone’s study involved short, simple games with a Behaviorist drill and practice pedagogical approach. Malone provided a framework for creating engaging video games that did not require the long, involved simulations preferred by Papert (1998) and Kiili (2005).

Malone’s (1984) theory of game design included three aspects. First, the challenge aspect involved consistently placing the players in situations that forced them to try new strategies and struggle to progress, but not so difficult to stymie the players. Curiosity kept the players interested by giving them information at a complexity level that they were most ready for. Fantasy allowed the player to identify with the situation and immerse themselves into it. Malone found that if the players enjoyed a game’s fantasy, they were more engaged with a game featuring an endogenous fantasy. Likewise, if they disliked the fantasy being presented, an endogenous fantasy led to greater disengagement, once again pointing to the importance of careful design.

Two casual digital games being examined in this study are Shoot ’Em and Scramble ’Em produced by BuenSoft for teaching French language. These are individual player games. Shoot ’Em involves players to shoot at the letters of the French phrase in order, guided by the English-language translation shown at the bottom of the screen, by avoiding the attacks unleashed by those letters (similar with the invading aliens in Space Invaders). Further, the player’s time is limited in that if the letters reach the bottom of the screen, the player loses the game. Scramble ’Em is a word game in which players unscramble words and phrases.

The first game meets all three of Malone’s (1984) conditions of intrinsic motivation of challenge, curiosity, and fantasy, and Scramble ’Em meets none of the three elements. Shoot ’Em integrates the challenge aspect as players have to shoot letters correctly in time, and embeds the fantasy element of defending against an alien invasion. Although this is a simple fantasy, the game is supported with a variety of sound effects, backgrounds, and animations, all serve to generate sensory curiosity. Scramble ’Em does not involve the player in fantasy, nor has the same level of challenge. Though the potential to fail does exist for an individual phrase, there is no way to lose the game. Failing to unscramble some or all of the phrases only results in a low final score. The game has no animation with limited sound effects. Without variety the effects cannot surprise the player, and thus, contribute little to the player’s sensory curiosity.

Both games are simple and short enough to be integrated into a classroom environment using in-class individual learning centers, laptops, or computer laboratories. Malone’s theory has been chosen because it specifically addresses short, simple digital games that adopt to a Behaviorist approach rather than complex simulations or overall learning environments. As such, the theory is most applicable to games that can be easily integrated into existing classroom structures.

Methodology

A quasi-experimental design was used as it incorporated experimental sessions along with obtaining supportive descriptive data. Participants in the study include students attending a community college located in southeastern United States. Community college students were appropriate participants because of the prevalence of college-level language classes. After the research was approved by the university’s Institutional Review Board or research ethics committee, we began the recruitment process. Participants were recruited by going to various class meetings with the instructors’ permission and through word of mouth. At the class meetings, we asked interested students to sign up for appointment times to participate and offered a small cash incentive to all who participated in this quasi-experimental study of game design and content knowledge gain and engagement. Participants that have been eliminated because they scored too high on the pre-test for French language (90%) knowledge still received cash incentives. During the course of the experimental study sessions, no participants who began the study discontinued participation. In total, 195 participants were recruited and 59 were eliminated, leaving 136 valid participants.

Participants included the following demographics and experiences. Fifty-one were female and eighty-five were male. Ninety-nine of the 136 were of lower socio-economic status, as indicated by receiving Pell Grants, while thirty-seven were not. Ninety-one were racial minorities, while forty-five were white. The majority of the participants in the minority category were African Americans, with eighty-six of the ninety-one participants classified as minority. Participants varied in their experience with digital games, both in terms of frequency of game play and types of games played. Twenty-eight of the one hundred thirty-six reported having very little or no experience with digital games; fifty-three participants reported having moderate experience with digital games; fifty-two participants reported gaming frequently or very frequently; and three did not answer. Fourteen participants reported that they most often played casual and classic games with reflex elements such as Pac-Man, Space Invaders, and Tetris; twenty-one reported that they most frequently played casual and classic games without reflex elements such as Angry Birds, Farmville, and Bejeweled; thirty-six reported that they most frequently played in-depth action games like Call of Duty, Arkham City, and Left 4 Dead; twelve reported that they most frequently played in-depth strategy and role-playing games like Skyrim, Final Fantasy, and Starcraft; thirty-one reported they most frequently played several kinds of games; and twenty-two reported that they did not play video games or very rarely did so.

Experimental sessions were held in an on-campus computer lab and participants were randomly assigned a number that served as their identifier on the pre-test, post-test, and Game Engagement Questionnaire (GEQ). Participants assigned an even number (N=67) would play Shoot ’Em, and participants assigned an odd number (N=69) played Scramble ’Em. All participants interacted with the same vocabulary list, Buensoft French’s ‘Basic Phrases 2,’ which included not only words, but short phrases. The French language had been chosen because of a greater chance to find participants inexperienced with the language, and thus, easier to measure prior knowledge of the language.

Each experimental session lasted for approximately thirty minutes to one hour and began with a brief introduction to the vocabulary list, followed by a pre-test, fifteen minutes of gaming, a post-test, and the completion of Brockmyer et al.’s (2009) digital game engagement instrument. The questionnaire also asked participants basic demographic questions such as grade level, gender, minority status, and socio-economic status.

The pre- and post-test consisted of a multiple-choice format, with content-level questions on the first page and spelling questions of on the second page. Each test asked two questions about each French phrase: (1) one asked for the meaning of the phrase as written in French, and (2) one requested participants to select the correct French phrase when presented with the comparable English phrase. The second question about each phrase tested spelling knowledge by presenting four possible answers: (1) one correct in both spelling and meaning, (2) one correct in meaning with an incorrect spelling, (3) one correctly spelled with an incorrect phrase, and (4) one phrase that was both incorrect in meaning and spelling. The pre-test and post-test contained the same questions, but with a different order of questions and rearranged possible responses. Both tests were developed based upon a vocabulary list provided by Buensoft specifically for its French language program. The pre-test and post-tests’ validity were established through consultation with an experienced French instructor who determined that the instruments were correct in content and age-appropriate to the participants in the study.

The Brockmyer et al.’s (2009) Game Engagement Questionnaire consisted of nineteen statements such as “I lose track of time” to which participants responded “no,” “sort of,” or “yes.” The questions were not divided into separate sections or categories. The GEQ was selected because it specifically measured participant engagement in digital games, and because the instrument had been extensively tested for validity and reliability. The GEQ’s reliability testing yielded a Cronbach’s alpha of .85, a Rasch estimate of person reliability of .83, and a Rasch estimate of item reliability of .96. Rasch validation showed that “the modeled data accounted for 69.2% of the variability in the responses” (Brockmyer et al., 2009, pp. 628-629).

Findings

The pre-test and post-test consisted of twenty multiple-choice question each, with each correct question representing one point, yielding a twenty-point scale with a theoretical minimum of zero. Both tests’ results were normally distributed. The pre-test had skewness of −.317 and kurtosis of −.803, and the post-test had skewness of −.274 and kurtosis of −.230, both well within acceptable bounds.

The word and phrase meaning and spelling components of the pre-test and post-test each consisted of ten multiple-choice questions for each area (meaning and spelling), each worth one point, yielding a ten-point scale with a minimum score of zero. All results were normally distributed. The word and phrase meanings pre-test had a skewness of −.357 and kurtosis of −.509, and its post-test had a skewness of −.319 and a kurtosis of −.754, all within acceptable bounds. The spelling-only pre-test had a skewness of −1.074 and kurtosis of 1.031, and its post-test had a skewness of −.886 and a kurtosis of 1.133, all within acceptable bounds.

Brockmyer et al.’s (2009) Game Engagement Questionnaire (GEQ) consists of nineteen questions with three possible values each: yes, sort of, and no. To create a composite score, yes was scored as 3, sort of as 2, and no as 1, and the scores of all nineteen questions were totaled, creating a scale of nineteen to fifty-seven. GEQ results were normally distributed, with a skewness of .060 and kurtosis of −.810.

We also explored three research questions with five hypotheses. The first two research questions were quantitative in nature, dealing with differences in knowledge gain, as measured by post-test scores, and engagement, as measured by the Game Engagement Questionnaire. The last research question examined student perceptions arising from the descriptive open-ended responses.

Conclusions and Discussion

Results of the quantitative analysis and perceptions arising from the participants’ comments lead to many possible conclusions. The lack of significant differences in content knowledge gain and engagement between participants who play Shoot ’Em and Scramble ’Em does not conform to Malone’s (1984) theory and must be explored. Likewise, the overall lack of difference between pre-test scores and post-test scores among all participants bears examination. The differences that do exist, between the content knowledge gain of minority and non-minority participants, raise questions relevant to those who wish to use digital games for educational purposes.

One area of note is the difference between demographics concerning ethnicity. The results reveal significant differences between pre-test and post-test scores, but only for participants with and without minority status. Participants with minority status score significantly lower on general post-tests and on the word and phrase meanings post-tests than participants without minority status. However, this is the only area in which the study reveals a difference. This difference may be a case of the well-documented ethnic-group differences in multiple-choice test scores, in which participants of some ethnic groups, including African Americans, tend to score lower in general on multiple-choice tests (Edwards & Arthur, 2007).

In terms of learning content, there is no significant difference between pre-test and post-test scores overall, whether taken together or divided between word and phrase meanings and spelling. The experience of playing these games do not seem to affect content gain in any way, regardless of the game played or the participants’ socio-economic status, minority status, or sex. In short, neither Shoot ’Em nor Scramble ’Em seem particularly effective in teaching the material in this context. This result is different from those studies that saw improvement in second language acquisition using digital games (deHaan, 2005; Reinders & Wattana, 2011; Shirazi et al., 2016).

Furthermore, the question as to why participants playing the two games have not differed significantly in engagement must be explained. Scramble ’Em may have engaged the participants as a puzzle, even though the game does not meet Malone’s (1984) conditions for an engaging digital game. Word scrambles are popular and widely-published puzzles in non-digital form, which implies that they are engaging in their own way. Further, Shoot ’Em’s difficulty, as reported by multiple participants, may have negatively impacted participants’ engagement scores. Although both games are simple, that simplicity may have produced its own learning difficulty, as over 58% of the participants frequently played in-depth games of various sorts, and over 16% rarely or never played digital games. Additionally, participants may have required some time to adjust in using the keyboard controls for an arcade-type game such as Shoot ’Em.

Frustration with the difficulty level of the games, especially Shoot ’Em, may have distracted the participants from the content knowledge and negatively impact self-reported engagement scores. Malone’s (1984) element of challenge may have been too present in Shoot ’Em. Additionally, some participants commented that they have been distracted by the in-game elements of both Shoot ’Em and Scramble ’Em such as sound effects in both games, shots fired by the aliens in Shoot ’Em, and trying to beat the timer in Scramble ’Em, and thus, were unable to focus on the content.

Future Focus

The results of this study have failed to support Malone (1984). Perhaps this gives more weight to Papert (1998), Kiili (2005), and Gee (2005), who hold that open-ended, exploratory, user-guided games are superior for learning, and that casual, linear, drill-and-practice games are of little value as educational tools. Future studies could abandon Buensoft specifically, and other drill-and-practice casual games in general. Instead, comparing two or more in-depth, exploratory, self-guided games that follow Papert’s (1998) and Gee’s (2005) theories might produce better results. Later studies could also examine games specifically designed for mobile devices, comparing them against each other, against PC-based drill-and-practice games, and against more in-depth games. Similarly, a future study could compare drill-and-practice games to in-depth, exploratory, self-guided games, although such games would need to be shown to increase content knowledge beforehand. Despite the difficulties involved in making a rigorous and fair comparison of drill-and-practice and in-depth, exploratory, self-guided games, in terms of results and in terms of adherence to Malone’s (1984) theory, the resulting insights would be valuable.

The study has also experienced several limitations that could have impacted the results. For practical and logistical reasons, each participant’s role in the study has to be completed in one sitting. This proves to be a serious limitation because this allows immediate short-term memory to play a disproportionate role in the outcome. Additionally, the study’s instruction and reinforcement session differ substantially from the actual classroom learning environment, which more often features multiple days of instruction prior to a major summative assessment. In this case, long-term learning is of primary importance to educators. Participant selection was another issue. Because the participants were volunteers and not randomly selected, this could have affected the results. Finally, although the incentive made data collection possible, this could have led participants being there for the incentive only with little interest in the subject or study itself. Overall, even though the results of this study is inconclusive as to the effectiveness of drill-and-practice games, they may still have value as a form of reinforcement or study tool in education for specific content areas. However, this conclusion is a topic left open for future research and examination by using different types of games in various settings; taking special note to the level of difficulty, over emphasizing specific game elements, and the timing of instruction.