Using Kahoot to Inspire

Abstract

Today, there is growing interest in digital game-based learning due to the increase in the variety of educational and commercial games available. The literature indicates that digital educational games are effective at keeping learners engaged. This study examines the use of a digital educational game called Kahoot and supports the hypothesis that it can improve engagement and learning when immersed in traditional, lecture-based classrooms. The results of the mixed-methods study, using 96 undergraduates, suggest that learners’ performance and engagement are enhanced when using Kahoot versus traditional teaching methods. In addition, practitioner and future research implications are also discussed.

Keywords

game-based learning performance outcomes cognition digital games Kahoot

Introduction

Using digital games in education or digital game-based learning (DGBL) is becoming increasingly feasible due to the rise in variety and availability of digital educational and commercial games as well as their consumption by contemporary users (Gros, 2007; Hainey, Connolly, Stansfield, & Boyle, 2011; Pivec, 2007). For example, in 2015, 155 million Americans played video games, with at least two players in each game-playing household (Entertainment Software Association, 2015). Gaming is inherently motivating, which may allow learners to gain skills and knowledge, by leveraging entertainment and weaving it within learning environments (Becker, 2008; Bopp, 2006; Gee, 2003, 2005; Kiili, 2007, 2010; Miller, 2008; Paraskeva, Mysirlaki, & Papagianni, 2010; Rieber, 1996; U.S. Department of Education, 2010; Van Eck & Hung, 2010). Research studies as well as conceptual literature indicate that video or digital games in particular have great engagement and education-related benefits (Bogost, 2007; Griffiths, 2002; Paraskeva et al., 2010; Zarraonandia, Diaz, Aedo, & Ruiz, 2014).

Currently, digital educational games (DEGs) are emerging as interesting options for instruction, although they are not being used to their optimum potential in higher education (Godwin-James, 2014; Law & Sun, 2012). To institutionalize and adopt DGBL more extensively in higher education, it is important to research how using DEGs in higher education curriculum may effect learners’ experiences and learning (Epper, Derryberry, & Jackson, 2012). Using a report published by National Foundation for Educational Research, Perrotta, Featherstone, Aston, and Houghton (2013) indicate that although there is evidence that game-based learning can improve engagement and motivation, there is a gap pertaining to knowledge about its effects on performance outcomes.

This mixed-methods study seeks to contribute to literature by examining the value of using a DEG as compared to traditional teaching methods. The selected DEG, Kahoot, is gaining popularity in the educational arena, and according to Johan Brand (2016), CEO at Kahoot, 1.6 million Kahoots were played by 14 million players in over 100 countries in 1 month of January 2015 alone. However, at present, there is no literature on the use of Kahoot. Thus, it would be useful to investigate this DEG in the context of improving student performance outcomes in higher education.

In addition, to understand why such improvements, if any, took place, the perceptions of learners with respect to the interest and challenge factors pertaining to the DGBL learning must also be examined. Based on this, the study answered the following questions:

Do learners have higher performance outcomes when exposed to activities involving Kahoot versus not using it?

What factors within Kahoot do learners find interesting and challenging and why?

The following section provides overviews of key concepts pertinent to the study, including the theoretical frame. The “Methodology” section discusses elements of the study design as well as the rationale for selecting Kahoot. Thereafter, findings (quantitative and qualitative) are discussed. Finally, limitations and practitioners as well as research implications and conclusions are provided.

Background

DGBL refers to the inclusion of digital games as part of a curriculum and using them for different educational purposes like instruction and assessments (Nadolny & Halabi, 2015; Prensky, 2001; Van Eck, 2015). Within DGBL environments, learners usually participate in gameplay, explore various aspects of the game to satisfy the learning contexts created by the course designer or instructor, and engage in problem-solving activities (Kiili, 2005; Nadolny & Halabi, 2015; Pivec, 2007; Pivec, Oziabenko, & Schinnerl-Beikircher, 2014; Tsai & Fan, 2013).

McClarty, Orr, Frey, Dolan, Vassileva, and McVay (2012) define DEGs as digital games that represent a technology-based system within which players engage in game-generated competition that are guided by rules and have quantifiable outcomes. Such games seek to promote learning in an entertaining way by fusing educational content with gameplay and stories, while giving learners the opportunity to strategize with higher order thinking (Kickmeier-Rust & Albert, 2012; Kickmeier-Rust, Göbel, & Albert, 2008; Paraskeva et al., 2010; Pivec et al., 2014). Currently, DEGs are being used in a variety of instructional settings, primarily in military and medical training (Alexander, Brunyé, Sidman, & Weil, 2005; Binsubaih, Maddock, & Romano, 2009; Graafland, Schraagen, & Schijven, 2012; Nicolaidou et al., 2015; Ulicsak, 2015). Although many researchers support the use of DEGs (Bogost, 2007; Godwin-James, 2014; Griffiths, 2002; Paraskeva et al., 2010; Law & Sun, 2012; Zarraonandia et al., 2014), others focus on the issues within DEGs, such as low entertainment value that makes such games less interesting for learners (Bellotti, Ott, Arnab, Berta, deFreitas, Kiili, & Oliver, 2011), and the dilution of effectiveness when education and entertainment are combined (Khine, 2011). Thus, when selecting any DEG, a judicious mix of entertainment and educational value must be considered, which is why Kahoot was the ideal choice for this study, as it is gaining in popularity, is easy to use, is free, and has components of DEGs as described earlier.

In addition, after examining three DEG options (Kahoot, Quizziz, and Socrative) that could be a fit for in-class activities, Kahoot was found to be the best choice, based on the criteria shown in Table 1 and rationale provided by the participant instructors as adapted from Roediger (2015).

Table 1.

Comparison of Kahoot, Quizizz, and Socrative Based on Selection Criteria for Study Use.

Criteria and rationale	Kahoot	Quizizz	Socrative
Must have an account to administer and save games, preferably free, so that instructors can share the same account and any edits and usage may be monitored.	Yes	No	Yes
The Q and A should not appear on one screen. This was required to avoid the BlackBerry Prayer effect (Michaluk et al., 2010) where users are so glued to their device screens that they do not look up and /or participate in collaborative activities.	Yes	No	No
Ability to include images	Yes	Yes	Yes
Ability to be used as a competitive game	Yes	Yes	Yes
Ability to be self-paced or timed	Yes	Yes	Yes
Teacher controlled as the purpose of use was to facilitate learning, and not assessment. Having teachers control the pace was essential.	Yes	No	Yes
Ability to search for games made by others that may act as extra resources if needed	Yes	Yes	No
Help and support is available	Yes	Yes	Yes

Kahoot is inherently entertaining and educational web-based software for learning purposes that allows instructors and course designers to integrate instructional content through a quiz-like, gameplay construct. According to Singer (2016), it can be described as a television game show fused within a video game, where teachers act as game hosts and engage students. It provides players the opportunity for strategizing, problem solving, social interaction, and collaborative learning within face-to-face class settings. The gameplay structure in Kahoot allows for quantifiable outcomes by mimicking quiz-based gameplay. To use Kahoot, teachers create learning games known as Kahoots, using a series of multiple choice questions that can include videos, images, and diagrams. Players answer on their own devices, while games are displayed on a shared screen and earn points for selecting the correct answer and potentially more points for how quickly they select the correct answer. The Kahoot games are point-based games with winners and losers. In addition, Kahoot can be used as an application on mobile devices. It provides an entertaining way to conduct in-class assessment, allowing teachers and students to evaluate student learning.

Kahoot is gaining popularity within K-12 classrooms to enhance instruction in elementary math and junior high social studies and English (Knodel, 2016). According to Singer (2016), the company manufacturing Kahoot reported that 20 million out of 55 million elementary and secondary students in the United States used Kahoot in 2016. Kahoot offers increased collaboration over conventional DEGs that rely on individual play modes (Stewart et al., 2013), and wherein “subject mastery is generally emphasized over complex problem-solving” (L. Johnson, Smith, Levine, & Haywood, 2010, p.18). As a quiz-based game, Kahoot can mesh with any discipline or subject matter as opposed to those DEGs that are designed for specific subjects (L. Johnson, Levine, Smith, & Stone, 2010; L. Johnson et al., 2010; Stewart et al., 2013). Despite its perceived value for education based on such reports, there are no studies on Kahoot’s value and viability in a higher education setting.

Methodology

Mixed-Methods Approach

A mixed-method approach (Creswell, 2014; Creswell & Plano-Clark, 2011), using true experimental methodology for quantitative and phenomenology methodology for qualitative data, was used to examine learner performances and experiences within a DGBL environment. Mixed-methods approaches have been known to provide deeper, more comprehensive data and result in superior research (Halkier, 2011; R. B. Johnson & Onwuegbuzie, 2004; Schr⊘der, 2012). For this study, following Creswell’s (2014) convergent-parallel mixed-method approach, collection of quantitative and qualitative data was done at approximately the same time and then analyzed by converging information from the data. The three key identifiers of the true experimental design are randomization, control, and manipulation (Berry & Yost, 2013; R. B. Johnson & Christensen, 2014), which the study incorporated. The class sections were selected randomly, and the duration of usage of Kahoot was controlled within the experimental groups. Participants were randomly placed in experimental (using Kahoot) and control (not using Kahoot). In addition, the study used a phenomenological approach for qualitative data that were drawn from participant interactions and reactions to a DGBL environment, as suggested by several researchers of this methodology (Creswell, 2014; Holroyd, 2001; Husserl, 1931; Sadala & Adorno, 2001).

Setting and Participants

Participants were selected from one Midwestern Community College Campus with an approximate population of 5,100. The sample population were 96 students from Introduction to Business courses, enrolled in Associates Degree programs, recruited from five sections taught by three instructors who agreed to participate. Based on information from the course instructors and Introduction Discussion forum posts, participants were approximately 18 to 60 years of age and were mixed in terms of ethnicity. Based on survey data, 90% of the population had little to no experience with Kahoot. All of the instructors had no prior experience with DGBL or gaming in general, although they were knowledgeable in computer usage and basic educational technology.

Data Gathering

Course identifying numbers for the participating instructors were randomized to select sections for the experiment. Quantitative data were collected using the scores participants received for their final exams. Within the experimental sections, students were taught course materials using lectures for the first half of the semester (7 weeks); while for the remaining 7 weeks, Kahoot-based activities were used to complement the traditional teaching methods in class. This was done based on the suggestion of the instructors, given that the materials covered in the latter half of the semester were more difficult than the ones taught in the first half. It was the instructors’ desire to use Kahoot for the more difficult topics as they hoped it would help improve the learners’ retention and learning process. The control group courses did not use Kahoot at all.

Instructors created Kahoots or interactive and problem-solving based question and answer activities, using material from their lectures and the common text books assigned to the course. In the classrooms, these Kahoots were used as different group activities. These were usually done after a brief lecture, as opposed to full class lectures in the first 7 weeks. For example, some days, students played a Class-Jeopardy game. On other days, they played a group contest game based on which group had more correct answers or which group was more prompt in responding to the questions. At least once during the experiment, students played a round table game wherein groups designed their own questions based on the ideas generated by the Kahoot and quizzed other groups. The focus of these gameplays was on interactive and collaborative learning. For each activity, groups played against other groups for points.

The final exam scores were compared to examine the effects of the Kahoot intervention. The exam questions for both groups were same. The final exam was held at the end of the semester and had 100 multiple choice questions, worth one point each. This exam covered all materials for the entire semester for the course. Students had 2 hours to complete this exam. Students were not allowed to use any notes or consult the text book for this exam.

Qualitative data were collected using one survey with three questions and entries made in one reflection journal. The survey questions were as follows: “Have you experienced Kahoot type of learning technology before? If yes, what technology was it? If not, say NA. Write down three words or sentences that identify your feelings about using Kahoot? Would you like to engage in Kahoot-based activities again? Why or why not? Explain clearly, but briefly.”

Analysis

For the exams, independent sample t tests were conducted, and Cohen’s d effect size was calculated. Survey data were quantitized (R. B. Johnson & Christensen, 2014, p. 504) using term and phrase frequencies showing a comparative analysis of the perceptions of Kahoot as expressed in specific words/phrases. First, a list of all words/phrases used by the participants was drawn. As several of these were overlapping and more than one participant used them, the number of responses per word/phrase was counted, then codified based on the frequency, and represented using a clustered bar graph.

Results

Research Question 1: Performance Outcomes

The null hypothesis (H0) was that there will be no significant difference between the exam scores of learners using Kahoot and students from control groups. The alternative hypothesis (HA) was that learners using Kahoot will have significantly higher exam scores than control groups. Due to directional approach for the HA, one-tail, independent sample t tests were used. Alpha level for both tests was set at 5% (α = .05). Confidence level was set at 95%. Based on the results of the t test and Cohen’s d, the null hypothesis (H0) was rejected. Table 2 highlights the results of the t test using numbers and graph. See the Appendix for deidentified exam scores.

Table 2.

Descriptive Statistics and t Test Results.

	p < .00001	Significant at p < .05
t = 6.90809	N	Minimum	Maximum	Mean	Standard deviation
Experimental	48	43.00	100.00	79.5625	13.10174
Control	48	25.00	92.00	56.8333	18.65400
Valid N (listwise)	48

The effect size using Cohen’s d was large: Cohen’s d = (56.83 − 79.56)/ 16.116121 = 1.410389.

Research Question 2: Interesting and Challenging Factors

Data revealed that none of the participants had played Kahoot in their classrooms before, although some had minimal experience with other in-class gaming. This supported the researchers’ findings from literature that Kahoot has been underutilized in higher education, despite its perceived value for educational purposes as found in K-12 environments. All participants found the use of Kahoot to be extremely beneficial, and 100% of them agreed that they would like to engage in Kahoot-based activities again. When answering questions about using words/phrases to identify their feelings about Kahoot and why they would like to use it again, participants came up with 35 words/phrases in total, many of which were repeated by different participants, bringing the total number of words/phrases submitted to 103. Based on the codification, these were divided into seven categories that identified the general perceptions about the interesting and challenging aspects of Kahoot. Figure 1 shows a cluster graph of the results. As shown in Figure 1, participants found Kahoot to be highly entertaining, motivating, exciting, and helpful in providing assistance to their learning.

Figure 1.

Perceptions of Kahoot Value.

Most participants did not find using Kahoot difficult or challenging. Finally, only some participants found Kahoot to be competitive, even though it was used to create competition among groups in class. In this study, participants appear to have been more involved due to the entertainment factors and the perceived value they found in using DEG-based activities for their learning purposes. This was further collaborated by participants’ journals, excerpts from which are given later using pseudonyms.

One of the values seen was the ability of such activities to help review course materials and help them prepare in a better way for the exams. For instance, Emma stated, “I like the Kahoot technology, I think it is a better way to review the material, and get ready for the tests.” Kyle noted “It was fun and seemed to have helped refresh the reading.” Some participants felt that this helped them retain information better, thus assisting their working memory. For instance, Sherry explained how “Kahoot was very helpful it just refreshed what I learned and put it in my long term memory.” In a similar vein, Darren mentioned, “About the Kahoot I like that it reinforced the stuff we were learning and made it more likely to be remembered.” In addition, participants’ ability to read and take notes was also facilitated. Charlene confided how, “Before using Kahoot I mostly read the power points and took short notes, but after we started playing I would take better notes and read over them to make sure I understood the definitions.” Finally, using Kahoot facilitated more robust participation, even by shy students. For instance, Sam mentioned that, “I liked playing Kahoot in class. It involved people in class that are scared to speak out to tell the right answer when you ask questions,” while Emma stated, “It is also a good way to get people to participate in the class.”

Integration of Qualitative and Quantitative Findings

A mixed-method approach complemented quantitative data with qualitative findings to gain a comprehensive understanding of the phenomenon of using Kahoot-based intervention in classrooms. Although the quantitative data indicated that there were definite improvements in performance outcomes for the experimental group versus the control group, the qualitative analysis provided several pointers as to why this could be. The key factor in this was the element of motivation that the Kahoot usage provided that could be deemed as the catalyst for the improved performances. Despite some issues and concerns regarding technology-based instruction (Ariffin, 2012; Ertmer, 2005), technology integration is seen as a criterion for pedagogical expertise and superior instructional design (Borthwick et al., 2008; Jaipal-Jamani & Figg, 2014; Piersen, 2001).

Table 3 shows a synthesis of the key qualitative findings and how they relate to the quantitative findings of higher performance, with supporting literature. When examining the qualitative data from this study, the following factors were offered by participants as crucial to their engagement with the games that perhaps steered their motivation to higher performance.

Table 3.

Quantitative and Qualitative Findings Synthesis.

Qualitative	Quantitative
Interesting Kahoot contents, including audio-visual components, that helped students repeatedly read content and the associated engagement factors led to repeated interaction with such contents.	Repeated readings can lead to higher performance (Chafouleas, Martens, Dobson, Weinstein, & Gardner, 2004; Granic, Lobel, Engels, Anderson, & Norman, 2014; Schlickum, Hedman, Enochsson, Kjellin, & Felländer-Tsai, 2009; Therrien, 2004).
Socializing options within video games can enhance motivation.	Socializing-based motivation has a higher compelling impact on learners’ efforts and performances (Boggiano & Pittman, 2010; Lee, Hwa Hsu, & Chang, 2013; Peterson, 2013). Having a competitive essence in an activity may enhance engagement and allow learners to develop more effective problem-solving skills and participate more robustly in uninteresting or mundane educational activities, which may lead to higher learning (Annetta et al., 2009; Burguillo, 2010).

Qualitative

Quantitative

Interesting Kahoot contents, including audio-visual components, that helped students repeatedly read content and the associated engagement factors led to repeated interaction with such contents.

Repeated readings can lead to higher performance (Chafouleas, Martens, Dobson, Weinstein, & Gardner, 2004; Granic, Lobel, Engels, Anderson, & Norman, 2014; Schlickum, Hedman, Enochsson, Kjellin, & Felländer-Tsai, 2009; Therrien, 2004).

Socializing options within video games can enhance motivation.

Socializing-based motivation has a higher compelling impact on learners’ efforts and performances (Boggiano & Pittman, 2010; Lee, Hwa Hsu, & Chang, 2013; Peterson, 2013). Having a competitive essence in an activity may enhance engagement and allow learners to develop more effective problem-solving skills and participate more robustly in uninteresting or mundane educational activities, which may lead to higher learning (Annetta et al., 2009; Burguillo, 2010).

Discussion

As is evident from the results, the use of Kahoot positively affected learners’ performances. The effects were seen with respect to higher performance in exams and greater cognitive retention, a fact that several participants discussed in their reflection on the game. For the final exam, contents from 14 weeks were used. Thus, for the final exam, students had substantial cognitive load when compared to the rest of the semester, where assignments covered smaller chunks of subject matter. However, for 7 weeks preceding the final exam, content was taught using Kahoot. In addition, as it is possible that after a gap of several weeks, students would have forgotten some of what they learned during the first half of the semester (which did not use Kahoot), in order to perform well, students may have had to put in more effort for the final exam. It is possible that this effort was boosted by the use of Kahoot.

Cowan (2010) discussed storage-specific measuring in the context of the nature of working memory limitations and posited that forgetting some of the acquired learning may lead to shallower understandings, and learners would thus need to go back and reread materials. He also stated that the limits of working memory ability differ, based on how much a person can store and how effectively working memory is used. For instance,

An important example is in the use of attention to fill working memory with the items one should be remembering (say, the concepts being explained in a class) as opposed to filling it with distractions (say, what one is planning to do after class). (Cowan, 2010, p. 52)

With respect to the participants for this study, the time gap between learning activities would be enough to create several distractions, thus reducing their working memory capacity. Yet, the experimental students’ performance in the overall final exam was better than the control groups.

Qualitative results strongly indicated that participants were engaged by the use of Kahoot in the classroom. This supports evidence in the literature that students find digital game-related activities more entertaining and motivating, and that these factors in turn may assist learning (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005; Bogost, 2007; Griffiths, 2002; Hainey et al., 2011; Paraskeva et al., 2010; Zarraonandia et al., 2014). One unexpected result was that despite the game-show aspect of Kahoot and its focus on competition between teams, participants did not identify the game as a competitive experience. This is different than what is talked about in some studies and conceptual literature that emphasizes the competitive value of games as being key to participants’ interest and motivation (Annetta, Minogue, Holmes, & Cheng, 2009; Burguillo, 2010; Jayakanthan, 2002; Ladd & Fiske, 2003; Papastergiou, 2009; Vorderer, Hartmann, & Klimmt, 2003). Despite the lack of competition, students clearly were engaged, not only by the gaming experience but also by their perception of the value of the game for learning.

Implications, Limitations, and Conclusion

In terms of research implications, there are strong indications that the DGBL-based intervention using Kahoot was successful in delivering higher performance scores and inciting high interest and engagement in participants as opposed to traditional teaching methods. A limitation of the study is that it only examined the use of one DEG. However, when viewed through the lens of this study’s findings, it can be extrapolated that DGBL strategies using DEGs may positively affect performances. Investigation of other DEGs in this context may be beneficial. Based on the success of the study and suggested value of Kahoot as a DEG for enhancing performance outcomes, it is recommended that the study be continued to replicate the experiments. Additional limitations are that this study focused on learners and did not include the teachers’ perceptions, which may be crucial to get a deeper view of the value of this Kahoot-based intervention. Furthermore, it will be useful to examine the effects of Kahoot DEG usage in different time slots. For instance, it will be useful to see learner reactions when Kahoot is used only for the first half of the semester in some courses and vice versa in others. Finally, it will also be valuable to examine the use of Kahoot in a variety of subject areas and as assessment tools, as this study’s focus was on using it as a knowledge regulator.

In terms of practitioners’ implications, instructors should look to DGBL interventions as tools that can potentially increase engagement to a point where performance is positively impacted. To conclude, the ability of Kahoot to help learners with motivation and performance is highly encouraging and significant. Kahoot provides an example of a free game with limited barriers to use in traditional classroom environments. Kahoot and other similar games are, therefore, worthy of further examination, as they hold strong potential for accelerating the incorporation of DGBL in the classroom.

Appendix: Final Exam Scores Deidentified

Experimental	Control
99	44
91	86
81	62
82	86
81	62
94	76
80	70
73	64
95	92
85	92
88	42
100	66
74	70
90	92
92	66
79	38
92	74
94	80
86	62
78	46
86	56
82	67
81	52
81	28
75	37
79	29
84	27
80	25
84	35
61	40
64	34
72	43
82	40
91	46
88	33
80	76
89	44
67	78
53	76
88	60
84	50
54	54
63	54
50	70
80	47
55	53
43	49
89	55

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Author Biography

Papia Bawa has a PhD in Learning Design and Technology from Purdue University’s College of Education. She is also a Professor of Instructional Design in the college’s Masters in LDT Program. As a faculty and professional Instructional designer, she has 17 years of experience in academia. Her research focuses on learner-centered environments, cultural inclusivity issues in online and offline courses, and technology centered curriculum, including game-based learning. She has extensive experience in writing and publishing scholarly articles, poetry and creative non-fiction. She serves as the Managing Editor of The Atrium, Ivy Tech Community College’s refereed journal, as well as a reviewer for several scholarly journals such as Journal of Economics and Finance Education (JEFE), Curriculum and Instruction, Simulation and Gaming, Journal for Research Initiatives, American Educational Research Association (AERA), Educational Researcher, and Association for Educational Communications and Technology (AECT).

References

Alexander

Brunyé

T., Sidman

Weil

S. A.

(2005). From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in PC-based simulations and games. Retrieved from http://www.aptima.com/publications/2005_Alexander_Brunye_Sidman_Weil.pdf

Annetta

L. A.

Minogue

Holmes

S. Y.

Cheng

(2009). Investigating the impact of video games on high school students’ engagement and learning about genetics. Computers & Education, 53(1), 74–85. doi:10.1016/j.compedu.2008.12.020

Ariffin, M. M. (2012). Towards digital game-based learning (DGBL) in higher education (HE): The educators' perception. Developing Country Studies, 2(11), 228–236. Retrieved from http://www.iiste.org/Journals/index.php/DCS/article/view/3579/3628

Barab

Thomas

Dodge

Carteaux

Tuzun

(2005). Making learning fun: Quest Atlantis, a game without guns. Educational Technology Research and Development, 53(1), 86–107. doi:10.1007/bf02504859

Becker

(2008). Video game pedagogy: Good games = good pedagogy. In Miller

C. T.

(Ed.), Games: Purpose and potential in education (pp. 73–125). New York, NY: Springer Science+Business.

Belloti

Ott

Arnab

Berta

deFreitas

Kiili

Gloria

A. D.

(2011). Designing serious games for education: From pedagogical principles to game mechanisms. In Gouscos

Meimaris

(Eds.), Proceedings of the 5th European Conference on game based learning. Reading, England: Academic Publishing Limited. Retrieved from https://halshs.archives-ouvertes.fr/hal-00985800/document

Berry

C. A.

Yost

(2013). Study guide for nursing research: Methods and critical appraisal for evidence-based practice (8th ed.). St. Louis, MO: Mosby.

Binsubaih

Maddock

Romano

(2009). Serious games for the police: Opportunities and challenges. Special reports and studies series at the Research and Studies Center (Dubai Police Academy). Retrieved from https://staffwww.dcs.shef.ac.uk/people/S.Maddock/publications/BinSubaihEtal2009_special_report.pdf

Boggiano

A. K.

Pittman

T. S.

(2010). Achievement and motivation: A social-developmental perspective. Cambridge, England: Cambridge University Press.

10.

Bogost

(2007). Persuasive games the expressive power of videogames. Cambridge, MA: MIT Press.

11.

Bopp

(2006). Didactic analysis of digital games and game-based learning. In Pivec

(Ed.), Affective and emotional aspects of human-computer interaction—Game-Based and innovative learning approaches. The future of learning ( Vol.1). Amsterdam, the Netherlands: IOS Press.

12.

Brand

(2016). Kahoot!—Inclusive design. Retrieved from http://www.inclusivedesign.no/ict/kahoot-article172-261.html

13.

Burguillo

J. C.

(2010). Using game theory and competition-based learning to stimulate student motivation and performance. Computers & Education, 55(2), 566–575. doi:10.1016/j.compedu.2010.02.018

14.

Chafouleas

S. M.

Martens

B. K.

Dobson

R. L.

Weinstein

K. S.

Gardner

K. B.

(2004). Fluent reading as the improvement of stimulus control: Additive effects of performance-based interventions to repeated reading on students' reading and error rates. Journal of Behavioral Education, 13(2), 67–81. doi:10.1023/b:jobe.0000023656.45233.6f

15.

Cowan

(2010). The magical mystery four: How is working memory capacity limited, and why? Current Directions in Psychological Science, 19(1), 51–57. doi:10.1177/0963721409359277

16.

Creswell

J. W.

(2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Thousand Oaks, CA: SAGE.

17.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed method research (2nd ed.). Thousand Oaks, CA: SAGE.

18.

Entertainment Software Association. (2015). Essential facts about the computer and video games industry. Retrieved from http://www.theesa.com/wp-content/uploads/2015/04/ESA-Essential-Facts-2015.pdfcontent/uploads/2015/04/ESA-Essential-Facts-2015.pdf

19.

Epper

R. M.

Derryberry

&Jackson

(2012). Game based learning: Developing an institutional strategy. Educause Research Bulletin. Retrieved from https://net.educause.edu/ir/library/pdf/ERB1208.pdf

20.

Gee

J. P.

(2003). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan.

21.

Gee

J. P.

(2005). Good video games and good learning. Phi Kappa Phi Forum, 85(2), 33–37. Retrieved from http://search.proquest.com/docview/235184729?accountid=13360

22.

Godwin

J. R.

(2014). Games in language learning: Opportunities and challenges. Language Learning and Technology, 18(2), 9–19. Retrieved from http://llt.msu.edu/issues/june2014/emerging.pdf

23.

Graafland

Schraagen

Schijven

(2012). Systematic review of serious games for medical education and surgical skills training. British Journal of Surgery, 99(10), 1322–1330. doi:10.1002/bjs.8819

24.

Granic

Lobel

Engels

R., Anderson.

Norman

(2014). The benefits of playing video games. American Psychologist, 69(1), 66–78.

25.

Griffiths

(2002). The educational benefits of videogames. Education and Health, 20(3), 47–51. Retrieved from http://sheu.org.uk/sites/sheu.org.uk/files/imagepicker/1/eh203mg.pdf

26.

Gros

(2007). Digital games in education: The design of games-based learning environments. Journal of Research on Technology in Education, 40(1), 23–38. doi:10.1080/15391523.2007.10782494

27.

Hainey

Connolly

Stansfield

Boyle

(2011). The differences in motivations of online game players and offline game players: A combined analysis of three studies at higher education level. Computers & Education, 57(4), 2197–2211. doi:10.1016/j.compedu.2011.06.001

28.

Halkier

(2011). Methodological practicalities in analytical generalization. Qualitative Inquiry, 17(9), 787–797. doi:10.1177/1077800411423194

29.

Holroyd

(2001). Phenomenological research method, design and procedure: A phenomenological investigation of the phenomenon of being-in-community as experienced by two individuals who have participated in a community building workshop. Indo-Pacific Journal of Phenomenology, 1(1), 1–10. doi:10.1080/20797222.2001.11433859

30.

Husserl

(1931). Ideas: General introduction to pure phenomenology and to a phenomenological philosophy (trans. F. Kersten). The Hague, the Netherlands: Martinus Nijhoff Publishers. Retrieved from http://www.dhspriory.org/kenny/PhilTexts/Husserl/Ideas1.pdf

31.

Jayakanthan

(2002). Application of computer games in the field of education. The Electronic Library, 20(2), 98–102. doi:10.1108/02640470210697471

32.

Johnson

Levine

Smith

Stone

(2010). The 2010 horizon report. Austin, TX: The New Media Consortium. Retrieved from https://www.nmc.org/pdf/2010-Horizon-Report.pdf

33.

Johnson

Smith

Levine

Haywood

(2010). The 2010 Horizon Report: K-12 Edition. Austin, TX: The New Media Consortium. Retrieved from https://www.uakron.edu/dotAsset/8a0b91a7-453e-42c4-a684-6952412e95ce.pdf

34.

Johnson

R. B.

Christensen

L. B.

(2014). Educational research: Quantitative, qualitative, and mixed approaches (5th ed.). Los Angeles, CA: SAGE.

35.

Johnson

R. B.

Onwuegbuzie

A. J.

(2004). Mixed Methods Research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. doi:10.3102/0013189x033007014

36.

Khine

M. S.

(2011). Let the game begin. In Khine

M. S.

(Ed.), Learning to play: Exploring the future of education with video games (pp. 1–8). New York, NY: Peter Lang Publishing.

37.

Kickmeier-Rust

M. D.

Albert

(2012). An alien’s guide to multi-adaptive educational computer games. Santa Rosa, CA: Informing Science Press.

38.

Kickmeier-Rust

M. D.

Göbel

Albert

(2008). 80Days: Melding adaptive educational technology and adaptive and interactive storytelling in digital educational games. In Klamma

(Eds.), Proceedings of the First International Workshop on Story-Telling and Educational Games (STEG’08)—The power of narration and imagination in technology enhanced learning. Maastricht, the Netherlands. Retrieved from http://www.eightydays.eu/Paper/STEG08a.pdf

39.

Kiili

(2005). Digital game-based learning: Towards an experiential gaming model. The Internet and Higher Education, 8(1), 13–24. doi:10.1016/j.iheduc.2004.12.001

40.

Kiili

(2007). Foundation for problem-based gaming. British Journal of Education Technology, 38(3), 394–404. doi:10.1111/j.1467-8535.2007.00704.x

41.

Kiili

(2010). Call for learning-game design patterns. In Edvardsen

Kulle

(Eds.), Educational games: Design, learning and applications. New York, NY: Nova Publishers.

42.

Knodel

(2016, February 26). Teacher uses Kahoot! To enhance instruction. Dayton Daily News (Dayton, OH). Retrieved from http://www.daytondailynews.com/news/news/local/teacher-uses-kahoot-to-enhance-instruction/nqYjM/

43.

Ladd

H. F.

Fiske

E. B.

(2003). Does competition improve teaching and learning? Evidence from New Zealand. Educational Evaluation and Policy Analysis, 25(1), 97–112. doi:10.3102/01623737025001095

44.

Law, E. L.-C., & Sun, X. (2012). Evaluating user experience of adaptive digital educational games with Activity Theory. International Journal of Human-Computer Studies, 70(7), 478–497. doi:10.1016/j.ijhcs.2012.01.007

45.

Lee

Hwa Hsu

Chang

(2013). Factors influencing sociability in educational MMORPGs—A fuzzy AHP approach. Internet Research, 23(3), 298–315. doi:10.1108/10662241311331745

46.

McClarty

K. L.

Orr

Frey

P. M.

Dolan

R. P.

Vassileva

McVay

(2012). A literature review of gaming in education. Retrieved from Pearson website: http://researchnetwork.pearson.com/wpcontent/uploads/Lit_Review_of_Gaming_in_Education.pdf

47.

Michaluk, K., Trautschold, M., Mazo, G., Markham, J., Andres, C., Anglin, S., … Meador, P. (2010). CrackBerry: True Tales of BlackBerry Use and Abuse. Berkeley, CA: Springer New York.

48.

Miller

C. T.

(2008). Games: Purpose and potential in education. New York, NY: Springer Science+Business.

49.

Nadolny

Halabi

(2015). Student participation and achievement in a large lecture course with game-based learning. Simulation & Gaming, 47(1), 51–72. doi:10.1177/1046878115620388

50.

Nicolaidou

Antoniades

Constantinou

Marangos

Kyriacou

Bamidis

Pattichis

(2015). A virtual emergency telemedicine serious game in medical training: A quantitative, professional feedback-informed evaluation study. Journal of Medical Internet Research, 17(6), E150–E150. doi:10.2196/jmir.3667

51.

Papastergiou

(2009). Digital game-based learning in high school computer science education: Impact on educational effectiveness and student motivation. Computers & Education, 52(1), 1–12. doi:10.1016/j.compedu.2008.06.004

52.

Paraskeva

Mysirlaki

Papagianni

(2010). Multiplayer online games as educational tools: Facing new challenges in learning. Computers & Education, 54(2), 498–505. doi:10.1016/j.compedu.2009.09.001

53.

Perrotta

Featherstone

Aston

Houghton

(2013). Game-based learning: Latest evidence and future directions (NFER Research Programme: Innovation in Education). Slough, England: National Foundation for Educational Research. Retrieved from https://www.nfer.ac.uk/publications/GAME01/GAME01.pdf

54.

Peterson

(2013). Recent studies on computer gaming in CALL: An analysis of findings. Computer Games and Language Learning, 73–100. doi:10.1057/9781137005175_6

55.

Pivec

(2007). Editorial: Play and learn: Potentials of game-based learning. British Journal of Educational Technology, 38(3), 387–393. doi:10.1111/j.1467-8535.2007.00722.x

56.

Pivec

Oziabenko

Schinnerl-Beikircher

(2014). Aspects of game-based learning. CiteSeerx, 1, 10. doi:10.1.1.97.8613

57.

Prensky

(2001). Digital game-based learning. New York, NY: McGraw-Hill.

58.

Rieber

(1996). Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Educational Technology Research and Development, 44(2), 43–58.

59.

Roediger, A. (2015, July 1). A Lever and a Place to Stand: Kahoot or Quizizz or Socrative: Which Should I Use?. Retrieved from http://aleverandaplacetostand.blogspot.com/2015/07/kahoot-or-quizizz-or-socrative-which.html

60.

Sadala

M. L.

Adorno

R. C.

(2001). Phenomenology as a method to investigate the experience lived: A perspective from Husserl and Merleau Ponty’s thought. Journal of Advanced Nursing, 37(3), 282–293. doi:10.1046/j.1365-2648 (2002). 02071.x

61.

Schlickum

M. K.

Hedman

Enochsson

Kjellin

Felländer-Tsai

(2009). Systematic video game training in surgical novices improves performance in virtual reality endoscopic surgical simulators: A prospective randomized study. World Journal of Surgery, 33(11), 2360–2367. doi:10.1007/s00268-009-0151-y

62.

Schrøder

K. C.

(2012). Methodological pluralism as a vehicle of qualitative generalization. Journal of Audience and Reception Studies, 9(2), 798–825. Retrieved from http://www.participations.org/Volume%209/Issue%202/42%20Schroder.pdf

63.

Singer

(2016, April 6). Kahoot app brings urgency of a quiz show to the classroom. The New York Times. Retrieved from http://www.nytimes.com/2016/04/17/technology/kahoot-app-brings-urgency-of-a-quiz-show-to-the-classroom.html

64.

Stewart

Bleumers

All

Mariën

Schurmans

Van Looy

Centeno

(2013). The potential of digital games for empowerment and social inclusion of groups at risk of social and economic exclusion: Evidence and opportunity for policy (AA INFSO/SMART 2011/0054—JRC 32397-2011). Retrieved from Publications Office website: http://is.jrc.ec.europa.eu/pages/EAP/eInclusion/documents/DGEID1MainReportFINALforweb.pdf

65.

Therrien

W. J.

(2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education, 25(4), 252–261. doi:10.1177/07419325040250040801

66.

Tsai, C.-W., & Fan, Y.-T. (2013). Research trends in game-based learning research in online learning environments: A review of studies published in SSCI-indexed journals from 2003 to 2012. British Journal of Educational Technology, 44(5), E115–E119. doi:10.1111/bjet.12031

67.

Ulicsak

(2015). Games in education: Serious games. Retrieved from http://media.futurelab.org.uk/resources/documents/lit_reviews/Serious-Games_Review.pdf

68.

U.S. Department of Education. (2010, November). Transforming American education: Learning powered by technology. Retrieved from http://www.ed.gov/sites/default/files/netp2010-execsumm.pdf

69.

Van Eck

(2015). Digital game-based learning: Still restless, after all these years. Educause, 50, 13–28. Retrieved from https://er.educause.edu/articles/2015/10/digital-game-based-learning-still-restless-after-all-these-years

70.

Van Eck, R., & Hung, W. (2010). A taxonomy and framework for designing educational games to promote problem solving. Paper presentation at the Videogame Cultures & the Future of Interactive Entertainment Annual Conference of the Inter-Disciplinary. net Group, 7–9 July 2010, Mansfield College, Oxford, England. Retrieved from https://docplayer.net/19298628-A-taxonomy-and-framework-for-designing-educational-games-to-promote-problem-solving-richard-van-eck-university-of-north-dakota.html

71.

Vorderer, P., Hartmann, T., & Klimmt, C. (2003, May). Explaining the enjoyment of playing video games: the role of competition. In Proceedings of the second international conference on Entertainment computing (pp. 1–9). Carnegie Mellon University.

72.

Zarraonandia

Diaz

Aedo

Ruiz

M. R.

(2014). Designing educational games through a conceptual model based on rules and scenarios. Multimedia Tools and Applications, 74(13), 4535–4559. doi:10.1007/s11042-013-1821-1

Experimental	Control
99	44
91	86
81	62
82	86
81	62
94	76
80	70
73	64
95	92
85	92
88	42
100	66
74	70
90	92
92	66
79	38
92	74
94	80
86	62
78	46
86	56
82	67
81	52
81	28
75	37
79	29
84	27
80	25
84	35
61	40
64	34
72	43
82	40
91	46
88	33
80	76
89	44
67	78
53	76
88	60
84	50
54	54
63	54
50	70
80	47
55	53
43	49
89	55

Experimental	Control
99	44
91	86
81	62
82	86
81	62
94	76
80	70
73	64
95	92
85	92
88	42
100	66
74	70
90	92
92	66
79	38
92	74
94	80
86	62
78	46
86	56
82	67
81	52
81	28
75	37
79	29
84	27
80	25
84	35
61	40
64	34
72	43
82	40
91	46
88	33
80	76
89	44
67	78
53	76
88	60
84	50
54	54
63	54
50	70
80	47
55	53
43	49
89	55

Experimental	Control
99	44
91	86
81	62
82	86
81	62
94	76
80	70
73	64
95	92
85	92
88	42
100	66
74	70
90	92
92	66
79	38
92	74
94	80
86	62
78	46
86	56
82	67
81	52
81	28
75	37
79	29
84	27
80	25
84	35
61	40
64	34
72	43
82	40
91	46
88	33
80	76
89	44
67	78
53	76
88	60
84	50
54	54
63	54
50	70
80	47
55	53
43	49
89	55