Comparing Serious Games and Educational Simulations: Effects on Enjoyment,Deep Thinking,Interest and Cognitive Learning Gains

Abstract

Background and Aim. Serious games are generally considered to have positive effects on many aspects of learner engagement as well as on cognitive learning gains and subject-related interest. Yet few studies have examined which combination of game elements influence engagement and learning, and how these factors are related. For this reason, an experimental study was conducted to explore these aspects with regard to digital serious games.

Method. Twelve primary school classes with 153 students from 9 to 12 years of age participated in this experimental field study using group comparisons. The students were randomly assigned to interact either with an educational simulation or a digital serious game. The results were analyzed using t-tests and hierarchical linear regressions.

Results and Conclusion. Results show that there are no group differences in tested learning gains nor in self-reported cognitive learning gains or increase in interest. Although there are also no differences regarding enjoyment, self-reported levels of deep thinking are higher when learning with a serious game. While post-test knowledge is only influenced by prior knowledge, self-reported cognitive learning gains and increases in interest are both positively correlated with deep thinking and enjoyment. These results lead to the conclusion that learning with serious games does not always lead to the expected increases in all aspects of engagement and learning outcomes. Thus, research needs to address the interplay of game elements and their impact on engagement and learning in more detail.

Keywords

digital game-based learning engagement game attribute gamification learning serious game simulation

Introduction

Substantial amounts of research on serious games (i.e. games for educational purposes) exist, both with regard to digital and non-digital games (Abt, 1970; Klabbers, 2009; Kriz, 2017). Recent meta-analyses with regard to digital serious games have consistently shown positive effects with regard to learning gains and motivational gains (Clark, Tanner-Smith, & Killingsworth, 2015; Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012; Sitzmann, 2011; Vogel et al., 2006; Wouters, van Nimwegen, van Oostendorp, & van der Spek, 2013). However, serious game research has struggled to isolate the specific combinations of game characteristics that are responsible for these positive findings. Garris, Ahlers, and Driskell (2002) found six game attributes which were shown to be relevant for learning: fantasy, rules/goals, sensory stimuli, challenge, mystery and control. A more recent attempt has been made by Bedwell, Pavlas, Heyne, Lazzara, and Salas (2012), proposing a list of nine game attributes, which has been further reduced by Jabbar and Felicia (2015) to four core factors: motivational elements, interactive elements, fun elements and multimedia elements. However, these attributes are very general and a lack of knowledge still exists concerning the effect of combining these characteristics in different ways. Many authors agree that it is crucial for effective serious games to strive for a high degree of overlap between these kinds of game attributes and instructional objectives (Garris et al., 2002; Granic, Lobel, & Engels, 2014; Ke, 2008a; Wilson et al., 2009). However, success cannot be taken for granted (Michael & Chen, 2005; Ritterfeld, Cody, & Vorderer, 2009). One reason for this might be the way in which research on serious games is carried out. Learning with serious games is often compared to learning with conventional learning material instead of comparing different possible variations or applications of a game (Mayer, 2014; Tobias, Fletcher, Dai, & Wind, 2011). In effect, a large body of research on serious games suffers from the problem that game- and non-game-treatments are hardly comparable as they differ in too many respects. More recently, this problem has been partly resolved by research on gamified learning environments, where video-game characteristics are applied to non-game environments (Deterding, Dixon, Khaled, & Nacke, 2011; Kapp, 2012; Landers, 2014). Although gamified learning environments might suffer from the impression that game characteristics are mere add-ons, this approach creates gamified treatments that are better comparable to the non-gamified environments they are based on. In addition, studies on gamified learning environments have shown similar positive effects on user engagement as well as cognitive and motivational outcomes as studies on serious games (Hamari, Koivisto, & Sarsa, 2014; Kapp, 2012). In consequence, this type of research might be an interesting way forward in isolating game characteristics that are responsible for specific effects on learning.

Another explanation for the lack of consistent findings with regard to essential characteristics of effective serious games is provided by Landers’ (2014) theoretical model. It assumes that the effects of game elements on learning outcomes are mediated by learning-related behavior and attitudes (Landers, 2014). This needs to be taken into account when conducting empirical studies on serious games. Moreover, serious games are often embedded in broader educational activities. Thus, achieving learning outcomes depends not only on the game but also on the reflection and debriefing during which the game activities are reviewed (Garris et al., 2002; Granic et al., 2014; Ke, 2008a; Wilson et al., 2009). To sum up, further research on the effects of isolated game characteristics on learning outcomes is necessary to find out what constitutes effective serious games. Such research should also focus on learning requirements and learning processes.

Comparing Educational Simulations and Serious Games

One way to design comparable treatments is the comparison of serious games and similar educational simulations without game elements (Landers & Armstrong, 2017; Landers & Callan, 2012; Wang & Hannafin, 2005). Simulations can be defined as computer programs that represent dynamic models of natural or artificial systems, where users can change specific parameters and observe the impact on other parameters within the system (De Jong, 1991, 2011). Such a comparison is worthwhile because serious games and educational simulations are similar in many respects. They both have real world knowledge at their core, either represented as information or as dependencies between relevant variables. While educational simulations seek to follow rules that are as realistic and focused as possible, serious games are enriched by additional rules, visual metaphors, imaginary stories and challenging controls (Gredler, 2004; Rieber & Noah, 2008). Furthermore, educational simulations typically are narrowed-down interactive representations, including as few variables as possible. Relationships between variables are clearly defined and educational simulations typically avoid anything that might distract from the interplay between these core factors. By contrast, serious games have many additional features: each serious game relies on an enriched system of rules and game mechanics that constitute the playful interaction between both the serious games and with other players, which is not required in educational simulations. Serious games typically are about winning or achieving a competitive goal (for example, achieving as many points as possible) while educational simulations are about experimenting without competitive elements. Especially digital serious games may include additional aspects such as graphical presentations, game characters, sound-effects and a soundtrack, voice-acting, as well as controls that require a certain degree of dexterity, which would be considered as distracting in educational simulations (Boocock & Schild, 1968; Sitzmann, 2011). Similar aspects have been highlighted by research on gamified learning environments (Landers & Armstrong, 2017; Landers & Callan, 2012; Wang & Hannafin, 2005). Therefore, this study will compare a serious game to a similar educational simulation without game elements and examine the effects of a specific combination of game elements on engagement, interest and cognitive learning gains.

Studying the Impact on Process Variables: Enjoyment and Deep Thinking

In educational research, the term engagement has often been used for all kinds of positive student behavior. More recent studies have agreed on a more systematic understanding, defining engagement as a combination of behavioral, cognitive and emotional aspects of student involvement with a particular learning environment (Bouvier, Lavoué, & Sehaba, 2014; Boyle, Connolly, Hainey, & Boyle, 2012; Filsecker & Kerres, 2014; O’Brien & Toms, 2008). Some authors propose including additional aspects, such as perceived esthetics and usability (Fu, Su, & Yu, 2009; Wiebe, Lamb, Hardy, & Sharek, 2014). However, recent research has concluded that only emotional and cognitive engagement play a role with influencing learning, as behavioral engagement can be seen as prerequisite for these deeper kinds of engagement (Jabbar & Felicia, 2015). Within these two types of engagement, enjoyment and deep thinking have been identified as the most crucial aspects with regard to learning with serious games (Granic et al., 2014). These two aspects have impact on increased interest and cognitive learning outcomes (Dickey, 2005; O’Brien & Toms, 2008; Phillips, Horstman, Vye, & Bransford, 2014). Therefore, this study focuses on these aspects when comparing a serious game with an educational simulation.

The literature review of Jabbar and Felicia (2015) has summarized previous findings on the effects of specific game characteristics on student engagement. Motivational elements, such as points or ranking, seem to contribute to higher enjoyment, while results with regard to cognitive engagement are still inconclusive (Chang & Wei, 2016; Ke & Abras, 2013). Interactive elements, including problems and obstacles, can promote both emotional and cognitive engagement (Ke & Abras, 2013; Yang, Chen, & Chen, 2007). Fun elements, such as narratives, virtual characters and controls, are also positively related to both cognitive and emotional engagement (Ke, 2008a; Ke, Xie, & Xie, 2016). Challenge elements can improve enjoyment (Ke, 2008a) and deep thinking (Rubin-Vaughan, Pepler, Brown, & Craig, 2011). Multimedia elements, such as attractive visuals, affect especially emotional engagement (Rosas et al., 2003; Tan, Goh, Ang, & Huan, 2013). However, as noted earlier, only few studies exist on combinations of these game elements.

Studying the Impact on Outcome Variables: Interest and Achievement

Next to enjoyment and deep thinking, research needs to address the impact of specific game characteristics on different types of learning outcomes. Although these effects are likely to be mediated by engagement, many experimental studies have been able to show direct effects on tested learning gains, self-reported learning gains and increased interest without taking engagement variables into account (Clark et al., 2015; Connolly et al., 2012; Sitzmann, 2011; Vogel et al., 2006; Wouters et al., 2013). In several studies, game characteristics have been varied to test their effect on interest and learning outcomes. The review of Jabbar and Felicia (2015) shows mostly positive evidence for most characteristics, with little counter-evidence. With regard to motivational elements such as points, leaderboards, achievements, badges, rewards and feedback, research has shown overwhelmingly positive effects on cognitive learning outcomes and interest (Domínguez et al., 2013; Gillispie, Martin, & Parker, 2010; Tan et al., 2013). However, with regard to points and achievements, some studies remain sceptical by showing that an over-reliance on these aspects might primarily increase extraneous motivation and superficial engagement (Deci, Koestner, & Ryan, 1999; Mekler, Brühlmann, Tuch, & Opwis, 2017). With respect to interactive elements, it was found that role-playing, in particular, can have a positive impact on cognitive learning gains (Ke, 2008b; Kebritchi, Hirumi, & Bai, 2010; Zheng, Young, Wagner, & Brewer, 2009) and increase interest (Zheng et al., 2009). Fun elements, such as virtual characters/environments, challenges, controls and narratives, have also been found to enhance both cognitive learning gains and increases in interest (Hamari et al., 2014; Rubin-Vaughan et al., 2011; Seaborn & Fels, 2015; Sun Lin & Chiou, 2017). However, these elements could also be a factor in increasing extraneous cognitive load, therefore distracting from the task at hand (Mayer, 2014; Sweller, Ayres, & Kalyuga, 2011). For multimedia elements, some studies have shown that virtual environments and game characters yielded increases in interest (Jabbar & Felicia, 2015; Liao, Chen, Cheng, Chen, & Chan, 2011; Tüzün, Yılmaz-Soylu, Karakuş, İnal, & Kızılkaya, 2009), but others are more reluctant, as this might be a matter of personal preference (Landers & Armstrong, 2017). In these studies, the effects of individual game characteristics were investigated, but – as in the studies on student engagement – the effect of the combination of specific game elements, such as points, challenge and graphics, on interest and cognitive learning gain needs further investigation.

Hypotheses

Based on the findings reported above, it can be assumed that the combination of specific motivational, fun and multimedia elements in a serious game results in higher student enjoyment and deeper thinking as well as higher cognitive learning gains and increases in subject-related interest when compared to learning with a comparable educational simulation, where these characteristics are missing. In this study, this assumption leads to the following detailed hypotheses:

H1a: Students who learn with a serious game achieve higher subject-related knowledge gains than those who learn with a comparable educational simulation.

H1b: Students who learn with a serious game achieve higher perceived knowledge gains than those who learn with a comparable educational simulation.

H1c: Students who learn with a serious game show higher perceived increases in interest in the topic than those who learn with a comparable educational simulation.

H2a: Students who learn with a serious game report higher levels of deep thinking during learning than those who learn with a comparable educational simulation.

H2b: Students who learn with a serious game report higher levels of enjoyment than those who learn with a comparable educational simulation.

As proposed by Landers (2014), it is equally essential to test the impact of game characteristics on process variables, such as cognitive engagement in the form of deep-thinking, and emotional engagement, such as enjoyment. This assumption is in line with findings from general educational research (Anderman & Dawson, 2011; Fredricks, Blumenfeld, & Paris, 2004). This leads to the following detailed hypotheses:

H3a: There is a correlation between deep thinking and subject-related knowledge gains.

H3b There is a correlation between deep thinking and increase in interest.

H3c: There is a correlation between enjoyment and subject-related knowledge gains.

H3d: There is a correlation between enjoyment and increase in interest.

H3e: These correlations from H3a till H3d are stronger for students learning with a serious game than for students learning with a comparable educational simulation.

Method

Research Design and Sample

The study was designed as a randomized field experiment comparing pre- and post-test measures of two groups of students, with one group learning with a serious game and the other with a similar educational simulation without game elements. The experiment was part of a research project supported by Swiss National Science Foundation (project number: 13DPD3_134705) and was conducted in twelve primary school classes in Central Switzerland. They were recruited through an open call in a magazine for schools. The sample consisted of 153 students from 9 to 12 years of age (M = 10.5, SD = 0.67), 75 girls and 78 boys. None of these classes had explicitly covered the topic of the game – information literacy – in previous lessons. According to ethical standards, the students and teachers were comprehensively informed about the aims and methods of the study and could choose not to participate. In order to meet the methodological requirements of randomization, the students in each class were randomly assigned to a group learning with either a serious game (N=78) or an educational simulation (N=75). Power analyses suggest that these sample sizes are appropriate to adequately detect mean-score differences between these two groups with an effect size of d = .46, p = .05 and a power of 1-b= .80 for two-sided comparisons and with r = .22 for correlations calculated with the full sample and r = .32 for within one of the experimental groups.

All the students completed a pre-test on prior knowledge one week before the experimental treatment (t1). The treatment took place during three subsequent lessons of 45 minutes in the regular classroom. Every student was given a notebook with a pre-installed version of either the serious game or of the educational simulation. Before the treatment, all the students received the same short introduction about why it is important to be cautious about information online and what to look for when judging if information on a website is trustworthy, appropriate, complete and neutral. This introduction took about half an hour. After that, the students were asked to spend 45 minutes working with either the educational simulation or the serious game, practicing their ability to identify useful information on websites. During the debriefing, the whole class discussed the play and learning process and reflected on the learning objectives. Afterwards, a post-test to assess knowledge gains and a questionnaire to measure engagement and self-reported learning gains (t2) were administered.

Material

A serious game called AWWWARE (Müller, Petko & Götz, 2011) and a comparable educational simulation called BWWWARE that included no game elements were used for the experiment. Both include the same learning goals and materials and both were developed especially for this study. The only difference between them is the omission of all gamified elements in the simulation.

The goal of both this serious game and the comparable educational simulation is to promote information literacy, especially the ability to critically assess online information when searching the Internet (Virkus, 2003). The core-mechanic relies on an interlinked labyrinth of screenshots from real webpages. The student has the task of locating suitable webpages with information on specific questions, e.g. “What are the main reasons for traffic accidents?” by applying the criteria given in the introduction beforehand. Therefore, he or she must navigate the information structure. The aim is to avoid visiting and flagging inappropriate webpages and to identify and flag trustworthy, appropriate, complete and neutral ones. Students control the learning environment by moving the mouse cursor. In the serious game, this is represented by a small mouse-shaped kite which is held and indirectly controlled by a raven. The background is an attractive rural landscape (Figure 1). In the educational simulation, a normal curser is used and instead of the gamified background with graphic visuals, the students see just a flowered wallpaper background (Figure 2). Students working with the serious game receive immediate feedback in the form of metaphoric weather changes and they receive points. These elements are missing in educational simulation. At the end of both learning environments, a detailed record of the browsing behaviour gives further information and can serve as the basis for a detailed discussion on how to look for information in class. Leaderboards and points are only available in the serious game, but not in the educational simulation. The results of individual students are intended to give teachers the opportunity to compare and discuss different strategies for online information retrieval with the whole class.

Figure 1.

Serious game.

Figure 2.

Educational simulation.

While the interactive elements of gamification (i.e. procedures and resources) are the same for the serious game and the educational simulation, the serious game has the following additional characteristics, according to the gaming elements proposed by Jabbar and Felicia (2015):

Motivational elements (points and achievements): Players should seek and mark a certain number of appropriate webpages for a specific task. By doing so, players are rewarded with points and high scores for selecting relevant webpages while avoiding inappropriate ones. Furthermore, the players receive immediate feedback on whether the marked webpage is correct or not.

Fun elements (challenging controls): Players have to navigate using an indirect control through the raven and the kite. Moving the raven left and right has an impact on both the horizontal and the vertical position of the kite-shaped mouse cursor. This kite flutters in the wind and changes according to the direction of the wind, make controlling the kite more difficult. This slows down browsing behaviour and should be an additional challenge, while giving students more time to consider their web-browsing decisions.

Multimedia elements (Playful graphics, game character and visual metaphors): The learning environment consists of a representation of the real Internet, which is positioned in a graphic frame with a wide comic-landscape which is home to the game character, a black raven. Several aspects in this frame serve as visual metaphors for feedback on players’ strategies: facial expressions of the raven; green, yellow or red flags on the kite; changes in lighting and weather.

Measures

Student ratings on deep thinking, enjoyment, gain in interest and knowledge gain

No previous validated measures for primary school students existed to measure either deep thinking and enjoyment or students’ subjective knowledge gains and interest. Existing measuring instruments were only available for adults and were therefore too long or too complicated for primary school students (Fu et al., 2009; Rich, Lepine, & Crawford, 2010). Therefore, suitable scales were built, inspired from items of the EGameFlow scale (Fu et al., 2009). The items on deep thinking focus on the aspect of application of knowledge during the interaction with the serious game or the educational simulation respectively. The items on emotional engagement measure the experience of enjoyment and fun during the interaction with the serious game or the educational simulation. The items on subjective knowledge gains measure the subjective rating on learning progress during game, which can be considered as a rating of learning-related self-efficacy. Interest can be interpreted as the willingness to invest more time in learning about the topic. Learning-related self-efficacy and increased interest can be considered important dependent variables, both related to current and future academic performance (Alexander & Grossnickle, 2016; Schunk & DiBenedetto, 2016).

Each of these four aspects was surveyed with three items on five-point rating scales (strongly disagree – strongly agree) and checked for reliability with Cronbach’s α coefficients. The reliability for the subscales of enjoyment, knowledge gains and interest was good, and for deep thinking merely satisfactory (Table 1). Despite these lower values, this scale was retained due to its relevance with regard to the research questions. To assess the reliability together with the discriminatory power of this four-factor measurement model, a confirmatory factor analysis was carried out. Fit values indicate a well-fitting model, apart from the significant Chi2, which might be an effect of the sample size (Robust Chi2(46) = 65.162, p < .05, CFI = .974, TLI = .963, RMSEA = .060, SRMR = .040). For further analysis, mean-scores were computed for each subscale.

Table 1.

Index Variables and Items of Deep Thinking, Knowledge Gains, Enjoyment and Interest for Learners Working With Serious Game.

Index variable	Item
Deep thinking (3 items, Cα = .63)	‘For the game, it was important to be well acquainted with the Internet.’
Deep thinking (3 items, Cα = .63)	‘I had to know a lot about the Internet to get a lot of points.’
	‘I have to learn something about the Internet to get better in the game.’
Knowledge gains (3 items, Cα = .85)	‘I learned a lot about what information is good on the Internet in the game.’
Knowledge gains (3 items, Cα = .85)	‘I learned something about which Internet pages should be avoided in the game.’
	‘Through the game, I know better how to recognize dangers in the Internet.’
Enjoyment (3 items, Cα = .91)	‘The game was a lot of fun.’
Enjoyment (3 items, Cα = .91)	‘I would love to play the game again.‘
	‘The game was amusing.’
Interest (3 items, Cα = .85)	‘The game has increased my interest in the Internet.’
Interest (3 items, Cα = .85)	‘Now I want to know more about how to distinguish between good and bad Internet sites.’
	‘Through the game, I learned to become more attentive on the Internet.’

Pre- and post-test knowledge

Subject-related knowledge was measured with a standardized pre- and post-test consisting originally of 40 items. The test used screenshots of ten webpages, all related to the topic of the game (reasons for traffic accidents). For each screenshot, students had to respond to a set of four items, judging on a three-point scale whether this website can be considered as trustworthy, appropriate, complete and neutral (Table 2). The test questions were created based on a literature research on the subject of information assessment as a partial aspect of information competence. Three experts assessed the correct answers independently of each other. The answers were graded by comparing student answers to an expert rating. Matching answers were assessed with 1 point and answers deviating from the expert answers with 0 points. All items were checked for overall internal consistency (α ≥ .70) as well as individual item difficulty (20% - 80%) and sufficient item discrimination (≥ .20). After applying these criteria, ten items were selected for further analysis. The reduced test showed good internal consistency in both the pre-test (α = .72) and the post-test (α = .70). For further analysis, sum scores of the pre-test and post-test were computed for each student. To determine the individual knowledge-gain score, the difference between pre-test and post-test scores was calculated.

Table 2.

Example of Test Item: Judging Four Aspects of a Website.

This website is…(please select the correct answer)	Agree	Partially	Disagree
…trustworthy (what is written here is correct)	□	□	□
…suitable (it contains useful information for answering the research question)	□	□	□
…complete (you can find all necessary information on this website)	□	□	□
…neutral (there are facts, not personal opinions)	□	□	□

Statistical Analysis

Although the individual items were measured using ordinal rating scales, composite scores were treated as interval variables and parametric tests were used for the analysis. As the underlying concepts of the composite scales can be considered as continuous and all other assumptions regarding the statistical tests were checked and met – for example, the normality of sampling distributions and homogeneity of variance for t-tests as well as multicollinearity, independence of errors and normality of residuals for regression analyses - applying parametric methods of analysis can be considered as robust (Carifio & Perla, 2008; Norman, 2010). Welch’s t-test was used to compare mean-score differences between the two experimental groups with regard to tested learning gains, self-reported cognitive and motivational learning gains, as well as cognitive and emotional engagement. Non-parametric Wilcox tests were used to check these results, without major deviations from the parametric counterparts. This is why they are not explicitly reported in the results. To investigate the relationship between engagement and learning gains, Pearson correlations between all variables were conducted. Non-parametric Spearman correlations were used to check the results, also without major deviations from the parametric results. Furthermore, hierarchical regression analysis was employed to test the combined effects on tested knowledge and self-reported cognitive and motivational learning gains. Residuals were checked for multicollinearity, independence of error and normality, and all statistical prerequisites were met. In addition to tests for significance, effect size coefficients will be provided whenever possible. All statistical analyses were executed with R and additional R packages (stats, sjPlot, pwr, lavaan).

Results

Comparing Effects on Treatment Groups

Statistical comparisons showed no group differences in subject knowledge test-score in either the pre-test (t1, d = −.06) nor in the post-test (t2, d = .14) (Table 3). Students showed minor improvements in tested knowledge between the pre-test and post-test, with a small and non-significant effect favoring the serious game group (d = .21).

Table 3.

T-Test Comparing Test Scores of the Serious Game Group and Educational Simulation Group.

	Educational simulation			Serious game
	N	M	(SD)	N	M	(SD)	d	t	(df)	p
Subject knowledge t1	75	3.67	2.60	78	3.53	2.33	−0.06	0.35	147.88	.72
Subject knowledge t2	75	4.16	2.50	78	4.49	2.37	0.14	−0.83	149.63	.41
Knowledge gain t2-t1	75	.49	2.38	78	.96	2.21	.21	−1.26	148.97	.21

Note: scale 0-10

With regard to different types of engagement, students in the group working with the serious game reported significantly more deep thinking than students in educational simulation group. The effect size was small to medium (d = .32). However, with regard to enjoyment (d = - .06), subjective knowledge gain (d = - .03) and increase in interest (d = - .03), there was no difference between the groups (Table 4).

Table 4.

T-Test Comparing Serious Game Group and Educational Simulation Group.

	Educational simulation			Serious game
	N	M	(SD)	N	M	(SD)	d	t	(df)	p
Deep-thinking	74	3.17	1.01	77	3.49	.96	.32	−2.04	147.68	.04
Enjoyment	74	3.62	1.13	78	3.55	1.28	−.06	.35	149.26	.73
Subjective knowledge gain	74	3.74	1.01	78	3.71	1.10	−.03	.18	149.82	.85
Interest	74	3.52	1.01	77	3.49	1.14	−.03	.17	148.04	.87

Note: scale 1-5

Interactions Between Any Dependent Variables

To analyze the relationship between engagement variables and learning gains, Pearson correlations were conducted. After applying a Bonferroni correction for multiple comparisons, results indicated that there is no correlation between tested knowledge (both t1 and t2) and any dependent variables. However, self-assessed subjective knowledge gains and increase in interest are positively and significantly correlated. Both deep thinking and enjoyment are also positively correlated with self-assessed subjective knowledge gain. There is also a correlation between deep thinking and enjoyment as well as between deep thinking and group (Table 5).

Table 5.

Intercorrelation Using Pearson-Method.

Measure	1	2	3	4	5	6
1 Group
2 Subject knowledge t1	−.03
3 Subject knowledge t2	.08	.56***
4 Deep thinking	.18*	−.05	.03
5 Subjective knowledge gain	−.01	.07	.12	.38***
6 Enjoyment	−.03	.02	−.04	.36***	.64***
7 Interest	−.01	.04	−.03	.47***	.76***	.71***

Note: n = 150; *** = p < .001, * = p < .05. Coding of group (1 = serious game, 0 = educational simulation)

To explore these interrelations further, multiple hierarchical linear regressions were conducted to show the combined influence of treatment-group membership, pretest-scores and deep thinking as well as enjoyment on post-test scores, subjective knowledge gains or increase in interest. While the tested post-test scores were significantly affected by the pretest-scores, no influence of group on deep thinking or enjoyment was found. Table 6 shows the hierarchical regression model, with the full model (step 3) explaining 33% of variance of the tested subject knowledge t2 with prior subject knowledge t1 as the only significant factor.

Table 6.

Hierarchical Linear Regression: Influence of Different Variables on Tested Subject Knowledge t2.

Factor	B	SE	β	R²	ΔR²
Step 1				.32***
Group	.43	.33	.09
Subject knowledge t1	.56	.07	.56***
Step 2				.32***	.00
Group	.39	.34	.08
Subject knowledge t1	.56	.07	.56***
Deep thinking	.12	.17	.05
Step 3				.33***	.01
Group	.36	.34	.07
Subject knowledge t1	.56	.07	.57***
Deep thinking	.18	.18	.07
Enjoyment	−.14	.15	−.07

Note: n = 150; *** = p < .001

In contrast, the regression analysis predicting self-reported knowledge gains shows that this aspect was influenced by both deep thinking and enjoyment. Moreover, tested prior knowledge had no influence and, once again, group membership had no effect. In the full model (step 3), deep thinking explains 15% of the variance in self-reported learning gains, and enjoyment an additional 29%. In total, the model explains 45% of the variance (Table 7).

Table 7.

Hierarchical Linear Regression: Influence of Different Variables on Self-Reported Subjective Knowledge Gain.

Factor	B	SE	β	R²	ΔR²
Step 1				.01
Group	−.02	.17	−.01
Subject knowledge t1	.03	.04	.07
Step 2				.16***	.15***
Group	−.17	.16	−.08
Subject knowledge t1	.04	.03	.09
Deep thinking	.42	.08	.40***
Step 3				.45***	.29***
Group	−.06	.13	−.03
Subject knowledge t1	.03	.03	.07
Deep thinking	.19	.07	.18**
Enjoyment	.50	.06	.58***

Note: n = 150; *** = p < .001

A similar pattern can be observed for self-reported increase in interest. Deep thinking and enjoyment are significant predictors, while there is no effect from the group or the pre-test-score. Deep thinking explains 24% of the variance and enjoyment an additional 32%. In total, the final model accounts for 56% of the variance in increase in interest (Table 8).

Table 8.

Hierarchical Linear Regression: Influence of Different Variables on Increase in Interest.

Factor	B	SE	Β	R²	ΔR²
Step 1				.00
Group	−.03	.18	−.01
Subject knowledge t1	.02	.04	.04
Step 2				.24***	.24***
Group	−.20	.16	−.10
Subject knowledge t1	.03	.03	.07
Deep thinking	.54	.08	.49***
Step 3				.56***	.32***
Group	−.09	.12	−.04
Subject knowledge t1	.02	.02	.05
Deep thinking	.29	.07	.27***
Enjoyment	.54	.05	.61***

Note: n = 150; *** = p < .001

Summary and Discussion

This article compares learning with a serious game and learning with a non-gamified digital educational simulation, both focusing on the topic of critical information literacy online. The randomized experimental study examines differences when learning with a serious game or an educational simulation with regard to enjoyment and deep thinking, increased interest, and both self-reported and tested cognitive learning gains. Furthermore, the study explores the interplay of enjoyment and deep thinking, on the one hand, and learning outcomes, on the other.

Contrary to the hypotheses, no differences in tested cognitive knowledge gains or in self-perceived cognitive knowledge gains and interest were found when comparing students working with a serious game with those working with an educational simulation (H1a − H1c rejected). Students learning with the serious game reported more deep thinking but equal enjoyment when compared to students learning with an educational simulation (H2a confirmed, H2b rejected). Furthermore, enjoyment and deep thinking correlate with self-reported knowledge gain and increase in interest (H3a-H3d confirmed) but not with tested subject knowledge t2. With regard to the interplay of these variables, it makes no difference whether learning occurs with a serious game or an educational simulation (H3e rejected). Although no negative effects were visible in the serious game condition when compared to the educational simulation, most positive effects were too small to be of any significance.

As almost all the hypotheses with regard to group differences are rejected, results are challenging to explain. Previous research has repeatedly shown that learning with serious games leads to higher levels of enjoyment and deep thinking, ultimately resulting in higher overall motivational gains and learning gains (Clark et al., 2015; Connolly et al., 2012; Ke, 2008b; Landers, 2014). Our study found an increase only in deep thinking.

These differences with regard to deep thinking can possibly be explained as an effect of the indirect controls of the serious game, which were intended to slow down the decision-making process and to promote deep thinking. Apart from this effect, this study seems to support findings like those of Adams, Mayer, MacNamara, Koenig, and Wainess (2012) or Landers and Armstrong (2017), who also found no increases in motivation or learning gains in gamified learning environments.

The most obvious explanation for these missing effects is that the combination of competitive points, challenging controls and playful visuals for the serious game were not sufficiently different from the educational simulation to yield bigger differences with regard to enjoyment, further learning gains and increase in interest. It has been highlighted by different authors that gamification elements should not be superficial add-ons to simulations but instead should be ingrained more deeply in the grammar of the game environment (Granic et al., 2014; Landers, 2014; van Staalduinen & de Freitas, 2011). Game elements must be closely linked to the learning content. One of the reasons emphasized for intentional learning is that objectives, instructional methods and assessment must always be closely linked and coordinated with one another (Anderson & Krathwohl, 2001; van Staalduinen & de Freitas, 2011). From the perspective of multimedia learning, independent additional game elements might distract from the learning content and take up too many extraneous cognitive resources (Mayer, 2014; Paas, Renkl, & Sweller, 2003). Our study seems to support this notion.

Furthermore, these results point out the importance of the educational context. It is probably not only the game elements that are decisive but also how the game is embedded in the lessons (De Jong et al., 1999; Garris et al., 2002; Leutner, 1993; Mayer & Johnson, 2010). Various studies have shown that explanations, instructional support as well as time spent on task have an influence on learning gain and motivation (Landers & Landers, 2015; Leutner, 1993; van Merriänboer, Kirschner, & Kester, 2003; Zhang, Chen, Sun, & Reid, 2004). Future studies should therefore also consider aspects of the educational context more specifically.

In addition, it is important in serious game research to look not only at the effect of individual game elements, but also how they are combined. Add-on studies allow the controlled investigation of effects of individual game elements. However, it is the combination of elements in their coherence that makes an effective serious game (Landers, 2014). Research needs to find answers to questions regarding, on the one hand, the tension between entertaining game elements that promote fun and, on the other, instructional game elements that adequately support the cognitive learning process without disrupting the game and suppressing the fun (Ritterfeld & Weber, 2006). In this study, the combination of game elements might be responsible for an increased extraneous cognitive load, which could be an explanation for the lack of clearer advantages for the serious game in this study (Mayer, 2014). Especially for the development of new effective serious games, it is important that future research investigates which combinations of game elements do not distract too much from learning content but rather promote cognitive and emotional engagement. Such controlled experimental comparisons with variations of combinations of game elements are particularly suitable for this purpose (Phye, Robinson, & Levin, 2005). Nevertheless, there were no negative effects in this study, so this serious game does not seem to impact negatively on enjoyment, interest and learning.

Nevertheless, results need to be interpreted with caution as there are several limitations in the methods used in the study. The absence of visible effects might also be a problem caused by the short duration of experimental treatment. A longer learning period might possibly have yielded higher learning gains and greater differences between the experimental groups. In addition, both the content instruction at the beginning of the treatment and the debriefing were relatively short. Greater emphasis should be placed on instruction and debriefing, because various studies have shown that it is difficult to abstract complex concepts from a serious game alone and the learning process must therefore be supported (Garris et al., 2002; Simons, 1993). Studies on instructional elements or debriefing, e.g. on the effect of pre-training or coaching, have shown that this might increase cognitive capacity to deal with the learning content and reduce extraneous load (De Jong et al., 1999; Leutner, 1993; Mayer & Johnson, 2010).

Various limitations also refer to the measures: Engagement and learning gains were measured by means of questionnaires, in which self-assessments might be biased. Moreover, the brief rating scales focused on highly specific aspects of cognitive engagement (i.e. deep thinking and the application of knowledge) and emotional engagement (i.e. fun and enjoyment), and results might be different when employing different or more comprehensive measures. Furthermore, there were no validated measuring instruments for the survey of primary school students. Therefore, a separate instrument had to be developed. The tested learning gains were measured using a rather short scale, which was adequate for primary school students but did not produce as much variance as we had hoped. A more sophisticated test could possibly show differences and developments in more detail. Finally, we did not include gameplay log-data to monitor the actual interaction with the game. As the study strictly follows a randomized design, the results are reasonably robust to justify our interpretations.

On a more general level, the study could show that there is strong correlation between deep thinking and enjoyment on the one hand, and perceived cognitive learning gain and increase in interest on the other. This also mirrors a fundamental assumption that is often found in the literature (Ladd & Dinella, 2009; Landers, 2014). Although the engagement did not show a correlation with tested learning gains in our study, it seems to be worth the effort to promote cognitive and emotional engagement (Filsecker & Hickey, 2014). In addition, the study was able to contribute in showing which combination of game elements can be used to promote deep thinking. Further studies with different combinations of game characteristics need be carried out to identify successful strategies to promote emotional and cognitive engagement as well as motivational gains and increases in learning gains.

Footnotes

Acknowledgements

We would like to thank the reviewers for their comments on earlier versions of this article. This research has been conducted following the ethical requirements established by the German Educational Research Association.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Swiss National Science Foundation, project number: 13DPD3_134705.

Author Biographies

Nina Imlig-Iten, MA, is a lecturer for media pedagogy and media didactics at the Schwyz University of Teacher Education. Her research interests are game-based learning and media education.

Contact: nina.imlig@phsz.ch

Dominik Petko, Prof. Dr., is professor and director of the Institute for Media and Schools. He directs a number of research projects in the area of learning with digital media.

Contact: dominik.petko@phsz.ch

References

Abt

C. C.

(1970). Serious games. New York, NY: Viking Press. Retrieved from https://eric.ed.gov/?q=+serious+games+abt&id=ED039447

Adams

D. M.

Mayer

R. E.

MacNamara

Koenig

Wainess

(2012). Narrative games for learning: Testing the discovery and narrative hypotheses. Journal of Educational Psychology, 104(1), 235-249. doi:10.1037/a0025595

Alexander

P. A.

Grossnickle

E. M.

(2016). Positioning interest and curiosity within a model of academic development. In Wentzel

K. R.

Miele

D. B.

(Eds.), Handbook of motivation at school (pp. 188-208). New York, NY: Routledge. Available from www.routledge.com

Anderman

E. M.

Dawson

(2011). Learning with motivation. In Mayer

R. E.

Alexander

P. A.

(Eds.), Handbook of research on learning and instruction (pp. 219-241). New York, NY: Routledge. Available from www.routledge.com

Anderson

L. W.

Krathwohl

D. R.

(Eds.) (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy. New York, NY: Pearson Longman. Available from https://www.pearson.ch

Bedwell

W. L.

Pavlas

Heyne

Lazzara

E. H.

Salas

(2012). Toward a taxonomy linking game attributes to learning: An empirical study. Simulation & Gaming, 43(6), 729-760. doi:10.1177/1046878112439444

Boocock

S. S.

Schild

E. O.

(Eds.). (1968). Simulation games in learning. Beverly Hills, CA: SAGE. Retrieved from http://eric.ed.gov/?id=ED026857

Bouvier

Lavoué

Sehaba

(2014). Defining engagement and characterizing engaged-behaviors in digital gaming. Simulation & Gaming, 45(4-5), 491-507. doi:10.1177/1046878114553571

Boyle

E. A.

Connolly

T. M.

Hainey

Boyle

J. M.

(2012). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28(3), 771-780. doi:10.1016/j.chb.2011.11.020

10.

Carifio

Perla

(2008). Resolving the 50-year debate around using and misusing Likert scales. Medical Education, 42(12), 1150-1152. doi:10.1111/j.1365-2923.2008.03172.x

11.

Chang

J.-W.

Wei

H.-Y.

(2016). Exploring engaging gamification mechanics in massive online open courses. Journal of Educational Technology & Society, 19(2), 177-203. Retrieved from https://www-jstor-org.web.bisu.edu.cn/stable/jeductechsoci.19.2.177

12.

Clark

D. B.

Tanner-Smith

E. E.

Killingsworth

S. S.

(2015). Digital games, design, and learning a systematic review and meta-analysis. Review of Educational Research, 86(1), 79-122. doi:10.3102/0034654315582065

13.

Connolly

T. M.

Boyle

E. A.

MacArthur

Hainey

Boyle

J. M.

(2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. doi:10.1016/j.compedu.2012.03.004

14.

De Jong

. (1991). Learning and instruction with computer simulations. Education and Computing, 6(3), 217-229. doi:10.1016/0167-9287(91)80002-F

15.

De Jong

. (2011). Instruction based on computer simulation. In Mayer

R. E.

Alexander

P. A.

(Eds.), Handbook of research on learning and instruction (pp. 446-466). New York, NY: Routledge. Retrieved from https://www.routledge.com/Handbook-of-Research-on-Learning-and-Instruction/Mayer-Alexander/p/book/9781138831766

16.

De Jong

Martin

Zamarro

J.-M.

Esquembre

Swaak

Van Joolingen

W. R.

(1999). The integration of computer simulation and learning support: An example from the physics domain of collisions. Journal of Research in Science Teaching, 36(5), 597-615. doi:10.1002/(SICI)1098-2736(199905)36:5<597::AID-TEA6>3.0.CO;2-6

17.

Deci

E. L.

Koestner

Ryan

R. M.

(1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627-668. doi: 10.1037/0033-2909.125.6.627

18.

Deterding

Dixon

Khaled

Nacke

(2011). From game design elements to gamefulness: Defining gamification. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning future media environments (pp. 9-15). doi:10.1145/2181037.2181040

19.

Dickey

M. D.

(2005). Engaging by design: How engagement strategies in popular computer and video games can inform instructional design. Educational Technology Research & Development, 53(2), 67-83. doi:10.1007/BF02504866

20.

Domínguez

Saenz-de-Navarrete

de-Marcos

Fernández-Sanz

Pagés

Martínez-Herráiz

J.-J.

(2013). Gamifying learning experiences: Practical implications and outcomes. Computers & Education, 63, 380-392. doi:10.1016/j.compedu.2012.12.020

21.

Filsecker

Hickey

D. T.

(2014). A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement and learning in an educational game. Computers & Education, 75, 136-148. doi:10.1016/j.compedu.2014.02.008

22.

Filsecker

Kerres

(2014). Engagement as a volitional construct: A framework for evidence-based research on educational games. Simulation & Gaming, 45(4-5), 450-470. doi:10.1177/1046878114553569

23.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59-109. doi:10.3102/00346543074001059

24.

F. L.

R. C.

S. C.

(2009). EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52(1), 101-112. doi:10.1016/j.compedu.2008.07.004

25.

Garris

Ahlers

Driskell

J. E.

(2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467. doi:10.1177/1046878102238607

26.

Gillispie

Martin

Parker

M. A.

(2010). Effects of a 3-D video game on middle school student achievement and attitude in mathematics. Electronic Journal of Mathematics and Technology, 4(1). Retrieved from https://php.radford.edu/~ejmt/ContentIndex.php#v4n3

27.

Granic

Lobel

Engels

R. C.

(2014). The benefits of playing video games. American Psychologist, 69(1), 66-78. doi:10.1037/a0034857

28.

Gredler

M. E.

(2004). Games and simulations and their relationships to learning. In Jonassen

D. H.

(Ed.), Handbook of research on educational communications and technology (Vol. 2, pp. 571-581). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

29.

Hamari

Koivisto

Sarsa

(2014). Does gamification work?—A literature review of empirical studies on gamification. In Proceedings of the 47th Hawaii International Conference on System Sciences (pp. 3025-3034). doi:10.1109/HICSS.2014.377

30.

Jabbar

A. I.

Felicia

(2015). Gameplay engagement and learning in game-based learning. Review of Educational Research, 85(4), 740-779. doi:10.3102/0034654315577210

31.

Kapp

K. M.

(2012). The gamification of learning and instruction: Game-based methods and strategies for training and education. San Francisco, CA: John Wiley. Available from www.wiley.com

32.

(2008a). A case study of computer gaming for math: Engaged learning from gameplay? Computers & Education, 51(4), 1609-1620. doi:10.1016/j.compedu.2008.03.003

33.

(2008b). Computer games application within alternative classroom goal structures: Cognitive, metacognitive, and affective evaluation. Educational Technology Research & Development, 56(5-6), 539-556. doi:10.1007/s11423-008-9086-5

34.

Abras

(2013). Games for engaged learning of middle school children with special learning needs. British Journal of Educational Technology, 44(2), 225-242. doi:10.1111/j.1467-8535.2012.01326.x

35.

Xie

(2016). Game-based learning engagement: A theory- and data-driven exploration. British Journal of Educational Technology, 47(6), 1183-1201. doi:10.1111/bjet.12314

36.

Kebritchi

Hirumi

Bai

(2010). The effects of modern mathematics computer games on mathematics achievement and class motivation. Computers & Education, 55, 427-443. doi:10.1016/j.compedu.2010.02.007

37.

Klabbers

J. H. G.

(2009). The magic circle: Principles of gaming & simulation (3rd & Rev. ed.). Rotterdam, The Netherlands: Sense Publishers. Available from https://www.sensepublishers.com

38.

Kriz

W. C.

(2017). Historical roots and new fruits of gaming and simulation. Simulation & Gaming, 48(5), 583-587. doi:10.1177/1046878117732845

39.

Ladd

G. W.

Dinella

L. M.

(2009). Continuity and change in early school engagement: Predictive of children’s achievement trajectories from first to eighth grade? Journal of Educational Psychology, 101(1), 190-206. doi:10.1037/a0013153

40.

Landers

R. N.

(2014). Developing a theory of gamified learning: Linking serious games and gamification of learning. Simulation & Gaming, 45(6), 752-768. doi:10.1177/1046878114563660

41.

Landers

R. N.

Armstrong

M. B.

(2017). Enhancing instructional outcomes with gamification: An empirical test of the Technology-Enhanced Training Effectiveness Model. Computers in Human Behavior, 71, 499-507. doi:10.1016/j.chb.2015.07.031

42.

Landers

R. N.

Callan

R. C.

(2012). Training evaluation in virtual worlds: Development of a model. Journal for Virtual Worlds Research, 5(3), 1-20. doi:10.4101/jvwr.v5i3.6335

43.

Landers

R. N.

Landers

A. K.

(2015). An empirical test of the theory of gamified learning: The effect of leaderboards on time-on-task and academic performance. Simulation & Gaming, 45(6), 769-785. doi:10.1177/1046878114563662

44.

Leutner

(1993). Guided discovery learning with computer-based simulation games: Effects of adaptive and non-adaptive instructional support. Learning and Instruction, 3(2), 113-132. doi:10.1016/0959-4752(93)90011-N

45.

Liao

C. C. Y.

Chen

Z.-H.

Cheng

H. N. H.

Chen

F.-C.

Chan

T.-W.

(2011). My-Mini-Pet: A handheld pet-nurturing game to engage students in arithmetic practices. Journal of Computer Assisted Learning, 27(1), 76-89. doi:10.1111/j.1365-2729.2010.00367.x

46.

Mayer

R. E.

(2014). Computer games for learning: An evidence-based approach. Cambridge, MA: MIT Press. Retrieved from https://mitpress.mit.edu/books/computer-games-learning

47.

Mayer

R. E.

Johnson

C. I.

(2010). Adding instructional features that promote learning in a game-like environment. Journal of Educational Computing Research, 42(3), 241-265. doi:10.2190/EC.42.3.a

48.

Mekler

E. D.

Brühlmann

Tuch

A. N.

Opwis

(2017). Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Computers in Human Behavior, 71, 525-534. doi:10.1016/j.chb.2015.08.048

49.

Michael

D. R.

Chen

S. L.

(2005). Serious games: Games that educate, train, and inform. Boston, MA: Thomson Course Technology. Retrieved from http://www.cengageptr.com/Topics/QuickSearch?q=serious%20games

50.

Norman

(2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education, 15(5), 625-632. doi:10.1007/s10459-010-9222-y

51.

Müller

Petko

Götz

(2011). AWWWARE - a “game for teaching” to improve children’s internet literacy. In Wiemeyer

Göbel

(Eds.), Serious Games - Theory, Technology & Practice. Proceedings - Game Days 2011 (pp. 53–59). Darmstadt: Institut für Sportwissenschaft. Retrieved from http://www.serious-games.tu-darmstadt.de/publikationen_5/publikationen_6/index.de.jsp

52.

O’Brien

H. L.

Toms

E. G.

(2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 59(6), 938-955. doi:10.1002/asi.20801

53.

Paas

Renkl

Sweller

(2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1-4. doi:10.1207/S15326985EP3801_1

54.

Phillips

R. S.

Horstman

Vye

Bransford

(2014). Engagement and games for learning: Expanding definitions and methodologies. Simulation & Gaming, 45(4-5), 548-568. doi:10.1177/1046878114553576

55.

Phye

G. D.

Robinson

D. H.

Levin

(2005). Empirical methods for evaluating educational interventions. San Diego, CA: Elsevier Academic Press. Available from https://www-elsevier-com-s.web.bisu.edu.cn

56.

Rich

B. L.

Lepine

J. A.

Crawford

E. R.

(2010). Job engagement: Antecedents and effects on job performance. Academy of Management Journal, 53(3), 617-635. doi:10.5465/AMJ.2010.51468988

57.

Rieber

L. P.

Noah

(2008). Games, simulations, and visual metaphors in education: Antagonism between enjoyment and learning. Educational Media International, 45(2), 77-92. doi:10.1080/09523980802107096

58.

Ritterfeld

Cody

Vorderer

(Eds.). (2009). Serious games: Mechanisms and effects. New York, NY: Routledge. Available from https://www.routledge.com

59.

Ritterfeld

Weber

(2006). Video games for entertainment and education. In Vorderer

Bryant

(Eds.), Playing video games: Motives, responses, and consequences (pp. 399-413). Mahwah, NJ: Lawrence Erlbaum. Retrieved from https://www.researchgate.net/profile/Ute_Ritterfeld/publication/309901513_Video_Games_for_Entertainment_and_Education/links/53f706990cf2888a74976416/Video-Games-for-Entertainment-and-Education.pdf

60.

Rosas

Nussbaum

Cumsille

Marianov

Correa

Flores

. . . Salinas

(2003). Beyond Nintendo: Design and assessment of educational video games for first and second grade students. Computers & Education, 40(1), 71-94. doi:10.1016/S0360-1315(02)00099-4

61.

Rubin-Vaughan

Pepler

Brown

Craig

(2011). Quest for the Golden Rule: An effective social skills promotion and bullying prevention program. Computers & Education, 56(1), 166-175. doi:10.1016/j.compedu.2010.08.009

62.

Schunk

D. H.

DiBenedetto

M. K.

(2016). Self-efficacy theory in education. In Wentzel

K. R.

Miele

D. B.

(Eds.), Handbook of motivation at school (pp. 34-54). New York, NY: Routledge.

63.

Seaborn

Fels

D. I.

(2015). Gamification in theory and action: A survey. International Journal of Human-Computer Studies, 74, 14-31. doi:10.1016/j.ijhcs.2014.09.006

64.

Simons

K. L.

(1993). New technologies in simulation games. System Dynamics Review, 9(2), 135-152. doi:10.1002/sdr.4260090204

65.

Sitzmann

(2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489-528. doi:10.1111/j.1744-6570.2011.01190.x

66.

Sun Lin

H.-Z.

Chiou

G.-F.

(2017). Effects of comparison and game-challenge on sixth graders’ algebra variable learning achievement, learning attitude, and meta-cognitive awareness. EURASIA Journal of Mathematics, Science and Technology Education, 13(6), 2627-2644. doi:10.12973/eurasia.2017.01244a

67.

Sweller

Ayres

Kalyuga

(2011). Cognitive load theory. New York, NY: Springer. doi:10.1007/978-1-4419-8126-4

68.

Tan

J. L.

Goh

D. H.-L.

Ang

R. P.

Huan

V. S.

(2013). Participatory evaluation of an educational game for social skills acquisition. Computers & Education, 64, 70-80. doi:10.1016/j.compedu.2013.01.006

69.

Tobias

Fletcher

J. D.

Dai

D. Y.

Wind

A. P.

(2011). Review of research on computer games. In Tobias

Fletcher

J. D.

(Eds.), Computer games and instruction (pp. 127-222). Charlotte, NC: Information Age Publishing. Retrieved from http://www.infoagepub.com/products/Computer-Games-and-Instruction

70.

Tüzün

Yılmaz-Soylu

Karakuş

İnal

Kızılkaya

(2009). The effects of computer games on primary school students’ achievement and motivation in geography learning. Computers & Education, 52(1), 68-77. doi:10.1016/j.compedu.2008.06.008

71.

van Merriänboer

J. J.

Kirschner

P. A.

Kester

(2003). Taking the load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5-13. doi:10.1207/S15326985EP3801_2

72.

van Staalduinen

J.-P.

de Freitas

(2011). A game-based learning framework: Linking game design and learning. In Kihne

M. S.

(Ed.), Learning to play: Exploring the future of education with video games (Vol. 53, pp. 29-54). New York, NY: Peter Lang. Available from https://www.peterlang.com

73.

Virkus

(2003). Information literacy in Europe: A literature review. Information Research, 8(4). Retrieved from http://www.informationr.net/ir/8-4/paper159.html

74.

Vogel

J. J.

Vogel

D. S.

Cannon-Bowers

Bowers

C. A.

Muse

Wright

(2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34(3), 229-243. doi:10.2190/FLHV-K4WA-WPVQ-H0YM

75.

Wang

Hannafin

M. J.

(2005). Design-based research and technology-enhanced learning environments. Educational Technology Research & Development, 53(4), 5-23. doi:10.1007/BF02504682

76.

Wiebe

E. N.

Lamb

Hardy

Sharek

(2014). Measuring engagement in video game-based environments: Investigation of the User Engagement Scale. Computers in Human Behavior, 32, 123-132. doi:10.1016/j.chb.2013.12.001

77.

Wilson

K. A.

Bedwell

W. L.

Lazzara

E. H.

Salas

Burke

C. S.

Estock

J. L.

. . . Conkey

(2009). Relationships between game attributes and learning outcomes: Review and research proposals. Simulation & Gaming, 40(2), 217-266. doi:10.1177/1046878108321866

78.

Wouters

van Nimwegen

van Oostendorp

van der Spek

E. D.

(2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249-265. doi:10.1037/a0031311

79.

Yang

J. C.

Chen

Y. H.

Chen

C. H.

(2007). PILE: Physical Interactive Learning Environment. In Proceedings of the First IEEE International Workshop on Digital Game and Intelligent Toy Enhanced Learning (DIGITEL’07) (pp. 218-220). doi:10.1109/DIGITEL.2007.39

80.

Zhang

Chen

Sun

Reid

D. J.

(2004). Triple scheme of learning support design for scientific discovery learning based on computer simulation: Experimental research. Journal of Computer Assisted Learning, 20(4), 269-282. doi:10.1111/j.1365-2729.2004.00062.x

81.

Zheng

Young

M. F.

Wagner

M. M.

Brewer

R. A.

(2009). Negotiation for action: English language learning in game-based virtual worlds. The Modern Language Journal, 93(4), 489-511. doi:10.1111/j.1540-4781.2009.00927.x