Abstract
Background and Aim.
Method. Twelve primary school classes with 153 students from 9 to 12 years of age participated in this
Results and Conclusion. Results show that there are no group differences in tested
Keywords
Introduction
Substantial amounts of research on serious games (i.e. games for educational purposes) exist, both with regard to digital and non-digital games (Abt, 1970; Klabbers, 2009; Kriz, 2017). Recent meta-analyses with regard to digital serious games have consistently shown positive effects with regard to learning gains and motivational gains (Clark, Tanner-Smith, & Killingsworth, 2015; Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012; Sitzmann, 2011; Vogel et al., 2006; Wouters, van Nimwegen, van Oostendorp, & van der Spek, 2013). However, serious game research has struggled to isolate the specific combinations of game characteristics that are responsible for these positive findings. Garris, Ahlers, and Driskell (2002) found six game attributes which were shown to be relevant for learning: fantasy, rules/goals, sensory stimuli, challenge, mystery and control. A more recent attempt has been made by Bedwell, Pavlas, Heyne, Lazzara, and Salas (2012), proposing a list of nine game attributes, which has been further reduced by Jabbar and Felicia (2015) to four core factors: motivational elements, interactive elements, fun elements and multimedia elements. However, these attributes are very general and a lack of knowledge still exists concerning the effect of combining these characteristics in different ways. Many authors agree that it is crucial for effective serious games to strive for a high degree of overlap between these kinds of game attributes and instructional objectives (Garris et al., 2002; Granic, Lobel, & Engels, 2014; Ke, 2008a; Wilson et al., 2009). However, success cannot be taken for granted (Michael & Chen, 2005; Ritterfeld, Cody, & Vorderer, 2009). One reason for this might be the way in which research on serious games is carried out. Learning with serious games is often compared to learning with conventional learning material instead of comparing different possible variations or applications of a game (Mayer, 2014; Tobias, Fletcher, Dai, & Wind, 2011). In effect, a large body of research on serious games suffers from the problem that game- and non-game-treatments are hardly comparable as they differ in too many respects. More recently, this problem has been partly resolved by research on gamified learning environments, where video-game characteristics are applied to non-game environments (Deterding, Dixon, Khaled, & Nacke, 2011; Kapp, 2012; Landers, 2014). Although gamified learning environments might suffer from the impression that game characteristics are mere add-ons, this approach creates gamified treatments that are better comparable to the non-gamified environments they are based on. In addition, studies on gamified learning environments have shown similar positive effects on user engagement as well as cognitive and motivational outcomes as studies on serious games (Hamari, Koivisto, & Sarsa, 2014; Kapp, 2012). In consequence, this type of research might be an interesting way forward in isolating game characteristics that are responsible for specific effects on learning.
Another explanation for the lack of consistent findings with regard to essential characteristics of effective serious games is provided by Landers’ (2014) theoretical model. It assumes that the effects of game elements on learning outcomes are mediated by learning-related behavior and attitudes (Landers, 2014). This needs to be taken into account when conducting empirical studies on serious games. Moreover, serious games are often embedded in broader educational activities. Thus, achieving learning outcomes depends not only on the game but also on the reflection and debriefing during which the game activities are reviewed (Garris et al., 2002; Granic et al., 2014; Ke, 2008a; Wilson et al., 2009). To sum up, further research on the effects of isolated game characteristics on learning outcomes is necessary to find out what constitutes effective serious games. Such research should also focus on learning requirements and learning processes.
Comparing Educational Simulations and Serious Games
One way to design comparable treatments is the comparison of serious games and similar educational simulations without game elements (Landers & Armstrong, 2017; Landers & Callan, 2012; Wang & Hannafin, 2005). Simulations can be defined as computer programs that represent dynamic models of natural or artificial systems, where users can change specific parameters and observe the impact on other parameters within the system (De Jong, 1991, 2011). Such a comparison is worthwhile because serious games and educational simulations are similar in many respects. They both have real world knowledge at their core, either represented as information or as dependencies between relevant variables. While educational simulations seek to follow rules that are as realistic and focused as possible, serious games are enriched by additional rules, visual metaphors, imaginary stories and challenging controls (Gredler, 2004; Rieber & Noah, 2008). Furthermore, educational simulations typically are narrowed-down interactive representations, including as few variables as possible. Relationships between variables are clearly defined and educational simulations typically avoid anything that might distract from the interplay between these core factors. By contrast, serious games have many additional features: each serious game relies on an enriched system of rules and game mechanics that constitute the playful interaction between both the serious games and with other players, which is not required in educational simulations. Serious games typically are about winning or achieving a competitive goal (for example, achieving as many points as possible) while educational simulations are about experimenting without competitive elements. Especially digital serious games may include additional aspects such as graphical presentations, game characters, sound-effects and a soundtrack, voice-acting, as well as controls that require a certain degree of dexterity, which would be considered as distracting in educational simulations (Boocock & Schild, 1968; Sitzmann, 2011). Similar aspects have been highlighted by research on gamified learning environments (Landers & Armstrong, 2017; Landers & Callan, 2012; Wang & Hannafin, 2005). Therefore, this study will compare a serious game to a similar educational simulation without game elements and examine the effects of a specific combination of game elements on engagement, interest and cognitive learning gains.
Studying the Impact on Process Variables: Enjoyment and Deep Thinking
In educational research, the term engagement has often been used for all kinds of positive student behavior. More recent studies have agreed on a more systematic understanding, defining engagement as a combination of behavioral, cognitive and emotional aspects of student involvement with a particular learning environment (Bouvier, Lavoué, & Sehaba, 2014; Boyle, Connolly, Hainey, & Boyle, 2012; Filsecker & Kerres, 2014; O’Brien & Toms, 2008). Some authors propose including additional aspects, such as perceived esthetics and usability (Fu, Su, & Yu, 2009; Wiebe, Lamb, Hardy, & Sharek, 2014). However, recent research has concluded that only emotional and cognitive engagement play a role with influencing learning, as behavioral engagement can be seen as prerequisite for these deeper kinds of engagement (Jabbar & Felicia, 2015). Within these two types of engagement, enjoyment and deep thinking have been identified as the most crucial aspects with regard to learning with serious games (Granic et al., 2014). These two aspects have impact on increased interest and cognitive learning outcomes (Dickey, 2005; O’Brien & Toms, 2008; Phillips, Horstman, Vye, & Bransford, 2014). Therefore, this study focuses on these aspects when comparing a serious game with an educational simulation.
The literature review of Jabbar and Felicia (2015) has summarized previous findings on the effects of specific game characteristics on student engagement. Motivational elements, such as points or ranking, seem to contribute to higher enjoyment, while results with regard to cognitive engagement are still inconclusive (Chang & Wei, 2016; Ke & Abras, 2013). Interactive elements, including problems and obstacles, can promote both emotional and cognitive engagement (Ke & Abras, 2013; Yang, Chen, & Chen, 2007). Fun elements, such as narratives, virtual characters and controls, are also positively related to both cognitive and emotional engagement (Ke, 2008a; Ke, Xie, & Xie, 2016). Challenge elements can improve enjoyment (Ke, 2008a) and deep thinking (Rubin-Vaughan, Pepler, Brown, & Craig, 2011). Multimedia elements, such as attractive visuals, affect especially emotional engagement (Rosas et al., 2003; Tan, Goh, Ang, & Huan, 2013). However, as noted earlier, only few studies exist on combinations of these game elements.
Studying the Impact on Outcome Variables: Interest and Achievement
Next to enjoyment and deep thinking, research needs to address the impact of specific game characteristics on different types of learning outcomes. Although these effects are likely to be mediated by engagement, many experimental studies have been able to show direct effects on tested learning gains, self-reported learning gains and increased interest without taking engagement variables into account (Clark et al., 2015; Connolly et al., 2012; Sitzmann, 2011; Vogel et al., 2006; Wouters et al., 2013). In several studies, game characteristics have been varied to test their effect on interest and learning outcomes. The review of Jabbar and Felicia (2015) shows mostly positive evidence for most characteristics, with little counter-evidence. With regard to motivational elements such as points, leaderboards, achievements, badges, rewards and feedback, research has shown overwhelmingly positive effects on cognitive learning outcomes and interest (Domínguez et al., 2013; Gillispie, Martin, & Parker, 2010; Tan et al., 2013). However, with regard to points and achievements, some studies remain sceptical by showing that an over-reliance on these aspects might primarily increase extraneous motivation and superficial engagement (Deci, Koestner, & Ryan, 1999; Mekler, Brühlmann, Tuch, & Opwis, 2017). With respect to interactive elements, it was found that role-playing, in particular, can have a positive impact on cognitive learning gains (Ke, 2008b; Kebritchi, Hirumi, & Bai, 2010; Zheng, Young, Wagner, & Brewer, 2009) and increase interest (Zheng et al., 2009). Fun elements, such as virtual characters/environments, challenges, controls and narratives, have also been found to enhance both cognitive learning gains and increases in interest (Hamari et al., 2014; Rubin-Vaughan et al., 2011; Seaborn & Fels, 2015; Sun Lin & Chiou, 2017). However, these elements could also be a factor in increasing extraneous cognitive load, therefore distracting from the task at hand (Mayer, 2014; Sweller, Ayres, & Kalyuga, 2011). For multimedia elements, some studies have shown that virtual environments and game characters yielded increases in interest (Jabbar & Felicia, 2015; Liao, Chen, Cheng, Chen, & Chan, 2011; Tüzün, Yılmaz-Soylu, Karakuş, İnal, & Kızılkaya, 2009), but others are more reluctant, as this might be a matter of personal preference (Landers & Armstrong, 2017). In these studies, the effects of individual game characteristics were investigated, but – as in the studies on student engagement – the effect of the combination of specific game elements, such as points, challenge and graphics, on interest and cognitive learning gain needs further investigation.
Hypotheses
Based on the findings reported above, it can be assumed that the combination of specific motivational, fun and multimedia elements in a serious game results in higher student enjoyment and deeper thinking as well as higher cognitive learning gains and increases in subject-related interest when compared to learning with a comparable educational simulation, where these characteristics are missing. In this study, this assumption leads to the following detailed hypotheses:
H1a: Students who learn with a serious game achieve higher subject-related knowledge gains than those who learn with a comparable educational simulation.
H1b: Students who learn with a serious game achieve higher perceived knowledge gains than those who learn with a comparable educational simulation.
H1c: Students who learn with a serious game show higher perceived increases in interest in the topic than those who learn with a comparable educational simulation.
H2a: Students who learn with a serious game report higher levels of deep thinking during learning than those who learn with a comparable educational simulation.
H2b: Students who learn with a serious game report higher levels of enjoyment than those who learn with a comparable educational simulation.
As proposed by Landers (2014), it is equally essential to test the impact of game characteristics on process variables, such as cognitive engagement in the form of deep-thinking, and emotional engagement, such as enjoyment. This assumption is in line with findings from general educational research (Anderman & Dawson, 2011; Fredricks, Blumenfeld, & Paris, 2004). This leads to the following detailed hypotheses:
H3a: There is a correlation between deep thinking and subject-related knowledge gains.
H3b There is a correlation between deep thinking and increase in interest.
H3c: There is a correlation between enjoyment and subject-related knowledge gains.
H3d: There is a correlation between enjoyment and increase in interest.
H3e: These correlations from H3a till H3d are stronger for students learning with a serious game than for students learning with a comparable educational simulation.
Method
Research Design and Sample
The study was designed as a randomized field experiment comparing pre- and post-test measures of two groups of students, with one group learning with a serious game and the other with a similar educational simulation without game elements. The experiment was part of a research project supported by Swiss National Science Foundation (project number: 13DPD3_134705) and was conducted in twelve primary school classes in Central Switzerland. They were recruited through an open call in a magazine for schools. The sample consisted of 153 students from 9 to 12 years of age (M = 10.5, SD = 0.67), 75 girls and 78 boys. None of these classes had explicitly covered the topic of the game – information literacy – in previous lessons. According to ethical standards, the students and teachers were comprehensively informed about the aims and methods of the study and could choose not to participate. In order to meet the methodological requirements of randomization, the students in each class were randomly assigned to a group learning with either a serious game (N=78) or an educational simulation (N=75). Power analyses suggest that these sample sizes are appropriate to adequately detect mean-score differences between these two groups with an effect size of d = .46, p = .05 and a power of 1-b= .80 for two-sided comparisons and with r = .22 for correlations calculated with the full sample and r = .32 for within one of the experimental groups.
All the students completed a pre-test on prior knowledge one week before the experimental treatment (t1). The treatment took place during three subsequent lessons of 45 minutes in the regular classroom. Every student was given a notebook with a pre-installed version of either the serious game or of the educational simulation. Before the treatment, all the students received the same short introduction about why it is important to be cautious about information online and what to look for when judging if information on a website is trustworthy, appropriate, complete and neutral. This introduction took about half an hour. After that, the students were asked to spend 45 minutes working with either the educational simulation or the serious game, practicing their ability to identify useful information on websites. During the debriefing, the whole class discussed the play and learning process and reflected on the learning objectives. Afterwards, a post-test to assess knowledge gains and a questionnaire to measure engagement and self-reported learning gains (t2) were administered.
Material
A serious game called AWWWARE (Müller, Petko & Götz, 2011) and a comparable educational simulation called BWWWARE that included no game elements were used for the experiment. Both include the same learning goals and materials and both were developed especially for this study. The only difference between them is the omission of all gamified elements in the simulation.
The goal of both this serious game and the comparable educational simulation is to promote information literacy, especially the ability to critically assess online information when searching the Internet (Virkus, 2003). The core-mechanic relies on an interlinked labyrinth of screenshots from real webpages. The student has the task of locating suitable webpages with information on specific questions, e.g. “What are the main reasons for traffic accidents?” by applying the criteria given in the introduction beforehand. Therefore, he or she must navigate the information structure. The aim is to avoid visiting and flagging inappropriate webpages and to identify and flag trustworthy, appropriate, complete and neutral ones. Students control the learning environment by moving the mouse cursor. In the serious game, this is represented by a small mouse-shaped kite which is held and indirectly controlled by a raven. The background is an attractive rural landscape (Figure 1). In the educational simulation, a normal curser is used and instead of the gamified background with graphic visuals, the students see just a flowered wallpaper background (Figure 2). Students working with the serious game receive immediate feedback in the form of metaphoric weather changes and they receive points. These elements are missing in educational simulation. At the end of both learning environments, a detailed record of the browsing behaviour gives further information and can serve as the basis for a detailed discussion on how to look for information in class. Leaderboards and points are only available in the serious game, but not in the educational simulation. The results of individual students are intended to give teachers the opportunity to compare and discuss different strategies for online information retrieval with the whole class.

Serious game.

Educational simulation.
While the interactive elements of gamification (i.e. procedures and resources) are the same for the serious game and the educational simulation, the serious game has the following additional characteristics, according to the gaming elements proposed by Jabbar and Felicia (2015):
Motivational elements (points and achievements): Players should seek and mark a certain number of appropriate webpages for a specific task. By doing so, players are rewarded with points and high scores for selecting relevant webpages while avoiding inappropriate ones. Furthermore, the players receive immediate feedback on whether the marked webpage is correct or not.
Fun elements (challenging controls): Players have to navigate using an indirect control through the raven and the kite. Moving the raven left and right has an impact on both the horizontal and the vertical position of the kite-shaped mouse cursor. This kite flutters in the wind and changes according to the direction of the wind, make controlling the kite more difficult. This slows down browsing behaviour and should be an additional challenge, while giving students more time to consider their web-browsing decisions.
Multimedia elements (Playful graphics, game character and visual metaphors): The learning environment consists of a representation of the real Internet, which is positioned in a graphic frame with a wide comic-landscape which is home to the game character, a black raven. Several aspects in this frame serve as visual metaphors for feedback on players’ strategies: facial expressions of the raven; green, yellow or red flags on the kite; changes in lighting and weather.
Measures
Student ratings on deep thinking, enjoyment, gain in interest and knowledge gain
No previous validated measures for primary school students existed to measure either deep thinking and enjoyment or students’ subjective knowledge gains and interest. Existing measuring instruments were only available for adults and were therefore too long or too complicated for primary school students (Fu et al., 2009; Rich, Lepine, & Crawford, 2010). Therefore, suitable scales were built, inspired from items of the EGameFlow scale (Fu et al., 2009). The items on deep thinking focus on the aspect of application of knowledge during the interaction with the serious game or the educational simulation respectively. The items on emotional engagement measure the experience of enjoyment and fun during the interaction with the serious game or the educational simulation. The items on subjective knowledge gains measure the subjective rating on learning progress during game, which can be considered as a rating of learning-related self-efficacy. Interest can be interpreted as the willingness to invest more time in learning about the topic. Learning-related self-efficacy and increased interest can be considered important dependent variables, both related to current and future academic performance (Alexander & Grossnickle, 2016; Schunk & DiBenedetto, 2016).
Each of these four aspects was surveyed with three items on five-point rating scales (strongly disagree – strongly agree) and checked for reliability with Cronbach’s α coefficients. The reliability for the subscales of enjoyment, knowledge gains and interest was good, and for deep thinking merely satisfactory (Table 1). Despite these lower values, this scale was retained due to its relevance with regard to the research questions. To assess the reliability together with the discriminatory power of this four-factor measurement model, a confirmatory factor analysis was carried out. Fit values indicate a well-fitting model, apart from the significant Chi2, which might be an effect of the sample size (Robust Chi2(46) = 65.162, p < .05, CFI = .974, TLI = .963, RMSEA = .060, SRMR = .040). For further analysis, mean-scores were computed for each subscale.
Index Variables and Items of Deep Thinking, Knowledge Gains, Enjoyment and Interest for Learners Working With Serious Game.
Pre- and post-test knowledge
Subject-related knowledge was measured with a standardized pre- and post-test consisting originally of 40 items. The test used screenshots of ten webpages, all related to the topic of the game (reasons for traffic accidents). For each screenshot, students had to respond to a set of four items, judging on a three-point scale whether this website can be considered as trustworthy, appropriate, complete and neutral (Table 2). The test questions were created based on a literature research on the subject of information assessment as a partial aspect of information competence. Three experts assessed the correct answers independently of each other. The answers were graded by comparing student answers to an expert rating. Matching answers were assessed with 1 point and answers deviating from the expert answers with 0 points. All items were checked for overall internal consistency (α ≥ .70) as well as individual item difficulty (20% - 80%) and sufficient item discrimination (≥ .20). After applying these criteria, ten items were selected for further analysis. The reduced test showed good internal consistency in both the pre-test (α = .72) and the post-test (α = .70). For further analysis, sum scores of the pre-test and post-test were computed for each student. To determine the individual knowledge-gain score, the difference between pre-test and post-test scores was calculated.
Example of Test Item: Judging Four Aspects of a Website.
Statistical Analysis
Although the individual items were measured using ordinal rating scales, composite scores were treated as interval variables and parametric tests were used for the analysis. As the underlying concepts of the composite scales can be considered as continuous and all other assumptions regarding the statistical tests were checked and met – for example, the normality of sampling distributions and homogeneity of variance for t-tests as well as multicollinearity, independence of errors and normality of residuals for regression analyses - applying parametric methods of analysis can be considered as robust (Carifio & Perla, 2008; Norman, 2010). Welch’s t-test was used to compare mean-score differences between the two experimental groups with regard to tested learning gains, self-reported cognitive and motivational learning gains, as well as cognitive and emotional engagement. Non-parametric Wilcox tests were used to check these results, without major deviations from the parametric counterparts. This is why they are not explicitly reported in the results. To investigate the relationship between engagement and learning gains, Pearson correlations between all variables were conducted. Non-parametric Spearman correlations were used to check the results, also without major deviations from the parametric results. Furthermore, hierarchical regression analysis was employed to test the combined effects on tested knowledge and self-reported cognitive and motivational learning gains. Residuals were checked for multicollinearity, independence of error and normality, and all statistical prerequisites were met. In addition to tests for significance, effect size coefficients will be provided whenever possible. All statistical analyses were executed with R and additional R packages (stats, sjPlot, pwr, lavaan).
Results
Comparing Effects on Treatment Groups
Statistical comparisons showed no group differences in subject knowledge test-score in either the pre-test (t1, d = −.06) nor in the post-test (t2, d = .14) (Table 3). Students showed minor improvements in tested knowledge between the pre-test and post-test, with a small and non-significant effect favoring the serious game group (d = .21).
T-Test Comparing Test Scores of the Serious Game Group and Educational Simulation Group.
Note: scale 0-10
With regard to different types of engagement, students in the group working with the serious game reported significantly more deep thinking than students in educational simulation group. The effect size was small to medium (d = .32). However, with regard to enjoyment (d = - .06), subjective knowledge gain (d = - .03) and increase in interest (d = - .03), there was no difference between the groups (Table 4).
T-Test Comparing Serious Game Group and Educational Simulation Group.
Note: scale 1-5
Interactions Between Any Dependent Variables
To analyze the relationship between engagement variables and learning gains, Pearson correlations were conducted. After applying a Bonferroni correction for multiple comparisons, results indicated that there is no correlation between tested knowledge (both t1 and t2) and any dependent variables. However, self-assessed subjective knowledge gains and increase in interest are positively and significantly correlated. Both deep thinking and enjoyment are also positively correlated with self-assessed subjective knowledge gain. There is also a correlation between deep thinking and enjoyment as well as between deep thinking and group (Table 5).
Intercorrelation Using Pearson-Method.
Note: n = 150; *** = p < .001, * = p < .05. Coding of group (1 = serious game, 0 = educational simulation)
To explore these interrelations further, multiple hierarchical linear regressions were conducted to show the combined influence of treatment-group membership, pretest-scores and deep thinking as well as enjoyment on post-test scores, subjective knowledge gains or increase in interest. While the tested post-test scores were significantly affected by the pretest-scores, no influence of group on deep thinking or enjoyment was found. Table 6 shows the hierarchical regression model, with the full model (step 3) explaining 33% of variance of the tested subject knowledge t2 with prior subject knowledge t1 as the only significant factor.
Hierarchical Linear Regression: Influence of Different Variables on Tested Subject Knowledge t2.
Note: n = 150; *** = p < .001
In contrast, the regression analysis predicting self-reported knowledge gains shows that this aspect was influenced by both deep thinking and enjoyment. Moreover, tested prior knowledge had no influence and, once again, group membership had no effect. In the full model (step 3), deep thinking explains 15% of the variance in self-reported learning gains, and enjoyment an additional 29%. In total, the model explains 45% of the variance (Table 7).
Hierarchical Linear Regression: Influence of Different Variables on Self-Reported Subjective Knowledge Gain.
Note: n = 150; *** = p < .001
A similar pattern can be observed for self-reported increase in interest. Deep thinking and enjoyment are significant predictors, while there is no effect from the group or the pre-test-score. Deep thinking explains 24% of the variance and enjoyment an additional 32%. In total, the final model accounts for 56% of the variance in increase in interest (Table 8).
Hierarchical Linear Regression: Influence of Different Variables on Increase in Interest.
Note: n = 150; *** = p < .001
Summary and Discussion
This article compares learning with a serious game and learning with a non-gamified digital educational simulation, both focusing on the topic of critical information literacy online. The randomized experimental study examines differences when learning with a serious game or an educational simulation with regard to enjoyment and deep thinking, increased interest, and both self-reported and tested cognitive learning gains. Furthermore, the study explores the interplay of enjoyment and deep thinking, on the one hand, and learning outcomes, on the other.
Contrary to the hypotheses, no differences in tested cognitive knowledge gains or in self-perceived cognitive knowledge gains and interest were found when comparing students working with a serious game with those working with an educational simulation (H1a − H1c rejected). Students learning with the serious game reported more deep thinking but equal enjoyment when compared to students learning with an educational simulation (H2a confirmed, H2b rejected). Furthermore, enjoyment and deep thinking correlate with self-reported knowledge gain and increase in interest (H3a-H3d confirmed) but not with tested subject knowledge t2. With regard to the interplay of these variables, it makes no difference whether learning occurs with a serious game or an educational simulation (H3e rejected). Although no negative effects were visible in the serious game condition when compared to the educational simulation, most positive effects were too small to be of any significance.
As almost all the hypotheses with regard to group differences are rejected, results are challenging to explain. Previous research has repeatedly shown that learning with serious games leads to higher levels of enjoyment and deep thinking, ultimately resulting in higher overall motivational gains and learning gains (Clark et al., 2015; Connolly et al., 2012; Ke, 2008b; Landers, 2014). Our study found an increase only in deep thinking.
These differences with regard to deep thinking can possibly be explained as an effect of the indirect controls of the serious game, which were intended to slow down the decision-making process and to promote deep thinking. Apart from this effect, this study seems to support findings like those of Adams, Mayer, MacNamara, Koenig, and Wainess (2012) or Landers and Armstrong (2017), who also found no increases in motivation or learning gains in gamified learning environments.
The most obvious explanation for these missing effects is that the combination of competitive points, challenging controls and playful visuals for the serious game were not sufficiently different from the educational simulation to yield bigger differences with regard to enjoyment, further learning gains and increase in interest. It has been highlighted by different authors that gamification elements should not be superficial add-ons to simulations but instead should be ingrained more deeply in the grammar of the game environment (Granic et al., 2014; Landers, 2014; van Staalduinen & de Freitas, 2011). Game elements must be closely linked to the learning content. One of the reasons emphasized for intentional learning is that objectives, instructional methods and assessment must always be closely linked and coordinated with one another (Anderson & Krathwohl, 2001; van Staalduinen & de Freitas, 2011). From the perspective of multimedia learning, independent additional game elements might distract from the learning content and take up too many extraneous cognitive resources (Mayer, 2014; Paas, Renkl, & Sweller, 2003). Our study seems to support this notion.
Furthermore, these results point out the importance of the educational context. It is probably not only the game elements that are decisive but also how the game is embedded in the lessons (De Jong et al., 1999; Garris et al., 2002; Leutner, 1993; Mayer & Johnson, 2010). Various studies have shown that explanations, instructional support as well as time spent on task have an influence on learning gain and motivation (Landers & Landers, 2015; Leutner, 1993; van Merriänboer, Kirschner, & Kester, 2003; Zhang, Chen, Sun, & Reid, 2004). Future studies should therefore also consider aspects of the educational context more specifically.
In addition, it is important in serious game research to look not only at the effect of individual game elements, but also how they are combined. Add-on studies allow the controlled investigation of effects of individual game elements. However, it is the combination of elements in their coherence that makes an effective serious game (Landers, 2014). Research needs to find answers to questions regarding, on the one hand, the tension between entertaining game elements that promote fun and, on the other, instructional game elements that adequately support the cognitive learning process without disrupting the game and suppressing the fun (Ritterfeld & Weber, 2006). In this study, the combination of game elements might be responsible for an increased extraneous cognitive load, which could be an explanation for the lack of clearer advantages for the serious game in this study (Mayer, 2014). Especially for the development of new effective serious games, it is important that future research investigates which combinations of game elements do not distract too much from learning content but rather promote cognitive and emotional engagement. Such controlled experimental comparisons with variations of combinations of game elements are particularly suitable for this purpose (Phye, Robinson, & Levin, 2005). Nevertheless, there were no negative effects in this study, so this serious game does not seem to impact negatively on enjoyment, interest and learning.
Nevertheless, results need to be interpreted with caution as there are several limitations in the methods used in the study. The absence of visible effects might also be a problem caused by the short duration of experimental treatment. A longer learning period might possibly have yielded higher learning gains and greater differences between the experimental groups. In addition, both the content instruction at the beginning of the treatment and the debriefing were relatively short. Greater emphasis should be placed on instruction and debriefing, because various studies have shown that it is difficult to abstract complex concepts from a serious game alone and the learning process must therefore be supported (Garris et al., 2002; Simons, 1993). Studies on instructional elements or debriefing, e.g. on the effect of pre-training or coaching, have shown that this might increase cognitive capacity to deal with the learning content and reduce extraneous load (De Jong et al., 1999; Leutner, 1993; Mayer & Johnson, 2010).
Various limitations also refer to the measures: Engagement and learning gains were measured by means of questionnaires, in which self-assessments might be biased. Moreover, the brief rating scales focused on highly specific aspects of cognitive engagement (i.e. deep thinking and the application of knowledge) and emotional engagement (i.e. fun and enjoyment), and results might be different when employing different or more comprehensive measures. Furthermore, there were no validated measuring instruments for the survey of primary school students. Therefore, a separate instrument had to be developed. The tested learning gains were measured using a rather short scale, which was adequate for primary school students but did not produce as much variance as we had hoped. A more sophisticated test could possibly show differences and developments in more detail. Finally, we did not include gameplay log-data to monitor the actual interaction with the game. As the study strictly follows a randomized design, the results are reasonably robust to justify our interpretations.
On a more general level, the study could show that there is strong correlation between deep thinking and enjoyment on the one hand, and perceived cognitive learning gain and increase in interest on the other. This also mirrors a fundamental assumption that is often found in the literature (Ladd & Dinella, 2009; Landers, 2014). Although the engagement did not show a correlation with tested learning gains in our study, it seems to be worth the effort to promote cognitive and emotional engagement (Filsecker & Hickey, 2014). In addition, the study was able to contribute in showing which combination of game elements can be used to promote deep thinking. Further studies with different combinations of game characteristics need be carried out to identify successful strategies to promote emotional and cognitive engagement as well as motivational gains and increases in learning gains.
Footnotes
Acknowledgements
We would like to thank the reviewers for their comments on earlier versions of this article. This research has been conducted following the ethical requirements established by the German Educational Research Association.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by Swiss National Science Foundation, project number: 13DPD3_134705.
Author Biographies
Contact:
Contact:
