The Effects of Flow,Emotional Engagement,and Motivation on Success in a Gamified Online Learning Environment

Abstract

This study aimed to examine the factors that explain academic success in a gamified online learning environment considering flow, emotional engagement, and motivation. The gamified online learning environment was used by 40 undergraduate students, and the data gathered from them. A hypothetical path model showing the interaction of variables with each other was suggested and tested. The experience of flow and emotional engagement in the gamified learning setting had a highly significant impact on motivation. Furthermore, it was concluded that flow increased academic success through increasing motivation. In line with numerous studies in the literature, motivation was determined to have a positive effect on academic success. In addition, the results show that flow and emotional engagement explained 68% of variance of motivation; flow, emotional engagement, and motivation explained 22% of variance of academic success. It is suggested that subsequent studies should focus on the establishment and testing of models that would help to explain success in gamified settings which should incorporate game elements and player types in the structural model.

Keywords

gamification flow emotional engagement motivation academic success

Introduction

Gamification refers in very general terms to the utilization of video game components in nongame context (Deterding, Dixon, Khaled, & Nacke, 2011; Kapp, 2012). It is applied in many fields such as marketing, business, health, and education. In the educational context, Kocadere and Çağlar (2018, p. 12) described gamification as “an educational approach using game design principles in the learning environment to engender interest and motivation in learners.” To take advantage of the potential power of games, gamification utilizes game elements in the real world, capturing feelings similar to those in games but without leaving the real world (Kocadere & Samur, 2016). Likewise, Plass, Homer, and Kinzer (2015) emphasized that the use of games in learning aim to create a new world and achieve learning outcomes by bringing the learner into the created artificial conflict.

Numerous studies have been carried out examining the effects of gamification in the field of education. Studies focusing on gamification’s effect on the learning environment have emphasized its positive impact on learner performance (Mekler, Brühlmann, Opwis, & Tuch, 2013; O’Donovan, Gain, & Marais, 2013; Su & Cheng, 2015), participation in the learning setting (Barata, Gama, Jorge, & Gonçalves, 2013; Coetzee, Fox, Hearst, & Hartman, 2014; Hew, Huang, Chu, & Chiu, 2016), and collaboration (Moccozet, Tardy, Opprecht, & Leonard, 2013). Furthermore, studies have found that gamified settings result in engagement with the learning environment (Barata et al., 2013; Cheong, Cheong, & Filippou, 2013; O’Donovan et al., 2013; Wang, 2015), motivated learners (Hamzah, Ali, Saman, Yusoff, & Yacob, 2015; Su & Cheng, 2015), and a high sense of flow (Kocadere & Çağlar, 2015; Moccozet et al., 2013; Sillaots, 2014; Simões, Redondo, & Vilas, 2013). However, in addition to these positive results, some studies concluded that gamification does not impact matters such as improved learning outcomes (Coetzee et al., 2014), information recall (Hew et al., 2016), increased participation (Barata et al., 2013), internal motivation, autonomy, and sufficiency. Hanus and Fox (2015) reported that gamification had a largely negative impact on learner motivation and satisfaction. These contradictory results may be related to the design of the gamified learning environment, and various suggestions have been made in the literature as to how to design a gamified setting (Dicheva, Dichev, Agre, & Angelova, 2015; Hanus & Fox, 2015; Hew et al., 2016; Kapp, 2012).

It is noteworthy that, of the examined variables, those that have possible positive effects on success are also significant for learning. Many of the studies in the literature have focused on gamification’s effects on learners in terms of engagement, flow, motivation, performance, participation, and other psychological and behavioral aspects. However, very few studies have examined the interrelation of specific structures or examined how these effects have a share in the success of gamified environments. To fill this gap, the current study aimed to examine the impact hypothetically related variables such as flow, motivation, emotional engagement, and success (Csikszentmihalyi, 1997; Shernoff, Csikszentmihalyi, Schneider, & Shernoff, 2003) have on each other in a gamified online learning setting designed on the basis of suggestions in the literature. It is anticipated that an examination of the effects of these variables on success and identification of the variables that have a greater effect on success would help give direction to the design of learning settings. In this way, educators, researchers, and practitioners who want to use the gamification approach can guide their designs based on the variables that trigger success. Figure 1 visualizes the model developed to explain success in a well-designed gamified online learning environment. Solid lines represent direct effects, and the dashed lines indirect effects.

Figure 1.

A suggestion for a path model explaining effects on academic success in a gamified online learning environment.

Theoretical Background of the Model

Flow

Flow is the psychological condition of losing track of time and external factors due to total immersion in an activity (Csikszentmihalyi, 1997). Four feelings experienced during the sensation of flow are as follows:

An intense level of physical or mental engagement in the performed activity.

Total focus on tasks throughout the performed activity and inability to interact with external factors in the environment.

No feeling of concern or worry about completing the performed activity.

Feeling of time flowing relatively fast and change in the perceived concept of time.

The experience of these sensations of flow is one of the most important factors contributing to motivation (Csikszentmihalyi, 2014, p. 233). As such, the current study tested the hypothesis that “flow has a direct positive impact on motivation” (H2a).

It should be noted that certain conditions must be met to experience the flow and benefit from its positive effects. Csikszentmihalyi (1997, p. 39) reported that people may not experience the feeling of flow even while doing a job they enjoy and that there are various factors that affecting this feeling. The first and most important of these is the matching of the task’s level of difficulty with the individual’s capability. According to Csikszentmihalyi (2014, p. 231), if an individual’s perceived level of difficulty is higher than their perceived ability, they will worry about failure while they will become bored if this perception is reversed. Balance between difficulty and ability is the most important factor necessary to experience the feeling of flow. However, other factors also determine whether flow occurs or not. Open and clear goals about an individual’s role in the task and their ability to obtain prompt feedback throughout the duration of the task are other factors that affect the experience of flow. In his studies, Csikszentmihalyi (1997, p. 67) reached the conclusion that individuals experienced a deeper feeling of flow while playing games.

It is thought that as educational games and other game-like environments determine learners’ appropriate difficulty levels and their included challenges are clearly defined, such environments provide a suitable setting for an intensive feeling of flow (Csikszentmihalyi & Schneider, 2000). Gamified learning settings derive their foundations from the structural factors of the game and are effective with in inducing the feeling of flow (Crisp, 2014). Flow experienced in learning environments remain highly significant as it is known to be effective in developing motivation, cognitive absorption, learning, academic achievement, and creativity (Csikszentmihalyi, 1997). As such, the study also examined the hypothesis that “flow has an indirect positive effect on academic success” (H2b).

In addition to the effects of flow on academic success, flow is related to the attention paid to a particular task and entertainment. The four principle feelings experienced during flow can trigger emotional engagement with the learning environment (Shernoff et al., 2003) which can in turn affect the sensation of flow (Csikszentmihalyi, 1997, 2014). As such, an association between flow and emotional engagement was assumed, and our hypothetical model was formed.

Emotional Engagement

Different researchers and studies have defined emotional engagement using different variables. This term has been defined variously as,

A psychological condition in which care, attention, and effort are spent for the task of learning (Marks, 2000).

The attention paid to the learning environment and the joy taken from this setting (Shernoff et al., 2003).

The quality of the effort exhibited for learning outputs along with the willingness to engage and participate in learning activities (Kuh, 2009).

As understood from the definitions set out in the literature, emotional engagement is a multidimensional concept. As such, Fredricks, Blumenfeld, and Paris (2004) examined engagement as a meta-concept that consists of three dimensions: behavioral, cognitive, and emotional. Behavioral engagement is defined as conformity with in-class rules (Finn, 1993), while cognitive engagement is defined as the mental effort in order to understand the information in the learning environment and improve performance in skills perceived to be difficult by the learner (Newmann, Wehlage, & Lamborn, 1992). Emotional engagement is the collection of emotional reactions displayed by the learner for the learning environment (Skinner & Belmont, 1993). Sensations such as attention, love, excitement, happiness, and joy displayed by the learner for the learning environments are indicators of emotional engagement (Skinner & Belmont, 1993).

The literature shows that learners engaged in the learning environment display a high level of success and social interaction (Carini, Kuh, & Klein, 2006; Fredricks et al., 2004) and a low instance of dropping out (Ream & Rumberger, 2008). As such, educators and researchers should make an effort to prepare a learning environment where learners can actively interact and participate and display cognitive engagement for the setting in which learning takes place, which in turn stimulates learner engagement (Klem & Connel, 2004). Game-based and gamified learning settings are used more frequently in recent years to ensure the engagement of learners in the learning environment (Crisp, 2014).

This study largely focuses on emotional engagement as one of the constituents of engagement with the assumption that components utilized in the gamified learning environment have a greater effect on emotional engagement. It is acknowledged that emotional engagement is an important factor behind motivation and success related to the realization of a task (Shernoff & Hoogstra, 2001). As such, the study examined the hypotheses that “emotional engagement has a direct positive impact on motivation” (H1a) and “emotional engagement has an indirect positive impact on academic success.”

Motivation

Motivation is defined as the taking of an action with the aim of doing or completing something and has a significant effect on success and performance (Ryan & Deci, 2000). One model examining motivation in the literature is Keller’s ARCS model of motivation (1987). This model emphasizes that four conditions (attention, relevance, confidence, and satisfaction) should be provided in order to motivate the individual in the learning environment. According to this model, in order for the individual to be motivated, the following must occur: Their attention must be captured and sustained in the learning environment, and the learning setting should be entertaining and valuable for the individual and meet their needs. In addition, the individual should have confidence that they will successfully complete the tasks and feel satisfaction following success in the learning setting. This model also states that the clear expression of the tasks that need to be completed along with the rewards that would follow will positively affect the individual’s satisfaction in the learning environment (Keller, 2009).

The utilization of games in the learning setting is among a number of strategies used to stimulate motivation in Keller’s ARCS model. It is known that the feelings of flow and emotional engagement evoked in the individual by game and game-like settings positively affects motivation (Csikszentmihalyi, 1997; Shernoff & Hoogstra, 2001). Motivation is one of the most important variables that predicts success (Keller, 1987). As such, the study’s final hypothesis is that “motivation has a direct positive impact on academic success” (H3).

Related Works

Hamari et al. (2016) developed a model to examine the impact of flow, immersion, and engagement on learning in a game-based learning setting. In the model, flow was examined across two different dimensions including challenge and skill, and the impact of these dimensions on engagement and immersion and their association with success was examined. The model was tested via data obtained from 173 participants. The results indicated that engagement has a positive impact on success. Challenges contained within a game had a positive effect on success both directly and via increased engagement. While skill does not directly influence success, it had an indirect effect on success through increasing engagement.

Hew et al. (2016) examined the impact of game mechanics on the cognitive and behavioral engagement of learners. Learners were divided into two groups: The treatment group’s course contained gamified learning activities related to the course activities, while the control group’s nongamified course included course activities only. The treatment group made a more significant contribution to the discussion environment than the control group. However, no significant difference was observed between the two groups with respect to the number of clicks on the lesson materials. Data regarding cognitive engagement indicated that learners in the gamified setting have a higher level of engagement with difficult tasks and performed better quality educational duties.

Su and Cheng (2015) investigated the impact of a gamified learning setting on motivation and success in a study with 102 students enrolled in the fourth grade. Participants were classified into three groups, consisting of one treatment and two controls. The treatment group used a gamified mobile learning system, the first control group learning materials and a guide via a mobile instrument and the second control group learned subject without technology support. Preliminary and posttests indicated that the treatment group exhibited a higher level of success and greater motivation in comparison to the two control groups.

Ibanez, Di-Serio, and Delgado-Kloos (2014) investigated the impact of gamified learning environment on cognitive engagement and success. A moderate improvement was recorded in learning outputs, and positive results were reached in the fields of academic success and engagement. Only 2 of the 22 students dropped out from the lesson without reaching the necessary maximum score. Some students continued to enter the setting despite exceeding the maximum score, which the researchers interpreted as proof of cognitive engagement. Game components such as badges and leaderboards provide entertainment, and a large number of students entered the setting to assist their friends reach the maximum score.

In a study on 43 graduate and postgraduate students, Sillaots (2014) aimed to investigate the impact of gamification on the sensation of flow. Results indicated that the flow sensation experienced by learners in the gamification setting was relatively high and that postgraduates experienced a greater sensation of flow than graduate students.

Simões et al. (2013) emphasized that the sensation of flow is an indicator of engagement and suggested a framework for gamification design using flow theory as a foundation. School students in an online learning setting prepared based on this framework were included in the study. Learners in the gamified social learning settings experienced a higher level of flow sensation in comparison to learners in the nongamified social learning settings. The researchers accepted this finding as an indicator of the higher level of engagement displayed by the treatment group in comparison to the control group.

Denny (2013) examined the impact of a gamified online learning setting on the perception and engagement of learners in their study of 1,031 participants divided into a treatment and a control group. The treatment group won badges in return for the activities they performed in the setting, while the control group performed the same activities but did not win any rewards. The learners in the control group had a significantly higher number of written answers and frequency of entering the learning. This situation indicated that the gamified setting had a positive impact on learner engagement. The learning instrument was found to be significant in enabling learning, and 65% of learners in the treatment group found gamification entertaining.

This section summarized studies examining flow, engagement, and motivation in a gamified learning environment. Researchers have studied the effects of the aforementioned variables on learning in different environments. As was outlined in the theoretical background of the model, these variables are crucial for success. In addition to success, flow affects learners’ motivation, cognitive absorption, learning, and creativity (Csikszentmihalyi, 1997). The other important variable for learning environment is engagement as individuals who are engaged during learning process focus more of their attention on the learning activities. Engagement, with its cognitive, emotional, and behavioral dimensions, affects academic achievement, interaction with peers, and participation (Carini et al., 2006; Fredricks et al., 2004). In addition, motivation is one of the most important predictors of success. These influential variables have also been investigated in gamified environments. Research in gamified learning environments shows that learners are motivated (Su & Cheng, 2015), engaged (Denny, 2013; Hew et al., 2016; Ibanez et al., 2014; Simões et al., 2013), and feel a sense of flow (Sillaots, 2014) in these environments. Hamari et al. (2016) explored the effect of engagement and flow on success in a game-based learning environment. The authors concentrated on the big picture, which is based on a game, and examined the effects of the variables on each other. It is thought important to take into consideration the effect of flow and engagement on motivation and to discover the indirect–direct effects of these three variables on success in a gamified learning environment.

Method

The current study is based on the correlational research method. Within their university course, 40 students (age: 21–24 years) participated in a gamified online learning setting for 13 weeks; 28 of the participants were females and 12 were males. Data were collected using data collection instruments at the end of 13 weeks.

Design Process of the Gamified Online Learning Environment

The gamified learning environment was designed taking into account suggestions from the literature. The design was based on specialist opinion and subsequently on students’ opinions. The following suggestions and highlights from different research and application examples were taken into consideration in the design of the settings:

Different gamification components were integrated into the gamified setting in addition to the utilization of leaderboard and badges (Hanus & Fox, 2015; Kapp, 2012).

A story related to the learning tasks in the gamified setting and also including other components was used (Hanus & Fox, 2015; Hew et al., 2016; Kapp, 2012).

Challenges in the setting were made more complex as the user made progress in the system (Dicheva et al., 2015; Simões et al., 2013).

Learners were provided with the capability to monitor the progression in the setting.

Gamification components earning the learner status in the setting were used, and all learners were asked permission before their status was made visible to all users on the system (Dicheva et al., 2015).

Learners were given the freedom to make mistakes, and no penalties were given (Dicheva et al., 2015; Simões et al., 2013).

Competition and collaboration were used together (Li, Dong, Untch, & Chasteen, 2013; Sillaots, 2014).

Social engagement loops were established in the form of an invitation that would enable the participation of learners in the setting (Zichermann & Cuningham, 2011).

In light of these suggestions, the design process began with an examination of the teaching plan and weekly learning tasks, and the activities and the number of weeks needed for their completion were determined. As the educational task was examined thoroughly, the situations expected from the learners were stated in a clear and open manner as suggested by Werbach and Hunter (2012, p. 87). Subsequently, the most appropriate gaming component and which circumstances meet those aims were identified. Upon deciding the components and how the components will be earned, the story was created taking into consideration the characteristics of the study group and the components. Following the selection of the story, the components and the conditions in which they were assigned were updated accordingly. Details of the content of the visuals representing the story and the gaming components related to the story were decided, and a visual design specialist prepared the necessary visuals. Following these processes, the prepared story, the educational content, educational tasks, and gaming components were integrated into a learning management system (LMS). To determine which LMS was more suited to our purposes, needs, and possibilities, the Talent LMS, Moodle, Blackboard, and Wordpress systems were examined. Wordpress LMS was selected due to its more flexible structure and easy customization according to design choices. After choosing the LMS, all gamification components were integrated into the online learning environment using a Wordpress plugin. Screenshots of the gamified online learning environment are given in Figures 2 and 3.

Figure 2.

Screenshot of the profile page in gamified online learning environment.

Figure 3.

Screenshot of the leaderboard page in gamified online learning environment.

Components of the Gamified Online Learning Environment

A study on the different gamification components and learning tasks that cover these components was utilized. The components of team, gifting, collection, achievement, point, content unlocking, boss fight, level, and surprise were used.

The story focused on a pirate attempting to reach a treasure by passing from one island to another. The learner earned points as they meet educational goals in the setting and collected components that help the pirate find the treasure. The pirate aimed to move from one island to the next and consequently reach the final island to win the treasure. Gamification components such as level, badge, collection, achievement, gifting, and surprise were designed to be integrated into the story. Details regarding the components used in the setting and their use are outlined later.

Point: Points were earned according to the degree to which learners completed the educational tasks and challenges assigned. Tasks carried out by the learners were evaluated by the instructors based on their quality, and scores were assigned accordingly.

Level: The game consisted of 13 levels that the learner must pass through in one class period. To be compliant with the story, levels were visualized as 13 islands that the pirate passes through to reach the treasure.

Content unlocking: Levels were not unlocked until a certain point was reached in the previous level. The required number of points necessary to unlock the next level increased as the levels progressed. Therefore, as suggested in the literature, challenges increased along with the increased progress in the system (Simões et al., 2013).

Achievements: Learners earned achievements compatible with the story as they unlocked each level. For example, when meeting the challenge or learning task (i.e., reading the article of the week and preparing a video summarizing the subject of the week) in the first level, the learner earned points and was able to release the lock on the second level and thus open the content of the second week. As the second level was unlocked, the learner earned the achievement defined for that level, the raft necessary for the pirate to pass to the second island. Further earned components included sailboats, maps, and binoculars relevant to the story that made it easier to pass from one island to the next. In addition, achievements such as artillery, rifle, and swords were provided to help learners with difficult challenge encountered in the setting. Achievements were designed to help the learner so that when they incorrectly answered a question in the boss fight, they were not marked negatively.

Boss fight: While reading an article or doing homework was weekly challenges, learners faced a greater challenge at the end of the term in the form of a gamified exam compatible with the story. A game card (Figure 4) was used in the exam. The gamified exam was planned and designed to align with the story of a pirate utilized in the overall learning process. Four islands corresponded to four levels which were locked until learners were able to unlock them by answering questions correctly to collect points. Achievement items obtained by learners in the online learning environment could be used in the exam to erase incorrect answers as explained in the achievement component. Scores from all levels were added to the score collected in the process and therefore affected the final ranking of the learners in the leaderboard.

Figure 4.

Game card designed for the boss fight.

Badges: The function of badges was to earn status and reputation and were awarded to learners who finished the week with the highest scores, who were ranked among the top three in the leaderboard, or who shared content about the learning environment on social networks. For instance, a badge earned in one level was visualized to show the achievements earned by the pirate in that level.

Collection: Collection items were given to learners who clicked on the supplementary materials found in the setting. These items consisted of different types of jewelry integrated with the story.

Surprise: The surprise element was used to add a factor of chance into the environment and was programmed to appear as Aladdin’s magic lamp, one of the pieces of the treasure, at random depending on the actions of the learner in the system. A learner possessing this component earned the right to choose a question card. If the question was answered correctly, additional points were awarded and incorrect answers did not lose any points.

Leaderboard: The leaderboard was utilized to create competition. Learners’ position on the leaderboard was based on points they earned and they were ranked according to daily, weekly, monthly, and total points. All learners were ranked according to their points on the table.

Gifting: Learners were able to give gifts to each other by sending points to those who were unable to unlock content or by giving them achievements such as artillery and rifles for use in difficult challenges.

All components were integrated into the setting taking into consideration the story. As suggested in the literature, challenge components were designed to became more difficult as the system progressed (Simões et al., 2013). Learning tasks became more complicated and the scores necessary to unlock the content of the next level higher as the levels progressed.

Furthermore, in order to provide the social engagement cycle, learners were able to share notices regarding their presence in the settings on social networks.

Implementation of the Gamified Online Learning Environment

The study setting combined a LMS with gamification components as the online step of a mixed learning environment. The lesson was at the undergraduate level. Materials were presented to the learners in the online step in a face-to-face learning environment, and the various learning tasks were given weekly to the learners via an online learning environment. The aforementioned tasks concentrated on resources and applications regarding the understanding of the lesson’s content.

Learning tasks were used in two different ways: weekly mandatory tasks and optional tasks. Weekly mandatory tasks included activities such as doing weekly homework related to the learning content, taking quizzes, or reading articles. Optional tasks covered those such as reading extra materials embedded in the forum, summarizing them in the forum, writing discussion questions regarding the examined extra materials, or making contributions to other discussion subjects in the forum. Thus, the mandatory tasks that connect learners with the process and motivate those who do not take full responsibility for their learning and the optional tasks that provide a sense of independence for the learners and enable them to progress through the system at their own pace were both integrated into the setting. In addition, a portion of the learning tasks was organized as individual tasks and a part as collaborative tasks to reduce the sense of competition present in the setting. In addition to individual tasks such as reading the given book chapters, participating in discussions, doing the weekly homework and entering the brief gamified exams, learners also participated in tasks such as developing an end of the term project that requires collaboration. Studies in the literature stress the need to be careful about the issue of competition (De-Marcos, Domínguez, Saenz-De-Navarrete, & Pagés, 2014) and show that the use of competition and collaboration together in the gamified learning environment creates more effective results (Li et al., 2013; Sillaots, 2014).

Following the completion of the tasks, learning products were examined by the instructors weekly and graded on the basis of quality. Subsequent to the entry of points into the system, learners were able to unlock the levels in the setting by their points earned, progress to the next level, and earn achievements relating to the story of the gamified setting and exchange points. In addition, the learners were ranked on the leaderboard by points earned. Based on their position on the leaderboard, they were able to win status badges, social network badges, and collection pieces relating to the story based on their activities in the system and surprise components.

Opinions of Experts and Learners on the Design of the Gamified Online Learning Environment

After the gamified online learning environment was designed, the opinions of three experts, one specializing in gaming and story design research and the other two on gamification, were obtained. A form explaining the gamification components and how they were used in the setting was created, and the experts gave feedback by completing the form based on the compatibility of the use of components with the idea of gamification. In accordance with the expert opinion forms, feedback was obtained on the idea of the integration of various components used in the design into the setting and the suitability of the functioning of gamified setting with respect to the nature of gamification. In line with the suggestions made by the experts, changes were made in the following fields throughout the design process:

Nonmandatory elective learning tasks that enable the learner to work in the setting whenever they like and to collect small points were added to the initial plan of granting points through the completion of weekly learning tasks such as doing homework or reading an article. As such, learners with low scores who were unable to progress to the next level were given the ability to collect more points, and the necessary setting was provided to learners wanting to complete a greater number of learning tasks.

Experts suggested that the leaderboard must be used carefully and not in every situation. As opposed to the original plan of listing the top 10 learners, the experts suggested that the top 3 be listed and that all learners should be asked their opinions on the matter.

The difficult challenge component was added to support the story.

Learners were then asked in the first week of the application how many people should be listed on the leaderboard. The majority of learners wanted to see the whole class ranked on the leaderboard. As such, the leaderboard was used without any limitations on the number of people that could be added.

In addition to the leaderboard, learners were asked what the collection pieces could be and which gamification components they would like to see in such a setting. In line with the feedback obtained from learners, the surprise component was added to the setting. It was decided that the collection pieces would be the precious treasure collected by the pirate.

Data Collection

Three different scales were used for data collection on learners’ flow, emotional engagement, and motivation in the gamified online learning environment. Evaluation was then used as a base for data regarding the success variable. Data were collected at the end of the implementation process.

Flow Experience Scale

To determine the level of flow felt by learners in the gamified online learning environment, the Flow Experience Scale developed by Rheinberg, Vollmeyer, and Engeser (2003) and adapted by İşigüzel and Çam (2014) was used. The scale consists of two factors including scale flow experience and concern. The scale is of the 7-point Likert type and consists of a total of 13 items. The first 10 items are related to the flow experience, and the remaining 3 items contain those related to the concern scale. İşigüzel and Çam (2014) took support from three language experts for the scale’s translation validity. Feedback was obtained from 25 students for clarity of the scale items. They reported the adaptation process of the scale in detail in their study. For this study, the scale’s flow experience dimension was used. For the original study group, the internal consistency coefficiency for the scale’s flow experience factors is Cronbach’s alpha = .88. In this study, Cronbach’s alpha of flow experience factor was .86.

Student Engagement Scale

The emotional engagement factor of the Student Engagement Scale developed by Sun and Rueda (2012) and adapted by Ergün and Usluel (2015) was used to determine the emotional engagement of learners in the gamified online learning setting. The scale consists of three factors including behavioral engagement, emotional engagement, and cognitive engagement and contains a total of 19 items. The scale items are a 5-point Likert scale. A 6-item emotional engagement factor was used within the scope of this study. The internal consistency coefficient for the scale’s emotional engagement factor is Cronbach’s alpha = .90. Ergün and Usluel (2015) took support from five field experts for the scale’s translation validity. Feedback was obtained from university students for clarity of the scale items. More detailed information can be found about the reliability and validity of the scale in their study. In this study, Cronbach’s alpha of emotional engagement factor was .89.

Motivation Scale

Motivation was measured using the Motivation Scale–Course Interest Survey developed by Keller (1987) and adapted by Varank (2003). The scale was developed based on the ARCS motivation model and consists of the four principle factors of attention, relevance, confidence, and satisfaction (Keller, 1987). The scale contains 34 items in the 5-point Likert type. For original scale, the Cronbach’s alpha of each factor is as follows: attention: .84, relevance: .84, confidence: .81, and satisfaction: .88. The internal consistency coefficiency of the scale is Cronbach’s alpha = .95. In this study, Cronbach’s alpha of attention factor was .82, relevance factor .86, confidence .90, and satisfaction .92.

Determination of Academic Success

Evaluation carried out by the instructor was used as a base for data regarding academic success variables. A visa exam was carried out within the scope of the lesson, and students were asked to prepare a project in which they could apply the things they learned during the duration of the lesson. Academic success scores were calculated by taking the average of visa and project results.

Data Analysis

Data were analyzed using the path analysis technique which is used to test the causal relationships among three or more variables (Fraenkel & Wallen, 2013). Designed by Çokluk, Sekercioglu, and Büyüköztürk (2010), the process can be used as a grounding in structural equity modeling is summarized in Figure 5.

Figure 5.

Structural equity modeling process (Çokluk et al., 2010).

In this study, the direct and indirect relationships between the variables of flow, emotional engagement, and motivation were defined. Next, the path model that will be tested was determined. Preliminary analysis was carried out based on data derived from the sample, and the suitability of the data set for path analysis was controlled. Suggestions offered by LISREL were evaluated but no modifications were made.

Preliminary Data Analysis

To perform path analysis (Çokluk et al., 2010), assumptions such as the provision of the necessary sample, determination of lost and extreme values if any, normal distribution of data, linearity, and multicollinearity problems should be first tested.

Assumption regarding sample size

Kline (1998) stated that a sufficient number of samples for analysis is 10 times greater than the number of variables placed in the model. Despite the provision of different samples in different resources, the literature confirms the sample size mentioned by (Bryman & Cramer, 2001; Kline, 1998). Thus, a sample size of 40 was considered suitable as the path analysis contained four variables.

Normal distribution assumption

To examine data normality for each variable in the study, the ratio of skewness and kurtosis coefficients to standard error was examined. Values between −2 and 2 are considered to indicate normal distribution (George & Mallery, 2010). Skewness and kurtosis values are given in Table 1.

Assumption regarding multicollinearity

Table 1.

Normality Values Regarding Variables.

Variables	Skewness / standard error	Kurtosis / standard error
Flow	−.419	−.358
Emotional engagement	−1.072	.854
Motivation	−.050	−1.041
Academic success	−.563	−.393

Path analysis requires the absence of strong relationships, or multicollinearity between the independent variables. The variance increase factor (VIF) and tolerance values were examined to determine multicollinearity. A VIF value smaller than 10 and a tolerance value greater than 0.1 indicate multicollinearity (Çokluk et al., 2010). VIF and tolerance values were within the desired range (Table 2).

Table 2.

Tolerance and Variance Increase Factor Values Regarding Independent Variables.

Independent variables	Tolerance	Variance increase factor
Flow	.376	2.658
Emotional engagement	.621	1.609
Motivation	.299	3.346

In addition, the multivariate linearity assumption was tested by examining the scatter diagram of variable pairs, and no lost or extreme values were found in the data set.

Results

To test the suitability of the path model (Figure 1), fit indices were examined. The level of the suitability belonging to the path model was evaluated taking into account the indices suggested in the literature. As such, in order to assess model coherence, χ², χ²/SD, root mean square error of approximation (RMSEA), goodness of fit index (GFI), comparative fit index (CFI), and normed fit index (NFI) were calculated.

The χ² value is a conformity index (Çokluk et al., 2010). To prove a high level of conformance, the p value (significance value) should not be significant and be ideally close to 0 (Hoyle, 1995). However, as this value is affected by sample size to a large extent, it is suggested that the ratio of this value to the standard deviation also be examined (Çokluk et al., 2010). A χ²/SD value lower than 2 for small samples and lower than 3 for large samples signals excellent conformity (Tabachnick & Fidell, 2001).

Another index used in the evaluation of the path model is the RMSEA value. The RMSEA value ranges between 0 and 1 and its approximation to 0 indicates that there is no difference between the covariances of the population and the sample (Çokluk et al., 2010). RMSEA values smaller than .05 indicate the presence of perfect conformity (Schumacker & Lomax, 2004).

GFI, CFI, and NFI values were also examined. Values of .90 or above indicate perfect conformity (Schumacker & Lomax, 2004; Tabachnick & Fidell, 2001). Based on values calculated (Table 3), it was concluded that the created path model seems to exhibit a high level of conformity with the data obtained from the research (for χ² p = .36, χ²/SD = 1.011, RMSEA = .017, GFI = .98, CFI = .99, and NFI = .97).

Table 3.

Conformity Indices Regarding the Path Model.

Conformity indices	Value	Cutoff points for acceptance
χ²	.36	>.05
χ²/SD	1.01	<2
RMSEA	.017	<.05
GFI	.98	>.90
CFI	.99	>.90
NFI	.97	>.90

Note. RMSEA = root mean square error of approximation; GFI = goodness of fit index; CFI = comparative fit index; NFI = normed fit index.

Following the determination of conformity indices, the standardized beta coefficients and t values obtained at the end of the analysis were examined (Figure 6). Emotional engagement had a significant direct positive effect on motivation (β = .64, p < .05). Similarly, flow had a significant direct positive impact on motivation (β = .96, p < .05). When the obtained beta coefficients were taken into account, the flow had a strong effect on motivation. In addition, the flow and emotional engagement explained 68% of the variance in motivation (R²= .68). The direct effect of motivation on academic success was also positive (β = .56, p < .05). The emotional engagement and flow and the motivation variable, all of which had a direct effect on academic success, explained 22% of the variance in academic success (R²= .22).

Figure 6.

The path model that explains academic success in the gamified online learning setting.

The indirect effects of emotional engagement and flow on academic success through motivation were also tested. Flow had a significant indirect positive effect on academic success (=.54, p < .05). However, the indirect impact of emotional engagement on academic success was not significant (=.36, p > .05). Table 4 provides the path coefficients, t values, and standard deviation coefficients for the mentioned direct and indirect effects.

Table 4.

Values Regarding the Pathways in the Path Model.

Dependent variable	Predictor variable	Direct effect	Indirect effect	Standard deviation	t
Motivation	Emotional engagement	0.64*	–	0.3	2.19
Motivation	Flow	0.96*	–	0.15	6.53
Academic success	Motivation	0.56*	–	0.16	3.4
Academic success	Emotional engagement	–	0.36	0.2	–
Academic success	Flow	–	0.54*	0.18	–

*p < .05.

Discussion

Motivation experienced by learners throughout the learning activity period largely stemmed from the sensation of flow and emotional engagement experienced in this setting. Flow was an especially strong predictor of motivation. Furthermore, it was observed that the sensation of flow, engagement, and motivation in the gamified learning environment positively affected the learners’ success. Our hypotheses are summarized in Table 5.

Table 5.

The Hypotheses of the Study and Findings Regarding the Hypotheses.

Hypotheses	Rejection status	Statistical proof
H1a: Emotional engagement has a direct positive effect on motivation	Fail to reject	β = .64, p < .05
H1b: Emotional engagement has an indirect positive effect on academic success	Reject	β = .36, p > .05
H2a: Flow has a direct positive effect on motivation	Fail to reject	β = .96, p < .05
H2b: Flow has an indirect positive effect on motivation	Fail to reject	β = .54, p < .05
H3: Motivation has a direct positive effect on academic success	Fail to reject	β = .56, p < .05

Hypothesis 1a on the positive effect of emotional engagement on motivation in a gamified online learning setting was accepted, and the effect was found to be significant. An individual who enjoys being in an environment feels happy and pays attention to the given tasks and is therefore affectively engaged in the setting works more proactively and intensively for an aim determined by themselves. Although there are no studies in the literature on the effect of emotional engagement on motivation in the gamified learning setting, Shernoff and Hoogstra (2001) emphasized the impact of emotional engagement on motivation. In the current study, that the effect of emotional engagement on success through increased motivation was not significant and thus Hypothesis 1b was rejected. Despite reaching the conclusion that emotional engagement had a positive impact on success through increasing motivation, this impact was not statistically significant. However, Shernoff and Hoogstra (2001) stressed that the emotional engagement experienced in learning settings has an effect on success. It is thought that the design of gamified learning settings will provide emotional engagement and remains important due to its significant effect on motivation.

Flow had a relatively high direct positive impact on motivation, confirming Hypothesis 2a. Csikszentmihalyi (2014) reported that studies carried out since the 1960s have found that flow is one important factor triggering motivation. Throughout the duration of the flow, the individual focuses entirely on the task: External environmental factors do not interact with the individual’s consciousness, and the individual concentrates on the realized task, is not worried, and does not realize how much time has passed. Keller (2009) pointed out that in order for a learner to be motivated in the learning setting in the ARCS motivation model, their attention needs to be captured, the learning environment should meet the needs of learners, and the individual should be self-confident and happy in the learning setting. Full concentration on the realized task by the individual experiencing flow affects the attention dimension defined in the ARCS motivation model, while the pleasure sensation from the realized task and not feeling worried affects the satisfaction dimension defined in the model. As tasks are assigned to suit the individual’s skills in a setting that induces the flow, this affects the trust dimension placed in the motivation model. In addition to the direct impact of flow on motivation, its indirect effect on success was positive and significant (H2b). In a model formed in a game-based learning setting, Hamari et al. (2016) concluded that flow has a significant positive effect on success.

Hypothesis 3 held that motivation has a positive on success in a gamified online learning environment. According to our results, this effect was significant. Motivation is a vital element that explains the success of the learner in a learning environment because motivation helps direct the learner to achieve their goals and encourages them to make the effort until they reach their goals (Keller, 2009). Various studies in the literature indicate that motivation felt in the learning environment positively affects success (Keller, 2009).

Emotional engagement and flow explained 68% of the variance in motivation, while flow, emotional engagement, and motivation explained 22% of the variance in success. These percentages are relatively good values for studies carried out in the field of social science (Field, 2009). As success is affected by numerous variables including individual and environmental factors, it is possible to say that the explanation rate of emotional engagement and motivation in success in the learning environment is relatively good. In the light of these implications, when desiring to trigger motivation, learning environments could be designed based on the principles of flow highlighted by Csikszentmihalyi (1997) or by taking into consideration the indicators of emotional engagement.

Conclusion

This study’s most important result is that the experience of flow in the learning setting and emotional engagement triggers motivation, which is among the most important predictors of success. It is known that gamified learning environments are effective in triggering flow and engagement (Sillaots, 2014; Simões et al., 2013). In general terms, games and similar settings present clear and explicit steps for individuals to reach their aims as levels progress in line with the individual’s skills, challenges increase in difficulty, and immediate feedback is provided. This provides the necessary conditions to enable the experience of flow. When taking into consideration the fact that individuals in the 21st century have greater ability to multitask and prefer dynamic learning environments instead of static learning contexts involving a pencil and paper, games and similar setting appeal to these individuals via their dynamic structure and presentation of complex challenges and thus enable individuals to affectively engage in the setting.

When the effects of flow and emotional engagement on both motivation and success are taken into consideration, it is thought that the environments designed to provide flow and emotional engagement will be effective for learning purposes. To motivate individuals and improve learning, it is suggested that the design of learning settings should be based on flow principles along with the use of elements that would capture the attention of learners based on their characteristics which they would enjoy once the characteristics of the target group has been determined. It is thought that this investigation of factors that enable flow and emotional engagement in gamified and game-based settings and how these change based on the learner type will also contribute to the literature. It is suggested that subsequent studies should focus on the establishment and testing of models that would help to explain success in gamified settings which should incorporate game elements and player types in the structural model. Gamified learning environments can be designed based on the MVP model (Keller, 2017), which explains the basic structures of a psychological environment with motivation and learning. In addition, the effects of these learning environments can be explored.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Hacettepe University Scientific Research Coordination Unit through the project titled “The Design of a Gamified Online Learning Environment and Examination of Its Effects” (SDS-2016-9265).

Şeyma Çağlar Özhan is a research assistant at the Department of Computer Education and Instructional Technology (CEIT) at Bartın University, Turkey. She received her master’s degree from Hacettepe University in 2016 and continues her doctorate education at the CEIT department of Hacettepe University. Her research interests include gamification and social-emotional learning.

Selay Arkün Kocadere is an assistant professor at Hacettepe University in Ankara, Turkey. Following her bachelors’ and master’s degree in Mathematics Education, she received her MSc and PhD in Computer Education and Instructional Technology. Interested in technology-enhanced mathematics education, online learning, and mentoring, Dr. Kocadere’s recent research focuses on gamification and educational games.

References

Barata, G., Gama, S., Jorge, J., & Gonçalves, D. (2013, October). Improving participation and learning with gamification. Proceedings of the First International Conference on Gameful Design, Research, and Applications (pp. 10–17). New York, NY: ACM.

Bryman

Cramer

(2001) Quantitative analysis with SPSS release 10 for windows: A guide for social scientists, London, England: Routledge.

Carini

R. M.

Kuh

G. D.

Klein

S. P.

(2006) Student engagement and student learning: Testing the linkages. Research in Higher Education 47(1): 1–32. doi:10.1007/s11162-005-8150-9.

Cheong, C., Cheong, F., & Filippou, J. (2013, June). Quick quiz: A gamified approach for enhancing learning. Paper presented at Pacific Asia Conference on Information Systems, Jeju Island, Korea.

Coetzee, D., Fox, A., Hearst, M. A., & Hartmann, B. (2014, February). Should your MOOC forum use a reputation system? Paper presented at 17th ACM Conference on Computer Supported Cooperative Work Social Computing (pp. 1176–1187). New York, NY: ACM.

Çokluk

Ö. S.

Sekercioglu

Büyüköztürk

(2010) Sosyal bilimler için çok degiskenli istatistik: SPSS ve LISREL uygulamalari [Multivariate statistics for the social sciences: SPSS and LISREL applications], Ankara, Turkey: Pegem Academy.

Crisp

G. T.

(2014) Assessment in next generation learning spaces. International Perspectives on Higher Education Research 12: 85–100.

Csikszentmihalyi

(1997) Finding flow: The psychology of engagement with everyday life. Choice Reviews Online 35(3): 35–1828.

Csikszentmihalyi

(2014) Flow and the foundations of positive psychology: The collected works of Mihaly Csikszentmihalyi, New York, NY: Springer.

10.

Csikszentmihalyi

Schneider

(2000) Becoming adult: How teenagers prepare for the world of work, New York, NY: Basic Books.

11.

De-Marcos

Domínguez

Saenz-De-Navarrete

Pagés

(2014) An empirical study comparing gamification and social networking on e-learning. Computers and Education 75: 82–91. doi:10.1016/j.compedu.2014.01.012.

12.

Denny, P. (2013, April). The effect of virtual achievements on student engagement. Proceedings of the Conference on Human Factors in Computing Systems—CHI’13 (pp. 763–772). New York, NY: ACM.

13.

Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011, September). From game design elements to gamefulness: Defining gamification. Proceedings of the 15th International Academic MindTrek Conference on Envisioning Future Media Environments - MindTrek ’11 (pp. 9–15). New York, NY: ACM.

14.

Dicheva

Dichev

Agre

Angelova

(2015) Gamification in education: A systematic mapping study. Educational Technology & Society 18(3): 75–88.

15.

Ergün

Usluel

Y. K.

(2015) Çevrimiçi öğrenme ortamlarında öğrenci bağlılık ölçeği’nin Türkçe uyarlaması: Geçerlik ve güvenirlik çalışması [The Turkish adaptation of student s engagements scale in online learning environment: A study of validity and reliability]. Eğitim Teknolojisi Kuram ve Uygulama 5(1): 20–33.

16.

Field

(2009) Discovering statistics using SPSS, London, England: Sage publications.

17.

Finn

J. D.

(1993) School engagement and students at risk, Washington, DC: National Center for Education Statistics.

18.

Fraenkel

J. R.

Wallen

N. E.

(2013) How to design and evaluate research in education. Journal of Chemical Information and Modeling 53(9): 1689–1699.

19.

Fredricks

J. A.

Blumenfeld

P. C.

Paris

A. H.

(2004) School engagement: Potential of the concept, state of the evidence. Review of Educational Research 74(1): 59–109.

20.

George

Mallery

(2010) SPSS for windows step by step: A simple guide and reference, Boston, MA: Pearson.

21.

Hamari

Shernoff

D. J.

Rowe

Coller

Asbell-Clarke

Edwards

(2016) Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior 54: 170–179.

22.

Hamzah

W. M. A. F. W.

Ali

N. H.

Saman

M. Y. M.

Yusoff

M. H.

Yacob

(2015) Influence of gamification on students’ motivation in using e-learning applications based on the motivational design model. International Journal of Emerging Technologies in Learning (iJET) 10(2): 30–34.

23.

Hanus

M. D.

Fox

(2015) Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education 80(2): 152–161.

24.

Hew

K. F.

Huang

Chu

K. W. S.

Chiu

D. K. W.

(2016) Engaging Asian students through game mechanics: Findings from two experiment studies. Computers and Education 92: 221–236.

25.

Hoyle

R. H.

(1995) Structural equation modeling: Concepts, issues, and applications, London, England: Sage Publications.

26.

Ibanez

M.-B.

Di-Serio

Delgado-Kloos

(2014) Gamification for engaging computer science students in learning activities: A case study. IEEE Transactions on Learning Technologies 7(3): 291–301. doi:10.1109/TLT.2014.2329293.

27.

İşigüzel

Çam

(2014) The adaptation of Flow Short Scale to Turkish: A validity and reliability study. Journal of Human Sciences 11(2): 788–801.

28.

Kapp

K. M.

(2012) The gamification of learning and instruction: Game-based methods and strategies for training and education, New York, NY: John Wiley & Sons.

29.

Keller

J. M.

(1987) Development and use of the ARCS model of instructional design. Journal of Instructional Development 10(3): 2–10.

30.

Keller

J. M.

(2009) Motivational design for learning and performance: The ARCS model approach, Tallahassee, FL: Springer Science & Business Media.

31.

Keller

J. M.

(2017) The MVP model: Overview and application. New Directions for Teaching and Learning 2017(152): 13–26.

32.

Klem

A. M.

Connell

J. P.

(2004) Relationships matter: Linking teacher support to student engagement and achievement. Journal of School Health 74(7): 262–273.

33.

Kline

R. B.

(1998) Principles and practice of structural equation modeling, New York, NY: Guilford Press.

34.

Kocadere

S. A.

Çağlar

Ş.

(2015) The design and implementation of a gamified assessment. Journal of E-Learning and Knowledge Society 11(3): 85–99. doi:10.1145/320473.320476.

35.

Kocadere

S. A.

Çağlar

Ş.

(2018) Gamification from player type perspective: A case study. Journal of Educational Technology & Society 21(3): 12–22.

36.

Kocadere

S. A.

Samur

(2016) Oyundan oyunlaştırmaya [Game from Gamification]. In: İşman

Odabaşı

Akkoyunlu

(eds) Eğitim Teknolojileri Okumaları, Ankara, Turkey: TOJET, pp. 397–411. Retrieved from http://www.tojet.net/e-book/eto_2016.pdf.

37.

Kuh

(2009) The national survey of student engagement: Conceptual and empirical foundations. New Directions for Institutional Research 2009(141): 5–20.

38.

Dong

Untch

R. H.

Chasteen

(2013) Engaging computer science students through gamification in an online social network based collaborative learning environment. International Journal of Information and Education Technology 3(1): 72–77.

39.

Marks

H. M.

(2000) Student engagement in ınstructional activity: Patterns in the elementary, middle, and high school years. American Educational Research Journal 37(1): 153–184.

40.

Mekler, E. D., Brühlmann, F., Opwis, K., & Tuch, A. N. (2013, October). Do points, levels and leaderboards harm intrinsic motivation? Proceedings of the First International Conference on Gameful Design, Research, and Applications (pp. 66–73). New York, NY: ACM.

41.

Moccozet, L., Tardy, C., Opprecht, W., & Leonard, M. (2013, September). Gamification-based assessment of group work. Proceedings of the 2013 International Conference on Interactive Collaborative Learning (pp. 171–179). Piscataway, NJ: IEEE.

42.

Newmann

Wehlage

G. G.

Lamborn

S. D.

(1992) The significance and sources of student engagement. In: Newmann

(ed) Student engagement and achievement in American secondary schools, New York, NY: Teachers College Press, pp. 11–39.

43.

O’Donovan, S., Gain, J., & Marais, P. (2013, October). A case study in the gamification of a university-level games development course. Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference (pp. 242–251). New York, NY: ACM.

44.

Plass

J. L.

Homer

B. D.

Kinzer

C. K.

(2015) Foundations of game-based learning. Educational Psychologist 50(4): 258–283.

45.

Ream

R. K.

Rumberger

R. W.

(2008) Student engagement, peer social capital, and school dropout among mexican American and non-latino White students. Sociology of Education 81(2): 109–139.

46.

Rheinberg

Vollmeyer

Engeser

(2003) Measuring flow experience. In: Stiensmeier-Pelster

Rheinberg

(eds) Tests and trends, Göttingen, Germany: Hogrefe, pp. 261–279.

47.

Ryan

R. M.

Deci

E. L.

(2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. The American Psychologist 55(1): 68–78.

48.

Schumacker

R. E.

Lomax

R. G.

(2004) A beginner’s guide to structural equation modeling, Mahwah, NJ: Psychology Press.

49.

Shernoff

D. J.

Csikszentmihalyi

Schneider

Shernoff

E. S.

(2003) Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly 18(2): 158–176. doi:10.1521/scpq.18.2.158.21860.

50.

Shernoff

D. J.

Hoogstra

(2001) Continuing motivation beyond the high school classroom. New Directions for Child and Adolescent Development 93: 73–87. doi:10.1002/cd.26.

51.

Sillaots, M. (2014, October). Achieving flow through gamification in the course of computer games. Paper presented at European Conference on E-Learning, Berlin, Germany.

52.

Simões

Redondo

R. D.

Vilas

A. F.

(2013) A social gamification framework for a K-6 learning platform. Computers in Human Behavior 29(2): 345–353.

53.

Skinner

E. A.

Belmont

M. J.

(1993) Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology 85(4): 571–581.

54.

C. H.

Cheng

C. H.

(2015) A mobile gamification learning system for improving the learning motivation and achievements. Journal of Computer Assisted Learning 31(3): 268–286.

55.

Sun

J. C. Y.

Rueda

(2012) Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education. British Journal of Educational Technology 43(2): 191–204.

56.

Tabachnick

B. G.

Fidell

L. S.

(2001) Using multivariate statistics, Needham Height, MA: Allyn & Bacon.

57.

Varank, I. (2003). The effects of computer training on Turkish teachers’ attitudes toward computers and the effects of computer-supported lessons on Turkish students’ reported motivation to lessons (Unpublished doctoral dissertation). Florida State University, Florida.

58.

Wang

A. I.

(2015) The wear out effect of a game-based student response system. Computers & Education 82(2): 217–227.

59.

Werbach, K., & Hunter, D. (2012). For the win: How game thinking can revolutionize your business. Philadelphia, PA: Wharton Digital Press.

60.

Zichermann

Cunningham

(2011) Gamification by design: Implementing game mechanics in web and mobile apps, Sebastopol, CA: O’Reilly Media, Inc.