Abstract
This article aims to explore gamification tools in services and higher education environments and their role in creating student engagement. The research adopts a qualitative method based on quasi-experimental design. Participants from a higher education institution are exposed to gamification activities during a full semester. Researchers use the sentiment analysis (SA) technique based on a text-mining approach to analyse the data. Findings reveal that participants perceive gamification in service settings as a useful tool. The global SA reveals a positive sentiment about the gamification approach that contributes to increasing participants’ engagement. This study’s novelty arises from quasi-experimental research to measure gamification activities’ impact on students’ engagement, measured through SA of their opinions.
Introduction
This article aims to understand the role of gamification in student engagement in a higher education setting, following recent calls for further research about the role of technology development on stakeholders engagement (Leonidou et al., 2018; Loureiro et al., 2020c; Watson et al., 2018) and customer engagement (Bilro & Loureiro, 2020; Hollebeek et al., 2020). Student engagement may be defined as behavioural and psychological involvement in learning activities (Appleton et al., 2008). The literature already addresses engaged students as an active part in the process of searching, sharing and learning new skills and theories about a topic (e.g., Damnjanovic et al., 2020; Iosup & Epema, 2014), especially when using gamification tools in educational environments (e.g., Faiella & Ricciardi, 2015; Galbis-Córdova et al., 2017; Looyestyn et al., 2017). However, pieces of evidence about how the use of gamification can successfully create engagement in participants of a service environment, such as higher education service, are still scarce (Kayimbaşioǧlu et al., 2016; Martín-Gutiérrez et al., 2017).
The academic discussion on the topic has focused the attention on gamification as a way to increase attachment (Oleksy & Wnuk, 2017), to build co-creation experiences (Nobre & Ferreira, 2017), to foster the intention of engagement and brand attitude (Lim & Puspita, 2020; Yang et al., 2017) or to develop interaction and participation at higher education environments (Piñeiro-Otero & Costa-Sánchez, 2015). Contrariwise, several authors have discussed the subversive ludification of society that gamification may represent (e.g., Fuchs, 2018; Woodcock & Johnson, 2017). The adoption and practice of game-based techniques or gamification in higher educational environments may promote positive behaviour changes, leading to increased motivation, relevance or immersion (Kapp, 2012; Kapp & Coné, 2012). However, gamification’s use as a tool to leverage student engagement in services is still an under-explored topic (e.g., Leclercq et al., 2018; Rosado-Pinto & Loureiro, 2020).
This article adopts a quasi-experimental approach to test the causal effects between an intervention and its outcome (Harris et al., 2006), aiming to answer the following research question: Does gamification tools influence students’ engagement in services, such as higher education settings? Specifically, this research aims to shed light on gamification’s implications in students’ engagement in higher education environments, analysing the data through a sentiment analysis (SA) technique based on a text-mining approach. For that, authors lay hands to stimulus-organism-response (SOR) theoretical framework (Eroglu et al., 2003; Mehrabian & Russell, 1974; Roschk et al., 2017). This research shows gamification’s potential outside online environments and in distinct practical application areas, collecting evidence of its positive impact in service settings. The findings unveil that students understood the experience as an educational element, and only after as entertainment. The findings also show that our participants’ overall experience was positive or very positive and that participants feel engaged by using gamification tools in service settings such as educational environments.
The article’s contribution is twofold. First, the findings add to the existing stakeholder and engagement literature, namely on services and technology transformation topics. Second, the authors claim that services, specifically higher education institutions (HEIs), can adopt new technological approaches such as gamification to improve students’ service (i.e., education). As far as authors are aware, this article is one of the pioneers to further understand students’ engagement process through technological tools such as gamification in an extended period (such as a semester) in an actual/physical environment.
The remainder of this article is structured as follows. The article reviews the relevant literature on gamification in educational settings and student engagement in the next section. Next, the authors present the research methodology and the results of our study. Finally, we discuss the findings and this article’s theoretical and managerial implications, along with research limitations and opportunities for further research.
Theoretical Background
Gamification and Its Use in Educational Settings
The gamification concept is still in its early days and open to a more established definition (Hamari et al., 2014). The Oxford dictionary describes gamification as applying typical game playing elements (such as competition with others, point scoring, among others) to distinct fields (Oxford, 2019). Literature defines gamification as the use of game design elements in non-game contexts (Deterding et al., 2011; Priya & Kalpana, 2014), by incorporating game elements into non-game settings, or by using game mechanics, dynamics and frameworks to encourage specific behaviours (Ip et al., 2011; Pagowsky, 2012; Sheldon, 2012; Stott & Neustaedter, 2013). Additionally, other authors consider gamification more broadly and describe it as the concept of using game-based processes and game thinking to involve people, to motivate actions, to promote learning processes, or to solve problems (Chaudhary, 2010; Kapp, 2012).
In this article, the authors adopt the definition of Deterding et al. (2011, p. 11), who claim that gamification is ‘the use of game design elements in non-game contexts’. It is one of those non-game contexts where gamification techniques are evolving is the educational setting, specifically the Higher Education (e.g., Faghihi et al., 2014; Galbis-Córdova et al., 2017). In this article, we also accept Robson and colleagues’ definition of gamification: ‘The application of lessons from the gaming domain to change behaviours in non-game situations’ (Robson et al., 2015, p. 2). Both definitions point out how to use gamification in different surroundings, such as HEI.
Various studies have observed participants’ behavioural changes in gamified environments (Barata et al., 2017; Koivisto & Hamari, 2014; Turan et al., 2016), with several others reinforcing the positive aspects of gamification’s application at the HEI level (Subhash & Cudney, 2018; Yildirim, 2017). Still, since its beginning, gamification has sparked controversy between academics and practitioners devoted to human-computer interaction (Mahnič, 2014; Woodcock & Johnson, 2017). This controversy is reflected in some literature addressing the topic, putting in evidence that gamification’s effect on motivation or participation is lower than the expectations created by the hype (Broer, 2014; Fuchs, 2018), or not showing a positive relationship between gamification tools usage and improved success (Frost et al., 2015). Although gamification in educational settings is a growing phenomenon, the literature reveals that not sufficient evidence exists to support the long-term benefits of gamification in educational contexts (Dichev & Dicheva, 2017) and that the knowledge about the gamification outcome in an educational context is still scarce (Dichev & Dicheva, 2017; Dicheva et al., 2015). However, the levels of motivation and engagement among individuals joining gamified scenarios seem to increase, leading to improved performance and positive results (Looyestyn et al., 2017).
This study intends to contribute to the ongoing debate shedding light on gamification’s role in students’ engagement. For that, authors lay hands to SOR theoretical framework (Eroglu et al., 2003; Roschk et al., 2017) initially proposed by (Mehrabian & Russell, 1974). The framework in use represents the role of stimuli (i.e., in-class gamification activities) in influencing consumers’ emotional and cognitive states (i.e. the organism), which, in turn, result in approach or avoidance behaviour (i.e. student engagement).
Students’ Engagement in Gamified Environments
The literature already shows the relevance of distinct stakeholders’ engagement, such as consumers, shareholders, providers or students, from various viewpoints (Loureiro et al, 2020c), and engagement between companies and consumers (e.g., Brodie et al., 2011; Hollebeek et al., 2014; Loureiro et al., 2020b; Sprott et al., 2009). The topic of engagement is also discussed and studied in other areas such as psychology, sociology or education (Garczynski et al., 2013; Loureiro et al., 2020a; Morimoto & Friedland, 2013). Literature defines student engagement as a behavioural and psychological involvement in learning activities (Appleton et al., 2008). Student engagement is also seen as a multidimensional construct, comprising three dimensions: cognitive engagement, behavioural engagement and emotional/affective engagement (Fredricks et al., 2004; Jimerson et al., 2003).
First, cognitive engagement can be recognised as expending additional effort to understand multifaceted concepts and/or overcome upscaled expertise (Finn & Zimmer, 2012; Fredricks et al., 2004). Diverse authors evidence the relevance of the cognitive engagement dimension in many online contexts (e.g., Burkey, 2019; Ghosh, 2019; Putman et al., 2012; Zhu, 2006). Cognitive engagement in an online context can be perceived as the attention and effort students apply to interact with peers and tutors in discussions, comments or posts (Bilro et al., 2018; Putman et al., 2012). Moreover, it includes students’ upscale expertise, such as analysing, reviewing or reasoning (Putman et al., 2012; Shukla & Sharma, 2018; Zhu, 2006). Second, behavioural engagement can usually be perceived in technological contexts based on discussions or replies (Amegbe et al., 2017; Bilro & Loureiro, 2020; Leclercq et al., 2018). The literature debates if the number of interactions among peers is a measure of behaviour engagement (Goggins & Xing, 2016) and the linkage between discussions among students and achievements (Ramos & Yudko, 2008). Lastly, the literature defines emotional engagement as the students’ psychological response to academic settings, such as boredom or enjoyment from the learning activities (Finn & Zimmer, 2012). The relationship between tutors and students, the connection between students and their peers, and students’ interests or enjoyment in making part of the online discussions also influences this dimension (Fredricks et al., 2004).
Methods
Qualitative research based on a quasi-experimental design is adopted in this study, as we aim to demonstrate causality between an intervention and its outcome (Harris et al., 2006). To operationalise the study, we conduct a quasi-experimental design without a control group (Shadish et al., 2002) and expose 91 undergraduate students to various gamification moments during a full semester in a specific curricular unit. Participants were all management degree students, ranging from 19 to 25 years old, and gender balance was achieved (57.14% female). We have built and assess a services scenario using a gamification tool, Kahoot!, ‘a game-based learning platform that makes it easy to create, share and play fun learning games or trivia quizzes in minutes. Users can play Kahoot! on any mobile device or computer with an internet connection.’ (Kahoot!, 2019). During the semester, researchers ask participants to use Kahoot! in class, as final modules quizzes, as an assessment tool for other colleagues’ group presentations or others. At no moment during the semester was explained to students why they were using Kahoot!. The goal was explained only at the end of the semester, and students opinions were collected. Researchers collected the data through one single open-ended question made available on Qualtrics, assessing their opinion about the advantages and disadvantages of this type of gamification-based tool: ‘What is your opinion about the use of Kahoot! inside the classroom? Which are the main advantages and disadvantages that you see in using this type of Gamification tools?’ Participants were first asked to provide informed consent, fill out a pre-experiment questionnaire and anonymity was guaranteed. Students access the form during a class through their own devices, answering individually. Researchers also conducted a post-experimental debrief to participants, as we are dealing with experience-based learning activities (Stewart, 1992). From a total of 91 participants, it was possible to collect 73 valid answers after blank responses or other non-suitable answers were eliminated (80.2% success rate). The sample size is considered adequate (Simmons et al., 2013) and in line with the current practise (e.g., Javornik, 2016; Zhao & Patrick Rau, 2020).
To extract, examine and transpose the vast amount of information into valuable knowledge, it became necessary to proceed with a text mining technique (Fan et al., 2006; Költringer & Dickinger, 2015; Srivastava & Sahami, 2009). Researchers performed a SA procedure to understand the participants’ sentiments and attitudes towards the gamification moments. Data and text mining techniques allow researchers to analyse information and process non-structured text to find relevant knowledge decoded into actionable information (Fan et al., 2006; Zhang et al., 2009). The text mining techniques usually include distinct actions such as text clustering, topic extraction, text categorisation, among others (Li & Wu, 2010). For this study, we use the MeaningCloud software, a powerful tool to extract meaningful knowledge from all types of unstructured content, allowing researchers to perform text analysis, text classification or SA (MeaningCloud, 2019b). MeaningCloud SA API uses semantic approaches based on advanced natural language process (NLP) in all aspects of morphology, syntax, semantics and pragmatics (MeaningCloud, 2019b). Distinct research has opted for this software in different scientific domains (e.g., Bilro et al., 2019; Kaur & Chopra, 2016; Martínez et al., 2016; Segura-Bedmar et al., 2015). This tool can analyse a vast amount of data through NLP. NLP aid computers to comprehend the human’s natural language, allowing machines to interpret the relevant elements of the human language sentence and produces an interpretation of the text so it can be analysed (Godbole et al., 2010; Loureiro et al., 2020b; Mostafa, 2013). To assess the validity and reliability of the text mining outcomes, the authors lay the hands into existing literature measuring and comparing MeaningCloud tool results. Previous studies highlight that MeaningCloud presents high validity and reliability when compared with several other SA tools (van Aggelen, 2015). van Aggelen (2015) argues that the operation made by MeaningCloud of decomposing the text provides concurrently valid and consistent SA, as the outcomes agree with other concurring tools, with only minor deviations. Moreover, literature also put in evidence that when compared with other tools, the results obtained by MeaningCloud are very good, presenting a high percentage of correctly classified text, with minor detected errors (Gonzalez-Marron et al., 2017).
Results
The data analysis was made using distinct text-mining procedures. The first procedure is topic analysis, and topic extraction was made from the data on MeaningCloud. Table 1 shows the 10 most mentioned topics discussed by the students in their responses.
Ten Most Mentioned Topics
The main topics in students’ answers are connected to the classes themselves, such as class, student, classroom or subject. These are followed by topics connected to match, competition or interactive. These results are in line with the expected outcome. Therefore, students perceive gamification firstly as an educational tool, and only after that, they mention the facets more connected to the game or the competition itself. Based on these results, it is possible to argue that students understood the experience they have had as an element of education, and only after as an element of entertainment.
The second procedure is to analyse the most mentioned words. To analyse and group the set of words most used, we resort to a WordCloud, a visual illustration of text data, used to portray keyword metadata or visualise the free form of text. For this purpose, HTML5 WordCloud was used. The respondents’ most mentioned word is ‘class’, followed by ‘student’ (see Figure 1).
Other words, such as game, competition, or winner, are also highly ranked. However, as seen in Figure 1, respondents focus their opinion on words connected to education, such as class, students, or subjects. Again, respondents seem to understand that the purpose of this type of tool is related to education rather than entertainment.
Moving on in our text-mining analysis, we perform the third procedure through deep categorisation of the available text. Deep categorisation ‘assigns one or more categories of a predefined taxonomy to different snippets of a text. By applying a powerful semantic rule technology, it provides maximum accuracy in the classification while allowing the fastest and most efficient definition of models’ (MeaningCloud, 2019a). The deep categorisation reveals that the respondents’ main categories are education, educational assessment, technology and computing, video gaming and mobile games. Again, the text categories are in line with the previous analysis, with a clear focus on educational environments (Table 2).
Text Categories
To complement the analysis, researchers use a fourth text-mining procedure as it became relevant to carry out a cluster analysis, to classify our object (in this case, the text corpus) into related groups that are similar to each other (i.e., clusters) (Li & Wu, 2010; Punj & Stewart, 1983). The software creates clusters and attributes a score to each cluster to identify the clusters with higher scores. Due to the large group of clusters that emerged, researchers decide to use only the clusters with higher scores, establishing a cut-off for clusters scoring 150 or higher (see Table 3).
Word Clusters
The clusters with the highest scores are ‘Good tool’ (602.60), ‘Good way’ (602.55) and ‘Paying attention’ (531.20). We can argue that students understand this type of gamification tools as a good approach to adopt in educational environments, namely inside classrooms. Students also highlight that it is a fun way to learn (255.64) and outweighs the disadvantages (245.31). Some students also refer to a more practical aspect of using this type of technology in class: internet Wi-Fi should always be available to not detract from the overall experience (244.80).
Finally, to understand the sentiment level of each participant in the study it is performed a global SA, which intends to map the overall sentiments expressed in the text (i.e., participants’ answers) (Cambria et al., 2013; Liu, 2015). Through the MeaningCloud software, it is possible to analyse the sentiments expressed in each comment or answer and attribute a polarity scale for each one, from Positive + (P+) to Negative + (N+). As seen in Table 4, most answers express positive sentiments (72.60%), and only 8.22 % of the answers reveal negative sentiments.
Sentiment Analysis Results
Based on these results, most participants have a positive sentiment about this practice in higher educational settings. Moreover, less than 10% of respondents revealed negative sentiments towards the experience, and around 20% reveal neutral sentiments. Based on these results, we argue that this group of participants’ overall experience is positive or very positive and that participants feel engaged by using gamification tools in service settings such as educational environments.
Discussion
This study explores if gamification tools influence students’ engagement in services, specifically in Higher Education Settings. The literature already points out some behavioural changes observed in participants of gamified environments, namely online (e.g., Chaudhary, 2010; Koivisto & Hamari, 2014; Turan et al., 2016). With this research, this article contributes to the ongoing discussion by putting in evidence the potential of gamification outside online environments and in distinct areas of practical application, collecting evidence of its possible positive impact in service settings. Moreover, it is possible to realise that students perceive gamification as an educational tool, and only after that as a game or a competition itself. The findings show that students understand the experience as an educational element, and only after as entertainment. Based on the results, it is also possible to argue that students understand this type of gamification tools as a good approach to implement in educational environments, namely inside classrooms.
This study also shed light on the impact of gamification activities on student engagement, which can be seen as a behavioural and psychological involvement in learning activities (Appleton et al., 2008). Our findings show that participants’ overall experience is positive or very positive and feel engaged using gamification tools. Additionally, by resorting to SA, it is possible to find and measure the participants’ general sentiment about the gamification approach to support the learning process. SA is an exciting way to facilitate understanding of how participants feel about something, about their emotional states and their opinions, answers or comments expressed freely, without limitations or restrictions (Altrabsheh et al., 2014; Liu, 2015). Most of our participants’ sentiments expressed (72.60%) were positive, revealing that this type of gamification tool and its direct application can contribute to greater interest and motivation.
Conclusions and Implications
Theoretical Contributions
This article proposes to study how gamification’s technology-based approach could influence students’ engagement with the contents made available in services. Using a gamification tool, we have built and assess a services scenario to collect students’ motivations, interests and engagement in higher education settings. The findings may contribute to theory development and improvement of services and HEI.
First, this article underlines that gamification influences students’ engagement in services, such as higher education settings. When asked, students have reported high levels of engagement behaviours when taking part in the Kahoot! activities. However, the implementation of gamification-based activities within an educational environment requires special attention to ensure the best possible outcome from the whole experience for participants and providers. Following Huang and Soman (2013), caution should be made about the preparatory actions to be taken, such as the definition of objectives, the understanding of the target audience, or identifying the available resources that could support the application of gamification elements.
Second, this study uses a novel approach by applying the S-O-R framework to explain the role of gamification in student engagement in higher education settings, particularly the use of Kahoot! as a gamification tool. The student engagement, grounded in the S-O-R environmental psychology framework, posits how environmental stimuli are perceived and processed, ultimately impacting attitudinal and behavioural responses (Donovan & Rossiter, 1982; Mehrabian & Russell, 1974). Our study empirically extends the SOR framework to the study of students’ engagement. Research employing this framework suggests that technological experiences resulting from controlled stimuli may be critical to engagement (Mosteller & Mathwick, 2014). Psychological engagement, defined as a cognitive and affective commitment to a functional relationship (Mollen & Wilson, 2010), may manifest from the gamification process experienced.
Managerial Implications
From a managerial perspective, students’ engagement in gamification activities shows relevant managerial implications due to the interest and behavioural engagement students reveal in these activities. First, this study aims not to analyse the gamification tool but highlight the importance of this type of gamification-based tool to stimulate participants and increase their interest in the contents made available through an active and engaging offer. Kahoot! is a straightforward and interactive online tool, which can be considered an entertainment tool. Managers should bear in mind that it can be a powerful tool to stimulate participation and interest when used in a service setting. In our study, participants consider it a ‘good tool’ and a ‘good way to pay attention’ and, most of all, a ‘fun way to learn’ which is also an exciting condition to promote participant satisfaction and engagement (Borrás-Gené et al., 2019). Second, practitioners (either college administrations, lecturers or others) may understand and promote this type of tools as a leverage to create participation and engagement in their audiences. Gamification providers, such as firms, brands or HEI, need to master the best way to apply gamification elements in a service setting properly and, above all, to be aware of this technology-based approach advantages or disadvantages.
Limitations and Future Research
This article focuses on gamification tools’ effect on students’ engagement in a service setting such as HEI. Although this research is based on a rigorous method and data analysis, unveiling relevant findings that broaden literature knowledge, is not without its limitations. First, limitations arise from using a quasi-experimental design, which has recognised boundaries (Harris et al., 2006), such as the lack of randomisation or the presence of temporal confounders. Second, while data mining and text mining have many potential benefits and values, there are still some technical limitations to their capabilities. Data and text mining can analyse and assign a comment to one group of ideas (e.g., topic analysis, clusters) based on its content characteristics (word similarities defined in the NLP). However, that information can contain multiple cognitive aspects or contradictory categorisation cues that may result in missing data assigned to specific groups of ideas (Seifert, 2004). Third, using a single gamification-based tool (Kahoot!) can also be perceived as a limitation. Other gamification tools could be used, and/or a mix of tools and methods could contribute in an aggregated way to influence the participant’s engagement process. Since applying a single gamification-based tool has shown positive results in participants’ engagement, it can be relevant to test different approaches and scenarios, seeking to identify additional effective methods that may contribute to participants’ interest and engagement.
Consistent with our research, future researchers may consider building on the S-O-R framework applied to student engagement to investigate the structural paths linking environmental stimuli, psychological processes and engagement-related responses. Moreover, further research can also be performed by comparing distinct types of gamification tools with the same sample to understand who ensures the best learning outcomes. Finally, future research may also use different university courses in distinct cultural contexts to strengthen the findings and consider students’ emotional, cognitive and social dimensions and their role in these gamification educational approaches.
Code Availability
MeaningCloud 3.4.2.0.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The authors received no financial support for the research, authorship and/or publication of this article.