Abstract
Background. Programming is a challenging and highly important topic for introducing learners to our digital world. Research has shown that motivation plays a key role in programming performance.
Intervention. This research explores the effects of badges on the
Methods. This quasi-experimental study followed a pre-test/post-test design to measure the effects of badges on the
Results. Findings are inconclusive as the quantitative results suggest that badges did not increase
Discussion. We analyse and discuss the results within the context of motivation and the programming discipline, and discuss external factors that may impact the relationship between
Conclusion. Further research is necessary to investigate whether
Background
Programming remains challenging for students with little or no programming experience (Boyer, Phillips, Wallis, Vouk, & Lester, 2009; Gomes, Santos, & Mendes, 2012; Jenkins, 2001; Kunkle & Allen, 2016; Pears et al., 2007; Watson & Li, 2014). In technical higher education, gamification, the use of game design elements for purposes and contexts other than normally expected (Deterding, Dixon, Khaled, & Nacke, 2011), is increasingly being used to motivate and engage students (Iosup & Epema, 2014). Gamification has been employed to teach core programming concepts (Arawjo, Wang, Myers, Andersen, & Guimbretière, 2017), improve programming skills (Barrón-Estrada, Zatarain-Cabada, & Lindor-Valdez, 2017), motivate students to develop specific skills (Rojas-López & Rincón-Flores, 2018) and to engage students (Ibanez, Di-Serio, & Delgado-Kloos, 2014).
Digital badges represent a form of gamification used to motivate and engage learners (Abramovich, Schunn, & Higashi, 2013; Filsecker & Hickey, 2014; O’Donovan, Gain, & Marais, 2013; Pedro, Santos, Aresta, & Almeida, 2015), encourage participation (Boticki, Baksa, Seow, & Looi, 2015; Davis & Singh, 2015; Denny, 2013; O’Donovan et al., 2013), promote self-regulated learning (Cucchiara, Giglio, Persico, & Raffaghelli, 2014), and encourage competition (Preist, Massung, & Coyle, 2014). They are also considered powerful motivators in the online environment (Denny, 2013), however, more evidence is needed to support their value and contribution (Facey-Shaw, Specht, van Rosmalen, Börner, & Bartley-Bryan, 2018a; Falkner & Falkner, 2014) including their motivational effects within different environments and across domains (Abramovich et al., 2013; Denny, 2013), in which subject areas they are best used or how they are to be used (Boticki et al., 2015).
We respond to the call for more evidence by exploring whether badges can contribute to the intrinsic motivation levels of introductory programming students. An intrinsically motivating activity is engaged in for its inherent satisfactions rather than for outcomes such as pressures or rewards (Ryan & Deci, 2000a). Underpinning our study on gamification and motivation is self-determination theory (SDT; Ryan & Deci, 2000b) which provides a framework for examining human motivation through its focus on innate psychological needs (competence, autonomy and relatedness) and the environments fostering or undermining motivation.
A subset of SDT, cognitive evaluation theory (CET), further explores how social conditions supporting an individual’s competence, feeling of autonomy, and the need for relatedness can foster intrinsic motivation, whereas conditions which control behaviour, or challenges that are not optimal and lack connectedness may undermine intrinsic motivation (Ryan & Deci, 2000b). Reid, Paster, and Abramovich (2015) suggest that learner motivation via badges depend on how learners interpret feedback (controlling or informational), their expectancy-values (high or low) and their performance (fewer or more badges), and that while badges may not solve the learner motivation challenge, they can serve to measure learning progress and as assessment under the right conditions. Hakulinen, Auvinen, and Korhonen (2015), who found that achievement badges affected student behaviour positively and motivated students, suggest that badges are most useful when challenges are voluntary and students can turn off badges. Though badges are often perceived as extrinsic rewards (Attali & Arieli-Attali, 2015; Hanus & Fox, 2014), Cavusoglu, Li, and Huang (2015) suggest that earning badges indicate competence, promotes autonomy and fosters connectedness, which motivates greater participation, enhances intrinsic motivations and facilitates the internalization of extrinsic motivations, albeit in an online community environment. In their discussion on the risks of gamification reducing intrinsic motivation, Hakulinen et al. (2015) state that badges could be controlling (e.g. using them for grading) or informational (e.g. using them for optional goals and challenges) depending on context, criteria and individual differences.
L. Hassan (2017) discusses how the motivational affordances of gamification can influence motivation through the provision of both extrinsic rewards, externally provided to change behaviour and intrinsic rewards, which internally influences, for example, autonomy, purpose, mastery and relatedness. In matching motivational mechanisms to gamification elements, Sailer, Hense, Mandl, and Klevers (2013), in a study conducted within a simulated game environment, support the thinking that badges motivate through goal-setting and by their ability to foster feelings of competence. Research also suggests that gamification aligns with motivational processes such as goal-setting, outcome expectations, values and social comparison (Morrison & DiSalvo, 2014). Badges have a social function, can promote social status (Immorlica, Stoddard, & Syrgkanis, 2015), and social engagement and collaboration (Giannetto, Chao, & Fontana, 2013) which can lead to them being more desirable.
Intervention
This paper aims to explore the motivation levels of undergraduate introductory programming students within the context of badges. In our study, we seek to examine motivational levels over time by reporting the results of a four-year study involving students pursuing an a first-year programming course. The study has progressed beyond badges which may have been viewed as controlling to community-building badges with a social function. Research recommends longitudinal studies to fully understand how gamification impacts motivation (Alsawaier, 2018). Pitt, Bell, Onofre, and Davis (2019) conducted a multi-year project to address challenges of badge implementation by connecting learning over the long-term in several contexts. In our first year, a baseline study was carried out without badges. In subsequent years, the course was offered with various versions of digital badge implementation to aid us in understanding the relationship between gamification and motivation with badges.
The article provides an overview of the badge design activities and data collection procedures employed in the study followed by the results and discussion. We then present our conclusion and recommendations for future work. This paper contributes to ongoing empirical research on gamification used in a motivational context within the challenging domain of introductory programming.
Methods
Overview
This quasi-experimental research followed a pre-test/post-test design to measure the effects of badges on the intrinsic motivation levels of university-level introductory programming students over a period of four years. It reports the results of a baseline study without badges and four subsequent offerings of the course with various versions of digital badges. The study sought to address the following question:
To what extent do badges affect the intrinsic motivation of introductory programming students?
The first year Programming 1 course covers concepts ranging from data representation in memory, variables and constants to flowcharts, control structures and arrays. A Moodle-based learning management system (LMS) provided core and supplemental resources, such as course notes, worksheets, videos, useful links and other material to support the learning process.
Sample
Most students in the course pursued the first-year undergraduate computing degree while less than 5% pursued an undergraduate actuarial science degree. Convenience sampling was used due to the structure of the class sessions. Students were invited to participate in the surveys within the first and last two weeks of the semester. The research was approved by the university’s Research Ethics Committee. Ethical procedures regarding voluntary participation and informed consent were observed.
Instruments
The research study employed the Intrinsic Motivation Inventory (IMI; Ryan, 1982), a multidimensional instrument used to assess the subjective experience of participants while conducting an activity. This instrument is often used in experiments related to intrinsic motivation (Amorose & Horn, 2001; Vos, van der Meijden, & Denessen, 2011) and has received strong validity support (McAuley, Duncan, & Tammen, 1989,; Monteiro, Mata, & Peixoto, 2015). In all our studies, this survey was used to assess changes in the intrinsic motivation of introductory programming students.
Survey items were slightly modified for domain and time context. For example, the original item “This activity was fun to do” was changed to “Programming will be fun to do” for the pre-test and “Programming was fun to do” for the post-test. Participants stated their opinion on a 7-point Likert-type scale ranging from 1 (Strongly Disagree) to 7 (Strongly Agree) involving 37 items. Demographic data such as gender, age and course of study, were captured. The baseline study, conducted January to April 2014, acted as a control for measuring the effectiveness of digital badges in later experiments.
Badge design and experiments
Data from four badge experiments was collected from 2015 to 2017. Table 1 captures the number of valid comparative responses used for analysis after dissemination of the pre-test and post-test IMI questionnaires, along with demographic characteristics related to age and gender.
Participants in Experiments.
Badges were initially created using the Open Badge Designer tool 1 and uploaded to Moodle. 2 In later experiments, images from openclipart.org were used to create badges. Students were formally advised of badges via an uploaded document listing badge names, categories, icons, description and criteria for achievement. A clickable Badges link allowed viewing of available badges.
In experiment 1 (Facey-Shaw, Börner, Specht, & Bartley-Bryan, 2015), our badge design focused on three main categories – attendance, participation and flowcharts, given past experience with the course. Weekly attendance badges were manually issued while students’ ability to draw flowcharts and pass a quiz earned them flowchart badges. Participation badges were earned for providing solutions to challenges, viewing course resources, taking quizzes and reflecting on the programming experience. An overall course category was also added, based on badge achievement in the other three categories, leading to 27 badges (Table 2). Each category included an easy-to-attain starting badge and a more difficult master badge (Qin, Rau, & Salvendy, 2010). Badges were not used for grading.
Examples of Badges and Their Categories Used in Exp. 1.
Three badges were hidden from the course and document view to surprise students (de-Marcos, Domínguez, Saenz-de-Navarrete, & Pagés, 2014). Activities to earn badges were not all enabled at once. For example, self-test quizzes and challenges were accessible only after coverage of related topics. Badges were awarded automatically or manually by the main instructor when the criteria were met. During the September semester when most students pursued the course, approximately 5 to 6 instructors tutored 25 students on average in each class whereas lecture sessions hosted over 100 students. During the January semester, 3 instructors taught the course. One researcher of this study was also an instructor for the Programming course. Achievement notification was sent via automatic email or as an LMS message. An example of a student’s badge achievement is found in Figure 1. Badge recipients were encouraged to share their thoughts on badges earned during the first few minutes of the lecture sessions for student awareness, and to encourage the social functions of badges, particularly since students participated in activities within Moodle outside of formal class periods. Experience-sharing involved the lecturer identifying or inviting recent badge awardees to reflect and share their reactions upon badge achievement.
Example of a student’s badge achievements in Exp. 1.
Our badge categories and type changed after the first, second and third experiments. New Moodle features and feedback from experiments informed subsequent changes. For example, in the Participation category in Exp. 1, many students earned the first badge in the pair but not the second. To minimize possible frustration with not earning these badges, an intermediate badge level e.g. Reader, Enthusiastic Reader and Super Reader badges, was introduced in Exp 2. In Exp. 1, students complained that badge notifications via email were not sufficiently described. Descriptive details were subsequently added. Hidden badges previously included to encourage experimentation, such as in Eleftheria et al. (2013), were removed. Previously available attendance badges, also used in O’Donovan et al. (2013), were later removed as student feedback revealed that such badges were not valued (Denny, 2013) due to minimal effort for achievement. A Moodle upgrade allowed students to view peer badges, a feature not previously available.
Table 3 shows how badges evolved over the course of the experiments (Facey-Shaw, Specht, Bartley-Bryan, & van Rosmalen, 2018b). The first two experiments were focused heavily on content-type badges whereas Exp. 3 introduced community-building badges such as Helpful Programmer and Programming Motivator to strengthen networking and social considerations. Some previous participation and course badges were removed. Badges linked to an individual’s competence were maintained throughout the experiments, such as the Challenger or Pseudocode Creator badges to recognize students’ proficiency in problem solving. Challenges were also provided for which students had the option of solving any, or all, to earn the challenger badge. The relatedness subscale, which focuses on interpersonal interactions, was introduced in the IMI Survey in Exp. 3 to capture social elements.
Badges Used in 2015 – 2017 Experiments.
By Exp. 4, course badges were removed to allow students options in activities in which they could participate. This decreased the focus on the number of badges earned and on content-related badges while facilitating exploration on how competition, via a leaderboard, would influence intrinsic motivation. Figure 2 provides an example of the badges used in Exp. 4. The visual appeal of badges, for example, use of colours (Cucchiara et al., 2014) and unique symbols (Börner, Kalz, Ternier, & Specht, 2013) may influence interaction with the badge system. Over time, our badge designs were more colourful and symbolic to appeal to learners (Figures 1 and 2). The leaderboard, with 10 levels to promote achievement and friendly competition was incorporated via the Badge Ladder tool. Students could earn experience points set up with each leaderboard level by viewing various course resources. Other social badges were introduced to reward efforts at helping other students to understand aspects of the content.
Example of badges used in Exp. 4.
Focus groups and interviews
Qualitative data, via focus group sessions, was obtained in Exp. 1 and Exp. 2 to obtain greater understanding of students’ badge experience (Facey-Shaw, Specht, & Bartley-Bryan, 2018c). For Exp. 1, four of 15 invited students showed for the session. Three additional students were available afterwards for interviews. In Exp. 2, two focus group sessions were held. For wider representation of the sample, the first group consisted of 8 students, identified by their instructors as top-performing or high achievers while the second comprised 9 students of mixed achievement levels. Students were asked about their badge experience and how badges shaped their behaviour. Their responses were recorded and transcribed.
Post-badge survey and Moodle activity reports
Table 4 presents a list of questions appended to the post-test survey to capture badge feedback. Similar questions were adapted to later versions of the experiment, for example questions on attendance and flowchart badges were taken out when those badges were removed. Questions such as positive experience with badges and whether badges motivated students to learn programming were consistent throughout the experiments. Moodle activity reports were used to assess the degree of participation in badge-related activities. Reports included number of views on a particular activity, number of badge recipients and grader reports for quizzes.
Survey Items on Badges.
Results
Our study has sought to address the extent to which badges had an effect on the intrinsic motivation of introductory programming students. Table 5 provides a description of the mean score differences for each IMI subscale in the baseline study (Control) and all experiments, as well as the paired T-test results. The results reveal that the mean scores for the self-report interest/enjoyment subscale (p > .05) for all experiments were not statistically significant. The mean differences for the control group were also not significant. Mean score differences for the experiments were significant (p < .05) for the perceived competence, effort/importance, perceived choice (except for Exp. 4) and the value/usefulness subscales, however, the direction of the magnitude suggests lower perceptions of competence, efforts and value. Perceived choice was also statistically significant for the first three experiments but in the negative direction.
Mean Intrinsic Scores and Paired T-Test Results (Confidence Level = 95%).
Overall, an examination of the pre-test scores show slightly higher mean values for the interest/enjoyment, perceived competence, effort/importance and value/usefulness subscales for all but the exp. 4 group with relatively smaller participants. In other words, all pre-test values, with the exception of pressure/tension and to a lesser extent, perceived choice, start relatively high. As a consequence, there is not much space for improvement of these values. We also note that the direction of the control group scores, in almost all cases, is the same as for the experimental groups.
It would appear that our survey results do not confirm the effectiveness of badges as an intrinsic motivational factor, given the non-significance of the scores for the interest/enjoyment subscale for the various experimental groups and the control group. We find it useful, however, to further unpack the results of the mean differences to investigate any patterns that may exist among the various subscales. Figure 3 provides line graphs for each subscale showing the mean of differences across each experiment.
Stacked graph showing means of differences across experiments.
Across the control study and the first two experiments, the interest/enjoyment scale trends downwards but increases slightly by the end of Exp 4. Similar trends exist for the other subscales with an upward trend by the end of the research period. The distances between the graphs for the respective subscales are also interesting. Interest/enjoyment is closely situated with pressure/tension, with similar direction in the graphs except for Exp 2 where the mean difference for interest/enjoyment trended downwards while the mean difference for the pressure/tension recorded an increase.
As indicated above, the directions for the perceived choice, value/usefulness and the perceived competence subscales were fairly similar across the experiments. There was a noticeable dip in the values for the effort/importance subscale for Exp 3. Also, of note is that the greatest distance was between the interest/enjoyment and the effort/importance subscales.
For the focus group results, students generally found the badges motivating, including those of their peers. For example, they reported that “badges showed that you were accomplishing something”, “a good incentive”, and “every time I went onto Moodle, I would look around to see what is new and what I’m able to do now to get a badge”. There was, however, variation in the motivation effects. Very few reported being already motivated, in one instance reporting “already motivated, don’t think I can be motivated more”. Interestingly, one student reported “when I started to not get the badges, it was demotivating”, while another stated that “I think I could have gotten more badges maybe if there were some more rewards attached to them, perhaps percentage grades.
Other qualitative results revealed increased informal interactions among students whose interest in badges sparked after conversations with their peers. Students were not interested in certain badges, such as those given for merely attending classes. A number of recommendations were suggested, including carefully designed competition badges which would enable them to challenge their peers, physical badges which could become a conversation point among their peers and optional badges which would allow students to choose their level of engagement to accomplish their own individual learning goals. Further details of the results of these qualitative findings are available in Facey-Shaw, Specht, & Bartley-Bryan (2018c).
An examination of key motivation-related statements on the post-badge survey (Table 6) indicates fairly good support for the inclusion of badges, based on the mean values and quartile ranges, particularly for the first two experiments.
Motivation-Related Questions on Post-Badge Survey.
a: I feel good about my badge accomplishments
b: I found badges to be useful to me
c: Badges helped me to meet my goals for this programming course
d: Badges enhanced my learning experience in programming
e: Badges motivated me to learn programming
Across experiments, there is a slight increase then decline in mean scores, with a few more persons feeling good about their badge accomplishments than being motivated by badges to learn programming for Exps 1, 2 and 4. We examine possible reasons and implications of these and previous results in the ensuing section.
Discussion
Jenkins (2001) suggests that in the early weeks of students’ university life, their expectations of course demands and attitudes towards assessments will most likely change as the course progresses, even as they learn to become responsible for self-directed programming practice. Given the lead author’s own experience with the course, one explanation for the results could be that students’ initial expectations of introductory programming became more realistic over time as became acquainted with the formal concepts and demands of program design and implementation. Further investigation is needed to examine student expectations and the impact on their motivation. Research shows that introductory programming students vary in their abilities, background and aptitude for learning (Carter et al., 2010). This diversity may also help to explain why some students may be unfamiliar with programming but have high anticipation in addition to others who may have prior experience with programming and yet are highly motivated to pursue the course.
A strong correlation between performance and perceived competence in programming was found by Gomes et al. (2012) who recommended stimulating student awareness in what they already know and are able to do. Badges were designed to represent achievement and give feedback on tasks that students were able to accomplish. Very few students indicated that they were already motivated and did not want to participate in activities leading to badges, however, time management and a heavy workload challenged students who had both school and work responsibilities. Some students also reported being unaware of the badges. This may have led to the statistically significant drop in scores for perceived competence. One also cannot ignore the realization that as programming tasks became progressively more challenging, some students may not have had the confidence in their ability to conduct the programming exercises.
The fairly low values for pressure/tension values appear to be consistent with the expectation that pressure should negatively predict intrinsic motivation. For perceived choice, though there is variation in high and low pre-test scores, the slightly lower values for the post-test scores suggest that programming students may not necessarily feel they are participating in the programming course by choice. This is unsurprising given that the course is a core requirement for their degree. Glover (2013) suggests that simple game elements could engage unmotivated learners but if initial intrinsic motivation is high, then providing extrinsic motivation through rewards could lead to demotivation, as supported in the literature (Hanus & Fox, 2014; Randall, Harrison, & West, 2013).
Despite game mechanics being aligned with learning objectives, Hanus and Fox (2014) found student motivation levels in a gamified course decreased over time when compared to a non-gamified course, suggesting that the novelty of badges and the controlling effects of such rewards could harm motivation. Our qualitative results indicate that once students were aware of the existence of badges, whether from their instructor or their peers, it became a motivator for its achievement though few students stated they were already motivated (Facey-Shaw, Specht, & Bartley-Bryan, 2018c). Badge designers should devise ways to ensure that badges do not make students less motivated. Empowering students to be in control of their own learning is a key element supporting badges as a motivational tool (Davis & Klein, 2015).
Kyewski and Krämer (2018), using an e-learning platform as the study context, posited that their way of awarding badges did not influence motivation and concluded that over time students became less intrinsically motivated. The authors recommend more badge research given several limitations of their study. With the average student in our own study undertaking four or five courses per semester, the combined workload appeared to have limited student participation as the course progressed, as expressed in the focus groups. In the programming course alone, for example, students had to undertake four in-course assessments, in addition to a three-part project and a final exam.
A significant number of our students also work full-time and part-time to support their studies financially which impact on their ability to carry out badge-related activities designed for completion outside of scheduled class times. Also, focus group results suggest peak periods during the semester may not be the best time to expect students to undertake badge activities unless these activities contribute to course grades despite a willingness of students to want to participate. The badge-earning period may need to be carefully paced to allow for engagement in activities without students feeling overburdened. Context such as differences in instructor characteristics and variations among students’ readiness and willingness to engage is also important in gamification implementations (Alsawaier, 2018). These explanations indicate that external factors may impact the gamification-motivation relationship.
Figure 3, which provides a graphical display of the mean differences across all experiments, reveals some interesting relationships among the subscales. Given that each experiment had separate groups of students with different sets of badges, these patterns require further analysis as it is difficult to isolate the effect of badges on the individual subscales.
In programming education, balancing task difficulty with learning support can affect student motivation (Settle, Vihavainen, & Sorva, 2014) and depending on whether students perceive themselves as competent in programming, their lack or presence of programming skills can decrease or increase motivation at different points throughout the course (Carbone, Hurst, Mitchell, & Gunstone, 2009). Settle et al. (2014) also point out that effort is directly associated with motivation and given the constant practice required in programming, motivation will need to be maintained in order for students to achieve success. To help steer motivation in the right direction (Carbone et al., 2009), careful consideration should be given to badge use. Deci, Koestner, and Ryan’s (2001) cognitive evaluation theory which considers rewards as informational, conveying self-determined competence, or controlling, conveying low perceived self-determination is also useful to help us understand motivation. The activities leading to badge awards, such as attending classes, reading course notes, drawing flowcharts, etc. may have been viewed as behaviour-controlling events which did little to enhance intrinsic motivation (Deci, Koestner, & Ryan, 2001) in the early experiments hence their subsequent removal.
The fairly average post-badge survey results indicate there is still the potential for badges to motivate students. We reflect here on related qualitative investigations in which students provided more insightful results and reported that badges were indeed motivating (Facey-Shaw, Specht, & Bartley-Bryan, 2018c). Students felt that badges represented an accomplishment though few considered themselves already motivated and did not necessarily participate in activities leading to badges. Some badges, such as attendance badges, were not considered valuable as students wanted badges that required achievement effort and not those that “everyone can earn.” Some students were motivated through the sharing of other’s achievements in class but did not want to share badges outside the context in which they were earned. Achieving badges appeared to be competitive for some, with students either wanting to earn more badges than their peers or recommending that competitive elements be introduced in the badge systems.
Among the key insights arising from their qualitative feedback is the need to disseminate knowledge on available badges and their earning requirements. This information should be provided early in the semester or within the badge implementation period to encourage greater interaction. Engaging learning activities should be provided and allowing students to choose which badges to earn may serve to address individual motivational requirements, and provide more social interactions which could stimulate greater participation.
The inconclusiveness of this and other similar studies (Arawjo et al., 2017) suggests that finding the right mix of autonomy, competence and self-determination is challenging and may only occur in some students and in some circumstances, given the diverse nature of the programming students. Akasaki et al. (2016) posit that mixed gamification results may be due to individual preferences and personality, and that most gamification approaches are designed for average, rather than diverse people. Designing badge-based instructional environments to achieve motivational success with diverse students appears to be the foremost consideration for instructors and badge designers. Individualized badges within traditional classrooms may prove impractical given the level of effort and resources that would be required (Hanus & Fox, 2014), however, increased motivation has been achieved in studies which incorporate adaptive gamification which provides personalized experiences for students (M. A. Hassan, Habiba, Majeed, & Shoaib, 2019). In short, there is a call for individualized (Kyewski & Krämer, 2018), customized badges (Akasaki et al., 2016), with dynamically adapted badge criteria (Hakulinen et al., 2015) that takes into account individual learner differences (Fanfarelli & McDaniel, 2015).
The role of badge design in enhancing motivation requires further exploration, particularly as it relates to specific gamification elements that can promote intrinsic motivation (Alsawaier, 2018; Hanus & Fox, 2014). Employing effective badge design strategies is recommended (McDaniel & Fanfarelli, 2016; Randall et al., 2013) while keeping in mind learner motivation and goals (Goodyear & Nathan-Roberts, 2017). Undertaking longitudinal studies to fully understand gamification effects on motivation and engagement (Alsawaier, 2018) and investigating how gamification affects different motivational outcomes such as autonomy and psychological need satisfaction could also improve badge system design (Sailer, Hense, Mayr, & Mandl, 2017). Zhou, Chen, Fan, and Ji (2019), who report increased participation with badge use, recommends focusing on design rather than competition for motivating learners. Combining qualitative and quantitative design may help to understand long-term effects (Alsawaier, 2018) and provide a foundation for further analysis.
Technological and implementation issues associated with badge systems can also provide insights into their effectiveness. An in-depth evaluation of the structural features of our badges through the lens of an IT implementation framework (Facey-Shaw, Specht, Bartley-Bryan, & van Rosmalen, 2018b) highlight various enabling and challenging factors with students’ gamification experience. For example, how and when students are notified of badges, whether students can view peer badges, and the features and limitations of the badge platform can impact learner interaction with the badge environment, participation, engagement and overall badge experience. Ironing out technical badge implementation challenges prior to implementation (McDaniel, 2016), and testing and adapting after implementation (Pitt et al., 2019) can help to increase success.
Limitations and Future Research
We acknowledge the small sample size used in the control group which acted as a baseline for the subsequent experiments. Due to the university environment and the design of classes, convenience sampling was used in our experiments. We therefore cannot generalize the results from this study. Despite these limitations, this work provides insights into the use of badges within a programming context. Overall, our study adds to the literature on the contextual use of badges and its effects on intrinsic motivation, in particular since the effects have been examined over an extended period of time. Our study also provides opportunities for further work in the design of an intrinsically motivated learning environment for introductory programming which incorporates gamification as a means for motivating and engaging learners. This could include examining how badges work with small groups (Chou & He, 2017) as well as examining elements of the badge design to examine the effectiveness of particular features. Promoting motivational learning through badge design (Goodyear & Nathan-Roberts, 2017) is also recommended. Future work should also include how intrinsic motivation for a topic develops over time. Without this understanding, it will be difficult to investigate, not just badges, but any kind of intervention which aims to affect intrinsic motivation.
Conclusion
This study was done to examine the effects of digital badges on the intrinsic motivation of university-level introductory programming students. A baseline study without badges was first carried out along with four subsequent studies with badges. Our study is one of few that have examined the effects of badges on intrinsic motivation over an extended period of time, and is even more noteworthy given the focus on introductory programming students for whom motivation to learn programming is a well-known challenge.
Our quantitative findings show statistically significant mean differences between the pre-test and the post-test intrinsic motivational survey for various of the intrinsic motivation subscales for most experiments but in a negative direction. We are, however, cautious against making a causal claim that badges generally reduced intrinsic motivation. Unrealistic expectations at the beginning of the course, workload, progressively challenging tasks and variability in student abilities are possible explanations for these results. In contrast, badge survey and qualitative data indicated that badges received mostly positive support for motivating programming students.
The challenge for us may be sustaining, rather than increasing motivation throughout the semester when students are immersed in the real demands of the course. We agree with others that an adaptive approach which provides individualized learning could provide a challenging, but potentially effective, approach to the use of badges in enhancing intrinsic motivation. The pool of increasing literature in gamification studies, particularly in the well-known challenging area of programming, can serve to benefit from this study as we continue to investigate whether gamification, through the use of digital badges, can foster positive intrinsic motivational results.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Notes
Author Biographies
Contact: lshaw@utech.edu.jm
Contact: m.m.specht@tudelft.nl
Contact: p.vanrosmalen@maastrichtuniversity.nl
Contact: jmbartley@utech.edu.jm