Effect of a Serious Educational Game on Academic and Affective Outcomes for Statistics Instruction

Abstract

The overuse of lecture-based approaches for instruction in university courses may have limited student access to knowledge, particularly the transfer of complex concepts, such as central limit theorem in statistics. This study seeks to contribute to empirical research regarding the effectiveness of serious educational games (SEGs) to increase undergraduates’ conceptual understanding and affective interest in statistics. An experimental design was used to test the efficacy of an SEG, Deadly Distribution, which simulates a real-world context to learn and interact with statistics concepts, compared to traditional notes and homework problems, as supplements to instruction in addition to class lectures. Students who played the game had similar increases in academic growth of conceptual knowledge as students who studied traditional course material. Furthermore, this treatment group had a significant increase in affective outcomes compared to the control group. These findings extend the current literature, which is mixed and sparse, on the effectiveness of SEGs in the undergraduate classroom. In an undergraduate introductory statistics course, an SEG might be an effective substitute for traditional study time of course materials outside of class to increase their affect toward the subject matter and produce similar gains for students who might not otherwise study.

Keywords

serious educational game simulation game statistics instruction experimental design teaching using computer technology

Introduction

The lecture-based approach to undergraduate education has had challenges with the transmission of knowledge to students for decades (Holmes & Gee, 2016). With empirical research to demonstrate the effectiveness of active learning strategies for improving educational outcomes in undergraduate education, some scholars have called to end the “chalk and talk” lecture as the primary instructional strategy for postsecondary education (Freeman et al., 2014; Lambert, 2012; Mazer & Hess, 2017). The discussion of the use of educational gaming in higher education setting has occurred concurrently with the call for increased active learning experiences. However, game-based learning has yet to become mainstream in higher education (Holmes & Gee, 2016). Following a meta-analysis of the effects of serious educational games (SEGs), serious games, and simulations on achievement, cognition, and affective behaviors, Lamb, Annetta, Firestone, and Etopio (2018) call for additional research into the use of these learning technologies in the postsecondary environment after they found significant positive effects of digital games in Grades 6 to 12, but only negligible effects at the university level. Yet, scholars have also demonstrated that playing commercial videos games can increase critical university graduate skills such as adaptability, resourcefulness, and communication skills (Barr, 2017). Overall, the challenges of implementation of SEGs into existing undergraduate curriculum and instruction might be a reason for the continued mixed findings regarding the effectiveness of SEGs in postsecondary settings.

Much of the empirical work demonstrating the effects of digital games on academic and affective outcomes has been centered in the K-12 educational sector (Lamb et al., 2018). However, the higher education sector fundamentally differs from the K-12 sector in two major ways. First, structural barriers to using game-based learning have included the relatively short class meeting times and perceived burden to students of incorporating out-of-class gameplay with course readings (Holmes & Gee, 2016). Second, there are curricular and instructional barriers to effective game use at the postsecondary level such as designing games for increasingly complex content and integrating games in a lecture-dominant instructional approach (Lamb et al., 2018). This study seeks to fill the gaps recently identified in the empirical literature regarding the use of SEG in postsecondary education contexts by evaluating the effects of an SEG on content knowledge and affective behaviors in an undergraduate statistics course. Specifically, this study adds to the general knowledge of the effects of SEGs on student learning and affective outcomes in higher education settings, while also demonstrating the use of SEGs in a specific mathematics course, undergraduate statistics. In current reviews of the use of games and simulations in higher education, simulations—many used in nursing, biology, and business courses—are overrepresented in empirical studies, and SEGs are underrepresented (Vlachopoulos & Makri, 2017). Undergraduate statistics is an appropriate choice for addressing these gaps because these courses contain complex content that have historically been presented using a mixture of lecture and paper–pencil homework activities. These instructional strategies may not be adequate for overcoming student anxieties associated with statistics or for developing an understanding of statistics that allows them to apply concepts effectively (Chance & Rossman, 2006; Mvududu & Kanyongo, 2011). The use of an SEG as an out-of-class practice activity compared to a traditional paper–pencil activity can also demonstrate an implementation strategy for SEGs in courses with complex content requiring out-of-class practice for mastery.

An understanding of statistics is a requirement across many disciplines of study at the college level; however, over the past decade, there has been a call for a focus on statistical education at all levels of education to ensure that citizens are informed in an increasingly quantitative information laden society (Ben-Zvi & Garfield, 2004; Mills, 2002). Statistics education has shifted focus from technical aspects of computation to statistical literacy; this type of literacy is conceptual in nature, contextualized in wider social problems, and requires application of concepts to support or contradict claims (Gal, 2004). There are many difficulties and misconceptions that must be overcome to engage students in this type of understanding and application of statistics. Without effective simulations or models of important statistical concepts, students can develop further misconceptions related to more abstract statistical subjects (Boyle et al., 2014).

Previous research has also documented affective challenges of undergraduate statistics education. Students often view statistics education as boring and difficult leading them to feelings of reduced self-efficacy in the subject. For example, Nichols (2017) found that undergraduate students found less value in statistics after completing a statistical methods course. Statistics anxiety also poses an affective barrier to students’ learning process. Statistics anxiety includes more than math anxiety. It can involve additional factors such as anxiety over interpretation of statistical data, fear of asking for help, and even fear of the statistics instructors (Baloğlu, 2003). Statistics anxiety has been defined as an “anxiety reaction to any situation in which a student is confronted with statistics in any form and at any time” (p. 28) and can be considered an obstacle in students finishing degree paths which require a knowledge of statistics (Onwuegbuzie, DaRos, & Ryan, 1997). This anxiety acts to reinforce a general low interest and motivation in learning statistics which, in turn, increases learning difficulty (Conners, Mccown, & Roskos-Ewoldsen, 1998). As Chance and Rossman (2006) concluded, the abstract nature of statistics, specifically the fact that “so many statistical concepts and methods are based on the issue of what would happen if a random process … were repeated indefinitely” (p. 1), is very hard for students to grasp, even when they are fully engaged and comfortable with the topic.

Some of the concepts that have been identified as particularly challenging for students include the central limit theorem (CLT) and its foundational concepts, sample, population distributions, sampling, and confidence intervals (DelMas, Garfield, & Chance, 1999; Mills, 2002; Watkins, Bargagliotti, & Franklin, 2014).Based on previous research, game-based learning could be an effective means for teaching students regarding introductory abstract statistics concepts including CLT, confidence intervals, and the sampling distribution because this learning is enhanced when students can apply their knowledge to confront faulty ideas or misconceptions in an interactive, applied situation. Chance and Rossman (2006) also concluded that simulation-type serious games could be very effective in helping students understand abstract statistics concepts, assuming they were properly designed to guide students through the concepts and processes. Meanwhile, Mvududu and Kanyongo (2011) found that the use of simulations of real-world examples may help students relate statistical concepts to real-life situations and help lower students’ statistics anxiety, and Hildreth, Robison-Cox, and Schmidt (2018) found that a simulation-based undergraduate statistics performed as well as a “consensus curriculum” based exclusively on the normal distribution. However, the authors suggest that adding a technology component may improve outcomes. A similar conclusion was found by Hagtvedt, Jones, and Jones (2007) related specifically to software simulations.

This experiment was designed to test the effectiveness of a newly developed turn-based strategy game, Deadly Distribution, used to increase student knowledge of CLT and engagement, absorption, and interest in statistics. Deadly Distribution was designed to provide students with an engaging statistics simulation within a real-world context that would allow them to learn and interact with statistics concepts, without the consequences of failure present in traditional methods such as worksheets and tests. This study meets calls to more carefully construct educational serious games particularly for the complex content of postsecondary courses (Lamb et al., 2018; Lane & Tang, 2000; Mills, 2002, 2004), more appropriately demonstrate the application of the CLT and sampling distribution in courses (Watkins et al., 2014) and more rigorously test game-based learning for statistics using experimental designs (Boyle et al., 2014; DelMas et al., 1999; Mills, 2002, 2004; Watkins et al., 2014).

Purpose and Research Questions

This study seeks to build on the previous empirical work that examines the effect of an SEG on learning academic outcomes in an undergraduate-level statistics course and on student affective behaviors including interest in statistical concepts, immersion in studying of statistics, and engagement and enjoyment of SEGs. This study also seeks to extend the research on the use of simulation to teach statistical concepts including CLT, use of sampling distributions, and use of confidence intervals. The intervention is an SEG that engages students in a series of turn-based decisions in which their input results in statistical feedback from the game that is used by the students to make epidemiological decisions; thus, expanding traditional simulations typically used to teach CLT into authentic problem-oriented game play requiring students to apply the statistical concepts in a meaningful way.

We proposed the following research questions:

What is the difference in knowledge of statistical concepts between students who engage in an SEG and students who engage in studying the traditional materials in an undergraduate course?

What is the difference in student interest in statistics following playing an SEG compared to engaging in studying traditional materials in an undergraduate course?

What is the difference in absorption, engagement, and interest of students who play an SEG compared to students who engage in studying traditional materials in an undergraduate course?

To what extent do demographics influence academic outcomes and affective behaviors in undergraduate statistics learning?

Background and Review of the Literature

Impact of Educational Games on Academic Outcomes

With the increasing popularity of using SEGs as an instructional strategy, researchers have begun to elucidate how SEGs contribute to student learning. Game genre has been compared to a pedagogical stance or approach that is frequently described for other types of instructional strategies. A game genre has helped students navigate disciplinary content and achieve specific types of educational outcomes (Foster, 2011; Lamb, 2016; Lameras et al., 2017). Squire (2006) has proposed that game genres can be divided into two major categories: exogenous games that essentially teach through transmission and drill and practice, and endogenous games in which learning happens through experimentation, discovery, and meaning making. Endogenous games have a specific educational value because they offer designed experiences in which students “learn through the grammar of doing and being.” Simulation games fall into the endogenous category of SEGs because they require learners to apply knowledge in a decision-making exercise while immersed in an artificial environment. Learners receive feedback from the simulation game to learn the consequences of their action (Sitzmann, 2011). Researchers have drawn parallels between this type of learning using SEGs and the experiential learning view of cognition due to their ability to allow students to construct meaningful knowledge of an abstract concept and test the abstract concept to complete the learning process (Garris, Ahlers, & Driskell, 2002; Gee, 2003, 2005; Kolb, 2015).

The connection of game design and genre to theories of learning suggest that SEGs should be able to facilitate student learning. Although games have been introduced as a method of instruction, there has been disagreement regarding their impact on student learning, their superiority to traditional teaching methods, and disagreement as to which type of game genre is most effective in achieving learning objectives (Vogel et al., 2006). Review articles have demonstrated that the body of empirical research regarding games is mixed, and this creates difficulty in drawing conclusions regarding the effectiveness of games in improving student learning. Randel, Morris, Wetzel, and Whitehill (1992) also found mixed evidence of student learning; in a review of 68 studies, 38 showed no difference between games and traditional instruction, 27 showed a positive impact compared to traditional instruction, and 3 games favored traditional instruction. More recent literature reviews have also shown mixed evidence related to learning using games in an academic setting (Connolly, Boyle, Macarthur, Hainey, & Boyle, 2012). A primary issue in game research has been that empirical studies vary widely in the type of game investigated, the purpose for which the game was used, and in the designs of the studies themselves, and this has made drawing conclusions using literature review a difficult process (Connolly et al., 2012; Girard, Ecalle, & Magnan, 2013; Randel et al., 1992). Several meta-analytic studies have been conducted to address this issue, and while four of these studies showed significant results favoring games over traditional instruction, these research groups still echo the call for additional empirical work to be conducted (Clark, Tanner-Smith, & Killingsworth, 2016; Lamb et al., 2018; Sitzmann, 2011; Vogel et al., 2006; Wouters, van Nimwegen, van Oostendorp, & van der Spek, 2013). Two empirical studies comparing SEGs to traditional methods of instruction within the subject of mathematics also showed mixed results with regard to student achievement. Kebritchi, Hirumi, and Bai (2010) found that high school students who used an SEG as a mathematics instruction supplement had significantly higher achievement than students who engaged in traditional practice. However, Ke (2008) found that while mathematics achievement increased with the use of an SEG, that achievement was not significantly different from students who engaged in traditional paper-and-pencil practice.

Impact of Educational Games on Affective Behaviors

The introduction of games into the learning environment was initially based on the increasing popularity of video games among school-age students and young adults. Instructors sought to harness the student engagement and enjoyment of games to promote mastery and interest in academic content. The movement to incorporate digital games into the classroom increased, as researchers identified modern middle-, high-, and college-aged students as the “Net Generation”, students who thrive on being connected, receiving near-immediate feedback, and engaging in experiential learning (Oblinger, Oblinger, & Lippincott, 2005). Instructors found themselves in direct competition with a highly engaging medium and were finding it critical to combine time that students were spending playing video games with time they wanted them to spend on academic work (Trotter, 2005; Worley, 2011). Specific to higher education, Lameras et al. (2017) conducted a content analysis of 165 sources and found that motivation and attention were critical conceptual dimensions linking entertainment to learning features in SEGs.

One component of academic motivation is students’ interest in the subject matter that they are studying. Incorporation of games into an instructional program has been previously shown to increase student subject matter interest across grade bands. Ke (2008) found that elementary-aged students’ scores on the Attitudes toward Mathematics Inventory was significantly higher for students who engaged in mathematics digital games compared to students who participated in paper-and-pencil drills. Middle-grade students also showed increased interest in mathematics and science and more positive attitudes toward the subjects when SEGs were included as a part of the instruction (Moreno, Mayer, Spires, & Lester, 2001; Plass et al., 2013). Teachers of high school math students reported that the digital mathematics game suite DimensionM™ changed the students’ state of mind about mathematics, made them want to learn more about math, and diminished their phobia of mathematics (Kebritchi et al., 2010). College-level students also showed improved interest in science and technology subject matter interest with the inclusion of SEGs and technology activities in coursework as compared to traditional coursework in those subjects (Moreno et al., 2001; Vu, Crow, & Frederickson, 2014).

Beyond providing students an opportunity to connect to academic content, games also provide students a way to immerse themselves in narratively rich worlds and take on alternate roles in which they become scientists, doctors, and mathematicians (Barab, Gresalfi, & Ingram-Goble, 2010). Several empirical studies have shown that SEGs have the potential to improve both engagement and enjoyment in academic subjects. Brom et al. (2017) found that masking a chemistry task as a beer brewing challenge in an SEG improved affective-motivational outcomes in undergraduate college students. SEGs have also shown the ability to engage students on cognitive, behavioral, and emotional levels more than traditional instructional methods (Annetta et al., 2009). Barab et al. (2010) concluded that games are a form of transformational play that result in significantly higher levels of both engagement and enjoyment; students aged 9 to 12 years who played multiple curricular units of the educational game Quest Atlantis™ scored almost two standard deviations higher than control group students in engagement measures, and 86% of students who played the game reported that they enjoyed or strongly enjoyed the activity compared to only 22% of students in the control group. The supermajority of the control group students, 95%, reported that they completed class activities compared to only 35% of the game students reporting that they were motivated by grades; 65% of the game students instead reported that they engaged in the activity because they wanted to be doing it.

Changing Context of Undergraduate Statistics Education

This study is situated in a large, public university in the United States. The demographic makeup of the United States has been rapidly changing since the 1990s, particularly with regard to the percentage of people of color in the population. Census estimates project that sometime between 2050 and 2075, all of the groups previously considered as the ethnic “minority” will actually become the majority of the U.S. population. The Educational Testing Service estimates predict that this will be reflected in undergraduate institution enrollments; their prediction is that if undergraduate populations expand, the majority of the expansion will come from students of color (Swail, 2002). An increase in the number of students who speak English as a second language will likely accompany an expansion of Latino and Asian students of color; an increase in this population will have implications for instructional methods and strategies. In addition to a changing racial population, U.S. public universities are also seeing an increasing number of female students. From 2003 to 2013, the number of female undergraduate students in the United States increased by 19%, and the number of male students increased by 22%; however, in 2013, females represented a majority of all undergraduate students at 57% (U.S. Department of Education, 2016). The percentage of degrees awarded to women by statistics departments has also been increasing, going from 38% to 43% over the past decade (Scheaffer & Stansy, 2004).

Student populations of undergraduate institutions in the United States are not just shifting demographically. Students currently entering as undergraduates are part of a generation group known as “millennials,” but are also widely referred to as the “Net Generation” because they have grown up as digital natives (Worley, 2011). Although the “Net Generation” of students may have a comfort level with technology, they are not homogeneous with regard to technology expertise. Many students may not be able to, or even have an interest in, articulating how the technology works, and this can have an impact on their educational experience. Their use and comfort with technology encompasses a wide range from basic users to power users (Kennedy, Judd, Dalgarnot, & Waycott, 2010). In addition, these students view education as a consumer product, often desiring to complete coursework with the minimal amount of effort to obtain a desired mark. Educators of these students will find themselves frequently competing for the students’ attention with activities that students of previous generations had no access to, making engagement in learning activities more critical than it has ever been (McGlynn, 2007).

Method

Research Design

This study follows an experimental design in which participants were randomly assigned to treatment and control groups to calculate an effect as the difference between the outcomes from participants who received treatment and a reasonable counterfactual, an absence of treatment (Shadish, Cook, & Campbell, 2002). All students were administered a pretest prior to the course unit, then received in-class lectures and homework which included additional textbook problems. Toward the end of this unit, students were invited into a lab for the experiment in which the treatment group “studied” with the game, and the control group “studied” with lecture notes from the instructor and previously assigned homework problems. Both groups completed the posttest immediately following the same duration of “study” time. The counterfactual reasoning (Campbell, 1975; Shadish, 1995; Shadish & Cook, 1999; Shadish, Cook, & Campbell, 2002) in this case was a control group which studied existing, traditional undergraduate course materials, lectures notes, and textbook practice, for the same duration as the treatment group game play. The pre- and posttest instrument, which assessed knowledge of the content, was not formatted to mirror game play, lectures, or homework problems and was designed to test conceptual thinking and application. A bank of questions aligned to each objective randomly rotated across pre- and posttests. The survey instrument which measured affective outcomes was only administered to participants as a posttest as items asked them to respond based on their experience with the assigned lab activity. These procedures helped control for any testing bias across groups meaning less error in the analyzed effect which was not already accounted for by random assignment and the timing of the experiment (Campbell, 1957; Shadish, Cook, & Campbell, 2002). This ability to control the influence of extraneous variables on outcomes with strong internal validity leads to the generalizability of study findings to a similar target population (Campbell, 1957; Shadish, Cook, & Campbell, 2002).

Deadly Distribution Game Design

Deadly Distribution is a turn-based strategy game based on actual epidemiology models (Kermack & McKendrick, 1927) and requires students to accomplish authentic statistical analysis to succeed at locating and eradicating deadly diseases infecting the populous of the fictional nation of Kalgana (see Figure 1). Instructional content was collected from a textbook (see Newbold, Carlson, & Thorne, 2013) and from a senior, university faculty expert who has designed and taught a relevant statistics course. This game was designed to help teach complex concepts within an entry level, required undergraduate statistics course which often has a large enrollment of around or more than 100 students.

Figure 1.

Map of Kalgana as main user interface with population, money and outbreak tracking, and menu options.

The game is structured with four levels of progressively higher complexity. The first level acts as a simple introduction to the game’s mechanics and to the statistical tools within the game. The second level begins the introduction of explicit instruction on statistical concepts and their relation to player actions within the game world. This instruction covers random sampling, sample size considerations, standard deviation, and the CLT. In the third level, these concepts are expanded upon with explicit instruction on sample means, margins of error, and confidence intervals. For each of these levels, the player is also given instructional, corrective feedback based on their actions to reinforce the instructional content. The fourth (and most complex) level acts as both a review of the concepts previously introduced and an assessment of the student’s understanding of those concepts in relation to the events depicted within the game world. It lacks the feedback of the previous levels, instead, leaving it up to the player to work their way through the challenges using only the understanding they developed over the course of the game.

In each turn of the game, players can establish population studies to identify and track the infected, distribute medication and vaccines to combat the diseases, and if all else fails, quarantine a region to prevent the diseases from spreading. Using their understanding of statistical concepts, students must analyze the data provided by their studies to determine which treatments to use and when to seal off regions to prevent outbreaks. All of this must be accomplished while balancing their actions against a limited budget that depletes with every action they take. Forcing them to carefully analyze the data they have and to make choices based on that data, rather than merely guessing, also more accurately simulates the limitations that exist on studies in the real world. Figure 2 is a screenshot of study data to assess the need to purchase vaccines for one city, Eldrat, within Kalgana.

Figure 2.

Example data on the estimated number of infected and dead to adjust costs for future studies and vaccines.

Players can set a sample size and confidence interval which will then result in data being collected and presented on the following turn. These data are displayed both as a histogram and as a week-by-week trend showing the estimated number of infected found through the study and any treatments the player may have used. Between these two graphs, the player can judge how fast the disease is spreading and how well their treatments are affecting it, eventually using that knowledge to eradicate the disease in each region.

Participants

The participants were undergraduate students from a 4-year, research intensive institution who were enrolled in two sections of “Elements of Statistics,” an entry level, required statistics course in a department of economics. Both sections were taught by the same instructor during the same semester. Each section of this course typically enrolls more than 100 students every semester and uses a traditional in-class lecture approach. All students from both sections of the course were solicited to participate in the study. In return for their participation in all phases of the study, students would receive extra credit points toward their course grade and entrance into a raffle for a tablet, which yielded a very high participant rate. A total of 218 students consented to participate in the study and were randomly assigned to treatment and control groups. This sample size is sufficient power (.80) to demonstrate results with medium effect sizes for the analyses used in this study (see Cohen, 1992). Past findings for the influence of serious games or serious games using mathematics on student academic and affective outcomes are mixed (Wouters et al., 2013) and are only descriptive with smaller samples for mathematic games (see Connolly et al., 2012). This purposeful sample is the maximum number of students available to maintain the research design for stronger internal validity which increases generalizability to similar populations.

Procedure

During the Spring 2016 semester, a couple of weeks prior to any course instruction on related content, participants were asked to complete a presurvey that contained test items for three learning objectives around the CLT (see Table 1). Next, following the delivery of relevant lectures in class, all students in both the treatment and control groups were assigned specified times to attend a study session in a computer lab on the university campus. Lab sessions were homogeneous for either control group participants or treatment participants.

Table 1.

Descriptions of Learning Objectives Included in Statistics Assessment to Measure Academic Outcomes and Aligned With Regular Course Content and Deadly Distribution Game.

Objective 1
1.1	Explain the purpose of inferential statistics in terms of generalizing from a sample to a population.
1.2	Explain how sample size affects margin of error.
1.3	Recognize the relationship between the Normal distribution and the Central Limit Theorem.
1.4	Apply Central Limit Theorem to solve real-life problems.
Objective 2
2.1	Understand the concepts of estimation, confidence interval, and confidence level.
2.2	Explain how sample size, level of confidence, and standard deviation affect confidence interval length.
Objective 3
3.1	Generate a method for determining sample size for estimating a population mean or proportion.
3.2	Determine the sample size necessary for estimating the population mean within a specified margin of error.
3.3	Balance accuracy of a study (sample size) with the cost of the study.

The treatment group was asked to play a newly built game called Deadly Distribution for a period of 90 minutes. Deadly Distribution, developed by a panel of content and game experts, is a turn-based strategy game in which data from sampling distributions are used to make decisions about how to apply vaccinations, treatments, and quarantines to stop the spread of a disease throughout fictional geographic regions. Students engaged in gameplay for 90 minutes and were observed by project assistants during this time to ensure that participants were engaged in play for the full 90 minutes.

Students assigned to the control group engaged in studying course materials for 90 minutes instead of playing the Deadly Distribution game. Study materials included PowerPoint slides and homework problems provided by the instructor and additional practice problems provided through MyStatLab™ (Pearson Education). MyStatLab™ is an online statistics practice platform that has three primary features: reviews of the textbook chapters, multiple choice conceptual questions that are contextualized in a scenario, and fill-in-the blank computational questions based on simulated data sets. Students can receive instant feedback and access correctly completed example problems and questions in the MyStatLab™ platform. These are materials that the students regularly use as part of their normal study in this course during the timeframe in which CLT was taught. Following the 90-minute lab period, either treatment or control, students received a link via e-mail to a postsurvey. The postsurvey contained both a content knowledge assessment as well as a survey of their experience with their assigned lab activity.

Survey Instruments

The presurvey contained academic questions for three objectives in statistics addressed by the Deadly Distribution game and control group study materials. The knowledge assessment contained items selected by the research team and course instructor to test overall student understanding of content within the unit. The assessment questions were compiled from existing test banks appropriate for the course, content, and study (see Anderson, Sweeney, & Williams, 2014a, 2014b, 2014c; McClave, Benson, & Sincich, 2012a, 2012b). To minimize the likelihood of previous exposure to assessment items, the test banks utilized came from instructor materials aligned to texts that were not utilized in the study participants’ course. Each assessment item was reviewed by the course instructor for content and objective alignment during the design phase of the experiment. The assessment was aligned to objectives to ensure that the full range of the content was tested. Furthermore, in Deadly Distributions, as students advanced in weeks (i.e., levels), each objective was presented with increasing complexity. For this reason, the test results were separated by objective rather than based on psychometric properties. For the purpose of this study, the students’ ability to correctly answer conceptual questions on content aligned with course materials and the treatment was the goal of this instrument. Table 1 details the content covered in each objective which was used to align instructional materials with Deadly Distributions. Students were randomly assigned two of three questions for each subobjective resulting in 8 questions for Objective 1, 4 questions for Objective 2, and 6 questions for Objective 3, for a total of 18 questions related to statistical concepts. In addition, the presurvey included questions related to students’ interest in statistics, their level of participation in this statistics course, and their perceived academic self-efficacy (Table 2).

Table 2.

Factor Loadings for Affective Behaviors (Full Sample Control and Treatment Groups).

Variables	Interest in statistics (α = .919)	Engagement in activity (α = .936)	Absorption in activity (α = .867)
I am now more likely to read articles about statistics.	.729
I am now more likely to search the web for information about statistics.	.779
I am now more likely to pay attention to news stories about statistics.	.806
I am now more likely to watch a television about statistics.	.770
I am now more likely to research the facts concerning statistics.	.853
I would describe this activity as very interesting.		.636
I believe this activity could be of some value to me.		.682
I would be willing to do this activity again.		.684
I believe doing this activity could be beneficial to me.		.846
I think this is an important activity.		.717
I had fun when I was doing this activity.		.831
I enjoyed doing the activity.		.823
Doing the activity excited my curiosity.		.736
Doing the activity made me curious.		.597
Doing the activity aroused by imagination.		.637
While doing the activity, I was able to block out most other distractions.			.775
While doing the activity, I was absorbed in what I was doing.			.653
While doing the activity, I was immersed in the task I was performing.			.703
While doing the activity, my attention did not get diverted very easily.			.715

Table 3.

Descriptives for Variables in Two-Way ANOVA for Statistics Academic Outcomes.

	Control				Intervention
	Min	Max	Mean	SD	Min	Max	Mean	SD
Objective prescores
Objective 1	0.00	5.00	2.816	1.341	0.00	6.00	2.611	1.305
Objective 2	0.00	4.00	1.230	0.936	0.00	3.00	1.073	0.945
Objective 3	0.00	5.00	2.398	1.271	0.00	5.00	2.246	1.235
Objective postscores
Objective 1	0.00	8.00	4.058	1.680	0.00	7.00	3.400	1.395
Objective 2	0.00	4.00	1.460	1.032	0.00	3.00	1.158	0.903
Objective 3	0.00	5.00	3.046	1.329	0.00	6.00	2.621	1.256

Maximum score possible: Objective 1 = 8, Objective 2 = 4, Objective 3 = 6 (Control Group N: Presurvey = 113, Postsurvey: 87; Intervention Group N: Presurvey = 110; Postsurvey = 95).

The postsurvey contained the previously described academic questions as well as questions designed to assess students’ interest in the subject of statistics. The postsurvey also included questions to assess affective behaviors including immersion in the intervention or control activities and engagement and enjoyment of the intervention or control activities (Agarwal & Karahanna, 2000). Factor analysis using principal components analysis with varimax rotation was conducted in order to construct composite variables for the subject matter interest, immersion, and engagement and enjoyment affective behavior constructs (Table 2).

Data Analyses

A series of statistical analyses were used to answer each research question. Prior to the parametric techniques, these data were inspected for assumptions of normality, homogeneity of variance, independence of observations, linearity, and missingness and outliers (see Lopez, Valenzuela, Nussbaum, & Tsai, 2015; Mertler & Vannatta, 2002). First, the difference in content knowledge from pre- to posttest for treatment compared to control was analyzed using a two-way analysis of variance (ANOVA). Separate two-way ANOVAs were applied by objective (see Table 3 for descriptive statistics of objectives) to demonstrate any possible differences in learning gains because of the difficulty of content or a student’s ability to advance to higher levels (i.e., more depth of content and additional objectives) in the game or the ability to study all materials within the lab time provided. Second, three t tests were used to compare the treatment to control group on the posttest survey constructs: interest, engagement, and absorption. Finally, based on the results of the ANOVAs and t tests, using multiple regression, a set of predictors were applied to explain the variance of three dependent variables (three separate models) for the treatment group only: interest, Objective 1, and Objective 3. In this final analysis step, these predictors attempt to explain some of the growth and differences found for the treatment group only (See Table 7 for correlation coefficients of variables included in regression analysis).

Limitations

This study has limitations due to its design, procedures, and analysis. This study was designed to have a single comparison group which we argue as a control or reasonable counterfactual given traditional instructional approaches and materials provided in undergraduate entry level statistics courses. However, this design could be improved with alternative comparison groups as well as longitudinal time points or more repeated measures (see Shadish, Cook, & Campbell, 2002). There were no pretest measures of affective outcomes. Furthermore, the pretest was administered prior to in-class lectures so that the pre- to posttest growth included the entire unit rather than immediately prior and after the lab activity. In essence, the content assessment growth contains a teaching effect, possible out of class study effect, in addition to the lab activity. While the difference between a randomly assigned control and treatment group may account for this variance from these added forms of learning growth, we purposely included the full timing of the unit to have a more comprehensive and authentic measure of student progress toward content mastery. The content assessment needs further tests of psychometric properties which was not possible in this experiment or its pilot (see Embretson & Reise, 2000). The content assessment was compiled from existing test items which were well aligned with both course materials and game to match the purpose and intended use within this study. The representativeness of the sample is unknown compared to similar courses and students at 4-year, research intensive institutions. Yet, the strength of the internal validity allows for generalization to students from the target population (Campbell, 1957; Shadish, Cook, & Campbell, 2002).

Results

Research Question 1: What Is the Difference in Knowledge of Statistical Concepts Between Students Who Engage in an SEG and Students Who Engage in Studying the Traditional Materials in an Undergraduate Course?

Two-way ANOVA was conducted to compare pre- and posttest scores between students assigned to the deadly distribution intervention group and students assigned to the control group. This analysis showed that there were near significant differences between intervention and control group students in overall knowledge of statistical concepts as measured by total score on the statistics questions, F(1, 180) = 3.171, p < .10, partial η²= .017. Students in the control group had a larger mean increase from pre- to posttest compared to the intervention group. Bonferroni-adjusted post hoc tests showed that both the intervention group—mean difference = 1.295, F(1, 180) = 20.321, partial η²= .101, p < .001—and control group—mean difference = 2.034, F(1, 180) = 45.950, partial η²= .101, p < .001—had significant increases in overall knowledge. There were also near significant differences between intervention and control group students in Objective 1, applying knowledge of sampling distributions to real-world problems, F(1, 180) = 3.280, p < .10, partial η²= .018. Bonferroni-adjusted post hoc tests showed that both the intervention group—mean difference = .821, F(1, 180) = 19.323, partial η²= .097, p < .001—and control group—mean difference = 1.310, F(1, 180) = 45.071, partial η²= .200, p < .001—had significant increases in knowledge of Objective 1 (Figure 3), with control group students having the larger pre- to posttest increase. There were no significant differences between control group students and intervention group students in knowledge of Objective 2 (Figure 4), and neither group had significant pre- to posttest score changes in this objective. There were no significant between-group differences in knowledge of Objective 3 (Figure 5); however, Bonferroni-adjusted post hoc tests showed that both the intervention group—mean difference = .379, F(1, 180) = 7.495, partial η²= .40, p < .05—and control group—mean difference = .575, F(1, 180) = 15.788, partial η²= .081, p < .001—had significant increases in knowledge of Objective 3.

Figure 3.

Pre- and post-mean scores for treatment and control groups for Objective 1, understanding sample, sample size, and normal distribution related to central limit theorem. Error bars are equal to the standard deviation of each group. Control group pretest 95% CI = 2.47 to 3.02 and posttest 95% CI = 7.99 to 9.14. Deadly distributions group pretest 95% CI = 2.32 to 2.84 and posttest 95% CI = 3.09 to 3.71.

Figure 4.

Pre- and post-mean scores for treatment and control groups for Objective 2, understanding confidence intervals (CIs), with no significant growth from pre to post for either group. Error bars are equal to the standard deviation of each group. Control group pretest 95% CI = 1.11 to 1.51 and posttest 95% CI = 1.26 to 1.66. Deadly distributions group pretest 95% CI = 0.87 to 1.26 and posttest 95% CI = 0.96 to 1.35.

Figure 5.

Pre- and post-mean scores for treatment and control groups for Objective 3, estimating and applying sample sizes, with significant growth from pre to post for both groups but no difference in growth between groups. Error bars are equal to the standard deviation of each group. Control group pretest 95% CI = 2.20 to 2.74 and posttest 95% CI = 2.77 to 3.32. Deadly distributions group pretest 95% CI = 1.98 to 2.50 and posttest 95% CI = 2.36 to 2.88.

Research Questions 2 and 3: What Is the Difference in Student Interest in Statistics Following Playing an SEG Compared to Engaging in Studying Traditional Materials in an Undergraduate Course? What Is the Difference in Absorption, Engagement, and Interest of Students Who Play an SEG Compared to Students Who Engage in Studying Traditional Materials in an Undergraduate Course?

T tests were used to determine differences between treatment and control groups in their affective behaviors or perceptions about their assigned supplemental activity and interest in statistics (see Table 4 for descriptive statistics of affective variables). Students who participated in the game reported that they were significantly more absorbed (mean difference = 1.067, t = 5.803, p < .001) and engaged in the activity (mean difference = .916, t = 5.054, p < .001) as well as interest in statistics (mean difference = .435, t = 2.072, p < .05) (Figure 6). Although the results of research questions one on the academic benefit of the game over traditional studying were mixed, students in the treatment group were consistently higher in their affect toward the game as an instructional activity as well as overall interest in statistics.

Figure 6.

Postsurvey increased affective behaviors for treatment compared to control group. Error bars are equal to the standard deviation of each group. Absorption in activity 95% CI of the mean difference between control and Deadly Distributions groups = −1.43 to −0.70. Engagement in activity 95% CI of the mean difference between control and deadly distributions groups = −1.27 to −0.56. Interest in statistics 95% CI of the mean difference between control and deadly distributions groups = −0.85 to −0.02.

Research Question 4: To What Extent Do Demographics Influence Academic Outcomes and Affective Behaviors in Undergraduate Statistics Learning?

Finally, in addition to the effectiveness of the game compared to traditional study methods, we hypothesized that the game design could influence affective and academic outcomes when learning more complex concepts in statistics. Study variables and game play data were regressed on academic outcomes and interest in statistics using only the treatment sample. Academic outcomes were reduced to Objectives 1 and 3 because they showed student growth. In addition, interest in statistics was used as a final affective outcome for students’ increases in engagement and absorption in the game as an instructional activity. Three separate models were analyzed for each of these dependent variables (see Table 5 for descriptives and Table 6 for results). For student responses about interest in statistics, females, and seniors compared to sophomores were less likely to have increased interest. However, students with a first language other than English were more likely to have greater interest compared to native English speakers in this course. Success on Mission/Level 3 compared to Mission 1 had a negative influence on their interest in statistics. Other than Mission 3 success, the average minutes and number of attempts in the game were possible metrics of success in their application of statistics and predicted their overall interest in the content. For the academic outcomes, who completed Mission 4 compared to Mission 1, were more likely to score higher on Objective 1. For Objective 3, absorption in the game positively predicted performance. Prior performance and affective behaviors were the largest predictors for Objective 3.

Table 4.

Descriptives for Affective Variables in ANOVA.

	Control				Intervention
	Min	Max	Mean	SD	Min	Max	Mean	SD
Interest in statistics	1.00	6.00	3.38	1.31	1.00	7.00	3.81	1.52
Engagement in activity	1.00	6.00	3.68	1.16	1.00	3.00	4.60	1.29
Absorption in activity	1.00	6.50	4.29	1.27	1.00	6.00	5.36	1.29

Note. Scale 1 = Strongly Disagree, 7 = Strongly Agree (Control Group N = 87; Intervention Group N = 95). SD = standard deviation.

Table 5.

Descriptives for Variables in Multiple Regression for Deadly Distribution Sample (N = 94).

	Min	Max	Mean	SD
Objective prescores
Objective 1	0	6	2.61	1.31
Objective 3	0	5	2.25	1.24
Student background
Class participation	0	2	1.36	0.73
More than one statistics course	0	1	0.05	0.23
Female	0	1	0.41	0.49
Freshman	0	1	0.14	0.35
Junior	0	1	0.13	0.33
Senior	0	1	0.01	0.10
First language not English	0	1	0.13	0.33
Game mission success
Mission 2	0	1	0.64	0.48
Mission 3	0	1	0.49	0.50
Mission 4	0	1	0.12	0.32
Average attempts per mission	1.00	4.50	1.52	0.64
Average minutes per attempt	2.78	45.00	15.34	7.08
Affective behaviors
Engagement in activity (game)	−2.80	1.87	0.00	1.00
Absorption in activity (game)	−3.63	1.37	0.00	1.00
Engage × Absorb	−2.24	2.24	0.00	1.00
Technology use
Identify as gamer	−1.94	2.10	0.00	1.00
Identify as computer user	−2.90	1.96	0.00	1.00
Dependent variables
Interest in statistics	1	7	3.81	1.52
Postscore Objective 1	0	6	3.40	1.39
Postscore Objective 3	0	6	2.62	1.26

Note. SD = standard deviation.

Table 6.

Results from Multiple Regression of Deadly Distribution Sample (N = 94) on Interest in Statistics and Learning Objectives 1 and 3.

	Model 1 Interest in statistics		Model 2 Postscore Objective 1		Model 3 Postscore Objective 3
Variables	β	t	β	t	β	t
Intercept		5.346***		1.267		.024
Objective prescores
Objective 1	.062	.748	−.021	−.170	−.115	−1.040
Objective 3	−.016	−.210	.167	1.389	.485	4.605***
Student background
Class participation	.116	1.488	.064	.533	−.076	−.724
More than one statistics course	−.113	−1.482	.026	.222	−.016	−.149
Female	−.233	−2.398**	.181	1.212	.222	1.740∼
Freshman	.008	.100	−.031	−.254	−.066	−.611
Junior	.060	.770	.115	.967	−.099	−.969
Senior	−.161	−2.229*	−.020	−.179	.109	1.108
First language not English	.222	2.897**	−.047	−.398	.043	.438
Game mission success
Mission 2	−.012	−.115	.208	1.303	.258	1.813∼
Mission 3	−.194	−1.979*	−.114	−.759	.027	.205
Mission 4	.002	.020	.253	2.110**	.027	.205
Average attempts per mission	−.230	−1.578	−.021	−.096	.093	.489
Average minutes per attempt	−.125	−.601	.135	.422	.251	1.075
Affective behaviors
Engagement in activity (game)	.157	.527	−.156	−.341		NS
Absorption in activity (game)	−.280	−1.202	.193	.539	.248	2.086*
Engage × Absorb	.597	1.325	.051	.074		NS
Technology use
Identify as gamer	−.006	−.063	−.102	−.658	.082	.589
Identify as computer user	.131	1.602	.126	1.002	.132	1.202
r²		68%		24%		37%
Adjusted r²		59%		3%		22%

Note. NS = not significant and removed from model due multicollinearity issues.

∼p < .10. *p < .05. **p < .01. ***p < .001.

Table 7.

Correlation Coefficients of Variables Included in Multiple Regression of Deadly Distribution Sample (N = 94) on Interest in Statistics and Learning Objectives 1 and 3.

	First language not English	Freshman	Junior	Senior	Female	Presurvey participate} \cr \rotate{in class	Presurvey more than} \cr \rotate{one statistics course } \cr \rotate{in college	PreObj 1	PreObj 3	Success on } \cr \rotate{Mission 2	Success on } \cr \rotate{Mission 3	Success on } \cr \rotate{Mission 4	Average attempts } \cr \rotate{per mission started	Average minutes } \cr \rotate{in mission	Average minutes } \cr \rotate{per attempt	Z-score: } \cr \rotate{gamer	Z-score: computer } \cr \rotate{user	Z-score: absorption } \cr \rotate{in activity
First language not English	1	0.033	0.046	−0.039	0.069	0.020	0.162	0.075	0.027	−0.044	−0.119	−0.139	0.153	−0.063	−0.097	−0.128	−0.006	−0.213*
Freshman	0.033	1	−0.151	−0.041	0.228*	0.089	−0.103	0.201	0.142	0.109	−0.084	0.046	−0.066	0.284**	0.259*	−0.128	0.017	0.124
Junior	0.046	−0.151	1	−0.039	−0.124	−0.069	−0.099	0.050	−0.023	0.155	0.136	0.059	−0.026	−0.065	−0.029	0.166	0.297**	0.125
Senior	−0.039	−0.041	−0.039	1	−0.086	0.090	−0.027	−0.046	0.144	−0.138	−0.102	−0.038	0.024	−0.057	−0.076	0.008	0.056	−0.031
Female	0.069	0.228*	−0.124	−0.086	1	−0.053	0.047	−0.026	0.043	−0.220*	−0.306**	−0.307**	0.092	0.255*	0.118	−0.611**	−0.164	−0.281**
Presurvey participate in class	0.020	0.089	−0.069	0.090	−0.053	1	0.099	0.065	−0.065	−0.073	0.026	−0.051	0.003	0.133	0.019	−0.078	−0.023	0.208*
Presurvey more than one statistics course in college	0.162	−0.103	−0.099	−0.027	0.047	0.099	1	−0.111	−0.113	0.106	−0.082	−0.095	0.158	−0.222*	−0.210*	−0.009	−0.140	−0.105
PreObjective1	0.075	0.201	0.050	−0.046	−0.026	0.065	−0.111	1	0.284**	0.179	0.108	0.094	−0.293**	0.192	0.225*	0.076	−0.035	0.322**
PreObjective3	0.027	0.142	−0.023	0.144	0.043	−0.065	−0.113	0.284**	1	−0.151	0.012	−0.018	0.049	0.093	0.014	0.026	−0.043	−0.010
Success on Mission 2	−0.044	0.109	0.155	−0.138	−0.220*	−0.073	0.106	0.179	−0.151	1	0.471**	0.274**	−0.571**	−0.066	0.337**	0.203	0.013	0.184
Success on Mission 3	−0.119	−0.084	0.136	−0.102	−0.306**	0.026	−0.082	0.108	0.012	0.471**	1	0.306**	−0.469**	−0.194	0.160	0.338**	0.163	0.184
Success on Mission 4	−0.139	0.046	0.059	−0.038	−.307**	−0.051	−0.095	0.094	−0.018	0.274**	0.306**	1	−0.152	−0.162	−0.025	0.253*	0.085	0.287**
Average attempts per mission started	0.153	−0.066	−0.026	0.024	0.092	0.003	0.158	−0.293**	0.049	−0.571**	−0.469**	−0.152	1	−0.368**	−0.730**	−0.126	0.038	−0.270**
Average minutes in mission	−0.063	0.284**	−0.065	−0.057	0.255*	0.133	−0.222*	0.192	0.093	−0.066	−0.194	−0.162	−0.368**	1	0.806**	−0.122	0.018	0.039
Average minutes per attempt	−0.097	0.259*	−0.029	−0.076	0.118	0.019	−0.210*	0.225*	0.014	0.337**	0.160	−0.025	−0.730**	0.806**	1	0.001	0.009	0.123
Z-score: gamer composite	−0.128	−0.128	0.166	0.008	−0.611**	−0.078	−0.009	0.076	0.026	0.203	0.338**	0.253*	−0.126	−0.122	0.001	1	0.384**	0.381**
Z-score: computer user composite	−0.006	0.017	0.297**	0.056	−0.164	−0.023	−0.140	−0.035	−0.043	0.013	0.163	0.085	0.038	0.018	0.009	0.384**	1	0.059
Z-score: absorption in activity	−0.213*	0.124	0.125	−0.031	−0.281**	0.208*	−0.105	0.322**	−0.010	0.184	0.184	0.287**	−0.270**	0.039	0.123	0.381**	0.059	1

*p < .05. **p < .01.

Discussion

The purpose of this study was to test the effects of a turn-based strategy game on knowledge in statistics and affective outcomes for undergraduate students. This study fills a gap in the body of research in SEGs by testing a game specifically designed for undergraduate students in a course containing complex content (Lamb et al., 2018). Much of the research examining the effectiveness of SEGs has been conducted in the K-12 setting; however, there has been a call for changing instructional practices to engage and motivate “Millennial” students in the undergraduate setting by incorporating more graphics, visualizations, simulations, and games in a technology enriched curriculum (McGlynn, 2007; Worley, 2011). While there has been a call to increase the use of technology and games in the undergraduate setting, Brom et al. (2017) conclude that the evidence of effectiveness of SEGs in the college setting remains mixed. This experimental test of the Deadly Distribution game contributes to the body of empirical evidence on the effectiveness of SEGs in postsecondary settings by demonstrating that a game-based instructional approach can improve affective outcomes as well as academic outcomes. Students who engaged in gameplay had significant increases in their academic understanding of objectives related to the CLT, and that these increases were not significantly different from traditional homework approaches. However, the key finding of this study is that students’ affective behaviors were significantly improved compared to traditional homework; students who engaged in playing the Deadly Distribution game reported significantly higher levels of engagement, absorption, and interest in statistics. In their game enjoyment model for the development of education games, Fjællingsdal and Klöckner (2017) suggest that quality educational games must capture players’ attention by applying motivational elements found in commercial games, and that, frequently, these motivational elements are paramount in SEGs that depend on pleasure and motivation to increase learning outcomes. In addition, these findings are congruent with other empirical work reporting positive effects of SEGs on student interest and attitudes about mathematics (Annetta et al., 2009; Barab et al., 2010; Kebritchi et al., 2010; Moreno et al., 2001; Vu et al., 2014).

This study also makes two additional contributions to the research in the field of SEGs. The results presented here were obtained using an experimental design, which addresses a problem frequently highlighted in the SEG literature (Kebritchi et al., 2010). This study addresses gaps in empirical research of the use of SEGs as an instructional approach in undergraduate courses. While Lamb et al. (2018) found negligible effects of SEGs on cognitive and affective outcomes in university settings, this study demonstrates that improvement in academic outcomes with the use of the Deadly Distributions SEG mirrors traditional paper–pencil practice activities while producing significantly higher affective outcomes. This study also demonstrates a plausible method for incorporating SEGs as a supplemental, out-of-class practice activity (Holmes & Gee, 2016). In addition, the use of an introductory statistics course with high enrollment provided a large enough group of participants to allow for disaggregation of data by important student background variables including gender, year in college, speaking English as a second language, and comfort with technology and digital games. The results demonstrate that the Deadly Distribution game generated significant, positive interest in statistics for students who speak English as a second language. This finding is important given the increasing diversity of U.S. undergraduate student populations (Swail, 2002). However, female students who played the Deadly Distribution game showed significantly less interest in statistics compared to male students. This finding replicates previous empirical work that similarly showed that female students found statistics courses less engaging compared to male students in the same courses (Nichols, 2017). This finding is important given that female students have come to represent the majority of undergraduate students in the United States and that the percentage of women earning statistics degrees has been increasing over the past decade (U.S. Department of Education, 2016; Scheaffer & Stancy, 2004). Finally, the results show that previous experience with digital games or comfort as a computer user had no impact on interest in statistics or academic outcomes among Deadly Distribution players. This may indicate that the students in this course all have an adequate base of technology skills to navigate through a new digital game with no external instruction, or it may indicate that the game was easily accessible to students of varying technology expertise levels.

Although this study contributes to the growing body of empirical research into the effectiveness of SEGs, there are areas that need additional investigation. While it has been previously documented that statistics anxiety can be an obstacle to motivation, interest, and degree completion (Conners et al., 1998; Onwuegbuzie et al., 1997), this study did not directly measure student anxiety. In future studies of SEGs for statistics and mathematics, student anxiety would be a valuable proximal outcome to measure. This study also demonstrates the need for additional examination of the effects of SEGs on different subgroups of students, particularly female students. The Deadly Distribution game was not successful in increasing female students’ interest in statistics, and, in future studies, it would be worthwhile to collect exit interview data to elucidate the reasons that females had less interest in statistics compared to males. In a recent study of a mathematics SEG in a middle school context, Garneli, Giannakos, and Chorianopoulos (2017) found that girls reported more difficulty with the gameplay than boys, and these difficulties led them to quit playing the game. In this study, girls who practiced the same concept using traditional paper and pencil exercises had increased learning compared to those who played the mathematics game. The insights of Garneli et al. were based on qualitative exit interviews which suggest that this type of data would be invaluable in improving game designs to engage specific subpopulations of students. In addition, the persistence of interest in statistics and engagement in statistics activities were not measured longitudinally, and it is possible that the effects would not persist beyond this course, and this would be an interesting area for future research. Finally, this study was limited to a single-player, noncollaborative game environment; however, there were anecdotal indications from the research assistants that students wanted to engage in collaboration with their peers to problem solve. While research assistants were instructed to redirect students to play individually, this observation suggests that a collaborative game design may further improve student outcomes.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Meredith Wronowski is a visiting assistant professor in the Department of Educational Leadership at Miami University. Her research interests include the effects of federal policy on teacher turnover, policy and school segregation, and school improvement.

Angela Urick is an assistant professor in the Jeannine Rainbolt College of Education, Department of Educational Leadership and Policy Studies. She studies how to lead and measure school improvement.

Alison S. P. Wilson is a doctoral candidate in the Jeannine Rainbolt College of Education, Department of Educational Leadership and Policy Studies, at the University of Oklahoma. Her research examines school structures, policies, and practices that contribute to the opportunity gap in schools to inform school leadership practice and education policy.

William Thompson is a senior instructional game designer in the K20 Center at the University of Oklahoma. He translates learning objectives and content into game mechanics in an effective and engaging manner.

David Thomas is the director of Technology at TEL Library and an instructional game designer in the K20 Center at the University of Oklahoma. He creates adventure by transforming instruction into engaging digital worlds.

Scott Wilson is the director of Innovative Technologies in the K20 Center at the University of Oklahoma. He bridges research and practice in schools with a focus on instruction, through game-based learning, and student readiness.

Francisco J. Elizondo is an educational video game producer in the K20 Center at the University of Oklahoma. He coordinates game developers, instructional designers, and subject matter experts to produce reliable and engaging gaming experiences for students.

Ryan Ralston is a senior software engineer in the K20 Center at the University of Oklahoma. He designs and develops educational games and the auxiliary systems to integrate the games in the classrooms.

References

Agarwal

Karahanna

(2000) Time flies when you’re having fun: Cognitive absorption and beliefs about information technology usage. MIS Quarterly 24(4): 665–694.

Anderson, D. R., Sweeney, D. J., & Williams, T. A. (2014a). Discrete probability distributions. In Essentials of modern business statistics (pp. 209–250). Independence, KY: Cengage Learning.

Anderson, D. R., Sweeney, D. J., & Williams, T. A. (2014b). Continuous probability distributions. In Essentials of modern business statistics (pp. 251–282). Independence, KY: Cengage Learning.

Anderson, D. R., Sweeney, D. J., & Williams, T. A. (2014c). Sampling and sampling distributions. In Essentials of modern business statistics (pp. 283–323). Independence, KY: Cengage Learning.

Annetta, L.A., Minogue, J., Holmes, S.Y., & Cheng, M. (2009). Investigating the impact of video games on high school students' engagement and learning about genetics. Computers and Education, 53, 74–85.

Baloğlu

(2003) Individual differences in statistics anxiety among college students. Personality and Individual Differences 34(5): 855–865.

Barab

S. A.

Gresalfi

Ingram-Goble

(2010) Transformational play: Using games to position person, content, and context. Educational Researcher 39(7): 525–536.

Barr

(2017) Video games can develop graduate skills in higher education students: A randomised trial. Computers & Education 113(Supplement C): 86–97. doi:10.1016/j.compedu.2017.05.016.

Ben-Zvi

Garfield

(2004) Statistical literacy, reasoning, and thinking: Goals, definitions, and challenges. In: Ben-Zvi

Garfield

(eds) The challenge of developing statistical literacy, reasoning and thinking, Dordrecht, the Netherlands: Springer, pp. 3–15.

10.

Boyle

E. A.

MacArthur

E. W.

Connolly

T. M.

Hainey

Manea

Kärki

van Rosmalen

(2014) A narrative literature review of games, animations and simulations to teach research methods and statistics. Computers & Education 74: 1–14. doi:10.1016/j.compedu.2014.01.004.

11.

Brom

Děchtěrenko

Frollová

Stárková

Bromová

D’Mello

S. K.

(2017) Enjoyment or involvement? Affective-motivational mediation during learning from a complex computerized simulation. Computers & Education 114: 236–254. doi:10.1016/j.compedu.2017.07.001.

12.

Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54(4), 297–312. https://dx-doi-org.web.bisu.edu.cn/10.1037/h0040950.

13.

Campbell, D. T. (1975). “Degrees of freedom” and the case study. Comparative Political Studies, 8, 178–93.

14.

Chance, B., & Rossman, A. (2006, July). Using simulation to teach and learn statistics. In Proceedings of the Seventh International Conference on Teaching Statistics, The Hague, the Netherlands: International Statistics Institute.

15.

Clark

D. B.

Tanner-Smith

E. E.

Killingsworth

S. S.

(2016) Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research 86(1): 79–122. doi:10.3102/0034654315582065.

16.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.

17.

Conners

F. A.

McCown

S. M.

Roskos-Ewoldsen

(1998) Unique challenges in teaching undergraduate statistics. Teaching of Psychology 25(1): 40–42.

18.

Connolly

T. M.

Boyle

E. A.

Macarthur

Hainey

Boyle

J. M.

(2012) A systematic literature review of empirical evidence on computer games and serious games. Computers & Education 59(2): 661–686. doi:10.1016/j.compedu.2012.03.004.

19.

DelMas

R. C.

Garfield

Chance

(1999) A model of classroom research in action: Developing simulation activities to improve students’ statistical reasoning. Journal of Statistics Education 7(3): 1–2.

20.

Embretson

Reise

(2000) Item response theory for psychologists, Mahwah, NJ: Lawrence Erlbaum Associates.

21.

Fjællingsdal

K. S.

Klöckner

C. A.

(2017) ENED-GEM: A conceptual framework model for psychological enjoyment factors and learning mechanisms in educational games about the environment. Frontiers in Psychology 8: 1085, doi:10.3389/fpsyg.2017.01085.

22.

Foster

A. N.

(2011) The process of learning in a simulation strategy game: Disciplinary knowledge construction. Journal of Educational Computing Research 45(1): 1–27. doi:10.2190/EC.45.1.a.

23.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M.P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences of the United States of America, (pp. 8410–8415).

24.

Gal

(2004) Statistical literacy. In: Ben-Zvi

Garfield

(eds) The challenge of developing statistical literacy, reasoning and thinking, Dordrecht, the Netherlands: Springer, pp. 47–78.

25.

Garneli

Giannakos

Chorianopoulos

(2017) Serious games as a malleable learning medium: The effects of narrative, gameplay, and making on students’ performance and attitudes. British Journal of Educational Technology 48(3): 842–859. doi:10.1111/bjet.12455.

26.

Garris

Ahlers

Driskell

J. E.

(2002) Games, motivation, and learning: A research and practice model. Simulation & Gaming 33(4): 441–467.

27.

Gee

J. P.

(2003) What video games have to teach us about learning and literacy. Computers in Entertainment 1(1): 20–20. doi:10.1145/950566.950595.

28.

Gee

J. P.

(2005) Learning by design: Good video games as learning machines. E-Learning and Digital Media 2(1): 5–16. doi:10.2304/elea.2005.2.1.5.

29.

Girard

Ecalle

Magnan

(2013) Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning 29(3): 207–219. doi:10.1111/j.1365-2729.2012.00489.x.

30.

Hagtvedt

Jones

G. T.

Jones

(2007) Pedagogical simulation of sampling distributions and the Central Limit Theorem. Teaching Statistics 29(3): 94–97.

31.

Hildreth

L. A.

Robison-Cox

Schmidt

(2018) Comparing student success and understanding in introductory statistics under consensus and simulation-based curricula. Statistics Education Research Journal 17(1): 103–120.

32.

Holmes

J. B.

Gee

E. R.

(2016) A framework for understanding game-based teaching and learning. On the Horizon 24(1): 1–16. doi:10.1108/OTH-11-2015-0069.

33.

(2008) Computer games application within alternative classroom goal structures: Cognitive, metacognitive, and affective evaluation. Educational Technology Research and Development 56(5): 539–556. doi:10.1007/s11423-008-9086-5.

34.

Kebritchi

Hirumi

Bai

(2010) The effects of modern mathematics computer games on mathematics achievement and class motivation. Computers & Education 55(2): 427–443. doi:10.1016/j.compedu.2010.02.007.

35.

Kennedy

Judd

Dalgarno

Waycott

(2010) Beyond natives and immigrants: Exploring types of net generation students. Journal of Computer Assisted Learning 26(5): 332–343. doi:10.1111/j.1365-2729.2010.00371.x.

36.

Kermack

W. O.

McKendrick

A. G.

(1927) A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 115(772): 700–721.

37.

Kolb

D. A.

(2015) Experiential learning: Experience as the source of learning and development, 2nd ed. Upper Saddle River, NJ: Pearson Education.

38.

Lamb

R. L.

(2016) Examination of the effects of dimensionality on cognitive processing in science: A computational modeling experiment comparing online laboratory simulations and serious educational games. Journal of Science Education and Technology 25(1): 1–15. doi:10.1007/s10956-015-9587-z.

39.

Lamb

R. L.

Annetta

Firestone

Etopio

(2018) A meta-analysis with examination of moderators of student cognition, affect, and learning outcomes while using serious educational games, serious games, and simulations. Computers in Human Behavior 80(Supplement C): 158–167. doi:10.1016/j.chb.2017.10.040.

40.

Lambert, G. (2012). Twilight of the lecture: The trend toward “active learning” may overthrow the style of teaching that has ruled universities for 600 years. Harvard Magazine. Retrieved from http://harvardmagazine.com/2012/03/twilight-of-the-lecture.

41.

Lameras

Arnab

Dunwell

Stewart

Clarke

Petridis

(2017) Essential features of serious games design in higher education: Linking learning attributes to game mechanics. British Journal of Educational Technology 48(4): 972–994. doi:10.1111/bjet.12467.

42.

Lane

D. M.

Tang

(2000) Effectiveness of simulation training on transfer of statistical concepts. Journal of Educational Computing Research 22(4): 383–396.

43.

Lopez, X., Valenzuela, J., Nussbaum, M., & Tsai, C. (2015). Some recommendations for the reporting of quantitative studies. Computers and Education, 91, 106–110.

44.

Mazer

J. P.

Hess

J. A.

(2017) What is the place of lecture in higher education? Communication Education 66(2): 236–237. doi:10.1080/03634523.2017.1287411.

45.

McClave, J. T., Benson, P. G., & Sincich, T. T. (2012a). Sampling distributions. In Statistics for business and economics (pp. 277–312). London, England: Pearson Education Limited.

46.

McClave, J. T., Benson, P. G., & Sincich, T. T. (2012b). Inferences based on a single sample: Estimation with confidence intervals. In Statistics for business and economics (pp. 313–270). London, England: Pearson Education Limited.

47.

McGlynn

A. P.

(2007) Millennials in college: How do we motivate them? The Hispanic Outlook in Higher Education 17(25): 34–36.

48.

Mertler, C. A., & Vannatta, R. A. (2002). Advanced and multivariate statistical methods: Practical application and interpretation (2nd ed.). Los Angeles: Pyrczak.

49.

Mills

J. D.

(2002) Using computer simulation methods to teach statistics: A review of the literature. Journal of Statistics Education 10(1): 1–20.

50.

Mills

J. D.

(2004) Learning abstract statistics concepts using simulation. Educational Research Quarterly 28(4): 18–33.

51.

Moreno

Mayer

R. E.

Spires

H. A.

Lester

J. C.

(2001) The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction 19(2): 177–213.

52.

Mvududu

Kanyongo

G. Y.

(2011) Using real life examples to teach abstract statistical concepts. Teaching Statistics 33(1): 12–16.

53.

Newbold

Carlson

W. L.

Thorne

(2013) Statistics for business and economics, Boston, MA: Pearson.

54.

Nichols, R. (2017). The effectiveness of inquiry-based vs. didactic teaching methods on student performance in undergraduate statistics (Dissertation). Florida Gulf Coast University, ProQuest Dissertations Publishing. Retrieved from https://search-proquest-com.ezproxy.lib.ou.edu/docview/1925938923?accountid=12964 (10286086).

55.

Oblinger

J. L.

Lippincott

J. K.

(2005) Educating the net generation, Boulder, CO: EDUCAUSE.

56.

Onwuegbuzie

DaRos

Ryan

(1997) The components of statistics anxiety: A phenomenological study. Focus on Learning Problems in Mathematics 19: 11–35.

57.

Plass

J. L.

O’keefe

P. A.

Homer

B. D.

Case

Hayward

E. O.

Stein

Perlin

(2013) The impact of individual, competitive, and collaborative mathematics game play on learning, performance, and motivation. Journal of Educational Psychology 105(4): 1050–1066. doi:10.1037/a0032688.

58.

Randel

Morris

Wetzel

Whitehill

(1992) The effectiveness of games for educational purposes: A review of recent research. Simulation & Gaming 23(3): 261–276.

59.

Scheaffer, R.L. & Stansy, E.A (2004). The state of undergraduate education in statistics: A report from the CBMS 2000. The American Statistician, 54(4), 265–271.

60.

Shadish, W. R. (1995). The logic of generalization: Five principles common to experiments and ethnographies. American Journal of Community Psychology, 23, 419–428.

61.

Shadish, W. & Cook, T.R. (1999). Comment-design rules: More steps toward a complete theory of quasi-experimentation. Statistical Science, 14(3), 294–300.

62.

Shadish, W., Cook, T. D., & Campbell, D. (2002). Experimental and quasi-experimental design for generalized causal inference. Boston, MA: Houghton Mifflin.

63.

Sitzmann

(2011) A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology 64(2): 489–528. doi:10.1111/j.1744-6570.2011.01190.x.

64.

Squire

(2006) From content to context: Videogames as designed experience. Educational Researcher 35(8): 19–29.

65.

Swail

W. S.

(2002) Higher education and the new demographics: Questions for policy. Change 34(4): 14–23.

66.

Trotter

(2005) Despite allure, using digital games for learning seen as no easy task. Education Week 25(10): 1–19.

67.

U.S. Department of Education, National Center for Education Statistics. (2016). Digest of Education Statistics, 2014 (NCES 2016-006), Chapter 3.

68.

Vlachopoulos

Makri

(2017) The effect of games and simulations on higher education: A systematic literature review. International Journal of Educational Technology in Higher Education 14(22): 1–33. doi:10.1186/s41239-017-0062-1.

69.

Vogel

J. J.

Vogel

D. S.

Cannon-Bowers

J. A. N.

Bowers

C. A.

Muse

Wright

(2006) Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research 34(3): 229–243.

70.

Crow

S. R.

Frederickson

(2014) Using games in online education: Is it a winning strategy? Online Journal of Distance Learning Administration 17(4): http://www.westga.edu/∼distance/ojdla/winter174/vu_crow_fredrickson174.html .

71.

Watkins

Bargagliotti

Franklin

(2014) Simulation of the sampling distribution of the mean can mislead. Journal of Statistics Education 22(3): www.amstat.org/publications/jse/v22n3/watkins.pdf .

72.

Worley

(2011) Educating college students of the net generation. Adult Learning 22(3): 31–39. doi:10.1177/104515951102200305.

73.

Wouters

Van Nimwegen

Van Oostendorp

Van Der Spek

E. D.

(2013) A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology 105(2): 249–265. doi:10.1037/a0031311.