Innovation Diffusion

Abstract

This research project investigated student reaction to playing the DIFFUSION SIMULATION GAME (DSG) and how an instructor, who is a novice in playing online games, implemented the DSG in an online higher education course. The goal of this research project was to determine whether playing the DSG helps students learn and apply course content. In addition, the authors sought to understand how to implement and use the DSG effectively as an integral part of the online Managing Technological Change course. Results indicated that student reaction to playing the game was very positive and students felt the game helped them better understand the change process. For example, students felt the DSG provided insight into school culture, the amount of time needed to implement a change, and the need for a change agent to be flexible. The instructor learned the importance of clearly identifying learning goals when using the DSG in an online course and realized that debriefing is a critical component of using games or simulations for learning.

Keywords

change agent change process debriefing diffusion of innovations DIFFUSION SIMULATION GAME feedback learning managing technological change online course participant perspectives players simulation games strategy student reaction survey web-based simulation

Although many games and simulations are used in educational settings, questions remain about the effectiveness and value of games and simulations for learning. Simulations provide users with a safe and reality-based environment, in which they can gain valuable experience, that is typically less expensive than arranging for real experiences and that avoids possible risky or unsafe environments. However, not all simulations are well designed and developed. Poorly developed simulations can result in failure and frustration for the student; moreover, not all learners accept simulations as an instructional method (Cruickshank & Telfer, 1980).

One area where games have been used frequently is in business-related courses, but even in this area, the majority of business faculty members have never used a business simulation game (Faria & Wellington, 2004). Simulation games have also been widely used in the military for tactical analysis, in the medical field for surgical training, and in aviation for flight training. Simulator Networking and Modular Semiautomated Forces were used to prepare the military for Desert Storm (Smith, 2010); CathSim, Ultrasim, MIST VR, Cinemed, and Human Patient Simulator are used in medical education for phlebotomy and IV insertion training, analysis of ultrasound images, laparoscopic surgery training, and teaching in acute care clinical skills (Lane, Slavin, & Ziv, 2001). In aviation, all pilots today are trained in flight simulation (Crookall, 2010). Brooks-Young (2010) argued that if the value of gaming is perceived as positive by the U.S. military and the medical profession, then educators also should take a close look at games for learning. However, Ebner and Holzinger (2007) found little evidence that games and simulations are used in higher education and Becker (2007) characterized the use of games by some K-12 teachers as “rewards given after the ‘real’ work is done” (p. 479) and suggested that many educators may not know how best to use games for learning.

One reason why educators may be reluctant to use games in online or traditional face-to-face learning environments is that educators may find it difficult to know where to find out about what games are available and to learn how games can be used effectively (Becker, 2007). A second possible reason is the difficulty in adequately evaluating what students have learned from playing the game (Dempsey, Haynes, Lucassen, & Casey, 2002; Moreno-Ger, Burgos, & Torrente, 2009).

As a novice game-player, the instructor for the Managing Technological Change online course decided to use the DSG to learn more about how to use simulations and to investigate student reaction to playing an online simulation. In this evaluation study of the implementation of the online DIFFUSION SIMULATION GAME (DSG; 2009), participants completed a survey to gauge their reactions after playing the game and wrote a reflection paper about the experience. Results indicate that using the DSG allowed participants to apply the theories they were learning in a realistic scenario. It was evident from survey results that the use of the DSG in the online course was received positively by students. A weakness of the research design was the lack of emphasis on debriefing. This oversight was due to the fact that the course instructor was a novice game-player, unaware of available DSG debriefing aids, and underestimated the importance of debriefing when using games for learning. (The terms game and simulation are used interchangeably when referring to the DSG throughout the article.)

Review of the Literature

One characteristic that makes a game a game is the set of particular rules that players must follow (Hlodan, 2008; Shaffer, 2006). In addition, Shaffer (2006) suggested that game-players have specific roles and these roles also are defined by rules. Arguing that games encourage active learning, Gee (2007) included three aspects in this type of learning: (a) experiencing the world in new ways, (b) forming new affiliations, and (c) preparing for future learning. These characteristics of games, that they have rules, roles, and provide a platform for active learning, are complimented by the fact that with a game, the player can fail and start over, often learning through an iterative process.

In fact, it is frequently the case that the purpose of a simulation game is not to win the game (Rogers & Goodloe, 1973), but to help students develop skills in decision making and problem solving as well as providing a low-risk environment in which players can try out innovative and creative strategies. In this sense, simulations allow players to make mistakes and learn from them. In playing a game multiple times, the player develops critical learning skills, skills that Gee (2007) defined as consciously attending to, reflecting on, critiquing, and manipulating principles and patterns within a particular environment. Thus, as players learn and experience the rules of a game by assuming a specific role, they see and learn from the consequences of the game-playing decisions they make (Sottile & Brozik, 2004). Games can therefore help develop problem-solving skills, a set of skills often identified as a critical learning outcome for 21st-century students.

Another important learning goal in education is to help students transfer knowledge learned in the classroom to real-world situations. The use of simulation games that replicate real-world situations help meet this goal (Sottile & Brozik, 2004). Realism, then, becomes a critical aspect of simulation games, especially because authentic real-life environments and learning experiences are difficult to create in traditional classrooms (Klopfer, Osterweil, Groff, & Haas, 2009). Crookall and Thorngate (2009) referred to action knowledge; the idea that simulations help players use their knowledge appropriately in a realistic setting, thus encouraging optimal learning.

Through the use of simulation games, additional learning outcomes may be achieved including introducing students to planning, having students experience teamwork, engaging students in critical thinking, and helping students measure their comprehension and understanding (Faria & Wellington, 2004). One very positive aspect of using games for learning is that students typically enjoy them and are often more motivated and engaged in the learning process when games are used (de Freitas, 2006).

In spite of the number of advantages associated with games in education, many educators have never used or are skeptical about their benefits in learning (Cotton, Ahmadi, & Esselborn, 1997; Devlin-Scherer & Sardone, 2010). Faria and Wellington (2004) studied current and former simulation game users and those who had never used simulation games. All participants in their study were members of business faculties. From the perspective of current and former game users in the study, games were seen as beneficial for both students and instructors. One benefit indicated by the study participants was that simulation games provided a way for students to apply the theories they were learning. Participants also noted that, for students, simulation games provided experiential learning. With respect to instructors, participants noted that simulation games offered interactive and dynamic exercises. Moizer and Lean (2010) investigated the relatively low use of simulations and games in higher education and offered a diffusion model for encouraging more faculties to use games. Their model includes both intrinsic and extrinsic strategies as well as suggestions for how to eliminate barriers to game adoption in higher education.

When instructors elect to use a simulation or game for instructional purposes, they need to be mindful and ensure that the game aligns with course content (Cotton et al., 1997; de Freitas, 2006). Moizer, Lean, Towler, and Abbey (2009) found that one major barrier to the use of simulations and games in higher education centered on whether a faculty member perceived that a specific simulation or game was suitable for a particular course and would help students achieve learning outcomes.

Best practices for using games are identified in a publication sponsored by the Software and Information Industry Association (SIIA; Wilson, 2009). These best practices encompass how administrators can best encourage teachers to use games, how to provide professional development and support for teachers, and how to implement the use of games in classrooms. The SIIA study found, for example, that the common experience of playing a game can enhance whole-class participation in discussions and other activities related to the game’s content. Stemming from another project called The Teaching and Games Project, a grid was developed to balance characteristics of gaming in relation to the curriculum (British Educational Communications and Technology Agency, 2007). Not surprisingly, games that are most closely linked to the curriculum and that are well-developed games in and of themselves are viewed as the most valuable in the classroom because “playing the game as the designer intended is the means by which children will achieve the learning objectives of the lesson” (British Educational Communications and Technology Agency, 2007, p. 49).

One simulation designed to teach specific course content is the DSG, based on Rogers’s diffusion of innovation theory (Rogers, 1995) and other related change theories (DSG, 2009). The DSG first emerged as a board game in the 1970s to teach change management strategies to master’s level students in the Instructional Systems Technology (IST) program at Indiana University in a face-to-face classroom setting (Molenda & Rice, 1979). Since that time, the DSG has been used regularly in some courses at Indiana University and has been adopted by numerous departments of education throughout the United States. The DSG has been found effective in achieving its affective and cognitive objectives and has received recognition from a number of national educational and instructional technology organizations (Molenda & Rice, 1979). In 2002, the paper-based DSG was converted to a web-based version for online master’s students at Indiana University. The web-based DSG subsequently has been used for both online and face-to-face classes in the IST program at Indiana University (Frick, Kim, Ludwig, & Huang, 2003).

The DSG simulates a fictional junior high school, with the player acting as a change agent with the goal of persuading as many staff and faculty members as possible to adopt a peer-tutoring program within a 2-year virtual time frame. The player selects a strategy for each “move” and receives feedback from the DSG about each move’s effectiveness in the adoption process. Changes in perceptions about the peer-tutoring program progress along a continuum from awareness to adoption (Frick et al., 2003). Frick et al. (2003) reported results from a questionnaire on respondents’ reactions to the online DSG from two groups of students, one group were members of a face-to-face class and the second group were members of an online seminar. In general, both groups of students had positive experiences playing the DSG. These students felt the DSG was interesting and realistic, and they were motivated to learn more about the change process. Recently, the web-based DSG has gone through a revision for improving its usability (Lara, Myers, Frick, Aslan, & Michaelidou, 2010).

Merely playing a simulation game, however, is not adequate. As the instructor of the Managing Technological Change course learned, debriefing afterwards is an essential and critical component when using simulations in the classroom (Crookall & Thorngate, 2009; Kriz, 2010) because through debriefing, students improve their understanding of relationships between the simulation and real situations. Kriz (2010) argued that debriefing strengthens student competencies, provides for multiple perspectives to be discussed, and leads to common, shared knowledge. When the simulation is used in a face-to-face classroom, debriefing can be conducted verbally in the class after all participants have played the game. Debriefing can also be conducted asynchronously in an online discussion board. Oertig (2010) set up an online discussion forum in Moodle for classroom debriefing and found that most students participated. In addition, he felt the quality of the debriefing conversations was more in depth and substantive than when done verbally in a whole-class session. An advantage of online discussions is that the debriefing information is written and thus provides students and the instructor a record of the conversation. Frick et al. (2003) also reported using an asynchronous online communication tool for debriefing, where players discussed the strategies they used during the game, based on feedback received and recorded on each player’s respective game log.

In the current environment of Web 2.0 tools, online games and simulations are becoming more plentiful and teachers may be more willing to explore the potential benefits and value of games for learning (Brooks-Young, 2010). In addition, some researchers argue that more educators should embrace new digital games, social networking, and simulations (Klopfer et al., 2009). The combination of the desire by the instructor to use the DSG in online courses and the relatively low number of studies related to the use of games in higher education were the impetus to investigate the use of the DSG in the Managing Technology Change online course.

Research Questions

In an effort to determine whether using the DSG is a valuable addition to the Managing Technological Change online course, this research project investigated student reaction to the DSG. The goal of this research project was to determine whether playing the DSG helped students learn and apply course content. In addition, we sought to better understand how best to use the DSG for learning from an instructor’s perspective. The specific research questions were as follows:

Research Question 1: What are the perceptions of students regarding their ability to apply course content by playing the DSG?

Research Question 2: What are student reactions to playing the DSG as part of an online Managing Technological Change course?

Research Question 3: What does an instructor need to do to use the DSG in the Managing Technological Change online course effectively?

Method

On learning of the existence of the DSG when searching for resources to be used in a course called Managing Technological Change, the instructor of the course accessed the free online version of the DSG and played the game. Seeing a strong connection between the DSG and course content, the instructor decided to use the game as part of the course and developed an assignment that incorporated students playing the DSG and writing a reflective paper about their experiences. Throughout the course, multiple change models and theories were investigated. The instructor decided to conduct a study on the use of the game to determine whether students thought the DSG helped them learn course content and to gauge student reaction to playing the DSG as part of an online course.

Eighteen students volunteered and signed consent forms for the study. These students were taking a course called Managing Technological Change as part of their graduate program in instructional technology at a large Midwestern university. Participants were asked to complete an online, anonymous Simulation Game Student Reaction Survey (see appendix) after they played the DSG and to write a reflective journal about their experiences. The reflective journal asked what they liked and did not like about playing the game, what game strategies they used, and their opinion about whether the game helped them learn course content.

The decision was made to purchase the commercial form of the game because of the availability of game logs. Each participant was provided with a unique username and password. Having unique usernames allowed participants access to game logs that recorded each move and its associated feedback and also allowed the researcher to capture game logs after students had finished playing the DSG. The free version of the DSG may be played using a generic username and password; however, game logs are not available with the free version.

Although 18 students participated in the research study, only 15 game logs were retrieved and only 14 students completed the survey instrument. An even split between males and females completed the survey, 7 of each gender. Eighteen reflective journals were available for the study.

The DSG and the curriculum of the Managing Technological Change course are closely aligned because both deal with the change process. In addition, the theories used to design and develop the DSG are theories studied in the course. When playing the DSG, the player assumes the role of a change agent and is presented with a situation where a new instructional methodology is being introduced and implemented in a fictional school. The player has a virtual 2-year window in which to implement the new methodology. The player may interact with 22 teachers, staff, and administrators and may use a variety of game strategies. The player makes “moves” in the game by choosing strategies. Each strategy chosen can be more or less successful and each move uses up a specific number of weeks in the virtual 2-year timeline. One important aspect of the game is that it is structured to represent the stochastic nature of the change process; that is, the effect of a particular move is only partially predictable and includes a random element. Thus, choosing the same move at different times during the game will most likely result in quite different results depending on when in the game the move is selected and where each of the potential adoptees is in the continuum of stages (from awareness to adoption). The goal of the game is achieved if a player has 10 or more adoptees by the end of the game (Frick et al., 2003). The game ends when all 22 school personnel have adopted the change or the 2-year time frame has been reached.

Specific game strategies that a player can make as his or her “moves” when playing the DSG include the following:

Getting information (personal, community, lunch mates, social)

Talking to people

Asking for help

Doing a site visit

Using media (print, mass media)

Conducting training or materials workshops

Doing a presentation

Doing a demonstration

Doing a pilot

Compulsion

Confrontation

Participants were allowed to play the game multiple times and to access their game logs. A game log is created each time the game is played. The game log is a record of each move and includes the feedback provided to the player for each move and the impact of the move on specific teachers, staff, and administrators. Therefore, a player may review their game log, identify both positive and negative moves, and continue to play the game or stop the game and play again. Each time the game is played, a new game log is created that overlays the previous game log. Participants played the DSG toward the latter part of the semester, after nine different change models had been investigated and discussed.

After playing the game, participants were asked to complete the Simulation Game Student Reaction Survey designed to measure reaction to the game. The survey consisted of 16 questions related to playing the DSG plus 4 demographic questions and an open-ended comment area. The survey questions were based on a similar survey as reported in Faria and Wellington (2004). Because we were unaware that a specific debriefing protocol was available for the DSG, study participants completed a reflective journal after playing the game rather than participating in a more formal debriefing session. The instructor also created an online discussion forum and invited participants to post questions, comments, and reactions related to playing the DSG.

Results

In playing the DSG, two participants were highly successful, in having all 22 teachers, staff, and administrators adopt the change. Three participants convinced between 15 and 20 of the school’s teachers, staff, and administrators to adopt the change, four participants convinced between 9 and 14 teachers, staff, and administrators to adopt the change, and six participants had their game end abruptly due to the use of the “compulsion” or the “confrontation” strategy. The average number of adopters was 16.6. According to a game message that displays when the game is over, if players score at least 11 adoptees, they have a better score than the average number of adoptees gained by past DSG players of the free online version. Table 1 contains the numbers of adoptees.

Table 1.

DIFFUSION SIMULATION GAME Results

Won the game	15-20 adoptees	9-14 adoptees	Zero adoptees
2	3	4	6

Individual scores for participants are listed in Table 2.

Table 2.

DIFFUSION SIMULATION GAME Results by Participant

Participant ID	Score (number of adoptees)
1	0
2	0
3	0
4	18
5	22
6	9
7	20
8	12
9	13
10	0
11	14
12	20
13	0
14	0
15	22

Mean scores, standard deviations, and the distribution of scores from the Simulation Game Student Reaction Survey are presented in Table 3. Results from the survey are presented in more detail with each research question.

Table 3.

DIFFUSION SIMULATION GAME Student Reaction Survey Results

Survey question	M	SD
1. The DIFFUSION SIMULATION GAME was appropriate for this course.	4.57	0.514
2. The DIFFUSION SIMULATION GAME was realistic.	3.57	1.016
3. The DIFFUSION SIMULATION GAME allowed me to apply what I am learning in this course.	3.71	0.994
4. Using a simulation added variety to this course.	4.43	0.646
5. Playing the DIFFUSION SIMULATION GAME was fun.	4.00	1.038
6. Playing the DIFFUSION SIMULATION GAME was too time-consuming.	1.93	0.829
7. It was easy to exit and reenter the game.	4.07	0.829
8. I liked having the game log available.	4.29	0.611
9. Reviewing my game log provided me insight into how I played the game.	4.29	0.611
10. Reviewing my game log was helpful to me in playing the game.	3.86	0.864
11. The result of at least some of my choices was not what I expected to happen.	4.29	0.469
12. I felt that the result of some of my choices did not align with my understanding of the change process.	3.43	1.089
13. I got frustrated playing the DIFFUSION SIMULATION GAME.	3.14	1.351
14. It was easy for me to “win” the game; to achieve adoption of the change within the 2-year time frame.	2.07	0.997
15. Making a decision that did not give me points helped me better understand the change process.	3.00	1.038
16. Making a decision that did not give me points helped me make better decisions later in the game.	3.29	1.204

The reflective journals not only provided insight into how participants played the game, but also into their overall reactions toward playing the game. In their reflective journals, participants were asked to answer several questions related to how they played the game, including what their initial game moves were; which change theories, if any, they used to make game decisions; whether they felt that the DSG helped them learn course content; and the major obstacles encountered when playing the game. Participants also were given the opportunity to provide additional comments related to playing the game.

What Are the Perceptions of Students Regarding Their Ability to Apply Course Content by Playing the DSG?

Survey results from Item 3 indicated that participants felt they were able to apply what they were learning in the course during the game (M = 3.71, SD = 0.994). Several participants stated that they played the game multiple times. One participant played a total of 7 times. Representative statements made by participants in their reflective journals that point to their ability to apply course content to the DSG included the following:

[The DSG] helped me understand many of the change models better. The real-world connection made it helpful. [I] liked the responses I received after making a move. [I] could replay and make better decisions based on what I knew from previous games.

[The DSG] was frustrating, but it did give me personal insight on how actual events influence the change process and [the] time required to implement a change.

[I] was skeptical at first about playing a simulation game as a tool to help learn about educational change models. After playing, I was pleasantly surprised. The game did provide insight into change theories and captured the social, political, and group dynamics of an educational environment.

The activity provided insight into the school building climate and what is experienced. It was a fun and stimulating exercise and much more in depth than I originally thought.

It is a great way to put the theories into practice.

Additional statements in the reflective journals provided evidence that participants made connections between the game and the change process with comments, including “[I] learned change is hard,” “I think this game gave me more insight [into] the change process as a whole . . . than it did helping me understand a particular change model,” and “This game gives us some ideas about how difficult it is to make a change occur.”

One participant stated that playing the DSG was helpful in making changes in real life. Another participant felt the game did a good job in depicting resistance to change and offered “good practice for real life.” A third participant stated,

[I] learned that even though you have a plan in place . . . there will be things that occur that changes your plan. You need to be patient, think things through, and adapt as needed to make sure the change process can move forward.

Two participants were more negative and failed to see a connection between the DSG and change theories. One participant stated, “[I] don’t really feel it helped me understand any of the models or theories better.” The other participant made a similar comment, “[The DSG] did not really help me understand any of the change models because I didn’t see the connection between the two.”

What is Student Reaction to Playing the DSG as Part of an Online Managing Technological Change Course?

Survey results from Items 1, 2, and 5 indicated that participants found the game to be appropriate for the course (M = 4.57, SD = 0.514), realistic (M = 3.57, SD = 1.016), and fun (M = 4.00, SD = 1.038), respectively. In addition, results from Item 4 showed participants thought the DSG added variety to the course (M = 4.43, SD = 0.646). However, Item 13 indicated participants were frustrated at times (M = 3.14, SD = 1.351), and Item 11 showed participants got unexpected results from some of their choices and decisions (M = 4.29, SD = 0.469). For example, a slight overall agreement that some results from moves did not align with the participant’s understanding of what was “supposed” to happen according to change theory models was found, thereby resulting in a mismatch between what the participant thought would happen in the game and what actually happened.

Participants did not find it easy to win the game as indicated from Item 14 (M = 2.07, SD = 0.997), but liked the fact that they had the option to play the game multiple times (Item 7) and that they were able to view a game log from their previous game sessions asked in Item 8 (M = 4.29, SD = 0.611).

What Does an Instructor Need to Do to Use the DSG in the Managing Technological Change Online Course Effectively?

Before including any game or simulation in a course, an instructor needs to play the game and determine whether the game and the game’s goals align with the specific course content (Cotton et al., 1997; Kebritchi, 2010). Once that determination is made, a specific course assignment needs to be designed and developed. Students need to understand why they are playing a simulation or game, the connection between the simulation or game and the course, and any special instructions for playing (Kriz, 2010). While the majority of participants in this study would be classified as successful in achieving the goals of the DSG because the average number of adoptees was 16.6, some students may benefit from a more thorough explanation of the game prior to playing. For example, the instructor may want to provide a prerecorded video clip of how to login, where to find instructions for how to play the game, and how to use the game log.

In terms of using the game within a class, three suggestions were made by study participants. One suggestion was to play the game twice, once at the beginning of the course and a second time at the end of the course to see if players would have a better score after learning about the change process. A second suggestion was to have students play the game in small groups of two or three people rather than playing it alone. The third suggestion was to have students play the game using only one specific change model. An instructor may want to consider providing students one or more of these options and investigate how each option may impact student learning of course content.

Discussion

Comments made in the reflective journals provided evidence of a strong alignment between the goals of the Managing Technological Change course and the structure, goals, and content of the DSG. Participants in this study applied their knowledge of change theories when playing the DSG and were successful in doing so. We know this because the average number of adoptees was 16.6 and 10 adoptees is a winning score according to the DSG game developers.

Positive overall participant reaction was tempered with one participant who was negative about the game and felt it was unrealistic and “a waste of time.” Other participants, however, stated that it was fun and interesting. Statements participants made in relation to enjoying the game included the following:

“It was fun to try things without having real-life consequences.”

“Extremely fun, even though it was very stressful.”

“I enjoyed playing the game. It was fun to be learning, but not feel like it.”

Instructors need to be cognizant of the fact that some students may not enjoy or accept a gaming environment as a way to learn (Cruickshank & Telfer, 1980). Engaging in a preliminary discussion about the positive and negative aspects of game-playing may provide a means for instructors to listen and attend to multiple perspectives about game-playing and make adjustments if necessary.

The initial move of the DSG is dictated by the game itself. In the initial move, the player must obtain information from any 5 people using the “get information” strategy. Eight participants indicated that one of the people they chose to get information about in their initial move was the principal. Ten students included chair people in their first move (e.g., the chair of the language arts department or the mathematics department). Nine out of the 18 participants chose to select repeatedly the “get information” move until information about all 22 teachers, staff, and administrators was obtained. From the standpoint of change theories, where communication is an essential and critical component in successful change, it makes sense that participants gathered basic information on multiple potential adoptees in their initial moves.

Participants indicated that they made game decisions based on their perceptions of relationships between school personnel as well as from the consequences of their prior moves. Some participants concentrated on convincing school personnel they felt were most influential, while other participants tried to determine moves that would result in the least resistance or gain the most adopters. Eleven participants identified specific change models they used to help them make DSG game decisions. While Rogers’s diffusion of innovation theory was one of the theories mentioned, the most frequently cited change model was the Concerns-Based Adoption Model (Hall, Wallace, & Dossett, 1973).

Several comments made by participants about their game decisions illustrate multiple game strategies. One participant based moves on “individuals, their relationship, and influence with others, and the consequences of one move to determine the next move.” Another participant based moves on “power and who could be the most influential.” A third participant indicated game decisions were made “based on which move would be more appropriate, would cause the least resistance, and would get the most adopters the fastest.” A fourth participant indicated that game decisions were strictly “trial and error.” Clearly, multiple strategies were used when making game decisions.

Major obstacles encountered by participants included the 2-year game time limit, the difficulty in getting the principal to become an adopter, and their own failure to pay attention to specific school personnel. Some participants mentioned that their own biases as to who they thought would be important in supporting the change caused them problems in gaining adopters. Three participants indicated that they stopped playing the game due to confusion, frustration, or time constraints; however, most participants played the game multiple times, trying new strategies each time they played.

As mentioned earlier, participants encountered situations where the result of a particular move did not align with what the participant was expecting, thus causing a discrepancy between game results and his or her understanding of the change process. These discrepancies, however, are representative of what happens in real life, where results are often unpredictable due, in part, to the complexity of the change process and the many social, political, organizational, and human factors that are involved. Thus, the fidelity between the DSG and a real change process appears to be high.

Enfield, Myers, Lara, and Frick (2011) investigated the fidelity between the DSG and its underlying theoretical framework by analyzing successful and unsuccessful game strategies from games played using the free version of the DSG. Their findings show some disparity between game strategies, expected results as predicted by the diffusion of innovations theory, and actual game results. These researchers suggested that revisions may be needed in the record-keeping aspects of the DSG to provide more detail about the game-playing process. Access to more than one game log per player may be beneficial in analyzing game results because comparisons could then be made between results and game-playing decisions and resultant patterns across multiple games by the same player.

Although we were limited to viewing one game log per participant because only the log of the last game played was available, interesting patterns are evident through an analysis and tabulation of specific moves. For example, the two participants who gained all 22 school personnel as adopters used the “site visit” move more frequently than all, but one other participant. Similarly, neither of these two participants used the “pilot the innovation” option, nor did they choose to use the “ask for help” option. In addition, these two participants had fewer overall moves than other participants, with 24 and 36 total moves, respectively. In contrast, other participants averaged 44 moves. On one hand, the two participants who gained all 22 adoptees may have had a better overall understanding of the change process than other participants. Alternatively, they may have simply been lucky in choosing options that helped them progress through the change process quickly and successfully. A third possible interpretation is that these two participants played the DSG multiple times, learned from their repetitive play, and were eventually able to deduce a successful sequence of moves. Future research might investigate the impact of playing the DSG multiple times by capturing game logs after each game, recording the number of adoptees gained, and comparing the sequence of moves coupled with the feedback received from each game.

We found it interesting that the “talk to” move was the most frequently chosen option, being chosen an average of almost 16 times per game. That “talk to” is chosen often may be understood in the context of our daily lives. When people are unsure of what to do, they often rely on the advice and guidance of others. Therefore, using “talk to” may simply be a way for those playing the DSG to try to gain more knowledge about a particular person or to find out more about the current status of the change. The least chosen moves were “compulsion,” chosen only 4 times and “confrontation,” chosen only 3 times. One participant chose both “compulsion” and “confrontation” during the game.

For the six participants who ended the game with the “compulsion” or the “confrontation” move, it was impossible to determine how many actual adoptees they had at the end of the game. Both of these moves try to force the innovation to be adopted. Comments in participant reflection journals suggest that as some participants neared the 2-year time limit in the game, they resorted to a drastic move option (i.e., “compulsion” or “confrontation”) in an attempt to increase their number of adopters. This strategy, however, failed, as the participant ended the game with few or no adopters. How the “confrontation” and “compulsion” moves are resolved in the DSG closely parallels what often happens in real life when a change is forced on an organization or group; when people are forced to accept a change, they often strongly resist, may try to undermine the change effort, or may abandon it altogether.

After playing the DSG, participants had a number of suggestions for improvement in the game itself as well as how it is used as a course assignment. Several participants indicated that they would have liked to have had more of an explanation about how a particular move related to a change theory. For example, one phase of Rogers’s diffusion of innovation theory (Rogers, 1995) is called “trialability,” defined as the ability to experiment with, or try out, an innovation before adopting it. In the game, moves such as the “demonstration” move or the “pilot” move afford a trial period. It may be beneficial for DSG feedback to include statements that clearly match the game move to relevant aspects of the diffusion of innovation theory. Another suggestion was to redesign the game to include multiple school scenarios, such as a large suburban school district and a small rural one, to compare the change process in two different environments.

As a course assignment, participants suggested playing the DSG in groups, at the beginning and end of the course, similar to a pre- and posttest design or according to a specific change theory. An instructor may want to consider providing students one or more of these options and investigate how each option may impact student learning of course content. Playing in small groups can support collaborative decision making (British Educational Communications and Technology Agency, 2007) and teamwork (Drake, Goldsmith, & Strachan, 2006) and may lead to a better understanding about the change process because members of the group can discuss moves both before and after they are made. The strength of a pretest and posttest is that it is one way to “directly assess the impact of the activity” (Chin, Dukes, & Gamson, 2009). Having students use a particular change model when playing the DSG may afford a basis for comparing strengths and weaknesses of change models or a means to identify common components among change models.

Instructors need to design an evaluation strategy to measure what students learn from playing a game. Moreno-Ger et al. (2009) suggested possible options for assessment, including traditional testing, interviews, essays, or observation. They suggest, however, that reviewing game logs may be much more beneficial to instructors because a game log may contain detailed information about how long a student spent playing a game, how many attempts were tried, and, as is the case of the DSG, the sequence of moves made and corresponding feedback. Chin et al. (2009) proposed ungraded surveys, paper-and-pencil assessments, reflection papers, and journal writing. Perhaps the most critical aspect of assessment of learning from simulations or games is debriefing (Crookall, 1992, 2010; Peters & Vissers, 2004).

One area that could have been improved in the present study is debriefing. The reflective journals did not provide enough of an opportunity for participants to discuss with each other the course content in relation to the game. Although a discussion forum was set up in the online course that invited comments, few participants took advantage of the opportunity to discuss their individual results, compare their results with others, and further explore what they learned about the change process from playing the DSG. In future classes, more attention needs to be given to debriefing because debriefing often fosters deeper understanding of course content and can lead to a shared understanding of that content (Peters & Vissers, 2004). In much the same way that Mawdesley, Long, Al-jibouri, and Scott (2011) made changes to their teaching methods when using simulations by adding group presentations and pre- and postquestionnaires, various strategies need to be investigated to strengthen and improve the learning that can take place through the use of a simulation coupled with a strong debriefing protocol. Unfortunately, the instructor of the Managing Technological Change course was unaware that debriefing guidelines were available for the DSG. In the future, greater attention will be directed to designing and implementing a debriefing plan to occur after students play the DSG, using the debriefing guidelines available from the developers of the DSG as a starting point.

Limitations of the study

Limitations of the study include the use of a convenience sample, access to only one game log per participant, lack of survey completion by some participants, and small sample size that severely restricts statistical analysis of data. First, study participants were a convenience sample because they were students in a course taught by the lead author. Second, only the final game log was available for analysis. In other words, if a participant played the game multiple times, it was only possible to capture and analyze the final game log. Each time the game is played, the previous game log is overlaid. The game logs were captured only after the simulation game assignment was completed. Having multiple game logs per participant may have afforded a better understanding of why some participants were more successful than others in gaining adoptees. Third, some participants elected to not complete the survey instrument resulting in 14 usable surveys. These self-reporting surveys assume that respondents are truthful and candid; however, survey data need to be viewed with this caveat. Clearly, the low number of study participants limits the statistical analyses that can be conducted. The means, standard deviations, and distribution of scores from the survey were reported.

Conclusions

The use of the DSG in the graduate-level online course Managing Technological Change is advantageous from multiple perspectives. First, the DSG is directly linked to course content. Playing the DSG gives students the opportunity to apply what they are learning and better understand theoretical concepts. Second, the simulation provides an environment in which students can experiment with different strategies and tactics as they assume the role of a change agent and experience the effects of their game decisions. Third, using a simulation often provides a new and different type of assignment that students perceive as fun and pragmatic. Last, as the DSG is available online, it is easy for students to access and use. From all these perspectives, using the DSG was a positive experience for both instructor and students.

While negative comments about the game were made by a few participants, the majority of participants overwhelmingly thought that playing the game was worth their time and effort. The lack of a carefully structured debriefing plan presents an area that can benefit from future research. Future studies regarding the DSG may want to investigate how debriefing is best structured and encouraged in online learning environments to help students draw conclusions about the game as well as reach consensus about the relationships between playing the DSG and learning about the change process.

Another area worth further research is to investigate how playing the DSG multiple times may affect final game results. Adding a question to the survey instrument to ask participants how many times they played the DSG is one way to gather data to help address this issue.

In the DSG, potential adopters move along a continuum of four phases from awareness, through interest and trial/appraisal, to adoption. Data gathered for this study only included the number of teachers, staff, and administrators who reached the adoption phase. An analysis of game results based on “closeness to adoption” may yield another measure of game performance. For example, if one player had 12 people at the trial phase and 4 adopters and a second player had 12 people at various stages from awareness to trial/appraisal and 6 adopters, the first player may have outperformed the second player because more people were closer to adoption even though the total number of adopters is higher for the second player. Future research that investigates these nuances of adoption may better inform a determination of student performance in the game.

Cruickshank and Telfer (1980) reminded us that student background knowledge may have a bearing on game achievement; therefore, another area worthy of additional research is the exploration of this relationship. In this study, perhaps the two students who “won” the DSG had more knowledge about the change process and change models than other students. Thus, having an assessment to measure course content knowledge prior to playing a simulation may help educators better understand why some students are more successful than others when playing a simulation.

Overall, results from this study indicate that simulations can be a win-win component of online courses. While educators need to insure that any simulation used matches their course content, simulations can be motivating, challenging, and engaging for students.

Footnotes

Appendix

Acknowledgements

The authors wish to thank the reviewers for providing high-quality feedback on the manuscript that helped improve the overall content and clarity.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Bios

Cindy L. Kovalik received her master’s and doctorate in educational psychology with a concentration in instructional technology from Kent State University (KSU). A novice game-player, she is currently learning about using games for learning and investigates technology integration, online learning teaching strategies, and visual literacy. She serves on the editorial review board of the Journal of Research on Technology in Education.

Contact: ckovalik@kent.edu.

Chia-Ling Kuo received her master’s in computer education and technology and doctorate in instructional technology from Ohio University. She joined KSU in 2005. Currently, she teaches educational technology, portfolio review, and advanced practicum. Her research interests include the integration of technology into curriculum, online learning and teaching, electronic portfolio development and assessment, wireless technology in education, and issues surrounding educational technology.

Contact: ckuo@kent.edu.

References

Becker

(2007). Digital game-based learning once removed: Teaching teachers. British Journal of Educational Technology, 38, 478-488. doi:10.1111/j.1467-8535.2007.00711.x

British Educational Communications and Technology Agency. (2007). Emerging technologies for learning (Vol. 2). Coventry, England: Author.

Brooks-Young

(2010). Teaching with the tools kids really use: Learning with web and mobile technologies. Thousand Oaks, CA: Corwin Press.

Chin

Dukes

Gamson

(2009). Assessment in simulation and gaming: A review of the last 40 years. Simulation & Gaming, 40, 553-568. doi:10.1177/1046878109332955

Cotton

Ahmadi

Esselborn

(1997). Assessing simulation games for the classroom. Assessment Update, 9, 6-9.

Crookall

(1992). Debriefing. Simulation & Gaming, 23, 141-142.

Crookall

(2010). Serious games, debriefing, and simulation/gaming as a discipline. Simulation & Gaming, 41, 898-920.

Crookall

Thorngate

(2009). Acting, knowing, learning, simulating, gaming. Simulation & Gaming, 40, 8-26. Retrieved from Education Research Complete database

Cruickshank

Telfer

(1980). Classroom games and simulations. Theory Into Practice, 19, 75-80. Retrieved from Education Research Complete database

10.

de Freitas

(2006). Using games and simulations for supporting learning. Learning, Media, and Technology, 31, 343-358.

11.

Dempsey

J. V.

Haynes

L. L.

Lucassen

B. A.

Casey

M. S.

(2002). Forty simple computer games and what they could mean to educators. Simulation & Gaming, 33, 157-168. doi:10.1177/1046878102332003

12.

Devlin-Scherer

Sardone

(2010). Digital simulation games for social studies classrooms. Clearing House, 83, 138-144. doi:10.1080/00098651003774836

13.

DIFFUSION SIMULATION GAME. (2009). [Developed by Department of Instructional Systems Technology (Online game).] Bloomington. Indiana University. Retrieved from https://www.indiana.edu/~simed/istdemo/

14.

Drake

Goldsmith

Strachan

(2006). A novel approach to teaching teamwork. Teaching in Higher Education, 11, 33-46. doi:10.1080/13562510500400115

15.

Ebner

Holzinger

(2007). Successful implementation of user-centered game-based learning in higher education: An example from civil engineering. Computers & Education, 49, 873-890. doi:10.1016/j.compedu.2005.11.026

16.

Enfield

Myers

R. D.

Lara

Frick

T. W.

(2011). Innovation diffusion: Assessment of strategies within the DIFFUSION SIMULATION GAME. Simulation & Gaming. Retrieved from http://sag.sagepub.com/content/early/2011/05/21/1046878111408024, doi:10.11777/1046878111408024

17.

Faria

A. J.

Wellington

W. J.

(2004). A survey of simulation game users, former-users, and never-users. Simulation & Gaming, 35, 178-207.

18.

Frick

T. W.

Kim

K.-J.

Ludwig

B. M.

Huang

(2003, October). A web simulation on educational change: Challenges and solutions for development. Paper presented at the meeting of the Association for Educational Communication and Technology, Anaheim, CA. Retrieved from http://www.indiana.edu/~tedfrick/aect2003/frick_kim_ludwig_huang.pdf

19.

Gee

J. P.

(2007). What video games have to teach us about learning and literacy. New York, NY: Palgrave Macmillan.

20.

Hall

Wallace

Dossett

(1973). A developmental conception of the adoption process with educational institutions (Report No. 3006). Austin: The University of Texas at Austin, Research and Development Center for Teacher Education.

21.

Hlodan

(2008). Digital games: Learning through play. Bioscience, 58, 791. doi:10.1641/B580905

22.

Kebritchi

(2010). Factors affecting teachers’ adoption of educational computer games: A case study. British Journal of Educational Technology, 41, 256-270. doi:10.1111/j.1467-8535.2008.00921.x

23.

Klopfer

Osterweil

Groff

Haas

(2009). The instructional power of digital games, social networking, simulations, and how teachers can leverage them. Retrieved from http://education.mit.edu/papers/GamesSimsSocNets_EdArcade.pdf

24.

Kriz

(2010). A systemic-constructivist approach to the facilitation and debriefing of simulations and games. Simulation & Gaming, 41, 663-680. doi:10.1177/1046878108319867

25.

Lane

J. L.

Slavin

Ziv

(2001). Simulation in medical education: A review. Simulation & Gaming, 32, 297-314.

26.

Lara

Myers

Frick

T. W.

Aslan

Michaelidou

(2010). A design case: Developing an enhanced version of the DIFFUSION SIMULATION GAME. International Journal of Designs for Learning, 1, 40-54. Retrieved from http://scholarworks/iu/edu/journals/index.php/ijdl/index

27.

Mawdesley

Long

Al-jibouri

Scott

(2011). The enhancement of simulation-based learning exercises through formalised reflection, focus groups and group presentation. Computers & Education, 56, 44-52. doi:10.1016/j.compedu.2010.05.005

28.

Moizer

Lean

(2010). Toward endemic deployment of educational simulation games: A review of progress and future recommendations. Simulation & Gaming, 41, 116-131. doi:10.1177/1046878109359052

29.

Moizer

Lean

Towler

Abbey

(2009). Simulations and games: Overcoming the barriers to their use in higher education. Active Learning in Higher Education, 10, 207-224. doi:10.1177/1469787409343188

30.

Molenda

Rice

J. M.

(1979). Simulation review. Simulation & Games, 10, 459-467.

31.

Moreno-Ger

Burgos

Torrente

(2009). Digital games in eLearning environments. Simulation & Gaming, 40, 669-687. doi:10.1177/1046878109340294

32.

Oertig

(2010). Debriefing in Moodle: Written feedback on trust and knowledge sharing in a social dilemma game. Simulation & Gaming, 41, 374-389. doi:10.1177/1046878108325982

33.

Peters

V. A. M.

Vissers

G. A. N.

(2004). A simple classification model for debriefing simulation games. Simulation & Gaming, 35, 70-84. doi:10.1177/1046878103253719

34.

Rogers

E. M.

(1995). The diffusion of innovations (4th ed.). New York, NY: Free Press.

35.

Rogers

Goodloe

A. H.

(1973). Simulation games as method. Educational Leadership, 30, 729-732. Retrieved from Education Research Complete database

36.

Shaffer

D. W.

(2006). How computer games help children learn. New York, NY: Palgrave Macmillan.

37.

Smith

(2010). The long history of gaming in military training. Simulation & Gaming, 41, 6-19. doi:10.1177/1046878109334330

38.

Sottile

J. M.

Brozik

(2004, January). The use of simulations in a teacher education program: The impact on student development. A critical review. Paper presented at the Hawaii International Conference on Education. Retrieved from http://www.eric.ed.gov/PDFS/ED490383.pdf

39.

Wilson

(2009). Best practices for using games and simulations in the classroom: Guidelines for K-12 educators. Software and Information Industry Association (SIIA). Retrieved from http://www.siia.net/index.php?option=com_docman&task=doc_view&gid=610&tmpl=component&format=raw&Itemid=59