Simulation Gaming Can Strengthen Experiential Education in Complex Infrastructure Systems

Abstract

Background Despite federal directives to strengthen the resilience of critical infrastructure systems, existing education programs have not kept pace with ambitious policy goals. As post-war infrastructure ages, it is increasingly necessary for graduates to master systems thinking to understand the complex and interdependent nature of infrastructure. Whereas in traditional physical science and engineering courses, learning would take place in laboratory exercises, the scale and criticality of infrastructure present obstacles to experimental and experiential learning activities.

Aim This article describes the experience of an educational simulation game, called the LA Water Game , to teach management of ageing water infrastructure as a complex socio-technical system.

Method A total of over 200 participants in 16 workshops completed an introductory lecture, experimental scenario development, experiential game play, and participated in reflective group discussion. Qualitative data was collected during game play and debriefing interviews and was used to assess participant learning outcomes.

Results Participant feedback affirmed that simulation gaming can reinforce the experimental, experiential, and reflective phases of the Kolb Learning Cycle. Subjects displayed cognitive and affective engagement, intrinsic motivation, and often reported improved understanding of complex systems attributes, including interdependencies, feedback loops, nonlinearity, and stochasticity.

Keywords

critical infrastructure complex systems experiential learning simulation games and systems thinking

Introduction

A diverse range of disciplines use simulation games for teaching and training, including: military system operations (Caillois, 1961; Huizinga, 1944; Perla & McGrady, 2011; Sabin, 2012; Smith, 2010), economic systems (Doyle, Radzicki, & Trees, 2008), environmental systems (Rijcken, Stijnen, & Slootjes, 2012; Stave, Beck, & Galvan, 2015), governmental systems (Nishikawa & Jaeger, 2011), public policy (Duke, 2011; Mayer, 2009), disaster management (Kobes, Helsloot, de Vries, & Post, 2010), water resource management (Chew, Zabel, Lloyd, Gunawardana, & Monninkhoff, 2014; Medema, Furber, Adamowski, Zhou, & Mayer, 2016; Rusca, Heun, & Schwartz, 2012; Savic, Morley, & Khoury, 2016), history (Corbeil & Laveault, 2011; Hofstede, De Caluwé, & Peters, 2010), sociology (Greenblat, 1971) and engineering (Potier, Lagarrigue, Lalanne, Lelardeux, & Galaup, 2016). Despite their popularity, the experiential nature of simulations and games makes learning difficult to assess (Chin, Dukes, & Gamson, 2009; Corbeil & Laveault, 2011; Wolfe & Roberts, 1986). Girard, Ecalle, and Magnan (2013) found that the following attributes can promote learning in serious games: a) cognitive and affective engagement, b) intrinsic motivation, c) flow state (Csikszentmihalyi, 2014), and d) stimulation without distraction from learning. Nevertheless, there exists a subset of cases involving complex systems simulation games wherein participants consistently demonstrate their belief in misconceptions that lead to failure (Dörner, 1996).

Unfortunately, when existing education programs and approaches fail to teach students complex systems thinking skills (e.g., Bosch, Nguyen, & Ha, 2014), the consequences extend beyond the sanctuary of simulation games. Real complex systems are typically characterized by non-linearity, interdependencies, feedback loops, and stochasticity (Checkland, 1999; Jolly, 2015; Meadows, 2008; Sterman, 2000). Thus, effective simulation games must draw attention to these qualities. Some of the earliest examples of complex systems games include, a business game (Bellman, Clark, Malcolm, Craft, & Ricciardi, 1957) developed by consultants of the Rand Corporation and Booz, Allen & Hamilton, and the Beer Game (Forrester, 2007) created by professors of the MIT Sloan School of Management. Bellman’s business game represented a business firm as a whole and aimed to reveal interdependent relationships between multiple firms that are competing for a known consumer market. In the Beer Game, reinforcing feedback loops and information delays result in an amplification of order imbalances called the bullwhip effect. The Beer Game has stumped supply chain management students since its creation in the 1960s (Goodwin & Franklin, 1994; Sterman, 1989, 1994). Similarly, a recent study demonstrated that college students were unable to identify complex systems traits exhibited in a non-serious game (Wasserman & Banks, 2017).

This article demonstrates the value of a simulation game as a teaching method for engineering and infrastructure students to learn and develop the systems thinking skills required to support infrastructure resilience (Seager et al., 2017). We developed a simple systems dynamics model to represent the problem of maintaining the quality of the Los Angeles water distribution infrastructure over a 75-year period. Although the technical model is not specific to Los Angeles, the frequency of water main breaks during the severe California drought of 2014 allowed the game facilitators to draw upon popular news articles and events to make game play emotionally more salient for players. The simulation model features all four components of complex systems mentioned above: 1) non-linearity is represented in the ageing of the water system over time, 2) interdependent relationships exist in the way that public opinion is impacted by water rates and water system quality, 3) feedback loops are present in the dependence of funds for maintenance on public opinion, and 4). stochasticity is present in the random occurrence and expense of water main breaks. The resulting LA Water Game was tested in 16 workshops and played by more than 200 individuals. Workshops were organized around the four aspects of the Kolb Learning Cycle: abstraction, experimentation, experience, and reflection (Kolb, 2014; Kolb & Kolb, 2009; McKenna, Yalvac, & Light, 2009).

Game Development and Play

The challenge of decaying infrastructure demands a novel, innovative engineering education approach that engages all four parts of the Kolb Learning Cycle to transform students’ understanding of complex systems (Vanasupa, Stolk, & Herter, 2009). Existing teaching methods that emphasize homework problem sets, lab bench experiments, and experience as classroom discussion are inadequate because the scale and critical nature of infrastructure services prohibits opportunities for students to experiment with real complex infrastructure systems. Current games for engineering education (Bodnar, Anastasio, Enszer, & Burkey, 2016), define learning objectives on technical details and system optimization. Although there are several complex systems instructional games set in ecological systems (i.e. Seager & Selinger, 2009; Spierre, Seager, Selinger, & Sadowski, 2011), infrastructure games remain focused on complicated systems while neglecting complex systems (Poli, 2013).

The LA Water Game creates the opportunity for groups to explore the management of an abstract infrastructure water system as a simulated game, without the risks and consequences of real failures (Figure 1). The game emphasizes infrastructure complexity in relation to public opinion, funding, and the quality of the water system in Los Angeles, California (see Table 1 for full list of game parameters). In the LA Water Game, the participants’ objective is to manage the Los Angeles water distribution system for a 75-year period without violating financial, quality, or public approval constraints.

Figure 1.

Guests from the Office of Naval Research, the Space and Naval Warfare Systems Command (SPAWAR), the Energy Systems Technology and Evaluation Program (ESTEP), and Naval Facilities visited the ASU Decision Theater and played the LA Water Game. Photo credit: Emily Herring

Table 1.

The LA Water Game Parameters.

Variable	Description	Scale	Initial Value
System Quality	Overall system condition for provision of clean water and reduced frequency of pipe breaks	Range: 0% to 100%	100%
Available Funds	Sum of the annual public fees collected less the yearly preventative maintenance allocation and the total emergency repair costs	Millions of dollars	$50 Million
Public Opinion	Rating rises or falls in relation emergency repairs and fee changes	Range: 0% to 100%	100%
Annual Fees	Public fees collected and added to available funds	Millions of dollars per year	$50 Million per year
Emergency Costs	Yearly cost to repair spontaneous leaks and breaks in the water distribution system	Millions of dollars	$0 Million
Maintenance	Yearly amount allocated towards preventative maintenance and system repairs	Millions of dollars	Decision variable
Fee Changes	Percent increase or decrease to the rate of annual public fees collected, compounded annually	Range: ± 0% to 100%	Decision variable

The simulation is programmed in Vensim© (Ventana Systems Inc., 2010) systems dynamic modeling software that allows modelers to intervene at each time step by adjusting decision variables. The game can be facilitated within a 75-minute class period or spread out over several days. Whereas the data depicted in the simulation model is not calibrated to estimate the real maintenance needs of the Los Angeles water distribution system, the curvature of model relationships have been empirically observed. For example, the decay of water system quality proceeds from 100 percent (brand new) to zero percent in a S-shaped curve consistent with that observed in real water distribution systems (e.g., Cromwell et al., 2001; Damodaran, Pratt, Cromwell, Lazo, & David, 2005; Thomson, Flamberg, & Condit, 2013).

Figure 2 shows how the simulation parameters are interrelated. For example, in the absence of maintenance, infrastructure system quality decays exponentially with respect to time and is modeled with a logarithmic function. Since distribution pipes are located under the ground, the public does not observe changes in infrastructure quality and the public approval is not sensitive to quality shifts. Instead, public opinion declines as a result of increasing fees and the occurrence of emergency breaks. The frequency of water main breaks increases as infrastructure quality declines, and the emergency cost of these breaks are modeled with a beta random distribution.

Figure 2.

The influence diagram for the systems dynamic simulation model for the LA Water Game.

Each decision period, players decide on increases or decreases in the rate of fees charged to the public, and how much to spend on maintenance. As participants make decisions regarding public water fees and allocation of maintenance funds, the game facilitators enter their decisions in the Vensim© simulation to update the model. The challenge facing players is that ageing infrastructure requires more maintenance, necessitating increases in fees, and consequently damaging public opinion. Due to its complex nature, participants must learn to manage competing agendas and balance the dynamic relationships between each of game parameters described in Table 1.

Historical Context of the LA Water Game

In January 2014, the governor of California declared a state of emergency in response to a multi-year drought, placing restrictions on water usage, requirements for water use reporting, and calling for residents to reduce water consumption. Nonetheless, six months later, on the day new water restriction measures went into effect, a 90-year old water main near the University of California Los Angeles (UCLA) campus burst, releasing 20 million gallons of water and causing $13 million in damages (Gordon, 2015). While this break was a particularly acute example, a series of prior and subsequent incidents formed an alarming trend.

Operation of the water system is provided by Los Angeles Water & Power, which budgets for its maintenance and repair. However, spontaneous breaks require immediate attention and emergency funding that may exceed available maintenance funds. Whereas the public may bristle at a steady increase in rates for water services and pipe maintenance, in the face of an emergency like the UCLA Flood, funds must be redirected for immediate use and repair. From a financial perspective it may be cheaper to fund these emergency repairs for acute pipe breaks as they arise, rather than pro-actively maintain and replace the ageing components throughout the system – despite the unexpected service interruptions and collateral flooding damage that undermines public confidence in the infrastructure managers.

Complex Systems Thinking Skills Represented in the LA Water Game

The LA Water Game integrates four components of complex systems thinking into its design, providing a concrete experience and touchstone for engineering students to understand the complexities of decisions made regarding infrastructure, as summarized in Table 2.

Table 2.

Complex Systems Dynamics in the LA Water Game.

Dynamic	Manifestation
Interdependency	Physical components of infrastructure (water pipes) are interdependent with social systems (public opinion) and economic systems (fee rates and availability of funding).
Feedback Loops	Water system quality output becomes input for public approval rating; public approval rating affects fee change; fee change affects availability of funds; funds affect system quality (Marlow, Moglia, Cook, & Beale, 2013).
Stochasticity	If the water system quality decreases, the system experiences semi-random emergency breaks to varying degree of damage.
Nonlinearity	Water system quality is nonlinear over time; the increase or decrease in system quality over five year increments does not produce linear, predictable changes in public approval or fees.

In the process of making decisions within the game, participants test hypotheses to understand how the system components are related. Since participants have only two variables to manipulate, they can test all the combinations of their decision variables early on. For example, in the beginning of the game first generation participants often realize that they must test how much money is enough to maintain the quality of water service. Since many participants have no prior experience, first generation participants often decide upon an arbitrary amount to assign to maintenance (e.g., one half the annual fees collected, thus 25 million per year) and their selection of the amount to spend on annual maintenance has a relatively low weighted impact on the other parameters (public opinions and system quality). However, as the game continues and the system ages, later generations must formulate and test additional hypotheses to determine how much investment is required to mitigate rapid declines in quality.

Game Play Sequence and Objectives

Facilitators must be well-versed in the historical context and story of California’s drought and water system problems, as well as the relationships between the parameters described in Table 1. Facilitators control the environment and are tasked with narrating the game, engaging participants, maintaining attention and participation, and integrating additional components of infrastructure complexity into cohort discussions and debate. Gamification relies upon the facilitators’ artful ability to elevate participants’ experience such that they feel emotionally invested in and affected by the outcome and are motivated to achieve success in the form of sustaining public opinion through stable maintenance fees, availability of funds for repair, and acceptable quality levels of the water system.

To begin, facilitators divide the participants into three groups. The three groups represent the first, second, and third generations to inherit the LA water distribution system. Starting with the first generation, each group plays the role of the LA water manager for five sequential turns before passing the role to the next generation. Each turn represents a length of five years in the real world where the managing group is responsible for selecting how much to spend on annual system maintenance, as well as decide whether or not to increase the LA residents’ water fees. The first generation begins the game with a new water distribution system outfitted with clean pipes. They must decide how to set maintenance fee rates, accrue or spend available funds, and maintain the quality of the water system during their tenure. Since the game moves in intervals of five years, so each generation has a maximum of five consecutive decision-making opportunities. After 25 simulated years, the first generation retires, and second generation assumes control for the next 25 years, and the third generation for the remaining 25 years.

If any of the three performance constraints are violated, the generation managing the system is dismissed – i.e., no longer permitted to make game decisions. Facilitators then have the option of promoting the next generation early or recalling a prior generation from retirement. Thus, participants who desire to complete all 25 years of their generation’s service must avoid low public approval ratings, maintain a minimum quality condition, and avoid bankruptcy.

Since the game is analogous to the LA system, facilitators have access to a portfolio of real newspaper articles and videos which they may choose to interject at pivotal moments of the game. For example, when a generation raises ratepayer fees, facilitators can evoke an emotional response by pulling up a LA Times article on local billing complaints (Lopez, 2015), while implying that the complaints described in the article are the result of the decision to increase fees. When small emergencies occur in the game, news stories about minor pipe breaks in LA county (Mejia, 2015) and a NBC Los Angeles article on a water main break that sent asphalt chunks flying (Arambulo & Arvin, 2015) are available. Additional articles feature more severe events, such as the UCLA flood covered by USA Today, NBC News, and other sources (Gordon, 2015; Loyd & Guinyard, 2014; Moloshok, 2014; Woodyard, 2014) or complaints from 60,000 overcharged customers (Ezzeddine & Vara, 2014). Although game variables will not correspond to the details of these articles, by including stories and vivid images of the real LA county failures throughout the game, the students’ emotional experience may correspond to the real predicament faced by LAWPD water managers.

The LA Water Game teaches participants about a specific kind of feedback loop that is often a cause for poor decision-making – a time lag. Because participants learn the mechanics of the system as they play, they must become responsible for operation constraints in real time, learning on the job how sensitive their system is to the decisions they make. Participants, particularly those in the first generation, do not have all the required information when asked to make their initial decisions, and yet they must decide on maintenance fund allocation and public fee amounts. In addition, the system is not nearly as sensitive to changes in the beginning of the games as it becomes at later points in the game, for the second and third generations. Thus, participants confront a time lag feedback loop because they often learn how the infrastructure system responds to changes after the system is already failing. By that time, it is often too late for the present generation to recover from poor management decisions, which are inherited by future generations.

Workshop Debriefing and Assessment Methodology

Procedures

The LA Water Game was facilitated in 16 workshops for over 200 participants. Each workshop contained between 7 and 27 participants. Participants consisted of heterogeneous groups of undergraduate and graduate students, faculty, and active-duty military personnel. Students’ backgrounds represented a diverse range of disciplines, including engineering and non-engineering majors such as, industrial design, data analytics, computer science, and business. During each workshop, participants completed an introductory lecture, experimental scenario development, and then were divided into three groups for experiential game play. In all 16 workshops, participants engaged in reflective debriefing, led by the game facilitator during and post-game play.

Data was gathered from participant observations (Robson & McCartan, 2002) and debriefing interviews (Lederman, Stewart, & Golubow, 2001). Upon evaluation, workshop themes emerged and were categorized by two attributes that Girard et al. (2013) identified with learning in serious games: a) cognitive and affective engagement, b) intrinsic motivation.

Qualitative data collection

For assessing participants throughout game-play, Chin et al. (2009), recommends the qualitative data collection methods detailed in classic works such as Writing Ethnographic Fieldnotes (Emerson Robert, Fretz Rachel, & Shaw Linda, 1995) or In the Field (Smith & Kornblum, 1989). In addition to these sources, this study used qualitative methods (Tracy, 2012) for analyzing qualitative data gathered from participant observations and debriefing interviews. Participant observational data was structured by the framework proposed by Bogdewic (1992), which covers space, actors, activities, objects, acts, events, time, goals, and feelings. During debriefing interviews, participants were brought together for guided recall, reflection, and analysis.

Debriefing

Historically, debriefing interviews have been used by military campaigns and war gaming activities (Lederman, 1992). Among simulation gaming methodology literature, debriefing is generally accepted for aiding the learning process (Crookall, 2010; Lederman, 1992) and facilitating reflection (Thiagarajan, 1992). Interim and post-game debriefing sessions were incorporated in the LA Water Game to assist participants to relate their experiences from the game to those of real-life situations. Debriefing conversations were held at two phases of the game. Intermediate debriefing was conducted when a generation passed on the LA water manager role to the sequential generation. During intermediate debriefing, the facilitator asked the earlier generation to reflect on the decisions they made while they were the LA manager and to self-evaluate their job performance. After the game, the facilitator led a post-game joint debriefing session in which all participants were asked to analyze the evolution of the game and to relate their performance to a real-life situation. Participants then reflected on the differing perspectives offered by the three generations. The facilitator prompted participants to discuss alternative courses of action and asked participants to hypothesize the potential consequences of these options.

Open-ended questions were designed to induce reflection and analysis of the events and processes in the simulation game, their contributions to these processes, and to develop their systems thinking skills relevant to other real-life situations. These questions were based off of Thiagarajan’s (1992) 7-point debriefing sequence, including recollection of what happened and their feelings, hypothesize cause-effect relationships, and determine real-world relevance or further applications of principals learned during game play. In accordance with Peters and Vissers (2004) exploratory/development debriefing classification, the LA Water simulation game provides a context for exploration and experimentation while the facilitator guides participants to use the opportunities provided by the simulation game to explore boundary conditions, and a chance to change these conditions if necessary.

Findings From the LA Water Game Pilot Workshops

The game was evaluated as a tool for teaching complex systems thinking skills with particular focus on two assertions that effective educational games: 1) foster cognitive and affective engagement, and 2) increase intrinsic motivation that allows for longer training periods. For each finding highlighted, we provide examples from qualitative participant observations gathered during game play or debriefing interviews. Table 3 describes the diverse set of student and researcher teams with whom we have tested the LA Water Game in 2016 and 2017.

Table 3.

The LA Water Game Piloted Workshops.

ID	Workshop	Location	Date
1	Systems Engineering Class	ASU campus, Tempe, AZ	04/26/2016
2	Naval Postgraduate School (NPS)	NPS campus, Monterey, CA	05/09 - 05/11/2016
3	Thunderbird Executive Education Crisis Management Instruction & Decision Theater team	ASU campus, Tempe, AZ	06/01/2016
4	Navy Enterprise Partnership Teaming with Universities for National Excellence Veteran student	Decision Theater, Tempe, AZ	07/15/2016
5	Resilient Infrastructure Graduate Research Seminar	ASU campus, Tempe, AZ	09/06/2016
6	Water Sustainability Climate Research Team	Decision Theater, Tempe, AZ	09/09/2016
7	ASU Luminosity Lab	Decision Theater, Tempe, AZ	10/21/2016
8	Decision Theater Coding Team	Decision Theater, Tempe, AZ	11/23/2016
9	Systems Engineering Class (subset)	ASU campus, Tempe, AZ	01/09/2017
10	Office of Naval Research, SPWAR, ESTEP, and NAVFAC visitors	Decision Theater, Tempe, AZ	01/11/2017
11	Systems Engineering Class (full class)	ASU campus, Tempe, AZ	03/02/2017
12	ASU Spirt of Service Scholars Workshop	Decision Theater, Tempe, AZ	03/25/2017
13	Infrastructure Socio-Eco-Tech Seminar	Decision Theater, Tempe, AZ	03/27/2017
14	Infrastructure Grad. and Undergraduate Students	Purdue, West Lafayette, IN	04/19/2017
15	Frontiers of Resilience Workshop	George Mason University	05/11/2017
16	Naval Postgraduate School (NPS)	NPS campus, Monterey, CA	06/01 – 06/03/2017

Couple Cognitive Engagement With Affective Engagement and Motivation

Annetta, Minogue, Holmes, and Cheng (2009) suggested that cognitive engagement in training, coupled with affective engagement and motivation (Baker, D’Mello, Rodrigo, & Graesser, 2010; Knight et al., 2010; Sitzmann, 2011) is correlated with learning in serious games. During game workshops, we observed the following signs of cognitive engagement among players: paying attention, experimenting, deliberating, and negotiating. All participant names have been changed to protect their privacy.

Paying attention

Prior to the start of every game, facilitators opened the room for questions. Participants asked questions about the game rules and how to interpret the game progress metrics and graphs. We observed participants visually track game progress and dynamics on the projected screen. Participants demonstrated alert behaviors, learning forward in their chairs, pointing at the screen graphics as they changed, and directly incorporating reported data into their conversations. Figure 3 shows how each generation grew more engaged during their turn as the LA water manager in workshop 11. During the first generation’s turns (Figure 3A), participants from the second and third generations listened attentively to the first generation as they deliberated, repeating spoken words, phrases, and numerical figures amongst their own groups. As facilitators shared news articles and game updates, participants asked clarifying questions and nodded along with facilitators’ directions.

Figure 3.

A, B, and C show the 1^st, 2^nd, and 3^rd generations from workshop 11. The 1^st generation (A) seated themselves across at two tables with just one student running the calculator simulation. In the 2^nd generation (B), students are leaning in to the discussion. By the 3^rd generation’s turn (C), students are standing crowded around a table immersed in heated debate. (D) One 3^rd generation student got up from the group table, stood at the facilitators’ laptop for immediate access to the simulation controls, and proceeded to tab through the simulation graphs.

Experimenting

Participants attempted various combinations of maintenance investment and public service fees to maintain public opinion over the course of the game. When given access to the single-player version of the LA Water Game, participants from the second and third generations tracked the game progress with their personal computing devices and simulated multiple future scenarios to anticipate future events and test the efficacy of different response strategies for when their turn arose.

Deliberating

Participants engaged in conversations, debates, and problem solving with their own generation and other generations. First generation participants often spent the most time to plan out their decisions, attempting to forecast anticipated effects of a given fee amount or maintenance investment. By the third generation’s turn, the system was often in a state of disrepair, the public approval rating was below 25 percent and the city budget was in debt. Thus, third generation participants often spent the least amount of time forecasting and calculating, given that they had less freedom to operate within their given constraints.

Negotiating

Participants who inherited a broken system often negotiated with game facilitators. In workshop 10, a second-generation participant, declared that the second generation would not accept the role of the LA water manager unless facilitators guaranteed they would be given at least 30 years without fear of being fired. By the end of the 30 years, this group was able to raise water fee rates, invest in the infrastructure, and eventually public approval (which remained below the 25 percent threshold) began to increase. While this interaction might appear to bend the game’s rules, the player’s negotiation was acceptable since this interaction helped the group explore the boundaries and constraints of the system – a skill critical to understanding complexity.

In addition to the listed indicators of cognitive engagement, participants also exhibited signs of affective engagement and motivation. Starting each workshop, participants were optimistic and confident in their ability to accomplish the game requirements. They laughed, smiled, and sat leaning forward, expectant and ready to participate. By the time the second generation inherited the manager position, participants began to demonstrate physical indications of stress. They sat with their arms folded, brows furrowed, and their chins rested on their hand while contemplating the data on the screen. Participants expressed anxiety after the sudden emergency costs and by the non-linear decays in system quality, often letting out strained laughs and using common expressions to convey resigned, emotional responses to decisions made both by their own and other generations.

Below is a workshop excerpt of participant observations that conveys affective engagement between two students through their dialog and physical gestures. The participants’ conversation highlights their learning process and their iterative approach to decision making that helps them to understand the system’s underlying feedback loops.

Tyler:

Start super high (increasing the fee change by a large amount early on in the game). Everyone is going to hate you.

Zach:

So… drop the maintenance? [entering the decision]

Tyler:

There goes your public opinion.

Zach:

Yeah. It is already gone. [laughing]

Tyler:

Now, drop it by 20% every decision cycle because they (the public) already hate us. So let’s just see what happens.

Zach:

Exactly. Oh wait. You mean the fee (change) or maintenance?

Tyler:

The–the maintenance price and then the fee change is minus 20% but the maintenance should be the same.

Zach:

How about minus 10%?

Tyler:

Ok. Sure. [throwing hands in concession]

Zach:

Minus 0.1 [mumbling while entering the fee change decision]

Tyler:

Just want to see what the results are.

Zach:

This is per year, so the–that actually will make a really big impact. [hypothesizing before inputting the decision]

Tyler:

Ohhhh… ok.

Zach:

[clicking forward] Wow. You see like how here. [pointing at screen]

Tyler:

See the public opinion goes right back up… [pointing at screen]

Zach:

Yeah.

Tyler:

…and you dropped the fee just a little bit. And we are still at a ridiculous amount of money. You recover just about–about half of what you lost.

Zach:

Yeah. [laughing]

Tyler:

Right? [laughing] But, look at our funds![grinning and laughing together]

Zach:

Dude! Yeah!

Tyler:

[laughing, throwing hands upwards into a V-victory position]

Zach:

Look at the rate.

Tyler:

What a bank account.

Zach:

Look at the rate. [repeating himself in a louder voice to be heard over Tyler’s excited laughing]

Zach:

Oh my god.

Tyler:

[pointing at screen] Do it again. Let’s see what happens the next time.

The above transcription exemplifies two participants displaying signs of cognitive and affective engagement throughout game play. From the onset, Tyler and Zach experiment with a high fee change to increase annual revenue from fees. When Tyler and Zach realize they have an exorbitant amount of funds, they decided to reduce the public fees and negotiate the magnitude of the fee change. Both participants were motivated and expressed affective engagement through laughing and hand gesturing.

Intrinsic Motivation

The simulation game encouraged participants to continue practicing for longer than a traditional lecture period. After students played through three generations of the game, many participants requested to restart the simulation and play the game over again. The majority of participants expressed dissatisfaction with the first game results and some students insisted on playing the game several more times. When time allowed, facilitators agreed to the students’ requests to replay the game. Of the 16 workshops, 10 of the workshops had time to play through multiple attempts. Because it took a full 75-minute class period to play one round of the game (including the time required to introduce and debrief the game), some students, such as those in workshops 2 and 16, were given the opportunity to repeat the game during a second class period. With each voluntary repetition, students devoted additional time to experimenting with the complex system components (non-linearity, stochasticity, interdependencies, and feedback loops) featured in the game.

We believe that students requested to play the game over because the game capitalized on intrinsically motivating features and allowed students to work towards a meaningful purpose, while pursuing mastery and autonomy (Garris, Ahlers, & Driskell, 2002; McGonigal, 2011; Pink, 2011). Students’ disciplinary backgrounds may also have correlated with the purpose or goals they pursued as the LA water manager, as indicated in the following anecdote.

In workshop 7, participants included graduate and undergraduate student researchers representing academic disciplines such as computer science, chemical and electrical engineering, accounting, industrial design, business, and data analytics. Members of the first generation, led by an industrial designer and business major, made small investments in maintenance to avoid negative public opinion toward fee changes. When the water system experienced its first stochastic break, ratings for public opinion plummeted as the system required an increased fee change to fix the break and save funds for future maintenance. The first generation invested only enough to ensure that the quality of the system was acceptable until their retirement. In their final decision, the industrial designer and business students decided to decrease the fee change and “give money back to the public”, thus making the public approval rating rise, rather than stockpile for future generational maintenance.

In contrast, the second generation was led by a chemical engineering student who was eager to take on management of the LA water system. This student argued for increases in fees and maintenance spending. However, public opinion ratings declined so much that his generation was fired.

Following the game workshop, the chemical engineering student insisted on repeatedly playing the LA Water Game to achieve 100% quality for the full 75-year period. To accomplish this, the engineer and industrial design students played the game on a facilitator’s laptop for an additional hour after the workshop ended. When asked about why the quality constraint was important, he responded, “I’m a chemical engineer. We can’t have emergencies at a nuclear power plant. The public’s opinions and complaints don’t matter if they aren’t alive.” This student prioritized public safety over public opinion, which may be a result of internalizing lessons in engineering ethics learned in other courses. Nevertheless, students who requested to continue playing the game until achieving a satisfactory outcome are exhibiting mastery-based learning (Bekki, Dalrymple, & Butler, 2012).

Reflections

After processing qualitative data from participant observations and debriefing sessions, a common pattern was identified throughout the workshops. Prior to playing the game, participants were not aware of the reinforcing loops inherent to the system. In the first generation, participants expressed curiosity and eagerness to succeed, as was demonstrated by their questions and physical demeanor. As each generation played the role of the manager, it was only through a trial and error process that participants were able to test hypothesizes, receive feedback, and define the system’s underlying complex relationships.

After each decision was made, the simulation advanced forward 5 years, and feedback was presented to the group in the form of data visualizations. Through this, participants were able to identify patterns in the system – one participant from a second generation remarked that, “the public opinion seems more affected by rate increases [fee changes] than to the actual rate [fees paid annually].”

Debriefing interview discussions showed that the players had identified the complex relationships presented in the game. Through hypothesis testing, participants were able to identify the interdependencies between game variables, the nonlinear nature of infrastructure deterioration, stochastic emergency breaks, and the reinforcing feedback loops within the game. When asked what decisions they would make if they played again, one participant suggested that, “The higher you get your rate [annual fees collected] as soon as possible, the better it is, because you just flatten it [fee change] off to zero and you keep the rate [annual fees collected], which is already high, for the rest of the game.”

Two examples illustrate the eagerness of participants to go beyond the parameters of the game to realize additional layers of complexity regarding water infrastructure. During the workshop 2 debrief, NPS participants described new strategies for balancing competing complex dynamics. As a result of their experience playing the simulation game, they recommended a modification of the computational influence diagram. They requested that a new decision variable be added which would allow players to allocate funds to educating the public on water infrastructure. These participants argued that by educating the public, the public would better understand how their fees are being used to maintain the quality of the distribution system. As a result, the public upset might be less sensitive to rate increases.

A similar theme emerged in the workshop 5 debrief, in which participants in the Systems Engineering class acknowledged that they began playing the game with the assumption that the public would pay for a well-maintained system. However, through their participation in the simulation game this misconception was exposed through the relationship between public opinion, funding, and system quality. Participants shared their realization that the public often takes for granted both the existence of infrastructure systems and their successful function. Without technical expertise, the public is unaware of the maintenance requirements of the infrastructure systems which are integral to daily life. Like the NPS participants, the engineering participants considered how they might engage and educate the public in the process of maintaining infrastructure.

By highlighting the most critical elements of the system, participants were able to recognize the underlying, and sometimes counterintuitive, relationships governing system interactions. Participants commented on the transferability of feedback loops connecting public funding, public opinion, and system quality to other systems, particularly transportation and road maintenance. Participants observed non-linear relationships between allocation of funds toward maintenance and system quality in the first generation and admitted to being lulled into a false sense of correlation between these system dynamics. However, as later generations inherited the ageing system, the relationship between maintenance and system quality became nonlinear, requiring a more adaptive approach to fee changes and system maintenance all while managing public opinion. Participants described their reactions to stochastic water breaks and acknowledged the complex relationship between these spontaneous bursts, public opinion, and availability of funding for repairs.

Conclusion

Experiencing the LA Water Game gives participants a foundational understanding of the complex nature of infrastructure management and the systems thinking approach that is required for long-term infrastructure management. By engaging all four stages of the Kolb Learning Cycle, participants learn abstract conceptualizations of complex systems, experiment with parameters that leverage changes in the social-technical system, gain concrete experiences rooted in affective responses to goal attainment (or failure), and reflectively observe both their own actions and those of other generations. The game also demonstrates complexity around water system infrastructure. Although the simulation game itself is a cartoon-like abstraction of the real system, it provides a foundation for students to begin developing complex systems thinking skills which are critical for managing interdependent infrastructure systems.

Footnotes

Acknowledgements

The authors would like to thank the Arizona State University Decision Theater for use of their facilities and a special thanks to Madeline Sawyer and Greg Moon for their contributions to developing the model and software that drive the LA Water Game. This research was approved by the Arizona State University Institutional Review Board.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the Office of Naval Research NEPTUNE program, and by the National Science Foundation under Grant Award 1323401 and 1441352.

Author Biographies

Lauren R. McBurnett, is the systems engineering researcher for the Decision Center for Educational Excellence at Arizona State University. In addition to her contributions designing, developing, and facilitating the LA Water Game, she has designed simulations and games for training, education, and research teams while working as a research analyst for the Center for Naval Analysis FFRDC in Arlington, VA.

Contact: lauren.mcburnett@asu.edu

Margaret M. Hinrichs, PhD, is a researcher in the School for the Future of Innovation in Society and a program manager at the Decision Theater at Arizona State University. Her work focuses on the interpersonal communication processes of organizations, as well as the uses of models as boundary objects to engage diverse stakeholders in collaborative decision making.

Contact: margaret.hinrichs@asu.edu

Thomas P. Seager, PhD, is an Associate Professor in the School of Sustainable Engineering & the Built Environment at Arizona State University. He leads research teams working at the boundaries of engineering and social science to understand innovation for resilient infrastructure systems, including the life-cycle environmental consequences of emerging energy technologies, novel approaches to teamwork and communication in socio-technical integrative settings, and game play to teach creativity, and systems thinking in engineering education.

Contact: thomas.seager@asu.edu

Susan Spierre Clark, PhD, is the Policy, Planning and Sustainability Specialist at the Institute for Research and Education in eNergy, Environment and Water (RENEW) at the University at Buffalo. Her work focuses on decision-making for improving the sustainability and resilience of critical infrastructure systems to climate change.

Contact: sclark1@buffalo.edu

References

Annetta

L. A.

Minogue

Holmes

S. Y.

Cheng

M. T.

(2009). Investigating the impact of video games on high school students’ engagement and learning about genetics. Computers & Education, 53(1), 74-85. doi:10.1016/j.compedu.2008.12.020

Arambulo

Arvin

(2015, November 12). Water main break debris shatters SUV window. NBC Los Angeles. Retrieved from https://www.nbclosangeles.com/news/local/Multiple-Water-Main-Breaks-Flood-SoCal-Streets-346635902.html

Baker

R. S.

D’Mello

S. K.

Rodrigo

M. M. T.

Graesser

A. C.

(2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223-241. doi:10.1016/j.ijhcs.2009.12.003

Bekki

J. M.

Dalrymple

Butler

C. S.

(2012, October). A mastery-based learning approach for undergraduate engineering programs. In Frontiers in Education Conference (FIE), 2012 (pp. 1-6). IEEE. doi:10.1109/FIE.2012.6462253

Bellman

Clark

C. E.

Malcolm

D. G.

Craft

C. J.

Ricciardi

F. M.

(1957). On the construction of a multi-stage, multi-person business game. Operations Research, 5(4), 469-503. doi:10.1287/opre.5.4.469

Bodnar

C. A.

Anastasio

Enszer

J. A.

Burkey

D. D.

(2016). Engineers at play: Games as teaching tools for undergraduate engineering students. Journal of Engineering Education, 105(1), 147-200. doi:10.1002/jee.20106

Bogdewic

S. P.

(1992). Participant observation. In Crabtree

B. F.

Miller

W. L.

(Eds.), Research methods for primary care, Vol. 3. Doing qualitative research (pp. 45-69). Thousand Oaks, CA: Sage. https://us.sagepub.com/en-us/nam/doing-qualitative-research/book9279#features .

Bosch

Nguyen

(2014). Can advancements in economic and managerial practice be achieved without systems thinking education as the foundation? Business Systems Review, 3(2), 33-53. doi:10.7350/BSR.D07.2014

Caillois

(1961). Man, play, and games. University of Illinois Press. Retrieved from https://www.press.uillinois.edu/books/catalog/64rcw3ts9780252070334.html

10.

Checkland

(1999). Systems thinking. In Rethinking management information systems (pp. 45-56). Retrieved from http://www.research.lancs.ac.uk/portal/en/publications/systems-thinking(1b037eb1-a434-4001-a469-001497291a11).html

11.

Chew

Zabel

Lloyd

G. J.

Gunawardana

Monninkhoff

(2014). A serious gaming approach for serious stakeholder participation. Retrieved from https://academicworks.cuny.edu/cc_conf_hic/174

12.

Chin

Dukes

Gamson

(2009). Assessment in simulation and gaming: A review of the last 40 years. Simulation & Gaming, 40(4), 553-568. doi:10.1177/1046878109332955

13.

Corbeil

Laveault

(2011). Validity of a simulation game as a method for history teaching. Simulation & Gaming, 42(4), 462-475. doi:10.1177/1046878108325451

14.

Cromwell

Nestel

Albani

Paralez

Deb

Grablutz

(2001). Financial and economic optimization of water main replacement programs. Denver, CO: AWWA Research Foundation. Retrieved from http://www.waterrf.org/Pages/Projects.aspx?PID=462

15.

Crookall

(2010). Serious games, debriefing, and simulation/gaming as a discipline. Simulation & Gaming, 41(6), 898-920. doi:10.1177/1046878110390784

16.

Csikszentmihalyi

(2014). Toward a psychology of optimal experience. In Flow and the foundations of positive psychology (pp. 209-226). Springer Netherlands. doi:10.1007/978-94-017-9088-8_14

17.

Damodaran

Pratt

Cromwell

Lazo

David

(2005). Customer acceptance of water main structural reliability. American Water Works Association. Retrieved from https://books.google.com/books?hl=en&lr=&id=awru2Zpy_A8C&oi=fnd&pg=PP11&dq=Damodaran,+N.,+Pratt,+J.,+Cromwell,+J.,+Lazo,+J.,+%26+David,+E.+(2005).+Customer+Acceptance+of+Water+Main+Structural+Reliability.+American+Water+Works+Association.&ots=BdvaMlbWmr&sig=p6Tkir5eJ-4uGMDCxOO1tJPhfkY#v=onepage&q&f=false

18.

Dörner

(1996). The logic of failure: Recognizing and avoiding error in complex situations. Basic Books. Retrieved from https://www.basicbooks.com/titles/dietrich-dorner/the-logic-of-failure/9780201479485/

19.

Doyle

Radzicki

Trees

(2008). Measuring change in mental models of complex dynamic systems. Complex Decision Making, 269-294. doi:10.1007/978-3-540-73665-3_14

20.

Duke

R. D.

(2011). Origin and evolution of policy simulation: A personal journey. Simulation & Gaming, 42, 342-358. doi:10.1177/1046878110367570

21.

Emerson Robert

Fretz Rachel

Shaw Linda

. (1995). Writing ethnographic fieldnotes. Chicago, IL: University of Chicago. doi:10.7208/chicago/9780226206851.001.0001

22.

Ezzeddine

Vara

(2014, March 11). DWP website for overbilling complaints. NBC Los Angeles. Retrieved from https://www.nbclosangeles.com/news/local/Site-Set-Up-for-DWP-Overcharges-248466041.html

23.

Forrester

J. W.

(2007). System dynamics—A personal view of the first fifty years. System Dynamics Review, 23(2-3), 345-358. doi:10.1002/sdr.382

24.

Garris

Ahlers

Driskell

J. E.

(2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467. doi:10.1177/1046878102238607

25.

Girard

Ecalle

Magnan

(2013). Serious games as new educational tools: How effective are they? A meta-analysis of recent studies. Journal of Computer Assisted Learning, 29(3), 207-219. doi:10.1111/j.1365-2729.2012.00489.x

26.

Goodwin

J. S.

Franklin

S. G.

(1994). The beer distribution game: Using simulation to teach systems thinking. Journal of Management Development, 13(8), 7-15. doi:10.1108/02621719410071937

27.

Gordon

(2015, July 9). UCLA claims $13 million in flood damage from water line break. Los Angeles Times. Retrieved from http://www.latimes.com/local/education/la-me-ln-ucla-flood-20150709-story.html

28.

Greenblat

C. S.

(1971). Simulations, games, and the sociologist. The American Sociologist, 6(2), 161-164. Retrieved from https://www-jstor-org-443.web.bisu.edu.cn/stable/27701744

29.

Hofstede

G. J.

De Caluwé

Peters

(2010). Why simulation games work-in search of the active substance: A synthesis. Simulation & Gaming, 41(6), 824-843. doi:10.1177/1046878110375596

30.

Huizinga

(1944). 2003. Homo Ludens: A study of the play-element in culture. doi:10.2307/2087716

31.

Jolly

R. D.

(2015). Systems thinking for business: Capitalize on structures hidden in plain sight. Systems Solutions Press. Retrieved from http://www.worldcat.org/title/systems-thinking-for-business-capitalize-on-structures-hidden-in-plain-sight/oclc/905232880

32.

Knight

J. F.

Carley

Tregunna

Jarvis

Smithies

de Freitas

Mackway-Jones

(2010). Serious gaming technology in major incident triage training: A pragmatic controlled trial. Resuscitation, 81(9), 1175-1179. doi:10.1016/j.resuscitation.2010.03.042

33.

Kobes

Helsloot

de Vries

Post

(2010). Exit choice, (pre-) movement time and (pre-) evacuation behaviour in hotel fire evacuation—Behavioural analysis and validation of the use of serious gaming in experimental research. Procedia Engineering, 3, 37-51. doi:10.1016/j.proeng.2010.07.006

34.

Kolb

A. Y.

Kolb

D. A.

(2009). The learning way meta-cognitive aspects of experiential learning. Simulation & Gaming, 40(3), 297-327. doi:10.1177/1046878108325713.

35.

Kolb

D. A.

(2014). Experiential learning, Experience as the source of learning and development. FT Press. doi:10.1002/job.4030080408

36.

Lederman

L. C.

(1992). Debriefing: Toward a systematic assessment of theory and practice. Simulation & Gaming, 23(2), 145-160. doi:10.1177/1046878192232003

37.

Lederman

L. C.

Stewart

L. P.

Golubow

(2001). Using debriefing interviews to collect qualitative data on dangerous drinking: A case study. Communication Quarterly, 49(2), Q7. Retrieved from https://search.proquest.com/openview/334cae918210e6595cd96e309bec3aef/

38.

Lopez

(2015, May 30). DWP complaints keep pouring in. Los Angeles Times. Retrieved from http://www.latimes.com/local/california/la-me-lopez-bradnmary-20150531-column.html

39.

Loyd

Guinyard

(2014, July 30). 20M Gallons of Water Lost in UCLA Main Break. NBC Los Angeles. Retrieved from https://www.nbclosangeles.com/news/local/Water-Main-Break-Floods-UCLA-Campus-Sunset-Boulevard-Closed-269200881.html

40.

Marlow

D. R.

Moglia

Cook

Beale

D. J.

(2013). Towards sustainable urban water management: A critical reassessment. Water research, 47(20), 7150-7161. doi:10.1016/j.watres.2013.07.046

41.

Mayer

I. S.

(2009). The gaming of policy and the politics of gaming: A review. Simulation & Gaming, 40(6), 825-862. doi:10.1177/1046878109346456

42.

McGonigal

(2011). Reality is broken: Why games make us better and how they can change the world. Penguin. doi:10.5860/choice.49-0680

43.

McKenna

A. F.

Yalvac

Light

G. J.

(2009). The role of collaborative reflection on shaping engineering faculty teaching approaches. Journal of Engineering Education, 98(1), 17-26. doi:10.1002/j.2168-9830.2009.tb01002.x

44.

Meadows

D. H.

(2008). Thinking in systems: A primer. Chelsea Green. Retrieved from https://www.chelseagreen.com/product/thinking-in-systems/

45.

Medema

Furber

Adamowski

Zhou

Mayer

(2016). Exploring the potential impact of serious games on social learning and stakeholder collaborations for transboundary watershed management of the St. Lawrence River Basin. Water, 8(5), 175. doi:10.3390/w8050175

46.

Mejia

(2015, July 9). Two water main breaks in Hollywood cause street damage and road closures. Los Angeles Times. Retrieved from http://www.latimes.com/local/lanow/la-me-ln-water-main-breaks-hollywood-20150709-story.html

47.

Moloshok

(2014, July 30). Surf’s up! Water main break floods UCLA campus. NBC News. Retrieved from https://www.nbcnews.com/slideshow/surfs-water-main-break-floods-ucla-campus-55755906

48.

Nishikawa

K. A.

Jaeger

(2011). A computer simulation comparing the incentive structures of dictatorships and democracies. Journal of Political Science Education, 7(2), 135-142. doi:10.1080/15512169.2011.564915

49.

Perla

P. P.

McGrady

E. D.

(2011). Why wargaming works. Naval War College Review, 64(3), 111. Retrieved from https://search.proquest.com/openview/12a6fef640e0b83bf2c7d4e1fdff6ab0/1?pq-origsite=gscholar&cbl=34989

50.

Peters

V. A.

Vissers

G. A.

(2004). A simple classification model for debriefing simulation games. Simulation & Gaming, 35(1), 70-84. doi:10.1177/1046878103253719

51.

Pink

D. H.

(2011). Drive: The surprising truth about what motivates us. Penguin. doi: 10.1002/casp.1085

52.

Poli

(2013). A note on the difference between complicated and complex social systems. Cadmus, 2(1), 142. Retrieved from http://www.cadmusjournal.org/files/pdfreprints/vol2issue1/reprint-cj-v2-i1-complex-vs-complicated-systems-rpoli.pdf

53.

Potier

Lagarrigue

Lalanne

Lelardeux

C. P.

Galaup

(2016, October). Undesigned cooperation within a serious game observations during a mechanical engineering course. In International conference on Collaboration Technologies and Systems (CTS), 2016 (pp. 143-147). IEEE. doi:10.1109/CTS.2016.0041

54.

Rijcken

Stijnen

Slootjes

(2012). “SimDelta”—Inquiry into an internet-based interactive model for water infrastructure development in The Netherlands. Water, 4(2), 295-320. doi:10.3390/w4020295

55.

Robson

McCartan

. (2002). Real world research (2nd ed.). Malden: Blackwell. Retrieved from https://www.wiley.com/en-us/Real+World+Research%2C+4th+Edition-p-9781118745236

56.

Rusca

Heun

Schwartz

(2012). Water management simulation games and the construction of knowledge. Hydrology and Earth System Sciences, 16(8), 2749. doi:10.5194/hess-16-2749-2012

57.

Sabin

(2012). Simulating war: Studying conflict through simulation games. A&C Black. Retrieved from https://www.bloomsbury.com/us/simulating-war-9781441185587/

58.

Savic

D. A.

Morley

M. S.

Khoury

(2016). Serious Gaming for Water Systems Planning and Management. Water, 8(10), 456. doi: 10.3390/w8100456

59.

Seager

T. P.

Clark

S. S.

Eisenberg

D. A.

Thomas

J. E.

Hinrichs

M. M.

Kofron

Alderson

D. L.

(2017). Redesigning resilient infrastructure research. In Resilience and risk (pp. 81-119). Dordrecht, The Netherlands: Springer. doi: 10.1007/978-94-024-1123-2_3

60.

Seager

T. P.

Selinger

(2009). Experiential teaching strategies for ethical reasoning skills relevant to sustainability. In IEEE International Symposium on Sustainable Systems and Technology, 2009. ISSST’09 (pp. 1-6). IEEE. doi:10.1109/ISSST.2009.5156721

61.

Sitzmann

(2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel psychology, 64(2), 489-528. doi:10.1111/j.1744-6570.2011.01190.x

62.

Smith

Kornblum

(1989). In the field (3rd ed.). New York, NY: Praeger. Retrieved from https://trove.nla.gov.au/work/16601316?selectedversion=NBD6471102

63.

Smith

(2010). The long history of gaming in military training. Simulation & Gaming, 41(1), 6-19. doi:10.1177/1046878109334330

64.

Spierre

S. G.

Seager

T. P.

Selinger

Sadowski

(2011, May). Using non-cooperative games to simulate ethical tensions in climate policy negotiations. In 2011 IEEE International Symposium on Sustainable Systems and Technology (ISSST) (pp. 1-5). IEEE. doi:10.1109/ISSST.2011.5936882

65.

Stave

K. A.

Beck

Galvan

(2015). Improving learners’ understanding of environmental accumulations through simulation. Simulation & Gaming, 46(3-4), 270-292. doi:10.1177/1046878114531764

66.

Sterman

J. D.

(1989). Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Management Science, 35(3), 321-339. doi:10.1287/mnsc.35.3.321

67.

Sterman

J. D.

(1994). Learning in and about complex systems. System Dynamics Review, 10(2-3). 291-330. doi:10.1002/sdr.4260100214

68.

Sterman

J. D.

(2000). Business dynamics: Systems thinking and modeling for a complex world (No. HD30. 2 S7835 2000). Retrieved from http://www.sidalc.net/cgi-bin/wxis.exe/?IsisScript=BAC.xis&;method=post&formato=2&cantidad=1&expresion=mfn=057031

69.

Thiagarajan

(1992). Using games for debriefing. Simulation & Gaming, 23(2), 161-173. doi:10.1177/1046878192232004

70.

Thomson

Flamberg

Condit

(2013). Primer on condition curves for water mains (No. EPA/600/R-13/080). Retrieved from https://trid.trb.org/view/1290548

71.

Tracy

S. J.

(2012). Qualitative research methods: Collecting evidence, crafting analysis, communicating impact. John Wiley. Retrieved from https://books.google.com/books?hl=en&lr=&id=KAzBiMCS6uwC&oi=fnd&pg=PT14&dq=tracy+2012+qualitative+research&ots=IdDXPEOZUw&sig=L8RrWANsaHTc33AmXHTcoCcUEto#v=onepage&q=tracy%202012%20qualitative%20research&f=false

72.

Vanasupa

Stolk

Herter

R. J

. (2009). The four-domain development diagram: A guide for holistic design of effective learning experiences for the twenty-first century engineer. Journal of Engineering Education, 98(1), 67-81.

73.

Ventana Systems, Inc. (2010). Vensim Professional Software (Version 5.9) [Computer software]. Retrieved from http://www.vensim.com/

74.

Wasserman

J. A.

Banks

(2017). Details and dynamics: Mental models of complex systems in game-based learning. Simulation & Gaming, 48(5), 603-624. doi:10.1177/1046878117715056

75.

Wolfe

Roberts

C. R.

(1986). The external validity of a business management game: A five-year longitudinal study. Simulation & Games, 17(1), 45-59. doi:10.1177/0037550086171004

76.

Woodyard

(2014, July 30). Water, water everywhere: UCLA cleans up from main break. USA Today. Retrieved from http://www.usatoday.com/story/news/nation/2014/07/29/water-main-break-ucla/13342473/