Optimising the Learning Potential of Simulations Through Structural Transparency and Exploratory Guidance

Abstract

Background. Simulation-based learning environments are used extensively to support learning in complex business systems. Nevertheless, studies have identified problems and limitations due to cognitive processing difficulties. In particular, previous research has addressed some aspects of model transparency and instructional strategy and produced inconclusive results.

Aim. This study investigates the learning effects of using transparent simulations (that is, showing users the internal structure of models) and exploratory guidance (that is, guiding learners so they are able to explore the simulation by themselves, supported by specific cognitive aids) from a mental models perspective.

Method. A test based on a simulation experiment with a system dynamics model, representing a supply chain system, was performed. Participants are required to use the simulator to investigate some issues related to the bullwhip effect and other supply chain coordination concepts.

Results. Participants provided with the more transparent strategy and offered the more exploratory guidance demonstrated better understanding of the structure and behaviour of the underlying model. However, our results suggest that while exploratory guidance is a beneficial method for understanding both model structure and behaviour, making only the model transparent is more limited in its effect.

Keywords

simulation-based learning environment system dynamics simulation experiment model transparency instructional strategy exploratory guidance inquiry learning business simulation supply chain

Simulation-based learning (SBL) refers to the activity of humans interacting with an external, formal model (simulation model) for the purpose of learning. Simulation-based learning environments (SBLE) are used as tools that support subjects in experimenting, building, and testing their understanding of problems. In other words, simulation models trigger a process by which learners can improve their mental models needed to develop competence, confidence, and expertise (Davidsen & Spector, 2015).

Research has suggested that the use of SBLEs frequently facilitates inquiry-based learning (Chang et al., 2020; de Jong & van Joolingen, 1998; Eckhardt et al., 2013; Vreman-de Olde et al., 2013) and they are appropriate for promoting critical reasoning (Develaki, 2017) about dynamic, complex systems (Huang et al., 2019). In particular, studies have shown that SBL with system dynamics (SD) can support and enhance learning (Alessi, 2009). System dynamics is a scientific approach for computer-based modelling and simulation developed to facilitate our understanding and management of complex, dynamic systems (Davidsen & Spector, 2015). The models are expressed graphically so as to facilitate an effective description about the systems they represent.

Mental models have been commonly used in systems thinking and modelling literature (Forrester, 1961; Huff & Jenkins, 2002; Senge, 1990; Sterman, 2000). Students can use the simulation program to support the construction and improvement of their mental models. They form an initial mental model and develop it into a target conceptual model (the same one underlying the simulation model). Students involved in SBL use the simulator as requested by the learning task, formulate hypotheses and construct their own mental models of the problem situation, compare their own mental models with the target conceptual model, and successively change and improve the hypotheses and hence advance the mental models of the situation gradually (Landriscina, 2013). Moreover, SBL facilitates improve the learners’ mental models by engaging in inquiry that is otherwise impractical or even impossible. The cost in time and resources for each learning iteration are reduced. Thus, the number of iterations can be increased, potentially resulting a more detailed understanding of the problem at stake (Groesser, 2012).

The Use of Simulations in Operations and Supply Chain Management Education

SBLEs are promising tools for teaching in the business management domain. Simulation tasks have increasingly become an element of academic programme activities (Goi, 2019; Moizer & Lean, 2010). Studies have shown that students perceive simulation as a more effective teaching method than text-based case study and lecture (Farashahi & Tajeddin, 2018; Hallinger & Wang, 2020; Prado et al., 2020; Tunstall & Lynch, 2010). The operations and supply chain domain is particularly suited to simulation because it is concerned with the management of dynamic systems and processes, such as production, inventory control, or distribution. According to Wood (2007), simulations are more appropriate for capturing the dynamic nature of such systems than, for example, static case studies or problem sets. Similarly, Pasin and Giroux (2011) investigated the effects of simulation on operations management education and concluded that although simple decision-making skills can be acquired with traditional teaching methods, simulation games are more effective when students have to develop decision-making abilities for managing complex and dynamic situations. Webb (2014) also reported that by using simulations, students develop a practical understanding of the processes and complexities of supply chain management. Angolia and Pagliari (2018) found that students were engaged by the detailed nature of the simulation, which aided their conceptual learning. Students must think in terms of systems instead of isolated components when learning about production, inventory control, and supply chain concepts. For instance, simulations may help students to experience the bullwhip effect (the tendency for production and order variability to increase as one moves upstream in the supply chain from the customer demand signal) and to appreciate the importance of supply chain coordination (Lee et al., 1997; McCullen & Towill, 2002).

Simulator Transparency and Instructional Guidance

However, whilst the benefits of SBLEs are often discussed in the literature, evidence regarding the impact of their utilisation remains limited (Lean et al., 2015). Furthermore, a need exists for more research addressing how the learning potential of such environments might be enhanced (Davidsen & Spector, 2015). According to the literature, the effectiveness of SBLEs depends on many factors related to the learning context, e.g., simulation task, prior knowledge, interest, instructional method, and type of assessment. The present study focusses on two aspects: simulator transparency and instructional strategy.

Simulator Transparency

Simulator transparency refers to the extent to which the structure of the underlying computational model is shown to students using the simulation. Simulations can be differentiated in terms of the commonly made distinction between black-box (or opaque) simulations and glass-box (or transparent) simulations. In black-box simulations, students can explore a system’s behaviour, but the underlying computational models remain hidden and can only be inferred by what appears on the screen. Glass-box simulations have alternatively been proposed to obviate this problem, as the relations among variables are accessible by the students. Transparent simulators have been used in SD learning environments, which provide stocks-and-flows diagrams (SFDs) detailing the causal structure of the underlying system. By using this approach, students may trace the cause and effect structure and understand the relationships between structure and behaviour (Milrad et al., 2003), and they have the potential to understand even counterintuitive system behaviours (Groesser, 2012). However, according to Alessi (2000) and Davidsen and Spector (2015), the drawback of a glass-box approach is that the high visibility of SD models can only benefit students who are able to read and interpret an SFD.

Research shows that structural knowledge provided by transparent simulations has the potential to improve learning and task performance. The study of Machuca (2000) provided evidence that transparent simulations are beneficial to support management learning on organisation’s complexity. Größler et al. (2000) studied the effect of structural information on student performance during a business simulator. The control group used the simulation as a black box, whereas experiment groups received information about the feedback structure of the simulation. The results indicated that familiarity with the feedback structure led to better outcomes. In simulation experiments conducted by Qudrat-Ullah (2007), prior to performing the simulation task, students in the experiment group discussed the main feedback loops in the game’s underlying model. That discussion improved students’ mental models and task performance.

Pavlov et al. (2015) conducted a pilot study where a structural debriefing protocol was introduced to help students learn about the causal structure of a black-box simulation. The students successfully completed all the steps of the structural debriefing but required considerable time to do so. Kopainsky and Alessi (2015) investigated whether model transparency, combined with prior exploration strategy, would improve learning. The authors concluded that participants provided with the more transparent strategy demonstrated better understanding of the underlying model, but their performance was equivalent to those in less transparent conditions. The implication of the previous study is that students must not only identify the structure of the simulator model but also have to recognise the relationship between structure and behaviour. The ability to infer behaviour from structure in complex, dynamic systems is a very advanced skill. Consequently, instructional support is required to facilitate effective learning (Davidsen & Spector, 2015).

Instructional Guidance

Instructional guidance refers to the support provided to students during simulation in the form of questions, hints, procedures, steps, or materials. The associated issue is about how much and what type of instructional guidance needs to be offered in order to optimise the learning potential of simulations.

Education literature has revealed a debate about the benefits of inquiry learning versus direct instruction approaches (Alfieri et al., 2011; Clark, 2009; Lazonder & Harmsen, 2016; Wen et al., 2020). Direct instruction and inquiry learning can be viewed as the two poles of a continuum ranging from purely receptive learning to purely discovery learning (Romiszowski, 1981). According to Huang et al. (2019), exploratory instruction is appropriate for promoting understanding about phenomena new to the learner. In SBL, direct instruction occurs when detailed instructions are given, and students must discover the underlying concepts through a sequence of carefully programmed steps. In the inquiry learning method, general questions or learning goals are presented, and students are free to explore the SBLE. Studies have found that students achieve deeper understanding of subject matter when using scientific reasoning (Clement, 2008). In simulations, students develop scientific reasoning as they formulate hypotheses, define actions and scenarios, conduct experiments, interpret simulation results, and draw conclusions (Friedler et al., 1990). Some studies, however, have shown that students encounter difficulty in inquiry learning (Kirschner et al., 2006; Klahr & Nigam, 2004; Mayer, 2004; Renkl, 2005).

Research on inquiry learning with computer simulations (for example: de Jong, 2006; de Jong & van Joolingen, 1998) indicates that students operating in complex simulation environments generally have difficulty in all phases of the inquiry process. Cognitive load (CL) theory provides an explanation for those problems (Sweller, 2020; Sweller et al., 2011). Cognitive load is defined as the total quantity of activity imposed on working memory at a given moment. Learning is compromised when the total CL exceeds available working memory capacity. In a simulation task, the complexity of the model may exceed the working memory limits of participants. To obviate these CL problems and increase the effectiveness of simulation, de Jong et al. (2018) suggest that instructional method be integrated with ‘cognitive tools’ aimed at guiding and supporting students’ activities.

Previous studies (Elsawah et al., 2017; van Borkulo et al., 2012) propose to enhance model-based learning, students need to be walked through each element of the model step-by-step. This approach to managing model complexity (known as model progression) recommends a progression of activities that gradually reveals more details and aspects of the underlying simulation model (graduated complexity), allowing participants to incrementally build their mental models (Davidsen & Spector, 2015). Mulder et al. (2015) conducted experimental studies to examine how model progression and worked examples can promote inquiry learning. Mulder and colleagues found positive effects due to both increased model complexity and availability of worked examples. In a multivariate inquiry task, Moon and Brockway (2019) found that guiding participants to investigate the effect of one variable at a time facilitated learning.

Recent research on SBLEs has addressed the simulator transparency issue and instructional strategies aimed at promoting inquiry learning. Nevertheless, although the studies reported provide important advances, they are not conclusive and call for further investigation (Davidsen & Spector, 2015).

Research Hypotheses

This research focuses on the issue of which model transparency and instructional guidance conditions are the most suited to optimise the learning potential of simulation for teaching the bullwhip effect and supply chain coordination (SCC), leading to enhanced student understanding of their main concepts.

Under a model-based learning perspective, students use the simulation program to form an initial mental model and develop it into a target conceptual model (the same one underlying the simulation model). Thus, the learning outcome refers to the students’ understanding of the simulated system in terms of mental model similarity. As the simulation model is known by the researchers in advance, it can be compared with the participants’ mental model in order to evaluate how that elicited mental model matched the simulated reality.

The analysis of the students’ understanding of the simulation model (representing some concepts of supply chain coordination) considers two components: the comprehension of model structure (CMS) and the comprehension of model behaviour (CMB). Comprehension of model structure is related to the similarity between the structure of students’ mental models and the structure of the supply chain system as represented in the simulator. Comprehension of model behaviour reflects how students are able to infer the dynamical behaviour of the simulation model.

The expected relations and hypotheses are based on the following variables:

Level of model transparency (LMT). This variable represents the transparency level of the simulator. Level of model transparency indicates the extent to which students have access to the variables and relations included in the simulation model when they perform the simulation task.

Level of exploratory guidance (LEG). This variable represents the exploratory level of the instructional method used to guide students throughout the simulation task. Level of exploratory guidance indicates the extent to which students may choose methods and activities to explore the SBLE.

Comprehension of model structure. This variable indicates the extent to which students understand the structure of the simulation model. Comprehension of model structure measures the similarity between the structure of the external representation of the students’ mental models and the structure of the simulation model.

Comprehension of model behaviour. This variable indicates to what extent students are able to understand and infer the dynamical behaviour of the SCC simulator. Comprehension of model behaviour measures the similarity between the student’s expectation on model behaviour (inference of student’s mental models) and the actual behaviour of the simulation model.

The present study assumes that by teaching SCC concepts through a transparent simulation, students will benefit from a more effective understanding of the model structure and behaviour, which leads to the first and second hypotheses:

Hypothesis 1: The level of transparency of the SCC simulation model positively influences the level of comprehension of the model structure.

Hypothesis 2: The level of transparency of the SCC simulation model positively influences the level of comprehension of the model behaviour.

It is assumed that, preventing high CL situations, exploratory guidance (that promotes scientific reasoning and inquiry learning) is more effective for teaching SCC concepts through an SBLE environment, and thus students will benefit from a more effective understanding of the model structure and behaviour, which leads to the third and fourth hypotheses:

Hypothesis 3: The level of exploratory guidance of the instructional method positively influences the level of comprehension of the SCC model structure.

Hypothesis 4: The level of exploratory guidance of the instructional method positively influences the level of comprehension of the SCC model behaviour.

In this research, we assumed that students assess the behaviour of the SCC simulator more accurately if the structure of their mental models is as similar as possible to the structure of the simulation model, which leads to the fifth hypothesis:

Hypothesis 5: The level of comprehension of the model structure positively influences the level of comprehension of the model behaviour.

Method

Simulator Overview

System dynamics has been used as a methodology for designing SBLEs (Alessi & Kopainsky, 2015) and also appears to be an effective and relevant tool to create underlying formal simulation models for research purposes (Größler, 2001; Repenning, 2003). Moreover, SD emphasises transparent (or glass-box) simulations rather than opaque (or black-box) simulations.

The purpose of this SBLE is to teach concepts related to SCC and the bullwhip effect. By using this SBLE, students would be able to appreciate why supply chains can demonstrate complex behaviour and the performance effects caused by the lack of overall coordination. In particular, they would be able to observe the tendency for production and order variability to increase as we move upstream in the supply chain from the customer demand signal. This phenomenon (termed the bullwhip effect) is considered a fundamental aspect of the potential behaviour of real supply chains (Lee et al., 1997; McCullen & Towill, 2002).

The SD model incorporated in the simulator represents the supply chain of a beer distribution system. It is based on the famous board game (the beer game) first developed in the 1960s at the Massachusetts Institute of Technology’s Sloan School of Management (Senge, 1990). The model conceptualisation and construction considered some elements of the models presented by McGarvey and Hannon (2004) and Maani and Cavana (2008).

Figure 1 presents an SFD of the simulator model. The model is based on a simple supply chain consisting of three stages as follows: the retailer who sells beer to the consumer, the distributor who sells beer to the retailer, the brewery that makes the beer and sells it to the distributor. Two types of items are moved in the supply chain: the cases of beer that move from the brewery to the retailer and the consumer, and the orders that move from the consumer and the retailer to the brewery. As shown in Figure 1, separate parts of the model handle these different items.

Figure 1.

Stocks-and-flows diagram of the simulator model.

Each stage keeps a certain inventory (safety stock) to hedge against uncertainty in supply and demand. It is essentially a make-to-stock supply chain. The retailer receives demand from the consumer. If the retailer has the beer in stock, the demand is satisfied; otherwise, the sale is lost. Based on the demand forecast (historical average sales), the retailer determines the desired safety stock and decides what orders to place to the distributor (the difference between the desired inventory and the actual inventory). Once the retailer places an order, it takes a few days for the order to reach the distributor. The distributor will try to satisfy orders from stock. Any unsatisfied orders are backlogged. Once sent from the distributor, the beer takes a few days for the beer to reach the retailer. The distributor must also determine the desired safety stock, decide how many cases of beer to order from the brewery, and so on. Finally, the brewery must estimate the desired inventory and decide how many cases of beer to manufacture.

The supply chain performance is measured in terms of overall earnings (total earnings = total net sales revenues - total inventory cost - total stockout cost). The overall performance results from the trade-off of two competing pressures, one to satisfy the orders (avoiding stockout costs) and one to minimise inventories.

The simulator interface includes three screens. The first screen (the control panel presented in Figure 2) is separated into three sections: one section allows participants to adjust simulation parameters; a second section includes three graphics that present the historical behaviour over time for inventory, supply flows, and order backlog for each of the supply chain stages; and a third section with a table presenting performance measures (sales revenues, average inventory, inventory cost, stockout cost, and earnings). A second screen provides a description of each of the variables in use. A third screen shows the SFD of the simulator model (see Figure 1).

Figure 2.

Simulator interface (control panel).

The participants define the experiment scenario, adjust the parameters accordingly, and run the supply chain simulator for a period of six months. Then, they observe the graphics to analyse the behaviour over time and appreciate the overall performance indicators. The following simulation parameters are adjustable: 1. Step change in demand: an increase in demand will be introduced after 20 days in the simulation. 2. Size of pulse: a sudden ‘pulse’ increase in demand will be introduced after 20 days in the simulation and repeated every 30 days. 3. Demand information factor represents how the perceived consumer demand is being shared and used to influence the order policy (if it is set to 0, the demand information that is used is coming from the next downstream stage; if it is 1, the final consumer demand is used to set the order policy). 4. Supply line factor represents how the supply line (beer in production or in transit) is taken into account when orders are being placed to a supplier. The model sets a target supply line using Little’s Law. If it is set to 0, the status of the supply line is ignored, so orders in production or in transit are ignored. If the value is 1, then all the beer previously ordered that is in production or in transit is taken into account when orders are being placed. 5. Retail cover consists of the number of days of retail sales desired to be held at the retail inventory. The parameters brewery, distributor, and retailer order fractions (the percentage of the shortage of beer at the stages that managers order each day) represent how quickly deviations from the actual inventory to the target inventory levels are being reacted to when orders are being placed. If this value is small, then the supply chain tends to react slowly to changes in inventory level. If this value is large, then the supply chain reacts more aggressively to changes in inventory.

Students may observe the bullwhip-type behaviour in the results. Even with small changes in consumer demand, the variability in orders and production rates up the chain can be huge.

Research Variables

This section summarises the use of the variables that were defined in the research model.

Level of model transparency

This variable features two degrees. In the low degree (low LMT), the students perform the simulation task without accessing the SFD of the simulator model. In the high degree (high LMT), the students, during the simulation task, have access to the SFD representing the simulator model.

Level of exploratory guidance

This variable features two degrees. In the low degree, the students perform the simulation task by strictly following the step-by-step instructions provided. For each issue being addressed, they replace the parameters with the values given, run the simulator and observe the results. The students are not encouraged to reason scientifically. In the high degree, the students are free to explore the simulator, performing all the simulation runs they need. This experiment condition is designed to foster scientific reasoning and promote inquiry learning with appropriate aids. They are asked to investigate the supply chain performance effects of the selected factors. The students reflect on each issue, formulate some hypotheses about model behaviour, and define experimental scenarios in order to test those hypotheses (including the parameters to be adjusted and corresponding values). They run the simulator, interpret the results, test the hypotheses, attempt to explain model behaviour, and determine further steps before the cycle repeats.

Comprehension of model structure

This variable measures the similarity between the structure of the external representation of the students’ mental models and the structure of the simulation model. As the structure of the simulation model is known by the researchers in advance, it can be compared with the participants’ mental model in order to evaluate how that elicited mental model matched the simulated reality. As described in the next section, the students were asked to complete a causal-loop diagram representing the simulator model (see Figure 5) by writing some missing concepts (eleven) and links (eight). This diagram represented the elicited structure of the students’ mental model. Each concept or link answered was worth one point. Thus, CMS ranges from 0 to 19.

Comprehension of model behaviour

This variable measures the similarity between the students’ expectation on model behaviour (inference of the studenta’ mental models) and the actual behaviour of the simulation model. In the final questionnaire, the students responded to multiple-answer questions related to the dynamical behaviour of the model (20 questions). Each question was worth one point. Thus, CMB ranges from 0 to 20.

Participants, Apparatus, Procedure, and Facilitation

In order to test the hypothetical model, a 2x2 experimental design (see Figure 3) with four treatment groups was defined: two groups (A and C) of participants who interact with an opaque simulator (low LMT), two groups (B and D) who use a transparent simulator (high LMT), two groups (A and B) who receive ‘step by step’ instructions (low LEG), and two groups (C and D) who receive exploratory guidance (high LEG).

Figure 3.

Treatment groups in the experiment.

The experiment involved four classes of supply chain management courses with 93 students in total. One of the authors acted as instructor in these classes. Each of the four different treatments was assigned randomly to one of the four classes: groups A (with 24 students), B (with 23 students), C (with 22 students), and D (with 24 students). Group A participants were submitted to the basic conditions (low LMT and low LEG), and thus comprised the control group. All the participants, from both experimental and control groups, were familiar with the concepts included in the simulator, as they attended lecture classes on supply chain coordination and the bullwhip effect. The participants did not know the business game and had no previous experience with the simulator.

The present experiment was carried out individually in class with one participant per computer. All students were given a full experimental guide, including an introduction reviewing the SCC and bullwhip effect concepts; presentation of the simulator; objectives, instructions for accessing, starting, and running the simulator; instructions for performing the simulation task; SFD of the simulator model (for experimental groups B and D); and sheets for recording scenarios and results. As the students’ participation in this experiment was considered as part of the course assessment process, they were motivated to follow the instructions.

In the simulation experiment, the participants were asked to investigate the impact of some factors on supply chain performance. The following five factors were considered: demand irregularity, demand information sharing, supply line control, order fraction, and inventory cover.

Past research has found that students operating in complex simulation environments generally have considerable difficulty in all phases of the inquiry process (de Jong, 2006). To obviate these problems, previous studies (Elsawah et al., 2017; Mulder et al., 2015; van Borkulo et al., 2012) suggest that participants need to be guided through the simulation model (model progression) and investigate the effect of one variable at a time (Moon & Brockway, 2019). According to Mulder et al. (2015), complementing model progression with worked examples enhances students’ inquiry performance and learning. Consequently, the students from groups C and D address the five factors sequentially and one at a time. They are also guided first to analyse performance effects caused by the factor being addressed and then to look at potential interactions with other factors previously analysed. Additionally, the students from groups C and D received a demonstration on how to investigate the supply chain effects of the first factor (demand irregularity) by defining hypotheses and scenarios, adjusting the parameters, running the simulator, and interpreting the results.

The experiment procedure involved three sessions and had the following steps (see Figure 4).

Figure 4.

Experimental procedure.

Session 1

This session involved only the participants from experimental groups B and D. The literature (Groesser, 2012) pointed out that the extra information provided by the transparency of SD models can only benefit learners who are able to read and interpret SFDs. Thus, the students received a lecture on SFDs, so that they were able to read and interpret the SFD representing the simulator model. Furthermore, as suggested from previous studies (Größler et al., 2000), giving the students a lecture on the model structure led to better simulation performance. Consequently, then, the instructor described the simulator SFD to the students in a form of a step-by-step guided tour.

Session 2

In this session, the students performed the simulation task. They read the introduction with the overall description and the objectives of the simulation task. The participants then read the instructions for accessing, starting and running the simulator. Some simulation runs were conducted to familiarise participants with the game interfaces and commands. The participants performed the simulation task, following the instructions included in the experimental guide. Students from experimental groups A and B adjusted the parameters and ran the simulator according to step-by-step instructions that guided them to discover the performance effects of those parameters. Students from experimental groups C and D were free to explore the simulator. They analysed the questions to be addressed, defined hypotheses and scenarios, adjusted the selected parameters, ran the simulator, interpreted the results, defined new hypotheses scenarios, and repeated the process. As mentioned before, the students from groups C and D were guided through worked examples on how to perform this inquiry process according to model progression approach. During the simulation task, the participants from experimental groups B and D were encouraged to read the SFD. The students were also instructed to ask for support if they found any difficulties in reading the SFD (groups B and D) or in any stage of the inquiry process (groups C and D).

Session 3

First, the students received instruction in how to read and draw a causal-loop diagram (CLD). Then they individually answered a two-part questionnaire to measure learning using the simulator model. In the first part, they were asked to complete a CLD representing the simulator model (see Figure 5 presents the incomplete CLD, and Figure 6 shows the complete CLD). Students had to fill in the empty boxes with the corresponding concepts, which they selected from a list of eleven missing concepts. They also had to identify eight missing links between the concepts and draw the corresponding arrows and signs that indicated the cause-and-effect relationships. In the second part, the students responded to twenty multiple-choice questions related to the behaviour of the model.

Figure 5.

Causal-loop diagram of the simulator to be completed by students.

Figure 6.

Complete causal-loop diagram.

No pre-test was given (before the simulation task) because the topic tested (simulator model) was new material for all the students. They did not previously know about the questionnaire they would be required to answer.

Results and Discussion

Table 1 presents the mean values, standard deviations, and sample sizes for the variables CMS and CMB corresponding to the four experimental treatments. Table 2 displays the results of an independent-samples t test of significance for differences in means between pairs of treatment groups.

Table 1.

Means and Standard Deviations for Variables CMS and CMB.

			CMS - Comprehension of Model Structure		CMB - Comprehension of Model Behaviour
Treatment	Description	N	M	SD	M	SD
A/CG	Low LMT, low LEG	24	6.167	2.761	5.833	1.659
B	High LMT, low LEG	23	9.609	5.408	6.217	2.194
C	Low LMT, high LEG	22	8.000	4.546	7.227	2.581
D	High LMT, high LEG	24	14.292	4.379	8.667	2.929

Note. LMT = Level of model transparency; LEG = Level of exploratory guidance.

Table 2.

Independent-Samples t Test of Significance for Differences in Means Between Pairs of Treatment Groups.

	CMS - Comprehension of Model Structure			CMB - Comprehension of Model Behaviour
Pair	Mean Difference	SD	Significancep	Mean Difference	SD	Significancep
B-A	3.442**	1.261	0.010	0.384	0.569	0.504
C-A	1.833	1.121	0.111	1.394**	0.646	0.038
D-A	8.125***	1.057	0.000	2.833***	0.687	0.000
D-B	4.683***	1.439	0.002	2.449***	0.753	0.002
D-C	6.292***	1.318	0.000	1.439*	0.813	0.083

p < .1. **p < .05. ***p < .01.

The lowest mean values for the variables CMS and CMB were found in participants from group A (control group), which were submitted to the basic conditions (opaque simulation and ‘step-by-step’ guidance). This result may be explained by the lack of essential information concerning the structure of the simulator model. Additionally, the ‘step-by-step’ guidance did not foster scientific reasoning and inquiry learning, and this may have led participants to misinterpret cause-and-effect relationships and model behaviour.

On average, the participants from group D (transparent simulation and exploratory guidance) exhibited the highest values for the variables CMS and CMB. As shown in Table 2, the mean values of CMS and CMB for group D were significantly different from the equivalent values for group A (pair D-A). These results suggest that students learn SCC concepts more effectively if the SBLE combines transparency and a higher degree of exploratory guidance. These processes combined gave participants from group D a significant cognitive aid that accelerated their learning about the relationships between structure and behaviour of the simulated system.

The students from group A and those from the group B (transparent simulator and step-by-step guidance) were submitted to equivalent guidance conditions (‘step-by-step’ instructional guidance). The difference between these treatment groups is that while students from group A used an opaque simulator, students from group B interacted with a transparent simulator. As expected, the comparison between treatment groups A and B presented in Tables 1 and 2 demonstrates that simulator transparency did cause the participants to improve their CMS. The statistical testing presented in Table 2 (pair B-A) provides evidence that the average responses from those groups are significantly different (CMS mean difference = 3.442, p = 0.010). On the other hand, the comparison between group D and group C (opaque simulation and exploratory guidance) reveals an even stronger difference in means for CMS (mean difference = 6.292, p < 0.000). Consequently, the variable LEG seems to positively moderate the impact of LMT on CMS. Students demonstrate, on average, an increasing comprehension of the model structure if they combine a transparent simulation with a higher degree of exploratory guidance.

As shown in Table 2, it is clear that the difference in means for CMS between group C and group A is not significant (mean difference = 1.833, p = 0.111). This means that when students are performing the opaque simulation, the mean value of CMS for those students submitted to exploratory guidance is not significantly different from the equivalent value for those submitted to ‘step-by-step’ guidance. However, the difference in means for CMS between group D and group B is very significant (mean difference = 4.683, p = 0.002). Thus, the variable LMT seems to positively moderate the impact of LEG on CMS. These results suggest that by using only exploratory guidance, students do not learn about the model cause-and-effect relationships more effectively. In other words, even though students from group C apply a scientific approach (hypothesis formulation and testing) supported by specific cognitive aids, they are not more capable of deriving structure from behaviour.

There were very few differences between the means of the CMB for group A and group B. The independent-samples t test between groups A and B displayed in Table 2 (pair B-A) does not suggest that by increasing simulator transparency, the participants revealed a higher CMB on average. Although the CMB mean increased with the use of the transparent simulator, as presented in Table 2, it is not significantly different (CMB mean difference = 0.384, p = 0.504). Therefore, the alternative hypothesis, that transparent simulations will yield the same results as opaque simulations in foster their comprehension of model behaviour, cannot be rejected. As students from group B were explained the SFD of the simulator model, they took advantage of that structural knowledge and were more able to identify and represent the key variables and the feedback processes into a causal-loop diagram. Nevertheless however, these students (from group B) were not more successful in comprehending the model behaviour. These results seem to evidence a learning difficulty, frequently mentioned in systems dynamics literature (Davidsen & Spector, 2015), that it is difficult to develop an understanding of how the behaviour of a complex system emerges from its underlying causal structure. On the other hand, the comparison between group D and group C indicates a positive difference in means for CMB (mean difference = 1.439, p = 0.083). Even though that difference is not very significant, it seems to suggest that the variable LEG also positively moderates the impact of LMT on CMB. Students demonstrate, on average, an increasing comprehension of the model behaviour if they combine transparent simulation with a higher degree of exploratory guidance.

As seen in Tables 1 and 2, there was a significant difference between the means of the CMB for group C and group A (mean difference = 1.394, p = 0.038). Moreover, the difference for CMB between groups D and B displayed in Table 2 (pair D-B) indicates a stronger significant difference than that found between groups C and A. Consequently, these results seem to suggest that by increasing the level of exploratory guidance, the students demonstrate, on average, a higher CMB. But they may even learn about the dynamics of the SCC system more effectively if they combine exploratory guidance with transparent simulations. Thus, the findings indicate that LMT also positively moderates the impact of LEG on CMB.

As presented in Table 3, regression analysis for CMS on the independent variables shows a highly significant effect for LMT (β = 0.462, p < 0.001) and a significant effect for LEG (β = 0.312, p = 0.001). Regression analysis of CMB (1) shows a low significant effect for LMT (β = 0.175, p = 0.071) and a highly significant effect for LEG (β = 0.373, p < 0.001). The regression model was refined by performing a stepwise procedure in order to exclude the variables that did not seem to significantly explain the dependent variables and to preserve the most significant explanatory variables. As presented in Table 3 and Figure 7, regression analysis of CMB (2) on the most significant independent variables (including CMS) shows a strong effect for CMS (β = 0.482, p < 0.001) and a significant effect for LEG (β = 0.221, p = 0.016).

Table 3.

Regression Results for All Independent Variables. Regression CMB (2) Is Obtained Through a Stepwise Procedure.

	Dependent Variables
	CMS - Comprehension of Model Structure		CMB - Comprehension of Model Behaviour (1)		CMB - Comprehension of Model Behaviour (2)
Independent Variables	Standardized Beta	Significance p	Standardized Beta	Significance p	Standardized Beta	Significance p
LMT - Level of Model Transparency	0.462***	0.000	0.175*	0.071
LEG - Level of Exploratory Guidance	0.312***	0.001	0.373***	0.000	0.221**	0.016
CMS - Comprehension of Model Structure					0.482***	0.000
Adjusted R2	0.305		0.156		0.336

p < .1. **p < .05. ***p < .01.

Figure 7.

Regression model with explanatory variables obtained through a stepwise procedure (standardised Betas)

These results confirm four of the five hypotheses. Performing the task with a transparent simulator significantly improved the comprehension of the model structure, supporting hypothesis H1. As we hypothesised, the results suggest that the SFD gave participants a powerful tool that accelerated their learning about the structure of the simulated system.

As indicated by Tables 1 and 2, the mean value of CMB for experimental group B (high LMT and low LEG) was not significantly different from that for experimental group A - the control group (low LMT and low LEG). Also, the regression (see Table 3) did not show a significant effect (at 0.05 significance level) of LMT on CMB. Consequently, our research does not provide full support of hypothesis H2 (the level of transparency of the SCC simulation model positively influences the level of comprehension of the model behaviour). The transparent simulation provides students more relevant information about the model structure, as they previously received an explanation on the SFD and were able to visualise it during the task. Nevertheless, these results suggest that by using only a transparent simulator, students do not learn about the system behaviour any more effectively than they would otherwise.

Regression results also support hypothesis H3 (the level of exploratory guidance of the instructional method positively influences the level of comprehension of the SCC model structure) and hypothesis H4 (the level of exploratory guidance of the instructional method positively influences the level of comprehension of the SCC model behaviour). We also found strong evidence that improved comprehension of the model structure led to better comprehension of the model behaviour, in support of hypothesis H5. Therefore, as expected, enhanced mental models in terms of structure similarity seem to improve students’ understanding of the model dynamics.

Conclusion

This study is based on an educational experiment aimed at testing hypotheses about the impact of simulator transparency and instructional guidance on students’ learning. The simulation experience involves a simulator based on SD, designed to foster student understanding of the bullwhip effect and other supply chain coordination concepts. It was predicted that (1) making the simulator more transparent (by showing and explaining students the SFD of the simulator model) and (2) providing exploratory guidance (a type of instructional strategy that includes specific cognitive aids to prevent from high CL situations and promote scientific reasoning and inquiry learning) would benefit learning.

Those hypotheses take into consideration a mental models perspective, as students’ learning is measured in terms of mental models structure and behaviour similarities.

The results from regression analysis confirm four of the five hypotheses. The hypothesis H1 (the level of transparency of the SCC simulation model positively influences the level of comprehension of the model structure) is supported. This finding is consistent with some of the literature on learning from transparent models (Größler et al., 2000; Kopainsky & Alessi, 2015). Students who received a prior explanation of the SFD and were encouraged and supported in using that diagram during the simulation demonstrated, on average, a better understanding of the underlying model structure. However, hypothesis H2 (the level of transparency of the SCC simulation model positively influences the level of comprehension of the model behaviour) is not fully supported, thus suggesting that by using only a transparent simulator, students do not learn about the model behaviour any more effectively than they would otherwise. These findings are consistent with those reported by Kopainsky and Alessi (2015) who concluded that participants provided with the more transparent strategy demonstrated better understanding of the underlying model, but their performance was equivalent to those in less transparent conditions. The testing of hypotheses H1 and H2 evidences that the students submitted to those transparent conditions are able to acknowledge certain cause-and-effect relations, but they fail to mentally simulate the model behaviour. This conclusion is in line with a learning difficulty frequently mentioned in systems dynamics literature (Alessi & Kopainsky, 2015; Davidsen & Spector, 2015): that it is difficult to develop an understanding of how the behaviour of a complex system emerges from its underlying causal structure. Particularly, various studies (Alessi & Kopainsky, 2015; Fischer et al., 2015; Strohhecker & Größler, 2015; Sweeney & Sterman, 2000) reported that learners tend to misunderstand stocks and flows. The SCC model used in this experiment involves essentially stock-flow relationships and delays related to the flows and accumulations of product inventories and orders throughout the supply chain system. As such, not only were the students unfamiliar with SD and SFDs, they may have experienced inherent difficulties in recognizing the relationship between stock-flow structures and behaviour.

Hypotheses H3 (the level of exploratory guidance of the instructional method positively influences the level of comprehension of the SCC model structure) and H4 (the level of exploratory guidance of the instructional method positively influences the level of comprehension of the SCC model behaviour) are supported. The findings evince a strong impact of the exploratory instructional process on learning about the SCC concepts. Students achieve a deeper understanding of the model structure and behaviour when they have opportunity to explore the simulation themselves and receive appropriate facilitation, such as worked examples, support on scientific reasoning (hypothesis formulation and testing), and model progression (gradually increasing task complexity). These results reinforce two assumptions already articulated in previous research: (i) The idea of using simulations to foster scientific reasoning and inquiry learning as pointed out, for example, by Clement (2008) and Nersessian (2008). Students develop a type of scientific reasoning as they construct a mental model of the system, predict its behaviour on the basis of mental simulation, formulate hypotheses, define actions and scenarios, conduct experiments, interpret simulation results, and draw conclusions. (ii) The importance of worked examples and model progression to obviate students’ difficulties in all phases of the inquiry process and enhance learning as evinced, for example, by Elsawah et al. (2017) (model progression) and Mulder et al. (2015) (model progression and worked examples).

Additionally, the analysis of difference in means between treatments indicates that students’ comprehension of the dynamical behaviour of the supply chain system is not improved when they are not submitted to the exploratory guidance condition. A possible explanation is that students receive critical information included in the SFD but act as passive knowledge recipients, as they internalise that information without making use of it to infer the system dynamics. One implication of the present study is that appropriate instructional support is required to facilitate students’ ability to recognise the relationship between structure and behaviour.

Finally, hypothesis H5 (the level of comprehension of the model structure positively influences the level of comprehension of the model behaviour) is also supported. This hypothesis was derived from the research question of whether or not improved mental models of systems lead to better inference of the systems behaviour. This finding is consistent with previous studies on this topic (for example Capelo & Dias, 2009).

This study offers contributions to the SD field by reinforcing the importance of combining management flight simulators with specific instructional techniques as a basic strategy to improve learning on complex systems. This experiment also evinces the effectiveness and relevance of SD modelling and simulation for experiments in the field of management education research and reinforces the usefulness of the mental model construct as a means of investigating how subjects learn about complex systems.

The results of this research provide useful contributions to the management education field by showing how best to use a simulator to improve learning on supply chain coordination concepts. Our findings strongly confirm that a business simulator for learning purposes can be significantly enhanced with the introduction of transparency and exploratory guidance. With these conditions, an SBLE offers students opportunities to better learn about complex business problems. This is because the students, through an active learning process, develop a systemic and dynamic understanding of the supply chain model by progressively creating and mentally simulating a causal model (mental model) that represents the critical cause-and-effect relations.

Limitations and Implications for Future Research

The present study indicates that simulation-based learning can be enhanced by introducing transparency and exploratory guidance. However, these results should be interpreted with some precaution. An important drawback of transparent simulations is that the high visibility of a model can only benefit students who are able to read and interpret the description of that model (by using SFDs in the case of SD models). Consequently, educators must assure that students comprehend the graphical code used to express the model. Another aspect is the detailed complexity of the model description. As the model used in this experiment involves a relatively limited number of variables and relations, the SFD accessed by students included the complete model. In the case of extremely complex models with a high number of variables and relations, learning might be enhanced by showing only those parts that mainly determine the dynamical behaviour of the system being represented. Future research could explore the learning effects of showing SFDs with different levels of detail complexity. The current experiment applies an instructional strategy (designated as exploratory guidance) intended to foster inquiry learning and to obviate cognitive load problems, which includes worked examples, model progression, and free exploration of the simulator. However, the time available to the simulation task and therefore to learn about the underlying model cannot be neglected. That procedure requires considerably more time than a direct instruction strategy. Extended research should investigate alternative support procedures to make such a learning approach more feasible for classrooms. A possible solution might be to accelerate the learning that occurs between simulation runs by applying a debriefing with an instructor or peers as suggested by Davidsen and Spector (2015). Furthermore, several methodological potential limitations to the current investigation exist. The research method was based on a quasi-experimental design, as there was no random allocation of students to the experimental and control groups. Each group (including the control group) consisted of students from a class; otherwise, the participants would have different educational activities in the same classroom, which was difficult to control and could jeopardise the experience. Standardised student-teacher relationships, class size and classroom features were also difficult to control. As this experiment was considered as part of the course assessment process, the authors assumed that students would be motivated to follow the instructions. However, as no effective mechanisms were applied, the researchers could not be certain that the students performed the task accordingly to the experimental guide. Despite these limitations, this study should be of interest to the many educators who teach through SBLEs.

Footnotes

Ethics Statement

The present research was approved by the Ethical Committee of the Institution. All the participants were expressly informed about the objectives of the study and that all the information collected would be treated as strictly confidential and anonymous.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was partially supported by The Foundation of Science and Technology (FCT) of Portugal, project UID/SOC/04848/2019.

ORCID iD

Carlos Capelo

Author Biographies

Carlos Capelo is an assistant professor and teaches strategic and operations management at Universidade Europeia, Lisbon, Portugal. His main research areas are related to strategic learning and business modelling and simulation.

Contact: carlos.rosario@universidadeeuropeia.pt

Ana Lorga Silva is an associate professor at ULHT, Lisbon, Portugal. She holds PhD degrees in Mathematics Applied to Economics in Management and in Computer Science from ISEG-CNAM.

Contact: ana.lorga@ulusofona.pt

References

Alessi

S. M.

(2000). Designing educational support in system dynamics-based interactive learning environments. Simulation & Gaming, 31, 178–196. https://doi.org/10.1177/104687810003100205

Alessi

S. M.

(2009). Modeling a system and teaching a system: Enhancing what students learn from modeling. In Blumschein

Hung

Jonassen

D. H.

Strobel

(Eds.), Model-based approaches to learning: Using systems models and simulations to improve understanding and problem solving in complex domains (pp. 199–214). Sense Publishers.

Alessi

S. M.

Kopainsky

(2015). System dynamics and simulation/gaming: Overview. Simulation & Gaming, 46, 223–229. https://doi.org/10.1177/1046878115596390

Alfieri

Brooks

P. J.

Aldrich

N. J.

Tenenbaum

H. R.

(2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017

Angolia

M. G.

Pagliari

L. R.

(2018). Experiential learning for logistics and supply chain management using an SAP ERP software simulation. Decision Sciences Journal of Innovative Education, 16(2), 104–125. https://doi.org/10.1111/dsji.12146

Capelo

Dias

(2009). A feedback learning and mental models perspective on strategic decision making. Educational Technology Research and Development, 57, 629–644. https://doi.org/10.1007/s11423-009-9123-z

Chang

Liu

Wen

Tseng

Chang

Chiang

Hwang

Yang

(2020). The impact of light-weight inquiry with computer simulations on science learning in classrooms. Computers & Education, 146, Article 103770. https://doi.org/10.1016/j.compedu.2019.103770

Clark

R. E.

(2009). How much and what type of guidance is optimal for learning from instruction? In Tobias

Duffy

T. M.

(Eds.), Constructivist theory applied to instruction: Success or failure? (pp. 158–183). Routledge, Taylor and Francis.

Clement

J. J.

(2008). Creative model construction in scientists and students. The role of imagery, analogy, and mental simulation. Springer.

10.

Davidsen

Spector

(2015). Critical reflections on system dynamics and simulation/gaming. Simulation & Gaming, 46, 430–444. https://doi.org/10.1177/1046878115596526

11.

de Jong

. (2006). Technological advances in inquiry learning. Science, 312, 532–533. https://doi.org/10.1126/science.1127750

12.

de Jong

Lazonder

Pedaste

Zacharia

. (2018). Simulations, games, and modeling tools for learning. In Fischer

Hmelo-Silver

C. E.

Goldman

S. R.

Reimann

(Eds.), International handbook of the learning sciences (pp. 256–266). Routledge, Taylor and Francis. https://doi.org/10.4324/9781315617572

13.

de Jong

van Joolingen

W. R

. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68(2), 179–201. https://doi.org/10.3102/00346543068002179

14.

Develaki

(2017). Using computer simulations for promoting model-based reasoning. Science & Education, 26, 1001–1027. https://doi.org/10.1007/s11191-017-9944-9

15.

Eckhardt

Urhahne

Conrad

Harms

(2013). How effective is instructional support for learning with computer simulations? Instructional Science, 41(1), 105–124. https://doi.org/10.1007/s11251-012-9220-y

16.

Elsawah

McLucas

Mazanov

(2017). An empirical investigation into the learning effects of management flight simulators: A mental models approach. European Journal of Operational Research, 259(1), 262–272. https://doi.org/10.1016/j.ejor.2016.10.011

17.

Farashahi

Tajeddin

(2018). Effectiveness of teaching methods in business education: A comparison study on the learning outcomes of lectures, case studies and simulations. The International Journal of Management Education, 16, 131–142. https://doi.org/10.1016/j.ijme.2018.01.003

18.

Fischer

Degen

Funke

(2015). Improving stock-flow reasoning with verbal formats. Simulation & Gaming, 46, 255–269. https://doi.org/10.1177/1046878114565058

19.

Forrester

(1961). Industrial dynamics. Pegasus Communications.

20.

Friedler

Nachmias

Linn

M. C.

(1990). Learning scientific reasoning skills in microcomputer-based laboratories. Journal of Research in Science Teaching, 27, 173–191. https://doi.org/10.1002/tea.3660270208

21.

Goi

(2019). The use of business simulation games in teaching and learning. Journal of Education for Business, 94(5), 342–349. https://doi.org/10.1080/08832323.2018.1536028

22.

Größler

(2001, 23–27 July). Musings about the effectiveness and evaluation of business simulators [Conference session]. Proceedings of the 19th International Conference of the System Dynamics Society, Atlanta, GA.

23.

Größler

Maier

F. H.

Milling

P. M.

(2000). Enhancing learning capabilities by providing transparency in business simulators. Simulation & Gaming, 31, 257–278. https://doi.org/10.1177/104687810003100209

24.

Groesser

S. N.

(2012). Model-based learning with system dynamics. In Seel

(Ed.), Encyclopedia of the sciences of learning (pp. 2303–2307). Springer.

25.

Hallinger

Wang

(2020). The evolution of simulation-based learning across the disciplines, 1965–2018: A science map of the literature. Simulation & Gaming, 51(1), 9–32. https://doi.org/10.1177/1046878119888246

26.

Huang

Spector

J. M.

Yang

(2019). Educational technology: A primer for the 21st century. Springer. https://doi.org/10.1007/978-981-13-6643-7

27.

Huff

Jenkins

(Eds.). (2002). Mapping strategic knowledge. Sage.

28.

Kirschner

P. A.

Sweller

Clark

R. E.

(2006). Why minimally guided instruction does not work. Educational Psychologist, 41, 75–86. https://doi.org/10.1207/s15326985ep4102_1

29.

Klahr

Nigam

(2004). The equivalence of learning paths in early science instruction: Effects of direct instruction and discovery learning. Psychological Science, 15, 661–667. https://doi.org/10.1111/j.0956-7976.2004.00737.x

30.

Kopainsky

Alessi

S. M.

(2015). Effects of structural transparency in system dynamics simulators on performance and understanding. Systems, 3, 152–176. https://doi.org/10.3390/systems3040152

31.

Landriscina

(2013). Simulation and learning: A model-centered approach. Springer.

32.

Lazonder

A. W.

Harmsen

(2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 86(3), 681–718. https://doi.org/10.3102/0034654315627366

33.

Lean

Moizer

Warren

(2015). The use and impact of simulations in management education. The International Journal of Management Education, 13(3), 349–349. https://doi.org/10.1016/j.ijme.2015.11.001

34.

Lee

H. L.

Padmanabhan

Whang

(1997). The bullwhip effect in supply chain. Sloan Management Review, Spring, pp. 93–102.

35.

Maani

Cavana

(2008). Systems thinking, system dynamics: Managing change and complexity. Pearson Education.

36.

Machuca

J. A. D.

(2000). Transparent-box business simulators: An aid to manage the complexity of organizations. Simulation & Gaming, 31, 230–239. https://doi.org/10.1177/104687810003100207

37.

Mayer

R. E.

(2004). Should there be a three-strikes rule against pure discovery learning? American Psychologist, 59(1), 14–19. https://doi.org/10.1037/0003-066X.59.1.14

38.

McCullen

Towill

(2002). Diagnosing and reduction of bullwhip in supply chains. Supply Chain Management: An International Journal, 7(3), 164–179. https://doi.org/10.1108/13598540210436612

39.

McGarvey

Hannon

(2004). Dynamic modeling for business management. Springer.

40.

Milrad

Spector

J. M.

Davidsen

P. I.

(2003). Model facilitated learning. In Naidu

(Ed.), Learning and teaching with technology: Principles and practices (pp. 13–27). Kogan Page.

41.

Moizer

Lean

(2010). Toward endemic deployment of educational simulation games: A review of progress and future recommendations. Simulation & Gaming, 41, 116–131. https://doi.org/10.1177/1046878109359052

42.

Moon

J. A.

Brockway

(2019). Facilitating learning in an interactive science simulation: The effects of task segmentation guidance on adults’ inquiry-based learning and cognitive load. Journal of Research on Technology in Education, 51, 77–100. https://doi.org/10.1080/15391523.2019.1566038

43.

Mulder

Y. G.

Lazonder

A. W.

de Jong

(2015). Simulation-based inquiry learning and computer modeling: Pitfalls and potentials. Simulation & Gaming, 46, 322–347. https://doi.org/10.1177/1046878115577159

44.

Nersessian

N. J.

(2008). Creating scientific concepts. The MIT Press.

45.

Pasin

Giroux

(2011). The impact of a simulation game on operations management education. Computers & Education, 57, 1240–1254. https://doi.org/10.1016/j.compedu.2010.12.006

46.

Pavlov

Saeed

Robinson

(2015). Improving instructional simulation with structural debriefing. Simulation & Gaming, 46, 383–403. https://doi.org/10.1177/1046878114567980

47.

Prado

A. M.

Arce

Lopez

L. E.

García

Pearson

A. A.

(2020). Simulations versus case studies: Effectively teaching the premises of sustainable development in the classroom. Journal of Business Ethics, 161, 303–327. https://doi.org/10.1007/s10551-019-04217-5

48.

Qudrat-Ullah

(2007). Debriefing can reduce misperceptions of feedback: The case of renewable resource management. Simulation & Gaming, 38, 382–397. https://doi.org/10.1177/1046878107300669

49.

Renkl

(2005). The worked out example principle in multimedia learning. In Mayer

R. E.

(Ed.), The Cambridge handbook of multimedia learning (pp. 229–245). Cambridge University Press.

50.

Repenning

(2003). Selling system dynamics to (other) social scientists. System Dynamics Review, 19, 303–327. https://doi.org/10.1002/sdr.278

51.

Romiszowski

A. J.

(1981). Designing instructional systems: Decision making in course planning and curriculum design. Kogan Page.

52.

Senge

(1990). The Fifth Discipline: The art and practice of the learning organization. Doubleday/Currency.

53.

Sterman

. (2000). Business Dynamics: Systems Thinking and Modeling for a Complex World. Irwin McGraw-Hill.

54.

Strohhecker

Größler

(2015). Performance in tangible and in cognitive stock-flow tasks: Closer than expected. Simulation & Gaming, 46, 230–254. https://doi.org/10.1177/1046878115577160

55.

Sweeney

Sterman

(2000). Bathtub dynamics: Initial results of a systems thinking inventory. System Dynamics Review, 16(4), 249–286. https://doi.org/10.1002/sdr.198

56.

Sweller

(2020). Cognitive load theory and educational technology. Educational Technology Research and Development, 68, 1–16. https://doi.org/10.1007/s11423-019-09701-3

57.

Sweller

Ayres

Kalyuga

(2011). Cognitive load theory: Explorations in the learning sciences, instructional systems and performance technologies. Springer.

58.

Tunstall

Lynch

(2010). The role of simulation case studies in enterprise education. Education Training, 52(8/9), 624–642. https://doi.org/10.1108/00400911011088953

59.

van Borkulo

S. P.

van Joolingen

W. R.

Savelsbergh

E. R.

de Jong

. (2012). What can be learned from computer modeling? Comparing expository and modeling approaches to teaching dynamic systems behavior. Journal of Science Education and Technology, 21(2), 267–275. https://doi.org/10.1007/s10956-011-9314-3

60.

Vreman-de Olde

de Jong

Gijlers

. (2013). Learning by designing instruction in the context of simulation-based inquiry learning. Journal of Educational Technology & Society, 16(4), 47–58.

61.

Webb

G. S.

(2014). Teaching supply chain management complexities: A SCOR model based classroom simulation. Decision Sciences Journal of Innovative Education, 12(3), 181–198. https://doi.org/10.1111/dsji.12038

62.

Wen

Liu

Chang

Chiang

Yang

Hwang

(2020). Students’ guided inquiry with simulation and its relation to school science achievement and scientific literacy. Computers & Education, 149, Article 103830. https://doi.org/10.1016/j.compedu.2020.103830

63.

Wood

S. C.

(2007). Online games to teach operations. INFORMS Transactions on Education, 8(1), 3–9. https://doi.org/10.1287/ited.8.1.3