Effect of Prior Exploration as an Instructional Strategy for System Dynamics

Abstract

In complex simulation-based learning environments, participants’ learning and performance may suffer due to demands on their cognitive processing, their struggle to develop adequate mental models, failure to transfer what is learned to subsequent learning or activities, and fear of failure. This study investigates an instructional strategy addressing those four problems, which we call prior exploration strategy. It was implemented in a simulation requiring participants to optimize a developing nation’s per capita income. The prior exploration strategy allows participants to manipulate and see the results of a simulation model in practice mode before they manage a similar simulation in a more final mode. The strategy was assessed in an experiment comparing participants using the prior exploration strategy with participants studying equivalent content in a non-exploratory fashion. The dependent variables were performance within the simulation and improvement of participants’ understanding. The prior exploration strategy significantly improved participants’ performance, as measured by per capita income. It also significantly improved some aspects of the participants’ understanding (e.g., their understanding of the nation’s debt accumulation), but not others (e.g., their understanding of the need to balance the nation’s health, education, and infrastructure investments; those that appear to have complex interrelations).

Keywords

cognitive load complex systems dynamic decision making feedback processes improvement of understanding instructional strategies learning environments mental models non-linearities per capita income performance prior exploration simulation games system dynamics time delays transfer of learning

Complex dynamic systems are characterized by multiple feedback processes, time delays, non-linearities, and accumulations (Sterman, 2002). Most people, even experts, find them difficult to understand and manage successfully (Brehmer, 1992; Funke, 1991; Jensen, 2005; Moxnes, 2004; Rouwette, Größler, & Vennix, 2004; Sterman, 1989a; Sterman & Booth Sweeney, 2007). Much of the research in system dynamics strives to improve the design of computer simulations and games as tools to improve human decision making and management of complex systems.

Computer simulations certainly have potential to overcome limitations on cognitive resources (e.g., Sterman, 2000; Tennyson & Breuer, 2002). Simulation models without instructional overlays (e.g., interfaces that provide guidance, feedback, and tools to support learning), are, however, generally considered insufficient, the more so as systems increase in complexity (Alessi, 2000a; Spector & Davidsen, 1997). Simulation/games or interactive learning environments (ILEs), which do include an instructional overlay, are one strategy to transfer the insights gained from a formal simulation model to a wider audience (Machuca, 2000; Spector & Davidsen, 1997). In a typical ILE, participants study textual instructions that describe the structure of the system they should manage. Participants then progress to a simulation-based, decision-making interface (the core of the ILE), where they must solve the task (i.e., manage the system) presented in the instructions. A variety of instructional strategies can be applied when designing the user interface of an ILE. These strategies include providing explanations of observed behavior, giving hints before users take action, and providing feedback following those user actions (Alessi, 2000a). These strategies work well in some simulations, especially simulations representing simpler systems. However, evaluations of the effectiveness of these instructional strategies are limited and give mixed results regarding users’ understanding and dynamic decision-making performance (see Sawicka & Rydzak, 2007, for a review).

Learners’ success in making decisions in more complex ILEs is likely to be hampered by four barriers:

The complex models underlying the ILE impose too much load on learners’ cognition. Cognitive load theory provides a clear theoretical explanation for this phenomenon (Sweller, 2005). Learners’ difficulties in complex environments are exacerbated when overall cognitive load is high and alleviated when good design decreases cognitive load. Designers can do that, most easily, by decreasing extraneous cognitive load, such as that due to the complexity of a user interface, without sacrificing the interesting and important details of the content itself (Sweller, van Merrienboer, & Paas, 1998). In situations where the content is innately and unavoidably complex, research and principles from simulation design and instructional design in general suggest that complex content be carefully sequenced beginning with simpler or more general aspects and gradually increased in complexity (e.g., Alessi, 2000b; Bruner, 1960; Reigeluth, 1999).

Learners must, but often cannot, form an adequate mental model to support decision making. Mental model theory asserts that people learning about a phenomenon or situation will form a model in their mind of what is important and how things work (Doyle & Ford, 1998; Seel, Al-Diban, & Blumschein, 2000). Such models can take many forms (e.g., images, sets of rules, procedures) and unfortunately they are not always correct. If learners in a simulation-based ILE (or in a real situation) have errors in their mental model, such that it differs from the actual underlying model, their understanding and performance in the ILE, or in the real situation, will suffer.

As people acquire knowledge and skills in an ILE, they must apply them (i.e., must transfer what they have learned) to subsequent activities, either those later in the ILE or those in real life (about which the ILE teaches). However, such transfer is often unsuccessful. People frequently acquire knowledge or skills yet fail to transfer them to applicable situations. On a basic level, transfer of learning is simply being able to apply what has just been learned at the present time to activity at some later time. That later time may be 5 minutes later. For example, if a learner sets parameters in a simulation and observes a particular outcome, using that just-acquired knowledge to do the next simulation run with sensible parameter choices (such as to test a new hypothesis) is an example of transfer, albeit very short term and near transfer. A key principle of near learning transfer is the similarity principle (Gagné, 1954; Osgood, 1949), which says that the more similar the stimuli and responses of two situations, the more likely transfer of learning will occur. In contrast, far transfer, which has been likened to generalization in learning, is more dependent on variation in situations, stimuli and responses (Barnett & Ceci, 2002), and designing such variation into an ILE should improve far transfer to real world activities.

When faced with important decisions (even in a simulation), learners often demonstrate the anxiety typical of risk (or loss) avoidance. People vary greatly in their willingness to take risks. However, a general rule is that people decrease their risk taking when they perceive the stakes as being high (e.g., dangerous or potentially very expensive) and increase their risk taking when the stakes are low (nothing consequential will happen). Kermer, Driver-Linn, Wilson, and Gilbert (2006) suggested that people will strive to avoid losses unless they perceive the amount of potential gain as much greater than the amount of potential loss. They also suggest that people generally expect the emotional effect of losses will be much greater than they in fact turn out to be.

Prior Exploration as an Instructional Strategy in ILEs

Using these four barriers to dynamic decision making (and managing systems), we designed an instructional strategy for ILEs. The strategy comprises a simulation-based prior exploration phase that precedes the simulation-based decision-making or management phase. The prior exploration phase incorporated four critical design features. First, it started with reduced complexity, so as to minimize initial cognitive load, and gradually increased complexity so as to manage cognitive load at a reasonable level. Second, it employed user controllable visuals (sliders controlling dynamic graphs) to induce an accurate mental model of how key input variables affected outcome variables. Third, it employed the similarity principle (by maintaining maximum similarity between the actions and visuals of the prior exploration phase and the subsequent management phase) to foster near transfer of learning. Fourth, the prior exploration phase incorporated reversible decisions; participants could move sliders up and down, and observe the results across all 50 simulated years. They were allowed to move back in time and try again. This appeared safer to participants than the management phase (in which they could only go forward in time). The possibility to go back in time should encourage exploratory behavior, even potentially risky exploration, because it is necessary for learners to see what can lead to catastrophic outcomes in order later to avoid such outcomes. The four barriers (cognitive load, mental model errors, poor transfer of learning, and risk avoidance) are the theoretical foundation. These four design features are an example of putting theory into practice via the prior exploration phase. We did not, however, treat the four features as separate independent variables in the study reported here. Rather, the use of the prior exploration strategy, which combined all four features, was the independent variable.

Kopainsky and Sawicka (2011) provided evidence that including prior exploration in an ILE improves both performance in the management phase of the ILE and learners’ understanding of a problem and its solution. That evidence was, however, limited to one specific task (reindeer management, a system dynamics simulation model with only one stock variable) and to a rather small number of participants. In this research, we investigate whether the prior exploration strategy improves performance and understanding in more complex systems, namely, those with more than one accumulation variable.

Toward this end, we designed a simplified experimental version of an ILE that we have been developing and refining since 2006. That ILE, called BLEND (the Bergen Learning Environment for National Development), its underlying simulation model, and our initial pilot tests are described in detail in Alessi, Kopainsky, Davidsen, and Pedercini (2008); Kopainsky, Pedercini, Alessi, and Davidsen (2010); and Kopainsky, Alessi, Pedercini, and Davidsen (2009). To summarize, participants in BLEND (the original ILE) play the roles of government ministers in a virtual developing nation who must make and revise budget decisions concerning domestic expenditures, taxes, and borrowing. They see the effects of their budget decisions in different national outcomes, including economic indicators (e.g., income, national debt, government revenue), social indicators (e.g., population, literacy, life expectancy), and environmental indicators (e.g., water quality and forest preservation). The ministers work simultaneously and interact as they make their decisions. The goal of BLEND is for participants to experience and understand that national development processes are characterized by non-linear relationships, delays, feedback loops, and multiple sources of causation (all key principles of system dynamics) requiring design of long-term policies across government ministries.

Our use of BLEND in classes and workshops made it clear, however, that people do not learn those important (and difficult) principles by mere exposure to them. As in many simulation games, players found the management tasks difficult and they frequently relied on either trial-and-error decision making or their traditional (often faulty) decision-making strategies. It was clear that better instructional strategies were needed.

The prior exploration strategy seems an ideal one, given its design based on the four previously discussed barriers to learning. We implemented the strategy in an experimental version of BLEND. This version contains only five key stocks and allows participants to manage the nation individually (rather than in conjunction with several other ministers) in the role of the nation’s prime minister, with authority for all the key national decisions. We tested its effectiveness in an experiment that compared participants using the prior exploration strategy with participants studying equivalent content in a non-exploratory fashion. We investigated the effects of prior exploration as an instructional strategy on both participants’ performance and understanding.

Performance, one of our two dependent measures, refers to the success of participants’ problem solving within an ILE. In addition to performance, our second dependent measure was participants’ understanding. While performance is primarily a measure of initial learning, understanding is more indicative of learning transfer, because the tasks and questions that measured understanding were different from the activities within the ILE. It is a measure of near transfer in contrast to far transfer (Laker, 1990) because the concepts are the same and the context is very similar. Measuring both performance and understanding is important when evaluating the effectiveness of an instructional strategy because the relationship between understanding and performance is not straightforward. Good understanding of the complex dynamic problem is necessary for consistently making good decisions, that is, for good performance. However, good performance can also be achieved without understanding, for example, through trial and error. Furthermore, although understanding is necessary, it is not always sufficient for good performance. Practice, for example, is often needed in addition to understanding. In general, we should strive to implement instructional strategies that not only lead to improved performance, but also do so as a result of better understanding.

The next section describes the laboratory experiment. The results section presents the experiment’s outcomes for performance and understanding with 51 introductory level system dynamics students. Finally, we discuss the implications of our findings for ILE design and theory.

Method

A pilot test of the experiment is described in Kopainsky et al. (2009). As the results were promising, we left the procedure unchanged and only refined the measures for assessing understanding.

Task and Decisions

Participants play the role of the prime minister in Blendia, a virtual sub-Saharan African Nation that, at the outset, is one of the poorest nations in the world (per capita income of US$300 per person per year). Their task is to achieve and maintain the highest possible per capita income in the course of 50 simulated years (see Appendix A for the complete instructions). The time horizon of 50 years is necessary because it takes more than two decades for investment decisions to significantly impact the development pattern of a nation (Arndt, Jones, & Tarp, 2009). Behavior patterns such as worse-before-better or better-before-worse only become visible with a reasonably longtime horizon.

The prime minister in Blendia has far reaching financial responsibilities and full decision-making authority regarding

Expenditures for education (an explicit decision);

Expenditures for health (an explicit decision);

Expenditures for roads (an explicit decision); and

Borrowing to finance the above expenditures (an implicit decision resulting from the three previous ones).

The simulation model used for the task is based on an extensive cross-country analysis that identified the role of a country’s resources for its long-term economic development (Pedercini, 2009) and it is described in detail in Kopainsky et al. (2009). The model depicts the development of per capita income over time as a consequence of reinforcing processes between capital accumulation through private sector development and capital accumulation through infrastructure and human development. Economic development can be severely hampered by the accumulation of debt as a consequence of too aggressive borrowing strategies. All variables are calculated on a per capita basis (e.g., capital per capita, debt per capita, and government development budget per capita), so that demographic development (i.e., the change in population) need not be taken into account. Expenditure and borrowing decisions are made every 5 years. The simulation starts in equilibrium and the prime minister remains in office throughout all 50 simulated years no matter how poor a participant’s performance. The simulation model and the user interface for the experiment were designed and implemented in LiveCode (previously called Runtime Revolution). The instructions (Appendix A) describe the structure of the simulation model underlying the experiment and the decisions participants could make.

Experimental Design

Treatments

The study compared participants using a simulation-based exploratory activity (the prior exploration strategy) before encountering a more complex management task (the experimental group) with participants who were given a textual introduction (in place of the prior exploration strategy) before doing the same management task (the control group).

During the prior exploration phase, participants in the experimental group first explored the effects of education expenditures alone (Step 1), health expenditures alone (Step 2), and road expenditures alone (Step 3), after which they explored the combined effect of all three decisions taken together (Step 4). Figure 1 shows the interface for the prior exploration phase for Step 1, where participants could move a slider for desired budget for education up or down, and observe the effect of any changing education expenditure in Year 0 (the year 2010) on the behavior of several indicators over the entire 50 years time horizon. The other expenditure categories were held constant at their initial levels.

Figure 1.

User interface of Step 1 in the prior exploration phase.

It is important to emphasize that the interface shown in Figure 1 uses dynamic graphs. As participants move the slider (in the case of Figure 1, the desired budget for education), the graphs at the bottom of the page (e.g., per capita income—interest payments) change immediately for the entire 50-year period, in accordance with the slider movement.

After each step, participants were asked to record their observations and to explain the resulting national outcomes indicated by the graphs. This pause after each exploration activity provided participants with the opportunity to reflect and was intended to prevent the impression that they are merely manipulating parameters in a trial-and-error fashion. The expectation is that through reflection, participants gain greater insight into the complexities of the system (Spector, Christensen, Sioutine, & McCormack, 2001). These reflection opportunities are considered part of the instructional strategy, and were not outcome measures.

After all four steps of the prior exploration phase, the experimental group proceeded to the management phase. The interface for the management phase is shown in Figure 2. The overall task is the same, to manage the nation well. Now, however, participants must more finely manage all three variables (education, health, and roads) and do so 5 years at a time. That is, they can modify the three expenditures every 5 years. To be clear, in the prior exploration activity, participants only set expenditures for the first year and observed outcomes across the subsequent 50 years. In the main management task, participants set expenditures, observed the results across the next 5 years, changed the expenditures, observed results across another 5 years, and so on for 10 successive 5-year periods.

Figure 2.

Interface for the management phase.

While the interface for the prior exploration phase used dynamic graphs (which changed immediately as the sliders were moved by the participants and which were reversible), the graphs in the management phase changed only when a participant had made all decisions and chose to progress 5 years. In the main management task, the participants could not reverse their decisions once implemented. As a participant moves the sliders, nothing happens to the graphs. Only when the participant clicks the button to “Simulate for the next 5 years” does the model recalculate and update the graphs for that next 5-year period. Thus, although the underlying model is the same for both the prior exploration activity and the main management activity, participants interact with it differently. The management activity is more like decision making in real life, including (in many cases) being irreversible and risky. The prior exploration strategy is reversible and therefore “feels” less risky to participants. However, it does not reveal the optimal management strategy. It only provides participants with a tool to discern it on their own.

Participants in the control group studied the same instructions as the experimental participants. They then engaged in an alternative task to the simulation-based prior exploration activity. They were provided with a text that verbally described the relationships between key variables and the reaction of outcome variables (e.g., per capita income) to changes in input variables (e.g., expenditure for education). The text was designed to require approximately the same study time as was required for the prior exploration activities (see Appendix B). Thus, we hoped to provide the control participants with a comparable amount (both time and content) of instruction as the experimental participants. The control participants then proceeded to the simulation-based management phase, which was exactly the same as that given to the experimental participants.

Hypotheses

Based on our theoretical framework and on previous studies using prior exploration, we expected participants in the experimental group to perform better and to have better understanding of the system. Concerning performance, our null hypothesis predicted no difference in performance between the experimental and the control group and our alternative hypothesis predicted a difference favoring the experimental group. Concerning understanding, our null hypothesis predicted no difference in understanding between the experimental and the control group and our alternative hypothesis predicted a difference, once again favoring the experimental group.

Participants

Data were collected with 51 introductory level system dynamics students in the fall 2009 and spring 2010. The students were recruited from the University of Bergen in Norway, the University of St. Gallen in Switzerland and the University of Freiburg in Germany. Participants from all three locations were assigned evenly and randomly to the experimental (25 participants) and control group (26 participants). That is, at each university, about half of the participants were randomly assigned to the experimental condition and half to the control condition. Participants were between 23 and 30 years old and 60% male and 40% female. The experiment was conducted in English, which was a foreign language for all participants at all three universities. Their random assignment to the experimental and control groups should, however, prevent any language-related bias in the results.

Procedure

Participants were assigned randomly to the experimental or control group. Before starting the task, all participants received the same pre-briefing. It emphasized that they were about to manage a virtual nation over a longtime horizon. They were then presented with the general schedule of the experiment.

The participants proceeded at their own pace and required between 45 and 90 minutes to complete all the activities. They worked at separate computers with no communication between them.

In addition to the simulation activities, participants completed several questionnaires designed to explain their performance and assess their understanding of the system. The first occurred immediately after the participants had been introduced to the nation of Blendia in the instructions. They were asked

to describe the problem situation in Blendia at the beginning of their term as Prime Minister; this included identifying the key variables relevant to the problem and explaining the relationships between them;

to describe their strategy for increasing per capita income while maintaining low interest payments on debt; this included explaining which policies they would implement and why they thought these would have the desired effect.

After the management phase, the participants were again given a questionnaire. They were asked

to revise their description of the problem situation in Blendia which was copied into the answer field;

to revise their description of a strategy to manage the nation which was copied into the answer field;

to report their interest in, prior knowledge of, and experience with national development issues and the use of simulations for national planning; this was used to control for the effects of participants’ backgrounds on performance and understanding.

Debriefing

The experimental session was followed by a plenary debriefing session that included an exchange of participants’ experiences while performing the experiment, collaborative development of the underlying model structure, and a discussion of the short- and long-term effectiveness of different expenditure strategies. The three groups of participants were taking classes in which the experimental activity was pedagogically relevant and the debriefing served to reinforce their understanding of the underlying system dynamics principles and their application to the respective courses, including correction of any misunderstandings students may have had at the end of the simulation. Qudrat-Ullah (2007) has shown that debriefing can reduce misperceptions of feedback. This study, however, focused on the effects of the prior exploration strategy on performance and understanding, so the debriefing followed and did not affect the measurement of those dependent variables. Its purpose was to ensure that the experimental activities contributed to the participants’ education in their respective courses.

Measurement of Learning Outcomes

To test the hypotheses, we compared measures of performance and understanding to benchmark values. For performance, we calculated the optimal quantitative values for each budget category and decision period using Vensim’s® policy optimization algorithm. Those values are the best that participants could have obtained if they made the best decisions possible. We evaluated participants’ performance by subtracting interest payments on debt from the per capita income they attained and comparing these values to the optimal ones. The distance to the optimal value has been used in previous studies (e.g., Diehl & Sterman, 1995; Moxnes, 2004; Paich & Sterman, 1993; Sterman, 1989b) for the statistical analysis of experimental and control groups’ performance and the quality difference between them.

This assessment of performance was objective, as it was based on straightforward calculations on outcome variables in the simulation and optimization by an unbiased computer program. In contrast, our assessment of understanding (described next) was more subjective, as it was based on participants’ open-ended questionnaire responses and raters’ evaluation of those responses.

To assess understanding, we compared participants’ verbal responses on the questionnaires to responses to the same questions by experts. The experts’ responses contained descriptions of the key relationships in the simulation model and of the necessary steps for solving the national development planning task in an optimal manner.

Although rating of verbal descriptions always includes some subjectivity, the following procedures were used to do the ratings in as unbiased a manner as possible. Participants’ verbal responses were first printed on one side of an index card and the participant number was printed on the reverse side to enable blind scoring (primarily so raters would not know the condition a participant was in). A scoring protocol was created, comprising a list of phrases showing evidence of understanding either detail or dynamic complexity (Senge, 1990). Detail complexity represents the overall amount of content learned, for example, by the number of variables or concepts and the number of links between them that participants wrote in their verbal responses. Dynamic complexity refers to the presence of specific concepts that reflect important system dynamic concepts including feedback, delays, non-linearities, and multiple causation. The scoring protocol awarded points to each of these elements with the maximum number of points determined by the expert text.

Two of the authors evaluated and discussed the expert texts, identifying 16 relationships between important variables (detail complexity). These relationships are summarized in the upper half of Table 1 and are labeled Relationships (detail complexity). Participants received one point for each relationship identified, the maximum being 16.

Table 1.

Coding Scheme for Measuring Understanding.

Relationships (detail complexity)	The goal is to maximize per capita income minus interest payments
	Per capita income depends on capital and total factor productivity
	Capital increases with investment
	Investment increases with per capita income
	Investment increases with education
	Investment increases with health
	Investment increases with roads
	The prime minister can regulate expenditure on education, health, and roads
	Available budget equals tax revenue minus interest payments
	Tax revenue equals per capita income times tax rate
	A deficit arises when the desired budget exceeds the available budget
	A surplus arises when the desired budget is below the available budget
	Deficit leads to borrowing
	Borrowing leads to debt
	Debt leads to interest payments
	Surplus leads to paying down debt

Characteristics of successful strategy (dynamic complexity)	Balance resources (education, health, roads)
	Invest in education early
	Invest in roads early
	Invest in health later
	Borrow early
	Pay down debt later

We similarly identified six characteristics in the expert texts that represented dynamic complexity and the understanding of stock and flow variables and their interactions (the part of Table 1 labeled Characteristics of successful strategy [dynamic complexity]).

Participants received one point if their description included the concept of balancing the expenditures on education, health, and roads. Neither roads, health, nor education alone can stimulate per capita income very much. Per capita income grows fastest when the three resources are balanced.

Participants received one point each if their description included education and roads requiring early increases in expenditures and health requiring a somewhat delayed increase in expenditures. Many years must pass before expenditures on education affect the nation’s per capita income. Knowledge, skills, techniques, and capabilities embodied in labor can be acquired through education and training, but they require time to show their effect. The same holds true for expenditures on health. The time necessary to achieve improvements in average life expectancy (a major indicator of health in a nation) is, however, considerably shorter than the time required to increase the average adult literacy rate (a major indicator of the quality of education in a nation). The results of expenditures on roads become visible fairly soon. To attain balanced growth, the optimal strategy is to prioritize education and roads in the early years, education because it has the longest implementation time and roads because those expenditures have a growth-stimulating effect fairly soon.

Finally, participants received one point each if they included the notion that borrowing early (to provide funds for expenditures) was important and that, at a later time, debt should begin to be paid off. The danger in not paying off debt is that with increasing debt, interest payments increase and these payments are deducted from tax revenue every year. Too aggressive borrowing or borrowing over too long a time period can easily bankrupt the nation and destroy any improvements in per capita income. With reasonable debt in the early years and adequate allocation to education, health, and roads, the economy starts growing so well that debts can be paid back and spending can be increased even more in all three budget categories. This allows per capita income to increase considerably without having the negative effects of debt and interest payments.

Scoring was fairly liberal in all cases. Any phrase suggesting participants understood a particular concept in Table 1 was awarded a point. One of the authors rated all participants’ responses and another author rated 20% of them to assess inter-rater reliability, which was .76 (Cohen’s κ). The reliability of the scale used was .79 (Cronbach’s α).

Results

The data were not normally distributed. We therefore present results based on non-parametric tests. Participants from the three different universities demonstrated no statistically significant differences in performance, understanding, interest in, and knowledge about national development planning, or other background information based on a Kruskal–Wallis test at α = .05. We therefore analyze and report on the entire participant pool as one group, consisting of 25 experimental and 26 control participants.

On average, participants reported a high interest in development issues with a mean of 3.8 on a 5-point scale from 1 (not at all interested) to 5 (extremely interested) and a slightly lower knowledge about national development issues with a mean of 3.3 on a 5-point scale from 1 (very poor knowledge) to 5 (very good knowledge). A two-tailed Mann–Whitney test (α = .05) revealed no significant difference between the experimental and control group for either interest or knowledge and no significant effects due to participants’ other background variables, such as their practical experience, whether they had ever used simulation to study national development issues or had ever taken classes in national development economics.

The time participants spent on the entire experiment varied considerably from one participant to another. However, participants in the control group spent significantly less time than participants in the experimental group (based on a two-tailed Mann–Whitney test at α = .05) for either time on the entire experiment or for the control group’s reading activity versus the experimental group’s prior exploration activity. We will elaborate on this difference further in the discussion section.

Performance

Figure 3 presents median performance for the experimental group (the solid black line) and the control group (dotted black line) across the 50 years of the simulation. The gray line represents the optimal solution.

Figure 3.

Optimal and median performance of the experimental and control groups.

Figure 3 illustrates that the experimental and the control group show considerable differences. They start out the same, but diverge as the simulated years progress. Median performance in the experimental group shows continuous improvement in the stated performance goal, per capita income minus interest payments (also referred to as per capita income corrected for interest). The average behavior pattern for the control group, on the other hand, is an initial slight growth of that goal followed by a decline caused by high debt and the resulting high interest payments.

Figure 4 displays graphs of the performance goal for individual participant, both for the experimental and the control group. A separate line is plotted for each participant. The lines show participants’ per capita income corrected for interest across the 50 simulated years. The figure illustrates that the participants in the experimental group manage to increase per capita income (corrected for interest payments on their debt) better than the control group. Although the experimental group had some unsuccessful performers, most experimental participants’ performance converges toward fairly high per capita income values (corrected for interest). The performance of the participants in the control group is more varied and ranges from fairly successful participants to unsuccessful ones who buy high values of per capita income with high debt, the result of which is that the per capita income corrected for interest becomes negative halfway through the simulation. The range of per capita income values (corrected for interest) at the end of the simulation is much greater for the control group than that of the experimental group. It appears the major impact of the prior exploration strategy is to help avoid the vicious cycle known as the reinforcing debt loop.

Figure 4.

Overview of individual participants’ performance in the experimental and the control group.

To investigate whether those differences are statistically significant, we compared per capita income minus interest for the two groups using a two-tailed Mann–Whitney test. The difference between the experimental and the control group was statistically significant at the 5% level (p = .05). We therefore reject the null hypothesis (no difference in performance between the experimental and the control group). On the other hand, we retain the alternative hypothesis that performance in the experimental group is better than in the control group.

Understanding

Table 2 presents participants’ descriptions of the problem situation and their proposed solution strategy. The percentages in the cells of the table indicate the percentage of participants in either the experimental or control group who described a specific relationship or characteristic of the successful strategy before (MTP1) and after interacting with the simulation (MTP2).

Table 2.

Percentage of Participants Describing Correct Relationships and Characteristics of the Successful Solution Strategy.

	Experimental group	Control group	Experimental group	Control group	Experimental group	Control group
	MTP1 (%)	MTP1 (%)	MTP2 (%)	MTP2 (%)	MTP2-MTP1 (%)	MTP2-MTP1 (%)
Relationships (detail complexity)
The goal is to maximize per capita income minus interest payments	36	23	40	23	4	0
Per capita income depends on capital and total factor productivity	32	19	36	19	4	0
Capital increases with investment	12	8	12	8	0	0
Investment increases with per capita income	4	0	4	0	0	0
Investment increases with education	28	15	28	15	0	0
Investment increases with health	28	15	28	15	0	0
Investment increases with roads	36	15	36	15	0	0
The prime minister can regulate expenditure on education, health, and roads	32	54	36	54	4	0
Available budget equals tax revenue minus interest payments	24	8	24	12	0	4
Tax revenue equals per capita income times tax rate	24	54	24	54	0	0
A deficit arises when the desired budget exceeds the available budget	4	4	4	4	0	0
A surplus arises when the desired budget is below the available budget	0	4	0	4	0	0
Deficit leads to borrowing	28	8	28	8	0	0
Borrowing leads to debt	32	31	40	31	8	0
Debt leads to interest payments	32	35	36	35	4	0
Surplus leads to paying down	8	8	8	8	0	0
Total relationships					24	4
Characteristics of successful strategy (dynamic complexity)
Balance resources	4	0	4	8	0	8
Invest in education early	32	35	28	27	−4	−8
Invest in roads early	44	15	44	19	0	4
Invest in health later	12	4	20	4	8	0
Borrow early	48	27	52	23	4	−4
Pay down debt later	16	0	24	0	8	0
Total strategy					16	0

Note. MTP1 = Measurement Time Point 1 (pre-simulation); MTP2 = Measurement Time Point 2 (post-simulation).

According to Table 2, a fairly high percentage of participants, irrespective of their experimental condition, understood the goal of the task (“the goal is to maximize per capita income minus interest payments”) and what can be influenced (“the prime minister can regulate expenditure on education, health, and roads”). A comparable percentage were able to identify the key stocks in the system (capital, education, health, roads, and debt). While they were able to identify the capital stock (“per capita income depends on capital and total factor productivity”), they failed to mention the relevant inflow (“capital increases with investment”). Similarly, many participants described the debt stock and that it increases with borrowing. However, few were also able to describe the outflow that will decrease the stock (i.e., the “surplus leads to paying down” relationship).

A fairly high percentage of participants were able to describe the direct consequences of their decisions (“investment increases with education/health/roads” relationships). They were also quite clear about how they could generate revenue domestically (tax revenue = per capita income × tax rate). However, few participants described the indirect consequences of their decisions, such as the correct mechanisms of the budget (i.e., the relationships “a deficit will occur if the desired budget is greater than the available budget” and “a surplus will occur if the desired budget is less than the available budget”). Another instance of an indirect consequence of participants’ decisions is the fact that per capita income closes a reinforcing private sector development feedback loop among per capita income, investment, capital and per capita income.

The missing focus on the flows is confirmed by the descriptions of the characteristics of successful strategies. Many participants realized that they needed to finance the important early expenditures for education (because of the long delay) and roads (because of the rather immediate impact on growth) through borrowing. However, few participants mentioned the importance of paying down debt at a later time, in order to avoid exponential growth of interest payments on the debt.

The majority of participants failed to recognize the importance of the non-linearities in the system, with only a minority mentioning that the three resources (education, health, and roads) need to be balanced for maximum growth and that as a result, expenditures for health must be increased a bit after expenditure increases in education and roads.

While these general statements are true for both the experimental and the control group, the two groups nevertheless differed from each other. At Measurement Time Point 1 (after the instructions, but before interacting with the simulation), the two groups showed no statistically significant differences with respect to their understanding of relationships (detail complexity; two-tailed Mann–Whitney test at α = .05). However, the difference in the number of relationships between Measurement Time Points 1 and 2 (the increase in the number of described relationships) was significantly higher for the experimental group, while it was not for the control group (two-tailed Wilcoxon test α = .05). The increase in the number of described relationships was due to an increase in descriptions of relationships, such as the goal of the task and the decision mechanism, or the determinants of per capita income and the debt stock. The prior exploration strategy did not, however, appear to increase the understanding of the relationships that were only indirectly linked to their decisions (i.e., the correct budget mechanisms; that investment increases with per capita income).

Concerning the number of good strategy descriptions, the two groups already differed significantly at Measurement Time Point 1, an unfortunate random variation in the composition of the two groups (based on a two-tailed Mann–Whitney test at α = .05). The difference between the two groups persisted at Measurement Time Point 2. The change between Measurement Time Points 1 and 2 was not significant for the comparison between the total number of strategy descriptions (Wilcoxon test at α = .05). However, at Measurement Time Point 2, a significantly higher proportion of participants in the experimental group (Mann–Whitney test at α = .05) described the need to borrow in the early years, which is crucial for solving the national development planning task.

Given these results (the experimental group showing a significantly greater increase in the number of relationships descriptions between measurement time points, and showing a significantly greater increase in one of the good strategy descriptions), the null hypothesis (no difference in understanding between the experimental and the control group) is not supported. However, the alternative hypothesis, that understanding in the experimental group is better than in the control group, was only partially supported.

Discussion

These results provide some evidence that engaging participants in the free and safe exploration of important variables is an instructional strategy that improves their learning from a simulation-based interactive learning environment. This is true not only for initial learning (performance during the ILE), but also for a measure of learning transfer (evidenced by their understanding of the underlying model and how to manage it). Both performance and to some degree understanding were significantly better in the experimental group than in the control group. The significant difference in performance agrees with results from similar studies (Kopainsky & Sawicka, 2011; Langley & Morecroft, 2004; Skraba, Kljajic, & Borstnar, 2007) and with results from our own pilot applications of the materials both with introductory level system dynamics students (Kopainsky et al., 2009) and with members of the actual target audience of our interactive learning environments (an unpublished study from an experimental session with planners from different government ministries in Swaziland). This is promising insofar as the simulation model underlying our ILE is more complex than the models from previous studies documenting the effectiveness of the prior exploration strategy.

More importantly, the experimental participants’ improved performance appears to be for the right reasons, given that they exhibited better understanding of some principles. In line with recent methodologies (e.g., Doyle, Radzicki, & Trees, 2008; Karakul & Qudrat-Ullah, 2008; Kopainsky & Sawicka, 2011; Rouwette, Vennix, & Mullekom, 2002; Schaffernicht & Groesser, 2011), we also assessed participants’ understanding and changes thereof in the national development task.

Participants in the experimental group improved their description of the structure of the problem (measured in the number of described relationships) more than the participants in the control group. However, the prior exploration strategy was clearly not as effective in improving understanding of dynamic complexity (strategy characteristics) as it was in improving understanding of detail complexity (number of described relationships). From a system dynamics point of view, prior exploration, therefore, seems to be more effective in developing an understanding of feedback mechanisms than of the behavior these mechanisms give rise to. This agrees with previous findings that people have great difficulties inferring behavior even from the simplest structures (e.g., Booth Sweeney & Sterman, 2000; Moxnes & Saysel, 2009; Sterman & Booth Sweeney, 2007). Although we were not able to overcome the latter difficulty fully, our results suggest that improved understanding of the problem structure itself, that is, without its behavioral implications, is necessary for improving performance.

Limitations

Several limitations should be considered when interpreting the results or suggesting their implications for theory and practice.

First, although we had designed the control group’s reading task to be equivalent to the prior exploration activity in both content and study time, the data do not show that to be entirely the case. The reading time was significantly less than the prior exploration time. Furthermore, the question about the content equivalence of the conditions arises. One might contend that the reading activity identified variables, but not their relationships, and that the prior exploration activity allowed experimental participants to actually test those relationships. The prior exploration strategy indeed has this advantage, but this is at the same time the very reason for designing the strategy. One could also make the counter-argument that the reading activity, which did state relationships, did so directly, while participants in the prior exploration condition must infer relationships. Nevertheless, the difference in time on task definitely favored the experimental condition and it might be an alternative explanation for improved understanding. An improved experiment must control for time on task better, perhaps by limiting the time allowed for prior exploration. Finally, one should consider that the greater time spent in prior exploration might be an indicator of the value of prior exploration, because participants were allowed to use it as much or as little as they wanted, and they obviously chose to use it a lot (compared with simple reading).

Second, while our measure of understanding relationships was based on 16 possible statements, our measure of understanding good strategies was based on only six possible statements. As the reliability of a measure increases with the number of data points (such as the number of items in a test), it is likely that our measure of understanding relationships is more reliable than our measure of understanding good strategies. Although this is a fair criticism of the latter measure, it also suggests that it might be too conservative a measure, and that using a model with more identifiably effective strategies would improve the power (and likelihood of significance) of the understanding strategies analysis.

Third, although our design of the prior exploration strategy was based on four cognitive learning principles (reduction of cognitive load through gradually increasing complexity, improvement of mental models through controllable visualization, improvement of learning transfer through the similarity principle, and decrease of perceived risk through time-reversible simulating), we did not measure those constructs and cannot attribute the improvements we did observe to one or another of them. Separate measurement of the four constructs would have required participants to devote too much time in those measures, rather than concentrating on the learning activity. In addition, measurements of some of the constructs are not yet well established and would have required creating new measures for this study, which in turn would have required substantial validation of the new measures. Our purpose was not a study and article on measurement techniques, but to evaluate the effectiveness of an instructional strategy.

The fourth and last limitation we point out is that the changes in participants’ text from before the simulation to after the simulation were rather small. This may account for our mixed findings for understanding, especially the lack of difference in increase for understanding strategies. This limitation is to some extent inherent in the pre- and post-test methodology (in contrast to a post-test only methodology). On the one hand, an advantage of having a pre-test is that it accounts for different levels of entering knowledge or skills. On the other hand, a disadvantage of having a pre-test is that participants learn from it, or when answering the same open-ended question feel that they do not really need to add anything to what they said earlier. Alternative measurement methodologies (such as objective questions or asking more explicit short-answer questions) might thus have improved our ability to detect improvements in understanding.

Conclusions and Implications for Future Research

As others have pointed out (de Jong & van Joolingen, 1998), and as our own data show, participants in simulation-based learning environments need instructional support. Without it, many participants engage in unproductive trial and error. Furthermore, the literature describes evidence that simply providing opportunities for exploration or other instructional strategies tends to be ineffective because participants fail to take full advantage of them (Größler, Maier, & Milling, 2000). The current prior exploration strategy implemented an instructional strategy in which participants were explicitly presented with an exploratory tool, were asked to use it, and afterward were asked to reflect upon what they learned from it. The exploratory tool was designed to make the relationships between important variables visually clear through dynamic graphs, and to do so in a way that was risk free for the participants. Our measures do not allow us to separate the contribution (to outcome improvement) of the various factors implicit in the design of the prior exploration strategy (i.e., reducing cognitive load, improving mental models, improving learning transfer, or reducing perceived risk). Still, it does appear that this particular technique of requiring use of a risk-free visual exploratory tool is effective in improving performance. Our results, however, only show partial improvement in understanding.

Future research should therefore focus on both improving the prior exploration strategy and on improving our measures of understanding. One way to improve the prior exploration strategy might be to provide a user interface that makes the structure underlying the observed behavior more transparent (see, for example, Davidsen, 1992). Existing empirical data provide mixed evidence about the effectiveness of revealing the structure underlying a simulation-based game (Größler et al., 2000). Nevertheless, increased transparency holds potential for improving understanding by helping learners to construct better mental models.

Improving the measurement of understanding, in turn, must go beyond indirect methods such as comparing novice and expert descriptions. Changes in understanding during learners’ interactions with simulation-based games, including how their existing knowledge hinders or contributes to the acquisition of new knowledge (Kopainsky & Saldarriaga, 2012) must be developed. In addition, it would be useful to measure intermediate variables that influence both performance and improved understanding, such as cognitive load and mental models, the very constructs that underlie the design of our prior exploration strategy and the strategies of many other researchers.

Footnotes

Appendix A

Appendix B

Acknowledgements

We are very grateful to Prof. Dr. Markus Schwaninger (University of St. Gallen) and PD Dr. Pablo Pirnay-Dummer (University of Freiburg) who gave us access to their students. We would also like to thank the reviewers for their useful comments and suggestions on earlier drafts.

Authors’ Note

The views and conclusions expressed in this article are those of the authors alone and do not necessarily reflect the views of the Swiss National Science Foundation or the Norwegian Research Council.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: During the research phase of this manuscript, part of the work of one of the authors (B.K.) was supported by the Swiss National Science Foundation under a Fellowship for Advanced Researchers. During the preparation of the manuscript, one of the authors (B.K.) received support from the Norwegian Research Council through the project “Simulation Based Tools for Linking Knowledge With Action to Improve and Maintain Food Security in Africa” (Contract Number 217931/F10).

Author Biographies

Birgit Kopainsky is a senior research fellow at the University of Bergen, Norway. She holds a PhD in agricultural economics from ETH Zurich and a master’s degree in geography from the University of Zurich in Switzerland. She is passionate about simulation models, dynamic decision making, and food systems research. She has worked in several sub-Saharan African countries and teaches at ETH Zurich as well as at the University of Bergen.

Contact: birgit.kopainsky@geog.uib.no.

Stephen M. Alessi is an associate professor and teaches instructional design and technology at the University of Iowa in the United States. His research includes the application of cognitive learning theory to the design of simulations, multimedia, and online instruction. He is co-author (with Stanley Trollip) of Multimedia for Learning: Methods and Development.

Contact: steve-alessi@uiowa.edu.

Matteo Pedercini is the director of modeling and capacity development at the Millennium Institute, in Washington, D.C. He holds a PhD and a master’s degree in system dynamics from the University of Bergen. He teaches system dynamics applied to development planning at the University of Bergen, Norway. His research interests include development planning, system dynamics, and scenario analysis. He has been consulting for several governments and international agencies in the field of development planning and has created numerous simulation models to support medium- and long-term strategic planning.

Contact: mp@millennium-institute.org.

Pål I. Davidsen (professor, Magister Artium) is educated in mathematics and information science with a speciality in system dynamics (SD). His research interests are in SD methodology, in the application of SD modeling, analysis, and policy design across a wide variety of fields in social sciences and in the development of SD-based multi-user interactive learning environments. He served as an associate chair of the Pre-College Education Project, Sloan School of Management, MIT (1990-1991) and as a visiting scholar in the System Dynamics Group, MIT (1986 and 1990-1991). He is a former president of the System Dynamics Society (2003) and was a managing editor of the System Dynamics Review (2007).

Contact: pal.davidsen@geog.uib.no.

References

Alessi

S. M.

(2000a). Designing educational support in system-dynamics-based interactive learning environments. Simulation & Gaming, 31, 178-196.

Alessi

S. M.

(2000b). Simulation design for training and assessment. In O’Neil

H. F.

Andrews

D. H.

(Eds.), Aircrew training and assessment (pp. 497-509). Mahwah, NJ: Lawrence Erlbaum.

Alessi

S. M.

Kopainsky

Davidsen

P. I.

Pedercini

(2008, March). A system dynamics-based multi-user domain for improving national development planning. Paper presented at the Annual Meeting of the American Educational Research Association, New York City, NY.

Arndt

Jones

Tarp

(2009). Aid and growth: Have we come full circle? (Discussion Paper). Helsinki, Finland: World Institute for Development Economics Research, United Nations University.

Barnett

S. M.

Ceci

S. J.

(2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128, 612-637.

Booth Sweeney

Sterman

J. D.

(2000). Bathtub dynamics: Initial results of a systems thinking inventory. System Dynamics Review, 16, 249-286.

Brehmer

(1992). Dynamic decision making: Human control of complex systems. Acta Psychologica, 81, 211-241.

Bruner

J. S.

(1960). Process of education. New York, NY: Vintage Books.

Davidsen

P. I.

(1992, July). The structure-behavior graph. Understanding the relationship between structure and behavior in complex, dynamic systems. Paper presented at the 10th International Conference of the System Dynamics Society, Utrecht, The Netherlands.

10.

de Jong

van Joolingen

W. R.

(1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68, 179-201.

11.

Diehl

Sterman

J. D.

(1995). Effects of feedback complexity on dynamic decision making. Organizational Behavior and Human Decision Processes, 62, 198-215.

12.

Doyle

J. K.

Ford

D. N.

(1998). Mental models concepts for system dynamics research. System Dynamics Review, 13, 253-265.

13.

Doyle

J. K.

Radzicki

M. J.

Trees

W. S.

(2008). Measuring change in mental models of complex dynamic systems. In Qudrat-Ullah

Spector

J. M.

Davidsen

P. I.

(Eds.), Complex decision making (pp. 269-294). Berlin, Germany: Springer.

14.

Funke

(1991). Solving complex problems: Exploration and control of complex systems. In Sternberg

Frensch

(Eds.), Complex problem solving: Principles and mechanisms (pp. 185-222). Hillsdale, NJ: Lawrence Erlbaum.

15.

Gagné

R. M.

(1954). Training devices and simulators: Some research issues. The American Psychologist, 9, 95-107.

16.

Größler

Maier

F. H.

Milling

P. M.

(2000). Enhancing learning capabilities by providing transparency in business simulators. Simulation & Gaming, 31, 257-278.

17.

Jensen

(2005). Learning and transfer from a simple dynamic system. Scandinavian Journal of Psychology, 46, 119-131.

18.

Karakul

Qudrat-Ullah

(2008). How to improve dynamic decision making? Practice and promise. In Qudrat-Ullah

Spector

J. M.

Davidsen

P. I.

(Eds.), Complex decision making: Theory and practice (pp. 3-24). Berlin, Germany: Springer-Verlag.

19.

Kermer

D. A.

Driver-Linn

Wilson

T. D.

Gilbert

D. T.

(2006). Loss aversion is an affective forecasting error. Psychological Science, 17, 649-653.

20.

Kopainsky

Alessi

S. M.

Pedercini

Davidsen

P. I.

(2009, July). Exploratory strategies for simulation-based learning about national development. Paper presented at the 27th International Conference of the System Dynamics Society, Albuquerque, NM.

21.

Kopainsky

Pedercini

Alessi

S. M.

Davidsen

P. I.

(2010). A blend of planning and learning: Simplifying a simulation model of national development. Simulation & Gaming, 41, 641-662.

22.

Kopainsky

Saldarriaga

(2012, July). Assessing understanding and learning about dynamic systems. Paper presented at the 30th International Conference of the System Dynamics Society, St. Gallen, Switzerland.

23.

Kopainsky

Sawicka

(2011). Simulator-supported descriptions of complex dynamic problems: Experimental results on task performance and system understanding. System Dynamics Review, 27, 142-172.

24.

Laker

D. R.

(1990). Dual dimensionality of training transfer. Human Resource Development Quarterly, 1, 209-223.

25.

Langley

P. A.

Morecroft

J. D. W.

(2004). Performance and learning in a simulation of oil industry dynamics. European Journal of Operational Research, 155, 715-732.

26.

Machuca

J. A. D.

(2000). Transparent-box business simulators: An aid to manage the complexity of organizations. Simulation & Gaming, 31, 230-239.

27.

Moxnes

(2004). Misperceptions of basic dynamics: The case of renewable resource management. System Dynamics Review, 20, 139-162.

28.

Moxnes

Saysel

A. K.

(2009). Misperceptions of global climate change: Information policies. Climatic Change, 93, 15-37.

29.

Osgood

C. E.

(1949). The similarity paradox in human learning: A resolution. Psychological Review, 56, 132-143.

30.

Paich

Sterman

J. D.

(1993). Boom, bust, and failures to learn in experimental markets. Management Science, 39, 1439-1458.

31.

Pedercini

(2009). Modeling resource-based growth for development policy analysis (Doctoral dissertation). University of Bergen, Norway.

32.

Qudrat-Ullah

(2007). Debriefing can reduce misperceptions of feedback: The case of renewable resource management. Simulation & Gaming, 38, 382-397.

33.

Reigeluth

C. M.

(1999). The elaboration theory: Guidance for scope and sequence decisions. In Reigeluth

C. M.

(Ed.), Instructional design theories and models: A new paradigm of instructional design (Vol. 2, pp. 425-453). Mahwah, NJ: Lawrence Erlbaum.

34.

Rouwette

E. A. J. A.

Größler

Vennix

J. A. M.

(2004). Exploring influencing factors on rationality: A literature review of dynamic decision-making studies in system dynamics. Systems Research and Behavioral Science, 21, 351-370.

35.

Rouwette

E. A. J. A.

Vennix

J. A. M.

Mullekom

T. V.

(2002). Group model building effectiveness: A review of assessment studies. System Dynamics Review, 18, 5-45. doi:10.1002/sdr.229

36.

Sawicka

Rydzak

(2007, July). Incorporating delays in the decision-making interface: An experimental study. Paper presented at the 25th International Conference of the System Dynamics Society, Boston, MA.

37.

Schaffernicht

Groesser

S. N.

(2011). A comprehensive method for comparing mental models of dynamic systems. European Journal of Operational Research, 210, 57-67.

38.

Seel

N. M.

Al-Diban

Blumschein

(2000). Mental models and instructional planning. In Spector

J. M.

Anderson

T. M.

(Eds.), Integrated and holistic perspectives on learning, instruction and technology: Understanding complexity (pp. 129-158). Dordrecht, The Netherlands: Kluwer.

39.

Senge

P. M.

(1990). The fifth discipline: The art and practice of the learning organization. New York, NY: Doubleday.

40.

Skraba

Kljajic

Borstnar

(2007). The role of information feedback in the management group decision-making process applying system dynamics models. Group Decision and Negotiation, 16, 77-95.

41.

Spector

J. M.

Christensen

D. L.

Sioutine

A. V.

McCormack

(2001). Models and simulations for learning in complex domains: Using causal loop diagrams for assessment and evaluation. Computers in Human Behavior, 17, 517-545.

42.

Spector

J. M.

Davidsen

P. I.

(1997). Creating engaging courseware using system dynamics. Computers in Human Behavior, 13, 127-155.

43.

Sterman

J. D.

(1989a). Misperceptions of feedback in dynamic decision making. Organizational Behavior and Human Decision Processes, 43, 301-335.

44.

Sterman

J. D.

(1989b). Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Management Science, 35, 321-339.

45.

Sterman

J. D.

(2000). Business dynamics. Systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill.

46.

Sterman

J. D.

(2002). All models are wrong: Reflections on becoming a systems scientist. System Dynamics Review, 18, 501-531.

47.

Sterman

J. D.

Booth Sweeney

(2007). Understanding public complacency about climate change: Adults’ mental models of climate change violate conservation of matter. Climatic Change, 80, 213-238.

48.

Sweller

(2005). Implications of cognitive load theory for multimedia learning. In Mayer

R. E.

(Ed.), The Cambridge handbook of multimedia learning (pp. 19-31). Cambridge, UK: Cambridge University Press.

49.

Sweller

van Merrienboer

J. J. G.

Paas

F. G. W. C.

(1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251-296.

50.

Tennyson

R. D.

Breuer

(2002). Improving problem solving and creativity through use of complex-dynamic simulations. Computers in Human Behavior, 18, 650-668.