The Effects of Situation Variability in a Simulation-Based Training for Implicit Innovation Knowledge

Abstract

Background. During the last decades, the use of simulations for training purposes has sparked wide interest. However, it is unclear how training format characteristics may affect learning, resulting in a lack of evidence-based guidelines for training developers and organizations.

Aim. We explore to what extent variation in the situations presented during a simulation training may improve learning outcomes. We test this research question in the context of a simulation-based training for improving innovation knowledge.

Methods. A sample of 131 business students was invited to participate in a study with a pretest and two posttests (within 48 hours after and 4 weeks later) and three conditions: a control group without training, an experimental training group with low situational variation, and an experimental training group with high situational variation.

Results and Conclusion. Compared to the control group, high but not low situational variation improved innovation knowledge. Participants’ prior innovation knowledge did not moderate the results. Hence, our findings indicate that ideally a simulation training includes multiple situations that offer learners various opportunities to practice innovation challenges.

Keywords

innovation knowledge leaning outcomes simulation-based training situation variation

Background

A growing body of evidence suggests that simulation-based training can be effective for a variety of learning outcomes, sometimes outperforming traditional methods (Gegenfurtner et al., 2014; Sitzmann, 2011; Vogel et al., 2006). Simulations are often considered more appealing and engaging than traditional learning settings such as classroom lectures or textbooks materials, leading to higher completion rates (Liaw, 2008; Ong & Lai, 2006). As a result, simulation-based training has become a popular method to improve cognitive outcomes (P. H. Anderson & Lawton, 2009; B. W. Mayer et al., 2011; Rogers, 2011; Scherpereel, 2005).

Researchers have focused on design characteristics to maximize the learning effectiveness of simulation-based training (Bedwell et al., 2012; Garris et al., 2002; Rutten et al., 2012; Sanchez & Van Lysebetten, 2017; Sitzmann, 2011; Wilson et al., 2009). Some design characteristics have been connected to learning effectiveness, such as learner control (Landers & Reddock, 2017), multimedia elements (Gegenfurtner et al., 2014), or engagement with training materials (Sitzmann, 2011).

One design characteristic that remains underexplored is situational variation. In simulation-based training learners are presented with challenging situations to consider and respond to. Variation in these situations typically comes from differences in their surface characteristics, while still focusing on the same underlying learning principles (Lievens & Anseel, 2007). To date, situational variation has remained largely unexplored in simulation-based training research. While some scholars have suggested that presenting multiple situations might improve learning (van Merriënboer et al., 2006), it is unclear how much variation would benefit learning. Understanding the impact of situational variation on learning effectiveness could explain some of the different effects reported in the simulation-based training literature (Sanchez & Van Lysebetten, 2017). Adding situational variation could provide a simple, cost-effective strategy for organizations, but this necessitates research-based guidance on how much variation is effective. Therefore, the aim of the current article is to examine the effects of low versus high levels of situation variation in a simulation-based training on performance on an innovation knowledge test.

Simulation-Based Training for Innovation Outcomes

Given variations in how simulations are defined in the research literature (Cannon-Bowers & Bowers, 2009; Garris et al., 2002; Sitzmann, 2011) we clarify our definition of simulation-based training as formats that involve instructions and situations based in reality and delivered through a computer (Bell et al., 2008). Increased popularity of simulation-based training (Cannon-Bowers et al., 2010; Gegenfurtner et al., 2014; R. E. Mayer, 2011; Training Industry Report, 2017) has been fostered by their potential to enhance learner motivation and the learning process (Bell et al., 2008; Sitzmann, 2011). The realistic virtual environments of simulation-based training can be perceived as fun, attractive, and approachable by learners, making learning goals more intrinsically motivating (Malone, 1981; Vogel et al., 2006). Given learner’s openness to simulations, many may experience higher self-efficacy because they feel capable repeating and performing tasks in a low stakes environment until their skills are satisfactory (Bandura, 1977; Tennyson & Jorczak, 2008). Meta-analyses have demonstrated that simulation-based trainings can stimulate self-efficacy, motivation, and learning (Gegenfurtner et al., 2014; Sitzmann, 2011).

Several practical advantages of simulation-based training have contributed to its rising popularity. For instance, a trainer’s presence is not always required because the training is delivered from a computer and learners may have flexibility to complete the training at a time and place that is convenient for them (Bell et al., 2008; Sun et al., 2008). In addition, simulation training can be designed to be customizable to individual preferences such as pausing or adjusting the pacing of a training that can be controlled by a user in a simulation (Bouhnik & Marcus, 2006; Landers & Reddock, 2017). The customizability of simulations can be further beneficial when designing rare events on the job that are harder to train, such as implicit knowledge about the innovation process (Cannon-Bowers & Bowers, 2009).

Simulation-based training has been researched across domains and for various cognitive outcomes including decision making for military commanders (Beal & Christ, 2004), clinical reasoning for medical doctors making diagnoses (Consorti et al., 2012), and critical thinking skills and mental models for business students learning to manage organizations (Lovelace et al., 2016; Palmunen et al., 2013; Scherpereel, 2005). These studies suggest that participating in simulation-based training can improve cognitive skills, understanding one’s environment, decision making, and performance. Given these and other positive findings that utilized simulation training for complex cognitive skills (Lainema & Nurmi, 2006; Siewiorek et al., 2013), we analyze advancements in implicit innovation knowledge using simulation training.

It has been established that innovation is a crucial source of organizational success (N. Anderson et al., 2004, 2014), which has allowed innovation to be widely pursued (Yuan & Woodman, 2010). One approach to improving innovation is to stimulate employees’ innovation potential where employees are actively involved in the ideas, actions, and decisions that may produce innovative solutions (Scott & Bruce, 1994). Much research has been devoted towards enhancing employee-driven innovation (N. Anderson et al., 2014; Birdi et al., 2016). However, the impact cognitive factors have on innovation and how to advance implicit knowledge and underlying knowledge structures (e.g., schemas, mental models, cognitive frameworks) of innovation, have been neglected in research (N. Anderson et al., 2014; Birdi et al., 2016). This is a relevant topic to pursue in research given evidence that suggests an individual’s effective decision making and performance is associated with accurate knowledge structures and insights in key principles of that domain (Chi et al., 1981; Dane et al., 2012; Gary & Wood, 2011; Gavetti & Levinthal, 2000).

When individuals encounter situations, they form mental representations of these experiences (Gentner & Stevens, 1983). In these mental representations, individuals can generate inferences and manipulate analogies, which helps them to develop implicit knowledge and connect this knowledge into a broader cognitive map of knowledge structures (i.e., mental models; Gary & Wood, 2011; Johnson-Laird, 1983). Studies have shown that knowledge structures help individuals understand causal relationships between contributory factors and situation outcomes. It could be assumed that accurate cognitive frameworks of the innovation domain may provide resources for individuals to overcome innovation challenges and may guide individuals in their decision-making during innovation projects. These knowledge structures may lead to better approaches and individual performance when confronted with complex innovation projects. Given the sparse research on advancing implicit innovation knowledge (Dane & Pratt, 2007), there is no practical guidance on selecting and designing effective learning environments for this purpose. This study aims to identify effective training formats for implicit innovation knowledge to guide the design of future trainings.

Hypotheses and Research Questions

As a baseline hypothesis, we first test whether the simulation training has a main effect on the targeted outcome, performance on an innovation knowledge test. Because there are compelling arguments for positive and negative effects of including more variation in simulation tests, we articulated competing hypotheses for variation:

Hypothesis 1. Learners in a simulation training for innovation will have better innovation knowledge than learners who receive no training.

Working memory (i.e., the cognitive system that allows individuals to retain and process information; Baddeley, 2003) has a limited capacity (Baddeley, 1992). The mental activity demanded from working memory is defined as cognitive load (van Gog et al., 2010). When this limited capacity of working memory is reached there is cognitive overload and the individual is not able to concentrate or focus, leading to an inability to process relevant information, integrate new and prior knowledge, and hindering learning (Gathercole & Alloway, 2012).

Cognitive load theory suggests that training design should reduce cognitive load and the demands placed on working memory (Paas et al., 2003). To this end, a training should; (a) find an optimal level of complexity and difficulty in the materials (i.e., intrinsic load), (b) reduce factors that do not facilitate learning such as distracting and unnecessary stimuli (i.e., extraneous load), and (c) support cognitive load for relevant learning materials involved with mental model formation (i.e., germane load). This, simulation design should optimize intrinsic load and germane load and minimize extraneous load.

Offering a variety of situations in a simulation may induce germane load and stimulate the learning processes necessary for cognitive learning (i.e., mental model acquisition). When a learner resolves different versions of the same task, the learner will receive more opportunity to practice and to apply the cognitive processes of selecting, organizing, and integrating relevant information into the knowledge structures, which implies deep cognitive learning. Situational variations also provide more opportunity to detect common principles and similarities across different situations, distinguish relevant information from irrelevant cues, and identify the range of situations in which the principles can be applied (J. R. Anderson, 1982; Van Gerven et al., 2006). Higher situational variation may give leaners more opportunities to integrate the underlying structures, insights, and principles into richer and accurate mental models, which drive learning. Thus, on one hand, we hypothesize:

Hypothesis 2a. A high level of situational variation in a simulation-based training will lead to more innovation knowledge compared to a low level of situational variation.

However, from a cognitive load perspective, it can be argued that training is more complex when there is more situational variation because the learner must process more information. Greater complexity induces a higher intrinsic load, which requires more mental activity from the learner’s working memory. Since the working memory has a limited capability to process information (Baddeley, 1992; Cook, 2006), too much situational variation could result in cognitive overload (i.e., with high intrinsic load, the training becomes too complex for the learner). This could undermine training effectiveness because the learner might become unable to concentrate and process the relevant information, which is vital for effective learning. Thus, on the other hand, we provide a competing hypothesis:

Hypothesis 2b. A high level of situational variation in a simulation-based training will lead to less innovation knowledge compared to a low level of situational variation.

Finally, we also examine whether prior innovation knowledge moderates the effect of variation level. Prior knowledge may provide an advantage to learners as it can support the germane load induced by the training, thereby facilitating learning because it requires less mental effort during the training. Also, prior knowledge may lower intrinsic load and extraneous load because learners with prior knowledge understand the provided situations (due to existing mental models). This makes it easier to process the varying information offered in the situations and helps learners avoid distracting stimuli during training. These reductions in cognitive load may increase mental capacity in the working memory to optimize the learning process. Accordingly, we expect that learners with adequate prior innovation knowledge may benefit more from a high level of variation due to their ability to filter the provided information. Thus, we hypothesize:

Hypothesis 3. Learners’ prior innovation knowledge will moderate the effect of variation level (i.e., low versus high) on innovation knowledge. Learners with high prior knowledge will benefit from high situational variation more than learners with low prior innovation knowledge.

Materials and Methods

Sample

This research included 131 students (45.9% male, 54.1% female) enrolled in a business and management program at a Belgian university. Participants ranged from 19 to 29 years of age (M = 21, SD = 2). The majority held a secondary degree (88.8%), 10.4% bachelor’s degree, and 1% doctoral degree. A sizeable number of participants did not participate in later phases of the study, which is common for multi-week online training settings (Brown, 2001). Table 1 shows the final sample size per condition for each study phase after attrition. Of the original 131 participants, 63% performed the first post-test at Time 3 (48 hours later), and 44% performed the second post-test at Time 4 (4 weeks later).

Table 1.

Final Sample Distribution by Condition at Each Time.

Condition	Time 1	Time 2	Time 3	Time 4
Control	43	- -	31	22
Experimental Low SV	46	37	25	17
Experimental High SV	42	33	26	19
Total n	131	70	82	58

Note. SV = Situational Variation Training.

This research has been conducted following the ethical requirements established by the Belgian national board of ethics. There was no financial or course-related incentives; participation was voluntary. Students were informed they could stop participating at any time.

Research Protocol

The study involved four phases administered online via e-mail invitations, see Figure 1. Phase 1 included a pre-test for all participants. Two weeks later in phase two, participants in the experimental condition received a link to the training. In phase 3 a first post-test was given to all participants. Two weeks later in phase 4 a second pots-test was given to all participants.

Figure 1.

Study overview.

Control Group

The control group completed the pre-test and both post-tests but did not receive any training.

Experimental Groups

The experimental groups completed the pre-test followed by a training simulation. One group (i.e., low situational variation) received three situations and the other (i.e., high situational variation) received six situations in the training. The number of situations was determined by the withdraw point of participants in a pilot test. Participant feedback revealed the training was too long, when it contained nine situations, to retain full attention. Six situations were used in the high situation variation as this was near the natural withdraw point. Half the situations were used in the low situation variation group because one would have provided no variation and the variation for two situations would have been limited (see Gary et al., 2012).

Attrition and Pre-Training Differences

We performed one-way analyses of variance (ANOVAs) and chi-square tests to measure the effect of attrition on the different study conditions with regard to demographic characteristics, performance scores for training motivation, and prior innovation knowledge at Time 1. We found no significant differences due to attrition (all p > .05). In addition, participants who only completed the pre-test at Time 1 were excluded from further analysis on the basis of missing data.

We also performed one-way ANOVAs and chi-square tests to identify pre-training (Time 1) differences across conditions for age, gender, level of education, training motivation, or prior innovation knowledge (all p > .05). None of the analyses showed significant results, indicating that random assignment was effective for ensuring the three groups were similar before training.

Intervention

The simulation training consisted of situations mimicking real-life innovation challenges. Participants were instructed to write actions they would take to solve the innovation situation, and explain the effectiveness of their approach. Next, participants received seven possible response actions and rated the effectiveness of each action on a seven-point scale, for which they received automatic feedback. This feedback included effectiveness ratings from innovation experts. Next, participants received a comprehensive and structured expert model with instructions to compare their solution to the expert model, see Figure 2. The procedure repeated for each situation. Multiple-choice questions requesting feedback on the situation were used as an attention check.

Figure 2.

Different steps of the innovation simulation training.

Training materials

The innovation situations in the simulation were developed using an inductive approach by two researchers, following the protocol for situational judgment test development (Lievens & Anseel, 2007; Lievens et al., 2008). For the purposes of this study, nine of the initial 70 situations were selected (i.e., six for training and three as performance measures). These situations captured innovation challenges such as selecting innovative ideas, gathering support to implement ideas, managing production complications, and communication issues with clients. Moreover, to ensure the realism of the simulation task, the situations were supplemented with audio, graphics, and interactive activities. Appendix A (see Supplemental Material) provides images of a situation used in the simulation task.

Feedback materials

For each situation in the training participants were asked to rate seven response actions. Input was provided by 56 innovation experts1 and 18 laypeople2 to generate these response actions, which were reviewed and rated on their effectiveness. Response actions with low agreement between raters using intra-class coefficients (ICC < .80) were excluded. The average effectiveness score from experts was used as the effectiveness rating in the study.

Expert models

The expert models that participants used to make comparisons with their own decisions were generated for each situations by 10 innovation experts using a think-aloud protocol (Ericsson & Simon, 1993; Hoffman et al., 1995). Interviews were transcribed and consistent structures and similar information in the experts’ solutions were highlighted and aggregated to design a comprehensive and structured expert model for each situation (see Day et al., 2001). Each expert model yielded: (a) a comprehensive approach to problem solving; i.e., with a sequence of steps showing how the specific innovation situation can be effectively resolved, (b) various innovation principles, such as rules of thumb that are in line with the experts’ simplified mental models of the innovation process and are applied when confronted with complex and uncertain situations (Gary & Wood, 2011), and (c) explanations of why these steps and principles would be effective. The expert models were similar in length and level of detail. Appendix B (see Supplemental Material) provides an example of an expert model used in the simulation training.

Measurements

Demographic data

At Time 1 we obtained data about the participants’ gender, age, and educational level.

Training motivation

As a control, participants were asked to complete a training motivation questionnaire (Tharenou, 2001) at Time 1. The scale consisted of seven items such as, “I try to learn as much as I can from Training & Development programs.” Agreement was on a Likert type scale from 1 = strongly disagree to 5 = strongly agree. The scale’s Cronbach’s alpha was .79.

Innovation knowledge

The simulated innovation situations were used to measure participants’ innovation knowledge. At Time 1, one simulated innovation situation was used to measure the participants’ prior innovation knowledge. For both post-training innovation knowledge tests, we used a simulated innovation situation that was also used during the training.

During these simulations, participants provided written responses that were scored by two independent raters who were trained to use the coding scheme. The coding scheme guided raters to determine if (1) an effective action was proposed in the solution, (2) the solution was structured and detailed, (3) the solution held correct and comprehensive explanations, and (4) innovative principles were used. The maximum possible score was 20 points.

Participants’ scores on the innovation knowledge tests were at Time 1: M = 4.94, SD = 3.49, ICC = .91, n = 131; Time 3: M = 5.72, SD = 2.71, ICC = .93, n = 82; and Time 4: M = 7.97, SD = 4.77, ICC = .94, n = 58. The inter-rater reliability scores were adequate and in line with previous studies (see Jones, 1981).

Analogical learning

To determine if participants could apply their learned knowledge directly to a new situation with the same solution, we presented participants with a novel innovation situation at Time 3 and at Time 4 (Gentner et al., 2003). The same coding schema and methodology were used to score the situations. The mean scores were at Time 3 (M = 7.65, SD = 3.48, ICC = .93, n = 82) and Time 4 (M = 8.02, SD = 4.17, ICC = .90, n = 58).

Results

Descriptive Statistics and Correlations

Table 2 presents the means, standard deviations, and correlations between the study variables. A significant negative relation was observed between gender and innovation knowledge at Time 3 (r = −.21; p < .05). Prior innovation knowledge had significant positive relationships with innovation knowledge (r = .23, p < .01) and analogical learning (r = .31, p < .01) at Time 3. Training motivation did not have a significant correlation with innovation knowledge at either post-test (for Time 3, r = .04, p > .05; and for Time 4, r = .20, p > .05). Furthermore, we found that innovation knowledge at Time 3 and Time 4 significantly correlated (r = .73; p < .01), and there was also a significant positive correlation for analogical learning between Time 3 and Time 4 (r = .50; p <.01). In addition, innovation knowledge at Time 3 had a moderate positive relation with analogical learning at Time 3 (r = .60; p < .01) and Time 4 (r = .62; p < .01), while there was a significant correlation at Time 4 (r = .70; p > .05).

Table 2.

Means, Standard Deviations, and Correlations.

	M	SD	1	2	3	4	5	6	7
1. Gender	- -^a	- -^a	- -
2. Age	21.04	1.74	−.41**	- -
3. Training Motivation	3.90	0.39	.10	−.07	- -
4. Prior Knowledge	4.94	3.49	−.01	.11	−.04	- -
5. Innovation Knowledge (T3)	5.72	2.71	−.21^*	−.05	.04	.23^*	- -
6. Innovation Knowledge (T4)	7.97	4.77	−.01	−.03	.20	.15	.73**	- -
7. Analogical Learning (T3)	7.65	3.48	.20	.10	.16	.31**	.60**	.56**	- -
8. Analogical Learning (T4)	8.02	4.17	−.16	−.04	.05	.21	.62**	.70**	.50**

Note. Gender: 0 = male, 1 = female. T3 = Time 3; T4 = Time 4.

Means and SDs could not be calculated since they are categorical variables.

p < .05, **p < .01.

Test of Hypotheses

Table 3 shows the means and standard deviations for innovation knowledge and analogical learning for Time 3 and Time 4.

Table 3.

Means and Standard Deviations for Cognitive Outcome Tests by Condition.

Condition	Innovation Knowledge				Analogical Learning
	T3		T4		T3		T4
	M	SD	M	SD	M	SD	M	SD
Control	4.77	1.80	6.05	4.82	6.77	3.23	7.77	3.02
Experimental Low SV	5.52	2.68	8.53	4.95	8.31	3.79	7.75	5.25
Experimental High SV	7.04	3.03	9.68	3.87	8.00	3.27	8.53	4.48

Note. Possible score range 0-20. T3 = Time 3; T4 = Time 4; SV = Situational Variation Training.

First, we conducted ANOVAs with training condition as the independent variable and innovation knowledge and analogical learning as the dependent variables for both Time 3 and Time 4 (Hypothesis 1). For innovation knowledge, we found a significant difference between the three conditions at both Time 3, F(2, 79) = 5.59, p = .005 and at Time 4, F(2, 55) = 3.41, p = .04. There was no significant difference in analogical learning across conditions, neither at Time 3, F(2, 80) = 1.62, p = .20 nor at Time 4, F(2, 54) = .21, p = .814.

We proceeded with a contrast test conducted via a General Linear Model-procedure for the dependent variable of innovation knowledge measured at Time 3 and Time 4 to further examine whether there was a difference between the combined mean score of the two training conditions and the mean score of the control condition. Since there was no significant differences between the training conditions for analogical learning, no further analyses were conducted for this outcome.

Innovation knowledge was significantly higher in the training groups compared to the control group for both post-tests. There was a mean increase of 1.51, 95% CI [.34, .2.67], p = .01 for Time 3 and a mean increase of 3.07, 95% CI [.58, 5.54], p = .02 for Time 4. Our findings indicate that participants scored significantly higher on the innovation knowledge test immediately and 4 weeks after completing a simulation training when compared to a group that received no training. However, this was not found for analogical learning. In sum, we only found partial support for Hypothesis 1.

Table 4 shows the results of the Tukey post-hoc tests for innovation knowledge as a dependent variable for Time 3 and Time 4. These analyses were conducted to compare differences across the three conditions for innovation knowledge to test for the competing Hypotheses 2a and 2b. Innovation knowledge mean scores at Time 3 and Time 4 increased from the control group, to the low situational variation group, and to the high situational variation group (see Table 3). However, the only significant differences were the mean increase in innovation knowledge at Time 3 comparing the control group and high situational variation group (2.26, 95% CI [.63, 3.9], p = .004; see Table 4) and the mean increase at Time 4 comparing the control group and high situational variation group (3.64, 95% CI [.19, 7.09], p = .037; see Table 4). No post-hoc tests on analogical learning were conducted since one-way ANOVAs did not reveal significant differences.

Table 4.

Post-hoc Tukey Tests (ANOVA) for Innovation Knowledge.

Condition	Comparison	Mean Difference ^a	Significance	CI
Time 3 (N = 82)
Control	Low SV	−0.75	.53	−2.40, 0.91
	High SV	−2.26	.004**	3.90, 0.63
Low SV	High SV	−1.52	.10	−3.24, 0.20

Time 4 (N = 58)
Control	Low SV	−2.49	.22	−6.04, 1.07
	High SV	−3.64	.04^*	−7.09, −0.19
Low SV	High SV	−1.16	.73	−4.83, 2.52

Note. CI = Confidence Interval; SV = Situational Variation Training.

Difference between the marginal means of the condition and its comparison.

p < .05. **p < .01.

Simulation training with high situational variation was effective in improving innovation knowledge initially and 4 weeks after the training in comparison with a control group that received no training. However, we did not find significant differences between the training condition with low situational variation and the control condition or between the two training conditions with different situational variability (i.e., low vs. high) for either post-test. In summary, we found partial support for Hypothesis 2a suggesting that a high level of situational variation can improve innovation knowledge compared to a control group. So the competing Hypothesis 2b was not supported.

Hypothesis 3 was tested via multiple regression analyses to examine the interaction effect of situational variation level (i.e., low vs. high) and prior innovation knowledge on the dependent variables of innovation knowledge and analogical learning (controlling for both main effects).

There was no significant moderator effect of prior innovation knowledge, as evidenced by the coefficients of the interaction term for Time 3 (b = .29, SE = .27, p = .28) and Time 4 (b = .69, SE = .51, p = .19) for innovation knowledge as the dependent variable. Similarly, when analogical learning was the dependent variable, the coefficients of the interaction term were not significant for Time 3 (b = .02, SE = .32, p = .96) or Time 4 (b = .50, SE = .56, p = .38). Consequently, Hypothesis 3 is not supported.

Discussion

This study aimed at improving the understanding of an important but underexplored training design characteristic, namely situational variation in simulation training. More specifically, we explored whether a high level of situational variation is more effective than a low level and whether this difference is dependent on the learners’ prior knowledge.

First, our results supported that innovation knowledge can be advanced by completing simulation-based training. Participants who received simulation training were better at solving innovation problem situations than those who did not. The learning effect was observed immediately after training and also 4 weeks later. However, participants were not able to apply their knowledge to solve new related situations, neither immediately nor 4 weeks after training. A possible explanation might be that the small sample sizes (partly caused by participant attrition) prevented the detection of differences for analogical learning. Another explanation might be that it is difficult to train a complex cognitive outcome such as innovation knowledge when only providing artificial situations. Others have suggested that learners might not incorporate relevant information into their knowledge structures due to a lack of verbalization of what is processed during simulation training (Leemkuil & Jong, 2011; Wouters et al., 2008), and this hinders its application in new situations. Thus, we can assume that simulation tasks should ideally be part of a broader training context (Sitzmann, 2011; Wouters & van Oostendorp, 2017) with supplemental instructions (e.g., a debriefing possibility in which learners can discuss their learned insights).

Second, our findings suggest that only simulation training with an extensive set of varying innovation process situations yielded a significant increase in innovation knowledge compared to a control group. However, no significant differences for analogical learning were found (i.e., participants were unable to apply learned innovation knowledge to new and unfamiliar innovation situations). This seems contrary to the findings of Gary et al. (2012) who reported positive effects by adding only one situation to improve strategic decision making. However, our findings suggest that no cognitive overload of the working memory was induced. Greater variability did not hamper learning but rather improved innovation knowledge. Thus, it can be assumed that more opportunities to practice supported the germane load produced by the training and led to cognitive learning. The results also suggest that the higher level of situational variation did not produce a higher level of intrinsic load; training with more variability was not too complex to impede innovation knowledge, as otherwise no learning would have occurred. Of the two possibilities we investigated, we found more evidence supporting the effectiveness of using a higher level of situational variation.

Third, further drawing on theories of mental model formation (Moreno & Mayer, 2005) and cognitive load theory (Sweller, 1988), we expected that prior knowledge would play a moderating role. Participants with low and high levels of prior knowledge would benefit more from receiving training with low and high variability, respectively. Our results did not support this hypothesis. Prior knowledge did not seem to influence the optimal number of situations during training. However, given that our sample only consisted of students with no prior experience of making decisions or leading organizational innovation projects, we did not detect significant variance in prior innovation knowledge (M = 4.94, SD = 3.49). This minimal variance in prior knowledge, combined with our small sample size, is a probable explanation for why we did not observe a moderating effect of prior knowledge. Future research could investigate the moderating role of prior knowledge on the level of variability in improving cognitive outcomes, with a sample containing differing degrees of expertise (e.g., experts, novices, and naïves).

Practical Implications

Our study has direct implications for the design of simulation-based training for cognitive outcomes. Given that organizations prefer efficient employee training, it would be beneficial if training designers focus on developing simulation-based training that enables participants to transfer knowledge and skills they learn into different context. In order to attain this goal, we recommend developing and implementing more extensive training designs with multiple situations that have variable surface characteristics but the same underlying structure. Irrespective of prior participant knowledge, this is a better format than presenting a smaller variety of situations. This should help build a broader knowledge foundation in learners, which can improve their understanding and performance. This recommendation aligns well with a meta-analysis for leadership training. Lacerenza et al. (2017) demonstrated that longer training duration (i.e., more extensive training) was more effective to train leadership skills; however, caution is needed. In our pilot test, participants tended to drop out when we utilized a training with nine situations. This indicates that simulation developers need to strike a balance between variation and brevity. Therefore, we would recommend incorporating an option for the learner to control when to pause and continue the training. This is supported by previous research suggesting integrating learner control into simulation training designs (Landers & Reddock, 2017) to prevent learners from becoming demotivated by feeling overwhelmed.

Limitations of the Present Study

Our results should be evaluated in the light of some limitations. First, we did not measure the underlying theoretical mechanisms of mental model formation (i.e., the cognitive processes of selecting, organizing, and integrating) or cognitive load experienced by the learners. Second, only students participated. We should be cautious in generalizing these findings to a business or organizational context. Third, there was attrition for both post-tests resulting in a smaller sample than expected. Although we took multiple actions to counter attrition (e.g., professors emphasized the importance of participation, a motivational information brochure was sent prior to the program start, and multiple reminder emails were sent), a sizeable number of participants still dropped out. A possible explanation is that the program was time intensive, so participants were less motivated to complete the post-tests. Also, encountering technical difficulties (e.g., slow internet, error notifications, etc.) during online training results in higher attrition levels (Sitzmann et al., 2010). Despite this, training conditions were still comparable regarding gender, age, training motivation, and prior knowledge. A final shortcoming of the study is that due to the online learning context, we had limited control regarding how, where, and when participants completed the different study phases, which may have decreased internal validity. We tried to counter lack of control by emphasizing that participants were required to complete the different phases at a location where it was quiet and they were not distracted. However, as there is no clarity on the impact of the real-life learning environment when learning via a simulation training, further research is needed to specify how this influences the learning process.

Suggestions for Future Research

Future research should explore the underlying cognitive mechanisms influencing learning processes during simulation-based training. For instance, by applying recent technologies such as eye tracking, researchers could clarify which information learners focus on during training (Gegenfurtner & Seppänen, 2013). Including a cognitive load measure would also help to understand how much cognitive load the training produces and how this impacts learning. This would clarify the impact of cognitive load and attention span during training.

To better comprehend the impact of situational variation, it may help to explore a curvilinear relation of situational variation and learning effectiveness. Too little variation is not beneficial, but our pilot study suggests that excessive variation is also not ideal. For this purpose, a study with more levels of situational variation should be carried out.

Another underexplored simulation characteristic that might impact the participants’ experience is the role they are required to adopt in different situations. In other words, the perspective from which the participant is involved in the situation. In typical business simulations, participants are often required to take on the role of an employee or manager. For the purpose of developing a richer and broader mental model, it might be interesting to let participants solve situations from the perspectives of different stakeholders and adopt multiple roles. Rather than changing the context, the decision-making role can vary the perspective of the situation and thus the knowledge structures may be expanded.

An additional avenue for future research could be examining the option of micro-training formats (De Jans et al., 2017; Lukosch et al., 2016) to advance cognitive learning outcomes. As we observed during our pilot study, participants tend to drop out when they find the training too time and attention intensive. One way to solve this may be to provide different situations separately over a longer timespan instead of offering them during a single session, in essence a micro-training format. This option could reduce the likelihood of cognitive overload of the working memory since only one situation would need to be solved at a given time.

Conclusion

A growing body of evidence suggests that simulation-based training can be effective for a variety of learning outcomes, sometimes outperforming traditional learning settings (Gegenfurtner et al., 2014; Sitzmann, 2011; Vogel et al., 2006). Due to its appealing design characteristics and their potential for deeply engaging learners, simulation-based training has become a popular method to improve cognitive outcomes. Organizations seeking to develop effective simulations are often confronted with various design question, for which evidence-based guidelines are not yet available. The current study provides a first answer to a very basic, but important design question: Does situational variation in training simulation make a difference in learning effectiveness? Using an experimental design, we showed that high situational variation during a simulation training improved complex cognitive outcomes such as innovation knowledge, irrespective of the learners’ prior knowledge about innovation. This finding provides a first step towards a more nuanced understanding of how design characteristics may influence learning through simulation.

Supplemental Material

Appendix_A – Supplemental material for The Effects of Situation Variability in a Simulation-Based Training for Implicit Innovation Knowledge

Supplemental material, Appendix_A for The Effects of Situation Variability in a Simulation-Based Training for Implicit Innovation Knowledge by Saar Van Lysebetten, Frederik Anseel and Diana R. Sanchez in Simulation & Gaming

Footnotes

Acknowledgements

This research was supported by a Baekeland scholarship of Flanders Innovation & Entrepreneurship and CREAX granted to the first author. We would like to thank VLAIO and CREAX to make this research possible.

Authors’ Note

Frederik Anseel is now affiliated with the University of New South Wales, Sydney, Australia.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Saar Van Lysebetten is a human capital consultant, which obtained her PhD at Ghent University, Belgium. Her research focuses on innovation, mental models, and simulation- and game-based training.

Contact: saarvanlysebetten@gmail.com

Frederik Anseel is professor of Management and Associate Dean Research at UNSW Business School, Sydney. His research examines the psychological microfoundations of organizational learning, innovation and entrepreneurship and has been published in leading management journals.

Contact: f.anseel@unsw.edu.au

Diana R. Sanchez is an assistant professor of Psychology at San Francisco State University. Her research explores the use of technology in the workplace with an emphasis on the use of simulations and virtual environments for assessment and training purposes.

Contact: sanchezdianar@gmail.com

References

Anderson

J. R.

(1982). Acquisition of cognitive skill. Psychological Review, 89(4), 369–406. https://doi.org/10.1037//0033-295x.89.4.369

Anderson

De Dreu

C. K.

Nijstad

B. A.

(2004). The routinization of innovation research: A constructively critical review of the state-of-the-science. Journal of Organizational Behavior, 25(2), 147–173. https://doi.org/10.1002/job.236

Anderson

Potočnik

Zhou

(2014). Innovation and creativity in organizations: A state-of-the-science review, prospective commentary, and guiding framework. Journal of Management, 40(5), 1297–1333. https://doi.org/10.1177/0149206314527128

Anderson

P. H.

Lawton

(2009). Business simulations and cognitive learning: Developments, desires, and future directions. Simulation & Gaming, 40(2), 193–216. https://doi.org/10.1177/1046878108321624

Baddeley

(1992). Working memory. Science, 255(5044), 556–559. https://doi.org/10.1126/science.1736359

Baddeley

(2003). Working memory: Looking back and looking forward. Nature Reviews Neuroscience, 4(10), 829–839. https://doi.org/10.1038/nrn1201

Bandura

(1977). Social learning theory. Prentice Hall.

Beal

S. A.

Christ

R. E.

(2004). Training effectiveness evaluation of the Full Spectrum Command game (Technical Report No. 1140). U.S. Army Research Institute for the Behavioral and Social Sciences.

Bedwell

W. L.

Pavlas

Heyne

Lazzara

E. H.

Salas

(2012). Toward a taxonomy linking game attributes to learning an empirical study. Simulation & Gaming, 43(6), 729–760. https://doi.org/10.1177/1046878112439444

10.

Bell

B. S.

Kanar

A. M.

Kozlowski

S. W. J.

(2008). Current issues and future directions in simulation-based training in North America. International Journal of Human Resource Management, 19(8), 1416–1434. https://doi.org/10.1080/09585190802200173

11.

Birdi

Leach

Magadley

(2016). Factors influencing innovative behavior. Journal of Product Innovation Management, 33, 19–35. https://doi.org/10.1111/jpim.12250

12.

Bouhnik

Marcus

(2006). Interaction in distance-learning courses. Journal of the American Society for Information Science and Technology, 57(3), 299–305. https://doi.org/10.1002/asi.20277

13.

Brown

K. G.

(2001). Using computers to deliver training: Which employees learn and why? Personnel Psychology, 54(2), 271–296. https://doi.org/10.1111/j.1744-6570.2001.tb00093.x

14.

Cannon-Bowers

J. A.

Bowers

(2009). Synthetic learning environments: On developing a science of simulation, games, and virtual worlds for training. In Kozlowski

S. W. J.

Salas

(Eds.), Learning, training, and development in organizations (pp. 229–263). Routledge.

15.

Cannon-Bowers

J. A.

Bowers

Procci

(2010). Optimizing learning in surgical simulations: Guidelines from the science of learning and human performance. Surgical Clinics of North America, 90(3), 583–603. https://doi.org/10.1016/j.suc.2010.02.006

16.

Chi

M. T. H.

Feltovich

P. J.

Glaser

(1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152. https://doi.org/10.1207/s15516709cog0502_2

17.

Consorti

Mancuso

Nocioni

Piccolo

(2012). Efficacy of virtual patients in medical education: A meta-analysis of randomized studies. Computers & Education, 59(3), 1001–1008. https://doi.org/10.1016/j.compedu.2012.04.017

18.

Cook

M. P.

(2006). Visual representations in science education: The influence of prior knowledge and cognitive load theory on instructional design principles. Science Education, 90(6), 1073–1091. https://doi.org/10.1002/sce.20164

19.

Dane

Pratt

M. G.

(2007). Exploring intuition and its role in managerial decision making. Academy of Management Review, 32(1), 33–54. https://doi.org/10.5465/amr.2007.23463682

20.

Dane

Rockmann

K. W.

Pratt

M. G.

(2012). When should I trust my gut? Linking domain expertise to intuitive decision-making effectiveness. Organizational Behavior and Human Decision Processes, 119(2), 187–194.

21.

Day

E. A.

Arthur

Jr. Gettman

(2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86(5), 1022–1033. doi:10.1037/0021-9010.86.5.1022

22.

De Jans

Van Geit

Cauberghe

Hudders

De Veirman

(2017). Using games to raise awareness: How to co-design serious mini-games? Computers & Education, 110, 77–87. https://doi.org/10.1016/j.compedu.2017.03.009

23.

Ericsson

K. A.

Simon

H. A.

(1993). Protocol analysis. Cambridge, MA: MIT press.

24.

Garris

Ahlers

Driskell

J. E.

(2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467. https://doi.org/10.1177/1046878102238607

25.

Gary

M. S.

Wood

R. E.

(2011). Mental models, decision rules, and performance heterogeneity. Strategic Management Journal, 32(6), 569–594. https://doi.org/10.1002/smj.899

26.

Gary

M. S.

Wood

R. E.

Pillinger

(2012). Enhancing mental models, analogical transfer, and performance in strategic decision making. Strategic Management Journal, 33(11), 1229–1246. https://doi.org/10.1002/smj.1979

27.

Gathercole

S. E.

Alloway

T. P.

(2012). Working memory and learning. In International Journal of Psychology, Vol. 47 (pp. 592–592). East-Sussex, England: Psychology Press.

28.

Gavetti

Levinthal

(2000). Looking forward and looking backward: Cognitive and experiential search. Administrative Science Quarterly, 45(1), 113–137. https://doi.org/10.2307/2666981

29.

Gegenfurtner

Quesada-Pallares

Knogler

(2014). Digital simulation-based training: A meta-analysis. British Journal of Educational Technology, 45(6), 1097–1114. https://doi.org/10.1111/bjet.12188

30.

Gegenfurtner

Seppänen

(2013). Transfer of expertise: An eye tracking and think aloud study using dynamic medical visualizations. Computers & Education, 63, 393–403. https://doi.org/10.1016/j.compedu.2012.12.021

31.

Gentner

Loewenstein

Thompson

(2003). Learning and transfer: A general role for analogical encoding. Journal of Educational Psychology, 95(2), 393–408. https://doi.org/10.1037/0022-0663.95.2.393

32.

Gentner

Stevens

A. L.

(Eds.). (1983). Mental models. Lawrence Erlbaum.

33.

Hoffman

Shadbolt

N. R.

Burton

A. M.

Klein

(1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Decision Processes, 62(2), 129–158.

34.

Johnson-Laird

P. N.

(1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press.

35.

Jones

(1981). Inter-rater reliability in the assessment of group exercises at a UK assessment centre. Journal of Occupational Psychology, 54(2), 79–86. https://doi.org/10.1111/j.2044-8325.1981.tb00046.x

36.

Lacerenza

C. N.

Reyes

D. L.

Marlow

S. L.

Joseph

D. L.

Salas

(2017). Leadership training design, delivery, and implementation: A meta-analysis. Journal of Applied Psychology, 102(12), 1686–1718. https://doi.org/10.1037/apl0000241

37.

Lainema

Nurmi

(2006). Applying an authentic, dynamic learning environment in real world business. Computers & Education, 47(1), 94–115. https://doi.org/10.1016/j.compedu.2004.10.002

38.

Landers

R. N.

Reddock

C. M.

(2017). A meta-analytic investigation of objective learner control in web-based instruction. Journal of Business and Psychology, 32(4), 455–478. https://doi.org/10.1007/s10869-016-9452-y

39.

Leemkuil

Jong

T. D.

(2011). Instructional support in games. In Tobias

Fletcher

J. D.

(Eds.), Computer games and instruction (pp. 353–369). Charlotte, NC: Information Age Publishing.

40.

Liaw

S. S.

(2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education, 51(2), 864–873. https://doi.org/10.1016/j.compedu.2007.09.005

41.

Lievens

Anseel

(2007). Creating alternate in-basket forms through cloning: Some preliminary results. International Journal of Selection and Assessment, 15(4), 428–433. doi:10.1111/j.1468-2389.2007.00401.x

42.

Lovelace

K. J.

Eggers

Dyck

L. R.

(2016). I do and I understand: Assessing the utility of web-based management simulations to develop critical thinking skills. Academy of Management Learning & Education, 15(1), 100–121. https://doi.org/10.5465/amle.2013.0203

43.

Lukosch

Kurapati

Groen

Verbraeck

(2016). Microgames for situated learning: A case study in interdependent planning. Simulation & Gaming, 47(3), 346–367. https://doi.org/10.1177/1046878116635468

44.

Malone

T. W.

(1981). Toward a theory of intrinsically motivating instruction. Cognitive Science, 4, 333–369.

45.

Mayer

B. W.

Dale

K. M.

Fraccastoro

K. A.

Moss

(2011). Improving transfer of learning: Relationship to methods of using business simulation. Simulation &Gaming, 42(1), 64–84. https://doi.org/10.1177/1046878110376795

46.

Mayer

R. E.

(2011). Applying the science of learning to multimedia instruction. In Mestre

J. P.

Ross

B. H.

(Eds.), Psychology of learning and motivation (Vol. 55, pp. 77–108). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-387691-1.00003-X

47.

Moreno

Mayer

R. E.

(2005). Role of guidance, reflection, and interactivity in an agent-based multimedia game. Journal of Educational Psychology, 97(1), 117–128. https://doi.org/10.1037/0022-0663.97.1.117

48.

Ong

C. S.

Lai

J. Y.

(2006). Gender differences in perceptions and relationships among dominants of e-learning acceptance. Computers in Human Behavior, 22(5), 816–829. https://doi.org/10.1016/j.chb.2004.03.006

49.

Paas

Renkl

Sweller

(2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4. https://doi.org/10.1207/s15326985ep3801_1

50.

Palmunen

L. M.

Pelto

Paalumäki

Lainema

(2013). Formation of novice business students’ mental models through simulation gaming. Simulation & Gaming, 44(6), 846–868. https://doi.org/10.1177/1046878113513532

51.

Rogers

(2011). Developing simulations in multi-user virtual environments to enhance healthcare education. British Journal of Educational Technology, 42(4), 608–615. https://doi.org/10.1111/j.1467-8535.2010.01057.x

52.

Rutten

Van Joolingen

W. R.

Van Der Veen

J. T.

(2012). The learning effects of computer simulations in science education. Computers & Education, 58(1), 136–153. https://doi.org/10.1016/j.compedu.2011.07.017

53.

Sanchez

Van Lysebetten

(2017). Findings from a meta-analysis on training games and learning outcomes: Future directions. In Armstrong

M. B.

(Chair), Sanchez

D. R.

(Co-Chair), Bauer

K. N.

(Co-Chair), & Kraiger

(Discussant), Gaming and gamification IGNITE: Current trends in research and application [Symposium]. 32nd Annual Conference of the Society for Industrial and Organizational Psychology, Orlando, FL, United States.

54.

Scherpereel

C. M.

(2005). Changing mental models: Business simulation exercises. Simulation & Gaming, 36(3), 388–403. https://doi.org/10.1177/1046878104270005

55.

Scott

S. G.

Bruce

R. A.

(1994). Determinants of innovative behavior: A path model of individual innovation in the workplace. Academy of Management Journal, 37(3), 580–607. https://doi.org/10.2307/256701

56.

Siewiorek

Gegenfurtner

Lainema

Saarinen

Lehtinen

(2013). The effects of computer-simulation game training on participants’ opinions on leadership styles. British Journal of Educational Technology, 44(6), 1012–1035. https://doi.org/10.1111/bjet.12084

57.

Sitzmann

(2011). A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Personnel Psychology, 64(2), 489–528. https://doi.org/10.1111/j.1744-6570.2011.01190.x

58.

Sitzmann

Ely

Bell

B. S.

Bauer

(2010). The effects of technical difficulties on learning and attrition during online training. Journal of Experimental Psychology: Applied, 16(3), 281–292. https://doi.org/10.1037/a0019968

59.

Sun

P. C.

Tsai

R. J.

Finger

Chen

Y. Y.

Yeh

(2008). What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1183–1202. https://doi.org/10.1016/j.compedu.2006.11.007

60.

Sweller

(1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4

61.

Tennyson

R. D.

Jorczak

R. L.

(2008). A conceptual framework for the empirical study of instructional games. In O’Neil

Perez

(Eds.), Computer games and team and individual learning (pp. 39–54). Oxford, UK: Elsevier.

62.

Tharenou

(2001). The relationship of training motivation to participation in training and development. Journal of Occupational and Organizational Psychology, 74(5), 599–621. doi:10.1348/096317901167541

63.

Training Industry Report (2017). Training Magazine (November/December), 20–33.

64.

Van Gerven

P. W. M.

Paas

van Meriënboer

J. J. G.

Schmidt

H. G

. (2006). Modality and variability as factors in training the elderly. Applied Cognitive Psychology, 20(3), 311–320. https://doi.org/10.1002/acp.1247

65.

van Gog

Paas

Sweller

. (2010). Cognitive load theory: Advances in research on worked examples, animations, and cognitive load measurement. Educational Psychology Review, 22(4), 375–378. https://doi.org/10.1007/s10648-010-9145-4

66.

van Merriënboer

J. J. G.

Kester

Paas

. (2006). Teaching complex rather than simple tasks: Balancing intrinsic and germane load to enhance transfer of learning. Applied Cognitive Psychology, 20(3), 343–352. https://doi.org/10.1002/acp.1250

67.

Vogel

J. J.

Vogel

D. S.

Cannon-Bowers

Bowers

C. A.

Muse

Wright

(2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34(3), 229–243. https://doi.org/10.2190/FLHV-K4WA-WPVQ-H0YM

68.

Wilson

K. A.

Bedwell

W. L.

Lazzara

E. H.

Salas

Burke

C. S.

Estock

J. L.

Orvis

K. L.

Conkey

(2009). Relationships between game attributes and learning outcomes: Review and research proposals. Simulation & Gaming, 40(2), 217–266. doi:10.1177/1046878108321866

69.

Wouters

Paas

van Merriënboer

J. J. M.

(2008). How to optimize learning from animated models: A review of guidelines base on cognitive load. Review of Educational Research, 78, 645–675. https://doi.org/10.3102/0034654308320320

70.

Wouters

van Oostendorp

(2017). Overview of instructional techniques to facilitate learning and motivation of serious games. In Wouters

van Oostendorp

(Eds.), Advances in game-based learning. Instructional techniques to facilitate learning and motivation of serious games (pp. 1–16). New York, NY: Springer.

71.

Yuan

Woodman

R. W.

(2010). Innovative behavior in the workplace: The role of performance and image outcome expectations. Academy of Management Journal, 53, 323–342. https://doi.org/10.5465/amj.2010.49388995

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.17 MB