Mental Models: Structural Differences and the Role of Experience

Abstract

Mental models are mental representations of the external world that humans constantly use when they interact with the environment and systems within it. These mental models are in part constituted by an underlying structure of associated concepts that are modified as a person gains experience with a system or domain. Video games provide a context that encourages the development of sophisticated mental models. The current research sought to understand how mental model structures differ between video game players of varying experience levels. Participants were recruited both over internet forums and through Mechanical Turk. Mental model structures were measured using relatedness ratings between pairs of concepts that were derived from players with high levels of experience playing League of Legends. Relatedness ratings were transformed into Pathfinder networks that were used to analyze mental model structures. Results revealed structural differences in mental models between experience levels. A three-stage model of mental model structure development is proposed to explain the results, which suggest that some structural characteristics appear earlier in mental model development than others. The role of mental model structural characteristics is discussed in light of both the design of training programs and video games.

Keywords

declarative memory experience knowledge organization knowledge structure mental models pathfinder network topics training video game

People do not interact with technological artifacts; they interact with their mental models of those artifacts (e.g., Norman, 1983). Consequently, people frequently use mental models as they complete daily tasks. Simple tasks, such as driving to work, operating a computer, and cooking a meal require the use of mental models for successful completion (Gentner, 2002; Norman, 1983). Mental models are also useful when performing more complex tasks, such as air traffic control and the medical care activities by physicians and nurses (Mogford, 1997; Smith & Koppel, 2014). The pervasiveness of mental models in routine human activity, especially as it relates to interaction with technology, reveals the importance of understanding how these models are formed, used, and adjusted over time. To further our understanding of mental models, the current research examined how the memory structures of mental models differed between people who have varying degrees of experience playing a video game.

Mental Models

Defining the term

The roots of the term mental model can be traced back to Kenneth Craik’s (1943) book The Nature of Explanation. Craik suggested that humans hold small-scale models of reality in their minds that are used to reason and solve problems in their environment. Later, Johnson-Laird (1983) posited that people create analogical models of aspects of the environment, which they use in deductive problem solving. Another pioneer of mental models, Norman (1983), suggested that people construct internal representations of the systems with which they interact and that these representations “provide predictive and explanatory power for understanding the interaction” (p. 7). He described mental models as knowledge about objects and the environment gathered from interaction with that environment. Rouse and Morris (1986) warned that defining mental models too broadly could result in a definition that is no different from general knowledge and provides no additional theoretical or practical utility. We suggest a useful definition of mental models that avoids these concerns. First, mental models are representations that are both stored in long-term memory and accessible to working memory. Second, they are memory representations of systems or situations. Finally, mental models help predict the future states of, as well as reason with, those systems and situation.

Influence on behavior in human-systems interaction

Mental model theory not only describes representations in long-term memory but also how models influence behavior. Gentner and Stevens (1983) proposed that the concept of a mental model is one that has usefulness in understanding of the behavior of humans interacting with systems. Much research related to mental models supports Gentner and Stevens’ claim. Kieras and Bovair (1984) found that users who developed a more complete mental model of a device showed increased performance and learning. Slone (2002) reported that people with different mental models of internet searching engaged in different types of web-searching behaviors. Dinet and Kitajima (2011) found similar results in their study, which reported that types of mental models in children were predictive of performance in web-searching tasks. Mental model completeness was found to be predictive of how much users trusted an adaptive cruise control mechanism (Beggiato & Krems, 2013). Bader and Beyerer (2011) suggested that the state of a person’s mental model can predict gaze patterns during human-computer interaction tasks. Fein, Olson, and Olson (1993) found that groups who acquired mental models yielded higher performance using a complex device than those who did not.

Characteristics of mental models

Mental models have been observed to have a number of distinct characteristics, in addition to those noted above—that they involve knowledge representations in memory of external phenomena and that they have predictive power. The primary mechanism of prediction in mental models is generally thought to be mental simulation (Gentner, 2002). Mental simulation involves completing a series of actions in working memory often in a visual format (Landriscina, 2013). Simulating a model allows the individual to predict future states of a system, which can alter how the user chooses to interact with that system. Gentner (2002) suggested that people simulate future states of a system when needed and simply retrieve stored knowledge about future states of the system when such knowledge is available. The latter event is also related to automatic processing characterized by quickly executed, low-effort actions that rely on well-defined existing knowledge and differs from slow mental simulation that places high demands on cognitive resources (Anderson, 1983; Kahneman, 2011; Shiffrin & Schneider, 1977).

Another central characteristic of mental models is that they have an organized structure in long-term memory. Mental models can contain numerous types of informational content—static declarative information, such as names and functions of system components; causal information about how these components interact with one another to produce effects (Mayer, Mathias, & Wetzell, 2002); and procedural knowledge about how to operate functions of the system to produce desired outcomes (Zhang, 2013). This procedural knowledge, much like the knowledge gained from repeated mental simulation, contributes to automatic systems of cognitive processing (Anderson, 1983; Kahneman, 2011; Shiffrin & Schneider, 1977). The format of mental model content is flexible. Model representations in memory have been suggested to be conceptual or propositional (Doyle & Ford, 1998) as well as image-like (Rouse & Morris, 1986).

Mental model structures may be organized as networks with information chunks and concepts existing as nodes and links representing associative connections between those nodes (Doyle & Ford, 1998; Lokuge, Gilbert, & Richards, 1996). The associative connections between nodes can be semantically meaningful (e.g., Node X is subordinate to Node Y, such as Bird has Feathers) but are often framed in terms of general relatedness (Node X is highly related to Node Y). The structure of a mental model provides the foundation from which behaviors with a system arise. Mental simulations and predictions rest upon the underlying model content and structure (Jones, Ross, Lynam, Perez, & Leitch, 2011; Wilson & Rutherford, 1989). The organization provided by a mental model structure is not only used for mental simulation and prediction, though. It has been demonstrated that knowledge that is organized is easier to recall than unorganized knowledge (Cooke, Durso, & Schvanevedlt, 1986; Tulving, 1962). This is consistent with spreading activation theory as presented in the seminal work of Collins and Loftus (1975) and Quillian (1966). Mental model structures both facilitate mental simulation and aid in recall of declarative knowledge. However, in order to fully understand mental model structure, it is necessary to also examine how structures are modified and developed over time.

Model construction and development

Studies of people in the process of developing mental models have allowed researchers to begin to understand the mental model construction process (Mayer et al., 2002; Katzeff, 1990; Savage-Knepshield, 2001). Early versions of the model are often based on previously developed models of similar phenomena (Cool, Park, Belkin, Koenemann, & Ng, 1996; Marchionini, 1989; Savage-Knepshield, 2001). People also make full use of new system features by recognizing the need to manipulate an existing model or create a new model, when the current model is insufficient, by assimilating new information into an existing model, eliminating old information, and adjusting existing information (Zhang, 2009).

A second finding derived from the observation of people developing mental models is that constructing a mental model often involves multiple stages. For example, Mayer et al. (2002) proposed (a) a first stage, which involves identifying each component of the system including all of the possible states that each component can take independently of other components, and (b) a second stage, in which the components are integrated into a model on the basis of the causal relationships among model components, including how each component interacts with other components. Katzeff (1990) proposed that mental model construction consists of three phases: construction, testing, and running. These phases represent the mental model being created, edited, and used for prediction and mental simulation, respectively.

With the focus on different stages of mental model development has come an interest in comparison of the mental models of novices and experts. Researchers have proposed a number of characteristics that are identified with mental model structures at different levels of experience with a system or domain, as can be seen in Table 1. In addition, greater similarity of mental model structure of a novice and that of an expert is related to an improvement in the novice’s performance (Day, Arthur, & Gettman, 2001; Goldsmith, Johnson, & Acton, 1991; Kraiger, Salas, & Cannon-Bowers, 1995).

Table 1:

Mental Model Structural Characteristics by Experience

Characteristic	Low Experience	High Experience	References
Level of abstraction	Associations are based on surface features and concentrated around concrete objects.	Associations are based on conceptual features and based around abstract concepts.	Bradley, Paul, & Seeman, 2006; Graham, Zheng, & Gonzalez, 2006
Density	Networks contain fewer links compared to those with more experience.	Networks contain more links compared to those with less experience.	Bradley et al., 2006; Zielinska, Welk, Mayhorn, & Murphy-Hill, 2015
Centralization	Subnetworks of concepts are less common and make up less of the overall network.	Subnetworks are more common and take up a substantial amount of the network.	Gillan, Breedin, & Cooke, 1992
Semantic	Associations are made based on meanings of words in natural language instead of domain language.	Associations are made based on meanings of words in the language of the domain.	Cooke & Schvaneveldt, 1988; Sebrechts et al., 1983
Procedural	Associations are not based on concepts frequently used together in procedures.	Subgroups and associations are often based on frequently used procedures.	Kay & Black, 1984

Table 1 provides a summary of relevant literature related to the novice and expert stages of mental model development. Much of the research has been performed in only a few domains such as computer-science-related experience (e.g., Cooke & Schvaneveldt, 1988; Gillan et al., 1992; Kay & Black, 1984; Zielinska et al., 2015) and education (e.g., Bradley et al., 2006). The nature of the domains could limit or shape how model structures are developed in a way that is not generalizable to other domains with different types of content. In addition, many of the analyses of mental model characteristics, such as subnetwork-based and natural-language-based structures, are made without any accompanying inferential statistical analysis (e.g., Cooke & Schvaneveldt, 1988; Gillan et al., 1992; Graham et al., 2006). The lack of inferential statistics weakens the claims that can be made about mental model structural differences. Lastly, with regards to model structures at different levels of system or domain experience, the literature rarely investigates beyond the simple expert-novice distinction. There is little examination of mental model structures at finer grains of experience. As a consequence, we lack a more complete understanding of how mental model structures develop over time and with experience. The current research sought to examine mental model structures in a context that differs from those typically studied and at more than two levels of experience. The typical areas of study mentioned earlier of computer-science-related activities and training are well-defined domains that offer mostly structured learning that is not typically social in nature. It could be beneficial to extend research to a domain with other characteristics, such as video games. Video games differ from these domains in that they contain both structured (e.g., tutorials) and unstructured (e.g., unguided exploration) learning. Video games can also be played socially and competitively. These are some ways in which video game play differs from text editing (Kay & Black, 1984), computer programming (Cooke & Schvaneveldt, 1988), and instructional training (Bradley et al., 2006), for example.

Mental models of video games

Video games have advanced rapidly in recent decades and are a pervasive source of entertainment in modern society, with an estimated 155 million gamers in the United States and four out of every five households containing at least one member who plays video games (Electronic Software Association, 2015). Many of those who play video games do so regularly: 42% of Americans play at least 3 hours per week (Electronic Software Association, 2015). People spend less time engaging with other sources of entertainment, such as television, as video games rise in popularity. Consumers in the United States alone spent more than $25 billion on video games in 2014.

In addition to their influence in the consumer entertainment industry, video games have been used for educational purposes (Papastergiou, 2009), training of complex skills (Gopher, Well, & Bareket, 1994; Schlickum, Hedman, Enochsson, Kjellin, & Felländer-Tsai, 2009), improvement of cognitive ability (Feng, Spence, & Pratt, 2007; Green & Bavelier, 2007; Li, Polat, Makous, & Bavelier, 2009), and rehabilitation of physical and mental disabilities (Deutsch et al., 2011; Larose, Gagnon, Ferland, & Pépin, 1989; Rand, Kizony, & Weiss, 2008). Video games designed or augmented for educational purposes, that is, serious games, have shown some advantages to traditional instruction methods for engendering learning (Wouters, Van Nimwegen, Van Oostendorp, & Van Der Spek, 2013). The complexity of and learning the skill development supported by video game play are likely major contributors in the rise of video game popularity in various research and educational domains. Given the propensity of video games to produce declarative learning and procedural skill acquisition (Anderson, 1983; Gee, 2003), it should be expected that they also produce an underlying mental model.

Video games offer a potentially productive domain of research for mental model theory. Playing complex video games requires learning and using both declarative and procedural knowledge. Additionally, individuals who play multiple video games must engage in the transfer of mental models from one game to another. Research has shown that effortful processing can alter how easily information is learned (Hasher & Zacks, 1979). Video games have been shown to be particularly motivating, especially within the framework of Self-Determination Theory (Ryan, Rigby, & Przybylski, 2006). Given the motivational nature of video games, it is reasonable to think that video games encourage players to engage in effortful processing more often than with many other technologies. The mental model structures of video game players should reflect this. Model structures might be more complex, rich, stable, or optimized due to the increase in effortful processing and time spent learning, although the relationship between mental model development and other aspects of skill development is unclear. Lastly, the large range in time spent playing video games (Electronic Software Association, 2015) is likely to result in a wide range of mental models, from those with almost no experience to those who have spent thousands of hours engaging in game play. This provides researchers with a potentially large, naturally occurring pool of participants with mental models that vary by stages based on different amounts of experience.

Unfortunately, little research has focused on the development of mental models during video game play. In one exception, Graham et al. (2006) performed a pilot study examining how mental models of participants playing a real-time strategy video game changed over time. They posited that novice players would construct models of video game elements based on the physical features perceived and that over time the model would develop into a structure based on object function. Although there was a limited sample of five participants, they found that at least some of the initial model structures could be categorized as being built around physical features. As model structures changed over time, though, they did not follow the author’s hypothesis and produced a variety of structure types, some of which could not be categorized. This line of exploratory research sets the ground work for research into how additional time spent playing a video game alters mental model structure. Additionally, Graham et al. (2006) limited their study to the structure and organization of mental model content while ignoring the amount of content. In addition to measuring how experiencing a video game alters model structure, effects on the amount of content should also be examined.

Research Questions

The development of mental models in video games is an important area of research, both theoretically and practically, which has been underexplored. The current research was designed to investigate the relation between specific characteristics of mental model structures at multiple levels of experience. Additionally, the domain of video games has been left largely untouched but presents a profitable area of research for mental model theory. The current research conducted a study that examined the following broad research question: How do mental model structures differ across three levels of experience playing a video game?

Method

The current research consisted of a preliminary study and a main study. The preliminary study determined the materials needed for the main study. Associations between concepts in a mental model structure can be measured by presenting all possible pairs of words from a list and requiring participants to conduct ratings of relatedness or similarity on a numerical scale (e.g., Cooke et al., 1986; Cooke & Schvaneveldt, 1988; Goldsmith et al., 1991; Kraiger et al., 1995). This study used relatedness ratings to measure mental model structure in this way. However, prior to the presentation of relatedness ratings, a list of concepts needed to be generated that could be compared. Goldsmith and Kraiger (1997) detailed one process for generating a list of concepts. The primary methods by which a list can be generated are (a) examining related source materials (e.g., training manuals) and (b) interviewing domain experts or those with extensive experience. Given the lack of previous research in the domain, as well as the lack of sophisticated instructional materials, the current research used the interview method to elicit a list of concepts related to a video game. Due to the difficulty of defining and finding experts, though, the interview method was conducted with participants of varying experience levels.

Preliminary Study

For the preliminary study, four participants took part. The ages of participants spanned from 20–24 years (M = 21.5, SD = 1.7, male = 4, female = 0). The Electronic Software Association (2015) reported that the most frequent game players who played socially spent an average of 6.5 hours per week playing with other people, including online play. People who spent on average at least 6 hours per week playing the chosen video game League of Legends, which will be described in the next section, were deemed eligible to participate. Although 6 hours per week was the minimum requirement for eligibility, most participants far exceeded this requirement, as is indicated in Table 2. The sample was a convenience sample. Participants were approached by the experimenter and asked to participate in the study. Participants were not compensated for taking part in this portion of the study.

Table 2:

Characteristics of Interview Participants

Participant ID	Gender	Age	Hours Played per Week	Total Hours Played
1	Male	21	40	1,300
2	Male	21	20	384
3	Male	24	8	100
4	Male	20	20	3,500

Note. Data concerning the number of hours played per week and in total are estimates for time spent playing the video game League of Legends.

A paper questionnaire was used to record the following demographic information:

gender

age

estimated hours per week spent playing League of Legends, and

estimated total hours spent playing League of Legends.

The questionnaire also included two interview items. The interview items (a) asked participants to engage in a concept listing task wherein they listed as many terms related to the video game League of Legends as they could in a 5-minute time frame and (b) asked participants to indicate which of the listed terms are related or often used together when accomplishing goals in the game. The video game that was used for both the preliminary study and the main study was League of Legends. League of Legends is currently the most popular online video game with over 67 million players every month and over 27 million players every day according to publisher Riot Games (“Our Games,” n.d.). In addition to its widespread appeal, it also fosters one of the most active professional gaming communities. League of Legends is an online team-based arena game in which players assume the role of a character matched with a team of players and pitted against another set of human-controlled characters. All characters have a unique set of abilities that players use to destroy the opponents’ units and structures. Players are able to choose a new character for each game session. The popularity, complexity, and diversity of player experience made the game an excellent fit for the goals of the present study. Participants were greeted and given the demographic portion of the questionnaire. The preliminary study resulted in a large list of 143 concepts that could be used for the main study.

Main Study

Pathfinder

Prior to analyzing mental model structures, these structures need to be represented in a meaningful way. Mental model structures can be represented as Pathfinder networks. Pathfinder analysis uses a dissimilarity matrix as an input and provides a network representation of the matrix data as an output. Pathfinder networks contain three central features: nodes, links, and link weights. Nodes are input terms or objects. Links are weighted paths between nodes that represent a meaningful connection. Link weights, which are often hidden in the graphical representation, indicate the strength of the connection (i.e., link) between two nodes. The Pathfinder analysis eliminates links between pairs of nodes until the only remaining links are those that represent the strongest connections between nodes. This is accomplished by using a triangular inequality theorem method such that if the link between any two nodes is weaker than some other path of links that connects those two nodes, that link is eliminated. Thus, a Pathfinder network represents the most meaningful links between nodes given the parameter settings. There are two primary parameter settings, which act as constraints on the network. The q parameter determines the maximum number of links that are allowed in a path of links, such as to limit to maximum length of a path. The Minkowski r distance parameter determines how path weights are calculated. The most common parameter settings for similarity data, and those that were used in this study, are q = n – 1, r = ∞. More details on the Pathfinder algorithm can be found in Schvaneveldt, Durso, and Dearholt (1989). Pathfinder networks have been shown to be a valid method of representing mental model structures (Day et al., 2001; Goldsmith et al., 1991) and, in some cases, have even been demonstrated to be more valid than the raw data itself (Cooke et al., 1986).

Objectives

This study sought to compare mental model structures of League of Legends players of varying levels of experience. More specifically, this study aimed to compare these structures by eliciting and analyzing their distinctive structural characteristics using Pathfinder analysis.

Design

A cross-sectional design was implemented. A cross-sectional design collects data from a sample of individuals in a population at a single point in time and allows different portions of the sample to be compared. The cross-sectional design of the study did pose a limitation in that longitudinal changes in mental models could not be directly observed. However, this design did allow a diversity of mental models to be examined simultaneously.

Independent variables

The independent variable was considered the amount of experience participants had with League of Legends. The independent variable contained three levels of experience (low, medium, high). Participants were split into three groups of experience based on the number of total hours that they indicated they had played League of Legends (low = 1–100 hours, medium = 101–1,000 hours, high = 1,001 or more hours). These thresholds resulted in a relatively balanced split of low (n = 40), medium (n = 52), and high (n = 66) experience groups. Establishing experience thresholds to create groups has been utilized in prior mental model research (e.g., Cooke & Schvaneveldt, 1988). Due to the cross-sectional design, participants could not be randomly assigned to groups but rather were grouped based on their current experience. Age and gender were examined to take note of any grouping bias. A one-way between-subjects ANOVA revealed that the effect of age was significant F(2, 155) = 7.79, p < .001, η² = .09. A Fisher’s LSD post hoc test revealed participants in the high experience group (M = 37.08, SD = 7.28) were significantly older in age than those in the low experience group (M = 31, SD = 7.97) (p < .05), but no effect was detected for the medium experience group (M = 34.77, SD = 7.95). A chi-square test of independence revealed that there was no significant effect of gender between groups, χ²(2, n = 158) = 1.29, p = .53.

Dependent variables

The primary dependent variable of interest was relatedness rating responses. The questionnaire used in this study contained a set of relatedness ratings based on a list of concepts derived from the preliminary study. From the large list of concepts that were generated in the preliminary study, 20 were chosen for relatedness rating comparisons. Two judges, who had some expertise in research methods and cognitive psychology, categorized all concepts into two groups: abstract (e.g., ideas, strategies) and concrete (e.g., objects). Interrater agreement was 76%. For the categorizations on which experimenters agreed (which were the only ones eligible for the final list), 25 concepts were categorized as concrete and 84 as abstract. The experimenter then chose 10 terms that were categorized as abstract and 10 that were categorized as concrete, resulting in a list of 20 concepts. Concepts were chosen in part based on the frequency with which they were listed. For example, if two concepts were being compared, other factors being held equal, frequency acted as a tiebreaker and the concept that was written down by more participants was selected. Experimenters also attempted to select a wide range of concepts that addressed different aspects of the game, based on their judgement. Prior to conducting the ratings, participants were shown the 20 individual concepts that they would subsequently compare. This is important as it provides a contextual frame of reference for making meaningful comparisons (Clariana, 2010; Cooke & Schvanevedlt, 1988). Goldsmith and Kraiger (1997) indicated that more concepts are generally associated with higher validity and that their research has typically included at least 20 concepts. Ratings of the pairs of concepts used a scale of 1 (high unrelated) to 9 (highly related). In order to ensure that participants were completing relatedness ratings correctly, they were given initial practice trials and an attention check trial. The practice trials consisted of four relatedness rating items where the terms were common kitchen items (e.g., spoon and pot). The attention check trial was a single relatedness rating item that consisted of a repeated item from the set of 190 pairs of terms and was presented as the final trial (although participants were unaware that this was the final trial).

Relatedness rating responses were used as the inputs for Pathfinder analyses that provided a representation of mental model structure. Each of the structural characteristics identified earlier were operationalized so that they could be analyzed within Pathfinder networks. The degree to which mental model structures were organized around abstract connections (e.g., ideas, strategies) was operationalized as the proportion of links in a Pathfinder network where at least one of the nodes connected to a link was categorized as abstract. This is similar to how Bradley et al. (2006) operationalized abstract structures. Mental model structural density was considered the total number of links in a given Pathfinder network. This method is similar to how Zielinska et al. (2015) and Bradley et al. (2006) compared Pathfinder networks. The degree to which structures were organized around semantics, the meaning of words in natural language as opposed to the language of a domain, was determined by a two-step process. First, two judges, who were doctoral graduate students in an applied cognitive psychology program at North Carolina State University, categorized all 190 pairs of concepts as being related based on semantics in natural language or domain language. Judges were able to reconcile all discrepancies in order to establish 100% interrater agreement. These links were then counted in all individual Pathfinder networks, and the proportion of network connections that were categorized as natural language-based was determined. Participants in the preliminary study identified procedurally related concepts (e.g., steps/strategies taken to accomplish a goal). Each pair of connected concepts that were identified as being procedurally related were counted in all networks and the proportion of connections in a network that were procedurally related indicated how procedurally based a network was. Lastly, the extent to which mental model structures were organized with central nodes and subnetworks was operationalized. This study used a modified version of the criteria for central nodes and subnetworks found in Gillan et al. (1992). A subnetwork was counted if a combination of links in a Pathfinder network met the following conditions:

a series of nodes that begins and ends with the same node,

all nodes in the network connect to at least two other nodes in the network,

at least half of the nodes in the network connect to at least three other nodes in the network,

no node in the series is separated by more than two links,

a series of nodes is not considered a subnetwork if it contains half or more of the nodes in the entire network (i.e., 10 or more), and

a series of nodes is not considered a subnetwork if the network would contain more than five subnetworks.

These criteria were intended to identify small neighborhoods of tightly interconnected links that could be considered a subnetwork of links. Using these criteria helped ensure that only sets of links that met our definition of a subnetwork would be counted. A central node was considered a node with at least three links that was not part of a subnetwork. Subnetworks and central nodes were counted for all individual Pathfinder networks.

Participants also rated their experience with games in the same genre as League of Legends, multiplayer online battle arenas (MOBAs), on a scale of 1 (no experience) to 5 (high experience). This measure was taken to ensure that prior experience with similar games would not influence ratings more for one level of experience than another.

Participants

One hundred and sixty participants took part in the study. Two participants’ data were not included in the analysis because practice trials and the attention check indicated that their data were not valid. The remaining 158 participants (male = 133, female = 25) that participated in the study ranged in age from 18 to 80 years (M = 34.8, SD = 8). Participants were individuals with at least some experience playing League of Legends. The average estimated total number of hours of experience that participants had playing League of Legends was 1,423 (SD = 1,810). The estimated average number of hours per week spent playing was 11.9 (SD = 14.16). Participants were recruited through two methods including internet forums and Amazon’s Mechanical Turk. Those who participated were compensated $4 for participating in the study on Mechanical Turk. Participants who took part in the study through internet forums were given the opportunity to enter a raffle for one of three $25 Amazon gift cards. Participants were required to be at least 18 years old to participate.

Apparatus

Consent forms were developed for both forum and Mechanical Turk participants and were distributed to participants at the start of each session. An online questionnaire was developed that was used to collect all of the data in the study. A demographic questionnaire was also used to collect the same information collected in the preliminary study.

Procedure

Participants were first asked to read and agree to a consent form. Participants then completed the demographics questionnaire. Next, participants completed the relatedness ratings portion of the questionnaire, which included the practice trials, full set of randomized relatedness rating items, and attention check trial. Lastly, the Mechanical Turk participants were given a code to enter in order to receive compensation, and forum participants were given the opportunity to enter an email address into the raffle. This procedure was part of a larger study that included other survey items that are not reported here.

Data analysis and hypothesis

A one-way multivariate analysis of variance (MANOVA) was conducted in order to examine any differences between experience groups (low, medium, high) or the dependent variables described above. A MANOVA was chosen in order to protect against the inflation of Type 1 error associated with multiple ANOVA follow-up comparisons (Cramer & Bock, 1966). The analysis was aimed at answering the following hypothesis: There are one or more mean differences between experience levels (low, medium, high) or the dependent variables: percentage of abstract connections, total number of links, percentage of natural-language-based connections, number of central nodes, number of subnetworks, and percentage of procedurally based connections.

Results

Video Game Experience

As mentioned previously, participants indicated the extent of their experience with video games that are similar to League of Legends (i.e., MOBAs). A one-way between-subjects ANOVA revealed that there were no significant differences in prior MOBA experience between experience levels (low, medium, high), F(2, 155) = .73, p = .48. Relatedness ratings between terms were used as the matrix inputs for a Pathfinder analysis. Pathfinder networks were generated for each individual participant’s data. Additionally, mean aggregate Pathfinder networks were generated (see Figure 1).

Figure 1.

Aggregate Pathfinder networks for each level of experience. Descending: high experience, medium experience, low experience.

Each Pathfinder network has an associated coherence value. The coherence statistic measures the extent to which the network is internally consistent. A network with a low internal consistency suggests that the pairwise inputs (i.e., relatedness ratings) were not reliably obtained. Although there is no generally accepted threshold for internal consistency, the creators of the Pathfinder software used in this study suggest that a coherence of less than .15 indicates a network with low internal consistency (Pathfinder Networks, n.d.). Coherences for the low, medium, and high experience groups’ mean aggregate networks were .4, .38, and .46, respectively. Additionally, the mean coherences for each individual’s network in the low, medium, and high groups were .24, .33, and .37, respectively. All of these coherences exceed .15, and the trend for the individual networks follows what was to be expected, that networks are more coherent for individuals who have more experience. Additionally, individual relatedness ratings generated from Mechanical Turk participants correlated with those generated by forum participants at r = .86, supporting the notion that participants recruited from different sources did not produce substantially different relatedness ratings.

Mental Model Structural Characteristics

Means were examined at the descriptive level for each of the structural characteristics across the three experience groups (see Figure 2). A descriptive examination, prior to any significance testing, of means revealed noticeable differences between groups that were more prominent for some variables than others. Descriptive statistics also showed the mean proportion of natural-language-based connections was higher for the low experience group than for the medium and high experience groups. The mean proportion of abstract links and the number of subnetworks were markedly higher in the higher experience group than the low and medium experience groups. The mean proportion of procedural links revealed a step-like pattern whereby each experience group had a higher proportion of procedural connections, ascending from the low experience group to the high experience group. There was also a slightly lower number of total links for the high experience group, whereas the number of central nodes revealed no clear differences between groups.

Figure 2.

Bar charts displaying means for each structural characteristic measurement for each experience group. Error bars represent standard errors.

A one-way MANOVA was conducted in order to examine any differences between experience groups (low, medium, high) for the various structural characteristics. A statistically significant overall MANOVA effect was found, Wilks’ Lambda = .63, F(12, 300) = 6.53, p < .001, partial η² = .21, indicating a difference in structural characteristics between experience levels. Before follow-up ANOVAs were conducted, homogeneity of variance was tested using Levene’s F test at p < .001 for all dependent variables. Homogeneity of variance was considered satisfied for all variables. As can be seen in Table 3, ANOVAs were found to be statistically significant for percentage of natural language-based connections, F(2, 155) = 6.72, p = .002, η² = .08; percentage of abstract connections, F(2, 155) = 8.2, p < .001, η² = .1; percentage of procedural connections, F(2, 155) = 16.98, p < .001, η² = .18; and number of subnetworks, F(2, 155) = 10.34, p < .001, η² = .12. ANOVAs were not found to be statistically significant for total number of links, F(2, 155) = 1.02, p = .36, η² = .01, and number of central nodes, F(2, 155) = .18, p = .84, η² = .002.

Table 3:

Structural Characteristics—Follow-Up Univariate Results

IV	DV	SS	df	MS	F	p	Partial η²
Experience levels	% natural language	.032	2	.016	6.72	.002	.08
	Density	1499.88	2	749.94	1.02	.36	.01
	% abstract	.066	2	.033	8.2	<.001	.1
	% procedural	.346	2	.173	16.98	<.001	.18
	Central nodes	.95	2	.475	.18	.84	.002
	Subnetworks	53.97	2	26.98	10.34	<.001	.12

A series of post hoc analyses using Fisher’s LSD were conducted in order to examine differences between individual experience groups and the dependent variables for the significant univariate tests. Post hoc analysis found that Pathfinder networks of participants in the low experience group (M = .21, SD = .06) were found to have a greater percentage of natural-language-based connections than in the medium (M = .18, SD = .05, p = .003) and the high (M = .18, SD = .04) experience groups (p = .001). Results also indicated that participants in the high experience group (M = .82, SD = .06) had a greater percentage of abstract connections than in the low (M = .77, SD = .07, p < .001) and medium (M = .79, SD = .06, p = .008) experience groups. Participants in the high experience group (M = .43, SD = .1) were found to have a greater percentage of procedurally based connections than the low (M = .31, SD = .11, p < .001) and the medium (M = .38, SD = .1, p = .007) experience groups, while participants in the medium experience group were found to have a greater percentage than the low experience group (p = .002). Lastly, participants in the high experience group (M = 2.5, SD = 1.7) were found to have more subnetworks than in the low (M = 1.3, SD = 1.5, p = .001) and the medium (M = 1.2, SD = 1.5) experience groups (p < .001).

Discussion

Differences in Mental Model Structures

The primary aim of this study was to assess if and how mental model structures differ between multiple levels of experience. The study found that some structural characteristics systematically differed between three levels of experience with the video game used in this study. Each of these structural characteristics were drawn from the literature and had received some degree of empirical support. As this study used a cross-sectional design methodology, it is not equipped to directly address issues of mental model change over time but rather to identify group differences. However, differences between groups categorized on the basis of a temporal measure such as degrees of experience can be accounted for by developmental explanations and theories. Therefore, many of the explanations posited here account for the group differences found in this study by appealing to theories of mental model development. Additionally, as mentioned earlier, research has established Pathfinder as a valid method of representing mental model structures and thus conclusions about such structures can be drawn from Pathfinder networks (Cooke et al., 1986; Day et al., 2001; Goldsmith et al., 1991). This section interprets the findings of this study related to each of the mental model structural characteristics examined in this study.

Several researchers have suggested that expert mental model structures are denser (i.e., contain more connections) than those of novices (Bradley et al., 2006; Koponen & Pekhonen, 2010). Bradley et al. (2006) suggested that experts may have mental model structures containing more connections than novices due to their increased domain knowledge. However, differences in knowledge organization, rather than amount of content, would leave densities equal. This latter explanation sufficiently accounts for the results found in this study that densities did not differ between experience levels. Gillan et al. (1992) suggested that central nodes represent important concepts that link many aspects of the structure together. It may be true, similar to the case of structural density, that more experienced individuals have undergone central node change that is not additive but rather the simple replacement of central nodes. This would result in mental model structures with equal numbers of central nodes between experience levels and thus account for the results of this study that experience groups did not differ in the number of central nodes.

The current study found differences in the semantic relations in mental model structures, with the least experienced group having more links to natural language concepts than did the medium and high experience groups. Individuals approach a system or domain utilizing existing mental models of language (Gentner, 2002). As individuals discover mismatches between their mental models of terms and phrases and those in the novel system or domain, they are likely to adapt existing models or create new models for a novel use of language (Cool et al., 1996; Marchionini, 1989). Language, both spoken and written, is a fundamental means of communication in many domains and systems that individuals encounter frequently. Therefore, it is plausible that individuals encounter and react to any mismatches between mental and system models relatively quickly, resulting in a sharp decrease in natural-language-based connections, which reaches an asymptote early in experience. This could explain our results showing that the lower experienced group had more links to natural language concepts than the two other experience groups. If mismatches are discovered early in experience, then we would not expect to find many differences between the medium and high experience groups.

Kay and Black (1984) proposed that procedurally based mental model structures might represent the formation of memory around frequently performed tasks. In the context of spreading activation theory (Anderson, 1983), objects and concepts that are present in the environment or in working memory during frequently performed tasks have the connections between their associated nodes strengthened each time the task is performed (or even mentally simulated). Humans often quickly learn a solution and use that solution repeatedly to perform tasks and solve problems (Luchins, 1942; Schwartz et al., 2002). Given the propensity for humans to perform the same tasks frequently, the nodes associated with the repeated task would be likely to become associated early in the development of a mental model structure. Over time, especially in a competitive domain such as League of Legends, individuals encounter more advanced problems to solve and, consequently, learn novel procedures to accomplish these more complex goals. This would account for the results of the present study, which suggest that the proportion of procedural connections in mental models is higher at each level of experience. Anderson’s (1983) Adaptive Control of Thought–Rational (ACT-R) model also supports the notion that declarative knowledge can be proceduralized over time with repetition or practice. Individuals continuously discover procedures that accomplish desired goals and repeatedly perform those procedures while simultaneously organizing the associated mental model structure around procedural connections.

Previous research has indicated that an increase in domain experience is associated with an increase in the extent to which mental model structures are based around abstract concepts. Graham et al. (2006) posited that inexperienced individuals first encode aspects of a system based on the information that is most readily available (i.e., surface features). More experienced individuals are able to, over time, encode and organize the functional purposes of various concepts and objects. Of course, the extent to which the system allows the user to easily perceive its features and achieve goals within it, what Norman and Draper (1986) called the gulfs of evaluation and execution, could also factor into how easily functions are learned. Functional purposes are often, by their very nature, conceptual or abstract. Bradley et al. (2006) suggested that this experiential difference in structure abstraction is due to experts organizing knowledge differently. Although this latter explanation may not sufficiently explain the findings of this study, the interpretation of Graham et al. (2006) provides a starting point. The findings of the present study indicate that the low and medium experience groups were comparable in the number of abstract connections in their knowledge networks, and both had fewer abstract connections than the high experience group. This difference could be due to a developmental pattern. Functional concepts, as pointed out by Graham et al. (2006), require knowledge gained after repeated exposure and interaction. Surface features of objects, such as appearance, are readily available and can be quickly encoded. It may be the case that individuals first encode surface features of objects, encode isolated functional operations, and then finally encode associations between different objects, concepts, and functions. This is consistent with the two-stage theory of mental model construction posited by Mayer et al. (2002). The encoding of object functions and subsequent associations between different functions could take considerable time and effort relative to encoding readily apparent surface features. If this is the case, then low and medium levels of experience would be expected to be similar in this aspect of their mental models.

Gillan et al. (1992) suggested that mental model subnetworks aid in the recall of a set of memory chunks for a given context. Initially it might seem surprising that differences in procedurally related connections were found between low and medium experience groups, whereas differences in subnetworks were found only between medium and high experience groups. Procedurally organized sets of concepts are likely reinforced through repetition, and small sets of procedurally related concepts would appear to constitute a subnetwork. The primary difference between a string of procedurally related concepts, such as a script, and a subnetwork is that such a string is linear, but a subnetwork contains a large degree of nonlinear connectivity (Gillan et al., 1992). This added structural complexity requires interconnections of concepts beyond those found in a linear script (e.g., abstract connections) in order to establish a truly interconnected subnetwork of concepts. Based on this account, the increase in the number of subnetworks would be most pronounced for those with higher levels of experience. Additionally, it could be that some subnetworks begin to form because the mental model is constructed around a new understanding of abstract concepts. For example, several concepts might become interrelated (i.e., a subnetwork) because they are all part of a new abstract category.

Three-Stage Theory of Mental Model Construction

The results discussed in the previous section indicate differences in how mental model structures are organized across individuals of different experience levels. As mentioned previously, the design of this study does not lend itself to direct claims about longitudinal change. However, a developmental theory can plausibly explain the cross-sectional group differences found in this study. Based on the differences discussed in the previous section, a three-stage model of mental model structure development is proposed (see Figure 3). Stage 1, in which individuals have low experience with a system or domain, is characterized by a low proportion of procedural connections and abstract connections and few subnetworks but a high number of natural language connections. Stage 2, in which individuals have a medium amount of experience, is characterized by a high proportion of procedural connections but a low proportion of natural language connections and abstract connections and few subnetworks. Lastly, Stage 3, in which individuals have a high amount of experience, is characterized by an even higher proportion of procedural connections, a high proportion of abstract connections, and a high number of subnetworks but a low proportion of natural language connections.

Figure 3.

Three-stage model of mental model structure development. The three blocks represent three stages of experience with a system or domain. Block text indicates structural characteristics that are found in low degrees, high degrees, or very high degrees.

This model is not in competition with the models of mental model development discussed earlier, such as the two-stage model of Mayer et al. (2002), Zhang’s (2009) model, and Katzeff’s (1990) model, but rather complements them by positing specific structural changes that occur during mental model construction. Additionally, this model is consistent with skill development models of automatic and controlled processing as proposed by Anderson (1983), Kahneman (2011), and Shiffrin and Schneider (1977). In our three-stage model, there are a higher number of procedural connections for those individuals who have more experience. This is to be expected in a dual-processing approach to skill development as information processing shifts from controlled to automatic and rule-based procedural knowledge increases with experience (Anderson, 1983; Kahneman, 2011; Shiffrin & Schneider, 1977).

Applied Implications

Training

There is ample evidence that mental model structures can be used to both design and assess the effectiveness of skill training and education programs (Day et al., 2001; Goldsmith & Kraiger, 1997; Stout, Salas, & Kraiger, 1997; Trumpower & Sarwar, 2010; Trumpower, Sharara, & Goldsmith, 2010; Upchurch, 2013). However, Cooke and Schvaneveldt (1988) proposed that, in order to effectively use mental model structures in the design and execution of training programs, structural signifiers of expertise need to first be identified. These signifiers are components of mental model structures that denote expertise and thus can be used to inform the design of training regimens. Research on the use of mental model structures in training often uses overall measures of Pathfinder network similarity as a signifier (i.e., the similarity between expert and novice networks). The use of mental model structures at a finer grain could provide additional utility for formative uses such as training design. The structural characteristics of mental models discussed in the present research represent possible signifiers of expertise that transcend the holistic nature of the expert-novice network similarity approach. The three-stage approach outlined earlier provides a framework that suggests temporal priorities for training with regards to mental model structural characteristics. This is to say, in using the three-stage approach, training program design can be informed both by specific structural signifiers of expertise and also by when these signifiers appear in a structural development timeline. For example, a training program might expose trainees to domain language during early training sessions while addressing abstract concepts in later training sessions, consistent with both the structural characteristics and temporal order identified in the three-stage model.

League of Legends is characterized by specific aspects that should be considered when applying the results of the present study. Although these results may generalize to many types of tasks, the recommendations provided here are likely to be most effective for tasks that share key similarities with League of Legends. Training programs most likely to benefit from these recommendations are those for which tasks are team-based, involve interpersonal communication, encourage competition, and require problem solving for both open-ended and narrowly defined problems, among other potential factors.

Game design

Just as mental model structural characteristics can be applied to training programs, they can also be applied to video game design. Graham et al. (2006) suggested that video game difficulty progression design can be informed by mental model structure development. They suggested that instead of increasing difficulty in a linear fashion, as many games do, difficulty should be increased by requiring the player to adapt their current mental model or develop a new one. This design suggestion is even more plausible given the results of the current research. The structural characteristics described in this study offer a number of avenues through which game designers can increase game difficulty by manipulating mental model accommodation and assimilation. As Graham et al. (2006) pointed out, this route is preferable to traditional approaches of increasing game difficulty because it focuses on manipulating mental models and not merely exhausting physical and cognitive abilities, which are limited. Additionally, this approach is open to a variety of qualitative shifts in difficulty, avoiding the repetition that often accompanies linear difficulty increases, which demand that players execute the same tasks and complete the same goals while merely increasing the cognitive and physical workload. Additionally, game designers should consider how game elements might affect the construction of mental models, choosing when to provide or not provide support for the construction of mental model structures.

Limitations and Future Research

The conclusions drawn from Pathfinder analysis, however valid, are based on a single indirect measure of mental model structure. Future research using alternate methods of eliciting mental model structures would prove valuable in testing the three-stage model presented here. Although a general model of mental model structural development is proposed here, a few areas of concern suggest limitations to the research. The present study was a cross-sectional design and thus cannot make direct claims about intraindividual change. The three developmental stages we propose in our model, for example, were not directly observed but rather were proposed to explain our cross-sectional findings. Therefore, longitudinal research should be performed to further establish the proposed developmental nature of individual differences in mental model structural characteristics. Additionally, the small sample size of the preliminary study greatly constrains the extent to which it can be analyzed for similarities and differences in concept generation by experience level, gender, and age. Future research that collects larger sample sizes for concept generation could reveal additional details about the content of mental models, beyond what was presented in this study. The sample for a concept generation task could also be selected so that the distribution of game experience for concept generation participants is similar to that of the main study.

The present study also examined mental models in a single context and with a single system. Future research in differing domains would provide evidence serving to either strengthen the generalizability of these findings or reveal their contextual bounds. This study also limited its scope to three levels of experience, which were broadly inclusive of different degrees of experience. Future research at finer grains of experience levels could produce a more detailed understanding of where differences in structural characteristics lie. Lastly, devising and testing novel training interventions that support the development of sophisticated mental model structures based on the findings of this study would provide evidence for the utility of the structural approach.

Footnotes

ORCID iD

Caleb S. Furlough

Caleb S. Furlough works in industry as a human factors consultant. He received his PhD in human factors and applied cognition from North Carolina State University in 2017.

Douglas J. Gillan is a professor of psychology at North Carolina State University. He received his PhD in biopsychology from the University of Texas at Austin in 1978.

References

Anderson

J. R.

(1983). The architecture of cognition. Cambridge, MA: Harvard University Press.

Bader

Beyerer

(2011). Influence of user’s mental model on natural gaze behavior during human-computer interaction. In 2nd Workshop on Eye Gaze in Intelligent Human Machine Interaction (pp. 25–32). Palo Alto, CA: ACM.

Beggiato

Krems

J. F.

(2013). The evolution of mental model, trust and acceptance of adaptive cruise control in relation to initial information. Transportation Research Part F: Traffic Psychology and Behaviour, 18, 47–57.

Bradley

J. H.

Paul

Seeman

(2006). Analyzing the structure of expert knowledge. Information & Management, 43, 77–91.

Clariana

R. B.

(2010). Multi-decision approaches for eliciting knowledge structure. In Ifenthaler

Pirnay-Dummer

Seel

(Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 41–59). Boston, MA: Springer.

Collins

A. M.

Loftus

E. F.

(1975). A spreading-activation theory of semantic processing. Psychological Review, 82, 407–428.

Cooke

N. M.

Durso

F. T.

Schvaneveldt

R. W.

(1986). Recall and measures of memory organization. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12, 538–549.

Cooke

N. J.

Schvaneveldt

R. W.

(1988). Effects of computer programming experience on network representations of abstract programming concepts. International Journal of Man-Machine Studies, 29, 407–427.

Cool

Park

Belkin

N. J.

Koenemann

K. B.

(1996). Information seeking behavior in new searching environment. CoLIS, 2, 403–416.

10.

Craik

(1943). The nature of exploration. Cambridge, UK: Cambridge University Press

11.

Cramer

E. M.

Bock

R. D.

(1966). Multivariate analysis. Review of Educational Research, 36, 604–617.

12.

Day

E. A.

Arthur

Jr. Gettman

(2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86, 1022–1033.

13.

Deutsch

J. E.

Brettler

Smith

Welsh

John

Guarrera-Bowlby

Kafri

(2011). Nintendo wii sports and wii fit game analysis, validation, and application to stroke rehabilitation. Topics in Stroke Rehabilitation, 18, 701–719.

14.

Dinet

Kitajima

(2011). Draw me the Web: Impact of mental model of the web on information search performance of young users. In Riveill

(Ed.), 23rd French Speaking Conference on Human-Computer Interaction (pp. 35–41). New York, NY: ACM Press.

15.

Doyle

J. K.

Ford

D. N.

(1998). Mental models concepts for systems dynamics research. System Dynamics Review, 14, 3–29.

16.

Electronic Software Association. (2015, April). Essential facts about the computer and video game industry. Retrieved May 5, 2016 from http://www.theesa.com/wp-content/uploads/2015/04/ESA-Essential-Facts-2015.pdf

17.

Fein

R. M.

Olson

G. M.

Olson

J. S.

(1993). A mental model can help with learning to operate a complex device. In Ashlund

Mullet

Henderson

Hollnagel

White

(Eds.), INTERACT’93 and CHI’93 conference companion on human factors in computing systems (pp. 157–158). Amsterdam: ACM Press.

18.

Feng

Spence

Pratt

(2007). Playing an action video game reduces gender differences in spatial cognition. Psychological Science, 18, 850–855.

19.

Gee

J. P.

(2003). What video games have to teach us about learning and literacy. New York, NY: Palgrave/St.

20.

Gentner

(2002). Mental models, psychology of. In Smelser

N. J.

Bates

P. B.

(Eds.), International encyclopedia of the social and behavioral sciences, (pp. 9683–9687). Amsterdam: Elsevier Science.

21.

Gentner

Stevens

(1983). Mental models. Hillsdale, NJ: Lawrence Erlbaum Associates.

22.

Gillan

D. J.

Breedin

S. D.

Cooke

N. J.

(1992). Network and multidimensional representations of the declarative knowledge of human-computer interface design experts. International Journal of Man-Machine Studies, 36, 587–615.

23.

Goldsmith

T. E.

Johnson

P. J.

Acton

W. H.

(1991). Assessing structural knowledge. Journal of Educational Psychology, 83, 88–96.

24.

Goldsmith

T. E.

Kraiger

(1997). Applications of structural knowledge assessment to training evaluation. In Ford

J. K.

(Ed.), Improving training effectiveness in work organizations (pp. 73–96). Mahwah, NJ: Erlbaum.

25.

Gopher

Well

Bareket

(1994). Transfer of skill from a computer game trainer to flight. Human Factors: The Journal of the Human Factors and Ergonomics Society, 36, 387–405.

26.

Graham

Zheng

Gonzalez

(2006). A cognitive approach to game usability and design: Mental model development in novice real-time strategy gamers. CyberPsychology & Behavior, 9, 361–366.

27.

Green

C. S.

Bavelier

(2007). Action-video-game experience alters the spatial resolution of vision. Psychological Science, 18, 88–94.

28.

Hasher

Zacks

R. T.

(1979). Automatic and effortful processes in memory. Journal of Experimental Psychology: General, 108(3), 356–388.

29.

Johnson-Laird

(1983). Mental models. Cambridge, MA: Harvard University Press.

30.

Jones

Ross

Lynam

Perez

Leitch

(2011). Mental models: An interdisciplinary synthesis of theory and methods. Ecology and Society, 16, 46.

31.

Kahneman

(2011). Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.

32.

Katzeff

(1990). System demands on mental models for a fulltext database. International Journal of Man-Machine Studies, 32, 483–509.

33.

Kay

D. S.

Black

J. B.

(1984). Changes in knowledge representations of computer systems with experience. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 28, 963–967.

34.

Kieras

D. E.

Bovair

(1984). The role of a mental model in learning to operate a device. Cognitive Science, 8, 355–273.

35.

Koponen

I. T.

Pehkonen

(2010). Coherent knowledge structures of physics represented as concept networks in teacher education. Science & Education, 19, 259–282.

36.

Kraiger

Salas

Cannon-Bowers

J. A.

(1995). Measuring knowledge organization as a method for assessing learning during training. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37, 804–816.

37.

Landriscina

(2013). Simulation and cognition. Simulation and learning. New York, NY: Springer.

38.

Larose

Gagnon

Ferland

Pépin

(1989). Psychology of computers: XIV. Cognitive rehabilitation through computer games. Perceptual and Motor Skills, 69, 851–858.

39.

Polat

Makous

Bavelier

(2009). Enhancing the contrast sensitivity function through action video game training. Nature Neuroscience, 12, 549–551.

40.

Lokuge

Gilbert

S. A.

Richards

(1996). Structuring information with mental models: A tour of Boston. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 413–419.

41.

Luchins

A. S.

(1942). Mechanization in problem solving: The effect of Einstellung. Psychological Monographs, 54, i–95.

42.

Marchionini

(1989). Making the transition from print to electronic encyclopedias: Adaptation of mental models. International Journal of Man-Machine Studies, 30, 591–618.

43.

Mayer

R. E.

Mathias

Wetzell

(2002). Fostering understanding of multimedia messages though pre-training: Evidence for a two-stage theory of mental model construction. Journal of Experimental Psychology: Applied, 8, 147–154.

44.

Mogford

R. H.

(1997). Mental models and situation awareness in air traffic control. The International Journal of Aviation Psychology, 7, 331–341.

45.

Norman

D. A.

(1983). Some observations on mental models. In Gentner

Stevens

(Eds.), Mental models (pp. 7–14). Hillsdale, NJ: Erlbaum.

46.

Norman

D. A.

Draper

S. W.

(1986). User centered system design. Hillsdale, NJ: Erlbaum.

47.

Our Games. (n.d.). Retrieved June 1, 2016, from http://www.riotgames.com/our-games

48.

Papastergiou

(2009). Digital game-based learning in high school computer science education: Impact on educational effectiveness and student motivation. Computers & Education, 52, 1–12.

49.

Pathfinder Networks. (n.d.). Frequently asked questions. Retrieved December 17, 2016, from http://interlinkinc.net/FAQ.html

50.

Quillian

M. R.

(1966). Semantic memory. Cambridge, MA: Bolt Beranek and Newman Inc.

51.

Rand

Kizony

Weiss

P. T. L.

(2008). The Sony PlayStation II EyeToy: Low-cost virtual reality for use in rehabilitation. Journal of Neurologic Physical Therapy, 32, 155–163.

52.

Rouse

W. B.

Morris

N. M.

(1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin, 100, 349–363.

53.

Ryan

R. M.

Rigby

C. S.

Przybylski

(2006). The motivational pull of video games: A self-determination theory approach. Motivation and Emotion, 30(4), 344–360.

54.

Savage-Knepshield

P. A.

(2001). Mental models: Issues in construction, congruency, and cognition (doctoral dissertation). The State University of New Jersey, Rutgers, New Brunswick.

55.

Schlickum

M. K.

Hedman

Enochsson

Kjellin

Felländer-Tsai

(2009). Systematic video game training in surgical novices improves performance in virtual reality endoscopic surgical simulators: A prospective randomized study. World Journal of Surgery, 33, 2360–2367.

56.

Schvaneveldt

R. W.

Durso

F. T.

Dearholt

D. W.

(1989). Network structures in proximity data. The Psychology of Learning and Motivation: Advances in Research and Theory, 24, 249–284.

57.

Schwartz

Ward

Monterosso

Lyubomirsky

White

Lehman

D. R.

(2002). Maximizing versus satisficing: Happiness is a matter of choice. Journal of Personality and Social Psychology, 83(5), 1178–1197.

58.

Sebrechts

M. M.

Black

J. B.

Galambos

J. A.

Wagner

R. K.

Deck

J. A.

Wikler

E. A.

(1983). The effects of diagrams on learning to use a system (Report No. 2). Learning and Using Systems. New Haven, CT: Yale University Press.

59.

Shiffrin

R. M.

Schneider

(1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychological Review, 84, 127–190.

60.

Slone

D. J.

(2002). The influence of mental models and goals on search patterns during web interaction. Journal of the American Society for Information Science and Technology, 53, 1152–1169.

61.

Smith

S. W.

Koppel

(2014). Healthcare information technology’s relativity problems: A typology of how patients’ physical reality, clinicians’ mental models, and healthcare information technology differ. Journal of the American Medical Informatics Association, 21, 117–131.

62.

Stout

R. J.

Salas

Kraiger

(1997). The role of trainee knowledge structures in aviation team environments. The International Journal of Aviation Psychology, 7, 235–250.

63.

Trumpower

D. L.

Sarwar

G. S.

(2010). Formative structural assessment: Using concept maps as assessment for learning. Fourth International Conference on Concept Mapping, 22, 132–136.

64.

Trumpower

D. L.

Sharara

Goldsmith

T. E.

(2010). Specificity of structural assessment of knowledge. Journal of Technology, Learning, and Assessment, 8(5), 1–31.

65.

Tulving

(1962). Subjective organization in free recall of “unrelated” words. Psychological Review, 69(4), 344–354.

66.

Upchurch

(2013). Adaptive performance: The role of knowledge structure development (doctoral dissertation). Rice University, Houston, Texas.

67.

Wilson

J. R.

Rutherford

(1989). Mental models: Theory and application in human factors. Human Factors: The Journal of the Human Factors and Ergonomics Society, 31(6), 617–634.

68.

Wouters

Van Nimwegen

Van Oostendorp

Van Der Spek

E. D.

(2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265.

69.

Zhang

(2009). The construction of mental models of information-rich web spaces: The development process and the impact of task complexity (doctoral dissertation). University of North Carolina, Chapel Hill.

70.

Zhang

(2013). The development of users’ mental models of MedlinePlus in information searching. Library & Information Science Research, 35(2), 159–170.

71.

Zielinska

O. A.

Welk

A. K.

Mayhorn

C. B.

Murphy-Hill

(2015). Exploring expert and novice mental models of phishing. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 1132–1136.