Abstract
Background. Extensive research exists in the application of gaming simulation to education, experimentation and policy. Systems engineers have begun to utilize gaming simulation
Aim. The purpose of this research is to test the hypothesis that the use of gaming simulation will
Method. A
Results. As the research has matured, informal free-form
Conclusion. The conclusion of this article points to
Keywords
Gaming has long been used as a problem-solving tool, often providing an alternative method of approaching a challenge. During the past decade, gaming simulations have become common across many fields. As a discipline, systems engineering has been relatively slow in adopting gaming simulation. This research is aimed at determining the effectiveness of gaming simulation to a specific systems engineering product, the Concept of Operations (CONOPS). The current CONOPS development process illustrates a key gap identified by systems engineering professionals; many believe that the weakest link in system development is often between what the user desires and what the development team believes is needed. A shared vision or understanding of the environment in which the new system will be deployed seldom exists.
This article will briefly describe CONOPS, the general application of gaming simulation to CONOPS, and a specific method to allow system users to create and execute CONOPS in a 3D immersive environment. This article presents an update to early research in progress (Cloutier et al. 2009; Cloutier et al. 2010; Cloutier et al. 2012). It is an extension of a paper presented at the Third International Engineering Systems Symposium of the Council of Engineering Systems Universities (Korfiatis, Cloutier, & Zigh, 2012). The research is ongoing and this article presents progress made since the Symposium.
Systems Engineering
Driven by advances in science and technology, systems have grown significantly more complex over the past few decades. This complexity is especially evident in the omnipresence of software and the multidisciplinary nature of today’s systems. Many systems being developed require a variety of engineers working together to meet the needs of users, and systems engineering provides an interdisciplinary approach to produce such complex systems. The goal of systems engineering is to carry out development with a holistic view, utilizing a variety of processes and tools, examining both hard and soft sciences, and integrating among domain specific engineers. This article will focus on CONOPS, a specific product developed during an early part of the systems engineering development lifecycle.
Concept of Operations
CONOPS is typically written by end users and is intended to inform those defining the requirements that will drive system design. This document describes the current operational environment, how and why the operational environment should change, user needs, and representative scenarios for system operations. In short, CONOPS describes how a system will work from the user’s perspective (INCOSE, 2011).
The completed CONOPS is generally a lengthy document containing text and images that are intended to describe future system operations in the form of scenarios. Based on surveys and state of practice assessments, a number of shortcomings with current CONOPS development processes have been identified (Cloutier et al., 2009). Writing CONOPS can be a subjective, labor-intensive, and time-consuming task involving multiple iterations of natural language descriptions and prototypical representations. The resulting artifact is typically a static representation of the user identified need, which may not satisfy the original intent, is potentially skewed towards one perspective, and is not referenced and updated during system development (Roberts & Edson, 2008).
Challenge areas for improving CONOPS include reduction in development time, enhancement of user-developer collaboration, and extension of static conceptual representations to dynamic artifacts. It is believed that improving CONOPS practices will significantly reduce development time by enabling faster requirements elicitation and reducing rework due to misunderstanding.
Introducing Model-Based Systems Engineering
For the past decade, the systems engineering community has been undergoing a revolution. The Model-Based Systems Engineering (MBSE) initiative was launched by INCOSE to promote and institutionalize the transformation of a typically document-driven system development lifecycle to one driven by the creation and maintenance of models. Considerable work has been carried out to strengthen MBSE during requirements elicitation, design, and testing, yet little research has been dedicated to providing a structured approach to operational scenario modeling and model-based CONOPS development.
A goal of this research is to introduce MBSE tools and methodology to CONOPS development. As discussed below, users will be able to create CONOPS graphically, using a gaming simulation to construct conceptual models and auto-generate visualizations. These conceptual models, which capture the needs of future users, will be structured in such a way as to make them accessible to other standard MBSE tools.
Gaming Simulation and Immersive Environments
Today’s 3D games have advantages that can be leveraged to improve the development of CONOPS. Working in immersive environments often leads people to gain a better understanding of concepts and allows for multiple perspectives to be observed, meaning that a user could adapt their viewpoint and vary the information displayed to them. Immersive multiplayer environments and gaming simulations are also recognized for their improvement of distributed collaboration (Carmichael, 2011). Research in learning, problem solving, and design has shown that 3D environments can promote collaboration and reduce negotiation time (Lin, Low, Ng, Bu, & Liu, 2003). Finally, Macedonia (2002) points out that the next generation of workforce has spent years immersed in games and has unique characteristics predisposing them to working in a digital world.
A Gaming Approach Early Systems Engineering Development
Previous research has proposed a number of gaming methods and tools to improve outcomes of early systems engineering related activities. In the fields of civil construction and architecture, virtual environments and gaming have been used successfully across many projects. It is easy to see how a virtual rendering can be used by concerned parties to provide a mental model of how a project will look when completed. A number of companies offer tools for architects to transform 2D schematics into 3D renderings. Adding gaming elements to these 3D representations through the popular virtual environment SECOND LIFE has allowed those concerned with visual impact assessments to “walk around” and explore proposed designs (Panagopoulos, Jankovska, & Straupe, 2012).
In the automotive industry, Mansurov and Vasura (2001) have developed a visual interface for the elicitation and validation of customer needs. This software presents a virtual environment of a vehicle’s cockpit, allowing the user to interact with key elements of the vehicle. Although the user is playing in the environment, the software translates their activities to standard automotive engineering model-based artifacts. This type of tool demonstrates how virtual environments can enable users to experiment with system alternatives and provide feedback to engineers on proposed designs.
For many years, defense companies have been using games and virtual environments for testing systems prior to development, which have proven to be valid substitutes for tests that could be expensive and potentially harmful to human life (Linden Labs, 2009). Gaming environments such as AMERICAS ARMY have also been used as experimentation platforms for new systems, allowing personnel to evaluate new technologies and provide feedback that improves design before systems are developed (Nieborg, 2004).
Rhoads and Gilman (2004) have adapted existing war gaming and constructive simulation software to enhance CONOPS for employment of Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance Nodes. This game was presented to subject matter experts and a controlled experiment was conducted, yielding vulnerabilities of operating procedure and shortcomings in military information processing capabilities.
X-PLANE is flight simulator that allows the gamer to select a real life aircraft and control the aircraft, environment, and a number of other factors. Although X-PLANE is marketed as a computer game, what differentiates it from other flight simulators is the complexity of its underlying aircraft, flight and environment models. Invisible to the casual gamer, X-PLANE allows users to create their own aircraft and fly it, with flight performance dependent on the aircraft’s geometry and other characteristics. The level of realism provided by X-PLANE, together with how easily it integrates with current aircraft design tools, has led aerospace engineers to use it during early systems engineering (Ribeiro, 2010).
Although these researchers have all made contributions to the CONOPS development process, few have taken full advantage of the advances made in computing power, graphics technology and gaming simulation readily available in today’s games. Based on an extensive literature review and assessment of current CONOPS processes and technology (Cloutier et al., 2009; Mostashari, McComb, Kennedy, Cloutier, & Korfiatis, 2011), we have identified a need for quickly and graphically articulating CONOPS to realize a shared mental model and understanding across the set of diverse stakeholders.
The primary research question of this work centers on a determination of effectiveness of a gaming simulation for use during CONOPS development. The hypothesis to be tested states that the use of a gaming simulation will improve the CONOPS development process. The plan to test this hypothesis involves having system stakeholders gather in an immersive environment and visually model scenarios that a system will encounter during operation. A control group will develop a traditional textual CONOPS and questionnaires will be used to evaluate the quality of CONOPS and ease with which it was created.
As of the publication of this article, research is underway and qualitative and quantitative results from experiments are forthcoming in future publications. This article will introduce the novel gaming simulation research instrument that will be used to address the hypothesis, the Integrated Concept Engineering Framework (ICEF). Discussion will focus on the anticipated process by which users will collaboratively play with ICEF to form CONOPS, reflections on ICEF as a gaming simulation, and preliminary qualitative results of user assessment of ICEF as a gaming simulation systems engineering tool.
The Integrated Concept Engineering Framework
ICEF was developed as a proof-of-concept prototype to investigate the application of gaming simulations to CONOPS development. ICEF is meant to be played by a distributed group of system stakeholders in an immersive environment. Upon logging into ICEF, players are directed to the main screen, as seen in Figure 1.
ICEF main menu.
At the main menu, players are asked to select a domain within which they want to develop and execute their scenario. Each domain is linked to domain-specific databases containing a variety of relevant scenario primitives. For example, military primitives include the objects (soldier, enemy, weapon) and actions (question, locate, shoot) that would be needed to model military operations. Additionally, in the event that an object or action is unavailable, players have the ability to insert a generic placeholder. The databases currently contain hard-coded primitives that need to be developed by game facilitators, but future development of ICEF will allow the players to build their own primitives as they play. Once the domain has been selected, the players are led to the authoring environment, seen in Figure 2.
ICEF authoring environment.
In the authoring space, players work together to drag-and-drop objects into a scene and define how they interact with each other using actions. Players further define objects and actions with attributes such as speed and acceleration. Actions are organized chronologically to develop a script of how the scene plays out. When scene construction is completed, additional scenes can be added by selecting a new terrain and defining the transition between scenes. Once the users construct their scenario, the model is executed, initiating a visual animation. This animation allows players to observe their creation in motion, leading to a greater understanding and validation of what is happening in the scenario. At any time during execution, the players can stop the animation and alter the scenes to better capture their expectations. As evidenced by Stewart (2007), this type of immersive, virtual playback can provide a new perspective to users and allow them to identify misunderstanding or poor design early in the development process. ICEF has demonstrated the feasibility of developing user-directed, auto-generated animations from pre-coded primitives, and user feedback has shown positive effects during CONOPS development (Korfiatis, 2013).
If data-rich attributes are associated with primitives, ICEF is designed to utilize external software such as MATLAB to carry out mathematical calculations and simulate the scenario. Attributes such as ammunition, weapon type and armor can be associated with a soldier. When the solder is directed to shoot at an enemy, ICEF sends these variables to a MATLAB lethality model and calculations are carried out yielding the results of the firefight, which are fed back into ICEF and drive the animation. As seen in Figure 3, both the mathematical and visual results of the firefights are displayed to the players.
ICEF simulation results.
Although a large part of scenario development is collaborative, a competitiveness exists when stakeholders are trying to ensure that their needs are prominent among conflicting concerns. In the traditional method of CONOPS development, prioritization of concerns is managed asynchronously through editing drafts of the textual CONOPS. By the nature of real-time collaboration, players are forced to have discussions when disagreements arise. McComb (2007) has shown that as team members interact, shared mental models form. The ICEF environment acts as a collaborative gaming simulation to help players reach such shared mental models (Mostashari et al., 2011).
Detailed discussion on ICEF’s use and its development can be found in (Cloutier et al., 2012; Korfiatis et al., 2012). The remainder of this article will describe feedback related to ICEF as a gaming simulation.
The Integrated Concept Engineering Framework as a Gaming Simulation
Duke and Geurts (2004) provide a common definition for a gaming simulation as “an operating model of a real-life system in which actors in roles partially recreate the behavior of the system”. In ICEF, users represent their interests in a future system while building and testing a model of a real-life scenario in a gaming environment. ICEF contains a number of gaming elements identified by previous researchers. Table 1 represents some characteristics of gaming simulations and how ICEF can be compared against these attributes.
Game Characteristics of ICEF.
In addition to its gaming characteristics, ICEF can be described using common gaming simulation approaches. In his investigation of gaming simulation, Kriz (2003) lays forth the design cycle seen on the left-hand side of Figure 4. On the right-hand side, the appropriate stages of ICEF are traced to Kriz’s approach.
ICEF mapped against Kriz’s (2003) gaming simulation approach.
Consistent with Figure 4, ICEF is used to construct typical scenarios that may be encountered by a system as a facsimile of the real world. During scenario generation and playback, ICEF gamers are playing the game; each time a scenario is run, the users are able to change aspects of the scenario and re-execute. During each of these iterations, ICEF users are trying to reach their objective, to design a scenario that unambiguously states their desired needs of the future system.
ICEF Feedback and Evaluation
To date, a prototype of ICEF has been exposed to both systems engineering and gaming communities through conference presentations. It has also undergone two early testing workshops aimed at generating feedback to help guide ICEF development and direct the research. The remainder of this article will focus on this feedback framed against areas of interest in the gaming simulation domain.
During the first workshop, the participants were provided an overview of the research and were asked to free-form play with ICEF. A discussion followed that provide an opportunity for participants to ask questions and provide feedback. The goal of the first workshop was to measure reception of the research among the user community and derive desired capabilities to fuel continuing development of ICEF. The second workshop was carried out with more direction. Participants were broken into groups and handed a specific scenario to model in ICEF. Each group was observed by a researcher and a debriefing session provided the opportunity for feedback. A logging function and persistent database produced additional data indicating how the workshop participants interacted with the software. The goals of the second workshop included gathering additional feedback to plan for future development for ICEF, as well as gathering information in preparation for two planned controlled experiments. Select feedback from testers, and reflections by researchers, is presented below.
Fidelity and Realism
The most common feedback received has been regarding the fidelity and realism of ICEF. These are two related points that are frequent topics of discussion in the gaming simulation community. The relationship between fidelity and effectiveness is a difficult concept to understand fully. Feinstein and Cannon (2002) have conducted an extensive literature review of fidelity in training and education games, concluding that higher fidelity does not necessarily translate to more effective learning and may indeed hinder training. Zyda (2005) identified that a game thought to have insufficient fidelity to make it useful as a training tool ended up being effective to soldiers struggling with traditional training. To a certain extent, the proper level of fidelity of a game depends on perspective.
Demands on sub-classes of fidelity (functional, physical, psychological, etc.) may be set at different levels within the same game. During the first ICEF workshop, participants agreed that the quality of game graphics was sufficient for the intended purposes. During the second workshop, participants echoed that sentiment, however, they were concerned that the functional fidelity of ICEF may be too low, leading to misrepresentation of operational scenarios. During the development of CONOPS, detailed data may not exist, therefore lower level model fidelity may suffice.
Peters, Vissers, and Heijne (1998) form a link between game validity and fidelity that is dependent on gaming objectives and context. In examining gaming simulation as a tool to explore possible options to solve a problem, we note that game environments should be less restricted in correspondence to the real world, as players should be free to explore a range of alternative solutions. This argument leads to a more in-depth discussion on validity.
Game Validity
This research was presented at a conference for Game Design of Complex Systems, providing an opportunity to engage a community of expert game developers who apply gaming to engineering problems. The conference track consisted of short presentations followed by an open discussion, during which a common theme centered on game evaluation and validity. Feinstein and Cannon (2002) provide a thorough literature review on types of validity, centered on a game’s use as a method of learning or assessment. Peters et al. (1998) focus on simulations and games as a model of a reference system, with validity measuring the correspondence between the results of the use of models. In this regard, further experimentation must be carried out using ICEF across a large number of users, all faced with the same challenge. Convergence to or divergence from a common methodology across a variety of ICEF users must be observed to allow researchers to gather metrics relating to the validity. As Meijer (2009) argues, freezing the inputs (rules, roles, objectives, constraints, load and situation) across a number of ICEF sessions would allow researchers to examine the relationship between repeatability of an ICEF session and the validity of its outcomes.
In their examination of validity in gaming simulations developed for research, Vissers, Heyne, Peters, and Geurts (2001) focus on validity as the delta between conditions in the game and the real world. Concerns include the applicability of results within a gaming simulation to real world situations, as well as the level of accuracy in the behaviors and activities taking place during a gaming simulation. We suggest that requirements on this type of validity are context specific and the accuracy of a scenario model is relative to the needs of its users.
ICEF has been developed as a research tool, aimed at addressing the research questions described above. However, if the hypotheses of this research prove to be correct, and if ICEF were to reach the market as a commercial gaming simulation, internal and external validity would need to be examined closely on a case by case basis. It is important to keep in mind that the purpose and users of a gaming simulation will have an impact on where the valid/invalid line lies. The context specific dependency of ICEF has driven its development as a framework, allowing users to specify the fidelity of underlying models as a lever for controlling overall realism and validity. Further study of validity will be possible once a large number of ICEF sessions have taken place. Detailed discussion regarding further development and experimentation with ICEF can be seen in (Korfiatis, 2013) and (Korfiatis & Cloutier, 2013).
Usability
To examine usability of ICEF, user feedback was matched against Nielsen’s (1995) heuristics. A heuristic of significant focus was consistent visibility of system status. To ensure that ICEF provided appropriate and timely feedback to its users, a status bar was integrated into the interface, providing messages confirming user actions and ICEF’s reactions. As seen in Figure 5, when users add a new scene, ICEF provides a “New Scene created” message. The status bar in Figure 6 and Figure 7 show additional status messaging when errors occur, which also speaks to the error recognition, diagnosis and recovery heuristic. Yellow messages indicate that an error has occurred due to a user error, which can be remedied with human interaction. Red messages indicate an error on the part of ICEF.
ICEF multiple scenes.
ICEF user error handling.
ICEF error handling.
As stated by Nielsen (1995), although error recognition, diagnosis and recovery are important elements of usability, error prevention through mindful design of user interfaces can be even more effective. This is particularly relevant to ICEF as its users may not be technologically advanced. Where certain user behavior would produce errors, the ICEF development team restricted the user or forced mitigating actions. Simple design choices such as limiting the movement of characters and requiring unique naming of objects prevented errors that could have affected usability and in turn reduced effectiveness of ICEF as a gaming simulation.
Debriefing
Debriefing is crucial to the effectiveness of gaming simulation as a learning tool. Although ICEF is not an educational gaming simulation, it does provide a learning opportunity, primarily the transfer of knowledge between stakeholders and development engineers of anticipated system operational scenarios. Citing personal correspondence with Joe Wolfe, Crookall (2010) describes debriefing as an opportunity for players to share learnings from a game. The success of ICEF as a systems engineering tool is dependent on this post-gameplay sharing. Future experimentation with the user community will require extensive debriefing, allowing players to compare scenarios developed within ICEF and allowing researchers to measure the effectiveness of ICEF as a collaborative CONOPS development tool. This debriefing will be facilitated through the animations created as an output of the ICEF modeling process. It is envisioned that following a controlled experiment, two opportunities exist for debriefing. As a systems engineering tool, groups of users will be able to replay ICEF-generated operational scenarios and discuss how they meet specific needs and what knowledge they were able to gain through creation of CONOPS. As a gaming simulation, players within a group will be able compare their in-game experience with that of other group members, highlighting the effectiveness of a gaming environment for the creation of CONOPS.
Following the feedback sessions discussed above, players testing ICEF conducted an abbreviated, informal debriefing session to assess their experience with ICEF in terms of its theoretical and practical application to reality (Connolly, Stansfield, & McLellan, 2006). Based on Kriz’s (2010) interpretation of the six phases of debriefing, the following observations were made:
Phase 1: How did you feel?
Players were asked how their experiences made them feel about ICEF, the process they used to develop CONOPS, and the interactions they had with their group. Younger, more experienced gamers felt that the game was easy to play and found value in its use as a medium for CONOPS development, organically adopting it as a tool for meaningful collaboration. Older experienced gamers seemed interested in, but distracted by, the gaming aspect of ICEF. They entered the environment with good intentions and motivation but soon found themselves veering off course, using ICEF more as an imaginative game rather than a constructive method for developing CONOPS. Older inexperienced gamers proved to be skeptical of the entire process and resorted more to talking about CONOPS and then using ICEF, rather than interacting with the game directly throughout the development process. Few of the younger players in the feedback sessions were inexperienced with games, so little information was collected on this subset. Future experiments will require a sample that includes younger, inexperienced gamers to determine the effects of age versus gaming experience.
Phase 3: Connection between gaming situation and reality
Two types of comments were made in regards to comparing the gaming simulation to reality. The first was related to the realism of ICEF, which was discussed above. The second was related to how the process of CONOPS creation with ICEF compared to the current CONOPS development process. The majority of participants saw value in the use of ICEF to think about the problem at hand and collaboratively develop a solution. A number of comments indicated that although ICEF was a useful tool, it would only be applicable to brainstorming prior to the traditional writing process of CONOPS. The current defense industry acquisition process reinforces reluctance to the drastic changes a gaming simulation like ICEF would introduce. This is an important consideration for future development of ICEF and how open-minded different communities would be to the use of ICEF in real world system development.
Phase 5: What would happen if…?
This topic of debriefing led to many of the improvements that have been implemented in ICEF since the feedback sessions. Major areas of conversation included the ability to extend existing databases to include unanticipated objects and relationships, as well as the ability to support multi-form collaboration. Discussion was also focused around how players approach gameplay. The ultimate goal of ICEF is to promote the collaborative development of CONOPS. Although games where a group of people work together to achieve a common goal are widespread, players saw ICEF as a balance of collaboration and competition. They believed that fellow players would be competing to have their point of view dominate the CONOPS. This competition does exist in the traditional CONOPS development process, but decisions are often made subjectively without transparency. The projected benefit of using ICEF would be that priorities are captured by the gaming simulation and decision making can be conducted in the open.
As discussed above, future experiments will include a detailed, structured debriefing exercise, and results from these sessions will be described in future publication of experimental results.
Future Work and Conclusion
This article has described the theoretical background for the use of gaming simulations to address a shortcoming in the systems engineering process, as well as a description of typical scenarios a user will encounter while using ICEF. In addition to expanding ICEF capabilities, future work will include the creation of CONOPS development metrics to provide the basis for a quantitative evaluation of the effectiveness of ICEF. Data relating to these metrics and qualitative survey data will be collected during two laboratory experiments of quasi-experimental design to determine both game and research validity. Following further maturation of ICEF and additional testing with stakeholders, a full scale debriefing will be explored.
CONOPS development is a critical step in successful systems engineering processes. Although systems are growing in complexity, no significant advances to CONOPS development have been made in decades. Gaming simulation has the potential to improve the way stakeholders reason about operational concepts and develop CONOPS, leading to better understanding of what a system needs to do to satisfy its stakeholders. We believe that the technology exists to enable visualization of operational concepts in collaborative, virtual environments.
Footnotes
Acknowledgements
The authors would like to acknowledge the contributions of research partners associated with this work including Ali Mostashari, Sara McComb, Abhi Deshmukh, Drew Hamilton, Jon Wade, Mark Blackburn, Deanna Kennedy, Behnam Esfahbod, Peizhu Zhang, Keith Hall, John Santanello, Patrick Pape, Jim O’Brian and Sarah Weeks. Additional research support was provided by the management team of the Systems Engineering Research Center. The authors would also like to acknowledge the support of Julia Lo, Geertje Bekebrede and Heide Lukosch of the CESUN Third International Engineering Systems Symposium and the numerous reviewers of this article.
Author Contributions
All authors contributed to this article, in content and form. PK performed most of the literature review and wrote the manuscript, with editing and contributions from RC and TZ. PK developed the experiments, which were executed by PK, RC and TZ, and analyzed by PK and TZ. PK, RC and TZ developed software architecture, and TZ managed and directed software development efforts.
Declaration of Conflict of Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported, in whole or in part, by the Systems Engineering Research Center (SERC). SERC is a federally funded UARC managed by Stevens Institute of Technology. Partners in this research have included Texas A&M, Purdue University and Auburn University. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of Stevens Institute of Technology and/or any agency or entity of the United States Government.
Author Biographies
Contact:
Contact:
Contact: