The Journal of Defense Modeling and Simulation: Applications,Methodology,Technology

The theory of change links a strategy’s theory about the source of the conflict and the intervention philosophy to the implementation of programs and activities in the field. This paper focuses on strategic assessment at the national level, a step above the operational assessment at the departmental or agency level. The paper first discusses the importance of understanding a strategy’s theory of change and how policymakers at the national level think about the problem and how to intervene. The paper uses counterterrorism and conflict resolution strategy as examples. The author discusses two special challenges: how policymakers frame the desired outcome, and multiple theories. The author argues for strategic assessment officers to discuss a strategy’s theory or theories of change so that policymakers and national-level leaders can better understand how different theories drive their intervention plans and strategies. By improving policymakers’ understanding of their theory of change, and improve the clarity of the desired outcomes, strategic assessment officers can better measure the effectiveness of national-level strategies.

Military planners and commanders participate in two activities, design and assessment, that are currently separate and distinct processes. Military design aims at environmental understanding and crafting well-justified interventions. Assessment requires the discernment of whether these interventions are working. However, the logical and practical connections between design and assessment are under-theorized. We argue that the exercise of military design and assessment must become indistinguishable in practice. Assessment must be a continuous action as commanders and staff officers understand the environment, craft an array of interventions, and determine if these interventions are working.

In support of efforts to predict nation-states’ health, this paper develops a methodology that addresses incorporating disparate data from a variety of sources and utilizing both subject matter expertise and mathematical insights to effectively choose variables to use in a predictive model. The developed model can then capture the interrelatedness and complexities reflective of an actual operational environment. When selecting the indicators to be included in the composite indices, it is critical to evaluate the correlation that exists (or does not exist) between indices. If significant correlation is found, it could result in “double-counting” one or more variables, meaning bias exists in the analysis. Once the indicators have been selected and composite indices formed, in order to effectively evaluate a country’s health, we have developed a regional assessment framework and provided an example risk assessment based on the country of Syria. The results of this research provide a robust analysis approach to understanding and, ultimately, assessing a nation-state’s health. While the initial steps rely heavily on subject matter expert input for identification of indicators for each Political, Military, Economic, Social, Information index, once a solid list of indicators has been selected and weighted, future analysis can be readily conducted following the Region Mean Assessment Framework plan.

Over nearly two decades of war, the Department of Defense has spent tens of millions of dollars to survey the public in war-torn areas such as Iraq, Afghanistan, Philippines, Syria, and the Horn of Africa. Most often these surveys are conducted by a private polling firm, which, in turn, conducts the field work. Since field work occurs in contested or enemy-dominated terrain, direct supervision of the field work is not possible. This lack of oversight is concerning to senior leaders, who may inform decisions and assessments using this polling data. This article reviews some best practices for the expeditionary operations research system analyst when faced with the task of planning, contracting, executing, analyzing, and reporting public perception polling data in conflict affected areas. Additionally, this work codifies some of the insights uncovered during a community of practice meeting in March 2018 to include the use of latent data sources such as social media. The purpose of this effort is to provide analysts the tools to increase senior leader confidence in the data that is oft relied upon to inform operations.

Assessors often become invaluable assets to commands, especially during the execution of critical or non-standard missions. Unfortunately, due to the nature of these missions and staff turn-over, assessors often start from ‘scratch’ as young officers in nascent assessment programs. Years of lessons learned are often overlooked. This paper brings together a team of analysts with separate experiences as assessors in Army and Joint assignments from the Combined Joint Task Force (CJTF) level to the Brigade level to share insights gained through experience that apply in nearly all assessment contexts. The authors focus on three primary areas. First, they show how assessors must understand context and should often use reason and qualitative data more than rely on numerical metrics. They then explore how assessors can help staffs and leaders accurately identify trends amidst noisy and chaotic operational environments. Finally, they demonstrate how the assessment team can leverage and assist the staff to enable better assessments.

Evaluation theory provides a rigorous foundation for the practice of military operation assessment. Government and industry assessors have used evaluation theory to improve the effectiveness of assessment across a wide range of fields. This article focuses on the relationship between evaluation theory and military assessment. We briefly survey the major evaluation approaches with a focus on connecting the theoretical models to practical security-related applications. These evaluation approaches include expertise-oriented, program-oriented, decision-oriented, and participant-oriented models. Within the overarching framework of these approaches, we consider alternative monitoring and evaluation designs in detail, including descriptive designs (case study, cross-sectional, time-series), quasi-experimental designs (interrupted time-series, comparison group, case study), and experimental designs (posttest-only, pre-post). Then, we discuss quantitative and qualitative methods for analyzing and reporting uncertainty with respect to each design alternative, with an emphasis on mixed-method approaches. Throughout the review, we make the relationship between evaluation theory and operation assessment practice explicit through examples, and we suggest more detailed references where appropriate.

A digital transformation has been implemented across a range of industries to drive affordability, agility, quality, and efficiency. Advancements in digital technologies are unleashing innovations that provide an opportunity to transform the engineering practice. Digital engineering is the Department of Defense’s (DoD’s) initiative to transform the way it designs, develops, delivers, operates, and sustains complex systems in a formidable and changing threat environment. It is defined as “an integrated digital approach that uses authoritative sources of systems’ data and models as a continuum across disciplines to support lifecycle activities from concept through disposal.” This paper sets the strategic direction for the digital engineering vision, in which five strategic goals are presented to guide DoD’s transformation efforts. This paper also describes the digital engineering initiatives that are in progress across the Office of the Secretary of Defense, Air Force, Army, and Navy. While these efforts have begun to realize initial benefits, challenges and next steps are also presented in order to realize the vision.

This article describes Department of Defense (DoD) Systems Engineering Research Center (SERC) efforts leading to and supporting the DoD Digital Engineering (DE) initiative. Topics include the SERC’s initial collaboration with Naval Air Systems Command (NAVAIR) in 2013 as well as ongoing SERC DE research. The article also identifies future research needed to continue to develop the DE ecosystem for system of systems acquisition, which will require rapidly changing mission strategies to address ever-evolving threats.

The USA faces a multitude of threats to its national security and international interests in an era of exponential technology growth and unprecedented access by anyone with a smartphone. Traditionally, the acquisition of US defense systems has relied on sequential methods of conceptual design and development. While successful in the past, these methods are time consuming and in danger of creating vulnerability gaps that could limit or constrain US response options. The challenge is clear. Either the US Department of Defense (DoD) evolves the way it plans, develops, buys, and manufactures new weapons systems, or it cedes the high ground to a rapidly changing global environment. Adapting and expanding advances in high performance computing (HPC), developing and employing complex physics-based software tools for high fidelity modeling and simulation, and implementing a vision that combines these elements with other processes are critical enablers the DoD is pursuing. This paper describes the synergy of three major DoD efforts designed to address needs in the areas of acquisition program development and execution: the DoD High Performance Computing Modernization Program (HPCMP) (the DoD HPCMP began as an Office of the Secretary of Defense program in 1992; in October 2011, leadership transferred to the Assistant Secretary of the Army for Acquisition, Logistics and Technology; the US Army Corps of Engineers Engineer Research and Development Center manages the program, https://www.hpc.mil/index.php) Computational Research and Engineering Acquisition Tools and Environments program; the Engineered Resilient Systems program; and the DoD Digital Engineering vision.

This paper provides an overview of modular open systems approach (MOSA) design principles and open business practices in Department of Defense (DoD) programs, as well as selected examples of MOSA implementations by the Military Departments. The National Defense Authorization Act (NDAA) for Fiscal Year (FY) 2017 (Public Law 114-328) Section 805(a) requires DoD to implement MOSA, and an increasing body of evidence indicates MOSA can enable outcomes that merit consideration and advancement. Most DoD programs had already incorporated MOSA to some extent before the NDAA required it, but the formal acknowledgment in law further emphasizes the currency of the approach.

MOSA has multiple, similar definitions. For the purposes of this paper, MOSA is defined as a method to design systems with highly cohesive, loosely coupled, and severable modules that DoD can compete separately and acquire from independent vendors. MOSA is an approach rather than a technical or warfighting requirement. The approach allows the Department to flexibly acquire full capabilities and individual components – including systems, subsystems, and software – and create end item systems and services that can respond to changing threats and missions, allow for technology advances, upgrade and sustain when necessary, and maintain appropriate security assurances.

Through this paper, the Office of the Deputy Assistant Secretary of Defense for Systems Engineering (ODASD(SE)) discusses considerations for implementing MOSA, provides three examples of specific implementations by the Air Force, Army, and Navy, and discusses the vision for the future of MOSA application in DoD. The information is based on a literature review of published DoD reports and public documents, interviews with DoD Services discussing lessons, the Services’ written responses to the authors’ requests of exemplars, and review and comments from stakeholders across DoD. The authors concluded that the DoD Services and Agencies’ initiatives contribute to the MOSA body of knowledge and practice.

The US Department of Defense’s acquisition strategy incorporates directives to encourage the use of open architectures and modular solutions through the Modular Open Systems Approach (MOSA). The ways in which open standards are currently implemented, and programmatic guidance regarding the adoption of modular approaches, are inadequate, however, because of limitations on how modularity is objectively viewed to achieve its perceived benefits. Furthermore, current examples of implementations of modular concepts largely do not consider interdependencies at the enterprise level. This paper reviews ongoing research on modularity and openness, to synthesize best practices, community driven knowledge, and technical and programmatic catalysts that can better shape the appropriate adoption of MOSA. These items will be part of a comprehensive decision-making framework that can provide guidance to program managers in defense acquisition.