Abstract
Analog research of human or combined human and robotic missions is an established tool to explore the workflows, instruments, risks, and challenges of future planetary surface missions in a representative terrestrial environment. Analog missions that emulate selected aspects of such expeditions have risen in number, expanded their range of disciplines covered, and seen a significant increase in their operational and programmatic impact on mission planning. We propose a method to compare analog missions across agencies, disciplines, and complexities/fidelities to improve scientific output and mission safety and maximize effectiveness and efficiency. This algorithm measures mission performance, provides a tool for an objective postmission evaluation, and catalyzes programmatic progress. It does not evaluate individual sites or instruments but focuses at mission level. By applying the algorithm to several missions, we compare the missions' performance for benchmarking purposes. Methodically, a combination of objective data sets and questionnaires is used to evaluate three areas: two sections of closed and quantitative questions and a third section dedicated to the level or representativeness of the test site. By using a weighted metric, the complexity and fidelity of a mission are compared with reference missions, which yield strengths and weaknesses in mission planning.
1. Analog Mission Performance Metrics Description
Analog research paves the way for planetary missions to Mars, the Moon, and beyond by exploring the risks and challenges of an interplanetary mission in a controlled environment (Mahoney, 2009; Dorizon et al., 2016). The awareness of such analog missions and their number has increased in recent years. The Austrian Space Forum (OeWF), therefore, has proposed an algorithm with which to compare analog missions across agencies, disciplines, and complexities/fidelities to improve the scientific output and mission safety and maximize the effectiveness and efficiency of analog missions. We also strive for an improved understanding of analog missions, as set forth in the Vienna Statement on Analog Planetary Research (OeWF, 2016).
The importance of this endeavor was revealed during Desert Research and Technology Studies (DRATS) that were performed in 2010 (Eppler et al., 2013), where “the effort to understand the effectiveness of different planetary exploration approaches has led to the collecting of numerical metrics on a variety of test parameters” (Eppler et al., 2013). This initiative was also employed to ensure an efficient use of National Aeronautics and Space Administration (NASA) budgeting and hence show how such an algorithm could be put into use in various aspects of analog research.
Eppler et al. (2013) further pointed out that numerical data sets can facilitate decision-making processes and grant insights on where to increase time efficiency. Hence, we introduce a standardized algorithm to measure the performance of planetary analog simulations that enable objectification and provide the means for comparing the performance of planetary analog missions across agencies and disciplines The standardized algorithm also provides a mission internal analysis of the respective strengths and the possibility to improvement.
Taking previous OeWF analog missions into consideration and other contributions of other agencies and organizations, we studied and analyzed the main features and aspects that characterize this overall field of research. We identified performance parameters that are decisive for analog missions and developed a novel algorithm that could provide an objective evaluation. These parameters can be considered as inputs of a system for which we aimed to determine a cause-and-effect relationship between outputs and outcomes. We selected the Performance Matrix as the proper evaluation tool, which is defined as the process of collecting and analyzing information regarding the performance of an individual, group, or organization that is representative of their actions, abilities, overall quality, and success. This tool is used by managers and key decision-makers to understand how and to what extent the inputs are enabling progress toward outputs and outcomes.
However, the identified parameters cover a wide variety of elements of analog research, which at first were not easy to quantify with regard to their impact on outcome or to be correlated. Some research elements can be easily quantified, whereas others may represent a very specific aspect of a particular mission. These considerations brought us to define the three main sections of the Analog Mission Performance (AMP) metrics as follows: Level of representativeness of the test site, Dichotomous, and Quantitative sections.
For this first release of the algorithm, we defined for the first two sections two main key performance indicators (KPIs) that we believe characterize two main aspects of analog missions: complexity and fidelity. This is based on the assumption that analog missions aim to simulate future planetary surface operations in a scenario, which is as representative to actual flight missions as possible and hence have maximum complexity and fidelity.
These KPIs further allow for the plotting of a two-dimensional graph, a comparison between different missions, a measurement of missions' performance, facilitation of an objective postmission evaluation, and catalyzation of a programmatic evolution. The proposed algorithm also enables a well-structured mission planning, when the above-listed parameters are taken into account before the mission. Furthermore, the algorithm serves as a measure and benchmarking tool for funding entities to determine those research endeavors that should be supported. Hence, research teams that are aware of this algorithm will have a competitive advantage.
Developed especially for planetary analog missions, the AMP metrics algorithm specifically excludes: small-scale tests with low fidelity in terms of operations, such as laboratory instrument calibrations or platform demonstrations (e.g., thermal vacuum testing), computer simulations that include virtual reality, and actual flight missions.
Since there is a nondiscrete transition between laboratory instrument validations and, for example, instrument calibrations in representative environments, it should be noted that the AMP metrics have been designed to measure multidisciplinary planetary surface mission simulations. Thereby, it also takes operational constraints and environmental challenges into account in addition to the scientific and engineering performance evaluations.
2. Workflow for the Analysis of Analog Missions Using the AMP Metric
This section focuses on presenting the workflow behind the analysis and interactions performed by the involved parties that lead to the evaluation and comparison of analog missions. The overall process is summarized in Fig. 1, the first step of which is to contact the analog mission project leaders and provide them with a questionnaire for required information.
Deployed workflow for the AMP metrics evaluation. AMP, Analog Mission Performance; KPIs, key performance indicators; OeWF, Austrian Space Forum.
Once the questionnaire is completed, it is sent back to the OeWF for analysis of the data, scoring, and comparison with other missions. As presented in the Supplementary Data, the questionnaire is divided into five sections, the first four of which are used only to collect information about the mission, whereas the last is used by OeWF to collect feedback about the questionnaire itself to facilitate improvement of future queries.
Of the four main sections, only the first two sections follow a scoring process that assigns points to one or both of the two KPIs identified and used in this research, complexity and fidelity.
2.1. Section A: Level of representativeness of the test site
This includes three sets of two tables each of which will describe nominal conditions on the targeted planetary scenario, the expected/ideal conditions on the selected test site, and the actual scenario faced during the mission as defined with regard to weather and surface features. The tables of the nominal and expected scenarios are used to define a first preliminary score; each table will assign a maximum of four points to the fidelity parameter, depending on the number of matching features between nominal and expected scenarios, using the following equations:
n° Total features = 7 (for the weather) and 8 (for the surface)
Then, each matching feature is compared with those provided in the actual-observed-scenario. If the match is also confirmed in the third set, no adjustments are made to the score, whereas if the match is not confirmed in the actual scenario, the final value is computed with Equation 1 by comparing the nominal scenario with the actual-observed-scenario.
The difference between the ideal scenario on the test site and the observed scenario can be used as a reference for future missions in the same area.
2.2. Section B: Quality performance indicators
This consists of 43 closed questions, each of them to be used to assign 0 or 1 point to one or both of the main KPIs.
Table 1 summarizes the maximum scores that can be assigned to each KPI.
Fidelity and Complexity Scores Summary
KPI = key performance indicator.
The scoring process will then provide a set of two numbers that can be represented as a point in a bidimensional graph in which the axes are the two KPIs—complexity and fidelity—allowing an easy and visual comparison of the missions. Figure 2 shows an example of such a bidimensional graph by way of random data to visualize a possible outcome and examples of the analog missions, to which the AMP metrics could be applied: Haughton–Mars Project, Pavilion Lake Research Project, Mars-500, PolAres missions by the OeWF, Arctic Mars Analog Svalbard Expedition, NASA Extreme Environment Mission Operations, and DRATS.
Shows a possible example of outcome (random data have been used) of the values of the KPIs for the selected mission excluding the environment features. AMASE, Arctic Mars Analog Svalbard Expedition; DRATS, Desert Research and Technology Studies; HMP, Haughton–Mars Project; NEEMO, NASA Extreme Environment Mission Operations; PLRP, Pavilion Lake Research Project.
However, the analysis of the analog mission is not solely based on the complexity and fidelity KPIs. In addition, the 21 quantitative performance indicators (QuantPIs) questions will be analyzed to evaluate other performance parameters and define missions' strengths, weaknesses, opportunities for improvements, and threats (e.g., the analysis outcome of a mission might show a strong safety and security aspect, but a low mission preparation).
After the first analysis of the questionnaire, an interview with the mission project leaders will take place to discuss the results and further investigate or clarify doubts about the provided data or the feedback provided by the team leads in the questionnaire. This method also limits the risk of ambiguousness and ensures the same standards for the provided data for each mission that is being compared.
The main outcome of the overall process (questionnaire + interview) will be a better understanding of mission performances that could be used to address new challenges in the mission design by increasing its complexity and fidelity or further improve some other mission-related aspects. At the same time, the process provides continuous feedback for the questionnaire itself that could lead to further improvement and development of its capabilities, finding gaps, and aspects not covered in the current version.
Overall, the proposed process intrinsically foresees a feedback loop between all the parties involved that aims to promote the evolution of analog activities.
3. Methodology of the AMP Metrics
As an evaluation tool for collecting the data input for the AMP metrics, a dedicated questionnaire was developed. The following sections describe the selected mission performance parameters, each section of the questionnaire, and their respective methodology.
3.1. Mission performance parameters
Complexity and fidelity are the two main parameters selected as KPIs for the evaluation of analog missions. The overall simulation of extraterrestrial surface operations poses challenges and reveals issues, which have to be considered before a crewed mission to Mars. For this reason, several analog missions have been performed by space agencies and institutes around the world, each of them focusing on a few specific objectives at a time. Each challenge faced during a mission increases its complexity, such as including time delay into the communication between the simulated planetary surface and mission control on Earth or the usage of spacesuit simulators during extravehicular activities (EVAs). Each challenge is addressed in the questionnaire. For each challenge included in the analog mission, the complexity score gets higher (e.g., to include a time delay in communications for simulating a crewed mission on Mars will yield an additional point for the complexity score).
At the same time, performing the simulation in an environment as close as possible to the actual planetary surface increases the level of fidelity. The previous motivations lead to the selection of complexity and fidelity as main KPIs for the AMP metrics.
Through the obtained data, it is possible to further evaluate and compare the mission efficiency with regard to operational costs, time, planning, and outreach impact. These parameters are not direct measures of the success of an analog mission, but rather contribute to the postmission analysis. Whether the mission had dedicated success criteria or the success criteria were achieved is included in the questionnaire and contributes to the planning efficiency section. Thus, such parameters can aid institutions to pursue a programmatic evolution and improvement over several analog missions.
3.2. Level of representativeness of the test site
This section collects information regarding the test site and its representativeness with respect to the nominal environment on the targeted planetary surface that was simulated in the analog mission. Questions about the average temperature range, the average precipitation expectancy, the wind conditions, and additional features (e.g., whirlwinds also known as dust devils for Mars simulations) allow for assessment of the expected environmental conditions at the test site. Hence, the results of this section can either lower or improve the level of fidelity of the analog mission and contribute to the fidelity score. Dense vegetation or strong precipitation at the test site would, for example, lower the fidelity score for an analog mission that aims to simulate present-day Mars (Sections 1 and 2.1).
3.3. Dichotomous section
This section is composed of closed questions with the objective of collecting specific information (quality performance indicators [QualPIs]) related to the following categories:
field operations at the simulated extraterrestrial environment,
Earth operations (i.e., remote mission support/control),
planning strategy,
long-term development,
preparation (i.e., crew training, experiment familiarization),
safety and security, and
soft factors.
Each section is composed of multiple questions that contribute to one or both of the KPIs, complexity and fidelity, which aim to represent the performance of the mission.
3.4. Quantitative section
In this quantitative section, questions related to general facts about the mission (i.e., size of the crew, allocated budget, and mission duration), planning efficiency, and public impact (i.e., papers, citations, interviews, tweets, and posts) collect numerical input for the AMP metrics to assess the QuantPIs. The obtained information will be used to evaluate the mission efficiency and performance in the following areas: operational costs per day, time efficiency, budget composition, mission crew composition, and public impact. The results of this section provide insights on how the evaluated analog missions can improve their efficiency and with regard to conducting scientific experiments and performing surface operations with time-delayed communication.
4. Analysis of Selected Analog Missions
In this section, the AMP metrics are applied as described in the section before. The evaluated missions are three analog missions of the OeWF: MARS2013, AMADEE-15, and AMADEE-18, the most recent mission of the OeWF.
The required information used for the AMP metrics questionnaire was extracted from the official mission reports and published articles (OeWF 2013, 2015b) or was provided directly by mission project management (G. Groemer, personal communication, 2016; S. Gruber, personal communication, 2018) and yielded the analyses discussed subsequently.
4.1. OeWF MARS2013
4.1.1. Mission introduction
In 2013, the OeWF performed a 1-month Mars analog simulation in the northern Sahara near Erfoud, Morocco. A small crew, directed by the Mission Support Center located in Austria, conducted experiments in the fields of engineering, planetary surface operations, astrobiology, geophysics, and life sciences.
The main objective was to gain operational experience in the conduct of long-duration, complex planetary simulations. The Morocco desert represented a suitable location not only for its topographic and geological resemblance to martian surface features but also its accessibility and expected security level and reliability of the hosting country.
4.1.2. Complexity and fidelity scores
The analysis from the QualPIs in the questionnaire provides scores for the complexity and fidelity of each aspect of the MARS2013 mission, respectively (Table 2).
Complexity and Fidelity Scores of the MARS2013 Mission, Obtained from the Dichotomous Questions in the Analog Mission Performance Metrics Questionnaire
Where the complexity of the field and Earth operations is very high, due to the developed infrastructure, standard operating procedures (i.e., instructions on how to conduct routine operations), and implemented hardware, the fidelity can be increased by using a mock-up of a base habitat, rather than tents. The standing Flight Plan Team of the OeWF assures a high score in the planning strategy section, both for the fidelity as well as for the complexity, as their products (flight plans and schedules) are representative to what would be expected for an actual flight mission.
The need for additional field crew members—excluding experimenters such as the analog astronauts—at the MARS2013 test site to support field operations (e.g., with setting up the communication infrastructure) is one weakness that lowered the fidelity of the mission. During an actual mission on Mars, of course, only a small crew of astronauts would be involved in conducting operations on the planetary surface with no support crew whatsoever.
The presence of vegetation at the site in Morocco lowered the level of representativeness of the test site, as the targeted planetary body for this simulation was present-day Mars, where signs of vegetation are yet to be found. However, the remoteness of the site provided additional benefits and opportunities for human factor studies on the effects of isolation.
4.2. OeWF AMADEE-15
4.2.1. Mission introduction
In 2015, one of the most recent OeWF analog missions took place on a rock glacier in Kaunertal, Austria. The 2-week Mars simulation mission was conducted by field crew located on the glacier and supported by a Mission Support Center in Innsbruck. The small crew performed experiments in preparation for future human Mars missions in the fields of engineering, planetary surface operations, astrobiology, and geosciences. The mission goals were to investigate the limitations and opportunities for the study of a martian rock glacier with human explorers and state-of-the-art instrumentation. The goal here was to assess novel mission support strategies, decision-making workflows, and near real-time data analysis for flight planning.
4.2.2. Complexity and fidelity scores
The complexity and fidelity scores in Table 3 reflect the fact that the infrastructure and standard operating procedures of the OeWF provided the means for high-fidelity simulations of planetary surface operations, with a standard that can be kept and improved over more years. The fidelity score of AMADEE-15 is less than that of MARS2013. This is due to the fact that, during the AMADEE-15 mission, the field crew stayed in a hotel, which led to increased interaction with the general public, while at the same time to a decreased perception of isolation of the field crew and fidelity in the selected scenario. However, during the simulation times (i.e., when only time delayed communication to the Mission Support Center (MSC) was allowed and scientific experiments were conducted), this interaction was kept to a minimum (OeWF, 2015a).
Complexity and Fidelity Scores of the AMADEE-15 Mission, Obtained from the Dichotomous Questions in the Analog Mission Performance Metrics Questionnaire
The lesson learned from team leads in the wake of the AMADEE-15 mission showed the need for structured reporting systems and more training of all team members before the mission to avoid “training on the job.” This was also reflected in the QualPIs since the scores on questions related to mission preparation and long-term development were low, which further confirmed the lessons learned about the need for developing reporting structures for mishaps and showed the potential for improvement in the training and certification process as well as staffing of the MSC. As a consequence of this outcome, the OeWF introduced the Analog Mission Basic and Advanced training courses to certify and train both the personnel of the MSC as well as the field crew. Those training cycles resulted in a significantly more professional and efficient execution of the AMADEE-18 simulation.
4.3. OeWF AMADEE-18
4.3.1. Mission introduction
The AMADEE-18 analog Mars mission was conducted by the OeWF in cooperation with the Oman Astronomical Society in the Dhofar region, Sultanate of Oman. During the 1-month mission in February 2018, a carefully selected and trained field crew conducted experiments from various disciplines (e.g., engineering, planetary surface operations, astrobiology, geophysics/geology, life sciences) in preparation for future human Mars missions.
The aim of AMADEE-18 was to implement the so-called “exploration cascade” to efficiently deploy experiments with the intent to emulate and investigate the methodologies to be used for the search for life on Mars. For more details about the AMADEE-18 mission, see the work of Groemer et al. (2018).
4.3.2. Complexity and fidelity scores
In Table 4, the complexity and fidelity scores of each section are listed for the AMADEE-18 mission. Compared with previous OeWF missions, AMADEE-18 obtained the highest fidelity score. This is mainly due to the high representativeness of the test site as well as the increased fidelity of the field operations in Oman. For the first time, a mock-up of a base habitat was assembled as opposed to tents to increase the fidelity of the mission. However, during simulation mode, safety personnel were also in the vicinity of the analog astronauts during EVAs. Although they kept out of sight and were only allowed to interact in emergency situations with the analog astronauts, this modus of operation reduced the fidelity and will be changed for future missions.
Complexity and Fidelity Scores of the AMADEE-18 Mission, Obtained from the Dichotomous Questions in the Analog Mission Performance Metrics Questionnaire
The standing teams of the OeWF represent one of the strengths of the OeWFs analog missions, as they assure a high level of professionalism because they have to undergo a dedicated training and certification process. This is reflected in the high scores in the Earth operations, preparation, and planning strategy sections.
4.3.3. Comparison of selected analog missions (quantitative analyses)
The additional QuantPIs studied for this part do not represent a direct measure for the mission success, but rather provide more detailed insights into the mission and enable comparison to other analog missions (e.g., Fig. 3). However, the comparison is performed only if the mission goals, as requested in the questionnaire, are comparable and compatible (e.g., Mars500 mainly focused on isolation effects on crew members, whereas AMADEE-18 on surface operations).
The general modus operandi for OeWF mission was that the simulation and implemented time delay (to account for the light travel time between Mars and Earth) start each day at a predefined checkpoint. Thus, not all the time available at the test site during the mission is spent in “simulation mode.” The analysis on time efficiency in Fig. 4 shows that 22.5% of the time available during MARS2013 was spent in “simulation mode,” which means that all communications during that mode were channeled through the time delayed server. In comparison to the achieved efficiency during MARS2013, the percentage of available time spent in simulation mode was, at 10%, less during AMADEE-18 (OeWF, 2018). This was because the temperatures in Oman were higher than expected, which yielded an adaption in the daily activity plan and finally shorter EVA times as well.
The budget composition is also highly influenced by the choice of test site. Due to the remoteness of the site in Morocco, the expenses for logistics made up 45% of the overall budget (see Fig. 3). The costs for housing and meals, mainly for the field crew, were even higher for AMADEE-15 since the field crew slept in a hotel close to the test site. Another important cost factor, which varied for the OeWF analog missions, was the costs for base habitat and infrastructure. This variation is based on different arrangements with alternating host countries (as in Morocco and in Oman). In some cases, a base habitat must be provided by the OeWF, as during AMADEE-15 (OeWF, 2015b).
The public impact of an analog mission is measured by the published articles (in print as well as online media) and TV reports (Fig. 5). Each category is again divided into national (i.e., the home country of the executing institute) and international publications. Here again, the selection of the test site has a big impact. When comparing MARS2013 with AMADEE-15, it can be seen that the number of national TV reports and articles was higher for AMADEE-15 due to the strong regional reference in Austria. However, the national coverage of MARS2013 was higher. The same tendency is also applicable for the AMADEE-18 mission, which achieved the highest coverage of all OeWF missions so far.
Numbers of published articles (online and print) and TV reports, national (Austrian) and international, respectively. During each AMADEE-15 and Mars2013, one international TV report was broadcasted.
5. Conclusion
The AMP metrics provide a means by which to compare and assess different aspects of planetary analog missions. In particular, it reveals areas in which institutions can increase the level of complexity and fidelity of their operations. Furthermore, applying the AMP metrics to several analog missions conducted by a single institute provides insight into the long-term trends and developments of such missions.
Throughout this work, the AMP metrics were applied to different analog missions of the OeWF to (1) demonstrate the AMP metrics workflow, (2) assess the strengths and opportunities to improve in the assessed areas, and (3) present the long-term development and research efforts of the OeWF.
The results of the AMP metrics for the OeWF analog missions show that the establishment of both infrastructures and procedures with which to conduct complex planetary analog missions was accomplished, which helped to increase their fidelity over several years' time. Furthermore, a significant increase in the public impact recognized as well.
However, the time efficiency decreased, due to more complex operations and environmental challenges (i.e., extreme temperatures) during the most recent AMADEE-18 mission. Hence, the AMP metrics revealed this issue, which the OeWF will address for future analog missions.
Further analyses of analog missions from different institutions and agencies are planned with the intent to compare work from different missions across agencies and establish a common understanding of analog planetary missions.
Footnotes
Author Disclosure Statement
No competing financial interests exist.
Funding Information
No funding was received for this work.
Supplementary Material
Supplementary Data
Abbreviations Used
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
