Abstract
Function allocation and distributed task coordination are complex challenges facing many multi-team systems. These challenges are intensified in the case of human expeditions to and exploration of Mars, due to the impact of one-way light-time communication delays that can exceed 20 minutes. Research to identify, enhance, and support new requirements for task coordination and communication include considerations to mitigate the impact of delays through improved state monitoring and crew coordination and knowledge sharing techniques. Effective coordination for human cislunar and Mars exploration operations, including servicing, assembly, and maintenance activities, require effective and adaptive function and task allocation constrained by available bandwidth and crew member workload capability. The authors describe some of their previous research and ongoing activities, including improvements to time-delayed information and data displays to support mission control and spaceflight crew member situational awareness when conducting both routine operations and real-time responses to emerging anomalies.
Keywords
Introduction
This paper sets out to further understand the flexible roles of ground-based operators and spaceflight crew members when performing complex tasks with increasing communication time delays. Role flexibility is governed by the question “who should be doing what, when?” in response to both nominal activities and off-nominal events. Flight rules have been established by NASA to define authoritative roles and responsibilities from spacecraft launch to re-entry. Flight rules for missions to the Moon and Mars will assign responsibility based on the communication delay, expected sensemaking and decisionmaking times of operators and crew members, and action execution time (Love & Reagan, 2013). Currently, information coordination and event (or fault) detection, isolation and response (EDIR) is facilitated by distributed supervisory coordination (DSC) between ground-based teams and the crew, also referred to as a system of multi-teams (Lungeanu et al., 2022). DSC is an extension of Sheridan’s human supervisory control (HSC) model where task complexity and system dynamics require more than a single centralized command and communication network (Caldwell 2005a). No human spaceflight mission has experienced round trip time delays more than approximately two seconds. However, missions to Mars reflect up to three orders of magnitude longer delays. Using JPL NAIF’s SPICE data for the Mars2020 reference mission, the time delay during the transit and planetary expedition phases increases from 0 to over 20 minutes, as shown in Figure 1 (Acton, 1996; Acton et al., 2017). Missions to Mars are scheduled for launch when Earth and Mars are closest, roughly every two years. Mission operations on Mars are also constrained when Mars is blocked by the Sun, or Mars solar conjunction, leading to communication blackouts for weeks at a time.
Nonlinear change in light time delay over the course of the Mars2020 mission.
In the presence of communication time delays, delayed feedback will lead to instability in the supervisory control loop as the reference input and process output are misaligned and diverging (Sheridan, 1993). The divergence is evident in the stale status information displayed to the mission’s groundbased operators. Combined with sensemaking and decisionmaking processing times, the once appropriate command could be out of phase with the condition of the spacecraft when the command is finally received.
For near-Earth missions, there is an adaptive hand-off of authority between the Mission Control Center (MCC) and the crew depending on the task and who is in a better position to perform the task (Caldwell & Onken, 2011). Examples of the control hand-offs are detailed in the Hubble Space Telescope (Hubble) servicing missions and the International Space Station’s (ISS) CanadArm operations sections of this paper. Efficient control handoff and coordination is reliant on information freshness and shared situational awareness between and within MCC and the crew, both of which are hindered by growing physical separation and communication time delays. During highly procedure-driven tasks on the ISS, such as an extra-vehicular activity (EVA), the crew members involved in task performance are often dependent on MCC for step-by-step instructions with little to no decision-making autonomy (Caldwell & Onken, 2011). On the other hand, when the crew has a better vantage point in tasks such as taking photos based on ground-initiated requests or controlling a robotic arm to capture an incoming free flight vehicle, the crew member can exercise judgment and make decisions without consulting each step with MCC (Dempsey, 2018). Even in low Earth orbit, MCC could not feasibly hand-control a robotic arm like the crew to perform a fine, time critical maneuver due to latency involved with transmitting commands and receiving feedback. With increased communication delays, traditional roles and responsibilities between MCC and the crew will become more adaptive with tools to assist the crew during procedure-driven tasks without extended crew idle time waiting for instructions.
One advantage of human exploration over robotic space missions is the ability to have real-time human-based EDIR closer to the point of exploratory action (i.e., on the surface of the Moon or Mars) and additional actions and resource allocations devoted to problem-solving and resolution tasks. An on-orbit or on-surface crew can detect an anomaly in real-time; however given the relatively smaller team size, crewmembers have limited resources and a broader but less in-depth knowledge base. Earth-based mission control teams are subject to delayed and limited information, but a larger network of resources and expertise is available. Coordination and control of resources and tasks for space-based EDIR have not been demonstrated by dispersed multi-teams with a dynamic time delay. Understanding innovative responses and coordination in analogous operations (such as deep-sea, polar construction, or oceanic drilling) and previous crewed missions (such as Hubble servicing missions and EVAs supported by the CanadArm robotic extension) could be the first step towards future space exploration endeavors to Mars with effective multi-team function allocation, adaptive roles in task performance, and improved information displays for EDIR amidst a complex dynamic task, resource, and time critical environment.
Coordinating Multi-Teams In Space Operations
Multi-teams in the context of space exploration are a system-of-systems (SoS) comprised of multiple physically distributed human-computer systems and human teams with varying domain expertise coordinating tasks, resources, and information towards shared goals (Lungeanu et al., 2022). Interfaces between the teams that facilitate communication (such as the MCC voice loops between the crew, flight director, crew communicator, front room personnel, and backroom subsystem experts) are crucial to SoS performance focused on crew safety and mission success (Dezfuli et al., 2010).
As it exists from the 1960s through the early 2020s, MCC is organized in a centralized control configuration “front room” supported by sub-teams in a distributed control configuration represented as “backroom experts”; in turn, the MCC is connected to the spaceflight crew through a special console with dedicated communication priority, known as CAPCOM (Caldwell, 2005a, b). Connecting these teams are information networks consisting of relay satellite constellations and ground station networks for nearly continuous communication and data sharing capabilities. Even near-Earth communication is still subject to latencies, loss of signal, and software or hardware malfunctions. For example, in February of 2013 there was a communication and command control outage between mission control and the ISS crew for almost 3 hours due to the inability to point the station’s antenna at a NASA tracking and data relay satellite (Halvorson & Today, 2013).
The communication interface between space and ground is weakened by the intrinsic time delays for missions beyond low Earth orbit (LEO). A weak communication interface combined with traditionally highly procedure-driven tasks constitutes evolving the supervisory relationships between mission control and crew to allow for flexible crew autonomy. The recommendation is not to use fully automated computer agents – leaving the crew out of the loop, but improve information displays and human-computer interactions to support a more autonomous crew by utilizing the principles of HSC and DSC for efficient task allocation.
Human Supervisory Control and Distributed Supervisory Coordination
In Sheridan’s HSC model, the human supervisor’s modes (Plan – Teach – Monitor – Intervene – Learn) form three nested loops with the supervised agent (Sheridan, 2012):
the inner loop – monitoring: acquiring measurements, estimating process state, and detecting anomalies
the middle loop – intervention and teaching: action to abort or error rectification, set new goal state
the outer loop – learning back to planning: after learning from the previous subtask, make improvements to strategy for next subtask
For this application the supervised agent can be automated components on the spacecraft or a crew member performing a highly procedural task.
The timescales for operating and revising each loop ascends from the inner to outer loop (Sheridan, 2012). The ground-based human operator in MCC is monitoring the state via downlinked telemetry with updates as a function of the spacecraft’s distance from Earth and associated computer processing latencies. These delays are exacerbated in the middle loop information exchange cycles where corrective action commands are transmitted to set a new goal state. The time for a ground-based human operator to acquire and synthesize measurements, identify an anomaly, determine corrective action, and transmit commands, describes a Class III event timescale of 10-1000s per task plus the two-way light time delay (Caldwell, 2005b), as shown in Table 1.
Characteristic durations of events in the spaceflight environment (Caldwell, 2005b).
Commanding and control solely from the ground is not a viable option for time sensitive operations in cislunar space or at Mars. However, training the small crew to operate at the breadth and level of expertise as MCC is also not viable.
A functional diagram of the current DSC model for MCC is depicted in Figure 2. Figure 3 includes the three-dimensional nature of human-human and human-system communication interfaces with a near-Earth spacecraft. A direct extension of the DSC model incorporating the space-based crew risks a breakdown in multi-team coordination (Class IV in Table 1), particularly if no consideration is given for the distinct roles and capabilities of on-board crew with different task demands and workload constraints, combined with significant communication time delays between the ground and the crew. Instead, dynamic and task sensitive function allocation for effective DSC relies on controller synthesis of individual knowledge, skills, and team coordination represented by the six dimensions of expertise defined in Garrett and colleagues (2009). The six dimensions of expertise are subject matter expertise, situational context, interface tools, expert identification, communication effectiveness, and information flow (Garrett et al. 2009).
DSC model in mission control (Caldwell, 2005).
Three-Dimensional coordination and communication interface with crewed spacecraft (after Caldwell and Wang, 2003).
The six dimensions of expertise in MCC operations are supported by the digital voice and intercommunications system (DVIS) which allows flight controllers in each sub-team to listen to multiple communication channels, or voice loops, simultaneously. The four types of voice loops in Figure 4 are the Air (or Space) to Ground loop for CAPCOM to crew as well as broadcasting crew response to all controllers, the Flight loop between front room controllers and the mission flight director, the Front-to-Back loop for front room controllers to communicate with their respective subsystem expert teams, and the Conference loop for less urgent side conversations between front room controllers (Patterson et al., 1999). DVIS loops have proven to be an effective interface for ground-based teams to investigate suspected problems, coordinate information and knowledge with respect to the six dimensions of expertise, and generate a resolution path.
Illustration of four types of voice loops used by mission control (Caldwell, 2008).
Previous human spaceflight missions with strong MCC and crew coordination, as well as multi-site communications are useful to describe and predict communication needs for future missions. Hubble servicing missions required coordination and communication between MCC at Johnson Space Center in Texas, the Hubble flight control team at NASA Goddard Spaceflight Center in Maryland, and the Space Shuttle crew. A second example is the coordination and communication between MCC, the Robotics Mission Control Centre in Quebec, Canada, and the crew onboard the ISS to conduct servicing, assembly, and maintenance (SAM) operations using the robotic arm system.
Dynamic Function Allocation In Multiteam Space Operations
NASA Mission Control, Goddard Spaceflight Center, and Space Shuttle Coordination in Hubble Serving Missions
The first Hubble servicing mission in December 1993 demonstrated and validated the feasibility of on-orbit maintenance. This shuttle mission was considered one of the most challenging because astronauts had to perform multiple tasks on a rigid schedule for the first time in the history of human spaceflight. Mission success criteria included minimizing risk to the flight crew, shuttle, and telescope while ensuring the primary tasks were completed to leave the telescope in best possible operating state. Servicing task priority was determined by time required, resources, and procedure complexity constraints (NASA, 1993).
In total there have been five servicing missions each involving extensive coordination between MCC, Space Telescope Operations control center at NASA Goddard in Maryland, and the space shuttle crew. MCC monitored the space shuttle and astronaut activities. The Space Telescope Operations commanded telescope operations by placing Hubble into safe hold mode and maneuvering Hubble to connect with the shuttle’s arm. After each installation the Space Telescope Operations team validated that the component operated as expected, otherwise additional service from astronauts was required (NASA, 1993).
After the first servicing mission, NASA Goddard began an effort to modernize operator telemetry displays and data storage systems into a web-based graphical user interface (Rifkin, 1997; NASA, 1999). In missions prior, flight controllers never had instant access to stored data. Before the new display system, data had to be requested from NASA Goddard and receiving it could take up to 24 hours (Rifkin, 1997). After the redesign, telemetry could be downlinked in real-time, converted to engineering units, and merged to create data streams to analyze current spacecraft states as well as trends (Rifkin, 1997; NASA, 1999).
In response to plans for humans to return to the Moon and venture to Mars, necessitates a new age of graphical data displays that recognizes the time-delayed data noting information freshness, time to expiration, and flagging trends that are approaching system control limits.
NASA Mission Control, Canadian Space Agency, and ISS in Canada Arm Coordination
The Canada robotic arm is a critical system for conducting SAM operations on the ISS, grabbling incoming free-flight vehicles, and maneuvering / supporting astronauts performing EVAs. Flight controllers transmit a series of preplanned, automated commands to govern the motion of the robotic arm. On the ISS there are workstations with hand controllers and video displays for the crew to control the robotic arm (Dempsey, 2018). The decision as to whether the crew or flight controllers should operate the arm is based on the following factors: can the operation be performed methodically by a sequence of computer commands, how far in advance the operation is planned, and how dynamic the operation is (Dempsey, 2018). Various cameras as well as computer models of the arm movement provide shared in-situ awareness for both the flight controllers and crew members when performing their respective tasks.
An example of a complex, dynamic activity is capturing a free-flying spacecraft to dock with the ISS. The series of tasks is coordinated between MCC, Canada’s robotic mission control center, and the onboard crew. Crews undergo extensive training for capturing visiting vehicles as this activity requires timecritical actions and reactions to unexpected events (Dempsey, 2018). The motion of the rendezvousing vehicle is not always steady or predictable so grappling maneuvers with the robotic arm requires expert hand/eye coordination and responses in a timescale beyond what the ground controllers can perform. Once the target spacecraft approaches the ISS, the crew is tasked with monitoring the vehicle, maneuvering the arm in response to the vehicle’s dynamics, and capturing the vehicle. From there the flight controllers regain control of the robotic arm and dock the vehicle (Dempsey, 2018).
The multi-team system control handoff describes how ground-based flight controllers and the space-based crew perform dynamic function allocation based on who is better suited for the task at that time. The ISS crew monitoring the motion of an oncoming vehicle has the benefit of receiving frequent and real-time feedback, enabling corrective actions, especially fine adjustments, with the shortest delay relative to the ground controllers. SAM procedures in cislunar space and at Mars will also require frequent and real-time feedback for time-critical actions. Given the time delay, only the crew with support from automated agents can perform these complex activities with suitable compensatory and pursuit control precision. There are multiple considerations to ensure effective crew-supported SAM operations in such conditions, especially since bandwidth is neither capable in throughput nor as reliable as terrestrial broadband communications (NASA, n.d.).
Future Multi-Team Space Operations with Dynamic Autonomy
Questions for future human space exploration of Mars include how much autonomy should the crew have and how should responsibilities and decisions be allocated between and within the multi-teams (Sheridan, 2012). In multi-team operations, dynamic autonomy shifts based on the task environment, expertise and time required to perform the task, and the expertise of and time available for the team in the immediate situation. Operators hand off autonomy when another team or system has better local information, multidimensional expertise, and situational awareness available within the time and task constraints of a time-critical event. For example, in the launch phase of a mission the time available to detect an anomaly and complete abort preparations is too short for onboard crewmembers or ground-based controllers to effectively perform manually, therefore the task is allocated to computers (Caldwell & Onken, 2011). As demonstrated by the coordination of tasks between mission control and crew for controlling the Canada robotic arm, the crew is in a better location for timely and precise movements using hand controllers than using a series of commands from ground. In missions to Mars, the crew is expected to have more autonomy to not only perform simple mechanical repairs but also structure activities and goals such as exploring and conducting experiments.
Computer “teammates” should be designed to support procedural activities – typically overseen by MCC, such as answering multi-domain technical questions and coordinating available resources with crew input (National Academies of Sciences, Engineering, and Medicine, 2021). Crewmembers can consult computer “teammates” for task-related information like component location and critical values as well as validate component operations like the Hubble Space Telescope Operations team did during the service missions. Researchers at UC Davis and JSC proposed a helmet mounted display as a human-computer interface for increasing astronaut situational awareness during an EVA (Moses et al., 2023). The results from the demonstration in NASA’s Neutral Buoyancy Lab highlighted that improvements to information displays and computer interactions require iterative processes for adapting to different users and dynamic operating environments. Testing these tools early in the design phase promotes human-computer team training and developing trust between the crew and the computer teammates.
Love and Reagan (2013) compile findings from highfidelity spaceflight analog tests that studied impacts of delayed voice communication during mission operations. One critical impact was on extended crew time dedicated to communications instead of completing the tasks scheduled. Additionally, simulation crew members rated back-and-forth delayed communications as unacceptable when there was a high workload task that required timely actions. Results from spaceflight simulations and human-computer interaction evaluations will help update message content and frequency, flight rules for decision-making, and between MCC-crew and crew-computer systems.
Conclusion
Crewed missions to Mars relying on multi-team operations is highly sensitive to limited time, resources, and expertise for function allocation. A crew operating on the surface of Mars faces communication time delays with Earth-based support that can exceed 20 minutes. However, the current MCC communication and control structure yet to be demonstrated in human spaceflight missions with round trip time delays more than approximately two seconds. Efforts towards mitigating the impact of delays on highly procedural task performance and responses to emerging anomalies begins with understanding adaptive function allocation, knowledge sharing, and coordination in analogous multi-team operations and previous crewed missions. In addition to understanding team knowledge sharing and coordination, further research is needed to enhance human-computer team interactions and predictive information displays for a variety of dynamic environmental conditions and events.