Workload,awareness and automation in multiple-robot supervision

The rationale behind single-human supervision of multiple robots

A robot or group of robots may be supervised by a group of people. For example, the General Atomics Aeronautical Systems MQ-9 Reaper UAV requires a pilot and a payload operator. Alternatively, robots can be supervised by a single human. In their analysis of the multiple unmanned vehicle deployment, Riley and Endsley ¹⁴ have suggested that the reduction in the ratio of human supervisors to unmanned vehicles deployed is a likely solution to performing functions such as reconnaissance and surveillance. This is because using a smaller human to robots ratio can help to make efficient use of possibly limited human resources. This addresses the motivation to reduce the number of personnel and the labour costs in work related to unmanned ground and aerial robots. ¹⁵ In some applications, human supervisors can be still exposed to the dangers of the robots’ mission environment despite a standoff distance between robots and supervisors. The development of a multiple-robot system that requires only a single human for robot supervision can help to reduce the number of people exposed to such dangers. For instance, in the description of their experience at ground zero in the aftermath of the September 2001 attacks, Murphy et al. ¹⁶ revealed that fire chiefs preferred as few as possible people in the disaster environment. Presumably, this was to prevent more casualties if the rubble collapsed further or if fires in the rubble resulted in explosions.

The cognitive challenges in multiple-robot supervision

While there are benefits in single human supervision of multiple autonomous robots, some drawbacks do exist. Supervising a single robot can already be challenging. The addition of more robots under the human’s supervision is likely to heighten this challenge. Consequently, the supervisor would experience more workload with the addition of each robot. ^{14,17 –20} There is then a greater propensity for the supervisor to fail to perform critical tasks. ²¹ Awareness for the actions of robots, the status of each robot as well as their progress in fulfilling the mission objective is critical when deploying robots. ²² However, maintaining the necessary situation awareness for every robot may be difficult as each robot competes for the supervisor’s limited attention resources. ² Furthermore, the supervisor has to periodically switch attention from one robot to another. As attention is shifted, effort is needed to acquire awareness about the situation of the robot being attended to. This constant effort to realign the supervisor’s understanding of the robot’s situation requires time and may contribute to stress during time-critical deployments. That effort, in addition to maintaining a mental model of each robot’s status, progress and situation, distinguishes multiple-robot supervision from that of a single robot. The intensification of cognitive workload and competition of supervision attention resources can lead to a reduction in the supervisor’s level of awareness. ²¹ Impaired situation awareness in turn heightens the possibility for errors. ²³

Mental and physical workload

Workload is sometimes referred to as cognitive workload in order to highlight emphasis on mental exertion. Quite simply, it is the degree of being busy. ²¹ Although various attempts have been made to describe it, ^24,25,26 there is yet to be a consensus in terms of a definition for cognitive or mental workload. ²⁷ However, definitions of workload are predicated on the assumption of limited mental capacity for perception and response selection. As this finite supply of mental resources is depleted, mental workload increases. ²¹

Yet, workload not only pertains to mental activity but is related also to physical exertions. Some degree of physical activity is required even when using automated systems. Such activity may not be highly laborious but can nonetheless require movements of some complexity and time to execute. Therefore, neglecting the effects of physical effort even for tasks which are mostly cognitive, can be detrimental to system design.

In his review of studies on combining physical and mental workload, Tomporowski ²⁸ indicated that aerobic activity in the low-to-moderate-intensity range can enhance information processing. This boost in mental performance may be seen in simpler tasks involving simple detection, visual search and discriminative choice response. ²⁹ Alertness may be temporarily heightened following short-duration, submaximal physical exercises. However, it remains unclear if mental performance can consistently be boosted with the same physical exertions as the system user becomes increasingly experienced. In addition, physical activity has not been definitively found to benefit task aspects which are more complex, such as problem-solving and response selection. ^28,30 Designers may wish to hold off on specifically incorporating physical tasks into the system in an effort to better cognitive performance. They may, however, feel less apprehensive about physical effort, especially if the activity is light and not prolonged.

The effects of high workload

Wickens and colleagues ²¹ presented four strategies that system users may adopt when too much workload is experienced:

System performance may simply be allowed to degrade without any coping strategies.

The degradation of system performance when workload exceeds capacity has been reflected in studies of human–robot systems. For example, heightened workload has been linked to longer mission completion times, increased number of collisions and supervisor errors in a multiple mobile robot deployment study. ¹⁷ In the study where robots were deployed to search for victims in an urban search and rescue scenario, the number of victims has been found to decrease as the workload increased. ³¹

The causes of workload in multiple-robot supervision

A number of factors have been suggested to affect supervisor workload within the domain of human-supervised robot deployment. These include task complexity, interface design and the levels of available autonomy. However, when a single human supervises a group of robots, it is the size of that group (or group size) that typically has the largest effect on workload. The following paragraphs discuss these factors.

First, in a study on multiple UAV deployment, high levels of task complexity (due to the processing of more targets) have been noted to increase supervisor workload. ³² Poor interface design can also contribute to increased workload. Chen et al. ¹⁹ suggested that in terms of the supervisory control of multiple robots, the effectiveness and usability of the interface can affect supervisor workload and system performance. For example, a study on soldier–robotic swarm interactions suggested that use of multimodal (visual, auditory and tactile) display techniques influences the effectiveness and efficiency of supervisors. ³³

The level of autonomy available in the robotic system can affect supervisor workload as well. ^2,34 Automation is often employed to address the problem of high workload. The use of automation to aid the human supervisor during multiple-robot deployment has been reported by Lif et al. ⁵ who provided some anecdotal evidence that participants in their experiment found robot autonomy useful. Wickens et al. ²¹ pointed out that the appeal of automation in high-time stress situations or in settings where there is a need to simultaneously handle multiple tasks is its potential in reducing workload.

The focus amongst researchers in terms of contributors to workload is the number of robots that a single-human supervisor can simultaneously deploy. ³² Studies which have attempted to relate group size to supervisor workload are numerous. For instance, Ruff et al. ³⁵ examined the interaction between a human supervisor and up to four UAVs. They found that as more UAVs are deployed, perceived workload increases as well. Adams ¹⁷ presented a study in which participants deployed one, two and four mobile robots in a task to transport materials and saw that workload was significantly higher when four robots were deployed. In another effort, Trouvain and Wolf ²⁰ examined workload as participants were tasked to deploy two, four and eight robots in separate runs to inspect areas within a given map. They found that workload increased significantly when participants deployed eight robots. Furthermore, the rise in workload when group size increases is expected to be exponential especially when it is necessary for the human to coordinate robot activities. ³¹

These studies are just a few examples of the works comparing workload levels as the human supervisor is tasked to deploy various numbers of robots. Many other research efforts ^{15,18,31,32,36,37} have reported similar results to the ones presented above, reinforcing the link between increased group size and heightened workload. Given that workload can only be increased up to a limit, it becomes evident that only a limited number of robots can be deployed by a single human supervisor. ²⁶ The link between increased group size and added workload also suggests that solutions are required to address the desire to deploy more robots to boost system capabilities while preventing the workload demands induced by each robot from depleting the human supervisor’s cognitive resources.

The second section has introduced the concept of workload, pointing out that it consists of a mental and physical component. The section has also listed the behaviours that system users may adopt when faced with high-workload situations. These behaviours may not necessarily be elegant coping mechanisms and system performance will likely deteriorate should workload remain high. To help users manage workload, system designers should consider aspects such as task complexity, interface design and the available levels of autonomy in the system. More importantly, however, they should limit group size as its role in affecting workload is significant.

Defining situation awareness

Together with workload, situation awareness is also critical in influencing the ratio of human supervisors to robots. ³⁸ According to Endsley, ³⁹ situation awareness consists of three levels: perception, comprehension and projection. Cues are first perceived with human senses. These cues are used to comprehend the situation. This understanding of the situation is then employed to project what will happen in the near future.

Endsley’s definition of situation awareness is highly popular amongst researchers and is suitable for various applications. However, it does not point specifically to what the human supervisors of robots should be aware of. Drury et al. ²² presented a framework to define awareness for human–robot systems. The framework consists of five types of awareness, but only two are directly related to awareness required by the single human supervisor of multiple robots. The other three types of awareness include awareness associated with human and human collaboration, robot awareness of human commands and awareness that robots require to coordinate amongst themselves. The first of these two related types of awareness is human awareness of the robots. Drury and colleagues suggest that this awareness refers to knowledge of robot locations, identities, activities, statuses and surroundings. The second type of awareness is the knowledge required for the overall mission. This refers to the understanding of the goals of the mission as well as the progress made towards the goals. The framework presented by Drury and colleagues helps define situation awareness within the multiple-robot deployment and supervision domain.

The effects of poor situation awareness

Diminished situation awareness can result in a few problems such as poor decision-making, errors as well as slow response to the situation. The paragraphs below discusses these possible effects.

Situation awareness has been identified as ‘the major factor driving the quality of the decision process’. ⁴⁰ The comprehension of a particular situation is the basis of an effective choice. ⁴¹ Its absence can result in poor decision-making and subsequently, erroneous task execution. The study by Yanco et al. ²³ on four teams participating in a rescue robotics competition provides an example. One of the human supervisors in that event was found to have had poor awareness of the robot’s orientation, resulting in more robot collisions. The errors were due to the lack of awareness about the robot’s on-board camera’s orientation. Unbeknownst to the operator, the camera had been pointed off the centre. The supervisor’s lack of awareness resulted in wastage of more than half the team’s runtime and also fewer victims found. At the same event, the inability of another robot to detect the presence of Plexiglass resulted in poor situation awareness for the robot’s supervisor. This resulted in the robot being commanded to charge through the Plexiglass panel.

The possibility for errors is especially high when workload exceeds a critical level. ⁴⁰ Endsley and Jones proposed that this is because the supervisor can only attend to a portion of the pertinent information. Alternatively, it may also be due to flawed or imperfect information perception and assimilation. Measures are necessary to prevent cognitive workload from exceeding the critical level. Errors have also been known to be closely related to stress, ²¹ which itself is a contributor to diminished situation awareness. The link between stress and impaired situation awareness is presented later in the section on the causes of poor situation awareness.

In their discussion of the situation-awareness requirements for deploying multiple collaborating robots, Riley and Endsley ¹⁴ pointed out that the lack of situation awareness can also retard intervention response. System problems may be less quickly detected and remedied as system users use more time to become mentally realigned with the situation to establish the nature of the problem. Burke and Murphy ⁴² reported that operators with good situation awareness were nine times more likely to spot victims during an urban search and rescue (USAR) task.

The causes of poor situation awareness

There are many factors that can contribute to decreased situation awareness. Endsley ³⁹ suggested that such factors can include interface design, system capability, stress, workload, complexity and automation. In addition, Jones and Endsley ⁴³ found that factors like inadequate mental models and inexperience can hinder situation awareness. This suggests that a lack of adequate training is also detrimental. The above-mentioned factors are discussed here. Some of these factors are interrelated and so these correlations are also pointed out within this discussion.

Using automation to address workload and the loss of situation awareness

The use of automation can mitigate workload by performing certain functions for humans. ^21,53 Traditionally, it is thought that the more automation is used, the lower is the level of workload. ³⁴ This may not necessarily imply that automation is good only for replacing the human entirely or that automation is used only for jobs that humans cannot perform at all. Rather, automation can serve to assist (instead of replace) humans and also to address tasks that humans can do well only at the expense of increased workload. ²¹ Squire and Parasuraman ¹⁵ suggested that supervising multiple robots will ‘consequently increase the cognitive workload demands on operators and automation will be needed to support effective and safe human-system performance’.

Implementing automation is thought to be a commonly used approach to allow the deployment of greater numbers of robots without overloading the supervisor’s attention resources. ⁵⁴ This is because the supervisor is relieved of mentally processing aspects of the task handled by the automation. Automation also replaces the supervisor’s need to select the relevant response for the robot, ² reducing the number of physical actions that must be performed (i.e. the physical component of workload). If the number of responses that the supervisor has to perform is reduced, then it could be possible to increase the number of robots that can be supervised by a single human. ¹⁷

Workload can also be reduced with other methods besides the removal of some aspects of mental processing or response requirements from the human. A multiple-resource model to mental processing ²¹ suggests that perceptual cues related to task execution may be provided along more than one sensory modality or channel. In such a manner, information can be processed mentally in parallel rather than in a serial manner when information is restricted to being presented along only one sensory modality. An example of information presentation with multiple-sensory modalities can be found in military aviation. Successful missile lock on an enemy target is indicated to the pilot via an audio tone in the cockpit. That event is thus communicated to the pilot via the auditory channel, leaving the visual channel free to process other aspects of flying the aircraft.

Haptic displays are also relevant when considering the multiple-resource theory. Rösch et al. ⁵⁵ presented a set-up for using a force-feedback joystick to control a mobile robot via the Internet. The joystick vibrated according to the magnitude of the force acting on the robot’s wheels, giving the remote operator information regarding the resistance faced by the robot. The multiple-resource model has been tested within the domain of multiple-robot supervision. An experiment conducted by Dixon et al. ⁵⁶ showed that participants in a dual-UAV supervision task experienced less workload when information was presented via the auditory channel in addition to the typical visual sensory channel. However, it is important to note that cues presented in parallel even via different sensory modalities may not share attention resources perfectly. Tasks which are high in complexity can negate the benefits touted by the multiple-resource model. ²¹ Furthermore, the selection of responses to two competing tasks occurs only serially. The response to one of these competing tasks is inevitably delayed, but the selection of a response may still occur in parallel with perception. ⁵⁷

The reduction of workload through automation use can also assist in mitigating workload. As described earlier in the section on the effects of poor situation awareness, excessive workload can limit situation awareness. Therefore, the prevention of excessive workload through automation can assist also with mitigating losses in situation awareness. There are, however, also automation techniques more specifically targeted at achieving good situation awareness.

For instance, Calhoun et al. ⁵⁸ introduced automation to enhance the human supervisor’s situation awareness of the mission space while deploying multiple UAVs. Their effort produced automation to address the problem of disorientation when the human switches from interacting with one UAV to another. The resulting change in UAV camera imagery may cause confusion regarding the newly selected UAV’s position and heading. What Calhoun and colleagues did to address such confusion was to display the disengagement of interaction from one UAV by zooming out from a computer animation of the UAV. This would create a view in the animation, showing the deselected UAV’s position in the mission space. The animation would then zoom in towards the newly selected UAV, informing the human about this UAV’s position and heading relative to the previously selected UAV and to the mission space. This animation aid was found to have assisted in enhancing spatial awareness and reduced the time needed by experiment participants to acquire desired views of the ground.

Another example of automation aimed at boosting situation awareness sought to support airspace monitoring involving multiple combat UAVs. In that effort, Fern and Shively ⁵⁹ found that a graphical map overlay on the tactical situation display significantly assisted in promoting situation awareness of the airspace compared to a text-based Internet chat room style interaction.

Model-based approaches to automated evaluation of human–robot systems

This review of relevant work on human–robot interaction has so far focused on empirical studies. However, computational studies also contribute to understanding human–robot systems by revealing subtle interactions between factors that may not be as easily detected with empirical techniques. Some of these computational studies have been geared towards the use of mathematical models. These models may be proposed based on the understanding of operator behaviour and making some assumptions about interaction episodes. It is also possible to derive models based on sets of data recorded from empirical studies. The resultant model (either proposed or derived) can then be used to evaluate the correctness of operator actions as well as the performance of the human–robot system.

An early example of a proposed model is that by Crandall et al. ⁶⁰ for predicting the number of robots that a single human can supervise. Crandall’s team termed this number ‘fan-out’ and has claimed that it depends on neglect time (NT) and interaction time (IT). NT is defined as ‘the expected amount of time that a robot can be ignored before its performance drops below a threshold’ while IT means ‘the expected amount of time that a human must interact with a robot to bring it to peak performance’. The model assumed that the human interacted with the robots one at a time in sequence.

Breslow et al. ⁶¹ extended upon this model by considering the possibility of a highly dynamic mission environment. They postulated that instances where multiple robots will simultaneously require the human’s attention are likely. This leads to temporary overloading and can result in erroneous response from the human. The proposed model from Breslow’s team considers the available time which the human has for remedying a robot’s problem, the IT needed to make the necessary corrective actions, the time taken to realize that a particular robot requires attention as well as the time spent on manual operations not directly related to the robot in focus. The experiment conducted showed that their model could accurately predict instances where multiple robots would suddenly require interaction with the human. This model may thus be a useful tool for providing early notification of converging attention demands.

Another example of the model-based approach is the study by Boussemart and Cummings. ⁶² The resulting model was derived from the data recorded from an earlier empirical study. The model used hidden semi-Markov models (HSMM) to generate a predictor of operator actions in the domain of human-supervised automated systems. The HSMM predictor is also able to model the duration of a certain behaviour, thereby additionally evaluating the timeliness of operator actions. The system would score poorly in terms of model accuracy if consecutive operator actions and the timeliness of those actions do not match well with the predicted response. Consequently, suitable automation can be invoked to alert or assist the human when there is a mismatch between anticipated and actual responses.

Such system models can represent individual preferences and nuances so that automation can even be designed with the ability to compensate for individual weaknesses. If the design of automated system software also considers the models for operator behaviour, then such software may achieve validation from users more readily. ⁶³

Still, it should be noted that such computational models may not yet represent the full myriad of (beneficial and improper) operator responses. It is thus possible for a model to score some responses poorly even when those responses are appropriate. The onus to ensure that operator actions are correct should not at this time, fall solely upon such models.

When comparing model-based approaches with empirical methods in human–robot interaction, the latter appears to be more common. This is likely due to the difficulty in producing an accurate and robust model of human interaction with automation. However, in addition to attempting to model the deployment on a macro scale, the modelling approach can also be applied to specific techniques such as that by Calhoun et al. ⁵⁸ (described in the section on using automation to address workload and the loss of situation awareness). The automation aid by Calhoun’s team may then be implemented to occur at different speeds according to the model of a particular user’s interaction style for instance.

Issues with automation use

Despite the potential of automation for reliving workload, supporting situation awareness and evaluating operator actions during runtime, it can sometimes instead impose workload and degrade situation awareness. When it is improperly applied, automation can, in particular, result in diminished situation awareness. ⁴⁰ Highly automated systems can also cause problems such as the reduction of user vigilance, ¹⁴ complacency, ^64,65 being out of the loop ⁴⁵ as well as skill loss or degradation. ¹⁹

The decrease in user vigilance that stems from automation-induced prolonged inactivity can result in users being disinterested in task monitoring. Events and changes go undetected because of a lack of direct task control. ⁶⁴ This relationship lends further support to the work mentioned in subsection on mental and physical workload which reported improved cognitive performance that accompanies low to medium physical workload.

Complacency or overreliance results from a sense of overtrust in the automation’s ability to perform the task flawlessly. Users of highly reliable automation may be prone to being complacent. ⁶⁴ Yet, the failure of such reliable systems is always possible. According to Wickens et al., ²¹ ‘an operator who expects that automation is doing its job will be less likely to monitor the job it is doing – losing awareness of the evolving state of, or surrounding, the automated system’. A study by Chen and Joyner ⁶⁶ found that participants in a target detection task relied too much on an automatic aid for target detection, scoring more poorly than when the automatic target recognition feature was unavailable.

When automated systems are working, they may inhibit users from practicing the skills required to perform their tasks manually. This results in skill degradation over time and the inability to apply the required know-how when automation fails. This know-how may include the procedure to efficiently call up system information and acquire situation awareness. When users lack the skills to address automation failures, the consequences can be more severe than if no automation had been available in the first place. ⁶⁷

In the domain of USAR, the out-of-the-loop syndrome is, in particular, one of the most pertinent effects of situation-awareness reduction that stems from automation use. For example, Riley and Strater ⁴⁴ found that significantly higher level 3 situation-awareness scores were seen when participants deployed two robots using only teleoperation than when the robots were deployed with more autonomy. They explained that being out of the loop when more autonomy was used, may have lowered supervisor attention for the robots. This in turn degraded perception and comprehension necessary to select suitable robot actions. Case in point, automating the robot’s navigation such that the human is monitoring robot movements instead of driving the robot remotely, can diminish awareness about where the robot has been or how it arrived at the current location. ⁵⁰ Even if the human supervisor remains vigilant throughout the deployment, sufficiently high levels of automation can hide the cause of some autonomous behaviours. ¹⁹ This leaves the supervisor confused when the robotic system exhibits unexpected behaviour. ⁶⁸

The effects of the out-of-the-loop syndrome are thought to be at their most extreme when system designers implement automation to allow technology to handle as much of the system’s objective(s) as possible. Kaber et al. ⁵³ have noted that designers of many robotic systems tended to adopt the belief that maximum autonomy is the most appealing approach. Such designers do not realize that this kind of implementation can render human intervention impossible and demotes the role of the human to a mere observer instead of a supervisor or an operator. ² Furthermore, Kaber and Endsley ⁶⁹ recommended that human–machine systems employed for dynamic applications will benefit from periods of manual control (i.e. temporarily scaling the level of autonomy to no autonomy at all). Such instances of manual control may assist in preventing skill degradation.

Perhaps another method for combating complacency and subsequently being out-of-the-loop is the use of psychophysiological measures ⁷⁰ such as eye tracking to ascertain the level of human engagement with the task. In the event that the psychophysiological data suggests that the human’s level of engagement is below an acceptable threshold, the system can provide prompts or scale autonomy to a lower level to encourage re-engagement.

Maximizing autonomy may not even be worth the engineering effort. Wilson and Neal ⁷¹ conducted an experiment to compare the effort of implementing levels of autonomy with increases in efficiency. The experiment involved tasking a human to act as a ‘shepard’ to direct the robotic ‘sheepdog’ to herd a robotic ‘sheep’. Wilson and Neal claimed that the reduction in human interaction with the sheepdog did not warrant the effort in developing very high levels of autonomy.

Sliding autonomy

An increasingly popular method for applying autonomy in supervised robotic systems is sliding autonomy. Sheridan and Verplank ⁷² have proposed a spectrum of autonomy based on 10 possible autonomy levels. These autonomy levels begin from the lowest extreme where the human performs the whole task manually (level 1), to autonomy at a middle level where the system executes an action upon human approval (level 5), to the highest autonomy level where the automation handles the whole task (level 10). A human–machine system can be designed such that its level of autonomy is fixed. Such fixed autonomy is also referred to as static autonomy. With sliding autonomy, however, the system’s user can dynamically adjust its level of autonomy according to the needs of the situation. ⁷³

Parasuraman et al. ⁷⁴ compared the use of sliding autonomy, static autonomy and no autonomy in an experiment involving the supervision of unmanned ground and aerial vehicles. In the static autonomy condition, the UAV constantly assisted the human with an automatic target recognition function while that function was only invoked in the sliding autonomy condition when system performance fell below a set threshold. Their study showed that sliding autonomy assisted in promoting situation awareness and was also the most helpful condition for detecting changes and mitigating workload.

A benefit of such adjustable autonomy is that a high degree of automation can be present in the system to assist the human in coping with high-workload situations. However, that level of automation can be scaled back when possible, for the human to re-engage with the task. Such scaling back of autonomy and re-engagement of the human with the task may lower the risk of the human becoming out of the loop. ⁷⁰

Sliding autonomy is useful for supervised robot deployment because there will very likely be situations where a robot’s on-board intelligence or sensors are insufficient to handle the task at hand. ⁷⁵ Such instances are ideal for the scaling back of robot autonomy to allow for manual control of the robot(s) by the human. Take, for instance, navigation by a ground robot through an area with many large holes that the robot can fall into. The deployed robot may be equipped to detect obstacles visible from the ground up but not depressions on the ground. The human, at this point, can suspend autonomous robot navigation and select to teleoperate the robot through the challenging terrain.

Regardless of robotic system’s intended application, designing for the system’s autonomy to be dynamically adjustable is advocated because it can be difficult if not impossible to anticipate failure of the system’s automation. ⁷⁵ Should the function for the scaling back of autonomy still be available despite the failure of higher level autonomy features, the robot can be saved from incapacitation. It may even be possible to continue with the mission, albeit possibly with some handicap due to the loss of high-level autonomy features.

Deployments of multiple robots can benefit from sliding autonomy as the level of autonomy on all the robots do not have to be the same. For instance, the human may lower the autonomy on one robot individual to teleoperate it out of an impasse, while the rest of the robots explore the mission environment autonomously.

It is worth noting that while it is beneficial for autonomy levels to be adjustable, the levels of autonomy which are available within the system could be much more important. Kaber and Endsley ⁶⁹ described their study on the effects of static autonomy levels and automation allocation cycle time (AACT). They defined AACT as a ratio between the duration in which a preset level of autonomy is available and the duration where autonomy was not. To ascertain their effects in a simulated multitask scenario, the two factors were examined independently and in conjunction with each other. Kaber and Endsley noted that the batch processing level of autonomy may facilitate good system performance, improve situation awareness and moderate workload in static autonomy systems. Under that level of automation, the options to be performed are generated and selected by the human while the actual implementation is done by the automation. However, that level of autonomy did not offer the best combination with an AACT that has also enabled optimal results; that is, the benefits are not ‘additive’. Thus, Kaber and Endsley suggest that the identification of the appropriate autonomy level is crucial even for systems capable of sliding autonomy.

What, how and when to automate

The designer of a robot system or any system involving automation will have to consider what aspects of the task to automate, how automation can be implemented to assist and also when the installed automation feature should be invoked. This section explains the importance of these concerns and describes them in greater detail.

When to automate

It is also important to recognize the opportunities that occur during robot deployment that automation can be invoked. The circumstances behind these opportunities can thus become automation triggers. This article uses the second part of Feight et al.’s ⁷⁹ framework to present when automation can be activated. Five types of automation triggers are possible: (1) operator based, (2) system based, (3) environment based, (4) task based and mission based and (5) spatiotemporal.

First, a trigger may be classed as operator based because of the user’s activation of the automation. It is also considered operator based if automation is initiated due to the system’s recognition of the operator’s performance or workload. Byrne and Parasuraman ⁷⁰ pointed out that system evaluation of operator workload is possible via psychophysiological measurements such as electroencephalogram, heart rate variability and palmar skin conductance. However, such measures should be used in conjunction with other methods like those for assessing operator performance as environmental factors and individual tendencies can affect psychophysiological data.

The second type of automation activation is system-based triggers. With this type of trigger, automation is applied by the system because of certain system states or modes. For instance, if a robot within a multiple-robot system becomes incapacitated, the remaining robots can coordinate amongst themselves to formulate a new plan which takes into account the system’s new handicapped state.

Third, triggers can be environment based because they are brought on by changes like surrounding light, temperature or humidity levels. Environment-based triggers also include events that occur within the system’s workspace.

Next, triggers can be task or mission based if automation is engaged because the mission has reached a particular phase. For instance, the return-to-deployment coordinates behaviour on multiple-robot system can also be triggered once the robots have completed their mission objectives. The fifth type of trigger in the framework of Feigh et al. is that of the spatiotemporal variety. Such triggers pertain to automation state that is altered because of changes in the system’s location or simply because the time calls for it.

Within the system, multiple triggers can be present. The activation of each trigger does not have to be simplistic but can depend on the status of other triggers as well. In a system of multiple autonomously coordinating robots, for instance, an incapacitated robot may serve as the criteria to trigger an automatic response from other robots to take over its unfinished task. If, however, the trigger for ‘return to base’ has been set, then remaining robots can ignore the trigger to assist the incapacitated individual.

The considerations for what, how and when to automate have been summarized in Table 1.

OPEN IN VIEWER

Table 1.

A summary of what, how and when to automate.

What, How and When to Automate
WHAT	HOW	WHEN
1. Information acquisition 2. Information analysis 3. Decision selection 4. Action implementation	1. Modification of function allocation 2. Modification of task scheduling 3. Modification of interaction 4. Modification of content	1. Operator-based triggers 2. System-based triggers 3. Environment-based triggers 4. Task- and mission-based triggers 5. Spatiotemporal triggers

This section has introduced the application of automation to decrease workload and assist in acquiring situation awareness via empirical techniques as well as model-based approaches. It has also presented possible issues such as decreased user vigilance and being out of the loop when automation is misused. To avoid such potential pitfalls, the designer may consider imbuing the system with the ability for adjustable autonomy. The designer must also be aware of the types or classes of automation available, how such automation can be implemented and when during runtime the implemented automation should be invoked.