Perspectives on the system-level design of a safe autonomous driving stack

Driving is an intrinsically multi-agent activity, even though the focus of attention is often on the task of the single agent under one’s control. Even a seemingly standalone task, such as automated lane-keeping1

OPEN IN VIEWER

Fig. 2.

A simulation of an automated lane keeping system (ALKS) scenario.

OPEN IN VIEWER

Fig. 3.

Five’s physical self-driving platform traversing a complex London intersection.

Ideas from control theory have a played a central role in the design of automated agents in transportation systems [13], motivated by the need for robust and stable operation in the face of uncertainty [30] and scenario variation. As autonomous vehicles come to be deployed in dynamic environments and situations requiring tight interaction with other intelligent agents, we find that they can push the limits and assumptions that normally underpin classical methods of policy synthesis, such as trajectory optimisation- or search-based approaches. In current practice, this challenge is being attacked from many different perspectives, ranging from the addition of special case treatment and switching logic within a modular sense-plan-act architecture [10,28] to adopting fully end-to-end learning architectures [5]. Our philosophy is to seek a balance between the use of state-of-the-art learning-enabled components along with architectures that allow for safety considerations to be built in by-design.

This approach reflects the subtlety of the currently existing specifications of the task of driving and the gaps inherent in such specifications, which were originally designed for human interpretation. Firstly, there are often exceptions to stated rules, context-dependent conflicting rules, and guidance of an ‘open nature’, all requiring interpretation in context. For instance, Rule 163 of the UK Highway Code [1] states that after starting an overtaking manoeuvre one should “move back to the left2

Approaches such as Responsibility-Sensitive Safety (RSS) [27] address similar concerns by defining weak requirements from first principles, such as from considerations of vehicle physics and basic visibility limits. While this is aligned with our thinking, we seek more expressivity in the requirements and a more direct integration between this and policy synthesis. Therefore, we employ a multi-level policy synthesis architecture and we combine multiple optimisation methods each tuned to different specifications but together achieving the objectives of safety-by-design. Likewise, even within just the prediction modules, our architectures are designed both to be interpretable to the design and testing teams, as well as amenable to verification with respect to stated run-time performance requirements. Lastly, making such structure explicit allows for the possibility of incorporating other interaction mechanisms, such as game theoretic schemes, to model interactive phenomena, e.g. altruistic yielding in tight merging scenarios [9].

Safety is sometimes viewed within the Artificial Intelligence community as only a matter of achieving high reliability (e.g., overcoming adversarial examples). However, as Nancy Leveson and others have argued [17], reliability is neither the only necessary condition nor a sufficient condition for safety. Instead, one needs a system-theoretic approach to understanding hazards and overcoming issues arising in the ‘outer-loops’. In practice, when one considers the system components being discussed above, this calls for scenario-based designs and evaluations in the context of careful analysis of Operational Design Domains (ODDs) and their characteristics. So, going beyond mere risk analysis of individual components, we seek to understand system-level behaviours emerging from realistic forms of component-level errors and performance degradation, and the interaction thereof.

Performing this analysis through extensive real-world testing is a seemingly impossible task due to the enormous size of the space defined by variables that needs to be covered in testing, the long tail of rare events that are the most crucial to be analysed, and the cost attached to all of that even for a limited ODD. Moreover, it is becoming increasingly accepted that disengagement statistics alone may be a poor descriptor for system safety and trustworthiness [16]. This calls for employing – as a crucial part of the development process – large-scale high-fidelity simulation in which the most important variations of a scenario could be synthesised for deeper analysis.

The scale at which high-fidelity simulation needs to be employed could make traditional approaches difficult and computationally costly: simulating the world in sufficient detail (e.g. to capture all the visual light-reflective and radar-reflective properties) followed by accurate simulation of all the sensor characteristics (e.g., RGB cameras, LIDAR, RADAR). On the other hand, the alternative of creating a simplified simulation environment specifically for testing prediction and planning at the level of objects on a map fails to capture the subtlety of the real world and the physical platform. Thus, we explore Perception Error Models (PEMs) that allow for simulations to be well calibrated to the real world conditions in which the system will subsequently be validated without the need to simulate at the level of light transport and sensor physics, hence focusing attention of the entire safety analysis process on the relevant parts of the design space which may be the most sensitive.

This speaks to how one evaluates the system as a whole. On the one hand, simulation-based testing would always be limited by issues of realism, while other formal approaches to verification and validation can suffer from scaling issues in open environments. So, developing a system-level methodology for large-scale simulation-based development work flow is a crucial area of ongoing development.

1.4. Paper organisation

The structure of the remainder of this paper is to summarise a few representative published works from Five, focusing attention specifically on motion planning and prediction issues and their overlap with perception models, mainly because it is here that the multi-agent nature of the problem becomes most salient. An exciting area of research not covered here is to further investigate advances in multi-agent aspects of other autonomous driving problems, e.g. active perception especially when informed by interaction, federated learning and ad hoc interaction at a higher level.

The rest of this paper is organised as follows:

In Section 2.1, we describe a general approach to imposing safety specifications into planning, and propose an efficient implementation based on this. In particular, we discuss the following two papers:

2.1. Safe planning

Instead of attempting to post-validate the safety of a planning system that does not have safety built-in, e.g. a manually-tuned rule-based system, a better approach is to directly consider safety specifications at design time, such as by employing a constrained optimisation scheme with safety constraints. For autonomous driving, safe and assertive driving style could be captured with hard constraints that delimit the domain in which driving is feasible (e.g. kinematic feasibility) and safe (e.g. avoiding the road boundaries or colliding with other road users); and soft constraints that capture desirable qualities of driving (e.g. continuous progress to the target, and having low jerk in both acceleration and steering). Moreover, different contexts suggest additional constraints that capture the subtlety of interaction and expected behaviour in these particular contexts.

With nonlinear dynamics and non-convex constraints, a natural formulation of this planning problem is with constrained Nonlinear Programming (NLP), usually implemented with a receding horizon. Here, hard constraints get introduced into the optimisation objective function through Lagrange multiplier terms (for equality constraints) and KKT conditions (for inequality constraints), and the gradient of this augmented objective function is relied upon to lead to convergence to a solution. However, it is well known for this kind of formalisation that convergence is uncertain and sensitive to initialisation, and solutions tend to be only locally-optimal [25].

2.1.1. Improving convergence rate and solution quality

In [8] we warm-start the NLP problem by the solution of an ‘easier’ approximation of the planning problem formulated as a Mixed-Integer Linear Programming (MILP) problem with receding horizon. This is achieved by linearising and simplifying the dynamics and constraints, reformulating the objective function, then solving (almost) optimally using Branch and Bound. The solution of this MILP problem is then used as a starting point for the NLP solver. This initialisation stage along with a one-shot NLP stage leads to a two-stage optimisation framework (2s-OPT) as shown in Fig. 4 that results in improved convergence, faster NLP solving times, and higher solution quality [8].

OPEN IN VIEWER

Fig. 4.

2s-OPT architecture. The first stage solves a mixed-integer linear programming formulation of the planning problem with receding horizon, which solution warm-starts a nonlinear programming solver. ©[2022] IEEE. Reprinted, with permission, from [8].

This multi-level optimisation scheme also allows us to address complex road rules that admit formal description, such as in Signal Temporal Logic (STL). As described in [21], the satisfaction of STL formulae can be encoded as constraints in a MILP formulation, allowing frameworks such as the above to accommodate encodings of road rules and specifications from the highway code.

2.1.2. Improving runtime efficiency

While our two-stage optimisation framework produces high quality solutions, it suffers from the non-trivial additional time required by the solver of the first optimisation stage to converge. This trade-off between quality and runtime efficiency could be accommodated in certain environments but would bar the applicability of the framework to high-speed, interactive settings.

To address this issue, we ‘distil’ the performance of the 2s-OPT planner offline with a deep neural network, such that the network learns to produce similar outputs to the two-stage optimiser for the same input, and augment that with a light-weight runtime optimisation stage, resulting in the PILOT framework [20]. To that end, we use imitation learning where 2s-OPT acts as the expert. We employ DAgger [23] as a data augmentation scheme and run the learning loop using the CARLA simulator [7]. The neural network has convolutional layers that consume visual representations of the driving scene (and how it is predicted to evolve over the planning horizon), in addition to feed-forward layers that take into account other scalar quantities. The desired output of the network is a smooth, feasible trajectory for the full planning horizon.

At runtime, in order to retain some of the guarantees with respect to specifications, we use the output of the network to warm-start an efficient NLP stage. While this hybrid solution would not guarantee the satisfaction of all the specification at all times (that is, if the NLP stage fails to converge within the allocated time budget as discussed earlier), the quality of the initialisation produced by the deep network along with the high-frequency planning allowed by the improved runtime performance lead to very encouraging empirical results [20]. Figure 5 shows the training and runtime pipelines of PILOT.

OPEN IN VIEWER

Fig. 5.

PILOT training and runtime pipelines. At the training phase, 2s-OPT is used as the expert to train an imitation deep neural network. At runtime, the output of the network is a seed to a nonlinear programming optimisation stage that rectifies the network’s output and establishes some of the guarantees. ©[2021] IEEE. Reprinted, with permission, from [20].

Further performance gains could be attained at the neural network training phase through adversarial techniques, e.g. using robust adversarial training [4].

While interpretability by itself is not a sufficient condition for safe prediction systems, it is a necessary requirement and a stepping stone towards a less opaque and more introspective stack. Thus, we developed a number of interpretable, model-based prediction systems which we describe next.

We developed an integrated goal recognition and motion planning system, called Interpretable Goal-based Prediction and Planning (IGP2) [2]. IGP2 infers posterior probabilities of possible targets extracted from a map and corresponding trajectories of other vehicles using rational inverse planning. The goal and trajectory predictions inform a sampling-based motion planner employing Monte Carlo Tree Search (MCTS), which repeatedly forward-simulates the present states to determine the best sequence of actions for the ego vehicle. By modelling the actions of all vehicles using high-level manoeuvres and macro actions, IGP2 generates manoeuvre plans over extended horizons for which we can extract intuitive explanations based on rationality principles. We tested IGP2 in diverse simulated urban driving scenarios, demonstrating its ability to robustly discover goals and driving intentions of nearby vehicles and exploit this information to generate efficient driving plans for the ego vehicle [2]. An open-source implementation of IGP2 has been released.3

We extended IGP2 by adding the ability to infer the presence of occluded objects. Our approach, Goal and Occluded Factor Inference (GOFI) [12], jointly models the probability of existence of occluded objects along with the goals of other vehicles. If an observed vehicle’s behaviour seems rational under the hypothesis of an occluded object being present but seems irrational otherwise, GOFI uses this information to increase its belief that an occluded object is present. Like IGP2, GOFI uses the inferred goal and occluded object probabilities to inform an MCTS planning procedure. In a range of simulated driving scenarios, we showed that GOFI is able to reduce the number of collisions with occluded objects or other vehicles [12].

Towards verifiable prediction for autonomous vehicles, we developed a goal recognition method called Goal Recognition with Interpretable Trees (GRIT) [6]. Our agenda in this work was to develop a method that is fast, accurate, interpretable, and formally verifiable against specified properties. Unlike previous methods, GRIT is the first method which satisfies all four of these objectives. GRIT infers posterior probabilities over the goals of vehicles using decision trees which are learned from vehicle-trajectory data. We evaluated GRIT across four urban driving scenarios from two vehicle-trajectory data sets. Our experiments showed that GRIT is computationally fast and achieved similar goal recognition accuracy to deep learning based methods and higher accuracy than several other baselines [6]. We also showed that GRIT produces trees which are human-interpretable and which can be formally verified against specified properties by mapping the trees into propositional logic and applying Satisfiability Modulo Theory (SMT) solvers [6].

Based on our goal recognition and motion planning system IGP2, we ask whether we can generate intuitive dialogue-oriented explanations automatically in a way that builds transparency and trust in our system. IGP2 is particularly well-suited for this task as it was designed from the start with interpretability in mind. We are working towards building an explanation generation system on top of IGP2 that can automatically create intuitive explanations of vehicle decisions in response to natural language explanation queries by human users. A recent prototype and results were presented in [11].

Finally, considering the requirements of a safety-critical and real-time motion planning system, we developed a modular, light-weight motion prediction system called Flash [3]. Building on top of inverse planning and goal recognition, Flash has a hybrid architecture that involves both analytically-modelled and data-driven components. This system is designed to exhibit multiple properties of interest: multi-modality, predicted spatial uncertainty, accuracy and computational efficiency. In contrast to many end-to-end regression-based systems, Flash’s control-based trajectory generator provides guarantees that kinematic and motion profile constraints are respected, and its modularity facilitates component-focused improvements. In [3], we performed a detailed analysis of the system components with experiments stratifying the data based on behaviours, such as change lane versus follow lane. We demonstrated state-of-the-art performance in terms of trajectory error, especially for longer horizons that are crucial for motion planning. We also showed the ability to isolate performance factors by analysing the impact of the motion profile prediction component and by interpreting the goal recognition outputs.

While prediction and planning systems normally assume the correctness of their input (the output of perception modules), no real world sensor systems or real world perception systems are able to produce noise-free output. Understanding the limits of perception systems in a systematic way 1) facilitates developing prediction and planning systems that can handle the noisy real world, and can influence the design of the perception system through its impact on the downstream task of planning [18]; and 2) makes simulations more realistic, closing the gap between simulation and reality.

2.3.1. Perception error models

For complex, real-world perception tasks, e.g. compute-intensive object detectors, modern perception pipelines often involve large deep learning models and process quite high-dimensional data [15]. Characterising the emergent error behaviour of an autonomous driving system under test that employs such pipelines usually requires large-scale simulation testing, and hence requires computationally-expensive sensor rendering in addition to the cost of running these pipelines, with quality dependant on the realism of the simulator.

As an alternative, perception error models (PEMs) serve as a surrogate for the compute-intensive components and pipelines of the task under test, and thereby enable efficient large-scale testing using simplified low-fidelity simulators without the computational cost of executing the expensive deep learning models [14]. Such perception error models could be created using engineering knowledge of the task [19], from real world data [29], or even using simulated data [24]. An example of a simulation pipeline with a PEM is shown in Fig. 6.

OPEN IN VIEWER

Fig. 6.

Block diagram of a simulator loop for a backbone task (e.g. LIDAR detector). The compute-intensive stages could be circumvented during end-to-end testing with a surrogate perception error model. The outputs of the surrogate model and the backbone task are fed into the model for the downstream task (e.g. a planner) in the same way because their outputs (e.g. detections) are elements of the same vector space. The simulator updates the world state based on the output of the downstream task (e.g. the new plan), which is used both by the expensive simulator to produce sensor readings (e.g. LIDAR point clouds) and by the efficient low-fidelity simulator to produce low-dimensional inputs (salient variables) required by the surrogate model.

We demonstrate the efficacy of PEMs in [24] through the reduced computational cost of an autonomous driving task in the CARLA simulator [7] when using trained, efficient surrogate models of state-of-the-art LIDAR detectors whilst maintaining the accuracy of the simulation as measured by planning metrics of the behaviour of the ego vehicle and metrics of the performance of the perception system. Figure 7 shows an example of using this model where the surrogate PEM model performs similarly to the actual LIDAR detector from the point of view of missing detections.

OPEN IN VIEWER

Fig. 7.

An example of a false negative vehicle detection that leads to a collision in a CARLA simulation. Ego is the black vehicle in the bottom. Detected objects are shown by red bounding boxes for the ground truth annotation, the LIDAR detector output, and the neural surrogate model output.