Stanford Seminar - Continual Safety Assurances for Learning-Enabled Robotic Systems

Introduction and Challenges of Safety in AI and Autonomy

• The long-term goal is to develop robot algorithms that operate with guaranteed safety and performance in new and uncertain environments, applicable to various fields such as autonomous drones, cars, and space exploration (30s).
• Machine learning and AI are becoming increasingly pervasive in autonomy stacks, particularly for perception, trajectory forecasting, planning, and control, due to their ability to capture real-world complexity (56s).
• However, the inclusion of machine learning has also introduced new safety challenges, such as the need for safety assurances in systems that operate in uncertain environments (1m21s).
• Recent incidents, such as Cruise cars being rolled back from San Francisco and robots crashing into humans on factory floors, have highlighted the severity of these safety challenges (1m28s).
• The US government has passed an executive order to prioritize discussion on AI safety, and other governments have followed suit, emphasizing the need for safety assurances in AI and autonomy (1m36s).
• There is a tension between enabling systems to leverage machine learning capabilities while maintaining safety, which is the focus of the discussion (1m48s).
• Most machine learning systems are designed without specific regard to safety, and safety issues are often addressed with post-hoc solutions, referred to as "safety bandages" (2m14s).
• These safety bandages are not scalable, can be conservative and degrade performance, and may not work in new deployment conditions (2m49s).
• To overcome these challenges, safety is viewed as a continuous process, formally ingrained in different stages of the learning process, from design and training to deployment and iterative improvement (3m26s).
• Algorithms are developed to programmatically incorporate safety requirements in the training process, learning inherently safe and robust controllers and policies for robotic systems, referred to as "design time safety methods" or "design for safety" (3m47s).
• The goal is to develop methods for detecting out-of-distribution and anomalous situations in learning-enabled robotic systems, adapting their behavior to maintain safety, and learning from past failures to improve safety over time (4m13s).
• A continual safety assurance framework is proposed, where assurances are provided during design time, monitored and adapted during operation time, and continuously improved over the system's life cycle (4m40s).
• The framework involves learning provably safe controllers from data, adapting these controllers online under new deployment conditions, and stress-testing policies or controllers to minimize safety-critical failures (5m2s).

Safety Analysis and Reachability Analysis

• Safety analysis aims to determine whether and how a robot can prevent its trajectory from entering an undesirable set of states, referred to as the failure set (5m39s).
• Two key aspects of safety analysis are quantifying which configurations of the robot are doomed to fail versus which are safe, and how to keep the robot in safe configurations, referred to as the "whether and how of safety" (5m55s).
• Control theory provides powerful frameworks for safety analysis, including Hamilton-Jacobi reachability analysis, which is used to mathematically characterize and automatically compute safety requirements (6m20s).
• Hamilton-Jacobi reachability analysis assumes a robotic system with dynamics, state, control, and disturbance, and computes the backward reachable tube, a set of initial states from which the robot will be driven to an undesirable set of states despite best control efforts (6m55s).
• The backward reachable tube represents the unsafe configurations for the robot and should be avoided, as illustrated by a simple example of a quadrotor moving longitudinally up and down in a room (7m41s).
• A system's trajectory can be analyzed to determine if it will eventually crash into a ceiling or floor, making it impossible to avoid collision, or if it can be controlled to stay within a safe set, with the contrast between these two scenarios represented by a light red region and a blue region, respectively (8m2s).
• The blue region, or safe set, is the area where the system has a controller or policy to keep it inside at all times, and reachability analysis can provide both this safe set and the safety controller (8m15s).
• Reachability analysis involves defining a failure set implicitly with a function L of x, which is negative inside the failure set and positive outside, representing the safety reward the robot gets at state x (8m41s).
• The cumulative reward of a trajectory is given by the minimum safety reward along the trajectory, which is different from typical optimal control and reinforcement learning problems where the cumulative reward is the sum of rewards (9m15s).
• The sign of the cumulative reward can be used to determine whether the system trajectory was ever safe or not, and control can be used to formulate a game between the control and disturbance to maximize the safety reward (9m36s).
• The value function corresponding to this game captures the closest the system will ever get to the failure set, and if the value function is negative, the system must have entered the failure set at some point (10m18s).
• The backward reachable tube, or unsafe set, is the set of states where the value function is less than or equal to zero, and this value function can be computed using the principle of dynamic programming, resulting in a partial differential equation (10m43s).
• The partial differential equation, known as the Hamilton-Jacobi-Bellman equation, relates how taking a particular action affects the system's value or distance to the failure set, and solving this equation provides the value function (11m21s).
• The value function can be visualized, with more red indicating a more negative value function and more blue indicating a more positive value function, representing the states closer to the ceiling or floor (11m29s).
• Hamilton-Jacobi reachability analysis is a method that can capture the requirements of safety analysis, providing a set of safe states and a controller to keep the system inside those safe states, with the help of a safety value function V, whose sign indicates whether the system is safe or unsafe, and whose gradient tells a safe controller for the system (13m9s).
• The safety value function is more negative towards the lower end than the positive end, even though the failure set is both the ceiling and the floor, because gravity is pushing the system more in one direction, making it more unsafe when closer to the floor (12m7s).
• The safety controller is a function that, at any state X, pushes the system towards higher and higher values, which means more safe, by aligning with the gradient ascent direction of the value function (12m47s).

Neural Approximations of Safety Value Function and Deep Reach

• The Hamilton-Jacobi reachability analysis can be applied to general nonlinear autonomous systems, but it has challenges, including scalability, as it is computationally hard to scale these methods beyond even five-dimensional systems, and it is not immediately clear how to interface these methods with real-world data and machine learning models (13m46s).
• To address these challenges, neural approximations of the safety value function can be used, which involves learning a neural approximation of the safety value function that takes as input the state and time of the system and outputs the corresponding safety value function (14m43s).
• The Deep Reach method is a self-supervised learning method that relies on the fact that the true safety value function must satisfy a partial differential equation, which can be used as a signal to train the safety value function (15m6s).
• In the Deep Reach method, the safety value function is trained by randomly sampling some state and time, propagating it through the neural network, computing the value function, and computing the PD violation error, which is then used to backpropagate and update the neural network (15m27s).
• The goal is to optimize the neural network parameters to learn a more accurate representation of the safety value function, which is consistent with a partial differential equation, allowing for the explicit inclusion of safety requirements in the training process and the learning of inherently safe controllers from data (15m42s).
• This approach has two key advantages: it can explicitly bake in safety requirements and learn inherently safe controllers, and neural representations are easily scalable to higher dimensional systems, enabling the synthesis of safe controllers for a broader class of autonomous systems (16m1s).
• The three aircraft conflict resolution problem is used as an example, where two evader vehicles and a pursuer vehicle with uncertain behavior need to be safeguarded against, and the failure set is defined as any configuration where two aircraft are in close proximity (16m29s).
• Due to the high dimensionality of the problem, a direct computation of the unsafe set is not scalable, and a common approach is to compute pairwise collision sets between aircraft and take their union as an approximation (17m1s).
• However, with Deep Reach, it is possible to directly compute the high-dimensional unsafe set, capturing more configurations than the approximation, including three-way interactions between aircraft that could not be captured earlier (17m39s).
• The Deep Reach framework has also been applied to autonomous driving, specifically in the context of urban driving, where an autonomous car needs to navigate around a stranded vehicle in its lane while avoiding oncoming traffic (18m36s).
• In this scenario, a learning-based controller was initially used, but it was not able to capture the nuanced intersection point between the two cars, leading to collisions, whereas Deep Reach can bake in safety requirements to prevent such collisions (19m12s).
• A safety controller was demonstrated in a car scenario, where it automatically adjusted its behavior to oncoming traffic, making the white vehicle wait behind the standard vehicle if the orange driver was aggressive, and then crossing the lane once it was safe (19m31s).
• This behavior emerged automatically from the learning-based system because the safety requirements were embedded in the learning process.
• In the latest work, Deep Reach was applied for learning safe controllers for lagged locomotion, a hybrid system with both continuous and discrete controls (20m29s).
• The safety controller successfully avoided collisions with obstacles, even when deliberately pushed into a collision by Shuang, and sometimes changed its walking pattern completely to maintain safety (20m53s).

Probabilistic Safety Assurances and Conformal Prediction

• To provide safety under neural network representations, guarantees were put on hold, and probabilistic safety assurances for Deep Reach were explored (21m19s).
• The key idea behind probabilistic assurance is that the learned safety value function induces a candidate safe policy for the robot, which should ideally achieve the same value as the original policy (22m3s).
• The gap between the two value functions can be used to calibrate the learning error, and finding a bound on the maximum learning error is of interest (22m25s).
• To provide assurances, computing the bound Delta is necessary, and various methods have been explored, including neural network verification methods and scenario optimization (23m3s).
• Conformal prediction is a method that is particularly exciting due to its simplicity and beauty, and it will be discussed further (23m17s).
• Conformal prediction provides a probabilistic bound on Delta, which cannot be computed exactly but can be approximated with high confidence, resulting in a high-confidence safe set after correcting the value function (23m42s).
• This method is illustrated in the context of a multi-vehicle collision avoidance problem, where the learned backward reachable tube is corrected by conformal prediction to obtain a certified safe set (24m20s).

Adapting Neural Safety Representations and Online Adaptation

• Neural safety representations combine traditional safety analysis methods with dynamic learning capabilities, allowing for the incorporation of safety constraints in the learning process and scalability to higher-dimensional systems (24m47s).
• These representations have two key advantages: they enable the incorporation of safety constraints in the learning process and are easily scalable to higher-dimensional systems (25m0s).
• However, in real-world scenarios, safety constraints, dynamics, control authority, and environment are subject to change, requiring the system to dynamically adapt safety controllers to maintain system safety (25m27s).
• Neural safety representations can be adapted online by inputting additional data, such as uncertain system or environment parameters, to learn the safety value function as a function of these parameters (25m58s).
• This approach is referred to as parameter-conditioned safety value functions, which can be quickly activated online to maintain safety in response to changing deployment conditions (26m13s).
• An example of this approach is demonstrated in a simple drone delivery problem, where the drone must navigate to its goal location while avoiding collisions with obstacles in uncertain wind conditions (26m34s).
• By learning a parameter-conditioned value function as a function of the uncertain wind intensity, the safe set corresponding to different wind conditions can be obtained, allowing the drone to adapt its route accordingly (26m55s).
• The drone can start with a more direct route to its goal location in low wind conditions but adapt its route in response to changing wind conditions to maintain safety (27m15s).
• A robotic system's safe set can change due to external factors such as high wind intensity, and using parameter condition representation, the safety controller can quickly adapt to take a longer but safer route to its goal location (27m36s).
• The safety value function can be adapted directly from high-dimensional observations such as RGB images or LiDAR scans, allowing the robot to dynamically construct its safety value function and maintain safety in unknown environments (28m4s).
• This approach is called observation condition reachable set, and it has two key adaptation components: dynamically adapting the safety value function using LiDAR scans to adapt to unknown obstacles, and constantly estimating the uncertainty in the robot's dynamics (28m30s).
• The uncertainty estimation component allows the robot to adapt its safety value function to account for factors such as rugged terrains or slippery surfaces, and it can be used as a safety filter for a nominal policy (28m52s).
• The framework is agnostic to the underlying nominal policy and can be used with various RL-based policies, MPC-based policies, and in different environments with dynamic obstacles and adversarial humans (29m59s).

Stress Testing Learning-Based Policies and Identifying Visual Failures

• The neural safety representations can enable adaptation to dynamic obstacles and environment uncertainty, and can be used to stress test learning-based policies to identify data regimes where they might cause safety-critical failures (30m28s).
• Stress testing is particularly important for vision-based controllers, which are becoming increasingly ubiquitous in autonomy stacks but are challenging to handle using traditional safety analysis methods due to their high dimensionality and complicated nature (31m21s).
• The problem of stress testing learning-based policies can be formalized to identify data regimes where they might cause safety-critical failures (31m35s).
• A robotic system with dynamics and a visual sensor is considered, where the sensor provides visual observations such as RGB images or point clouds, which are used as input by a vision-based controller to apply control to the robot (31m37s).
• The goal is to find the set of images or point clouds that lead to the failure of the overall closed-loop system, not just the vision-based controller (32m7s).
• The failure discovery problem is cast as a reachability problem, and the corresponding backward reachable tube is used to extract visual failures (32m25s).
• The robot's sensor function and vision-based controller are cascaded to obtain an equivalent state-based policy for the robot, which simplifies the closed-loop system for stress testing (32m39s).
• The backward reachable tube is computed to find the set of all states that will lead to failure despite the best control action, but in this case, a specific controller is used instead of the best control action (33m11s).
• The backward reachable tube is used to find the images seen by the robot along the failure states, which gives the visual failures for the system (33m48s).
• An example of this work is the collaboration with Boeing, which designed a vision-based controller for an autonomous aircraft taxiing on a runway using only RGB images from a camera on the right wing (34m3s).
• The goal is to keep the aircraft on the runway, and the failure set is defined as outside the runway boundary (34m27s).
• The backward reachable tube is computed for the aircraft under the vision-based controller, and the starting configurations that will lead to failure are shown in red, while the safe configurations are shown in blue (34m37s).
• Representative images that will cause failure are shown, and analysis of one such image reveals that the vision-based controller confuses runway markings with the center line of the runway, causing the aircraft to drive towards the marking (34m56s).
• A proposed framework can identify semantic failures in vision-based control systems, such as those that lead to system-level failures, but not component-level failures, as seen in the example of an aircraft leaving a runway due to a vision-based control failure (35m33s).
• The framework can detect prediction errors in vision-based controllers, with high prediction errors not always leading to system-level failures, and low prediction errors sometimes triggering system-level failures (35m53s).
• The goal is to target component-level errors that lead to system-level failures, and the framework can combine with parameter condition reachable sets to obtain failures as a function of different environment latents, such as time of day or cloud conditions (36m31s).
• For example, a state that is a failure during the morning due to runway markings may be safe at night, improving aircraft safety, and the framework can find failures as a function of cloud conditions, providing a diverse set of failures (36m41s).
• The framework was applied to an autonomous indoor navigation pipeline, which was trained entirely in simulation and worked well in the real world, but had interesting failure modes, such as the vision-based controller learning a correlation between light-colored surfaces and traversability (37m13s).
• This correlation led to the controller thinking it could go through light-colored walls, resulting in a collision, and the framework can be used to identify such failures and improve the vision-based controller (38m12s).
• The ultimate purpose of identifying failures is to use them to improve the vision-based controller, such as by training an anomaly detector (39m1s).
• A detector can be used to determine the probability of failure of a system given an image, and if the anomaly detector triggers a fail, a fallback controller can be used to preserve safety in an online mode (39m10s).
• The detector flags a failure input, slowing down the robot, and once the failure is resolved, the system goes back to the learning-based controller to maintain system safety (39m32s).
• Designing the anomaly detector and fallback controller are interesting questions, and a catalog of failures can provide a starting point for addressing these questions (39m52s).
• Targeted incremental training of the controller on failure data can improve performance, as shown by a significant reduction in unsafe volume after training (40m11s).

Other Safety Research Projects and Imitation Learning

• The key goal of the research is to design autonomous systems that can leverage modern machine learning methods while maintaining safety, considering safety in different stages of the learning process (40m33s).
• Other projects in the lab include Deep Reach, a paradigm to learn controllers for robotic systems, and exploring other learning paradigms such as imitation learning (41m14s).
• Imitation learning suffers from the compounding error problem, and to address this, researchers have been injecting adversarial disturbances during data collection to learn safety-aware imitation learning policies (41m28s).
• The adversarial disturbance is computed using reachability analysis and pushes the system towards safety-critical states, allowing the robot to learn corrective actions from those states (42m1s).
• The hypothesis is that by visiting more safety-critical states, the robot can learn to correct errors and improve safety during test time (42m25s).
• A hypothesis is proposed to recover from certain states, and an experiment is conducted using the same aircraft texting problem to test the imitation policy, which goes from image to action, and the results show that injecting safety information significantly improves the safety of the test policy at test time (42m59s).
• The demonstration and rollout of the policy for the vanilla method and the safety-guided imitation policy are compared, showing that the vanilla method cannot recover from errors near the boundary of the runway due to lack of data, while the safety-guided method can learn recovery behavior from such states (43m37s).
• The data used for both methods are exactly the same, and the safety-guided imitation policy is also applied to a real crazy fly, resulting in better sim-to-real transfer, specifically in the context of safety (44m12s).
• The idea of using safety-critical information to guide the exploration of a learning agent is proposed as a way to design more robust policies, and an example is shown using safety information to guide the exploration of a sampling-based planner, such as MPPI (44m48s).
• The use of safety information can also help with exploration during the design phase, achieving the same performance with much fewer samples, and this direction is being actively pursued (45m27s).

Language-Based Safety Constraints and Vision-Language Models

• The challenge of determining who gives the safety constraints and who tells what is safe or unsafe is discussed, and the use of language as a medium for flexibly defining these constraints is proposed (45m39s).
• A vision-language model is used to convert natural language feedback into physical constraints that are more interpretable by traditional safety analysis methods, such as reachability analysis or control barrier functions, to design a safety controller that adheres to these natural language feedback (46m12s).
• An experiment is conducted using a vision-language model to convert user-given safety constraints into a safety constraint, and a safety controller is computed to avoid a coffee spill on the floor (46m34s).
• The concept of using tools like VMS and LLMs to define rich safety constraints and semantic safety constraints for robotic systems is discussed as a starting point for further development (47m6s).

Further Exploration of Safety Value Function Parameterization and Scalability

• The parameterization of the value function for safety is explored, with the possibility of using latent parameters instead of explicit physical parameters, allowing for more complex inputs to be processed (48m35s).
• The use of encoder-decoder frameworks or simultaneous learning can be employed to preprocess complex inputs and make them more manageable for the safety value function (49m7s).
• The scalability of deep BRE (Bayesian Reinforcement Exploration) is discussed, with the complexity of the computation scaling with the dimensionality of the system, but with the potential for better performance if the value function has a lower-dimensional manifold (49m31s).
• Deep BRE relies on the fact that many systems of interest have a lower-dimensional representation of the value function, allowing it to overcome some of the challenges associated with the curse of dimensionality (50m10s).
• The possibility of using smarter gridding methods to improve the performance of vanilla HJB (Hamilton-Jacobi-Bellman) is mentioned, with the idea that using a parametric representation with fewer parameters could be more efficient (50m29s).

Challenges in Incremental Training and Monotonic Improvement

• The concept of training a vision-based controller using anomalies detected during training is discussed, with the question of whether such a controller would be transferable to a different environment, such as a different runway (50m46s).
• Incremental training of networks can lead to failures in certain regions of the state space where the system was previously fine, due to the lack of monotonic improvement in machine learning methods as more data is added (51m1s).
• This issue is significant for safety, as it means that the performance of the system may not be completely monotonic over the previous dataset, even if the overall metric improves (51m41s).
• The lack of monotonic improvement in incremental training makes it challenging to ensure safety, as the system may fail in some regions even if it was previously safe (52m12s).
• Researchers are exploring ways to achieve monotonic improvement in machine learning models as more data is added, which is an open question in the field (52m36s).

Co-optimizing Safety and Performance and Levels of Safety

• Safety and performance are often at odds, and current methods for ensuring safety, such as safety filtering, can be myopic and do not consider the long-term effects of actions on performance (53m17s).
• A potential solution is to design controllers that co-optimize safety and performance using dynamic programming, which can optimize both requirements simultaneously (53m36s).
• There is a trade-off between computation and the level of safety assurance, and researchers may need to compromise on safety to achieve more scalable computation (54m8s).
• Companies often think of safety in terms of levels or tiers, rather than absolute safety, but the methods for computing these levels can be ad hoc or heuristic-based (54m44s).
• Levels of safety have a hierarchy, with different severities associated with various incidents, such as collisions, and this hierarchy is considered when evaluating safety levels (54m53s).
• There is limited academic work on the hierarchy of safety levels, but it is an interesting area of study (55m11s).

Adversarial Situations and Game Theory in Autonomous Driving

• An example of an adversarial situation is when a car is convinced to squeeze into a space, and this situation raises questions about the symmetry of controllers in such scenarios (55m25s).
• In a hypothetical situation where a human and an autonomous vehicle (AO) swap controllers, it is unclear whether the system would still work, and this scenario enters the realm of game theory (55m37s).
• The assumption of symmetric information is made, assuming that both cars can be controlled and that there is a level of cooperativeness between drivers to avoid collisions (55m55s).
• In reality, oncoming traffic cannot be controlled, and the assumption of cooperativeness may not always hold true (56m7s).
• The avoidance of adversarial situations by human drivers may be more robust than expected, and there is a distribution of behaviors that can be classified as either adversarial or cooperative (56m33s).
• A behavior-level classification of vehicles is often performed by analyzing their past trajectories to determine whether they are behaving cooperatively or adversarially (56m43s).
• Based on this classification, different planning strategies are employed to navigate around the vehicle, and this process involves a level of behavior planning (56m54s).
• Cars in the city are not necessarily shy and have become more assertive over time (57m10s).