What You Don't Know Can Hurt You: How Well do Latent Safety Filters Understand Partially Observable Safety Constraints?

What Y ou Don't Know Can Hurt Y ou: How Well do Latent Safety Filters Understand Partially Observable Safety Constraints? Figure 1: We design a series of controlled experiments to test how latent safety filters behave under partially observable constraints. Left: We find that safety filters that rely on latent state representations trained-on and deployed-with only RGB inputs behave unreliably when they must enforce constraints, such as temperature limits, that are not easily observable. Right: Training with rich, safety-relevant multimodal supervision shapes the latent state representation to enable safe control (e.g., lifting the pan before overheating), even when the robot is deployed with only RGB inputs at runtime. Abstract-- Safe control techniques, such as Hamilton-Jacobi reachability, provide principled methods for synthesizing safety-preserving robot policies but typically assume hand-designed state spaces and full observability. Recent work has relaxed these assumptions via latent-space safe control, where state representations and dynamics are learned jointly through world models that reconstruct future high-dimensional observations (e.g., RGB images) from current observations and actions. This enables safety constraints that are difficult to specify analytically (e.g., spilling) to be framed as classification problems in latent space, allowing controllers to operate directly from raw observations. However, these methods assume that safety-critical features are observable in the learned latent state. We ask: when are latent state spaces sufficient for safe control?