Closed-Loop Policies for Operational Tests of Safety-Critical Systems

Abstract--Manufacturers of safety-critical systems must make the case that their product is sufficiently safe for public deployment. Much of this case often relies upon critical event outcomes from real-world testing, requiring manufacturers to be strategic about how they allocate testing resources in order to maximize their chances of demonstrating system safety. This work frames the partially observable and belief-dependent problem of test scheduling as a Markov decision process, which can be solved efficiently to yield closed-loop manufacturer testing policies. By solving for policies over a wide range of problem formulations, we are able to provide high-level guidance for manufacturers and regulators on issues relating to the testing of safety-critical systems. This guidance spans an array of topics, including circumstances under which manufacturers should continue testing despite observed incidents, when manufacturers should test aggressively, and when regulators should increase or reduce the real-world testing requirements for an autonomous vehicle. I. INTRODUCTION Confidence must be established in safety-critical systems such as autonomous vehicles prior to their widespread release. Establishing confidence is difficult because the space of driving scenarios is vast and accidents are rare. Automotive manufacturers can build confidence by conducting test drives on public roadways and make the safety case based on the frequency of observed hazardous events like disengagements and traffic accidents. Each manufacturer must devise a testing strategy capable of providing sufficient evidence that their system is safe enough for widespread adoption. Real-world testing that is too aggressive may yield hazardous events that diminish confidence in system safety. However, a manufacturer that is reluctant to test their product may forfeit opportunities to identify and address shortcomings, and may ultimately not be able to compete in the market. The fundamental tension between the desire to thoroughly test a product and the urgency to forego further testing in favor of bringing the product to market is not unique to the automotive industry.