Abstracting the Geniuses Away from Failure Testing

The heterogeneity, complexity, and Scale of cloud applications make verification of their fault tolerance properties challenging. Companies are moving away from formal methods and toward large-scale testing in which components are deliberately compromised to identify weaknesses in the software. For example, techniques such as Jepsen apply fault-injection testing to distributed data stores, and Chaos Engineering performs fault injection experiments on production systems, often on live traffic. Both approaches have captured the attention of industry and academia alike. Unfortunately, the search space of distinct fault combinations that an infrastructure can test is intractable.