Microsoft & Others Catalog Threats to Machine Learning Systems

In May 2016, Microsoft introduce a chatbot on Twitter, dubbed "Tay," that attempted to hold conversations with users and improve its responses through machine learning (ML). A coordinated attack on the chatbot, however, caused the algorithm to start tweeting "wildly inappropriate and reprehensible words and images" in the first 24 hours, Microsoft stated at the time. For the software giant, the attack demonstrated that the world of ML and artificial intelligence (AI) would come with threats. Last week, the company and an interdisciplinary group of security professionals and ML researchers from a dozen other organizations took a first stab at creating a vocabulary for describing attacks on ML systems with the initial draft of the Adversarial ML Threat Matrix. The threat matrix is an extension of MITRE's ATT&CK framework for the classification of attack techniques.