Artificial intelligence (AI), and especially its sub-field of Machine Learning (ML), are impacting the daily lives of everyone with their ubiquitous applications. In recent years, AI researchers and practitioners have introduced principles and guidelines to build systems that make reliable and trustworthy decisions. From a practical perspective, conventional ML systems process historical data to extract the features that are consequently used to train ML models that perform the desired task. However, in practice, a fundamental challenge arises when the system needs to be operationalized and deployed to evolve and operate in real-life environments continuously. To address this challenge, Machine Learning Operations (MLOps) have emerged as a potential recipe for standardizing ML solutions in deployment. Although MLOps demonstrated great success in streamlining ML processes, thoroughly defining the specifications of robust MLOps approaches remains of great interest to researchers and practitioners. In this paper, we provide a comprehensive overview of the trustworthiness property of MLOps systems. Specifically, we highlight technical practices to achieve robust MLOps systems. In addition, we survey the existing research approaches that address the robustness aspects of ML systems in production. We also review the tools and software available to build MLOps systems and summarize their support to handle the robustness aspects. Finally, we present the open challenges and propose possible future directions and opportunities within this emerging field. The aim of this paper is to provide researchers and practitioners working on practical AI applications with a comprehensive view to adopt robust ML solutions in production environments.

Machine learning has enabled tremendously exciting technologies, but at the same time it raises questions as to how it should be deployed in a responsible and trustworthy manner. How can machine learning be made secure, reliable, robust, fair, and private? This workshop will explore the information-theoretic foundations of these aspects of machine learning. The workshop will include invited talks by experts on these topics from both academy and industry, student poster presentations, and time for fruitful discussions. Keynote talks will be given by Tara Javidi, Ilya Mironov, Todd Coleman, and Ayfer Ozgur.

Ensuring trustworthiness in machine learning (ML) models is a multi-dimensional task. In addition to the traditional notion of predictive performance, other notions such as privacy, fairness, robustness to distribution shift, adversarial robustness, interpretability, explainability, and uncertainty quantification are important considerations to evaluate and improve (if deficient). However, these sub-disciplines or 'pillars' of trustworthiness have largely developed independently, which has limited us from understanding their interactions in real-world ML pipelines. In this paper, focusing specifically on compositions of functions arising from the different pillars, we aim to reduce this gap, develop new insights for trustworthy ML, and answer questions such as the following. Does the composition of multiple fairness interventions result in a fairer model compared to a single intervention? How do bias mitigation algorithms for fairness affect local post-hoc explanations? Does a defense algorithm for untargeted adversarial attacks continue to be effective when composed with a privacy transformation? Toward this end, we report initial empirical results and new insights from 9 different compositions of functions (or pipelines) on 7 real-world datasets along two trustworthy dimensions - fairness and explainability. We also report progress, and implementation choices, on an extensible composer tool to encourage the combination of functionalities from multiple pillars. To-date, the tool supports bias mitigation algorithms for fairness and post-hoc explainability methods. We hope this line of work encourages the thoughtful consideration of multiple pillars when attempting to formulate and resolve a trustworthiness problem.

Concerns about the societal impact of AI-based services and systems has encouraged governments and other organisations around the world to propose AI policy frameworks to address fairness, accountability, transparency and related topics. To achieve the objectives of these frameworks, the data and software engineers who build machine-learning systems require knowledge about a variety of relevant supporting tools and techniques. In this paper we provide an overview of technologies that support building trustworthy machine learning systems, i.e., systems whose properties justify that people place trust in them. We argue that four categories of system properties are instrumental in achieving the policy objectives, namely fairness, explainability, auditability and safety & security (FEAS). We discuss how these properties need to be considered across all stages of the machine learning life cycle, from data collection through run-time model inference. As a consequence, we survey in this paper the main technologies with respect to all four of the FEAS properties, for data-centric as well as model-centric stages of the machine learning system life cycle. We conclude with an identification of open research problems, with a particular focus on the connection between trustworthy machine learning technologies and their implications for individuals and society.

A team of U.S. computer scientists is receiving a $10 million grant from the National Science Foundation to make machine learning more secure. The grant establishes the Center for Trustworthy Machine Learning at a consortium of seven universities, including the University of California San Diego. Researchers will work together toward two goals: understanding the risks inherent to machine learning; and developing the tools, metrics, and methods to manage and mitigate these risks. The science and arsenal of defensive techniques emerging within the center will provide the basis for building more trustworthy and secure systems in the future, as well as fostering a long-term research community within this essential domain of technology, researchers say. "This research is important because machine learning is becoming more pervasive in our daily lives, powering technologies we interact with, including services like e-commerce and Internet searches, as well as devices such as Internet-connected smart speakers," says Kamalika Chaudhuri, a computer science professor at the Jacobs School of Engineering, who will be leading the UC San Diego portion of the research.

San Diego, Calif., Oct. 24, 2018 -- A team of U.S. computer scientists is receiving a $10 million grant from the National Science Foundation to make machine learning more secure. The grant establishes the Center for Trustworthy Machine Learning at a consortium of seven universities, including the University of California San Diego. Researchers will work together toward two goals: understanding the risks inherent to machine learning; and developing the tools, metrics and methods to manage and mitigate these risks. The science and arsenal of defensive techniques emerging within the center will provide the basis for building more trustworthy and secure systems in the future, as well as fostering a long-term research community within this essential domain of technology, researchers said. "This research is important because machine learning is becoming more pervasive in our daily lives, powering technologies we interact with, including services like e-commerce and Internet searches, as well as devices such as Internet-connected smart speakers," said Kamalika Chaudhuri, a computer science professor at the Jacobs School of Engineering, who will be leading the UC San Diego portion of the research.