Apache Kafka and the four challenges of production machine learning systems

#artificialintelligence

Machine learning has become mainstream, and suddenly businesses everywhere are looking to build systems that use it to optimize aspects of their product, processes or customer experience.


Apache Kafka and the four challenges of production machine learning systems

#artificialintelligence

To learn more about cutting-edge data science tools like Apache Kafka, check out the Strata Data Conference in Singapore, Dec. 4-7, 2017--early price ends October 20.



Secure Deep Learning Engineering: A Software Quality Assurance Perspective

arXiv.org Artificial Intelligence

Over the past decades, deep learning (DL) systems have achieved tremendous success and gained great popularity in various applications, such as intelligent machines, image processing, speech processing, and medical diagnostics. Deep neural networks are the key driving force behind its recent success, but still seem to be a magic black box lacking interpretability and understanding. This brings up many open safety and security issues with enormous and urgent demands on rigorous methodologies and engineering practice for quality enhancement. A plethora of studies have shown that the state-of-the-art DL systems suffer from defects and vulnerabilities that can lead to severe loss and tragedies, especially when applied to real-world safety-critical applications. In this paper, we perform a large-scale study and construct a paper repository of 223 relevant works to the quality assurance, security, and interpretation of deep learning. We, from a software quality assurance perspective, pinpoint challenges and future opportunities towards universal secure deep learning engineering. We hope this work and the accompanied paper repository can pave the path for the software engineering community towards addressing the pressing industrial demand of secure intelligent applications.


More on "Why machine learning projects are hard to manage"?

#artificialintelligence

This article (Why are Machine Learning Projects so Hard to Manage?) by Lukas Biewald got me thinking about some more issues that make ML projects hard to manage. Here are a few observations I have made. As mentioned in Lukas' article, setting expectations is itself a problem. Another problem is that traditional software development methodologies and tools may not always help. Additionally, all of the failed attempts, and any errors need to be just as well documented as the final correct and successful model.