A Scalable End-to-End Anomaly Detection System using Azure Batch AI

#artificialintelligence

To complete the pipeline of an end-to-end solution I've created a walkthrough on GitHub that includes submitting and scheduling prediction jobs in addition to training of models. The solution comprises several Azure cloud services and Python code that interacts with those services. The scheduling component allows continuous training and scoring in a production environment. The diagram below shows the proposed solution architecture where the main components are Azure services that can easily connect to each other through configuration or SDKs. This is a general solution and only one way of designing predictive maintenance solutions.


Towards Experienced Anomaly Detector Through Reinforcement Learning

AAAI Conferences

This abstract proposes a time series anomaly detector which 1) makes no assumption about the underlying mechanism of anomaly patterns, 2) refrains from the cumbersome work of threshold setting for good anomaly detection performance under specific scenarios, and 3) keeps evolving with the growth of anomaly detection experience. Essentially, the anomaly detector is powered by the Recurrent Neural Network (RNN) and adopts the Reinforcement Learning (RL) method to achieve the self-learning process. Our initial experiments demonstrate promising results of using the detector in network time series anomaly detection problems.


A note about finding anomalies – Towards Data Science

#artificialintelligence

This article is inspired by the research done during studies in the university. Goal of this article is to act as a note and reminder that finding anomalies is not a trivial task (currently). Anomaly detection refers to the task of finding observations that do not conform to the normal, expected behaviour. These observations can be named as anomalies, outliers, novelty, exceptions, surprises in different application domains. The most popular terms that occur most often in literature are anomalies and outliers.


Statistical Anomaly Detection for Train Fleets

AI Magazine

We have developed a method for statistical anomaly detection which has been deployed in a tool for condition monitoring of train fleets. The tool is currently used by several railway operators over the world to inspect and visualize the occurrence of event messages generated on the trains. The anomaly detection component helps the operators to quickly find significant deviations from normal behavior and to detect early indications for possible problems. The savings in maintenance costs comes mainly from avoiding costly breakdowns, and have been estimated to several million Euros per year for the tool. In the long run, it is expected that maintenance costs can be reduced with between 5 and 10 % by using the tool.


Anomaly detection: Machine learning for real-time decision making

#artificialintelligence

Ever since the rise of big data enterprises of all sizes have been in a state of uncertainty. Today we have more data available than ever before, but few have been able to implement the procedures to turn this data into insights. To the human eye, there is just too much data to process. Tim Keary looks at anomaly detection in this first of a series of articles. Unmanageable datasets have become a problem as organizations are needing to make faster decision in real-time.