Time-Series Anomaly Detection Service at Microsoft

arXiv.org Machine Learning

Large companies need to monitor various metrics (for example, Page Views and Revenue) of their applications and services in real time. At Microsoft, we develop a time-series anomaly detection service which helps customers to monitor the time-series continuously and alert for potential incidents on time. In this paper, we introduce the pipeline and algorithm of our anomaly detection service, which is designed to be accurate, efficient and general. The pipeline consists of three major modules, including data ingestion, experimentation platform and online compute. To tackle the problem of time-series anomaly detection, we propose a novel algorithm based on Spectral Residual (SR) and Convolutional Neural Network (CNN). Our work is the first attempt to borrow the SR model from visual saliency detection domain to time-series anomaly detection. Moreover, we innovatively combine SR and CNN together to improve the performance of SR model. Our approach achieves superior experimental results compared with state-of-the-art baselines on both public datasets and Microsoft production data.


Good paper on multidimensional outlier detection on time series

@machinelearnbot

In this study, we examine the issues of anomaly detection in multi-dimensional time-series data. We propose time-series data cube to capture the multi-dimensional space formed by the attribute structure. This facilitates the detection of anomalies based on expected values derived from higher level, \more general" time-series. Anomaly detection in a time-series data cube poses computational challenges, especially for high-dimensional, large data sets. To this end, we also propose an e cient search algorithm to iteratively select subspaces in the original high-dimensional space and detect anomalies within each one.


Towards Experienced Anomaly Detector Through Reinforcement Learning

AAAI Conferences

This abstract proposes a time series anomaly detector which 1) makes no assumption about the underlying mechanism of anomaly patterns, 2) refrains from the cumbersome work of threshold setting for good anomaly detection performance under specific scenarios, and 3) keeps evolving with the growth of anomaly detection experience. Essentially, the anomaly detector is powered by the Recurrent Neural Network (RNN) and adopts the Reinforcement Learning (RL) method to achieve the self-learning process. Our initial experiments demonstrate promising results of using the detector in network time series anomaly detection problems.


A note about finding anomalies – Towards Data Science

#artificialintelligence

This article is inspired by the research done during studies in the university. Goal of this article is to act as a note and reminder that finding anomalies is not a trivial task (currently). Anomaly detection refers to the task of finding observations that do not conform to the normal, expected behaviour. These observations can be named as anomalies, outliers, novelty, exceptions, surprises in different application domains. The most popular terms that occur most often in literature are anomalies and outliers.


Statistical Anomaly Detection for Train Fleets

AI Magazine

We have developed a method for statistical anomaly detection which has been deployed in a tool for condition monitoring of train fleets. The tool is currently used by several railway operators over the world to inspect and visualize the occurrence of event messages generated on the trains. The anomaly detection component helps the operators to quickly find significant deviations from normal behavior and to detect early indications for possible problems. The savings in maintenance costs comes mainly from avoiding costly breakdowns, and have been estimated to several million Euros per year for the tool. In the long run, it is expected that maintenance costs can be reduced with between 5 and 10 % by using the tool.