Anomaly Detection

Gartner Supply Chain Executive Summit -- IBM (NYSE: IBM) today launched Business Transactional Intelligence (BTI), an AI-powered solution that offers anomaly detection and visualization capabilities for mitigating supply chain disruptions and accelerating data-driven decision making. BTI, part of IBM's Supply Chain Business Network, enables companies to garner deeper insights into supply chain data to help them better manage, for example, order-to-cash and purchase-to-pay interactions. The technology does this, in part, using machine learning to identify volume, velocity and value-pattern anomalies in supply chain documents and transactions. Machine learning is a method used to teach artificial intelligence how to learn from data, spot patterns and make decisions on its own. This enables companies to discover potential issues faster and resolve them before they escalate and impact the business.

Anomaly detection covers a large number of data analytics use cases. However, here anomaly detection refers specifically to the detection of unexpected events, be it cardiac episodes, mechanical failures, hacker attacks, or fraudulent transactions. The unexpected character of the event means that no such examples are available in the data set. Classification solutions generally require a set of examples for all involved classes. So, how do we proceed in a case where no examples are available?

An advantage of using a neural technique compared to a standard clustering technique is that neural techniques can handle non-numeric data by encoding that data. Anomaly detection, also called outlier detection, is the process of finding rare items in a dataset. Examples include finding fraudulent login events and fake news items. Take a look at the demo program in Figure 1. The demo examines a 1,000-item subset of the well-known MNIST (modified National Institute of Standards and Technology) dataset.

In the earliest days of big data, collection was the top priority. Business leaders needed to find innovative ways to collect as much information about customers and operations as possible. Now that this goal has been accomplished, a new problem has arisen. There is enough data available to optimize user experience, network performance, business operations, and more, however, between 60 and 73 percent of that data never gets put to good use. There is an overwhelming amount of different metrics and systems to track, making it increasingly difficult to evaluate business patterns and, more importantly, deviations.

Note: This post is part of a broader work for predicting stock prices. The outcome (identified anomaly) is a feature (input) in a LSTM model (within a GAN architecture)- link to the post. Options valuation is a very difficult task. To begin with, it entails using a lot of data points (some are listed below) and some of them are quite subjective (such as the implied volatility -- see below) and difficult to calculate precisely. As an example let us check the calculation for the call's Theta -- θ: Another example of how difficult options pricing is, is the Black-Scholes formula which is used for calculating the options prices themselves.

Azure Stream Analytics is a fully managed serverless offering on Azure. With the new Anomaly Detection functions in Stream Analytics, the whole complexity associated with building and training custom machine learning (ML) models is reduced to a simple function call resulting in lower costs, faster time to value, and lower latencies.

Built-in machine learning (ML) models for anomaly detection in Azure Stream Analytics significantly reduces the complexity and costs associated with building and training machine learning models. This feature is now available for public preview worldwide both in the cloud and on IoT Edge. Azure Stream Analytics is a fully managed serverless PaaS offering on Azure that enables customers to analyze and process fast moving streams of data, and deliver real-time insights for mission critical scenarios. Developers can use a simple SQL language (extensible to include custom code) to author and deploy powerful analytics processing logic that can scale-up and scale-out to deliver insights with milli-second latencies. Many customers use Azure Stream Analytics to continuously monitor massive amounts of fast-moving streams of data in order to detect issues that do not conform to expected patterns and prevent catastrophic losses.

In this part of the assignment, we will implement an anomaly detection algorithm using the Gaussian model to detect anomalous behavior in a 2D dataset first and then a high-dimensional dataset. Multivariate Gaussian Distribution is an optional lecture in the course and the code to compute the probability density is given to us. However, in order for me to proceed on with the assignment, I need to write the multivariateGaussian function from scratch. Some of the interesting functions we had utilized here are from numpy linear algebra class. The official documentation can be found here.

Correlated anomaly detection (CAD) from streaming data is a type of group anomaly detection and an essential task in useful real-time data mining applications like botnet detection, financial event detection, industrial process monitor, etc. The primary approach for this type of detection in previous researches is based on principal score (PS) of divided batches or sliding windows by computing top eigenvalues of the correlation matrix, e.g. the Lanczos algorithm. However, this paper brings up the phenomenon of principal score degeneration for large data set, and then mathematically and practically prove current PS-based methods are likely to fail for CAD on large-scale streaming data even if the number of correlated anomalies grows with the data size at a reasonable rate; in reality, anomalies tend to be the minority of the data, and this issue can be more serious. We propose a framework with two novel randomized algorithms rPS and gPS for better detection of correlated anomalies from large streaming data of various correlation strength. The experiment shows high and balanced recall and estimated accuracy of our framework for anomaly detection from a large server log data set and a U.S. stock daily price data set in comparison to direct principal score evaluation and some other recent group anomaly detection algorithms. Moreover, our techniques significantly improve the computation efficiency and scalability for principal score calculation.

When a person drives, there are many things that are quickly noticed and then ignored. What gains attention are those things that might be a danger. A pedestrian who might walk out into the road, a light turning yellow, an adjacent car drifting into the same lane, all of those need special attention. The same thing is true in the world of business computing. For instance, a sudden increase in sales is great, but the company needs to track that anomalous increase back to its cause in order to identify and replicate the reason.