Anomaly Detection

In Machine Learning is normal to deal with Anomaly Detection tasks. Data Science frequently are engaged in problem where they have to show, explain and predict anomalies. I also made a post about Anomaly Detection with Time Series, where I studied an internal system behaviour and I provided anomaly forecasts in the future. In this post I try to solve a different challenge. I change domain of interest: swapping from Time Series to Images.

The Gartner Security & Risk Management Summit is just a few days away, and I'm delighted to have the opportunity to chat with attendees about how anomaly detection and machine learning can help give your organization a more proactive security posture. You don't need to have been in the cybersecurity space for long to be bewildered by and unsure about vendor claims around artificial intelligence, machine learning, and analytics. At Interset (acquired by Micro Focus in February of this year), we have regular conversations with security professionals who struggle to understand which techniques and tools are effective in boosting breach defense in the real world. Ultimately, these conversations lead to an important question for us: How can you implement user and entity behavioral analytics (UEBA) in a way that will enable an efficient security operations center (SOC)? There are multiple factors that go into an effective UEBA implementation, but it's helpful to start with ensuring that the math and machine learning powering the solution are suitable for your security objectives.

A.I. based automated Anomaly detection system is gaining popularity nowadays due to the increase in data generated from various devices and the increase in ever evolving sophisticated threats from hackers etc. Anomaly detection systems can be applied across various business scenarios like monitoring financial transactions of a fintech company, highlighting fraudulent activities in a network, e-commerce price glitches among millions of products, and so on. Anomaly detection system can work well in managing millions of metrics at scale and filter them into a number of consumable incidents to create actionable insights. What is the alert frequency (5 minutes/ 10 minutes/ 1 hour or 1 day): Alert frequency is very much dependent on the sensitivity of the process which will be being measured, including the reaction time and other metrics. Some applications demand low latency: like detecting & intimating the suspicious fraudulent payment transactions to users in case of any misuse of the card within minutes. In the case of some applications it can be less sensitive to changes and not so severe, like total inbound & outbound calls from cellular towers, which can be aggregated to an hourly level rather than measuring at 5-minute intervals etc.

Data-driven anomaly detection methods typically build a model for the normal behavior of the target system, and score each data instance with respect to this model. A threshold is invariably needed to identify data instances with high (or low) scores as anomalies. This presents a practical limitation on the applicability of such methods, since most methods are sensitive to the choice of the threshold, and it is challenging to set optimal thresholds. We present a probabilistic framework to explicitly model the normal and anomalous behaviors and probabilistically reason about the data. An extreme value theory based formulation is proposed to model the anomalous behavior as the extremes of the normal behavior. As a specific instantiation, a joint non-parametric clustering and anomaly detection algorithm (INCAD) is proposed that models the normal behavior as a Dirichlet Process Mixture Model. A pseudo-Gibbs sampling based strategy is used for inference. Results on a variety of data sets show that the proposed method provides effective clustering and anomaly detection without requiring strong initialization and thresholding parameters.

Gartner Supply Chain Executive Summit -- IBM (NYSE: IBM) today launched Business Transactional Intelligence (BTI), an AI-powered solution that offers anomaly detection and visualization capabilities for mitigating supply chain disruptions and accelerating data-driven decision making. BTI, part of IBM's Supply Chain Business Network, enables companies to garner deeper insights into supply chain data to help them better manage, for example, order-to-cash and purchase-to-pay interactions. The technology does this, in part, using machine learning to identify volume, velocity and value-pattern anomalies in supply chain documents and transactions. Machine learning is a method used to teach artificial intelligence how to learn from data, spot patterns and make decisions on its own. This enables companies to discover potential issues faster and resolve them before they escalate and impact the business.

Anomaly detection covers a large number of data analytics use cases. However, here anomaly detection refers specifically to the detection of unexpected events, be it cardiac episodes, mechanical failures, hacker attacks, or fraudulent transactions. The unexpected character of the event means that no such examples are available in the data set. Classification solutions generally require a set of examples for all involved classes. So, how do we proceed in a case where no examples are available?

An advantage of using a neural technique compared to a standard clustering technique is that neural techniques can handle non-numeric data by encoding that data. Anomaly detection, also called outlier detection, is the process of finding rare items in a dataset. Examples include finding fraudulent login events and fake news items. Take a look at the demo program in Figure 1. The demo examines a 1,000-item subset of the well-known MNIST (modified National Institute of Standards and Technology) dataset.

In the earliest days of big data, collection was the top priority. Business leaders needed to find innovative ways to collect as much information about customers and operations as possible. Now that this goal has been accomplished, a new problem has arisen. There is enough data available to optimize user experience, network performance, business operations, and more, however, between 60 and 73 percent of that data never gets put to good use. There is an overwhelming amount of different metrics and systems to track, making it increasingly difficult to evaluate business patterns and, more importantly, deviations.

Note: This post is part of a broader work for predicting stock prices. The outcome (identified anomaly) is a feature (input) in a LSTM model (within a GAN architecture)- link to the post. Options valuation is a very difficult task. To begin with, it entails using a lot of data points (some are listed below) and some of them are quite subjective (such as the implied volatility -- see below) and difficult to calculate precisely. As an example let us check the calculation for the call's Theta -- θ: Another example of how difficult options pricing is, is the Black-Scholes formula which is used for calculating the options prices themselves.

Azure Stream Analytics is a fully managed serverless offering on Azure. With the new Anomaly Detection functions in Stream Analytics, the whole complexity associated with building and training custom machine learning (ML) models is reduced to a simple function call resulting in lower costs, faster time to value, and lower latencies.