Azure Stream Analytics is a fully managed serverless offering on Azure. With the new Anomaly Detection functions in Stream Analytics, the whole complexity associated with building and training custom machine learning (ML) models is reduced to a simple function call resulting in lower costs, faster time to value, and lower latencies.
In this walkthrough I show how an end-to-end anomaly detection system can be implemented for IoT use cases. The scenario is intentionally kept simple for illustration purposes and to allow generalizing to different scenarios in industry. The solution is built on Microsoft's Azure stack and includes multiple cloud services that allow handling data streaming, data processing, model training/predicting, and data storage. The main component here is Batch AI, a cloud service that enables users to submit parallel jobs to a cluster of high performing virtual machines. The business problem addressed in this walkthrough is: monitoring sensor measurements of multiple devices and predicting potential anomalies that might lead to failures across these devices.
We knew Azure is one of the fastest growing Cloud services around the world it helps developers and IT Professionals to create and manage their applications. When Azure HDInsight has huge success in Hadoop based technology, For Marketing Leaders in Big data Microsoft has taken another step and introduced Azure Machine Learning which is also called as "Azure ML". After the release of Azure ML, the developers feel easy to build applications and Azure ML run's under a public cloud by this user need not to download any external hardware or software. Azure Machine Learning is combined in the development environment which is renamed as Azure ML Studio. The main reason to introduce Azure ML to make users to create a data models without the help of data science background, In Azure ML Data models, are Created with end-to-end services were as ML Studio is used to build and test by using drag-and-drop and also we can deploy analytics solution for our data's too.
Stream Analytics helps to develop and deploy solutions to gain real time insights from devices, sensors, and applications by real time stream processing in the cloud. Stream Analytics enables to perform real time analytics for Internet of Things solutions, stream millions of events per second, provide mission critical reliability and performance, also deliver real time dashboards and alerts over data from devices and applications, correlate across multiple streams of data and use SQL based language for development. Stream Analytics customers deploy and monitor streaming jobs. Applications of stream analytics includes personalized, real-time stock-trading analysis and alerts offered by financial services companies, real-time fraud detection; data and identity protection services, analysis of data generated by sensors and actuators, web clickstream analytics, customer relationship management (CRM) alerts, supply chain alerts, transportation alerts. Apache Flink is an open source platform for distributed stream and batch data processing.
Anomaly Detection is an API built with Azure Machine Learning that is useful for detecting different types of anomalous patterns in your time series data. The API assigns an anomaly score to each data point in the time series, which can be used for generating alerts, monitoring through dashboards or connecting with your ticketing systems. The machine learning based API enables: - *Flexible and robust detection:* The anomaly detection models allow users to configure sensitivity settings and detect anomalies among seasonal and non-seasonal data sets. Users can adjust the anomaly detection model to make the detection API less or more sensitive according to their needs. This would mean detecting the less or more visible anomalies in data with and without seasonal patterns.