Apache Kafka on Azure HDInsight was added last year as a preview service to help enterprises create real-time big data pipelines. Since then, large companies such as Toyota, Adobe, Bing Ads, and GE have been using this service in production to process over a million events per sec to power scenarios for connected cars, fraud detection, clickstream analysis, and log analytics. HDInsight has worked very closely with these customers to understand the challenges of running a robust, real-time production pipeline at an enterprise scale. Using our learnings, we have implemented key features in the managed Kafka service on HDInsight, which is now generally available.
Household names such as Adobe, Jet, ASOS, Schneider Electric, and Milliman are amongst hundreds of enterprises that are powering their Big Data Analytics using Azure HDInsight. Azure HDInsight launched nearly six years ago and has since become the best place to run Apache Hadoop and Spark analytics on Azure. We will monitor the cluster and all the services, detect and repair common issues and respond to issues 24/7. Your big data applications can run more reliably as your HDInsight service monitors the health and automatically recovers from failures. Isolate your HDInsight cluster within VNETs and take advantage of transparent data encryption.
Apache Spark is an open-source parallel processing framework that supports in-memory processing to boost the performance of big-data analytic applications. Spark processing engine is built for speed, ease of use, and sophisticated analytics. Spark's in-memory computation capabilities make it a good choice for iterative algorithms in machine learning and graph computations. Spark is also compatible with Azure Blob storage (WASB) so your existing data stored in Azure can easily be processed via Spark. When you create a Spark cluster in HDInsight, you create Azure compute resources with Spark installed and configured.
Kyvos Insights, an OLAP-on-Hadoop startup, is announcing today that it can now run on Microsoft's Azure HDInsight. Customers can now provision an HDInsight cluster, then set up an Azure virtual machine, install Kyvos on it, and federate that Kyvos instance with the HDInsight cluster. With the announcement, the Kyvos/HDInsight combination is now both compatible and officially supported. Welcome to the OLAP party Kyvos joins Big Data vendors Platfora and AtScale in offering an OLAP cube approach to Hadoop analytics. Platfora doesn't call itself out as an OLAP solution -- instead it guides customers through the creation of "lenses," which run on Platfora's own in-memory platform, including dedicated hardware.