To demonstrate a practical use of an ML pipeline, this example uses the sample HVAC.csv data file that comes pre-loaded on the default storage for your HDInsight cluster, either Azure Storage or Data Lake Storage. HVAC.csv contains a set of times with both target and actual temperatures for HVAC (heating, ventilation, and air conditioning) systems in various buildings. The goal is to train the model on the data, and produce a forecast temperature for a given building.

In this episode, we chat with Katherine Kampf, PM on Azure Big Data team, about the newly introduced ML Services in Azure HDInsight. ML Services includes highly scalable, distributed set of algorithms such as RevoscaleR, revoscalepy, and microsoftML that can work on data sizes larger than the size of physical memory, and run on a wide variety of platforms in a distributed manner. Learn more about the collection of Microsoft's custom R packages and Python packages included with the product. Any R or Python open-source machine learning package can work side by side with any proprietary innovation from Microsoft.

Learn how to use Apache Hive to process Twitter data. The result is a list of Twitter users who sent the most tweets that contain a certain word. The steps in this document were tested on HDInsight 3.6. Linux is the only operating system used on HDInsight version 3.4 or greater. For more information, see HDInsight retirement on Windows.

Today, we announced the general availability of R Server for Azure HDInsight. This gives Azure HDInsight the most comprehensive set of ML algorithms and statistical functions in the cloud that also leverages Hadoop and Spark. R is one of the most popular programming language that helps millions of data scientists solve their most challenging problems in fields ranging from computational biology to quantitative marketing. R Server for Azure HDInsight is a scale-out implementation of R integrated with Spark clusters created from HDInsight. This gives you the familiarity of the R language for machine learning while leveraging the scalability and reliability built into Spark.

In this session from the Microsoft Machine Learning & Data Science Summit, explore how parallelized ML algorithms in Azure HDInsight R Server and Spark machine learning library contrast and complement each other. The session also covers advanced algorithms such as deep neural network learning libraries available in the broader Spark ecosystem.

Almost 12 months after releasing the public preview of Spark for Azure HDInsight Microsoft has announced general availability of the proposition to the industry, as well as extending Cortana's offering to the Xbox. Making the announcement on the Azure blog, Oliver Chiu, Product Marketing Manager for Hadoop/Big Data and Data Warehousing, outlined the improvements made on the offering as well as the company's efforts to make big data easy and more approachable. The company claims the Hadoop and Spark cloud service is now an enterprise-ready solution which is fully managed, secured, and highly available. "Since we announced the public preview, Spark for HDInsight has gained rapid adoption and is now 50% of all new HDInsight clusters deployed," said Chiu. "With GA (General Availability), we are revealing improvements we've made to the service to make Spark hardened for the enterprise and easy for your users. This includes improvements to the availability, scalability, and productivity of our managed Spark service."