Don't look now but Apache Spark is about to turn 10 years old. The open source project began quietly at UC Berkeley in 2009 before emerging as an open source project in 2010. For the past five years, Spark has been on an absolute tear, becoming one of the most widely used technologies in big data and AI. Let's take a look at Spark's remarkable run up to this point, and see where it might be headed next. Apache Spark is best known as the in-memory replacement for MapReduce, the disk-based computational engine at the heart of early Hadoop clusters.

Editor's Note: Download this Free eBook: Getting Started with Apache Spark 2.x – from Inception to Production In this blog post, we will give an introduction to machine learning and deep learning, and we will go over the main Spark machine learning algorithms and techniques with some real-world use cases. The goal is to give you a better understanding of what you can do with machine learning. Machine learning is becoming more accessible to developers, and data scientists work with domain experts, architects, developers, and data engineers, so it is important for everyone to have a better understanding of the possibilities. Every piece of information that your business generates has potential to add value. This overview is meant to provoke a review of your own data to identify new opportunities.

Learn how to utilize some of the most valuable tech skills on the market today, Scala and Spark! In this course we will show you how to use Scala and Spark to analyze Big Data. Scala and Spark are two of the most in demand skills right now, and with this course you can learn them quickly and easily! This course comes with full projects for you including topics such as analyzing financial data or using machine learning to classify Ecommerce customer behavior! We teach the latest methodologies of Spark 2.0 so you can learn how to use SparkSQL, Spark DataFrames, and Spark's MLlib!

My journey into Deep Learning In this post I'll share how I've been studying Deep Learning and using it to solve data science problems.

The aim of this course is to provide a practical understanding of advanced Machine Learning algorithms in Apache Spark to make predictions and recommendation and derive insights from large distributed datasets. This course starts with an introduction to the key concepts and data types that are fundamental to understanding distributed data processing and Machine Learning with Spark. Further to this, we provide practical recipes that demonstrate some of the most popular algorithms in Spark, leading to the creation of sophisticated Machine Learning pipelines and applications. The final sections are dedicated to more advanced use cases for Machine Learning: streaming, Natural Language Processing, and Deep Learning. In each section, we briefly establish the theoretical basis of the topic under discussion and then cement our understanding with practical use cases.

Ground level definitions out of the way, what has Databricks been doing to add to unified utopia? The company has this month announced Apache Spark open-source cluster-computing framework. This means that the company is the vendor to support Apache Spark 2.3 within a compute engine, Databricks Runtime 4.0, which is now generally available. In addition to support for Spark 2.3, Databricks Runtime 4.0 introduces new features including Machine Learning Model Export to simplify production deployments and performance optimizations. "The community continues to expand on Apache Spark's role as a unified analytics engine for big data and AI. This is a major milestone to introduce the continuous processing mode of Structured Streaming with millisecond low-latency, as well as other features across the project," said Matei Zaharia, creator of Apache Spark and chief technologist and co-founder of Databricks.

This blog post describes our experience debugging a failing test case caused by a cross join query running "too fast." Because the root cause of fail test case spans across multiple layers--from Apache Spark to the JVM JIT compiler-- we wanted to share our analysis in this post. The vast majority of big data SQL or MPP engines follow the Volcano iterator architecture that is inefficient for analytical workloads. Since Spark 2.0 release, the new Tungsten execution engine in Apache Spark implements whole-stage code generation, a technique inspired by modern compilers to collapse the entire query into a single function. This JIT compiler approach is a far superior architecture than the row-at-a-time processing or code generation model employed by other engines, making Spark one of the most efficient in the market.

Try Apache Spark 2.1 & Zeppelin in Hortonworks Data Cloud by Vinay Shukla Wanna try Spark 2.1 Now? Well, you are in luck… Hortonworks Data Cloud ("HDCloud") for AWS gives you a quick way to launch a Spark cluster in the cloud. Read More Machine Learning & its Impact on the Future for Insurance by Cindy Maike First and foremost, machine learning WILL change the way insurers do business. The insurance industry is founded on forecasting future events and estimating the value/impact of those events and has used established predictive modeling practices – especially in claims loss prediction and pricing – for some time now. Read More A Reference Architecture for the Open Banking Standard… by Vamsi Chemitiganti Financial services firms specifically deal with manifold data types ranging from Customer Account data, Transaction Data, Wire Data, Trade Data, Customer Relationship Management (CRM), General Ledger and other systems supporting core banking functions. When one factors in social media feeds, mobile clients & other non traditional data types, the challenge is not just one of data volumes but also variety and the need to draw conclusions from fast moving data streams by commingling them with years of historical data.

Apache Spark has become the de facto standard for processing data at scale, whether for querying large datasets, training machine learning models to predict future trends, or processing streaming data. In this article, we'll show you how to use Apache Spark to analyze data in both Python and Spark SQL. And we'll extend our code to support Structured Streaming, the new current state of the art for handling streaming data within the platform. We'll be using Apache Spark 2.2.0 here, but the code in this tutorial should also work on Spark 2.1.0 Before we begin, we'll need an Apache Spark installation.

R is ubiquitous in the machine learning community. Its ecosystem of more than 8,000 packages makes it the Swiss Army knife of modeling applications. Similarly, Apache Spark has rapidly become the big data platform of choice for data scientists. Its ability to perform calculations relatively quickly (due to features like in-memory caching) makes it ideal for interactive tasks--such as exploratory data analysis. R (SparkR) is the latest addition and support for it certainly lags the other three languages. In Spark 1.x there was no support for accessing the Spark ML (machine learning) libraries from R. The performance of R code on Spark was also considerably worse than could be achieved using, say, Scala.