Project Hydrogen -- Combining AI and Big Data Workloads

Project Hydrogen is a major Spark initiative to unify state of the art AI and big data workloads. It enables the support for running deep learning and machine learning frameworks in distributed way on Spark thereby improving the performance and fault recovery of the frameworks. Artificial Intelligence(AI) refers to a set of technologies which enable computers to simulate human intelligence. Their performance can be iteratively improved based on the information they collect. Big data refers to massive, complex and high velocity of data.