Project Hydrogen Unites Apache Spark with DL Frameworks

The folks behind Apache Spark today unveiled Project Hydrogen, a new endeavor that aims to eliminate barriers preventing organizations from using Spark with deep learning frameworks like TensorFlow and MXnet. It's tough to overstate the impact that Apache Spark has had in the emergent big data field. The software not only replaced MapReduce in the Apache Hadoop stack, but it also presented a unified framework for running data transformation, machine learning, SQL query, and streaming analytic workloads. However, the elegance of Spark's "single framework" approach breaks down when one tries to plug other distributed machine learning frameworks into the loop. While Spark MLlib provides solid machine learning functionality, the data science field is moving extraordinary fast and many data scientist want to explore the capabilities of emerging deep learning frameworks, such as TensorFlow, Keras, Caffe2, MXnet, and others.