Spark's New Deep Learning Tricks

Imagine being able to use your Apache Spark skills to build and execute deep learning workflows to analyze images or otherwise crunch vast reams of unstructured data. That's the gist behind Deep Learning Pipelines, a new open source package unveiled yesterday by Databricks. Deep Learning Pipelines, which was unveiled at the Spark Summit conference in San Francisco Tuesday, will essentially provide a way to extend the Spark MLlib library to popular deep learning frameworks like TensorFlow and Keras. This will allow Spark users to leverage existing work they've done in MLlib, and to execute deep learning models directly in Spark's existing machine learning library, says Reynold Xin, co-founder and chief architect at Databricks, the commercial outfit behind Apache Spark. "It's a library to integrate essentially all deep learning libraries with Spark to make deep learning substantially easier without having to actually learn about the specifics of deep learning," Xin tells Datanami.