Databricks brings deep learning to Apache Spark

Databricks is giving users a set of new tools for big data processing with enhancements to Apache Spark. The new tools and features make it easier to do machine learning within Spark, process streaming data at high speeds, and run tasks in the cloud without provisioning servers. On the machine learning side, Databricks announced Deep Learning Pipelines, which are designed to make it possible for data scientists and AI novices to implement neural nets in their big data processing. It provides developers with high-level APIs designed to help with tasks like loading images, tuning a model's hyperparameters, and modifying a more general model to help in a specific case. The company has also integrated Spark's Structured Streaming feature into the beta of its enterprise product to accelerate the processing of real-time data.