This video was recorded at Scala Days Berlin 2016 follow us on Twitter @ScalaDays or visit our website for more information http://scaladays.org Abstract: What new superpowers does it give me? The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting -- in several different frameworks. We'll show what it's like to work with Spark.ml, and compare it to other widely used frameworks (in R and python) along several dimensions: ease of use, productivity, feature set, and performance.

Alexy Khabrov's talk at LX Scala caused the Scala community to raise their arms for a revolution. After his talk, Alexy discussed his thoughts on how to ease the transition for Python data scientists into the Scala community, what Scala can learn from Python as a programming language, and how machine learning will influence the future of the Scala community within the next five years.

SQL is referred to as a declarative language as opposed to an imperative language like the 3GL's. You give it a high-level goal, and it figures out which machine learning algorithm to use, and tunes the hyperparameters for you. Are there other declarative machine learning systems out there? Their purpose is to allow non-Spark developers to write machine learning programs in languages they are comfortable in (like Python), yet be able to compile down to Spark Scala when the time comes to deploy to production.