Auto-scaling scikit-learn with Spark

Data scientists often spend hours or days tuning models to get the highest accuracy. This tuning typically involves running a large number of independent Machine Learning (ML) tasks coded in Python or R. Following some work presented at Spark Summit Europe 2015, we are excited to release Scikit-learn integration package for Spark that dramatically simplifies the life of data scientists using Python. Python is one of the most popular programming languages for data exploration and data science, and this is in no small part due to high quality libraries such as Pandas for data exploration or scikit-learn for machine learning. Scikit-learn provides fast and robust implementations of standard ML algorithms such as clustering, classification, and regression. Scikit-learn's strength has typically been in the realm of computing on a single node, though.