Introducing Aardpfark: Exporting Spark ML Models to PFA

The common perception of machine learning is that it starts with data and ends with a model. In real-world production systems, the traditional data science and machine learning workflow of data preparation, feature engineering, and model selection, while important, is just one aspect. A critical missing piece is the deployment and management of models, as well as the integration between the model creation and deployment phases. This is particularly challenging in the case of deploying Apache Spark ML pipelines for low latency scoring. While MLlib's DataFrame API is powerful, elegant, and works well in batch scoring scenarios, it is relatively ill-suited to the needs of many real-time predictive applications, for two main reasons.