Quick! Quick! Exploration!: A framework for searching a predictive model on Apache Spark - DataWorks Summit

Research and development of machine learning (ML) algorithms are a hot topic in data analytics. Novel OSS ML libraries are continuously proposed such as Google TensorFlow and XGBoost of Washington U. As choices of ML algorithms and libraries are increasing, model selection is getting a serious pain of data analytics in a bunch of business use cases. Despite the development of ML technologies, achievement of high accuracy essentially requires hyper parameter tuning in big search space. Data scientists have to execute ML algorithms hundreds to thousands times by switching OSS and hyper parameter configurations, which last several days. Data preprocessing is also one of data scientists' big headache because model selection among a bunch of ML OSS requires format conversion and saving the converted data to storage for each OSS.