Supplemental Material for " Model Selection for Production System via Automated Online Experiments " A Experiment Details