Michael Manapat: Counterfactual evaluation of machine learning models

PyData Seattle 2015 Machine learning models often result in actions: search results are reordered, fraudulent transactions are blocked, etc. But how do you evaluate model performance when you are altering the distribution of outcomes? I'll describe how injecting randomness in production allows you to evaluate current models correctly and generate unbiased training data for new models. Stripe processes billions of dollars in payments a year and uses machine learning to detect and stop fraudulent transactions. Like models used for ad and search ranking, Stripe's models don't just score---they dictate actions that directly change outcomes.