Review for NeurIPS paper: FrugalML: How to use ML Prediction APIs more accurately and cheaply

Additional Feedback: The paper covers the interesting topic of efficient API-reuse and, in general, presents a solid method with promising results. The result section is insightful, but am I missing how the conditional accuracies are estimated. From the paper I extract that you learn a model which performs instance-wise predictions, correct? How much left-out training data of the particular dataset (or other datasets) do you use for this? How easy/difficult is this task and do the results vary on the used datasets?