Adaptive Experimental Design and Counterfactual Inference

Yet, experimenters are steadily shifting toward Adaptive Experimental Design (AED) methods with the goal of increasing testing throughput or reducing the cost of experimentation. AED promises to use a fraction of the impressions that traditional A/B/N tests require to yield high confidence inferences or to directly drive business impact. In this paper, we share lessons learned regarding the challenges and pitfalls of naively using adaptive experimentation systems in industrial settings where non-stationarity is the norm rather than the exception. Moreover, we provide perspectives on the proper objectives and system specifications in these settings. This culminates in a high level presentation of an AED framework for counterfactual inference. To provide a robust and flexible tool for experimenters with performance certificates at minimal cost, our methodology combines cumulative gain estimators, always-valid confidence intervals, and an elimination algorithm.