Improved prediction of future user activity in online A/B testing

In online randomized experiments or A/B tests, accurate predictions of participant inclusion rates are of paramount importance. These predictions not only guide experimenters in optimizing the experiment's duration but also enhance the precision of treatment effect estimates. In this paper we present a novel, straightforward, and scalable Bayesian nonparametric approach for predicting the rate at which individuals will be exposed to interventions within the realm of online A/B testing. Our approach stands out by offering dual prediction capabilities--it forecasts both the quantity of new customers expected in future time windows and, unlike available alternative methods, the number of times they will be observed. We derive closedform expressions for the posterior distributions of the quantities needed to form predictions about future user activity, thereby bypassing the need for numerical algorithms such as Markov chain Monte Carlo. After a comprehensive exposition of our model, we test its performance on experiments on real and simulated data, where we show its superior performance with respect to existing alternatives in the literature. 1 Introduction The problem of predicting the size of a population from which random samples are drawn has a long history in the statistics literature. Originally motivated by applications in ecology, where the goal is typically to determine the number of distinct species of animals within a population (Fisher et al., 1943; Good, 1953; Burnham and Overton, 1979), a variation of this problem has recently received considerable attention also in the genomics literature, where scientists are interested in predicting the number of future rare variants to be observed within a genomic study (Ionita-Laza et al., 2009; Zou et al., 2016; Chakraborty et al., 2019; Masoero et al., 2022).