Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning

The recent surge of interest in few-shot learning (FSL), driven by its potential applications in many real-world scenarios, has led to a proliferation of new methods and novel experimental protocols (Sung et al., 2018; Snell et al., 2017; Bendou et al., 2022; Zhang et al., 2020). If in conventional machine learning it is common to benchmark methods using a fixed split into training and validation sets, FSL presents unique challenges due to its reliance on extremely small and, consequently, biased training datasets. In fact, the performance of FSL can dramatically depend on the choice of the given labeled training samples (Arnold et al., 2021). One question that FSL shares with conventional machine learning is that of the best performing methods. Especially relevant to FSL, the high variance of measured performance based on the choice of labeled data has led practitioners to quickly adopt the standard of aggregating statistics over a large number of artificially generated tasks, stemming from a single (or a few distinct) dataset(s). The predominant approach is to generate artificial few-shot tasks by randomly sampling the same dataset with replacement, i.e. permitting the same samples to appear across multiple tasks. The outcome of these numerous tasks is the calculation of an average accuracy and its associated confidence interval (CI) for each method, thereby providing researchers with a statistically relevant basis for comparing the efficacy of different methods. By allowing the same samples to appear in multiple tasks, the computed CIs account for the randomness of the sampler but not the data itself.