TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications

We introduce TabRepo, a new dataset of tabular model evaluations and predictions. TabRepo contains the predictions and metrics of 1206 models evaluated on 200 classification and regression datasets. We illustrate the benefit of our dataset in multiple ways. First, we show that it allows to perform analysis such as comparing Hyperparameter Optimization against current AutoML systems while also considering ensembling at no cost by using precomputed model predictions. Second, we show that our dataset can be readily leveraged to perform transfer-learning. In particular, we show that applying standard transfer-learning techniques allows to outperform current state-of-the-art tabular systems in accuracy, runtime and latency. Machine learning on structured tabular data has a long history due to its wide range of practical applications. Significant progress has been achieved through improving supervised learning models, with key method landmarks including SVM (Hearst et al., 1998), Random Forest (Breiman, 2001) and Gradient Boosted Trees (Friedman, 2001). While the performance of base models is still being improved by a steady stream of research, their performance has saturated and state-of-the-art methods now leverage AutoML techniques (He et al., 2021) or new paradigms such as the pretraining of transformer models (Hollmann et al., 2022). AutoML solutions currently dominate tabular prediction benchmarks (Erickson et al., 2020; Gijsbers et al., 2022). Auto-Sklearn (Feurer et al., 2015a; 2020) was an early approach that proposed to select pipelines to ensemble from the Sklearn library and meta-learn the hyperparameter-optimization (HPO) with offline evaluations.