Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice

In active learning (AL), we focus on reducing the data annotation cost from the model training perspective. However, testing'', which often refers to the model evaluation process of using empirical risk to estimate the intractable true generalization risk, also requires data annotations. The annotation cost for testing'' (model evaluation) is under-explored. Even in works that study active model evaluation or active testing (AT), the learning and testing ends are disconnected. In this paper, we propose a novel active testing while learning (ATL) framework that integrates active learning with active testing.