Towards Enabling Meta-Learning from Target Models

Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. It is often realized through optimizing a meta-model with the evaluation loss of task-specific solvers. Most existing algorithms sample non-overlapping \mathit{support} sets and \mathit{query} sets to train and evaluate the solvers respectively due to simplicity ( \mathcal{S} / \mathcal{Q} protocol). Different from \mathcal{S} / \mathcal{Q} protocol, we can also evaluate a task-specific solver by comparing it to a target model \mathcal{T}, which is the optimal model for this task or a model that behaves well enough on this task ( \mathcal{S} / \mathcal{T} protocol). Although being short of research, \mathcal{S} / \mathcal{T} protocol has unique advantages such as offering more informative supervision, but it is computationally expensive.