We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric latent prior based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling. Experiments demonstrate the effectiveness of Exemplar VAEs on density estimation and representation learning. Importantly, generative data augmentation using Exemplar VAEs on permutation invariant MNIST and Fashion MNIST reduces classification error from 1.17% to 0.69% and from 8.56% to 8.16%.

Figure 1: Random samples drawn from Exemplar V AEs trained on different datasets. Exemplar V AE are generated and shown. Define Cache: initialize cache = [] insert( i, c): insert value c with index i into cache update( i, c): update the value of index i to c kNN(c): return indices of kNNs of c in cache for n in {1,...,N } do Cache.insert( Table 1: The number of active dimensions computed based on a metric proposed by Burda et. The exemplar V AE generates a new sample by stochastically transforming an exemplar.

We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric latent prior based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling.

Additional Feedback: I read all the reviews and the rebuttal. I agree with the authors that the proposed method is different from learned pseudo-exemplars in the embedding space as in VampVAE, and this work uses real exemplars in the image space. However, I am not convinced that randomly sampling exemplars in the data space with some heuristics based on LOO and trivial exemplar subsampling as regularizations on toy datasets is a significant contribution extending the exemplar-based prior in VampVAE. A possible limitation of the proposed Exemplar VAE is that, the generative model might not learn much beyond reconstruction, instead, it only produces some random samples that stay close to epsilon-ball of training data points. It's possible that Exemplar VAE even performs no better than a deterministic autoencoder with tiny Gaussian noise added to latent codes and k-means regularization in the latent space. VampVAE doesn't have this issue.

And since the ideas are rather incremental Meta-reviewer recommendations: The paper is borderline. R2 is considering increasing the score. R4 recommends rejection based on the lack of novelty compared to the VampPrior and that the paper conducts small-scale non-challenging experiments that doesn't require approximate nearest-neighbor search. He proposes to run the method on ImageNet but I believe this cannot be a condition for acceptance since not everyone has the potential for running such experiments. Furthermore, the paper already covers quite a lot in experiments with MNSIT, Omniglot, FashionMNIST and CelebA, and classification. I believe that R4's concern for novelty are successfully addressed in the rebuttal.

Recent efforts in fine-tuning language models often rely on automatic data selection, commonly using Nearest Neighbors retrieval from large datasets. However, we theoretically show that this approach tends to select redundant data, limiting its effectiveness or even hurting performance. To address this, we introduce SIFT, a data selection algorithm designed to reduce uncertainty about the model's response given a prompt, which unifies ideas from retrieval and active learning. Whereas Nearest Neighbor retrieval typically fails in the presence of information duplication, SIFT accounts for information duplication and optimizes the overall information gain of the selected examples. We focus our evaluations on fine-tuning at test-time for prompt-specific language modeling on the Pile dataset, and show that SIFT consistently outperforms Nearest Neighbor retrieval, with minimal computational overhead. Moreover, we show that our uncertainty estimates can predict the performance gain of test-time fine-tuning, and use this to develop an adaptive algorithm that invests test-time compute proportional to realized performance gains. We provide the $\texttt{activeft}$ (Active Fine-Tuning) library which can be used as a drop-in replacement for Nearest Neighbor retrieval.

We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric latent prior based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower bound on log marginal likelihood. To enhance generalization, model parameters are learned using exemplar leave-one-out and subsampling.