Example Selection For Dictionary Learning

A BSTRACT In unsupervised learning, an unbiased uniform sampling strategy is typically used, in order that the learned features faithfully encode the statistical structure of the training data. In this work, we explore whether active example selection strategies -- algorithms that select which examples to use, based on the current estimate of the features -- can accelerate learning. Specifically, we investigate effects of heuristic and saliency-inspired selection algorithms on the dictionary learning task with sparse activations. We show that some selection algorithms do improve the speed of learning, and we speculate on why they might work. 1 I NTRODUCTION The efficient coding hypothesis, proposed by Barlow (1961), posits that the goal of perceptual system is to encode the sensory signal in such a way that it is efficiently represented. Based on this hypothesis, the past two decades have seen successful computational modeling of low-level perceptual features based on dictionary learning with sparse codes. The idea is to learn a set of dictionary elements that encode "naturalistic" signals efficiently; the learned dictionary might then model the features of early sensory processing.