Unsupervised Feature Learning and Deep Learning Tutorial

Sparse coding is a class of unsupervised methods for learning sets of over-complete bases to represent data efficiently. While techniques such as Principal Component Analysis (PCA) allow us to learn a complete set of basis vectors efficiently, we wish to learn an over-complete set of basis vectors to represent input vectors \mathbf{x}\in\mathbb{R} n (i.e. The advantage of having an over-complete basis is that our basis vectors are better able to capture structures and patterns inherent in the input data. However, with an over-complete basis, the coefficients a_i are no longer uniquely determined by the input vector \mathbf{x}. Therefore, in sparse coding, we introduce the additional criterion of sparsity to resolve the degeneracy introduced by over-completeness.