Gradient-based Training of Slow Feature Analysis by Differentiable Approximate Whitening

Finding meaningful representations in data is a core challenge in modern machine learning as the performance in many goal-directed frameworks such as reinforcement learning or supervised learning is directly and strongly influenced by the quality of the former. Usually, features are either provided (e.g. by expert knowledge) or acquired through learning. Most currently successful approaches for either deep supervised learning [1] or reinforcement learning [2] rely on a training signal, i.e., a classification label or reward signal, to provide sufficient indication which features of the input data should be extracted to increase performance. However, in most real-world scenarios labels have to be acquired by expert knowledge and reward signals are sparse. In unsupervised representation learning one tries to find and apply a principle by which to extract meaning from data without assuming the availability of any goal-driven metrics. Examples for such principles are based on reconstruction error (principal component analysis, autoencoder networks [3]), statistical dependence of extracted features 1 (independent component analysis [4]), indistinguishability of synthetically generated data from samples of the input distribution (generative adversarial nets [5]), fitting the probability distribution of input data (variational autoencoders [6]), (graph-)neighborhood preservation (locally-linear embedding [7], Laplacian eigenmaps [8]), and temporal coherence (slow feature analysis [9], regularized slowness optimization [10]).