This post is about a simple tool in deep learning toolbox: Autoencoder. It can be applied to multi-dimensional financial time series. Autoencoding is the practice of copying input to output or learning the identity function. It has an internal state called latent space which is used to represent the input. Usually, this dimension is chosen to be smaller than the input(called undercomplete).
Welcome to Part 3 of Applied Deep Learning series. Part 1 was a hands-on introduction to Artificial Neural Networks, covering both the theory and application with a lot of code examples and visualization. In Part 2 we applied deep learning to real-world datasets, covering the 3 most commonly encountered problems as case studies: binary classification, multiclass classification and regression. Now we will start diving into specific deep learning architectures, starting with the simplest: Autoencoders. The code for this article is available here as a Jupyter notebook, feel free to download and try it out yourself.
An Autoencoder is neural network capable of unsupervised feature learning. Neural networks are typically used for supervised learning problems, trying to predict a target vector y from input vectors x. An Autoencoder network, however, tries to predict x from x, without the need for labels. Here the challenge is recreating the essence of the original input from compressed, noisy or corrupted data. The idea behind the Autoencoder is to build a network with a narrow hidden layer between Encoder and Decoder that serves as a compressed representation of the input data.
We have some algorithm that's given a handful of labeled examples, say 10 images of dogs with the label 1 ("Dog") and 10 images of other things with the label 0 ("Not dog")--note that we're mainly sticking to supervised, binary classification for this post. The algorithm "learns" to identify images of dogs and, when fed a new image, hopes to produce the correct label (1 if it's an image of a dog, and 0 otherwise). We have some algorithm that's given a handful of labeled examples, say 10 images of dogs with the label 1 ("Dog") and 10 images of other things with the label 0 ("Not dog")--note that we're mainly sticking to supervised, binary classification for this post. The algorithm "learns" to identify images of dogs and, when fed a new image, hopes to produce the correct label (1 if it's an image of a dog, and 0 otherwise). This setting is incredibly general: your data could be symptoms and your labels illnesses; or your data could be images of handwritten characters and your labels the actual characters they represent.
Non-linear manifold learning enables high-dimensional data analysis, but requires out-of-sample-extension methods to process new data points. In this paper, we propose a manifold learning algorithm based on deep learning to create an encoder, which maps a high-dimensional dataset and its low-dimensional embedding, and a decoder, which takes the embedded data back to the high-dimensional space. Stacking the encoder and decoder together constructs an autoencoder, which we term a diffusion net, that performs out-of-sample-extension as well as outlier detection. We introduce new neural net constraints for the encoder, which preserves the local geometry of the points, and we prove rates of convergence for the encoder. Also, our approach is efficient in both computational complexity and memory requirements, as opposed to previous methods that require storage of all training points in both the high-dimensional and the low-dimensional spaces to calculate the out-of-sample-extension and the pre-image.