Autoencoder Node Saliency: Selecting Relevant Latent Representations

The autoencoder is an artificial neural network that learns hidden representations of unlabeled data. With a linear transfer function it is similar to the principal component analysis (PCA). While both methods use weight vectors for linear transformations, the autoencoder does not come with any indication similar to the eigenvalues in PCA that are paired with eigenvectors. We propose a novel supervised node saliency (SNS) method that ranks the hidden nodes, which contain weight vectors for transformations. SNS is able to indicate the nodes specialized in a learning task. The latent representations of a hidden node can be described using a one-dimensional histogram. We apply normalized entropy difference (NED) to measure the "interestingness" of the histograms, and conclude a property for NED values to identify a good classifying node. Keywords: networks, node selection Autoencoder, latent representations, unsupervised learning, neural By 1. Background and Motivation The autoencoder is an artificial neural network model that aims to find an encoding for a dataset in a reduced dimension [1]. The model is unsupervised because class labels (i.e. The encoding of autoencoders constructs a powerful representation and often learns useful properties of the data [2, 3]. The unsupervised feature extraction provided by the encoding of autoencoders is a key factor in the success of pattern recognition [2, 4, 5, 6, 7, 8, 9]. For example, theoretical studies [10] suggest that we may need deep architectures to efficiently model complex distributions and obtain better performance on challenging pattern recognition tasks. Training the autoencoders becomes a successful approach to solving the difficult optimization problem, which arises from building a multi-layer neural network [11, 12, 13, 10].