Activation Functions in Deep Learning: From Softmax to Sparsemax -- Math Proof

The objective of this post is three-fold. The first part discusses the motivation behind sparsemax and its relation to softmax, summary of the original research paper in which this activation function was first introduced, and an overview of advantages from using sparsemax. Part two and three are dedicated to the mathematical derivations, concretely finding a closed-form solution as well as an appropriate loss function. In the paper "From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification", Martins et al. propose a new alternative to the widely known softmax activation function by introducing Sparsemax. While softmax is an appropriate choice for multi-class classification that outputs a normalized probability distribution over K probabilities, in many tasks, we want to obtain an output that is more sparse.