Neural Networks gone wild! They can sample from discrete distributions now!

In this post you will learn what the Gumbel-softmax trick is. Using this trick, you can sample from a discrete distribution and let the gradients propagate to the weights that affect the distribution's parameters. This trick opens doors to many interesting applications. For start, you can find an example of text generation in the paper GANS for Sequences of Discrete Elements with the Gumbel-softmax Dis.... Training deep neural networks usually boils down to defining your model's architecture and a loss function, and watching the gradients propagate. However, sometimes it's not that simple: some architectures incorporate a random component.