Properties of the Concrete distribution

Like the more well-known Dirichlet distribution, it includes the uniform distribution on the simplex, but is otherwise distinct from the Dirichlet distribution. There is a temperature parameter, named by analogy with the Boltzmann (or Gibbs) distribution in thermodynamics. In the zero temperature limit, samples become concentrated at the corners of the simplex and the distribution approximates a categorical distribution: the limit of a continuous distribution approximates a discrete distribution, hence the portmanteau "Concrete". There are additionally K parameters that are easily interpreted as unnormalized probabilities for the K categories in this limit; a single constraint reduces these to K 1 independent parameters. The Concrete distribution has applications in machine learning for stochastic neural networks with discrete random variables, such as random sampling from discrete distributions and estimation of parameter gradients through backpropagation. One way of constructing the Concrete distribution is from combining Gumbel distributions through the softmax function, hence its alternative name of the Gumbel-softmax distribution. The differentiability of the softmax function, unlike the argmax function, allows for backpropagation.