Diffusing Differentiable Representations