A random energy approach to deep learning

The study of ensembles of random systems can provide several insights on the properties of complex systems, such as heavy ions [29], ecologies [15], disordered materials [17], satisfiability in computer science [18] and machine learning [30]. Indeed the collective behaviour of a system composed of many interacting degrees of freedom often does not depend on the specific realisation of the wiring of the interactions, but only on the statistical properties of the resulting energy landscape. In these circumstances, any realisation of a random system that shares the same statistical properties enjoys the same "typical" collective behaviour. The Random Energy Model (REM) [6] is probably the simplest exemplar of this approach. It makes minimal assumptions on the network of interactions, because interactions of any order can occur among the variables [6]. It features a phase transition between a random (high temperature) phase and a low temperature frozen phase, which reproduces the gross features of more complex systems such as spin glasses.