Revisiting Deep Learning as a Non-Equilibrium Process

Last year, the best paper award for ICLR 2017 went to "Re-thinking Generalization" by Chiyuan Zhang et al. The key take away of his teams discovery is that the nature of Deep Learning systems is remarkably very different from other classical machine learning systems. One of the biggest misunderstanding about Deep Learning is that it is just a higher dimensional form of curve fitting and thus solved from the perspective of optimization techniques. This is incorrect notion can be due to the fact that the way Artificial Neural Networks (ANN) is taught to many is that it is just a larger form of logistic regression. Alternatively, for the more experienced machine learning expert, everything can be framed from the viewpoint of an optimization problem.