Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Originality: The method is original in the deep learning literature. Though limitations of ODEs cannot cross paths is quite well-known, this paper views this deficiency from a modeling perspective and removes it while keeping within the ODE framework. Quality & Clarity: The motivations for ANODE are well-explained and the experiments are well-chosen. The prose is very well written, and with many simple visualizations that support their claims. Significance: Given the interest in ODE-based modeling, this work has enough impact for a NeurIPS paper.

This paper connects a well-known result about the limits of diffeomorphisms, and applies it to the recent neural ODE model. The authors do experiments to show how adding extra channels reduces the computational cost of these models as well. R1 makes the valid point that the theoretical result was shown in 1955, and that the engineering trick of making layers wider is resnets existed previously. However, I'd say that the main contribution of this paper is in connecting these ideas to neural ODEs, and giving a possible explanation of why wider layers help in resnets. This paper also pushes forward our practical understanding of training neural ODEs. However, the paper and rebuttal avoided reporting absolute (probably poor) classification results.

We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.