Review for NeurIPS paper: A shooting formulation of deep learning

Summary and Contributions: In this paper the authors propose a novel perspective on parameterizing and training continuous-depth neural networks. The approach taken relies on a couple of ideas and techniques which together make this work quite innovative and different from established deep learning techniques: The authors identify/assert that having constant or piecewise parameterized parameters for continuous-depth neural networks is a shortcoming that is worth solving. Yet, to avoid making the parametrization infinite-dimensional, they propose a "complexity" regularizer on the parameters \Theta(t). While this does not lead to an explicit representation of the parameters, it does constrain their implicit dimensionality if done correctly. By considering the adjoint sensitivity method for the regularized continuous depth dynamics, the authors recognize and describe a set of optimality constraints that need to be fulfilled for \Theta(t) to minimize the training loss.