Universal flow approximation with deep residual networks

Since then, they have received continuously growing attention. ResNets have a recursive structure x k 1 x k R k( x k) where R k is a neural network and the copying of the input x k is called a skip connection. This structure can be seen as the explicit Euler discretisation of an associated ordinary differential equation (ODE) and this inspired intensive research. However, all of those works only consider the connection of ResNets to a relatively small class of ODEs. We show that by simultaneously increasing the number of skip connection as well as the expressivity of the networks R k the flow for an arbitrary right hand side f L 1 null I; C 0, 1 b (R d; R d)null can be approximated uniformly by deep ReLU ResNets on compact sets. Further, we derive estimates on the number of parameters needed to do this up to a prescribed accuracy under temporal regularity assumptions. We also give a self-contained introduction to the preliminaries regarding neural networks and differential equations. Here, we give an elementary proof for a quantitative universal approximation theorem for deep ReLU networks and see that weak ODEs with right hand side in L 1null I; C 0, 1 b (R d; R d)null are globally well posed. Finally, we discuss the possibility of using ResNets for diffeomorphic matching problems and propose some next steps in the theoretical foundation of this approach.