Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

A standard ODE solver can be used to solve all the hidden states and final states (output from the neural network), starting from an initial state (input to the neural network). The continuous neural network described in (2) exhibits several advantages over its discrete counterpart described in (1), in terms of memory efficiency, parameter efficiency, explicit control of the numerical error of final output, etc. One missing component in the current Neural ODE network is the various regularization mechanisms commonly employed in discrete neural networks. These regularization techniques have been demonstrated to be crucial in reducing generalization errors, and in improving the robustness of neural networks to adversarial attacks. Many of these regularization techniques are based on stochastic noise injection. For instance, dropout [3] is widely adopted to prevent overfitting; injecting Gaussian random noise during the forward propagation is effective in improving generalization [4, 5] as well as robustness to adversarial attacks [6, 7].