Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset

Neural networks are most often trained under the assumption that data come from a stationary distribution.