The paper is well motivated and quite clear. I like the distinction between statistical, functional and heuristics methods of normalization. Also, investigating normalization techniques that do not rely on mini-batch statistics is an important research direction. I have however a few remarks concerning ON: 1) How does it compares to Batch Renormalization (BRN)? Both methods rely on running averages of statistics, so I think it would be fair to clearly state what are the differences between the two methods and to thoroughly compare against it in the experimental setup, especially because BRN introduces 1 extra hyper-parameter, while one need to tune 2 of them in ON. 2) How difficult is it to tune both decay rates hyper-parameters?

The input is a single text file and the model learns to predict the next character in the sequence. Hoping to see many public char-rnn micro-api s with different models spring up, so we can experiment together more easily. This code implements multi-layer Recurrent Neural Network (RNN, LSTM, and GRU) for training/sampling from character-level language models. The input is a single text file and the model learns to predict the next character in the sequence. The context of this code base is described in detail in my blog post.