NaNs in Layer Normalization • /r/MachineLearning

I'm experimenting with Layer Normalization on recurrent neural networks (GRUs) and I've noticed that the gradient tends to behave erratically. I've applied gradient norm clipping, but even with that every few updates I get some NaN in the gradient, which I "fix" by throwing the gradient away for that update and shrinking the parameters (that is, instead of updating with the true loss gradient I update with a L2 regularization gradient). Ultimately, the optimization process still makes progress, but I'm wondering if I'm losing some performance because of this issue. Is there any way to mitigate it?