Limitations of the Empirical Fisher Approximation