Pitfalls of Static Language Modelling