The low-resource label analysis and representation visualization further illustrate the effectiveness of our pre-training strategy.

Despite their prevalence in deep-learning communities, over-parameterized models convey high demands of computational costs for proper training.

This simple generative task requires a model to copy binary symbols over long-range dependencies, ignoring the tokens in between. We find that Transformer FFLMs suffer from a long tail of sporadic reasoning errors, some of which we can eliminate using various regularization techniques.