B matrices to avoid training interference.

Despite these achievements, LLMs often underperform in domain-specific tasks without training on relevant domain data.

With Monarch matrices, Kronecker factorizations, and post-training quantization, we achieve non-vacuous generalization bounds for LLMs as large as LLaMA2-70B.

We have open-sourced our code.

LLMs that exploit full model access remain largely unexplored.

The pretraining data of today's strongest language models is opaque; in particular,