LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing

Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often downgrades performance due to the restricted parameter space. In this work, we introduce Latent Crossing (LaX) - a simple yet effective plug-andplay module that enhances the capacity of low-rank models by enabling information flow across low-rank subspaces.