Small Batch Size Training for Language Models: When Vanilla SGDWorks, and Why Gradient Accumulation Is Wasteful

Open in new window