Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck

Open in new window