Brief Review -- Scaling Language Models: Methods, Analysis & Insights from Training Gopher

RMSNorm (Zhang and Sennrich, 2019) instead of LayerNorm, and The relative positional encoding scheme from Dai et al. (2019) is used rather than absolute positional encodings. Relative encodings permit us to evaluate on longer sequences than that is trained on, which improves the modelling of articles and books. The relative positional encoding scheme from Dai et al. (2019) is used rather than absolute positional encodings. Relative encodings permit us to evaluate on longer sequences than that is trained on, which improves the modelling of articles and books.