DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging

This renders them impractical to use in a wide range of use-cases, limiting who can benefit from them to a handful of big corporations. As an attempt to mitigate this issue, Touvron et al.