Best of arXiv -- June 2021

The past month in ML research literature has brought surpising results such as the revival of MLPs as a competitive architecture for Computer Vision or the questioning of Batch Normalization as an all-good innocuous layer. Transformers are also (of course) on the plate: for self supervised learning on vision, as well as for sentence representation techniques and character-level language modelling. This is a monthly selection of recent ML research literature, backed by Zeta Alpha, where we're always keeping a close eye at the latest papers. Why Very simple MLP-based architectures suddenly work way better than they should, this has interesting implications and advances our knowledge about what makes Deep Learning work. Key insights You can probably solve ML by just scaling up. Well okay this is an oversimplification and exaggeration, but Rich Sutton's Bitter Lesson seems to be aging better than fine wine so far.