BatchNormalizationProvablyAvoidsRankCollapse forRandomlyInitialisedDeepNetworks

Given the rich literature on random matrices, it is not surprising to find that the rank of the intermediate representations in unnormalized networks collapses quickly with depth.