Review for NeurIPS paper: What Do Neural Networks Learn When Trained With Random Labels?

Weaknesses: Even though the authors dismiss performing experiments with image augmentations (L 50) as it would introduce a supervisory signal, it could be beneficial to investigate it in the paper. Even though augmentations do add a prior on the expected data distribution, it could be worthwhile to investigate the effect. This is of course another step away from the i.i.d. Along the same lines, I would expect that with increasing kernel size of the convolutions, the correlation between patches increases and with that potentially the misalignment score. If this understanding is correct I would also expect that the experiment in Figure 1 would look very different if only the last layers were transferred instead of the first layers.