Paper review: CONVERGENT LEARNING: DO DIFFERENT NEURAL NETWORKS LEARN THE SAME REPRESENTATIONS? - A Blog From a Human-engineer-being

This paper is an interesting work which tries to explain similarities and differences between representation learned by different networks in the same architecture. To the extend of their experiments, they train 4 different AlexNet and compare the units of these networks by correlation and mutual information analysis. My discussion: We see that different networks learn similar representations with some level of accompanying uniqueness. It is intriguing to see that, after this paper, these are the unique representations causing performance differences between networks and whether the effect is improving or worsening. Additionally, maybe we might combine these differences at the end to improve network performances by some set of smart tricks.