On the information bottleneck theory of deep learning

Last week we looked at the Information bottleneck theory of deep learning paper from Schwartz-Viz & Tishby (Part I,Part II). I really enjoyed that paper and the different light it shed on what's happening inside deep neural networks. Sathiya Keerthi got in touch with me to share today's paper, a blind submission to ICLR'18, in which the authors conduct a critical analysis of some of the information bottleneck theory findings. Sathiya gave a recent talk summarising results on understanding optimisation and generalisation, 'Interplay between Optimization and Generalization in DNNs,' which is well worth checking out if this topic interests you. Definitely some more papers there that are going on my backlog to help increase my own understanding!