Invariance, Causality and Robust Deeplearning

Why do Neural networks fail to generalize to new environments, and how can this be fixed? Many real world data analysis problems exhibit in-variant structure, and models that take advantage of this structure have shown impressive empirical performance, particularly in deep learning. Most machine learning problems have an invariant structure. Image classification tasks, for example, are usually invariant to translation, rotation, scale, viewpoint, illumination etc. An example of statue class is shown below. It seems intuitive the machine learning models should capture the invariances of the problem at hand to perform better. We will look at why is it so in details below. Anyways, there are many works that empirical support of this over range of applications (Cohen & Welling, 2016; Fawzi et al., 2016; Salamon & Bello, 2017).