Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

A deep neural network (DNN) with piecewise linear activatio ns can partition the input space into numerous small linear regions, where diffe rent linear functions are fitted. It is believed that the number of these regions rep resents the expressivity of the DNN. This paper provides a novel and meticulous perspe ctive to look into DNNs: Instead of just counting the number of the linear regio ns, we study their local properties, such as the inspheres, the directions of t he corresponding hyper-planes, the decision boundaries, and the relevance of the su rrounding regions. W e empirically observed that different optimization techniq ues lead to completely different linear regions, even though they result in similar cl assification accuracies. W e hope our study can inspire the design of novel optimizatio n techniques, and help discover and analyze the behaviors of DNNs. In the past few decades, deep neural networks (DNNs) have ach ieved remarkable success in various difficult tasks of machine learning (Krizhevsky et al., 2012; Graves et al., 2013; Goodfellow et al., 2014; He et al., 2016; Silver et al., 2017; Devlin et al., 2019). Albeit the great progress DNNs have made, there are still many problems which have not been thoro ughly studied, such as the expressivity and optimization of DNNs. High expressivity is believed to be one of the most important reasons for the success of DNNs. It is well known that a standard deep feedforward network with pie cewise linear activations can partition the input space into many linear regions, where different li near functions are fitted (Pascanu et al., 2014; Montufar et al., 2014). More specifically, the activat ion states are in one-to-one correspondence with the linear regions, i.e., all points in the same li near region activate the same nodes of the DNN, and hence the hidden layers serve as a series of affine transformations of these points.