An empirical study of the relation between network architecture and complexity

Although we lack a rigorous definition of complexity, many agree that certain factors contribute to the complexity in a classification problem including: the dataset size in relation to the dimensionality of the data, the intrinsic ambiguity of the classes, and how compactly the decision boundary can be expressed [1]. The capacity of a network describes the complexity of the functions it can potentially model. Naturally, data with high complexity require a model with high effective capacity. In recent findings counter to the conventional wisdom, Zhang et al. [2] showed that high effective capacity models can both memorize and generalize well whereas Neyshabur et al. [3] showed that networks with higher capacity also generalize better. 1 In this paper, we perform an empirical study to characterize how generalization error relates to network capacity when the complexity of the data changes . Our aim is to improve our understanding of the capacity of various deep neural architectures and potentially help guide the design process. Instead of utilizing theoretical bounds to calculate an architecture's capacity [4] or measures based on the 1 We are concerned with a low generalization error, not a small generalization gap (the difference between test and training error).