For many initialization schemes, parameters of two randomly initialized deep neural networks (DNNs) can be quite different, but feature distributions of the hidden nodes are similar at each layer. With the help of a new technique called {\it neural network grafting}, we demonstrate that even during the entire training process, feature distributions of differently initialized networks remain similar at each layer. In this paper, we present an explanation of this phenomenon. Specifically, we consider the loss landscape of an overparameterized convolutional neural network (CNN) in the continuous limit, where the numbers of channels/hidden nodes in the hidden layers go to infinity. Although the landscape of the overparameterized CNN is still non-convex with respect to the trainable parameters, we show that very surprisingly, it can be reformulated as a convex function with respect to the feature distributions in the hidden layers.

Additional Feedback: Overall, I have the impression that the paper tries to advocate for the fact that the landscape of convolutional neural networks is convex with respect to the features at each layer. Unless I miss some important details in the paper, I think this claim is not true. Even for finite networks, understanding the convexity of the problem with respect to the features at each layer would require some first understanding of the space of features realizable by the network at every intermediate layer, which is also not treated here. In the main paper, the authors present a theoretical result to support their claim. However, if I understand correct the way their proof works is just by a change of variable where the new variables hide the non-convexity of the system. Thus I believe this cannot be used as a valid argument to say that the whole system is convex with respect to the original distribution of interest.

This paper studies the landscape of overparametrized convolutional networks and argues that their training dynamics can be analyzed by comparing the trajectories of feature distributions. Using "network grafting" as a metric, it shows feature distribution trajectories of two networks with the same architecture but different initializations remain close during training. The paper also shows that although the landscape is non-convex with respect to the trainable parameters, it can be reformulated as a convex function with respect to the features. They find that the paper is well written and proposes a novel and appealing perspective for analyzing training dynamics. However, there was lack of clarity about the claim of convexity, which the authors clarified in the rebuttal. I think adding those clarifications to the paper is needed.

For many initialization schemes, parameters of two randomly initialized deep neural networks (DNNs) can be quite different, but feature distributions of the hidden nodes are similar at each layer. With the help of a new technique called {\it neural network grafting}, we demonstrate that even during the entire training process, feature distributions of differently initialized networks remain similar at each layer. In this paper, we present an explanation of this phenomenon. Specifically, we consider the loss landscape of an overparameterized convolutional neural network (CNN) in the continuous limit, where the numbers of channels/hidden nodes in the hidden layers go to infinity. Although the landscape of the overparameterized CNN is still non-convex with respect to the trainable parameters, we show that very surprisingly, it can be reformulated as a convex function with respect to the feature distributions in the hidden layers.