Reviews: Visualizing the Loss Landscape of Neural Nets

Overview: Visualizing the loss landscape of the parameters in neural nets is hard, as the dimension of parameters in neural network is huge. Being able to visualize the loss landscape in neural net could help us understand how different techniques are helping to shape the loss surface, and how the loss surface "sharpness" vs "flatness" are related to generalization error. Previous works include interpolating a 1D loss path between the parameters of two models to reveal the loss surface along the paths. However, 1D visualization sometimes can be misleading and losing critical information about the surrounding loss surface and local minimums. For 2D visualizations, usually 2 directions are selected for interpolating the loss surface along the 2 directions from an origin. However, as the neural net has a "scale-invariant" problem, which means the weights in different layers can appear in different orders but remain the same effect.