Benign overfitting in leaky ReLU networks with moderate input dimension

Intuition from learning theory suggests that fitting noise during training reduces a model's performance on test data.