In order to find the best performing network, I'm training it with many different randomly selected hyper parameters. While deep and wide architectures obviously overfit extremely, they also yield slighty better test performance than other architectures. For example, a network with 4 hidden layers with 800, 960, 1150 and 1380 units respectively results in a mean squared error of 3.28 while an architecture with only 2 hidden layers and 300/210 units results in a MSE of 3.32. How can this be explained? I don't feel like using an overfitted network is sensible, however it does show better test performance than less overfitted networks.