Do Convolutional Networks Perform Better With Depth?

"Double descent does not happen through depth." The double descent curve, tells that increasing model capacity past the interpolation threshold can lead to a decrease in test error. Increasing neural network capacity through width leads to double descent. How does increase or reduction in-depth play out towards the end? A group of researchers from MIT have attempted to explore this question in their work titled, "Do Deeper Convolutional Networks Perform Better?".