Is Deeper Better only when Shallow is Good?

While current works account for the importance ofdepth for the expressivepower ofneural-networks, itremains an open question whether these benefits are exploited during a gradient-based optimization process.