Is Deeper Better only when Shallow is Good?