We thank the reviewers for their time and constructive feedback on the submission, which we will incorporate to

We observe that Θ is full rank. We find that they are positive-definite as expected. Supervised Differentiable Programming" by Chizat and Bach is an important contribution and we will absolutely Sec 2.2 in V1, V2) are restricted to single-hidden-layer networks. It is still an open research question to determine what are the main factors that determine these performance gaps. We will expand discussion around this.