We find that they are positive-definite as expected. Supervised Differentiable Programming" by Chizat and Bach is an important contribution and we will absolutely Sec 2.2 in V1, V2) are restricted to single-hidden-layer networks. It is still an open research question to determine what are the main factors that determine these performance gaps. We will expand discussion around this.

We thank the reviewers for their time and constructive feedback to improve the paper. R1: - Depth dependence: We agree that understanding the role of depth is another interesting analysis. This is consistent with [4, Table 1]. We will mention these key architectural details described in the SM in the main text. R2: We thank the reviewer for encouraging and positive feedback on our submission!