Appendix: On Infinite-Width Hypernetworks

To further demonstrate the behavior reported in Figure 1 (main text), we verified that it is consistent regardless of the value of the learning rate. However, for earlier epochs, the performance improves for shallower and wider architectures.(a) As a consequence of Thm. 1, we prove that Sec. 3, terms of the form in Eq. 5 represent high order terms in the multivariate Taylor expansion of As a consequence of Thm. 1, we prove that In this section, we prove Lem. 3, which is the main technical lemma that enables us proving Thm. 1. To estimate the order of magnitude of the expression in Eq. 7, we provide an explicit expression for By Eqs. 14 and 10, we see that: T Lemma 2. The following holds: 1. F or n Lemma 3. Let k 0 and sets l = {l The case k = 0 is trivial. By Eq. 16, it holds that: n Lemma 4. Let h(u; w) = g (z; f ( x; w)) be a hypernetwork.