The following proof is from Daniely and V ardi [15], and we give it here for completeness. By Lemma A.1, there exists a DNF formula We construct such an affine layer in Lemma A.2. At least one of the k size-n slices in z contains 0 more than once. We define the outputs of our affine layer as follows. Pr [z represents a hyperedge ] = n (n 1) ... (n k + 1) null 1 n null Pr null z Z null 1 2 log(n) .

Gaussian, and the weight matrix is non-degenerate.

This is the Picard rank one analogue to Proposition 5.3 in the main text. The number in each dimension is given in Table 1.

Algebraic varieties are the geometric shapes defined by systems of polynomial equations; they are ubiquitous across mathematics and science.

Unfortunately, even for standard linear networks in regression setting, a comprehensive characterization of the implicit bias is still an open problem.

Recent works (Shah et al., 2020; Chen et al., 2021) have demonstrated that neural

Such models range broadly in both complexity and biological plausibility.

We generate EquiDyn through a certain random sampling procedure.