Thm. 3.2] we can construct an SCM

The coefficients in this RF expansion are optimized to fit the given data of input-output pairs.

This illustrates the dependence of the classifier on the representation's robustness, i.e., the predictor can only be robust if

Other concurrent works address, e.g., learning from soft interventions with polynomial mixing [

However, SBC has drawbacks: The test statistic is somewhat ad-hoc, interactions are difficult to examine, multiple testing is a challenge, and the resulting p-value is not a divergence metric.

Often, however, the causal relationship may be entirely absent or account only for a part of the initially observed correlation.

Independent Component Analysis (ICA) aims to recover independent latent variables from observed mixtures thereof.

This view of the data generating process is also adopted and promoted by popular existing works.

This paper studies the role of activation functions in learning modular addition with two-layer neural networks. We first establish a sharp expressivity gap: sine MLPs admit width-$2$ exact realizations for any fixed length $m$ and, with bias, width-$2$ exact realizations uniformly over all lengths. In contrast, the width of ReLU networks must scale linearly with $m$ to interpolate, and they cannot simultaneously fit two lengths with different residues modulo $p$. We then provide a novel Natarajan-dimension generalization bound for sine networks, yielding nearly optimal sample complexity $\widetilde{\mathcal{O}}(p)$ for ERM over constant-width sine networks. We also derive width-independent, margin-based generalization for sine networks in the overparametrized regime and validate it. Empirically, sine networks generalize consistently better than ReLU networks across regimes and exhibit strong length extrapolation.