On the Anisotropy of Score-Based Generative Models

We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture's output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models. The only difference is the choice of subspace: the left "sphere" lies in a subspace aligned with the network's geometry, G Despite identical setups, their quality differs consistently across repeated trials, suggesting that alignment with architectural geometry controls generalization. Neural networks generalize through inductive biases, i.e., biases that guide learning beyond training data (Goyal & Bengio, 2020; Wilson & Izmailov, 2020).