Neural networks are known to develop latent representations that are $aligned$, namely structurally similar across networks trained with different architectures, training protocols, or training datasets. We study this phenomenon in a controlled setting, where we train an ensemble of networks on regression and classification tasks using training sets perturbed by independent realizations of a noise process. We show that the signal-to-noise ratio (SNR) and the training sample size influence the alignment in qualitatively similar ways in networks trained on real-world datasets and in an extremely simple $linear$ network with a single hidden layer, for which the alignment can be estimated analytically. Across linear and nonlinear networks, regression and classification tasks, and both synthetic and real-world data, we consistently observe that alignment varies monotonically with SNR but non-monotonically with training sample size. In particular, the alignment is minimized near the interpolation threshold, and a stronger alignment does not necessarily correspond to better generalization error. These findings reveal a non-trivial dependence of alignment on data quality and quantity, decoupled from generalization performance.

Reproducibility319 The backbone recommendation model, DLRM by Naumov et al. [2019], has an open-source PyTorch320 implementation available on Github which includes an implementation of CE. For CCE you need a321 fast library for K-means. We recommend the open-sourced implementation by Johnson et al. [2019]322 for better performance, but you can also use the implementation in Scikit-learn [Pedregosa et al.,323 2011]. The baseline result should be straightforward to reproduce as we closely follow the instructions324 provided by Naumov et al. [2019]. For the CE methods, we only need to change two functions in325 the code: create_emband apply_emb. We suggest using a class for each CE method; see Figure 3.326 For the random hash function, one could use a universal hash function or numpy.random.randint.327

For each algorithm we report a range, e.g.

However, categorical features present a unique challenge as they require embedding a typically vast vocabulary into a smaller vector space for further calculations.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/