Fast Training of Sinusoidal Neural Fields via Scaling Initialization

Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high training cost, which prevents a broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized to maximize the training speed. We find that the standard initialization scheme for SNFs--designed based on the signal propagation principle--is suboptimal. In particular, we show that by simply multiplying each weight (except for the last layer) by a constant, we can accelerate SNF training by 10 . This method, coined weight scaling, consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory. Neural field (NF) is a special family of neural networks designed to represent a single datum (Xie et al., 2022). Precisely, NFs parametrize each datum with the weights of a neural net which is trained to fit the mapping from spatiotemporal coordinates to corresponding signal values. For neural fields, the training speed is of vital importance. Representing each datum as an NF requires tedious training of a neural network, which can take up to several hours of GPU-based training (Mildenhall et al., 2020). This computational burden becomes a critical obstacle toward adoption to tasks that involve large-scale data, such as NF-based inference or generation (Ma et al., 2024; Papa et al., 2024). To address this issue, many research efforts have been taken to accelerate NF training, including fast-trainable NF architectures (Sitzmann et al., 2020b; Müller et al., 2022), meta-learning (Sitzmann et al., 2020a; Chen & Wang, 2022), and data transformations (Seo et al., 2024).