ASMR: Activation-sharing Multi-resolution Coordinate Networks For Efficient Inference

Department of Computer Science, University of Toronto jasonlcl@connect.hku.hk Coordinate network or implicit neural representation (INR) is a fast-emerging method for encoding natural signals (such as images and videos) with the benefits of a compact neural representation. While numerous methods have been proposed to increase the encoding capabilities of an INR, an often overlooked aspect is the inference efficiency, usually measured in multiply-accumulate (MAC) count. This is particularly critical in use cases where inference throughput is greatly limited by hardware constraints. To this end, we propose the Activation-Sharing Multi-Resolution (ASMR) coordinate network that combines multi-resolution coordinate decomposition with hierarchical modulations. Specifically, an ASMR model enables the sharing of activations across grids of the data. This largely decouples its inference cost from its depth which is directly correlated to its reconstruction capability, and renders a near O(1) inference complexity irrespective of the number of layers. Experiments show that ASMR can reduce the MAC of a vanilla SIREN model by up to 500 while achieving an even higher reconstruction quality than its SIREN baseline. Neural networks have been proven to be very effective at learning representations of various modalities of data such as images, videos, 3D shapes, neural fields, and many more. In particular, Sitzmann et al. (2020); Mildenhall et al. (2021); Park et al. (2019); Li et al. (2024) have demonstrated that simple coordinate networks, taking in a coordinate system and outputting the modality-specific data, exhibit state-of-the-art (SOTA) expressivity as an implicit neural representation (INR). However, while numerous methods have been proposed to improve the encoding capabilities of an INR, an aspect that is often overlooked is the network's cost of inference Currently, hybrid INRs that make use of explicit representations such as Plenoxels (Fridovich-Keil et al., 2022) and Instant-NGP (Müller et al., 2022) are the best at low-cost inference as they remove the need to learn a complex neural network.