Abstract--We introduce NeRFCom, a novel communication system designed for end-to-end 3D scene transmission. Comp ared to traditional systems relying on handcrafted NeRF semanti c feature decomposition for compression and well-adaptive c hannel coding for transmission error correction, our NeRFCom empl oys a nonlinear transform and learned probabilistic models, en abling flexible variable-rate joint source-channel coding and effi cient bandwidth allocation aligned with the NeRF semantic featur e's different contribution to the 3D scene synthesis fidelity. E xperi-mental results demonstrate that NeRFCom achieves free-vie w 3D scene efficient transmission while maintaining robustness under adverse channel conditions. Index T erms --Neural radiance field (NeRF), 3D scene transmission, semantic features, nonlinear transform coding. IRTUAL reality (VR) and augmented reality (AR) construct 3D scenes to provide users with immersive experiences [ 1 ]. However, traditional 3D scene synthesis techniques often rely on manual scene modeling, and the complex workflow increases the cost of deploying 3D technologies.

Information theory and machine learning are inextricably linked and have even been referred to as "two sides of the same coin". One particularly elegant connection is the essential equivalence between probabilistic generative modeling and data compression or transmission. In this article, we reveal the dual-functionality of deep generative models that reshapes both data compression for efficiency and transmission error concealment for resiliency. We present how the contextual predictive capabilities of powerful generative models can be well positioned to be strong compressors and estimators. In this sense, we advocate for viewing the deep generative modeling problem through the lens of end-to-end communications, and evaluate the compression and error restoration capabilities of foundation generative models. We show that the kernel of many large generative models is powerful predictor that can capture complex relationships among semantic latent variables, and the communication viewpoints provide novel insights into semantic feature tokenization, contextual learning, and usage of deep generative models. In summary, our article highlights the essential connections of generative AI to source and channel coding techniques, and motivates researchers to make further explorations in this emerging topic.

The emerging field semantic communication is driving the research of end-to-end data transmission. By utilizing the powerful representation ability of deep learning models, learned data transmission schemes have exhibited superior performance than the established source and channel coding methods. While, so far, research efforts mainly concentrated on architecture and model improvements toward a static target domain. Despite their successes, such learned models are still suboptimal due to the limitations in model capacity and imperfect optimization and generalization, particularly when the testing data distribution or channel response is different from that adopted for model training, as is likely to be the case in real-world. To tackle this, we propose a novel online learned joint source and channel coding approach that leverages the deep learning model's overfitting property. Specifically, we update the off-the-shelf pre-trained models after deployment in a lightweight online fashion to adapt to the distribution shifts in source data and environment domain. We take the overfitting concept to the extreme, proposing a series of implementation-friendly methods to adapt the codec model or representations to an individual data or channel state instance, which can further lead to substantial gains in terms of the bandwidth ratio-distortion performance. The proposed methods enable the communication-efficient adaptation for all parameters in the network without sacrificing decoding speed. Our experiments, including user study, on continually changing target source data and wireless channel environments, demonstrate the effectiveness and efficiency of our approach, on which we outperform existing state-of-the-art engineered transmission scheme (VVC combined with 5G LDPC coded transmission).