Neural Compression of Atmospheric States

This paper presents a family of neural network compression methods of simulated atmospheric states, with the aim of reducing the currently immense storage requirements of such data from cloud scale (petabytes) to desktop scale (terabytes). This need for compression has come about over past 50 years, characterized by a steady push to increase the resolution of atmospheric simulations, increasing the size and storage demands of the resulting datasets (e.g., Neumann et al. (2019), Schneider et al. (2023), Stevens et al. (2024)), while atmospheric simulation has come to play an increasingly critical role in scientific, industrial and policy-level pursuits. Higher spatial resolutions unlock the ability of simulators to deliver more accurate predictions and resolve ever more atmospheric phenomena. For example, while current models often operate at 25 - 50 km resolution, resolving storms requires 1 km resolution (Stevens et al., 2020), while resolving the motion of (and radiative effects due to) low clouds require 100 m resolution (Satoh et al., 2019; Schneider et al., 2017). Machine learning models for weather prediction also face opportunities and challenges with higher resolution: while additional granularity may afford better modeling opportunities, even the present size of atmospheric states poses a significant bottleneck for loading training data and serving model outputs (Chantry et al., 2021). To put the data storage problem in perspective, storing 40 years of reanalysis data from the ECMWF Reanalysis v5 dataset (ERA5, Hersbach et al. (2020)) at full spatial and temporal resolution (i.e.