EMPEROR: Efficient Moment-Preserving Representation of Distributions

Popular choices such as global average pooling [1] and CLS-style attention pooling [2] are computationally attractive but collapse the underlying distribution of features without guarantees on what information is preserved. This heuristic reduction can hinder interpretability, robustness, and data efficiency, and has motivated alternatives that try to encode more distributional structure [3, 4, 5, 6, 7]. However, most existing approaches emphasize empirical performance over principled recoverability or quantifiable fidelity to the original feature distribution. In this paper, we propose EMPEROR, an Efficient Moment-Preserving Representation of Distributions, that treats a layer's features as samples from a finite positive measure and encodes that measure through its moments. The core idea is to replace ambiguous, high-dimensional moment estimation with sliced moments: we project features onto multiple directions, fit lightweight univariate Gaussian mixture models (GMMs) to each projection, and aggregate the resulting slice parameters into a compact descriptor. Theoretically, sliced moments determine the multivariate measure under mild conditions (via Carleman + Cram er-Wold), and specializing to GMMs yields explicit, stable moment formulas.