Symmetry Guarantees Statistic Recovery in Variational Inference