The Breakdown of Gaussian Universality in Classification of High-dimensional Mixtures