Rethinking Crystal Symmetry Prediction: A Decoupled Perspective