Outlier-Robust Group Inference via Gradient Space Clustering