Robust Adverse Weather Removal via Spectral-based Spatial Grouping