Towards Distribution-Agnostic Generalized Category Discovery