Normalizing flows for SU($N$) gauge theories employing singular value decomposition