Long-time asymptotics of noisy SVGD outside the population limit