Unlabeled Debiasing in Downstream Tasks via Class-wise Low Variance Regularization