Personalized Collaborative Learning with Affinity-Based Variance Reduction