CHOIR: Collaborative Harmonization fOr Inference Robustness