You Only Debias Once: Towards Flexible Accuracy-Fairness Trade-offs at Inference Time