Fine-tuninglanguagemodelstofindagreementamong humanswithdiversepreferences Appendix