A Reference LPF methods on AlpacaFarm 568 Having defined and validated the pairwise feedback simulator and evaluations in AlpacaFarm, we

A.1 Methods that directly learn from pairwise feedback To start, we describe the step of training the surrogate reward model. We adapt this approach in AlpacaFarm as a two-step method. In Appendix F, we include our preliminary study of multi-round expert iteration. We find exactly this result with the simulator. Figure 5: Our simulated annotators are cheap and match well with human annotators.