Appendix for " Preference-grounded Token-level Guidance for 657 Language Model Fine-tuning " 658 Table of Contents