Annotation-Efficient Preference Optimization for Language Model Alignment