Towards Efficient and Exact Optimization of Language Model Alignment