Sharp Analysis for KL-Regularized Contextual Bandits and RLHF