Adaptively Private Next-Token Prediction of Large Language Models