Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes