Probabilistic Insights for Efficient Exploration Strategies in Reinforcement Learning