Lifting the Veil: Unlocking the Power of Depth in Q-learning