Explicit Explore-Exploit Algorithms in Continuous State Spaces