A Natural Extension To Online Algorithms For Hybrid RL With Limited Coverage