Hybrid Reinforcement Learning from Offline Observation Alone