Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning