Balancing optimism and pessimism in offline-to-online learning