Efficient Offline Reinforcement Learning: The Critic is Critical