When is Offline Two-Player Zero-Sum Markov Game Solvable?