On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures