Offline RL with Discrete Proxy Representations for Generalizability in POMDPs