Learning How to Infer Partial MDPs for In-Context Adaptation and Exploration