Inferring the Future by Imagining the Past