1 Supplement 1.1 Model Architectures Figure 1: Model Architectures for Latent Integration

Farmworld is an open-ended gridworld environment designed with two goals in mind: high customiz-ability and support for diverse solutions. Maps can be hand-crafted, or randomly generated. RGB images are used, agents'see' exactly what we see: units visibly lose health by damage patterns Agents have partially-observable observations: they do not see the entire map. Reward Agents get an individual reward of 0.1 for each timestep that they are alive. To this end, we augmented DIA YN and called this DIA YN*. We subtract by the batch mean so that on average, the expected agent reward equals only what is provided by the extrinsic environment.