Often a task is a composition of previously learned tasks (e.g.

While previousbenchmarks insimilar domains havebeenprimarily focusing onthelanguage grounding of object properties (e.g., "table"), relations (e.g., "on"), and planning (e.g., object search and manipulation) [6,7],inthispaper,wehighlights theadditional challenge forunderstanding human instructions withambiguities (i.e., recognizing the subgoal) based on physical states and human actionsandgoals. Each episode in HandMeThat contains twostages.

In this paper, we show that this task can be addressed effectively by leveraging instead 2D pre-trained models for instance segmentation.

Table A: Configuration of the main parameters used for executing each episode of the PIN task containedinthePInNEDdataset.

Instance segmentation, which seeks to obtain both class and instance labels for each pixelinthe input image, isachallenging task incomputer vision. State-ofthe-art algorithms often employ a search-based strategy, which first divides the output image with a regular grid and generate proposals at each grid cell, then the proposals are classified and boundaries refined.

Farmworld is an open-ended gridworld environment designed with two goals in mind: high customiz-ability and support for diverse solutions. Maps can be hand-crafted, or randomly generated. RGB images are used, agents'see' exactly what we see: units visibly lose health by damage patterns Agents have partially-observable observations: they do not see the entire map. Reward Agents get an individual reward of 0.1 for each timestep that they are alive. To this end, we augmented DIA YN and called this DIA YN*. We subtract by the batch mean so that on average, the expected agent reward equals only what is provided by the extrinsic environment.