A Environment Details A.1 Product Scraping We use ScraperAPI [

Attributes such as "dry skin" can fit the The product title refers to y described in 3. 15 Figure 5: The Amazon Mechanical Turk interface for the instruction writing task. Table 7: Two examples of failed human trajectories. Often, the item does not contain the desired options even though the item's title The task instruction is shown at the end of Appendix. Each trajectory is then manually re-scored against a human criteria; the purpose of this is to determine how representative the reward function is of a human's judgment towards Instead of the exact matching approach, points are awarded if (1) the picked product's On the other hand, if cyan is desired as an option for a goal product, but the user picks blue even though cyan is available as a choice, then no points are awarded. Table 8 reflects our observation that our reward function is similar to a human's score, with a This under-scoring is a result of our reward function's exact matching criterion.