We thank the reviewers for their thoughtful and constructive feedback on our manuscript. This should help both contextualize each task's difficulty and illustrate what it involves. Reviewer 3 noted the Section 2 task descriptions could be better presented. We have reformatted it so that "the order We also changed our description of IMP ALA to match Reviewer 5's suggestion. Regarding the task suite, Reviewer 4 raised a thoughtful consideration on whether "most of the findings translate when Some 3D tasks in the suite already have '2D-like' semi-counterparts that do not require navigation, '2D-like' because everything is fully observable and the agent has a first-person point of view from a fixed point, without Spot the Difference level, was overall harder than Change Detection for our ablation models.

We thank the reviewers for their thoughtful and constructive feedback on our manuscript. This should help both contextualize each task's difficulty and illustrate what it involves. Reviewer 3 noted the Section 2 task descriptions could be better presented. We have reformatted it so that "the order We also changed our description of IMP ALA to match Reviewer 5's suggestion. Regarding the task suite, Reviewer 4 raised a thoughtful consideration on whether "most of the findings translate when Some 3D tasks in the suite already have '2D-like' semi-counterparts that do not require navigation, '2D-like' because everything is fully observable and the agent has a first-person point of view from a fixed point, without Spot the Difference level, was overall harder than Change Detection for our ablation models.

We thank the reviewers for their insightful comments and encouraging feedback. Reviewer 1 raises two concerns about speedups which we believe to be based on a misunderstanding. A 2 reduction in this metric seems significant. We will clarify this in the paper. Secondly, the reviewer suspects that Tables 6 and 7 show timings for the slower GLOO backend.

Previous "learning to infer" models were only able to perform reconstruction on 2d slices.

We appreciate suggestions from R6, 7, 8 and will include these in the paper. We have included most competitive methods with comparable settings to ours at the submission time. We will include the shown Algorithm 1. S sampled such that all attributes are present? However, our framework can compose features from any set S by solving Eq (10) even with missing attributes in S . Please notice that they are different.

This paper points out a widespread problem with angular losses, and proposes a simple, elegant scheme to address the problem (regularizing each embedding to lie on a shell), getting moderate but consistent improvements across a range of problem settings and datasets. As pointed out by Reviewer 5, the majority of the theoretical results were already known in Section 3.3 of "Heated-Up Softmax Embedding" (2018, unpublished, https://arxiv.org/abs/1809.04157). That paper, however, did not really propose a solution to the problem, merely noted its existence. Reviewer 5 also complains that the interaction with the Adam optimizer is under-explored in this work. "Improved Deep Metric Learning with Multi-class N-pair Loss Objective," also regularized the L2 norm of embedding vectors (towards 0; see their Section 3.2.2).