Weaknesses: (a) The paper falls short in providing a clear theoretical backing for using the U-Net structure in the generator. The explanation in Section 2.4 seems to suggest that the U-Net structure was selected due to the benefits of its bottleneck passing low-level features, but later states that it is only passing high-level features. Clarification of this section is needed. As the brain encodes the LVC signals into the HVC signals, the information from HVC signals should also be contained in the LVC signals. Why then, do we need the high level representation for the decoding problem?

This paper was well received, and the reviewers praised it for its clarity and contributions. The idea to separate shape and semantics in the reconstruction is an interesting approach, and one that has proven quite useful. This work is likely of interest to a wide range of the NeurIPS community, those interested in computer vision as well as neuroscience. The reviewers pointed out a few places that could be clearer, but these points (for the most part) could be handled in minor revisions. One point remained unanswered about the loss of information in the HVC, and was discussed by the reviewers after seeing the rebuttal.