We propose a novel extension of the encoder-decoder framework, called a review network. The review network is generic and can enhance any existing encoderdecoder model: in this paper, we consider RNN decoders with both CNN and RNN encoders. The review network performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a thought vector after each review step; the thought vectors are used as the input of the attention mechanism in the decoder. We show that conventional encoder-decoders are a special case of our framework.

First, as R4 suggested, "symbolic35 tree" was more approachable for people in the ML community. Second, the symbolic tree is declared by the user using36 decorators and serves to represent high-level program constructs, which is different from the AST that represents all37 the syntactic structures for the program. For example, the full Python AST contains information about objects' class38 methods, whereas our symbolic representation does not.39 R4: "Second, most of their tool/language design could be summarized as adding some kind of non determinis-40 tic/parametric choice ... It's extension to ML does not introduce anything particularly new ..."41 We agree with R4 that symbolic programming and non-deterministic programming are well-studied topics in the PL42 community. However, we would like to emphasize that this work is the first to introduce such concepts to AutoML43 to significantly reduce engineering effort, which is a novel and useful contribution. For example, PyGlove leverages44 symbolic manipulation to decouple the search algorithm, search space and child program, which enabled us to unify45 the interface among search methods with and without weight sharing. To enable symbolic programming in Python,46 PyGlove implements an object model for maintaining the consistency of program state during symbolic manipulation.47 R4 "Provide the grammar in the main text"48 We understand the "grammar" here as a reference to the formal definition of the search space specification. We will49 revise current Appendix Table 3 into a formal definition, and add it to the "search space" sub-section.50

We thank the reviewers for their insightful feedback, and we appreciate the opportunity to improve our paper. We will1 address typos and notational inconsistencies in the updated version.2 Response to Reviewer 1:3 We would like to emphasize that Theorem 1 is the most important contribution of our paper due to its generality.4 By considering the set of all possible classifiers, it provides lower bounds on adversarial robustness for any pair of5 class-conditional distributions. As we show in our experimental results in Section 6, we are able to obtain lower bounds6 for arbitrary real-world datasets by constructing the empirical distribution for these. In our estimation, these results7 serve to provide theoretical validation for adversarial training for low perturbation budgets as well as to highlight the8 gap to optimality for higher budgets.9

We thank the reviewers for their encouraging and constructive comments. We are pleased that they find the paper well1 written and acknowledge the novelty and originality of the proposed task, which "has a potential to spark interest"2 (R1) and "may lead to future papers studying it" (R2). Regarding the proposed framework, R1 and R2 not only find it3 "sound" and "novel" but also stress the "re-implementation ease" from which "practitioners may benefit" (R1). Still,4 the reviewers raise points of improvement (R1, R3) and suggest a discussion about a related task (R2). We carefully5 address these comments below.

Figure 1: (a) Precision (blue) and recall (orange) for Figure 2: (a) Real data covers five modes (1-5) and several neighborhood sizes k. Both metrics were evaluated using 20k real and of varying sample count. Figure 1a illustrates the effect of varying k in the setup used in Figure 4b of the submission (truncation sweep 4 in StyleGAN, VGG-16 features, 50k samples). In general, different k yield consistent results and affect mainly the 5 saturation towards 0 or 1. Therefore, selecting k is a tradeoff between under-or overestimating the manifolds.