Summary and Contributions: Neural template models [1, 2] are interesting with interpretability and controllability. Such models can be trained in a VAE framework with CRF as the posterior. Due to the non-differentiability of discrete templates, this paper investigates using Gumbel-Softmax as the gradient estimator for the posterior distribution against other gradient estimators such as REINFORCE and PM-MRF used by previous work. Empirically, the gumbel estimator demonstrates lower variance and better performance on unsupervised paraphrasing and data-to-text generation than comparable baselines. ACL 2020 ------------ After Rebuttal -------------- Thank you for the response!

The authors propose a general framework for designing new MCMC samplers, including methods that use stochastic gradients. Their approach is to define a stochastic dynamical system whose stationary distribution is the target distribution from which we want to sample from. The stochastic dynamical system is represented through a stochastic differential equation that is simulated through an epsilon-discretization approach. As the step-size parameter epsilon goes to zero, the bias in the simulation vanishes. The proposed approach can handle stochastic approximations obtained by sub-sampling the data in mini-batches.

This paper is an essentially theoretical contribution regarding convergence rates for the so-called "Hogwild"-style algorithms for stochastic gradient descent. In these algorithms, the gradient step is produces asynchronously over different chunks of the dataset in parallel, with results updating current weights as they are completed, independent of other parallel updates. Previously, demonstrating theoretical convergence has been difficult and somewhat brittle. They show that one of their proven variants, "Buckwild" provides significant real-world speedups by using lower precision arithmetic to compute the gradient steps. As far as the paper goes, it is generally good. I had little trouble reading and understanding the paper (I think), and they make a point to explain the maths in an intuitive fashion, insofar as it is possible.

Summary: This paper analyzes the stochastic version of normalized version of normalized gradient descent (NGD), which is the first effort to explore the efficacy and property of stochastic normalized gradient descent (SNGD). In order to verify the benefits of NGD in training non-convex optimization problems, this paper introduces a new property, local-quasi-convexity, to prove its convergence to a global minimum. Particularly, they prove that NGD finds an \epsilon-optimal minimum for local quasi convex functions within O(1/ \epsilon 2) iterations. In addition, this paper introduces a new setup: stochastic optimization of locally-quasi convex functions, in which the gradient is estimated using a minibatch of examples. Empirically, this paper reports experimental results by training deep neural networks by comparing with the-state-of-the-arts methods, minibatch SGD and Nesterov's accelerated gradient method.

Summary and Contributions: This paper consider a regression with reject option problem, where one may abstain from predicting at some "hard" instances, with an emphasis on the case where the rejection (abstention) rate is prescribed. The first contribution is a characterization of the optimal prediction rule (knowing the true distribution of the data) given the rejection rate epsilon, which is obtained by predicting using the regression function, and abstaining when the conditional variance at the input point exceeds its (1-epsilon)-quantile. (This is done by first considering a variant where rejection is associated to a fixed penalty, then using the standard correspondence between penalized and constrained problems.) Motivated by this characterization, the authors propose a plug-in approach, which relies on (1) an estimator of the regression function, (2) an estimator of the conditional variance and (3) an estimator of the quantiles of the conditional variance (taken to be the empirical quantile of the estimated conditional variance on a separate set of data inputs). This plug-in approach is shown to be "consistent" (in that its prediction accuracy and rejection rate converge to that of the best predictor with prescribed rejection rate), provided that the previous estimators are consistent in appropriate senses (L 2 for regression function and L 1 for the conditional variance). Finally, the plug-in approach is applied to the k-Nearest Neighbors (k-NN) algorithm, for which nonparametric rates of convergence for Lipschitz regression function and conditional variance (and some "margin condition" describing the mass of the conditional variance around the optimal threshold) are provided using convergence rates of k-NN.

The paper shows that any functional class that can be learned in polynomial time by some algorithm can be learned in polynomial time by deep neural networks using stochastic gradient descent. This sheds light, in part, on the empirical success of deep learning, and makes an important contribution toward furthering our understanding of efficient learning of neural networks. Authors complement the result with several extensions, including (a) showing that the results hold even when polynomial noise is added to the gradients or when weights can be of polynomial precision, (b) showing that a network of size O(n 2) and depth O(log(n)) can learn parities using SGD in poly time, and (c) lower bounds for descent algorithms characterized in terms of novel properties of networks that may be of independent interest. The paper reads very well, and the results and insights in the paper are very compelling.

Weaknesses: I have some major concerns with the evaluation part of the paper. A simple baseline could be a loss based selection method. Simply select training points based on loss change. A recent paper [DataLens IJCNN 20] shows that a simple loss based selection outperforms both influence functions and representer selection on mislabelled data identification when the mislabeled data is small. As the fraction of mislabelled data increases, influence function works better than loss based method.

I recommend accepting this paper. The authors managed to address the concerns that the reviewers had. Everyone is in agreement that this paper should be accepted.

Weaknesses: - The MCAR assumption is difficult to justify in practice. This is good, however, could the authors clarify some of the following points regarding their method in the context of MCAR missingness. By definition, MCAR implies that one can simply ignore any rows of data containing missingness and restricting the analysis to so called "complete cases" will still result in unbiased estimates of the parameter of interest. In light of this, and the bounds on \epsilon implying that there will always be complete cases in the data as n - \infty (if this were not true, the parameters of interest would not be identifiable) what is the advantage of the proposed EM algorithm over simply doing complete case analysis and using some of the older tools cited in the paper that can be run on complete data. I apologize if I missed this, but it doesn't seem like there's a baseline comparison to such a complete case analysis or to the alternative of directly maximizing the observed data likelihood by integrating according to patterns of missingness.

We thank the reviewers for their comments and interest. R1 Assigned_Reviewer_1). R2 proposes a baseline method to compare with. Our interpretation of the comment is that in the expression Y - Z t beta _2, R2 uses Z to denote the feature-vector and Y a 0-1 label, so this proposal corresponds to standard least-squares regression (with lasso). Generally, logistic (lasso) regression is preferable for binary responses [1]. As we already evaluated our approach against the latter method (Figure 1b), the proposed comparison seems unnecessary given the space constraints.