Update after rebuttal: I think the rebuttal is fair. It is very reassuring that pseudocode will be provided to the readers. I therefore keep my decision unchanged. Original review: In the paper "Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo" the author(s) consider the problem of inference for deep gaussian processes (DGPs). Given the large number of layers and width of each layer, direct inference is computaitonal infeasible, which has motivated numerous variational inference methods to approximate the posterior distribution, for example doubly stochastic variational inference (DSVI) of [Salimbeni and Deisenroth, 2017] The authors argue that these unimodal approximations are typically poor given the multimodal and non-Gaussian nature of the posterior.

This paper proposes an oracle model for parallel algorithms for stochastic convex programming. There are two main oracle models that are studied: In the first one, the algorithm has access to an stochastic unbiased estimator of the gradient for a given feasible point, and in the second one one can query an stochastic proximal oracle (that is, the proximal operator for a random instance f(.,z), All oracle access are obtained from i.i.d. Interestingly, the paper introduces sample access architectures that can reflect parallelism, delayed updates or intermittent communication, and it turns out the obtained lower bounds are optimal for most of these cases. However, in some situations the matching upper bound comes from an unnatural algorithms, so it is left as open problem whether the natural algorithms used in practice are optimal as well.

The paper proposed a noval and scalable approach to solve the Laplacian K-modes problem. The primary strategy is to adopt another way to relax the original discrete problem and further minimize a upper bound of the objective. The method is guaranteed to converge and evaluated to show its superiority over some benchmark clustering algorithms in both clustering performance and algorithmic efficiency. The statements are clear and content is easy to follow. The technical part is also correct.

Updated comments: I carefully read the authors' response and the paper again. I was mistaken when I read Algorithms 1 (Eq(2.3)) The authors control the variance by simply averaging a batch of unbiased stochastic gradients. In this paper, the authors considered the problem of zeroth-order (non-)convex stochastic optimization via conditional gradient and gradient methods. However, all the techniques are already known but none are mentioned in the paper.

After rebutal; I do not wish to change my evaluation. Regarding convergence, I think that this should be clarified in the paper, to at least ensure that this is not producting divergent sequences under resaonable assumptions. As for the variance, the author control the variance of a certain variable \hat{g} given g but they should control the variance of \hat{g} without conditioning to invoke general convergence results. This is very minor but should be mentioned. The authors consider the problem of empirical risk minimization using a distributed stochastic gradient descent algorithm.

Review after rebuttal: I thank the author(s) for their response. While I still believe that this paper is a minor increment beyond what has already been done on SGLD, I agree that the message might be useful for some. I also appreciate the effort the authors have made in improving the manuscript based on reviews' suggestions, particularly their efforts to include relevant numerical experiments to ML scenarios, and recommendations beyond the CV approach which has been studied to exhaustion and rarely applicable in practice. Based on this, I've adjusted my decision to marginally above threshold. Original review: In the paper "The promises and pitfalls of Stochastic Gradient Langevin Dynamics" the authors revisit the Stochastic Langevin Gradient Dynamics (SGLD) approach to approximately sampling from a probability distribution using stochastic gradients (specifically subsampling). The authors compare a number of different classes of approximate inference method, including SGLD, LMC (known by some as Unadjusted Langevin Algorithm or ULA) and Stochastic Gradient Langevin Dynamics Fixed Point (SGLDFP) -- the latter being a variant of SGLD with a control variate exploiting the unimodality of the distribution, similar to what has been presented in [3, 25 and others].

This paper focuses on large-scale machine learning tasks in a distributed computing setup with the objective of making the underlying optimization procedures communication efficient. In particular, the paper considers the problem of reducing the overall communication during the collection of the stochastic gradient of the objective function at hand. The authors propose a sparsification approach where, during an iteration of the optimization procedure, each coordinate of the stochastic gradient is independently replaced with zero value with a certain probability. The non-zero coordinates are then properly scaled to ensure that the sparsified gradient remains an unbiased estimate of the true gradient. Now the (scaled) sparse gradient vector can be encoded up to a desired level of precision.

The paper considers k-means under differential privacy. The main idea of differential privacy (applied to k-means) is that changing one point in the input should not significantly change the centers that are computed. This immediately means that any approximation algorithm for the differentially private k-means problem needs to have additive error: The introduction includes a simple example of points in k locations ( optimum has zero cost). Now when moving one of these points to a different location, the centers may not change significantly; ergo one of the solutions has to have a large additive error; basically an error of D 2 where D is the largest pairwise distance. Thus, a common approach for this problem is to compute an approximation that is both multiplicative and additive. The best known result prior to this paper achieves a quality of O(k)*OPT O (d 0.51 k 1.51).

The paper describes algorithms for distributed k-center/k-median/k-means clustering. For distributed k-center, it is known that any constant approximation needs Omega(z) communication cost, where z is the number of allowed outliers. This can be avoided by using a bicriteria approximation where more than z outliers are allowed, but the cost is compared to the best solution with at most z outliers. It is known that using 2z outliers is sufficient to obtain a O(1)-approximation. The paper at hand now improves the violation with respect to the number of outliers to (1 eps)*z.

This paper studies deficiencies of volume sampling, and proposes a modification based on leverage scores, or renormalizing the current ellipse before performing volumne rejection sampling. It improves the number of unbiased samples required to guarantee 1\pm\epsilon accuracy by a factor of \epsilon {-1}, and also demonstrates the good empirical performances of its routines on datasets from LibSVM (in Supplementary materials E). Both linear regression and volume sampling are well studied topics, and the observations made in this paper are quite surprising. The paper clearly outlines a class of matrices that are problematic for volume sampling, and then proves the properties of the revised methods. The proposed methods also exhibit significant empirical gains over other methods in the small sample size regime, which are arguable the more important cases. I believe these contributions are of significant interest to the study of both randomized sampling and randomized numerical linear algebra.