Overview The paper introduces a robust online change point detection algorithm for non-stationary time-series data. Robustness comes as a by product of minimizing \beta-divergence between data and fitted model as opposed to using KL divergence as in standard Bayesian inference. In the generalized Bayesian inference the posteriors are intractable. The paper mitigate this problem by resorting to structural variational approximation, which is proved to be exact as \beta converges to zero. The paper also discusses systematic approaches to initialize \beta and refine it online.

This paper considers the problem of optimizing over measures instead of parameters directly ( as is standard in ML), for differentiable predictors with convex loss. This is an infinite dimensional convex optimization problem. The paper considers instead optimizing with m particles (dirac deltas). As m tends to infinity this corresponds to optimizing over the measure space. Proposition 2.3 shows existence and uniqueness of the particle gradient flow for a given initialization.

I could go either way on this paper, though am slightly positive. The short summary is that the submission gives elegant expectation bounds with non-trivial arguments, but if one wants constant factor approximations (or 1 eps)-approximations), then existing algorithms are faster and read fewer labels. So it's unclear to me if there is a solid application of the results in the paper. In more detail: On the positive side it's great to see an unbiased estimator of the pseudoinverse by volume sampling, which by linearity gives an unbiased estimator to the least squares solution vector. I haven't seen such a statement before. It's also nice to see an unbiased estimator of the least squares loss function when exactly d samples are taken.

The paper analyzes ability of the soft-max, if used as the output activation function in NN, to approximate posterior distribution. The problem is translated to the study of the rank of the matrices contating the log-probabilities computed by the analyzed activation layer. It is shown that the soft-max does not increases the rank of the input response matrix (i.e. The authors propose to replace soft-max by the so called sigsoftmax (i.e. It is shown that the rank of sigsoftmax matrix is not less the rank of soft-max.

This paper starts by developing a notion of local "separability" of a loss function, which they use to get l_infty convertence rates, in terms of the separability parameters, for low and high dimensional settings. These rates are then applied to then applied to a probabilistic classification problem with both a generative and discriminative approach. After computing the teh separability parameters for each, they can apply the theorems to get l_infty convergence rates for the discriminative approach (logistic regression), as well as two generative approaches (for the cases that x y is isotropic Gaussian and gaussian graphical model). They next consider l_2 convergence rates. The discriminative rate is trivial based on the support consistency and the l_infty rates.

This paper proposes an efficient way to approximate the softmax computation for large vocabulary applications. The idea is to decompose the output matrix with singular value decomposition. Then, by selecting the most important singular values you select the most probable words and also compute the partition function for a limited amount of words. These are supposed to contribute for the most part of the sum. For the remaining words, their contributions to the partition function is only approximated.

The paper proposes a scheme for adaptive choice of learning rate for stochastic gradients descent and its variants. The key idea is very simple and easy to implement: given the loss value L at the global minimum, L_min, the idea is to choose learning rate eta, such that the update along the gradient reaches L_min from the current point i.e. solving L(theta-eta*v) L_min in eta, where v is for example dL/dtheta in gradient descent. Finally, to make the adaptive learning rate pessimistic to the possible linearization error, the authors introduce a coefficient alpha, so the effective learning rate used by the optimizer is eta*alpha. The authors empirically show (on badly conditioned regression, MNIST, CIFAR-10, and neural computer) that using such adaptive scheme helps in two ways: 1. the optimization performance is less sensitive to the choice of the coefficient alpha vs the learning rate (in non-adaptive setting), and 2. the optimizer can reduce the loss faster or at worst in equal speed with commonly used optimizers. At the same time, the paper has some shortcomings as admitted by the authors: 1.

However I feel that the presentation could be improved and more details about the (complicated) implementation should have been included. For instance, first p(x z_n ; \theta) is called the "encoder", then q(z x ; \phi) is called the "encoder" - which is it, and why is it given this name?

The main idea of the paper is to learn a predictive model p(y x;theta) such that the task's objective function f is directly optimized. In contrast, traditional approaches learn p(y x;theta) to minimize a prediction error without considering f. The main technical challenge in the paper is to solve a sub-optimization problem involving argmin w.r.t.

For the valuable problem of large-scale and sparse stochastic inference on simplex, the authors proposed a novel Stochastic gradient Markov chain Monte Carlo (SGMCMC) method, which is based on the Cox-Ingersoll-Ross (CIR) process. Compared with the commonly-used Langevin diffusion within the SGMCMC community, the CIR process (i) is closely related to the flexible Gamma distribution, and therefore more suitable for inferring a Dirichlet distribution on simplex, since a Dirichlet distribution is just the normalization of Gamma distributions; (ii) CIR has no discretization error, which is shown to be a clear advantage over the Langevin diffusion on simplex inference. Besides, the author proved that the proposed SCIR method is asymptotically unbiased, and has improved performance over other SGMCMC method on sparse simplex problem via two experiments, namely inferring a LDA on a dataset of scraped Wikipedia documents and inferring a Bayesian nonparametric mixture model on Microsoft user dataset. I think the quality is good; the presentation is clear; as far as I know the proposed technique is original and of great significance. Therefore I vote for acceptance. However, the experiments are okay, but not strong.