Technical quality: It seems the nested-batch method is likely to introduce overhead by keeping all previously sampled points in memory, especially since mini-batch k-means is usually run for many iterations? And the computational cost of checking whether a point is already sampled grows as the number of iteration grows as well. How did this not seem to have an effect in your experiments, as comparing to the original mini-batch algorithm? The experiments in Figure 1 may be a little misguiding: it shows that nested-mini-batch achieves same level of k-means cost faster than the other compared methods; however, this may only mean that it plateaued faster. As time increases, it's possible that the other algorithms will achieve a lower k-means cost eventually (they reach a plateau with a lower k-means cost).

Two popularly used time series prediction models, autoregressive (AR) and dynamic linear models (DLM), are both time consuming to learn, especially for high dimensional time series prediction problem with missing values. However, matrix factorization is relatively efficient for large-scale matrix. The authors model the high dimensional time series as matrix and induce the constraints as regularization terms, then formulate the problem as a regularized matrix factorization problem and solve it by adopting the off-the-shelf solvers. The temporal regularized matrix factorization(TRMF) framework proposed by the paper sounds interesting. Inherited from the properties of MF, TRMF is able to deal with missing values and can be scalable to high-dimensional time series datasets.

Comparing with previous work, including generalized phase retrieval problems, this paper has the following differences: 1) Solves a second order function incorporating absolute values of measurements 2) No step size normalization (or variants) 3) No gradient truncation The key intuition of this paper is that for a measurement ai, x with large magnitude, in the local region near global optima, the sign of ai, x and ai, z are likely to be same. In the case, locally the gradient update of reshaped WF (RWF) is as the same as an equivalent least squares problem. For the objective function including all the measurements, if most the components have reasonably large measurements, those with small measurements contribute less, hence using gradient descent good initialization should give good results. Such intuition is formalized in equation (34) and (35). Yet I have a question -- why don't one consider further truncating components with small magnitude when computing gradient? Just like in robust regression, TWF, etc, we can throw out "wrong" directions.

This paper proposes a new objective function to solve mixed linear regression problem, but fails to explain many important issues: (1) What is the intuition of the introduction and advantage of the objective function? The answer between line 39 and line 40 is not good. Because if it is modeled as finite mixture model as in many references, "objective value is zero when {w_k}_{k 1,2,...,K} is the global optima and y's do not contain any noise" is also true. The following is a example. It seems there is no probabilistic interpretation for the objective function in Eq.(1).

In my view, the main reason the proposed lower bound is interesting is that it offers a potential way to speed up training for multi-class models with a very large number of classes. While it is useful to understand other properties of the lower bound, the paper could be improved by emphasizing this primary use case in machine learning. Figure 1c and Figure 3 need a more clear explanation of what is being displayed, and why it is important. In particular, what value is being plotted on the y-axis, and at what setting of the parameters w. Here is how I understand it, for Figure 1c say: Blue Line - value of Eq. (13) at the setting of parameters w that maximize 13 Red Line - value of Eq. (13) at the setting of parameters w that maximize 14 Green Line - value of Eq. (13)? at the setting of parameters w that maximize the Bouchard lower bound (?) Red dashed line - value of Eq. (13)? at parameters w based on the given iterations of training?

In this paper the author studied theoretic properties of the robust k-means (RKM) formulation proposed in [5,23]. They first studied the robustness property, showing that if the f_\lambda function is convex, the one outlier is sufficient to break down the algorithm; and if f_\lambda need not be convex, then two outliers can breakdown the algorithm. On the other hand, under some structural assumptions on the non-outliers, then a non-trivial breakdown point can be established for RKM. The authors then study the consistency issue, generalising consistency results that are known for convex f_lambda to non convex f_\lambda. My main concern of the paper is that the results appear very specific and I am not entirely sure whether they will appeal to a more general audience in machine learning.

STRENGTHS: - The problem addressed by the authors is important - The proposed algorithm is relatively novel and seems effective - While I didn't verify the proofs, the arguments appear correct. WEAKNESSES: - I found the application to differential privacy unconvincing (see comments below) - Experimental validation was a bit light and felt preliminary RECOMMENDATION: I think this paper should be accepted into the NIPS program on the basis of the online algorithm and analysis. However, I think the application to differential privacy, without experimental validation, should be omitted from the main paper in favor of the preliminary experimental evidence of the tensor method. The results on privacy appear too preliminary to appear in a "conference of record" like NIPS. TECHNICAL COMMENTS: 1) Section 1.2: the dimensions of the projection matrices are written as A_i \in \mathbb{R} {m_i \times d_i} .

After reading the author's rebuttal I have increased the technical quality to 2 and after reading the the other reviews I increased the potential impact to 3. The authors replied to many questions but not to all, in particular the answer was not satisfactory to the question about the parameter K which is one of the crucial parameter in any segmentation algorithm. Why they did not provide the results using the suggested automatic method in Fig4 instead of cyclying on possible (wrong) number of clusters? I would have expected to see in the results the performances with at least one auto-tuning heuristic to asses its generality (at least the one suggested by the authors). In the following the issues found in the paper: 1) In Eq(2) when constructing the adjecency matrix, the ranges of the distances d(...) and \delta(...) are the same? In the line 114 d(s) is a measure of heterogeneity, in line 125 of distance and in Eq(2) of color distance.

Summary: This paper proposes a stochastic online learning method for the task of structured prediction. In this setting, the learner doest not get the correct structured output during training. Instead, it only gets bandit feedback from the labeler. The paper first proposes an online learning algorithm that learns model parameters via stochastic gradient descent; generalizes the learning method to pair-wise comparison of structured outputs; provides an optimization approach with Cross-Entropy Minimization; and theoretically analyzes the convergence property of the optimization approach. Pros: The paper proposes an online stochastic learning algorithm for minimizing the expected loss of structured predictions; gives a method of learning from pair-wise comparisons; and theoretical analyze the convergence rate.

The extension of SGGMC from previous work (SGRHMC)[1] are in two folds. First, the proposed method use Geodesic flow rather than Riemmannian manifold. Second, the proposed method leverage a symmetric splitting integrator (ABOBA) scheme. However, unfortunately none of extensions have a clear and convincing novelty as far as I can see. The drop-in replacement of D and Q are not surprising.