Weaknesses: The ideas of the paper could be interesting however the paper loses some points in terms of presentation. Also some claims are not really justified. For example the title mentioned "An Improved Analysis" but it was never really explained in detail why the proposed analysis justifies the word "improved". There are some limitations of existing papers in the Intro but this should be more clear in the main contributions of the work. In line 74, the authors mentioned: "To the best of our knowledge, this is the first convergence (and acceleration) guarantee for SGDM in the multistage setting."

The paper studies the convergence of SGD with momentum, which is of strong research interest. It shows that SGD with momentum converges as fast as SGD for smooth strongly-convex/non-convex objectives, and faster in a multi-stage scenario with learning-rate decay. While the core contribution was liked by all reviewers, Reviewer 3 brought a serious issue in the proof of Lemma 1 to our attention, which forms the foundation for the main results. After the feedback and additional clarification by the authors and longer discussions, we share the impression with the authors that the issue can be fixed by replacing E[m k] by v k throughout the paper and adjusting minor constants. We expect trust the authors to perform these changes and should any issues remain, withdraw the paper. Additionally, we hope the detailed feedback with improvement suggestions from the 4 reviews will be implemented for the camera ready version.

Unlike standard prediction tasks, survival analysis requires modeling right censored data, which must be treated with care. While deep neural networks excel in traditional supervised learning, it remains unclear how to best utilize these models in survival analysis. A key question asks which data-generating assumptions of traditional survival models should be retained and which should be made more flexible via the function-approximating capabilities of neural networks. Rather than estimating the survival function targeted by most existing methods, we introduce a Deep Extended Hazard (DeepEH) model to provide a flexible and general framework for deep survival analysis. The extended hazard model includes the conventional Cox proportional hazards and accelerated failure time models as special cases, so DeepEH subsumes the popular Deep Cox proportional hazard (DeepSurv) and Deep Accelerated Failure Time (DeepAFT) models.

In sequence to sequence model for each n, you do not really have to train a different model. You could just train a different softmax for each n and the rest of the parameters can be shared for all the problems of different lengths.

Summary: The paper proposes a novel approach to computationally efficient maximum likelihood learning in exponential families. In general, finding the maximum likelihood solution is intractable. From a convex optimization perspective, the sticking point is the need to calculate an integral wrt the currently proposed EF parameter. By assuming that MCMC is fast-mixing for all allowed parameters, the author(s) are able to show that the integrals needed for proximal gradient descent can be calculated with sufficient precision that, when combined with the results of Schmidt et al. (2011), a fully-polynomial randomized approximation scheme for calculating the MLE can be obtained. Both the convex and strongly convex cases are considered, which lead to different types of guarantees: the former on the likelihood error, the latter on the parameter error.

We thank all the reviewers for their valuable comments and suggestions. We will revise accordingly in the final version. Response to Reviewer_1 The conditions on the initial solution can be satisfied in two ways: (i) Do multiple random initializations and use the initial solution leading to largest Q function. This is a heuristic that practitioners often do for EM algorithm; (ii) Use result of tensor methods, e.g., [9], as an initial solution, for which we can obtain rigorous theoretical guarantees. We would like to suggest the use of gradient ascent implementation for two reasons: (i) For complex models, it can be hard to derive the closed form of the exact M-step; (ii) For high dimensional models, the exact M-step is not always well defined.

Weaknesses: The major weaknesses of the paper are listed below: 1. There are some potential inaccuracies in the description of the algorithm. For example, in Section 3.1, the first equalities in the two lines of equations after line 210 should be \approx instead, right? And does the notation p_{\tau_B} ' denote the sub-gradient of p_{\tau_B}? In general, some more explanations about the linearization here would be helpful.

This paper proposes a new parametrization of the multivariate linear regression problem. It shows that under this new parametrization, it is easier to employ sparsity inducing penalty terms on the inverse covariance matrix. The paper suggests a sequential relaxation algorithm. The reviewers noted the novelty of the approach and numerous strengths. The simulation experiments (in the supplementary material) explore the method in the context of several connectivity scenarios. However, one weakness is the exploration of the performance of the model on real data scenarios.

Weaknesses: While the problem setting and proposed approach are interesting there are some drawbacks in the execution of this idea. First much of the experimental detail is left to the supplementary material and makes the main paper appear lacking in results. Concerningly, few of the transductive baselines outperform the main baselines (see Cora table in the Appendix for Deepwalk features) reported in the main body of the paper and thus their omission is questionable. Furthermore, the chosen datasets as the paper recognizes are either small graphs or contain only a single graph and as a result its difficult to assess how scalable the proposed approach is to larger real world graphs. The biggest weakness in this reviewers opinion is that its unclear why the MCMC scheme proposed is a natural or superior choice to existing approaches to training EBMs in the literature. Training EBMs have seen a resurgence of late and there have been multiple approaches that provide significant computational benefit [1] [2] [3] are few recent examples.

Weaknesses: - Distinction between F and constituent f_i being convex or strongly convex: why does this matter in practice? In the general learning case, f_i(x) is usually of the form f(x, \xi_i) where \xi_i is the i-th scenario or data. So, many common properties like smoothness, convexity etc. are naturally assumed to "flow" from f to the sample/population average F. What practical scenario do you envision where f_i could be distinct functional forms that are not convex but F is? where would an example like this make sense: f_1 (x) -x 2 (concave), f_2 (x) 2x 2 leading to F(x) x 2 (convex)? Is there any guarantee that doing something like projecting all iterates on to your \cal{S}_{x_0} set (defined on pg 3 of the supplement) to enforce this assumption will not lead to sub optimal results? How does your analysis of the progress made in the shuffled SGD steps compare to progress that would have been made w.r.t epoch count K using deterministic full gradient descent with a proper step-length determination like Armijo rule?