Mathematical & Statistical Methods
Review for NeurIPS paper: Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics
Weaknesses: The specific empirical evaluation chosen is the primary weakness of the paper. From a neuroscience perspective, the validation of parameter recovery on synthetic data is a necessary first step, but not a sufficient one. Given that [a] the task is primarily of neuroscientific interest and [b] a simpler (though also bayesian belief-updating) fit model is given in the cited prior work, the lack of comparison of cross-validated performance against that prior model is surprising. We should either see better cross-validation performance to the models in prior work, or similar performance but more insight / explanation of the underlying mental computation. This would show us a real payoff of the new insights here.
Review for NeurIPS paper: Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics
The paper describes a novel technique for inverse rational control. The reviewers all agree that this is great work that makes an important contribution. There is one important weakness though: the experiments. More comprehensive experiments would be desirable to increase the impact of the work. Nevertheless, this is still good work.
Review for NeurIPS paper: Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
Weaknesses: No Explanation of Transformations of Stochastic Processes: I was under the impression that transforming / reparameterizing a stochasic process is non-trivial. Thus, I was expecting Equation 7 to include a second derivative term. I'm not saying that Equation 7 is wrong, per se---transforming just the increments agrees with intuition. However, the problem is that the paper provides no explanation or mathematical references for stochastic processes and their transformations. There are *zero* citations in both Section 2.2 and Section 3.1.
Review for NeurIPS paper: Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
One reviewer recommend borderline rejection, but in my opinion the authors successfully addressed his concerns in the rebuttal. Recommendations: The authors are encouraged to clearly explain the reviewers' concern on potential similarities of the approach with the Kalman filter with nonlinear outputs. Also the issues related to background and related work and motivation for continuity.
Reviews: Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses
The paper studies large-scale convex optimization algorithms based on the Newton method applied to regularized generalized self-concordant losses, in particular in ill-conditioned settings, providing new optimal generalization bounds and proofs of convergence. The reviewers found the contributions of high quality and were satisfied with the clarifications provided by the author response.
Reviews: Minimal Variance Sampling in Stochastic Gradient Boosting
Update: I read authors' responce RE:sampling rate does not tell the whole story - i was suggesting to add information about on average how many instances were used for each of the splits (because it is not equal to sampling rate * total dataset size). I am keeping my accept rating, hoping that authors do make the changes to improve the derivations/clarity in the final submission Summary: this paper is concerned with a common trick that a lot of GBDT implementation apply - subsampling instances in order to speed up calculations for finding the best split. The authors formulate the problem of choosing the instances to sample as an optimization problem and derive a modified sampling scheme that is aimed at mimicking the gain that would be assigned to a split on all the of the data by using a gain calculated only on a subsampled instances. The experiments demonstrate good results. The paper is well written and easy to follow, apart from a couple of places in derivations(see my questions).
Reviews: Minimal Variance Sampling in Stochastic Gradient Boosting
The authors propose a non-uniform sampling strategy for stochastic gradient boosted decision trees. In particular, sampling probability of the training data is optimized towards maximizing the estimation accuracy of the splitting score of decision trees. The optimization problem allows an approximate closed-form solution. Experiment results demonstrate superior performance of the proposed strategy. The reviewers agree that the paper can not only help understand sampling within GBDT from a more rigorous perspective but also improve GBDT implementations in practice.
Reviews: Understanding the Role of Momentum in Stochastic Gradient Methods
INDIVIDUAL COMMENTS / QUESTIONS 1) I really appreciate how the paper ties up loose ends by unifying the analysis of several momentum-based methods in the stochastic setting. I am not very closely familiar with the literature analyzing momentum methods, but there's a lot of work out there (e.g., the line of research studying momentum methods in the continuous time limit). A brief review would be very helpful to position the paper within the existing work. To me this implies that the analysis would go through for more general functions. I don't find it obvious that it would.
Reviews: Understanding the Role of Momentum in Stochastic Gradient Methods
The reviewers agree that the topic tackled in the paper is interesting and the mathematical results are promising. Overall, this submission is a good attempt in deriving a mathematical understanding of QHM, but the results are often only partially investigated and commented. For instance, in section 3 the main result (i.e. the convergence rate for quadratics) is really hard to parse and is poorly commented in the sense that its practical value is unclear. The paper also makes a number of conjectures that are not backed up and the authors are therefore advised to tone down their claims. This includes "we conjecture that the optimal convergence rate is a monotonically decreasing function of nu" as well as the quality of the approximation in Section 4. In conclusion, all three reviewers liked the paper but also highlighted some shortcomings, therefore justifying acceptance as a poster but not an oral.