I think the optimization approach proposed in the paper is novel enough to justify the high score, and that the experimental evaluation is sufficient to show that the optimization method is doing something sensible. I also think that the new experiments from the rebuttal will improve the paper, and will help address some of the concerns raised in the original reviews. Comparison with related work: The experiments in this paper show state-of-the-art results. However, in the original submission, it was difficult to tell the extent to which quality and sample efficiency improvements over NASNet/AmoebaNet were coming from the optimization algorithm, as opposed to search space improvements or longer training times. The authors directly address both points in their rebuttal, and I believe the experiments they mention will help explain where the efficiency improvements come from.

This paper introduces a new algorithm (BARRONS) for the online portfolio optimization problem. This is in contrast to the Cover's universal portfolio algorithm, online newton step, and exponentiated gradient, which each achieve at most two of these three goals. The algorithm itself is an application of the online mirror descent framework, using the less-classical log-barrier regularizer. The paper provides good intuition for how the algorithm is able to avoid dependence on the gradient norm: the gradient norm is only big if a previously poorly-performing stock starts to perform well. As a result the learner has a kind of "surplus" of regret that it can fall back on while adapting to the new stock.

Paper Summary Starting from the observation that batch-normalization induces a particular form of scale invariance on the weight matrix, the authors propose instead to directly learn the weights on the unit-sphere. This is motivated from information geometry as an example of optimization on a Riemannian manifold, in particular the Stiefel manifold V(1,n) which contains unit-length vectors. As the descent direction on the unit sphere is well known (eq 7), the main contribution of the paper is in extending popular optimization algorithms (SGD momentum and Adam) to constrained optimization on the unit-sphere. Furthermore, the authors propose orthogonality as a (principled) replacement for L2 regularization, which is no longer meaningful with norm constraints. The method is shown to be effective across two families of models (VGG, wide resnet) on CIFAR-10, CIFAR-100 and SVHN.

I would suggest elaborating on the optimization landscape more in the paper --Finally, the mixture of two gaussians is a very special case where EM converges since the landscape does not have bad local optima. The paper misses discussions on the following relevant results: (a) Jin, Chi, et al. "Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences."

Summary: The authors propose a bilevel method for metric learning, where the lower level is responsible for the extraction of discriminative features from the data based on a sparse coding scheme with graph regularization. This effectively detects their underlying geometric structure, and the upper level is a classic metric learning approach that utilizes the learned sparse coefficients. These two components are integrated into a joint optimization problem and an efficient optimization algorithm is developed accordingly. Hence, new data can be classified based on the learned dictionary and the corresponding metric. In the experiments the authors demonstrate the capabilities of the model to provide more discriminative features from high dimensional data, while being more robust to noise.

This paper proposes a framework which can learn classifiers that satisfy multiple notions of fairness such as fairness through unawareness, demographic parity, equalized odds etc. The proposed framework leverages ideas from two different lines of existing research namely, distribution matching and privileged learning, in order to accommodate multiple notions of fairness. This work builds on two prior papers on fairness - Hardt et. The proposed method seems interesting and novel, and the ideas from privileged learning and distribution matching have not been employed in designing fair classifiers so far. The idea of proposing a generalized framework which can handle multiple notions of fairness is quite appealing. The paper, however, has the following weaknesses: 1) the evaluation is weak; the baselines used in the paper are not even designed for fair classification 2) the optimization procedure used to solve the multi-objective optimization problem is not discussed in adequate detail Detailed comments below: Methods and Evaluation: The proposed objective is interesting and utilizes ideas from two well studied lines of research, namely, privileged learning and distribution matching to build classifiers that can incorporate multiple notions of fairness.

This paper studies the online learning (stochastic and full-information) problem of bidding in multi commodity first price auctions. The paper introduces a polynomial time algorithm that achieves a regret of \sqrt{T log(T)} that has a near optimal dependence on T. The main challenge that the paper has to deal with is to find a computationally efficient algorithm for computing the best biding strategy given a known distribution.The authors first demonstrate that natural approaches for solving this problem exactly are not computationally efficient (this is not a formal np-hardness proof). Then, they provide a FPTAS for solving the problem using dynamic programming. Once they have a FPTAS for the offline problem, their results hold for the stochastic online setting using existing reductions. I haven't carefully looked in to the details of their analysis of the dynamic programming, but I think the effectiveness of it here is interesting and surprising -- specially given that the variation of this problem for the second price auctions is hard to approximate.

This paper proposes a way to differentiate the process of submodular function minimization thus enabling to use these functionals as layers in neural networks. The key insight of the paper consists in the usage of the interpretation of discrete optimization of submodular functions as continuous optimization. As a concrete example the paper studies the CRF for image segmentation and creates and the graphcut layer. This layer is evaluated on the Weizmann dataset for horse segmentation and is reported to bring some improvements. I generally like the paper very much, find the description of the method clear enough.

In the submission, the authors aim at developing a black-box boosting method for variational inference, which takes a family of variational distributions and finds a mixture of distribution in a given family that approximates a given posterior distribution well. The main keyword here is black-box; white-box, restricted approaches exist. In order to achieve their aim, the authors formulate a version of the Frank-Wolfe algorithm, and instantiate it with the usual KL objective of variational inference. They then derive a condition on the convergence of this instantiation that is more permissive than the usual smoothness and is based on the reformulation of the bounded curvature condition (Theorem 2). They also show how the constrained optimization problem included in the instantiation of Frank-Wolfe can be expressed in terms of a more intuitive objective, called RELBO in the submission.

The authors propose a mathematical programming approach to build interpretable machine learning models. In this case, the interpretable model is a system of Boolean rules in disjunctive (or conjunctive) normal form which is constructed using column generation for the linear relaxation of a mixed integer program (MIP) designed to minimize the number of positive samples classified incorrectly and the complexity of the learned system subject to complexity constraints. To remedy the fact that there are exponentially many potential clauses to optimize over, the authors propose a standard column generation approach that prices potential columns to add and solves a secondary integer program to find such potential columns. The authors also note that the column generation can also be done via heuristics or a greedy algorithm. Once the linear programming program is solved or reaches its time limit, the approach then solves the global mixed integer formulation to get a final set of rules.