Hessian matrix but as a result the convergence and generalization guarantees remain largely mysterious for MAML.

First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. The paper proposes to use random functions as an approximation to kernel functions and then proposes to do stochastic gradient descent. Convergence rates and generalisation bounds are derived. Experimental results on large datasets are presented. The idea of introducing random functions to approximate kernel functions and then using SGD is very interesting.

The general perception is that kernel methods are not scalable, so neural nets become the choice for large-scale nonlinear learning problems. Have we tried hard enough for kernel methods? In this paper, we propose an approach that scales up kernel methods using a novel concept called " doubly stochastic functional gradients ". Based on the fact that many kernel methods can be expressed as convex optimization problems, our approach solves the optimization problems by making two unbiased stochastic approximations to the functional gradient--one using random training points and another using random features associated with the kernel--and performing descent steps with this noisy functional gradient. Our algorithm is simple, need no commit to a preset number of random features, and allows the flexibility of the function class to grow as we see more incoming data in the streaming setting. We demonstrate that a function learned by this procedure after t iterations converges to the optimal function in the reproducing kernel Hilbert space in rate O (1/t), and achieves a generalization bound of O (1 / t). Our approach can readily scale kernel methods up to the regimes which are dominated by neural nets. We show competitive performances of our approach as compared to neural nets in datasets such as 2.3 million energy materials from MolecularSpace, 8 million handwritten digits from MNIST, and 1 million photos from ImageNet using convolution features.

We explore combining dropout with robust training methods and obtain better generalization.

For example, a backdoor attack uses one trigger to manipulate the model's outputs on all inputs,

The key idea of COPML is to securely encode the individual datasets to distribute the computation load effectively across many parties and to perform the training computations as well as the model updates in a distributed manner on the securely encoded data. We provide the privacy analysis of COPML and prove its convergence.