Reviews: Learning to Optimize in Swarms

This paper introduces a new meta-learning algorithm that combines population-based and point-based optimization. While population based approaches have been very popular in very rugged landscapes, current meta-learning methods are point-based and thus not suitable for optimizing such functions. This work presents two contributions, (1) a new architecture for population based meta-learning. This architecture, while more complicated, can be summarized as follows: each particle is composed of a set of 4 features (gradient, momentum, velocity, and attractions), an attention mechanism is applied to those features together with the hidden state. The outputs of the attention mechanism for all particles are fed into an inter-particle attention together with a similarity matrix.