Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

In this paper, the authors proposed a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism based on 2D Gaussian model. Furthermore this mechanism sequentially focuses on salient regions and takes additional glimpses within those regions in multi-label image classification. This sequential attention model also supports multiset prediction, where a reinforcement learning based training procedure allows classification to be done on instances with arbitrary label permutation and multiple instances per label. Pros: 1) This paper proposes a novel saliency based attention mechanism that utilizes saliency in the top layer (meta-controller) with a new 2D Gaussian based attention map. This new attention map models the regional /positional 2D information with a mixture of Gaussian distributions, which is more general than the standard attention layer (in DRAW, Show-attend-tell), where attention is enforced based on softmax activation. This mechanism is intuitive as it's inspired by human-level attention mechanism.

Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label. Papers published at the Neural Information Processing Systems Conference.