Differentiable Patch Selection for Image Recognition

Neural Networks require large amounts of memory and compute to process high resolution images, even when only a small part of the image is actually informative for the task at hand. We propose a method based on a differentiable Top-K operator to select the most relevant parts of the input to efficiently process high resolution images. Our method may be interfaced with any downstream neural network, is able to aggregate information from different patches in a flexible way, and allows the whole model to be trained endto-end Figure 1: Examples of large images where patch extraction using backpropagation. We show results for traffic allows (top-left) to focus on details for fine-grained recognition, sign recognition, inter-patch relationship reasoning, and (bottom-left) to reason across patches, and (right) to fine-grained recognition without using object/part bounding efficiently capture very localized information.