Targeted Dropout

Neural networks can represent functions to solve complex tasks that are difficult -- if not impossible -- to write instructions for by hand, such as understanding language and recognizing objects. Conveniently, we've seen that task performance increases as we use larger networks. However, the increase in computational costs also increases dollars and time required to train and use models. Practitioners are plagued with networks that are too large to store in on-device memory, or too slow for real-world utility. Progress made in network sparsification has presented post-hoc (after training) methods that allow us to remove parameters in a neural network and then fine tune the leftover parameters to achieve a similar task performance.