This leads to a new MU paradigm, termed prune first, then unlearn, which infuses a sparse model prior into the unlearning process.

Neural Information Processing Systems http://nips.cc/

Have you retrained or moved careers due to your previous career path being at risk of an artificial intelligence takeover? Please include as much detail as possible. Did you have a dream profession that you have decided not to pursue because of fears it will be thwarted by AI? Optional Please include as much detail as possible.

For problems with a very large vocabulary size, the embedding and the softmax matrices can account for more than half of the model size. For instance, the bigLSTM model achieves great performance on the One-Billion-Word (OBW) dataset with around 800k vocabulary, and its word embedding and softmax matrices use more than 6GBytes space, and are responsible for over 90% of the model parameters. In this paper, we propose GroupReduce, a novel compression method for neural language models, based on vocabulary-partition (block) based low-rank matrix approximation and the inherent frequency distribution of tokens (the power-law distribution of words).

This leads to a new MU paradigm, termed prune first, then unlearn, which infuses a sparse model prior into the unlearning process.