Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources. Neural network quantization has significant benefits in reducing the amount of intermediate results, but it often requires the full datasets and time-consuming fine tuning to recover the accuracy lost after quantization. This paper introduces the first practical 4-bit post training quantization approach: it does not involve training the quantized model (fine-tuning), nor it requires the availability of the full dataset. We target the quantization of both activations and weights and suggest three complementary methods for minimizing quantization error at the tensor level, two of whom obtain a closed-form analytical solution. Combining these methods, our approach achieves accuracy that is just a few percents less the state-of-the-art baseline across a wide range of convolutional models.

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining.

Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Deep learning frameworks have often focused on either usability or speed, but not both.

We have revised the paper as suggested by the reviewers. As mentioned by reviewer #1, the hardware requirements of PyTorch were not clearly explained. Reviewer #1 wondered about attribution. Since more than 1000 people have contributed to PyTorch, we can't Instead we've highlighted the people who had a profound impact on the library We've also acknowledged the impact of the community We fixed the typographical error line 87 pointed out by reviewer #1. We realized that didn't cite the "Automatic differentiation in PyTorch" work from Paszke, Gross, Chintala, Chanan, Y ang, DeVito, Lin, Desmaison, Antiga and Lerer submitted at the 2017 NIPS autodiff workshop.

As a result, they may suffer from distribution mismatching issue.

Neural Information Processing Systems http://nips.cc/

Using two new tasks similar to Raven's Progressive Matrices, we evaluate the usefulness of the representations learned by 360 state-of-the-art unsupervised disentanglement models.