r/MachineLearning - [Research] deep gradient compression implementation

Those who are working on data parallel distributed training for neural networks, you might have come across this paper in your life. This work solves the communication bottleneck in data parallel DNN training by reducing the data transmission size by sending only gradient values that crosses a threshold. I was fascinated by the ideas in the paper and wanted to quickly try it out. So I made a version of DGC on MXNet. I tried training ResNet-110 on CIFAR-10 with DGC.