Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training

Therefore, from a system-level perspective, the design ethos of a system-efficient communication-compression algorithm is that we should guarantee that the compression/decompression of the algorithm is computationally light and takes less time, and it should also be friendly to efficient collective communication primitives.