The momentum acceleration technique is widely adopted in many optimization algorithms. However, there is no theoretical answer on how the momentum affects the generalization performance of the optimization algorithms.

Stochastic gradient methods have been a widespread practice in machine learning.

Therefore, from a system-level perspective, the design ethos of a system-efficient communication-compression algorithm is that we should guarantee that the compression/decompression of the algorithm is computationally light and takes less time, and it should also be friendly to efficient collective communication primitives.

Algorithm Distributed 1: Input: stepsize { t}with 1 =0; number M; compressorC( ).

Numerical experiments are conducted tosupport our theoretical results.

Numerical experiments are conducted tosupport our theoretical results.

Stochastic gradient methods have been a widespread practice in machine learning.

Then dynamics ofwt is like a physical process - a satellite's motion around the earth (see illustration in Fig.1): according to