Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays

"virtual iterates" and delay-adaptive stepsizes, which allow us to derive state-of-the-art guarantees for both convex and non-convex objectives.