b7bb35b9c6ca2aee2df08cf09d7016c2-Reviews.html

The paper presents an approach to building a parameter server for distribute ML systems that presents a view to each client where parameters have a bounded degree of staleness. Using a combination of caches, the client interface guarantees that all updates to the parameter array occurring after a fixed deadline (the current clock/iteration/tick minus a fixed delay) are visible along with more recent updates if possible. Thus the interface presents a view of parameters that integrates most updates along with best-effort service for more recent updates. It is shown that this simple semantic preserves the theoretical guarantees of cyclic delay methods while being significantly faster in practice. Empirical analysis on several problems with multiple cluster configurations show that the advantage is due to a combination of increased efficiency (over BSP) and optimization progress per update (over Asynchronous).