How would you describe why the image in Figure 1 looks like a clay colored sparrow? Perhaps the bird's head and wing bars look like those of a prototypical clay colored sparrow. When we describe how we classify images, we might focus on parts of the image and compare them with prototypical parts of images from a given class. This method of reasoning is commonly used in difficult identification tasks: e.g., radiologists compare suspected tumors in X-ray scans with prototypical tumor images for diagnosis of cancer [13].

Neural Information Processing Systems http://nips.cc/

We introduce Bayesian distributed stochastic gradient descent (BDSGD), a high-throughput algorithm for training deep neural networks on parallel computing clusters. This algorithm uses amortized inference in a deep generative model to perform joint posterior predictive inference of mini-batch gradient computation times in a compute cluster specific manner. Specifically, our algorithm mitigates the straggler effect in synchronous, gradient-based optimization by choosing an optimal cutoff beyond which mini-batch gradient messages from slow workers are ignored. The principle novel contribution and finding of this work goes beyond this by demonstrating that using the predicted run-times from a generative model of cluster worker performance improves over the static-cutoff prior art, leading to higher gradient computation throughput on large compute clusters. In our experiments we show that eagerly discarding the mini-batch gradient computations of stragglers not only increases throughput but sometimes also increases the overall rate of convergence as a function of wall-clock time by virtue of eliminating idleness.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Becauseofarichvarietyof applications, themaximization ofanonnegativesubmodular function with respect toacardinality constraint (MCC) has a long history of study (Nemhauser et al., 1978). Furthermore, this ratio is optimal under thevalue oracle model (Nemhauser and Wolsey,1978).

Neural Information Processing Systems http://nips.cc/