To this end, a fairly common distributed learning framework,namely dataparallelism,distributesthe(huge)data-setsovermultiple workermachines to exploit the power of parallel computing.

Fifteen years ago, when I returned home after fighting in Iraq, a friend asked me to describe the bravest thing I saw anyone do. I had led a Marine platoon in the Second Battle of Fallujah, in 2004, and had seen plenty of heroism--Marines dragging their wounded off machine-gun-swept streets, or fighting room to room to recover a comrade's body. But none of these compared to the cumulative heroism of the 19- and 20-year-old infantrymen who placed their bodies across that fatal funnel--a doorway with a potential enemy inside--every day. Clearing the enemy from the city, house by house, was a game of Russian roulette played on a grand scale. You never knew who might be waiting on the other side of the door.

We develop a distributed second order optimization algorithm that is communication-efficient as well as robust against Byzantine failures of the worker machines. We propose COMRADE (COMunication-efficient and Robust Approximate Distributed nEwton), an iterative second order algorithm, where the worker machines communicate only once per iteration with the center machine. This is in sharp contrast with the state-of-the-art distributed second order algorithms like GIANT [34] and DINGO[7], where the worker machines send (functions of) local gradient and Hessian sequentially; thus ending up communicating twice with the center machine per iteration. Moreover, we show that the worker machines can further compress the local information before sending it to the center. In addition, we employ a simple norm based thresholding rule to filter-out the Byzantine worker machines. We establish the linear-quadratic rate of convergence of COMRADE and establish that the communication savings and Byzantine resilience result in only a small statistical error rate for arbitrary convex loss functions. To the best of our knowledge, this is the first work that addresses the issue of Byzantine resilience in second order distributed optimization. Furthermore, we validate our theoretical results with extensive experiments on synthetic and benchmark LIBSVM [5] data-sets and demonstrate convergence guarantees.