Sphynx: ReLU-Efficient Network Design for Private Inference

Deep learning inference is often outsourced to external cloud services to mitigate the high cost of executing state-of-theart deep networks on user devices [1]. However, outsourced inference raises essential concerns about data and model privacy: users may not trust a cloud service provider with their data, and cloud service providers may not want to share their models, trained at enormous expense, with users. Private inference (PI) provides a solution to this problem by guaranteeing user and model privacy using cryptographic techniques [2, 3]. Under PI, a user can perform inference using a model hosted in the cloud without the cloud learning anything about her data and, conversely, without the user learning anything about the cloud's model parameters. PI techniques thus far reported in the literature have leveraged a range of cryptographic protocols, including homomorphic encryption (HE), additive secret sharing (SS), and garbled circuits (GC). However, these all incur heavy computational overheads, resulting in several orders-of-magnitude increase in inference latency compared to standard "plaintext" inference. Prior work has demonstrated that non-linear network operations like the Rectified Linear Unit (ReLU) and max-pooling are the key bottlenecks. For example, Ghodsi et al. [4] estimate that ReLU layers in MiniONN [5] are four orders of magnitude more expensive than convolution layers, while in D