Linearizing Models for Efficient yet Robust Private Inference