Towards Robust Interpretable Surrogates for Optimization