Mind Your Solver! On Adversarial Attack and Defense for Combinatorial Optimization