Towards Robust Influence Functions with Flat Validation Minima