Empirical influence functions to understand the logic of fine-tuning