Rethinking harmless refusals when fine-tuning foundation models