Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models