Investigating the Semantic Robustness of CLIP-based Zero-Shot Anomaly Segmentation