On the Problem of Consistent Anomalies in Zero-Shot Anomaly Detection