Task-Agnostic Attacks Against Vision Foundation Models