Task Vectors are Cross-Modal