From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance