Self-Supervised Speech Representations are More Phonetic than Semantic