Avoid Overthinking in Self-Supervised Models for Speech Recognition