Feature Extractor or Decision Maker: Rethinking the Role of Visual Encoders in Visuomotor Policies