Unifying Specialized Visual Encoders for Video Language Models