Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers