Mamba State-Space Models Can Be Strong Downstream Learners