Rethinking Imbalance in Image Super-Resolution for Efficient Inference