A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning