Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models