Specialized Foundation Models Struggle to Beat Supervised Baselines