Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners