What Matters in Learning from Large-Scale Datasets for Robot Manipulation