Is Diversity All You Need for Scalable Robotic Manipulation?