Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation