Learning to Act Through Contact: A Unified View of Multi-Task Robot Learning