Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks