USA-Net: Unified Semantic and Affordance Representations for Robot Memory