Learning Affordances at Inference-Time for Vision-Language-Action Models