Empowering Large Language Models on Robotic Manipulation with Affordance Prompting