Highlighting the Safety Concerns of Deploying LLMs/VLMs in Robotics