Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models