On the Worst Prompt Performance of Large Language Models