Are Humans as Brittle as Large Language Models?