On the Brittleness of LLMs: A Journey around Set Membership