AI Is Not Your Friend

Recently, after an update that was supposed to make ChatGPT "better at guiding conversations toward productive outcomes," according to release notes from OpenAI, the bot couldn't stop telling users how brilliant their bad ideas were. ChatGPT reportedly told one person that their plan to sell literal "shit on a stick" was "not just smart--it's genius." Many more examples cropped up, and OpenAI rolled back the product in response, explaining in a blog post that "the update we removed was overly flattering or agreeable--often described as sycophantic." The company added that the chatbot's system would be refined and new guardrails would be put into place to avoid "uncomfortable, unsettling" interactions. But this was not just a ChatGPT problem. Sycophancy is a common feature of chatbots: A 2023 paper by researchers from Anthropic found that it was a "general behavior of state-of-the-art AI assistants," and that large language models sometimes sacrifice "truthfulness" to align with a user's views.