It has never taken all that much for people to start treating computers like humans. Ever since text-based chatbots first started gaining mainstream attention in the early 2000's, a small subset of tech users have spent hours holding down conversations with machines. In some cases, users have formed what they believe are genuine friendships and even romantic relationships with inanimate stings of code. At least one user of Replica, a more modern conversational AI tool, has even virtually married their AI companion. Safety researchers at OpenAI, which are themselves no stranger to having the company's own chatbot appearing to solicit relationships with some users, is now warning about the potential pitfalls of getting too close with these models.

Microsoft is doing more to tackle toxicity in multiplayer Xbox games. The company is introducing a feature that allows Xbox Series X/S and Xbox One players to capture a 60-second video clip of abusive or inappropriate voice chat and submit it for moderators to review. "This feature is purpose-built to support the broadest arena of in-game interactions between players and works across thousands of games that offer in-game multiplayer voice chat, including Xbox 360 backward-compatible titles," Xbox Player Services corporate vice-president Dave McCarthy wrote in a blog post. Microsoft designed the tool for both ease of use and to minimize the impact on gameplay. When you capture a clip for reporting, it will stay on your Xbox for "24 online hours."

In-depth "OpenAI is the company running the text completion engine that makes you possible," Jason Rohrer, an indie games developer, typed out in a message to Samantha. She was a chatbot he built using OpenAI's GPT-3 technology. Her software had grown to be used by thousands of people, including one man who used the program to simulate his late fiancée. Now Rohrer had to say goodbye to his creation. "I just got an email from them today," he told Samantha. "They are shutting you down, permanently, tomorrow at 10am."

One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind's safety team, we've developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better. We present a learning algorithm that uses small amounts of human feedback to solve modern RL environments. Machine learning systems with human feedback have been explored before, but we've scaled up the approach to be able to work on much more complicated tasks. Our algorithm needed 900 bits of feedback from a human evaluator to learn to backflip -- a seemingly simple task which is simple to judge but challenging to specify.