OpenAI releases webcrawler GPTBot, how to block it

CEO says OpenAI CEO Sam Altman said language and cultural inclusivity is "very important" to his company's mission as it builds and trains powerful artificial intelligence systems. OpenAI has launched web crawler GPTBot to improve artificial intelligence models. "Web pages crawled with the GPTBot user agent may potentially be used to improve future models and are filtered to remove sources that require paywall access, are known to gather personally identifiable information (PII) or have text that violates our policies," the company said in a post on its website. "Allowing GPTBot to access your site can help AI models become more accurate and improve their general capabilities and safety," OpenAI wrote. A web crawler is a type of bot.