Google joins OpenAI and Anthropic in promising new scraping controls

Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley. As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley's proposal will change that by getting multiple AI companies to recognize and honor a single common signal. Publishers have to set up a new user agent in the robots.txt