Google will let publishers hide their content from its insatiable AI

Google has announced a new control in its robots.txt The control is a crawler called Google-Extended, and publishers can add it to the file in their site's documentation to tell Google not to use it for those two APIs. In its announcement, the company's vice president of "Trust" Danielle Romain said it's "heard from web publishers that they want greater choice and control over how their content is used for emerging generative AI use cases." Romain added that Google-Extended "is an important step in providing transparency and control that we believe all providers of AI models should make available." As generative AI chatbots grow in prevalence and become more deeply integrated into search results, the way content is digested by things like Bard and Bing AI has been of concern to publishers.