Online publishers face a dilemma: Allow AI scraping from Google or lose search visibility

As the US government weighs its options following a landmark "monopolist" ruling against Google last week, online publications increasingly face a bleak future. Bloomberg reports that their choice now boils down to allowing Google to use their published content to produce inline AI-generated search "answers" or losing visibility in the company's search engine. The crux of the problem lies in the Googlebot, the crawler that scours and indexes the live web to produce the results you see when you enter search terms. If publishers block Google from using their content for the AI-produced answers you now see littered at the top of many search results, they also lose the privilege of appearing in other Google search programs like snippets and Discover. Google uses a separate crawler for its Gemini (formerly Bard) chatbot, but its AI Overviews are generated using data from its main crawler.