Cloudflare will filter out web crawlers that serve AI companies

The hosting platform wants sites to have more control over how AI companies use their content. Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare's stance is becoming more defensive by default. Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge, Matthew Prince, Cloudflare's CEO and co-founder shared in a statement. Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent.