New data shows AI bots pushing deeper into the web, prompting publishers to roll out more aggressive defenses. The viral virtual assistant OpenClaw--formerly known as Moltbot, and before that Clawdbot--is a symbol of a broader revolution underway that could fundamentally alter how the internet functions. Instead of a place primarily inhabited by humans, the web may very soon be dominated by autonomous AI bots. A new report measuring bot activity on the web, as well as related data shared with WIRED by the internet infrastructure company Akamai, shows that AI bots already account for a meaningful share of web traffic. The findings also shed light on an increasingly sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to keep them out.

Last year, internet infrastructure firm Cloudflare launched tools enabling its customers to block AI scrapers. Today the company has taken its fight against permissionless scraping several steps further. It has switched to blocking AI crawlers by default for its customers and is moving forward with a Pay Per Crawl program that lets customers charge AI companies to scrape their websites. Web crawlers have trawled the internet for information for decades. Without them, people would lose vitally important online tools, from Google Search to the Internet Archive's invaluable digital preservation work.

The 28-year-old founders of TollBit, a New York-based startup that is all of six months old, think we're living in the "Napster days" of AI. Just like people of a certain generation downloaded digital music, companies are ripping off vast swaths of the internet without paying the rights holders. They want TollBit to be the iTunes of the AI world. "It's kind of the Wild West right now," Olivia Joslin, the company's co-founder and chief operating officer, told Engadget in an interview. "We want to make it easier for AI companies to pay for the data they need."

Perplexity, a company that describes its product as "a free AI search engine," has been under fire over the past few days. Shortly after Forbes accused it of stealing its story and republishing it across multiple platforms, Wired reported that Perplexity has been ignoring the Robots Exclusion Protocol, or robots.txt, Technology website The Shortcut also accused the company of scraping its articles. Now, Reuters has reported that Perplexity isn't the only AI company that's bypassing robots.txt Reuters said it saw a letter addressed to publishers from TollBit, a startup that pairs them up with AI firms so they can reach licensing deals, warning them that "AI agents from multiple sources (not just one company) are opting to bypass the robots.txt