AI Scraping and the Open Web

Tussles between websites and scrapers are not new. Almost since there has been a web to scrape, people have been scraping it and using the data to make search engines, caches and archives, analytics platforms, research datasets, and more. And for almost as long, some websites have objected and tried to stop the scraping with a mix of technical and legal measures. Broadly speaking, scrapers cause two kinds of problems for websites. First, they create bad traffic: millions of automated requests that no human will ever see.