Google, Microsoft, Anthropic, and OpenAI Announce New Crawling Standard to Protect Website Content
Google LLC, Microsoft Corp., Anthropic PBC, and OpenAI Group PBC have recently unveiled plans to support a new website crawling standard aimed at enabling content creators to block AI developers from utilizing their material for model training.
This new technology seeks to provide website operators with the ability to manage how artificial intelligence interacts with their content, utilizing a familiar mechanism: the “robots.txt” file. Traditionally used to guide search engine crawlers about accessible content, this file can prevent scraping entirely or restrict access to specific pages as needed. Furthermore, it allows webmasters to manage the frequency of crawler visits.
Previously, website operators had to make numerous adjustments to their robots.txt file to prevent various AI providers from scraping their data for training purposes. This required adding separate entries for each AI provider, complicating compliance and accessibility. As many AI firms may employ multiple scrapers, the process could become exceedingly cumbersome for website owners.
The newly proposed crawling standard will simplify this task. By enabling operators to block all data-collection scrapers for generative AI models with a single entry in their robots.txt file, the process will become much less labor-intensive. However, it is important to note that this block will not extend to models designed exclusively for non-generative tasks.
The existing structure of robots.txt files adheres to the standardized Robots Exclusion Protocol, which Google, Microsoft, Anthropic, and OpenAI aim to enhance with their new crawling standard. A draft proposal for this initiative has already been submitted to the Internet Engineering Task Force (IETF), a nonprofit organization responsible for maintaining the Robots Exclusion Protocol.
In addition to this development, Microsoft has announced its own crawling feature known as AI Grounding. This feature will allow an AI model to access a webpage for the purpose of answering user queries, as opposed to collecting data for training. Website operators will have the capability to restrict this access as well, extending this functionality to third-party AI tools that utilize Microsoft technology for processing prompts.
OpenAI has committed to providing similar controls, with both OpenAI and Microsoft planning to adopt the new crawling standard in the coming months. Meanwhile, Google anticipates following suit next year. Although Meta Platforms Inc. has expressed support for the initiative, it has yet to confirm its intentions to implement the standard. This collaborative effort of major tech firms underscores the growing importance of content protection in the evolving landscape of artificial intelligence and web technology.