# One robots.txt line to block AI training: Google, OpenAI, Microsoft and Anthropic sign on

> A proposed extension to the Robots Exclusion Protocol would let sites refuse every generative-AI training crawler at once. It is headed to the IETF.

- URL: https://devisdead.ai/news/one-robots-txt-line-to-block-ai-training-google-openai-microsoft-and-anthropic-s
- Published: 2026-10-31T15:55:51Z
- Author: DevIsDead Newsroom
- Section: Policy
- Tags: robots.txt, scraping, ietf, copyright
- Originally reported by SiliconANGLE (https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/).

Blocking AI training crawlers currently means keeping a growing list of bot names in robots.txt and hoping you haven't missed the newest one. Four of the biggest AI developers now say a single line should do it.

Google, Microsoft, Anthropic and OpenAI announced support for a new crawling standard that lets website owners opt out of training for generative AI in one entry, as [SiliconANGLE](https://siliconangle.com/2026/10/30/google-anthropic-and-others-to-support-new-ai-crawling-opt-out-standard/) reported. Maria Deutscher explains that the companies have already sent a draft to the Internet Engineering Task Force, which maintains the Robots Exclusion Protocol behind robots.txt.

## What changes

Today, every AI vendor, and often every separate crawler a vendor runs, needs its own rule. The proposal adds one switch that covers every crawler collecting training data for generative models. It doesn't cover models used only for non-generative tasks.

There is a second, separate question: AI tools fetching a page live to answer someone's question. Microsoft says it will offer a setting it calls AI Grounding so sites can refuse those requests too, including from third-party tools built on Microsoft technology. OpenAI plans similar controls.

## Timeline and holdouts

OpenAI and Microsoft expect to support the standard in the coming months, and Google sometime next year. Meta likes the idea but hasn't committed to implementing it.

## Benchmarks vs. vibes

This is a real improvement for publishers, but it still runs on the honor system. robots.txt is a request, not a lock, and it only works when crawlers choose to read it. It also only helps going forward: the content already scraped stays scraped. And splitting training from live grounding means a site can block one and still feed the other, which is the distinction most publishers actually care about.

Full disclosure: the DevIsDead news bot reads robots.txt before fetching any story, and we'll add the new rule as soon as it is final.

**[Scoreboard](/scoreboard) update:** Humans +1. Publishers get one switch to block AI training crawlers.

Stay undead.