One robots.txt line for AI crawlers is overdue, but it has no teeth

What does it say about the web that the biggest AI companies had to agree on a way to be told "no" before anyone could count on them hearing it? The new shared robots.txt signal is good news. It is also a reminder of how much of the internet still runs on manners.

What was agreed, in short

According to aiweekly.co, citing The Verge, Google, Microsoft, Anthropic and OpenAI said on October 30 that they will honor a single robots.txt policy token, __usage_ai-training. One line would let a site opt its content out of AI model training across all of those companies' crawlers, instead of one rule per provider. Another outlet, zglg.work, says the draft has gone to the IETF, with Microsoft and OpenAI implementing in the coming months and Google following next year.

That is the whole deal: a shared vocabulary, not a new set of obligations.

The fix for a real mess is welcome

Credit where it is due. Today a publisher who wants out has to keep track of every crawler name, add a rule for each, and update the list whenever a new bot shows up. That is a tax on the people who make the content, and the small ones probably pay it hardest. One token that all the big names understand is the kind of boring plumbing the web needs. The fact that it goes through the IETF, and not a blog post, also matters: standards bodies are slow, but they tend to produce things other companies can implement without asking permission.

We would also say the "four giants agree" framing slightly oversells it. Agreeing on a syntax costs these companies very little. Agreeing to be audited, or to pay, would cost them far more, and the reports we found describe nothing of the kind.

A rule nobody can check is a suggestion

Here is the problem. robots.txt has never been enforced by anything except the good behavior of the crawler. Golem notes that most AI crawlers dutifully ignore it, which is the reason it is supposed to become a binding standard. In our view, that makes the new token a fix for the paperwork, not for the behavior: whatever its benefits for publishers, it still rests largely on the goodwill of the companies involved.

Think about who this actually binds. The four signatories now have a public commitment, and that has some value: a broken promise is at least visible. But the scrapers that worry publishers most are probably the ones that never signed anything. A token they never agreed to read is not going to stop them.

There is a second, quieter weakness. One write-up, from tech-insider.org, lists "opt-outs arrive after the data is gone" among the criticisms of the standard. We think that is right. A signal that applies from now on does nothing about the models already trained on yesterday's crawl. It is a door being closed after the house has been walked through.

Where this leaves publishers

Our reading is that the token is necessary and nowhere near sufficient. What publishers want is leverage, and leverage would come from things the standard, as described, does not seem to offer: a way to verify compliance and a price for access. Without them, an opt-out is a polite way to say "please ask first," and the companies with the most to gain from not asking are exactly the ones deciding whether to listen.

So the practical advice is simple. Add the token when the crawlers start supporting it, because it is free and it creates a record of your intent. But do not treat it as protection. If your content has real value, pair it with technical blocking and, where you can, licensing terms. And watch the next step closely: the moment a regulator or a court treats an ignored robots.txt as evidence of something, this line could become much more than a courtesy. Until then, it is a good idea with no teeth.