Four AI companies agree to read one robots.txt switch: the IETF draft names it __usage_ai-training, Anthropic already honors it, Microsoft and Google say 2027

Sources first. This article rests on three primary documents: OpenAI's official post Making it easier for websites to express AI training preferences, published October 30, 2026; Google Search Central's post Supporting a shared AI training control for websites, published the same day and authored by John Mueller; and the IETF Datatracker draft draft-kelley-aipref-training-preference-signal-00, authored by Kevin Kelley, last updated 2026-10-30 and expiring May 3, 2027. Microsoft's and Anthropic's participation comes from The Verge's reporting. I could not locate their own blog posts, so every sentence about them below is labelled as a media report.

What the draft actually changes. The abstract says it describes how a service operator binds usage preferences such as no AI training to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line. The next sentence is the important one: it introduces no new directive names and no new grammar, and the path language, longest-match precedence and group structure of robots.txt are unchanged. So this is not a new protocol. It is a manual for expressing a new meaning with the old one.

What the token looks like. Section 4.1 defines it as __usage_ai-training, conforming to the robots.txt product-token syntax of letters, underscores and hyphens. Consumers must match policy tokens exactly apart from case, and the draft adds a note that some deployed parsers match more flexibly, which is precisely why the double-underscore prefix exists: to make accidental matches unlikely. One line of rules, built on an acknowledgement that deployed parsers do not behave identically.

The genuinely awkward part is the wildcard. Section 4.2 is blunt: wildcard groups, meaning User-Agent: * with a Disallow underneath, must not be interpreted as matching the policy token. The consequence is direct: if your robots.txt only has User-Agent: * and Disallow: /, then under this draft you have expressed no preference about AI training. Absence, not refusal. Only an explicit User-Agent: __usage_ai-training line with an Allow or Disallow beneath it counts. That single clause decides how site-wide rules have to be written.

Crawler rules and usage preferences are two different things. Section 4.3 requires consumers to interpret RFC 9309 signals exactly as they would without this document: ExampleBot still checks only its own rules. Section 4.4 adds that the policy token is not a directive for crawling behavior, though a consumer may choose to let preferences inform crawling, for instance by refraining from fetching resources whose operators expressed a preference against the purposes it is crawling for. Put plainly: this governs whether content is used for training, not whether it gets fetched. The latter still belongs to each product token.

Precedence is spelled out too. Section 4.5 says specific expressions should take precedence over general ones, that consumers may document that they treat rules matching proprietary tokens as more specific than __usage_ai-training, and, more concretely, that private agreements may override preferences expressed through this mechanism. Read together, those three sentences define its position: a set of public defaults that contracting parties can vary, and that each consumer interprets for itself.

What OpenAI said. The official post states that OpenAI will honor a common signal in robots.txt, framed explicitly as voluntary, on the grounds that operators previously had to identify individual crawlers and maintain separate rules for each provider. The timeline is restrained: the aim is to implement the signal in the coming months, with another update published when it ships. The post also says the new signal will not replace existing crawler controls, that the bots documented on developers.openai.com continue to apply, and it closes by encouraging other providers to honor the signal. This is a commitment, not a launch.

Google is more specific because it already owns Google-Extended. John Mueller's post says that when a site uses the policy token to opt out and does not explicitly specify Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended, and that other providers can handle it similarly, so a site can opt out once or pick and choose per provider. One reverse clause is easy to miss: websites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks via grounding in Gemini Apps. So a narrow gap still exists between opting out of training and keeping answer-surface visibility.

The timelines do not line up, and that is the detail worth remembering. Google writes that it aims to support the new control along with other AI providers in 2027; OpenAI writes that it aims to implement in the coming months. The Verge's version is finer: Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says coming months. The same draft, with roughly a year of spread in implementation. The author, Kevin Kelley, is described by The Verge as a technical lead for standards at Anthropic, so the drafter moved first, which is not unusual in standards history but does mean site operators will spend the next year maintaining both spellings.

Microsoft's detail deserves separate treatment because it may end up being the most useful one. The Verge reports that Microsoft will offer more granular controls letting publishers opt out of AI training or of AI grounding, the step that points a model at a verified source to reduce hallucination, and that sites still appear in Bing search results regardless of their preferences. If that description is accurate it is more useful than one blanket token: what most content sites actually want is not to vanish, but to stay out of training while keeping the chance to be cited and clicked.

Context matters or the significance gets misjudged. Google's own post says the proposal reflects feedback from publishers, civil society groups and regulators, and is consistent with the EU Code of Practice for General Purpose AI under the EU AI Act, which recognises robots.txt as a primary method for publishers to express machine-readable preferences. The Verge notes two competing routes: the Really Simple Licensing standard that lets publishers charge crawlers, and Cloudflare blocking AI crawlers by default. So in late 2026 the industry is running three schemes in parallel: voluntary declaration, paid licensing, and infrastructure-level blocking. This announcement is the mildest of the three.

Boundaries, stated so this is not mistaken for protection already in force. First, the draft is an Active Internet-Draft (individual), and the IETF Datatracker states plainly that it is not endorsed by the IETF and has no formal standing in the IETF standards process. Second, OpenAI says coming months and Google says 2027, so until then the rule has no practical hold on your site. Third, the Microsoft and Anthropic positions come only from The Verge; I did not find either company's own post. Fourth, even if everything ships, Section 4.5 lets private agreements override the signal and Section 4.4 lets each consumer decide whether it affects crawling. Fifth, this article does not forecast whether the mechanism will change how content is actually used, nor whether robots.txt is an adequate governance tool; the draft answers neither question.