The four largest AI developers have quietly agreed to honor a single AI training opt-out signal buried in a website's robots.txt file. For the first time, OpenAI, Google, Microsoft and Anthropic are lining up behind one rule for how the open web says no. The fight over who owns the internet's training data just entered a decisive new phase.

By Blumefield | October 31, 2026

One rule, four rivals

For three years, publishers who wanted to keep their work out of artificial intelligence models faced an ugly choice. They could block the crawlers entirely and vanish from search, or they could leave the gates open and watch their archives feed systems they never agreed to train. This week that calculus changed. OpenAI, Google, Microsoft and Anthropic have each signaled that they will honor a single AI training opt-out instruction placed inside a website's robots.txt file, the plain text rulebook that has governed web crawling since the 1990s. For the first time, the four companies that dominate the frontier of AI are converging on one way for the open web to say no.

The mechanism is deceptively small. A site owner adds a line reading User-Agent: __usage_ai-training to the robots.txt file, follows it with Disallow: /, and the four companies pledge to treat every page on that domain as off limits for model training. An optional Allow: line can carve out exceptions, so a newsroom could withhold its reporting while releasing its press kit. The syntax borrows the grammar web operators have used for decades, which is precisely the point. Nobody has to learn a new system to use it.

The commitments arrive on a staggered schedule rather than in one coordinated launch. Anthropic says it has already begun implementing support. OpenAI has told publishers that its support will arrive in the coming months. Microsoft has named a date, February 1, 2027, in a Bing announcement, and Google has said it intends to recognize the control in 2027, framing it as a draft proposal it will adopt alongside other providers. The absence of a single switch date is the first hint that this is a convergence of self interest, not a treaty.

What the AI training opt-out actually controls

Underneath the robots.txt line sits a slower, more deliberate effort at the Internet Engineering Task Force, the volunteer body that has standardized the web's plumbing for decades. Its AI Preferences working group has been drafting a vocabulary for expressing how content may be used, and the shared AI training opt-out leans directly on that work. The group's draft, authored by Paul Keller of Open Future and Mozilla's Martin Thomson, defines a short list of usage categories that a publisher can permit or refuse.

Three categories carry most of the weight. The label train-ai covers using a work to modify the learned parameters of a generative model, which is training in the ordinary sense. A second label, ai-use, covers feeding a work into a model at the moment of a query, the retrieval step that powers answer engines. A third, search, covers indexing a page in order to point users back to it. The AI training opt-out targets only the first of these. A publisher can refuse to be a training example while remaining fully discoverable.

That precision is the quiet achievement here. Earlier opt-out gestures treated the web as a single switch, on or off. By naming the specific use a publisher objects to, the standard lets a site owner draw a line that matches how the technology actually works, separating the model that learns from a work from the index that merely finds it.

Why search is the escape hatch

That distinction is not an accident. It is the concession that made the agreement possible. Google has spent two years offering a separate control called Google-Extended, which lets sites refuse training without disappearing from the results pages that still drive most of the web's traffic. The company has been explicit that the control does not affect a site's ranking or inclusion in search. The new shared token extends that logic across the industry in one line.

For the AI companies, keeping search outside the AI training opt-out protects the one thing they cannot afford to lose. Discovery is the lifeblood of the open web, and any standard that forced publishers to choose between refusing training and accepting search invisibility would simply be ignored, the way blunt crawler blocks have been. By splitting training from indexing, the four firms hand publishers a refusal they can actually use, and hand themselves a signal publishers are far more likely to adopt.

There is a wrinkle. Google has indicated that if a site uses the shared token to opt out but does not separately address Google-Extended, it will read that silence as a Google-Extended refusal, which also touches the grounding that feeds its Gemini apps. The overlap shows how fast a tidy standard runs into each company's own tangle of crawlers, products and legacy controls.

A standard with no enforcement

For all the coordination, the AI training opt-out is a request, not a lock. Robots.txt has always been a voluntary convention, formalized as RFC 9309 but backed by nothing stronger than good faith. OpenAI has described the token as a coordination mechanism rather than a security control, which is a careful way of saying that a scraper determined to ignore it faces no technical barrier at all. The flood of lesser known crawlers harvesting data for resale will not feel bound by a signal the big four chose to honor among themselves.

The draft itself is candid about its limits. It is an individual submission, not an endorsed standard, and the IETF notes bluntly that such drafts carry no formal standing in its process. The vocabulary document warns, in plain terms, that it does not ensure preferences are actually followed. And crucially, adding the line today does nothing about the past. A robots.txt entry cannot reach into datasets already scraped or models already trained. Years of archives that shaped today's systems remain inside them, beyond recall, and no opt-out written now can pull them back out.

The real contest over the web's data

The timing explains the sudden cooperation. The relationship between AI developers and the people who make the content those models learn from has curdled into open conflict, a breakdown Blumefield has tracked through the year. Lawsuits over training data are grinding through the courts. Publishers have watched referral traffic from AI powered answers collapse even as their work is absorbed into the systems delivering those answers. Some, like Reddit, have begun walling off their archives and striking paid licensing deals rather than trusting any honor system at all.

A voluntary AI training opt-out lets the largest firms show regulators, courts and lawmakers that the industry can govern itself before someone forces the question through legislation. It is cheaper to offer publishers a checkbox than to litigate the ownership of the internet's text one case at a time. That motive does not make the gesture empty, but it does explain why four fierce competitors found it easy to agree.

Whether the settlement holds depends on adoption and good faith, neither of which is guaranteed. If publishers embrace the token and the four companies honor it, the web gains its first shared language for consent in the AI era. If the signal becomes a box the big players tick while smaller scrapers run wild and already trained models stay untouched, it will read less like a truce and more like public relations. Either way, the AI training opt-out marks the moment the industry conceded that taking everything by default was no longer tenable. Readers tracking the economics of that shift can follow it in Blumefield's continuing coverage of the data wars.

Primary sources: IETF AI Preferences vocabulary draft, OpenAI, Google Search Central, and Microsoft Bing.