1 linein robots.txt to say no to AI training, once supported
Websites may soon get one simple way to tell every big AI company not to use their pages to train AI. OpenAI, Google and Microsoft said on 30 October that they will support a shared signal, and Anthropic is already building it in.
Key points
- How it works: a site owner adds a short rule to robots.txt A small text file on a website that tells automatic visitors, like search engine and AI crawlers, which pages they may or may not use., the file that already tells Web crawler A program that visits web pages automatically and saves what it finds, for search engines or for training AI. where they can go. A new line, User-Agent: __usage_ai-training, followed by Disallow means 'don't train on this', and Allow means training is fine. You can also mix them, for example blocking the whole site but allowing a press kit folder (ITmedia, IThome).
- It only covers training. It does not stop crawlers from visiting or search engines from showing your pages. If a site adds nothing, that counts as no stated preference, not a yes or a no.
- Timing differs. Anthropic is first. OpenAI aims for the next few months, Google and the others aim for 2027, and Microsoft says it will start reading the signal on 1 February 2027. Google will also treat a 'don't train' signal as a no for its own Google-Extended setting unless a site says otherwise. Meta said it supports the idea but gave no plan.
- It is still a proposal. The idea comes from an Internet-Draft An early, public proposal for how something on the internet should work. Anyone can write one, and it has no official standing until a standards group approves it. written by Anthropic's Kevin Kelley, which has no official standards status yet. Google says the switch also fits the EU AI Act's code of practice, which treats robots.txt as a machine-readable way to say no.
Why it matters: Today every AI company has its own Opt-out A way to say no to something that would otherwise happen by default, like having your website used to train AI., so site owners have to chase each one. One shared switch would make saying no to AI training a single line, if the companies follow through.
Sources (2)
Was this useful?
Tap to vote. Hold, right-click or press Shift+Enter to add a reason or a note first.
