OpenAI commits to a shared robots.txt signal for AI training opt-outs

OpenAI says it will honor a common signal in robots.txt that lets a website say whether its content may be used for foundational generative AI training. In its announcement, OpenAI ties the signal to the IETF's AI Preferences working group. It says other participants in that group are announcing the same signal alongside it.

Your site works the same as it did yesterday. OpenAI says it aims to implement the signal "in the coming months" and will share implementation details when it does.

What the announcement says

• The signal lives in robots.txt, the file you already use to manage crawlers.

• It covers one use of your content: foundational generative AI training.

• It is voluntary. OpenAI writes that it will "encourage other AI providers to honor the signal", so it only reaches providers that choose to take part.

• It does not replace OpenAI's existing crawler controls, which "will continue to apply and be respected".

What it leaves open

OpenAI's post does not name the other providers taking part, and it gives no implementation date. It links to the signal but does not explain how OpenAI will read it. It also does not say what happens when the training signal and a rule for an individual crawler disagree. Until OpenAI publishes its implementation update, any answer to that is a guess.

What to do with it

The useful part is the scope. OpenAI says that until now, site operators had to identify individual crawlers and keep separate rules for every provider. One preference aimed only at training gives you a cleaner way to decline model training without touching the crawlers behind AI search and retrieval.

That separation is where brands lose visibility. A blanket block meant to keep content out of training can also stop an assistant from fetching your pages and citing them when someone asks about your category. If you want to be named in ChatGPT answers, keep the training choice and the search choice as separate decisions.

For now:

• Audit robots.txt for rules that block AI crawlers wholesale, and work out which of them were only ever meant as training opt-outs.

• Make the training decision as a business decision about your content, apart from your decisions on search crawlers.

• Hold off on adding the new signal until OpenAI publishes its implementation details. Its post does not yet say how the signal will be read.

We will update this page when OpenAI publishes its implementation.