AI training opt-out
AI training opt-out refers to the mechanisms allowing a content publisher to prohibit the use of its pages to train language models. It is expressed through robots.txt directives, specific metadata, or declared content policies.
In practice
OpenAI, Google, and Anthropic respect specific robots.txt directives — GPTBot, Google-Extended, and ClaudeBot respectively — allowing publishers to block crawling of their content for training corpora. This may reduce visibility in future generative responses but preserves content exclusivity. The decision depends on the business model: a paywalled publisher benefits from blocking to drive subscriptions; a service provider seeking awareness benefits from permitting crawls.