Four of the biggest AI developers have backed a single robots.txt rule that lets a website say it does not want its pages used to train AI models. Instead of blocking each company’s crawlers one by one, a site can add a group headed “User-Agent: __usage_ai-training” and list the paths it wants kept out of training. The token is set out in an Internet-Draft written by an Anthropic engineer and published on 30 October. It is an individual draft, not an adopted standard.
Timelines differ. Microsoft says the new preferences take effect as Bing crawls from 1 February 2027, without changing a site’s eligibility for Bing search results. OpenAI says it aims to support the signal in the coming months. SiliconANGLE reports that Anthropic already honours the token and that Google plans support during 2027.
Why it matters: publishers have long complained that opting out of AI training means tracking a patchwork of crawler names. One line is simpler. But it is voluntary and relies on crawler operators to comply, and it does not on its own cover AI tools that fetch pages live to answer questions; Microsoft handles that use, which it calls grounding, with separate controls.