Openai, google, microsoft and anthropic support common ban on ai training

Currently, a site has to prescribe rules for robots or tokens of each company separately: for openai this is gptbot, for anthropic - claudebot, for google - google-extended. The new mechanism replaces them with one rule: the strings "user-agent: __usage_ai-training" and "disallow: /" close the entire resource for training, but it is also possible to limit individual sections. The proposal was prepared by kevin kelly from anthropic and submitted to ietf. So far, this is an individual draft without the status of a standard, discussion in the aipref working group continues.

The terms are different for companies. According to it home, which kod.ru refers to, anthropic has already started implementing; openai plans to add support in the coming months. Microsoft will start applying the settings from february 1, 2027, gradually, when re-scanning sites, google plans to do this during 2027. If there are no individual rules for google-extended, the general ban will apply to it, but google-extended's own settings will retain priority.

Microsoft also separated training from using materials when generating answers for bing and copilot. Such use is prohibited by a meta tag or http header, while pages remain in regular bing search. The company warns that the ban will reduce the site's presence in ai answers and possible traffic from these services. Without the owner's instructions, microsoft considers both options acceptable.

Meta representative andy stone supported the idea of a common signal, but did not report on implementation plans. The new directives express the site owner's preferences and do not technically protect against copying: their observance depends on the companies themselves.