Google, Microsoft, Anthropic and OpenAI have announced they will support a new standard that lets websites block all crawlers collecting training data for generative models with a single robots.txt entry, SiliconANGLE reports. Today a separate entry is needed for each provider and each crawler. The draft has been submitted to the IETF, which maintains the Robots Exclusion Protocol. Microsoft and OpenAI will support it in the coming months and Google next year, while Microsoft is also adding a setting for the use of pages in its tools' answers.

What it means: For website owners, controlling what is used to train models has so far been tedious and incomplete. Every new crawler needs a new entry, and many people never update their list. A single signal recognised by the big providers simplifies the process considerably. The distinction between training and using a page to answer a user's question also matters: a publisher may not want its content to train models but may want to appear as a source in answers and receive visits. In the EU, the copyright directive lets rights holders opt out of text and data mining in a machine-readable way, and the AI Act asks providers of general-purpose models to respect such reservations. A recognised standard makes that opt-out clearer and easier to prove. For Greek news sites, online shops and companies with original content, it is worth deciding their policy now and applying it as soon as the standard is final, while checking that they do not accidentally block search engines too.

Read the original on SiliconANGLE

Related service: Custom software

Related stories