Radio Hoosh: AI Deep Dives

OpenAI, Google, Microsoft and Anthropic back one shared robots.txt setting that lets websites say whether AI companies may train on their content, replacing separate blocks for each AI crawler.

Show Notes

AI-generated: both hosts are AI voices (Google NotebookLM), and the conversation is based on the sources listed below.

In this episode we look at a new shared opt-out for AI training. OpenAI, Google, Microsoft and Anthropic have agreed on one simple robots.txt setting that websites can use to say whether AI companies may learn from their content. Today, a website owner who wants to keep their pages out of AI training has to block each AI company separately. With this setting, one line covers all of them. We also go through how OpenAI says it will follow the new rule.

Why it matters: for the first time, the biggest AI companies agree on one shared way for websites to say yes or no to AI training. It arrives while news publishers and AI companies are fighting in court over the use of their articles, so a common standard for consent and control over web content matters to publishers, site owners and anyone following AI and copyright.

  • One robots.txt line can now opt a website in or out of AI training across several AI companies
  • OpenAI, Google, Microsoft and Anthropic all back the shared setting
  • It replaces blocking each AI crawler one by one
  • OpenAI has explained how it will follow the setting
  • The move comes during court fights between news sites and AI companies

Sources:

What is Radio Hoosh: AI Deep Dives?

Radio Hoosh takes each day's biggest AI story (a new model, a research paper, a lab announcement or a developer tool) and unpacks it in a 20–45 minute conversation. Both hosts are AI-generated voices (Google NotebookLM), and episodes are produced automatically from public sources such as Hacker News and Hugging Face Daily Papers, then published by Kasra Qasempour. "Hoosh" is Persian for intelligence; a Persian edition is available as رادیو هوش.