{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Media Copilot","title":"How AI scrapers are breaking the internet’s honor code, with TollBit’s Toshit Panigrahi","html":"","width":"100%","height":180,"duration":2954,"description":"There's something in TollBit's latest State of the Bots report that really jumped out at me. The major AI companies—notably Meta, OpenAI, and Google—have essentially given themselves permission to ignore robots.txt for their user agents. In the big picture of AI scraping it might seem like a detail, but it feels to me like a turning point from liberally interpreting the mechanisms for protecting content to brazenly flouting them.I put this question directly to TollBit CEO Toshit Panigrahi when he joined me on this week’s Media Copilot podcast. A little background on TollBit: It’s a startup that specializes in helping publishers monetize their content when AI bots come calling. As everyone knows, the AI internet is rising fast—people are using chatbots to search the web, and those chatbots are giving them answers, not links. And if there was any doubt that this was a fad or not very impactful, Tollbit's most recent State of the Bots report for the first quarter 2025 puts that notion to rest. AI scraping is way up, and it's very quickly becoming comparable to scraping from search engines like Google or Bing.That's one of several eye-opening, and in some cases worrisome, data points in the report, and Toshit and I dive deep into the most important ones. Like I mentioned, AI companies are now starting to openly ignore long-standing internet standards, effectively letting them crawl any \"publicly available\" content with impunity. What happens if this trend is left unchecked, and what can publishers do about it? We get into all the details, and their implications.If you're stressed about what agents are going to mean for the media ecosystem, you definitely don't want to miss this one.This episode covers:Why AI bots are ignoring robots.txt and what it means for publishers right nowHow the rise of agent-based scraping is quietly reshaping the open webThe surprising data behind how much AI scrapers take vs. how little they give backWhat a “new value exchange” with AI...","thumbnail_url":"https://img.transistorcdn.com/4EiFqLM4OC9vg9_Tigcvzf0FJU4e68DVprGgpAUDU4M/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS8zZGY3/ZTlmNDY3NDc0NjVm/NmNjMjNmZGM1ODNh/Y2JiYS5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}