{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Oxide and Friends","title":"Death by Uptime","html":"","width":"100%","height":180,"duration":3637,"description":"We hit a new (and disturbing!) failure mode recently when a production rack that had been up for several months saw every (!) compute sled's service processor become simultaneously unresponsive. Bryan and Adam were joined by the members of the Oxide team who debugged the vexing issue -- and reached its surprising root cause.\nIn addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Cliff Biffle, Matt Keeter, and Will Chandler.\nPreviously, on Oxide and Friends:\nOxF s05e03 – Holistic Engineering with Robert Mustacchi\nOxF s04e14 – Rebooting a datacenter: A decade later\nOxF s01e26 – The Pragmatism of Hubris\nOxF s05e20 – Debugger-Driven Development (omdb)\nOxF s05e07 – Transparency in Hardware/Software Interfaces\nOxF s05e31 – Futurelock\nOxF s05e33 – A Grown-up ZFS Data Corruption Bug\nSome of the topics we hit on, in the order that we hit them:\nhubris #2304: STM32H7 Ethernet driver stops yielding CPU after many packets\ngist — Summarizing the Hubris side of investigations\nMatt's blog: Hunting a spooky ethernet driver bug\nIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!","thumbnail_url":"https://img.transistorcdn.com/cVYTzMGAZ_7gNn2i7iJYSuWlQOM4wHdJAzEuWrnLo9A/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80OGI3/NjA5NWY4Yjc4NzE0/ZGI5ZmMzZmY4MjFi/YThiYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}