{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Tech on the Rocks","title":"How Denormalized is Building ‘DuckDB for Streaming’ with Apache DataFusion","html":"","width":"100%","height":180,"duration":3721,"description":"In this episode, Kostas and Nitay are joined by Amey Chaugule and Matt Green, co-founders of Denormalized. They delve into how Denormalized is building an embedded stream processing engine—think “DuckDB for streaming”—to simplify real-time data workloads. Drawing from their extensive backgrounds at companies like Uber, Lyft, Stripe, and Coinbase. Amey and Matt discuss the challenges of existing stream processing systems like Spark, Flink, and Kafka. They explain how their approach leverages Apache DataFusion, to create a single-node solution that reduces the complexities inherent in distributed systems.\nThe conversation explores topics such as developer experience, fault tolerance, state management, and the future of stream processing interfaces. Whether you’re a data engineer, application developer, or simply interested in the evolution of real-time data infrastructure, this episode offers valuable insights into making stream processing more accessible and efficient.\nContacts & Links\nAmey Chaugule\nMatt Green\nDenormalized\nDenormalized Github Repo\nChapters\n00:00 Introduction and Background\n12:03 Building an Embedded Stream Processing Engine\n18:39 The Need for Stream Processing in the Current Landscape\n22:45 Interfaces for Interacting with Stream Processing Systems\n26:58 The Target Persona for Stream Processing Systems\n31:23 Simplifying Stream Processing Workloads and State Management\n34:50 State and Buffer Management\n37:03 Distributed Computing vs. Single-Node Systems\n42:28 Cost Savings with Single-Node Systems\n47:04 The Power and Extensibility of Data Fusion\n55:26 Integrating Data Store with Data Fusion\n57:02 The Future of Streaming Systems\n01:00:18 intro-outro-fade.mp3","thumbnail_url":"https://img.transistorcdn.com/UA-KG4zXoEeEh4XuiVMdD9yAhMnDJ2YVfAhHWI5inpc/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80ZWI0/YjY4MDc1YmU2MDI0/MTNkNjYwZWYwYzky/ZGEyNi5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}