{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"Tech on the Rocks","title":"Reinventing Stream Processing: From LinkedIn to Responsive with Apurva Mehta","html":"","width":"100%","height":180,"duration":3493,"description":"Summary\nIn this episode, Apurva Mehta, co-founder and CEO of Responsive, recounts his extensive journey in stream processing—from his early work at LinkedIn and Confluent to his current venture at Responsive.\nHe explains how stream processing evolved from simple event ingestion and graph indexing to powering complex, stateful applications such as search indexing, inventory management, and trade settlement.\nApurva clarifies the often-misunderstood concept of “real time,” arguing that low latency (often in the one- to two-second range) is more accurate for many applications than the instantaneous response many assume. He delves into the challenges of state management, discussing the limitations of embedded state stores like RocksDB and traditional databases (e.g., Postgres) when faced with high update rates and complex transactional requirements.\nThe conversation also covers the trade-offs between SQL-based streaming interfaces and more flexible APIs, and how Responsive is innovating by decoupling state from compute—leveraging remote state solutions built on object stores (like S3) with specialized systems such as SlateDB—to improve elasticity, cost efficiency, and operational simplicity in mission-critical applications.\nChapters\n00:00 Introduction to Apurva Mehta and Streaming Background\n08:50 Defining Real-Time in Streaming Contexts\n14:18 Challenges of Stateful Stream Processing\n19:50 Comparing Streaming Processing with Traditional Databases\n26:38 Product Perspectives on Streaming vs Analytical Systems\n31:10 Operational Rigor and Business Opportunities\n38:31 Developers' Needs: Beyond SQL\n45:53 Simplifying Infrastructure: The Cost of Complexity\n51:03 The Future of Streaming Applications","thumbnail_url":"https://img.transistorcdn.com/UA-KG4zXoEeEh4XuiVMdD9yAhMnDJ2YVfAhHWI5inpc/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80ZWI0/YjY4MDc1YmU2MDI0/MTNkNjYwZWYwYzky/ZGEyNi5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}