{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"Ship faster, measure better: experimentation in the age of AI","html":"","width":"100%","height":180,"duration":1568,"description":"SummaryHow do you know if the thing you just shipped actually worked? On this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Kevin Yang, Executive Director and Head of Experimentation at JPMorgan Chase, who has spent six years building experimentation across Chase's digital platforms. Kevin shares how his team turned experimentation into more than a billion dollars of estimated value, why the losing experiments matter more than the winners, and the simple chart exercise he uses to prove that a million-dollar change is invisible without a control group. He and Ashley also dig into measuring engagement without chasing vanity metrics, planning for failure to defeat confirmation bias, and why AI is pushing experimentation into a golden era. It's a practical look for product managers, data scientists, and engineers at how a bank operating at massive scale makes better decisions.Chapters00:00 Welcome to the experimentation edge01:45 Kevin's role leading experimentation at chase04:15 Why chase invested in experimentation06:45 A billion dollars and the value of losers12:45 Plan for failure to beat confirmation bias14:30 The million dollar change you can't see18:45 Sharing learnings and experimentation wrapped20:45 Engagement without vanity metrics22:00 Experimentation's golden era with AI23:30 Why AI needs more experimentation, not lessTakeawaysChase estimates over a billion dollars of value from experimentation, and most of the lasting learning comes from the losing tests, not the winners.A control group is non-negotiable: at scale, a change worth millions is invisible under noise and seasonality, and no one can spot it by eye.Treat engagement carefully. For a bank, more time in the app isn't a win; trust, fast task completion, and healthy repeat engagement are.Plan for failure before you run a test. A pre-built playbook for a loss prevents confirmation bias and keeps teams from gaming the metrics.AI is ushering in a golden...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}