{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"PayPal's $180 million experimentation win","html":"","width":"100%","height":180,"duration":1899,"description":"Summary\nGaurav Sethi joins Ashley Stirrup on The Experimentation Edge to explain how PayPal's experimentation program went from 700 to 800 tests a year with four week readouts to 2,587 experiments a year and roughly $180 million in measured impact. Gaurav inherited a reported win rate of 55% to 60%, four times the industry average, and traced it to experiments logging assignment data instead of exposure data, carrying 25% to 30% dilution. The conversation covers the exposure event his team introduced, the instrumentation and metric standards that cut readouts to 24 hours, a carousel test that a multi armed bandit resolved in 51 days instead of a projected 700, and why cost avoidance from losing experiments belongs in the ROI number. It closes on what changes when the thing you are testing is an AI agent rather than a button. Useful for product managers, engineers, data scientists and growth leaders building or defending an experimentation program at scale.\n\nChapters\n00:00 Cold open\n00:54 Welcome Gaurav Sethi of PayPal\n01:30 Elmo, PayPal's homegrown experimentation platform\n03:53 $180 million in revenue impact and cost avoidance\n04:28 The win rate that was too good to be true\n06:29 Instrumentation standards and the exposure event\n08:56 The carousel test: 700 days down to 51\n11:29 Designing experiments so every result teaches you\n13:45 Building the platform is only half the job\n15:07 Education, office hours and executive support\n18:28 Exposure events and joining transactional data\n24:14 AI for experimentation, and experimentation for AI\n\nTakeaways\n- A win rate of 55% to 60% against an industry average of 11% to 14% was a tracking problem, not a performance one: assignment data carried 25% to 30% dilution and reached significance on users who never saw the test.\n- The exposure event fixed it. Fire an event as close to render as possible so the platform knows exactly when a user entered the experiment, instead of logging on page load.\n- Standardized instrumentation...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}