{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"Why Twilio ships on signals instead of significance","html":"","width":"100%","height":180,"duration":1463,"description":"Summary\nWanli Lau spent eleven years at Expedia Group, most recently running consumer product analytics on a program that shipped more than 1,500 A/B tests a year. Five months ago she moved to Twilio to lead global R&D analytics. In this episode she walks Ashley Stirrup through what carried over and what did not.\nThe centerpiece is a test that looked like a win. At Expedia, an account sign-up takeover on the front page drove conversion sharply up, and quietly harmed return rate and engagement across every geography. That result is where Wanli's insistence on guardrail metrics and a trade-off decision framework comes from. She also explains the parts of B2B experimentation that have no B2C equivalent: choosing between user-level and account-level randomization, the interference risk when two colleagues at the same company see different pricing, and the many-to-many problem of one developer belonging to several accounts. And because Twilio's customer counts are lower than a consumer business, her teams often ship on signals and team conviction rather than waiting for statistical significance.\n\nChapters\n00:00 Cold open\n01:02 Welcome Wanli Lau of Twilio\n01:23 Inside Wanli's role at Twilio\n04:03 Eleven years and 1,500 tests a year at Expedia\n04:58 Why B2B experimentation differs from B2C\n05:33 Cross-functional collaboration and good hypotheses\n06:39 Teaching teams to experiment rigorously\n07:48 The Expedia sign-up takeover that won on conversion\n11:09 Guardrail metrics and the trade-off framework\n14:52 Designing experiments to maximize learning\n17:46 Diagnosing where a feature failed\n19:10 North Star metrics and account-level randomization at Twilio\n21:40 Learning from signals instead of significance\n22:11 Where experimentation goes next at Twilio\n\nTakeaways\n- A test that wins on the primary metric can still be a loss. Expedia's sign-up takeover raised conversion and damaged return rate and engagement at the same time.\n- Name the primary, secondary and guardrail...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}