{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"The four questions Early Warning asks before any A/B test","html":"","width":"100%","height":180,"duration":1303,"description":"Summary\nWhat separates a valid A/B test from an expensive guess? Priya Singhee, VP of Enterprise Analytics & Data Science at Early Warning — the bank-owned consortium that fights payment fraud and operates Zelle, which processed a trillion dollars last year — joins host Ashley Stirrup to share the experimentation playbook she built leading storefront analytics at Wayfair. She walks through the four questions to ask before launching any A/B test, why 85 to 90% of tests are supposed to fail, how pre-registration and kill criteria stop p-hacking before it starts, the pitfalls that fake wins (novelty effects, hidden heterogeneity, multiple comparisons), and how to roll out winners with gradual ramps and long-running holdouts. A practical episode for product managers, engineers, data scientists, and growth leaders building rigorous experimentation programs.\n\nChapters\n00:45 Meet Early Warning: fraud detection, Zelle, and a trillion dollars in payments\n02:00 Wayfair and optimizing every step of the storefront funnel\n03:05 The four questions to ask before any A/B test\n05:15 Test setup best practices: hypotheses, guardrails, power, and pre-registration\n07:40 Why 85 to 90% of tests fail and why that's a learning agenda\n09:25 Novelty effects, hidden heterogeneity, and the multiple comparisons problem\n12:55 Pre-registration, kill criteria, and stopping p-hacking\n15:10 Rolling out winners: gradual ramps and long-running holdouts\n17:15 Causal inference when you can't A/B test\n20:05 The case for more A/B testing, not less\n\nTakeaways\n- Run the four-question framework before any test: clean randomization, a plausible effect size for your traffic, a reversible and cheap change, and a falsifiable hypothesis.\n- Treat A/B testing as a learning agenda: 85 to 90% of tests are supposed to fail, and a suspiciously high win rate is a red flag, not a trophy.\n- Pre-register the full analysis plan, including hypothesis, mechanism, primary metric, exact statistical test, and subgroups, so...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}