{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"Realtor.com on using your AI as a junior data scientist","html":"","width":"100%","height":180,"duration":1205,"description":"Summary\nWhat do you do when your biggest experiment win turns out to be a loss? Whitney Perez, Director of Product Management at Realtor.com, joins host Ashley Stirrup to share the checkout bundling test that posted a 300% attach rate and still lost revenue, the 30/30/30 rule she uses to set expectations for a new experimentation team, and how AI is turning an English major into an aspirational data scientist. This episode is for product managers, engineers, and data scientists building experimentation programs from the ground up.\n\nChapters\n00:00 Cold open and welcome\n01:10 From growth hacker to Realtor.com\n02:50 Three foundations for a new experimentation team\n04:20 The 30/30/30 rule\n05:05 The 300% bundling win that lost revenue\n07:10 You don't need a stats degree to experiment\n09:20 Cascading North Star metrics\n12:50 Do the homework before the experiment\n13:50 The wishlist: instrumentation, embedded knowledge, culture\n15:50 AI as an aspirational data scientist\n18:45 Keeping a human in the loop\n\nTakeaways\n- A winning decision metric is not enough. Realtor.com's bundling test hit a 300% attach rate, but funnel fallout from the extra step made it a net revenue loser. Set secondary metrics and their thresholds before launch.\n- Expect the 30/30/30 rule: roughly a third of tests win, a third are inconclusive, and a third lose. The math is the math, and the losers carry most of the learning.\n- Start a new team on foundations: what a clean test and an A/A test look like, which surfaces should not be tested, and a peer review program that lets people graduate to more complex experiments.\n- You don't need a stats background to run good experiments. Teach the simplest definition of a good test, then let people learn by doing.\n- AI can make anyone an aspirational data scientist for analyzing results and spotting opportunities, but it can be confidently wrong. Keep a human in the loop and sanity check output the way you'd peek at a freshly launched test.\n\nConnect with the...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}