{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"The Experimentation Edge","title":"What Wealthsimple decides to test and what to just ship","html":"<iframe width=\"100%\" height=\"180\" frameborder=\"no\" scrolling=\"no\" seamless src=\"https://share.transistor.fm/e/d787b3ac\"></iframe>","width":"100%","height":180,"duration":2174,"description":"Summary\nDelina Ivanova, Director of Data Science at Wealthsimple, joins host Ashley Stirrup to share how Canada's fast-growing financial services company decides what deserves an experiment and what should simply ship. With 4 million clients and a steady stream of new brokerage and banking features, Wealthsimple has spent the last 2 years formalizing experimentation across 3 data science teams.\n\nDelina explains her rules of thumb for skipping a test, why lagging metrics like deposits and revenue force teams to pick the right early signals, and what a margin promotion taught her team about messaging, targeting and combined offers. She also shares how to diagnose a losing test, her plan for meta-learning and uplift modeling, and how AI makes it cheaper to test ideas before investing in a full build. For product, data and growth leaders who want experiments that measure the thing they are trying to drive.\n \n \nChapters\n00:00 Wealthsimple, 4 million clients and a never-ending experimentation world\n02:46 Two years of formalizing experimentation\n03:42 What Misplay taught her about when you need a test\n06:00 Rules of thumb: when to test and when to just ship\n08:21 Ronny Kohavi, Bing and the value of a millisecond\n09:26 How Wealthsimple's 3 data science teams run experiments\n12:15 Engagement features, lagging metrics and early signals\n14:49 Launching margin and finding an education gap\n16:57 Why surveys need experiments\n19:10 Sharing results and building systemic learning\n21:56 The margin promotion that taught a messaging lesson\n26:23 Experiments as a way to learn about people\n27:39 Messaging, targeting or experience: diagnosing a losing test\n30:35 Meta-learning and uplift modeling\n33:42 Using AI to test ideas faster and cheaper\n \n \nTakeaways\n- Skip the test when you are removing a clear blocker for clients, and test when the change is optional or the outcome is uncertain\n- Deposits and revenue can lag 60 to 90 days in financial services, so pick early metrics that are...","thumbnail_url":"https://img.transistorcdn.com/D9kLs0HSsqR4ttk_5ESEdC1jX-wmD76GK-OHmb3a9B8/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS80YTFk/MGU1MjJlODhlNjJh/MTdlZTZkN2Q1ODY5/OTdjYy5wbmc.webp","thumbnail_width":300,"thumbnail_height":300}