A/B Testing Secrets from LaunchDarkly | Why Data Can't Replace Human Curiosity

What if your dashboard is lying to you? 📊❌ In this episode of UX Stories, host Lihong Hicken sits down with Aaron Montana, Head of Experimentation & Product Analytics at LaunchDarkly, to explore the real intersection of engineering, A/B testing, and user research — and why the future of testing might include machines testing for machines. 🤖 Whether you're a product manager, engineer, or UX researcher, this conversation will challenge the way you think about features, data, and what it really means to understand your users. 🕐 TIMESTAMPS 00:00 – Welcome & Introduction 01:20 – What a dashboard can never fully explain 💡 02:05 – The role of human curiosity in experimentation 03:05 – What is LaunchDarkly and who uses it? 03:27 – Why feature flags are the best foundation for A/B testing 05:45 – Bridging the gap between engineers and marketers 08:25 – Quantitative vs. qualitative: what's the right sequence? 09:25 – How to design a research study from scratch 11:15 – Qualitative finds problems; quantitative solves them 15:48 – The psychological challenge of killing your own feature 😬 18:45 – Why high-velocity teams are already running experiments — just badly 22:14 – Vibe coding vs. vibe testing: the speed gap 🚀 24:50 – Is qualitative research becoming too expensive? 26:20 – Synthetic data and AI panels — promise vs. reality 🤔 28:08 – Early-stage vs. late-stage testing: when to use AI 30:00 – Are we about to test for machines, not humans? 🤯 32:31 – The future: machines running experiments for machines 💬 If technology is the rain and our ecosystem is a garden — more jobs, more testers, more ecosystems to validate. 🌱 Like, Subscribe & hit the 🔔 if you're building products and want to ship with confidence — not just speed. UXResearch #ABTesting #LaunchDarkly #ProductManagement #UserTesting #VibeCoding #FeatureFlags #Experimentation #StartupGrowth #UXStories

What is UX Stories Podcast?

If you want to understand your customers more deeply and unlock growth that doesn't rely on guesswork, this series is for you. Founders, operators, and product leaders tune in to learn how real conversations shape better products, better decisions, and better outcomes.

In each episode of UX Stories Podcast, you'll explore how fast-growing teams uncover meaningful insights using smarter feedback loops and authentic user conversations. Inspired by the approach behind TheySaid, this show digs into why traditional surveys fall short and how modern teams gather richer, more actionable signals that drive real business impact.