Summary
What do you do when your big launch increases engagement and tanks conversion? On this episode of The Experimentation Edge, host Ashley Stirrup talks with Lina Blackman, Director of Product Analytics at Squarespace, about the blank template launch that flopped — and how its learnings became Blueprint, Squarespace's AI-guided website builder. Lina explains how her embedded analyst team runs 150–200 experiments a year for 3 million customers, the two questions she asks every time a test loses, why teams only need one or two big wins a quarter, how Squarespace calibrates statistical certainty to business stakes, and where AI belongs (and doesn't) in the A/B testing workflow. For product managers, data scientists, and experimentation leaders who want to extract more learning from every test.
Chapters
00:00 Introduction: Lina Blackman, Director of Product Analytics at Squarespace
01:45 Squarespace's business and 3 million website customers
02:30 Decentralized analysts, centralized experimentation program
04:15 150–200 experiments a year: onboarding, mobile, checkout, pricing
04:55 The blank template disaster that became Blueprint AI
07:45 Two questions for every losing test
09:30 Moving ship-first teams up the experimentation maturity curve
12:30 A/B test logs and insights rituals
13:30 North Star metrics and the KPI tree
16:35 AI in the A/B testing workflow — and what stays manual.
Takeaways
- Stated preference lies: users asked for a blank canvas, but behavior demanded guided design — and only the experiment could referee.
- Close every losing test with two questions: did it work for a granular segment, and is the idea worth further investment?
- One or two big wins a quarter is a healthy hit rate when you run 150–200 experiments a year.
- Calibrate certainty to stakes — tight bounds on revenue and pricing tests, wider bounds on engagement tests so teams don't spin on noise.
- Hand AI the mundane parts of the workflow (tracking, assignment setup), but if AI runs the brief and the analysis, ask why you're running the test at all.
Sponsor
Growthbook helps you ship features with confidence by bringing experimentation and feature flagging into one open-source platform. No more guessing whether that new checkout flow actually moved the needle, waiting weeks for data team bandwidth, or flying blind on rollouts.
Growthbook gives you a single place to run A/B tests, manage feature flags, and analyze results against your existing data warehouse.
With powerful stats built in, it takes the complexity out of experimentation, helps you catch regressions before they hit every user, and makes it easy to test ideas that keep your product improving and your metrics moving in the right direction.
What is The Experimentation Edge?
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.