Subscribe
Share
Share
Embed
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
The Experimentation Edge - Edd Saunders
===
Edd Saunders: [00:00:00] your one test might not have a huge amount of impact, but when you look back over a year, the 100 or so tests you run might, ~the, the,~ the growth will have compounded over time
Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions.
This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest.
Ahley Stirrup: Hello, and welcome to today's episode. Today we're excited to have Edd Saunders, product experimentation manager at JobLeads. Edd, welcome to the show.
Edd Saunders: Thank you. Nice to be on here.
Ahley Stirrup: Yeah. So Edd, maybe you could kick things off by telling us a little bit about JobLeads.
Edd Saunders: Yeah, of [00:01:00] course. So I'm Edd. I'm product experimentation manager at JobLeads. JobLeads is, ~we're working towards a,~ we're turning ourselves into a career coaching service. So just to give you a bit of background the company started almost 20 years ago, and it started out as a...
So effectively ~a it was a,~ a job board, but better, and it became a premium job board. And then as customer tastes and demands have changed over time, we've now moved away from the kind of premium job board as our core service. It's now free for everyone. And now we're offering a range and a suite of tools to help job seekers find jobs.
And then where we're moving to in the future is what I'm part of building, is effectively helping connect job seekers to jobs and help them stay in those jobs or find better jobs or improve their time in, within those jobs. Via tools like helping people negotiate salaries or better positions, those sorts of things.
So yeah, it's as you can imagine, the kind of job seeker world is exploding at the moment. Obviously good for us, not so good for the wider economy. But there's... ~A-~ and we've got, the shakeup with AI. It means the [00:02:00] space we're in is it's growing massively, and it's quite an exciting space to be part of.
'Cause effectively my mission is to help connect job seekers to jobs. So the work that I do, ~the delivering,~ the deliverance of my work helps have real world impacts on real people. So for me, it's very nice.
Ahley Stirrup: Yeah. Yeah, I'll bet. That's pretty cool. Are you doing a lot with the AI as part of that?
Edd Saunders: We do some, yep. So we've got a bunch of AI features. What I'm working on at the moment, so I'm part of the user acquisition team, so my job is effectively to turn new visitors into registered customers, and then turn registered customers, you activate them, so turn them into paying customers.
So we don't really use AI to ~in~ lieu of a proper good customer experience. We use it to enhance the customer experience, and in my experimentation work, I'm using it very heavily for building MVPs effectively. And if we can validate them using AI, 'cause it's cheap and easy, then we go away and build the feature properly in the background using our huge data resource that we've got.
Ahley Stirrup: [00:03:00] Got it. Got it. So you're using it for prototyping, is that what you
mean?
Edd Saunders: yeah. And then it really useful for discovery really useful for the kind of analysis phase. I use it very heavily for prototyping, like you say, Claude Design, Claude Code. It's now... I don't write tickets anymore so I've got skills to do it for me. ~It's~ it's brilliant, yeah. It's very time-saving for me.
Ahley Stirrup: Terrific. Terrific. And can you tell us a little bit about your team and how JobLeads does experimentation?
Edd Saunders: Yeah. Experimentation, they only started testing really in, I think it was the beginning of January 2025, or maybe the end of 2024. So the organization is relatively new to experimentation. I was, I believe, the second dedicated experimentation hire. However, because it is a relatively new team because we're hiring so many people at the moment, especially over the last year, a lot of people have come aboard with previous testing experience which has helped shifted the culture.
But I've also been doing a lot of work on the experimentation operations side of things. So giving people the [00:04:00] tools and access and capability to run experiments, but also without having to do too much admin work at the kind of front end, but also being able to collect and share the knowledge that we're gaining from each team just to break down those silos.
I delivered the other day ~a, a n-~ a knowledge base, which effectively is taking the learnings from every single experiment we're running across the company and synthesizing it so that anyone can learn. Anyone, ~d-~ no matter what your position is within the company it's open to everyone. You can dip into that as a additional resource to help your to help your work, be you're in marketing, customer support, product wherever.
Ahley Stirrup: Yeah. Yeah, I think that's definitely an interesting trend in the whole experimentation space is just how do you unlock as much insight as possible out of all the experiments you're running? How many
Edd Saunders: and it's,
Ahley Stirrup: Sorry, go ahead. Go ahead.
Edd Saunders: It's a way of keeping people excited about experimentation as well. What I've seen from back in my consulting days, I was consulting side ~about,~ for about six years, and I've recently moved back in-house, is there's a lot of interest [00:05:00] in experimentation. It spikes at the beginning when you start, and then dwindles slowly over time when people realize the reality of every idea they have might not be something that gets implemented.
It might be proved, your great idea might be proved to actually be terrible. But the way I'm framing it isn't in terms of, look at what's won, look at what's lost. It's look at what we've learnt and look at what better decisions ~we've,~ we're empowered to make because we've run an experiment rather than invested a huge amount of time and energy and resource into just delivering something and moving on.
And for me, that's the exciting part. Getting people involved, getting people understanding, helping keeping that cycle moving. Yeah, good times.
Ahley Stirrup: Yeah. Yeah, there's no question that's where the real power of experimentation is when you start to compound, right?
Edd Saunders: 100%.
Ahley Stirrup: experimentation isn't about going and getting a home run, it's about stacking singles.
Edd Saunders: Yeah, that's, I think that's a really nice way of framing it. It's like your one test might not have a huge amount of impact, but when you look [00:06:00] back over a year, the 100 or so tests you run might, ~the, the,~ the growth will have compounded over time.
Ahley Stirrup: Yeah, no question. So how many different teams there are, are you supporting?
Edd Saunders: I'm attached to the user acquisition team, so my primary focus is with them. Experiment operations stuff I'm doing is universal, so any team. I think we're about 130 people across the company. Obviously not everyone is in product side. So there's probably about six or seven teams who would benefit from the operational work that me and partner are doing.
Yeah, so it's large impact organizational-wide.
Ahley Stirrup: Yeah. And usually one of the challenges with that is just helping people, with the rigor of experimentation. Do-- Are they designing the experiment correctly and all that? So is that something you work on?
Edd Saunders: Yeah, so a big piece of this operations was a brand new workflow and in standardizing the kind of input. Because once you can standardize that you then don't have to start chasing your tail in terms of making sure that, that you got the rigor of [00:07:00] the experiment design properly.
~We've also~ we've got a new model now where every team has a department ~an- ~analyst attached to them, so they can be the arbiter of what's a good or poor experiment design, which has really helped.
Ahley Stirrup: Terrific. Terrific. Can you tell us about a time in your career, it doesn't have to be at JobLeads where you had an experiment where you had a lot of learnings?
Edd Saunders: Yes. Okay, so my favorite story I like to tell ~is~ is a few years back I was working with a, a large established pizza company whose name I won't mention just now, but you can probably guess. They wanted to explore personalization as a method of improving their web experience. The, the angle we took was how can we use personalization to improve the customer experience, make it a bit more friction-free, make it a bit more seamless?
And this is, yeah, you can tell this is a few years back because ~it was pre- it~ I think it's just as ChatGPT was becoming popular, but it's still when personalization was the big buzzword that everyone was chasing. But ~the,~ the angle we went down was, so yeah, trying to make a friction-free experience.
What we saw from the data, from the customer journey maps [00:08:00] that we built in terms of drop-offs and the funnels, was that the average customer, it took them, I think it was around 22 distinct inputs from starting their session to or completing their order, and obviously you have to go through their email, seek confirmation, and they have to wait for it to be delivered, blah, blah, blah.
So yeah, 22 distinct steps to make their food order. What we also noticed is that people ~the,~ the biggest bulk of I think it was probably about 50% actually of traffic was returning users, and returning users would place the exact same order week after week. There might be some variation with the sides or drink or whatever, but their typical pizza was the exact same thing over and over again.
What we'd also learnt from previous experiments with part of this program was that people wouldn't just have this brand's website open when they're ordering. They'd have two or three other takeaway brands open. So in the UK we have Deliveroo and Just Eat.
They're two of the big ones. So typically someone would be multi-tabbing between, three different places, [00:09:00] trying to see which gets to their basket the first in order to place the order. Because what we realized is people were hungry, so they're hungry and angry and getting angry 'cause their blood sugar's getting low, right?
~So the...~ Knowing all that, we decided to see if we could reduce the number of clicks or inputs it took from landing and starting their session to completing the order, because that's the kind of logical conclusion you draw from that, right? And they also wanted to explore personalization. So what we did is we spun up this...
It's actually a really cool technical solution. I won't dive too much into that now. But we built them ~a,~ a database which collects every single individual item that this, that customers are ordering, saves it against their user ID, and then when they came back for their, the next week for their returning session, we served the exact same order back to them in the form of this really nice, nicely designed widget.
And it was literally, you land on the site, you can click one button, your entire order is added to your basket, and you're redirected to the checkout. So you go from 22 clicks to, I [00:10:00] think it was five or six. So massive amount of time saving. It had absolutely zero impact, zero impact on user behavior.
It didn't increase purchases. It didn't decrease purchases. People saw it and just thought, "Nah, I'm not using
Ahley Stirrup: So are they just not using it?
Edd Saunders: ~The, there was,~ it got ~a,~ a bit of interest, not as much as we were expecting. And because of that, the result of that is it didn't have any impact on the critical metrics we're trying to measure, which was typically revenue or revenue per session in that regard.
But it, what we learned from it is aside from the fact that, personalization for them specifically probably wasn't the best solution at this point. We did a few iterations, but we came to the conclusion that there's not really much point in forcing personalization on these people 'cause it just doesn't work.
It's a waste of money. But the other thing we learned is that the exploration is still a massive part of customers' delight. Their enjoyment of the experience, and They feel like they want to have control over their experience. So by [00:11:00] making it easier to find their order again, we actually took away some of their control.
Maybe they felt like they weren't getting the exact same deal as before, or maybe they just wanted to explore different options, and I think that was part of it as well. People would kid themselves that they wanted something different this time, but actually they'd revert back to the exact same old pizza they bought time after time again.
And yeah it didn't work. But ~we s- we've,~ we learned a lot and saved a lot of money. So that's one positive we had from this from this track of experimentation.
Ahley Stirrup: that... Yeah, boy, is that a great example of where you're just like, "I'm sure this is gonna be a winner. I just know it." And yeah.
Edd Saunders: obvious, isn't it? It's so logical. It's the logical conclusion from all our all our research.
Ahley Stirrup: It's pretty funny 'cause like I know I personally would want that feature. I'm not gonna go to three different websites. I'm gonna order from the place around the corner, and I just want it as painless and quick as possible. And yeah.
Edd Saunders: I want my exact same thing that I always have.
Ahley Stirrup: Yeah, exactly. That would keep me up at night wondering why that feature didn't work, and yeah, it's some interesting learnings you've got [00:12:00] there.
So that ~l-~ leads into a great question, which is so let's say you start working with a new product manager and they really haven't done experimentation before, and they have a new feature that they wanna test. How would you guide them on designing the experiment so they get the most learnings? One of the things I notice with people new to experimentation is they often think that, 90% of what they test is gonna be a winner, and in reality it's more like 20%.
And so you have a very different mindset with experiment design if you're designing for a loser,
Edd Saunders: Yeah.
Ahley Stirrup: on. Yeah.
Edd Saunders: I like this question a lot because part of my job as a consultant was to train others. And typically training people who are brand new to experimentation, never run a test before, but they're all full of ideas. ~So the-~ I think ~the,~ the trickiest part is getting people out of the solution space thinking and moving them into the problem space thinking.
So the solution space thinking is trying to come up with ideas and just chucking stuff at the wall and hoping for the best. And we all have this [00:13:00] natural internal bias that we believe our ideas are better ~than ~than others for ~some-~ for whatever reason. So I think everyone new to this has to go through this kind of ego death, you might call it, or ego flattening experience where...
Yeah, humbling, yeah, exactly. Where their brilliant idea actually has zero impact whatsoever. But the way I do that is I like to guide people in terms of putting themselves in the shoes of the customer and then understanding the problems the customer's facing, and then working out how we can develop solutions to solve those customer problems.
Because what you'll notice when you're problem-focused like that, the output of your experimentation program will be one, more effective in terms of more, like, better win rate. But two, the learnings you get from that are gonna be far better because that piece of intelligence you get from identifying ~a-~ an assumption about a problem and validating it is useful for anyone in the organization.
Not just product teams, but marketing teams, they can adjust their marketing based on that. Customer support teams, because they know [00:14:00] what, the typical fears of these customers are. Leadership teams because they can know how to steer the ship in terms of the problems that we're trying to solve.
But yeah, I like to run an exercise called customer journey mapping- and it's a good introduction to analytics tools as well. So it's like, if my goal is to get on, let's say on JobLeads, my goal is to get a new customer to be- become a paid customer, which means they need to land on the site, they need to go through the kind of sign-up process, they need to obviously then go through the payment gateway, and then they're redirected to the sort of introductory type stuff.
The first thing I want to understand is the customer journey map. So where are people coming from, as in which channels and sources? Where are they landing on the site? What is their primary action after that? And then looking at all the individual kind of micro steps they need to take in order to get through that next gate of, acquisition to activation.
And then from that, we can plot the... or we can work out the funnel drop-off rates in a very granular [00:15:00] detail. So let's say you land on the JobLeads website, you click a sign-up button, you can choose to sign up by email. I want to know the drop-off points between all these points, because then what it helps me and the new person to understand is where is the biggest the biggest leak happening, and it's a method of pre-prioritization, so you can zoom in on the biggest weaknesses.
But then once we've got that quantitative data, we can layer on some qualitative data, and what I mean by that is things like heat maps, session recordings click maps, scroll maps, just to understand the how behind the what. So how is a customer interacting with the site? We know what, where they're dropping off, et cetera.
But I wanna see... I wanna put myself in their shoes and just see exactly what they're seeing, and these tools help with that. And then the next layer is more kind of user-based research. So that could be user interviews. It could be reviews from third parties, social media for job leads.
Actually, social media reviews is a big [00:16:00] thing for us 'cause it just gives us a huge amount of intelligence and helps us steer where the product should go. But with all those kind of three sources, I can then zoom in on where's the biggest problem, how are they interacting as in put myself in their shoes and see what the problem is, it actually is, and then hear from their words what they believe the problem is.
And then from that, I can summarize that in terms of problem statements. So as a new user, I don't register for job leads because I don't trust the brand, things of that nature. And then there's an exercise that I love doing called problem mapping. So if you're familiar with assumption mapping, very similar concept, just to build upon that.
So what we're doing in the first~ first~ instance is taking that data and trying to come to conclusions based on the problems that real users are facing, which is stopping them from achieving our goal of registering or becoming a paid subscriber. Once we've measured those out, once we've got a good log of those, we prioritize them on a two-by-two matrix, which on the [00:17:00] horizontal axis has we judge based on how much we know that this problem is real or not.
So it starts from zero on the left side all the way up to 10 on the right side, and effectively, it's down to what data points we've got. So if the problem that we've come up with is pure assumption, and that's fine at this stage if it's pure assumption, it goes on the left side. So it's a zero.
We don't know for sure. We don't have any data to back up this problem. If we've got a combination of that quantitative, qualitative user journey maps, maybe some meta-analysis from previous ~ex-~ experiments, that will go all the way to the 10 And basically on the right side, I'm hoping people can visualize this 'cause I'm talking a lot, but ~from 5 to 10 is,~ from 5 to 10, that's the known side of things.
So we know for sure that it's a problem. From 0 to, 4 up to 5 in the middle, that's unknown. And then on the vertical axis, we've got impact. So if we... anything below the line, the horizontal line, that's low impact. So if we have to solve that problem, it would have a very, [00:18:00] meaningless or it's not really worth the effort to do.
If it's above the line it would have a higher impact, so the effort is worth more than the... So the reward's worth more than the effort. And this bit is quite subjective. You can plot this with data. It takes a long time, And that's fine. We're trying to embrace the human element of this process because ultimately we're optimizing for, for humans to build better human experiences right?
But the outcome of this exercise is that in your top right quadrant, you've got a bunch of problems which are both high impact and you know for sure that they're problems. Then in the top left quadrant, you've got a bunch of stuff which is high impact but unknown. So in terms of prioritizing those problems, you take the high impact known stuff, you then turn those into potential hypotheses, and then come up with solutions to test those hypotheses, and that's your kind of top chunk of ~of~ of experiments to run first, followed by the stuff that is unknown.
What you want to do there is keep a log of them, but you maybe could do some additional research [00:19:00] or look for some other sources or, maybe s- commission some other user research to validate them or not. Or if they're simple to do, just run them as experiments. But going back to why I'm going through this process with new people it's to help them really understand that we are trying to solve real user problems, and this is a really good exercise to do that.
And the other thing which it's useful for is what I've noticed is people new to kind of CRO or experimentation, at some point or another, they're worried that they can't come up with new and fresh ideas. ~They think it's~ they think it's ~the,~ the ideas won't come ~in,~ in abundance.
And I've noticed that if you ~d- if you~ live in the solution space, that is what happens. Your kind of ideas run dry pretty quickly. If you come up with the problems, if you understand the problems you're trying to solve, naturally you'll come up with multiple hypotheses for each problem that you've identified.
You'll come up with different solutions for each hypothesis that you come up with. So one problem can spring a nice tree of ideas. And it's a really nice way of just keeping it fresh.
Ahley Stirrup: Boy, [00:20:00] that's a really powerful example.
Edd Saunders: Dumped a lot of stuff on you there.
Ahley Stirrup: Yeah, no, and the part I think's so important is that mind shift from solutions like, "Oh, I've got, five great ideas," to really putting yourself in the the head of your customer and the problems that they're facing. A lot of examples of learnings from past episodes have been where people had a great idea of this thing they wanted to insert into somebody's, I'll call it a buyer's journey.
It's not always buying, but, whatever the task is. And if that task isn't aligned with what the customer's trying to actually achieve, it's generally a loser.
Edd Saunders: Yeah, agreed. It's a bit like on, on e-commerce stores these days, the majority... okay, I don't know if it's the majority, but it feels like the majority. You go on the site, and all of a sudden there's, bam, "Get 10% off if you give your email." And it's like, what problem of mine is that solving?
That's clearly someone on a marketing team, and no offense to marketers, I understand the pressure that they're under. [00:21:00] They've been told that they need to increase the, I don't know, the efficacy of their paid spend or affiliate spend or whatever, and offering a discount is usually a surefire way of seeing that number go up, that line go up.
It's just served right at the beginning of the user experience is off-putting for a lot of people.
Ahley Stirrup: I couldn't agree more. That's one of my personal pet peeves. It's like, "I just got here. I don't wanna give you my contact info. I don't want another thing to unsubscribe from tomorrow."
Edd Saunders: Yeah. That's what I like about Black Friday ~bec-~ 'cause all these brands in my inbox, I can just go unsubscribe, unsubscribe.
Ahley Stirrup: That's so funny. Yeah. When you look at JobLeads and kind of the future of experimentation there what do you see are the biggest opportunities?
Edd Saunders: Definitely velocity. So we're ramping up at the moment. So from when it started, it was around about I think 0.3 experiments launched per month. And we're now at, I think it was about 2.8 when I calculated the other day. So it's ramping up. And on the kind of marketing department, which is where I'm [00:22:00] attached to, I want to increase that velocity as well.
And my personal goal is, one, velocity, but also get more people involved in the process of coming up with ideas and h- and giving them the tools to be able to launch their own experiments. For me, that's a very powerful thing. If a content manager or just a normal marketer can feel like they ha- they can have some input into what the product looks like via the form of an experiment, and they've gone through this sort of process of understanding that they've got to find a problem, they've got to find a solution to solve that problem, then turn it into an idea, and then see that their ideas come to life on the website and it's launched and it's validated, and it is absorbed into the, into, into the production environment, I think that's a very powerful thing.
So yeah, really it's ... I guess you could call it democratizing experimentation, giving the power to the people.
Ahley Stirrup: Yeah. Yeah, I love that one. And what do you think are some of the most important things that you need to do in order to enable that? And I assume some of that is technology and some of that is like [00:23:00] education and things like that.
Edd Saunders: Yeah, ~I think it's,~ I think it's education, but I think habit change is ~the~ probably the bigger one. People are under so much pressure with their own tasks, and if you come along and say to someone who doesn't even do experimentation or product as their main job that, "Actually, I'm now forcing you to come up with ideas and run experiments," there's ~g- there's~ gonna be a lot of resistance, right?
And that's partly why I built the knowledge base, so people can understand that, we don't run experiments just to see a KPI go up. We run it to learn something. So if I feel if people can see the impact that their idea has in terms of the learning that's now shared with the organization, and it can say, "Hey, Ashley, you had that great idea a few weeks ago.
We ran it as an experiment, and the company now knows X, Y, Z because of your insight," I think that's a really powerful thing.
Ahley Stirrup: Yeah, I love that. That's terrific. Do you see yourselves using AI in other ways to enable experimentation or enable better analysis, things like that?
Edd Saunders: Yeah, our workflow's changed massively already because of AI. One is the kind of, [00:24:00] yeah, like I said earlier, the creation of the basic stuff like tickets and keeping things ~mov-~ moving smoothly like that, and standardizing tickets and making sure everything has the correct the correct fields filled out that I need for the knowledge base.
But there's the kind of ~c-~ the big, sexy stuff that you get with AI, but there's also the kind of really basic stuff which I think not many people talk about, which is I used Claude the other day to help me set up an n8n automation, which does something as simple as when someone moves a, a ticket from one status to the next status, it looks into the body of the ticket, and in any linked tickets, extracts a Figma link, and then populates the link to design field.
So it's just super simple stuff like that which I find this AI is really useful for. And it's one of those things that I need to keep the knowledge base, up and running. It's good for the kind, kind of documentation and consistency of that documentation, and it's like a 30-second job if I did it myself, but it's one of those [00:25:00] really low priority jobs that no one wants to do.
So if I can do stuff like that to ~remove- to, to~ ease the burden a bit, those little stepping stones are gonna help people over the entire organization run experiments and and learn from those experiments.
Ahley Stirrup: Yeah, I love that. I actually think that's one of ~the,~ the s-secret things about AI. Like you said, it's underappreciated that it enables a new kind of automation. With a little bit of AI and a little bit of automation, you can remove a lot of friction. So that's a terrific example.
Yeah. Edd, thank you so much for being on the show today. I feel like ~we,~ we learned a ton. I loved that whole framework that you have for both coming up with ideas and evaluating them. I think that's a, a great thing for our listeners to think about and apply to their own business.
So thank you so much for joining the show today.
Edd Saunders: My pleasure. Nice to chat.
[00:26:00]