Subscribe
Share
Share
Embed
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
Ashley Stirrup (00:33)
Welcome Kameron to the show. Kameron is the head of growth engineering at Fyxer Fyxer is a fast growing ⁓ AI assistant for your email. And that's a pretty exciting product. And Kameron has been doing a tremendous amount of experimentation over the last year. And so we're really excited to have him on the show. Welcome Kameron.
Kameron Tanseli (00:55)
Thank you for having me.
Ashley Stirrup (00:56)
Maybe to start, you could just talk us through a little bit about the mindset towards A-B testing at Fyxer.
Kameron Tanseli (01:02)
Yes, it's a little bit different from where I've worked previously, where you come into an established business and you've got tens of thousands of unique visitors coming through and maybe you're in an e-comm and you're pushing conversion rates within a step funnel. When you're coming into a business like Fyxer that's just come out of an accelerator million in revenue, your main aim is to try and grow as fast as possible rather than
you eking out as like an AB testing team might attend 15 % lift in a conversion rate metric here or there. Your goal suddenly becomes here's the entire business. Let's use experimentation to de-risk it, but also to grow as fast as possible. And when we think about AB tests for startups, I often get met with, hey, we're just not there yet. Like we need to be way bigger in order to start doing this.
But I'd rather have a 50-50 test out to give me some sort of direction or insight rather than just say the word yoloing, shipping to a hundred percent constantly, because there is that risk there as well. And we can sort of see that at Fyxer where thanks to Growth Book, we can see on the insights tab that we only have around a 25 % win rate. So if I did ship everything from a year and a half ago,
I'm not sure where the business would be in terms of error, because we're testing risky stuff as well. It's not just, you know, button colors, it's pricing models, it's limiting usage, it's changing the core parts of the models and how they work in the product as well. So that's the big difference there.
Ashley Stirrup (02:45)
Yeah, yeah, that's a, it's a definitely a different mindset when you're measuring for 10 % gains or something like that versus a 1 % gain. So that's, that's pretty interesting. And when you do decide to run a rigorous AB test and when do you decide to do something that's more simple, like just, you know, tracking before after or something like that.
Kameron Tanseli (03:09)
it depends on the risk appetite. So the further we move up the funnel, we start to get a bit more cautious because of the lower marketing spend going into that. we're big on, you know, paid, well, we big on paid last year. We're now more towards PLG. And with that kind of spend, you have to be a bit more cautious and we have the sample size there in order to do those.
Still not 5 to 10 % lifts, but 10 to 20%, even 30 % above lifts there. So that's where we decide the further up the funnel, the more users, the more rigorous we can become, because we can measure smaller effect sizes. When it's in the product, couple of hundred users, if the risk is minimal, we might just go and ship it or do a non-inferiority, which doesn't require that much runtime compared to a standard A-B test.
Yeah, it's, it's basically risk appetite. Some, if it's a, I know, um, Alexi, fellow growth engineer, he has a really good article on how, like when to A-B test. And it's a really fun flow chart that I like to show around, which is if it's a strategic change or a very simple change with next to no downsides, you should just go and do it because you're to go and do it anyway. Um, so yeah, that's.
Those are the times when we do it before and after. If we do feel that there's some risk, we'll do a non-inferiority or 80-20 or whatever there is to measure any sort of downside there.
Ashley Stirrup (04:41)
Yeah, and I think I read that you have a team of four and that across the four of you, you ran 360 experiments last year.
So that's a pretty huge number. Is that bigger than you've done in other places? Is that because AI is helping you go faster?
Kameron Tanseli (04:59)
AI is definitely helping us go faster. But when you break down the 360 and then you look at it across the entire product, it actually becomes not that impressive. I've talked to CGOs and they brought up the 500 number of the entire order. And then the next question to me is why aren't you running a thousand? Cause really like 360, so I'll break it down weekly and it's around like two or three a week.
Ashley Stirrup (05:20)
Mm-hmm.
Kameron Tanseli (05:28)
And then you assume we've got the engineers across the entirety of the product. So two, three experiments on the entire surface area from landing page to using the product in the inbox. It's actually not that many. And that's why this year I'm trying to get the team a lot faster as well. So can we break through a thousand with a team of around eight? Right now we're at six people now, including myself. By the end of the year, hopefully 13.
And hopefully we're north of a thousand tests with the main sort of focus being on developer performance. So as I mentioned in Kyle's article, we have cloud agents for cleaning up old experiments, but the shift in the last few months, especially from like cursor and tempo and codecs is to start running virtual desktops. Because for AB testing at least, which is quite visual.
the CRO related things. Anytime you use Codex or cloud agents, they're doing the code changes. There's a terminal. They made the change. Great. I, as a developer, still have to pull down the code, run the app, check that the upsell panel looks correct, check that I'm happy with the copy and so on and so forth. With cursor desktop, it's running the app. It's showing me a video of what the experiment is going to look like. And then I can just sign it off without having to do the extra manual QA myself.
So it means I can parallelize four to five to six different experiments. As long as they're fairly simple, there are obviously other experiments in the back end, which are even simpler to do one or two lines for a config change. But that's really how I see the dev team going to 1,000 experiments this year.
Ashley Stirrup (07:10)
Got it. So it's not really like a mindset change. mean, I'm sure you're encouraging people to test everything, but it's really more just the state of AI kind of making it easier to go turn these things on.
Kameron Tanseli (07:24)
Pretty much. There's also what used to be is a lot of manual effort in data analysis. Whereas now we have tools like dot, or we have code code running over our database schema in order to write more complex queries. So we do need to dig into post experiment, what's happening here, because growth book and.
is feeding off post-hoc and that feeds into BigQuery, we all have one source of truth that we're pulling from, which is BigQuery. So if we do need to do an advanced analysis, we can dive in there with the help of a data science agent and quickly get results, but also research for upcoming experiments too. So we're saving time on that end at the start of the experimentation and afterwards as well and during the development.
Ashley Stirrup (08:10)
Got it, got it. And so when you're at a business that's in hyper growth like Fyxer, how can you tell whether it's the new features that you're delivering that are really helping or whether it's just the overall momentum of the business?
Kameron Tanseli (08:23)
Great question. With a company like Fyxer, there'll be some inputs, right? Like ad spend and the usual rates of people starting subscriptions and that coming through. So you're able to sort of estimate the growth that you're going to have. With areas like PLG, it can become a bit more fuzzy because with PLG features, especially when you're launching them, it maybe takes a while.
for them to start to take off. But when it does, you can really see it. My method so far has been, if I'm launching something and I don't see it impact the global revenue, like an ARR chart, I'm not doing something correctly. So for us with a lot of our PLG features, especially late last year, we managed to unlock that. And that's when we seen that big
increase, same with moving to a trial-based model when I joined back in Jan. That was the huge impact. And we just keep trying to run for that and keep doing big things. Cause the worst thing you can do is try and maintain the status quo and do those five to 10 % uplifts.
Ashley Stirrup (09:32)
Right. And so I got a little lost there. It sounds like you had to unlock when you went with the free trial. It also sounds like with PLG, some of those features, take a while before you start to see the results. And so like, does that force you to give those experiments more time?
Kameron Tanseli (09:52)
Yes. Those, so what's interesting is those features, you can definitely do it before and after. So like you launch it, you assume that it's not going to do anything negative or it's a, add-on to your business. So like you can launch it for two weeks, three weeks and just be like, we didn't cause a bunch of Chan introducing this. Because more often than not, no one's going to use your feature. The amount of times I've launched something, no one cares about it. So.
Ashley Stirrup (09:59)
Mm-hmm.
Yeah.
Mm-hmm.
Yeah.
Kameron Tanseli (10:19)
What I'm doing is I'll launch a PLG feature in and then I'll work on experiments immediately after that to see if I can increase usage. That's the, that's the parts that I'm really testing quite fast. The main surface area has been shipped. Now I'm running tests on it. And then I'm looking on like Metabase, as the daily active usage of, you know, the percentage of our existing user base that's using that feature. As that starts to increase.
That's when I'm like, okay, is it causing an impact to the revenue? Are we getting the PLG that we expect now that we have users being driven to it? But doing it in one big go is often a fallacy. Like no one's going to use your feature yet.
Ashley Stirrup (10:56)
Yeah.
Yeah. And so it sounds like you kind of deliver the main feature and then you iterate on it and see like, how do I increase engagement in this things like that? Yeah.
Kameron Tanseli (11:12)
Yeah, it's trying
to get as many surface areas for new additional testing. That's my aim with PLG.
Ashley Stirrup (11:18)
Yeah.
And how do you differentiate between a PLG feature and just a product feature?
Kameron Tanseli (11:26)
When I'm planning PLG features, what I like to do is break it down into two diagrams. So we have a habitual loop, which is your trigger, your action, your variable reward. And that drives you to continually use the product. And then we have the growth loops, which is the actions that you're doing in the habitual loop. Are they able to create something like a shareable artifact, which can bring in new users? With that clear distinction.
I'm able to assess our product feature list and go, OK, this is something that we have lots of daily active users using. So every day I can ask permission for you to share something. And then I'm turning that feature into a PLG feature by creating shareable artifacts from that. Now, not everything works just because it's in a diagram and it looks really cool in theory. But how we approach testing that is we'll draw a new line that goes out.
and we'll come up with a theoretical growth loop. And then we'll go through as many hypotheses and assumptions that we've made and then try and invalidate all of those. So for example, like we had a scheduling feature, it's like a Calendly sort of link. And I was like, you know, when people book, we can send them a booking confirmation and that will lead them to, you know, sign up back to Fyxer. And so I came up with a bunch of hypotheses. was like,
If we send this booking confirmation, people will sign up and maybe they'll reschedule and they'll do all these things. We started sending out these booking confirmations as an experiment and people were like, why are you sending out booking confirmations? I get the Google Calendar invite email and Outlook Calendar invite email and then you've obviously given me an extra Fyxer email and Fyxer's main goal is to not overload your inbox. So we were like, ⁓ damn, we're not able to loop this back in now. ⁓
Ashley Stirrup (13:12)
Right.
Kameron Tanseli (13:12)
in
order to close that loop. So we need to think of something else ⁓ in order to grow scheduling. And that's how we pivot and grow it.
Ashley Stirrup (13:20)
Yeah, that's a great story. Overall, you've had tremendous results. I believe last year you grew the business from a million to 30 million, is that right?
Kameron Tanseli (13:28)
study five.
Ashley Stirrup (13:30)
35, yeah, even better. Yeah. And so like, I'm sure it's difficult to say how much was, you your team's work versus others, but as you kind of look at the year, like how much of a contribution do feel you made?
Kameron Tanseli (13:31)
35, yeah.
If you think that, know, POG now accounts for more than half of the ARR that we're doing, I want to say a lot, but we unlocked that later in the year. So I can't say that I've made an impact of all the revenue that we generated, but definitely we've accelerated it at the start and at the end, I would say.
Ashley Stirrup (14:04)
Right.
Yeah.
Kameron Tanseli (14:09)
Our marketing strategy is, you know, a big proponent of that. we're huge on meta, probably if you've been spammed with our ads. Um, and we've managed to have some unlocks. think the culture of the business as well is experimentation first, which is great. The product has significantly improved. Um, as the product team has adopted AB testing with the mindset of how do we improve our alarms?
Ashley Stirrup (14:18)
a little bit.
Kameron Tanseli (14:39)
through A-B testing rather than just pure evals and fine tuning.
Ashley Stirrup (14:43)
Yeah, that's a trend we're seeing a lot more in our customer base is that you just, you know, it's one thing to do an eval, but it's another thing to get it in front of real people and see how they react. Yeah. And so you said it's a culture of experimentation at the company overall. was there some leader in particular that really kind of drove that there?
Kameron Tanseli (14:56)
Exactly.
All three founders were pretty bullish on experimentation. It definitely helped. The growth team started running tests immediately. And then those tests, we had winners and the business was like, I really like this idea of us being able to measure the impact of our work. Let's do it everywhere. So all three founders, Rich, Arch, Matt, they all sort of adopted it because it's
Ashley Stirrup (15:25)
Yeah.
Kameron Tanseli (15:31)
felt like the thing that was the right thing to do at that time. And we could show progress, which was very motivating for the rest of the business. Cause we have a public channel from day one. always been huge visibility on experimentation, launch loss, why it lost. So the rest of the business can also feed in. And since then it's become this huge visibility hub. The founders love it because there's one place to look at what's happening in the business.
And this is what I mean where experimentation is core to what's happening at Fyxer.
Ashley Stirrup (16:03)
Yeah. And when you say there's one place, that, did you say that was Slack or somewhere else?
Kameron Tanseli (16:09)
We push into Slack and we try as much as possible to automate that. So automating from growth book with an API, but also we have cursor that writes product release docs. And then that gets put into topic experiments, which is the of the channel. Paid marketing also puts into their CRM. So it's like the whole business is contributing to this one sort of change log.
Ashley Stirrup (16:11)
Mm-hmm.
Wow, that sounds great. And so it sounds like you had a pretty big unlock somewhere in the later part of last year with the PLG motion. Like, what does that look like? Was there like one particular breakthrough or was it just a series of small incremental winds? How did that work?
Kameron Tanseli (16:51)
We took a feature that was already working and it was a series of small incremental win. We're like, yeah, people get stuck at this point. So we need to optimize the email that they click through from. We optimize that so people are clicking through. And it was like, in a very linear way, unlocking each step until people started to sign up, add seats. And then we started unlocking that, additional error.
Ashley Stirrup (17:16)
what was it about? About improving the kind of the onboarding journey for people? that what did it?
Kameron Tanseli (17:21)
was about improving the mechanisms for the virality itself, getting people to back to Fyxer.com. You know, we live in their other tools, so it's really important that we drive people back to Fyxer because that's where they can sign up and then add seats to their orgs. So it was very much testing different messaging, different emails, different signup pages. We have custom signup pages depending on where you come from.
Ashley Stirrup (17:39)
Yeah.
Kameron Tanseli (17:49)
as well. And that's been a huge thing for us is being really personalized because we know a thing or two from where you've come from and what it relates to.
Ashley Stirrup (17:54)
Yeah.
Right. And you kind of alluded to it, but it sounds like getting companies to sign up is pretty big and then kind of virally within those companies that is important to you. Yeah. Yeah. That's pretty fascinating. Cause usually you hear about viral and you think of more B2C products. So that's pretty exciting to see it in a, you know, in more of a business context. So.
Kameron Tanseli (18:09)
Yes.
Well, all the work is happening in the inbox, if you imagine. Fyxer tries to inject itself into that, whether it's your meeting, whether it's your email threads, whether it's scheduling. So the possibility of P.O.G.able things is actually quite a lot, if you think about it, in the context of your work.
Ashley Stirrup (18:27)
Right, of course. Yeah.
Right. Yeah, that makes a lot of sense. And so like, you already talked a little bit about, ⁓ kind of learning from losers, but, you know, how do you decide, okay, this is a loser and it's time to move on versus this is a loser and I want to, but I want to keep iterating on it.
Kameron Tanseli (19:02)
Yes.
I think with the amount of surface area that we're testing across, if we have really high conviction, like we know that there's a huge drop off area and we try one thing and it failed, I'm obviously going to be very, what's the word? Yes, determined. Like if, if, you know, 70 % of people are dropping off on a step, I know that there's a huge win there.
Ashley Stirrup (19:23)
Determined?
Kameron Tanseli (19:31)
If the conversion rate is 80 % are coming through, then if I've tested a few things, like, this probably isn't the major area which I should be focusing on, or I need to do something massive. And because the team is constantly working, like if we do want to retry something or try it from a different angle, we will. I don't think we see repeated testing as wasted effort. We retry failed tests from last year and now they win as well, which is
been an interesting sort of thing to happen as our customer changes and other tests have made it so we have a different environment now and product and feel. we do test a lot. Some things, you know, if there's sig negative two, three times that we've tried something, maybe we'll back off and go, let's just rethink the approach entirely. And that has happened recently.
Ashley Stirrup (20:20)
Yeah, yeah, that's all very fascinating. Do you think the role of experimentation is gonna change as your product matures? You know, it's one thing with a really early stage product, you do a bunch of things and you really can move the needle, but over time you'd imagine that the changes would lead to kind of smaller, more incremental improvements.
Kameron Tanseli (20:42)
For the more established features, we'll build out teams specifically for doing those exploitation style experiments to get those marginal gains. With AI and the industry and how it's going, can't really, there's not a whole lot of features where that's happening because everything's being disrupted all the time. And at Fyxer, we're working on some huge things in the pipe now. So even more so.
I feel like I'm still doing the big stuff even as we go series B and beyond. So not yet, but probably once we're 200 people and very established.
Ashley Stirrup (21:17)
Yeah.
Yeah, and you've got some huge goals for growth for this year as well, right?
Kameron Tanseli (21:27)
Yes, we want to soar past 100, 150 million in error.
Ashley Stirrup (21:32)
Yeah, that's just incredible to talk about a company that's gone from kind of one to 35 to 100 plus million in just a couple of years' time. Almost unfathomable. So if someone were to say move into a growth engineering role, let's say that it's their first role at a new company, early stage, one to 10 million in revenue somewhere in there.
What kind of advice would you give them in terms of how they should approach the job?
Kameron Tanseli (22:04)
You're going to be really bad at it for the first few months. Even I'll be bad at it. I've been doing it for like nine years. If I joined a new company, I'm to start launching stuff and people that are already been there are going to be like, what are you doing? Like, or this doesn't, this won't resonate with our customer at all. But eventually after a few months, you'll start to learn about the customer, learn what works, and then you'll really find your group.
Obviously, hopefully you have a gut instinct for UX and UI and some sort of other discipline, whether it's design or data science to help you, because you really should be T-shaped coming into this role. If you're not, you're going to struggle or you're going to have to adapt really fast. But yeah, you're going to suck. You're not going to come out the gate swinging.
Ashley Stirrup (22:48)
Right.
Yeah, and that's really just about learning your user and learning the basically the use cases that they're going through. Yeah.
Kameron Tanseli (23:01)
Yeah. And it's been a trend for
me, like joining Fyxer was B2B SaaS, prosumer AI, like completely new space. Joining Newman was B2C health tech. And then before then I was in B2B, like pure SaaS and each time like sucked in the first few months, trying to adapt and learn what the customers liked, what integrations they were looking for and so on and so forth.
Ashley Stirrup (23:09)
Yeah.
Yeah.
Yeah.
And like, what are some of the key learnings that you gathered in those early months that then helped you be successful later?
Kameron Tanseli (23:32)
⁓
I want to say a lot of it happened very fast and I've just sort of consumed it. So trying to say what the learning source is a bit difficult, but definitely. So I was going to say definitely moving from B2C health tech where you're a bit more advertising in your language to try to be a value-based tool was definitely a hard, harder transition. And I had to sort of undo that.
Ashley Stirrup (23:42)
Yeah.
It's until...
Kameron Tanseli (23:59)
marketing style language and just be genuinely helpful. know, I'm there as an assistant to help you set up. I kept in the first few months being like, discounts and cost savings and pricing plans. It's like that works for subscription, you know, boxes that come to your house every month for not for B2B SaaS, especially, you know, when you're dealing with AI credits and usage and these sorts of stuff.
Ashley Stirrup (24:01)
Cut.
Yeah, yeah, that's really interesting. So just learning as much the language to talk to your customer with was important. Yeah, that's not what I would have immediately thought of. I would have more thought about like, what does my customer care about? And ⁓ they always ignore these kinds of features, but it's also about kind of the style of how you show up with them.
Kameron Tanseli (24:32)
Yes.
Exactly. It's because you're going to have most zeroes trying to convince someone to take action on something. So you need to learn to chat that and go find out what they care about is a big portion of that you're right. And the signals that they look for as well in the funnel. Because once you understand that, you can generally form some sort of consensus of where you should look next.
Ashley Stirrup (24:53)
Right.
Yeah.
Yeah, yeah, that makes a lot of sense. So you kind of alluded to a pretty interesting stack of tools that you're using today. Could you kind of walk us through what you're using?
Kameron Tanseli (25:20)
Yes. Um, weirdly it changes quite a bit. Uh, the AI system moves so fast. We've now moved over to Claude. So we have, uh, Claude, we're not, we have some people in the team using Claude code, but mostly we use Claude as an assistant. I really love the diagramming within the chat. Uh, sometimes they're doing data analysis in there and it's really nice to see those visualizations.
But we're using Claude with shared skills across the team. So can we take a growth book experiment using the MCP, turn it into a shareable Slack post, and then post it into Slack? Can we analyze the result or use dots through Claude? Like Claude is very much the hub where we're doing things. And we can have requests come off from Slack bots as well. The Claude Slack bot curse the Slack bot.
Then we have Cursor for development. And as I mentioned before, the Cursor desktop cloud agents are very useful for smaller experiments where you can just one shot them now with Opus or GPT 5.4. And then in the PR, we're also using Cursor. It's basically full stack Cursor in terms of development. So we can share skills, plugins, and rule sets.
And that is enforced by the CTO who wants everyone to share everything. But it's also a nice, it's good to have the CTO enforce that because it means that as a team, we can then share skills and plugins, which is the core to automating a dev process. And then post that we have cursor automations, which we can set up when a PR is opened, when a growth book experiment or every day check for growth book tests that are still left in the code, clean them up.
And we just continuously create new automations as we see steps and processes come up. And then obviously growth book for the analysis. And then go from there.
Ashley Stirrup (27:17)
Yeah, and you mentioned dot. I'm not familiar with that. What is that?
Kameron Tanseli (27:22)
So dot is an agent that we've given our table schemas to. So the data team, shout out to them. They have meticulously written markdown files for what the tables are, the columns, and how it should map on what their relationships is. And now that they have that, most of their team is also quite automated. So they're able to write most of their queries through code code. It understands relationships. It's able to.
write the query, give us the results. And we have that through Slack as well. And that's essentially what dot is, is the consumer side. It's basically called code with a dbt schema and we're able to add it in Slack. And it's able to answer questions that are quite complex. You know, it can do segmentation and it can do survival analysis. It's basically running Python, NumPy, SciPy, all of that in order to compute those results.
Ashley Stirrup (28:04)
Got it.
Kameron Tanseli (28:18)
So for non-technicals, it's been a huge unlock. Non-technical stakeholders will overload your data team over time with requests. And engineering teams will get less prioritized requests because we can write SQL. So eventually it's like, okay, you need the non-technicals questions answered. can just, you know, DIY your own. Now it's the data team can give us a bit more time because DOT is answering the non-technical stakeholder questions.
Ashley Stirrup (28:45)
Got it. And you mentioned having like an AI data scientist. Did you mean dot or what were you referring to there?
Kameron Tanseli (28:53)
Dot is an AI data scientist, if you prompt it well, but the data team themselves have particular skills, Claude skills, not like that they do have skills, but yeah, that make it act as a data scientist. But also a lot of their manual work that they used to do has been automated.
Ashley Stirrup (29:13)
Got it. And is some of that talking to a growth book with the skills?
Kameron Tanseli (29:19)
Only, so one of the things that they do use the growth book API for is when we see a metric go down negatively, they'll use growth book API to find when experiments may have gone out. Because what stakeholders want to answer is, you know, did the growth team tank conversion rate this week and is that why, or is it like a channel issue? So the data team is able to go, Hey Claude.
Ashley Stirrup (29:43)
Mm-hmm.
Kameron Tanseli (29:45)
what was released last week and did it affect conversion rate and it's able to put that together into a report and then send it. So that's where they're using it.
Ashley Stirrup (29:51)
Got it.
Got it. That's super helpful. It ⁓ sounds like a really exciting time that you're using so many different tools to really make the whole process a lot faster. Yes. Well, we're out of time. Kameron, I really appreciate you coming on the show. This is just such a fabulous story. I can't wait to see where Fyxer goes next. And I'm looking forward to becoming a user. I'm going to go sign up right after the show.
Kameron Tanseli (30:04)
Yes.
Thank you for having me.
Ashley Stirrup (30:23)
Thank you, thank you so much.