Subscribe
Share
Share
Embed
How Experimentation Leads to Annual Growth Every Year at Fanatics
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
Ashley Stirrup (00:00)
Welcome to today's episode. I'm excited to welcome Medha Umarji VP of Growth and Experimentation at Fanatics. And Medha, maybe you could start things off with just telling us a little bit about Fanatics and the scale of the business there.
Medha Umarji (00:15)
Sounds great, Ashley, I'm very excited to be here. Thanks for having me. I've been at Fanatics for about 10 years now. Fanatics, ⁓ you might have heard about us. You might have seen billboards and posters about us at all these different sporting events. We are basically ⁓ a sports merchandising ⁓ retailer, and we're like a pure play e-commerce company, but we're starting to get more into ⁓ in-venue retail and things like that.
I work for Fanatics Commerce, which is one branch of the entire Fanatics umbrella. We also have Fanatics Betting and Gaming, some other kind of Fanatics collectibles, Fanatics Live, ⁓ our other sister companies at Fanatics. At Fanatics Commerce, our scale is pretty big. I want to say we are growing. uh
We have about 60 million visits a month. It's a pretty, we have about 900 sites, all the major sports leagues and like WWE, NFL, NBA, NHL, all the soccer stuff is with us. So we are, ⁓ I wanna say the most dominant player in the sports licensing and merchandising world. So it is an exciting time.
to be here and we've learned a lot. ⁓
Ashley Stirrup (01:31)
Yeah, and I didn't realize
it's a multi-billion dollar business. That's pretty incredible. And so I decided to have some fun with today's episode and got myself a hat off of the site. I'm a big Warriors fan. Sadly, they're not in the playoffs anymore, but I can still root for them. And so it was a good excuse to buy some of your merch.
Medha Umarji (01:37)
Yeah.
⁓ my god.
Yeah.
I'd be interested to know your experience of shopping with us. We work hard, but... Okay.
Ashley Stirrup (02:01)
It was very smooth. thought about the whole
checkout process as I went through it. And yeah, it was super smooth. So it was great. So maybe to kick things off, could you tell us a little bit about the experimentation program at Fanatics?
Medha Umarji (02:08)
I'm happy to do that.
Yeah, experimentation at Fanatics has evolved from like a boutique CRO team, know, testing like just little conversion nudges and things like that into like a full fledged, mature, you know, program that now tackles like all stages of the entire like e-commerce life cycle, you know, from, from merchandising to pricing to...
fulfillment, delivery, customer satisfaction, and just a whole plethora of aspects around the entire e-commerce business, really. We started out by doing about, I wanna say 10 experiments a month, and we are close to 100 a month at this point over the last 10 years. So it's definitely been a growth journey.
Ashley Stirrup (03:05)
Yeah, that's terrific. And what led to making experimentation that important? Why did you invest to grow it at basically 10x?
Medha Umarji (03:16)
⁓ I think that, you know, as I mentioned before, since we're a pure play e-commerce company, all the signals that we can get from the, you know, from our customers are really via testing and learning. And, you know, there was almost no option for us, but to kind of really lean into experimentation to learn about our customers, you know, in addition to the user research and things like that.
And the other interesting thing at Fanatics has always been that our leadership has been so invested in the data and just the trajectory of ⁓ findings about customers that, I don't know if that answers your question, but yeah, like basically we, just to give you an example, mean, we try to test each and everything that is hitting our site, right? And it's not just,
whether we're launching a new payment method or we're launching a new feature in checkout, but it also is things like the tariffs are coming, we need to test our prices, what impact is this going to have on the business? And ⁓ the good news is everyone from our CEO down is very invested in data. We all have our opinions and ⁓ everybody feels very strongly about them. However,
Our culture is such that data sort of takes the precedence. mean, nobody's arguing. As long as we're convinced and we think that the data makes sense, all the decisions are very much data-driven at Fanatics. And so that sort of buy-in from the top down has really helped the adoption of experimentation and the growth here.
Ashley Stirrup (04:58)
Yeah, I think it's so interesting how some leadership teams really lean into experimentation and others don't. Do you have any thoughts on why that is?
Medha Umarji (05:09)
I think, I mean, just from my personal experience, I think that it really depends on the context of the businesses, you know, because we, and the context of the problems that you're trying to solve too, right? For example, you know, there are some problems that are uniquely suited to the experimentation ⁓ structure, you know, whereas there are some problems that have longstanding implications that are just not going to be measured by experimentation. And in many of those,
situations, you might find that leadership has to sort of take a stance and make a call on like, okay, this is what we're going to do because it's right for the customer, it's right for branding, it's right for the business. At Fanatics, we've kind of ⁓ evolved our understanding of the long ranging impacts of things that we do on the site. ⁓
And we don't have as many of those sort of decisions to make. But I think my thinking is that it really depends on the problems that we're trying to solve. ⁓
Ashley Stirrup (06:13)
Yeah, that makes
total sense. Yeah, it's one thing when you're kind of refining something that you've already got. It's another when you're making a new big bet. I you always want to track everything, but you do it differently when it's like a brand new Greenfield thing. But experimentations had a massive impact on the fanatics business, isn't that right?
Medha Umarji (06:21)
Right.
Yeah, so I want to say like over the last 10 years, ⁓ every year, consistently we are delivering about 8 % of the total growth at Fanatics. You know, the experimentation team and the product team sort of working together. We've had, ⁓ I want to say, quite a landslide of success at Fanatics. Fanatics.
Ashley Stirrup (06:55)
Yeah,
and for a multi-billion dollar business, that's a massive impact, 8 % growth on that. Yeah, so congratulations. ⁓ And so it's no wonder, but I guess your CEO is pretty engaged in your monthly experimentation meetings.
Medha Umarji (07:03)
Yeah, I think. Thank you.
Oh absolutely, our CEO, CPO, Everybody everybody I want to say in the C-suite is very engaged with the outcome of experiments and you know I think whenever we have something launching, you know the first thought is that people have now is like hey how are we going to test this, how are we going to measure the impact and
And that's exactly where you want to be, you know, just from an experimentation team perspective, where you don't have to deal with like convincing people to test things. It's more of like, you know, how do we do it? Right. Rather than why or, you know, and whatnot. So, yeah.
Ashley Stirrup (07:48)
Right.
And your CEO is also open to having his mind changed with data,
Medha Umarji (07:53)
yeah, all the time. Yeah, we'll notice it. know, like, because, you know, we all have our intuitions and, you know, we'll try things and, ⁓ but, you know, yeah, our CEO is like so data driven. Like he literally like consumes Excel's like, you know, he's like, do not shy away from putting data on the slides. Like I want to see everything. And he will question everything. Like nothing gets by him. You know, like he, he's a very like.
data geeky kind of person, think. But I think it helps us, right? Because we are, you know, that's exactly on the same page with all the conclusions. You know, I think, and the good news for us really is that, you know, it's kind of like this whole notion of focus and context, right? Cause like we are very hyper-focused on that small little test that we are running, but the leadership team has sort of this broader context of like where the business needs to go, what we need to do and.
Ashley Stirrup (08:20)
Yeah.
Medha Umarji (08:44)
And they will really help us sort of fine tune, know, okay, like we're testing this piece of the puzzle, but you know, there are these other three things that kind of go into this whole, you know, ecosystem that we have to also account for. And therefore, you know, let's either broaden the scope, let's, you know, let's try testing a couple of other things, you know. So there's always this really interesting dialogue around our experimentation results.
Ashley Stirrup (09:11)
Yeah, that's a
really interesting kind of strategic unlock when you've got your top leadership thinking strategically and they can partner with you to help figure out how to frame what to experiment, what to test and where the biggest opportunities are.
Medha Umarji (09:26)
Exactly.
Ashley Stirrup (09:27)
The other thing you've told me that I think is just so interesting is that if your CEO is demonstrating that he recognizes he can be wrong and that if the data says something different that you're going to do something different, it just sets the culture for the whole company.
Medha Umarji (09:43)
It's exactly such a culture, ⁓ you know, ⁓ center. Yeah, exactly. Those were the words I would use because, you know, it's like top down. ⁓ Everybody is, you that humility is very much a part of the culture ⁓ here.
Ashley Stirrup (09:58)
Yeah. Yeah. I just think that's such an interesting, interesting topic. We could spend the whole podcast on humility. could you tell us a story of a time when you ran an experiment and you had a surprising result?
Medha Umarji (10:03)
Thank
Yeah, so this one is interesting. You might have seen this on other e-commerce sites too, but we tend to lean into ⁓ ads and ⁓ other partner initiatives just to bring in some incremental revenue. we tested turning, but everybody hates ads because they just clutter up the experience. They just look so, they make.
you know, they make your site look less attractive, you know, and what we did was we tried to turn off ads because we really wanted to understand like what is the impact to the customer ⁓ in this particular, you know, feature. And so we turned off ads on all our grid pages and the first result, and remember everybody's rooting for the ads turn off variant to win because we really, you know, care about that. But when we turned it off and it was positive, you know.
And we were all so excited. But then we have a very rigorous double-click on anything that is looking positive. We try to establish causality between what we're seeing at the very top level metrics to down to the, hey, did we see more people browse more grid products? Did we see them scroll more? Were more products viewed? And did we see a higher increase in?
traffic from grid to cart. There's all these little micro metrics that we track. And we did see a whole lot of that movement. And then there was a lot of question about, what is driving this lift in total revenue? And so what we did was, which is again, another thing that we do here to reduce our false positive risk is we replicate our outcomes. We try to replicate it if we are not able to explain it.
And so we ended up turning off this test and rerunning it, you know, and we weren't able to replicate it. It was actually really flat. you know, going back to like just the historical time, because this is not the first time we had tested it, right? We've tested it about six or eight times now and we've always seen them be really flat, you know, and that's, think what happened here. The test was completely inconclusive in the second run.
I think our hypothesis there, it was surprising to us, A, from a methodological perspective that, this is a real instance of us running into that 5%, like it was 95 % stat sig on the first run and it's kind of inconclusive in the second one, so that means it's kind of in that gray area. But also secondly, I think people just, I the customer insight we got from that is that people just have banner blindness after a while.
You know, like it doesn't degrade the customer experience as much as we think it does. You know, mean, like qualitatively it looks terrible. Like it doesn't look great, but you know, it doesn't. mean, people just sort of gloss over the banners at some point, the sponsor, then the, you know, I suppose, mean, you know, it's hard to know for sure, but I think that that's probably what's happening because.
Ashley Stirrup (13:00)
Yeah.
Yeah, well, I think there's two really interesting takeaways from that story. You know, the first is just, yeah, 95 % means that one out of 20, you're gonna have, you know, a false positive. And so if it really matters, go back and test it again.
Medha Umarji (13:26)
Yeah.
Ashley Stirrup (13:27)
⁓ But then the second thing is I love how you talked about how you looked at all the other associated metrics and like, okay, if we really did see this result, we should see some of these other metrics moving in the same direction and we don't.
And that to me, you know, because the most important thing from experimentation is learning. And a single metric isn't really going to help you learn. It's understanding, okay, how did the user behavior change? What was, how did the user experience change? And the fact that you're looking at all those things shows that you're really kind of tracking all that and looking to not just get a number and declare a win, but get to get insight out of every experiment.
Medha Umarji (14:07)
Yeah, and it really goes back to sort of, I mean, there's like this inherent assumption whenever you're running an A-B test that whatever you're finding as the outcome is causal, right? And I think what we really have learned over the years is that, yes, I mean, it is causal, but we absolutely do.
have to do a double click there because A, sometimes we might not have the power we'd like to have. We might not have the sample we'd like to have. We might not have the testing environment. There might be outliers or suddenly there's a draft pick and now all our data is doing its own thing. So you almost owe it to yourself to have that double click on any test outcome.
Ashley Stirrup (14:48)
Right. Yeah.
Yeah. And speaking of learning, you ⁓ have a wiki at Fanatics and you kind of have a process for how you kind of test and iterate. Is that true?
Medha Umarji (15:05)
Yeah, and you know, I'm very excited to talk about the Wiki all the time. Whoever joins Fanatics new as a company, you I always like to share them with our Wiki. I think the interesting thing there is like, you know, we started off with like all these PowerPoints and, you know, there would be these Word documents of test results floating around. And then we realized that, you know, experimentation was such a core decision making sort of function at Fanatics that, you know, it just it didn't make sense to have it live in these, you know, sort of ⁓
directories somewhere. So we built a wiki and you know the good thing about the wiki is you know we document the results, we provide all this like causal sort of interpretations that we have and also you know we have a very robust sort of next steps and recommendations section that we you know we've also built some automations that like takes that converts it into a backlog and you know and we're constantly sort of you know iterating and learning from those things you know whether it's a win, whether it's a loss.
whether it's an inconclusive outcome, like we will take nuggets of learnings from each test and then they become their own. And that sort of feeds into this ⁓ experimentation flywheel, if you will, where we're, because the feature is already built, remember, there's no tech dependency at that point. so, because otherwise our velocity will slow down if we're just waiting for tech to build new features for us, right? So we take the existing features, we...
build a roadmap on iterations and then ⁓ our roadmap on a regular basis, if there's a new feature that comes in, that takes the highest priority, but then all these other features that have come up from the Wiki just feed our roadmap whenever we're open. Whenever there's any bandwidth, we'll just slot in these iterations. ⁓ So that's why I love the Wiki. I like to talk about it so much. It's because...
Ashley Stirrup (16:58)
Yeah.
Medha Umarji (17:01)
It's sort of become our growth engine, you know.
Ashley Stirrup (17:04)
Yeah, so it sounds like you're putting a lot of rigor into your process and really documenting your findings and documenting next steps. Does the Wiki make it easier for people to say what other tests have we run like this test?
Medha Umarji (17:09)
Yes.
Yes, exactly. so what we do is, know, so we do, by the way, I mean, this is an aside, but the Wiki is really helping us as we move towards AI, because we now use Glean and ⁓ Claude and, you know, and there's connections with that relation on all of those. And so, you know, people are able to build their own little like, you know, feature based kind of ⁓ meta analysis.
But one of the things that I challenge my team to do on a regular basis is like, if you run more than three tests on a particular feature, then you ought to build like a little ⁓ meta table of all the outcomes for that feature. Like, hey, we tested this verbiage, three different swings at the messaging, or we tested three different price points, or we tested three different colors. Let's put it in a little consumable like nugget.
that then becomes much more easy to distribute and people don't have to read each brief. They can just get a quick summary as they look at the meta-analysis. ⁓ And that is, I feel, one of the other kind of value adds that we provide ⁓ as well.
Ashley Stirrup (18:19)
Yeah.
Yeah, that makes a lot
of sense. So is the Wiki itself your backlog or you kind of do you have like a Jira or something where you're kind of tracking all this? Yeah.
Medha Umarji (18:33)
⁓ yeah,
we have a very sort well fleshed out JIRA board, you know, with all our information in there. I think we're learning, know, we have now with all the AI stuff happening, there's a lot of automation between JIRA and Confluence. You know, the other thing that we do really put in the Wiki is all the experience call-outs, you know, because one of the things that you'll notice as you test is that you'll
the result will tell you something that you did not anticipate at all. And you'll be like okay, why is this metric even impacted? This wasn't a part of the initial hypothesis, whatever. And then you go back to the screenshots, you go back to the videos that you take when the experiment is running and you're like, my gosh, look at that, this particular metric. In one test, we suddenly saw our expedited shipping rates go up.
You know, which is great. mean, we're not complaining, but it's like, why? You know, this wasn't a part of the whole story, right? But then you go back and you're like, you know what? As we change this thing on the PDP, this particular vibrancy was elevated, you know, and that's why, you know, it caught people's eye and it's actually, you know, trending well. mean, so, you know, at least now we're able to kind of tie these pieces together. So like just documenting the actual, you know, visual experience is a big part of the Wiki as well, you know.
So, yeah.
Ashley Stirrup (19:57)
Yeah, yeah, that makes
a ton of sense. So just so many opportunities to learn from every experiment. Yeah. So I'm sure sometimes you have to deal with people that maybe don't want to wait for the experiment results or they think they know better. How do you deal with situations like that?
Medha Umarji (20:03)
Yeah.
Yeah, that one's always tricky. I think it's, ⁓ you know, there's this sort of, know, if you're a high growth company, I mean, there's a lot of pressure on different teams to really deliver value. especially if experimentation sort of then becomes a bottleneck for, you know, launching features and doing things, you know, there's definitely a deeper conversation. ⁓
to be had, I think you learn, being in experimentation for many years, you kind of learn to get over your own opinions very quickly, because you're right only about 10 % of the time or whatever. I think when it comes to new teams or new kind of things that are getting onboarded, there's definitely that skepticism that creeps in. they're like, you but I realize one of the things that
I strongly believe is that everybody actually has valid points when they come to you because I think part of their hesitation could be that they're worried that the test would not measure what they are trying to measure, because they could just be launching, let's say it's a branding play and they want to just invoke customer sentiment and there's no way that your test is really going to capture that. And so what we do is now we've sort of broadened our scope a little bit and we say that, hey, you know what?
We highly suggest you run this experiment. However, we also suggest you do X, Y, and Z, like work with the customer research team, you work with this and the other. And to that end, Ashley, the other thing that we've done is we've kind of launched a couple of different frameworks. So we have a framework that we call the do no harm framework. It's basically these non-inferiority guardrails. So as long as you're not hurting all our primary KPI, you could, and you believe strongly that
this is going to have a better impact after the purchase, or it's going to be better for customer brand perception or things like that, we've now opened up that track for you. And the other thing that we've done is we've also built a small sample framework, which basically says you're not going to the high standard of orders and metrics that we've placed on the rigor that we use for our regular experiments.
However, these smaller things that you want to do, let us build something that supports you in your decision making. And that way you could still have whatever rigor is affordable to you and still make changes, meaningful impact to the site experience.
Ashley Stirrup (22:47)
Yeah. And probably one of the most dangerous things you can allow us, you know, people to just go make changes thinking, well, this isn't going to hurt. And if it actually does, you want to know. yeah, do no harm tests can be really powerful that way. so, ⁓ as we wrap up, do you have any, ⁓ final advice for somebody early in their, ⁓ career and experimentation? You know, how do they make experimentation strategic at a new company that they joined?
Medha Umarji (22:56)
Exactly.
Yeah.
So I think that when it comes to experimentation, I mean, I'll say a couple of things. One, I think the proof is in the pudding. I think that once you start running some experiments, you find teams that are more inclined towards data. You start driving adoption with the people that are already sort of in that frame of mind. And then,
you start to publicize the insights that you get and you start talking about it, you do more road shows, you get more awareness and publicity. But then also, not publicity, but yeah, like just more kind of awareness. But I think the other aspect that I've realized over the years is I tend to not be too persnickety about the data and the experiments. For example, if your test is getting 300 orders,
and I need 30,000 orders in my test, you know? But I urge you to compare your 300 with your 400 in the B, you know? And it's like, okay, there's actually a 100 order delta here, guys. So this is telling you something, you know? So let's, I mean, I think the more sort of democratic you make it and the more open, you know, the lower barrier for adoption that you can have, at least at the initial outset of launching a program, I think.
Ashley Stirrup (24:17)
Right.
Medha Umarji (24:34)
you're going to get a lot more traction if you are open-minded. If your barriers are too high, then people are just not in the mindset to adopt.
Ashley Stirrup (24:38)
Yeah.
Yeah,
yeah, I think those all are great suggestions. And one thing that I was thinking as I was hearing you is that, yeah, you know, find the teams that are the most open minded and then just grow, try to grow visibility. I would imagine also, you know, showing people.
surprising results. Like I think a lot of people will be surprised that their win rate's only 10 or 20 percent. And so when you start to do that, people start to realize, ⁓ my intuition isn't as strong as I thought it was. I really need data here.
Medha Umarji (25:08)
Right.
Exactly. think that, you know, one of the most common things I say is like, your odds of winning at like roulette or poker are probably higher than your odds at winning at experimentation, you know, and like it's, it's literally like gambling.
Ashley Stirrup (25:31)
Yeah, that's pretty funny. Yeah, yeah, yeah.
I'm gonna use that one. That's a really good one. Yeah. It definitely changes your mind about how you think about it if you think about that. Like, ⁓ I've got better luck at Blackjack than I do at experimentation.
Medha Umarji (25:50)
And if you like a challenge, you know, then you like, and that's all more fun, right? Like you.
Ashley Stirrup (25:54)
Yeah, yeah.
But I mean, but to put that into business terms, if you're running 10 experiments and one or two are winners and a bunch of flat and then one or two are losers, if you can avoid the one or two losers, if you can turn one of the flats into a winner, I mean, you've probably doubled your overall revenue growth from that. So.
Medha Umarji (26:12)
Yes.
Absolutely. And I think that one of the things that's understated in experimentation is just the risk ⁓ avoidance aspects of experimentation. That is, honestly, a big part of my job is trying to just cut down on any risks that we are taking. Even if it's not a statsig, if you think, it's a do no harm, we could roll it out. But is it really a do no harm? Are we risking the experience in certain ways? I think that is an unquantifiable.
quantified thing for us right now. yeah, that's kind of the next frontier is trying to quantify the risk that we are actually helping the business avoid.
Ashley Stirrup (26:57)
Yeah, well,
I've certainly talked to lots of people where experimentation saved them millions of dollars by quickly identifying something that otherwise was going to cost them a lot. And many times those are silent killers because your overall business is growing. You're getting enough wins that you're getting growth, but you don't realize what your growth could be if you were eliminating the losers. Yeah.
Medha Umarji (27:03)
Right.
Exactly.
Ashley Stirrup (27:22)
Well, thank you so much, Medha. I really enjoyed having you on the show today and ⁓ definitely encourage everybody to ⁓ follow Medha on LinkedIn. She promises to start posting some more and we'll be highlighting this episode there as well. So thank you very much.
Medha Umarji (27:39)
Okay, awesome. Thanks Ashley. Great chatting today.