Subscribe
Share
Share
Embed
Mary Alfheim, VP Data and Analytics at Headspace, joins host Ashley Stirrup to explain why more engagement is not always better when your members come to you with real symptoms. She walks through how Headspace experiments on Ebb, its AI companion, by defining inputs, outputs and guardrails for every test, with clinicians in the loop and stop rules that let teams self-serve safely.
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
The Experimentation Edge - Mary Alfheim
Recorded 2026-09-24
Ashley Stirrup: Hello and welcome to today's episode. I'm excited to have Mary Alfheim, Vice President Data and Analytics at Headspace. Welcome to the show, Mary.
Mary Alfheim: Hi Ashley, thanks so much for having me.
Ashley Stirrup: Yeah, excited to have you on. We've got a lot to cover today, so this is gonna be fun. Why don't we start with a little bit of a background on Headspace?
Mary Alfheim: Sure, so Headspace, great brand awareness, mental health and well-being company, but probably best known for its meditation and its mindfulness app. So we offer guided meditations, sleep content, wake-up content, mindfulness exercises, and stress management tools. But we've also expanded into broader mental health services. So that includes like clinical care, coaching, therapy, psychiatry. We have B2B businesses where we partner with employers and health plans, but also a really strong D2C brand where you can subscribe and use our app directly too.
Ashley Stirrup: Well, it sounds like it must be a really rewarding business to be in and I know that I spend a lot of time thinking about gosh, I need to get more sleep. So I think that's pretty important these days.
Mary Alfheim: You should check it out. I think I have some subscriptions I can share too.
Ashley Stirrup: I love that. We definitely have to talk about that afterwards. Could you tell us a little bit maybe about your career and your role at Headspace?
Mary Alfheim: Absolutely. So, like you said, I'm VP of data and analytics at Headspace, but I am building off of over 20 years at this point in marketing and product analytics and data science, mostly focused on consumer subscription businesses. So I've been at Headspace since the beginning of the year. Before this, I had a similar role leading all of data and analytics at Scribd, where we have UGC content as well as ebooks and audiobooks. Before that, led product analytics at Prime Video at Amazon with a similar role at HBO, which was super fun, was part of the team that got to launch HBO Max way back when in its first branding. And then before that, I spent the first half of my career in financial services. And I worked at TIAA, TIAA-CREF back then, and that's really where I learned the most about data.
Ashley Stirrup: Wow, that is amazing. So I would imagine with a business like Headspace, it's, you know, a little bit of a different business in terms of what it means to do optimization versus, you know, say Prime Video.
Mary Alfheim: That's a great question. Yes, and I think that comes down to the different ways that the company is built. So I mentioned we have like a B2B business and we have a D2C business. So when I think about optimization, there's two different kinds of customers that we're looking for, first of all. And then secondly, which will this will come up when we're talking about actually running experiments, we are not just trying to drive engagement at all costs. So at a place like HBO or Prime Video, yeah, like, you know, putting costs aside, more engagement is generally better. You want people to watch the next show, read the next book, watch the next movie. You can look at things like how many minutes someone's spending with your product, but that's not the case for Headspace because more is not always better. We want people to get better if they are coming to us with actual symptoms. And so the optimization is health outcomes, not just more and more time in the app, right? So now we have this dual mandate that we're really trying to drive.
Ashley Stirrup: Yeah. And how do you measure outcomes?
Mary Alfheim: Great question. There are lots of different ways to measure outcomes. Some are at a population health level, some are, you know, using third-party data sets or match data sets. But one of the ways that apps like ours really try to hone in at the individual level is by self-reported data and taking snapshots of a user's self-reported, you know, scales of how they feel in terms of anxiety or depression, for example, and take those snapshots over time and be able to kind of correlate that with any interventions, treatments, or content that they've engaged with over that time.
Ashley Stirrup: Yeah, that makes a lot of sense. So I'd imagine you're, you know, for a typical experiment, you're looking at both kind of traditional metrics as well as these kind of more outcome based.
Mary Alfheim: Exactly. So like I said, like we can go into so much more detail, but what's so critical about the way we experiment here is we don't just look at some primary metric, like if we're moving trial conversion or retention or something like minutes on, let's say, sleep content. We really do have to make sure that we're respecting these guardrails of those outcomes and making sure that we can measure that in our experiments. Because again, more content is not always better. It's not the end-all be-all, it's not the only metric.
Ashley Stirrup: And I would imagine some of what you're doing is looking at kind of things that you can measure short term and then trying to predict longer term outcomes. Is that something you've had to work on?
Mary Alfheim: Absolutely. So we have a couple of challenges or opportunities with the way that we test. So if you're not dealing at the scale of something like Amazon, right, where we have more than 200 million people on Prime, you don't just like fill up your sample immediately. And so we have kind of a sample size or a sample time issue. If you're looking to directly measure annual resubscriptions, for example, that could take a long time to really get a read and decide what to do. And at the same time, like I mentioned, we really want to measure not just are people likely to resubscribe or retain, but are we moving the needle on their actual self-reported health? We don't always have the sample for that either. And so what we have to do is kind of work backwards then with our science teams and with our clinical researchers to like put much more science behind it to say, well, when we do look over time, the kinds of people who interact with this kinds of content or interact at this level of frequency or breadth, their outcomes are associated with this level of improvement or, you know, this level of, in some cases, lowering of symptoms, you know, depending on which direction we're looking at. And then we have to make inferences, right? We don't relitigate it with every test, but we do have to sometimes think: how do we put the science behind the relationship here, behind these different variables, so we can make really good conclusions without having to wait years for everything to start to populate.
Ashley Stirrup: Yeah. So it sounds like some of this is kind of science based. Is some of it also like are you building, you know, predictive models that okay, if I see them doing X, that probably means they will start to improve over time, that kind of thing?
Mary Alfheim: So we're working in two different realms. So like I said, Headspace has clinical care. We have these amazing clinical researchers and so and actuarial sciences, we have lots of folks who are looking at science over time in these kind of cross-sectional views to really put that rigor into place. Yes, absolutely. And that's kind of the backbone for where we are today. But part of what my team is doing within the data science organization is starting to build more of those kind of composite or surrogate metrics that we can use more of on a day-to-day basis within our tool of choice and allow product managers or anyone who's running the test to start to self-serve.
Ashley Stirrup: Yeah. And let's say, you know, you were talking to a friend at a business that was kind of more traditional, I don't know, let's say Airbnb. And you know, because a lot of people try to predict like, okay, what's my lifetime value of a customer or long term retention? And great, they, you know, in the Airbnb case, right, they stayed at this particular location. Was it a great experience or not? And how do you, you know, kind of look at the short term metrics to predict the long term? Is there any kind of advice that you would give people in terms of things they should go research to help them build better models to predict the long term metrics from the short term?
Mary Alfheim: Hmm, that's a really great question. I think I'm gonna draw on some of what we did at Amazon there. Amazon has fleets of economists that help design these kinds of composite metrics. And I think the first thing that I would encourage anyone to do is to think broadly and think about your business. So the composite metric, you should test a few. Let me put it that way. You may have a really strong hypothesis for what you think should be a predictive measure, but go out, back test it, you know, see what that could tell you from previous experiments. But put in the actual gold standard RCT, randomized control trial, validations into place. Don't just rely on those observational models because you may very well get counterintuitive results, results that may not hold for every population that you're testing on, or results that don't carry over time. So you have to kind of expand your thinking and think of even of these metrics as not just a one and done, but a constant curation. Even if you're thinking about all of the science behind building these out, your business could still change over time. Your customers, their tastes and preferences, their behaviors, your competition could change such that the relationship could continue to change. And so I think it's important to just continue to stay, stay curious, collect a lot of data, and design thoughtful experiments where you can have more controls or a long-term kind of holdout to truly measure the causal impact.
Ashley Stirrup: Yeah, yeah, that makes a lot of sense. And I think part of what I'm hearing is do the data science, do the modeling, but, you know, get a variety of different metrics and use a bit of common sense along with that.
Mary Alfheim: And validate it. If I go back to a lot of like marketing testing, for a long time there was this conversation around, you know, multi-touch attribution models. And you know, they give you just-in-time information, and that's how you can optimize it down to a campaign or asset level. You need that information. But they're biased and it doesn't really work with social and there are all these holes in it. So we have to go back to media mix models or randomized control at some sort of DMA level, gives you an actual causal impact, but it's only one point in time and it doesn't give you the detail you need. I think over time, people have started to combine both of these methods, right? Like use your observational models, use your predictive models, make those decisions, sure, for day-to-day optimizations, but continue to validate them with those point-in-time RCTs and make sure that your assumptions and what you're seeing in the model, it stands up, or understand the magnitude that you have to start to tune or adjust those results by.
Ashley Stirrup: Yeah, that makes a lot of sense. Can you talk a little bit about the business impact experimentation has had on for Headspace?
Mary Alfheim: I sure can. Experimentation at Headspace has been absolutely fundamental to our ability to drive engagement with our app and with our conversational agent, Ebb. That's our AI companion. It's our empathetic conversational chatbot. It's built into our app, helps members reflect on, process their emotions, other conversations. But experimentation, and especially in this part of our experience, again, it's been fundamental to driving engagement, but with safety. With safety. People can jailbreak chatbots. They can use them for things that they were not intended for. They could be having moments of crisis that don't call for AI, that you absolutely need humans in the loop. This is kind of fundamental and existential to what we do at Headspace. Like care comes first. We are not going to just keep experimenting and trying to drive engagement and shift people only towards this one, you know, this chat, this companion in our app when what they might really need is human care. So I would say this has been a great example of how we have started to organize ourselves around what I call like inputs, outputs, and then guardrails, which we were starting to talk about. So we weren't doing a ton of experimentation about a year ago. And that's okay sometimes. Sometimes things are one-way doors, a huge redesign, or something that you're not gonna change or go back or iterate or learn from. But this is a place where we did wanna keep evolving, evolving safety standards, evolving the rules and the prompts behind it, in some cases, even the infrastructure that was running it. So it was a great example of a place where we did want to experiment and make sure that we were not harming anyone, first and foremost. We want people to use it, but not for things that it's not meant to be used for. So this is an example where we've really defined, well, what's an input, what's an output, and what's a guardrail. So inputs, we can directly impact, measure what it is that we may be changing. And we have this concept of that. Are we changing the prompts? Is it a new safety rule? Is it again like the infrastructure, the architecture that it's running on? And then we can look at some of the outputs. Very different. Are we having more sessions, longer sessions, more messages sent per session? Can even collect like user feedback like NPS. That's really what will help us drive success. But through this process, and because of what we were testing on, that's when we defined it, that's when we defined those guardrails. So we also make sure that we are not leaving humans who need acute care in the lurch. So human clinicians have been in the loop on all these experiments and doing clinical QA so that we can monitor against these guardrails, monitor for interactions that really do require more action or a different level of care. And so going through this really forced us to define what those different metrics were and where there was like stop rules in place, which I think is so extensible to really any kind of experiment. So another like a very simple example could be I want to drive engagement, but I don't want to do it at the expense of like cost. I don't want to drive cost up, let's say. Right. You can say, fine, help yourself, self-serve this experiment. If you get positive lift on engagement, and there's no harm to your cost guardrail, feel free to roll it out. You're ready to go. If you have harm to your guardrail, then there can be clear rules put into place to roll it back or go to some other sort of review mechanism.
Ashley Stirrup: Yeah. We have a customer, Khan Academy, and they did some really interesting work around data labeling. So they looked at all the questions their AI-powered tutor was getting and labeled them based on how engaged they thought the student was. So, you know, more engaged, they're asking questions where they're really trying to learn the subject, and less engaged, they're trying to just get the chatbot to give them the answer so they could move through the quiz. And I was just thinking about that relative to your business and, you know, boy, there's kind of a whole nother level here, right? Like, when should we intervene and route them to a person or, you know, is the conversation going in a direction it shouldn't or something like that?
Mary Alfheim: Exactly. 'Cause these are real humans and that's what comes first and people, you know, we're all learning how to use AI and talk to AI and some of us maybe trying to use it for clinical care, for advice about medications, things that it's not designed to do. Right? That is where human care is so critical. And so we're trying to make sure that we can provide care or provide conversational ability to people who may, again, maybe they want to reflect on something that's happened, trying to think about how better to use sleep content, think about what their emotions are. That's again very different from someone who may be in crisis, trying to take very literal advice from an agent like this. It's not where we want people to be.
Ashley Stirrup: Yeah, it's such a hot topic in general, around A/B testing LLM-powered features, 'cause, you know, you might roll it out initially and it might perform great and then you might get drift over time. So there's kind of a longer term element to it. But it's also kind of a gold standard for measuring the impact, you know, like is this new model better, really better than the other model? Is this new prompt actually performing better? And that, you know, evals can kind of be QA for is it generating an answer that seems good? But you can't tell without an A/B test, is it really leading to the change in behavior expected from humans or not.
Mary Alfheim: Exactly. And there can be surprises when you actually roll these things out, as you know, and that's kind of the point, to learn. I think tests are they're most powerful when you really are looking not just to validate an opinion you already had, but really trying to learn something new and look around those corners. Like what is that new behavior that could be happening here?
Ashley Stirrup: Yeah, the other interesting thing is that, you know, there's lots of LLM-powered examples that you can measure pretty easily initially. For example, Zapier, and they have an AI widget that helps guide you in creating a workflow. And, you know, did the workflow get completed? Did it get published or not? Typeform's another example, you know, creating surveys. Those are very easy to measure. And then there are things that are much harder to measure. In the case of Khan Academy, they had to kind of create a new type of data. They had to go through a whole data labeling project. For Headspace, were they more in the, you know, were there already pre-existing metrics that you could measure or did you have to create some of your own metrics?
Mary Alfheim: I'd say we're still creating quite a few of those. Like you, again, you can think about standard kinds of engagement. How many people get into an onboarding flow, how many messages do they send? How long could a session last up until we know where that may be capped? So those are all routine kinds of metrics that we've defined and that we can experiment against. But I would say as the product evolves, the metrics need to evolve with it. That's, you know, that's why it's also so important that a data team doesn't design inputs and outputs and guardrails in a silo or by themselves, or try to just impose that on any team that is running their experiments. A product team and the people who own what you're testing on, the service or the feature, they know what the intention is, they know what the roadmap is. They are deeply empathetic to their members, their customers, their users. It should be a conversation. It should be a conversation to think through what is it I'm truly trying to drive. Can I measure that? And if I can't, how do I start to collect the data to be able to do that?
Ashley Stirrup: Yeah, it's so interesting to me. When I first started in experimentation, I wouldn't have guessed that experimentation could be such a good opportunity to bring teams together and create a common set of decision criteria, like, okay, we're trying to optimize these metrics, but only if these other metrics still perform well, or, you know, things like that. And so it's a great forum for everybody to kind of come to the table, build some trust, and get a common sense of what good looks like.
Mary Alfheim: I totally agree. I totally agree. And you can extend it over time too. So in my last role, one of the mechanisms that we developed was not just a forum where data and engineering and product or marketing, you know, could come together to think through, well, what's the experiment design? How do I interpret these results? Let's make a call. We also had these office hours, like we did it in a public forum and then also captured all of these notes in like a knowledge repository so that others could learn from it over time. We could start to have examples of like what were well-designed tests versus poorly designed tests, because we don't get it right all the time. Like not just a hypothesis that doesn't pan out, but sometimes you screw the test up, or you learn something about how things are supposed to work. Or maybe, you know, two years ago there was a ton of testing on an onboarding flow, and you can learn something about what worked or not for your population. It gives you ideas of the future or gives you ranges that you could expect for outcomes. But it helps, I think, to kind of raise the bar in an organization to inspire more tests and really kind of model what good can look like and encourage the conversation and the creativity.
Ashley Stirrup: Yeah. Yeah. That's definitely one thing I've learned from doing the podcast. It's just how important what I call building a culture of experimentation is. But, you know, it's really like how do you, you know, do experiments with rigor? How do you share results and let other teams benefit from the learnings of, you know, the first team? As well as just creating a mindset of how do we test this and let's test everything and, you know, things like that. So super interesting.
Mary Alfheim: Or if you don't test everything, what do we feel as an organization should be tested and what makes sense to maybe move forward with?
Ashley Stirrup: Yeah, yeah, absolutely. Absolutely. Yeah. You're never gonna be testing everything. But at least having the mindset of can I test this and is there something I can learn as I roll this feature out so that I can understand my customer base and their user experience even better as I introduce new features. So
Mary Alfheim: It's part of what's so fun about data science. I think it's one of the most fun parts about what a data team actually does. Because sometimes we're surprised by results. It's also an opportunity to be a cheerleader. So sometimes tests, putting aside, maybe you set the test up wrong, sometimes they don't show results. And sometimes that can be really disappointing. Really, really disappointing. But I always try to still celebrate that as a learning. If you ran a test and did not get the result or the impact that you expected, you at least now you have more information now to make your next move that you would not have had otherwise if you had just assumed something would work or it would work at a level, you know, that you expected. It can be easy for people to think, well, it was a failure. It didn't move the needle. Hey, I mean, fair enough. You gotta get back to the drawing board. Like we're trying to run businesses here. But because of the test, now you know. And you are able to iterate and move on.
Ashley Stirrup: I think that's such an important point. You know, like sometimes there are metrics that are just really hard to move. And so step one is learn this metric is really hard to move. And then step two is you start to explore, well, what ways can I move it? I had a guest on who talked about not just build a test but build an opposite version of it too. And so like if you're simplifying a screen, actually make it more text-heavy as well and run both, and maybe the less text-heavy one doesn't move the needle and maybe the more text-heavy moves it in the negative way, but at least it shows you that you could move the metric in any direction. And then that helps guide you in terms of how you think about the next. To me, I love that as an example of where you're getting information from experimentation.
Mary Alfheim: You know, that's a great example of building an experience and trying to make that detectable effect big enough to see if you move the needle. The other thing I love about what you brought up is it can also, like, experiments like that can also set off a feedback loop within your data science organization to say, hey, I don't have the right predictive metrics in place, or my metric isn't variable or like sensitive enough to tell me what's going on. And so that's actually a very interesting piece of information for a science team to say, hey, even if I think that, hey, minutes or, you know, minutes across these different content types, let's say that should be the right metric. Hey, it's just not moving or it's not moving enough. How can I build more composite metrics that maybe take more signals into account into some sort of weighted version that could show more sensitivity, right? So that I can make a call. And that can be very exciting to a data scientist, like to go back and say, like, we have to keep reinventing this. We want to get the science right, but we also understand that like the business has to make a decision. And the longer we go, the longer these tests run, we're missing out on rolling out the right experience, right? We're missing out on what's right for the customer and what's right for the business. We have to get better too. We have to get smarter.
Ashley Stirrup: Yeah. I love that. That is such a great example. Thank you for sharing that. Is there an experiment you can share with us where you had a lot of learning?
Mary Alfheim: You know, I have so many experiments to talk about with lots of learning. I'd kind of like to tell you about a few of them, if we have some time. So the one that comes to mind, it's not from Headspace, it's from a previous role, media subscription business. Our product team had spent more than a year, about maybe a year and a half, on developing this entirely new UI. Basically a whole new way to navigate, to find your content, how users were gonna find something to watch. We had aligned on the measurement plan, what metrics we were going to look at, when we would read results, what success looked like. All this was aligned ahead of time. We even planned to monitor for a long time over this test because we expected some of our existing users, probably, they don't like change. It's very possible that you could see a dip in engagement and then a rebound over time as people habituated to it and that they adapt. And so we thought we had all of our ducks in a row here, but it didn't go exactly the way that we planned. We did not see the results that we would expect. And it was hard to disambiguate some of the results that we had. So let me tell you why and like what the takeaway was. We tested, I think, too much. And we kind of waited for this very large, what ended up being a combination of many changes, not just like one feature change. We waited too long to test. So instead of testing early and iterating on every piece, we waited for it all to be done. And what that meant was by the time we got there, the business objectives had actually changed a bit. First of all, so our measures of success or what a successful test looked like wasn't necessarily aligned to where the business wanted to go in the next 18 months. So that had to create a whole new feedback loop for so many different experiments, conversations. But then also when the results weren't showing exactly what we thought they would, it was really hard to figure out what part of the experience wasn't performing because we packed so much into it. So we may say it's a UI redesign, but it's actually a combination of many different components. So which one was it? And so we kind of had to go back and start to test things over and over again in that iterative manner to get to a new place. So I always say faster is better, more discrete changes is better, and don't wait for some big grand reveal. Truly like test as you go. So that was a huge learning for me early, like relatively early in my career.
Ashley Stirrup: That's a great example, yeah. So important.
Mary Alfheim: There's another one. Another hard-earned lesson or some battle scars. My team was partnering on a test in the payment space. So I've talked a lot about like subscription services. So, you know, subscription services partner with Google and Apple stores and then payment processors like, you know, Zuora, Braintree, Stripe, et cetera, to actually like collect money, figure out who can sign up, who can convert from a free trial, you know, who's gonna bill through from an annual or monthly plan, whatever it is that you have. But these are complicated systems. When a payment processor is trying to balance fraud detection, or whose card is likely to go through, who's likely to charge back, what fees could be associated with where you're routing traffic in different parts of the world. They're balancing all of these different considerations that to you, if you are just kind of testing one payment processor for another, may seem really similar, like maybe something you don't have to dive in as much. So I was in an organization where we wanted to test two different processors and see, you know, what could that do to the business? And our guardrails were around things like our chargebacks, frauds, et cetera. And what we learned here is because we didn't know as much about the payment processors themselves and how they worked, we didn't get our randomization right. So let me explain that. We thought we were A/B testing, send half of our traffic to one, half of our traffic to another. Seems straightforward. But when we started to see results and saw like our challenger really underperforming, you know, we wanted to roll that back, and this was a huge surprise, what could go on? Once we really started to peel back all of the different layers, we realized, well, the challenger payment processor was actually not really random. They didn't have as much information about their traffic as the control did. Because you can't provide the history of any payments that have gone through before. So really, like, it was a painful learning. We had to, like, we didn't understand exactly how the mechanism of the treatment worked as well as we should. So my takeaway from that is you have really got to know what your treatment is and question and question again if you're getting your randomization right. Because we wasted a lot of time rolling that back and having to then stratify our sample to get like what is actually a well-randomized read.
Ashley Stirrup: Interesting. And just to make sure I understand, so the control was a payment processor you've been working with for a long time. And so they had a lot more data on your customer base. And so they were better at predicting fraud and, you know, high-risk transactions versus the new provider that didn't have all that history. And so it just wasn't able to perform as well in the short term. Is that right?
Mary Alfheim: Exactly. So it needed more time to kind of learn the audience. And so it was kind of like behind the eight ball relative to the control. But as a data team, I don't think we fully understood that when we helped figure out like this randomization. It seemed straightforward. Send half the traffic here, half the traffic there. But we didn't get it right because we didn't fully understand, we didn't really understand the treatment and this technology behind it as well as we should. And it's only because the results didn't quite make sense and our partners, like in engineering and in product, they were our partners and really questioning this and asking the next set of questions, like, well, does this hold for this kind of cohort or this kind of tenure? You know, it was that partnership that helped us even uncover this. And so always include your partners. Don't just tell, you know, this is not just for your data team. You have to know what that treatment really is and what that experience is, especially for some of those more complicated tests.
Ashley Stirrup: Yeah. Boy, such a great example of, you know, the importance of kind of rigor and trust and communication and the ability to dive into the data after the fact. And
Mary Alfheim: Yeah. And knowing your customer experience. I don't think that at when we had just started that, we did not appreciate how much already went in to so many of these decisions with like what that customer journey truly was by the time that they got even to the point where we would be forking the traffic.
Ashley Stirrup: Yeah. And I would imagine these kinds of learnings apply very well to any kind of AI-powered feature, right? 'Cause in some ways none of us knows how these things totally work. And so it requires that kind of collaborative, let's all take a look at the results from multiple perspectives to truly see is this getting the result we want and then is it continuing to get the result we want over time.
Mary Alfheim: Exactly. So one of the learnings that we took from this too is we started, instead of just this being like an experiment where we would get some sort of magnitude of an effect that we could expect for the business, we turned it into a portfolio of experiments that we would have to run. And so we then started to design, well, now that we know that this could look different for different cohorts, we stratified different samples and we ran different experiments for new customers versus people who may be resubscribing, right? Who, people who had history versus no history, so we really could randomize. And we also wanted to make sure that we ran it at different points in time and collected repeated results to see how well it held over time and if the magnitude of difference really did hold or one could kind of catch up to another.
Ashley Stirrup: Got it. And as you kind of look forward for Headspace, how do you see experimentation evolving?
Mary Alfheim: It's so exciting. First, I think it's going to be faster and I think that we'll probably have bolder treatments and experiences for our members that we want to be testing. We did not do a ton of testing, like I mentioned, you know, a year ago. But one of the things I appreciate so much about the people I get to work with, and product in particular, such science-forward, measurement-forward partners, they know the value of testing inside and out. I don't have to get them to the table. You know, they're trying to figure out how to go even faster. Like, absolutely. Incredible, incredible scientists as partners there. So I think with that level of partnership, they are eager to test. They are eager to use, you know, AI on both ends of the loop. We can use it to help be more productive, build new experiences to test, but then also use that for faster interpretation of results. So I'm surely just looking forward to continuing to scale this up. At this point, you know, we're at a couple of dozen tests over a quarter. I like to look for almost like that plateau of effect size over time to figure out are we running the right amount of tests? You know, are we experimenting and iterating with our experience as much as we should? And I think we're on the ascent of that plateau right now.
Ashley Stirrup: Yeah. Yeah. Yeah, I think it's so interesting 'cause like obviously one of the most important things that you want out of an experimentation program is learning and how do you scale up the number of experiments you're doing, but make sure you're doing it in a way where you're actually scaling up your learning at the same time.
Mary Alfheim: Exactly. It has to be applicable. Don't test just so you can say you had a win and put it in your quarterly business review or a deck and say I drove this level of impact. Make it worth it.
Ashley Stirrup: Yeah. Yeah, yeah. I mean it's fine to scale do-no-harm testing and kinda look at that as a risk mitigation strategy and then have another portfolio of these are the experiments we're gonna run where we really are trying to learn about our users and the user journey and things like that. So
Mary Alfheim: Yeah, quick, quick story on that. Yeah, I think you'll hear a lot of practitioners talk about a portfolio of experiments. Have a bunch of small loop tests always on and a couple of big loop tests, like really big, big shots that could really move your business. Yes. But I would say that there's a third bucket. There's just a learning bucket. Take the pressure off of your team. Don't even expect this result to have to show up on your total of what you drove for the quarter, or it doesn't necessarily have to be something that you speak to in, again, like some sort of like business review. There should be some level of experimentation where you truly are just trying to learn. Maybe you don't even have expectations, but I think that is what can drive that curiosity or that next level of innovation or drive the bigger loop tests overall.
Ashley Stirrup: Yeah. Yeah, you're kinda hinting at something I'm really passionate about, which is that a lot of the best experimentation programs that I've talked to, they have an executive who is leading by demonstrating humility, demonstrating that they know what they don't know, and so that they're showing curiosity as they run experiments and like what can we learn from this? And they're asking those questions. They're not expecting a high win rate, but they're trying to get as much learning as possible from that and they're leading by example. And I think that's so powerful. So
Mary Alfheim: Very powerful, and not even expecting what a win rate should be.
Ashley Stirrup: Yeah. Yeah, well I mean, you look at a lot of the, you know, biggest, most successful companies in the world and their win rates are in the 10 to 20% range. So yeah.
Mary Alfheim: Yeah, great. That means they hit that plateau, right? You hit, you got your low-hanging fruit. You did the big things that like you knew you needed to do for that customer, user, member experience. And now it's harder. It's not so easy. It's not so straightforward. So you do have to be curious and creative and deeply empathetic to your customer.
Ashley Stirrup: Yeah. And still with a 10 or 20% win rate, you can still have a massive impact on the business by stacking wins over time. So yeah.
Mary Alfheim: You sure can. You sure can.
Ashley Stirrup: Well, Mary, thank you so much. This has been a fascinating episode. I particularly loved what you were saying about the data science team learning about new ways of, you know, kind of looking at different metrics and new ways of measuring things to help the business learn faster. So that was really good.
Mary Alfheim: Absolutely, the feedback loop can go right into science, not just product or marketing. I appreciate it. Thank you so much, Ashley.
Ashley Stirrup: That's right. So thank you so much.