Subscribe
Share
Share
Embed
A SaaS product design podcast for non-designers. The Love At First Try Podcast explores how SaaS products become unforgettable.
We unpack the idea of taste in product and brand design, deconstruct what makes beautiful products beautiful, and show how to merge growth with delight.
If you’re a SaaS founder, CEO, or developer building products people love, this is for you.
Jim Zarkadas (00:00)
Hey, I'm Jim, and this is the Love at First Try podcast, a podcast for SaaS CEOs and developers that truly want to learn more about design and care about it, but there are no designers that find it too complex. In every episode, we discuss how to design products that become sticky and unforgettable. We dive into the topics of taste, UX, growth, and conversions, and we share practical tips and frameworks you can add into your development process. Enough with the intro, so let's dive into today's episode.
Jim Zarkadas (00:25)
Yeah, welcome to the podcast, officially as well. Thanks for joining me today. And we always start, the first question that I always start with the episodes is who you are, like for somebody that never heard about you, who you are, what you're doing and what's your story essentially. And also the last part is what are you focused on right now and who are you helping at the moment?
Lucia Van Den Brink (00:47)
I hear my dog barking like crazy. What should we do? Okay. I'm Lucia van den Brink. I'm founder of The Initial and we install experimentation programs for in-house brands. And recently we started focusing on brands that do more than a hundred million a year. really, usually they're like international, multinationals ⁓ and really interesting companies for us.
Jim Zarkadas (00:49)
It's all good. Yeah, yeah, it's all good. ⁓
Lucia Van Den Brink (01:14)
But we really believe that those kind of companies, you know, they shouldn't rent experimentation. Like they shouldn't just have that as an external project, but they should really bring that in-house and make it, you know, part of their own competitive advantage. So that's what I do. People might know me from LinkedIn. ⁓ Usually that's what people say when they...
meet me they say like I know you're from LinkedIn I also run a community called women in experimentation and what yeah and how else can people know me yeah I guess just from being in a space and giving keynotes yeah
Jim Zarkadas (01:43)
⁓ I had no idea.
Yeah, yeah, yeah, yeah. So yeah, right now you said you're focused on the 100 million brands. Okay, that's interesting. the first question that I was thinking of kicking the podcast with on this part is our audience is...
product teams. So you could say it's like CEOs, developers, but also some designers as well. And it's mainly from companies making around one million of annual recurring revenue. This is kind of the core audience we're focused on. And let's say I'm a SaaS founder. I build the SaaS. ⁓
Usually, let's say I'm bootstrap, I build a SaaS, we're making one million per year and I have no experience with A-B testing. I know what A-B testing is, maybe I did some testing in the past like just for fun, but never really approached it in a systematic, ⁓ systematical, senior way. So...
Let's assume I'm that person, what would you tell me if we had a consulting session, let's say, what are some fundamental things that you would share in a first call with me, and the top mistakes you see people like me doing? So it's a pretty open-ended question because I'm really curious to see what are the top of mind things that come to you.
Lucia Van Den Brink (02:59)
Mm-hmm.
Okay,
so let's first translate that into like I work with these big brands, right? And what they do is they really systemize their decision making process. ⁓ and that has a lot to do with design as well. So, you know, should it be this color or that color? Should UX be a radio button or just text that you can click, you know? All those decisions, especially in bigger companies, if you get them wrong, it costs you millions.
Jim Zarkadas (03:09)
Mm-hmm.
Mm.
Lucia Van Den Brink (03:33)
is why all those bigger companies are testing with that. If you're doing like around one to five million, then you know maybe it's not so much about preventing... ⁓
Jim Zarkadas (03:33)
Mm.
Lucia Van Den Brink (03:45)
risk but maybe it's more about finding the levers for growth and usually you know the smaller the company the more low-hanging fruit there is especially if they have never experimented yet so what experimentation basically is is you make data-driven decisions right so you actually start from looking at
We do that, but ideally if you bring it in-house, would do the same. You start looking at insights and data to learn like, what is important to our customers? What kind of language do they use? What are they actually looking for? And why do they like us more than our competitors if they become clients of ours? ⁓ And then with that information, you start to come up with hypotheses like, hey, if our visitors...
are reading a lot of reviews, for example, before they go on to take a subscription with us. Maybe we should play that up bigger, right? That can be a very small example. So you kind of try to translate behavior you see into changes you can make on your website. And that's not just for...
Well, that's essentially for everyone. It's for designers, it's for founders, it's for products, teams, if there are any. All of those different roles and different types of people and different types of responsibilities can actually, even technical developers can benefit from taking on this scientific process to find out what really works and what moves the needle.
Jim Zarkadas (05:19)
⁓ Yeah, yeah, I see your point and like the difference between a other medium brand and a one medium brand is like that experimentation can have a different impact there and help you in different things. ⁓ Yeah, one question of these that I have is what spreads companies that build a real experimentation culture compared to companies that just do random tests here and there? Like how...
Yeah, how does for you because your whole POV, your whole point of view is that you need to build the experimentation as part of your culture. It's not something that you hire an agency like, hey guys, please do experimentation for us. You need to make your people think in this way. So how does the culture and the mindset look like for a company that is serious about experimentation? And from what I hear from you.
This assumption that many people have, including me by the way, this podcast episode is also a chance for me to educate myself more around experimentation. The typical excuse like, we have like low amount of visits, we just have like a few thousand visits, we're not like a big brand, we don't have the volume to do any testing. And I strongly feel that this is wrong in the end. It's a mindset problem there. So I'm curious like how more like, yeah, how real experimentation culture looks like actually, even in smaller companies.
Lucia Van Den Brink (06:36)
So I also had an interesting conversation with someone from booking.com that had worked there and he said it's essentially about scaling leadership. ⁓ If people like to listen to that, it's on my podcast. But he said, his name is Marta and he said that essentially, know, whenever you're just a small company, you can work with the intuition of the founders to grow because they know their audience and they know what's right. But as you start to hire
Jim Zarkadas (06:50)
Thank you.
.
Lucia Van Den Brink (07:05)
more people that might be not so much attuned to your visitors and all that stuff ⁓ and you also don't want to keep on relying on the original founders.
you want to use the process of experimentation to find out what actually works and what doesn't and how you can make the right decision. So what happens is you're starting to miss out on that intuition of the founders ⁓ because the company grows, but then you still want to find a way to do the right thing, to make the right decisions. And that's not to say that founders always make the right decisions, but somehow in those beginning stages, they often...
very quickly see, if we change this, suddenly we have more clients. But once you scale, you want to do that in a more systematic and scalable manner for also your teams to take on the role of making the decisions and making the right decisions. Because in the end, ⁓
Experimentation is seen as a way to make money, which is fine and it can definitely be. But what is even more important is that it becomes a strategic lever into understanding and learning what works for your company and what doesn't.
Jim Zarkadas (08:21)
Yeah, I love thinking here and one follow up question that I have on this is when we see experimentation, so one thing that I'm realizing is that what an experiment means for everybody can be different. Like if you ask me, things that I think about A-B testing is we have this idea of adding... ⁓
One more question in the onboarding flow, let's say, let's test if it's gonna create a big trouble for not. So that'll be an experiment, but it's always about let's add this element and test if it's a problem or not. But I feel maybe I'm wrong, but I'm curious to see like that experimentation can be more than that actually. So like what about launching a new feature? So the question here is,
what is an experiment for you and what are the types of experiments you've seen? So it's a bit more kind of a yeah the actual experiment could be landing page tests so especially when comes to products and not just the websites what are some type of experiments you've seen there?
Lucia Van Den Brink (09:15)
That's a good observation Jim because that is definitely true and you know we can question the term experiment a lot because it doesn't even necessarily need to be an A-B test. Like you can do much more types of research so that's one thing and you can also learn from that research if it's a good decision to make to proceed with. A-B testing is quite high in the level of evidence that you get and ⁓ the level of trustworthiness so that's why
especially the bigger brands kind of prefer that because you just have more certainty to say that hey this is actually better for us and we even know kind of the percentage that it will bring to us but like you say you can use it to do run small tests like you suggest like a forum extra yes or no but it can also be a way to validate a new feature and while you might have done a lot of research to create that feature
So usually if there is really a real big feature being introduced, there is usually also user interviews involved in creating that before spending a lot of dev resources. But then once you have...
The thing that comes to mind is not necessarily B2B SaaS, but was a news website that I actually was an in-house experimentation lead and that's also where my love for in-house comes from. But they knew that it was important for visitors to have just the latest news in their timeline. But the website didn't automatically refresh or show new articles. So that would be a really big feature we had to build. we beforehand tried to
We did research and we found people said this is the most important thing, this is our job to be done and we knew actually our website is not really ⁓ offering that because it doesn't auto refresh, like you cannot look to the page and you see something new popping up so that was the feature we wanted to build but then in that process we had to make a lot of decisions also based on design, how to interact with the elements we would add because at the beginning it was just like a small...
Jim Zarkadas (11:09)
again.
Lucia Van Den Brink (11:26)
blog on the page that would show like three to five new articles if they were there. ⁓
Jim Zarkadas (11:27)
you
Lucia Van Den Brink (11:32)
So then beforehand you already have a lot of research which are in a way like experiments as well to validate what you're doing. And then eventually we were really sure like we had found the right way, the right design and then you know it was really built. It took like I don't know took like half a year to build. don't know if it's like a lot of teams, a lot of different restructuring of the code. Really crazy. And then we run an experiment to see if it actually worked. Worked like a charm.
Jim Zarkadas (11:54)
Yeah.
Lucia Van Den Brink (12:01)
So that was really a big growth lever for this brand and came from everything that we had researched before that we knew already, that we validated the initial kind of sketches that we had of that feature. And then we run an A-B test to really understand, yes, it is positive. And also we can expect like a 2 % lift from that ⁓ on the metrics we were using, which you can imagine are different for a news website.
Jim Zarkadas (12:27)
Yeah, yeah, exactly. Yeah, the numbers are way different like number of visits and everything. Yeah. Yeah Okay One other belief that exists is that ⁓
Experimentation is an expensive sport in general. It's like, yeah, to do A-B experiments, it means we need to build two things and test them and so on. And there is this kind of a belief in general that experimentation can be an expensive thing. And I'm just curious about your reaction on this and your high level thoughts.
Lucia Van Den Brink (12:57)
mean, it can be expensive, but it doesn't need to be. let me just try to think of the cheapest setup that you can use. It's probably, there are some free tools out there. So I would go with that, or either like a free trial just to have this, you need to have a tool to run the experiments for you, right? So I would start with that.
then you can run like one, two experiments a month usually. And then you might think like, we need a designer or a dev, but that's also not necessarily the case because you could start with just removing things from your website. Honestly, this is one of the biggest growth levers out there. And there is a lot of ⁓ people that have analyzed databases of all the experiments they have ever run. And then they find like, hey, one of the best things you can do is actually remove things from pages and from
journey so and that's really easy to do even with tools they usually have like an editor that you can just click the element and say like remove this ⁓ even if you would need to learn how to write CSS that's really easy too like even I can do that ⁓ so you know as a founder you could start by investigating removing things from the website
Jim Zarkadas (13:52)
Okay.
yet.
Yeah.
Lucia Van Den Brink (14:14)
And maybe sometimes you will get a loser. So let's talk about that example of the field that you gave. Let's remove a phone number field in a journey and see what happens. And maybe it actually harms the conversions, you know, that can be. But then at least you have learned, OK, apparently this phone number field is important to us because maybe they follow up the...
phone numbers with calls and then they close the deals or whatever that is. Sometimes adding friction can be actually good for conversion rate as well which is interesting. ⁓ But you know that's a really easy way to start running experiments and I think potentially anyone could do that. It's not like a massive experimentation program but it's a start and sometimes starting with these things is actually the hardest part so I really recommend that.
Jim Zarkadas (15:04)
Yeah. On tools that you mentioned, what are some of your favorite tools? Like it's a, I'm not a tools person in general, where like people get obsessed with tools and tactical advice. I'm more into the strategic advice, but when comes to tooling.
Now that you mentioned actually this question came to me. One problem that I've had in the past is we use, for example, post-hoc and data would be corrupted. Like tracking wouldn't work even if the implementation was correct in theory. And I'm curious, like if that's something you've seen in general is struggle to get the data right and trust them. That's one question. The other one is what are some tools that are not like crazy price because I know like data tools tend to go wild when it comes to pricing. Like Full Story used to have the free plan. Now it's like freaking expensive. ⁓
Lucia Van Den Brink (15:43)
Yeah
Jim Zarkadas (15:44)
And yeah,
I'm curious if you have any. Post-hoc feels promising to me, but I have mixed feelings because I've seen it with some teams that it didn't work. So yeah, I'm curious about your experience of that.
Lucia Van Den Brink (15:54)
Yeah, so getting the data right is super important because otherwise, like you say, lose trust in the program and the things you're doing. But often with a lot of tools, you can just measure the number of clicks to the next page, which might be already like the easiest way to set up an experiment and measure like, do people actually go to the next step or do they actually reach the end of this funnel? So usually to set that up with tooling is pretty easy.
Jim Zarkadas (16:02)
thing.
Lucia Van Den Brink (16:22)
Now, if you want to connect that back to a bigger data warehouse, etc. That's where things get more complicated. But again, if you're just starting out. ⁓
I think just measuring the number of visitors that go to the correct URL or the next page is just a fine start. But there are some things you can do actually to measure data quality. So I always recommend running AA tests. So that's just an empty test. So you have a control and a variant that's basically the control. But then you just measure if you install, you want to measure the click to that next page as a metric.
Jim Zarkadas (16:42)
See you.
Lucia Van Den Brink (17:00)
that you measured that beforehand before you ever start running any experiment, just to know you can trust the data.
Jim Zarkadas (17:06)
Okay, that makes sense. And in terms of tools, do you have any top of mind? Like any tools that you like? If you want to do testing, then you should take a look at this, let's say.
Lucia Van Den Brink (17:16)
I mean Jim, I've worked with so many tools. In the end, I am not really, I don't really mind whatever tool it is that brands want to use, that people want to use, because the most important thing is they split the traffic.
That's also why I built actually a little tool advisor that if people just want to get like three recommendations for tools, they can fill in what they need. And then that spits out like, Hey, these are your three recommendations, because I don't want, you know, people to get stuck on selecting a tool. So usually that's a big bottleneck and this big thing, and we need like resources and we need to make the right choice for years. And sometimes companies make it so big that they actually never get to the, yeah, never get to the experimentation itself or do
Jim Zarkadas (17:51)
Hmm.
Hmm.
Do nothing. ⁓
Lucia Van Den Brink (18:03)
nothing and that's such a shame so that's why I kind of created that little helper if people want to use that they can find it on my LinkedIn profile but anyway I don't really care for tools the most important thing is they split the traffic and then you can build your test of course some tools are better at this and better at that and you have expensive ones you have cheaper ones you have ones that fits with Shopify which is probably not relevant for your audience but yeah
Jim Zarkadas (18:27)
Yeah, I a point like integrations and the ecosystem you're building into.
Yeah. Yeah.
Lucia Van Den Brink (18:32)
But I'm not really recommending this or that too. Whatever too it is, we'll make it work is more my approach.
Jim Zarkadas (18:40)
Yeah, yeah, yeah. Yeah, for me, like in terms of specific tools, we're using Framer for with all the clients, with all the teams we're working with. And they have integrated A-B testing, which is beautiful because you don't need to spend any time. just like, here's a page, here's another page, run this test. Cool. Took me like five minutes. I even started doing A-B tests. I was like, ah, this is cool. So yeah, for example, when we redesigned the hero section of the homepage, we run a test because it's like a critical change.
Lucia Van Den Brink (19:03)
Mm-hmm.
Jim Zarkadas (19:05)
Which leads to another question that I have on this.
which is how do you decide what to test and what not? I know my questions are very high level. We're going to go into specifics. So what I have in mind to go into in a few, a few minutes is get specific stories from our work and ask you like, what would you do there? Like how would you approach it so that we can get into more tactical and specific advice. But on this one is a high level way of thinking. How do you prioritize A-B test? Before you said like start testing to, ⁓ you can just remove stuff and run some tests. The thing is like you can have an
endless list of choices there, like I could do this, I could do that, then I could do that, but then you have like a roadmap you're building, so how to approach it in general to not just get lost into this ocean of experiments.
Lucia Van Den Brink (19:48)
Yeah, so...
Let's take that example of removing elements because I think that's a great start. ⁓ So imagine, like you said, you have things on your web page.
Jim Zarkadas (19:55)
Mm-hmm.
Lucia Van Den Brink (20:03)
Where are going to start removing things? So there are prioritization frameworks that you can use and it can be just like ice or pie if you like, but there is also the PXL framework, which is a little bit more sophisticated because it asks you, is it above the fold? Which basically means will it affect more users? And that's a very important thing to take into consideration because you want to create the biggest effect. ⁓
Jim Zarkadas (20:27)
Hmm.
Lucia Van Den Brink (20:28)
So I would recommend doing some some type of prioritization first if you have a list of ideas, if you have a list of OK, we can remove all those things and then you can also look to pages. So which pages actually have the highest potential and
And this is maybe where the data part already becomes a bit tricky because there are some calculations you can make to find out which page has the most potential for you to test on. And this is called bandwidth calculations. But this is not something I recommend if someone is just starting out. So then I would just say like, what are your five most important pages in your funnel? ⁓ And then start.
somewhere maybe in the middle, start at not at a homepage because that traffic is usually a little bit diluted like people are not that the quality of traffic is not always that high but if you start at a product page which b2b size usually also has or ⁓ already the funnel where people go through to sign up
Jim Zarkadas (21:13)
Mmm.
Mm-mm.
Hmm.
Lucia Van Den Brink (21:34)
those are usually really important pages. So then you have done your prioritization, you have looked at the most important pages for you, and then you also, you you don't want to do any segmenting. So you don't want to say like, we're going to run this test only for mobile and only for UK visitors. You want to actually have just as much traffic as possible if you're going to run an A-B test. And ideally you do those calculations, but they are like a little bit hard to explain. And in our programs where we
install and teach experimentation that's actually one of the biggest hurdles so I will leave that for now but people can figure out in Google how to do that if they want.
Jim Zarkadas (22:13)
Okay, yeah, that's useful. Yeah, Yeah, I totally hear you on the website and also like that you need like to pay attention to the quality of the traffic and other things. On this one, I want to...
going to a specific idea we have with one of the teams we're working on. So it's a product that is about building help centers and it's more enterprise, right? So they're selling to bigger companies, minimum pricing plan, like the lowest pricing plan is 100 or 150, something like that. And usually companies go for like three or 400 and so on. And the profile of clients is not SMBs, it's more like large organizations that are...
Lucia Van Den Brink (22:41)
Okay.
Jim Zarkadas (22:51)
that are buying the software to build internal knowledge basis too. So that for example, ⁓ call centers, they build knowledge basis for their agents so that they can easily find answers and then provide proper support on the phone. So this is the use case. Now we have an idea. We've seen that if people book an onboarding call,
they're more likely to convert. That's something we've seen from the sales and the customer success team. And this is an insight. So we're like, why not just make the call a mandatory step in the onboarding flow? Why not just force people to book a call? And this could be a filter of leads as well. And it's a scary thing to tweak the onboarding flow. By onboarding, I mean the sign-on flow that you add your email and password and then go through five steps and then you're into the admin and you can build your knowledge base. And...
For this one, I feel confident. like, yeah, let's just go and add this step. Now we're discussing this idea with the team actually, and feels like it's a good case for experimentation, for example, because we're like, so we force, so we add a skip button, or should we do another option where you don't ask them on the onboarding, but you ask them a bit later, you delayed and then you throw open a pop-up. And when they take specific actions, let's say when they create a second article into the, the knowledge base, then you open a pop-up and ask them, Hey, we can help you do this. We're going to hold your hand, book your onboarding code.
and most people are more successful when they do it. ⁓
My question on that is how would you approach it? There are two sides of me. The one which is like, I'm sure you've seen this typical kind of a designer but also founder, we're like, no, I want to trust my gut feeling. I know this is a good idea, which honestly nowadays that are more into business and like talking with founders, I want to feel like it's a stubborn approach. And then it's the experimental approach where...
I would say let's just test this. But it's to say, complicated to do if you're a newbie into the experimentation space. the question is how would you approach it? And one thing that I wanted to ask here is also, what do you measure? That's, I think, the most complicated question to answer because for this one, the goal that we have is increase the trial to paid conversion, right? So new people sign up for the product. We want to get more customers.
But this is really hard to measure from a Navy experiment. Like, again, did this change in the onboarding flow? What was the converse of trial to pay? You need to wait even for four months to see the data. Of course, you could take a look at it, but I'm always wondering, like, is there any more direct, something closer to that step that we can measure? So it could be ⁓ number of calls booked, or it could be the drop off on the sign up. And then you need to run this experiment maybe for a few months to see how it affects your revenue and your high level conversion rate. So you need to look at
various metrics and give it time because it's a bigger experiment like in terms of the impact. So this is where I am in terms of like think about it and questions that I have and I'm curious to hear your thoughts.
Lucia Van Den Brink (25:41)
So what would your hypothesis be if people book a call, they will be much more informed? Or what actually happens on the call?
Jim Zarkadas (25:52)
Yeah, so that's a good question. So the problem that people have is that building a knowledge base...
is a lot of, it's a big effort. usually what they, 50 % of the users have scattered documents and knowledge in emails, in people's heads and so on. And they need to take all these, put them into a website, create folders, organize it, and then customize the knowledge base. So there's a lot of effort. We have built self-service tools like import tools, I think customization settings and so on. But they love the call first because they interact with the support and they get told that, okay, this is a really good company.
to trust, that's one thing, and the other thing is that...
If they don't want to call, we sell them, they don't have to pay, but we sell them on the idea of recurring calls actually, ⁓ and it's implementation calls essentially. So we guide the people on how to build a knowledge base and we make it easy. They have this anxiety that this is going to take a lot of time, but that's a reason sometimes they drop off and delayed and why the average purchase time is really long. So yeah, they call it the onboarding call, but it's really about implementation, giving you guidance and making this whole thing.
easy because building your first knowledge base is a lot of it's a big effort.
Lucia Van Den Brink (27:03)
Okay, so the hypothesis is if more people take a call, ⁓ they will understand better how easy it is to implement the system for them, which will lead to more also free signups. Are they also there and then paid signups as well?
Jim Zarkadas (27:24)
⁓
Yeah, all signups, they don't have to put a credit card, but we don't have a free plan. It's all paid plans.
Lucia Van Den Brink (27:31)
Okay, it's all paid plans, so there is no free plan. But you said they don't need to put a credit card. How does that work then? Okay, so they have a trial. Okay, so I think your first main KPI would be measuring the trial and seeing either positive effect or no effect there.
Jim Zarkadas (27:33)
Yeah. Yeah, it's a 30 days trial. Yeah.
Lucia Van Den Brink (27:47)
And that's where you would base your decision on as well of the test. If the test is a winner, yes or no. And then later you come back and evaluate for the delayed metrics. So for example, we have the same with SEMrush, our client. They also have a free part of the tool that they can access. And then... ⁓
Jim Zarkadas (28:00)
Mm-hmm.
Lucia Van Den Brink (28:07)
When we run an experiment, we can immediately measure the number of free trials and increase there. But what we also do is we actually wait for two weeks to also measure the effect on paid accounts. So that's a delayed metric. And then we either change course from what we thought we were doing or not, of course, but we definitely take the delayed metric into consideration as well.
The other thing I think you mentioned is how to approach the actual A-B test because in order to test this hypothesis you can actually also skip and
not have people call at all and then you could measure, does that affect negatively and then you would also learn. That might be like the easiest thing to set up but it's not necessarily where you want to go strategically, it's the opposite where you want to go but that's kind of a negative test like removing elements from which you can also learn.
So if building a whole pop-up or building the flow where people have to take the call is really intense and takes a lot of resources, then sometimes it's easier to do a negative test to see if we do the opposite thing, what does that do? But you also had other solutions to work on testing this hypothesis, which were the pop-up.
which were maybe making it mandatory to book a call in some way. And here you already see that you have one hypothesis, but we already have three potential tests. And this is again where then you think about prioritization, ⁓ which solution that we're thinking of would make the biggest impact. So which one will be maybe higher on the page or affecting more users or is based on something we really already tested and know.
So there again comes that system of prioritization as well and then you can also think of okay which pages are the most useful for us but it sounds like this is just one page. Is this helpful Jim?
Jim Zarkadas (29:59)
⁓
Yeah, like I would love to go a bit deeper. For me, what's... You mentioned you would test a number of trials. So with this specific example, just adding this step.
could bring the number of trials down because some people may drop off and be like, no, I don't want to book a call. I'm going to cancel. But maybe these are not qualified leads. Another thing that is very tricky is that you see trials, but the big question is what are the qualified trials? And we're working on that. Like we've added questions in the onboarding flow to really see, is this a real company or they're signing up with a random email? Did they answer any of the questions? Like what tool did you use before? Were we asked some small quick questions to get some clarity or they're just tapping rather
of stuff, so this can tell us a bit about the quality of lead. And if we would look at the number of trials, I feel that would go down, but this is an assumption because we add more friction and some people may just not want the call. So yeah, this specific example...
I'm trying to answer the question, what is the right metric to look at? Like number of trials, I would look at it for sure, but not as a decision maker. Could be activation. Do these people really get active? Do they log in more than three times or do they in the end convert to trial? The thing is like, there are so many things you could measure. Yeah. And what, how do you decide? It's a, it's a tricky one.
Lucia Van Den Brink (31:28)
So you never decide on just one. And I think that's the answer because you will have one main KPI, which you will use as a guidance. But then you always have secondary KPIs to measure, for example, activation rate. That's a great one. But also the effect on the bait.
subscriptions that people eventually take ⁓ but maybe also how much they log in etc just to get a bigger holistic pictures now it's not to say you're going to measure as much as possible like you have to think about this beforehand so that you don't just start looking at the metrics that go up whenever you run a test but yet that you're really intentional with hey this is the hypothesis and these are the elements that we want to measure this on so
That's probably what makes this more of a full and holistic story here, Jim. It's indeed not about just taking one measure and deciding everything on that, because that would never tell the full story here.
Jim Zarkadas (32:30)
Yeah, what's some things that I'm learning now like from from what you're saying is
First, the concept of delayed metrics. You have like the direct metrics and the delayed metrics that take time. So that's very interesting on the mindset. And then you also have what you said is like, don't try to find only one. Like you don't need to measure only one. You should measure more. You can have a primary KPI, but you need to look into more metrics. And that's useful. And now it makes more sense because I have this tendency of trying to find only one. And that's what I've seen like most teams do as well. ⁓
Lucia Van Den Brink (32:56)
Yeah.
No, yeah.
And also you need to create a business case for this. as you say, we expect actually free trials might go down, but that is not necessarily a problem if we will get more paid customers out of it. So then you want to calculate what is the revenue and what is the percentage of... ⁓
Jim Zarkadas (33:14)
Exactly.
Lucia Van Den Brink (33:19)
free trials that actually goes down and does the revenue from the paid subscriptions actually compensate that? And you want to have a positive story there, right? Otherwise you're not going to implement it. you need to have all these metrics and then you also want to calculate, okay, if even one metric harms the other.
Is it still worth it? So a test we usually run that is super positive, but it's really sometimes a hard business case to make is showing customer support on, for example, a checkout page. So just showing a phone number and big corporations say, no, don't show the phone numbers. Then we are going to get a lot of calls and then we need to have more people in the call center and that costs money. ⁓ But then if you actually do so, that increases conversion with such a big
a step that actually compensates for all the costs that you will have in your customer service center. So usually this is also where we try to make the business case of, okay, yes, people will call more often, but in the end you're growing as an organization and you can just pay for those costs with the revenue you're making. you know, it's in the end a good call to implement this, but yeah, then you really need to make the business case a good one and make sure, you know, everyone is aligned on that.
Jim Zarkadas (34:20)
Hmm.
Lucia Van Den Brink (34:40)
and ⁓ work with multiple teams. So that's always interesting.
Jim Zarkadas (34:45)
Yeah, and like that's where I don't have the experience with the tools and I'm taking notes in the same time and take a look after the podcast episode as well is you can do like, you can look at product events and so on, but most of the times I would say, yeah, let's not generalize like many times, let's not say most. ⁓
you need to look at final metrics that are like on the core metrics of the companies, like trial to pay conversion, new revenue added per month and so on. ⁓ Because you may see that this helps a lot big ⁓ accounts convert, for example. So in the end, it's a business and you're doing all this to make more money. So you need to keep a ⁓
to be close to these metrics as well. And that's where it can get tricky because you need to find who were the people that joined that experiment and how much revenue did they add, which means that, yeah, I'm not sure like with Postdoc, for example, how that would work, but yeah, that's for a more to look into. It's too technical for the podcast. Yeah, exactly. Yeah, exactly.
Lucia Van Den Brink (35:42)
Yeah, it's probably a bit too specific. ⁓ of course, there's a lot of information out there on these things. And I think we're already giving the listeners
a good head start on this.
Jim Zarkadas (35:53)
Exactly, yeah, it's more about the mindset and yeah the technicality something to figure out because it depends on what billing system you're using and sometimes the the billing platforms they even have maybe testing themselves. yeah Okay, let me let me think about the the next one ⁓
me see my list. ⁓
Yeah, I'm thinking also about some other cases. Oof.
some other exams from another team we're working with, which is scheduling software. It's more for SMBs. It's a scheduling software for cleaning businesses. And we're actually running an experiment right now. So I'll tell you the experiments that we're running and then can give me, I can see your reaction. ⁓ The scheduling software for cleaning businesses. And right now we're really focused on optimizing turn for the first three months. We see that we have a high turn ⁓ within the first three months. And
Lucia Van Den Brink (36:31)
Okay, I'm curious.
Jim Zarkadas (36:45)
One thing we're doing is like we're getting all the data of who turned, what was the team size of how many cleaners did they have on the team? What did they use before? What is the reason they set on the turn survey? When they turned, they have to go through a small survey. What are the answers they gave there? And we're trying to find patterns, right? So we've seen that most people cancel. We have a big portion of the accounts on the same day. ⁓ A big portion of people cancel within the first two weeks. And even if you see a big turn within the first three months, it's mostly the first two weeks that is happening. So we found like some interesting data just to understand
and what's going on. And one of the ideas we had confidence on was a get started page. So right now, this one thing to mention is that these people are not technical. So we're talking about people who used to clean houses and decided that I want to out their game and become a business owner. And like, okay, I can do this. I'm going to hire other people and I'm going to build a cleaning business. And they dream about building a million dollar business many times.
They decide to use a software like ZenMate because they need some automation. Like if you just run your business pen and paper and Google calendar, it's very error prone. Like you may miss appointments, you need to remind your customers you're going to their house, they may not be there. So you really need some automated communication, for example. That's one of the problems they're experiencing. And right now when they sound up on ZenMate, they go through a few steps. We ask them to put their credit card, by the way. So we have intentional friction to qualify the leads because if they're not willing to put the credit card,
It means they're just kind of messing around. That's something we take the team test before I joined and they're really into this. They tested also without a credit card and they saw the quality going down and sucking their customer ⁓ support resources. And once they sign up, what's happening is that we just throw the users to the counter and be like, good luck. Now you can figure out your how to use the scheduling software. And we said, okay, what if we just make a nice page, a get started guide with the five, six things you should do on what people are
interested
in. first you can request a free migration. It's the first step that you can talk to our team and we can bring your data here so you can test the app with real data. Then you should go and set up your payment methods. You can charge your customers a line through ZenMate. So here's a button you can click and go to the payment method setting. So we have five, six steps where we sell ZenMate, like remind them all the benefits and at the same time we give them a path to follow. So that's an assumption we have is that if we just throw people there, they're not technical.
they're gonna get lost. So why not just give them a path and then we expect to see more people being active and getting value others than mate and in the end reducing the turn within the first two weeks. So that's kind of the experiment and right now the A-B test that we're running if I remember correctly is we have with the one variant is with the get started page the other one is without and we're measuring if they
added 10 appointments. So that's a metric that team came up with. ⁓ Is this the right metric to monitor? I'm not sure. So I'm curious about the case just to hear like your high level thoughts ⁓ on like what comes to your mind when you describe all this and how you approach it.
Lucia Van Den Brink (39:51)
So from what we just discussed, we can ask is there only one metric or is there more to measure? So you say 10 extra, what is it, calendar scheduled events? ⁓
Jim Zarkadas (39:59)
Mm-hmm.
Yeah, yeah, 10 appointments
that they added 10 appointments into their calendar
Lucia Van Den Brink (40:09)
And are you also measuring churn in those two weeks? Since the two weeks are the most important.
Jim Zarkadas (40:14)
I
the variants? No, actually, which sounds an obvious thing to look at,
Lucia Van Den Brink (40:17)
I mean that will be good. Yeah but it's hard. I know it's
sometimes hard to connect the dots there so I can guess to why that hasn't happened. ⁓ Anything else that might indicate that a customer is happy? Have you ever looked into, our happiest customers, use R2 in this and this way, they always click here, they always do this. Are those signals you might be able to measure as well?
Jim Zarkadas (40:41)
I lost you a bit on the first part of the question. You mean what actions they take with the one case and the other one.
Lucia Van Den Brink (40:45)
Yeah,
we're talking about there is probably a group of clients that's not so loyal and that leaves and then there's your most loyal customers in the software. What makes them different? How do they use the tool? And can we measure those as kind of microconversion? if they, I don't know, look at the calendar a lot, I don't know what it looks like, right? So it's a bit hard for me to suggest, but I would look for.
Jim Zarkadas (40:52)
Hmm.
Yeah.
Yeah, of course.
Lucia Van Den Brink (41:13)
Yeah, at least one main KPI could be the 10 extra events scheduled. And then underneath there, you want to have to one, two, three supporting metrics that can really tell you more of the full story, I would say.
Jim Zarkadas (41:26)
Hmm.
Yeah, I feel like that's a really nice takeaway is like, think about it a bit holistically, like find the key metrics and don't try to guess only one because the activation, it's good that we're measuring the amount of appointments because the end goals that they use more the software, this way they get more value and this is going to help the turn. ⁓ But we could have more stuff like how many people connected their credit card processor, for example, or ⁓ how many they turned. So yeah, what metrics to look at? That's, yeah, that requires more thought. It's outside of the
Lucia Van Den Brink (41:49)
Okay.
Jim Zarkadas (41:56)
episode but it's a very good guidance to think about actually because we have more experiments in the list we want to do.
And there are some things, there is a debate right now on this one that I could mention. Again, not like a strong debate. It's more like a trying to decide. Debate sounds a bit too much of what should be an experiment and what shouldn't be an experiment. So one of the ideas, for example, is ⁓ productors. You have this get started page. They click on set up payment methods and we're like, okay, we send them on a settings page where there are many UI elements. And it's not just about the payment methods. It's the billing settings, like the client
settings and in there there is a card about it so we're thinking what if we just add a very simple product tour where then when they go from the get started page they click on setup payment methods they open the settings and then we give them a nice tour they can click here you can do this you can do that just to make it easier and what I say as a designer of the team I'm like hey guys this is an obvious UX improvement we don't really need to test this like why test the product or it feels too much ⁓
What is your opinion about this in general? About the thinking that this is an obvious UX improvement, I see no reason to test it and run an experiment. Do you think that this is a valid way of thinking or not?
Lucia Van Den Brink (43:10)
So experimentation is really humbling and you know we're just most of the time really wrong. So if I look at all the experiments I've ever run, I've run more than a thousand experiments myself and I look at bigger experimentation programs, there's only a win rate of 20 to 30 percent.
While we actually think that everything that we do will be good, right? We have the intention of it to be better. But in fact, there is a really small amount of things that actually move the needle. So that's also what you'll learn if you start to experiment is that most of our ideas actually don't matter. Like they don't move the needle at all. Then there is also like 20, 30 percent that actually turns out bad, even if we have good intentions.
And then we have that magic 20, 30%. That is actually a win. So saying like, hey, this is an improvement. We should just do it.
is not always the best way to go but I am also not against you know there are also situations where you don't test things ⁓ but having a strong gut feeling about something is usually not the best indicator unfortunately it would be great if it was but unfortunately it's not
Jim Zarkadas (44:15)
No, I hear you. And the more I go into data, the more I realize that I can be wrong. Like, we, like humans, we have all these kind of beliefs and assumptions, right? And our actions.
are driven by all these. And beautiful thing with experimentation data is that they can challenge your belief in assumptions. Like one, for example, assumption that I have, which is not very related to what we're discussing on these ideas, is that if a cleaning business owner has more than 11 people in their team, there is no way they're to use Google Candler and notes. Like I cannot imagine running a team of 11 people and running my calendar pen and paper, but they do. And we that it's one of the most popular sources.
Lucia Van Den Brink (44:49)
Yeah, probably they do, yeah.
Jim Zarkadas (44:52)
when I went into the data. But if you could ask me without the data, would say, no, there is no way. And then you start acting with this belief. So I fully hear you on this. ⁓ Yeah. And if I try to go deeper into how I feel and where I'm at mentally,
I see myself having friction on running experiments sometimes because I have this belief that, ⁓ it's going to make things slow, like that it's going to add complexity. The typical stuff we started discussing at the beginning of the episode that experimentation is a bit of a, like a drug in a way. But if you just make it easy and part of the process, then I wouldn't have an issue because in this case of the product, it would be great actually to just run a simple experiment. Some users get the product or some people don't get it. And then you see how many people actually
Lucia Van Den Brink (45:31)
Yes, why not?
Mm-hmm.
Jim Zarkadas (45:37)
did set up their payment methods. And if you were to ask me like, would you do this? If it was easy, I would say, yeah, hell yeah, no problem. But my initial reaction to not do it is because I assume that it's gonna be tough. So I feel like there is a very interesting thing there of...
Lucia Van Den Brink (45:40)
Mm-hmm.
Jim Zarkadas (45:52)
make it easy, make it easy. Side reference on this, is a book, Atomic Habits, maybe you know it. And like one of the principles is make it obvious and make it easy if you want to make it a habit. And I feel like that's one of the mental analysis also from a behavioral kind of a psychology perspective on this one is just make experimentation as easy as possible. So you're going to really add it in the process. Yeah.
Lucia Van Den Brink (45:57)
Yes.
But maybe it's also mindset, it's also
something valuable to learn that ⁓ the easiest thing is actually to experiment because what if you implement this and you're actually wrong?
You will first of all never know, but it will hurt the clients, it will hurt the conversions. You will have to track back what is it that actually is harming. And that takes you much longer than actually running the experiments. So if you reverse it and see the risk that you're taking, doing the experiment is usually the easiest option actually.
Jim Zarkadas (46:35)
Mm-hmm.
Hmm
I love what you said. Yeah, that's a very good way to look at it because that's something that is I won't lie that something that's happening not just with teams that we are working with but with many teams that I've worked with in the past and I've seen with other people is that they do stuff but they don't really monitor the impact. In theory, they will say, yeah, we should take a look at what's happening but nobody really looks at it. We don't have like a very specific dashboard to look or somebody to own the looking part that this person is responsible to tell us did this work or not. And experimentation also solves this problem. It forces
people, it's part of the process to look at the results. But when you just see stuff, then yeah, you can see if the company is getting better and adding more revenue, but you cannot see what is the actual improvement ⁓ you did there. ⁓
Yeah, it's very interesting. Like I'm really happy with the episode because I feel like I'm also going through a mindset transformation and I hope like for listeners also it's going to be a bit of the same that are on the same page with me, which I can tell there are many people like with no experience that have similar beliefs and similar kind of a friction with the one that I personally have. ⁓
Lucia Van Den Brink (47:45)
Yeah, no, of course, I this every day. So that's why I can kind
of see it and explain the others. Well, the alternative as well.
Jim Zarkadas (47:54)
Yeah,
yeah, yeah, yeah. Let me see, we have 15 more minutes. So I just want to have some questions that I wanted to take. ⁓
Lucia Van Den Brink (48:09)
Would you mind if we stop like five minutes before because I immediately have a call after that and it was like a long session but then I can grab a tea before the next call. Yeah, thank you. But yeah.
Jim Zarkadas (48:14)
Yeah, yeah, Yeah, yeah, yeah 100 % yeah, so we have 10 minutes left so what
would be yeah 5 to 10 minutes I think I'm yeah, we can just wrap it up and I have actually two Two questions The the one is
It's more of a personal question to you. What are the biggest mind transformation you've seen from being a junior to being a senior person into experimentation? ⁓ What are some things that you saw that you went through in terms of changing your mindset?
the biggest transformations you had essentially.
Lucia Van Den Brink (48:56)
So maybe it was also in this episode that often juniors think like hey whatever we're going to change it's gonna be a good thing and then gradually you actually learn like okay that's often not even the case like not many things actually move the needle and I think also
Jim Zarkadas (49:03)
Hmm.
Lucia Van Den Brink (49:15)
There is this misconception that the biggest value will be seen in the money that we make from the tests and the learning. Well, in fact, the biggest value is how it shapes your organization, how it helps you make the right decisions and just do the right things in general, but also creates a nice environment for everyone to be.
empowered in the way that a designer can figure out with the help of A.B. which solution is actually the best one and learn from that and take that learning and continue with the next things while having this, while having learned this thing. in the end, experimentation is really a tool and just a process that should empower a lot of people to not only solve the problems that they're working on,
but also learn what is actually the solution. So sometimes we think we have solved something, but we also want to know if we actually did right. And that might be the empowering part. What I also see is that if you're just a mere designer or content manager inside a big organization, you might not have the...
capacity or the reach as a person to make things happen. While if you have the data and if you can really say like, hey, this is gonna get us a million because I've tested this and see the effect, that's when people actually start to listen. So whoever you are, if the organization really embraces experimentation, that means that... ⁓
even an intern can suggest an idea from their perspective and from their background and from the expertise that they probably have themselves as well, that might be a winner and help the organization grow through that. So in the end, you're using all the people inside an organization with experimentation to help the company grow.
Jim Zarkadas (50:50)
Hmm.
Hmm.
Yeah, I love it. And I think that I'll keep this as the last question. I don't think I need to go to another one. It was really good. And yeah, I fully see your point. I feel that the most useful part of the episode as well is...
What is the true value of experimentation? Indeed, like many people have this belief that yeah, just an A-B experiment to make money, but it's way more than that. It's actually a mindset transformation and a casual change that can translate into many, many things on how you operate and how you build, which of course can translate into revenue, but it's much, much bigger. Super. Thanks a lot for coming today. That was a really nice episode. Yeah, anytime.
Lucia Van Den Brink (51:40)
Thanks so much for having me, Jim.