Subscribe
Share
Share
Embed
Interviews with the experts behind the biggest apps in the App Store. Hosts David Barnard and Jacob Eiting dive deep to unlock insights, strategies, and stories that you can use to carve out your slice of the 'trillion-dollar App Store opportunity'.
David Barnard:
Welcome to the Sub Club Podcast, a show dedicated to the best practices for building and growing app businesses. We sit down with the entrepreneurs, investors, and builders behind the most successful apps in the world to learn from their successes and failures. Sub Club is brought to you by RevenueCat. Thousands of the world's best apps trust RevenueCat to power in-app purchases, manage customers, and grow revenue across iOS, Android, and the web. You can learn more at revenuecat.com. Let's get into the show.
Hello, I'm your host, David Barnard. My guest today is Alper Taner, head of performance marketing at a stealth mode app studio, working on productivity and utility apps. Alper has been in mobile growth for over a decade, leading UA and MarTech efforts across a wide range of verticals and managing eight-figure annual budgets. On the podcast, I talk with Alper about the competitive advantage of ignoring some best practices, the risk of drawing false conclusions when researching competitor ads, and why poor metrics are just facts until proven problematic.
Hey, Alper, thanks so much for joining me on the podcast today.
Alper Taner:
Yeah, thank you for having me.
David Barnard:
This has been a long time coming. You and I first met in person at MAU years ago and we've talked off and on about getting you on the podcast, but recently, we were talking about a lot of your contrarian ideas, just that you think a little bit differently and like to challenge the rules of thumb. And I thought that would be an especially fun thing to talk about. And so, let's kick it off with that. The podcast shares a ton of rules of thumb and like, "Hey, try this, don't try that. Oh, this didn't work for me, so you probably shouldn't do it. " And on the RevenueCat blog, we share all sorts of articles like that. But you went through, as you and I have been talking, preparing for this episode, and you ripped holes in some of the rules of thumb. So, at a high level, why don't you just start off, how do you think about these best practices and rules of thumb?
Alper Taner:
There are definitely best practices and rules of thumb that you should stick, I don't know, 80% of the time, but that doesn't mean they are universal for every single case. So, if I were the app developer listening all these ideas of dos and don'ts, I would prioritize it based on my current situation, data, and I would still try them just to see that it also doesn't work. Of course, it has to be reasonable, but idea is that your data should show you that instead of, "Oh, we heard someone said X, Y, Z," type of an approach. So, I think this applies across the way you test the creatives, the way you test incrementality, the way you test your bidding strategy. So, I think these are all meaningful tests to a certain extent, depending on what stage you're in. If you don't have your own learnings, of course, it's smarter to leverage others' learnings and it can just help you to prioritize.
If you hear something didn't work for 20 people, just deprioritize it. But it doesn't mean, oh, forget about this because it's not going to work for you because algorithms are constantly changing and improving and whatever someone tried six months ago can work today. And then we see that it's happening across multiple platforms within the platforms, as well, like the campaign A or specific targeting that was working really well stops working. This is like a day-to-day thing or your creative dies, where the creative that worked six months ago that you had to relaunch, it works great again. So, also, it potentially looks much better now because of the late conversions and stuff. So, it's not black and white. I think when you hear such advice, you should definitely take them as suggestions, but when it comes to execution and prioritization, you should do that based on your own data because you don't know the history of that particular account and why particular thing didn't work.
And there are several or a lot of reasons why a particular setup would work for one market or platform versus not. And that's why they're all individual cases. There are definitely, as I said, best practices that are universal, but I would always challenge them. I take them as, "Okay, great. Let me try it."
David Barnard:
Yeah. Well, I think you've just created a new rule of thumb. I like that, that you should spend 80% of your time on the best practices and learn because like you said, if 20 people tried it and it failed, maybe it is going to fail for you, but then take 20% of your energy, your budget, your efforts, and do the exact opposite of the best practices and figure out for yourself why they learn. And to your point, if you're early and you don't have a lot of resources, you don't want to over-index on being a contrarian. But that's the whole thing, is that there's alpha in the best practices and that's why they're the best practices. But then there's also alpha in the things people aren't trying or the things that certain people failed at, but you're going to succeed in that for something that's very specific to your app or your niche or your market or some other thing like that.
So, yeah, the new rule of thumb. Follow best practices 80% of the time and try crazy stuff 20% of the time. I like that. Let's go through some of these more specific contrarian takes. And one of the ones we were discussing is that sometimes, you just need to take something as a fact and not consider it a problem until it becomes a problem. And the example you gave me, and I'd love for you to elaborate on this, is that if your trial conversion is low and you're like, "Oh, trial conversion being low is a problem." Well, is it a problem? Are you getting cheaper installs? Tell me about that.
Alper Taner:
Yeah. Typically, people would say like, "Oh, we have, I don't know, installed to start trial ratio, I don't know, 5%." And usual healthier benchmark is more like 15% to 20% and such. And I'm like, "Okay, what have you done? What have you tried? And how did you interpret the results? How did you build on those results?" They're like, "Oh, we didn't because it's a problem." So, that's not a problem. It's just a fact. I think something becomes a problem after you systematically test it, iterate on the results, and try many different ways from, let's say, radical to iterative and you're still stuck at the same level, then I think I can consider that as a potential problem. Instead, you should always see what you can do or understand the root cause. So, diagnose any factual problem that you consider as a problem and then come up with an action plan and then measure the results to see the impact.
It's as easy as that. It might sound very straightforward, but I can tell you many people bypass this step-by-step funnel in figuring out stuff because when you follow such an approach, you realize that, oh, actually it's not a problem. Or I don't know, it's simple as people say like, "Oh, we tried, I don't know, Snap and it didn't work." I'm like, "Okay, how did you try it?" "Oh, we spent, I don't know, 5K in a week and it didn't work." I'm like, "Yeah, that's not a problem that Snap didn't work. You just didn't test the channel in a proper way that is optimal for the channel. You shouldn't test the channel just for a week." And same goes for, I don't know, creatives to channels or the markets, "Oh, it's too expensive for this market, or that channel didn't work." So, people like to come to conclusions fast.
And I think because of the priorities, because of this scarcity mindset, they want to reach conclusions really fast and they want to move on to the next. So, when they are in this mindset, I think then they're missing out on testing things properly. I think a lot of people focus on the execution, which is on paper great, but I believe the success of a test is at least 50% dependent on how you plan it, how you strategize it, how you hypothesize it, and how you execute it. And execution is just a what part. Maybe it's even sometimes less than 50%, but basically, how you execute matters more than you execute or not. Sometimes, people also say, "Oh, we tried, I don't know, let's say Lookalike, it didn't work." Because the way they tried Lookalike was at 10%, run it for a week, that's it.
No seed testing, no different ranges of testing and such. You cannot just call the whole thing doesn't work. If you don't slice and dice and try different things, that's my approach and that's how I deal with day-to-day business to understand and prioritize it based on, okay, how can I say I tested something and what information I need to call the shots that it doesn't work or it does work? So, this kind of thinking I think will force you to think of all the preliminary steps on what you should be considering and so on. Yeah.
David Barnard:
I like that root cause thinking. I actually did a talk at MAU a couple years ago about the top three subscription app metrics that really matter. And that was the thrust of my talk was that you need to look at the whole funnel, not just individual metrics. And like what you were saying, it's like if you have a low trial conversion rate or if you have a low install to trial rate, or if you overly focus on any one metric and you're like, "Oh, this is a problem." And you grind and grind and grind on that "problem" when maybe you can get like really cheap installs and the really cheap installs are why you have a low conversion, but hey, that's not a problem. It's not a problem when you get really cheap installs and your conversion rate's not great. That's maybe just a natural fact of your business that for whatever reason, you're getting incredibly cheap installs.
And the example I always use with this is like, my apps have gotten featured a ton by Apple over the years and getting featured is fantastic because it's totally free. Zero cost of install. But guess what? Those cohorts convert really poorly. So, do I have a conversion problem? I don't know. I know that those cohorts don't convert very well, but like should I overly focus on that or should I be more focused on top of funnel and like finding other ways? And so, a great reminder to not get overly focused on any one specific thing to the exclusion of like all the different moving parts in the funnel. And I like the way you phrased it, too, like it's just a fact. That's a fact. Right now, that's a fact. Is it a problem? We don't know. It's a fact. Figure out if it's a problem.
Don't just assume it's a problem.
Alper Taner:
Yeah.
David Barnard:
So, another thing you mentioned, you've been thinking a lot about lately is creative interpretation. And we had Eric Seufert on the podcast recently and he's intentionally a bit provocative saying like, "Don't even try. Don't even try to interpret the creatives." But you were telling me, you do think there are things that you could learn from trying to interpret why things won, why they didn't. So, what is your process for looking at the winning creatives and finding some value in that process?
Alper Taner:
I think looking at the output of a creative performance definitely matters because that's how you will iterate and find the next winners or the losers. It is important to look at the relative metrics, as well. So, maybe what he's referring to is more like, don't take it as your source of truth because platform is doing the modeling and such, but at the end of the day, the output/the performance of the creative matters and the approach should be more like what can we learn from it regardless of the output in a way, like good or bad, it's all about the learning. Okay, this creative for whatever reason performed better and other creative performed for whatever reason worse. Let's understand why. What is the hook rate? How is the thumbnail? What is the audio? What is the background and this and that. So, once we understand and then we start changing one variable at a time, then it becomes a lot more clearer on how we can engineer the success.
That's the whole point of, that's how you create winners from the winners because you have a winning concept and thanks to the interpretation of the output, you're able to create more winners because you look at the relative metrics, that's a guidance for what you should be testing and what you should be not testing anymore because whatever you do in that particular case doesn't work. However, I have seen from the same concept, like the third variation worked, but the first and second failed. It's the same concept. It's just slightly different messaging and such. So, that's why you would always test with few variations just to avoid such cases where you don't just jump to conclusion of a whole concept and so on. I think that's also important. So, there could be, of course, false positives and so on like nothing is perfect, but also, I'm not also saying like, "Oh, you should trust algorithm blindly and stuff like that."
What I'm just saying is we should be able to interpret the results and then drive conclusions to shape the next creative iterations based on that. So, that's when you need to interpret the output. And then what matters at the end of the day is your success rate and not the amount of creatives because I think LinkedIn nowadays is full of, "Oh, you know what? We just tested 500 creatives." "Oh, no, we tested 700. How much did you test?" Kind of race that is not so meaningful because those numbers alone don't matter if your success rate is very low. I would more look at your cost per successful creative that is beating the BAU ads performance. And it doesn't matter if you test 20 versus 200. Of course, okay, 20 versus 200 might matter, but it doesn't matter if you're testing a very high amount of creatives with a little success.
It matters more you test less amount of creatives, stronger hypotheses, stronger investment from an analyzing and hypothesizing way with a higher success rate than what matters more than just the total number of creatives that you are testing. I see potential issues when the creative amount is on the higher end, because then the mindset is like, "Hey, look, I just have to test these. Let's just move on." And then having less emphasis on why it worked, why it didn't work, and so on. And then sometimes, there's a disconnect because typically, a company that is testing 500 creatives a month tend to have a team and usually, there are, I don't want to call it silos, but the guy who is running the UA is not doing the old briefs of the creatives and then analyzing and then giving the briefs and all that typically. In the smaller startups, yes, but in the bigger teams, there is that.
And then so then the UA person's job is just to run them and maybe analyze them if they don't have the analyst team, then to give the creative, let's say the optimization manager to put them into briefs and get the next production. So, when the quantity is so high, I think the level of detail and sometimes, also the quality of analyzing and shaping the next input is also being jeopardized a bit because of the high quantity, which is not necessarily a win or something to be proud of alone if your success rate is low because also then the mindset shifts like, "Hey, look, I have 500 creatives. I can't spend 5K on all of them." Sometimes, this gets into, "Oh, we only spend $100 on these creatives and that's it. " Okay, but how did you decide on that threshold? And I think finding that threshold is such an important thing for every account.
For example, there's no universal number that I would give. It's very different from business to business, depending on your CPI, depending on your other CPX metrics. But what I would suggest is to look at your cumulative spend and cumulative CPX metrics, whether you're testing on a CPI or a cost per trial or just cost per subscription and such, and see when they stabilize. So, because I see a lot of accounts with high spend, they're sometimes just spending 10K, 20K only to realize, "Oh, this is actually not a good creative." I'm like, "Oh, really? I could tell it after 500 bucks, look at the data. It was already bad. It was already screaming. Why did you spend so much?" So, in order to avoid that inefficiency and scaling it more efficiently, you could easily look at cumulative spend and cumulative performance over time and to see where your data stabilizes because we are not going after the statistical significance.
No one has money for that, but we also don't want to spend just 20, 30 bucks to see, or we don't want to just let the algorithm decide which one to pick. So, there, the problem is the following, just because algorithm decides to deliver one creative over others doesn't make the other creatives bad creative. I think there's such a misconception or interpreting, I would say in a wrong way. That particular creative that got the 90% of the spend was maybe lucky, we call it lucky 5,000 impressions because Meta does some decision making around first 5,000 to 10,000 impressions. And if one of those 5,000 people who saw the ad didn't engage or whatever signals that Meta didn't receive, then it will not deliver anymore, but it will deliver the other one, which just got maybe one, two clicks and stuff like that.
And then I've seen that multiple times where you take out that dominant ad from that ad set and all the other ads suddenly bloom and you're like, "Oh, they were also great ads," but because of the other better ad or however you want to call it, which is not necessarily better ad, as well, because sometimes, Meta called the shots way too early, no conversions, nothing, it doesn't even give a chance. So, in those cases, we don't call that creative that got all the spend successful and all the rest bad labeling. I think this is not universal and this is really, I think, case by case basis. I've also seen ad sets with 50 ads live, "Oh, let's see what algorithm likes." And we will run it for a week and then we will see what happens. And then what happens in that week, 90% goes to the same ad and because they have this rules of thumb, oh, we run it for seven days and at certain, I don't know, daily budget, but then that's useless when the 90% of the budget goes to one creative. That's not a test.
You should, in those cases, for example, break the rules and like, "Hey, look, this reached my threshold, this is my decision threshold. Now, I want to see what other ads are going to perform for that minimum threshold that you decide based on your cumulative spend and cumulative performance over time." Yeah, so that's why there is no one rule that is valid for all, but I personally try to get spend on the other creatives if I see a dominant one. I believe that every creative should have an equal chance. So, I democratize a bit within the ad set. And because I've seen it countless times that the decision that made by Meta was way too early. Yes, we trust the algorithms, but we want to guide the algorithms. When we see a 80%, 90% of a dominant asset, let it be a creative, let it be a country and such, you're not fulfilling the potential of the other variables. And I think this is important.
For example, if you put the US with all the other smaller, let's say European countries, there's a 90% chance that at least 70% of the spend will go to US because of its size and everything. Also, you should never do that anyway. You should always tier by CAC or LTV depending on what you're optimizing towards and ideally, without those dominating variables. And if there's a dominating one, then you can always take it out, test it separately, give a chance to the rest because it's not meaningful to have, I don't know, let's say just a couple of thousand dollars spent with a really good CAC, really good ROS. Well, if you cannot scale that and then you try to scale by increasing that existing campaign's budget, but that country that you wanted to actually scale is not increasing its spend necessarily because if it's getting dominated by the other bigger countries or based on whatever logic that's depending on your setup.
So, in those cases then, yes, you can experiment with the value base, the rules, and stuff like that, and that we can talk about later in the testing and the learning side of things.
David Barnard:
What's the tactical move then there when you do have that "winning creative" that's getting 90% of the spend, is it better to pull the other creatives out because that ad set already knows how to target that specific ad to the right people, to the right geography, and everything else? Is it better to pull the "unperforming ads" and give them a shot somewhere else or is it better to pull the performing ad into a different set? What's the best way to actually implement that?
Alper Taner:
Depends on whether this creative is in a creative test ad set versus in a BAU ad set. If it's in a BAU ad set, then I would create another BAU ad set and then I would include those creatives that were under delivering into the new one and I wouldn't touch something that is working well. But if it is within the creative testing ad set, and if it already reached my threshold, then I would already pause it and then I would put it on the bench to get ready to upload to BAU ad sets whenever the team will do the creative refresh. So, it is case by case basis, but in general, the rule of thumb is don't touch something is working well, but depends on what case we are talking about.
David Barnard:
Yeah, that makes sense. I did want to step back and not to put words in Eric's mouth, maybe his point is more to not overly focus on a single hypothesis for why a creative won. So, going back to the whole, should you learn from creatives, because I do want to dig a little deeper into how you think about what you can learn. And the example he used was you look at a creative that has a dog in it and you're like, "Oh, people love dogs." And so, that's why the creative one and you maybe get overly focused. And then to your point, then you go create 200 new assets all with dogs, but maybe it wasn't the dog. Maybe it was just that it was something cute and a cat one would perform just as well or a panda one would perform even better or ocelot or like who even knows, right?
How do you think about forming a hypothesis around that? Because it seems like that's really the important step, is not being overly confident in why you think something won, but instead just allowing it to help you form a hypothesis for other things to test based on a multitude of reasons why that creative could have won.
Alper Taner:
Yeah. You should never just produce 200 variations of just from a one signal. I think it's always in phases. So, let's take the dog example. Okay, let's have the same dog, let's have a puppy, let's have another breed, let's have another animal. So, just to test and see if it's the animal or not. So, what we want to understand is what makes the difference? Is it the dog? Is it an animal? Is it something else? And once we figured it out, then we would slowly roll out the production and so on. So, it's never about, "Oh, the dog won. Now, we need 200 radiations of the dog." It's more about, "Okay, is it a dog?" We want to validate that. So, in every step, we want to do that and we want to do in both directions. That's why we have both iterative concepts and the radical concepts.
Iterative would be, "Oh, let's put another breed, let's put two dogs maybe, and it's even more effective." But in the radical one, you can have completely different hypotheses than what doesn't work based on what works today, but also based on other sources. So, when it comes to what creative to test, I think you should utilize both the first party data and the third party data. It's not always about the first party data. So, first party data, meaning your Meta creative output. That's one variable, then that's a valid, let's say, input. But you also have, let's say for particular copy, or let's say if you're testing particular, let's say visuals and such that also exists in your app. Okay, what is the most popular thing in the app that is more valuable for the users? You basically look at within your dataset, you look at your within your in-app analytics, what you can learn, what resonates with users.
You look at your high LTV users, why do they stick around? What do they use the most? What messaging resonates with them? And then because our goal in performance marketing is to find more of those high LTV users. So, testing test results from test campaigns is one variable, but in-app analytics data is another data input that we can utilize. And then we have the third party data where we have competitor concepts, what works for competitors, we have overall trending memes or concepts or audios and such. And then you could have category level concepts. They're not really direct competitor concepts, but other big players that are doing well within your category. What are they doing? How do they advertise their product? So, what do you can learn from them? So, then you bring and merge all these datasets together. Then you decide on, okay, you know what?
Of course, you give a lot more weight to the Meta test results, but it is not a 100%, "Hey, this is it guys, dogs, let's go." So, it is about merging that. So, when you hypothesize the next iterations, it also should come from what worked and what didn't work, not only for you, but also for others. You can check the Meta ad library to see how long competitors are running a particular ad and how many variations of that. If there's one concept with 15 variations, and that is a particular, the only one that has 15 variations compared to other ones having just two and three, you should be able to interpret that. Likely, that creative is performing well for them. That's why they're exploring more angles. So, it is important to analyze and put enough resources and attention to the planning part of the experiment exactly because of this way and not just act like, I don't know, growth bro and, "Hey, this worked well, let's do to a hundred of them and so on." So, I think this balanced effort planning and executing matters here.
David Barnard:
How do you think about analyzing competitor ads though? Because to your earlier point, there's a lot of people out there just following the playbook and creating 15 variations because they formed a bad hypothesis around... So, how do you actually dig into these competitor ads and try and figure out if it is actually working for them? And that's something I just see throughout the whole industry broadly is that it's too easy to overindex on something you observe without the data and say, "Oh, this must be working. These 15 creatives must be working because they've created 15 variations." That may have been like a dumb move and you just don't realize how poorly it's performing and they don't even realize how poorly it's performing because they don't have it instrumented well, they're not tracking deeper in the funnel. So, how do you think about looking at competitor data and getting any insights from that when you don't have the full picture of why, what their process is, how good their stuff is instrumented, how smart their people are, et cetera, et cetera?
Alper Taner:
Yeah. So, anything you see online, let's say, should be Omni as inspiration and not treated as this because I've also seen and heard people saying, "We tried but it didn't work, but I don't know, Leather is using it." So, then they justify and then they defend, "Oh, but it works for them." So, as you said, it doesn't mean anything for the brand exactly because of the reasons that you mentioned. So, it should be just treated as inspiration, as another input. That's why I talk about a pool of inputs that will shape together the next iterations. It is never about, "Oh, let's copy one-to-one, whatever comparator is doing. Oh, let's copy this trend." It comes down to how I can customize this to my own personas and in my own ways based on what worked for me before and what didn't work for me before.
So, at the end, this needs to all go in that processing machine that considers the persona of the past performance, the current performance, and all your learnings, and then you test it. So, it's never about, "Oh, compared to doing this. Great, let's do the same." I think that is probably why many people are getting upset. I don't know when they just copied the same paywall or same creative and then finding out, "Oh, it doesn't work for us." Well, why should it work? It's different audience, different funnel, different pricing, different messaging, different promises. So, although for example, on the web funnels, many of them look very alike, especially in the health and fitness space for many years, and then everyone copies from each other. So, pretty much a lot of the health and fitness web funnels are similar now. Some are missing, of course, some, I don't know, smaller elements here and there, but conceptually, they're the same. At the end of the day, you should just run to see. So, they should be treated as inspiration and as a variant in your whole test hypotheses and not as a test hypotheses itself.
David Barnard:
Yeah. And then one other elephant in the room in all of this is making sure that you're actually instrumenting things well and actually testing well and can actually interpret the results properly. Vahe recently had a tweet that I thought was amusingly provocative. He said, I won't quote it exactly, but it was something along the lines of half the people who say the Blinkist paywall works for them just aren't testing it properly. So, how do you think about in all of this experimentation and forming hypothesis and running all these tests, how do you think about even just validating your own process and testing and tooling and that side of things to make sure you're even drawing the right conclusions from the data?
Alper Taner:
Yeah. That's also, I think what we covered earlier, 50% of the issue with something working or not is about how they test it. So, I can totally get why he mentions that because I hear similar stories. They're like, "Oh, we tested X, Y, Z pricing because competitors the same, but our conversion rate dropped and our revenue dropped." You shouldn't take these stuff as the hard truth or your strategy. It should be only as a variant and as a source of inspiration. And in terms of how to interpret the results, everyone has their own in-app analytics tool or their third party tool, depending on what test are we talking about. If you're talking about, let's say the paywall tests and stuff like in-app analytics is the way to go. If you're talking about creative tests and so on, then we are talking about MMP as a source of truth, relatively speaking here.
But if you're talking about incremental impact of a channel, we are not looking at the last-click, obviously. So, that's also one thing that there is not one single source of truth for every answer you're looking for. So, if you want to compare creatives against each other, like MMP could be your best friend, but if you want to compare the impact of UAC versus Meta, MMP is likely not your best friend because one is a push channel, one is a pull channel and the competitiveness of the vertical and such. But typically, UAC is a lot cheaper because of the search and which of those higher proportion of those search is also coming from the brand search and so on. So, you wouldn't do a budgeting decision or forecast just based on MMP and such. So, you might want to rely on incrementality and MMM and conducting experiments around the GeoLift and the holdout and the blackout test and so on to come to such conclusions, knowing your baseline, knowing your seasonality.
Problems get different at different stages. You don't have to worry about incrementality when you're just spending 100K a month, I think. It might matter, but it matters a lot more once you get bigger because a lot of the decisions at such stage are driven by MMP/last-click, which is normal because when you are at that stage, you don't have all the data infrastructure set up to measure incrementality and so on. I'm maybe generalizing a bit, but typical startup don't have all those kind of tools in house or third party to make it happen. And once you get to certain level, I would say 500K+, then these are becoming a lot more important and prominent in decisions because then you start with the yearly budgets and then the forecasts and everything and the profitability starts mattering a lot more than before and so on, depending on, let's say your investor vision, as well.
At the end of the day, you should just stick to one tool for certain decisions while you should also cross check with other tooling time to time just as a baseline. I'm not saying, for example, you should use AppStore Connect as your baseline. No, but if your MMP and Meta are very different, but your App Store Connect, CPP data, and Meta matches, that should give you also some signals. Take it as they're a source of truth that you should be sticking to, but don't take them blindly as this is it like UAC is the best channel, let's double the budget only to see potentially the impact is output is not the double and so on. So, that's why then it comes down to multiple tools and multiple ways of interpreting it, for example, mentality and last-click also conflicts or MMM might be conflicting with day-to-day decisions, as well, depending on at what level you do MMM or at what cadence you feed the MMM and how you calibrate it.
David Barnard:
Having worked on so many different apps and audited so many different accounts over the years, do you have any horror stories of like the data team's SQL was wrong and for a year they were just like making really bad decisions or the MMP was not hooked up correctly or do you have any like just crazy horror stories from all these accounts that you've worked with or multiple crazy horror stories of just making bad decisions because the process wasn't set up?
Alper Taner:
Yeah, I think story was something like this. They switched to MMP and then they didn't implement the events on the Neva MMP. I think there are probably plenty, but there are also some must avoided all cost type of things such as I remember one project switched MMPs and they contacted me after a month and saying like, "Hi, we need some help. We cannot figure out what's happening." And the issue was so obvious. So, they switched MMPs, they didn't implement any events on the new MMP and all the MMP was still connected to Meta. They still had campaigns optimizing on Meta and this is like a sizable company, but the problem is they have a different team handling everything MMP. They have a different team handling Meta only, silos and stuff. And I was like, "How did you not realize this for almost a month?"
They're like, "Yeah, because it's not our thing and so on." So, this stuff is like, I think on a very extreme level. Other than that, I don't know. I've seen as crazy as someone spend 100K on a creative with $100+ and did not stop it before. I think it was like 90K something didn't stop. I was like, "Really?"
David Barnard:
With $100 cost per install?
Alper Taner:
Yeah.
David Barnard:
Whoa.
Alper Taner:
Relatively, it was not the crazy high compared to other ones, but it was high enough that you should have killed it after $2,000 max and not wait for 100,000. So, that happens when... It's not an excuse, but it tend to happen in large accounts with no automation guardrails. Yeah, let's keep it this way, no guardrails in place. And you're just really focused on execution and day-to-day business and you're missing out on hygiene checks, rules of thumb and all that kind of stuff because you're too focused on testing the maximum amount of creatives and so on.
David Barnard:
Maybe this is another good rule of thumb to come out of this podcast about not following the rules of thumb, but I don't know how you would divide it up, but maybe 80% of your growth team's efforts should be focused on testing and moving fast and all that stuff, but you want to leave 10% or 20% to be continually auditing your processes, continually auditing your data structures, continuing to audit your events, structures, and all of those processes that are in place to make sure that you don't end up in those situations.
Alper Taner:
Yeah. Another example I just remembered is on the positive side of things, we were doing creative testing and then we did find the winner at the end, thanks to interpreting the output and iterating on those output. So, it actually works when you do the iteration and find more winners. And we were able to, I think 5X the campaign budget within two weeks with the same and even decreasing cost per purchase. And we were not applying 20% increase per day. If you see that your daily budget is 2K, but I'll go spend 3K and you get the same performance, I was like, "Let me handle the bid and budget for this campaign because I see something, I see something very unusual." And I was so excited about it that we literally went hockey stick and the cost per purchase literally on a downhill and that's like any UA manager's dream.
And in those cases, but also the budget at some point was spending the double of the daily budget, but it is no problem as long as you have the performance. And talking about as a problem, like there are also companies who put, let's say, our spend on a Sunday as a problem, but then again, it's just a fact. My approach is, "Okay, what do you do for it? Have you decreased the bids and the budgets? How did you try to overcome this 'problem'?" And then they're like, "Oh, it's just the algo decides on that." I'm like, "Yeah, but you can guide the algo. If you put a lower budget, if you have some automation in place, then I'll go be like, okay, they don't want me to spend more and so on." And there are multiple ways to control it in a way.
You cannot control it 100%, of course, but we can guide the algorithms in that way. Another example is the capped inflated budget strategy where we have a million dollars a day budget at the campaign level, which is not a good advice unless you put a campaign spend limits in place. So, as long as you have the guardrails in, you won't be spending one million a day, of course. You will only spend up to your total campaign cap. And we tested this to bypass the pacing function, although we were still using the bid caps and the cost caps. And this was earlier this year when we tested it and it was also recommended by Meta. And basically, it did work quite well because we were able to capture a lot more higher quality traffic. We did not spend a million dollar a day. The spend varied across the days, but we were still able to get a stable performance with such a crazy budget that the algorithm is not pacing the day anymore because it's very large.
Because the way the daily budget works, it divides the budget to 24 hours in a way that, of course, it never does it equally in a way. You always have these fluctuations, the evening times that peaks up and so on. But by putting a million a day, again, always with the budget caps and spending limits at every level.
David Barnard:
Yeah, this is one that could get really dangerous.
Alper Taner:
Yeah, yeah, yeah.
David Barnard:
So, I think if I'm understanding this correct, is it at the creative or that the daily budget would be at the entire account level? Is that where you're setting the daily budget or where-
Alper Taner:
No, it's at the campaign level, just the campaign-
David Barnard:
Okay, at the campaign level. So, you set the campaign level budget to a million, and the reason you do that is that Facebook or Meta, Meta will try and throttle your performance to not overspend in the morning. So, let's say instead your budget was 100K and 100K is what you actually wanted to spend for the day. And the problem would be that Meta's like, "Ooh, they only want to spend 100K. We shouldn't serve too many ads this morning because we know the night's going to pick up. And so, we're going to throttle the morning ad delivery." And so, what you're saying is at the campaign level, you set that to a million, but then you very carefully go through each bid set and set caps there.
But then, how do you balance those bid caps so you still don't end up spending, or maybe you want to give it some freedom to maybe spend 200K or 300K. Yeah. So, then how do you dial in those bid caps where it does still allow the algorithm a little more flexibility and more aggressively spending?
Alper Taner:
Yeah. So, of course, you start conservatively. If your cost per purchase is 50, you want to start, I don't know, around 45 or 10%, 10% to 20% below and you might not get delivery, then you play it slowly. And by the way, this is not for, let's say not for beginners. So, why I'm saying that, because this works well at large accounts with large learnings that they're spending 50K a day anyway type of a thing and they want to see if they can spend more than 50K a day, if they can spend 100K a day, and this is one of the ways to unlock that. And again, with the campaign limits. So, for example, how do you ensure that you don't spend 200K, but you want to spend only 50K? You set a campaign spend limit and then you put a 50K as a, you say, regardless of how good is my performance, I don't want to spend more than 50K.
So, you put that 50K, but that 50K is not stopping your auction dynamics because auction dynamics are based on the bids and the budget. And the cap is, as far as I know, it is another control mechanism that is not going into the whole auction in terms of pacing and the bidding and all that stuff. So, it's just a guardrail. So, that's the way you control it. So, you have the bid as a control mechanism and then you have the safety mechanism by setting up a max limit. And then you want to spend another 50K, you're happy with the first 50K, then you increase your campaign limit to 100K. Then you give that room again and then now, you want to spend more, so you increase it. So, basically, you are still in control, you're just bypassing the auction dynamics in this way as an example.
And we tried this for multiple accounts and we had some great results. It didn't work for also some. They didn't get delivery or when they get delivery, it was too quick, too much, and then they had to start at a 10K and 20K and so on per day and so on. I don't know what happened later. I know the cases that it worked and then the cases that didn't work, I didn't follow up on the latest status because I'm also busy with my full-time job. But yeah, there are many this kind of things you can test. And this is, of course, one extreme. You should not test stuff only for sake of extreme testing, but there are simple stuff as the value-based rules. You can test those bid modifiers that exist forever, but now, it used to be available only through FMPs. Now, it's available within the UI.
If you have performance difference above 20%, 30% among the, let's say the placements or demographics and such, it is worth testing it. And yeah, so there are a lot of tests to do across all channels. So, when you want to look for room for growth, you should look at both vertically and horizontally. So, horizontally will be like at the channel level, okay, are we really maxing out at every channel? Do I have the response curves of all the channels? Can I really double on this channel or rather spend X percent on this channel and Y percent on another channel? This usually will come from the MMM output because MMM will consider all your various sources of data, although mostly it's looking at the impression spend and conversions and such. But it does know this is now there because you upload your last 24 months data and such.
And then when I talk about the vertical side, it is about, I don't know, on Meta, I gave already a few examples on, we talked about the audience testing, we talked about clustering based on CAC or LTV, depending on how we optimize, we talked about the value based. There are like 50+, I think, tests that you can do, but the prioritization of this test should come from your current data. And then on UAC, it should start as basic as, okay, what are you bidding on? Are you bidding on Firebase events or your MMP events? Are you still optimizing towards an install or are you optimizing towards a lower funnel and have you tested excluding your brand keyword? What is incremental value? So, at the different stages, so starting from... So, I think we talk a lot about the recent examples were more about large advertisers with large budgets where they can invest into in-house teams to build all this MMM or get a third-party tool to basically utilize the outsource the MMM.
But at a small scale, you can also just turn on and off. Large advertisers cannot afford to do that, but if you're a smaller advertiser, like if you're just spending... Okay, I don't want to number now because I said small advertiser. Let's say if you can afford to turn off the channel, regardless of your size, let's put it this way, if you can afford, just do it. If you cannot afford, then you have to use this kind of either the GeoLift or the blackout or holdout or there are tons of scientific statistical studies you can run for the sake of incrementality and such. But the most basic one is sometimes switch on and off and then observe the outcome. And again, there are no one right way to do things. It really depends on your size. It really depends on your markets vertical and such.
So, I know some of our folks within our industry, they have great advice that are applicable to most accounts, but they're not applicable to every single account, as well. So, that's why I mentioned at the beginning of the talk, take them as inspiration, take them as like, if you hear something, this didn't work at all, be like, "Great, I will try it, but later." But don't be like, "Oh, we should not even try that." Unless you have also evidence that it will also not work for you or there are the cues. Because at the end of the day, we have to prioritize how many things we can test. And yes, there are 100 things to test. Yes, it's good to utilize someone else's experience instead of you finding out by yourself, it's good to leverage other people's learnings. And there's definitely a good point. But what I'm just saying is just don't take out that test from your roadmap because someone said it didn't work for them, was my main message here.
David Barnard:
Yeah, this is a really fun... I feel like this was also a fun contrast even, or not a contrast, but a compliment to the signal engineering discussion in that, Thomas talked about and the industry's been talking more about signal engineering as a way to manipulate the algorithms, if you will, to coach the algorithms, to influence the algorithms. And then so many things that you talked about today are also ways, especially those last few comments around the manipulating big caps and things like that. The point is there's just so many ways to tinker and tinkering with bids and caps and all those things. And maybe for some focus on creative, let the algorithms do their thing and you get enough performance and you don't need a big team and like whatever, it works. And again, that's your point.
It's like, don't feel like you need to test all of this. Don't feel like you need to try the bid cap. But at some point, maybe you do need to scale and you're hitting a wall. There's a hundred different things you can try to break through that wall. And I think you've shared a lot of really fun contrarian or just maybe less talked about tactics to help break through those walls when you are struggling. When you've done the test and realized you do actually have a problem, it's not just a fact of the business and that there really is a problem, there's like so many different ways to tackle it and just try crazy stuff. So, it's been a super fun conversation.
Alper Taner:
Yeah. I also want to add one more thing to what I said in terms of the testing. Sometimes, people hear something and they're like, "Oh, we heard, I don't know, someone said mapping trial to purchase works better. Oh, should we also do that?" I'm like, "What is your reason? What is your diagnosis on... How did you come up with that hypothesis and why would you change stuff?" So, that's why also there's a misconception on, "Oh, let's just test everything," but that also sometimes contradicts with don't touch if something is working well. And the particular example I'm talking about is event mapping, for example, because you mentioned also signal engineering and stuff, it's also at the basic level. Sometimes, you don't need to come up with specific demographics triggered events. Sometimes, it's also about optimizing what event you map on Meta side. For example, sometimes... So, there are typically three options I see people do.
They either map their trial-to-trial, they map their trial-to-purchase, or they map their trial-to-subscribe to give better signal to the algorithm. And some people just do it purchase because everyone does it. So, it is great as long as it works for you. If you're happy, if you're profitable, if you've been using it for five years, if you scaled, don't touch it. Don't try to change it because your whole learnings is based on that. On the other side, if you're not happy with the purchase optimization because your cost per purchase is almost higher than your LTV, and then if you're not running the account because of that, then it's time to calibrate/try something else.
And of course, this should not be the main thing to look at. You should first analyze the account, understand why things didn't work and what. And one of the levers also could be on the event mapping/the signal engineering side of things because when you optimize towards a purchase where the Meta has a lot of information on and which is great because it goes after people who actually purchases things, but also, you should think about, "Okay, these people are purchasing anything."Just because someone bought a skirt doesn't mean that they're going to try your app. There's also a bit of the relevancy there.
So, I remember, I think it was sometime last year, had an app that had a really high CAC and because they mapped their trial-to-purchase, and then all we did was mapping their trial event to trial and their cost per purchase/the actual cost per trial went down by, I think, around 35%, only with that change. Same creative, same campaigns, just different event optimization. Also, their CPMs changed, as well, because if everyone is optimizing towards a purchase, and then the guy is, let's say this brand is optimizing towards purchase because his AOV is $200, he's willing to pay up to, let's assume $200 versus a cost per trial's worth for your business is $15, let's say, or $10. How are you going to compete with the guy who is willing to pay 200?
From the auction dynamics point of view, that guy is spending more and the way to give the autobid a signal is you increase your budget, you indirectly increase your bid when you have the autobid, when you don't have the bid cap and cost cap. And then that guy keeps increasing and then you're just getting crushed and he's maybe getting the highest intent and you're getting the lowest intent purchasers and stuff like that. So, there's a mismatch on what do you want to get and what you tell the algorithm. And then on the other side, when you map your trial-to-trial, you're honest about your intentions and the app because the only way for algorithm to know your business are the events you're sending to your events manager, to your Meta events manager. So, also the events that you map should also logically make sense and should ideally have a funnel logic, as well, that I've seen the other day, someone mapped subscription cancellation to add to cart, trial converted to something else, and there is a subscribe event, purchase event, there's everything.
So, for Meta, what is this? Is this a e-comm shop? Is this a subscription app? Is this whatever? And all these advertising businesses that are worth billions are built on machine learning because machine learns from all the signals. And if you send this wrong signals, that doesn't make a funnel, and then the algorithms are calibrating the whole auction dynamics because they're optimizing their own ECPMs with likelihood to convert based on your typical funnel and typical behavior. These are my personal opinion, but I'm just interpreting a story here. Algorithm is not confident to show that advertiser's ad to that user because there is much clearer signal on another advertiser's funnel based on this user's historic activity that is known to Meta. The Meta algo will rather show advertiser's B's ad to that user and not A, because he doesn't know if this user will convert or not.
And he knows that if this user will not convert on the advertiser's A's ad, he will not increase his budget and Meta optimizes towards the spend/the advertiser value. And for Meta to work at optimal level, signal engineering is one thing to improve what you're sending, but fundamentally, you should not send mixed signals for the sake of what you're potential thinking as signal engineering, as well. So, in general, keep it simple, keep it logical, don't send all possible events because you can. I think that's what also people do. "Oh, why don't we track also this because look, we can send this event." It's difficult to quantify what I just said, but it is just talking to Facebook product managers, knowing how this platform learns from the data, learns from the signal, and so on. This is my interpretation of that. And I see that the simpler accounts also do better because of that.
It used to be different, you had 50 types of campaigns and so on. Now, you have the broad interest and the lookalike that is all you need. Of course, there are more, but I'm just simplifying past versus now because algos know exactly whom to go after based on the signals that it gets and based on the signals that it has on the user. So, the signals that your app has is, if it's totally different than what users engage with all the other apps and all the other businesses, then there's a mismatch and that's a whole business model of Meta. It will find exactly the user. It learns from the users that it showed impression to. It gets the conversion as a feedback, and then it will try to find more similar people. So, when you send this mixed signals, mixed funnel, that doesn't make sense from the auction calculation point of view, then you're also potentially losing out on these opportunities.
So, having said that, if you're happy with your current event mapping, don't try to just challenge it for the sake of challenging. If you are unhappy and thinking there is a room to try something different, I would suggest create another event that gets triggered at the same place. For example, the trial start, add another trial start event in your MMP, then map that to, let's say to subscribe, then run a split test purchased versus a subscribe to see which event is more resonating. Of course, the history of each will not be the same, so I would at least wait a week or two so that the events manager will build some history and understand what does this event mean for their algo and so on. Again, it's difficult to quantify, wait one week versus two weeks, but it's just to build some data history and then run the test and then see how it is.
And that's exactly what we did for this particular project. And then we saw a 35% improvement just like that. So, sometimes, you shouldn't think too complex on, okay, but maybe they also come back on day three, then I will fire this super high LTV in predictor event that you can also do that, but also just step back, zoom out. Okay, what am I telling the algorithm? How does the algorithm know me? Is the algorithm X, Y, Z? So, it should start at this basic level and so on.
David Barnard:
Yeah. Which was to your broader point that there's so many things to play with, but don't break things just because people say you should try and break them. Form a hypothesis for why this would perform better and then systematically test it versus assuming because everybody said signal engineering is going to work that you should be doing that exact form of signal engineering. So, yeah, it's been so fun chatting through all of this. Anything you wanted to share as you wrap up? I know you do take on contracting work, but you're also working on your own app studio. If people did want to get in touch or are you taking clients or anything like that?
Alper Taner:
Thank you also for your insights and always happy to help, happy to chat when it comes to MarTech, when it comes to UA challenges, when it comes to attribution, CRO, ASO. That's what I've been dealing daily for the past 12 years because of my main project and always happy to chat. Always happy to learn from you, as well, and learn from the community. So, that's how we all grow.
David Barnard:
What's the best place to reach out? Just LinkedIn or what's the best place to get in touch with you?
Alper Taner:
Yeah, LinkedIn would be.
David Barnard:
Cool. All right. Well, such a fun conversation. I really appreciate you joining me.
Alper Taner:
Yeah, thank you so much.
David Barnard:
Awesome.
Thanks so much for listening. If you have a minute, please leave a review in your favorite podcast player. You can also stop by chat.subclub.com to join our private community.