​

[00:00:00] Phil: What's up, everyone. Today, we have the pleasure of sitting down with Ron Jacobson, co founder and CEO of Ruckerbox. 

Ron started his career as a software engineer before transitioning to product management at AppNexus, where he ran the platform analytics team and later the real time platform product team. He then took on the entrepreneurial plunge and co founding, uh, Rucker box, also bootstrapping it first as a programmatic advertising platform, then a multi touch attribution platform. And today they've added a suite of marketing measurement tools that also leverage marketing mix modeling. Ron, thanks so much for your time today.

It really pumped the chat.

[00:01:02] Ron: Yeah, no, excited to be here. So thanks. Thanks for having me on.



[00:02:54] Phil: I've jotted with a few guests who have a not so unbiased bone to pick with, uh, MTA multi touch attribution. And, uh, I think. Everyone kind of understands it's not perfect. We're not trying to reach the perfect attribution model. Um, but the issue I've had with trying to conquer this at a couple different companies is causality. Attributing revenue to a certain touch point for a lot of people means like this touch point caused this much revenue, but MTA doesn't actually answer that at all, right? Like at a high level, do you agree that MTA by default doesn't focus on causality and should be viewed more as like a. Credit distribution mechanism, not a causality mechanism.

[00:03:36] Ron: Yes, that's my very current answer. It's actually funny because I'm almost wondering if you'd like to listen in to some of our conversations in Toronto rocker box, because that phrase credit credit distribution mechanism is like literally Yeah. What my co founder and CTO Rick, who's beyond spectacular, has been saying to everyone, the rocker box for the past five years.

Um, that is the right way to think about MTA. It's not trying to determine the causal impact of marketing. Uh, it's more of a, let's look back retroactively and try to, um, distribute credit for marketing across different touch points. Now, I think stepping back, it really comes back to the question of what question are you trying to answer and then trying to figure out what data or what methodology can help you answer that question.

Because there are certain questions that MTA lends itself to very well, even disregarding anything about causality, even disregarding anything around credit distribution or redistribution. If you just try to think about it from the perspective of what could I do if I better understood how customers are engaging with my marketing?

That's how you should think about MTI. Um, and that, that's really how we advise clients to think about MTI. Um, a good example is like we have clients who will test into a net new channel. Um, OTT, CTV, linear TV. Um, and oftentimes they don't even know if it's doing anything. They're like, hey, is anybody seeing this?

Is this engaging with anyone? And that's a fair question to ask. I've had enough clients back in my programmatic days who had spent a lot of money on media that literally was being served to bots and fraud. Um, so even just wanted to know, Hey, this new OTT CTV campaign, I'm logging into Google Analytics.

I see nothing, um, because of a GA flaw or rather gap. Um, but that's a nice thing to look at. Hey, I actually can know for a fact it is on the path to purchase for some customers. Now, my idea that that's a small example, but I think that kind of starts to show how and where does understanding the path to conversion, um, fit into, um, kind of fit into the day to day operations of marketing teams, even if it isn't causal, even if it isn't, um, if it is primarily from at least a credit perspective.

Credit redistribution.

[00:05:38] Phil: Gotcha. Yeah. 

Let's chat about that. Like the value of that path to, to conversion. Like I, I get the idea of like building this fractional waiting to assign credit to each of those touch points to like get an understanding of visibility, um, that like led to a conversion. But like we said, the, the causality element there, like limits me understanding.

The value of it, because we can't say for sure that any of those touch points caused a conversion. Like if someone clicks on a Google link and converts, did that click cause the conversion sometimes for sure, but other times we know that it doesn't and maybe like the social ad they saw yesterday or the email they opened last week, like it prompted them to do the Google search and then click on that link, like we need to really ask what prompted the search in the first place to understand what caused you to like, Then decide to purchase. And without knowing this, like we aren't really understanding causality. We're simply observing sequences of actions like behavioral analytics without like grasping the underlying motivation. So like, what's the value of this path to conversion? If we can't say for sure that the conversion wouldn't have happened without that touch point in the first place.

[00:06:51] Ron: Um, I go back to, well, it's actually funny. Alana used to say to our sales team, um, when I was getting them chained up in rocker boxes, uh, when you're pitching MTA. And again, we have a whole suite beyond that, which I know we'll talk about. But if you're pitching MTA, pitch it without even thinking about the model.

Throw the model out. To me, the model comes after the fact, and it's a different lens from looking at last touch or first touch or even weight. And I think when you start to compare it to those models, we can talk about the virtues of it relative to using a different model, because that is frankly the default that brands are coming to rocker box with.

I'm still working with companies that are using last click internally. Um, and that's not causal either, right? Um, to the question that becomes from a path to conversion based modeling perspective. Yeah. Is last touch the best way? The best one to use versus 1st such for even weight versus a modeled approach.

I would argue with last. That's just not the best approach of all of those. And we could talk about it even later. Milder or better or worse. I think even way to modeling actually are pending on the media mix. Sometimes not not as dissimilar as 1 might think, but I would view from that perspective in the sense of, hey, the model is just.

Better way to think about, um, the credit redistribution than the last touch. But again, let's even remove the whole modeling out of it. And I think that's where if I'm in your shoes or any market issues, I would start asking this question of, um, or rather I'd present this kind of perspective. What questions can you answer?

By having a path to conversion. What about things like time to conversion? Uh, what about things like, um, path to conversion for net new customers versus retained customers? Retention has become a huge part of, um, well, it's always been a huge part, but especially post interest rates rising, um, it leads to a much more profitable way of, you know, building a business.

So looking at understanding what is the non paid media path that's leading your existing customers to convert? Um, does adding a new channel, Decrease or increase the speed for somebody to become a customer. Um, it's those sorts of questions that we find to be really valuable. Also, it's As an underlying data set, I kind of think of like the primitives.

What are the core data sets that you need as a marketing organization to do your job? Um, and a path to conversion is one of those. I've got even say beyond that, um, uh, full paths, not leading to conversions is one of those. Then you can start to look at conversion rates. Which is actually important as you're trying to understand the impact of marketing.

You can also then look at session data set. Um, so conversion rates, session rate, uh, session to, uh, sorry, I said session. So conversion rates, um, are different KPIs. But again, no, is this answering the question of, Is that dollar you spent incremental? Is this, uh, you know, would this person have converted with or without the marketing?

No, it's not.

[00:09:27] Phil: Yeah. Yeah. 

Let's, let's define incrementality then. Cause like, I'm, I'm curious. What your perspective is on this. And if you think mine, mine is wrong or kind of flawed there, but, um, my favorite way of defining incrementality is like business results for marketing campaigns or channels that wouldn't have happened otherwise.

It's kind of a flip on how most people think of reporting on marketing results, right? Like attribution is typically influenced revenue or influence pipeline that is attributed to. To marketing campaigns or channels so that we can pinpoint a certain dollar to a touch point or, or one of these channels, what's the difference for you in like incrementality?

Obviously, we talked about the causality there, but like with MTA, like when you have customers coming to you that are just like. We need MTA to like answer these questions, knowing that you guys added a bunch of other features to the MTA platform. Obviously you've kind of like expanded your philosophy or like some of the theory behind that.

Like how do you tackle both of those in a conversation with a Yeah.

[00:10:34] Ron: on the incrementality side to me. It's, um, would that dollar revenue have happened without marketing is kind of the way I think about it. Another way to think about it is. We need some sort of a baseline, um, that deserves to get some credit, credit for revenue, credit for conversions.

Um, and if you don't have a concept of a baseline, it's as if everything was driven by marketing. The only final perspective I would say about it, it's sort of like from more of a marginal perspective, that last dollar spent, um, what was essentially the marginal return on that, because you want to make sure that last dollar spent is still profitable as well, profitable against whatever goals you're trying to achieve as a business.

Um, now, I think where this becomes interesting, though, is, um, you know, they're gonna talk more about this, but different, different measure methodologies, some that are more geared towards uncovering incrementality for the most part, I would say the gold standard. There's testing medium. It's modeling is not.

It's not designed to determine incrementality, but I think when you start to look and understand the pros and cons of every methodology, you really start to understand. Or you should view it from how do I leverage the probe one to sort of augment another going back to the MTA conversation, um, causal or not, incremental or not, um, on a day to day basis, marketers need information to make decisions, right?

They need to know my upping or decreasing my budget in Facebook. Am I going to move some Facebook spend to medics that are etcetera? Um, something that we invest in was you call it, um, essentially, uh, I forget the exact, I think we call it credit, um, credit redistribution, which is funny because of the name that you said, but essentially allowing clients to bring in their own external waste to rockerbox and apply it as multipliers on top of our reporting.

I think this is really important because if you go and run a geo holdout test in Q1, and you know, it's a week later, a month later, a quarter later, you, you likely, well, we can talk about the testing, but. If you have the ability to apply the results of that test on top of the MTA reporting that Rockerbox provides, you have the ability to kind of get the best of both worlds, right?

You can take the incremental modifier, uh, multiplier, use that on a day to day basis on top of an MTA based reporting, and use it for in platform day to day optimizations as well. Um, and then kind of rinse and repeat whenever you're going to run that next holdout test or a scale up test. Um, so that's kind of how we see those 2 playing with each other.

Um, but it really just again comes back to, uh, I don't, I think the problem everyone has is everyone tries to figure out, like, what's the best methodology. And I think that's the complete wrong question. Uh, the right question is. What are you looking to get answered and how do you answer that? Um, and by the way, sometimes it's not even methodology.

Um, I go back to the example of if you want to answer, is this channel getting in front of new customers? There's really not a methodology that's there. You know, there is a path to conversion component to it, but the modeling part doesn't matter if you're trying to understand how do I make the best, the best, uh, best budget for the next quarter, that's very well suited for MMM.

If you're trying to say, you know, should I increase or decrease my spend on this creative today? MMM or incrementality testing are not going to help you at all. So it's kind of just start with a question and then figure out how to answer it versus starting with like these supposed answers and trying to work back into questions.

[00:13:42] Phil: Very interesting. 

When customers come to you with these questions, like, is there ever a part of you that's like trying to figure out why is this customer focused on this one question when they don't have the answers on these other basic building blocks yet? Like, how do you steer those

conversations to like, Hey, instead of focusing on like Time to deal or like any of these other vanity metrics or questions.

Let's, let's focus on the building blocks first. Like, how do you handle that with customers?

[00:14:14] Ron: so sure answer is no. Like I'm actually happier if somebody comes at least with some specific. Uh, data point or, uh, kind of hypothesis or hypothesis, they're looking at approved or disproved, even if in my mind, that's not the most important thing to approve or disprove. I'll hopefully talk to him about that.

I'd be like, Hey, why are you focused on, um, uh, you know, any metric, you know, uh, increasing brand lift, um, when, you know, your profitability or business is going down or like, I'm happy to have that conversation. But listen, at least they're coming. With like a specific problem they're trying to have solved.

And then we can kind of help them solution from how do you get, uh, the right data to be able to answer that question, the bigger problem I had. Uh, and I hear this all the time. Phil's clients come in and say, we ask, why are you, you know, why are we talking today? Um, uh, my, uh, I need better attribution and that means literally nothing.

It literally could not mean less. I don't blame anybody for saying that. I'm not trying to disparage them at all. I get why they're saying it. But, um, you know, it's, it's kind of like going to, uh, uh, a trainer and saying, you know, why are we here? Um, I need, I need to get fit. Well, what does it actually mean?

Like, are you trying to lose weight? Are you trying to get faster? Are you trying to lower your heart rate? Are you trying to live longer? Like what, what, you know, trying to lower your cholesterol? Like, what are we actually trying to do? And that's what we have all the time. My, I need better attribution. And that requires a lot of questions from us to really uncover what are they trying to really get at.

So we can help them figure out the right methodology. Maybe it's, you know, based on what you're saying, but you think that you need to have a combination of MMM for your wholesale, uh, probably some MTA for your DTC and down the line, you probably want to have a warehousing solution where this all gets into your data warehouse for the analytics team that you're going to be building out next year, like we can, we can kind of work through a right, um, uh, mix of our product suite for them, but yeah, that, that's a hard one.

Um, my attribution is broken, um, is a, is a question I, we hear way too often.

[00:16:12] Phil: We need better attribution. I feel like stems from some of the, I guess, like issues with MTA or like, there's so many different platforms trying to tackle some of

the harder things with MTA, right? Like,

[00:16:26] Ron: I would even, I would even push back there. Most of these clients aren't coming after having MTA. So like most of these clients don't even have any, let's even step back further. How are most clients coming to Rockerbox? There are three situations. One, um, they're using platform numbers. They're using the platform reported numbers.

Two, they're using Google analytics. Um, I guess a 2. 1 would be a combination of platform Google analytics or three. They've used a different measure methodology that doesn't work. And now they're coming to rocker box. Um, those are the three situations. The one where, um, we already have an MTA solution in house and it's not giving us what we need.

So come to us for MTA that that's significantly less rare. Um, or rather more rare. Um, there are actually reasons why that can happen. It could actually be pretty successful. I can say, Hey, we built the last touch MTA in house. And I'll be like, well, you're spending a lot of money on. Pinterest and Snapchat and tick tock.

And there's a lot of user attribution there, which is not getting into your model. And you're also investing in direct TV and OTC TV linear, which is not going to your model. So if you want to be doing MTA, we can help you build a much better path to conversion. Um, so that does happen often, but most of the time people are literally just starting from Google Analytics and platform reported numbers.

[00:17:33] Phil: One of the two, like, I guess, issues with MTA or like common critiques is this idea of offline channels and walled gardens and like most MTA, at least that I've seen, like, the common ideas that like, Oh, like all these offline touch points, they're just not going to be part of the model. And all these platforms like social platforms.

They have like walled gardens on all this engagement data, which won't be able to include that in the model. But you've disputed this by saying that MTA can in fact, measure these blind spots, if you view it as a combination of first party identity resolution and probabilistic identity resolution. Can you unpack that for us?

[00:18:15] Ron: Um, I think a lot of that, uh, a lot of the reason people come with that perspective. Thank you. Is because of third party cookies originally MTA, the first gen of MTA companies, Visual IQ, Convertro, Market Share, Adometry, companies you, we don't talk about it anymore, have all been bought out, um, we're all founded around 2010.

Um, they were all founded with people that are experts in programmatic advertising. This idea that you can target people, users one to one, um, and you can track every single impression with third party cookies. Amazing. You can build a perfect path to conversion using third party cookies. That was, uh, I call it the original sin of MTA.

This notion that third party cookies is the connecting pin of all the different marketing touch points. And I'll tell you why. I mean, you actually just told me you just did it. Um, there's no third party cookies for linear TV. Okay. There's no third party cookie for direct mail. There's no third party cookie for radio or for podcast, um, or for, um, social channels.

I was pausing because there kind of was, but there really wasn't. So even this like original way that all these companies, by the way, had really nice exits now comes, um, Bill's MTA on was on a flawed foundation, third party cookie. If we get rid of that and, and I wish third party cookies were gone yesterday.

Um, and then you ask the question, how would you measure these channels recognizing that there's no third party cookie? Then you have to actually have to start with a different approach. If you take a first principle approach to it, you can try to figure out how would you do it directly from the get go?

Um, let's take walled gardens, um, snapchat, pinterest, tiktok, reddit, um, to name a few. Uh, we've struck partnerships with all of them where we get deterministic. That means actual, um, impression level, uh, log files shared with rockerbox. Uh, with first party data in a private privacy compliant manner that we can connect with our customers data set.

Um, so we've, we've broken down the wall. Everyone talks about this walled garden. It's, uh, I made this joke internally. Uh, we should make some ads about, you know, like the Ronald Reagan, uh, uh, tear down that wall. We, we, we have torn down that wall. Um, let's take a look at it. It's not a good joke, but anyway, I, I think there's something there.

Um, we, uh, Direct mail, right? Direct mail. You send mail to an address. When you have somebody becoming a customer, they purchase a product, for an e commerce setting anyway, they give you an address. Well, that seems like a very joinable piece of information. OTTCTV. We've struck partnerships to get access to log files without their first party data or using IP addresses as a joining entity.

IP, not third party cookie. Um, for channels where you don't have any concept of user level data, take linear TV, for example, classic linear TV. Um, we take a modeling approach, uh, where essentially we get what's called post log files. These are CSVs saying where and when an ad was served. So at 1230 in Trenton, New Jersey on planet earth, an ad was served.

I can then go see. Who came to that client's website at 1230 in Trenton, New Jersey, and we start to build statistical models and determine of those users who arrived, who do we believe arrived from linear TV? Now, when you augment that also with post purchase survey data, saying, how did you hear about us?

When you augment that with promo code data that we can map back to channels, we could do a better job at reinforcing those models over time to make sure that they're really effective models for gauging the channel with linear TV. Um, so I guess the long and short of it is. Third party cookies, wrong way to do it, right way to do it.

You go channel by channel by channel. You figure out what's the most granular data you can get. Um, what's the best, uh, sort of metadata you can get that you can join upon. And if you don't have both of those in a granular way, you have to find a way to model against it. Mind you, this is similar to what Meta and Google themselves, um, sorry, uh, yeah, Meta and Google themselves, uh, have to do these days with model conversions, right?

Uh, their modeling conversion gap as well. So, um, I think it's just where a lot of, uh, a lot of this is heading for, for certain channels. Um, so I'm glad we've been doing this for, for quite a long time, actually.

[00:22:11] Phil: Very cool. Appreciate you walking us through that. Um, yeah, I think there's, there's some cool insights there that, um, folks don't know is even possible when we talk about this idea of multi touch attribution, um, on, on the topic of like, yeah, sorry, go for it.

[00:22:26] Ron: Well, I'll, I'll, I'll say, or just don't do MTA and do medium and small. Like we have that as well. So I'm happy to sell that as well. Mind you, there are flaws with medium and small. And so we can talk about that later, but like, uh, you know, I don't really care which measure methodology you use. I want the one that's best for the answers you're trying to get solved, I guess.

Yeah. Let's talk about it. Now. 

[00:24:25] Phil: Like, do you think it's fair to say that, uh, all three of these methods are useful in certain areas, but they all differ in their focus areas. And, um, I can give you my understanding of it and you can break this apart and tell me how you think about this. But MTA essentially optimizes the conversions journeys, final stages, like tracking interactions, leading to conversion, leading to purchase MMA on the other hand, analyzes like a bunch of historical data.

So not super useful for startups that are very, very early, but it's useful for identifying channels. For companies that have that historical data and strategies linked to business outcomes and adding, um, a ton of insights to like strategic planning. And it like, it's good for guiding test selection, but it doesn't show correlation, like it shows correlation, not causation.

Right. And incrementality testing. We talked about this one already a little bit, and we can unpack that one further. AB tests, holdout tests, like causality by measuring real impact of marketing actions. Maybe give us examples where like each model is best suited for a specific standpoint, having a platform that kind of caters to all three of these methodologies.

[00:25:37] Ron: Definitely. Um, we'll start with me. It's modeling. Uh, you, you're correct. You need historical data. Um, general rule of thumb is at least two years of data, the longer and the more data you can get, the better. Um, so yeah, if you're a company that's a week, a month, a quarter old, MMM is not even worth worth discussing.

I'd also say rule of thumb. You want to be spending at least 5 million bucks a year in marketing before you really start investing in medium. It's modeling. You want to be on at least a handful of channels. Like if you're just spending on Facebook and Google, you probably don't need them also to be honest with you.

Um, but essentially it's, we get all the historical data, all of your historical spend. Um, it could be even things like, uh, and different vendors have different approaches here, but it could be like impression counts for things like email or SMS. Um, you could also get some other exogenous variables. Um, consumer sentiment, inflation, et cetera, et cetera.

But essentially we're doing big, fancy statistical models. Try to take all this aggregate data and determine what's actually driving towards your business cycle is what medium it's models for. Um, main use case that we see from our clients is, uh, in forward looking budgeting and planning. So if you're doing budgeting annual basis.

quarterly basis, monthly basis, uh, weekly basis to, um, uh, we're using medium it's modeling as a method to help guide the development of that budget against a baseline. Even if it's finally can't help you uncover what is a baseline, um, component of your, uh, of your marketing mix. Um, what is it not useful for, um, day to day optimizations, anything more granular really than, Um, I would say channel tactic channel tactic could be like Facebook prospecting.

There are some ways to get a little bit more granular, so like channel tactic campaign, the most granular, but that's, it's a newer component of minutes modeling that is starting to get a little bit more developed, but not really there yet. Um, and generally more used by like the C suite, CMO, CFO, et cetera, et cetera.

Um, incrementality testing, um, meant for determining, uh, is that last dollar spent incremental or not? Ways that it's done is with either geo test. That could be a holdout test. I'm going to serve in two geos or rather I'm going to, um, stop serving in geo one and serve in geo two or a scale up test. I'm going to increase my spending in one geo versus another.

Um, what's great about it is it helps you uncover that answer. Was that dollar incremental or not? Cause you can compare the test versus the control. Negatives about it is there's a real cost to doing it. All of a sudden you're not serving media in a certain location. Um, that there's revenue that could be there.

Um, it's operationally more difficult because you actually have to make adjustments to your campaign structure to set up the test and then to run it for a period of time. Um, uh, it's hard to run multiple tests at the same time, and it's often a point in time answer. So you run an incrementality test for a month in January for, uh, for branded SEM.

That's like, uh, archetypal reason. Uh, that's the main, uh, the first thing everybody wants to test branded SEM. Cool. That result from January. Is it relevant in February, in March, April, May, June, July, et cetera, et cetera, um, MTA, we've talked about a lot in platform optimization, trying to understand, uh, and get insight from a path to conversion time to conversion.

Um, uh, and then also I would say applying results from eminem and medallion test as multipliers on top of MTA again to facilitate that day to day optimization for media buyers is how it was, how we think about

[00:28:57] Phil: Let's talk about this, the testing piece for startups. Like if, if, if we're in a startup, we have listeners that are across different sizes of companies, but, um, I think it's idealistic for startups to strive for like well thought out statistically significant experimentation design, because reality is like, not everyone has the volume of traffic. Or the time to wait for that volume to have detectable effect. So what advice do you have for startups to tackle this? And do you think that like geo based tests is the ideal solution for, for startups in their early days?

[00:29:31] Ron: in the early days. Absolutely. No. In the early days, you should not be talking to any measurement company. Um, don't talk to me. Don't talk to my competitors. Don't, don't think about this. You will know if something is working if you're running paid media. So my, my advice, your early startup, you need to find one thing that works, one channel that works.

And then you want to, um, pump the gas on that as, uh, as much as possible until it breaks essentially. Um, cause it's honestly so hard to find one thing that works. Um, that a lot of startups spread themselves way too thin, um, testing multiple channels at once, and they're never going to find anything that works.

So all the good companies I found have found one channel that works really, really well to start and they've scaled it. Um, as much as they could. So I would focus on there. And this even comes back to my earlier point, if you're, um, if you're spending like, uh, I'll overdo it a bit, but like imagine being a hundred million dollars a year on Facebook.

You don't even really need third party measurement, honestly, like you have one channel, like all you're doing is determining the impact of that one channel on your media, right? So there's no issue with duplication. There's no issue with, um, path to conversion. There's, you know, uh, you can cut the budget by 20 percent one day and see if there's an impact.

So if there's one channel, like you don't really have to do that much. I'm overdoing it because you're spending 100 million bucks a year. Like, mathematically, it's worth it to invest in something. Just, you know, given, you know, little percentages can have huge impacts. But the point stands is like, Uh, if you're on one channel, you don't need third party management.

Um, so going back to your case, early stage startup. Don't talk to me. Don't talk to my, uh, don't talk to anybody, just find one channel that works. Um, I'd say one of the biggest red flags I see time and time again is early stage startups that are coming to a rocker box with, again, that kind of my measurements, not working line.

And as we kind of peel back the onion, what I really realized is their business is not working, right? Um, none of their marketing channels are really effective. They're not profitable. They're not growing. And they come to a measurement company. Kind of hoping for us to be their answer. Like, Hey, nothing I'm doing is working.

Well, you know, rocker box will suddenly find what's working. Well, that, that's not the case. You know, if nothing is working, I can't find something that's working. Um, uh, in a way that that's basically, uh, it's bound for failure. Uh, the analogy I've used in the past is it's sort of like going, I think a measurement kind of was like a scale.

We measure what, you know, how much you weigh or not, it's like going to your scale, um, and being upset that, uh, you know, upset at the scale that like you didn't lose weight, you know, it's not the scale's fault. The scale is just measuring. If you lost weight, you got to go talk to, you know, change, change something else about your life.

Um, it's kind of the same with these startups that go to measurement companies because your business is not working. You want to come to, uh, come to us, come to Rockerbox. Once you have channels that are working effectively at scale, you have a systemized way of running your business, being methodological with, um, uh, not methodological.

Um, have a system for taking data, looking at results and making changes on a recurring basis, then applying measurement on top of it can be gas in the fire, uh, fire on the gas. Um, but don't come before that.

[00:32:29] Phil: Yeah, great advice. 

What, what do you think are like, well, what are your thoughts on companies that like are moving away from maybe what you, not necessarily what you, what you're building, but like this idea of, um, like I've chatted with leaders at Wistia and Ahrefs that like a common theme there is like, they don't do any MD, MTA, like they don't do MMM.

They do very little reporting. Let alone like they don't use Google analytics, even like they take all this effort required for sophisticated tracking and reporting. And instead they pour that into what they think will provide value to their audience, content, videos, experience, improving the product. They see revenue going up, so they know they're doing something right. And so they just like, they don't go down the MTA routes. Like I know this is like. A very niche example. And not every company has this benefit. They're a bit more like PLG SaaS, but like, what do you think about that?

[00:33:25] Ron: Well, just to confirm, they're not using any type of measurements, so it's not even an MTA or MMM or mentality. They're just running their business kind of, um, you know, finger on the pulse of what they think is happening, right?

[00:33:34] Phil: Yeah, the only measurement is revenue. Like they have revenue charts. They see that going up. They know they're doing something well. Like why would I spend a year or less like trying to figure out MTA to assign credit distribution or run all these sophisticated experiments to like try to find new channels?

Like, What we're doing works well. Revenue is going up. Like what if we just stop reporting on marketing for, for some of these companies, that revenue is going up.

[00:33:58] Ron: Listen, they have a very high, they have a very high class lack of problems that most companies don't 

[00:34:03] Phil: Yeah. 

[00:34:04] Ron: I'm not going to sit here and say these people should change what they're doing when what they're doing is obviously working. Um, you know, if you have an ability to grow your company without using any sort of measurement or reporting consistently and repeatedly over time, keep doing it.

Keep doing it until it breaks. That's my line when it breaks. If you can, if you can figure out the next thing that works great. Um, if you can't come, come, let's talk. But, uh, Yeah, I'm not gonna have, uh, I'm not gonna have the ego to say they should be working with rocket box of their businesses growing, growing very well.

Um, keep doing what you're doing.

[00:34:32] Phil: Fair enough. 

You mentioned holdout tests. Um, I want to like unpeel that a little bit. Um, one of the hesitations with even just experimentation as a whole within companies, you tease this out already. It was like, what if. We are not going to be servicing these ads, or we're not going to be doing this marketing activity to this percentage of our population.

And so just to get potential insights on how incremental this campaign was by default, we're missing out on potential revenue because we're not offering this to serve percentage of people. What are your thoughts on delayed holdout tests as a way to counter some of these objections? So let's say we're holding out two or 3 percent of an audience, but we only do it for a certain amount of time, let's say like two or three months. And in that third month, like we look at the data, we store that data snapshot. And then we serve that campaign to that audience that was held out. So hopefully that revenue that we missed out in those three months, then we still have that audience to, to, to surface it to you. What are your thoughts there?

[00:35:35] Ron: I mean, at that point, you should have an answer to your test. So you should know, is this audience incremental or not? Right? And if they are incremental, yeah, go back after the people you didn't serve. If they're not, you don't want to serve the media. So, like, in a way, you've already taken the cost there.

You've already taken, you know, your medicine. There's no point in my mind and not taking the lesson and applying it after the fact. Um, so that would kind of be my approach there.

[00:35:57] Phil: Do you think everything needs to be a test? Like let's say I'm leveling up. Email onboarding email is kind of my channel. I'm most comfortable with. So the first example that comes to mind, um, let's say I'm leveling up our email onboarding sequence from like just four old emails that aren't super valuable.

And we talked to a bunch of customers and we have these like use case segmented version of our emails. They're way better written. They look better. We know the new one is better. Do we really need to test it? Is that where like time based tests comes in? What are your thoughts on that? Like, does any, does everything need to be tested?

[00:36:34] Ron: I mean, listen, I think you have to know your company and know your people, right? Like there are companies and there are cultures of companies that are extremely important to answer this question. Like the classic example is Google with their blue button, right? And they went through like 50, 000, I'm making this up, whatever, 100 different variants that blue button to find the perfect blue button.

Um, so they very much are on one end of the spectrum. On the other side, you have Apple who, um, you know, we, uh, they view themselves as having taste. And they will pick the one that is correct and go with it. So both, you know, 3 trillion companies. So, um, I think it can be very successful on both ends of the spectrum.

I, I feel pretty confident that if you went to Apple and asked them to take Google's testing approach, it would fail. They don't have that culture. If you went to Google and told them to apply their taste, I think that would fail because they don't have, they don't have that, uh, that, that skill set. Um, so no, I don't think testing has to happen.

I think if you have the right people, the right, um, you know, uh, your, your organization understands that like testing is part of how you run your business, then, then fine. Um, but, uh, to your point, there's a cost, there's a cost of testing. Even if, even if you are an organization that is used to testing, there is a cost to the, the time to figure it out.

A time to implement it. Um, in a way it's somewhat also limits what you can do because you want to keep that original test clean and isolated. Um, so you can't run too many tests at one time. It's not fully true, but it's hard to run many tests at one time, uh, without having a kind of leak over, uh, effects from one to another, it's operationally difficult to, you know, all of a sudden you have to communicate really well to the whole company.

So everyone knows the whole company, but. Everyone involved in marketing, this test is going on. We have to wait this much time. Don't do X, Y, or Z. Somebody forgets, you know, uh, and then there's also exogenous effects, right? You can be testing and then your competitors can be adjusted. So it's not as if like, uh, you know, you're the only player in, uh, You know, you're the only player in the game, right?

You have competitors as well. Um, but, but again, like those are some negative testing. I do think companies should do it though, too. So I'm not trying to say like you shouldn't do it because of that. I mean, meta and Google, they all offer free tests. Like there's no reason at some, there's no reason at, at some amount to take them up on that offer.

Does that mean you run a test every single week? I don't think so. It doesn't mean you run a test when you have a specific. Question you're looking to get answered probably imagine you're in a medium X model and the outcome says one of your highest spending channel has, um, uh, very, very high ROI, which is great, but there's a very wide confidence interval.

So we're not really confident at all in that result. Um, now when you just see the high ROI, you're like, God, let me invest. Let me double down. When you see the high wide confidence, uh, band, you might want to say, you know, let's actually pause. Double run a test to validate that and if, you know, if it's validated, cool, we have 2.

2 pieces of data that confirm what we think and we're going to double down investing that channel. Um, so that's again, like all this in my mind starts with what questions do you want to have answered and then actually figure out how to answer it. Um, yeah, I think that's like the fundamental problem with a lot of this.

It's a lot of folks kind of start at the end of like, what's the methodology, um, and try to, uh, without actually knowing what they're trying to get out of it. It's like the companies who like build technology first and then try to find a use case for it versus what is the, what is the problem we're trying to solve?

And let's find technology to solve that use case. 

[00:39:53] Phil: Yeah.

I got one last question for you on, on testing. And then, uh, we can move on to, to some data topics there. But, um, this is kind of close to my heart cause we're, we're tackling some of these issues at my current startup. Um, 

I want to talk about, we're going to tease this out a little bit, but like time based tests and geo tests.

So oftentimes experimentation is thought of as, well, We need to run an AB test and we're doing 50 50. But when you don't have a ton of that data, like we talked about for startups, there are still options like time based and geo based that that would still at some point give you somewhat incremental, um, causality data.

Right. But I'm curious, like, if you have like other versions to throw in this bucket, and if you agree with the The definitions here, but time based essentially, like you said, we're running a campaign for a month and then we turn it off for a month. And this one assumes that everything else is the same during those two months.

And like, we know that that's not reality. Oftentimes there's like seasonality and external factors and concurrent

campaigns 

[00:40:55] Ron: you know, the world is crazy.

[00:40:57] Phil: the world is crazy. Yeah. Um, and then geo tests. Like instead of running your ad campaign to all states or your email campaign to all geos, you pick one segment. And, um, it's a bit better than time based tests, but like, you can still run into like geographic variability and demographic differences.

And the news affects different areas differently. Also, like, when is it okay to accept the potential limitations of these tests and how, how. Cautious. Should we be about assigning causality incremental numbers to like time based and geo tests?

[00:41:35] Ron: So I think we should go one by one. Like a classic A B test is not actually telling you incrementality um, or causality, right? Because again, to your point, sure, you're sending two different messages to two different users. It does not mean they would, they would or would not have converted without that message being sent to them, even if it is two different messages.

So like the true way that you're running a test is by not serving to some users. Um, serving to others and comparing and contrasting the, uh, you know, the performance Delta between the two of those, that's like the most accurate way to do it, all the exogenous factors aside. Um, uh, it's funny. Cause a lot of people talk about geo tests.

Let's find these two geos that are all that are perfectly the same. I mean, that does not exist, but anyway, like, um, but still you should be doing this and I think it's just about being honest about the flaws of these. I mean, to your, to your main question though, it's like, how do you use this in your day to day running a business?

I, there's not a generic answer. Like it depends on the size and scale and stage of your business and what you're trying to accomplish. I think it depends on the potential ramifications of the result of a test. An example is if a ramification of a test is that you could suddenly go and. Double your marketing spend.

If that failing kills your business, you want to be really, really thoughtful about that. If that feeling does not kill your business, that's different. Um, so it just depends on your company, your stage, uh, but your, yeah, the, the, uh, I don't think there's like a generic answer for this. Um, which is why it's so important to have really good people in house to understand your business, understand your business context.

It's why it's really important, frankly, for your measurement providers to understand your business context. A really important thing with like building on our medium smiles, for example, is unless I know your business, I can't build a model for you. A good example is we have, um, one of our clients is in, uh, uh, the golf, uh, golf industry, uh, they sell golf apparel.

Um, and, uh, you know, we, for them, sponsors are golfers, right? And what we realized was it's not enough to understand if one of their Uh, their sponsored golfers is playing in a tournament. It's what type of tournament is a mass is a major or just a regular tournament. That is a huge impact to the business.

Uh, and that was a huge impact to the efficacy of that sponsorship. And like, unless we knew that and understand the actual nuance, we could not build a model. That's actually accurate for their business. So, um, long story short, there's a lot of business context. There's not, uh, it's not a one, one answer fits all kind of situation.

[00:44:00] Phil: know that we both share the sentiments that, uh, 

a big issue in measurement and even in a lot of my AI for marketers episodes that like doesn't get enough attention is this idea of the underlying data, like your data warehouse. And, um, this one, I feel like is a bit further away for marketers because, um, we, as marketers have access to a lot of MarTech tools that have data within those tools, but. As marketers, we're not as close to the data team who is building and cleaning and normalizing data in the data warehouse. And so, like, this is one of the queries that record box focuses on this idea of storing, cleaning, normalizing data, all those, like, building blocks of attribution and measurements.

Like you, I think you called them primitive, primitive data, like

click sessions, conversion, spend all that stuff in your data warehouse. And Walk us through like how this enables customers to be way more flexible in how they decide to measure, uh, their their marketing spend. Um, yeah. Walk us through that, like versatility angle of, of how you tackle that with, uh, data warehousing.

[00:45:10] Ron: definitely. I think there are three. Three components to this. One is, do you have the right data? The second is, is that data properly categorized? And the third is how do you connect the data? Uh, I'll walk you through all of them. Uh, the first one is, do you have the right data? This goes back to that walled garden question you asked before.

If you're investing a lot of money in Pinterest and, uh, it's a very nice, beautiful video ad that you know what people are going to watch and they're going to get seduced by. What was that? Like sun kiss, I guess this is the Snapchat ad, but the guy in the, you know, the Skateboard, like great ad, right? Um, but if you don't have any data set speaking to, uh, which users actually view that ad, you can't measure the impact of it from a, uh, just full stop.

So, like, do you have the right data set? So, I think that's working with partners, Rockerbox in some cases. We've already done a lot of the heavy lifting to strike partnerships to get access to that data. Um, but there are other data sets as well. Um, did you want to make sure like you even have everything coming in top of funnel to your warehouse so you can start to do all the subsequent components.

Other parts of there are post log files or direct mail files, et cetera, et cetera. Next part is, is it properly categorized? We see this all the time with clients. They call. They use multiple different terms for the same thing. In Google, it's prospecting, in Facebook, it's upper funnel, and in Snapchat, it's branding.

Um, but they all mean the same thing organizationally. And unless you know that and can, you know, uh, kind of re rename them or join them together. Uh, you're going to be misreporting on your marketing because it marketer should be looking at what's the performance of our upper funnel encompassing of all those three different things, but, um, huge, huge problem also then kind of little problems of like, uh, I launched a retargeting campaign, but I spoke retargeting correctly, so that's suddenly not going to be categorized correctly.

Um, you see this all the time. So again, do you have the right data? Um, is it categorized correctly? And then how do you join and connect it? This is both. How do you connect the data within itself? So how do I join that direct mail ad that direct mail campaign with a conversion? You know, we do this. This comes to the concept of identity resolution and join keys.

There's also joint keys between marketing data and your own internal company data, which is super, super important. Fine. You get data from a marketing provider in your market provider in your warehouse. How do you connect it with your non marketing data? How do I connect the path to a conversion for order ID number one to five?

With the actual P and L for that order. Well, you have to have joint keys, either an order ID or product ID, et cetera, et cetera. So, um, we think about this nonstop at rocker box. In fact, we've, we, we basically split our business into two different product clients and data product line, which is just focused on this.

And then our analysis product line, where we actually start to get into MMM and MTA and in platform optimization and all those different types of marketing analysis. And we did this because we realized it's so important to really think of it as two separate problems. Um, And to frankly not give them necessarily equal weight, but to try to call it out of market.

If you don't have proper, proper data foundation, if it's not in your warehouse, um, you're, you're really going to be disadvantaged long term.

[00:48:09] Phil: Love that you guys are focusing on that. It's, uh, obviously. Not as fun of a topic is like, show me a dashboard and let me visualize what is driving revenue for my company is like, well, let's, uh, let's take a step back here. Do you have all the data you need? And are you joining that data properly? Do an idea rise?

Like, it's not a conversation that marketers are as comfortable with is like, uh, what colors can I add to my bar charts on this dashboard there? So I love that you guys are focusing there.

[00:48:37] Ron: I'll give you a really good example. Like one thing that gets prospects of rocket box excited is that we can use post purchase survey data and promo code data to help measure your marketing. Imagine people come to your site direct. But then in the post purchase survey, they say they heard about you via TV.

That's great. Let's give some credit to TV. Um, now, if you don't actually have that post purchase survey data, in a way that, you know, in a way that you could pass to your measurement providers, or in a clean, organized manner, it's almost as if it doesn't exist. Um, if you don't have that propagating to GTM, so you could pass it to third parties, it's almost as if it doesn't exist.

There are other ways around this, but like we've, we've seen this time and time again where, um, you know, there are all these things they want to do with measurement, but they don't have their data house in order. So we actually have to help a lot, honestly, consulting clients to get their, their internal house in order.

Before we can even start to do our job for them.

[00:49:22] Phil: Makes sense. Ron, this has been a super fun conversation. 

I got one last question for you. We asked this one to all of our guests. You're a co founder CEO. You're also a father and a former avid runner. One question, like I said, that we ask everyone is like, how do you remain happy and successful in your career?

How do you find this balance between all the things you're working on while staying happy?

[00:49:45] Ron: I don't think I find a balance. I think I struggle with it a lot. Um, uh, at the end of the day, when you start a company, you end up thinking about it all the time. I could be watching my kid play in the back of my mind is rocker box. So I'm not, uh, it's something I struggle with a lot to be honest with you.

Um, I will say, um, having a. A young child is, uh, it's a great, it's a great realization that, you know, he, he doesn't care if I had a good or bad day, he's going to cry when he's going to cry. He's going to want food. He's going to want food. So in a way, it's a, it's a, it's a nice forcing function. Um, but it's really hard.

I will say running when I, when I find when I'm diligent enough to do it consistently, um, is a huge, uh, huge help for me. Um, I have times where like, yeah. I'm having a bad day, whatever it is. And I just like, feel, I feel it in my body. I'm just like, this is, uh, something's not good. I go for a run, I come back, everything's a lot better.

So, um, that's, that's the one thing I do. And I wish I could do a lot more, a lot more of it.

[00:50:37] Phil: Yeah. Great advice. Yeah. I, uh, I definitely empathize with, uh, having a, a young kid, uh, at home and they, they live on their own schedule and, uh, it doesn't matter, uh, how good or how bad your day was, but they're always there as much as possible to, to make that day better at the, at the end of the

day. 

[00:50:55] Ron: great. Listen, they, they, they are great also. Um, but they, uh, they are needy.

[00:51:02] Phil: No doubt, Ron, this has been super fun. Uh, I'll send out links to record box, obviously.

Um, the blog is an awesome resource for folks. Um, yeah, thanks so much for, for your time, Ron. This has been super fun.

[00:51:14] Ron: Yeah, no, uh, thanks a lot. Phil enjoyed it and, uh, uh, let's do it again.

[00:51:18] Phil: Sounds good. Cheers. Cheer.