[00:00:00] Constantine: understanding full customer journey. Is impossible. [00:00:03] there is no way for us to track conversion and distribute credits between all the touchpoints because we don't know all the touch points. [00:00:10] And there are so many touchpoint that happen outside, like attribution window from different devices, from different browsers. [00:00:17] So what can we use as the next closest proxy metric? [00:00:21] if you're not interested, probably you're gonna just look at few products and you're gonna bounce. But if it's something relevant, you're gonna start your research. [00:00:28] we started measuring this time and measuring different actions [00:00:31] We give a score for every single visit, even if this visit does not end up with a conversion, and then we redistribute, the conversions. You have probabilistically, between all the upper funnel campaigns. [00:00:42] Phil: Intro Music [00:01:08] Phil: In this episode we cover: Why Marketing Attribution Still Matters Despite Its Flaws Why Geo Holdout Testing Fails Because Companies Get The Science Wrong Why Marketing Will Never Have True Causation How Visit-Based Attribution Unlocks Hidden Marketing Insights AND… How Marginal ROAS Exposes the Lie of Average Returns All that and a bunch more stuff – after a super quick word from 2 of our awesome partners. --- [00:03:37] Phil: Constantine really excited to chat with you today. Thanks for your time. [00:03:40] Constantine: Pleasure to be here. [00:03:42] Phil: I spent a lot of time, uh, researching your methodology and listened to several past podcast episodes and read a lot of your articles and your newsletter, and I'm excited to get your spin on existing measurement methodologies. [00:03:53] Chatted with a lot of different people about this stuff and the limitations. Um, but not a lot of folks have something [00:04:00] unique to, um, introduce to, to folks. And I think what you're building is really interesting. [00:04:05] Why Marketing Attribution Still Matters Despite Its Flaws --- [00:04:05] Phil: maybe we can chat about MTA, 'cause I know that you have. A bone to pick wi with MTA. A lot of folks have a bone to pick with MTA, this methodology itself, I feel like is the hardest to defend of all the different ways you can measure marketing today. Uh, most folks I, I've chatted with have woken up to like the lack of usefulness when it comes to MTA. [00:04:25] One of the biggest arguments against it, and I'm curious to get your take here, is like the lack of causality in, in its output. Like we're simply looking back on the digital trackable touch points that we can see, and those are diminishing every day. And then we're arbitrarily distributing credit and not really answering the question of what prompted that person to purchase or what caused, uh, like what, what drove revenue last quarter. [00:04:49] Do you think that MTA is pretty much useless in most contexts, or do you think that it's still applicable and there is still some value? Do you think it's impossible to measure the full customer journey in [00:05:00] 2025? What's your take there? [00:05:02] Constantine: Yeah. So in my opinion, MTA still has its own place, and like if I compare it to anything that exists on the market, it's still the both, the most, uh, like business applicable approach because there are many, like scientific methods. There are many like, like geo holdout testing, MMM, other like really scientific approaches where you take math, where you take a lot of data, you go to the lab, you make some tests, but when you try to apply this to real world business, usually it's not applicable. [00:05:35] Take, uh, geo holdout testing, right? It sounds like a nice idea. Uh, just withhold showing your ads in, like, for example, 50% of states and continue running your ads in the rest and just measure the uplift. But for some businesses, uh, this is like losing 1 million, $2 million during the test. So would you be willing to run a [00:06:00] test that's gonna cost you $1 million? And another pitfall here that you won't be able to run a geo holdout test for one small campaign because geo holdout test has such a thing as minimal detectable effect. And usually, like even for United States with a diversity of different states and cities, it's around 5%. By, by 5%, I mean that the channel should contribute towards at least 5% of total revenue of the company. [00:06:29] Phil: Hmm. [00:06:29] Constantine: So you by withholding like, uh, your ads. And if your ads are truly incremental for at least 5%, you're gonna be losing 5% of your total revenue. [00:06:41] Phil: Hmm. [00:06:41] Constantine: So like, the same with MMM, like mathematically, everything works fine, but when you apply it in reality, there is no way to test it. You just see some numbers and there is no way to test it. [00:06:54] For example, it shows that TV brings you like millions of dollars. How, how you gonna validate [00:07:00] this? [00:07:01] How you gonna, how you gonna extract the effect that this effect was cost exactly by tv, not by by anything else. I'm not even talking about the how complex it is to properly imp implement MMM, because essentially you need to have enormous amount of data for the last two, three years. [00:07:22] Literally about everything, about competition, about all the product pricing, changes, discounts, holidays, microeconomics, wars in the world, like, um, uh, presidential elections, like stock. Like you need to factor in everything to be able to build at least more or less accurate MMM model. And of course, like for companies like Coca-Cola, probably this is the only methodology that they can use to somehow evaluate. [00:07:51] But when it comes to like D two C brand or software service startup or any like, uh, lead generation [00:08:00] business, like MTA still is the most applicable business wise methodology that can be implemented here. But you need to know nuances. You just need to know that this methodology. Cannot be applied across channel. [00:08:15] You cannot measure your upper funnel campaigns the same way as you measure your lower funnel campaigns. You just need to know nuances, but you still can compare your lower funnel campaigns to lower funnel campaigns, mid funnel campaigns to mid funnel campaigns. But what is it like real contribution? You might not know, but you can know at least like which campaigns are working better. [00:08:36] At the same funnel level. You can, uh, compare different creatives, et cetera. And, uh, and all, all the platforms like Google Ads, Facebook ads, ads, and all other platform inside the platform, they use single touch or multi-touch distribution. Still, they have a lot of like, uh, complex identity graphs when they try to stitch as much as possible because they possess a lot of data. [00:08:59] But you, [00:09:00] you, you wouldn't see like MMM or geo holdout testing inside these platforms. Why? It's a good question. Like it's, uh, I always ask a question if like, uh, Google and Facebook, so willingly open source, different MMM technologies and they really believe in this technology, why wouldn't they implement it into their own product? And I have an article about this. So there, there are some concerns. They have some concerns, yeah. [00:09:30] Phil: yeah, I, I think I read that, that article actually, and [00:09:32] Simplified MMM is the Marketing Measurement Fantasy You're Being Sold --- [00:09:32] Phil: you said about mm m in a couple of different places that it, it feels like a utopian type of methodology just based on the number of inputs you can have in there. And a lot of folks will argue, well, like you can just start with inputs and outputs. [00:09:47] Like, what did you do over the last year? And you assign a date frame on that and what were sales in that output. And you assign a date frame on that. But your kind of argument to that is like, if you limit the [00:10:00] MMM model with only a couple of sources of inputs of data, like what is the value of the hypothesis that's [00:10:06] Constantine: so, so it, it, it was a huge hype with this like cookie Less Future and like everyone was starting to return to MMM like that was invented, I dunno, 60 or 70 years ago. And, um, most of the companies understood that like it's, it's enormous amount of work to implement MMM and for companies that. Invest like 1 million, 2 million, uh, dollars a month in digital advertising, it's unreasonable. [00:10:32] So probably they won't be able even to isolate the effect. So a lot of companies came up with a solution. Why don't we just simplify MMM, and instead of feeding all the data, we're gonna feed just data from ad platforms like impressions, cost, data, et cetera. And of course you get shiny, beautiful dashboards and very easy to implement. [00:10:52] And some people call it like next Gen MMM or far fast paced, MMM. So you can pull new data from ad platforms, feed it [00:11:00] into the model, and of course you see amazing results like the channel where you invest the most. And the channel that has the most of impressions looks, uh, really incremental, uh, on your dashboards. [00:11:12] But the reality is, if it's uh, so incremental, then you would be seeing it with your naked eye. Like sometimes this, uh, dashboard shows such. Enormous numbers that like when if you launched this campaign, you would be seeing it with an acai. So, uh, but in many cases, the biggest problem is the isolation of the effect because most of the brands they run like Facebook, Google brand campaigns, like many campaigns simultaneously. [00:11:38] And they don't do this sequentially, like in, for example, huge Coca-Cola campaign where they decided to invest like $50 million just to run new ads on Christmas. Of course, for such cases, it's possible to apply these technologies, but for most, like evergreen campaigns, for most of the D two C brands and uh, companies that run, uh, like Evergreen [00:12:00] campaigns like this technology is utopian, right. [00:12:03] Phil: Yeah. Yeah. You, you mentioned some of the drawbacks of incrementality with the, your example of, of geo testing already. Like you're basically telling businesses that we are gonna be losing revenue for the sake of figuring out what might be incremental and what might not be. [00:12:19] Geo Holdout Testing Fails Because Companies Get The Science Wrong --- [00:12:19] Phil: I think that unlike MMM and and MTA though, incrementality experiments are still trying to prove the causality by using treatment and control groups, whether you're using, you know, geo holdouts or, or you're doing this across different channels. [00:12:33] But the big limitation of this method that a lot of folks have chatted about is that you need to have a time window in your test to determine a winner. And that often ends up being shorter than the time it takes for a lot of customers to purchase in B2C when it's an expensive product, it all the time in B2B, right? [00:12:50] Incrementality often misses like how your marketing influences. Customer behaviors over extended periods of time, um, even like long-term holdout tests, like have [00:13:00] an expiration date also, right? [00:13:02] Constantine: Yeah. Yeah. And in many, like, uh, a lot of people tell that like geo holdout test is an AB test, so it's like 100% causation. [00:13:12] Phil: Mm-hmm. [00:13:12] Constantine: While it is, uh, it's not true because like real ab test, like real ab test is when you take the, I dunno, population of the us, you randomly split this population into two groups and randomly show campaigns to one group and do not show to another. [00:13:30] So the primary quality of the proper a b test is randomization. In geo holdout test, we do not do randomization. We decide which states or which cities behave similarly. And then we build synthetic control. So synthetic control is already like a, like a modeled control group. And then we compare results to this synthetic control. [00:13:53] And if any, anything happens within the regions that were selected for synthetic control. For [00:14:00] example, you just switched off your Facebook ads in 50% of states, but for some reason it was like cross cannibalization. For example, YouTube ads started showing to specific, uh, audience or something else happened or something happened in specific region or, uh, what I observe all the time, uh, because, uh, um, usually like when we, uh, work with, uh, some clients, we see how they run geo holdout tests in the background. [00:14:26] And what they usually do, they have like Facebook ads spending 50 k uh, a month and they shut down 50% of the states, but still the rest of the budget is automatically relocated to the. To, to the control group. So they forget that they also need to scale down the budget. And this way, like, wow, like it looks like the test is incremental. [00:14:48] They see different results. But in many cases, like simply because of, again, it's such a complex methodology and because of the human error, and there are so [00:15:00] few people who can properly accurately run incrementality like geo holdout test. Most of the, the time people get incorrect results. And as, as I mentioned, it's very costly. [00:15:10] So, um, this way people misuse it to calculate actual incrementality. And so any geo holdout test has a margin error. For example, you've measured your incrementality as 5%, but usually for 5% incrementality with 5% minimal detectable effect, margin error is plus minus 4%. So your actual incrementality is between 1 to 9%. [00:15:36] Phil: Hmm. [00:15:37] Constantine: And, and when you, like, uh, when you translate it into ROAS, sometimes it's like ROAS can be different like a times. So like you, you wanted it to measure incrementality and incremental ROAS, using geo holdout test, and now you get resolved that actually your ROAS is somewhere between 0.5 to 5.5 So it's either you are either, uh, not profitable [00:16:00] or very profitable and you have no idea. [00:16:01] So that's why the proper use of geo holdout, incre reality testing is when you are not sure at all whether your campaigns are incremental. For example, some people speculate that like brand campaigns might be not incremental. Okay? If you believe that your brand campaigns might be not incremental, so probably it worth to shut it down in 50% of states. [00:16:26] I. Because you believe that it's not incremental, you won't lose money if, if this is your belief, right? And then you run incrementality test and you prove it either right or wrong, but at least now you, you know, or you run like super, super upper funnel like TV campaigns where there are no clicks, no other metrics that you can measure. [00:16:44] For this case, geo holdout is also applicable, but for any other case where you know exactly that your ads are incremental, like measuring true incrementality, using geo holdout test is impossible. And this is a huge mistake that many companies [00:17:00] sell to. That's why like, uh, I tell that right now we have dark, dark era of marketing measurement because like people are so, um, confused because of this cookie future and everything that happened with GDPR, with all this, uh, technological tracking, prevention, et cetera. [00:17:21] And during such times when this is chaos. There are a lot of experts that appear on the market and they try to sell you like gold. While this gold is not gold, but no one knows because in marketing ecosystem. So marketing is a space that evolved very fast and there are not that many specialists who are good both in technical aspects and statistics and understanding how everything works. [00:17:46] And the same time, they're good businessmen. So sometimes we see either one side, like really good like data scientists, and they build these models and they test these models and it shows like good accuracy. But when you apply this to the business, it [00:18:00] doesn't work at all. Or we have like marketers like head of digital CMOs who really manage like huge budgets and they in a sense, they don't have technical knowledge and they just buy from this marketing websites that like we will measure true incrementality with geo holdout test, which is absolutely not true. [00:18:18] Phil: Hmm. Yeah. The, the grifters of the, the dark measurement age. I like how you, you put that, um, maybe the last one to touch on before we chat about like, what you guys are doing differently here. We, we talked about this a little bit at the, at the top of the episode here, when, when you were saying like, how do we measure like my posts on LinkedIn and the podcast stuff, and, and we don't have to unpack that, [00:18:38] Why Marketing Will Never Have True Causation --- [00:18:38] Phil: but there's like a qualitative source of attribution here. [00:18:42] Like lot, lots of folks will bring up the human bias when it comes to self-reported attribution methods like surveys and interviews. Lots of folks are, are really lazy. Like you've called out in a couple episodes, just like. They just select the first option and you're like, how did you hear about a survey if you don't actually have a conversation with them. [00:18:59] I [00:19:00] feel like this is really the only thing though that we can use to counter that like long-term impact of, of campaigns. If, if your budget is, is limited. What other things should we be looking at in terms of like what is the impact of term brand investments, like overall traffic growth? Is there a share of search? [00:19:17] Like what advice do you have there? [00:19:20] Constantine: Yeah, so all we have is like proxy metrics. So like, uh, there are a lot of speculations about like correlation versus causation, but in, in marketing we have only correlation. This is not true that we have causation [00:19:35] Phil: Hmm. [00:19:35] Constantine: like, um, um, marketing reality and customer journey, and psychology is. Like very complex. And it's not like in medical studies, even in medical studies, it's very hard. [00:19:45] We, we, we mostly observe correlations and we have also like different placebo effects, et cetera. But, uh, like, it, it's, it's not like in science where you have like empirical observation and you can do this and that. Because [00:20:00] like when you do this, uh, when you do even, you do an AB test, you need a methodology to measure the results of this AB test. [00:20:08] And even if you do, like, uh, some people say that geo like incrementality lift studies, for example in Google ads is, uh, the thing you need, like, it'll solve all the problems because essentially it's an AB test, right? It's an AB test. So Google will randomly show ads to one group and will not show to other group. [00:20:31] So in a sense, like by design, it's a proper AB test because it's randomly selected cookies. But the question is how do you measure results? And you still measure results based on last click attribution. So it's an AB test that will show, like, uh, still will show like, uh, how many, uh, last click conversions you're gonna, so it, it's, it's still like the measurement methodology that you use to measure the results of the AB test still based on [00:21:00] attribution. [00:21:01] So like, there is a lot of confusion in this space. That's why like, it's, I would say it's one of the most complex, uh, areas to work in. So even some of our investors, they say like, guys, this is really tough. This is a really tough market because, uh, you will encounter people that will have no clue what you are doing. [00:21:23] Phil: Yeah. [00:21:23] Constantine: And the same time, uh, like, uh, for you to educate the market, it might take like 10 years, 15 years. So it's a really tough space. [00:21:33] Your CMO Is Buying Fake Attribution Data (And Everyone Knows It) --- [00:21:33] Phil: It's a tough space too, because I feel like some CMOs, like you said, aren't super technical and they don't really care to take the time to learn the space. Like they would rather just get data, whether they trust it or not, show it to the board, get approval for these other campaigns, and then move on to the next thing. [00:21:52] Like this obsession about the trustworthiness of the data and the results on what we ran and where we should invest. The next dollar [00:22:00] isn't always at the top of the list of priorities for A CMO, like the marketing ops team or like the folks that work in Martech, like there's differences of like the, the level they care about, like the, the legitimacy of the [00:22:12] data, [00:22:13] Constantine: And, and it works. Uh, and it works for a few reasons. So, uh, for example, this next gen MMM platforms, why they're so popular? Because imagine you're a CMO and for some reason you invested like, uh, tons of, tons of money in some DV 360 display YouTube campaigns. And you don't have any data to justify your investment, [00:22:38] Phil: Yeah. [00:22:39] Constantine: and now you need someone, you need a third party because, uh, at the end of the year, you are gonna need to go to the board meeting or to report to CEO and you're gonna need to show like why you've invested millions of dollars into some campaigns that don't show any, like attributed revenue. [00:22:56] So you buy MMM solution and they justify your [00:23:00] numbers. I've heard a lot from my, uh, uh, colleagues who work in different e-commerce companies that some MMM providers even pro provide report, and they say, do like it. If you don't like it, we can make it different. So we can choose different options. So you decide like, what do you believe is incrementality of your YouTube ads, so we can, we can adjust it, so it's no problem. [00:23:20] Yeah. And, and, and, and it, it works, uh, for two reasons. First of all, because in many companies, the rotation of CMOs and head of digital, like. Almost every year a company might be changing. So that's why you need to survive one year and then you go to the next company. Because if you work in a company for 2, 3, 4, 5 years and we have some CMOs from our clients who work for five for 10 years, it's not enough to show something on paper. [00:23:49] You need to show, you don't need to show attribution. You don't need to show like, um, uh, efficient effectiveness of your like YouTube or Facebook ads. What you [00:24:00] need to show that you grow like 20, 30% year over year. And if you don't show this, you're fired. And it doesn't matter that your reports show, like this YouTube campaign brought us so many conversions if you don't grow year over year. [00:24:13] But to grow year over year and to care about it, you need to, to see long term to work in this company, at least for 2, 3, 5 years. But many, I see so many times it like this constant rotation. So a rotation of CMOs, they just. Build another report justify investment, but then they have at least one more year. [00:24:33] If, if like for some reason company didn't lose money, but then it becomes obvious. That's why, that's why I say like how to, so, so the biggest problem that we try to solve is how to measure that our attribution is accurate because no one knows what is accurate. Like no one, like there is no such thing as an accurate because you don't have anything to compare it. [00:24:57] There, there is not like gold standard, [00:25:00] like you cannot compare it like to something like that is correct. You cannot compare it to real world because you didn't know how real world works. We have only understanding when clients follow the attribution and they follow the strategies that they grow year over year. [00:25:13] So this is the only way, and that's why I take all these case studies that I see in the internet when someone says, oh, actually we evaluated our TikTok ads and we figured out that like actual roles is five times more than we expected, and we doubled like our spend on TikTok. And now we've got like, I don't know, 1500% more revenue from TikTok. [00:25:40] My question is, how much did you grow year over year? Because when you measure results of your attribution by your own attribution, it doesn't make any sense. [00:25:50] [00:26:00] [00:27:00] [00:28:00] [00:28:04] Visitor Scoring Transforms Marketing Attribution --- [00:28:04] Phil: Uh, okay, so let, let's unpack how you guys are doing this a bit differently. So you're calling this on the site visitor scoring for incrementality, and I wanna give you a chance to unpack this, uh, a little bit. So, segment stream is using machine learning to evaluate the incremental impact of each web visit using first party data, even if there isn't a trackable conversion. [00:28:26] You're analyzing that session and looking to see the depth of time spent on site, what they looked at, like how legit was that visit. Can you maybe unpack this approach to visit scoring and how you're essentially taking its stab at reconceptualizing the relationship between all the different marketing touch points that we can see to purchase decisions. [00:28:46] Unpacked that for us. [00:28:48] Constantine: Right. So actually, um, uh, we've been talking a bit before this podcast about like the naming for this [00:28:54] attribution because at one point we just wanted to call it, call it multi touch attribution, [00:29:00] but, uh, people became so wired. Uh, that multitouch distribution should be based on conversions that attract based on cookies and then distribute. [00:29:08] So like in this case, and everyone was saying Multitouch distribution doesn't work, et cetera. So should we associate ourselves with Multitouch distribution even, even though we are a multitouch distribution? Because there is a, a variety of different ways how you do attribution even in, in innovative way. [00:29:24] We also wanted to call it like common sense attribution. We wanted to call it like session scoring attribution. We wanted to call it single session attribution. Uh, we wanted to call it like conversion modeling. Uh, but eventually we ended up with visitor scoring because it's, uh, we have lead scoring, we have LTV scoring, and in a sense we are applying the same technology, but we move upwards. [00:29:51] And now we don't have leads. We don't have different information about different lead properties, but now we have visits and we can score visits. And we can [00:30:00] gain some information about, like, the possibility, how much revenue gonna each visit bring, and if we apply statistics. So essentially, like if you're gonna bring five visitors that have 20% probability to buy, statistically it's the same as bringing one visitor who made the approach, who has 100% probability. [00:30:22] So essentially at big numbers, this math plays out. So when we were designing the, our measurement approach, uh, like Elon Musk, we decided to work out from like, uh, from the like basic principles of understanding, like, let's understand how it works [00:30:47] Phil: Yeah, for first [00:30:48] Constantine: and, and let first principles, and let's make some assumptions. [00:30:52] So let's make the first assumption understanding full customer journey. Is impossible. [00:31:00] So even the best like multitouch attribution, like if you take like super, super advanced multitouch attribution, data driven, et cetera, it still applies credit to all the touchpoints that happened within the same device, within the same browser. Yes, you can apply some stitching mechanisms by IP address if it's allowed to track. You can have user ID when someone authenticates on a website, but fraction of such sessions is very limited. And even with IP address, people move all the time with their iPhones now, uh, like I'm moving from my home to work and my IP changes all the time. [00:31:37] So it, you, you can stitch a lot of similar users working in the same WeWork office and it doesn't prove that this is the same user, right? So, so our first assumption was like understanding and seeing full customer journey is impossible if it's impossible. That there is no way for us to track conversion and distribute cred credits [00:32:00] between all the touchpoints because we don't know all the touch points. [00:32:02] And there are so many touchpoint that happen outside, like attribution window from different devices, from different browsers. And nowadays even on mobile device, when you click on your, on some ads on Instagram or Facebook, you don't even open it in the website in Safari or Chrome. You open it inside INAP Facebook browser or INAP Instagram browser. [00:32:27] And when you're interested, probably you're gonna switch to default browser. So even like within the same device, you have multiple cookies and multiple, uh, different like browsers. So we took this assumption. So what can we use as an, as the next closest proxy metric? And we understand like, okay, first idea was we can take like, um, we can use like add to cart event. [00:32:54] Or we can use like checkout, initiated checkout event. But most of these events are also [00:33:00] quite lower funnel, so we uncovered that. Even add to cart, even though you have like thousands of such events, some people use add to cart just because they use it as a wishlist. Some people add something to cart only when they made up their mind and then they made a decision. [00:33:19] So what else can we measure? And we understood that, uh, we need to measure effort. Like how do we measure, for example, if I see some ads, if this ad is relevant to me, I would do something to like to research the brand, to research the product. So what, what can I do? Eventually I can click on the ad. I. So many people, uh, argue about the incrementality of post click conversions, but at the same time they say, oh, but post U conversions actually incremental. [00:33:54] Like for me, this is like some, um, some interest in logic [00:34:00] here because if, if someone, um, didn't put any effort even to check out your website, why are you making an assumption that like all these millions of impressions that you bombard on Instagram, like YouTube are actually incremental? And again, like this might be applicable for brands like Coca-Cola, et cetera, that just need to put this message into your, um, into your brain or you already know about this brand, they just need to give you information about some new product line, et cetera, or something else. [00:34:34] But this is, if it's like unknown D two C brand to make an assumption that someone would bother. Remembering your brand after, uh, after impression. Like, uh, for me it's, uh, I cannot understand this logic. For example, today, probably, uh, you've scrolled your Instagram or YouTube, like how many like brands you've remembered from [00:35:00] those you were, did not engage. [00:35:01] Can you name one? [00:35:03] Phil: Yeah, I would probably only remember the ones that I knew previously and, [00:35:08] Constantine: Right, right, right. That's, that's why like this, like so-called brand awareness, it makes sense for brands that already have like, uh, that already have this brand awareness. Um, and they want to strengthen it. But if it's like a known like startup known D two C brand that just wants to find relevant audience, like you need to target your audience properly and you need to measure like engagement properly. [00:35:35] And we found out that like, okay, our next assumption if no one clicks on the ad. Probably they don't want to put an effort, they're not interested at the moment. So, okay, then we go to clicks. If someone clicks an ad, is there a correlation between click and conversion? Probably there is some, but not that much because you can just put little, [00:36:00] like shiny ads on Facebook with like click bait ads. [00:36:03] Like they promise you something or like, look really funny. Or you can, uh, see an amazing photo of beautiful dress, like you click on the website and actually see that it costs like, uh, $15 and actually it's from Alibaba or something like that. So like clicks do not, uh, have very good correlation with conversions. [00:36:23] So we go next level. You click. Now you are on the website, so if you're not interested, probably you're gonna just look at few products and you're gonna bounce. But if it's something relevant, you're gonna start your research. You are gonna start like looking, for example, if it's a booking website, you start looking for different destinations. [00:36:43] If it's a travel agency, you're gonna start looking at different price, et cetera. Eventually you start investing time and actually, like we started measuring this time and measuring different actions and then we take the longest customer journeys that we [00:37:00] have using cookies and we compare like what are the behaviors that like we can identify from these upper funnel campaigns, which are very similar to those behaviors from the longest customer journeys that can show us that actually this user is interested in the product and, and depending on how customers behave and how we measure these patterns. [00:37:22] We give a score for every single visit, even if this visit does not end up with a conversion, and then we properly like redistribute, uh, the conversions. You have probabilistically, becau, between all the upper funnel campaigns. How can we do this? First of all, imagine like you have a lot of clicks from, I dunno, from Paraguay and generated 1000 score, but you have only one conversion from Paraguay or zero conversion from Paraguay [00:37:54] Phil: Mm-hmm. [00:37:55] Constantine: and you have 1000, uh, score generated from us and you have [00:38:00] 100 conversions. [00:38:01] So of course we don't like simply distribute, uh, score and value and conversions based on the amount of time and like what you've done on the website. If eventually from some, uh, like probabilistic stitching mechanisms, we cannot identify that this is relevant because if this like thousand score that is generated from Paraguay. [00:38:22] And we don't see any conversion from Paraguay. This score doesn't matter for us. The value of the score will be almost zero. So we also bucket up all the scores and we try to stitch this probabilistically to distribute real conversions that happen between, uh, and for, or for example, like if we see that, uh, it's a booking website and someone is booking a flight from us to Dubai, of course we're not gonna match these conversions, uh, conversions purchased for the, uh, the flight from, uh, US to Dubai. [00:38:54] We won't match them to upper funnel journeys where someone was looking for a flight from [00:39:00] Paris to Prague. So we use a lot of like contextual behavioral data also to understand that these customer journeys might be related. And we assign credit or some portion of the conversion to this. So it's very probabilistic model. [00:39:16] But again, we play here with statistics and with big numbers. And when you do this properly on big numbers, eventually we see that if you're gonna invest more in customers who actually from specific segments who actually behave a specific way on the website and from the same parameters, they also buy customers having the same parameters. [00:39:36] They also buy, eventually your revenue grows, and eventually you have much higher ROAS in your blended marketing mix. [00:39:43] Phil: Hmm, very interesting. [00:39:46] Visit-Based Attribution Unlocks Hidden Marketing Insights --- [00:39:46] Phil: So I, I wanna make sure I, I, I got this right, so you're. All of this like scoring that you're doing, step one is every single session you're scoring them based on activity, whether they buy or not, whether you can stitch [00:40:00] other sessions together or not. But you have a lot of data for sessions that you were able to stitch together and you can see what were the behavioral activities early on at the top of the funnel. [00:40:14] And you're scoring those in all of the sessions. So you're baking in this, like this whole journey in a sense, or the trackable journey from what you can see and you're deciding, I. Or essentially evaluating the outcome of that by comparing it like you're testing the usefulness of the scoring by comparing, uh, like, let's use Facebook for example. [00:40:34] Like you would compare the incrementality of Facebook based on conversions and a test versus the incrementality of Facebook based on the scores that you assign to every session. And, and the scores are a lot closer, a lot better, better aligned. Uh, do I have that right? [00:40:50] Constantine: The problem is like, um. Sometimes for your upper funnel campaigns, you might not even have conversions. So, uh, even with big clients, sometimes we [00:41:00] see, like, they have like many campaigns on Facebook, and yes, one, one of the campaigns might have like hundreds of conversions, but some campaigns have just 50 conversions. [00:41:09] And when you go to the ad group level, it's even less. And then you go at creative level and it's even less. And there is no way for Facebook to properly optimize because it's just not enough signals. And, and if you move, uh, one level, uh, closer to the click and you evaluate actual behavior, you're gonna get much more signals. [00:41:32] So usually we can have like 10 times more signals. And also, as I've mentioned, conversion is only attributed to the last cookie. And on platforms like Facebook on TikTok, uh. Usually people like come, they use mobile a lot and many people buy on desktop or, uh, people just browse with an in-app browser and then they switch. [00:41:54] So this conversion will, there is no technical possibility for this conversion even to be attributed to the initial [00:42:00] click. That's why, uh, with visit scoring or um, like single session scoring, we can call it this way, you always have a consistency in cookies. You always know that cookie is consistent within one visit, so you never lose the traffic source. [00:42:19] So as, as soon as visit ended, we can evaluate this visit, give it a score, and immediately attribute it to the traffic source that initiated this visit. So there is no way to lose information here. So this is like forward looking model and it might be that this user will convert. In the future from a different device or browser, we don't know, but we just, uh, evaluate the probability of this. [00:42:46] Phil: Mm-hmm. So assuming that those users didn't decline cookies, right? Um, in, in a lot of cases they're like, one question I have in my head is like, it's what you're [00:43:00] doing is almost like a session quality analysis, but you're doing this acROAS different stages of the customer journey and you're looking at the source of that session and how good that session was in terms of like, research or, or doing something and not just like bouncing after viewing a couple pages or whatever. [00:43:20] Um. [00:43:20] Stop Chasing Credit and Measure What Matters --- [00:43:20] Phil: I wanted to ask you like, do you think that like one way to describe this, 'cause you were thinking of like, how do I name this? Like it's almost like a session based last touch attribution. I know like you're doing multi-touch, but in a sense you're giving a lot of credit to that last touch, right? Like the touch that had the highest scored visit where the person did something. [00:43:40] Like maybe we have a session that's really solid, tons of time spent on site, multiple page views, pricing views for B2B, whatever. Let's say the source of that traffic was direct though, and, and, and maybe that's like the user's fifth session. Previously they saw your podcast episode with Phil or they saw your YouTube ad. [00:43:58] They never had a session that [00:44:00] was high quality until they decided to type your brand name in the search box and decide to look into you. In that case, you're doing last touch attribution to that source of direct traffic. No, like chat to [00:44:11] Constantine: like in, in a sense, like imagine you have a customer journey, which has three touch points that we cannot stitch. [00:44:20] Phil: right. [00:44:21] Constantine: So we evaluate each visit separately, [00:44:24] Phil: Mm-hmm. [00:44:25] Constantine: and essentially in a sense, it's the last touch because like within one session, it's only one touch. [00:44:31] Phil: Sure. Yeah. [00:44:32] Constantine: There cannot be a, so you cannot build multi-touch attribution for one visit. [00:44:36] Phil: Mm-hmm. [00:44:36] Constantine: So of course, like when we, uh, are able to stitch, so if we are imagine we still are able to stitch a huge portion of traffic and we see a lot of customer journeys. We, where we see, uh, five touches within the same cookie. So in this case, our algorithm works a little bit differently. So for example, first you came from Facebook and we evaluated that your probability to convert [00:45:00] increased from zero to 20%, for example, but then you returned from some from direct and now your probability increased to 25%. [00:45:09] But we will not assign 25% to direct because 20% were already generated by initial. So we'll assign only, um, fraction only the change that this particular and also it's possible to set up in a platform. So we have a setup to identify. Uh, to set up, um, what is, what does it mean a significant traffic source? [00:45:33] Because like in many cases, like companies do not care. I've seen some companies that care, but like most same companies do not care about pseudo, uh, free channels. You cannot scale organic. You cannot scale email. You cannot scale direct. Like, uh, there are many things that you cannot scale by putting $1 million. [00:45:57] So it doesn't make sense to, like, I, I've seen some, [00:46:00] uh, some marketers who even put like UTM parameters on their cart abandonment emails. Like, why would you ever do this? Like, why, like, marketing measurement is already so complex. Why would you put U 10 parameters to your like, uh, email campaigns? If you want really to measure incrementality of your campaigns, run an ab test. [00:46:20] Like in CRM, it's very easy. But like my personal opinion, just do same things. And like if you want to send an email, it's, it's like reasonable to send an email because running a test is also costly we discussed before, but don't put U 10 parameters because it's insignificant traffic source. [00:46:38] Significant traffic sources are traffic sources where you invest a lot of money and you require budget. Everything else is insignificant. You're gonna do this anyways. Or even if you're gonna do this anyways, it's not possible to scale it two times next month. So that's why like, uh, in many cases we recommend our clients, like everything that is organic, [00:47:00] like, um, brand campaigns, uh, email campaigns direct, it's insignificant traffic source. [00:47:05] So even if we're gonna be able to stitch a customer journey with five touch points where first touch point was significant, and after that insignificant touch points, all the score from, from all five visits is gonna be attributed to the first, uh, significant source. [00:47:20] Phil: Hmm. Interesting. [00:47:22] Marginal ROAS Exposes the Lie of Average Returns --- [00:47:22] Phil: And so just to be clear, you mean significant in the sense that we're gonna be doing those things anyways? Those are insignificant. The significant ones are the ones that we're investing dollars in and we're like deciding what that lever is, like, how much we're [00:47:35] Constantine: For me, significant. It's like people are too obsessed about measurement. Like, um, when I write a new article and I send it, uh, to, to my email list I would be sending at anyways. If there are zero conversions, five conversions, 10 or 15, I don't care whether it's 10 or 20, I will be [00:48:00] sending it anyways. [00:48:00] Phil: Mm-hmm. [00:48:01] Constantine: But when I'm investing like $1 million, I need to be very precise about measurement. [00:48:09] So when I say insignificant, it means that you should not be that obsessed about measuring exact ROAS of this channel. Just know that this channel is incremental. Just know that you should be doing this. If you, or if you have doubts that this channel is incremental, run an AB test because if you have doubts, probably the cost of this AB test will, so my approach is like this, like if, if you know that this is incremental, just do it. [00:48:35] Don't run ab test because our ab test costs a lot of money of cost opportunity, uh, of, um, uh, the cost of opportunity. So how much revenue you're gonna lose, [00:48:46] Phil: Yeah. [00:48:48] Constantine: but at the same time, you're gonna be doing it anyways. It's incremental. It's sudo free's almost free. You don't, like, you just have one person doing SEO or so you don't need like exactly to know, [00:49:00] is it ROAS four point x or six point x? [00:49:03] Just know that it's already 20 point x because you just have one guy doing this and it brings you so much money. But when it comes to investing money and when you, when you decide where to allocate the next 100 k. For me, it's very important if it's ROAS, like 0.8 or 1.1. [00:49:22] Phil: Mm-hmm. Yeah. Big difference with a hundred K there. [00:49:26] Constantine: It, it's, it's, it's another depth of what ROAS is and especially like, like what, what is ROAS? [00:49:32] 1.1? Because again, in the market, everyone is looking at average loss, [00:49:37] Phil: mm-hmm. [00:49:38] Constantine: and this is a, a very misleading metric. [00:49:43] Phil: Yeah. [00:49:43] Constantine: A very misleading metric because like imagine you invest $1,000 per day in Google ads and you get $2,000 in return, so your ROAS is 2 x, right? [00:49:58] Phil: Mm-hmm. [00:49:59] Constantine: Great. [00:50:00] You decide to scale, you decide to scale your budget two times you invest additional two or additional $1,000, but it gives you only $500 of incremental revenue. [00:50:15] So it means you are already burning money. You invested 1000, but got 500 in return. But if you're gonna look at average ROAS, you invested 2000 and you've got 2,500 in return, it's still profitable. You can continue investing. [00:50:29] Phil: Right. [00:50:30] Constantine: So marginal ROAS is like, it's very hard to calculate and I, I, I probably, I would say probably we are the only platform that actually calculates this. [00:50:39] So to understand at which point you need to spend, uh, stop spending because you should be optimizing your budget allocation, not based on average ROAS And this is a market standard. You just go inside your Google ads and Facebook ads, you see this average ROAS numbers and you see, oh, my ROAS is still 1.2. [00:50:56] I can still invest. No, it might be that your marginal ROAS is [00:51:00] already 0.3, but you don't know about it. [00:51:02] Phil: Very cool. Yeah, and you've written a lot about this too. We'll, uh, we'll link out, uh, different articles that, um, and, and you got some really good LinkedIn posts, uh, on that too. I, we could probably do a full episode on like marginal [00:51:14] Constantine: Yeah, it's, it's a huge topic. it's a [00:51:16] huge topic. [00:51:17] Phil: Uh, [00:51:17] Activating Attribution Insights Through Automated Bidding --- [00:51:17] Phil: I want to give you a chance to chat about activating attribution insights. [00:51:20] Like, I think one thing that SegmentStream does really differently from a lot of platforms, a lot of folks talk about like how to measure, and that's hard enough as it is. But what do you do with that information? Once you have insights from different platforms, different measurement, segment Stream has a way to take those results from your synthetic conversions, like the, the stuff that's come in from visit scoring, and then you can directly optimize the bidding in platforms like Google and Meta right now. [00:51:46] Right. [00:51:47] Constantine: Yeah, exactly. So, so we have two ways how we activate insights because like, uh, initially when we, once we've built the product, we understood that actually, like, yes, [00:52:00] you see all these reports. But in many cases, like analytics just stays in eyes of analysts. [00:52:08] Phil: Yeah. Or a screenshot in a PowerPoint slide and it's [00:52:11] Constantine: yeah, and, and we decided, um, we invested a lot of money into this, but we just wanted to know, like our product team wanted to know, like, do our clients really use analytics? [00:52:22] Phil: Hmm. [00:52:23] Constantine: Because like many companies invest like hundreds of thousands of dollars a year into different, uh, marketing measurement methodologies, but they just look at this reports and they're like, okay, nice. And we've built a metric, which is called adoption. So actually how this works. Like we check on a weekly basis results of attribution, and we give recommendation what exactly needs to be done. [00:52:48] So you like, it's very pre prescriptive. You need to take, you need to decrease like budget for this campaign from $100 a day. To $50 a day. And for this campaign you [00:53:00] need to increase from 100 to 150. So exact recommendation based on the numbers of the attribution. And then we, uh, uh, create an API connectors and we connected to apply, but we also pull calls data about everything so we know what happened. [00:53:13] So, and we look at the next week and we see that nothing happened. Like they didn't do anything. And like we understood like, uh, maybe 90, 95% of our clients, they were not following attribution. They were just looking into dashboards and they were doing some other stuff there. And maybe they were doing something, but it was not correlate like actual results. [00:53:33] What was applied was not correlate. So. Ad platform was not properly synchronized with analytics because ad platform uses different attribution ad platform uses its own pixels, its own target ROAS which is based on average ROAS and not on marginal ROAS So even if you're gonna, like, if you see like two x row and segment stream and you're gonna put Target ROAS to X on Google is, those are two different numbers based on absolutely different [00:54:00] methodologies. [00:54:00] So we decide how can we like synchronize, uh, attribution, how we can synchronize analytics with actually what happens inside ad platforms. And the first thing that we've implemented was a synthetic conversion. So essentially visit scoring attributes, conversions to visit that have not converted, but Facebook and Google, they do not have any conversions for visits that for them it's like, like wasted spend. [00:54:24] For example, it was a user who actually like, was researching a product who like almost made the decision to buy, maybe even send a link using WhatsApp to, uh, to a wife. But then there is no conversion from this device in browser. And what Facebook gets, Facebook gets, oh, this visit was completely useless. [00:54:44] I should not be buying more users like this. And this was like, uh, and this synchronic incorrect. This breaks the smart bidding algorithm. And then there is some other user out of the blue or from the retargeting that buys immediately, like [00:55:00] maybe the wife who clicked the link and eventually made it purchase. [00:55:05] And you have this conversion and you don't understand what's going on. Like it's completely broken customer journey. So to compensate for this, like we. Decided to create a synthetic conversion. So at the end of the visit, if we see that this visit is valuable, if we see that Facebook should be bringing more customers like this because this customer was really interested in the brand and the product and has very high probability to buy, even though they might not buy from the same device browser, we create a synthetic conversion, which is synthetic means like it's not real. [00:55:35] There was no real conversion, but we fire it as an event and we send it using conversions, API to Google or Facebook or LinkedIn or TikTok, et cetera. So, so this was the first way how we implemented this activation. So now you can optimize your upper funnel campaigns. And for some clients, uh, it was an amazing, elegant, [00:56:00] uh, like way to solve their problem of upper funnel because their upper, upper funnel were not getting any signals at all. [00:56:05] There were no last click conversions at all, and they were optimizing for landing page views or for clicks. Which are not correlating, uh, from conversion with conversions at all. So we've changed this and now they were optimizing for synthetic conversions for, uh, high value customers, for high value like potential customers. [00:56:24] Um, the next step of activation was actually applying budget recommendations. So we know that when we show that in our attribution, the ROAS of this campaign is 2 x and ROAS and marginal ROAS is, for example, 1.5 and marginal ROAS of another campaign is 1.2. Like we know exactly that budget for this campaign needs to be increased and for another campaign needs to be decreed. [00:56:50] But how to apply it in the ad platform [00:56:53] Phil: Hmm. [00:56:54] Constantine: because you need to make all these calculations. Okay. Okay. How to map. SegmentStream's marginal, [00:57:00] ROAS to Google, ads ROAS, how much we need to change the target so that we exactly match that. Our budget gonna decrease by 20%. So it's a lot of calculations because of mismatch of attributions, bid strategies, et cetera. [00:57:11] So we've created algorithm and now you have all these recommendations like list of recommendations, how exactly you need to adjust the budget. And then you just have apply button and you just click this apply button and we connect using API to Google to Facebook, and you can even create across channel portfolio. [00:57:28] And we just apply this budget automatically. So we exactly calculated to achieve like 1.5 marginal roles on segment stream. You need to set target roles on Google 2.5 and increase daily budget by 7.5%. And this way we uh. We're really proud of this, that on many clients we achieved like 80% adoption of insights. [00:57:52] A hundred percent is almost impossible because still, like pla sometimes platforms behave very unpredictably. Sometimes they [00:58:00] overspend, sometimes they use like the their own signals when they decided they need to beat higher, et cetera. But 80% adoption is already good. So our clients are able to apply these recommendations on a weekly basis. [00:58:13] And some of our campaign, uh, our clients have hundreds of campaigns. And before they just needed to go inside Google ads and apply this like, uh, changes in budget for every single campaign that took like hours and hours and, and at best they were able to do this monthly. Now they just can click apply button and all these changes are applied in a second. [00:58:35] Phil: Fascinating. That's super cool. I, I'm, I'm happy we got into that topic 'cause I feel like that really differentiates your, your offering from a lot of different measurement attribution platforms on the market. I love the synthetic conversion data sent back to the platforms themselves. Not only are you improving your bidding strategy from that, but you're teaching the algorithm better, like how to serve those ads to [00:59:00] better customers instead of just people [00:59:02] Constantine: Yeah, so, and also like measurement through action is the most accurate measurement because like, like an example before with marginal roles, if you just keep your like $1,000 a day investment and you have two, two 2000 in return, you just have two x ROASs. And you, there is no way to measure marginal roles. [00:59:22] The only way to measure marginal roles is by doing, we call it controlled budget shifts. You need to do to destabilize your system. And you need to do this budget shift to decrease budget, to increase budget, to understand this, like differences in incrementality. So that's why like I don't believe in static measurement. [00:59:42] Measurement should be dynamic. It should be al always like destabilizing the system, applying different budget changes to be able to, uh, allow the model to train on new data. [00:59:54] Phil: Very cool. Uh, two last questions for you, Constantine. [00:59:56] Scoring for Lead Management vs Attribution --- [00:59:56] Phil: We, we chatted a bit about, uh, like the difference between what you're [01:00:00] doing, visitor scoring and lead scoring, like something that's really well known in, in the B2B world, specifically as you were like walking through the stuff that you're building and how you're doing this differently. [01:00:10] Um, I'm, is there a way to like make this like a white box, like a transparent approach to how you're scoring and like also delivering those insights to your customers and saying, Hey, we've noticed that, uh, for the ones that we have, like a good picture of that journey, these three or five activities on site. [01:00:29] Lead to very high visitor scores and propensity to purchase. And not only are we gonna use that data to inform meta and like inform your budget, but we're also giving you that data so that you can send that to your content team and actually do more of that content or send it to your sales team and chat more about that in, in, in sales calls and stuff like that. [01:00:50] Maybe unpack how you're also like turning this into a, a lead scoring offering for [01:00:54] customers. [01:00:55] Constantine: We, we are moving to towards this. So like, uh, for example, even with Google, [01:01:00] once they've implemented smart biding, everyone was very skeptical. Like, we make our, we choose our keywords, we adjust our bids, we do everything ourselves, but now we should trust some black box. Why would we do this? So Google implemented, uh, I, I, I don't think there is much value in this, but it's like psychological like, uh, thing that Google implemented now you, when you come, like your be your, be smart, being changed, and you see that like on Sundays, small conversions, users from me, Mexico City Convert. [01:01:32] So they have this like, uh, tips, like what, what exactly like influence, but you cannot influence this much because it's dynamically changed all the time. So first of all, we released, uh, customer journey functionality where now you can. Ch like, uh, pick any customer, like any cookie, like any particular visitor, and you can exactly see what happened with this visitor. [01:01:55] Like all the events, like time that was spent, what was, and, and exactly at which [01:02:00] point, uh, score was assigned and when it was increased. So, uh, the next step probably we, we might, uh, we might release some other variables, like what exactly influenced the score, et cetera. So we already have some transparency where you can go into every single cookie and see exactly like why this particular cookie received like 0.25. [01:02:24] But of course, there are a lot of things under the hood, especially how we match this. Like I've mentioned, like, for example, if it was one, some, uh, sometimes, um, one of our clients opened like, um. Yeah, customer journey. And it was really like complex customer journey, like 30 minutes on the website like, and they asked why you haven't, why you have assigned like 0.00, like almost nothing. [01:02:50] Like it's really complex customer journey. Like they almost, they even like visited checkout page but haven't bought, but this was a visit from Indonesia [01:02:59] Phil: [01:03:00] hmm. [01:03:00] Constantine: and they don't have any purchases from Indonesia. So there are a lot of like mechanisms under the hood that scale down the score if it's not aligned with specific buckets because people do not move between location that much so that they browse your website from Indonesia made the purchase from us, like say, and there are many of things under the hood. [01:03:19] It's quite complex. [01:03:21] Phil: Super cool. [01:03:22] The Key to Happiness is to Stop Following-up --- [01:03:22] Phil: One last question for you Constantine. Really appreciate your, your time here. You're a CEO co-founder, frequent speaker. You're also an author, you're also a husband, a dad, paraglider, surfer, and a Husky dog dad of two. Uh, one question we ask everyone on the show is how do you remain happy and successful in your career, and how do you find balance between all the stuff you're working on while, uh, still trying to stay happy? [01:03:45] Constantine: Yeah, I would say, um, we started adopting a very interesting principle in the company. We do what we want to do, so I, I, we took it to quite extreme level. [01:04:00] For example, I'm doing a lot of sales in the company and at some point I told I don't like CRM systems. I don't like follow ups. I don't like chasing clients. [01:04:14] Phil: Yeah. [01:04:16] Constantine: I like writing content, posting on LinkedIn, having interesting conversations with people who are interested in what we do and have a problem, and also like strategizing and talking to our product team. So. So I decided I won't be doing anything else. So I stopped using CRM. I stopped following up, and at first my partner was a little bit like, oh, you've met with this client like one month ago. [01:04:45] Would you like to follow up? Like, no, I don't want, I, [01:04:47] Phil: Hmm. [01:04:48] Constantine: why? Because like, it's like an attribution. Like there is a reason why this client is not reaching out to me. They might be many different priorities and I will [01:05:00] be pushing, but they're gonna be resistance. And myself, I don't like when someone pushes me. [01:05:04] Phil: Yeah. [01:05:05] Constantine: So at, at, at some time when they it, they're gonna prioritize this, when it's gonna be huge pain. They're gonna remember our great conversation and they're gonna reach out again. And I will remember them and we will have another conversation. So I just stopped doing things that many companies say that you should be doing because this is a proper way to do sales, to do marketing, to do software development. [01:05:30] And we just started doing like what naturally flows and what you really like and you really enjoy. [01:05:38] Phil: Very cool. I feel like the, the, the thesis of your, of your answer is, uh, how do I stay happy at work? Uh, I, I stopped using a CRM. [01:05:45] Constantine: Yeah. Stop using CRM. Stop using task uh, management systems. Like, because like, this is the nature of priorities. If you forget about this, it's not a priority, otherwise you wouldn't have forgotten. [01:05:58] Phil: Super cool. I love [01:06:00] it. Constantine, really appreciate your time, man. This is super interesting. Love what you guys are building. We'll, uh, share out links, um, and, and, uh, yeah, I'll share out links to your newsletter, the site, obviously. Um, yeah, anything, uh, anything else you wanna plug for folks? I, [01:06:15] Constantine: Subscribe to my newsletter. [01:06:17] Phil: yeah. [01:06:18] Yeah. Been loving it, honestly, truly. [01:06:21] Constantine: Yeah, it was a pleasure. [01:06:22] Great [01:06:23] Phil: Thank you so much. I really appreciate your time. [01:06:25] Constantine: Thank you very much.