[00:00:00] ​Intro [00:01:00] Phil: What's up everyone? Today we have the pleasure of sitting down with Sunder Swami, author of the Experimental Newsletter and part-time Marketing and Data Science Advisor. Sunder started. Started his career as a software developer at Bloomberg. Before managing 19 a debt manager, he pivoted to growth marketing and data science consulting, where he worked with direct TV and an ed tech AI startup. [00:01:23] He then made the mega move to Uber where he spent five years building brand performance and lifecycle marketing data science teams. He moved over to a travel tech startup after that and helped them go from zero to a hundred K MRR. And today Sunder is a marketing and data science advisor. He helps BTC founders and marketers on the side. [00:01:43] He's also got an upcoming podcast that he's been working on and he's got an awesome newsletter where he shares frameworks, how to guides to help B2C marketers. Sundar, thanks so much for your time today, man. Pub the chat. [00:01:54] Sundar: Yeah. Phil, thank you for having me really excited to be, to be a part of this. [00:04:10] Marketing Incrementality Testing Reveals Meta Ads Ineffective at Uber --- [00:04:10] Phil: I wanted to start by maybe unpacking one of my favorite articles that you wrote. So it's, uh, you titled it eight startup marketing lessons from five years at Uber. [00:04:18] And my favorite quote in there is your Mike Tyson version of, uh, everybody has a plan until they get punched in the mouth. Uh, and for marketing, that version is everybody thinks their campaign is great until they run an incrementality test. Maybe you can share, we can start with like you sharing one experiment or a couple of experiments that come to mind. [00:04:36] When you look back on the most surprising, uh, outcomes from five years of experiments at Uber. [00:04:42] Sundar: Yeah. Um, yeah. So I think the biggest one that I would point out is we were running Facebook or, uh, you know, meta, uh, Uh, that was our staple, you know, performance marketing channel. And basically I was the performance marketing analyst [00:05:00] for the U S and Canada, so I was the lead and, and we'd gone into basically a series of weekly meetings and kind of really looked at our, our CAC was just swinging like 10, 20 percent a week on week, which is like pretty unstable. [00:05:16] It shouldn't swing that much. And so what we were observing was like, we were spending the same amounts or spend is the same. And. But our signups were just all, you know, all over the place. And so basically I was like, Hey, let me, let me go away for a couple of days. Let me take a look at it. Uh, looked at the data and basically figured out that Facebook was completely non incremental just from the data perspective, it looked like it was non incremental and I was also in charge of this saturation analysis. [00:05:47] for all of Uber, trying to figure out how penetrated were we in our target markets. And the answer was, you know, this was eight years after Uber launched. Like we were fairly well known, you know, anybody that wasn't using Uber, they weren't [00:06:00] using us because they didn't like us. Right. [00:06:02] Phil: They had bikes or they were walking more. [00:06:04] Sundar: or they just liked lifts, right. [00:06:06] To be honest, that was, that was the other big one. Um, but yeah, right. We, we had a lot of substitutes. So in these like major urban cities, like there's no reason for someone not to have signed up for us. So when you look at that. And you look at the data then as saying her kacks are swinging wildly. I basically pitched to the CMO with a blessing of my like manager and you know, I was like, Hey, let's run an incrementality test on Facebook, like the entire thing. [00:06:31] And this is just for acquisition in the U S we ran a three month acquisition, uh, sorry, an incrementality test. And. It came back that Facebook was virtually non incremental. I mean, it was just like, at that point, like no one had thought about it. No one had really like explored what would happen. And I love that we were able to do that. [00:06:49] And no one like batted an eye. They were like, this is what the data says. Let's go try it out, ran it for three months. And then we, we turned it off and, and gave back Uber about [00:07:00] 30 million a year. So yeah, and which, which, you know, a lot of people be like, that's not a lot, but it was just, again, on one channel in one country. [00:07:09] So imagine that sort of strategy, then replicating across the globe, the impact starts to add up. [00:07:15] Phil: Okay. Like a lot of people do this like test with paid ads where they'll shut it off for one month and then they'll see and compare it to like when they weren't shutting it off. But you set this up as an incrementality test. So you had a control group and a treatment group in the U. S. Folks were getting paid ads in the treatment group in the control groups. [00:07:36] Like folks were not getting ads at all. Like walk me through the setup there. Like, how did you put that together? [00:07:41] Sundar: Yeah, that's, that's exactly right. And I'm trying to remember if on Facebook, you could do that. In itself, like I think there's an, uh, like it's just a split cell test, right? An incrementality test is essentially a split cell test where one cell of your test gets zero spend and the other gets whatever you're spending. [00:07:58] I don't, so I don't remember if we did that [00:08:00] or a geo based test. Um, it's been, it's been four or five years, but In essence, it wasn't that hard to set up though. It was a very classic A B test. We didn't have to spend too much time, you know, analyzing the results, asking if this was valid. Like we, we also at the, because we were such big spenders too, we also had Facebook's data science team work with us to validate the results, which I don't think, I don't think they were super happy with that. But, um, but yeah, like this is again, a very standard. Um, I think, I think you can set it up just on Facebook. It's just an AB test. [00:08:36] Phil: I wonder how much of that had to do with like Uber's brand presence already at that time, like you, you said, right? Like are ads really driving people to. Like book another ride if like they kind of already know Uber, like really it's about their personal need and the timing that they're in right now. [00:08:55] And so I think that like, The brand probably played a big role in that, [00:09:00] but you know, a lot of the folks listening right now, like they don't work for an uber, they don't work for a company that has this like global presence. Um, I myself has never worked for, uh, an uber, like my, my closest comparative would be like my short stint at wordpress. [00:09:14] com and it was really the only time where I never had to worry about sample sizes for experimentation. It's the first time I had a data science team to support us on like experimentation design or remember the exact numbers, but like we had crazy traffic and crazy signups. And most of that was, uh, like people being confused between WordPress and wordpress. [00:09:35] com. That that's another story. Um, but [00:09:37] How to Run Marketing Experiments With Limited Data --- [00:09:37] Phil: I'm sure you, like you had crazy volumes of data at Uber as well, obviously. Um, but that's not the case. Like I said, for 90 percent of the companies and most of the folks listening right now. So this idea of incrementality and running experiments, it's. It's the gold standard. [00:09:51] It's the best way to prove what is driving revenue and what isn't, but it's a bit of a privilege to, right? Like not everyone has the [00:10:00] capacity or the ability to do that. What should you do when you're dealing with tiny sample sizes? Like a lot of our listeners are B2B, like, do you still need to measure ROI for everything that you're going to do? [00:10:10] Like, what's your advice for like earlier stage founders that don't have that volume? [00:10:15] Sundar: Yeah, that's a, it's a really good point. I know the gold standard is a true AB test. Um, but to me, incrementality is more the mindset of, Hey, we just want to validate and justify, right? Like just a lot of these companies will also at one point we'll have MMMs and like, you can't just trust what MMM says. [00:10:35] You also have to validate it with. With some sort of test design. So what I would say is there's two things like I, you know, a lot of data scientists will, will, will shit on pre post pre post does have a really good use case, not only in, in data science, but in life. I mean, I, I, I, I think examples. [00:10:57] Thousands of years ago, you would come up on [00:11:00] something and you'd be like, all right, well, let me try this. How do I feel before and after, right? You can isolate things to pre post. If you isolate it to a very specific one thing and make sure that the variable that's changing is just one thing, right? It's a different versus if I were at Uber and I had turned off Facebook without an AB test. [00:11:19] And then said, okay, well, all of the drop is a hundred percent correlated to this spend drop. Well, I could say that, but my confidence level is gonna be super low, right? Because we, uh, we had crazy seasonality. Uber is a very seasonal business, right? We, you know, in, in sometimes in months. Our CEO would say something and our stock would just plummet or go up, right? [00:11:40] Like there's a lot of brand thing. There's a lot of competition. There's a lot of regulation. Like the problem was that you couldn't isolate it. Now, if you're a company and you're saying, listen, I'm going to keep everything else flat. I'm not gonna, you know, I don't have a huge product release that could change my numbers. [00:11:57] You can do pre post, right? [00:12:00] And actually I had someone on, on my podcast who is at a huge marketplaces at fair, and he gave an example where they were doing a branded search test in France with pre post, but they isolated it to a specific keyword. In branded search in a specific region isolating. I mean, you're essentially going to be able to know that that is the true cause, right? [00:12:25] Especially if you see a large drop, it's not because there's all these confluence of factors all at once. It's because of that one thing. So, so isolate one thing. And the second thing is if you are a early stage company with small sample sizes, in my mind, you should be aiming for home runs. And home runs can be picked up with small sample sizes, right? [00:12:47] If you've got an MDE of 30, 40%, those are rare, but at that stage, that's kind of what you're trying to hit for. You're not trying to do these micro optimizations. And so it's a bit of a strategic [00:13:00] lever too, to. Aim for things that you can measure with small sample sizes because that is the sort of, you know, Whether it's a new ICP, new copy on the website and copy I mean like a fundamental re changing of your value props, not just like a little tweak. [00:13:15] These things do show up with small sample sizes. So, um, it's a bit more about having the discipline to control everything, go into the test saying, Okay, we're just isolating these factors. We're just making this one change. Here's how we've addressed and thought about how it could be impacted by other things. [00:13:34] But we're fairly confident, right? At the end of the day, A B test is just about confidence. Pre post is a less confident, A B test is more confident. So you just have to essentially gauge that confidence to risk level. [00:13:48] Phil: I love the point about the big bets. Like when you're, when you're smaller, like you are less obsessed about changing the color on this button on your homepage or changing the H one, like [00:14:00] redo the whole landing page. Redo the whole homepage, like launch a new part of your product or completely remove in the 50, 50 tests, like a book of free trial versus. [00:14:11] Like see an ungated, like prerecorded version of your demo. Like those are the bigger tests that should have higher minimal detectable effects where you need less like large volumes to say with 95 percent confidence that like, yeah, yeah, this version was better. Like in this case, you're shooting for much better differences and, and that detection doesn't have to, you know, require that large sample size. [00:14:37] Am I getting that right? [00:14:37] Sundar: Yeah, perfect. And I can just give another example at that travel tech startup that you mentioned. Um, that I was at, we must have had only like, uh, one or 200 signups a week. And we ran an AB test where we changed like a big part of our onboarding flow and saw a 40 percent drop. And like, it is very obvious what happened. [00:14:57] Right. But I pushed for that AB test because I didn't [00:15:00] want to then have to deal with the CEO after, uh, Discussing what happened. Like, it's just easier, even if it's like a poorly powered AB test, you can still then take, you know, you, again, it's about risk. And I was like, I'm willing to take the risk with a hundred people because I'll see if something massive drops if it doesn't. [00:15:20] And right. Like, if it's saying, Hey, like, you know, with, with 60 percent confidence, this thing worked or not, you can, you're going to roll it out anyway. So go ahead and roll it out. But at least now I caught the backend, which is, it was a, you know, 30, 40 percent drop with like 95 percent confidence. That's the stuff I'm trying to protect against less. [00:15:41] So, you know, trying to be super like rigorous about AB testing. [00:15:46] The Difference Between AB Testing and Incrementality Testing --- [00:15:46] Phil: You used the word A B testing a few times in there and incrementality as well. And, um, I, I saw some person like post this in one of the comments on, on your posts on LinkedIn too. And I actually had a similar thought. So I wanted to like unpack this and maybe this is just [00:16:00] a nomenclature thing and it's not a big deal, but I'm curious to get your take on this. [00:16:04] Like as I understand it, incrementality tests means having Hold that group and or like a control group that doesn't get exposed to the campaign. But in a B test compares two or more variations, typically of like smaller variables to determine which variable performs better. Um, so like. One of them is to validate causality or like prove the overall impact of a campaign or something. [00:16:30] And it's asking fundamental questions like our ads, even working at all, like your big test at Uber. The other, the AB test one is focused more on like comparative optimization, like fine tuning your campaign. Creative is typically like an AB test, but I've seen it. Like you just said to like use both terms interchangeably a little bit, like saying that. [00:16:49] An incrementality test is an A B test where one of the variations is just the holdout group. Instead of getting a different version of the creative, they're just not getting [00:17:00] that creative at all. Do you think that incrementality tests are under the umbrella of A B tests? And that the definition of an A B test is not variations of only small variables? [00:17:10] Curious your take there. Mm [00:17:12] Sundar: Um, yeah, I definitely fall into the camp of the conceptual understanding that I have of an AB test is one group gets one thing. And the other group, it's another thing. It's two different experiences, because if you look at the statistical tests that you're then running on an incrementality test versus an AB test, for me, they've always been the same, right? [00:17:35] Whether you go with the Bayesian approach or you go with, you know, frequent, just like whatever you you're using, it's still the same comparison. Right. You just compare, comparing and, and, and the test itself, the statistical test doesn't know that in an incrementality test, one person is getting nothing. [00:17:53] It just knows there's two groups. And measuring their differences. So to me, they're the, [00:18:00] they're the same. I would argue that incrementality test has this understanding. That there's a holdout group. Um, but you can also do incrementality test on a hundred percent, sorry, a hundred dollars to spend versus 200 to spend, right? [00:18:14] There's an incrementality to that level of spending. So now you're testing the incrementality of that marginal spend level. So an incrementality test, again, it's just how you have to be. Clear on what you're testing. It generally is understood that you're testing the channel effectiveness and the channel incrementality. [00:18:33] But I have seen it heard of like, Hey, we're doing an incrementality test on, you know, uh, one X spend versus two X spend. So I would, I, I, like we tend to complicate things for, for what I, I don't believe are necessary reasons. Like it's just. A versus B, A can be whatever you want it to be. B can be whatever you want it to be. [00:18:55] And in fact, you can throw a C in there too. I've done an incrementality test where I've done. [00:19:00] Zero spend one X spent and two X spend. Right. And so we essentially were budget neutral, which is kind of cool, but you can test these where we are on the curb. Um, because we're like, well, I mean, while we're at it, if we're going to do an incrementality test and go one X, we don't want to give that budget back. [00:19:16] Let's just two X the spend and see, see where we are on the curb. So, um, yeah, that to me is again, it's an ABC test, but it's also incrementality test. So, uh, yeah. Yes, the, the broad umbrella is AB test and then incrementality is a part of that. [00:19:32] Best Ways To Measure Marketing Campaign ROI --- [00:19:32] Phil: I want to ask you about multi touch attribution. I know that, like you said, while you were at Uber, your team built a custom multi touch attribution model, typically the case for bigger companies with data science teams. You're not using a third party all the time or just relying on a third party for this stuff. [00:19:48] You're building a custom model. Um, but you said it was part of more of a robust measurement strategy. Um, in your post about the five ways. that you measure marketing ROI, which is also one of my favorite posts. [00:20:00] Uh, interestingly, you don't include MTA at all in that post, like in the five ways to measure marketing ROI. [00:20:06] Why did you intentionally leave out MTA as a way to measure marketing ROI? Do you think that, um, I think you'll agree that it's like far from a perfect methodology, obviously, but it's still probably more useful arguably than pre post or like baseline analysis, like some folks call it, like, which is a method that you do have in, in that post. [00:20:26] So why, uh, do you, do you have a grudge against MTA? Why is it not in that post? [00:20:30] Sundar: No, no, I don't have a grudge. I, I think of the five methods that I talk about in that post as actually like a way to test an AB. I mean, wait, the way to measure an AB test. And again, there's an assumption that an AB test has to be measured by, um, like, like a, a frequentist or a Bayesian approach. Pre post in a way is an AB. [00:20:54] It's, you know, if you get very like annoying about it, like a was before [00:21:00] B is after, and I'm comparing a versus B, right? Like it's. So, so that was a way for me to also plot, um, different ways of measuring ROI impact on a, on a scale of like more accurate to less accurate and like difficulty in terms of setting it up. [00:21:19] And so if you look at all those five methodologies that I talk about, they are Almost like a campaign, right? It's like you use pre post for a campaign, diff and diff for a campaign, um, uh, causal inference for a campaign, A B testing for a campaign. These are all campaign specific. MTA and MMMs are just holistic marketing. [00:21:39] ROI measurement tools, right? It's telling you cross channel. How is your, um, or what is the, you know, the, the modeled ROI of, of what you're spending, so it's not a way to measure ROI of a campaign necessarily. We have used it before. It's actually pretty cool tool. Or at least MMM is I haven't used [00:22:00] MTA as much. [00:22:01] Uh, but MMM is an interesting tool even for campaign measurement. But again, MMM and MTA are more holistic measurement tools, less campaign. And that particular blog post was about campaign measurement. [00:22:13] Phil: Gotcha. Fair enough. [00:22:15] Um, one thing that you also said in that, um, that AMA that you did on, uh, on LinkedIn, which is actually how I discovered you and how we, uh, we started chatting, um, [00:22:25] How to Choose the Right Marketing Attribution Model for Multiple Channels --- [00:22:25] Phil: you mentioned that at Uber, you didn't use different attribution models for different channels. [00:22:31] Um, some experts that I've had on the show, um, actually kind of had this consensus that each channel needs its own attribution method. To account for its unique role in the customer journey, because like you can't track everything for different channels. So do you think it's fair that for startups, maybe our smaller teams channel specific methods would require its own attribution method for that channel. [00:22:58] But as you scale and you [00:23:00] manage multiple channels, this more unified approach that you had at Uber becomes more practical. Curious. Your take there. Yeah, [00:23:08] Sundar: I think throughout our, our chat today, there's going to be a theme of, you know, startups versus bigger companies. Um, so the idea of having attribution for individual channels to me. Intuitively does not make sense because it causes overlap and, and the whole point of attribution right or wrong. [00:23:31] And right now I'll use last click attribution is the best example is to, it's basically like a MECE framework. It's supposed to be mutually exclusive, collectively exhaustive. That's the whole point of attribution. If you're saying, well, you know, it's actually first touch for this channel, last touch for that channel, by definition, I could have something that's last touched on this channel also be first touched on that channel, which breaks the whole idea of attribution. [00:23:56] Now attribution also doesn't mean that is, we all know [00:24:00] that's not right, right? Like there's more to attribution than just what the numbers say, but the framework that you use to me is meant is designed to attribute. And if you don't attribute everything and you've got this like weird in between attribution then like you kind of I can't believe I'm not even sure what the purpose of then having multiple attributions per channel is because attribution is also designed from like a practitioner's perspective to for reporting for, for, um, you know, uh, forecasting things like that. [00:24:38] And if you've got this weird Hybrid model that doesn't capture everything but also maybe even double counting then like that trust breaks. And I would rather like one of the things we said at Uber, which I love and I still say is, Last click attribution is wrong, but it is consistently wrong. And you can take advantage of that consistency [00:25:00] and you can take advantage of the fact that, you know, Google is always going to be over indexed. [00:25:06] Uh, obviously anything non digital is going to be virtually zero and, or show up in organic. Um, but you can take advantage of that. You can, you can track that over time. And so when you introduce a new channel, right. In this idea where you have multiple introduction per channel, like. In last click, something will break or something will change. [00:25:27] And that's a good thing. Right. You'll see it go away from another channel. It should, you know, the same channels shouldn't have the same impact. Um, and so, yeah, that's why we were big on, on keeping one attribution approach, which was, we kept last click and we had MTA and we had MMM because they, and we had incrementality testing because there it's, you're supposed to triangulate across multiple signals. [00:25:51] Marketing is way too complex and humans are way too complex to, to just have one thing. [00:25:59] Using Propensity Matching for Marketing ROI Analysis --- [00:25:59] Phil: your post about [00:26:00] the five ways to measure marketing ROI, you explain and talk about causal inference with propensity matching. This is a really cool breakdown, um, how you come up with that. Put it in a lot of your newsletters. You put most of this complex stuff into layman terms for your average marketer to, to, to kind of understand, but you're, you break down the detailed method of causal inference and propensity matching, which in the way that you explain it mimics AB testing by comparing similar groups to estimate. [00:26:31] Causal effects. And the example case that you use is when Uber launched Uber one, like you don't always want to do a big test for like rollouts like this, like product, uh, experiences or like new big features within the product. Right. So how do you select the right variables for. This matching, like, especially with complex behaviors, like loyalty programs, like Uber one was, and like, maybe touch on how do you ensure non tactical stakeholders will then understand [00:27:00] the results of, um, the, the, the methodology that you put together. [00:27:03] Can you break that down for us? [00:27:04] Sundar: Yeah, so I'll start with your second question and then make sure I get to the first part of it. Um, the propensity, I mean, when you're explaining propensity matching virtually, What you have to do is say, Hey, stakeholder, I'm on your side, but you know, we, neither of us, like neither the data science team or the marketing data marketing team wants to just do a controlled rollout because it's, it's, you know, in, in Uber's case is actually going to affect, uh, network effects, right? [00:27:33] So there's actually some, some, some bad stuff there, but you go to them and you say, this is the only way we can prove the ROI of our program and understand its value. So you, you approach it from the perspective of. This is us working with you to, uh, find a way around the fact that we can AB test. And what we're doing is instead of comparing, you know, people just randomly who bought versus who [00:28:00] didn't, which are inevitably going to be biased, we're going to be comparing people that bought come to people that. [00:28:07] Look like they should have bought, but didn't, and whatever reason they didn't. And so, um, so, so you're trying to look at, you're trying to compare apples to apples instead of what would have been apples to oranges if you can't AB test. So that's, that's how I explain it. Um, and, and obviously every marketer is like, great, sounds like we have a chance of proving this is impactful. [00:28:28] So go ahead. And there's no risk of like holding hold back, which marketers hate. Marketers hate holding back. So, so propensity matching overcomes that. Um, and then from the variable selection, first party is always best. Because one, it's clean first party data, sorry, because it's one cleaner, but two, it represents the actions of what a customer is doing, right? [00:28:52] Like we could have gone down the rabbit hole of trying to get demographic data, but one that's expensive to then you have to do matching on that [00:29:00] demographic data. And also like with a lot of privacy stuff, it gets very complex, very quick. When you start with first party data, you might not know stuff like age, gender, race. [00:29:11] But you do know behaviors. And at the end of the day, what you're trying to do are influence behaviors. And so having variables like what day of week do they ride the most? What time of day? What car type, right? Were they Uber black? Were they Uber X or the Uber pool? Uh, where do they live? Right? Cause we have this beautiful ability to know every single city center in the country and in the world too, we could say city center. [00:29:39] So urban versus suburban, all of a sudden you've got all this information about them, how long have they been on the platform, right? All this first party data that. All of a sudden, you're like, I've got a pretty solid, comprehensive understanding of what they're doing. Even stuff like, you know, where they're going. [00:29:54] So people that are commuting, uh, you know, I lived in DC for a long time. So people are commuting [00:30:00] to, uh, the white house area during rush hour are probably government employees. You can start to put personas around these things. And so first party data is where you should always start. And it's usually a lot richer than people give it credit for. [00:30:15] It might not have the demographic data that people associate with, with segmentation, but, but, you know, behavioral segmentation is, is just as rich and, and oftentimes more accurate because you don't have to do the matching. [00:30:31] Phil: [00:31:00] [00:32:00] Yeah, it's a super cool way of thinking about it. I think the whole like propensity predictive modeling side of marketing, especially sitting on top of first party data is like almost an area where. Not just enterprise teams, but like teams that invest in that, like data support for marketing have access to, like most of the startups I chat with, like advise a small number of teams, like this idea of propensity modeling, like it's just not a thing. [00:32:58] Like it's like a few [00:33:00] far out future base thing, even if there is like one data engineer or like a data team, like they're They're stuck like building tables and like using reverse CTL to share that data out and all the CEO doesn't like this tool So you're stuck like pushing that into like a google studio so that they have access to like they're not like focusing on building cool shit like they would probably think that propensity modeling is um, [00:33:24] but I wanted to ask you about uh predictive modeling like [00:33:27] A Guide to Customer Lifetime Value Prediction Methods --- [00:33:27] Phil: you mentioned that the Modeling your team did at Uber built, uh, was built to predict things like churn lifetime value. [00:33:33] You mentioned a couple other ones in there and it was really effective in the short term, but less so in long term predictions. Uh, given like obviously Uber is fast changing nature of the business. You mentioned a ton of seasonality. Did you find that like building flexibility into the models was a requirement or was it more like. [00:33:54] Frequently like rebuilding them from scratch. Like how, how did you account for that? Like long term [00:34:00] like lack of accuracy for things like churn and LTV. [00:34:04] Sundar: Yeah, yeah, that's a, so I took a question. I, I think we took the approach of. We know longterm, I mean, especially at Uber, right? We were the fastest growing company for, for almost a decade. So there was just a pace of change that we could not keep up with. Then you introduce products every few years. And then, so it was more from like a, like sanity perspective. [00:34:30] Um, but also like, if you think about companies like Uber. People go in and out all the time. Like you don't have the luxury of one person sitting there building a model for 10 years. So it's also like you're continuously evolving because the people building the model are evolving and the priorities for the business. [00:34:46] So that's one thing. But if you think about how churn and retention work. They're always from a series of short term actions, right? Like you're only retained because you were [00:35:00] activated because you saw value in the product. And you only took the second trip because you took the first trip and you only took the third trip because he's, but like, you could tell very quickly if, if I had a driver, you know, tell him he's going to be there and he or she's going to be there in 10 minutes and it takes 20 minutes. [00:35:17] And then my, my price ends up being three X. What I said, it was going to be, there's a lot of variables very quickly. You, as you know, we could very easily understand what caused churn. So you didn't have to predict long term churn because churn is an inevitably a short term thing. Like people react, people don't remember. [00:35:37] Like, Oh, I've been burned a hundred times. They're like, I've been burned twice. I'm done with this company. Right? So you don't have to actually model churn out for that long. And same thing with retention. People are like, I loved my last experience. It actually made up for the experience before that. And all of a sudden you, you get this retention, so you didn't have to build retention models. [00:35:54] And so that was part of the challenge though, right? Is. Everyone talks about LTV curves. [00:36:00] I mean, if anyone has is measuring LTV more than like a year and a half, two years, I automatically just be like, you're wrong. Like, just don't even bother doing that unless. You're a Nike, but then you have something like the pandemic. [00:36:12] Then what do you do? All of your models get thrown on and you rebuild everything, right? So you've got, you know, eight year economic cycles. It's just, there's just so much, it's not, it's not worth doing anything more than a year or two. And so that's, so all of these factors made it so that, um, you didn't have to build for that long. [00:36:30] And so what we actually did for even predicting LTV is you could tell. Very quickly, the LTV, a customer based on what they did in their first week. And so all you had to do is build a predictive LTV model of like one week behavior to like model to your behavior for like previous cohorts. Um, but then again, if you want a two year LTV, that means you have to wait two years to even calculate LTV. [00:36:54] So all these things that it becomes, it ends up being an exercise of how quickly. Can you, [00:37:00] how shortly in time can you get a signal to calculate LTV? So it ends up being like a predictive LTV of like three month LTV, six month LTV, knowing that, you know, the ratios between six months and two years you monitor and hope they stay the same over time. [00:37:14] Um, and, and that's how you do it. So at the end of the day, it all comes down to like months. [00:37:21] Phil: very cool. [00:37:22] Proving Brand Campaign Impact Using Geographic Testing --- [00:37:22] Phil: I know that part of your team's role at Uber at some point anyways, was to tie lifts in brand awareness to bottom of the funnel metrics and, and revenue. Um, obviously, and I think you said this in one of your posts too, like you can't use MTA for a national TV campaign. You're missing like way too many offline touches. [00:37:42] You can't use incrementality for a national TV campaign either. It's an opportunity to create a. Control group in the national TV campaign. Um, but you say that you can use incrementality for anything except a national TV campaign. What do you mean by that? And, uh, maybe also touch on like, what did that [00:38:00] process look like for longer term, uh, horizon experiments for brand? [00:38:04] Sundar: Yeah. Um, so this was really fun because. I built Uber's brand data science team. And so we were a team where I was on the performance. So Uber did this weird thing where they split up performance marketing and brand marketing. So there's a VP of brand and a VP of performance. I was the first time where I sort of like crossed this line. [00:38:26] So I reported to the VP of performance, but exclusively worked with people under the VP of brand. Um, and I'm so glad I did. Cause it's like such an eye opener for, you To be on the brand side of things. It's such a different world. And I, and I love it. It's actually like something I'm super passionate about now is getting more people to understand that side, but, um, not to go on a tangent, um, but when you're measuring brand, right? [00:38:52] Like the problem is with national, um, you obviously can't, you can't measure when there's a national TV ad [00:39:00] because there's exposure everywhere. And so what we were trying to do was understand, okay. Maybe not a national, but there are campaigns, um, where you're trying to understand one, the first thing is, does your campaign even move the metric? [00:39:15] It's supposed to, you'd be surprised how many companies. Don't even have that answer, right? They just roll out a brand campaign or any marketing and they're like, we think it's working. So the first part of the methodology was to even understand, okay, is awareness moving. And then what you do is first, once you get awareness to move consistently and you can predictably move it because your creative is good enough, you've got good customer insights. [00:39:40] Then you set up again, like you have to set up as geo lifts generally because brand campaigns are usually multi channel omni channel campaign. And you. Hold out at certain part of your geo and you run all of your brand ad on, not on that. And then you run it on the other part [00:39:59] Phil: [00:40:00] Hmm. Okay. Hmm. [00:40:02] Sundar: impact over not, you know, eight weeks, but your, your, your minimum is 12 weeks. [00:40:08] And so these are 12 to, you know, often three to six month tests that you just have to get going. But it starts with this idea of, we, you have to have trust that if you move the metric that you're meant to move. It will move the bottom impact. Like there's, it doesn't really make a lot of sense where like, if you have an increase in awareness by 20 points, you're not going to get more sessions. [00:40:34] Like why? Like where that disconnect, I think it rarely happens. Um, And so we were trying to do that step by step by saying, listen, we think if we move awareness, it'll transit to sessions. If we can move consideration, then you start to see requests. If you start to see whatever the next stage of the funnel is, you're going to see business impact. [00:40:54] And if you do it consistently over time, we also saw that if you ran a [00:41:00] brand test and then turned it off, we would measure the post effect and the brand value would just drop. [00:41:06] Phil: Hmm. [00:41:07] Sundar: So like, that's the kind of stuff where we would go and be like, you can't do these two month brand activations. They don't work. [00:41:15] They're a waste of money. Like you have to be always on. And that influenced a lot of the strategy to be always on, um, which brand marketers love. Cause they're like, great. Finally you've been saying this for a while. And we loved, cause we were like, you know, this is a very cool, you know, thing that we're, we're measuring and working on. [00:41:31] So like, there was a really cool, like happy marriage eventually in the beginning, it was a little rough because. You're telling people that don't want to AB test or test that you have to do a holdout, but eventually you get to the synergy of we're all trying to prove ROI and, oh, wow, we're, we're proving it for the first time. [00:41:48] So like we all collectively want more budget. [00:41:53] Phil: Do you think it's ever okay to like, not measure something like the national TV ad example that, that we just use, [00:42:00] like, how do we even tackle that? Like, can we agree that campaigns like that, that are cross country, that don't come with a click, that is not something that you can put a UTM on? We'll never have proof of ROI. [00:42:16] Explaining Unmeasurable Marketing Results to Executives --- [00:42:16] Phil: how do you handle execs asking you to measure things that can't be measured? Is it just like a long form conversation and it's a change management approach? Like, how do you have that conversation of the trade offs between, we won't be able to measure this, but it should have brand impact. Yeah. [00:42:35] Sundar: it's a really good question and I'll, and I'll start with the, the worst scenario is. A marketing leader being surprised that there's no measurement. That is, that is the absolute worst scenario, right? Because all of a sudden it literally like these things happen. It's like, Hey, like our CEO of this multi billion dollar company was like, Hey, we just spent this much. [00:42:59] What [00:43:00] happened? What's the proof. And for the CMO to go, Oh, I thought we were going to measure it. Oh, like that is just that, that sucks. So part of the conversation is education around, let's go into it, agreeing we can't measure, and then don't ask for a measurement later. Like, like it's totally okay to your, to your first question. [00:43:22] It's totally okay to not measure things. It's not okay to, to expect data science to come up with stuff because you forgot to have that conversation that's on you. And, and I was pretty like, you know, at some point you have to be careful, right? But I, we were pretty strict about saying like, listen, we're telling you months in advance or weeks in advance, we can't measure this. [00:43:44] We're not going to make stuff up. And like, you kind of, I mean, this is a, uh, not a great way to phrase it, but you kind of need to let marketers get burnt a little, and then you'll find that they come back and they, they want that conversation and then you get brought into [00:44:00] planning meetings and prioritization meetings. [00:44:01] And so, yeah, it is a bit of change management. I, you know, my, my style is like, Hey, I want to help you as much as I can. But at some point we're all adults. You're going to have to learn the hard way. Um, I, I don't like that. It gets to that point, but that's, you know, oftentimes the point does, does get to, but that's, that's what the conversation needs to be around is, is education around, okay, what is our measurement strategy? [00:44:25] Okay. If you want to do a geo test and you're not going to be able to, uh, invest as much budget as you thought. And then it's like, no, we really want to invest in this budget. We're behind targets. Okay. then we can't measure it. Right. And it's this trade off. And like you try the best you can. And sometimes we would say, okay, what, what region what's been going on, you know, in some of the smaller markets that don't give as much love, there aren't as many product changes, aren't as many things going on. [00:44:51] You can do pre post. You can try to try to help them with that. Um, But it's just, again, under, like [00:45:00] being very upfront with level of risk, level of confidence, and, and making sure that no one is surprised by the level of confidence and level of risk. [00:45:12] Phil: Why do you think it is that marketing needs to prove their impact for everything that they do? Like at Uber, you led a team of data scientists who were focused on building measurement methodologies to prove the ROI of marketing. Sometimes it was brands, sometimes it was performance. Did Uber also have a team of data scientists dedicated to proving the ROI of the finance team? [00:45:34] Or the HR team, like, obviously I think the answer here is like, you know, marketing has advertising spends. And for a lot of companies that are big, like there should be measurements as much as possible, but [00:45:46] Why Marketing Faces Unfair ROI Pressure and How to Embrace Measurement --- [00:45:46] Phil: marketing is way more than just paid media. Like finance doesn't have to report on campaign ROI or streamlining payroll. [00:45:53] And HR doesn't justify the head count for like with revenue attribution models. Like why should marketing [00:46:00] strategy operations, creative, Be under this microscope of ROI when we all know that, or at least most of us know that building a brand and refining a martech stack or creating a customer journey, like they're, they're not clear dollar to dollar payback. [00:46:16] It's not a direct response thing. But we still get asked all the time, what's the ROI of this going to be like, what drove the most revenue last month? And you know, we, we don't like other teams skate by with metrics like employee satisfaction or process efficiency metrics. Like if product teams can claim credit for long term innovation with no immediate ROI, why can't marketing do the same for brand? [00:46:43] Sundar: Yeah. Ooh, a lot to unpack there. So I think this, like this sentiment and way of working feels like it's in the last decade. I think if you look at like early nineties and even before like marketing was king, [00:47:00] um, and people understood the importance of marketing, I mean, I think that's why you have the best marketers in the world now. [00:47:10] You know, got their marketing chops in the eighties, nineties, uh, brands, like any of the CPGs, uh, Coca Cola's Pepsi's like, they're just so good at marketing and storytelling. It's amazing because they had to, um, because you were selling a commoditized product. And so I think for the first time in tech, you've got this non commoditized product. [00:47:30] And to be fair, I think product is always asked to prove it's ROI. I don't know many product teams that get away with, yeah, we haven't shipped anything or move the needle. So, so product was, you know, the past 10 years has been asked to do that. And they've been able to do that very well. Right. If you start at like companies like a Facebook or any of the big tech companies, like they've been data driven experimental, like from, from like day one, like Facebook's growth team is legendary. [00:47:56] And so they. Did a really good job of [00:48:00] saying, Hey, look, we're able to prove our, our efforts and we know these things move the needle and I don't think they were putting it back on marketing, but I think there is a bit of a rivalry, right? Like, uh, marketing, getting a dollar means product is not getting a dollar. [00:48:14] And so there is a bit of, and I don't think product intentionally means to do this, but they're just. Able to be better at, um, proving their ROI because a lot of it is first party. They can AB test, et cetera, versus marketing. There's been this. Like influx of performance marketing that became so, and like people loved it for the past 10, 15 years. [00:48:40] They love that they can measure every single thing and they, every single click and, and, and, you know, UTM is only a thing of the past 10, 15 years. Right. So like, so performance marketing also then did it to itself. And then, and then you have these people trained to say, well, [00:49:00] performance marketing can do it. [00:49:01] Why can't brand? So you've got this like weird disconnect of people just forgot that brand and, and sorry, the last thing is brand never had to prove it's ROI. So you have a bunch of people that have never been asked to prove ROI, then competing against both product and performance marketing who have got really good at proving their ROI, or at least. [00:49:23] Perception perception wise, I don't think performance marketers are actually that great. Um, uh, but You have these two teams and then they just forgot and they don't know how. So, right. So there, so you're now starting to see these tech companies catch up on brand data science, and now you've got these PhDs and econometrics, um, people coming in to try to prove the value and everyone that is investing in brand. [00:49:49] Is seeing positive ROI, right. And they're able to prove it. And so I think you'll start to see a bit more case studies and you'll start to see a bit more papers around measuring the impact of brand, but [00:50:00] like you have this weird, perfect storm over the 10, 15 years where, yeah, you didn't have to. They didn't have to before they don't know how to, and then, you know, their, their rival teams all knew how to, and did really well. [00:50:14] Phil: think you said that leadership at Uber were actually the ones that said that since we're spending all this money, let's build a measurement team backed by data science to figure out the ROI of all of this cash that we're thrown into meta. Let's say the tables are flipped though. And, and like, maybe some of the listeners are, are in those shoes too. [00:50:33] And you're advising a company with very little data science resources and they're focused on marketing and the measurement game is very thin. [00:50:43] How to Build Strategic Data Science Teams for Marketing --- [00:50:43] Phil: How, how can you make the case to leadership to build this data science team, or, uh, maybe ask differently? Like, is there a situation where. You advise teams to not waste efforts trying to solve the perfect ROI measurement. [00:50:56] Like, what are your thoughts there? [00:50:59] Sundar: [00:51:00] Oh, um, yeah, I, I would say like, I don't think I've ever advocated for like perfect ROI measurement, because if you're spending that much time measuring, then like, you're probably not spending enough time growing. So it's got to be this like breakneck speed. which also I only work with consumer companies. [00:51:22] And so most of them, you know, the only way you survive as a consumer company is you're growing like crazy. You just, you can't afford to grow slowly. So when you're in that situation, like it's about, again, I'll go back to like communicating what the trade offs are. So it's like, Hey, we'll keep investing in our, you know, in meta. [00:51:42] But I have a hunch that it might not be working as well as we think. So I can either pull back off the gas. It might show up poorly for your Q2 numbers, and you got to go to the board and explain that. Or we keep going, but like, you know, it's that conversation. And I think the [00:52:00] problem is data scientists are not really great at having that conversation. [00:52:05] And so, you know, I don't advocate for T like companies doing this because you have to have a data science leader that can sit and go toe to toe with the C suite and S and have these conversations. And I, I find that really lacking. And I think a bit of a, Tension here. But like one of the biggest problems is that data analysts, data scientists have been sold that data science is a technical field and it's not, it is, it is a very human field that happens to be, you know, have technical aspects to it. [00:52:38] And so one of the things I encourage data scientists, data analysts to do is to think about it is like you are a marketer first, that is really good at data science. You're not a data scientist that is doing marketing. And [00:52:49] Phil: Very cool. [00:52:50] Sundar: yeah. And so like, that's the problem in which case, again, marketing falls into this trap of you don't have great analysts that can, and data scientists that can advocate for the [00:53:00] strategic thinking. [00:53:01] And so then you then have people push and And the last thing I'll add to is for some reason, everybody thinks they can do marketing. Like no, like a CEO doesn't go, Hey, you know what? I don't have a background in finance. I'm just going to go run them with cash numbers. I mean, I'll, I'll handle reconciliation this month. [00:53:20] But every single time they're like, you know what? Marketing's not going well, let me step in. And that's every time. And so you've got to have this pushback and you've got to have people that are able to push back and have this conversation around ROI and the trade offs. Um, and then on the flip side. [00:53:39] Actually, a good example is I went to this company that I was consulting with and they had an amazing CRM team, right? Like the best life cycle team I've seen outside of Uber. Like it was these five women that were just killing it. I like, I like loved working with them and they would run these daily deal campaigns. [00:53:57] And I was like, why are you running these daily deal campaigns? They're an e [00:54:00] commerce company. And they're like, that's how we've always done it. I'm like, do you No, if they're incremental, like, do you know any, like, and so they're like, no. And I was like, they're like, how do we even do that? So I was like, okay, so let's run, uh, for a couple of weeks, you know, third of the group gets zero, a third of the group gets four emails a week and third of the group gets seven. [00:54:19] And what we were able to prove is that the four and the seven definitely have positive impact. So yay, you're incremental already, like a huge thing for you to know that the work you're doing is valuable, but the seven is not more valuable than the four. In fact, you were getting more on subscribes, same revenue. [00:54:34] So now. You know, difference. And so now you can cut your workload by like a huge chunk, you know, three sevens of your workload just gets taken away because you don't have to keep pushing these cat pins out why they were never able to have that conversation with the CEO. I don't know. Um, I, I generally get the dynamic and I know it's not easy. [00:54:54] It's also easy for me. To come in as an advisor to have that conversation. But [00:55:00] these are, these are the trade offs that actually prevent teams from bringing in data science and ROI measurement at the right time. It's less, it's almost never technical. I've, I could, you could give me any set of resources and circumstances and I could find a way. [00:55:17] To prove some part of your ROI, but it's it's you know, the the weakest part of most marketing data stacks are the people [00:55:28] Phil: in your answer there, like one of my favorites was like, tell your advice to tell data scientists that at least work on, on marketing stuff, that you need to figure out a way to be a marketer first. And then have your data analysts or your data scientists hat on after like the technical components come after you understand what the hell the marketer is trying to do. [00:55:51] What are the use cases? What are the strategies? And I think that's like way easier said than done. I have a lot of friends in data science and [00:56:00] data engineering and they just hate marketing. Like they're allergic to it. They love product. Like put me on any product situation. Like. User experience, like UI design, like, I don't care what it is. [00:56:13] As long as it's the product, it's technical. Like I love working on code stuff. Right. But like, throw me on the marketing team and I have to help like a rev ops person or sales something, or just like work on the website. Like, it's just not as exciting for a lot of. Technical folks. [00:56:31] Finding Technical Talent That Embraces Marketing Challenges --- [00:56:31] Phil: How do you make that case that, Hey, marketing is actually fun. [00:56:35] Like we chatted before this and you're like marketer first, you think of a marketer before you think of the technical stuff, like, how do you make that transition? How do you find those technical people that are willing to do that? Because it's not the case, at least in my experience. [00:56:49] Sundar: Yeah, I It's an in an innate thing though. Like I think Like I didn't love marketing I [00:57:00] loved Uber and so I was willing to really understand marketing because at the end of the day, you know, and this is, and this is a challenge that I've had candidly since I've left Uber is to find another company that I'm that passionate about. [00:57:16] Right? So part of it was also, I was just, I was at a cultural moment at a company that, you know, once in a lifetime opportunity. So, so I get that, right? Not everybody's in the same situation where they're that motivated. But I would say you've got to be motivated by something. Either you're passionate about your own success in which it behooves you to understand product and your customer and marketing better, or you're passionate about the people you work with, in which case it's better for you to understand what they're doing, or you're obsessed about the company, in which case it's better, right? [00:57:46] Like there's no situation in, in. In which case it's not better to deeply understand marketing or whatever you're working on, even if you're a product, right? If you look at product and go, I'm only [00:58:00] interested in the UX and the technical component, you've defeated the whole purpose of the product is there in service of the customer. [00:58:07] And so. there are people that are innately customer obsessed and truly like, I like mean that and like, you know what it's meant to be and not just like a buzzword. And then you can put them in any situation. In fact, like you'll, I don't know if you experienced this when you're at WordPress, but like there's a lot of these people that can just jump ship from like one, one problem to another across any team. [00:58:27] And it's, and they always thrive. And it's not because they're good at it. It's because they're that curious and curiosity, I think is a, like, it's something that's hard to test for an interview, but if you could, it is almost the best predictor. And so, so it goes back to curiosity is what. I think made me interested. [00:58:50] Like I would, on my drives, I would take an Uber home every night. Um, one, cause we got free credits, but, uh, but every single night I'd be like, Hey, [00:59:00] you know, to the driver, like, what do you like? What do you not like? And I would tell them I worked for Uber. Right. And, and I would tell them like, listen, I'm trying to help you out. [00:59:07] And a good example of this is I talked to a driver and he was like, Hey man, like this is like my second day. You know, but like Uber told me to drive on like Friday nights. It was super stressful. And so I went and looked at the data and driver retention was like a significantly lower portion, lower, lower number. [00:59:28] If your first trip was during like a rush hour and our marketing comms would push you towards that. Because from our side, it's a business thing. That's like, you know, we want more, you want more drivers during the time. And I was like, Hey, like, I think we should test this. Like, instead of pushing for Friday night, let's do like Saturday afternoon. [00:59:47] It's a lot more calm, like go do your first cup. And like, it was like one or two percentage points, which for retention is massive and for, and for, for supply side retention, and it comes down to this curiosity. So [01:00:00] people that are like, you know, in places where they're like, they're not excited, we'll find a place that you're curious about what they're the product is doing, you're going to be infinitely happier when you like the product and you like the people you work with. [01:00:12] It's just, it's just human nature. So you've got to find a way to feed that curiosity. [01:00:18] Phil: So cool. Such a great answer. Yeah. Curiosity is something that comes up a lot on the show when we're talking about hiring and like who is good and marketing operations and marketing technology. But it's, it's really cool to hear your answer and how that translate to not just marketing, like folks that are curious. [01:00:34] across different business teams do really well because they care deeply about the customer. Um, yeah, it's a super cool answer. This has been super fun. Sundar, I feel like we can keep chatting for probably a couple more hours there. I'm pumped for your upcoming podcast. By the time this drops, uh, you're probably going to be releasing it shortly. [01:00:55] Um, I'm excited to get more of that B2C side of, of, of [01:01:00] marketing. B2B has been more of my jam, but, um, [01:01:04] Managing Work Life Balance Without Burning Out --- [01:01:04] Phil: I got one last question for you. You're a data practitioner, marketing advisor, or writer, upcoming podcaster. You're also a husband, father, a well traveled home chef, as well as an amateur chess master. Uh, one question we ask everyone on the show is how do you remain happy and successful in your career? [01:01:20] How do you find balance between all the stuff you're working on while staying happy? [01:01:25] Sundar: Yeah. And I know we had briefly talked about this before, but, um, Over the past couple of years. So, you know, March and April, we had some personal stuff happen to us and it really puts stuff into perspective. Um, you know, three deaths in three weeks and you, and, and, and one was 99, one was like 67 and one was 33. [01:01:44] So like, just like a real like eye opener on life. And I think for me, one thing I've, I've been very big on is like, really not, not trying to manage stress. But just preventing it. So before you get to the point of managing it, [01:02:00] like I just try to like, not even get there and it's an easier thing to say than do, but like what I've, my philosophy has always been that you can not, you can control things and you can't control things. [01:02:11] The thing you, things you can control, if you do your best effort and you try your best, that's okay. And if you can't control it, then like. it's, you have to figure out a way to not get stressed about it. Right. It's the things that you can't control that always get people. And for me, I've just built up, I think like some resilience towards that. [01:02:29] And that's kept me happy because, you know, I would say like many things like short term, there's going to be fluctuations. It's impossible to always be happy, but like you've got to look at a longterm trend. Right. And if you think about it, like stocks, like you're going to take some losses in the short term, but like, If you've got this longterm perspective on how you want to invest in yourself and the relationships that you want to invest in, like if you do that right, repeatedly every day, as much as you can in the longterm, you're going to be happy. [01:02:58] And so an example for us [01:03:00] is, you know, we left our families and we came to Amsterdam and we're not looking back. Like, like it's unlocked a level of happiness that if we had thought too much about it, we would have never even gotten to in a reach. So for me, it's just. You know, trying to put myself into situations and see where my energy takes me. [01:03:20] Um, and then being comfortable revisiting some assumptions, like for example, like I grew up in an Indian household and work and career are always the number one thing. And I'm very comfortable accepting and like, I will never put work first, like I, you know, I did at a certain point in my life. I've now got, um, you know, my wife and my kid. [01:03:42] And like, it's like pretty clear difference between how much I care about them and how much of a care about work. Um, it doesn't mean I'm going to be bad at my job or crappy or like, you know, it just means like, I don't care. I've been very ironclad on like my priorities and like we've talked about [01:04:00] like what brings me energy and if it doesn't bring me energy like again like with the stuff that happened in March like it's not that life is too short it's just life is just too unpredictable that like I'm not going to chase stuff that doesn't give me energy um there's no point in just sitting there being like unhappy because it bleeds into every part of your life and it makes you just like a miserable person so so yeah so long answer apologies um but [01:04:25] Phil: no, it's a super profound, powerful answer. Really appreciate it. My first thought that popped into my head when you're like, we, we left our family, moved to Amsterdam. Um, I cannot imagine raising a small child without like the grandparents living close by and helping out when they're sick from daycare or just like giving us breaks here and there. [01:04:46] Like how, how are you guys doing that without, uh, grandparents nearby? [01:04:50] Sundar: Yeah. Well, since we've had the kid, I've been on this solo preneur journey, so I've had a bit more flexibility. But also, this is like what I mean by we found this new level of happiness, like we're in a [01:05:00] system that lets you have that work-life balance. Like there's no part of me that, or my wife, you know, she's been a full-time job. [01:05:08] Super stressful, super high opposition. There's very understanding, right? So if the kid can't go to daycare, there's not a question of like, Oh man, should I tell them, should I take a sick day? Like you're in an ecosystem. That I think just puts families and people first. Um, and so we won't, we won't get into my thoughts on the U S even though it's where I'm from, but, um, you know, we're just, we're in an environment where people get it. [01:05:36] Um, and. We've actually, like, found that, like, you know, we would love our grandparents to be, like, be around because we want them to have a good relationship with our kids, but, um, we haven't, like, felt the need, like, we're, like, struggling without them, and, and I, you know, we know a lot of expat couples here, and they're kind of the same thing because, like, you're just not in this daily grind where you feel like you [01:06:00] need grandparents sometimes to help you out, like, you know. [01:06:03] You can, you know, it's, luckily for us, our kid has also just been great. So I think there's a little bit of luck at the draw there. He's, you know, he's a, he's a good kid. So [01:06:12] Phil: Awesome. I really appreciate your time. Sundar, Uh, I'll link you out to, uh, the newsletter and, uh, I'll be sure to help, uh, support the launch on, on the podcast, uh, excited to follow on that journey there and, uh, yeah, it's keep in touch, man. It's been super fun. [01:06:26] Sundar: So this has been awesome. Thanks so much for your time, Phil. And, uh, yeah, like you said, let's keep in touch. [01:06:31] Phil: Cheers.