[00:00:00] Phil: you've argued that predictive models, so aren't really useful for marketers because they only tell us what is likely to happen if we don't change anything like instead, marketers need to graduate to causal levers. [00:00:11] Tobi: I mean, I've been obsessed with this problem for the last 10 years, I think marketing is ripe and open to counterfactual and causal thinking. and it does, lead a little bit to the realization that, you know, predictive models may be an input to something, right? [00:00:24] But it's an intermediate step. if I say, this person has this kind of churn likelihood, What, what does it, what does it do for me, right? [00:00:32] I need to understand what reduces the churn causally, right? So yeah, sure. I can take this prediction, segment it, and then run an experimentation program on top of it, And I think there is better answers now, and I think we'll get into that know, correlation does not address the jobs to be done by the marketer, which ultimately is all about a causal language. [00:00:55] ​ [00:01:22] In This Episode --- [00:01:22] Phil: What's up everyone? Today we have the pleasure of chatting with Toby Connier, VP of AI at Growth Loop. Toby holds a PhD from Stanford in computational social science and worked at Facebook Research. He also founded two startups, predict Wise and Later Accurate and Ad Tech startup that predicted LTV, that was acquired by Phoenix Commerce. [00:01:40] In this episode, we cover why predictive models fail without causal inference, why dynamic allocation works better than fixed Horizon AB testing. The Boomerang effect and why uninformed AI sabotages early results, the power of composable decisioning, and how machine decisioning transcends marketing, all that, and a bunch more stuff after a quick word from two for awesome partners, [00:02:02] ​ [00:03:40] Phil: Tobi, thank you so much for your time today. I'm really excited to chat. [00:03:43] Tobi: Uh, Phil, so much to get into, uh, the disastrous effect of Agen ai. One thing I love to talk about, ~um, but you know what, uh, maybe we should start with an O to San Francisco. You'll start light and, uh, uh, it's, it's, it's a great place to be, although it gets a lot of unfair, uh, representation in the press. Uh, why don't we start there highlighting that part.~ [00:03:49] Phil: ~Yeah, yeah. You've been in the Bay Area for a while now, right? I, I was looking at your LinkedIn, like you studied at Duke and then you had internship in New York at Microsoft, but then it was Stanford and you've pretty much been in like Palo Alto Bay area for like over a decade now, right?~ [00:03:49] Tobi: ~Yeah, I stayed. Um, and I think there is something really cool about, you know, Palo Alto is a sleepy town, which you may know it was home of the first tech boom, ~ [00:03:49] ~but I think there is something to be said now about San Francisco actually being the epicenter of technology. And I think that matters. Um, and that makes, I think this area, particularly the urban area, particularly, um, cool and interesting to live in.~ [00:03:49] Phil: ~But originally from Europe and you said you were just traveling recently in, in France, you know, three different languages.~ [00:03:49] Tobi: ~Yeah, that's right. I'm, I'm European, um, which, uh, you know, depending on my day's mood, I, I think that's the answer to all my problems or, you know, the, the bane of my existence. Um, I'm originally from Germany. Um, and I came to grad school, which is initially supposed to be a year long thing, and I never really went back home.~ [00:03:49] ~Um, yeah, along the way. I lived in France for a year, um, and have been in the bay since 2013. So for a while~ [00:03:49] Phil: ~Very cool, very cool trajectory. Uh, for sure. I think a lot of folks end up staying in the Bay Area. I've had the pleasure of visiting, uh, multiple times. I have a cousin that has a startup down there. So, um, something that you did start doing a lot more recently also is writing on LinkedIn. So when we connected, I uh, discovered you through the multi-part series that you were writing on correlation versus causation after you joined Growth Loop.~ [00:03:49] ~And I'm excited to let you dig into that a little bit deeper today. Uh, ~the first thing I wanted to ask you about is the [00:03:51] 1. Why Predictive Models Fail Without Causal Inference --- [00:03:51] Phil: prediction trap, if you will. Um, you've argued that predictive models, so a lot of the propensity based models that, uh, companies are using today aren't really useful for marketers because they only tell us what is likely to happen if we don't change anything like the current status quo. [00:04:07] So we're asking things like who is likely to buy? Convert who's likely to churn. The function is observation, and the outcome is describing the future if we do nothing else. But marketing is all about doing stuff, new channels, new campaigns. So you're saying that instead, marketers need to graduate to causal levers. [00:04:26] So instead of asking who's likely to buy, we should be asking, how can I make them buy? And instead of saying who is likely to churn, we should be saying, how can I prevent someone from churning? So we're changing the function from observation to intervention, and the outcome is like instead of describing a future, if we do nothing, we're trying to describe how to change that future. [00:04:47] Do I have that right? Like maybe explain the prediction trap as you see it and why A highly accurate observation of a customer's current trajectory is way different than having the causal levers to actually influence that [00:05:00] trajectory. [00:05:00] Tobi: Yeah, I think first of all, Phil, that's very well put. I mean, I've been obsessed with this problem for the last 10 years, and it's actually part of my trajectory a little bit as well. You know, um, before joining Growth Loop, I was CEO of a prediction company that did, uh, customer lifetime value predictions. [00:05:17] And it did it very accurately, and it did it, you know, with, with, uh, with deep learning and, uh, you know, taking, um, user level information, proprietary data, event stream data, all this kind of stuff to get prediction accuracy to the new level. Um, and I, you know, I was fairly new to the, to the field of marketing. [00:05:36] Um, and, you know, we, we sold into D two C commerce marketers, right? Um, and you know, I, I remember that, you know, we had an easy time selling because we could prove that our LTV predictions were very accurate, [00:05:49] right? But I remember sitting there with li particularly, um, the lifecycle side with marketers that asked really good questions like, you know, what do we actually do with LTV [00:06:00] predictions? [00:06:00] Do we target, uh, you know, for, do we target high LTB customers more? Do we, do we give them more expensive products? Do we give them a discount? And, you know, we didn't have an answer to that because, you know, the, the, the nature of predictive models is, as you said, it represents the status quo, right? What will be this customer's trajectory if nothing changes, [00:06:22] right? [00:06:22] And quickly realize that obviously the world of marketing is a causal world, as you said, right? Like, that is the job of lifecycle marketers. Um, if I change X, what is can, what is the outcome? And y right? How can I maximize Y otherwise there is really no point for marketing if everything is status quo and forecasting on status quo. [00:06:46] And that's the extent of it, right? There is no marketing role [00:06:48] here. Um, and you know, I think we'll get there, but this is not, this is not limited to, to marketing. You know, I would say now where I'm now. Um, I don't understand, for example, [00:07:00] in grad school why I was not obsessed with outcomes-based thinking and reinforcement learning. [00:07:04] Why did I even, you know, waste so much of my energy on, on prediction problems, [00:07:08] right? It's, you know, I mean, um, I'll tell you that my, my, uh, my wife for example, works in diagnostics in, um, in healthcare or health tech, and it's the same problem there, right? You can predict that, um, you know, this is representative of a certain cancer, it can do early detection, but the real question is what intervention moves the needle, [00:07:28] right? Um, and I actually think, you know, the interesting thing about marketing is that, that I think marketers, you know, first of all, they have a really hard job. Um, and second of all, I think they, they, I, I think marketing is ripe for this. I think marketers get it right. I think marketers understand that, um, correlation based, and we should get a couple of examples. [00:07:50] But correlation based thinking really doesn't move the needle in a meaningful way. Um, I think the other thing that I've discovered is that boardrooms start to [00:08:00] care too. Boardrooms start to ask much more causal questions, right? Um, you know, framed in terms of ROI, but it's ultimately a causal question, which is, you know, what, what did the intervention cause, um, when it comes to the top line. [00:08:13] So I think there is this, um, this trajectory change of marketing as a, as a field. I think marketing is ripe and open to counterfactual and causal thinking. Um, and it does, you know, coming back to your, to your entry point, um, it does, you know, lead a little bit to the realization that, you know, predictive models may be an input to something, right? [00:08:36] But it's an intermediate, it's an intermediate step. Um, there is no point, you know, and I, again, let, let's put, uh, let's go to marketing. I'll give you one example and I'll turn it back over to you, right. Um, if I say, you know, this person has this kind of churn likelihood, right? What, what does it, what does it do for me, right? [00:08:55] I need to understand what reduces the churn [00:09:00] causally, right? So yeah, sure. I can take this prediction, segment it, and then run an experimentation program on top of it, which ultimately, you know, back to my days at Accurate, which is the name of that company that I founded. That was our answer. And I think there is better answers now, and I think we'll get into that deal. [00:09:16] Um, but, but that I think is what we mean by, by correlation or what I mean by, you know, correlation does not address the jobs to be done by the marketer, which ultimately is all about a causal language. [00:09:32] Phil: Yeah. Okay. May, [00:09:33] 2. How to Validate Causal Impact on Customer Lifetime Value --- [00:09:33] Phil: maybe we can use LTV as an example, uh, to, to kind of unpack this. I feel like LTV is is really close to your heart and, and you were really deep in, in that space. It's, it's often like the gold standard when marketers are thinking of outcomes for experiments or like a research design. Um, maybe like, do you have an example with LTV where like correlative impact on LTV was dressed up as like causal returns? [00:09:56] Like how, how can we think about that? [00:09:58] Tobi: Yeah. [00:10:00] Um, I, I do think it's a good example and I'll give you two. So, so one, um, you know, it's this kind of interesting, you know, I said that there is this step change in thinking. Um, it's also true that, you know, six months ago I had conversation with, um, a head of CRM or Lifecycle Marketing of a billion dollar outdoor brand. [00:10:19] Um, and, you know, essentially we, we asked, you know, or I asked, what, what, what do you do when you know that somebody is predicted to be high LTV? What do you do with that customer segment? And for him, the answer was obvious. He said, oh, you know, what do you mean we're gonna spend more money on that segment? [00:10:33] That's obvious, [00:10:34] right? And it was sort of that, that was still the old way of thinking, right? It was if you, if you wanna be technical, it's the wrong answer. So, but what does it mean? So, you know, let's, let's give a specific example. Let's say, um, all customers that are high LTV, um, have looked at a certain, um, product in the, you know, maybe as an entry point, you know, looked at genes, right? [00:10:58] A certain style of genes, right? [00:11:00] So now you say, okay, well everybody who has ILTV. Uh, has looked at this product in the, the welcome flow. For example, let's kind of make this the mandatory option in the welcome flow, right? Will this causally increase LTV? You don't know, right? It, it is not a causal statement. [00:11:18] It has there, there could be all kinds of reasons. You know, maybe it is true that, you know, um, the ad that you, that you, uh, uh, that you sent to Facebook advertise this. And you know, that segment was a wealthier segment for all you care about, right? Which we call spurious correlation, right? It's associative. [00:11:39] Yes, there is a relationship there. But now the key cause of question is if you tune that lever, right, um, will you get an outcome? And the answer causal outcome? And the answer to that is you don't know. [00:11:54] Phil: Yeah, it's, it's a really good example there. Um, like when I think back of my [00:12:00] earlier days in, in propensity modeling, like the first time I discovered this was, uh, my short stand at, at wordpress.com, we had big company full data team that was servicing and helping the marketing team. And they had built a homegrown ML propensity model that was, uh, attached to our CDP homegrown CDP. [00:12:18] And like we could try to predict the likelihood that a certain segment of users would churn or would like convert or would graduate from a free plan to a paid plan. And I feel like a lot of. Teams within the company we're obsessing about, uh, uncertainty or like, how accurate is that prediction going to be if I wanna like do this after? [00:12:39] And like you said, the focus wasn't on. Like what is it, what is, what is the thing that I want to do to like make that outcome change? And. [00:12:48] 3. Reducing Uncertainty Around Causal Effects by Optimizing Levers, Not Labels --- [00:12:48] Phil: Maybe you can chat about like the role of uncertainty here, because like, like I said, most teams are like tightening uncertainty around those predictions and labels. [00:12:56] And if we're gonna rely on the model and this user is gonna churn, like we need to [00:13:00] be at least 90% or 95% confident in that prediction. Like we're all focused on guessing that future user state. But you read a lot about how the goal should be reducing uncertainty around the outcome of the intervention. [00:13:12] Like we just talked about, not trying to guess that user's tape, because like what do we do with that information? So in other words, like instead of focusing on how confident we are that this user is gonna churn, we should be focusing on how confident we are that this intervention, this new campaign, this new email, this new message will prevent this user from churning. [00:13:31] How does focusing on the estimated effect of our action, like changing a channel or a new offer, make marketing more predictable? Can you unpack that for us? [00:13:39] Tobi: Yeah, sure. And I'll, I'll start, you know, I'll start by saying something that I said before, which is, marketing is not an enviable job, right? It is a, it's a hard job. Like what we're talking about is, you know, I think if you, if you wanna be a good marketer, you have to understand the fundamental problem of causal inference, [00:13:54] which is ultimately what we're talking about here, right? [00:13:57] What happens to customer X, uh, [00:14:00] uh, with that intervention, or exposed to that intervention, and what happens to customer X if you hadn't been exposed to that [00:14:08] intervention, Right. [00:14:09] That is an unanswerable question. Um, at the end of the day. Now to your question, what, you know, when it comes to uncertainty, um, I think this all relates to predictive models being maybe a, um, uh, an intermediate step that is, that is fine towards the end goal, right? [00:14:30] So you could say something like, um, and, and we, you know, we did this many times. So you could say something like, yeah, I can, I can build, um, associated traits that are associated with higher LTV and I can build hypotheses from there, the cause or result of which then I need to test to a b testing, right? [00:14:50] So that's fine. But now imagine, imagine kind of the role of uncertainty, right? Um, if you kind of think about uncertainty [00:15:00] for all these different steps. So now I have my predictive model that maybe generates these associative traits of LTV, right? Um, that has uncertainty with it like any. Right. Um, now you are essentially talking about, uh, you know, again, I think that's not a, that's maybe not a clean model, but how do you translate associations into testable hypothesis? [00:15:21] Right? That is a, um, you know, I'd say that's a statistical model, right? It has uncertainty, um, uh, in it, right? Then you, the next question is, uh, what is the uncertainty around the tests that you're doing? The AB testing, right? Um, it has uncertainty to it all that uncertainty compounds, and that's my issue with kind of treating these, these things as end goals in themselves, right? [00:15:47] We, we clearly said that they're not, they shouldn't be. They can't be. 'cause it's not what marketers are here to do. It's not the jobs to be done, the marketer, right? But the other issue is, if you think that way, then you have. A couple of [00:16:00] unrelated steps. Um, each of them come with uncertainty. That uncertainty compounds, right? [00:16:05] As opposed to if you have an outcomes based thinking and you could say, yo, really all I care about, you know, I wanna move X, what happens to Y? Right? I don't care about anything else. That, that's all, that's all I care about as a marketer, right? Um, let me try to, to actually distill all my uncertainty into that outcomes based model. [00:16:26] Um, that is easier said than done, but I think it maybe shows why, um, the sort of focus on predictive accuracy of this step one. And in general, the focus on predictive models, um, can lead to problems down the road because uncertainty will get out of hand, right? [00:16:44] Phil: Yeah, so, so [00:16:45] 4. Why Dynamic Allocation Works Better Than Fixed Horizon A B Testing --- [00:16:45] Phil: you mentioned AB testing as a way to validate that. So it's not just correlative. We, we know for sure if this intervention is actually gonna affect the outcome in this cohort of people that we're predicting might churn or not. Um, you, you've talked about dynamic allocation when it comes to experimentation as a better way than just doing the standard AB test. [00:17:06] Um, I, I, I was trying to like figure out like what are other terms to call, like dynamic allocation. I've seen some folks like call this like adaptive experimentation, but it's all like a bucket of reinforcement learning or contextual bas armed bandits. I've also seen folks like these are all terms like. [00:17:23] Data folks are super familiar with. I think a lot of marketers right now are hearing you say, like, be bandits. And they're like, what the heck are you talking about, Toby? Um, can you walk us through like bandits and dynamic allocation? Um, these are like specific types of reinforcement learning, right? [00:17:38] Tobi: Yes. Uh, that, that's right. I, and, and now we're could of at the heart of decision or decisioning science, which is really interesting conundrum. Well, you know, back in the day, um, what you did, and this is, this was also I think, the gold standard of the internet age, right? [00:17:54] UAB tested, you found the winner, and you scaled the winner, right? [00:18:00] Um, and obviously internet companies, first of all, are admirable to that world because there is this immediate feedback loop of data, [00:18:08] right? I mean, again, we're gonna talk about marketing here. One of the things that's so cool about marketing is that, which also makes a marketer's job even less enviable. [00:18:16] Unfortunately, there is all the data at your fingertips, right? There is so much data and it's just not equivalent to other fields, right? In medicine, you can't, AB test doesn't work, right? Like you can't, uh, for ethical reasons, for other reasons, but in most fields, actually, it doesn't [00:18:30] work, right? You can't just, um. [00:18:33] Uh, you know, test one treatment against another on cancer patients and scale the winning treatment. Right? So it's sort of interesting. I mean, many fields by definition are more stuck in this correlational world because there is no way to get to that causal world. Marketing is [00:18:48] not one of them. Um, right. [00:18:49] But to your question, um, what is the difference between dynamic allocation and your right, that is ultimately reinforcement learning and traditional AB testing? Well, traditional [00:19:00] B testing essentially splits the traffic in our world, right? In a random way. So, you know, uh, ID ultimately, it'll lead to something like 50% will see that intervention. [00:19:13] 50% will [00:19:14] see that intervention. But the key thing is random assignment. So for every user who flows to the system, the machine flips a coin or you flip a coin, right? And there's all kinds of really great statistical properties that come from this approach, which is now, if you just look at the. The mean return of each condition, right? [00:19:34] That's causal, that's causal difference, right? That's, you know, that's, um, probabilistically true, right? Um, so if I show, if I essentially randomize and I show you back to our old example, right? I show you the status quo, or I show you this pair of jeans that high LTV customers clicked on before their first purchase, right? [00:19:53] And I measured against LTV. Now that mean difference is causal great, right? Um, but [00:20:00] you are wasting, you're wasting time or traffic, right? In the inferior condition will get 50% of traffic allocation. And that there is a real cost to it, right? Like, that's the whole, that's the whole trade off of experimentation. [00:20:15] You as a marketer, you say, you know what? I found this great. I, I am convinced, you [00:20:19] know, I figured out that, um, sending handwritten cards to, to dark owners, if you're like a pet food company or something, you know, like congratulates them for their birthday. Um, doc's birthday will increase LTV by 15%. Like I'm freaking convinced about this, right? [00:20:35] Um, now you run the experiment, but if you're so convinced about it, remember you are withholding that treatment from the, the other 50%. You know, that's a real bummer. Um, now the whole point of dynamic allocation or reinforcement learning is that you could do both things in parallel. So as opposed to you run an experiment for a fixed timeframe and then you scale the winner, [00:21:00] right? [00:21:01] Um, you basically say, okay, well you start random because all you got, right? But dynamically, as soon as there is some data coming back to the system, you actually, um, pipe more traffic to the condition that is winning, right? Which is, um, which at the end of the day is more efficient. Um, but there is also a real trade off here. [00:21:23] You know, that that's worth noting. We, we call it exploration versus exploitation. Right. Um, these things are traded off of each other no matter what approach you choose. Now, the interesting thing, and that's why I started with decision signs, is, you know, if you wanna optimize in a metric, which again, is the marketer's job, right? [00:21:42] And again, I, the way that I would describe the lifecycle marketing job is you have to optimize LTB. I mean, what else is there, right? [00:21:49] Like, that's what you got. You start with a known customer profile in your CRM or CDP or whatever it is now. Um, and, and you max you, you max, you maximize the squeeze, right? [00:21:58] That's your job, [00:22:00] right? So ultimately, I think marketing is about optimization. Um, and then there's this really neat paper out there, um, that is, um, uh, I think it's Gary Va Kaufman Etal, um, information processing systems that shows that if you do experimentation first and scale later, so you have a fixed timeframe in which you do experimentation and then you scale, you actually suboptimal. [00:22:23] It's much better to do this dynamic allocation. [00:22:25] Phil: Hmm. [00:22:26] Tobi: So here is a question for you. Um, if that is the case, why do some of the most sophisticated companies in the US do this other approach of fixed timeframe experimentation and scale it up? You know, I mean, these guys, these companies like Netflix and, you know, these, these companies have a bazillion of PhDs that understand decisioning science much better than I do. [00:22:48] Um, and they, they kind of center on this wrong approach. [00:22:52] And that's an interesting question, isn't it? It's, I think it's one of the paradoxes of decisioning science is applied to, uh, to [00:23:00] marketing writ large. [00:23:02] Phil: So, so why is it that, that, that happens? Like, is it, like I, I'm, I'm listening to you explain it and I, I get it. Like I want, maybe I can unpack that, maybe I wanna make sure I do get it. So, normal AB tests 50 50 split randomized split, and we're basically, the trade off is we're essentially wasting a lot of potential customers on the weaker option while we're waiting for enough data to reach stat zg, um, dynamic allocation is. [00:23:31] Shifting more traffic towards whatever is the leading variant, like performing better. But we're still reserving some traffic and we're dynamically assigning that and we're still learning and avoiding being fooled by randomness. Like that's the part where it's like, I get it, but also how do we explain that? [00:23:49] Like AB test, I feel like is just so standard. Everyone just gets it 50 50 random, that's it. But in the dynamic side, it's like, yeah, we're not randomly assigned it [00:24:00] anymore. We're doing it dynamically, but it's still random and we're still gonna hit stat sig. Like is that the trade off there? Like why are some folks still doing regular AB tests when a lot of people know that there is a better way to do it out there? [00:24:13] Is it like explainability versus like what we know is actually better? [00:24:17] Tobi: It is explainability, but I think in a different way. And by the way, you, you brought up a really good point that I sort of omitted. You know, even in dynamic allocation, you essentially say, Hey, you want to build some noise into the allocation process so you can explore right options that even initially are maybe suboptimal. [00:24:36] And this is exactly this trade off between exploration, exploitation, or I can say it in a different language. Um, you can learn and you can optimize, but these things trade off each other. Right. So very simple example. Let's say you start this thing and you're like, oh, holy, holy cow. The, the handwritten note to the dock owners improves LTV by 20% biggest intervention I've ever come up with. [00:24:59] You know, [00:25:00] now I'm gonna be CMO of the company very soon, right? So, you know, you put more and more traffic there, right? But the more traffic you put there, the less you learn. So these things do trade off of each other, right? But again, I would make the point that marketing for most instances, if you're a lifecycle marketer, you're not in the business of measuring results. [00:25:21] You're in the business of optimizing. I think there is, there is edge cases to that. If you work with a software vendor, for example, you know, one of my last companies that was chief innovation officer of was, uh, um, did, uh, estimated delivery dates on the, on the, on the product pages, right? Based on supply chain data and all this good stuff. [00:25:40] Real time ml, really cool stuff, right? If you, if you sell that. Um, into a, into a consumer facing company. Then the role of the consumer facing company, I think is to really just measure that, right? You can't optimize anything but lifecycle marketer writ large. I think the goal is to optimize, right? You want to, that's, I mean, [00:26:00] every, every CMO has this, or every, you know, director of CRM or whatever it is, VP Lifecycle Marketing has this written to their North Star, right? [00:26:09] Increase L TB bags. We talked about it. That's [00:26:10] not controversial, right? Um, so in that case, dynamic allocation is more efficient and there's a mathematical proof in that paper that I mentioned, right? Um, there is something about explainability here though, and that's kind of interesting. So if you look at companies like Zalando and Expedia, and I mentioned, you know, these guys tend to do, um, experimentation and scale. [00:26:33] And the reason is in, that's kind of the psychological aspect of decisioning science is internal stakeholders, right? Dynamic bandits are harder to explain. So, you know, you, you, you, basically, your goal is to affect the change. [00:26:49] Phil: Hmm. [00:26:50] Tobi: say you work, you know, you, you let, again, I don't wanna call out specific companies, um, but let, let's say you've, you've discovered, you know, that, that really, there is a [00:27:00] big benefit in this LTV, the handwritten note example, right? [00:27:03] Like, let's say that that test comes back positive, right? You're not done yet. You still need to convince the CMO or whoever it is to actually enact that intervention. It's a net no intervention, right? So you'll have to have explainability of the process, right? And it just turns out that in boardrooms and at the executive level, um, that more simple approach of fixed learning and then scaling, it's just easier explained. [00:27:31] Phil: Hmm. [00:27:32] Tobi: So it's, it's, it's almost like, you know, it's, it's, it's a human bias that leads to a suboptimal outcome, right? But. If you're the person enacting this, like if you're the PhD who leads decisioning science, right? Um, you're paid to do the change. So your job is to actually convince the stakeholder. And if you do dynamic allocation and you show, you show that, uh, all the traffic goes to one side, but the stakeholder shut you down because it doesn't understand the [00:28:00] principles of dynamic allocation or armed bandits. [00:28:02] Yeah, I mean, you, you're nowhere, right? Like you won. But it's, but it's not, it did not turn into an actual win. And so, you know, that's one thing that I've been thinking about too that obviously relates to change management, change management and things like that. But it's, but decisioning science, I don't think just means, you know, how can we make the most optimal decision? [00:28:22] It actually also incorporates, you know, how do we make sure that the decision gets implemented companywide. And that's a psychological aspect. And I think that is why, um, in many of these companies we're stuck with a slightly suboptimal, um, approach, which obviously still beats the status quo. Right. Oh, again, the handwritten example. [00:28:44] If you, if you, you know, if you spare the cost of the control group because you do dynamic allocation and you find there to be a big winner, but then the change doesn't, is not implemented. Right. That's actually a worse outcome than you running the experiment and, um, and scaling the winner. [00:28:59] [00:29:00] Right. And that the paper doesn't discuss this because this is, you know, beyond mathematics, it's psychology. [00:29:04] But I've been obsessed about that too. How, [00:29:06] you know, where are the limits of decisioning science? And it's not, it's not, you know, math on, on, on, on Global Optima only. [00:29:15] Phil: Yeah, it is such a cool topic. I, I feel like there's a whole episode to do on like the, the change management side of experimentation and how ab tests are easy to explain. Everyone kind of grasps it 'cause it's been a concept forever. But when we introduce terms like bei bandits, like you start losing folks really quickly, there's probably a whole episode we can do around that. [00:29:34] ​ [00:31:38] 5. The Boomerang Effect and Why Uninformed AI Sabotages Early Results --- [00:31:38] Phil: did wanna let you talk about the boomerang effect, uh, related to this. Like there's, you know, a lot of self-learning systems that are plagued by the boomerang effect. You wrote a lot about this. Like they, they start by assigning random and uninformed treatments. How can these systems which lack causal priors, if you will, like, end up sabotaging big revenue [00:32:00] KPIs for weeks and months? [00:32:02] During this like initial uninformed learning phase, um, before they actually start learning and, and actually start driving revenue. Like maybe walk us through that. [00:32:10] Tobi: Yeah. Okay. Let, let's, let's do one segue, right? So in your world, now we're in this better world where we do dynamic allocation, [00:32:17] right? So that's the world that you're describing. So, you know, and I, again, I think I, I think that's probably where marketing almost is, um, particularly at innovative companies, if you don't have to deal with the organizational width that we just described of, um, of, uh, behemoth companies, right? [00:32:33] Um, so now we're saying great. We have, um, and again, maybe we bring it back to the example of LTV, just to make it a little bit more concrete, right? We have a bunch of different treatment conditions, like we can send the handwritten no to the dog's birthday. We send a, are you a dog owner? [00:32:49] Phil: Yeah. [00:32:50] Tobi: Okay. So tell me what other good things there are. [00:32:52] So I'm, I'm like as creative as a fridge. [00:32:54] Phil: Um, automated, like food deliveries or like treat [00:33:00] deliveries, uh, dog walker ads, I don't know. [00:33:03] Tobi: love it like a dog walker gift certificate maybe. So you can go, you know, you can take your spouse on a vacation that has been suffering from [00:33:10] this dog that you brought into the home. You know, the, the marriage is on the brink of collapse. Right. So, so, but there's, there's a bunch of really good ideas right now. [00:33:20] You know, we kind of expose the ideas to, and I like your word of reinforcement learning. I mean, the, the industry now calls it AI decisioning, but that's ultimately what it's, right. It's the system essentially starts random, it listens to the results, you know, and then it allocates traffic dynamically more and more to the winning conditions. [00:33:38] Right. Um, here's the problem though, and I'm gonna turn this back as a question to you. How did you, I mean, actually you chose it for me. So how did you choose the, the, the universe of treatments? How did you do that? [00:33:51] Phil: Uh, well for this example, it's just off the cuff, but personal experience, like my own personal bias. [00:33:58] Tobi: And that's probably how many marketers [00:34:00] do something like [00:34:00] that, right? Um, um, and I don't wanna discard this either. I'm not, you know, intuition and institutional knowledge is a, is a thing, right? It's, it's not, it's, you know, I don't wanna come here and say there's a, um, but there is a danger here, right? Um, the danger is that five strategies that you just told me about actually, backfire. [00:34:20] Okay? So let's make this more. So, you sent me this gift certificate for the dog walker, and I'm gonna say, what the hell? I mean, this guy thinks my marriage is on the brink of collapse because I brought this puppy, but my wife loves this puppy. You know, how, assuming is this, of this dog food company that's, you know, I'm never gonna buy there again, right? [00:34:39] Problem, right? I mean, the system is gonna learn it, [00:34:44] of course, right? You're gonna allocate less traffic to that condition. But in the meantime, you actually have a lowering of LTV, [00:34:53] because I just went to dog food, company B. That leaves me alone. It doesn't devil in my private life. And I mean, these are [00:35:00] silly examples [00:35:01] and you know, [00:35:01] hopefully they're [00:35:01] Phil: it makes [00:35:02] Tobi: and, and maybe they're not funny, but, but I've seen this many, many times in my career. [00:35:07] I've seen this with LTV right? Many, many times where you'll forget decisioning or not forget reinforcement learning or not, but your initial idea is backfire causally, right? Um, so, you know, that is one problem. Even if they don't backfire, let's say they're net neutral, right? You're still wasting space. [00:35:28] Right? And even worse, and that's kind of an interesting question. You, you are stuck in the, in the treatment suggestions that you made to me, right? How do you explore new things, right? [00:35:41] Um, these systems are not well set up to answer this. And this is. Where, you know, uh, an idea comes in that, that we've been building towards and I've been obsessed with for a long time, which is tying, I mean, I, we haven't talked about this. [00:35:54] I, uh, we're gonna hopefully talk about agentic AI a little bit. Uh, oh, I did it. I, I said it in my [00:36:00] opener. You know, I, I am probably a skeptic. um, um, but one thing that I, that I have been really focused on is how to tie agentic AI to reinforcement learning and decisioning. Um, and so there is this thing, um, that we call CCCG, the causal customer context graph. [00:36:25] Phil: Hmm. [00:36:26] Tobi: Um, and it's an agentic, you know, it's an agentic context graph, right? But imagine you had this thing listening to all your experiments that you've ever done, right? It could build, um, as a semantic layer, it could build previous effects. Causal effects of similar interventions. Super hard to do, right? But essentially, let's break it down to our problem maybe, and we call it the cold start problem, right? [00:36:53] You had to, I don't know if I can curse on this show, you had to invent all these different things. Can I curse on this [00:37:00] show? [00:37:00] Phil: Yeah. Yeah, [00:37:00] Tobi: You had to pull them out of your ass, right? That's the, that's the, um, but, but the, you know, now imagine that there is a system that says, look, you know, we don't know for sure, but we actually have previous causal statements encoded in this graph. [00:37:17] And don't send this, this, uh, uh, dark walker thing because it's pretty proximate to something that you ran an experiment on three years ago. And our prior is that this is gonna backfire, [00:37:29] right? So you can condense the space basically in something that is more prone to work. Now, you know, here is the problem. [00:37:36] If you do this based on correlational data alone, like basically you say to the agent, look at all the warehouse and data. And initialize, you know, give me 10, 15 starting ideas, right? This thing is gonna basically be as bad as the whole idea of, of correlational decisioning. You know, it's just basically gonna say, okay. [00:37:58] Um, [00:38:00] you, yeah. I mean, people that like walk their dog often spend more money on food. So we're gonna send them a doc, you know, it's a core. We are gonna send them a dog walker gift certificate, right? Still gonna backfire, but it's gonna base it off of correlational data. Um, you know, like a fun example if is, if you not the marketing, um, uh, core capacities of audience building, right? [00:38:23] If you put a gen on audience building and you just ask this thing, give me an audience that causally minimizes churn, it'll give you an audience of highly engaged people that always buy. Why? [00:38:34] Because correlation, they never churn, right? But causally it's, it's, it's nonsense. [00:38:39] It's probably the worst audience that you can pick. Right. So here's this idea of the CCCG, um, this causal customer context path, right? Um, if, if you embed and log the causal context that you have on every one of their customers in the right way, you can avoid some of these [00:39:00] things, right? And here's I think where it gets really interesting. Um, you could do this across companies that we work with, for example, that you work with as a, as a MarTech vendor, right? [00:39:13] The old data co-op model, for example, you say, look, um, you don't have to, but it's all Anonym anon, uh, anonymized data anyways, right? It's basically, um, telemetry data off of experimentation. That's really what we're talking about here, [00:39:26] right? You put into this data co-op, that's what they used to, it was a big business model in Silicon Valley 20 years ago, 15 years ago. [00:39:34] Um, and you benefit from essentially what everybody else has been testing on already. [00:39:39] To avoid that boomerang effect. Right. Um, and then I think the related pathway, and I'll, I'll turn it back over to you, but the related pathway is, and kind of touched on this too, is you get stuck in what we will call local optima or local maxima in these systems as well. [00:39:55] And you need a way to get out of these. [00:39:58] 6. Escaping Local Maxima and The Failure of Randomly Initialized Decisioning --- [00:39:58] Phil: Yeah, lo local [00:40:00] maximum is, or local maxima is a really interesting, um, problem, I guess. Like, or, or like one of the downsides of reinforcement learning too, like when I was like looking through how you're writing about the boomerang effect, like, it, it totally makes sense. Like we have this uninformed learning period as the system is basically starting blind and we have random initialization. [00:40:22] Um, and I immediately thought of like the episode that we did with, uh, the chief growth officer at Wealthsimple last year, Simone Gen. He like called out local maxima as like one of the most important things that marketers need to understand. And I could see it cleanly in that boomerang effect because like, we have this in this random initialization period, like the system is basically blind and it's like, it, it could latch onto the first like, okay, result, like this looks promising. [00:40:50] Let's just filter traffic dynamically towards that. Then in the long run, maybe that would've been like the local Maxima versus, you know, if we ran [00:41:00] that a bit longer, we could have found something that was a bit more global, like more LTV. Is that where you were kinda learning towards there? Like can you maybe explain like how that random initialization period could potentially latch onto the first tiny positive signals and, and what that could mean? [00:41:16] Tobi: I mean, any initialization can latch onto the most positive signal, right? Like that's how these things are defined, and I think you'll have it right, is there is a set of fixed interventions. It ultimately allocates the traffic to the intervention that maximizes returns, which is good, but by definition, that is a local optimum, right? [00:41:36] Or a local maximum. So the question for these systems is, what is your North Star? If, if it, in our example, it latches on to me, what, what was a good example that you, I, I like the, I like the dog walk. The, the, the dog food gift certificate, maybe. Is that one of the things that you brought? [00:41:56] Phil: Yeah. Yeah. I was thinking too of like grooming, like pet grooming. [00:41:59] Tobi: oh, I [00:42:00] love it. [00:42:00] Let's use [00:42:00] that one. Yeah. I love it. Love it. You know, dogs always, I mean, my in-laws have a dog. This thing always looks like, you know, 10 days of rain. You know, you get a, this really increases LTV, you look at your dog, you're much happier. You know, you, you, you're willing to causally increases the lever of, um, of LTV. [00:42:18] 'cause now you buy food there all the time, right? Um, but maybe there is an intervention out there that is even better. You sent a doc on a cruise or something like that, I don't know. Um, but how is that system gonna know that it should, you know, first of all, how can it dynamically generate new candidates? [00:42:36] I think this is a big open question. Um, and then also what is good enough, [00:42:42] Phil: Hmm. [00:42:42] Tobi: When it's an unknowable question, you can't know by definition. The only way to know that something is a global maximum is you've explored all possible different options, is not feasible, [00:42:54] right? Um, so it's, it's, again, I think it's another example of trade offs, which [00:43:00] marketing is full of. [00:43:01] And that's why it's such a fascinating field, right? There's, we've had many, I mean, there's trade off between learning and scaling or exploration exploitation, right? Um, there is trade off between optimizing on local versus always looking for global. Um, there is. There is the fundamental problem of causal inference, which ultimately is a trade off, right? [00:43:18] So it's this world is full of trade-offs and that is that that and all the data is there, which makes it so fascinating from an intellectual perspective. [00:43:28] Phil: Yeah, it's, uh, it, it's tricky. Like the, the downside too is like, you know, finding that global maximum might take forever and like, that's another trade off. There is the time. And so, yeah, it's a, yeah, it's, it's a really interesting space for sure. Um, you mentioned AG agentic a couple times there, so I, I do wanna let you get into that. [00:43:47] Like, [00:43:47] 7. Why Agentic AI Trained on Data Warehouse Correlations Reinforces Bias --- [00:43:47] Phil: there's this growing trend. Uh, I've seen a lot of companies like, let's let agenda AI loose on our data warehouse. We spent all this time like building this source of truth. Finally, like we've removed silos. Let's just like slap agent AI on, on a couple of, uh, uh, on our data warehouse. Um, you've actually called this a terrible idea today and, and also in the series that you wrote on LinkedIn. [00:44:10] Um, since a warehouse kind of only reflects the company's current status quo, which, uh, in, in some cases may have nothing to do with the causal relationship that you kinda chat out about already, like how does. Having AI train this way, risk accelerating potential negative dynamics, like aggressively promoting a product that correlates with high LTV but doesn't actually cause it like the dog walking example one there. [00:44:35] Maybe chat about like that agentic trap, if you will. [00:44:38] Tobi: Yeah, I mean, I, you know, I, I think what you outline is unfortunately a trend and I think it's lazy thinking. [00:44:44] Um, my hope and I, my belief, oh. Look, I think we've all sold into marketing and have been frustrated at times with the position of the marketer. [00:44:54] Um, but what I try to do here is to give marketers a lot of credit because it's, it's, [00:45:00] it's a very comp, it's, it's a very complicated business. [00:45:03] My hope is that the market will see [00:45:05] this as lazy thinking, and it's risky too. And that was your question, right? So imagine that, um, basically your data warehouse is full of correlations that causally backfire, right? So again, this example of, um, all the high LTV dog food customers go on a walk often. [00:45:28] Right. Um, but sending the dog walker with their dogs, which maybe, you know, from a survey or something like that, um, but sending the, uh, the, the gift certificate of a dog walker actually causally backfires because now I think, you know, you're way interfering in my private life, right? Um, so this is what these systems would recommend, and there is a, there is a good paper out there out of Duke that makes this point in a, in a different, uh, um, in a slightly different way. [00:45:56] Um, but it cannot distinguish if [00:46:00] you don't log it right. It cannot distinguish what is causal and what is correlational. Now, if you put the agentic machinery on top of it, right? We're not talking about this one example, getting into production and, you know, causally hurting your top line, right? We're talking about millions of examples like that, and that's, I think the danger with agentic. [00:46:21] Agentic can accelerate the things that are good. But it will also accelerate the things that are bad to be really plainspoken. And it doesn't distinguish between the two [00:46:31] if you don't. Right. This is why, you know, a more cleanly delineate delineated context graph that delineates between these are causal effects. [00:46:41] This is correlational. Um, you know, uh, is I think the answer here, and that's complicated to build. But if you don't do something like that, the risk I think is disastrous. And I think there's other, or the possible outcome is disastrous. And I think there is other real issues here. Um, one issue for example [00:47:00] is the lack of auditability. [00:47:02] Yeah. I mean, okay, you go to a land graph and you have an auditability suite, right? I mean, does it answer any of your questions? No. Uh, the nice thing about these decisioning systems about reinforcement learning. Is that I can audit them and we will, you know, that's a bigger point that I have in general. [00:47:16] You know, no auditing is perfect, but I can say, Hey, at any given point, and you brought it up to, you know, Bayesian multi-arm bend with thumbs and sampling at any given point, my probability, distribution of rewards over arms of outcomes, over arms look like this. And I sampled this value and then I made this decision. [00:47:34] Right? You can audit it. Um, what a agentic AI does, particularly if you let it go wild on decisioning, you can't audit in this way. [00:47:42] And, you know, if anybody else claims anything else, it's a lie, right? So, you know, I do think there is this way for these two different trends to interact with each other, but I've described how I see this, the decisioning element is ultimately done by an explainable algorithm, such [00:48:00] as the reinforcement learning techniques that you mention. [00:48:02] Um, you know. the the generation of the initialization, the cold star, and the generation of treatments that can be done by agent ai if it is done on the correct context. 'cause otherwise you're back in this world that we described, which is, you know, you're stuck in local, uh, local, local Optima. You might be stuck in local mini for all I care about. [00:48:25] Right? Um, basically all your options backfire and you know you're screwed. Um, and we're back in that world. So, so that's, that's sort of my concern without this semantic layer that teaches the machine what is causal and what is not. It all make up what is and what is not, and you don't wanna be in that world. [00:48:44] 8. The Power of Composable Decisioning --- [00:48:44] Phil: So where does like composability fit into this whole like, solution here? Like you mentioned the context graph and how, you know, letting ag agentic loose on the warehouse could be really dangerous. Uh, but something else that you wrote was that like, [00:49:00] journeys are causal language, like a customer journey is causal language and AI has to be able to speak that language to be effective. [00:49:10] Maybe chat about like the concept of composable decisioning where like the model learns in real time as a customer is moving through that journey without data ever leaving the warehouse. Uh, yeah. Chat about that, that, that problem of like decoupled learning and execution. [00:49:26] Tobi: Yeah, it's, it's a good point. I think it's a. It's a slightly different point because this goes kind of deep into implementation. Um, and it goes to one thing that, that, you know, certainly I believe should be the, the standard for enterprise grade. And I think most enterprise companies do understand this as their standard, which is how much can we do without any of our data leaving the system? [00:49:48] Right? Most decisioning systems are built something like, you know, you take the relevant data out, um, you train the model, um, and then you update the model on some sort of [00:50:00] cadence that you predefine, [00:50:01] right? Um, when we build this thing, uh, we build it according to our, um, you know, I think philosophy on composability. [00:50:12] Um, and there is a way, you know, uh, first of all. I'm not a big fan of building products from scratch. That's my [00:50:20] bias in my career. Um, I've been there, done that, you know, I've raised close to $6 million, hired really smart people, went into a dark room and built a product. Um, and you know, I think if you're, particularly if you come from an academic background, that that is sort of the bias, [00:50:37] but it, but it doesn't work and it shouldn't work, right? [00:50:40] Because, because marketing is here and has workflows and jobs to be done. Um, and I think right now, this is the journey. I think marketers right now think about their work in terms of journeys and orchestration. Um, maybe audience building, but you know, orchestration. So there was two big questions. One, yeah. [00:50:59] As [00:51:00] opposed to building a new, oh, Toby invented, you know, AI decisioning, suite tm, you know, please use it. And, you know, um, I, I much rather. Retrofit existing product that has usage because it addresses pain points of the customers with new capabilities. So that's a philosophical 0.1. And I think the philosophical point too is, you know, yeah, can we do it all in such a way that we still don't have to take any data out? [00:51:26] And, and you mentioned the other point, if I have to take data out, train a model, you have this, um, this asymmetry, right? You there is the model learning step, which, which is a whole different pipeline. And then there is the decisioning at time of inference step, and they're disconnected, right? So we basically said, let's try to get to this. [00:51:46] Parallelism of training and traffic allocation all happens seamlessly in the same breath. Essentially, every person who's exposed to this decisioning system will lend intelligence to the system in real [00:52:00] time. Right. Um, there is parallelism now. Uh, and also let's, let's use products that we know are there and use, because they address, um, they address customer pain points or buyer pain points. [00:52:12] And that just, that's just my bias to building AI products because I've been on the other side of this. And, um, you know, if you ask my investors, the outcome for accurate was not, was not, um, a cursor or a, uh, you'll get it. [00:52:28] Phil: Yeah, I, I love the self-reflection there, like going, getting funding and, and going in a dark room and, and building what you think people are just gonna love. And then just like asking people to, to, to end up using versus the flip side and thinking about it as journeys. I think, um, that's totally the right way to to, to go about it. [00:52:47] Um, there's a couple more things I wanted to get into, Toby, but [00:52:49] 9. How Machine Decisioning Transcends Marketing --- [00:52:49] Phil: I do wanna make time, uh, for like the, the topic of machine decisioning and how it transcends marketing. Um, you, you wrote about that in, in, in some of your posts also, like looking at the bigger picture here, like you said, that machine decisioning is actually way bigger than just marketing. [00:53:07] We just talked about the marketing use case here, specifically lifecycle marketers. How do you think shifting from like the human gut feel, our intuition to auditable machine decisioning help reduce systemic biases in, in critical areas like, like healthcare. You mentioned, um, like hiring. [00:53:25] We also have like criminal justice. Um, yeah. Unpack that for us. [00:53:29] Tobi: Yeah, I mean, I, you know, it's well documented that, um, uh, that human decisioning is biased, um, in, in foundational ways. I mean, this is, you know, the whole. Um, or, or suboptimal, even if it's not biased, you know, just in impacted by noise. Um, this is this whole, this the whole theory of behavioral economics and Don Kahneman, right? [00:53:53] Um, who is a noble laureate. And I think the first eco, uh, the, the, the, the first sort of, um, noble [00:54:00] laureate and the behavioral economics, uh, uh, economics realm. Um, but it's this idea that humans actually don't behave as rational actors, [00:54:08] Phil: Hmm. [00:54:09] Tobi: They can be biased, but there can also be situational factors that influences you. [00:54:13] And there is many examples, hiring, firing, right? You, you, you add your, you add a really good breakfast in the morning, and your kids, you know, you had really nice interaction with your kids, and you meet this candidate who doesn't meet your standards, but you know, you, you just have, you, you own a good day, right? [00:54:30] This is not, this is, this is just how the human brain works. I mean, there, there is nothing about, there is nothing to be done about it. Right. And then there is bias decisioning as well. And I think their examples are bound too. I think if you look at, one of my favorite examples is, uh, traffic pullovers, right? [00:54:46] You bet they're biased and they're biased by race, and it's provable, right? Um, at least the downstream effect that they're biased. Um, sentencing another example you brought up, right? Um, medical treatment allocation [00:55:00] is biased, right? Um, uh, so you have these two big factors that make human decisioning worse. [00:55:07] And here's the biggest issue. You can't audit it, right? If you hired, I'm not gonna say you as in new Phil, but, but if you hired this guy because you had a really good breakfast, or you let this guy go because you had this awful fight with your wife and it, you know, it, I'm, I'm sorry. It just, it, this is reality today. [00:55:25] It happens. Right. Um, how, how can you go and say, oh, these decisions are correct or not, or even forget about that. How can you go and say, here are the basis that Le led to this decision and let's revisit them. You can't because it's encoded in your brain and your brain doesn't remember, and your brain is not, you know, meant to be a system of record for these things that is auditable. [00:55:49] Just doesn't work like [00:55:49] Phil: Mm-hmm. [00:55:50] Tobi: right? I mean, you know, my brain doesn't work very well. I've been sick for two weeks where my brain really didn't work, uh, well at all. But the human brains, you know, don't work very well. [00:56:00] The, I think the, the, the promise of machine decisioning is, in the very least, the decisioning process becomes auditable. [00:56:06] Um, whether it's marketing, right? You can go to these systems and say, what did you know at the point you made the decision? And why the frick did you make the decision with numbers? Right? Um, um, and you can do it for all these different use cases as well. You can do it for sentencing, you can do it for. Um, uh, you know, for, for traffic stops. [00:56:27] You know, I think there's, there's actually one interesting, uh, um, factor in here as well that, that some of these things have a really, really delayed feedback, uh, reward feedback system, right? Um, uh, like maybe essentially the, one of the benchmarks of how success or sentencing is, what is the, um, what is the, the rate of criminalization, [00:56:48] Phil: Hmm. [00:56:48] Tobi: right? [00:56:49] That takes, I mean, how long does it take to measurement? Depends on your timeframe. It can take years, right? So these systems, and by the way, if that's the case, and we think about this in marketing too, it's a different subject. I'm not sure if you wanna come [00:57:00] back to that, but, um, then you're actually out of the world of multi-arm band. [00:57:04] You're kind of in the world of these much more unstable systems, mark of decisioning processes, things like that, right? They're just less stable. It's much harder. It's a much harder problem, right? Um, but still, I think, I think that is the potential in in human decisioning, uh, in, sorry, in machine decisioning. [00:57:21] Um, and I, it also gives me, um, you know, it, it gives me some hope because it is a way to tie agentic AI into something that improves outcomes in a meaningful [00:57:35] way. [00:57:35] Phil: Hmm. [00:57:35] Tobi: And this is certainly one other, you know, there there's two, these two big streams that I'm really interested in marrying, right? Agentic AI and, and reinforcement learning. [00:57:44] Um, and, you know, I, maybe I'll, I'll say two more words on, on, um, on, on why and, and where I think Agentic AI is a standalone, sort of lacks in atory ambition, [00:58:00] which is where [00:58:00] we opened, um. You know, to me, and I made the point in, in a recent conference as well, if you look at the first internet era, so, you know, mid nineties in Silicon Valley, um, there was this feeling of, yo yeah, we're gonna make a lot of money on this, but we actually gonna better the world. [00:58:18] If you talk to anybody who was there, you know, there was this, this, this civiliz atory ambition to make humankind better. And with Agen ai, you know, I haven't seen it at all. It's, it's, [00:58:29] it's like a monetization play. You [00:58:30] know, it's a bandwagon that, you know, you want to jump on or, um, or write something nonsensical, uh, on LinkedIn. [00:58:38] You know, I was, I remember I was in panel discussion with the VC who said, oh, yo, we figured out agenda AI besides observability, and this guy has no clue what he's talking about. It sounds good. It's sound bites. you [00:58:48] know, you just wanna be on the bw. But you don't think about, um, implications of this technology. [00:58:54] Right? And there's obviously all these controversies that I'm, that I'm not, that, that I'm not gonna get into because I'm tired of them, [00:59:00] like the Claude Bot controversy that we've been following, right? It's funny, but it, but it, I think it shows there is a void of intellectual voices. [00:59:08] Okay. Tying this to auditable decisioning in the way that we described here, I think gives agen AI a purpose, um, that actually can lead to civiliz atory improvement. [00:59:22] Right? Um, and the last thing that I'll say on this is, you know, I am a skeptic. I mean, I, I think, you know, if you wanna take my blanket statement off, if you take Agentic AI as a, as a monolith, um, and you make that statement that it, it has no civiliz story potential or no potential to improve civilization, obviously that's too broad. [00:59:41] Right? And I do, you know, what I miss is the, is intellectual voices, academic voices that accompany that. Um, there was a fantastic paper out there called The Agentic Economy out of Microsoft Research. That to me, you know, where, where it takes agentic AI as a monolith and it describes how agentic [01:00:00] economy can reduce income inequality and, you know, really lead to a civiliz story improvement. [01:00:05] Um, I don't agree with the assumptions of that paper necessarily because, you know, I think, um, you know, it, it is sort of hinging on this idea of equals, which, you know, I think that's to be, to be, I'm skeptical on that. [01:00:17] Um, but at least it is, it is an intellectual, um, contribution that otherwise I've been missing to me though, to kind of close the loop, tying those two things together and saying, you know, we have a auditable pathway to make sure that forget marketing for a second, that sentencing is more fair. [01:00:35] Right. Let's do marketing that, that I can do better experiences for my customers, that also increase my top line, right? Win-win. Right? Um, uh, if tying that to AgTech AI to help with the, the, the issues that these systems have, the shortcomings these systems have, I think would give it, um, a place in, you know, [01:01:00] redefining what, uh, um, what, what can be done in terms of civiliz story improvement. [01:01:05] Phil: It's such a cool answer. I I, I could listen to you talk about this for, for a full hour, Toby We'll, we'll only get to the, the paper that you mentioned there. It sounds super cool. Um, but yeah, I, I love your answer to, to ground us back into like, yeah, you know, we, we do marketing for a living, but there's, there's applications that are way bigger than this. [01:01:23] And, uh, I appreciate that. Um, [01:01:25] 10. Why Clear Priority Hierarchies Improve Executive Decision Making --- [01:01:25] Phil: we got one last question for you. Uh, Toby, you're obviously VP of ai, you're an entrepreneur, uh, startup exec. You're a product innovator, but you're also a dad of two young girls. Uh, one question we ask everyone on the show is how do you decide what deserves your energy at any given moment, and what's your personal system for staying aligned with what actually makes you happy? [01:01:45] Tobi: Oh, Phil, that's an easy answer. Um, my girls always deserve my attention, so if there's any, uh, um, if there's any, uh, question there that always goes in favor of the girls. Um, I also, you know, I, I think you can learn a lot from kids, [01:02:00] right? Even on the things that we're thinking about, sort of from a philosophical angle, um, human biases, human noise that leads to a suboptimal decision making. [01:02:09] Like obviously we're lost causes, right? We are older, you know, we, but you see it sort of developing in, in kids and, and it's, it's interesting. Um, but then obviously from a, from a personal perspective, I mean, my girls and my family is, is, uh, is uh, is the center of my life. And I, you know, I, you know, I think about marketing and, and causal questions and, you know, civiliz obligatory improvements for sure. [01:02:35] But at the end of the day, um, we do marketing for internet and I think that's okay. That's fine. Um, uh, you know, um, it's, I think it's not net negative and I think that's, that's probably a good, uh, compass to, to, to judge that what you wanna work on. And not, by the way, that's one of my compass, you know, I'm a pragmatist. [01:02:54] I, I, um, you know, do no harm is a good place to start, right? But [01:03:00] at the end of the day, it's also marketing for the internet. So our, our tolerance for error is, is fairly high. And, you know, we, um, you know, we we're, we're, we're, we're, we're not solving, uh, the, the fam the, the most recent feminine in Central Africa. [01:03:12] So, [01:03:13] so, you know, it's an easy answer. It goes to my girls. [01:03:17] Phil: I love it. Yeah. Great. Grounding answer there. Uh, I got a, a young, young boy and, uh, and a young girl, so, uh, totally relate to that. Um, To, has been super fun. I really appreciate your time and, and, and prep leading up to this. Uh, yeah. Tons of insights for, for folks in there. And, uh, yeah, well, like we said, we, we prepped, uh, using notebook here and, uh, we will share some of the, the graphs that, that came out of this. [01:03:41] But yeah, really appreciate your time, Toby, this, this is super fun. [01:03:44] Tobi: Uh, Phil, agreed. Uh, thank you also for your, for your prep and, and taking the correlation versus causation. Uh, seriously. Um, I am still thinking about how to communicate this, this, uh, in a way that is, that is digestible. Um, maybe this was a step towards it, maybe not, but I very much appreciate the [01:04:00] conversation. [01:04:00] Phil: Yeah, hopefully. Hopefully a nice step towards that for sure.