[00:00:00] [00:00:00] Phil: you've said that you're a vocal critic of data democratization. [00:00:03] Joshua: There's this sort of implicit assumption when you talk about data democratization, that the data itself has good integrity, that it's clear. And if only people had access to it, then we'd all make better decisions and we'd see the world the same way. [00:00:16] But people have different levels of sophistication, with respect to data, interpreting data. [00:00:21] people bring their own kind of hypotheses and they're just looking for the data to confirm what they already believe, [00:00:27] Data can be extremely powerful, but you have to have a certain level of skill and training and, and contextual knowledge to interpret the data in a responsible way. [00:00:38] [00:00:42] In This Episode --- [00:00:42] Phil: What's up everyone? Today we have the pleasure of sitting down with Joshua Cantor, co-founder and chief Data and analytics officer at Convert ml. [00:00:50] Joshua is former principal at McKinsey, and he's a four time CMO Caesar's Entertainment, then PetSmart, then International Cruise, acquired by R and [00:01:00] later on Cora. And this episode we explore why data democratization is breaking more than it's fixing, how confirmation bias corrupts marketing decisions at scale. [00:01:10] Why you're thinking about statistical significance completely wrong, why B2B marketing tests should be loud, and how to future proof your MarTech career without burning out all that and a bunch more stuff afterwards. Super quick word from two of awesome partners. [00:01:26] [00:02:32] [00:03:25] Phil: thanks so much for your time today. Really pumped the chat. [00:03:28] Joshua: I'm excited to be here. Thanks for having me. [00:03:31] Phil: There's a bunch of different angles we could have gone down for the episode today when we were, uh, doing the pre-interview questions for our chat today. We landed on one that I think has a lot of different jumping off points, and [00:03:43] Data Democratization Is Breaking More Than It’s Fixing --- [00:03:49] Phil: this idea that like data democratization might be overrated was really interesting to me. Um, some folks call it like making data accessible to anyone in your company. It's, it's, this idea is really flashy and it feels like something most companies try to achieve. I actually spent like four [00:04:00] years at a bi startup early in my career, and it was part of our mission statement, like, democratizing data for everyone in your company. [00:04:07] Give everyone access to like dashboards and data, but there's a dark side to giving non-data literate or non-technical users access to data. Just because more people can see the data doesn't mean they know how to interpret it. So yeah, you've said that you're a vocal critic of data democratization. Where does this belief come from, and what problems do you think it's causing inside of orgs today? [00:04:29] Joshua: I should start out by saying that I'm not really some, like evil, like anti-democratic. Like I, I do actually believe in the ideal of data democratization. I think it's a great idea. I, I also believe after 25 years of, of doing this stuff, and my first job, by the way, was outta college was I was a SQL jock, you know, writing, you know, database, you know, queries 16 hours a day, so, so. [00:04:59] After [00:05:00] a long, a long career, working closely with or in data, I've just found that, you know, the pitfalls are so numerous and so subtle that people just, like, they fall into them all the time and they, figuratively speaking, they hurt themselves or the organizations that they're working on, you know, are working on behalf of because they just don't understand the data itself, the raw data, the underlying data. [00:05:27] So the idea, there's this sort of implicit assumption when you talk about data democratization, that the data itself is, has good integrity, uh, that it's clear. And if only people could like, engage with it, if they only had access to it, then we'd all make better decisions and we'd see the world the same way. [00:05:47] And I'm, I'm in for Kumbaya, but I also know from experience, that's not how it plays out. I mean, for. You can give people a cross tab [00:06:00] of, of, you know, the same data, right? Just like pick two dimensions, put a fact in the, you know, in the, in the cross tab. Give people the, the, you know, the spreadsheet and ask them to tell you what it means. [00:06:12] And if you ask five people, you're gonna get five different answers about that. Now, that's, that's mostly a function of, and that's, that's not even like figuring out what the queries are there. The query's already been like executed, the data's already been sourced, it's already been presented. But people have different levels of sophistication, uh, with respect to data, interpreting data. [00:06:32] They bring their own hypotheses very often. What I have observed, what I've seen is that, and this comes from a dozen plus years of being a consultant, people bring their own kind of hypotheses and they're just looking for the data to confirm what they already believe, [00:06:47] Phil: Yeah. [00:06:48] Joshua: right? So, so it's not really a, an endeavor of discovery or interrogating the data or really truly understanding what the data says. [00:06:57] And by the way, nobody wants to, [00:07:00] not nobody, too few people want to acknowledge that data has limitations. There's some data that we simply don't have. There's, there's, you can, you can interpret the data and extrapolate, or I interpolate within certain boundary conditions, but then beyond that, you're getting a little bit out over your skis. [00:07:19] Um, and so, you know, I, I guess what I'm saying is, um, data can be extremely powerful, but you have to have a certain level of skill and training and, and contextual knowledge to interpret the data in a responsible way. And ultimately that's what I want. And, and yes, I want everyone to do that. I do. So, back to this democracy, I, I would love everyone to do that, but I also know that it's just not the way that it played out. [00:07:52] You know, even people who have data related roles all too often ignore, you know, sort of the gaps in the data [00:08:00] or they, um, you know, they only look for confirmation of what they believe, but they don't, they don't look for, you know, information that might, you know, support a different hypothesis, uh, or they just, you know, misinterpret stuff, which can happen. [00:08:18] So anyway, that's, that's a, I mean, I think you, you nailed it. And even in your articulation of the question, uh, it's, um, it's a great ideal. Um, but in practice, it, it's very rarely lives up to the hype. [00:08:30] Phil: Yeah. It's a, it's a bit utopian for sure. I feel like your take is less cynical and more realist, and it's based in experience, and [00:08:38] How Confirmation Bias Corrupts Marketing Decisions at Scale --- [00:08:48] Phil: I totally relate with this idea of like the confirmation bias when it comes to diving into data and having. data to reinforce what you already believe. Instead of analyzing data to find something new, discover new insights, and bring that to other people. We often go searching for evidence to support assumptions that we already have to back [00:09:00] up like our existing thinking and, and like you said, that could be really dangerous. How do you see this playing out day-to-day in marketing orgs? Like what role should marketing ops or just like data operations teams play in protecting the org from this happening? [00:09:16] Like is it a gatekeeping thing? Is it education, is it guardrails? Kind of all of the above. What are [00:09:22] Joshua: I mean, look, my opinion is that sort of the. You know, for organizations that have some scale, what I'm about to say will not apply to companies that have like, you know, there's like one person who like works in the data and you know, so not for like super small kind of organizations, but organizations that have real infrastructure. [00:09:41] You should have, you know what some companies call the data steward, right? So people who understand the data and are responsible for it. And when, when a business leader has a question, they should send the question to the person who understands the data to then translate the question in English, into sql, let's [00:10:00] just say, right? [00:10:00] Or, or whatever all app tool they're using, what, whatever kind of BI tool they're using, they'll, they'll understand the question and the business intent of the question, find the best way to kind of dimensionalize it with existing data and then come back with both a data exhibit, right? That. Tries to, you know, uh, approximates a, a direct answer to the question, but then also some interpretation that says, look, this is what I see. [00:10:27] Be aware that, you know, the reason it doesn't go beyond this is that we didn't capture whatever. And, you know, so here's what I think the answer to your question is supported by these data with the following caveats. Right? And you'll get an answer like that. Now, not all business people like that answer. [00:10:42] Some people just want, you know, give me the number, tell me yes. Or whatever. And, and that's a maturity, you know, it's a, it's a, it's a, some people don't have a tolerance for, or have a low tolerance for, uh, uncertainty. Um, and, you know, and they want the database to give them every [00:11:00] answer. You know, I, I do think there's like a continuum of people's attitudes, their or their, their, their relationship with data. [00:11:07] There are some people who have, who believe very strongly in their own ins, in their own instincts. I. And the data that they want to validate their instincts with the data. And that leads to this confirmation, you know, sort of phenomenon of our time. On the other extreme, there's people who, like, are big believers or like, they're almost evangelical about like, we're gonna test everything and we're gonna do experiments, and we're gonna gather all the data, and the data will tell us, right? [00:11:34] And, and there's that other extreme where it's almost like there's no intuitive value, right? We're gonna let the data guide us, and then they, they'll just, they won't be able to make it a decision until they have not just every, like, cell in the experiment populated with, you know, n equals 200 or whatever. [00:11:53] By the way, central limit theorem says you only need 30 N in any given cell. But, [00:12:00] but the point is, they all just gathered here. I I, I took one of my years ago, I, I, uh, I worked at a $7 billion retailer as the CMO and I came in and, um, you know. The head of merchandising who left right around the same time I came. [00:12:15] I didn't really work with the guy, but he used to commission research and he would do like 9,000 respondents in his research. And the reason that he would do that was he wanted to be able to do cross tabs of any given cell, anything that he could dream up later and have statistically significant results. [00:12:35] Because, and this is again, the guy on the, on the other end of the spectrum, he just didn't, his relationship with data was not one where he could like make an assertion that was kind of supported by the data, right? So I think you see this kind of, this continuum. You've got some people who like, like give this like superficial nod to the data by like double checking that what they were already gonna [00:13:00] do is somewhat supported by the data. [00:13:03] You got other people who like can't make a decision, they're like paralyzed until the data tells them what to do. [00:13:09] Phil: Yeah. [00:13:11] Joshua: I don't think either of those is, is the right, you know, place. I mean, I think where you want to end up is you recognize the data is a tool, right? It has certain kinds of uses. And by the way, analytics and analytical methodologies are also tools that you apply to the data. [00:13:27] And so you have a toolbox of things and you, you work with imperfect data. You work with somewhat dirty data. You make the best assumptions you can based on your experience and the data, right? And you, and you try to make good decisions using the data. But all of those tools, knowing how to use them and the limitations of your own data, those all require a level of sophistication on the part of the user. [00:13:54] And so that ultimately what I'm saying is, you know, data is super [00:14:00] important strategically, it makes the difference between, um, a. You know, sort of mediocre results and extraordinary results. When I was working in the casino industry before the retailer, uh, I got, there was a quote, it was like the Wall Street Journal, like wrote, like, they, they took a quote that I had given in a speech, and they were, it was like a bold assertion I made. [00:14:22] What I basically said was that the ways that the sophistication and use of, um, of customer data was more valuable than a gaming license. And you have to understand in the gaming industry, like the casino entertainment industry, gaming licenses are like, you know, they could cost you a half a billion dollars or a billion dollars. [00:14:45] So for me to go out there and say that, you know, the, the skillset around enterprise mastery of data was more valuable. Like, I was making quite a statement and I didn't intend to make the waves that I need. With [00:15:00] that, with that statement and the speech that I gave, but it got quoted and I. I got a couple phone calls after that. [00:15:06] Um, but I, but I still, I stand by it. I stood by it then and I stand by it now. Um, you know, the, the skillset and tool set, uh, and sophistication that organizations have around making strategic and intelligent use of their data is the, is the difference between kind of just chugging along and, you know, and leading your industry? [00:15:30] Phil: Yeah. Such a great answer. [00:15:31] You’re Thinking About Statistical Significance Completely Wrong --- [00:15:33] Phil: It's, it's always a tough balance, like between smart rigor and like that unnecessarily delay that comes with a lot of experimentation. Getting that big Stat sig number, I think you actually said this in, in our pre-interview chat, like, uh, people not really appreciating the significance of statistical significance, [00:15:51] Joshua: That's true. [00:15:52] Phil: it. [00:15:53] Joshua: People, I mean, right. It, uh. Yeah. I hate being punny on a, on a digital format like this, [00:16:00] but no, I mean, it's true. I don't think people really understand what statistical significance is. For example, if something is significant at the 0.05 or 95% confidence, like level, what that means is one time outta 20, you'll be wrong, right? [00:16:16] I mean, it's not like a perfect thing. What it means is data like this will show up erroneously one time out of 20. Are you comfortable with that? And at the 80% level, it'll show up one time out of five. Well, you'll, you'll get the wrong answer. There's a guy, there's a, a statistician called Bon Ferone, Italian statistician from, God, I don't know when, 16 hundreds or hundreds. [00:16:41] By the way, somebody's gonna fact check us and I'm gonna be off by several hundred years. But anyway, Bon Ferone, uh, introduced this, this notion that, um. You, you can't do your, your analysis. You can't, sorry. You can't do your experiment and then go looking for your [00:17:00] hypotheses after you've done your analysis. You have to have your hypotheses up front because if you are looking for patterns in the day and then you go cherry pick, you are going to draw conclusions that aren't truly as strong as you believe. And so he is got this adjustment, uh, that I'm not gonna bore your listeners with. It basically says you have to actually cut your, your statistical significance, um, based on, you know, having your hypotheses a priori. [00:17:29] But back to what I was saying about the significance of significance, there's three ways to be statistically significant, right? The way that everybody thinks about is effect size. Meaning you're comparing two things, and one of them, the, the metric you care about is like way up here. And the other one is way down here. [00:17:47] And that's statistically different. That's the one that everybody thinks about. That's effect size, right? But with a very small effect size, you can also be statistically significant if you have enough sample size. And so, uh, for [00:18:00] example, they discovered in the fifties or sixties or something like that, that aspirin reduces the, the rate of, you know, heart attacks, right? [00:18:09] You, you had to take like half an aspirin or low low dose of aspirin. The effect size there, it's really small. But when you're dealing with life and death, really small counts for a lot, right? So if you do a study and you have 10,000 people and you're preparing, you know, some of them got the aspirin, some of them didn't, and then there's a natural heart attack rate, and then you've cut that by. [00:18:33] 4%, or, I mean, I don't know what the effect, it wasn't big. And in fact, with a small sample, you wouldn't be able to know. The variance would, would be, it would be too noisy to be able to even measure a 4% in, in impact or whatever the, the effect size was. But with a large sample size, which is what they had, they were able to confirm with statistical significance. [00:18:53] You know what, this actually makes a difference. And when you're dealing with life and death, that's really important. And that's one of the things that in now is [00:19:00] part of, you know, medical science has accepted this and people understand it and it's, it's a really important life and death tool. So that's the second way, right? [00:19:08] So the first way is effect size. That's where it thinks about sample size. You can take a, a small effect size and you can get to statistically significant results if you have a large enough sample. 'cause that reduces your standard deviation, which means your variance is, and so then you can, you can measure, you can discern two things being different, even if they're very similar averages, uh, because your sample size is so in your, your standard C deviation system. [00:19:32] The third way is sort of cheating, but for completeness, I'll, I'll mention it, which is that you can, you can reduce your threshold for what you call significant, right? So a lot of people use alpha as 0.05, meaning that, you know, it has to be, you know, one in 20 chance of it being wrong or less. But you could also have alpha as 0.10 or 0.20 or 0.25, you could you move the goalpost again, it's sort of [00:20:00] cheating, but, but there are times when, you know, it should be 0.01, [00:20:03] Phil: Mm-hmm. [00:20:04] Joshua: right? [00:20:04] I mean, the, where you really need a high bar, uh, medical science would be a good example of that. Um, but, and then there are times when it really doesn't matter that much, right? You're testing copy in a marketing communication and you know, I. You want indicatively, you want to have some idea about what the right answer is. [00:20:22] You can just look at the, the averages and that'll give you some idea. But if you have different sample sizes, you want to compare these things and you want to know, Hey, what's the chance this one is in fact better than that one? Well, you can calculate it and you can decide this is significant, or this is not significant, but you have a, a yardstick that you've defined. [00:20:40] Standardly people use 0.05, but that's not universally the, the only option to use. So anyway, yes. Statistical significance is something that, that's probably way more than your listeners care about, but I use it to, to illustrate that people are banding about terms and phrases that they [00:21:00] don't deeply understand. [00:21:02] They believe that significance is important. They, some of them believe that if it doesn't achieve the statistical significance threshold, that the finding is not meaningful, which isn't necessarily true, right. So, so anyway, I'm just saying that there's, there's sophistication here and the important thing is to pair good business judgment with a solid understanding of the data. [00:21:27] And if those two things don't happen to reside in the same human brain, that's okay. That's all right. You know, and so my general philosophy is you want to have, you know, business people who can ask, you know, important questions, and then people who understand the data, who can provide meaningful answers. [00:21:50] Why B2B Marketing Tests Should Be Loud --- [00:21:50] Phil: Do you think it's fair to say that instead of testing minor changes where that detectable effect might be very small teams, [00:22:00] maybe especially in B2B, should be trying bold marketing experiments to create. Unmistakable detectable effects. And how do we operationalize that? Like do we choose three consistent metrics that align with business objectives and we track them across longer horizon terms? [00:22:18] Like [00:22:18] Joshua: Yeah. [00:22:18] Phil: thoughts there? [00:22:19] Joshua: Yeah. I mean, look, I spent maybe of my, my career gets longer every day. I don't know what I know, what I'm doing wrong. I'm getting older apparently, but I've been doing this since the nineties and, uh, so it's a long time. And, um, first 20 years I worked almost exclusively in B2B, sorry, B2C. Um, but the last five years I've worked in B2B, and there are some structural differences that really show up in the data. [00:22:45] Um, you know, when you're, when you, like I ran, uh, in the casino industry, I ran a loyalty program that had, uh, 30 million members something. And then when I was in [00:23:00] retail, um, you know, the, uh, the loyalty program there was like 50 million. I mean, it was just like, so the. And then in, in B2B, it's like you're, you're in a certain sector and you're talking to, we, we were talking to CTOs or CIOs and so, you know, how many of those are there? [00:23:23] 10,000, you know, for companies above a billion dollars of revenue or what? I mean, like, it's just a fundamentally different scale. And so you can do a lot more robust analysis and experimentation and those sorts of things. In B2C, you can run a test with a hundred thousand people in condition A and a hundred thousand people in condition B. [00:23:44] Right? And you can get your answer really fast. Um, but you can't afford to do that in B2B, uh, because A, you run outta sample, uh, and B you've only got one shot for certain like companies. And so, you know, [00:24:00] so the, the ways that you learn have to be different in B2B versus B2C. And I, I mean, I could go into more depth, but at a high level you can't. [00:24:12] There's a lot more sophistication on the B2C side, which is where I come from, but I've also spent the last five years working in B2B. And, um, I think that back to what you said, you know, you can only measure, uh, and when I say measure, I'm talking about like legit, like data in the database kind of. You can only measure, uh, and quantify the, the difference if you're, if you have a strong effect size, meaning strategy A works better than strategy B, which which is better than strategy, you can only quantify that if you have fundamentally different, um, and, and differently effective strategies. [00:24:54] Um, and you can't get into a lot of the nuance in B2C. You have an experiment. You might have like [00:25:00] five different factors and each factor might have three levels. And what fact, I mean. Let me explain what those terms mean. Like if you have a marketing communication, there's the, the sub, let's say it's an email, there's the subject line, that's one factor. [00:25:15] You might have three different versions. Those are three different levels, right? So you've got one factor with three levels. Then you've got, like, let's say you've got an offer and it's 10% offer, 30% off or 50% off. Those are like very different. Usually that's, that's a broader range than you would typically put in one experiment, but you got another factor, three different levels. [00:25:33] Then you've got the, are you gonna have embedded video or images or text? That's another factor. Then there's like, what's the, the, the, uh, major positioning or the message that, you know, and one is about, you know, we're your trusted provider and the other one is this is the cheapest thing you've ever seen. [00:25:52] And the other is, you know, don't risk it with people who you know. I don't know what else. Right? You got like the, the kind of the, the [00:26:00] thrust of the, you know, the, the pitch you're making. So you can have these various factors and you can test different levels. And so then if you, if you were to combine all of those together, you've got, let's say five factors, each of them with three levels. [00:26:15] I'm not gonna do the math now, but that's, you know, three to the fifth, right? It's a big number of possible. So the full factorial, if you were to do this full experiment is gonna be, you know, several thousand cells, whatever the number is. You can't do that in B2B. You can do that in B2C, although most people don't do it that way. [00:26:39] That's, that's a, doing that would be a full factorial, uh, experiment. And you don't need to, you can be a lot more efficient through, um, methods that are like a fractional factorial design. I don't think your audience cares to hear about experimental design, but there are ways to be much more efficient, uh, in your experimentation where instead of doing all, let's say [00:27:00] 2000 combinations, you just do, you know, 50 combinations. [00:27:04] But then because of where they're chosen in the end dimensional space, you can interpolate, uh, to simulate any one of the 2000 things. And you can build a simulation tool that allows you to say like, okay, well if I, for this segment, if I had it like this, what would I expect the results to look like? Now we're getting into, this is fancy stuff you can do with math, but by the way, don't try putting this in the hands of your CMO and ask them to like, just come up with it on their own or their product owner. [00:27:30] Like this is sophisticated stuff that requires experts who understand the data, the methodologies, the tools, et cetera. [00:27:38] [00:28:43] [00:29:38] [00:29:38] Why CMOs Who Speak Statistics Are the Ones Redefining the Role --- [00:29:38] Phil: It, it's really cool to hear you talk about some of those technical data aspect things, because the traditional assumption is that the average CMO isn't technically savvy or data savvy. Like we talk to a lot of attribution measurement vendors and they sell into CMOs and CFOs, and it's like two very different [00:30:00] conversations. You're a three time CMO and you're talking about experimentation design. I, I don't think that qualifies as like your average CMO background there. Maybe chat about like how that has like shaped your experience and your background. Now, going back to the entrepreneurial world, you're a co-founder at, at Convert ml, like obviously that helped shape that, that decision. [00:30:23] Maybe chat about that a bit. [00:30:25] Joshua: Sure. Yeah. Look, I mean, I, I think it's true that not every CMO like, uh, has an affinity for the data side of things. If I, if I, I would say I. If you oversimplify marketing as a discipline, uh, there's sort of like brand marketing on the one end, and then, you know, data-driven marketing or performance marketing on the other end, right? [00:30:54] And, uh, and you can exist anywhere on the continuum. And I think by, by nature and by training, I'm sort of [00:31:00] personally more maybe two thirds on the, on the end of the spectrum towards the data and performance market. And, but I've spent 15 years or so try to, you know, really develop skill and mastery on the brand side as well. [00:31:13] And I, I think I'm fairly balanced, but, but there are some marketers who are just like, they're a brand person. And for some businesses that's really the thing that they need most. Um, I do happen to believe though, for the kind of businesses that I, I work in, I think that the role, the strategic role of a marketing leader. [00:31:34] Requires fluency in data and in technology. Um, you know, we don't, it's, it's not mad men anymore, you know, and I'm, I'm not trying to disparage brand marketers, I'm just saying like, uh, there was a time when that was the totality of marketing and now, you know, in the age of, of, you know, digital marketing, um, and, you know, user [00:32:00] generated content and, you know, social media, um, and connected devices, and I mean, the things that you can do as a marketer have exponentially exploded. [00:32:12] And I think you have to keep your finger on the pulse. You don't have to be a master of everything, but you've gotta understand modern technology and you've gotta have some level of understanding of data because those new technology touchpoints. Not only give you new ways to engage with your end customer or your prospective customers, which is very exciting for anybody who tries to drive the top line of a business. Um, but it also creates data, right? Which gives you insight. So not only does it give you reach and access, but it also gives you insight. And so it's really, it's really the combination of those things that's so important. You mentioned Convert Melon. Yeah. I'm really excited, uh, to be a co-founder of ConvertML [00:32:52] It's a business that was born, um, about a year and a half ago. My, my co-founder and I, uh, [00:33:00] got together. We met and we were talking about how do you, uh, put, give people access. This is, now, it may sound like it's in conflict of what we've just been talking about, but how do you give people access to advanced statistical methods in a way that doesn't. [00:33:20] Make them vulnerable to misinterpreting the data. How do you, how do you kind of give them safe, reliable access to advanced statistical methods? And that what we found is that, um, you, we can what, what we are one can and we are a convert ml, um, making use of generative ai. Generative ai, uh, was, uh, you know, before it existed, in order to get access to these advanced statistical methods, you needed to hire a data scientist or a PhD or somebody who really deeply understood that stuff. [00:33:56] I don't have a PhD. I do, I do understand more than most [00:34:00] marketers need to. I understand data and advance, you know, data science methods, but you had to have somebody with that level of depth to really use the data and get that kind of insight outta the data. We could talk about the difference between like reporting and analytics. [00:34:16] Um, because most people just do cross tabs, that's just reporting. Whereas they don't use any statistics because they haven't understood, you know, when they apply and how to interpret them, and you know which ones to use where. And so the idea was, oh, so, so then pre pandemic, pre generative ai, you need, uh, a statistics expert to then apply those methods to your data. [00:34:43] Then generative AI happens, and generative AI is fascinating because it has the ability to emulate the sort of human interpretive function that gets applied. I mean, there's like art and science in a lot of this stuff. When you're interpreting the data, [00:35:00] you know, there's some cognitive intuitive leaps you have to make to see patterns and then call 'em out and say, well, what does this actually mean? [00:35:07] And humans do that. They do it with varying degrees of, you know, reliability and, and insight, but they do it. And generative ai, if it's prompted correctly or, or intelligently, can also do a reliably good job there. So it's really the combination. What we've done with Convert ML is we've said, look, we understand these advanced methods that require expertise, and what we can do is we can program into, into a generative AI engine knowledge about which methods to use and which circumstances when the data has these characteristics to use this method, when it has this characteristics, use this method. [00:35:47] And then it not only does the analysis, but then serves up a facts set and an interpretation of what it means. So it kind of closes the gap and it makes it, [00:36:00] it gives you a, a reliable, extremely insightful, um. View into your data without having to often hire a $300,000 a year data scientist and without having to go back for your PhD in statistics. [00:36:14] Um, which some people will get, will do, but it's not the path that everybody takes. And so that's the, that's really what Convert Amel does is it, it, it's, it, it harnesses the power of advanced and local methods and generative ai and through automation, combining these two things you can put in the hands of, you know, average to advanced, you know, business users, you can put in the hands like a reliably powerful statistical analysis engine that'll tell them, Hey, this matters. [00:36:42] This doesn't matter so much. Right. This is what you should go do next. [00:36:47] Phil: Very cool. So you're promoting data democratization, but in a safe, responsible way [00:36:53] Joshua: That's, [00:36:53] Phil: with [00:36:53] Joshua: really that last bit. Right, exactly. Because, uh, I, as I said earlier, I do believe in putting it in the hands of [00:37:00] everybody. I just happen to believe that there are a lot of ways you can go wrong. I. Um, and statistics really helps. But the problem is there's this big barrier where like, people don't want to go study statistics. [00:37:12] They've got a business to run, right? And so giving them access to a tool that helps them to get the answers they need in a timely manner without all of the, you know, hiring and management of data wonks, myself included, right? I say that affectionately, but you don't wanna have to go do that for most businesses. [00:37:31] And so that's the, that's really where Convert ML is designed, you know, to, to play. And, you know, whether it's survey data, like market research type stuff and, and segmentation, um, uh, or, or you know, the, or, or even even asking questions like in English and getting answers, right? You can use generative AI now to like ask a question in spoken English, and you get an answer that's rooted [00:38:00] in your data set. [00:38:00] Well. That's powerful, right? We have that in converter mail. We're not the only ones. People have stuff like that, but it's really, uh, it's really powerful. So the idea is to give people, um, safe, reliable, insightful, access to their own data. And so I mentioned that that's surveys or direct access to like full database, two different offerings, but same kind of tool set that you put on top. [00:38:28] How to Write Better Prompts for Data Analysis with GenAI --- [00:38:28] Phil: So you mentioned Gene AI a couple times there and. Um, like some of this I'm sure gets into like NLP two when you're able to convert some of those questions into SQL prompts and, and interact with your data. [00:38:41] Joshua: Yeah. [00:38:42] Phil: about like a good prompt and, and on like what is a good prompt, like in a world where gen AI is helping you interpret complex customer data, whether it's customer surveys or sitting on top of the warehouse, like what do you think makes a good prompt [00:38:59] Joshua: Um. [00:39:00] Good question. Um, first of all, um, you know what SQL stands for? [00:39:08] Phil: a structured query language? [00:39:10] Joshua: Yes, that's right. Not everyone knows that. A lot of people think the S is standard. Uh, and the reason I ask the question is English. The English language doesn't have that very important structure that, I mean, like if you try, if someone says, you know, how's my business doing? Like, that's a question that somebody might ask, but it gives you none of the structure that you would need if you were to try to turn that into sql. [00:39:35] Good luck. Right? Um, and so I think that, um, look inside of Convert ml, what we've done is we have certain offerings that are well structured, um, analytical processes, right? There's 12 steps to the process. Step one is this. Step two is this. Step three is this, and a lot of those are just data operations that if you know what to do. [00:39:59] It's [00:40:00] actually pretty straightforward. But then you get to step, let's say seven or eight, and at step seven or eight, you actually have to look at the data and interpret the data, and you have to like name a segment or describe a segment or what, you know, like you have to actually interpret it and say, what is it saying? [00:40:17] Um, and this is where historically like humans would go. But, but again, if this is something that generative AI does fairly well, if the prompt is well constructed. So what we've been doing at Convert ML is we've been combining kind of well-established, reliable, reliably goods statistical operations with these prompts that, that take care of the human interpretation part, right? [00:40:42] So there's automation of advanced statistics plus well con, well constructed, well crafted, uh, prompts, uh, or well engineered prompts that get you from, you know, the start of the process to the end Now. That's part of the secret sauce and what's under the hood at Converter Mill. [00:41:00] But, uh, people don't need to use converter mill. [00:41:02] I mean, you can, you can write your own prompts. The problem is the lack of structure in the English language. I mean, that, that's what, where we start, I started my answer to your question. 'cause when people are just kind of writing their own questions, it's up to them to, to provide enough context and to, and enough information for generative AI to have any idea what to do. [00:41:25] Um, and so, you know, I think about generative AI as, as a, uh, an extremely eager, um, but totally inexperienced new hire, fresh outta college, right? They'll work all night long and they'll generate anything you ask, but they have no idea if it's right. They have no experience to build on. Right. So it's up to you as the person asking the question or assigning the task. [00:41:57] It's up to you to give them the [00:42:00] guardrails and to give them the guidance about, Hey, this is the form I'm looking for. This is how to, to treat, you know, null values, or this is how to, you know, so, so unfortunately, maybe it's fortunately for some, but unfortunately for most average users, crafting a good data related prompt, um, requires you to do the thinking. [00:42:24] You have to, you have to give, like, you have to have some real rigor around it. And that's, it's more than I think we can ask of every user out there. People are gonna ask unsophisticated questions, not, not like thumbing their nose at the world, they just don't have the experience or the, the technical skill to do it. [00:42:45] And that's fine. But again, that's part of the reason why there's a gap in the world. There's a, there's a converter Mel shaped. You know, sort of void in the world. And that's what we're filling, right, by giving people kind of prebuilt prompts that they can [00:43:00] access in intuitive ways. And then all of the correct operations with well-engineered prompts will, that they can support and give them meaningful output. [00:43:09] Um, but, you know, regular users can do it too. They just have to know what they're doing and they have to know how to use generative ai. Not everyone ticks both of those boxes. [00:43:20] Phil: Yeah, definitely. And it's funny, like [00:43:23] Redesigning Marketing Teams for a GenAI World --- [00:43:31] Phil: some folks on the show when we talk about gen ai, say that it's going to be like the rise of the generalists. And some folks can encounter that and say that, you know, especially in this data context, that are specialists, like really niche specialists in one area will have more future-proofing potential because of gen ai. And I don't know, like it makes me think of like this idea of like instead of democratizing access to data. Do you think we need to reevaluate the role of like data literate specialists in marketing al roles or like marketing operations teams? Like what does [00:44:00] this ideal team structure look like? Is there a central data team or a decentralized embedded analyst who is a specialist on each marketing team? [00:44:09] What are your thoughts there? [00:44:10] Joshua: Yeah, no, those are, those are good questions. I think. Um. You know, if I, if I pull back a little bit and think about how organizations, how enterprises, or, you know, mid-size organizations really extract value from generative ai. Um, I don't think it's pixie dust. Uh, in other words, I, I don't think that you just keep your organization exactly the way it is and you give chat GBT to like your marketers or your data people or whoever. [00:44:44] I don't think that you, you keep everything the same and then give chat GBT to individuals and magically expect your business to transform. I don't think that's how it works at all. Um, in order to, to, you know, to harness [00:45:00] and extract the value, you know, the promised value of generative ai, uh, I think you need to take a look at the ways that you've been operating. [00:45:10] What are the core processes in your business? Right. You're, I would, I would hazard to assert that most businesses, their data architecture is the way it is because it's what served the needs of the business users, the da, the data consumers, and, and then your internal processes. And even your organizational design is all a function of, oh, it takes this long to do these tasks and in parallel, somebody else has to go and do these other tasks. [00:45:44] And then you have these kind of general work. There's a map of like processes that your whole business uses. And the org has built up around that. And the, the infrastructure and the data architecture have all been built up around that template of how things get. [00:46:00] And the reality is, and this is scary for some and exciting for others, um, generative AI can now fundamentally disrupt those core processes. [00:46:10] Not like throwing them out the window, but allowing you to take something that required five people in three weeks down to 90 minutes with one person in a laptop or whatever. And if that's the case, then you have to rethink your org structure. You have to rethink your data structure. You have to rethink, um, you know, your overall infrastructure. [00:46:34] You have to rethink a lot of things and not just rethinking what you already do. You have to introduce some things you didn't use to have. The ethical use of AI is a new discipline. Uh, you have to protect your organization from hallucinations. You have to make sure that you're, uh, protecting your proprietary LLMs from prompt injection and [00:47:00] other, like, there's new information security disciplines that have to be brought there. [00:47:03] So, so back to your question, like, you know. How is generative AI gonna change? Like the roles, is it specialists and generalists? I don't know, and it probably varies by company, but what I'd say is if an organization is serious about capturing the value at an enterprise level, not in an individual level, not saying, oh, my, my, my data analysts can do 30% more work. [00:47:28] Now you can do that. Just give 'em CHATT PT. But if you wanna transform your organization, you gotta take a look at your end-to-end processes and rethink how am I gonna get that core process done? And do I have the right team structure? Do I have, first of all, do I have the right process? You define the process first and how much of that can and should be done through generative ai. [00:47:49] What do the guardrails need to put around it? What's the data you need to use to support it? Then after you have those things answered, then you start to look at what is the team structure and what are the roles, and you probably don't [00:48:00] have as many boxes in that future state as you have today. Probably, and your business can be fundamentally faster, but you've got different roles. [00:48:10] Some of them are probably more specialized. Some of them, by the grace of God and chat, GBT are gonna be more generalized. Right? And some of them don't exist today that are gonna be necessary to protect the organization from the unknowns arising from this new operating. [00:48:27] Phil: Super cool answer. I love it. And [00:48:30] Why Generative AI Is Forcing a Career Reckoning in Marketing Ops --- [00:48:33] Phil: something I chat about a lot with listeners, like I ask a lot of new listeners like, what is the one thing that keeps you up at night right now? Or like, what's one thing you really wanna learn more about? And there's this like excitement, but also fear. With Jen AI right now in MarTech and marketing ops roles, a lot of folks are excited 'cause they're experimenting and playing. But to your point, like a lot of businesses are seeing in a couple of years this [00:49:00] where they can cut 50% of the workforce [00:49:04] Joshua: And that's terrifying. [00:49:05] Phil: tools. [00:49:05] Joshua: That's terrifying. I mean, it, it's terrifying. You know, maybe there's some, you know, uh, evil capitalist with a top hat and the money bags and he'd like laughing all the way to the bank. Maybe, maybe that person's excited. It is terrifying to think about, um, you know, the world I just described. [00:49:25] Needing fewer humans. Right. Um, I, I acknowledge that. I, I feel that deeply. Um, having said that, there's, it's not like gyne, right? Uh, it continues to be this like inexperienced, you know, aero prone, naive capability. And maybe it'll get better. It probably will, but you still need thoughtful humans to guide and interpret and validate what is happening in the system. [00:49:55] You're still gonna need that. It is exciting. This [00:50:00] is a transformative journey, and people who take the journey and the transformation seriously are gonna be imminently employable. Not everyone is gonna go along that journey, and I don't know how things play out, you know, for people who are not participating in this journey. [00:50:19] Uh, I, I. I wanna believe it's all gonna be just fine. And, and so I'm gonna just for today, make that assumption. But I, I, I think, um, I think, look, it is, it is scary. It is exciting. Um, I don't think it's all dystopia, and I don't think it's all utopia. I think it's gonna be a mixed bag just like everything else in human history to date. [00:50:42] Um, but I do think that if you want to participate in the excitement, you're gonna have to get scared and get involved and learn, get smart, right? So I think, I think that's, [00:51:00] it's, it's a choice. Some people are not into it and they're not gonna do it. They're gonna let somebody else figure it out, and that's fine. [00:51:06] But I also think that the first movers, um, are gonna, it's higher risk and higher reward. Um, and that's kind of where, you know, I. Convert is we're, we're, we believe we're moving quickly in, in a, in a space that is evolving and, and has a lot more evolution to, to go. Um, but, and then there will be other companies that kind of sit back and wait and watch maybe, maybe large companies that gobble up smaller companies that take the risks and the provi. [00:51:38] And that's, that's the way things work. And that's as it should be. [00:51:42] Phil: The cycle of consolidation and new startups. [00:51:46] Joshua: It, it totally happens. And, and, uh, yeah. But, but I, I do, I do kind of, I don't know how it's gonna play out for, for the business users. Right. Um, [00:52:00] I suppose if we had a time machine of the back a hundred years, right? Or 125, right. The industrial revolution, right. There were people whose job was to like, you know, hammer this, you know. [00:52:14] Dowel into this wood thing where somebody else had drilled the hole and then they hammer the thing in and you know, those jobs don't exist anymore. Right. And the jobs we have today, they couldn't have possibly imagined. Right? So, so I don't know what it's gonna look like in, I would say 125 years, but let's be honest, in, in 10 years, I don't know what it's gonna look like. [00:52:36] Um, I do think that there will be a place for people who are serious about participating. [00:52:44] Phil: On, on this point about like being serious about participating. Uh, [00:52:49] How to Future Proof Your Martech Career Without Burning Out --- [00:52:54] Phil: give us like a practical example that's like, what would do differently if you were starting? I. Maybe not restarting, but like, I dunno, five, 10 years into your data career in today's [00:53:00] MarTech gen AI landscape. What's one thing you encourage folks listening to be doing in terms of like taking this seriously? [00:53:07] I. [00:53:10] Joshua: Um, yeah. I think, let's see. When I graduated from college, uh, in the nineties, I had no interest in the business world. I had none. I could do math in my head. I got hired as a consultant and I ended up writing SQL code all day because I could do math, but I wasn't interested in the global economy. I didn't know the first thing about business. [00:53:40] And so I, I started out, I was working in financial services. I started by reading the American banker every day, which is pretty dry. And if you're not working in financial services, don't do it. Can't recommend it. But, but I just started out by giving myself a little bit of a diet of something wasn't interesting to me. [00:53:58] It certainly wasn't interesting the way [00:54:00] that Saturday morning cartoons were interesting to me when I was a teenager. Um, or, or other things. Right? Give yourself a little bit of a diet or, or subscribe to a newsletter or a podcast or whatever. Listen to people who are like doing this for a living. I mean, it's actually way easier today, um, to, to get a little bit of a taste here and there. [00:54:23] Find something or two or three things that you're kind of interested and then give yourself a month or two. Don't be an, don't set your target to be an expert, but like give yourself a personal assignment to learn about that thing. You don't know what a rag is. Figure out what a rag is, what the difference between a rag and an LLM. [00:54:44] Figure that out. You don't have to do that, but, but is set yourself. Start by just giving yourself a diet of something that maybe isn't that interesting to you, but like you feel like is somewhat relevant, you know, and then get a little bit of [00:55:00] exposure, figure out. Maybe there's some part of it you think is interesting. [00:55:02] And then go from there and then start to start to explore. And what you'll find, or what I found for myself is that the things that you initially start with are not the things that you're gonna end up. Like, you're gonna end up discovering something that you didn't know existed, and that's gonna be really interesting to you. [00:55:21] And then you can like, lean into that and maybe it'll shape the direction of your career if you're starting out, or frankly, even if you're not, you know, uh, even if you're, you've been doing this for a while, you may discover something. Um, you know, I'm no longer executive in multi-billion dollar enterprises now I'm, I'm a co-founder in a small, you know, data tech, MarTech, you know, AI company, right? [00:55:48] Um, and, and that's exciting, right? But not everybody's into that. But I think if people, if you're, if you are interested, don't try to, you know, uh, take it all in all at once. [00:56:00] Find a couple of things. Start doing some general research, subscribe to some things, listen to people, find people that you're interested in, the way that they talk, the way they present things. [00:56:09] This is a good one. Um, but, um, find people that you're interested and, and learn from them, and then branch out from there and give yourself a little bit of, you know, targets. Hey, by the end of the summer I want to know what this thing is. I wanna understand that generally. Um, and, and if you do that for a little bit, you'll, you'll find some things that you didn't know existed that are really, really fascinating and are part of the emerging economy. [00:56:35] Phil: That's such great advice, like the the rabbit hole discoveries of self-based learning. Anyone who's curious about something has gone down that path, whether it's like on YouTube or some blog somewhere and you discover [00:56:47] Joshua: I've got her, I've got, I've got two kids, right? One is a 4-year-old, and she's a Corona baby. And she's amazing. And I, and I love every second of it, but I've also got a son who is 20. He just finished his sophomore year in college. [00:57:00] And you know, it's, kids go off to college these days and they think they have to know their major and what their career is gonna be before they even get there. [00:57:08] And I've had to explain to my son, like, Hey, take it easy. College is actually not about, I mean, it is about learning, but it's actually more about learning about yourself, right? Try lots of things and discover what gets you excited, right? And then let that guide you in your next decision and then get your degree. [00:57:28] And when you're done, the people are gonna hire you. Don't care if you got a degree in, you know, Spanish literature or art history or MI microeconomics. They don't care. They care that you're an interesting human who, who you know is interested in the work and you know, they want to interact with you and they want, they, you've demonstrated a history of studying targets for yourself and achieving that, getting a college degree is that. [00:57:59] So [00:58:00] don't worry about being all over-focused. And the point that I'm trying to make to connect it back to the point we're just making is by learning about all these things. You, yes, you're learning those things, but you're really learning about what makes you tick. And that's something that all of us, none of us are too old to do that we all do that forever, you know? [00:58:18] Or, or you can stop anytime you like, but then you just sort of, you're on pause for the rest of your life. Who wants that, right? So, so continue to learn, continue to push, continue to ask questions, continue to find answers. And when you find those answers, it's about truth in the world, but it's also about truth inside of who you are. [00:58:37] Phil: Love it. I feel like this is the perfect segue to our last. Question. [00:58:40] How to Stay Sane and Motivated in a Career That Never Stops Moving --- [00:58:41] Phil: Joshua, you're a serial entrepreneur, three time, CMO plus multiple time as senior exec. You're also a husband. You just said a father of two. One question we ask everyone on the show is how do you remain happy and successful in your career? How do you find balance between all the stuff you're working on while staying happy? [00:58:59] Joshua: Uh, [00:59:00] that's a, that's a good question. You know, actually, there was an article just yesterday in the New York Times about happiness in American culture. And, um, I won't ruin it for people who should go out and read it. Uh, basically the authors say that that happiness in American culture has become very superficial, relative to the, where the concept, the concept of happiness is hundreds of years old. [00:59:22] It's not millennia old. It's, it's, um, it's younger than that. But they basically say that American culture has, has come to this sort of like fairly superficial, relatively speaking, more superficial concept of happiness today than it ever was. For me, it's more about fulfillment, um, and, um, do living, you know, like a moral life, but also doing things that, that I find to be meaningful. [00:59:51] Uh, and to me that's about being a, a husband and a father. Um, I love, um, cooking and baking with my 4-year-old. [01:00:00] Like, that's, there's like nothing better. Um, and my wife accuses me of trying to, you know, give her diabetes and, you know, uh, or, you know, flaring up her gluten and sensitivity. But I just, you know, it's just something, it's the Jewish grandmother inside of me, I think. [01:00:16] That just wants to feed people. That's a, that's an expression of love. Um, but I think, you know, spending time with, with loved ones, um, and doing something that's a shared experience. Um, my 4-year-old and I, this morning, um, after breakfast, but before school, we're doing a puzzle and man, you know, there's nothing better. [01:00:38] It's not about the puzzle, it's not about the food. It's not, it's about spending time with people you love and, um, helping them to grow. And while you're doing that, you know, doing a little bit of growing yourself. [01:00:51] Phil: Such a cool answer. It, it, it's funny, a lot of folks on the show are also founders or like they're starting something. And I can always [01:01:00] tell a little bit when someone has kids and when someone doesn't, and the people who don't have kids like have 17 different hobbies going on. And the people with kids are just like, are my kids? [01:01:11] Like I, [01:01:12] Joshua: Yeah. [01:01:13] Phil: this and that with my kids, like. [01:01:14] Joshua: know, I have to say it's a lot more socially acceptable. Uh, the dad jokes are a lot more socially acceptable when you actually have kids. I, I, I think that that humor has been part of my life since I was a kid. I get it from my dad, who gets it from his dad. Uh, but now that I have kids who are like, oh, it's just, uh, oh, yeah, of course he's a dad. [01:01:33] Um, yeah, my wife and I, we just went, we did pickleball the first time, uh, earlier, uh, I guess last week. Super fun by the way, and not, it's basically, I don't know if you've done it, it's like you take a racketball racket and a wiffle ball and then you go on a tennis court and then there's a game and it's, you know, but it's, it's, it's fun and it's not intimidating and you can do it if you're, you know, a teenager or you can do it if you're in your seventies. [01:01:57] So, um, it's sort [01:02:00] that's a, but you're, but you're right. I think, I think you can get lazy. I don't mean that in a judgmental way. Uh, you can, you can get very, it's easy to fill yourself up with just your family and your kids, um, but there's a whole world out there and, um, so, you know, travel and, and see it and experience it and, and pick up some things. [01:02:22] I'm, I'm doing, uh, uh, tonight I've got a class at a, uh, at a SU I'm in Arizona, so I'm going down to a SU I'm taking a class on, um, uh, on writing, uh, tonight my, my class. So, you know, trying to branch out a little bit. Um, but, uh, the core of it to your question is for me is family, uh, and, um, but I think there's a whole world of possibilities out there for people who are paying attention. [01:02:51] Phil: Love it. Love your answer, Joshua. Um, yeah, [01:02:54] About ConvertML --- [01:02:54] Phil: plug, uh, convert ML a little bit for, for folks that are curious to check it out. Convert ML AI right. [01:03:00] Joshua: That's right. Yeah. Convert ml.ai. Look, uh, we, we've been building, we're, we're a young company. Uh, I think I explained kind of earlier the, the, the thesis of the company, uh, is that, um, with the advent of generative ai, uh, we have the ability to, in mere mortals hands, like normal business people, we have the ability to put in their hands advanced statistical tools and methods, uh, that don't require them to be experts, but also enable them to get, uh, clear and meaningful, uh, insights into their data, uh, to run their business. [01:03:43] Uh, so that's, that's what we've been building. Uh, product set continues to expand. Uh, we launched, we, we launched externally about six months ago. Um, and, uh, yeah, we're, it's an exciting time. Uh, and uh, yeah, we hope, we hope people will be interested to [01:04:00] check us out. [01:04:01] Phil: Awesome. I appreciate your time today. This is super fun conversation. Uh, I know some folks probably got the, some laughs like, like I did, but yeah, really appreciate the, the data-driven scientific mind of A-A-C-M-O on the podcast here. Thank you so much for your time, Joshua. I. [01:04:17] Joshua: Thank you. Thanks for having me. [01:04:18] ​