Philippe Gamache  0:07  
What's up, guys, welcome to the humans of martech podcast. His name is John Taylor. My name is Phil Gamache. Our mission is to future proof the humans behind the tech so you can have a successful and happy career in marketing.

Philippe Gamache  0:26  
What's up? Arun today, we have the pleasure of sitting down with Pranav Bucha, co founder and CEO at paramark. Pranav started his career at a well known brand like PayPal and Dropbox, and he co founded pap lit, after that, the popular collaboration app to make school less boring. He's the former head of growth at Magento, as well as pilot.com Before becoming VP of Marketing at Bill. He's also a reforge instructor for a new marketing measurement course in March of last year, Brenau co founded paramark to help marketers measure in forecast the impact of their investments. Brunov Really excited to chat today. Thanks for your

Pranav Piyush  1:02  
time. Big Time list. Long time listener, first time caller. So thank you for having me. I'm excited to be here. Phil,

Philippe Gamache  1:10  
appreciate that this episode is brought to you by our friends at Knack launching an email or landing page in your marketing automation platform. Shouldn't feel like assembling an airplane mid flight with no instructions, but too often, that's exactly how it feels. Knack is like an instruction set for campaign creation, from establishing brand guard rails and streamlining your approval process to nack's no code drag and drop editor to help you build emails and landing pages. No more having to stop midway through your campaign to fix something simple. Knack lets you work with your entire team in real time, and stops you having to fix things mid flight. Check them out@knack.com that's K, N, A, K, and tell them we sent you. This episode was brought to you by our friends at customer IO, oversold on a legacy marketing automation platform that is still struggling to update its user interface. I've done a tour of duty with all the major marketing automation platforms, and many are definitely similar, but customer IO is the most intuitive and beautiful platform. I'm talking about the industry's top visual workflow builder to design and implement your unique messaging strategy, powerful AB testing features inside your workflows, not just on subject line sends, hold out testing functionality to see the incremental impact to your messages. Queue draft mode, so you can queue a messages and conditions in production with real users before anything is sent. Copy workflow items, so you don't have to repeat the building process again. And monitor campaigns, tests and key list membership growth from your personalized dashboard. The icing on the cake, marketers using customer IO have seen a 20% increase in conversion rates from strategic messaging. So stop using clunky, old tools and adopt a multi channel approach that creates joyful interactions with your customers. Start a free trial without a credit card@customer.io and tell them we sent you. We got a chance to meet ahead of this, and I've had the chance to dive in and read a lot of your content. I want us to start off with an interesting question around this idea of our analytical skills for marketers become overrated. So back in the day, marketing was thought of as like the arts and crafts department, right? It was a lot more art than science. But then there was this, like gold rush of data, and like data driven insights was kind of hot shit. And everyone has that on their LinkedIn. And I seen this, like shifting tide recently, though, or like a counter culture, where there was a recent tweet by Brian halogen, HubSpot co founder, went viral on Twitter. He basically said, marketers with analytical skills are overrated, but marketers with good tastes are underrated. Do you agree? What's your take there? It's

Pranav Piyush  3:52  
definitely a hot take. Short answer. I kind of agree with Brian, and I would totally sit in that camp, that I think having good taste is probably the biggest needle mover for a marketer now, right? So, like, put in a context, right? It's not a 01 it's not a it's a spectrum of skills, the spectrum of knowledge. And I would say that marketing is a creative function, you have to figure out how you're going to establish your presence in some humans mind that you have never actually met in the real world. That takes creativity, my friend, so Brian is 100% right about that. I will, however, say that the counterpoint to this is what people call data driven, what people call analytical skill set is absolute garbage, true understanding of the numbers and analytical skill set is very different from just being able to read a spreadsheet. Create formulas. Yeah, so Brian gets that because, guess what? He went to MIT and he's an engineer. Maybe, I don't actually know, maybe he's a salesperson, but he's good with numbers, and so he thinks that everybody is the reality is that 99% of marketers have no clue what terms like correlation mean, what terms like causation mean, what statistics means, what r square means, right? They haven't heard of this because, frankly, our education system has failed us to bring those basic concepts to life. So there's truth to the counter point to Brian as well. But I agree with him that if you have great taste and if you build great stuff, you are going to have a leg up over everybody else.

Philippe Gamache  5:46  
Yeah, interesting take. My perspective is that it's easy to say you're data driven. And we chatted with the head of growth at Wistia, and he actually shadowed over this idea of like, data driven and like, it's just this buzzword that people made up and like, data shouldn't drive everything that you do, and so it's kind of the spin on data driven and being more data informed, but actually being creatively driven. Like you need to be different, and you need to have something unique to say to, like, carve out some type of excitement in this space. So I think that resonates,

Pranav Piyush  6:23  
you know, one thing I'll add on that notion, I actually also think that that's a little bit of BS, right? Like data informed. So here's the thing, okay, you have a genius idea, right? It's amazingly creative. You launch it, it either works or it doesn't work. And if you can't figure out the difference between the two, then your creativity doesn't matter as much. So this whole like, we like to put things into clean boxes, right? Like, are you creative? Are you analytic? You gotta be both my friend and that is the hardest part about what you're doing with marketing or any other business function, same exact thing goes to product development. You build a feature and you launch it, and 10% of people of your customer base uses it. Is it a win? Is it not a win? Well, 10% that's a great number, but like, what does that mean in the broader context? Does it improve adoption? Did it improve revenue? Did it improve retention? Did you measure it through an AB test? Like, if you didn't do all of those things, then just saying that, hey, it's a great feature, and look all this amazing feedback. Well, if it didn't make you money, or if it didn't save you money, what did it do?

Philippe Gamache  7:37  
Yeah, let's talk about the measuring aspect there, because I think that gets lost in the noise. And a lot of folks think they have a good handle on ROI and reporting on stuff, but I know you're really big on this idea of incrementality, and I think it's poorly defined by a lot of different people, and it doesn't have really good mindshare with marketers like you mentioned, our education system has kind of failed us a lot with folks. So I'll give you my definition, and I'm curious, like, if you can kind of give me yours, and what your take here is, but my favorite way of defining incrementality is business results for marketing campaigns or channels that wouldn't have happened otherwise, and it's kind of a flip on how most people think of reporting and marketing results. Typically, a lot of folks, especially in my experience, is attribution, and that's typically influenced revenue or influenced pipeline that is attributed to marketing campaigns or channels. What's the difference between the two for you? Yeah,

Pranav Piyush  8:36  
it's a great question. It's a very semantic question. Yeah. So if you read the definition of the word attribution, you will see the word cause in there. So if we were to really think about the word attribution, I think it's a really amazing word, right? And it's about cause and effect. The reality is that the entire, you know, the last two decades of sort of the marketing technology industry and analytics tools and attribution tools have, frankly, sort of bastardized that term to mean something that it doesn't. So cause and effect is at the at the at the heart of the word attribution. That may not be how the world is using that word today. So that's one now, incrementality. I liked your definition. It's sales that would not have happened until and unless that marketing campaign or channel or strategy was put out in the world. Right? That is incrementality. One way that I talk to people is that if a prospect would have bought your product or service, regardless of the marketing that they see, then that marketing isn't incremental. Flip that, right? Vice versa. If a prospect would have never bought a product or service until, unless they saw a piece of marketing, that that marketing is very incremental. You can change that to sales. You can change that to partner. And that's the other interesting thing. For B to B, you have to look at it beyond just your marketing channels. You have to look at your scaled sales channels. You have to look at your partner channels, your affiliate channels, and so on and so forth. But I like your definition just simple and clean and clear cool.

Philippe Gamache  10:13  
So let's, let's touch on the MTA piece there, the multi touch creation. I know you have a bit of a bone to pick with that followed you on on LinkedIn for a bit now, and see you call out a lot of the, like you said, garbage around, kind of the definitions. And I don't know, like, I talked to a lot of folks, and it's like, the Holy Grail, like, attribution needs to be solved. We need to have multi touch attributions. We need to, like, get all the touch points in one place. And I think that, like still, though, a lot of folks do understand that it isn't a perfect science. Some of the issues with MTA that are well documented, and you've kind of mentioned in a lot of posts or channel biases, missing offline touches, the privacy impacts on like cookies, include IDs. But like you said, the real bone to pick with NTA is around this idea of causality, right?

Pranav Piyush  11:04  
That's 100% right? And it all you know. So when we were talking about the word attribution itself and the meaning of that word being cause and effect, and I do this in a very simple way, I ask people, somebody clicked on a link on Google, did that click cause the downstream conversion? And in some cases, maybe, in some cases, maybe not. So let's walk through an example, if I search for humans of martech and Google, and the first link that shows up is your organic SEO link, and I click through and then I subscribe to your podcast. Well, guess what? Your analytics tool, your MTA tool, will tell you your organic SEO efforts are great. What I will tell you as an outsider is what got me to put humans of martech in the search box and unless and until you recognize and acknowledge the fact that you don't know what got me to put that term in the in the in the search box, then we're not really talking about cause and effect. We're just talking about what I call behavioral analytics, which is not that different from your website analytics, your product analytics, right? Thing a happened, thing B happened, thing C happened, and then thing D happened. But you can't just by looking at that, say that thing a caused thing B, and thing B caused thing C, and thing C caused thing D. Now that takes some clarity of thought. And you know, unfortunately, because of all the pressures that marketing, ops and analytics teams feel, they don't even get the room to think about this. It's just like, well, we just need MTA, like, we need to show results next week. And you just go to the easy thing. And one of the things I will say is, like, the martech industry has made a lot of dollars by building the easy thing rather than by building the right thing? I think that needs to change, and it will change in the next five years. Yeah, I like

Philippe Gamache  13:08  
your perspective on like, when you're doing ROI reporting, it's not for a marketer to show to the CMO or their boss, like, what is the value of the campaigns that they did? Like, we should take a step back and think about it as when we're reporting to the board. And I think that, like, I haven't seen, like, I've had the pleasure of like, being part of some of those discussions and QBR presentations, and folks just gravitate to this idea, like, oh yeah, attribution, like whatever the team is telling me, like it must be true, like, they're using a tool to do MTA, if they're saying this campaign led to this many dollars, like, it has to be true. Like, the tooling has to be like, accurate there. But there is no deciphering of causality, especially in startups, and we're going back to like, incrementality here, but like, the causality element there is wouldn't have happened otherwise. And sometimes, like, marketing campaigns might correlate with a spike in signups, but it doesn't always mean, like you said, that the campaign caused those signups might have been something, you know, other factors that were at play that campaign, multiple campaigns going on at the same time. Obviously, like, the right answer to this, like you kind of teased out, is a well thought out AB tests, statistically significant experimentation design, right? But the reality, especially in startups, is, like, not everyone has the volume of traffic or the time to wait for that volume to have detectable effect. How do startups tackle this when, like, they're really early and, like yourself right now, like, probably not a ton of traffic compared to like, what you're gonna get next year. How do you tackle this?

Pranav Piyush  14:47  
It's a good question for an early stage business, right? If you're I would say like us, you know, four people, and by the time this launch. Is maybe a little bit more, and we're not even doing any paid media right now. Our channels are warm outbound events and a whole bunch of content and social. And when I think about I don't care about attribution now, I have a unique perspective, because I understand marketing. And if you understand marketing as a marketing as a founder, I think you'll be in the same boat. Attribution is not the question that you ought to be solving for right now. The real question is, how do you go from 100k in revenue to a million in revenue to 2 million in revenue, and you do whatever it takes, and you're doing all the marketing and sales yourself as a founder. So if you don't know that firsthand, no software is going to tell you, I'm sorry to disappoint people, right? So I would say, until you get to like, 234, 5 million in revenue, your intuition, your head of sales, is intuition, your head of marketing's intuition about what's working what's not working is probably better than trying to use and get to precise answers with software. It's a waste of time and effort. I should be able to count on my fingertips the channels. I should not have more than three channels, I would argue, really crack those three channels, and if you have fewer channels, your need for attribution, and ROI is like, non existent. Okay, so that's one. Now let's say I raise $5 million tomorrow, and I want to spend half a million of that on marketing over the next six months. What do I do now? I have real money, and I'm like, I'm not gonna, you know, double my outbound efforts through that. Probably want to do some scaled marketing, right? That's why marketing exists. Here's the simplest answer. You instead of running broad campaigns, you run a geo based test. And that's a very, very simple thing. You don't need paramark for this. You don't need any software for this, you go into, if you're doing non brand Google, you're doing LinkedIn, you're doing meta, whatever you're doing, you have options of targeting your spend by geography. You take your target account list, if I want to target 1000 accounts, and I look at where are their CMOs and their CFOs and their CROs located, create that list. It's very easy to do. Plug in clay here, Apollo here, a bunch of other online tools that help you do that. And this is a B to B example. And I find the geographical footprint right now I can see, okay, a bunch of my population is in SF and New York and whatever, whatever. And then I say, Okay, I'm going to run a test where I'm going to do all of my ads in Austin for a period of two months, because I see that 30% of my target list is in Austin or in Texas, and I'm not going to run the advertising in New York. And the same time, I'm going to see what is my organic visits from New York and what are my organic visits from Texas. I'm going to see what my lead volume is from New York and what is my lead volume from Texas. It's a very simple test. You don't need formal statistical training here. And even with a low volume, if you are investing the right way, if your message is resonating, if you're reaching the right people, you'll see a bump. If you don't see a bump, please stop spending that money. So it's not that complicated. And I would lean towards experimentation in the early days, because you don't have a whole bunch of historical data. Running these geo tests is so easy. So every marketer out there, please get into experimentation early. Yeah, that's what I would do.

Philippe Gamache  18:33  
Very cool. Yeah. I love the simplicity of geo tests. And I know you also mentioned in a couple other podcasts, time based tests, which is even easier than the geo tests, right? I'm curious if you have, like, other simple versions to throw in this bucket, and what your definition of the time based test versus the geo test is. But also, like, really curious to ask you, when is it okay to accept the potential limitations of these tests, for me, like the limitations of time based tests like you, you said you're running this campaign for a month, and then you turn it off for a month. This one doesn't work as well for email and some of the other channels, and it assumes that everything else is the same during those two months, except for that one campaign that you're turning off. But reality is, often there's the bunch of shit that will be different from month to month, especially in startups, right like seasonality, external factors, a bunch of other campaigns, web changes going on, and it's a bit better in geo tests. Like, instead of running your ad campaign to all states or your email campaign to all GEOS, you pick one segment and you do it in that one segment and not another one. You compare the conversion right there, while that is better than the time based test, like you still run into some potential geographic variability, like demographic differences, when's it okay to accept the potential limitations of these tests versus, like a more robust AB experimentation model?

Pranav Piyush  19:58  
Yeah, totally so. Couple of things to unpack here, right? One is, if you are doing experimentation on emails or in product or where you have a lot more control about how you're going to split the audience into control and test buckets, right? These are user based tests. You're saying, take 50% of the users and split them into control and test that actually gives you a lot of control, and the ambiguity and the limitations are probably kind of disappear, except that, do you have enough volume, right? If you're only talking about 10 users, what are you doing? Like you're not at the stage for doing testing, right? Like, just stop figure out how to go from 10 to 100 in different ways, right when you do experimentation on a surface that you don't control, and that is most of your marketing, you know, media, sort of channels, meta, Google, LinkedIn out of home, direct mail, podcasts out of home, you are sort of strong recommendation is geo testing. And yes, there are going to be differences between geographies, but you can actually pretty easily look at your traffic by state over the last six months and see the trend lines. And what you're going to notice is that you have a few geographies that follow the same exact trend line. And we think that there's a lot of variation in the real world, but look at the data, and if you look at the data, you'll be able to tell whether there is actually a lot of variation or not. Like, you know, somebody might hypothesize that, oh yeah. Like, there's a lot of seasonality in New York, but that seasonality doesn't happen in Texas as well. How do you know that? Like, let's go look at the data. And I, you know, I recommend this to you. Like, go to you, like, go to humans of martech and look at the last 12 months of traffic on your website. I'm pretty sure the seasonality trends are not there in the sense of, like, they're consistent across every geography, unless you have huge multi geo type of businesses, right where you got Europe and APAC and Africa and North America, then obviously you're not a small startup like you're probably much bigger, and there's a different answer. So that's the second point. The third point about time based testing. The concept is essentially saying, double down on something for a short period of time. So I'll give you an example. How would I do this? On LinkedIn, organic content, just my content, my feed. A great way of running this would be to say, Hey, I usually do one post a day and I skip Saturdays. I'm going to increase my frequency. I'm going to go to two posts a day for a month and see if that has any impact on my overall impression volume. Okay, that's great. The next month I'm going to say I'm going to go all video with the two posts a day, frequency, okay, let's see if that has any impact on my impression, volume. So you're more limited because you can only run one test at a time, right? Because it's time based test, you're comparing month zero to month one to month two. But it's still better than just saying, I'm just going to do stuff, and hopefully it'll work. So you have to have a big enough change for a time based test test to work. If you're testing on the fringes, like I'm just going to change the tone of my voice a little bit here and there, I don't think you're going to be able to detect the difference that you're hoping to see.

Philippe Gamache  23:21  
Yeah, I think that's super fair. This episode is brought to you by our friends at revenue hero, I can't think of anything worse than finding out a lead waited a week for a response from sales. That's why we recommend revenue hero. It's the easiest way to qualify leads based on Form Values or enriched data and route them to the right sales rep. Their product is packed with a bunch of behind the scenes superpowers that ensures qualified leads are assigned to the right reps, following your custom round robin rules and sending key data back to your CRM. That means more qualified meetings for your reps. We all know they want more of those, but more importantly, no more waiting time for your potential customers. They back all of this up with the best product support out there, offering 24, five support on Slack, connect for all customers No matter your pricing plan. So if you want to 3x your conversions with the same traffic, go to revenue hero.io and tell them we sent you. Your Sales Team will thank you for it. This episode is also brought to you by our friends at census, the number one data activation and reverse ETL platform loved by Activision, Canva, Sonos notion and more. As you might know, I'm pretty opinionated that the future of martech is composable, and that the single source of truth for your marketing data should be your data warehouse. Census helps marketers solve an age old marketing problem, getting real time complete access to your customer data without needing to write a line of code. Also, if you want your own face as a humans of martech style image, we're doing a fun monthly raffle with census for a personalized t shirt, enter to win at. At census.com/humans What are your thoughts on, like, this concept of, we kind of tease this out already a little bit like you said, you need a certain volume to care enough about even doing, like, some of these tests. But I asked this to a lot of recent guests, this idea of, like, does everything even need to be a test. And when I was@wordpress.com Like, we obsessed about everything that we could test. And if it wasn't, like, properly designed as an experiment, like, could we really roll it out with any type of confidence that this did have positive impacts on some of our metrics? Let's say, for example, like, when I was at WordPress, we really focused on on email and lifecycle was kind of like my my focus area in the first couple months. Let's say I'm, like, leveling up our email onboarding sequence from a four email nurture that's like three years old to this, like robust use gaze, segmented version that has, like, a full workflow for it. It's based on actually listening to customers, getting feedback on stuff we know the new one is better in theory. Do we really need to, like, roll this out as a test? Is that where, like, the time based tests might come in, like, we look at the baseline from the last x months and what the next couple of months are going to be like, What are your thoughts there?

Pranav Piyush  26:19  
It's a good question. I think there's a few things to unpack there, right? One is, this is stat. This is actually from booking.com and Amazon and Uber and Airbnb and all the meta analysis of all of the experimentation work that they have put out there. And what they usually find is that 70 to 80% of tests are going to come out to be basically failures. So first point, if you're doing a whole bunch of testing, seven to eight out of 10 tests are not going to be incrementally contributing to your business. So are you okay with that? And if you are, then that's a cultural positive, because you should then, then you have freedom to do more testing, because you're not always getting asked for like, well, if it didn't work, let's just stop it and then do something else, right? Like, that's not the idea. The idea is to constantly be learning. So generally, yes, when possible, do it as an experiment. But the counter to this is here, is it worth experimenting? And I'll give you an example. Email is a good one, if I know for a fact that emails are incrementally driving 3% of trial conversion. I'm just taking a random example from a SaaS business, right? Start a trial 30 days trial conversion, you've got a nurture email, and that if you stop the nurture email, your trial conversion goes down by about 3% okay, we have established that fact. So that's the first test that I would run, is, are those nurture emails even contributing anything? Then the question is, okay, what's the next goal? Are you trying to get that number to be even higher? Are you trying to increase trial conversion by X percentage? Is that the goal of whatever you're doing with nurture, or is it something else? Is it about making sure that there is high awareness of all the features on your paid product post trial? So it all actually starts with the hypothesis, what are you trying to achieve? Do you have a good way of measuring it, and what are you going to do if you are successful with this test? And that should dictate whether you run the test or not. And there's an opportunity cost, because you can't have unlimited number of tests running at any given time on any given surface. So you also have to ask a question of, well, is this in the top three? Is it in the top 10? And use that overall sort of culture and framework of experimentation to decide whether you should run a test or not. Very sort of tactical example, we relaunched our website three months ago or something, and we did not launch it as an AB test. That makes no sense for our website. Now, having said that, I did a whole bunch of qualitative research, we have great advisors. We have customers. I have a lot of marketing friends, and I spent time with them. I sent them figma box and said, tear it apart. Tell me what's working, what's not working. And we did that over three months to collect the feedback and then put out something. Guess what? Got unprompted feedback from folks like Rand Fishkin who just like, happened to chance upon our website. He's like, this is an amazing homepage. And I'm like, great. Okay, that's I don't need to run it as a statistically significant AB test. That's good enough, because I do trust Rand's taste. Yeah.

Philippe Gamache  29:42  
Does it need to be a test? No, if Fran says it's good, like a stamp of approval, ship it.

Unknown Speaker  29:47  
Ship it.

Philippe Gamache  29:49  
What do you think about I don't know if this is like a an overarching trend, and it's definitely not the majority of folks that I've had on the show, but there. There are these, like exceptions of companies that actually like, purposely don't invest in any type of ROI analysis or reporting, and they're like, allergic to any discussions about attribution. So Wistia and Ahrefs are two examples of that chatted with two growth leaders at both of those companies, and for now, they don't even use Google Analytics, like they do not track the impact of anything that they do other than revenue. Revenue is the only thing they look at. And they've both said, like, hey, if I'm doing something and the next month, revenue is going up in the right, like, I'm doing something right. Like, why do I need to obsess about GA and doing multi Dutch attribution to figure out, like, if was it this piece of content or that piece of content? So, like, I don't know this almost like a counterculture of just, like, why would I spend all this time, like, building out a data team and an analytics function to, like, make sure everything is like, driving revenue when, like, we're doing really good right now, we're focusing on quality content based on intuition, based on our users feedback. So like, why do we need to obsess about reporting? What are your thoughts? There

Pranav Piyush  31:12  
great question, and there's a lot of plg companies that fall in that same sort of bucket, right? So if you've got, and I spent time at Dropbox, like you could stop like you could probably get rid of the entire marketing and sales team, and it would still continue to make money for a long time, right? So that's the idea behind sort of what you're saying. So a couple of things. One is there's a certain scale at which that totally works, when you have really strong product and really strong in Product Marketing. And what I mean by that is, right, I'm a subscriber to Ahrefs. I get emails from them. I see their videos, I see their content. I see Tim everywhere on my feed, and yeah, they're putting out content that is resonating. And I would argue and tell me if I'm wrong, he is probably looking at the total impressions and reach that he is getting from his content. And he may not admit it, he may not be tracking that through Google Analytics, but he has some way of knowing that. Hey, it does look like my traffic on Ahrefs seems to be going up into the right and every time I post on LinkedIn, seems to get lots of engagement. Every time I put out a YouTube video, it gets 10s of 1000s of views. Guess what he's doing? He's doing what we're talking about through leading indicators, and through, I would argue, sort of good intuition looking at those leading indicators. Now, as refs is $100 million business, vistia is probably, you know, 70, $80 million business. I actually don't know the numbers, but I think that that's where they're going to and I think it can work when you have a really strong product and really strong founders who understand marketing. And I think astraps and vist are both in that cap. To go from 100 million to a billion, things are going to change, to go from self serve to also adding enterprise sales, things are going to change, right? And the final part, which I think is the linchpin here is the market and the competition. They are not immune to competition and the market. What does that mean? This is a famous example of Tesla never advertising, right? They did actually start advertising last year. Why did they start? Because the competition has changed, the biggest electronic vehicle manufacturer is now, I think, in China, and every major sort of regular auto company is entering the EV market, and everyone's advertising their EVs, so you don't have a choice but to advertise. So I think vistia Ahrefs are going to reach a similar point where they're going to have to put out more marketing dollars, because nobody's sitting in isolation and going, Oh yeah, let's just let, ah, reps take the whole market. Let's just let vistia take the whole market. The market doesn't work like that, right? You're going to have competition that because I'm that's going to mean that both of those companies are going to have to invest more in marketing in a differentiated way, and when you have 1520, different channels show up, suddenly, your intuition gets weaker and Tim doesn't scale, right? You see, what am I my point? Like when you have 100 marketers or 200 marketers in an organization that doesn't scale. You can't do everything off of intuition, as much as I would like, but yeah. Like, good luck. Like, look at Elon. I think they've even hired an agency, as far as I can tell. So yeah,

Philippe Gamache  34:55  
yeah. I saw a lot of backlash with the the the one ad they did. To for for one of their car commercials, like, folks are just like, this looks like, and then single car commercial,

Pranav Piyush  35:06  
that's right. And then they fired their entire team, right? So, yeah, started advertising. They launched a big campaign. They fired everybody on the team, and I think they're starting from scratch, but I'm pretty confident that he's going to have to figure out advertising if he wants to ward off the competition from the rest of the players, like, that's just that's life. Yeah,

Philippe Gamache  35:27  
we could do a full episode on on Elon's thoughts on on advertising, but I'd be remiss if I didn't ask you about hold that test. So this is very near and dear to my heart right now for some of the issues we're going through at I'm a full time startup, I'm curious to hear your thoughts on this. Like, this isn't actually common terminology for a lot of marketers. Like, I've chatted with some pretty seasoned marketers that are just like, hold out tests, like, what? What does that mean? And I think it, it's a really good word. How do you kind of describe it? Like at customer IO, sponsor of the show, and one of my favorite email tools, they have this baked into the product that's right self a lot of other tools, like, they don't really talk about holdout tests, and it's not really part of the product. Like@wordpress.com we had an internal ESP and we had a crazy work around for doing holdout tests, but really sophisticated data team, and the holdout test was like part of the like culture, almost like, if you're not doing a holdout test, when a new signup comes in and they're like, you're holding them out from from getting a lot of your other stuff, what are you even doing? But essentially, like this idea of removing two to 5% whatever the percentages, of your audience from ever receiving any of your campaigns, whether it's email or some of your other channels? What are your What are your thoughts on holdout tests?

Pranav Piyush  36:49  
Yeah, I think it's a great topic. A few different directions that we can go in. But first off, customer IO, huge fan of what they're doing, so shout out to the team. There a bunch of friends on that team, and I agree every marketing platform should have some sort of a holdout testing capability. So emails relatively easier to do, yeah, because you control the surface, right? And that's absolutely gold standard everybody should be doing hold out testing when it comes to in app or in email or known sort of user type of flows. So it's a no brainer to me, and customer IO makes it super easy, so go buy customer IO when it comes to so my thoughts are pretty simple, yes, yes, of course, you should do it. The challenge that happens is sometimes the fear is, well, those 5% of people are not going to get on emails. Yes, like, there's this, like, opportunity cost, and so yes, if you want to know the true impact of your emails, you have to be willing to live with that cost. Yeah, I will also argue you don't have to run that forever. You could run a holdout for three months or six months one year, understand the incremental contribution, go back to 100% and then do that again two years or three years from now. So there's a little bit of a it's not one size fit all sort of approach to hold out, you have to think about what is the question that you're really trying to solve for again, right? Like, why are you doing it? And once you know the answer of or once you've proven or disproven the hypothesis that you set out with a holdout test, then you could end the test, move on to life as usual, and then set up a new holdout test with a different hypothesis. Now I also want to give flowers where they're due. Meta is the only ad platform that has a native holdout testing called Conversion lift. And if you're advertising on meta, and you've not used conversion lift. Like, what are you doing? Like, go do that. And then Google has something good for YouTube and for display as well. So those are the two big platforms that have invested in incrementality testing, holdout testing, whatever you want to call it, it's like, true, randomly controlled trial sort of, type of experiences on those platforms. So one part of me wishes that in five or 10 years, every ad platform, every email platform, every martech platform, has holdout testing capabilities. That is how marketing should be run, very

Philippe Gamache  39:37  
cool that would be a fun a fun feature to live in, for sure, because they're like you said, it's easier for email, but like, if it's not baked into the product, there are still a ton of like, manual workarounds to do it. Like new users come in every single day, randomly selecting 2% of those users, putting them in a manual list and making them. Part of your like, global opt out group, so anything you're doing after that, they're not part of it. So there's a system for it in place, but it's like, Why? Why isn't it part of a lot more tools? I want to ask you about marketing mix modeling. We chatted MTA and incrementality, but we didn't touch on marketing mix modeling, yet, is it fair to say that, like all three of these reporting mechanisms are useful, but they differ in their focus areas? Would you say that, like incrementality centers on isolating the true impact of specific activities? MTA focuses on the contribution of individual touch points, and mmm looks at the broader impact of the entire marketing mix. Can you give us examples where like each method is best suited for?

Pranav Piyush  40:50  
Yeah, this is a hard one for me, because just my strong opinions on MTA, I will give it a shot.

Philippe Gamache  40:56  
Or MTA focus on mm no,

Pranav Piyush  41:00  
no. I think if you take away the word attribution from MPa, then I think about the underlying concept of behavioral analytics right as being incredibly important, because you are optimizing the bottom of the funnel conversion journey, the things that are captured in a digital sort of frame. And so that's how I think about tracking touches and clicks. You're trying to optimize the bottom of the funnel conversion journey with multi touch tracking. I don't call it attribution. MMMs and incrementality testing are essentially correlation and causation. That should be the mental model. MMMs give you a correlational analysis on your historical data set and tell you that, hey, this channel, this campaign, this strategy, is most correlated with your business outcomes, and these particular channels are least correlated with your business outcomes. That's incredibly helpful to get you to a starting point, but it's not enough, because correlation is not causation. So if you want to get to causality, then you have to run experiments. And incrementality, testing is one way of doing experiments. Hold out. Testing is another way of doing experiments. AB, testing is another way of doing experiments. And experimentation becomes critical if you want to get to causality. Now, MMMs and incrementality are telling you two different things. One is looking at the past and trying to predict the future. One is telling you, hey, let's run a real world example, sorry, real world test and see what that does. And when you do them together, actually very symbiotic. You use mmm to inform the types of tests you want to run, and you use the results of testing to feed back into mmm as what are known as priors. It's a fancy word of saying, What is my prior belief about a certain channel as a result of my incrementality test? So very symbiotic. And I would argue every sophisticated company should be investing in both. It's not a either R it's really you got to have both tools in your tool

Philippe Gamache  43:11  
set. Would you say that mmm is a bit better suited for later stage companies like is incrementality, something you should focus on in the first couple of, like, months or whatever, after launch, and mmm comes later.

Pranav Piyush  43:25  
Yes, I think that's totally fair. If you're earlier stage. Incrementality testing is a fantastic jumping off point, because you don't need a whole bunch of history. MMMs do require you to have a whole bunch of history. So if you don't have that history, there's nothing for the model to correlate with. So yeah, incrementality testing first, if you're earlier stage, if you're spending 20, 30 million, I would flip that and say, do mmm first, because you probably have lots of data already, so you don't want to wait for testing results to come in, right? You can see some insights right off the bat. Use that to inform your experimentation roadmap. And so it's Yeah, not a cookie cutter answer, but yeah,

Philippe Gamache  44:09  
yeah, it makes sense. Hopefully early companies aren't spending 20, 30 million on on meta too early.

Pranav Piyush  44:16  
Never know. You never know.

Philippe Gamache  44:17  
I got two last questions for you. Pranav, this has been a super fun conversation. Like I said, email is kind of really close to my heart, so I've got a question about prioritizing experiments and why email is never at the top of that list. This idea of, like, how do you prioritize the tactics and the channels that you want to measure or experiment with is really interesting. We talked a lot about, like, the methods of doing that, but the whole topic of like, how do you prioritize all of these ideas that you have? Email is often way at the bottom of that list, because it has this conception that email is free. Like, why should we need to test something? Because email is free. Email is not free. Many enterprise teams have a. Bunch of internal tooling and resources that are dedicated to email. Email development is expensive. Email Deliverability is time intensive. Those are extensive, expensive resources. And brands love designed emails, and it takes a ton of time to do those. Nowadays, companies are spending big bucks on martech to kind of stay out of the spam folder also, so email isn't free. Like, how can email get a seat at the table and be higher up on those? Like, prioritization for experiments?

Pranav Piyush  45:30  
Yeah, you know, it's really interesting, and we often have this discussion in our team about this perceived lack of leverage on organic or earned or owned channel, right? Everyone thinks about marketing as paid Yeah, like, let's go put more money in x and y and z. And I think if you there's a really fun one from the National Park Services. A couple of days ago, I saw somebody posting about the amazing work that their social media team is doing. Just hilarious captions on social for the National Park Services, right? Like millions of views, hundreds and 1000s of likes. It's like, if you figure out how to do email, organic social earned in a way that gets you a unique advantage over anybody else fighting the paid battle, then you know you have a significant upside in the business. So one is that is like inculcating that culture, that it is possible, and to show that possibility, you really have to look for role models and examples. Arby's is a great example. We were talking about this when Jon Stewart first resigned or left the Daily Show, Arby's tweeted at him and said, Hey, we got a job for you, right? That was hilarious. And guess what? Jon Stewart took that and put it on the show, and that is your classic example of what is possible with earned and owned, and, you know, social and I think email falls in the exact same bucket. And this is where I go back to like full circle, right? Brian's quote from the beginning of this episode about taste and creative figure out the crazy shit, the stuff that nobody's going to let you take risks with and pitch that, because if you just stay in the lane, it's a square. Here's our logo. Here's my personalization, dear Pranav. And here are the three bullets. And here's a video. I'm sorry to inform you, you're dying in the sea of sameness, yeah,

Philippe Gamache  47:38  
especially in B to B everyone, everyone is doing that for now. This has been super fun. I got one last question for you. You're a co founder, CEO. You're also wearing the CS and the sales hat these days. Early in your startup journey, you're also relearning a lot of the foundation elements of marketing, like building a website, doing copywriting, and recently, you've also dived into weightlifting and you're a father of two. Clearly, you've got a ton of shit going on right now. One question we ask everyone on the show is, how do you remain happy and successful in your career? How do you find that balance between all the stuff you're working on while staying happy?

Pranav Piyush  48:16  
It's a good question. I think it's a lifelong pursuit. And you know, the things that I do, like specifically, sort of, you know, lifting weights, that was a conscious decision to kind of give my brain something else to think about for an hour, and something really hard that I've never done in my life. Because if I can do that, then I can do everything else, right? So there's a little bit of that, I think a little bit of it is like just remembering that the pursuit of happiness is just a lifelong challenge. I'm very much into, you know, both ism and stoicism and all of those concepts, and that's a constant struggle. You're always suffering. You're always finding happiness and take joy in the little things, and not every day is going to go perfect. So, you know, be compassionate to yourself. Recently started meditating again with the waking up app. So here's another plug for the waking up app. And I don't know that's and, you know, I think the last part is like, really just staying connected with friends and family and being as sort of vulnerable as you can because that is what makes, ultimately, we're all gonna be in a, you know, six feet under the ground, or whatever your ending. You know, rights are. You're like this. There's nothing to really worry about in life and just constant remind that that's what it is. I'm sounding way too Zen, but in reality, I probably am not as stable and calm every day of the week. So I want to end on that note, but it's trying to stay connected with myself. What's working, what's not working, journaling, that's something that I've been doing for a couple of years has been fantastic. Yeah. So bunch of little life hacks. Yeah,

Philippe Gamache  50:02  
awesome. What's your number one parenting hack? I've heard that, you know, like, one of the main obstacles to building your own thing and starting out your own company is, like, I've got a little family. I've got little kids that I don't want to miss out on their life. Like, I don't want to work like, 1214, hour days. Like, how, how do you think about the I

Pranav Piyush  50:23  
don't know that I have a tip, per se. I like teaching a lot, and so one of the things that keeps me grounded is teaching my girls math, teaching them science, teaching them. I heard my seven year old one day, just like saying something like I was talking about branding or marketing. It's like, yeah, that's what you do, right? And I was like, proud father moment, oh my god. Like, she knows what that means. He's like, Yeah, you go to work and you build a brand. And I'm like, This is amazing. So just like little things like that, of just, you know, constantly be teaching and also be learning from them, right? What are they learning in school, and what are the conversations they're having with their friends, them trying their jokes on me to see if you know they can make Papa laugh. It's it's just being there making sure we have enough time. Because that's definitely been an issue where you have less time and you have to constantly kind of make sure you don't give up on that part of life. Love it. Thanks

Philippe Gamache  51:25  
for sharing. Pranav. Anything you want to plug for the audience, for we go,

Pranav Piyush  51:30  
No, I love humans of martech, so I hope that you all continue to grow your followership and listenership. I get a lot of value, and I think I've learned a lot from the episodes I'm sharing all the amazing guests. So yeah, nothing to plug. If people are interested, they know what to

Philippe Gamache  51:49  
do. Love it. I appreciate it. Thanks so much for your time. For now, it's been super fun. Absolutely you.

Philippe Gamache  52:00  
Music folks, thank you so much for listening this far. Really appreciate you being here. I just wanted to call out two quick things before I go, as well as give a shout out to other martech creators that you should check out. The best way to support the show is by signing up for our newsletter on humansmartech.com I send you a quick email every Tuesday morning letting you know what episode just dropped, but I also include my favorite takeaways. So if you don't have time to listen to that one, no sweat. I've got you covered with learnings anyway. And proceeds from sponsors this year have allowed me to venture into YouTube. So recently, launched a YouTube channel where I publish full length episodes. If you want to see my radio faced, check that out. But I also do clips from different episodes. So if you don't have an hour to sit down for an episode, you can search by specific topics and just listen to that one answer to from that question. And if you can't get enough from our tech content, I wanted to give a shout out to some of my friends and awesome creators in the space. We featured Justin Norris in episode 107 this year. And if you haven't checked out his podcast and newsletter, revops.fm you won't be disappointed. Justin is a technology hipster with a polished voice of reason, and he's interviewed big names like Wes co Joe Rowley and John Miller, but he's also got solo deep dive episodes like how to create a knowledge base for marketing ops and how to use AI, a guide for marketers. Check out our show at Rev ops.fm, that's it for now, folks, thanks so much for listening, and we'll catch you next week. You.

Transcribed by https://otter.ai