[00:00:00] Phil: You lead marketing measurement at Canva, [00:00:02] Matthew: We saw these challenges that existed with, attribution And we said that we think. It's a better way. [00:00:07] And you know, we sort of, I won't say abandoned it, but we really pushed it to the side in favor of MMM and experimentation. And I do think that to a degree we overcorrected. we use attribution not to not validate, but sense check our MM, if something is like really performant through the mm MM, but attribution is poor. that's where data science should live, right? In that, in that ambiguity, multi touch attribution, or attribution in general is valuable for understanding like how users find your product, what's the quality of users that come in through your marketing, Because MMM still aggregations, right? It's like. Aggregated data, not use level data. [00:00:44] ​ [00:01:10] In This Episode --- [00:01:10] Phil: What's up everyone? Today we have the pleasure of sitting down with Matthew Catino, marketing Measurement Science lead at Canva. In this episode, we cover canva's prioritization system for marketing experiments. What happened when they turned off branded search, how they connect marketing data into company forecasting their AI workflow automation for. [00:01:28] Geo experiments and why multi-touch attribution still matters at Canva. All that and a bunch more stuff after. A quick word from two of our awesome partners, [00:01:36] ​ [00:03:15] Phil: Matt, thank you so much for your time today. Really excited to chat. [00:03:19] Matthew: Yeah, I'm really, I'm really glad we got to figure this out. Time zone wise, just, uh, you being in New York, me being in Sydney, time zones don't overlap super well. Uh, so yeah, really glad that we got to find a time that works around our work and kid schedules. So yeah, thanks for having me on. [00:03:35] Phil: Yeah, I don't think we need to introduce folks to Canva. Super well-known product. Uh, obviously massive brand. I'm a daily user myself. Uh, Darrell uses Canva as well all the time. Um, maybe we can just kick off the conversation to. Day by talking about volume and scale, [00:03:51] 1. Canva’s Prioritization System for Marketing Experiments --- [00:03:51] Phil: Canva probably gets what, like I was trying to figure it out, like 600 million monthly visitors or like 200 million active users. [00:03:59] The numbers ~of data or users and data is~ [00:04:00] probably insane, and you could run hundreds of tests at any given time. Maybe you can chat about like some of the guardrails that your team is setting to avoid spreading the marketers or your own team, like too thin when it comes to. Producing conflicting outcomes with experiments like do you have a central experimentation team or council? [00:04:18] Is there a shared calendar of tests that are going out or like approval processes to balance speed with scientific rigor? Talk to us about that. [00:04:26] Matthew: Yeah, sure. So I think it's important also to differentiate like Canvas core product experimentation with what we do in marketing. Um. So Canva has a core experimentation platform for experimenting with features in the product, you know, um, uh, which is super sophisticated and has so many people working on it over a number of years, and I really wish in marketing that we could it leverage it. [00:04:52] But unfortunately, as we know, like marketing. Uh, a lot of it's outta the product, so we can't leverage this like, super [00:05:00] sophisticated, um, experimentation platform that, um, Canva has developed internally over a number of years. Um, so in, in marketing, we've kind of over the past few years developed our own internal tooling, um, to run experimentation out in the world, like doing, uh, geo tests and other kinds of experiments like that. [00:05:20] And you're right, like we're, we're, we're doing. Last count. We're doing marketing in at least 15 or so countries globally now. Um, and, um, across so many different channels that the, the, um, you know, the possible combinations are so many, right? We couldn't possibly experiment on everything all at once. Uh, so yeah, we do have to have some kind of prioritization framework and, um. We also don't want to create a process, as you said, which is so labor intensive and, and, and [00:06:00] resource intensive for both data science and marketers that it gets in the way of, you know, doing marketing better, right? We want measurement to sort of like be this like thing in the background that happens but doesn't like o occupy so much time that it takes away from like improving the, the creative and the strategy and all sorts of things. [00:06:19] So for us it's about, um. We've sort of developed a prioritization framework, which has got two big levers. The two big factors, which would be budget, because with budget comes opportunity cost, right? So if you are, um, spending a lot of money and you're spending it, uh, inefficiently, then there's a massive risk to that, right? [00:06:44] So like, um, we prioritize our highest budget channels first for experimentation to make sure that, um. Where we've got significant investment that we're the most confident we can be with those big investments. So, [00:07:00] um, in markets like the US and on our big, uh, digital channels, we're really make sure we prioritize those to make sure that we've got the, the most confidence that we can have in our higher spend channels. [00:07:13] And simultaneously, the other key factor is uncertainty. So, um. Uh, we live in a deterministic. We no longer live in a deterministic world, right? Where we, we, um, we don't, we don't know what the true impact is. It's more about being the most correct you can be, or the, or the least wrong that you can be rather than being right. [00:07:39] So we've got a few different ways of trying to figure out like, how confident are we in the, what we're spending on this channel. So. we'll do things like, we'll triangulate sources from like what the MMM is saying. A marketing mix model is saying what the platforms are saying. We'll look at, um, [00:08:00] conversion lift tests. [00:08:01] We run in platform. We'll look at engagement on the ads. So kind of figure that if engagement is really poor on the creatives that are running on that channel, it's really hard for the, um, channel to be operating efficiently in terms of like CPA, things like that, right? So. We also look at the confidence intervals that come out of the MM and say, you know, where the confidence intervals are wide because, you know, we haven't varied spend a lot historically, or for whatever reason, the MM is sort of like a lower confidence in that channel is a wide bound of possible values. [00:08:34] We know that those channels are the ones where there's also risk that we're spending incorrectly, right? Or not spending as well as we possibly could. For each market. That's the process basically, is we try and we try and figure out where is the, where's the most money spent in the most uncertain way based on triangulating all these different signals. [00:08:55] And um, what we're trying to do as well [00:09:00] is, historically experimentation has been a centralized function at Canva, like with the measurement team that I've run for a few years, has really owned, um, all of that experimentation globally. But as we expand, um, into new markets, we expand into new regions. We are hiring more locally so that we have like more bespoke and local approaches to creative and strategy in these regions. [00:09:27] There's, there's too much for sort of like a small, centralized team to run. So been thinking about how do we decentralize some of the knowledge and the skills around experimentation and measurement to empower. You know, data scientists embedded in those regions to sort of like partner with the teams locally, with the, with the strategists and the marketing folks locally. [00:09:54] And then that way they have this sort of like, there's not one central prioritization [00:10:00] roadmap, there's now a roadmap for each individual region that's grounded in sort of like their local expertise combined with the data. Right? Um, and that allows them to be more agile because what we don't want. Is, um, we, we want to have a set of best practices that guide experimentation and, and roadmaps and strategy, but we don't want to have this approval laden process. [00:10:22] That means that we can't act quickly, we can't be agile. Um, and the teams are sort of constantly waiting for like, heaps of approvals to happen to sort of like, you know, to lean in on where they could invest more or pull back on where they think things aren't working and where to experiment when they're unsure. [00:10:41] So that's kind of a really long answer, but that's what we're trying to work on is like prioritization, decentralization, and agility because um, yeah, as we expand globally, it's just too hard to have all those functions sitting with one or two people. Um, yeah, so that's how we [00:10:59] Darrell: Yeah. [00:11:00] No, that's great. And I, I, I think, uh, what really resonates with me is, you know, where you said we're trying to be at the least wrong as possible because a lot of the. A lot of the data and the measurement models, you know, are gonna be either directional or, um, and, and that's something that I hope marketers understand, right? [00:11:20] It's not, it's not gonna be an exact science, but, uh, you know, you, you can try to get as close as as you can. Um, ~but I wanted to shift gears a little bit too and, uh, you know, talk about, um, you know, experimentation and~ [00:11:27] 2. What Happened When Canva Turned Off Branded Search --- [00:11:27] Darrell: you've led marketing measurement in Canva through a period where experimentation. [00:11:31] Became like a core part of how company makes the company makes decisions. And for a lot of people, experiments are all about testing ads, but for seasoned ops folks, experiments become more of a way to win, trust, influence, budget, and prove impact. So, um, maybe you could, you know, share a little bit around your, some, you know, some of the experiments that, uh, at Canva that you've run or the teams have run that turned out to be a real inflection point. [00:11:57] Um, or maybe so something that you. [00:12:00] Uh, maybe surprised you that you learned, um, anything like that you can share with us. [00:12:05] Matthew: Totally. And I fully agree with that sentiment, right? Which is that, um, a lot of the experimentation and modeling approaches that we've adopted are new to a lot of folks, um, especially people that have come from, you know. Companies that have been very, you know, solely attribution driven. Um, this is a new philosophy, right? And, and the best way to sort of help people, um, bring in this new set of tools into the tool belt, right? [00:12:38] Because it's not about replacing anything, it's about bringing new tools into the arsenal, right? It is to run, is to do things which help make decisions, right? And to give people a clear next step and to show value. Um, the, the earliest example of that I can think of, um, which was like a real turning point for us in the [00:13:00] first major experiment that we read was, um, the question actually came from, uh, the CEO. [00:13:07] And she said, you know, I'm, why am I spending so much money on Google beating for my own company name, right? Like, does it, does that, why, why am I doing that? Like, does it make sense anymore? Right. Where. Canva has such a, um, a big community, such a, like, you know, this large organic growth, we're so well known. [00:13:29] Um, we, we rank pretty highly on searches when people search for Google. So like, there's a good like organic presence in our searching. Like is the, is searching for Canva and bidding on that term, actually driving value. Um, and, and that was the first experiment that we ran. The first, the first big one that got like real attention and real visibility was we. [00:13:49] We just turned it, we just turned it off or down in a couple of geographic regions within the US and said, well, what happens? Like, can we, can we turn this off or turn this down? Or, you know, [00:14:00] decrease what we're willing to bid for our term, um, and how does that impact growth and, and revenue? And the short answer is, it, it didn't, uh, in this, certainly not in the way that would justify the level of budget that we were investing in it. [00:14:13] Um, so we did, we did turn it down and, you know, we saved. Millions of dollars doing so reallocated it. And, um, the, the beautiful thing about that is that what other way do you arrive at that conclusion, right? Because traditional measurement methods such as attribution, they, they love, you know, branded search, right? [00:14:36] Because, um, the, uh, it's, you're capturing a high intent audience and, um, the conversion rate from a branded search would be super high. Right. So, um, traditional measurement methods weren't able to sort of answer this kind of question, and then we were able to, um, to do it in a way that's quite powerful and quite explainable. [00:14:59] You know, like a [00:15:00] lot of things like MMS are quite difficult to explain and that that's always a trade off with some approaches that like, they're really powerful but they're hard to explain. Experiment's quite easy. It's like, you know, group A, group B, group B, we turned it off and we saw that there was no impact and able to turn it down. So that was a really impactful one, certainly for us. Um, and that, that really kicked off the owl. That really gave like, sort of, um, a little weight and a lot of, uh, you know, to what, to the, to think the kinds of things that we were saying. And, uh, yeah. So [00:15:35] that [00:15:35] Darrell: And I lo, I also like how there wasn't too much disagreement about what should be done. It was more of like, let's find out, let's test and, you know, let's, let's let, let's let the data help us draw the right conclusions versus, you know, 'cause a lot of marketers have this debate around whether they turn off their, you know, the, the branded search, um. Do, do you think, like, [00:16:00] is that your recommendation pretty much? Like, you know, for other brands, maybe they're smaller, you know, um, or would you, would you, if you had to bet, would you bet that if they should turn it off too? [00:16:10] Matthew: It is a really good question. I certainly, I, I don't think that, I don't think it's reasonable to make the assumption that because one strategy works or doesn't work for one brand that. That's universally true, right? Um, I imagine that for the reason that it was, the reason that, um, we were able to turn it off, like the, the underlying rationale is that if someone searches for the word Canva, they'll see both the organic link in the top ranked search result, and they'll see the branded search ad, right? [00:16:50] They'll see the SEM result and the SEO. Right. And essentially we found that if the ad isn't there, [00:17:00] people just click on the organic link, right? Which is a cannibalization factor, right? But if you're a new and up and coming brand and um, people search for your thing, you may not rank particularly well for that term on organic search results. [00:17:19] So I think the best thing that you can do. Like if you, if you have the capability, sure, experimentation is the right thing to do, but I also think you can get a pretty good gut feel from just like stepping into the mind of the user, seeing what the experience is and being like, you know what, what is the journey here? [00:17:35] Right? And like, can I, if I don't run this ad, how would people, would people still find my product? And I feel like if you could answer, if you get a good feel for that, intuitively, that's a pretty good place to start. Um, so yeah, I definitely don't think that. And again, like Canva's, a freemium SaaS product, like, would I recommend this strategy for toothpaste? [00:17:54] Like, you know. [00:17:57] Phil: D different ballgame for like smaller [00:18:00] companies, especially in like niche, very competitive areas. Like, uh, I worked at a bunch of B2B companies where if you searched our brand term, like seven of our competitors would pop up at the top and at the bottom. And, uh, people were maybe already considering those solutions, maybe they weren't. [00:18:16] So we were like giving away traffic to, to some of those other folks. So yeah, it's, I, I [00:18:20] Matthew: companies can bid more on your terms too. [00:18:23] And the other thing I wonder is like if your, if your company name is a genuine word as well, you know what I mean? Like, [00:18:30] um, [00:18:31] Phil: Does that make sense? [00:18:32] Matthew: like it's, it's pretty hard for Google to recognize that, you know, if your company's called cat, you know, like, um, yeah, I dunno, that's a terrible example. [00:18:44] Phil: Uh, ninja Cat. I think there's [00:18:45] a, a Ninja. [00:18:46] cat. Yeah. Um, yeah, let's, uh, let, [00:18:49] 3. Structuring Global Measurement Teams for Local Decision Making --- [00:18:49] Phil: let's chat about like structuring measurement. Like you teased this a little bit in your previous answer, but, um, you said that like the, the hardest part of measurement isn't like the modeling, the attribution, it's everything around it. [00:19:01] Talk to us about like how, uh, Canva is structured today to make sure that when your team is pumping out insights and results, that it gets translated into action to the marketers. Can you share like how that structure has evolved over time and, and maybe like what didn't work along the way? [00:19:20] Matthew: Absolutely. And, and Canva itself has changed a lot in that time and we've had to evolve in response to that. So, as I sort of mentioned, like all measurement functions in terms of like modeling and experimentation, sit within a single team. And you know, at that time most marketing was operated out of headquarters, which is in Sydney. So. We're in the same time zone. Um, we can get into a physical room if we wanted to with everyone that we needed to communicate with. So having things centralized made sense, right? Sort of ran everything [00:20:00] out of a core group of people. Um, and then, you know, the last couple of years, canvas kind of recognized that to unlock that next stage of growth for our company, we, we need to adopt a more. [00:20:14] Localized approach in different regions, right? It's, it's not good enough to sort of like take the ads or take the product even it's working really well in, say, the US and Western regions and just ship it out all over the world. It doesn't work, right? It, it, it, it, it works for people that are super high intent, right? [00:20:36] But to capture everybody, which is, which is the goal, right? The goal of Canva is to be a design solution for everyone everywhere, right? And to do that, we need to have a more local approach to how to, not just marketing, but the product, the pricing, the content. Right. So, um, for example, if people log onto Canva and they don't find templates and [00:21:00] resources and images that are, that resonate with them, they won't come back. [00:21:04] They'll leave. Right. And, and, and so like the company has shifted that way, marketing has shifted that way. And then as a result, our team has have to shift that way. So. Um, we have found that trying to centralize everything is working less and less well because we just have to have, it's hard for a, a small number of people to have like, distributed knowledge around what's happening in all these different regions. [00:21:28] Like in terms of like what their strategy is, what, what, what marketing they're running, what the seasonal trends are in those markets. Um, the, the quality of the creative and, and, and the creative strategy that's operating all these different markets and that's needed to sort of like. Translate the insights of the model into like actionable recommendations. [00:21:47] So what we're working on is, as I said before, just trying to have a team that's, that's in charge of the technology, which is like the modeling, the experimentation, the core, and then best like a best practices [00:22:00] and then, um, empower these embedded data scientists and embedded teams with the ability to. To generate insights from the model that impact decisioning and empower them to be in charge of their own decisioning, operating within the, sort of the decision making frameworks that we create, um, in the tooling team. [00:22:23] And, and that's how our structure has evolved. So, um, so the team that I lead now is, is more about, um, the tooling, the assets, the technology, the research and development across, uh, across marketing science. Then, um, we now have this team that has embedded data scientists working in regions that will translate the insights to the models and that allows them to give more bespoken specific recommendations to those, to those local folks. [00:22:57] Um, [00:23:00] but yeah, I think like not only is like a structure of evolve, but just sort of like, it's really hard to find the. It takes time and expertise and experience to work out the right way to communicate stuff, like the right level of detail to give, you know, because like we might make a change to this like modeling parameter that causes a CPA to got by like 20% we're down by 20% and it's like, it's really easy to get lost in the weeds of that stuff. [00:23:29] It's like, oh yeah, we, we changed the Gaussian smoothing process. Uh, which you know, and then, you know, like. That like means that like, you know, now the, the ROAS is higher or lower or whatever, you know, and then it's really easy to get lost in that. So I, I find that focusing on what it means for people, what the insight and the action is, is the right way to do it. [00:23:54] And that's been the learning for us for the last few years. Yeah. [00:23:58] Darrell: Yeah. That's really great. [00:24:00] And yeah, it, that resonates with me too because I think, I think it was Scott Brinker that said that the, you know, marketing and measurement and ops is oftentimes. Uh, cyclical when it comes to, uh, centralization and decentralization. So, so at first it's centralized, then you decentralize, then it get, can get kind of like too wild. [00:24:20] So you have to kind of take back some of the control. So it's just a, uh, a a never ending kind of game of, of governance and then enablement like a, you know, there's like a balance. Uh, so I, I, I, I definitely agree with that. ~Why don't we talk about, you know, we've been talking about collaboration. Um, so, so why don't we talk about, uh, this question. ~ [00:24:33] 4. How Canva Integrates Marketing Measurement Into Company Forecasting --- [00:24:33] ​ [00:26:37] Darrell: How do you make sure the measurement team isn't just serving marketing, but also providing clarity to finance, leadership, and product? What do you think about that, Matthew? [00:26:46] Matthew: Yeah, that's a really good question. Um, we. This is like an evolving work stream for us at Canva, which is, um, what we're trying [00:27:00] and, and to be honest, like, I dunno the right answers for you today. Um, I, I will say that historically working with finance has been a really good way to sort of get us seat at the table decision making wise, like finance within marketing, you know, because if you can, if you can, um, demonstrate the ROI. Of things and show that this strategy's working, this strategy's not working. You have like a good, um, you have a good support system for, you know, generating action, right? Because if, if the finance folks here that something's not working and it's not giving good return on investment, they wanna make, they wanna drive change, right? [00:27:40] So I totally recommend that if you're starting out and you're not really sure how to get buy-in, but you're not really sure like what the right support is, talk to finance. Because I think that like. Measurement and finance speak a lot of the same language, right? They have a, they have a lot of, they, they're, they're trying to do the same sorts of things, approach from different [00:28:00] angles, right? [00:28:00] They have a lot of common understanding, common language. Um, what we're really trying to work on now is, uh, and a lot of this, uh, interesting, exciting changes driven by our fantastic new CFO, uh, Kelly Stackelberg is, uh, what we really try and do now is. Create connectivity between how we measure marketing for marketing with sort of our, uh, company's planning company's annual planning, company's long range planning, right? [00:28:31] So, um, uh, we'll have a forecast which forecast where we expect to land this year and over the next three years. But, um, there's not particularly good connectivity between how marketing measures. within marketing and for marketing with this sort of like company level, top line plan and long range planning over the next three years. [00:28:56] So what we really want to try and we're really trying to work on now [00:29:00] is figure out, you know, marketing's a big lever for growth, both in terms of a calendar year goals, but then also your long range goals. So you're trying to really understand and create that connectivity. To say that of, of the growth that we expect, this is the proportion that we expect to get from marketing and you know, if we invest in this way, this is where, how we expect, um, growth over the next few years to, to plan out. [00:29:27] Right? And what that allows then to do is to have all the different levers for growth across Canva, sort of like on a level playing field and, you know, represented in the same model. In the same way so that the company can then start to think about resource allocation and what levers it can pull to drive outsized growth. [00:29:48] Right? So that's what we're trying to work on now and, and it's a complicated problem, right? Like particularly over the long term, right? Because not only do you have to sort of think about, well, what's marketing [00:30:00] budget? What's marketing spend gonna look like in the next few years? You also have to start thinking about. [00:30:06] Is there a compounding effect to of the, you know, brand loyalty and memorability and, um, awareness, those sorts of brand health metrics. So we think about today, is there a compounding effect of that over time? Right. In terms of, you know, we like to believe that. If people know who we are, they love us, they understand what we stand for, our values, our, you know, what value we can bring to them, that that will have a long term impact on the growth of our company. [00:30:44] Right? But we're trying to quantify that is a difficult problem, but that's the sort of the path that we're trying to head down now is create this better connectivity to help the business understand how marketing interacts. What role it plays in top line [00:31:00] and then connect that to long range planning, which is an exciting thing to work on. [00:31:04] Um, well, exciting thing for my staff, data scientists to work on. [00:31:10] Phil: Yeah, super exciting. I, I think for a lot of listeners too, like this idea of finally having more people care about connecting marketing activities to long-term stuff that other departments are doing is super exciting and you. Like you mentioned finance a bunch of times there. We like, we like to pick on finance a little bit on this show here and I love the shout out that you have for them. [00:31:32] Like there is a lot of overlap with the terminology in data science, measurement and finance teams and I feel like you guys are playing a cool role. Bridging that gap between the marketing terminology because you know, the roas, the CPAs then qls. Like all of those things aren't a hundred percent relatable to like A CFO, but you kind of play in that role. [00:31:56] In between both of them I think definitely helps there. [00:31:59] 5. Using MMM Scenario Tools To Align Finance And Marketing --- [00:31:59] Phil: I'm curious to like get your take on those like conversations with finance, like when you're sitting down. With a finance leader to talk about growth or marketing versus efficiency, how do you structure that conversation? Like do you show scenarios, uh, probabilistic about like ranges of outcomes? [00:32:18] Do you tie every recommendation to a single KPI? Like, what are your thoughts are? How do you handle those? [00:32:24] Matthew: Yeah, there's a lot there. I think like, um, I think we do a pretty good job of see, of like understanding upfront, like what the goal is, right? Is it growth or efficiency or like. We, we, we tend to do a pretty good job each year of sort of aligning on what we think the right trade off is. So like we're, we're a good, where it feels comfortable, right? [00:32:47] In terms of we don't want to prioritize efficiency to the point that we're leaving growth on the table, but we also need to drive growth you reasonably profitably, right? So I actually find those [00:33:00] conversations go pretty well, right? 'cause as you said, you can, you can highlight ranges. You can say, you know, um. We've actually just deployed a tool, um, that takes the, um, output of our MMM. So, and then what it does is it allows finance or other stakeholders to select the, um, it allows 'em to select the, uh, spend objective or like the objective. Like, do I wanna maximize growth? Do I want to hit a certain, um. Like ROAS or profitability threshold? [00:33:38] Um, do I wanna maximize user growth? 'cause in different markets and different, you know, the, the, the strategy is different. So in the US we're really focused more on monetization, whereas in places like Southeast Asia, we're focused more on free user growth, right. Being a freemium product. Right. So we can, we can allow them to sort of like, um, [00:34:00] choose like what they wanna optimize for and. [00:34:03] Set their growth and efficiency trade offs and then it will generate scenarios, right? And then sort of just, it allows 'em to sort of like navigate and see the trade-offs that exist between different scenarios. And obviously like there's this expectation perfect, but right. But it's like if I care about efficiency broadly, this is how much I would spend, or broadly where I would distribute the money across markets. [00:34:25] If I care about user growth, I'm like, we're really interested in like free user growth. This is how I would blah. This is how I would distribute money across markets and things like that. So I find that that goes pretty well. We've got a pretty good shared, you know, understanding of that. And we can do scenario trade offs and that sorts of things. [00:34:42] Where it gets trickier is more around the, is more around the, the things that you expect to have the impact over a longer time horizon, right? Because when things have an impact on the short term. I would say performance marketing, we expect it to have a relatively short [00:35:00] term around time between investment and payoff. [00:35:02] Right? Um, finance is very comfortable in that space, right? Because it's like dollar in X excel's out, right? When you start playing in things that take, that are more like brand building and, and, and don't have immediate payoff. It's more about, it's more about, um. Just telling people who we are, but not necessarily expecting those activities to result in conversions. [00:35:28] That's where the thing get a bit trickier with finance, I find. 'cause it's like, what do you mean we're gonna spend all this money, but the revenue payoff this year is gonna be X. Right? It's like, it doesn't feel very good and, you know, um, so that's where it comes into that, that's where that long range planning comes into is we wanna be able to demonstrate that like we're investing now for like two, three years down the track. [00:35:50] Um. I find that's where I find, find things get a little bit trickier with finances when things don't have this immediate payoff. Um, but we we're fortunate enough to, like, we've been doing [00:36:00] this for a few years now with the same core group of people, so we have a good, like, shared understanding of what the goals are. [00:36:06] But yeah, definitely the, the, the longer range stuff is where things gonna be stickier with finance. 'cause they're like, I want things to pay off now. And it's like, well focus just on now, then you're gonna hamper our growth in the long term. But that's also. Harder to mathematically quantify, right? [00:36:23] Phil: Yeah, it's, it's always a tough conversation. Not even just finance, like, uh, other executives on, like the senior management team Also, uh, we've had some folks on the show try to explain it as when you invest in brand. You're actually lowering CAC in the short term, or at least medium turn, and like tho those results actually help you in those performance metrics. [00:36:46] And so kind of bringing it back to that shared understanding. But it, it's a tough one to unpack too, like when we talk about shared understanding stuff between marketing and finance, like one of the things that comes up a lot is [00:37:00] attribution. And I feel like we, we, we tease the attribution out a little bit earlier in the conversation, but. [00:37:06] 6. Why Multi Touch Attribution Still Matters at Canva --- [00:37:06] Phil: You lead marketing measurement at Canva, massive brand. There's a ton of stuff in the measurement toolkit now. Um, I've heard you on a couple other podcasts say that you've kind of went from just relying on, on attribution to now having a couple of different things in that measurement toolkit. We've spoken to plenty of different critics around multi Dutch attribution, a lot of supporters, a lot of folks that say, you know, you shouldn't be doing anymore. [00:37:32] You should be doing MMM only, or you should only be doing incrementality testing. How do you think about attribution today Inside Canvas, broader toolkit, like where does MTA still shine in your opinion? [00:37:45] Matthew: It's a good question. Um, I, I will say that I, I do think that an argument could be made that we overcorrected a little bit in the sense that, you know, [00:38:00] we saw these challenges that existed with, with attribution when it comes to making certain types of decisions. And we said that we think. It's a better way. [00:38:11] And you know, we sort of, I won't say abandoned it, but we really pushed it to the side in favor of MMM and experimentation. And I do think that to a degree we overcorrected. And that's something that we're thinking a lot about now, right? Um, I think it's like this black and white of saying that this tour is good, this tool is bad, or this is right, this is wrong. Easy to fall into that just for like pass money, but like heuristics, you know, whatever. But some, I don't think logically it's the, it's the right way to think about it. It depends on what you question you're trying to [00:38:46] answer, right? And it depends on like, um, it depends on what you're trying to do. And in marketing, good quality data is so [00:39:00] hard to come by. It's not always wise, it's just throw stuff away, right? Like, so a, a few different things would be, we use attribution from platforms or from internal attribution to sort of sense check, not to not validate, but sense check our MM, right? In the sense that if something is like really performant through the mm MM, but attribution either in platform or. [00:39:30] Internal attribution models is poor. That's, that's a time to stop and pause and go. Like, we have two different methodologies giving you two very different answers. Right? That's, that's where data science should live, right? In that, in that ambiguity, right? That's where we have the most value, and that is an opportunity to think about, like, how do we figure out where the truth is, like where the least wrong thing is, right? I also think [00:40:00] that, um, multi touch attribution, or attribution in general is valuable for understanding like how users find your product, what the journey is to finding your product, and then what they do from the ad through to your product, right? And then you can also look at interesting things and look at like, what's the quality of users that come in through your marketing, right? [00:40:23] Because MMM still aggregations, right? It's like. Aggregated data, not use level data. Um, but like perhaps you look at through attribution, you see that, yeah, you get lots of signups through Channel X, but they're lower value users, right? That, that aren't driving much value. You can't get those kinds of insights from an MMN, right? [00:40:47] Phil: Sure. [00:40:48] Matthew: Um, simultaneously, this, this other channel might be driving less users, but driving higher LTV users or higher value users. can also figure out where people get stuck in the, in the journey. Like maybe they come in from, [00:41:00] from this channel and they hit here and then they get lost, right? And then like that's an opportunity to improve your landing pages, prove your product flow, all that science sort stuff. [00:41:09] So for us, attribution's valuable for, for calibration exercise. Um, if we also see like that we launched new sets of creatives and. You know, through attribution, through the platforms or whatever, our CPA drops, right? Um, m m's gonna take some time to catch up to that. Whereas attribution's quite responsive to those kinds of things, right? [00:41:36] So in those situations, probably you should act quickly and invest some more money. If you are, the, the strategy is like, you know, had a step change is now working better rather than waiting three or four months for MM to catch up, right? So like. I would say generally probably overcorrected a little bit. [00:41:55] I think that there's still value to, there's definitely [00:42:00] value to attribution that it still exists. I think it's, um, but I think it's about being clear on what the strengths and weaknesses are of each tool, right? And how they compliment each other and how they can fill each other's gaps. Because the, the days where a single. And Google have published a bit about, this is the days where a single solution does everything is over like that, that you, that, that, that, that time has passed. Right? So don't, my recommendation is don't, don't throw out good data. Oh, any [00:42:36] anything's valuable in this long, deterministic world. [00:42:40] Darrell: A hundred percent. I agree. I agree. Um, [00:42:43] 7. How Canva Builds Feedback Loops Between MMM and Experiments --- [00:42:43] Darrell: why don't we shift gears a little bit and talk about experiments versus models. And a lot of people say that experiments are the gold standard when it comes to determining causality. Um, but you can't scale experiments infinitely, you know, and they're, you know, sometimes costly to run and you can't just run them all the time. [00:43:03] Um, and, but when it comes to models, models can scale. But come with complexity. So how do you think about balancing those two? Um, especially like inside Canva and, uh, you know, do you, you, do you use experiments to calibrate the models or the models to identify where to experiment? How do you think about that? [00:43:22] Matthew: Yeah, good question. Um, definitely agree. So the, the, the modeling is great for breadth, right? Like lots of countries, lots of channels. Um, experimentation is. Looks a little bit like for us, manual and time consuming. Um, and it's challenging to run multiple experiments in a given market at the same time. So, so the modeling's really good for breadth, right? [00:43:49] But the thing that I always come back to is when you think about what you're asking an MM to do, you're asking it to do quite a lot of heavy lifting, right? [00:44:00] You're asking it to estimate hundreds of parameters. For the, like seasonality, the efficiency of your marketing, your baseline, all this different stuff from not that much data, right? [00:44:13] Typically it's like daily grain data for the past two or three years, right? So maybe like a few thousand rows of data or few hundred rows of data. You, you're asking it to do a lot right? And, and to expect it to, to perfectly in with very high reliability. Understand the efficiency of each individual channel within a market when there's so much opportunity for bias is a lot. [00:44:44] You're, you're asking it to do too much, right? So MMS are only as good as the data that you give it. And they're subject to things like mis specified prior to, you know, um, uh, multi linearity where you have like channels that are [00:45:00] being spent. In like the investment strategy or the pacing for different channels is very similar. [00:45:04] So like if you always spend time on Facebook, when, when you, and when you spend time on TikTok and whenever you drop SPA on Facebook, you drop spa on TikTok, how is it supposed to tell the difference between those two things? It can't, right. So, you know, there's too much opportunity for bias, right? So for us it's like we need, experiments are very important to us to sort of, to calibrate the model, to sense check the model, and um, sorry. Yeah, totally. We, we've got a big experimentation program. We're trying to scale it more. Um, uh, I can talk a little bit about how we're using AI to try and do that as well. Um, but yeah, experimentation's core central to our philosophy, because m's, like best possible thing when you don't have an experiment, that's kind of the driving philosophy, right? [00:45:54] The best possible thing when you don't have an experiment. All trends, the experimentation know, is that they're time [00:46:00] bound. You know, it's like this experiment ran for two months, eight months ago. Like, how, how long is that experiment result still valid? Right? Um, well since then we've um, had three world wars, two new creatives, you know, like, who knows whether that result still valid, you know, so the short answer is they compliment each other. [00:46:22] They mm m. We use experimentation, uh, sorry, uh, modeling results and attribution and things like that to sort of figure out where we're least confident. That's where we prioritize experiments, feed that back into the model to create these like dynamic feedback loops. Um, but yeah, it's just certainly challenging at our scale, the number of markets that we have. [00:46:43] Phil: Yeah, no kidding. [00:46:45] 8. Canva’s AI Workflow Automation for Geo Experiments --- [00:46:45] Phil: Matt, can, can you talk about like where AI kind of fits in canvas's? Measurement workflows, like are you using AI for some forecasting stuff? Uh, an anomaly detection or automating, like reporting parts? Like what does that look like internally? [00:47:01] Matthew: Yeah, so, um, so fortunately our provider's pretty good anomaly detection, so they kind of like take that away from us, which is nice. Uh, shout out to recast. Um, uh, I would say, uh, where we're trying to use AI is twofold. The first one is to help decentralize some of the more technical things that are involved in experimentation. [00:47:29] So, for example, we're a, we're on a geo experiment. Um, you needed a reasonable amount of expertise, um, to understand like. Like the, the, the, the Python notebook or the R notebook that that generates the geo, the geo splits, the synthetic controls, all that sort of stuff. There sorts of parameters that you can tweak and if you wanna scale that process to a lot of people, it's quite challenging. [00:47:55] So we're trying to do is build, um, a natural language agent that [00:48:00] sits on top of this package so that people can, to say, I want to run an experiment in this country or this channel. Then it, it'll say like, how long do you wanna run it for, or what's daily spend? Um, and then you can sort of interact with it that way. [00:48:14] And then it's sort of like, there's still an element of expertise but not like super niche that you need to be able to interact with that thing now. And now everyone can run work for it. Now lots of people can run experiments and you know, we're sort of trying to have like 15 people ping one person say, I experiment on this. [00:48:31] And it's like that person's drowning in work. Right. So, so we're using AI in that way to sort of like, try and, um, uh, um, sort of democratize access to previously, um, tools with a high barrier to entry, right? So things around experimentation. Uh, we're also, um, working a lot with Snowflake and the Cortex product to try and introduce. [00:48:59] Phil: Hmm. [00:49:00] [00:49:00] Matthew: Natural language querying of the data warehouse, um, for stakeholders. So what we're trying to focus on, data science is creating a semantic layer in our warehouse, which has, you know, the core context that the model, that the model needs to answer questions in a reliable fashion. And then we're working towards a world where, so there's, you know, there's low hanging questions that sort of like, um, probably exist in a dashboard somewhere that someone can't find or. [00:49:30] Um, that don't take that long to answer individually, but when you get heaps of them, they take up heaps of time. Um, trying to create this like natural language interface for our stakeholders, um, so that they can, um, you know, answer questions or get answers to questions quickly, um, without requiring a data scientist, allowing a data scientist to focus on more complex work like developing experimentation, tooling, so on and so forth. [00:49:57] Um. the the [00:50:00] third avenue I'm kind of excited about that we are sort of dabbling with is, uh, using large language models to, um, to tag creative or to, to do meta tagging of creative so we can sort of understand like what are the features of creative that resonate well with folks, right? So you can sort of. [00:50:21] Um, use an AI to sort of like, consume your ads and, and, and generate feature, a feature set, a feature store of, like, this creative has this feature at this time in the video. Um, generate huge number of features and then you can train, you know, machine learning models to look at like, which features, features are most important for high performing creative. [00:50:46] Then you can use that to create like best practices, playbooks to drive creative strategy, creative direction. Um, and the beauty of this is that, you know, it's conducive multiple [00:51:00] languages. Um, could take multiple different mediums from statics to videos, um, to audios from podcasts. So, yeah, uh, that, that's something we're really excited about, um, developing. [00:51:11] Darrell: Love it. Love it. Yeah. And I love the practical application of ai and also it's just so good to talk to someone that's, you know, in the trenches really implementing. And to get your, your opinion. So it's been, it's been such a pleasure, Matthew, having you on. Um, and, uh, but we're running outta time, so, uh, wanna get to this question. [00:51:31] So, [00:51:32] 9. Why Strong Coworker Relationships Improve Career Satisfaction --- [00:51:32] Darrell: Matthew, you're a measurement lead at one of the best tech brands on the planet. You're also a father of two, a five-year-old and a two yearold. And you're an avid gardener, which is great. One question we ask everyone on the show is, how do you remain happy and successful in your career, and how do you find balance between all the things you're working on while staying happy? [00:51:52] Matthew: Yeah, that's a great question. I, I would say like, um, the thing that really gives me the most [00:52:00] satisfaction from. For my work. It's just the people that I work with. And, and I don't know if that's a practical advice to people because it's sort of thing that's sort of outside your control maybe. But I, I, I can't tell you how fortunate I am to work with such, such really richly intelligent people that are just also so great to work with. [00:52:20] And, you know, when work gets hard or life gets hard, knowing that you have these sort of, um, these people, the support system around you and at work, um. And that you can rely on other people to sort of like pick things up or to, to, to rally with you when you need to turn stuff around quickly is super helpful. [00:52:40] And my, my practical advice is that, you know, good relationships at work probably pretty highly correlated with, uh, with work satisfaction. So invest time in relationships in your workplace because I think that like. Um, maybe that is that, that that is something that's in your, in your control, right? Is investing time and developing good relationships in your [00:53:00] work. [00:53:00] Because I, I think ultimately, if, if you can, if you have a good belief that the, the people around you are well intentioned and want the same thing as you, even if you just in times of disagreement, those kinds of things carry you through, right? Denying that you're both trying to, you both want the same thing and you want good things for each other. [00:53:18] Like, that carries you through when you have differences of opinion or when things get hard. Um. Outside of work, it's starts to sort of where you have a hard day, you have a hard week, or things are a bit stressful and you walk outside and the kids don't care. You know, you know, they don't know, they don't care. [00:53:35] And you know, they're just like, dad could go on the trampoline, like, sure, we can go on the trampoline. You know, like, I think [00:53:40] like [00:53:41] Phil: Your 2-year-old doesn't care that like, uh, an experimentation model broke. Like they, they [00:53:45] Matthew: you know, shockingly, no. Um, and uh, yeah, so it's good to sort of like. Step outta this office. And then like, so I work, I work remotely and sort of step out, step out of this office and you know, um, straight away there's someone that's [00:54:00] demanding your attention on something. You know, they've got a much more tangible set of problems for you that you can solve, right? [00:54:07] Like, you know, um, then, you know, what's our roadmap for the next three years, right? Um, but yeah, just, uh, being outside, being with my kids, being with my wife, like all that stuff is just, uh. It's good to sort of step out and just sort of go like, they're just ads, you know, they're just ads, [00:54:26] you know. [00:54:27] Phil: So true. Great advice, Matt. Like super grounding, um, especially like in the technical role that you are right now. It's so cool to hear you say. You know, I'm heads down, I'm doing the work, but I actually think like the key to being happy at work is investing time in relationships with the people you work with. [00:54:46] And you know, when you step back like it, it makes sense, but it's so easy to get lost and the sauce of work and you're just like trying to get stuff out the door and it's like. You gotta like, find time to invest in the humans that you work [00:55:00] with. That [00:55:00] Matthew: The humans of [00:55:00] Phil: easier said than done. Exactly. Always bringing it back full circle. [00:55:06] That was super fun. Really appreciate your time. Uh, love everything that, that you shared. Uh, thank you so much for, for being so, uh, so open and, uh, yeah, we'll, uh, we'll be in touch. Enjoy your, uh, your holiday break and, uh, thanks again. Really appreciate it. [00:55:19] Matthew: Yeah. Great chat. Really appreciate it. And uh, yeah, thanks to you both have, you also have great holidays.