Retail Media Breakfast Club

What happens when four of the biggest retail media networks calculate ROAS in four completely different ways? And what does that mean for brands trying to compare performance, allocate budgets, and figure out which retail media investments are actually driving incremental sales?

In this episode, an expansion of my recent article for The Drum, I'm unpacking new research from Ovative Group that puts the ROAS calculations of Albertsons Media Collective, Home Depot's Orange Apron Media, Instacart, and Walmart Connect side by side. 

I'll explain why attribution methodologies can dramatically change reported results, why some brands are getting a misleading picture of performance, and what advertisers should be asking their retail media partners. Plus, I'm sharing some fascinating (and pretty pointed) reactions from the LinkedIn community, including why the industry might need to move beyond ROAS altogether.

This episode is sponsored by GrowthLoop

Timeline

[00:41] – Why I used to tell prospective clients I could get them any ROAS they wanted—and the catch behind that promise
[02:31] – Ovative Group's transparency guide: four retail media networks reveal how they calculate ROAS
[04:46] – Four networks, four attribution methodologies: last touch, equal credit, time decay, and duplicated attribution
[06:20] – The one question brands should be asking retail media networks about duplicated results
[09:31] – Why smaller brands rely on reported ROAS while larger advertisers can afford incrementality studies and media mix models
[12:17] – Why transparency matters, even when the industry can't agree on measurement standards
[15:15] – What's coming in part two: can retail media networks actually measure incrementality consistently?

Links & Resources

What is Retail Media Breakfast Club?

10 minutes of expert insights every weekday. Your morning ritual for staying ahead in retail media.

Four retail media networks, four ROAS calculations
===

[00:00:00] Kiri Masters: Morning, and welcome back to Retail Media Breakfast Club. I'm your host, Kiri Masters, and today I'm sharing a condensed version of a piece that I originally wrote for my column at The Drum on November 5th of this year. This [00:00:15] version, uh, is a little shorter, and it includes some comments from the LinkedIn discussion that it started, which is always my favorite way to come back, uh, to listeners of this show, is with a little bit of the reactions from the community [00:00:30] after I shared something initially.

[00:00:32] So excited to rehash this, and there is going to be a part two on this topic as well, which I'm gonna come back to at the end.

[00:00:40]

[00:00:41] Kiri Masters: So one of the questions that I heard the most often [00:00:45] from prospective clients when I was running my agency, Bobsled Marketing, from 2015 to 2022 was, "What kind of ROAS can you get us?"

[00:00:57] And, uh, in my more [00:01:00] petulant moments, I told them, "Any number you want." Because the system when it comes to ROAS is entirely gameable. We will simply bid on the long-tail, [00:01:15] high-intent keywords that always convert. If the assignment is to pump up the ROAS, we can absolutely deliver, but there's gonna be a catch.

[00:01:24] Your overall sales won't grow.

[00:01:28] Now, retailers' measurement choices [00:01:30] can move the number too. They can do things like count sales from the whole household instead of the shopper that saw the ad. They can stretch the attribution window out. The ROAS will go up, but nobody has actually sold an [00:01:45] extra unit. Earlier this year, I wrote a piece called Why ROAS Refuses to Die.

[00:01:52] Everyone agrees that this metric is flawed and limited, [00:02:00] but most brands keep on using it because it is fast, it's cheap, it's easy to explain to the boss. And I still think that there is room on the dashboard for [00:02:15] ROAS. and if ROAS is staying, the least the industry can do is explain what is inside it Earlier this month, the agency Ovative Group published its commerce media measurement [00:02:30] transparency guide.

[00:02:31] And what they did here was they asked seven retail media networks the same questions about how they calculate ROAS for on-site sponsored product ads and display ads

[00:02:43] Four of those seven [00:02:45] networks agreed to have their answers published side by side. Albertsons Media Collective, Orange Apron Media, which is the Home Depot, Instacart, and Walmart Connect

[00:02:56] So this is information that Ovative Group [00:03:00] collects that they use internally, that they use to normalize their own, uh, measurement results when they work with brands. And so instead of every brand asking the networks the same questions, Ovative put the answers all [00:03:15] in one place. Now, they did this through a collection of public information that the networks provide, as well as asking these networks to, you know, provide sort of on public record how they do things, and [00:03:30] all of these networks signed a release to have this information, aggregated in this report, even though a lot of this stuff is out there in the open publicly already

[00:03:40] so what did Ovative Group find? ~one thing that all four networks share is a 14-day default attribution window. Leah Logan, who is~

[00:03:43] ~Leah Logan, who is, um... ~One thing that all four [00:03:45] networks share is a 14-day default attribution window. In the LinkedIn comments when I first shared this article out on LinkedIn, I had a ton of responses. It was one of my best performing posts [00:04:00] of the month actually. Um, so I had a lot of interesting thoughts to wade through here.

[00:04:05] One of them was from Leah Logan, who is the head of growth strategy at the experiential sampling company Recess, and she questioned even the [00:04:15] 14-day attribution window. She said, "I find it interesting to have the same attribution window for purchases at a home improvement store as well as a grocery store.

[00:04:25] The trip frequency is vastly different, and buyer path for core items is very [00:04:30] different." Now, that's a good point, um, but it is, you know, one of these areas where retailers have, for whatever reason, decided to align on, one particular input. Where they [00:04:45] split is

[00:04:46] how a sale gets credited when a shopper sees more than one ad before buying. So run through what each of the networks do in that scenario where there is a multi-touch, uh, scenario happening. So Walmart Connect [00:05:00] gives full credit to a single campaign, the most recent touch, favoring clicks over views.

[00:05:06] So that's that classic last touch attribution model. Instacart splits the credit equally across the ads on the [00:05:15] path. Home Depot's Orange Apron Media uses time decay, giving more credit to the ads closest to the purchase. And Albertsons Media Collective uses last click or view with duplicated credit, so [00:05:30] one sale can count towards more than one campaign.

[00:05:34] Now,

[00:05:35] I spoke with Chelsea Lang, who is the director of retail media at Ovative Group, and she said that not a single retailer is using the same [00:05:45] approach to attribution methodology. And she sees that as the biggest driver of the gap in results because it decides whether a sale is counted once or more than once.

[00:05:57] So a brand who's running both display and [00:06:00] sponsored products on Albertsons could see one purchase that is credited to both of those campaigns. On Walmart, it goes to just the last one that the customer saw. So the same mix of ads could report very different ROAS [00:06:15] across each network, and neither of them would be wrong by its own rules.

[00:06:20] So when I asked Chelsea, "What's the biggest question that you think brands should be asking retail media networks?" She says, The [00:06:30] biggest question, if you could ask only one, would be, am I looking at duplicated results by channel?'

[00:06:37] Ovative works with both retailers and brand side media buyers, and so Chelsea [00:06:45] hears that comparison problem from both sides. The brands are asking whether a dollar at Walmart is the same as a dollar that, that they could spend at Instacart, for example.

[00:06:56] And if not, what math do I need to do [00:07:00] to make those numbers be, in her words, more apples to oranges than apples to pumpkins? Networks, on the other hand, the retailers hear the reverse, that advertisers telling them that they can't invest [00:07:15] more because your ROAS is half of what I see at another RMN, when those two numbers were never really built to be compared.

[00:07:23] So this is the sort of circular logic that's going around. The brands are blaming the retailers, the retailers are [00:07:30] blaming the brands, and, uh, where do we go from here? Are manual processes slowing down your RMN momentum? [00:07:45] With GrowthLoop's composable commerce media solution, retail media and ad teams can build and launch precise audiences in minutes, activate across DSPs and ad platforms, and [00:08:00] deliver transparent ROI reporting that drives advertiser revenue.

[00:08:06] learn how retail media leaders at Costco, Fanatics, and goPuff use GrowthLoop to deliver stronger results [00:08:15] for their brand partners fast. Visit go.growthloop.com/breakfast. That's [00:08:30] go.growthloop.com/breakfast

[00:08:30] Danny Satine, who is the head of retail media at the solution provider F11 and formerly group head of retail media at the UK home improvement retailer Kingfisher,

[00:08:39] pointed out in a comment on LinkedIn that that really costs the networks [00:08:45] something, that this lack of comparability makes it harder for the best-performing networks to prove that they're actually better

[00:08:54] And so that makes it hard for the brands because there could be some genuinely great ad [00:09:00] opportunities that they are under-investing in because they just don't look as good or performant as some of these metrics that place more emphasis on the last touch or [00:09:15] do something different And that makes a big difference because some of these brands might be under-investing in networks whose results don't look as good on paper as networks that are allocating credit to [00:09:30] multiple ad units.

[00:09:31] So Chelsea at Evative Group says that size of the brand really matters in terms of how they think about this and use this information. Larger [00:09:45] brands that have incrementality, access to incrementality studies or media mix models aren't making budget calls on attributed ROAS alone. They're the ones that can go out and afford third-party tools.

[00:09:58] These [00:10:00] models that are, you know, they're fairly expensive to run. They take a little bit of time. There's a lot riding on them. But there's thousands, if not hundreds of thousands of long-tail brands who don't have access to those larger studies, Chelsea says, and they [00:10:15] rely on the numbers that networks report Now I thought, you know, look, there's...

[00:10:19] I don't think there's anything shocking to anyone listening to this podcast about the findings here that, "Hey, surprise, not all these networks are [00:10:30] running the same numbers and showing the same, maths." But it-- I th- still commend what Evadeve Group did, which was getting these retailers on board and officially signing off [00:10:45] with releases to actually have their math compared side by side.

[00:10:51] So, um, I was interested in that, and I asked Chelsea, "What was it like actually getting involvement from the networks?" And she said, "Uh, actually it [00:11:00] wasn't that hard because most of them want to show their homework and show how things are done so that it can be transparent and clear." But, um, you know, obviously, as I mentioned, three out of seven of the [00:11:15] networks that they approached said that they couldn't participate in the guide.

[00:11:19] And one of those, obviously, is the big kahuna of, retail media ad spend, Amazon, and they, they couldn't get the releases, uh, approved. So, [00:11:30] uh, Chelsea still said, "Look, Amazon's methodology is very transparent. You can go and look up all of these things." It would probably be fairly easy to actually put this report into Claude or ChatGPT and then get the, the Amazon column [00:11:45] added to it.

[00:11:46] Um, or it can be a discussion point with your Amazon Ads sales team Some other limitations here, uh, it's for attributed ROAS on on-site placements only and [00:12:00] display placements. More channels and metrics are planned for later editions. And, um, Chelsea says, you know, everyone wants transparency, so it shouldn't really be that hard to recruit the next, [00:12:15] block of retailers here.

[00:12:17] So I think this, this is an interesting perspective because there's been so much talk about how difficult it is to get alignment. there's various groups working on this, including the IAB, the MRC, the ANA. [00:12:30] They've spent years trying to get networks to agree on how to measure various metrics. Um, and so what Evative did is they just said, "Let's not get everyone to agree on how to go forward.

[00:12:42] Let's just get them to say how they do their [00:12:45] measurement today." but even with that low bar, only four of those seven agreed. So I'm very supportive of this. I think that it was a great way to keep the conversation going, to get retailers to [00:13:00] volunteer and put their hand up and say, "We want to demonstrate that we are transparent, and we can do that by participating this, in this report."

[00:13:08] it's also a reminder of just how much is left on the advertiser's shoulders to [00:13:15] ask about this and dig into all of the math that goes into it, because you just can't take as a given that one retailer's ROAS is calculated the same way as the next one

[00:13:25] There was one final comment I wanted to share with you from LinkedIn, and this one is, [00:13:30] was from Rick Watrill, who is the founder of a company called Emlogix and the former chief data and analytics officer at Goodway Group. And he had a little rant on my LinkedIn post, which I thought was, [00:13:45] uh, very apt and, uh, interesting place to leave things for today.

[00:13:51] A little, you know, cheeky comment here. He says, "Let's combine the flaws of last click attribution with [00:14:00] different calcs to compare and sprinkle in biased methods. Then we ask advertisers and their agencies to understand and navigate all this across partners as if they have unlimited resources. And if none of that [00:14:15] works and they spin up their own measurement solutions that conflict or make us look bad, we'll partner with third parties to focus only on our silo and gaslight them into believing that it's not a fox-hen house [00:14:30] situation."

[00:14:31] Ouch. So that is, you know, the deeply cynical view of all this measurement discussion I think that a lot of people, um, silently share [00:14:45] Rick's, condescension here. Um, this is the state of where we're at. People don't believe that we're doing enough, to make things easier for advertisers. Retailers point the finger back at advertisers and say they're asking for too [00:15:00] much or that they benefit from these gray areas.

[00:15:04] Very murky situation, but hats off to Evadive for getting the conversation going, getting-- putting this out there. It was... I'm sure it wasn't easy getting all of these approvals in [00:15:15] place. and that's where I'm gonna leave it for today. I'm gonna give a little preview of what's coming up next on this topic, though, and that is that, um, a lot of the comments on my post came from readers saying, "We need to skip the comparison of ROAS and measure incrementality [00:15:30] instead.

[00:15:30] Just do incrementality." And so that sounds obvious, but, I'm curious about why more networks aren't reporting that way, and if we can truly do apples to apples comparison on incrementality. [00:15:45] So my, uh, theories there are either it's harder than it looks, or retailers don't have much reason to report on incrementality.

[00:15:56] So I'm gonna dig into that in part two. If you have a burning [00:16:00] POV on that, I'd love to hear from you. Reach out to me on LinkedIn or email me hello@retailmediabreakfastclub.com and I might be able to include your perspective in my next follow-up. Thank you so much for listening to Retail Media Breakfast Club.

[00:16:14] I'm Kiri [00:16:15] Masters, and I'll catch you tomorrow.

[00:16:16]