Subscribe
Share
Share
Embed
10 minutes of expert insights every weekday. Your morning ritual for staying ahead in retail media.
83% of retail media campaigns can be gamed on iROAS. Here’s how
===
[00:00:00] Kiri Masters: Hello and welcome back to Retail Media Breakfast Club. I'm Kiri Masters, and today I'm going to be reading to you an article that I published to my column at the drum on March 23rd about.
[00:00:12] Some new research that quantifies what [00:00:15] many of us in retail media suspected that methodological choices alone can swing incremental ROAS by a significant margin.
[00:00:25] Let's jump in.
[00:00:27]
[00:00:28] Kiri Masters: Liz Roche [00:00:30] has sat in rooms where two people from the same brand have completely different views on what their campaign metrics should look like. It's one reason that she thinks transparency matters more than standardization.[00:00:45]
[00:00:45] If the buyers themselves can't agree on what they want measured, the least a retailer can do is show it's working. Liz Roche is Vice President of Media and Measurement at Albertson's Media Collective, and [00:01:00] last week her team was among the authors of new research that shows exactly why that transparency is overdue across 42 real ad campaigns, the same media, the same audience, the same [00:01:15] creative, and the same spend.
[00:01:17] Produced incremental ROAS or I ROAS numbers that varied by an average of six and a half x and a median of two and a half X. That [00:01:30] is a huge swing in variance that depended solely on how the measurement was calculated. In 83% of those campaigns, the results could actually flip from positive to negative. Just based on [00:01:45] measurement methodology. This white paper titled I roas Demystified was a collaboration between Albertson's Media Collective EVA Group, and professors from Northwestern [00:02:00] University's Kellogg School of Management.
[00:02:02] It follows a paper from last year called ROAS Demystified, which unpacked the limitations of standard roas. This time the team turned their attention to I roas, the metric that the industry [00:02:15] has been championing as the more rigorous alternative, but they also found that it carries its own hidden variability.
[00:02:23] The paper isn't arguing that eye roll as is broken. It's arguing that the same label gets [00:02:30] slapped on meaningfully. Different calculations and that most brands don't know enough about what's happening under the hood to interpret results with confidence.
[00:02:40] We wanted to explore how different methodologies produce different results. We know [00:02:45] that there's a new retail media network cropping up every day and understanding what's working has become more and more challenging. Let's dig into what the research tested.
[00:02:55] The study took 42 onsite display campaigns across [00:03:00] Albertson's web properties and varied four methodological choices that any retail media network faces when calculating. I oas number one, how the test and control groups are filtered before matching. [00:03:15] Number two, which matching approach is used That could be clustering versus propensity score matching.
[00:03:22] Number three, what data features inform the match, for example, whether past brand sales are [00:03:30] included. And number four, how incremental revenue is calculated between the two groups. For example, observed sales versus a Bayesian time series model. Now, these choices when combined [00:03:45] and mixed produced 54 different ioas values per campaign without changing anything about how the campaign actually ran.
[00:03:55] Some specific findings stand out. Propensity score matching [00:04:00] produced roughly 12 times better match quality than clustering. but it also tended to produce lower IRO as estimates. Whether or not historical brand sales were included as a matching feature could also swing the eye roll as [00:04:15] from a dollar 23 to a negative 14 cents, and the two approaches to calculating incremental revenue diverged by an average of 90%.
[00:04:27] Derek Nelson, senior Director of Retail [00:04:30] Media consulting at Eva Group, said that customers are doing nothing different. Media is doing nothing different, just putting everything together differently. You end up with wildly different results.
[00:04:40] A friend of the show, Jordan Whitmer, managing director of Retail Media at Agency [00:04:45] Salt xc, sees this play out across his brand clients. He says, results still reflect how audiences are selected and campaigns are delivered.
[00:04:54] Especially when these systems are designed to show your ads to shoppers already [00:05:00] likely to buy. And that dynamic where measurement captures correlation with purchase intent rather than a genuine lift, is exactly what this research is trying to untangle. But don't throw the baby out with the bath water.
[00:05:14] The [00:05:15] natural question, giving this, finding that 83% of campaigns can flip from positive to negative is whether the industry should be using IO as as a headline, KPI at all. So this is what I [00:05:30] asked Liz and Derek, But they push back for some pretty good reasons. Derek's argument is that the methodologies are testing different things, new to skew, new to brand, new to category.
[00:05:43] Those three things, [00:05:45] for example, are pretty nuanced. He says, I ROAS is such a broad bucket term that it kind of catches a lot of things. The need to simplify has lost some of the nuance. That's overall a good thing, but it gets used [00:06:00] as shorthand, and his view is that it's on brands to ask the questions and decide what to do with the answers.
[00:06:08] Liz added that the responsibility doesn't sit with brands alone. She said, I think it's [00:06:15] also our responsibility to educate in this space. Not every one of those brands has a powerhouse data science team who can really make sense of all of this, and I think that's another reason why partnering with Kellogg and partnering with Ative on research [00:06:30] like this from her perspective, helps to make it conceptually more accessible.
[00:06:36] Liz framed Albert Albertsons as having a stewardship role here. She said, let's get to the actual truth of what's happening because everyone [00:06:45] wins. When we sell more units. Miracle Ads is the only retail media solution designed for both one P and three P Marketplace [00:07:00] brands. Why does that matter? Marketplace sellers demand a seamless advertiser experience that still offers full funnel ad formats, and retailers need a flexible solution that allows you to scale your media [00:07:15] business.
[00:07:15] Learn more@miracle.com. That's M-I-R-A-K l.com.
[00:07:23]
[00:07:24] Kiri Masters: The researcher's implications land differently depending on where you sit. For the top tier [00:07:30] CPGs with dedicated data science teams, this is confirmation of something that they've been wrestling with for a long time, and it could be a useful framework for structuring conversations with other retail media partners.
[00:07:43] Many of the largest brands [00:07:45] already. Rank and stack their retail media network partners using internal models. And Liz acknowledged that she's seen those models up close, and her focus is making sure Albertsons provides the transparency that those models [00:08:00] need to be properly calibrated. But a mid-market brand selling through Albertsons probably doesn't have a data scientist on staff, let alone one that can evaluate whether their IO as report used, propensity score matching [00:08:15] or clustering, or whether brand sales were included as a matching feature.
[00:08:19] As Liz noted, many of these brands receive a report and have no choice but to take that number at face value. To that end, this white paper's appendix. [00:08:30] Gives brands that don't have deep analytics capabilities, a starting point for asking the right questions. Questions about methodology. Who is included in the analysis?
[00:08:41] How is the control group built? So these are all [00:08:45] things that mid-market brands can use to inform their understanding. I've been reading Brene Brown's Atlas of the Heart Book recently, and a concept from the book came to me as I reflected on this conversation.
[00:08:59] [00:09:00] Brene Brown defines curiosity as recognizing a gap in your knowledge and becoming emotionally invested in closing it. She calls it a vulnerable act, a choice to embrace uncertainty over the safety of knowing. [00:09:15] The retail media industry has been operating in a comfort zone where familiar metrics like ROAS and I roas a treated as a settled number rather than a question to investigate.
[00:09:28] A lot of brands receive a report, they [00:09:30] see a positive figure and they move on. Retailers deliver results without always explaining the methodology. Underneath both sides are protecting themselves from the discomfort of admitting the numbers might not mean what they think or they might not [00:09:45] even really understand what they mean.
[00:09:47] Liz Roche described what genuine curiosity looks like in practice getting data scientists from both sides of the table to sit down and as she put it, seemingly unironically duke it out for about six [00:10:00] hours until they reach agreement on methodology.
[00:10:03] Now that requires vulnerability from the retailer whose methods are being scrutinized and from the brand, which might learn that their first conclusions were misplaced.
[00:10:12] Brene Brown also describes [00:10:15] curiosity as a tool that combats perfectionism and the need for self-protection in retail media, the self-protective instinct is strong. Retailers don't want to report lower numbers. Brands don't want to discover wasted [00:10:30] spend, and agencies don't want to question the metrics by which their performance is judged.
[00:10:35] I wrote about this dynamic in detail in a recent piece on why ROAS refuses to die. It is a collective action problem where everyone behaves [00:10:45] rationally. And the system stays broken. The antidote that this new research suggests is the willingness to sit in the gap for a bit.
[00:10:54] not to abandon eye roll as, but to ask what's behind it.
[00:10:58]