Retailgentic | Consumer Behavior & Retail Trend

This week is a research deep dive. ReFiBuy commissioned a behavioral study of how real people shop inside ChatGPT and Google's AI Mode, and Scot sat down with the researcher who ran it to walk through what was found. The full study is published today This is the conversation behind the numbers.

Highlights: 
  • 𝟴𝟰% 𝗼𝗳 𝗰𝗵𝗼𝗶𝗰𝗲𝘀 𝘀𝘁𝗮𝗿𝘁 𝗼𝗻 𝗮 𝗰𝗮𝗿𝗱: About 84% of product choices in the study came from a product card. That's 83.7%, and Scot rounds it up "if you allow."
  • 𝗣𝗼𝘀𝗶𝘁𝗶𝗼𝗻 𝘁𝗵𝗿𝗲𝗲 𝗯𝗲𝗮𝘁𝘀 𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻 𝘁𝘄𝗼: The first card wins by a wide margin, with roughly 43% of picks. But card three outperformed card two in both ChatGPT and AI Mode. Eric's take is that "it's not really a ranking." Scot's guess is curiosity, because card three is only partly visible in the carousel.
  • 𝗣𝗶𝗰𝘁𝘂𝗿𝗲𝘀 𝗯𝗲𝗳𝗼𝗿𝗲 𝗽𝗮𝗿𝗮𝗴𝗿𝗮𝗽𝗵𝘀: About 75% of shoppers skipped the AI's written answer and went straight to the product cards. They'd come back to the text later, but only as a second step.
  • 𝟵𝟴% 𝗼𝗳 𝗾𝘂𝗲𝗿𝗶𝗲𝘀 𝘁𝗿𝗶𝗴𝗴𝗲𝗿𝗲𝗱 𝗰𝗮𝗿𝗱𝘀: Purchase-intent prompts like "I want to buy X today" brought up product cards almost every time.
  • 𝗧𝗵𝗲 𝗳𝗶𝗿𝘀𝘁 𝗼𝗳𝗳𝗲𝗿 𝘄𝗶𝗻𝘀: When shoppers had to click out to a retailer, they leaned even harder toward the first offer than toward the first product card. The likeliest path is first card, first offer.
  • 𝗔𝗱𝘀 𝘁𝗵𝗮𝘁 𝗽𝗲𝗼𝗽𝗹𝗲 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗹𝗶𝗸𝗲: About 6 in 10 free-plan users saw ads or sponsored content on a task. 71% rated them at least as useful as Google search ads, and about 35% said an ad influenced what they considered buying.
  • 𝗧𝗵𝗲 𝗿𝗲𝘃𝗶𝗲𝘄𝘀 𝗽𝗮𝗿𝗮𝗱𝗼𝘅: Reviews were the main reason for a pick only 2% of the time. Afterward, about 75% said reviews mattered. Scot's explanation is that nobody reads them, but everyone takes the 4.8 over the 3.2.
  • 𝗧𝗿𝘂𝘀𝘁, 𝘂𝗻𝘁𝗶𝗹 𝗶𝘁 𝗯𝗿𝗲𝗮𝗸𝘀: Eric says the biggest surprise after two years of these studies is "the extent to which people trust, even when they say they don't." A mismatched price or a wrong spec breaks that trust quickly.
  • 𝗧𝗵𝗲 𝗸𝗲𝘆𝘄𝗼𝗿𝗱 𝗷𝗮𝗶𝗹 𝗲𝗿𝗮: Scot compares 20 years of chasing "men's running shoe" on Google with a world where shoppers let the AI choose for them.
  • 𝟰𝟬 𝗽𝗲𝗼𝗽𝗹𝗲, 𝗺𝗼𝗻𝘁𝗵𝘀 𝗼𝗳 𝘄𝗼𝗿𝗸: The study had 40 remote participants doing 30-minute think-aloud sessions on screen recordings. The team logged small behaviors down to seconds spent and scroll percentages.
Scot came out of the Amazon era believing the buy box was everything, and this study tells him ChatGPT has one too. Enjoy the conversation.

Timestamps:
01:57 Meet Eric Van Buskirk, Clickstream Solutions
02:15 Why ReFiBuy commissioned this study
02:48 Eric's background: SEMrush, Backlinko, Kevin Indig
04:05 Stated preference vs. observed behavior
06:05 How a behavioral study works
08:30 Why 40 people is enough
10:20 Who took part, and who got cut
11:55 The Amazon buy box and agentic commerce optimization
14:25 Finding #1: Pictures beat paragraphs
16:40 Card position: #1 wins, #3 beats #2
18:55 84% of choices come from a product card
20:35 Google SEO vs. AI: the "keyword jail" era
23:40 When do shoppers click out?
25:30 75% skip the text and go straight to the cards
27:25 Offer cards: the first offer dominates
30:20 Ads in ChatGPT: what shoppers really think
35:50 Trust and the reviews paradox
40:25 When bad data breaks trust
41:35 Where to get the study and work with Eric
44:55 Wrap-up

👉 Connect with Eric: https://www.linkedin.com/in/vanbuskirk/
👉 Learn more about Clickstream Solutions: https://clickstream.cc/

🧠 Want to stay ahead in AI commerce? Subscribe and follow along:
📰 Subscribe to the free Substack: retailgentic.ai
📺 Watch episodes on YouTube & Subscribe for updates: https://www.youtube.com/@Retailgentic

Creators and Guests

Host
Scot Wingo
CEO and Co-founder, ReFiBuy.ai

What is Retailgentic | Consumer Behavior & Retail Trend?

Retailgentic, presented by the ReFiBuy Media Network and hosted by Scot Wingo, is where Agentic Commerce meets retail innovation. We help retailers and brands prepare for the future of agent-driven shopping with actionable news, expert insights, dxis, forward-looking predictions, and practical guidance on Agentic Commerce Optimization (ACO).

Speaker 1:

The way people interact with cards was so fascinating. I've done a lot of these before, like I said, but they were never more about buyer behavior. They were more often like middle of the funnel or top of the funnel. And that's a very different thing to look at. Everything with their product cards especially was fascinating.

Speaker 1:

I started working with websites decades ago and I remember realizing the extent to which images and having text around images just immediately draws people's attention. And there's something very succinct, concise, and they love it, right? I mean, anywhere you put an image and have some text, it's gonna draw in users.

Speaker 2:

Welcome to the Retailgentic podcast, where we explore the intersection of retail, ecommerce, and AI agents, which we call Retailgentic. Our mission is simple. We are shining a bright light on the new Agentic AI shopping megatrend which will disrupt the trillion dollar digital retail market. The stakes are high. Google, OpenAI, Microsoft, Perplexity, Amazon, Apple, Meta, and Anthropic are all fighting to be the Consumer AI Shopping Agent winner.

Speaker 2:

There are gonna be some tough decisions for online brands and retailers to make. What happens to checkout? How should payments work? Could you continue to invest in retail media networks? What does the world look like when over half of your traffic is not from humans but from AI agents?

Speaker 2:

These existential questions are the tip of the iceberg. On this podcast, we're talking to industry experts, analysts, and influencers to get answers to these questions, identify trends, and best practices. This is Retailgentic, your podcast partner in navigating our AI shopping agent future. And now here's your host, Wingo.

Speaker 3:

Welcome to this episode of Retailgentic. I'm your host, Wingo. Today, I am joined by Erik Von Buskirk. He is the founder of Clickstream Solutions. They are an AI search and SEO consumer research firm.

Speaker 3:

Erik, welcome to the show.

Speaker 1:

Hey, great to be with you, Scot. Good to speak.

Speaker 3:

Yeah. So full disclosure, my company, RefiBuy, I'm a huge fan. I've seen your work published under, you've done for other folks. And we reached out because we were curious about how ChatGPT works. So we're going to get into that.

Speaker 3:

We're just publishing all that today. At the end, I'll tell folks where they can go to get this. But for now, Erik, tell us a little bit about your background. How did you get into the interesting world of understanding consumer behavior and how people do things online that they do?

Speaker 1:

Right, so I had been doing more on the data study side for a long time. I was at SEMrush as one of the first 15 employees eleven years ago. Then I did the largest ranking factor study to date about ten years ago with Brian Dean at Backlinko. And then from there, off and on, I did a lot more data studies. And then I was also doing client work for SEO with my company doing client work, and off and on doing research.

Speaker 1:

But then about two years ago, I started very heavily spending the vast majority of my time doing studies, which were primarily with Kevin Indi, who's probably one of the best known search optimizationSEO thought leaders in the space, doing these behavioral studies. The idea originally actually came from Brian Dean Abacklinko, because I'd scoped that out. He did one ages ago on how real users are behaving inside of Google. And so we decided, hey, let's do this now with AI mode and the new Google, the overviews too. Then since then, I've done like four or five of these primarily with Google.

Speaker 1:

This is only the second with ChatGPT.

Speaker 3:

Here on Retailgentic, we kind of read the tea leaves of what's going on with consumer behavior. And prior to working, talking to you, we had two buckets of data that I always talk about. We have stated preference, which is a bunch of survey data. And it's funny, I started this about eighteen months ago and a survey would come out every six months. So we were starved for surveys.

Speaker 3:

And now they come out literally, I can find 15 a week. Sometimes it's n equals 200, which is kind of useless. And then there's like n equals four or 5,000. The statisticians tell us the bigger the number, the better. And something like for The US audience, as you get into four and five thousand, it can become, you know, and if you read, there's always a margin of error, just like a poll kind of thing.

Speaker 3:

And then, so there's stated preference and then there's observed behavior. And observed behavior is typically through these things where they go and they get clickstream data. So they, through a variety of ways, they always say it's permission, but there's a lot of gray area in this ClickStream world. Panel data is another word for this, where they use browser extensions. Like if you, a famous one is there was this thing called the Honey browser, extension, and it was actually copying everything you did and selling it the data.

Speaker 3:

I don't know if it still does that or not. Very controversial. In the terms and conditions, you signed up for that, but most people don't read those. So there's a lot of, that's where the controversy comes from. And you have what I would consider a third bucket.

Speaker 3:

So explain to us, so, so you're, you're actually observing real humans do this. And I think we've all seen, at least in our world, you know, I've seen videos of these things where big companies like Apple and Google invite a bunch of consumers in. Or in movies, they're always there if you're looking at politics where they'll ask people how they feel about a candidate type thing. Yeah. You kind of replicate that in a smaller environment, but get a lot of the same statistics punch and observations from it.

Speaker 3:

Explain to us how these types of surveys work or this third bucket of observing how people act works.

Speaker 1:

I'm a bit surprised that people still haven't done more of these as public facing. And primarily this is done for internal research. So I think the main reason, quite frankly, is that it's not cheap. I mean, I have to be careful with my budget in order to do this, but it's a huge project. I mean, this thing that we did with you is months in the making.

Speaker 1:

And as we talk, people will maybe see how much is involved in this, but there's 40 people. That's quite a few people, and each of them spending thirty minutes. You're recruiting them. They're done remotely. So, yeah, this big budget internally, maybe it's worth $60,000 to spend to do something like this, but for something you're going to release and give away to the public, that can be a little harder than a data study to justify.

Speaker 1:

But they're recruited remotely and it's behavioral, and you have to be very careful with every single step of the way because you can't nudge people. You can't say, Oh, wait a minute. No, no, no. That's not quite right. Do it this way.

Speaker 1:

And that's been the biggest challenge for me is the way we do this tends to be a little... We need people to do quite a lot of things. And again, if there's no one in handhold, then we can't ask them too much. I mean, the one we did for you was one of the most complex because there was a lot of variables. We wanted to change things up as we went along.

Speaker 1:

And so, you know, we had some fails in the beginning and we had to just basically start over. We dropped one of the questions because we just felt like this was cognitive overload to have, you know, this many and this much. But all of this has been the same as what I've done with the others, which I don't know, maybe it's five other ones I've done in the last few years. And, that gives you some sense of what goes on. Key to understand is that people are speaking out loud and they are recording their desktop.

Speaker 1:

It's obviously anonymized completely. Even I don't know. I know demographics, but I don't know who they are. And it's a speak out loud protocol where we say, you know, when you found what you're looking for, please, you know, hover over it or click on it and tell us blah, blah, blah. Like, what did you find useful?

Speaker 1:

What is trustworthy? Things like that.

Speaker 3:

We've been trained that the higher the N, the better. And this is basically N equals 40. But it is... The problem with stated preference data is people say things and then don't do them. So for example, in e commerce, people have always said, if there was a green alternative, I would pay more.

Speaker 3:

And then you do that and no one buys it. Right. Because it's easy to say you're going to buy something, but at the moment of truth, you're like, well, yeah, just, yeah. So it's not, it's apples and oranges, but why does 40, you think, become a really good representation of how people behave with this?

Speaker 1:

My favorite way of describing all this is that it's directional, and you do have to look at a case by case basis. What are you trying to extrapolate to all young users and how they do shopping across in The United States? I mean, that's kind of a crazy thing to extrapolate from 40, but there's a lot of things that if there's a real strong pattern with 40 people, you start to say, Gosh, this is really being supported here, and we have a great idea that this may be the case for a larger group. And I actually like to think of these a lot as similar to focus groups. So for people familiar with focus groups, right, I mean, brands are extrapolating to a very large extent based on small in person groups.

Speaker 1:

As far as more, I mean, the more just gets too expensive, I think. I mean, the biggest one I did was maybe, I guess, 70, and it was completely overwhelming to do a study on that scale. Because we're annotating these microbehaviors. So little things people are doing, tiny things, including counting seconds and scroll percentages sometimes. Yeah, it's very precise.

Speaker 3:

Do you also measure where they are on the spectrum of their AI adoption, or you just kind of randomly pull a set of consumers out?

Speaker 1:

Depends on the study. For the one I just did for Refi Buy, it has a bias that's useful to know, but you have to have it, which is that they had to have at least a free or a go ChatGPT account. Otherwise, obviously, they couldn't perform the tasks, and that even limits the base because they have to have that. In the past, I really just tried to make it representational of a general population. It usually leans younger.

Speaker 1:

And that's probably because just like, to be honest, the people that want to do these kinds of studies, and they do it in part not just because they want to get paid, but because they find it interesting, especially what I'm doing, because I kind of... Once they get in, they see, Oh, this is cool. I get to actually learn about investigating all these different things in the tasks with an AI. They tend to be more educated, more thoughtful, more paying attention. And we have to actually exclude some people because they will start eating popcorn and you'll hear them crunching, crunching, or they'll say, Oh, honey, I think you better turn around the oven.

Speaker 1:

They're blasting the radio. We just can't really take that as someone that is doing more than trying to just get paid.

Speaker 3:

Interesting. Well, they just announced last Tuesday that they crossed 1,200,000,000 weekly active users. I think there was plenty of people to choose from out there in the world. So on our side, one of the reasons we decided to sponsor this and invest in it is, I come from the Amazon era. And in the Amazon era, there's this thing called a buy box.

Speaker 3:

When when you sell on Amazon, there's a buy box. And we had plenty of data very quickly at this company. I started ChannelAdvisor that if you owned the buy box, you got 90% of the sales. It became key thing when you saw on Amazon. If you don't own the buy box, you're basically lost the game.

Speaker 3:

And so then that, it's good to know that as a merchant because now you know what the game is, right? And you can kind of figure out, and it's a zero sum game. If you own the buy box, win. And if you don't, you actually lose the transaction. So it's very much a zero sum game.

Speaker 3:

But no one had done a similar study for ChatGPT. The idea was, we believe, you know, we call it Agentic Commerce Optimization because you've got your sentiment analysis where the answer engine will kind of say, hey, I'm looking for a skincare product. And they'll say, well, have you considered CeraVe, Aveeno, and so and so, and Vaseline or something? That's AI visibility. Do you get a textual mention by the engine?

Speaker 3:

Does it know your brand? It's kind of how to think about it. I kind of tie that to sentiment analysis. And where we live at Refi Buy is further down the funnel where we start at the bottom where we want your register to ring. And then we always know your catalog because it ends up being important for this.

Speaker 3:

Then from there, we want to make sure your product card shows up correctly and has the right offer. And then your offer card shows up correctly. So, so, you know, what we asked you to do is go on ChatGPT research this behavior around product card and the offer card. So let's dig into, did I state that right? Anything else that you want to add?

Speaker 1:

No, it's right. Guess the only thing that's important to say is that there are so many other things, well, not so many, but there are other things people can do inside that black box of ChatGPT or AI mode. And those shape how people use product cards and vendor offer cards. But that's the only other main thing I would say.

Speaker 3:

So let's walk through some of the results. As you get into it, what are some of the things? And we are releasing the full results of this, and you can go to refibuy.aistudy. We'll have some of the highlights on a sister publication of Retailgentic that will go out, a substack will go out around this. So what were some of the key findings here?

Speaker 1:

Sure. The way people interact with cards was so fascinating. I've done a lot of these before, like I said, but they were never more about buyer behavior. They were more often like middle of the funnel or top of the funnel, and that's a very different thing to look at. Or they were...

Speaker 1:

Some of them... It's really interesting to see how people interact with maps, so they were sometimes looking at people's interaction with maps. But everything with their product cards especially was fascinating. I started working with websites decades ago, I remember realizing the extent to which images and having text around images just immediately draws people's attention. And there's something very succinct, concise, and they love it, right?

Speaker 1:

I mean, anywhere you put an image and have some text, it's gonna draw in users. So indeed, that's what I found here, which was this was something very different than most of the other studies, which was to say, most of other studies, it's a reading exercise. So when people are just beginning to learn about a product, they wanna read the text. But it's always skim, of course, right? Like people skim a blog post.

Speaker 1:

Even Google search results, it's a little similar to that. But for this study, so interesting because the first thing they would look at would be the product cards. And then they might or might not, typically they would, also scroll through the text. And as was the case before, comparison grids, people love that. All of these things are sort of folks who are accustomed to knowing that it's a good way to compare and contrast, or that it's gonna be succinct and walls of text is not necessarily gonna be as succinct when you're trying to buy something.

Speaker 1:

So that was one of the main things that I found really interesting. And then probably the most interesting is we had to have a lot of team meetings about how to define things because, again, nobody else does this. And one of them is how do you define first card, second card, third card? And then there's the scroll because some are hidden. So, you know,

Speaker 3:

It's what carousel too. Yeah. What... Yeah, that's what mean.

Speaker 1:

Exactly. Yeah, the right left. So what the result was, was that the first position, because these scrolling horizontally or no scroll, was by far the most popular. But what kind of got me flummoxed was number two was less popular than number three. So, you know, I'm used to this world of, in the old days, of Google up and down, and it's a very clear ranking.

Speaker 1:

Well, what I realized after thinking and researching this more is just, and people that work with UI stuff would probably know this more, but it's not really a ranking, right? It's like position one is clearly you can see this is what the AI is promising you is the most important. But two and three, it's almost like just a layout issue. So it's not really a ranking, so to speak, which makes sense. And then also, what was completely fascinating was that the numbers of people that chose number one for AI mode as compared to ChatGPT was like extremely similar.

Speaker 1:

And, you know, this is a game where, okay, we say it's directional, but I just find it fascinating. And it also says to me that like, okay, it doesn't mean like my annotators and I had some bias or some mistake in how we're coding these things. Truly, it's one because it was one in AI mode and it was one in ChatGPT, but then two was not the second best. Three was.

Speaker 3:

So, yeah. There's also a little UI thing where, the card on three for most people isn't totally visible. So there may be a curiosity factor because you're kind of like, what's that? What's that one? And and it...

Speaker 3:

I think they do that so they can show people that it it scrolls horizontally, but it could create a curiosity factor that gives it a bump. I don't

Speaker 1:

For sure.

Speaker 3:

Yeah. Human behavior, you can't predict it. That's the, that's why you do your job. It gives you job Exactly.

Speaker 1:

Right. Yeah.

Speaker 3:

Yeah. So what does, so, so therefore kind of the punchline of this whole thing is that 83.7, I'm going to, I know you're, you're a scientist. I'm going round that up to 84% if you all out. Four percent of the product choice comes from a product card is basically what you discovered.

Speaker 1:

Yeah. Yeah. And it's also, of course, presented as being front and center by by both platforms. But but...

Speaker 3:

Yeah. Yeah. And by both platforms, you've previously done a study of AI mode and came up with similar type results of this one, but with a little bit of a different

Speaker 1:

I mean, this one, we did do AI mode. It's just that I think it ended up being like we targeted it as one third of the questions. I don't know if I ever did this, and it just does become much more complex. And it's part of why there's more complexity for users, because we have to tell them, Okay, this time, let's go to AI mode and do the task a similar way. But as far as comparing to the past, I mean, this was very unique because also the way people are prompting is going to trigger the product cards.

Speaker 1:

It was about 98% of the queries triggered a product card, whereas I don't know the numbers from before, but they were far lower because people weren't doing things like, I want to buy blah, blah, blah today. You know, that's just 98% of the time is going to trigger that. Whereas in the past, it's like, Okay, I want to travel to New York in three weeks or something. It's not going to show you that.

Speaker 3:

Yeah. Yeah. How about, you've done a lot of research of how traditional SEO works in Google. How is this different? It seems like the retailers we deal with are really struggling with this because they've had twenty years of Google era, right?

Speaker 3:

Where a lot of SEO, a lot of keyword optimization, the consumer was entering three or four keywords. If I nailed men's running shoe, that was the game, right? I got to get that keyword and that's the game. Explain how this behavior is different in today's world versus the old Google world. I call it the keyword jail era.

Speaker 1:

So I actually did another study. This one was entirely on my own, not for a client. And I did use data, and we were able to get Clickstream scroll data over time, capturing it in every second, and we did this on Google. It was hundreds of thousands of users that agreed and opted in to allow that. And it was fascinating.

Speaker 1:

Because yeah, it's true. This was during AIO. We already had AIO Overviews, but one of the things that's captured is, was an AIO Overview present versus was it not? And the behavior is just so remarkably different. So in a sense, I was able to see when it's not present, it's a little bit more like the old days versus today.

Speaker 1:

And we just saw the way in which the because people, when their mouse moves and it stays somewhere, that's a very direct signal that they're reading wherever it is. People tend to move their mouse where they're reading, and they just spent so much time on the top reading the AIO. And then it's quite interesting because what people do now, and this is our studies of regular Google show this too, is they compare... They really like especially to look at the top of the citation, right? I also ignored this too much, but that's why your meta descriptions are so important because, yeah, Google can overwrite them, but people do a lot of the same comparing and contrasting based on the title of what's appearing in your citation.

Speaker 1:

But at the end of the day, if there's an AI overview, they're not going to leave. So that's where everyone's up in arms because it's like Google's just hoovered up all the information in the world and it's not really sharing the, I don't know, the original writing and ownership should help the people that wrote it to get some audience. But instead, Google just hoovered that up.

Speaker 3:

So in the Google era, consumers would go, they would look at the Google, the old, this is before AI. They would look at all the Google results and they would like go on a journey of exploration. And they didn't really think of Google as like picking for them. Right. And now what we're seeing is they're basically saying, I trust the AI 85% of the time, I'm going to go with its recommendation, which is which is on our product card.

Speaker 1:

Does that? Yeah, exactly.

Speaker 3:

What you've what you've what we're seeing. Okay.

Speaker 1:

Yeah, exactly. Yes.

Speaker 3:

Yeah. And then So I'm curious. So that's like 84% of the time. What happens to other times? Do they just kind of keep scrolling or what's going on there?

Speaker 1:

So we had a really interesting thing happen, which is that... So one of the ones I did with Kevin, there was a big New York Times article that was fascinating, basically talking about the fact that websites are not getting clicks anymore and sort of being very direct and saying, This is not quite fair, and people are up in arms. But the reporter called Kevin Indig, who I did this with, and so they cited our study, which I think in that one, it was a little more clicking out because when people have to investigate more, they will click out of AI mode or AI overviews. But what was so fascinating was that the Google spokesman, after this was published, came back. And I guess she changed the article and said, well, we don't quite think that's representative.

Speaker 1:

And we think people leave Google a lot more. So they have this whole schtick going on of like, Oh, no, actually, we're sending all kinds of traffic to the open web. It's not much different than before. And people like myself and so many people like you and your audience are just like, Wait a minute. We can see clearly that you're not really sending the same amount of traffic as you used to.

Speaker 1:

But yeah, if people have to investigate, they leave more. If they have to buy and they can just click from a card to an external site to put it in the shopping cart, they'll do that. If there's a shopping cart that stays inside an AI platform in the future, if that happens more, I very much doubt they're gonna leave at all.

Speaker 3:

Yeah. Now, staying up at the research part of the phase Mhmm. When another stat that jumped out at me from the research is you have the little written part that the AI has, and then you have the product cards lower. It's much more engaging to your point about pictures. Jumped right into the product cards.

Speaker 3:

And I think we're saying 75% of the shopping kind of skipped a bunch of the words and just went right into the product card.

Speaker 1:

Yeah. Yeah. And again, that's different You're than what people are

Speaker 3:

watching them and you can kind of see they're speaking and they're like, Oh, look, you can kind of hear them not even read that? How do you know that's what's going on?

Speaker 1:

So we tell them in the instructions to speak out loud, to tell us what you're thinking about as you're looking at things. So there's that is one thing. And the other is, again, it's pretty remarkable the extent to which people use, no matter whether or not this study or not a study. Of course, it's why UX software works, but people really do move their mouse where they're reading. So even if they don't speak out loud, our annotators can understand that.

Speaker 1:

The way, and the other thing we do, which is just kind of helps double confirm things, is that I personally take all the videos and have them transcribed by an AI, and then I query for some of these things to help both sort of confirm what a real human is annotating and whatnot, but also to find patterns that a person wouldn't. So we can also tell that way, that people are talking about what they see in the cards and what are the patterns that they see. I don't really use those for results. It's more like, Oh, that's interesting. And then when we statistically analyze our results, I can kind of confirm that what we're finding makes sense based on what they spoke out loud.

Speaker 3:

Once they went to the product card, what happened as they went to the offer card? What did we learn?

Speaker 1:

So this is, I think, fairly important disclaimer, which is that these tasks that they were given were to find a product that you would buy in the next, I forget the exact time period, but you're looking to buy it today, right now. And that became tough at first because you don't want it to be artificial. You want it to be things people would naturally consider doing. So that's why we give them many, many chances, excuse me, many, many subcategories. So if it's like one of them was trousers, but we gave them, please choose which kind you want to choose.

Speaker 1:

That's a little more specific, but maybe shoes or sneakers. So because we're asking them to choose a product that you would buy today and the instruction is when you're finished and you decide, hover over the name, it meant that a lot of people didn't get as far as choosing the vendor because they didn't have to. On the other hand, for some of these questions, they were told, for your final choice, click out so that you go to the website where you would buy it. And I did that on purpose so that we could see where they were clicking out on these offer cards. And so there we saw an even higher bias towards number one.

Speaker 1:

Again, I don't want to overstate that too much, but yeah, because I feel like the users were their attention was mostly on picking a product and a little less so where to buy it. But yeah, they had to click out when in the cases when they had to click out, they would really often choose number one.

Speaker 3:

The first offer.

Speaker 1:

Yeah, even more so than with product cards. It became even more salient. Yeah.

Speaker 3:

Yeah. So I think the stat we have on the first product card let me pull this up. There's a lot of data here is 43%. So total pro a product card, there's not that many, there's like, you know, somewhere between three and eight was the 84% stat. But that first one, to your point, gets about half the clicks.

Speaker 3:

So, So the most likely scenario is first product card, first offer. Going be statistically where the bulk of the most traffic, and then it's going to go down from there. Maybe third product card, first offer, so on and so forth.

Speaker 1:

Yeah, that's right. I need to be careful with the percentages we're citing because we're both doing this live without pausing. You guys did such an incredible job with the layout, but it's a massive piece. It's like 60 pages to scroll through 45. So I'm not sure if all those percentages are exactly right right now.

Speaker 1:

But that's right. The number one is the really critical piece there. Yeah.

Speaker 3:

Yeah. Yeah. Got it. Okay. The other thing that we discovered we weren't really looking for was ads were popping up quite a bit because a lot of these folks were on the free plan.

Speaker 3:

What did we learn about the ads that folks saw? And if you haven't seen ads, most people listening probably on a paid plan. But on the free plan, there's a little bar and it's a pretty distinct, you know, this is another part of the results and it's lower for most people. And that's where the ads kind of cycle through. What did you learn about the ads on ChatGPT as we did this research?

Speaker 1:

Right. So this was sort of another reason why it limited I mean, wasn't an issue, actually, because where we recruit participants is a huge platform, so it wasn't a big issue, but it did limit who could participate because they actually could not have a pro plan, and we actually kicked people out when they started if they did. I think one person did end up doing it with pro plan, but... So that is a different kind of audience too because they either have free or they have Go, but they're not like hardcore ChatGPT users if they don't, I guess, if they don't have a pro. So let's see, I am actually looking at this survey now, and the main thing was the survey.

Speaker 1:

We did a lot of survey questions about the ads so that once someone said, Yes, I saw an ad, then we asked them a lot of follow ups. Roughly three out of four people said that they found them at least as useful as the ads that they're used to seeing in Google Search. So that's a positive experience that they have. Overall, I definitely think there was nothing really too negative about seeing ads in ChatGPT or AI mode. I think AI mode was just starting to show ads when we did this.

Speaker 1:

So we were getting responses there as well. Here's the percentages. So let's see, 61% reported seeing ads or sponsored content on a given task. Kind of interesting. It tells you something about how much ads or visibility for buying type queries.

Speaker 1:

71 said they were as useful as conventional Google. About half said they helped them at least a little. So that was interesting to me. I don't really have comparable data. I would have thought that people would have said, no, there's really nothing much helpful because they have such banner blindness, so to speak.

Speaker 1:

Let's see.

Speaker 3:

Well, ads are usually product cards too. So it looks kind of integrated. It's visually different than kind of a banner ad always was just like splattered in your face kind of thing.

Speaker 1:

Yeah. Although, I mean, now Google's kind of hiding the text ads so that people make mistakes and it may get them frustrated. But they didn't seem to have the frustration here about that. 35% said it influenced what they expanded or what they considered buying. So they really, you can see some ways in which they had a pretty meaningful impact on the way people did things.

Speaker 3:

Yeah. Yeah. Okay, so say those again, 70% saw them, 35% were influenced.

Speaker 1:

Okay. Two about said they saw ads or sponsored content. 71% rated the ads as at least as useful as conventional Google Search ads. About half said it helped them at least a little. About 35% said it influenced what they expanded or what they consider buying.

Speaker 1:

Half said the... Oh, this is interesting. Half said this distinction was very or somewhat clear, which means there's a lot of people that were a little bit confused by that.

Speaker 3:

Have to not, yeah.

Speaker 1:

Yeah, right. Have to not. But that's another one that I think deserves a caveat because, I mean, for some people, they saw that we were asking a lot of questionnaires about ads, and they might have thought, Gee, am I missing something here? But they should have seen them about half the time, so they still were a little bit uncertain sometimes. And there were times, too, when they thought, you know, a product card would, geez, is this an odd or not?

Speaker 1:

And it wasn't, right? But they just were thinking, is there some kind of bias here? Are they showing me something? Well, I don't know.

Speaker 3:

Yeah. Think the... So Sam Altman has kind of said the only way they would do ads is if it added to the experience, if it was incremental, right? And he always uses the Instagram ads as a you probably haven't studied this, but people generally feel that Instagram ads are kind of part of the content in the reels. So that's interesting.

Speaker 3:

You're basically kind of saying people see them as useful and that they helped. So, yeah.

Speaker 1:

Yeah. Yeah. Yeah. And

Speaker 3:

I think if we did the ones I look at are the Amazon ads. Do not find them useful. They're like getting in my way of getting to what I'm trying to get to.

Speaker 1:

Yeah.

Speaker 3:

So it seems like they're it's early days for their ad system. It did cross a billion dollars, but it sounds like they've put together something pretty good there that that's, you know, it's showing in a fair amount of spots and people are reacting to it pretty well. Yeah, yeah. Last couple of things. We talk a lot about trust and a lot of this indicates that people really trust the ChatGPT, right?

Speaker 3:

Because it's basically, if they're looking at the product cards, unlike the Google era where you would fan out and look at a million places, they're kind of saying, well, it's basically telling me these must be the ones that are the good ones. I will pick one of the product cards. And then they're going with the top offer. Did you get any other trust signals about how people feel about ChatGPT? Another thing ChatGPT is increasingly doing is showing review summaries around it.

Speaker 3:

Any interesting data from that?

Speaker 1:

Yeah, so that was really interesting, because that was one of the ones where it's sort of, you know, people don't always say what they do. Like you said at the beginning, that's why these studies are so interesting, because you can see things, we ask these questions as follow ups, and people tend to say they rely very, very heavily on reviews, but I think, again, everyone's always looking for a shorter way to do things and reviews, going through a lot of reviews, it can be a timely, time consuming process if you're trying to make a faster decision. By the way, the average task here was five minutes and they weren't complex products, and they were familiar to them. So I don't think people were really spending the time to look, but I think they probably felt like reading through the reviews is a much more time consuming thing. And yes, my goodness, do they trust what's being told to them is good or bad?

Speaker 1:

That's been the most astonishing thing to me after doing this almost two years now. Literally number one is the extent to which people trust, even when they say they don't. Gosh, your question though originally was what again? Other

Speaker 3:

Other reviews. Reviews. Trusting reviews.

Speaker 1:

Trusting reviews, yeah. What else? I I think I want to leave it in terms of trust with how much they just trust what's being fed to them by AI in so many ways. It's really incredible. And yes, some of these UI elements are important.

Speaker 1:

Anytime they can compare things side by side in grid for products, that's, again, something they love to do. So that's about what I'd say there.

Speaker 3:

One stat we teased out of the data, so we'll take credit for this one, is that reviews were the primary reason they picked a product in only 2% of the time. But in your after I think you guys did a you asked people afterwards. 75% of people said they were important or influenced the decision. And what they probably mean there is not many people read them, but I think, you know, like, like I do this, we did some DoorDash last night. And if I see a 3.2 versus a 4.8, I'm I'm going with the 4.8,

Speaker 1:

right? Yeah.

Speaker 3:

Because a 3.2 on a food thing, it's just like, woah, it's kind of like getting into the very dicey area.

Speaker 1:

Yes.

Speaker 3:

Okay.

Speaker 1:

Yeah, exactly.

Speaker 3:

Yeah. So we're real, this is kind of interesting. We had a theory kind of based on how people shop on Amazon and some of the data you've done on AI mode. And I think it basically proved the theory and it's good to have, it's always good to kind of, you know, it wasn't exact, like the behavior on the product cards was really interesting. I thought the first card would be a little bit higher, but it is what it is.

Speaker 3:

And this is why we do these things. But, you know, just kind of summarize what we learned is 84% of the buying decision of where people buy a product is from the product cards. So they're quite essential. And then number two, once they hit that, when they go to the offer card, they are picking the first item. So that does have much more of a concentration than the first product card.

Speaker 3:

And then we also learned that basically embedded in that, people are trusting the AI's recommendation. They're also skipping the text and going right into the product card for the most part.

Speaker 1:

Or they come back. They come back. So it's kind of secondary. So they do read It's of come back

Speaker 3:

later and read it. Yeah. Yeah. Yeah. But it's a secondary thing.

Speaker 1:

Yeah.

Speaker 3:

Yeah. And then the reviews kind of played a role and then ads are actually there and supporting the experience and not detracting from it.

Speaker 1:

Yeah. Yeah. And another thing that speaks to what I think a lot of people would be interested in is that it was interesting how much people were upset when they saw something that seemed inaccurate to them, or they saw it, like, let's say, I don't know, price was one thing in one place and another and different another place. So that, you know, even though it might seem trivial, it really gave them pause. So they would get really upset like, What?

Speaker 1:

Or they would say, Why is ChatGPT showing me this? Like, I specifically asked to have no pockets in my bag or something. And it was almost like, Geez, I'm handing over all this trust, but they would see something that didn't match up with perhaps from product specs or something that the vendor's providing or whatever, and then that would make them really startled and perhaps really lose some of that big trust they had.

Speaker 3:

Yeah. Yeah. And I think for a merchant, you know, obviously, we're big believers in you got to make sure your product is represented right on ChatGPT. You get a product card in there, the right image, the right price, and then on the offer that it's all dialed in. Awesome, Erik.

Speaker 3:

This is great. If people want to learn more, go to refibuy.aistudy. And then here at Retailgentic, we're going to publish a lot of content around this as well. Erik, if people want to learn more about your studies, where should they go?

Speaker 1:

Probably LinkedIn, I guess. It's just because it's a central place for that. So yeah, just search my name in LinkedIn, everything's I don't necessarily put everything on my website because these are done for clients, so I don't want to steal their fire.

Speaker 3:

So far, most of your clients have been vendors, but we were talking about how you're opening up a new shingle where if there were any retailers or brands that wanted to really dig into, let's say you were brand A and you're a CPG and I don't know, I'll pick dairy. Let's say you sell butter and dairy products, you want to see, you know, you're up against a competitor. Erik and team can, you know, help you understand across all these either, you know, could be all the different selling surfaces out there, how you're positioning versus others, and what consumers how they're reacting to what you're doing. So a lot of really interesting things there. We strongly we're very pleased with the results and how this came together.

Speaker 3:

And the rigor, the scientific rigor that you you kind of approach it almost like an academic, but one that understands industry enough that you're going to kind of we've worked with academics on this thing, they take it into like a really weird area that's not as, you know, doesn't, it starts to lose its applicability. Like they'll suddenly say, well, we got interested in synthetic humans. So we get rid of the

Speaker 1:

real ones. Wait a minute, know? Yeah.

Speaker 3:

So you have, you kind of have, you've done a, you do a really good job of kind of walking, you know, so you've got your Venn diagram is the math part of this, which is hard, the consumer behavior, but also kind of bringing it back to reality, which is like one of the legs that's usually missed on that. So we would highly recommend working with Erik, and really appreciate you working with us on this study.

Speaker 1:

Well, thanks so much. And when people get a chance to see this, I mean, I have to say the design, the layout that you all did, I mean, we did everyone on your team and myself included, spent a lot of time putting this together, but that layout is the best one I think of anyone I've worked with. When you No, do something so I don't. And people will see this because months went into this. And if you don't spend that attention and make it really long form, it's like, Oh, man, you're going to miss out on the nuances of all of it.

Speaker 1:

And then the last thing I would just say is that if people are interested in doing a private study, it is the direction I want to go in. It's super similar to doing competitive research with Ahrefs or SEMrush for shopping or something like that. But the difference is that this is inside a black box, so you can't really capture the friction data. So you're instead looking at very specific tasks that only are going to surface your brand and your top five competitors, and then what is people's reaction to you versus your competitors? So that's the idea in a nutshell.

Speaker 3:

Yeah, very true. Awesome. Erik, thanks for taking so much time to help us build the study. The results are amazing. And thanks for coming on the podcast.

Speaker 1:

All right, Scot. Thanks so much.

Speaker 2:

Follow, rate and share to stay up to date with Retailgentic, your guide to the future of AI shopping agents.