Retailgentic | Consumer Behavior & Retail Trend

In this episode of Retailgentic, Scot sits down with Luca to unpack a research paper that blends academic rigor with real-world implications:
“LLMs Reproduce Human Purchase Intent via Semantic Similarity licitation of Likert Ratings.”

The paper, co-authored with Colgate-Palmolive researchers, explores whether AI can accurately simulate human reactions to product concepts, enough to replace or accelerate traditional consumer panels, which are slow, expensive, and hard to scale.

This one goes deep, but in ways that any retail or AI leader should care about. Scot and Luca discuss:
  • Colgate’s Challenge: How to test product concepts faster and at scale.
  • Synthetic Consumers: AI models that react to products like human panels.
  • Accuracy Breakthrough: Reaching ~73–74% agreement with real consumers.
  • Fixing LLM Failure Modes: Why naive prompts don’t work, and what does.
  • Bayesian Reasoning: Adding uncertainty so AI stops being confidently wrong.
  • Smarter A/B Testing: Using AI to pre-screen ideas before running live experiments.
  • Digital Clones: Future consumers earning money by sharing preference data safely.
  • Simulated Populations: Matching real audiences for testing and predictions.
AI isn’t just helping brands write copy or generate images, it’s beginning to think like their customers. If synthetic consumers continue to evolve at this pace, product development, A/B testing, and personalization may look completely different in just a few years.

Timestamps:
03:00 — Luca’s background: Rocket Internet, HelloFresh, Lazada, Stitch Fix
08:00 — How PyMC Labs was founded & why Bayesian modeling matters
16:00 — Bayesian thinking explained in simple terms
19:00 — High-stakes decisions & why probabilistic reasoning matters
24:00 — Colgate’s challenge: testing product concepts at scale
26:00 — How synthetic consumer panels work
30:00 — Accuracy results: humans vs. AI (~73–74%)
32:00 — Why naive LLM prompting fails (“mode collapse”)
35:00 — How reasoning → scoring solves accuracy issues
38:00 — Example: synthetic consumers evaluating PyMC’s own website redesign
44:00 — How AI can pre-screen ideas for smarter A/B testing
48:00 — Where AI cannot replace causal testing
49:00 — Digital clones & monetizable consumer preferences
52:00 — Future benchmarks, new LLMs & evaluation methods

👉 Connect with Luca: https://www.linkedin.com/in/lfiaschi/
👉 Learn more about PyMC Labs: https://www.pymc-labs.com
👉 Check out the paper: https://arxiv.org/pdf/2510.08338

🧠 Want to stay ahead in AI commerce? Subscribe and follow along:
📰 Subscribe to the free Substack: retailgentic.ai 
📺 Watch episodes on YouTube & Subscribe for updates 
🎧 Listen wherever you get your podcasts: Spotify, Apple, etc: https://retailgentic.transistor.fm

#Retailgentic #AgenticCommerce #SyntheticConsumers #AIShopping #ProductTesting #CPGInnovation #GenerativeAI #BayesianModeling #Ecommerce #FutureOfRetail #DataScience #LLMs #AIResearch

Creators and Guests

Host
Scot Wingo
CEO and Co-founder, ReFiBuy.ai
LF
Guest
Luca Fiaschi

What is Retailgentic | Consumer Behavior & Retail Trend?

Retailgentic, presented by the ReFiBuy Media Network and hosted by Scot Wingo, is where Agentic Commerce meets retail innovation. We help retailers and brands prepare for the future of agent-driven shopping with actionable news, expert insights, dxis, forward-looking predictions, and practical guidance on Agentic Commerce Optimization (ACO).

You know, I was one of the customers of
PyMC Labs, uh, in, uh, uh, five to six

years ago when, when they really started.

And with EloFresh, we used them for one
of our projects because we were using the

open-source library PyMC internally in
our, like, uh, the data science teams.

And so I reached out to Thomas to help
me with the practical implementation

of a media mix model at that time.

And so after that, we became friends,
and I've always have been advising

him on how to grow the company.

And at the beginning of the
year, he asked me to join

full-time as business partner and

Welcome to the Retailgentic Podcast,
where we explore the intersection

of retail, e-commerce, and AI
agents, which we call retailgentic.

Our mission is simple.

We are shining a bright light
on the new agentic AI shopping

megatrend, which will disrupt the
trillion-dollar digital retail market.

The stakes are high.

Google, OpenAI, Microsoft,
Perplexity, Amazon, Apple, Meta, and

Anthropic are all fighting to be the
consumer AI shopping agent winner.

There are gonna be some tough decisions
for online brands and retailers to make.

What happens to checkouts?

How should payments work?

Should you continue to invest
in retail media networks?

What does the world look like when
over half of your traffic is not

from humans, but from AI agents?

These existential questions
are the tip of the iceberg.

On this podcast, we're talking to industry
experts, analysts, and influencers

to get answers to these questions,
identify trends and best practices.

This is Retailgentic, your podcast
partner in navigating our AI

shopping agent future, and now
here's your host, Scot Wingo.

Welcome to this episode of Retailgentic.

Some of our most popular shows are
where we take a research paper that's

centered around LLMs and/or agentic
commerce that's a bit out there, kind

of future thinking, because this is
research, so it's not in production

right now and not something we can use.

But it's fun to think about.

We boil it down, and then we apply
our digital retail lens and experience

to this concept and see if, if it's
interesting and what it means for the

future of where this is all going.

Uh, this is some of the ways that I, I
get pretty good at predictions, is trying

to, like, understand where, where academic
is, uh, the academic world, because

we're always a couple steps behind.

In this series of interviews, we've had
three guests kind of in this category.

We've had Luca Chi- uh, Chian, uh,
from UVA's Darden School of Business.

Uh, he talked about the
psychology of agentic shopping.

We had Amin from Columbia Custom AI, and
he talked about simulated buying agents.

And then we had Ronnie from
ChatGPT/OpenAI on how people use ChatGPT.

On today's show, I'm excited
to welcome Luca Fiaschi.

He is with a firm called PyMC
Labs, which is a very big in the

open source statistical world,
and he co-wrote a paper called…

Let me pull it up for you to see.

Uh, it is called

LLMs reproduce human purchase
intent via semantic similarity

elicitation of Likert ratings.

It's a mouthful.

You can see his name is up
there, uh, in the upper right.

He's one of, uh, looks like nine
different authors on this one.

You'll also notice that some of
the folks with attribution on here

are from Colgate-Palmolive, um,
which I thought was interesting.

Um, what caught my attention about this
paper, other than the Colgate part,

uh, was that it-- there's this f-very
futuristic idea here that thanks to

LLMs now and agents, um, you can create
simulated humans that act close enough

to real humans in how they think about
products and how to buy them and what they

like and don't like, that they could be…

If this, if this paper be-becomes true,
um, and if they prove their thesis,

which we'll talk about, did they or did
they not, um, then if they can, then

that means we can use these simulated
humans for things like potentially A/B

testing or product development in the
future, or maybe a bunch of other things.

Maybe you could show it some
different ads, and it could tell

you which ones it likes more.

Um, uh, Luca has a really good example,
uh, in the talk where they've used

it kind of beyond what's in the paper
that I thought was really interesting.

So I, I learned a lot from this one.

Um, I learned, uh, some new statistical
things and different ways, weaknesses

of LLMs, uh, and then how to counteract
those, uh, and also about how this

approach, uh, worked and in some
ways doesn't work for certain things

that I thought maybe it could.

So I hope you enjoy this as much as I do.

Again, this is kind of gonna be
like a two or three years out, we'll

see things like this happening.

Um, and, but I think it's interesting
to keep track of this 'cause this

world moves very quickly, and this
is a little glimpse into the future.

I hope you enjoy the
conversation as much as I did.

Luca Fiaschi, welcome to
the Retail Gentic Podcast.

Hello, Scott.

Thank you for inviting me.

Absolutely.

Um, you put out a paper, uh, and
that paper is one of my favorites.

It is called LLMs Reproduce Human
Purchase Intent, which I totally

understood, and then it gets a
little, little kind of crazy for

me, but we're gonna get into that.

Uh, via semantic similarity,
elicitation of Likert ratings.

Um, so this is well thumbed through,
so I, I really enjoyed the paper.

And, uh, when I reached out, I was
expecting an academic, but when I, uh,

looked at your LinkedIn, I saw you've
been, uh, in industry for quite a while.

You, you do have a PhD of course, but
you've, you're actually not an academic.

Uh, you, you know, I guess you're
both an academic and you actually are,

like, an operator, uh, you know, actual
real, uh, internet person doing stuff.

So excited.

And you have a lot of retail experience,
which was also exciting 'cause that's,

that's our whole topic here is tr-
figuring out the intersection of

retail and agentic commerce, and I,
I think, you know, you're sitting

right at the, the middle of that.

Um, so let's jump into your background.

So, uh, let's start at education,
uh, and then kinda go through

some of your, your work history.

Sure.

Thank you.

And, you know, I've always been
excited actually about combining the

cutting edge of research with, uh,
uh, applications in the industry.

It's actually what really motivated me.

And, you know, I started many years
back, um, more than 15 years ago,

uh, in Germany, where I got my PhD
at the University of Heidelberg.

At that time, uh, my PhD
was an industry, uh, PhD.

We were working with, uh, this
pharma company, German pharma company

called Bayer, in, uh, uh, basically
developing a new type of microscopes

for, um, like, high throughput screens.

And it, it was a PhD under Department
of Physics, uh, but it was all

computer science and machine learning.

And I had the good fortune to hook up on
this topic, which was, uh, uh, machine

learning really early on, uh, really
by meeting my doctoral father, uh, by

chance during, uh, an exchange program.

And, um, there I really picked up this
idea of, uh, really working on, on the

bleeding edge of what research can do
and transferring that really, really

quickly into, uh, industry applications.

And, you know, the academic
world completely wasn't for me,

so, like, when I got my PhD, I
decided to move for the industry.

I worked with a VC fund, uh, called
Rocket Internet, which, uh, used to be,

uh, one of the largest VC fund in Europe-

Yeah.

And that was these famous guys that
would take, uh, it was two brothers or

something, and they would take ideas
from the US and they would, like, spin

up companies and incubate them there.

So they, uh, they had many companies.

And, and weren't they involved in
Skype and just, like, all kinds of

big internet companies in Europe?

Yes.

So I do think that they were, um, either
investors or, like, they, they, uh,

participated in the building of Skype.

They started it and

spun it out.

Yeah.

Yeah.

So they were very famous because of, uh,
um, their company was acquired by eBay,

and then became eBay Europe at some point.

So, um, the Samwer Brothers, um,
uh, which, uh, I actually worked

quite closely with some of … We
are one of the tour with Oliver.

Um, they, they were, like, really pioneers
at that time, and they had this strategy

of, A, it's, uh, actually 30% about
the ideas and more 70% about execution.

So they would take these ideas, um, of the
very early commerce days, and they would

just, like, spin it up across the globe.

And they usually worked in pretty
difficult market where you have

to do a lot of customizations.

For example, this brought me to
work with them on a venture which

then became part of Alibaba.

Uh, the ventures was called Lazada,
which was in Southeast Asia.

And it's, it's … In, as an
idea, it's a clone of Amazon.

It's, uh, e-commerce.

And they, uh … It was exciting as
a very young, uh, you know, person

at that time like me, because they
would really literally tell you, "Hey,

here you have hundreds of millions
of dollars to play with," which

comes with specific money, and go and
conquer in like, uh, this new market.

And the internet penetration in those
markets is actually pretty high, like

in Southeast Asia, because these, uh,
countries, they, um, they start digital

first, and then, uh, you know, and it
was, like, figuring out things like how

do you do logistics and be super efficient
with forecasting is actually pretty

complicated, especially in these massive
countries where you have very complicated,

uh, logistics situations like shipping
to islands and so on and so forth.

So you get an idea that starts from
the US, and then you add a lot of

complexity and execution on top, and
that's why you actually needed to have

very good data science across different
domains, from operations to marketing

optimization and so on, to make sure
you succeed in, in those markets.

And then, uh, the, uh, Rocket Internet
gave me really the opportunity to work

in many different business models and
understand a lot of different problems

that the enterprise have today, like
from, uh, e-commerce retailers to food

delivery systems, uh, to, uh, things like
subscription systems and so on, so forth.

So it's just been really
exciting throughout my career

Yeah.

And you worked at HelloFresh and Stitch
Fix where there are two subscription

businesses, and I imagine it's really
interesting to think through the

lifetime value of customers and churn
modeling and, and I'm sure being a

data scientist that you were-- you
turned your attention to those things.

Is that a fair guess?

Uh, what I would-- what I'd say is that
interestingly enough, HelloFresh is,

uh, also a Rocket Internet company.

It was, uh, actually
created by Rocket Internet.

And, um, they, uh, that's why I say
like I worked for like almost 10

years with Rocket Internet ventures,
because it, it was actually, I

started with HelloFresh at very early
beginnings where it was 20 people team.

Um, and, uh, uh, you know, mothership
Rocket helping them develop, and then

the founder asked me to build their
data team in the US because the US

is the largest market of HelloFresh.

And so when I arrived,
it was just one person.

I left, it was 75 people with different
sub-teams handling things from analytics,

experimentation, customer, uh, like
understanding customer analytics is

part of one of the things we did.

Marketing analytics, to things
like recommendation engine,

forecasting, data infrastructure
and platform and so on and so forth.

So it was a, a very large scale set
up at that time, uh, and, uh, one

of the best in the industry, I would
say, with that type of industry

Cool.

So it sounds like you were kinda like
the go-to guy at Rocket Inter- Rocket

Internet when they had a startup that
was starting to get some momentum.

They're like, "Let's get Luca
in here to run some analysis and

build a team," to kinda like help
solidify that, that business.

Yeah, yeah.

My role was, uh, um, basically to
collect learnings and practices

across the enterprise and across
different batches of Rocket.

I helped the founders build their data
teams, uh, projects, mostly around growth.

How do you, um, allocate
budgets in marketings?

How do you, um…

We had our own ad tech infrastructure too,
so I was partly responsible to build some

of our ad tech infrastructure, our bidding
engines, our marketing recommendation

systems, and so on, so forth.

Going back to Heidelberg, uh, you don't
seem to have a German accent, so, uh,

where, where are you from before then?

I'm or- originally Italian,
from this area of Italy called

Tuscany, and it's a beautiful area.

I've lived there.

I, I, unfortunately, I lived eight year
in Germany, I never picked up German.

As Mark Twain said, "Life is too short."

We actually had another Luca
on the podcast from Italy that,

uh, is in the agentic world
as well, so it's interesting.

He's at the Darden University.

Luca, Luca, uh, Cha- Chaian.

I don't know if you know him or not.

I'll have to introduce you.

You'll have a Luca party.

Um, then, uh, and, and today, you know,
uh, for this paper, you're at a company,

and let's, let's see if I say this right.

Pi MC?

Is that how you say it?

PyMC

Labs.

Correct.

PyMC, PyMC Labs, yep, okay.

Yeah, so, so tell us about that.

You know, you're, you're one of the…

Let's see, what are you?

You're one of the founders, right?

No, I'm, I'm a partner.

Yeah.

Oh.

So the company was founded by- A

partner.

Okay, partners.

Yeah.

Yeah.

So, but the company was, uh, uh,
founded by, uh, my business partners,

Thomas Wiecki, about five years ago.

Um, so Thomas is very visionary person,
so his original idea is to build a company

around, uh, the open source library
that he developed that's called, uh,

PyMC, and, uh, basically find a place
for, uh, uh, academics or researcher,

uh, that work with him to actually
make money out of the work with it.

And, um, you know, I was one of
the customers of PyMC Labs, uh,

in, uh, five to six years ago
when, when they really started.

And with EloFresh, we used them for one
of our projects because we were using the

open source library PyMC internally in
our, like, uh, the data science teams.

And so I reached out to Thomas to help
me with the practical implementation

of a media mix model at that time.

And so after that, we became friends
and I've always been advising

him on how to grow the company.

And at the beginning of the year, he asked
me to join full-time as business partner

and, um, we have complementary skill set.

I'm more on the machine learning
engineering side of the house,

whereas, uh, like a great statistician.

And so he comes from the Bayesian
angle, I comes from the agents

generative AI world of the house,
and we have combined the two ideas

together in several different projects.

And what…

The way we work with clients is
a typical consulting approach.

So even the, um, um, paper that
we are gonna discuss later on came

from an engagement we had with one
of our clients, which is Colgate.

And, uh, you know, we developed these
systems with them, which marries some

ideas from Bayesian statistics with actual
generative AI eng- agents techniques

Yeah.

So to help listeners, um, yeah,
so, so tell us about what the

PyMC open source library does.

And, um, you know, I know you're gonna
talk about Bayesian, so, so when you

do that, like, let's, let's define,
you know, uh, for, for layman's terms,

what a, what a Bayesian, um, model is.

Yeah.

So Bay- Bayesian it's, uh, it's very
good exercise also for me because

I, I traditionally I'm not, I'm not
actually a Bayesian person myself.

I, I actually came across the topics.

Yeah.

I have a feeling you have a good way
to summarize it ver- If I asked a

Bayesian person, they probably couldn't
make it short, but I'm sure you can.

Yeah.

That's right.

That's exactly right.

So I would define Bayesian as a way of
thinking about probability of statistics

that comes from first principles.

So you start defining, uh, probability
distributions about your beliefs,

and then you define a model that
basically represent the world.

And out of this model, then you
can use to s- sample predictions.

Um, for example, this could be
a forecasting models about what

will happen in the future for,
uh, the revenue of your companies.

And then you can get, um…

You can encode belief about, uh, the
markets, about what competitors are

gonna do, and then you can get out
probability distributions on what, uh,

the world will look like, yeah, in the
future and how the companies are gonna

perform with confidence intervals.

And so it, it's something that it's a
little bit different than traditional

machine learning engineering type of
the house, a way, way of thinking.

As machine learning engineers, you
think always about the most accurate

model, and you don't care what the model
specification is really, as, uh, much as

it performs on a test set very accurately.

In a Bayesian world, you actually want
to think about the structure of the

model because you want something that's
interpretable, and you don't want the

model just to be accurate, but you want
the probability dis- uh, that it is

outputting to be, uh, as tight as possible
and to give you credible intervals

that are both, um, representative
of the coverage and tightness, which

means that they are very, um- It, it
represents your uncertainty about the

world in the future and adapt to it.

So

Got it.

Okay.

So when I see, uh, I see
all these surveys, right?

So there's these stated preference
surveys where people are like 80% of,

uh, you know, the survey respondents are
using AI for, for buying things online.

And if you look in the small
print, it'll say N equals 4,000.

So they've taken a…

They've taken that and then applied
a Bayesian framework to scale it

up to the population of the US.

Is that kind of like, is that
Bayesian or not Bayesian?

It, it depends.

So it's a way, it's a way, it's a way
of thinking, uh, it's a way of thinking

that allows you to generalize things
as, as statistics, as statistics.

You know, in … Th- there is
two world in the statistics.

There is the frequentist approach where I
have a sample set, and then I extrapolate

everything from a finite sample.

The Bayesian way of the world,
it's a little bit different.

So they … You think about an abstract
model of how the world will looks like,

and then your data is just evidence
that supports or falsifies the model.

And, and therefore, you use
the data to reconcile, uh, what

the world is gonna look like.

So i- in, in more like technical
terms, in the Baye- in the, um, like

traditional statistics, frequentist
one, you have P values, while in the

Bayesian world, you have confidence
intervals and probabilities, actually

credible intervals and probabilities
that allows you to define in a more

nuanced way what different alternatives
of the world will look like.

Now, that has a lot of implications from
the bus- from business terms, actually.

When I, when I try to explain in abs-
abstract, it sounds really weird, but

w- like in, uh, uh … It's, it's a
technique that's used very much when you

need to take a, a high stake decision.

For example, uh, some of
our clients are hedge funds.

They do derivatives pricings,
and they have billion dollars

decisions to make based on that.

So knowing the probability of something
to happen, uh, is extremely important

to them because they can value
the level of risk of specificity.

And so that's why we charge

Okay, cool.

And what's your role there at PyMC?

So I'm, I'm building the generative AI
vertical, and the idea is, uh, Bayesian

statistics is a bit of a Swiss Army knife
for certain set of business problems.

It's a technique that's very hard
actually to, um, use in practice.

So what we do is that, um, because
the, the language is complicated and

even the way that I try to explain
it, simplifying it today, you can

understand it's very complicated
a technique to actually explain.

While LLMs are universal translators,
and so what we can do is that we

can embed some of these Bayesian
models, uh, into LLMs to help them

make decisions in a principled way.

And, and these are, there's two benefits.

So LLMs are basically general, uh,
translators, so they can actually

help you build these complex Bayesian
models similar to the way they help you

code things in software engineering.

Uh, but with, uh, an understanding
of the causal structure of your data.

It's a principled way of
thinking about the world.

And the other things is that LLMs
can be, uh, often very, uh, confident

about, uh, things, but be very wrong.

And so Bayesian techniques allows
you to, um, have LLMs reasons about

uncertainty in a principled way.

Uh, so like, uh, correcting for some
of the biases that these LLMs have.

And within this world, um, you
know, uh, we also think about

ourself in a more abstract way as a
expert in making simulations, very

complex simulations, even while you
cannot explicitly build the model.

And so the papers we built is an example
of a simulations in a way that you

can do with AI, where you don't have
an explicit model of human behavior.

It would be too complex to
actually build something like this.

So you rely on LLM to learn from a,
uh, training data what the heuristics

of human decision-making are.

And, and that's the way you are
actually, um, physically you can do that.

You can build very-- these
very complex simulations

Okay, cool.

So, so you guys have clients
that come to you and say,

"I'd, I, I'd really like to…

You know, here's my business problem."

And that could be a hedge fund.

Colgate is obviously a brand.

And, and then it's your job to figure out
in this toolkit that your company offers,

which range from these Bayesian, you know,
statistical models that are very advanced.

You've brought in kind of LLMs and
AI as another tool in the toolbox

to kind of help, help figure out
the right way to approach this.

C- Correct.

So, and, uh, a, a lot of clients
come for, from, because they've

seen our open source tools.

We are very much, uh, you
know, contributor to different

open source libraries for
different types of applications.

And then they get curious about that.

They read our white papers, they
come on our website, and then we

start collaborations with them.

Yeah.

Red Hat is who where I
am, and that's kind of…

They, they, they took the
Linux distribution and added

an enterprise layer, uh, to
it, and that's where, you know.

And then they have a licensing model on
top of that, that, that kind of beefs it

up and makes it more enterprise ready.

And then they have-- But, like, the
bulk of what they do is also support

and consulting around, around.

So it's a, it's a way to monetize
open source that's pretty clever.

Yeah.

Um, so let's talk about the paper.

So what was the problem
that Colgate came to you…

Colgate-Palmolive, uh, came to
you and said, "Hey, we, we need

help figuring out this problem"?

Yeah.

So it's a very big problem
for them and, uh, it…

they invest tons of, uh, money every year.

So they're talking about, you know,
hundreds of millions of dollars,

which is product development.

And so the-- typically, the way that,
that it's done is that when they

think about the new concept for a new
product, before they go and produce

that product, um, they need to-- they,
they wanna evaluate whether it's gonna

resonate to a certain target audience.

There are a lot of different
considerations that goes into this

in terms of even, like, is-- can
we produce it at the right costs?

Um, what's the price
point for these products?

And so on and so forth.

And so these are very complex decisions
and workflows, and the idea was take

parts of these very complex decisions and
workflows and try to, um, automate them.

And one of the most time-consuming one
is really to do these consumer panels

where they give these products to, um,
to real humans and they, they ask, uh,

questions like, "Hey, would you buy this?

How unique you think it is?

What do you think when you--
about this new product compared

to competitive product?

At what price points would you buy it?"

So these type of questions, um,
they're typically, uh, conducted with

re-research of, like, 30 to 40 humans.

Um, assembling these
panels is time-consuming.

Uh, running these interviews is expensive.

It can cost you quite a bit
amount of money to work with

some of the consultancies.

And right now, like even where they're
using even a generative AI technique

to produce new product concepts,
it just doesn't scale anymore to

run all these type of interviews.

And so what Colgate wanted is a way to
screen new product ideas really, really

fast to prioritize them offline so that
they can focus only on the, I guess, value

one, and only those run with maybe humans
or just really bring it to productions.

And so that's what we did.

So we built basically a, a synthetic
consumer panel that is able to answer,

uh, questions about these product
concepts in a way that's very similar

to what a real human, uh, panel could
have answered about that product

Yeah.

Uh, let me-- I was gonna pull it up,
but I'll just show it on the screen.

Um, so this was an example from the
paper where you had a, a picture of a

product, um, and you know, the, this is
kind of the, the kind of thing they would

show people and say, "Here's a product.

What do you think about it?"

Um, and, uh, so for those that listen
to the audio, it's called Orofoam Mood

Infused Body Wash, and then it has,
like, a little description, a picture,

a product picture, and then four or
five bullets talking about the product.

Um, so the, their problem is that,
uh, they spend all this money, you

know, hundreds of thousands of dollars
getting, putting these things in

front of people, and I guess their
question is, "Can we do this with a,

you know, with a simulated human?"

Yeah.

Or they just said, they said to
you, "We need to figure out a

cheaper way to do this," and you
said, "Let's simulate humans."

Like how, how did that,
how did that kinda go?

Um, it-- That's a very good question.

So they, they're…

Colgate is actually a bit of the
forefront, uh, compared to other,

uh, companies in doing this.

Actually, they, um, had started
already pilots with, uh, other, um,

firms on developing this technology.

They also had an in- an internal
team working on something like this.

However, um, they, they had two problems.

One problem is that the results that they,
they were getting weren't that great,

and they weren't really sure, um, why
and what determines, what are the factors

that determines, like, uh, a good outcome.

Is it the type of LLMs?

Is it the way you prompt them?

Um, so there are different parameters in
these algorithms that you can actually,

uh, tweak to, to make it actually perform.

And the second thing is they weren't
sure is that how even do you measure

accuracy of something like this?

What does it mean accurate in itself?

And so when they came to us, they
came with the research questions

of, "Hey, can you help us figure
it out within this problem domain?

A technique that's actually good and
actually works so that even within our

enterprise, we can convince all the
different business partners to invest more

in this technology and then productize
it to become part of, uh, our workflows."

And, and that's the reason why we
came up with a, a research type of

engagement with them, which then
became an actual published paper.

Um, the paper is, uh, it's gonna
be submitted to the conference.

It's currently in archive
for everyone to check.

And we also open-sourced part of the
code base that we used to really make

sure that people can reproduce these
type of results, uh, by themselves.

If I was gonna try to run this
experiment, the baseline is humans, right?

So, so we're trying to do simul- si-
you know, simulations versus humans.

Is that kinda how you
set up the experiment?

You kinda said, "All right.

Here's how humans do on these
types of things, and, you know,

we're gonna, we're gonna try to
get as close to that as possible."

Um, yeah.

Is that, is that kind of
the setup of the experiment?

Yeah.

That's right.

So the question is how, um,
how do you measure accuracy?

And the idea is that Colgate has
pretty large datasets of past consumer

studies because they've been running
this business for many, many years.

And we use, uh, basically a dataset
which I believe, uh- I don't remember

the exact numbers, but over 8,000 unique
users, uh, and, uh, hundreds of products.

And, uh, basically the idea for us is
if we want the, um, LLMs to be accurate.

So accurate means that if they have
a new product, they would rank this

product, um, similarly to what human…

You would, would rank this
product so that you can actually

prioritize them on a scale.

This is our high priority
versus low priorities.

And, and then if you ask them to
rate these products on a scale from

one to five, the distribution of the
ratings of the LLMs look similar to the

distributions of, uh, the human panel.

And notice that there is a nuance here
that if you ask two panels of people, uh,

to rate these same products, and these
two panels are extracted exactly with

the same demographics and, uh, you know,
they are almost identical twins, they

would anyway rate the products not 100%
the same, but they will still correlate,

uh, like with a certain correlation fact.

Uh, but so even nuance, they
have an intrinsic variabilities.

And so in the way we had to evaluate this
was taking into account the variability

from our clients and the variability of
LLMs in providing the answers to make sure

that the results are aligned as possible.

And if we know that with, uh, these
historical data, we can reproduce

the results of humans, we have
confidence that, of course, this

can potentially generalize to new
products, uh, and to future situations.

You can never have 100%
confidence in science.

You can always falsify things, but that's
the way you actually do is you infer

with data that you have, and that gives
you confidence about future predictions.

Cool.

So that, that helps me
understand the setup.

Let's just go right to the punchline.

Did you get close to this or
you couldn't make it work?

Yes, we, we could, we could
prove that we could make it work.

Um, we actually proved that it, uh, works.

You know, we, uh, I do think the
main insight of the, uh, paper is

if you do the naive things, uh, of
prompting an LLM as a human, you get

okay-ish results but not good results.

Uh, and, uh, I think the insights of
the paper is how can you get something

that's better, and why the, if you
prompt an LLM naively to answer as

a human, it actually doesn't work.

And so that insights is what kind
of like help us, uh, and we discover

by using the paper, and we can get
into it a little bit if you like.

Is there a, uh…

So let's kind of talk
about, so humans are here.

Let's, let's say they're at 100.

Did you get halfway to human or
like where did you get on the scale?

And, and it's fine if you reject the
way I'm, I'm d- I'm framing this.

Yeah, yeah.

Yeah, and so humans in itself are not
at 100 because if you get two humans

partners that are actually identical,
they don't correlate with each other 100%.

They are something around
90%, uh, like correlations.

And-

Okay.

So the high water mark is 90%.

Okay.

The high water mark is 90%.

And, uh, um, the … I believe our
latest results published in the paper

is something like 73, 74%, which
is, uh, n- close to 90% of that 90%.

Is that enough where, uh, Colgate said
you solved this, or did they feel like

that it still wasn't close enough?

Yeah.

So I … It … The … To them,
uh, the, the most important, um,

outcome of this project was to get
the confidence that they needed to

convince, uh, their business stakeholders
and counterparts that that's a

technique that they can invest on to
automate their internal processes.

And, uh, their short answer is yes, they
continued investing into the platform,

um, which is proprietary to them that
they use to prioritizing products.

So that's based on this technique and,
uh, evolutions of these techniques.

We … And we are, you know,
still working with them.

They, they continue to develop this
algorithm and these ideas, uh, today.

So.

So you talked about naively asking
the LLMs, um, i- is not the way to go.

And why is that?

Like, the LLMs have been trained on
all of human knowledge, shouldn't

they just be basically people?

That's the … I'll play the
role of the naive, uh, the naive

person in this and explain to me
why, why that's not gonna work.

Yeah.

And so you, you get the, you get the
thing that's called mode collapse.

Uh, so basically if you ask the LLM,
um, like, "Give me a rating on this

product," and describe you the product,
and you tell the LLM, "Answer as a

person who lives in Brooklyn, uh, that's
a female between this and that age."

Or sorry, "Of this age, and she
makes around this bracket of income,"

and so on and so forth, they tend
to answer with the most One single

value, which is the most probable
value in, they've seen in the dataset.

And the reason why they do that is
that they haven't spent enough thinking

tokens into reason about what makes
that product a good fit for that person.

So one simple thing you can do is
actually to ask instead of asking, uh,

just the rating number to, um, give you
a reason for it, and you should ask for

the reason before you ask the number so
that the thinking token are spent upfront

to come up with, uh, uh, a good answer.

And the, uh, other thing that you can
do is to ask, uh, for just the reason

and then have a technique to remap
that reason onto the scale one to five

that gives you a soft wording, voting
score instead of out voting score.

Because the problem you
get is this mode collapse.

Otherwise, the response of
the LLMs will be very peaked.

They will always answer three or they
will always answer five, and you don't

get the variability that you would get
in a normal, you know, uh, distribution.

Can you, can you tell-- can
you give the LLM more context?

Like I've seen people kind of
say they'll, they'll take…

So I've seen retailers do this where
they'll know that they have, and I'll

just like, you know, they'll have like,
let's say they have a really simple

demographic, like one third soccer moms
and one third, you know, tech bros and

one third, uh, I don't know, whatever.

And they'll kind of say,
"You are a soccer mom.

You like these things and
you're very busy and, and…"

Will that influence the LLM enough or no?

It's still, it still will go home to
the most statistical probable thing.

It, it does, uh, but it will still
go home to the most probable things

for that type of subset of prompts.

And- So that's,

that's not gonna work.

Yeah.

Yes.

And, and the reason is, again,
it's about the thinking tokens.

So like LLMs, the output of
LLMs, it depends on two things.

Depends on your prompt that you give
them, but also depends on how much time

they spent reasoning about the answer.

And it's the same reason why reasoning
LLMs actually do work today, is because

they spend significant amount of time
considering different points of view on

the answer before they actually give it.

So if you just ask for one answer,
yes or no, it would probably give

not thinking about this enough.

And if you ask for to think about
the answer and give you an extended

one, then you get these variability.

So I understand why LLMs won't work.

How did you build a better synthetic,
uh, representation of how people react?

Like what, what was the
unlock, uh, to do that?

And does it involve Likerts?

Well, let, let me ask what a Likert is.

Like, that was new to me

So a L- Likert's, uh, s- scale is,
uh, a scale that's used in, uh,

um, basically consumer surveys.

And this simply means that to
the question you can answer

on a scale from one to five.

And, uh, instead of like, uh, um, giving
you like, uh, you know, uh, what's

supposed to scale from one to 10 or
it's giving you a yes or no answer.

So, uh, and, uh, the, um, and so
you, you have a distribution of…

If you ask the panel about
these questions, you have a

distribution of responses.

And so this distribution is
typically a sh- a certain shape.

And, um, the, the idea is that if you
get the distributions of responses

from a, uh, a synthetic consumer panel,
you want the, the two histograms to

look as, uh, similar as possible.

And you can measure
that in different ways.

Um, the way we choose to measure
this is to use, um, a, a scoring

which is called the KS divergence.

And the KS divergence is simply
a, a good score because it's

normalized between zero and one.

It tells you how much two
histograms are far apart.

And, uh, um, being normalized
between zero and one, it's

actually are easy to interpret.

So that's the way we choose
to, uh, to measure this.

And so the, the technique that we
developed, uh, instead of, uh, asking

you for one single token and answers,
again, ask, uh, the LLMs for, uh, an

entire sentence of what they like about
the products and what they don't like.

And if you do that, you can take that
answers, uh, and then remap them on a

score to one to five, uh, basically by
clustering them in the embedding space.

You can take a sentence and then
you can send it back to ChatGPT,

you can get an embedding of that
sentence, which means a vector.

And then you can look at the clusters
in the embedding space of these

answers called similar answers
that have the similar reasoning

will be, uh, close together.

And then you can, uh, aggregate those, uh,
answers instead of with the hard scoring,

you can use a soft scoring approach.

So maybe something that a nuanced
answer is something in between, give,

uh, a score of, uh, 50% to a four
and 30% to a five and 20% to a two.

It's not that you have a hard
score or you have just a five.

And that way of aggregating responses,
it's more representative of the nuanced

world that, uh, humans think about.

And so represent better the variability
of humans in terms of this KS divergence

Okay.

Let me just restate that
to make sure I got it.

The, um…

So if you get a full sentence or, or
more than, you know, a pretty good

explanation from the LLM, that gives
you enough sentiment in there that you

can map it better onto, you know, a…

You can, you can give it a better
mapping and create more divergence

of the responses that, that's
closer, 90% to the 90% of humans.

That, that's right.

Actually, th- you, um…

When we think that the similarity
between this technique and

sentiment analysis, I…

It's something that I haven't
thought about, but it's very, uh,

similar to sentiment analysis.

So you basically get the sentence from
the LLM, then you map that sentiment

into this score of one to five, which
is very similar to that technique there.

If you watch real humans and, you
know, you can imagine if you're just

seeing them, you know, write one or
five, it's not as good as them saying,

"Oh, I really love that color."

And you know.

So that's gonna be very, that's
a strong five versus maybe they

would have scored it, yeah.

Or, you know, it's okay.

Like that, that's negative
sentiment in there that you

only really get from a sentence.

Sometimes you only get it from in-
intonation of voice and body language.

That's probably where that
last 10% is, would be my guess.

Uh, just kinda like those little things
we get as being humans that, that are,

you know, not in the written language.

Um.

Yeah.

And, and then there's other advantage,
by the way, as you point out too, that

the hu- the LLMs or the humans, if
they answer with an extended answer,

they can tell you the reasons of why
they like and don't like the product.

Now, those reasons are valuable
because then you can, uh, um,

basically analyze them and use them
to improve the product Which is a buy,

so it's, you get it for free usually

Okay.

So,

so

you've-

And I, it's a funny story because,
sorry, this I have to tell you because

I got the results this morning.

It's actually really funny.

We are rebranding our website, and we are
choosing two different variants of the

colors of the website and the messaging
that we're gonna use on our website.

And so we did two things.

We did an internal survey within our team.

It's about 40-ish people.

And then we use our synthetic
consumer technology to evaluate

that, and the results are spot on.

And both the complaints that our team had
about some of the fonts of the new website

being too small or some of the messaging
not being great, uh, were exactly the same

as our synthetic consumers' equivalent.

And we also, in this case, instead of
using a generic panels, we prompted

the synthetic consumers to look like a
group of PhD students and professors,

which is more or less representative
of our, like, uh, component.

And so they gave a very good answer.

Does that freak you out?

Does it sometimes…

The statisticians are probably already
there, like, uh, but sometimes it does

freak me out that, like, we're not that
far away from LLMs at the end of the day.

Yeah.

It did, it did freak me out a little bit.

I wasn't expecting
survey so close, Randall.

Yeah.

That's the fun part of
this field we're in.

It's kinda like you, you…

I try not to think too much about it.

Again, r- it'd kinda be a bummer
Makes me wanna just go learn

to play guitar or something.

Do something very, very
human that an LLM can't do.

Um, so, so that was one of my questions.

So you've built this really interesting
framework, and, uh, did I hear

you say you've open sourced it?

Is it out on a GitHub somewhere?

Um, you, you get the code, uh,
for the, uh, for the reproducing

the results of the paper.

Of course, there is part of the things
we, we have to keep proprietary given

the, uh, collaboration with Colgate,
but they, they were really kind

to let us open source most of it.

Yeah.

But you guys now have this black box of
simulated humans, and you can have it…

You looked at a, a consumer product.

Now you can have it look at your website.

Um, and, uh, you know, presumably this
is pretty interesting because it is one

thing to go and, you know, hire a firm
to go do a $300,000 consumer study and,

and all the analysis and everything.

That's, that's the really high end
of this type of consumer research.

In my digital world, we're
frequently doing A/B test, right?

So we'll have a, you know, an audience
of 5% will get A and another five will

get B, and then we look at some metric
like conversion rate, and then we'll…

There'll, there'll be a, a hero and
a loser, and then the hero takes

over the last 90% or whatever it is.

Um, so now we can actually…

But in that example, you, you kind of
messed up the people that, the humans

that were the 5% that got the crappy,
the one, the one that didn't win, right?

The loser.

Um, so what's interesting is there could
be a world where you run it in a, in a…

Also, it takes time, right?

So this may be in the B2B context or even
consumer, sometimes it may take a week

or two to like figure out your A/B split.

So you guys do it, uh, you could do it
simulated instantaneously because you

can always scale up a simulation, right?

Uh, and then because it's j- you can di-
you can scale up a digital simulation.

Um, and then you also don't have the
problem of the testing always has

a, a set of real possible customers
that, that didn't see the best thing.

Um, so it also eliminates that, whatever
the cost is of those two things.

Um, you know, so that's,
that's pretty interesting.

Have you guys thought about
commercializing that or?

Yeah.

There, there are several companies
that approached us with the idea

of doing exactly that, and we're
working with a few of them.

We also have, um, uh, uh,
implementations that we may wanna

release, um, a- about these.

So we are considering releasing.

Uh, probably it's gonna happen
actually next week or so.

So the, the, the idea is, um, it's
fascinating- I do think that, um…

So there is two things.

Uh, there is a-- There are some caveats.

If you want to, um, AB tests will
ever remain the, the gold standard

for proving causality and the-- this
technique of simulations, the best thing

it can do, it can help you prioritize,
uh, um, like assets for AB testing and

the, uh, which is not a small thing.

So if you think about the difference
between the old world and the new

world, is that with generative AIs,
you can create thousands, millions of

assets, actually pretty, pretty quickly.

And so you can't AB test
possibly all of them.

Plus, as you said, you have
constraints, it's expensive.

So even just prioritize a small
subset of those, I think it's

already extremely interesting.

Now, can you, uh, reconstruct the
exact effect size in the simulations

that you would have when you run
the AB test, meaning the delta, the

incremental revenue or whatever?

Can it be estimated exactly?

That's probably not because, um,
we already know that inferring

causality from observational data,
uh, there is tons of theory that

tells you that's super hard to do.

But I do think that establishing these
correlations and prioritizing out of

a very large set of AB test, the top
ten percent or five percent is actually

probably possible to do, and then you
would have to run the AB test downstream.

So that's what we want to achieve.

Now, there are difficulties because the
question is, when you take an AB test

and you expose your audience online to
that AB test- The customer traffic that

you collect is not distributed at random.

It's not generally the
entire US populations.

So even how you instrument your offline
simulations to be representative

of, um, the audience that you
will see once you do the AB test

online is complicated in itself.

But actually here, we are thinking
about using Bayesian method and

techniques to up-sample and down-sample
specifically, uh, subsets of the synthetic

consumers to be as close as possible
to, uh, the, uh, online population.

So there's a calibration step that's
still we need to figure it out how

to make it happen, I understand.

Yeah.

You know how, um, so, so, so a lot
of the ad systems have a, uh, an

audience lookalike function where you
can upload your audience, and then

they have enough data that they can
create a, a, a map of, you know, that.

They must be doing Bayesian
stuff on their side for that.

But yeah, um, within this black
box piece, th- this simulation,

there could be a way where you like
have a map, you know, m- mimic my

audience functionality down the road.

That's, that's a very good…

That's a, that's a very interesting idea.

Yeah.

And the idea is that,
um, how do you do that?

We're still thinking about this, but how-

If you're Facebook, you, you
have all the data, right?

That's, that's how they do it.

Correct.

And in- indeed there, there are
startup that actually approached us.

They had a very clever way of
actually getting through this data.

So, um, we, we are not under NDA with
them, so I can, I can disclose this,

but basically they're thinking about
gamified systems where you, um, maybe

as a consumer, you link your personal
data, you get something reward out of

it, and then this personal data is used
to create a synthetic representation

of you that the brand can interface
with to get recommend- uh, to recommend

you personalized products and to ask
you questions about new products.

And so you will, in the future, I think
you will have a world where I can, uh,

get some money by sharing my personal data
and have, have a digital avatar of myself

interact with any brand in the world.

And the brand can ask me in a privacy
compliant, safe way, questions about my

preferences, my consumer preferences.

I get money out of that, and I
get great recommendations, which

is the best of both worlds.

Huh, that's crazy.

So you could create a digital clone
and have it go make money for you?

I like that.

Sign me up.

I'm, I'm, uh…

Sign me up to the beta.

Sounds like I could make money
while I sleep, which is always good.

Um, cool.

I wanna be…

I, I know you're busy.

Um, the…

All right, so we talked about this.

It sounds like you've got some
cool stuff coming up next week.

Anything else about the paper you
wanna share before we wrap up?

Um, I, I would say that, uh, y-
yeah, I mean, read, read the paper.

So we still, again, we got submitted
to, uh, one of the big conferences next.

Uh, check the code, reach out to us.

I mean, if you are interested in
these topics, uh, um, we are always

happy to understand new use cases
of s- some of these technologies and

keep developing these technologies.

So reach out and, uh, share your
ideas, thoughts, and we can probably

work on these problems together

Cool.

And if, uh, people want to find you
online, is it, um, do you have a place

you- so the website is, uh, pymc-labs.com,

and I'll, I'll put that in the show notes.

Um, so that's the company website.

Um, but you look like you're
pretty active on LinkedIn.

Is that a good place to find you?

Yeah,

yeah.

Feel, feel, feel, uh, uh, free to
reach out and DM me on LinkedIn.

I'll very responsive there.

Yep, yep.

And then, um, cool.

And that's, so when you post new
papers, they'll show up kind of on your

LinkedIn, and I think it was on, on your
website in a research area, if I recall.

You guys have a nice little library of
different research you've done there as

well.

Correct.

Uh, correct.

We have a pretty extensive
research blog, and we publish

continuously any content or material.

Uh, we have benchmarks as
well, so we, uh, we publish the

benchmark about LLMs and pricing.

So which LLMs are better at predicting
pricing, which LLMs are most strategic

about pricing competitions, you know,
like the Price Is Right and so on.

We have a little, uh, simulation
of the setup of, for The

Price Is Right, the TV show.

And we, um, as new models come up, we
make them compete into simulations to see

which one is, we make it a leaderboard.

So, so Price Is Right has,
like, six different games.

Are you just doing the closest without
going over the basic Price Is Right

or like the little, the Himalayan guy
or like wh- which, which one of the-

It's a, it's a good question.

Actually, I should, I should ask Alan
who has actually implemented this.

I, um-

It's probably closest without going over.

That- I, I

think it's- … that's what most
people- I think it's the basic one

that they have implemented so far.

But the, I, I mean, like I, I
thought it was a very fun exercise.

I think benchmark is generally
a big topics in, uh, uh, LLMs.

I do think if we want LLMs
to become better representing

humans, uh, then we also need
benchmarks for something like this.

And publishing this Price Is Right
was a way of helping researcher

of, uh, traditional models
developing this type of competition.

Yeah.

Very relevant to our world of, a lot
of, lot of people in retail are using,

thinking about using these for different
pricing, you know, systems of, you know,

how do you, how do you do dynamic pricing?

How do you think about,
uh, regional pricing?

You know, like a sweater in, uh, Los
Angeles may be worth less than one

in New York or something like that.

Interesting.

So, uh, so when a new model
comes out, like, y- you guys

run it through your little, your
battery of different things and…

Okay.

So Gemini 3.0

is rumored to be coming out this week, so
that's gonna be a big week for you guys.

You, you're, you ready?

Gonna, gonna run it through the paces?

Yeah, we're gonna do that.

We still need to run on GPT-5.1.

We, we're gonna publish the results soon.

So yeah, a lot of
exciting stuff coming up.

Yeah.

This is what's fun about this
LLM world, a lot of job security.

Everything's changing basically
every week, all the time.

So, awesome.

Luca, thanks for taking the time to come
and, uh, uh, answer my novice questions

about Likert's and, uh, your paper.

I really enjoyed it, and look
forward to my digital, uh,

clone making some extra money.

For sure.

Thank you very much,
Scott, for inviting me.

Have a good day.

Bye-bye.

Follow, rate, and share to stay up
to date with Retailgenic, your guide

to the future of AI shopping agents.