WEBVTT

00:00:00.088 --> 00:00:06.895
You know, I was one of the customers of
PyMC Labs, uh, in, uh, uh, five to six

00:00:06.895 --> 00:00:08.823
years ago when, when they really started.

00:00:09.494 --> 00:00:13.811
And with EloFresh, we used them for one
of our projects because we were using the

00:00:14.014 --> 00:00:19.472
open-source library PyMC internally in
our, like, uh, the data science teams.

00:00:20.038 --> 00:00:23.887
And so I reached out to Thomas to help
me with the practical implementation

00:00:23.887 --> 00:00:25.656
of a media mix model at that time.

00:00:26.211 --> 00:00:30.976
And so after that, we became friends,
and I've always have been advising

00:00:30.976 --> 00:00:33.581
him on how to grow the company.

00:00:33.983 --> 00:00:36.835
And at the beginning of the
year, he asked me to join

00:00:36.838 --> 00:00:38.485
full-time as business partner and

00:00:39.404 --> 00:00:43.633
Welcome to the Retailgentic Podcast,
where we explore the intersection

00:00:43.638 --> 00:00:49.360
of retail, e-commerce, and AI
agents, which we call retailgentic.

00:00:49.945 --> 00:00:51.104
Our mission is simple.

00:00:51.113 --> 00:00:54.263
We are shining a bright light
on the new agentic AI shopping

00:00:54.263 --> 00:00:58.644
megatrend, which will disrupt the
trillion-dollar digital retail market.

00:00:58.970 --> 00:01:00.365
The stakes are high.

00:01:00.663 --> 00:01:08.415
Google, OpenAI, Microsoft,
Perplexity, Amazon, Apple, Meta, and

00:01:08.415 --> 00:01:12.437
Anthropic are all fighting to be the
consumer AI shopping agent winner.

00:01:12.761 --> 00:01:15.860
There are gonna be some tough decisions
for online brands and retailers to make.

00:01:16.548 --> 00:01:18.034
What happens to checkouts?

00:01:18.523 --> 00:01:19.899
How should payments work?

00:01:20.615 --> 00:01:23.137
Should you continue to invest
in retail media networks?

00:01:23.750 --> 00:01:26.654
What does the world look like when
over half of your traffic is not

00:01:26.655 --> 00:01:29.206
from humans, but from AI agents?

00:01:30.274 --> 00:01:32.870
These existential questions
are the tip of the iceberg.

00:01:33.218 --> 00:01:37.303
On this podcast, we're talking to industry
experts, analysts, and influencers

00:01:37.629 --> 00:01:42.109
to get answers to these questions,
identify trends and best practices.

00:01:42.929 --> 00:01:48.128
This is Retailgentic, your podcast
partner in navigating our AI

00:01:48.139 --> 00:01:54.355
shopping agent future, and now
here's your host, Scot Wingo.

00:01:54.696 --> 00:01:57.007
Welcome to this episode of Retailgentic.

00:01:57.469 --> 00:02:01.153
Some of our most popular shows are
where we take a research paper that's

00:02:01.153 --> 00:02:06.251
centered around LLMs and/or agentic
commerce that's a bit out there, kind

00:02:06.251 --> 00:02:09.848
of future thinking, because this is
research, so it's not in production

00:02:09.848 --> 00:02:11.550
right now and not something we can use.

00:02:11.573 --> 00:02:12.677
But it's fun to think about.

00:02:13.087 --> 00:02:18.186
We boil it down, and then we apply
our digital retail lens and experience

00:02:18.250 --> 00:02:22.929
to this concept and see if, if it's
interesting and what it means for the

00:02:22.929 --> 00:02:24.735
future of where this is all going.

00:02:25.421 --> 00:02:28.959
Uh, this is some of the ways that I, I
get pretty good at predictions, is trying

00:02:28.959 --> 00:02:35.309
to, like, understand where, where academic
is, uh, the academic world, because

00:02:35.326 --> 00:02:37.012
we're always a couple steps behind.

00:02:38.050 --> 00:02:42.031
In this series of interviews, we've had
three guests kind of in this category.

00:02:42.033 --> 00:02:46.991
We've had Luca Chi- uh, Chian, uh,
from UVA's Darden School of Business.

00:02:47.278 --> 00:02:50.384
Uh, he talked about the
psychology of agentic shopping.

00:02:50.738 --> 00:02:57.113
We had Amin from Columbia Custom AI, and
he talked about simulated buying agents.

00:02:57.518 --> 00:03:03.328
And then we had Ronnie from
ChatGPT/OpenAI on how people use ChatGPT.

00:03:04.928 --> 00:03:08.500
On today's show, I'm excited
to welcome Luca Fiaschi.

00:03:08.690 --> 00:03:13.299
He is with a firm called PyMC
Labs, which is a very big in the

00:03:13.305 --> 00:03:17.571
open source statistical world,
and he co-wrote a paper called…

00:03:17.901 --> 00:03:19.260
Let me pull it up for you to see.

00:03:19.804 --> 00:03:21.470
Uh, it is called

00:03:26.086 --> 00:03:30.277
LLMs reproduce human purchase
intent via semantic similarity

00:03:30.425 --> 00:03:32.429
elicitation of Likert ratings.

00:03:32.539 --> 00:03:33.228
It's a mouthful.

00:03:33.493 --> 00:03:36.130
You can see his name is up
there, uh, in the upper right.

00:03:36.136 --> 00:03:39.487
He's one of, uh, looks like nine
different authors on this one.

00:03:39.868 --> 00:03:42.815
You'll also notice that some of
the folks with attribution on here

00:03:42.818 --> 00:03:46.054
are from Colgate-Palmolive, um,
which I thought was interesting.

00:03:47.303 --> 00:03:50.711
Um, what caught my attention about this
paper, other than the Colgate part,

00:03:50.752 --> 00:03:56.794
uh, was that it-- there's this f-very
futuristic idea here that thanks to

00:03:56.835 --> 00:04:03.277
LLMs now and agents, um, you can create
simulated humans that act close enough

00:04:03.281 --> 00:04:07.352
to real humans in how they think about
products and how to buy them and what they

00:04:07.353 --> 00:04:10.541
like and don't like, that they could be…

00:04:10.557 --> 00:04:15.044
If this, if this paper be-becomes true,
um, and if they prove their thesis,

00:04:15.056 --> 00:04:19.066
which we'll talk about, did they or did
they not, um, then if they can, then

00:04:19.067 --> 00:04:23.669
that means we can use these simulated
humans for things like potentially A/B

00:04:23.669 --> 00:04:28.732
testing or product development in the
future, or maybe a bunch of other things.

00:04:28.732 --> 00:04:30.887
Maybe you could show it some
different ads, and it could tell

00:04:30.887 --> 00:04:32.405
you which ones it likes more.

00:04:32.475 --> 00:04:37.240
Um, uh, Luca has a really good example,
uh, in the talk where they've used

00:04:37.240 --> 00:04:40.155
it kind of beyond what's in the paper
that I thought was really interesting.

00:04:40.934 --> 00:04:43.064
So I, I learned a lot from this one.

00:04:43.085 --> 00:04:47.797
Um, I learned, uh, some new statistical
things and different ways, weaknesses

00:04:47.797 --> 00:04:51.926
of LLMs, uh, and then how to counteract
those, uh, and also about how this

00:04:51.927 --> 00:04:55.352
approach, uh, worked and in some
ways doesn't work for certain things

00:04:55.353 --> 00:04:56.424
that I thought maybe it could.

00:04:56.822 --> 00:04:58.635
So I hope you enjoy this as much as I do.

00:04:58.641 --> 00:05:01.313
Again, this is kind of gonna be
like a two or three years out, we'll

00:05:01.313 --> 00:05:02.601
see things like this happening.

00:05:02.924 --> 00:05:06.034
Um, and, but I think it's interesting
to keep track of this 'cause this

00:05:06.037 --> 00:05:09.433
world moves very quickly, and this
is a little glimpse into the future.

00:05:10.128 --> 00:05:12.620
I hope you enjoy the
conversation as much as I did.

00:05:13.679 --> 00:05:17.777
Luca Fiaschi, welcome to
the Retail Gentic Podcast.

00:05:18.071 --> 00:05:18.808
Hello, Scott.

00:05:19.415 --> 00:05:20.316
Thank you for inviting me.

00:05:21.398 --> 00:05:21.996
Absolutely.

00:05:22.034 --> 00:05:27.075
Um, you put out a paper, uh, and
that paper is one of my favorites.

00:05:27.587 --> 00:05:31.671
It is called LLMs Reproduce Human
Purchase Intent, which I totally

00:05:31.672 --> 00:05:34.434
understood, and then it gets a
little, little kind of crazy for

00:05:34.435 --> 00:05:35.569
me, but we're gonna get into that.

00:05:35.878 --> 00:05:39.487
Uh, via semantic similarity,
elicitation of Likert ratings.

00:05:39.799 --> 00:05:43.816
Um, so this is well thumbed through,
so I, I really enjoyed the paper.

00:05:43.882 --> 00:05:48.242
And, uh, when I reached out, I was
expecting an academic, but when I, uh,

00:05:48.263 --> 00:05:53.205
looked at your LinkedIn, I saw you've
been, uh, in industry for quite a while.

00:05:53.206 --> 00:05:57.653
You, you do have a PhD of course, but
you've, you're actually not an academic.

00:05:57.688 --> 00:06:01.516
Uh, you, you know, I guess you're
both an academic and you actually are,

00:06:01.530 --> 00:06:07.429
like, an operator, uh, you know, actual
real, uh, internet person doing stuff.

00:06:07.429 --> 00:06:08.239
So excited.

00:06:08.249 --> 00:06:10.985
And you have a lot of retail experience,
which was also exciting 'cause that's,

00:06:11.244 --> 00:06:14.448
that's our whole topic here is tr-
figuring out the intersection of

00:06:14.448 --> 00:06:17.718
retail and agentic commerce, and I,
I think, you know, you're sitting

00:06:17.719 --> 00:06:19.046
right at the, the middle of that.

00:06:20.087 --> 00:06:22.560
Um, so let's jump into your background.

00:06:22.567 --> 00:06:26.970
So, uh, let's start at education,
uh, and then kinda go through

00:06:26.972 --> 00:06:28.279
some of your, your work history.

00:06:28.815 --> 00:06:29.137
Sure.

00:06:29.186 --> 00:06:29.711
Thank you.

00:06:30.047 --> 00:06:34.261
And, you know, I've always been
excited actually about combining the

00:06:34.263 --> 00:06:37.979
cutting edge of research with, uh,
uh, applications in the industry.

00:06:37.979 --> 00:06:39.835
It's actually what really motivated me.

00:06:40.391 --> 00:06:45.676
And, you know, I started many years
back, um, more than 15 years ago,

00:06:46.334 --> 00:06:50.431
uh, in Germany, where I got my PhD
at the University of Heidelberg.

00:06:50.965 --> 00:06:54.427
At that time, uh, my PhD
was an industry, uh, PhD.

00:06:54.459 --> 00:06:57.775
We were working with, uh, this
pharma company, German pharma company

00:06:57.776 --> 00:07:02.721
called Bayer, in, uh, uh, basically
developing a new type of microscopes

00:07:02.859 --> 00:07:06.436
for, um, like, high throughput screens.

00:07:06.486 --> 00:07:11.867
And it, it was a PhD under Department
of Physics, uh, but it was all

00:07:11.884 --> 00:07:13.474
computer science and machine learning.

00:07:13.474 --> 00:07:18.701
And I had the good fortune to hook up on
this topic, which was, uh, uh, machine

00:07:18.705 --> 00:07:24.956
learning really early on, uh, really
by meeting my doctoral father, uh, by

00:07:24.962 --> 00:07:27.194
chance during, uh, an exchange program.

00:07:27.914 --> 00:07:34.017
And, um, there I really picked up this
idea of, uh, really working on, on the

00:07:34.017 --> 00:07:38.907
bleeding edge of what research can do
and transferring that really, really

00:07:38.907 --> 00:07:41.264
quickly into, uh, industry applications.

00:07:41.954 --> 00:07:45.177
And, you know, the academic
world completely wasn't for me,

00:07:45.177 --> 00:07:49.784
so, like, when I got my PhD, I
decided to move for the industry.

00:07:50.179 --> 00:07:54.967
I worked with a VC fund, uh, called
Rocket Internet, which, uh, used to be,

00:07:54.967 --> 00:07:57.239
uh, one of the largest VC fund in Europe-

00:07:58.417 --> 00:07:58.706
Yeah.

00:07:58.778 --> 00:08:01.938
And that was these famous guys that
would take, uh, it was two brothers or

00:08:01.940 --> 00:08:05.493
something, and they would take ideas
from the US and they would, like, spin

00:08:05.494 --> 00:08:07.423
up companies and incubate them there.

00:08:07.437 --> 00:08:09.677
So they, uh, they had many companies.

00:08:09.677 --> 00:08:12.123
And, and weren't they involved in
Skype and just, like, all kinds of

00:08:12.124 --> 00:08:13.242
big internet companies in Europe?

00:08:13.410 --> 00:08:14.121
Yes.

00:08:14.131 --> 00:08:19.771
So I do think that they were, um, either
investors or, like, they, they, uh,

00:08:19.832 --> 00:08:21.254
participated in the building of Skype.

00:08:21.254 --> 00:08:21.294
They started it and

00:08:21.414 --> 00:08:22.113
spun it out.

00:08:22.191 --> 00:08:22.491
Yeah.

00:08:22.626 --> 00:08:22.974
Yeah.

00:08:22.974 --> 00:08:27.891
So they were very famous because of, uh,
um, their company was acquired by eBay,

00:08:27.910 --> 00:08:30.358
and then became eBay Europe at some point.

00:08:30.859 --> 00:08:35.775
So, um, the Samwer Brothers, um,
uh, which, uh, I actually worked

00:08:36.008 --> 00:08:39.453
quite closely with some of … We
are one of the tour with Oliver.

00:08:39.997 --> 00:08:44.552
Um, they, they were, like, really pioneers
at that time, and they had this strategy

00:08:44.552 --> 00:08:50.584
of, A, it's, uh, actually 30% about
the ideas and more 70% about execution.

00:08:50.638 --> 00:08:55.645
So they would take these ideas, um, of the
very early commerce days, and they would

00:08:55.936 --> 00:08:58.939
just, like, spin it up across the globe.

00:08:59.325 --> 00:09:03.053
And they usually worked in pretty
difficult market where you have

00:09:03.054 --> 00:09:04.440
to do a lot of customizations.

00:09:04.447 --> 00:09:09.188
For example, this brought me to
work with them on a venture which

00:09:09.239 --> 00:09:10.713
then became part of Alibaba.

00:09:10.959 --> 00:09:14.045
Uh, the ventures was called Lazada,
which was in Southeast Asia.

00:09:14.077 --> 00:09:17.591
And it's, it's … In, as an
idea, it's a clone of Amazon.

00:09:17.665 --> 00:09:18.992
It's, uh, e-commerce.

00:09:19.558 --> 00:09:25.507
And they, uh … It was exciting as
a very young, uh, you know, person

00:09:25.508 --> 00:09:30.184
at that time like me, because they
would really literally tell you, "Hey,

00:09:30.195 --> 00:09:34.928
here you have hundreds of millions
of dollars to play with," which

00:09:34.931 --> 00:09:39.598
comes with specific money, and go and
conquer in like, uh, this new market.

00:09:39.682 --> 00:09:44.389
And the internet penetration in those
markets is actually pretty high, like

00:09:44.394 --> 00:09:49.225
in Southeast Asia, because these, uh,
countries, they, um, they start digital

00:09:49.246 --> 00:09:54.282
first, and then, uh, you know, and it
was, like, figuring out things like how

00:09:54.282 --> 00:09:59.051
do you do logistics and be super efficient
with forecasting is actually pretty

00:09:59.053 --> 00:10:04.102
complicated, especially in these massive
countries where you have very complicated,

00:10:04.208 --> 00:10:07.755
uh, logistics situations like shipping
to islands and so on and so forth.

00:10:07.798 --> 00:10:12.290
So you get an idea that starts from
the US, and then you add a lot of

00:10:12.321 --> 00:10:15.681
complexity and execution on top, and
that's why you actually needed to have

00:10:15.719 --> 00:10:21.274
very good data science across different
domains, from operations to marketing

00:10:21.283 --> 00:10:25.479
optimization and so on, to make sure
you succeed in, in those markets.

00:10:25.518 --> 00:10:30.285
And then, uh, the, uh, Rocket Internet
gave me really the opportunity to work

00:10:30.304 --> 00:10:34.841
in many different business models and
understand a lot of different problems

00:10:34.842 --> 00:10:40.409
that the enterprise have today, like
from, uh, e-commerce retailers to food

00:10:40.409 --> 00:10:45.845
delivery systems, uh, to, uh, things like
subscription systems and so on, so forth.

00:10:45.910 --> 00:10:48.984
So it's just been really
exciting throughout my career

00:10:49.913 --> 00:10:50.212
Yeah.

00:10:50.234 --> 00:10:53.634
And you worked at HelloFresh and Stitch
Fix where there are two subscription

00:10:53.644 --> 00:10:57.261
businesses, and I imagine it's really
interesting to think through the

00:10:57.261 --> 00:11:01.363
lifetime value of customers and churn
modeling and, and I'm sure being a

00:11:01.363 --> 00:11:04.555
data scientist that you were-- you
turned your attention to those things.

00:11:04.556 --> 00:11:06.222
Is that a fair guess?

00:11:06.551 --> 00:11:10.578
Uh, what I would-- what I'd say is that
interestingly enough, HelloFresh is,

00:11:10.602 --> 00:11:12.815
uh, also a Rocket Internet company.

00:11:12.815 --> 00:11:15.753
It was, uh, actually
created by Rocket Internet.

00:11:16.385 --> 00:11:21.469
And, um, they, uh, that's why I say
like I worked for like almost 10

00:11:21.469 --> 00:11:25.740
years with Rocket Internet ventures,
because it, it was actually, I

00:11:25.741 --> 00:11:29.632
started with HelloFresh at very early
beginnings where it was 20 people team.

00:11:30.252 --> 00:11:36.462
Um, and, uh, uh, you know, mothership
Rocket helping them develop, and then

00:11:36.465 --> 00:11:40.606
the founder asked me to build their
data team in the US because the US

00:11:40.684 --> 00:11:42.499
is the largest market of HelloFresh.

00:11:42.858 --> 00:11:45.111
And so when I arrived,
it was just one person.

00:11:45.112 --> 00:11:51.258
I left, it was 75 people with different
sub-teams handling things from analytics,

00:11:51.258 --> 00:11:56.215
experimentation, customer, uh, like
understanding customer analytics is

00:11:56.227 --> 00:11:57.785
part of one of the things we did.

00:11:58.289 --> 00:12:01.916
Marketing analytics, to things
like recommendation engine,

00:12:02.331 --> 00:12:06.629
forecasting, data infrastructure
and platform and so on and so forth.

00:12:07.073 --> 00:12:11.654
So it was a, a very large scale set
up at that time, uh, and, uh, one

00:12:11.654 --> 00:12:14.490
of the best in the industry, I would
say, with that type of industry

00:12:15.542 --> 00:12:15.820
Cool.

00:12:16.036 --> 00:12:19.068
So it sounds like you were kinda like
the go-to guy at Rocket Inter- Rocket

00:12:19.069 --> 00:12:21.798
Internet when they had a startup that
was starting to get some momentum.

00:12:21.800 --> 00:12:25.000
They're like, "Let's get Luca
in here to run some analysis and

00:12:25.000 --> 00:12:27.636
build a team," to kinda like help
solidify that, that business.

00:12:27.637 --> 00:12:28.131
Yeah, yeah.

00:12:28.160 --> 00:12:32.092
My role was, uh, um, basically to
collect learnings and practices

00:12:32.092 --> 00:12:34.989
across the enterprise and across
different batches of Rocket.

00:12:34.993 --> 00:12:40.206
I helped the founders build their data
teams, uh, projects, mostly around growth.

00:12:40.803 --> 00:12:44.027
How do you, um, allocate
budgets in marketings?

00:12:44.075 --> 00:12:45.780
How do you, um…

00:12:46.096 --> 00:12:50.298
We had our own ad tech infrastructure too,
so I was partly responsible to build some

00:12:50.298 --> 00:12:54.774
of our ad tech infrastructure, our bidding
engines, our marketing recommendation

00:12:54.779 --> 00:12:56.312
systems, and so on, so forth.

00:12:56.726 --> 00:13:01.896
Going back to Heidelberg, uh, you don't
seem to have a German accent, so, uh,

00:13:02.276 --> 00:13:03.463
where, where are you from before then?

00:13:04.194 --> 00:13:07.311
I'm or- originally Italian,
from this area of Italy called

00:13:07.683 --> 00:13:08.976
Tuscany, and it's a beautiful area.

00:13:08.976 --> 00:13:09.524
I've lived there.

00:13:09.973 --> 00:13:13.534
I, I, unfortunately, I lived eight year
in Germany, I never picked up German.

00:13:14.034 --> 00:13:16.010
As Mark Twain said, "Life is too short."

00:13:16.196 --> 00:13:19.371
We actually had another Luca
on the podcast from Italy that,

00:13:19.373 --> 00:13:21.692
uh, is in the agentic world
as well, so it's interesting.

00:13:21.699 --> 00:13:23.233
He's at the Darden University.

00:13:23.249 --> 00:13:27.005
Luca, Luca, uh, Cha- Chaian.

00:13:27.394 --> 00:13:28.144
I don't know if you know him or not.

00:13:28.605 --> 00:13:29.207
I'll have to introduce you.

00:13:29.208 --> 00:13:30.123
You'll have a Luca party.

00:13:30.897 --> 00:13:36.690
Um, then, uh, and, and today, you know,
uh, for this paper, you're at a company,

00:13:36.733 --> 00:13:38.376
and let's, let's see if I say this right.

00:13:39.041 --> 00:13:39.810
Pi MC?

00:13:40.117 --> 00:13:40.856
Is that how you say it?

00:13:40.856 --> 00:13:40.866
PyMC

00:13:40.866 --> 00:13:41.312
Labs.

00:13:41.524 --> 00:13:42.058
Correct.

00:13:42.118 --> 00:13:43.130
PyMC, PyMC Labs, yep, okay.

00:13:43.643 --> 00:13:45.463
Yeah, so, so tell us about that.

00:13:45.463 --> 00:13:47.160
You know, you're, you're one of the…

00:13:47.216 --> 00:13:47.841
Let's see, what are you?

00:13:48.179 --> 00:13:50.035
You're one of the founders, right?

00:13:50.481 --> 00:13:51.636
No, I'm, I'm a partner.

00:13:51.644 --> 00:13:51.883
Yeah.

00:13:51.926 --> 00:13:52.015
Oh.

00:13:52.015 --> 00:13:53.507
So the company was founded by- A

00:13:53.507 --> 00:13:53.752
partner.

00:13:53.752 --> 00:13:54.230
Okay, partners.

00:13:54.398 --> 00:13:54.697
Yeah.

00:13:54.700 --> 00:13:54.943
Yeah.

00:13:55.012 --> 00:13:58.172
So, but the company was, uh, uh,
founded by, uh, my business partners,

00:13:58.194 --> 00:14:01.010
Thomas Wiecki, about five years ago.

00:14:01.549 --> 00:14:07.284
Um, so Thomas is very visionary person,
so his original idea is to build a company

00:14:07.284 --> 00:14:11.762
around, uh, the open source library
that he developed that's called, uh,

00:14:11.820 --> 00:14:18.200
PyMC, and, uh, basically find a place
for, uh, uh, academics or researcher,

00:14:18.711 --> 00:14:22.644
uh, that work with him to actually
make money out of the work with it.

00:14:23.313 --> 00:14:29.436
And, um, you know, I was one of
the customers of PyMC Labs, uh,

00:14:29.474 --> 00:14:33.448
in, uh, five to six years ago
when, when they really started.

00:14:34.091 --> 00:14:38.322
And with EloFresh, we used them for one
of our projects because we were using the

00:14:38.322 --> 00:14:44.073
open source library PyMC internally in
our, like, uh, the data science teams.

00:14:44.628 --> 00:14:48.552
And so I reached out to Thomas to help
me with the practical implementation

00:14:48.557 --> 00:14:50.252
of a media mix model at that time.

00:14:50.801 --> 00:14:55.601
And so after that, we became friends
and I've always been advising

00:14:55.601 --> 00:14:58.198
him on how to grow the company.

00:14:58.580 --> 00:15:03.085
And at the beginning of the year, he asked
me to join full-time as business partner

00:15:03.088 --> 00:15:05.819
and, um, we have complementary skill set.

00:15:05.858 --> 00:15:09.993
I'm more on the machine learning
engineering side of the house,

00:15:10.024 --> 00:15:13.152
whereas, uh, like a great statistician.

00:15:13.704 --> 00:15:17.666
And so he comes from the Bayesian
angle, I comes from the agents

00:15:17.684 --> 00:15:21.549
generative AI world of the house,
and we have combined the two ideas

00:15:21.549 --> 00:15:23.342
together in several different projects.

00:15:23.357 --> 00:15:23.712
And what…

00:15:23.791 --> 00:15:27.419
The way we work with clients is
a typical consulting approach.

00:15:27.431 --> 00:15:33.728
So even the, um, um, paper that
we are gonna discuss later on came

00:15:33.729 --> 00:15:37.279
from an engagement we had with one
of our clients, which is Colgate.

00:15:37.449 --> 00:15:41.952
And, uh, you know, we developed these
systems with them, which marries some

00:15:41.952 --> 00:15:47.513
ideas from Bayesian statistics with actual
generative AI eng- agents techniques

00:15:48.903 --> 00:15:49.162
Yeah.

00:15:49.162 --> 00:15:53.608
So to help listeners, um, yeah,
so, so tell us about what the

00:15:53.640 --> 00:15:56.551
PyMC open source library does.

00:15:56.593 --> 00:16:00.201
And, um, you know, I know you're gonna
talk about Bayesian, so, so when you

00:16:00.202 --> 00:16:04.145
do that, like, let's, let's define,
you know, uh, for, for layman's terms,

00:16:04.146 --> 00:16:07.012
what a, what a Bayesian, um, model is.

00:16:07.705 --> 00:16:08.020
Yeah.

00:16:08.022 --> 00:16:13.179
So Bay- Bayesian it's, uh, it's very
good exercise also for me because

00:16:13.192 --> 00:16:16.871
I, I traditionally I'm not, I'm not
actually a Bayesian person myself.

00:16:16.871 --> 00:16:19.427
I, I actually came across the topics.

00:16:19.974 --> 00:16:20.293
Yeah.

00:16:20.462 --> 00:16:22.932
I have a feeling you have a good way
to summarize it ver- If I asked a

00:16:22.933 --> 00:16:25.915
Bayesian person, they probably couldn't
make it short, but I'm sure you can.

00:16:26.385 --> 00:16:26.847
Yeah.

00:16:27.272 --> 00:16:27.984
That's right.

00:16:28.686 --> 00:16:29.834
That's exactly right.

00:16:30.048 --> 00:16:34.965
So I would define Bayesian as a way of
thinking about probability of statistics

00:16:35.031 --> 00:16:37.191
that comes from first principles.

00:16:37.200 --> 00:16:42.108
So you start defining, uh, probability
distributions about your beliefs,

00:16:42.129 --> 00:16:46.880
and then you define a model that
basically represent the world.

00:16:47.418 --> 00:16:52.304
And out of this model, then you
can use to s- sample predictions.

00:16:52.368 --> 00:16:55.749
Um, for example, this could be
a forecasting models about what

00:16:55.754 --> 00:16:59.465
will happen in the future for,
uh, the revenue of your companies.

00:16:59.878 --> 00:17:01.888
And then you can get, um…

00:17:02.137 --> 00:17:06.734
You can encode belief about, uh, the
markets, about what competitors are

00:17:06.739 --> 00:17:12.003
gonna do, and then you can get out
probability distributions on what, uh,

00:17:12.061 --> 00:17:15.142
the world will look like, yeah, in the
future and how the companies are gonna

00:17:15.173 --> 00:17:17.097
perform with confidence intervals.

00:17:17.483 --> 00:17:21.606
And so it, it's something that it's a
little bit different than traditional

00:17:21.640 --> 00:17:25.824
machine learning engineering type of
the house, a way, way of thinking.

00:17:25.899 --> 00:17:29.216
As machine learning engineers, you
think always about the most accurate

00:17:29.248 --> 00:17:34.541
model, and you don't care what the model
specification is really, as, uh, much as

00:17:34.542 --> 00:17:37.190
it performs on a test set very accurately.

00:17:37.627 --> 00:17:41.109
In a Bayesian world, you actually want
to think about the structure of the

00:17:41.115 --> 00:17:45.190
model because you want something that's
interpretable, and you don't want the

00:17:45.193 --> 00:17:49.449
model just to be accurate, but you want
the probability dis- uh, that it is

00:17:49.461 --> 00:17:54.325
outputting to be, uh, as tight as possible
and to give you credible intervals

00:17:54.360 --> 00:18:00.251
that are both, um, representative
of the coverage and tightness, which

00:18:00.251 --> 00:18:05.099
means that they are very, um- It, it
represents your uncertainty about the

00:18:05.105 --> 00:18:06.798
world in the future and adapt to it.

00:18:07.010 --> 00:18:07.359
So

00:18:07.870 --> 00:18:08.165
Got it.

00:18:08.669 --> 00:18:08.887
Okay.

00:18:09.186 --> 00:18:11.822
So when I see, uh, I see
all these surveys, right?

00:18:11.823 --> 00:18:16.085
So there's these stated preference
surveys where people are like 80% of,

00:18:16.121 --> 00:18:21.851
uh, you know, the survey respondents are
using AI for, for buying things online.

00:18:21.890 --> 00:18:25.278
And if you look in the small
print, it'll say N equals 4,000.

00:18:25.589 --> 00:18:26.500
So they've taken a…

00:18:26.904 --> 00:18:30.771
They've taken that and then applied
a Bayesian framework to scale it

00:18:30.777 --> 00:18:32.820
up to the population of the US.

00:18:32.961 --> 00:18:34.893
Is that kind of like, is that
Bayesian or not Bayesian?

00:18:35.264 --> 00:18:35.925
It, it depends.

00:18:35.932 --> 00:18:39.446
So it's a way, it's a way, it's a way
of thinking, uh, it's a way of thinking

00:18:39.463 --> 00:18:43.120
that allows you to generalize things
as, as statistics, as statistics.

00:18:43.122 --> 00:18:46.302
You know, in … Th- there is
two world in the statistics.

00:18:46.305 --> 00:18:51.482
There is the frequentist approach where I
have a sample set, and then I extrapolate

00:18:51.531 --> 00:18:53.301
everything from a finite sample.

00:18:53.650 --> 00:18:56.206
The Bayesian way of the world,
it's a little bit different.

00:18:56.210 --> 00:19:00.176
So they … You think about an abstract
model of how the world will looks like,

00:19:00.516 --> 00:19:05.133
and then your data is just evidence
that supports or falsifies the model.

00:19:05.699 --> 00:19:10.317
And, and therefore, you use
the data to reconcile, uh, what

00:19:10.317 --> 00:19:11.525
the world is gonna look like.

00:19:11.532 --> 00:19:16.954
So i- in, in more like technical
terms, in the Baye- in the, um, like

00:19:16.978 --> 00:19:21.280
traditional statistics, frequentist
one, you have P values, while in the

00:19:21.287 --> 00:19:25.451
Bayesian world, you have confidence
intervals and probabilities, actually

00:19:25.502 --> 00:19:29.846
credible intervals and probabilities
that allows you to define in a more

00:19:29.849 --> 00:19:33.975
nuanced way what different alternatives
of the world will look like.

00:19:34.473 --> 00:19:38.481
Now, that has a lot of implications from
the bus- from business terms, actually.

00:19:38.874 --> 00:19:44.117
When I, when I try to explain in abs-
abstract, it sounds really weird, but

00:19:44.326 --> 00:19:48.263
w- like in, uh, uh … It's, it's a
technique that's used very much when you

00:19:48.264 --> 00:19:50.685
need to take a, a high stake decision.

00:19:51.039 --> 00:19:53.758
For example, uh, some of
our clients are hedge funds.

00:19:54.188 --> 00:19:57.911
They do derivatives pricings,
and they have billion dollars

00:19:57.965 --> 00:19:59.864
decisions to make based on that.

00:19:59.879 --> 00:20:04.701
So knowing the probability of something
to happen, uh, is extremely important

00:20:04.710 --> 00:20:08.051
to them because they can value
the level of risk of specificity.

00:20:08.184 --> 00:20:09.946
And so that's why we charge

00:20:11.132 --> 00:20:11.787
Okay, cool.

00:20:11.787 --> 00:20:13.976
And what's your role there at PyMC?

00:20:14.603 --> 00:20:21.752
So I'm, I'm building the generative AI
vertical, and the idea is, uh, Bayesian

00:20:21.774 --> 00:20:26.400
statistics is a bit of a Swiss Army knife
for certain set of business problems.

00:20:26.859 --> 00:20:31.413
It's a technique that's very hard
actually to, um, use in practice.

00:20:31.840 --> 00:20:36.587
So what we do is that, um, because
the, the language is complicated and

00:20:36.587 --> 00:20:39.846
even the way that I try to explain
it, simplifying it today, you can

00:20:39.846 --> 00:20:43.098
understand it's very complicated
a technique to actually explain.

00:20:43.145 --> 00:20:48.096
While LLMs are universal translators,
and so what we can do is that we

00:20:48.097 --> 00:20:55.586
can embed some of these Bayesian
models, uh, into LLMs to help them

00:20:55.617 --> 00:20:57.953
make decisions in a principled way.

00:20:58.050 --> 00:21:00.609
And, and these are, there's two benefits.

00:21:00.613 --> 00:21:05.029
So LLMs are basically general, uh,
translators, so they can actually

00:21:05.035 --> 00:21:09.640
help you build these complex Bayesian
models similar to the way they help you

00:21:09.644 --> 00:21:11.942
code things in software engineering.

00:21:12.581 --> 00:21:16.126
Uh, but with, uh, an understanding
of the causal structure of your data.

00:21:16.127 --> 00:21:18.847
It's a principled way of
thinking about the world.

00:21:19.309 --> 00:21:25.531
And the other things is that LLMs
can be, uh, often very, uh, confident

00:21:25.806 --> 00:21:28.803
about, uh, things, but be very wrong.

00:21:29.403 --> 00:21:35.431
And so Bayesian techniques allows
you to, um, have LLMs reasons about

00:21:35.474 --> 00:21:37.061
uncertainty in a principled way.

00:21:37.204 --> 00:21:42.215
Uh, so like, uh, correcting for some
of the biases that these LLMs have.

00:21:42.272 --> 00:21:47.178
And within this world, um, you
know, uh, we also think about

00:21:47.272 --> 00:21:51.864
ourself in a more abstract way as a
expert in making simulations, very

00:21:51.864 --> 00:21:55.684
complex simulations, even while you
cannot explicitly build the model.

00:21:56.023 --> 00:22:00.023
And so the papers we built is an example
of a simulations in a way that you

00:22:00.023 --> 00:22:04.535
can do with AI, where you don't have
an explicit model of human behavior.

00:22:04.535 --> 00:22:07.373
It would be too complex to
actually build something like this.

00:22:07.810 --> 00:22:14.309
So you rely on LLM to learn from a,
uh, training data what the heuristics

00:22:14.310 --> 00:22:16.001
of human decision-making are.

00:22:16.583 --> 00:22:21.800
And, and that's the way you are
actually, um, physically you can do that.

00:22:21.866 --> 00:22:24.185
You can build very-- these
very complex simulations

00:22:25.076 --> 00:22:25.674
Okay, cool.

00:22:25.836 --> 00:22:28.039
So, so you guys have clients
that come to you and say,

00:22:28.039 --> 00:22:29.486
"I'd, I, I'd really like to…

00:22:29.520 --> 00:22:30.686
You know, here's my business problem."

00:22:30.797 --> 00:22:32.020
And that could be a hedge fund.

00:22:32.459 --> 00:22:34.188
Colgate is obviously a brand.

00:22:34.212 --> 00:22:38.304
And, and then it's your job to figure out
in this toolkit that your company offers,

00:22:38.304 --> 00:22:42.251
which range from these Bayesian, you know,
statistical models that are very advanced.

00:22:42.473 --> 00:22:46.492
You've brought in kind of LLMs and
AI as another tool in the toolbox

00:22:46.492 --> 00:22:49.277
to kind of help, help figure out
the right way to approach this.

00:22:49.475 --> 00:22:50.074
C- Correct.

00:22:50.096 --> 00:22:53.727
So, and, uh, a, a lot of clients
come for, from, because they've

00:22:53.729 --> 00:22:55.107
seen our open source tools.

00:22:55.109 --> 00:22:57.611
We are very much, uh, you
know, contributor to different

00:22:57.618 --> 00:23:00.336
open source libraries for
different types of applications.

00:23:00.756 --> 00:23:02.384
And then they get curious about that.

00:23:02.443 --> 00:23:06.387
They read our white papers, they
come on our website, and then we

00:23:06.390 --> 00:23:07.506
start collaborations with them.

00:23:08.309 --> 00:23:08.555
Yeah.

00:23:08.582 --> 00:23:10.484
Red Hat is who where I
am, and that's kind of…

00:23:10.526 --> 00:23:13.506
They, they, they took the
Linux distribution and added

00:23:13.506 --> 00:23:16.731
an enterprise layer, uh, to
it, and that's where, you know.

00:23:16.816 --> 00:23:20.124
And then they have a licensing model on
top of that, that, that kind of beefs it

00:23:20.126 --> 00:23:21.571
up and makes it more enterprise ready.

00:23:21.571 --> 00:23:24.522
And then they have-- But, like, the
bulk of what they do is also support

00:23:24.522 --> 00:23:26.056
and consulting around, around.

00:23:26.113 --> 00:23:29.127
So it's a, it's a way to monetize
open source that's pretty clever.

00:23:30.784 --> 00:23:31.059
Yeah.

00:23:31.481 --> 00:23:32.774
Um, so let's talk about the paper.

00:23:32.774 --> 00:23:35.171
So what was the problem
that Colgate came to you…

00:23:35.197 --> 00:23:38.020
Colgate-Palmolive, uh, came to
you and said, "Hey, we, we need

00:23:38.020 --> 00:23:40.181
help figuring out this problem"?

00:23:40.861 --> 00:23:41.150
Yeah.

00:23:41.151 --> 00:23:44.670
So it's a very big problem
for them and, uh, it…

00:23:44.767 --> 00:23:47.599
they invest tons of, uh, money every year.

00:23:47.611 --> 00:23:50.871
So they're talking about, you know,
hundreds of millions of dollars,

00:23:50.894 --> 00:23:51.875
which is product development.

00:23:52.200 --> 00:23:56.663
And so the-- typically, the way that,
that it's done is that when they

00:23:56.684 --> 00:24:01.457
think about the new concept for a new
product, before they go and produce

00:24:01.470 --> 00:24:06.214
that product, um, they need to-- they,
they wanna evaluate whether it's gonna

00:24:06.216 --> 00:24:08.079
resonate to a certain target audience.

00:24:08.083 --> 00:24:10.917
There are a lot of different
considerations that goes into this

00:24:10.918 --> 00:24:15.112
in terms of even, like, is-- can
we produce it at the right costs?

00:24:15.710 --> 00:24:18.041
Um, what's the price
point for these products?

00:24:18.041 --> 00:24:18.981
And so on and so forth.

00:24:19.503 --> 00:24:26.493
And so these are very complex decisions
and workflows, and the idea was take

00:24:26.604 --> 00:24:31.466
parts of these very complex decisions and
workflows and try to, um, automate them.

00:24:31.533 --> 00:24:35.617
And one of the most time-consuming one
is really to do these consumer panels

00:24:36.028 --> 00:24:42.628
where they give these products to, um,
to real humans and they, they ask, uh,

00:24:42.663 --> 00:24:44.581
questions like, "Hey, would you buy this?

00:24:44.626 --> 00:24:46.248
How unique you think it is?

00:24:46.544 --> 00:24:49.317
What do you think when you--
about this new product compared

00:24:49.317 --> 00:24:50.538
to competitive product?

00:24:50.887 --> 00:24:52.555
At what price points would you buy it?"

00:24:53.143 --> 00:24:58.355
So these type of questions, um,
they're typically, uh, conducted with

00:24:58.707 --> 00:25:02.545
re-research of, like, 30 to 40 humans.

00:25:03.047 --> 00:25:06.879
Um, assembling these
panels is time-consuming.

00:25:07.484 --> 00:25:10.160
Uh, running these interviews is expensive.

00:25:10.414 --> 00:25:12.592
It can cost you quite a bit
amount of money to work with

00:25:12.624 --> 00:25:13.969
some of the consultancies.

00:25:14.553 --> 00:25:21.374
And right now, like even where they're
using even a generative AI technique

00:25:21.672 --> 00:25:25.773
to produce new product concepts,
it just doesn't scale anymore to

00:25:25.789 --> 00:25:27.861
run all these type of interviews.

00:25:28.220 --> 00:25:32.533
And so what Colgate wanted is a way to
screen new product ideas really, really

00:25:32.536 --> 00:25:39.104
fast to prioritize them offline so that
they can focus only on the, I guess, value

00:25:39.104 --> 00:25:45.821
one, and only those run with maybe humans
or just really bring it to productions.

00:25:46.268 --> 00:25:48.042
And so that's what we did.

00:25:48.043 --> 00:25:52.809
So we built basically a, a synthetic
consumer panel that is able to answer,

00:25:53.224 --> 00:25:57.118
uh, questions about these product
concepts in a way that's very similar

00:25:57.161 --> 00:26:01.651
to what a real human, uh, panel could
have answered about that product

00:26:02.610 --> 00:26:02.796
Yeah.

00:26:02.910 --> 00:26:05.352
Uh, let me-- I was gonna pull it up,
but I'll just show it on the screen.

00:26:05.437 --> 00:26:09.887
Um, so this was an example from the
paper where you had a, a picture of a

00:26:09.888 --> 00:26:14.536
product, um, and you know, the, this is
kind of the, the kind of thing they would

00:26:14.537 --> 00:26:16.359
show people and say, "Here's a product.

00:26:16.359 --> 00:26:17.138
What do you think about it?"

00:26:17.168 --> 00:26:22.881
Um, and, uh, so for those that listen
to the audio, it's called Orofoam Mood

00:26:22.882 --> 00:26:27.083
Infused Body Wash, and then it has,
like, a little description, a picture,

00:26:27.087 --> 00:26:30.531
a product picture, and then four or
five bullets talking about the product.

00:26:31.366 --> 00:26:40.102
Um, so the, their problem is that,
uh, they spend all this money, you

00:26:40.102 --> 00:26:42.659
know, hundreds of thousands of dollars
getting, putting these things in

00:26:42.660 --> 00:26:46.144
front of people, and I guess their
question is, "Can we do this with a,

00:26:46.374 --> 00:26:47.885
you know, with a simulated human?"

00:26:49.852 --> 00:26:50.098
Yeah.

00:26:50.593 --> 00:26:52.849
Or they just said, they said to
you, "We need to figure out a

00:26:52.858 --> 00:26:55.136
cheaper way to do this," and you
said, "Let's simulate humans."

00:26:55.137 --> 00:26:57.907
Like how, how did that,
how did that kinda go?

00:26:58.805 --> 00:27:00.904
Um, it-- That's a very good question.

00:27:00.907 --> 00:27:01.606
So they, they're…

00:27:02.047 --> 00:27:06.046
Colgate is actually a bit of the
forefront, uh, compared to other,

00:27:06.106 --> 00:27:07.868
uh, companies in doing this.

00:27:08.050 --> 00:27:14.508
Actually, they, um, had started
already pilots with, uh, other, um,

00:27:14.654 --> 00:27:16.672
firms on developing this technology.

00:27:16.672 --> 00:27:20.763
They also had an in- an internal
team working on something like this.

00:27:21.368 --> 00:27:25.132
However, um, they, they had two problems.

00:27:25.158 --> 00:27:31.326
One problem is that the results that they,
they were getting weren't that great,

00:27:31.649 --> 00:27:37.867
and they weren't really sure, um, why
and what determines, what are the factors

00:27:37.905 --> 00:27:40.547
that determines, like, uh, a good outcome.

00:27:40.547 --> 00:27:41.948
Is it the type of LLMs?

00:27:42.325 --> 00:27:43.692
Is it the way you prompt them?

00:27:44.266 --> 00:27:47.920
Um, so there are different parameters in
these algorithms that you can actually,

00:27:48.108 --> 00:27:51.506
uh, tweak to, to make it actually perform.

00:27:52.147 --> 00:27:56.104
And the second thing is they weren't
sure is that how even do you measure

00:27:56.189 --> 00:27:57.548
accuracy of something like this?

00:27:57.551 --> 00:27:59.800
What does it mean accurate in itself?

00:28:00.352 --> 00:28:03.965
And so when they came to us, they
came with the research questions

00:28:04.006 --> 00:28:08.376
of, "Hey, can you help us figure
it out within this problem domain?

00:28:08.430 --> 00:28:13.026
A technique that's actually good and
actually works so that even within our

00:28:13.026 --> 00:28:17.440
enterprise, we can convince all the
different business partners to invest more

00:28:17.440 --> 00:28:22.370
in this technology and then productize
it to become part of, uh, our workflows."

00:28:22.417 --> 00:28:26.962
And, and that's the reason why we
came up with a, a research type of

00:28:26.962 --> 00:28:31.810
engagement with them, which then
became an actual published paper.

00:28:32.236 --> 00:28:35.464
Um, the paper is, uh, it's gonna
be submitted to the conference.

00:28:35.467 --> 00:28:37.844
It's currently in archive
for everyone to check.

00:28:38.209 --> 00:28:41.999
And we also open-sourced part of the
code base that we used to really make

00:28:41.999 --> 00:28:46.837
sure that people can reproduce these
type of results, uh, by themselves.

00:28:47.515 --> 00:28:51.346
If I was gonna try to run this
experiment, the baseline is humans, right?

00:28:51.347 --> 00:28:54.953
So, so we're trying to do simul- si-
you know, simulations versus humans.

00:28:55.222 --> 00:28:56.788
Is that kinda how you
set up the experiment?

00:28:56.788 --> 00:28:57.537
You kinda said, "All right.

00:28:57.538 --> 00:29:01.141
Here's how humans do on these
types of things, and, you know,

00:29:01.150 --> 00:29:03.807
we're gonna, we're gonna try to
get as close to that as possible."

00:29:03.879 --> 00:29:05.359
Um, yeah.

00:29:05.374 --> 00:29:07.356
Is that, is that kind of
the setup of the experiment?

00:29:07.656 --> 00:29:07.849
Yeah.

00:29:07.849 --> 00:29:08.265
That's right.

00:29:08.270 --> 00:29:11.253
So the question is how, um,
how do you measure accuracy?

00:29:11.288 --> 00:29:15.647
And the idea is that Colgate has
pretty large datasets of past consumer

00:29:15.647 --> 00:29:18.756
studies because they've been running
this business for many, many years.

00:29:19.296 --> 00:29:24.448
And we use, uh, basically a dataset
which I believe, uh- I don't remember

00:29:24.616 --> 00:29:30.884
the exact numbers, but over 8,000 unique
users, uh, and, uh, hundreds of products.

00:29:31.457 --> 00:29:40.050
And, uh, basically the idea for us is
if we want the, um, LLMs to be accurate.

00:29:40.052 --> 00:29:44.729
So accurate means that if they have
a new product, they would rank this

00:29:44.744 --> 00:29:48.299
product, um, similarly to what human…

00:29:48.410 --> 00:29:50.988
You would, would rank this
product so that you can actually

00:29:50.990 --> 00:29:52.490
prioritize them on a scale.

00:29:52.557 --> 00:29:54.894
This is our high priority
versus low priorities.

00:29:55.462 --> 00:29:59.874
And, and then if you ask them to
rate these products on a scale from

00:29:59.874 --> 00:30:05.402
one to five, the distribution of the
ratings of the LLMs look similar to the

00:30:05.402 --> 00:30:07.647
distributions of, uh, the human panel.

00:30:08.221 --> 00:30:13.888
And notice that there is a nuance here
that if you ask two panels of people, uh,

00:30:14.292 --> 00:30:18.720
to rate these same products, and these
two panels are extracted exactly with

00:30:18.732 --> 00:30:24.591
the same demographics and, uh, you know,
they are almost identical twins, they

00:30:24.592 --> 00:30:30.419
would anyway rate the products not 100%
the same, but they will still correlate,

00:30:30.485 --> 00:30:33.066
uh, like with a certain correlation fact.

00:30:33.243 --> 00:30:36.374
Uh, but so even nuance, they
have an intrinsic variabilities.

00:30:36.689 --> 00:30:40.365
And so in the way we had to evaluate this
was taking into account the variability

00:30:40.380 --> 00:30:45.569
from our clients and the variability of
LLMs in providing the answers to make sure

00:30:45.578 --> 00:30:47.298
that the results are aligned as possible.

00:30:47.298 --> 00:30:51.731
And if we know that with, uh, these
historical data, we can reproduce

00:30:51.735 --> 00:30:56.286
the results of humans, we have
confidence that, of course, this

00:30:56.533 --> 00:31:01.431
can potentially generalize to new
products, uh, and to future situations.

00:31:01.433 --> 00:31:04.634
You can never have 100%
confidence in science.

00:31:04.635 --> 00:31:09.333
You can always falsify things, but that's
the way you actually do is you infer

00:31:09.363 --> 00:31:13.371
with data that you have, and that gives
you confidence about future predictions.

00:31:14.671 --> 00:31:14.920
Cool.

00:31:15.164 --> 00:31:16.768
So that, that helps me
understand the setup.

00:31:16.768 --> 00:31:18.010
Let's just go right to the punchline.

00:31:18.011 --> 00:31:21.117
Did you get close to this or
you couldn't make it work?

00:31:21.306 --> 00:31:24.338
Yes, we, we could, we could
prove that we could make it work.

00:31:24.930 --> 00:31:28.646
Um, we actually proved that it, uh, works.

00:31:29.007 --> 00:31:33.890
You know, we, uh, I do think the
main insight of the, uh, paper is

00:31:34.374 --> 00:31:39.821
if you do the naive things, uh, of
prompting an LLM as a human, you get

00:31:39.869 --> 00:31:42.210
okay-ish results but not good results.

00:31:42.309 --> 00:31:47.899
Uh, and, uh, I think the insights of
the paper is how can you get something

00:31:47.963 --> 00:31:53.667
that's better, and why the, if you
prompt an LLM naively to answer as

00:31:53.667 --> 00:31:55.472
a human, it actually doesn't work.

00:31:55.935 --> 00:32:00.644
And so that insights is what kind
of like help us, uh, and we discover

00:32:00.653 --> 00:32:04.396
by using the paper, and we can get
into it a little bit if you like.

00:32:05.586 --> 00:32:06.562
Is there a, uh…

00:32:06.591 --> 00:32:08.766
So let's kind of talk
about, so humans are here.

00:32:08.829 --> 00:32:10.622
Let's, let's say they're at 100.

00:32:10.674 --> 00:32:13.528
Did you get halfway to human or
like where did you get on the scale?

00:32:15.133 --> 00:32:18.577
And, and it's fine if you reject the
way I'm, I'm d- I'm framing this.

00:32:19.047 --> 00:32:19.669
Yeah, yeah.

00:32:19.746 --> 00:32:24.517
Yeah, and so humans in itself are not
at 100 because if you get two humans

00:32:24.545 --> 00:32:29.150
partners that are actually identical,
they don't correlate with each other 100%.

00:32:29.207 --> 00:32:32.826
They are something around
90%, uh, like correlations.

00:32:33.339 --> 00:32:33.752
And-

00:32:33.752 --> 00:32:33.762
Okay.

00:32:33.785 --> 00:32:35.406
So the high water mark is 90%.

00:32:35.513 --> 00:32:35.668
Okay.

00:32:35.696 --> 00:32:37.305
The high water mark is 90%.

00:32:37.396 --> 00:32:42.121
And, uh, um, the … I believe our
latest results published in the paper

00:32:42.121 --> 00:32:48.969
is something like 73, 74%, which
is, uh, n- close to 90% of that 90%.

00:32:49.575 --> 00:32:54.377
Is that enough where, uh, Colgate said
you solved this, or did they feel like

00:32:54.459 --> 00:32:55.944
that it still wasn't close enough?

00:32:56.647 --> 00:32:56.960
Yeah.

00:32:56.960 --> 00:33:02.573
So I … It … The … To them,
uh, the, the most important, um,

00:33:02.628 --> 00:33:06.388
outcome of this project was to get
the confidence that they needed to

00:33:06.389 --> 00:33:11.284
convince, uh, their business stakeholders
and counterparts that that's a

00:33:11.291 --> 00:33:15.518
technique that they can invest on to
automate their internal processes.

00:33:15.580 --> 00:33:20.527
And, uh, their short answer is yes, they
continued investing into the platform,

00:33:20.966 --> 00:33:25.439
um, which is proprietary to them that
they use to prioritizing products.

00:33:25.441 --> 00:33:29.580
So that's based on this technique and,
uh, evolutions of these techniques.

00:33:29.594 --> 00:33:31.663
We … And we are, you know,
still working with them.

00:33:31.664 --> 00:33:35.700
They, they continue to develop this
algorithm and these ideas, uh, today.

00:33:35.834 --> 00:33:36.123
So.

00:33:36.815 --> 00:33:41.992
So you talked about naively asking
the LLMs, um, i- is not the way to go.

00:33:42.142 --> 00:33:44.207
And why is that?

00:33:44.257 --> 00:33:47.646
Like, the LLMs have been trained on
all of human knowledge, shouldn't

00:33:47.646 --> 00:33:49.303
they just be basically people?

00:33:50.891 --> 00:33:54.063
That's the … I'll play the
role of the naive, uh, the naive

00:33:54.070 --> 00:33:56.602
person in this and explain to me
why, why that's not gonna work.

00:33:56.902 --> 00:33:57.189
Yeah.

00:33:57.192 --> 00:34:00.227
And so you, you get the, you get the
thing that's called mode collapse.

00:34:00.296 --> 00:34:07.525
Uh, so basically if you ask the LLM,
um, like, "Give me a rating on this

00:34:07.538 --> 00:34:10.949
product," and describe you the product,
and you tell the LLM, "Answer as a

00:34:10.984 --> 00:34:16.199
person who lives in Brooklyn, uh, that's
a female between this and that age."

00:34:16.633 --> 00:34:20.208
Or sorry, "Of this age, and she
makes around this bracket of income,"

00:34:20.208 --> 00:34:25.445
and so on and so forth, they tend
to answer with the most One single

00:34:25.445 --> 00:34:28.603
value, which is the most probable
value in, they've seen in the dataset.

00:34:29.399 --> 00:34:33.427
And the reason why they do that is
that they haven't spent enough thinking

00:34:33.462 --> 00:34:39.112
tokens into reason about what makes
that product a good fit for that person.

00:34:39.136 --> 00:34:44.462
So one simple thing you can do is
actually to ask instead of asking, uh,

00:34:44.475 --> 00:34:49.811
just the rating number to, um, give you
a reason for it, and you should ask for

00:34:49.811 --> 00:34:54.297
the reason before you ask the number so
that the thinking token are spent upfront

00:34:54.831 --> 00:34:57.582
to come up with, uh, uh, a good answer.

00:34:58.186 --> 00:35:04.754
And the, uh, other thing that you can
do is to ask, uh, for just the reason

00:35:04.758 --> 00:35:09.223
and then have a technique to remap
that reason onto the scale one to five

00:35:09.653 --> 00:35:13.904
that gives you a soft wording, voting
score instead of out voting score.

00:35:14.108 --> 00:35:16.212
Because the problem you
get is this mode collapse.

00:35:16.629 --> 00:35:19.218
Otherwise, the response of
the LLMs will be very peaked.

00:35:19.291 --> 00:35:23.240
They will always answer three or they
will always answer five, and you don't

00:35:23.241 --> 00:35:27.500
get the variability that you would get
in a normal, you know, uh, distribution.

00:35:28.728 --> 00:35:32.563
Can you, can you tell-- can
you give the LLM more context?

00:35:32.563 --> 00:35:35.968
Like I've seen people kind of
say they'll, they'll take…

00:35:35.985 --> 00:35:38.917
So I've seen retailers do this where
they'll know that they have, and I'll

00:35:38.917 --> 00:35:42.074
just like, you know, they'll have like,
let's say they have a really simple

00:35:42.077 --> 00:35:47.546
demographic, like one third soccer moms
and one third, you know, tech bros and

00:35:47.547 --> 00:35:49.022
one third, uh, I don't know, whatever.

00:35:49.257 --> 00:35:51.943
And they'll kind of say,
"You are a soccer mom.

00:35:51.950 --> 00:35:55.227
You like these things and
you're very busy and, and…"

00:35:55.805 --> 00:35:58.126
Will that influence the LLM enough or no?

00:35:58.177 --> 00:36:02.825
It's still, it still will go home to
the most statistical probable thing.

00:36:03.145 --> 00:36:07.605
It, it does, uh, but it will still
go home to the most probable things

00:36:07.609 --> 00:36:10.144
for that type of subset of prompts.

00:36:10.758 --> 00:36:11.765
And- So that's,

00:36:11.816 --> 00:36:12.522
that's not gonna work.

00:36:12.601 --> 00:36:12.804
Yeah.

00:36:12.896 --> 00:36:13.209
Yes.

00:36:13.243 --> 00:36:15.884
And, and the reason is, again,
it's about the thinking tokens.

00:36:15.899 --> 00:36:21.188
So like LLMs, the output of
LLMs, it depends on two things.

00:36:21.196 --> 00:36:25.089
Depends on your prompt that you give
them, but also depends on how much time

00:36:25.125 --> 00:36:27.002
they spent reasoning about the answer.

00:36:27.479 --> 00:36:32.196
And it's the same reason why reasoning
LLMs actually do work today, is because

00:36:32.236 --> 00:36:36.418
they spend significant amount of time
considering different points of view on

00:36:36.419 --> 00:36:38.153
the answer before they actually give it.

00:36:38.240 --> 00:36:42.691
So if you just ask for one answer,
yes or no, it would probably give

00:36:42.796 --> 00:36:44.470
not thinking about this enough.

00:36:44.979 --> 00:36:48.312
And if you ask for to think about
the answer and give you an extended

00:36:48.313 --> 00:36:50.481
one, then you get these variability.

00:36:51.295 --> 00:36:53.187
So I understand why LLMs won't work.

00:36:53.459 --> 00:36:58.690
How did you build a better synthetic,
uh, representation of how people react?

00:36:58.691 --> 00:37:01.017
Like what, what was the
unlock, uh, to do that?

00:37:01.173 --> 00:37:02.529
And does it involve Likerts?

00:37:05.833 --> 00:37:07.520
Well, let, let me ask what a Likert is.

00:37:07.522 --> 00:37:08.202
Like, that was new to me

00:37:08.202 --> 00:37:13.497
So a L- Likert's, uh, s- scale is,
uh, a scale that's used in, uh,

00:37:13.532 --> 00:37:15.851
um, basically consumer surveys.

00:37:15.858 --> 00:37:19.731
And this simply means that to
the question you can answer

00:37:19.939 --> 00:37:21.744
on a scale from one to five.

00:37:22.428 --> 00:37:27.166
And, uh, instead of like, uh, um, giving
you like, uh, you know, uh, what's

00:37:27.167 --> 00:37:30.226
supposed to scale from one to 10 or
it's giving you a yes or no answer.

00:37:30.802 --> 00:37:37.962
So, uh, and, uh, the, um, and so
you, you have a distribution of…

00:37:38.025 --> 00:37:40.095
If you ask the panel about
these questions, you have a

00:37:40.097 --> 00:37:41.474
distribution of responses.

00:37:41.474 --> 00:37:45.094
And so this distribution is
typically a sh- a certain shape.

00:37:45.684 --> 00:37:50.888
And, um, the, the idea is that if you
get the distributions of responses

00:37:50.889 --> 00:37:55.975
from a, uh, a synthetic consumer panel,
you want the, the two histograms to

00:37:55.976 --> 00:37:58.238
look as, uh, similar as possible.

00:37:58.362 --> 00:38:01.124
And you can measure
that in different ways.

00:38:01.609 --> 00:38:07.908
Um, the way we choose to measure
this is to use, um, a, a scoring

00:38:07.908 --> 00:38:09.783
which is called the KS divergence.

00:38:10.309 --> 00:38:14.140
And the KS divergence is simply
a, a good score because it's

00:38:14.140 --> 00:38:15.508
normalized between zero and one.

00:38:15.508 --> 00:38:18.028
It tells you how much two
histograms are far apart.

00:38:18.626 --> 00:38:21.924
And, uh, um, being normalized
between zero and one, it's

00:38:21.961 --> 00:38:23.556
actually are easy to interpret.

00:38:24.065 --> 00:38:26.771
So that's the way we choose
to, uh, to measure this.

00:38:27.450 --> 00:38:33.276
And so the, the technique that we
developed, uh, instead of, uh, asking

00:38:33.287 --> 00:38:38.394
you for one single token and answers,
again, ask, uh, the LLMs for, uh, an

00:38:38.394 --> 00:38:42.412
entire sentence of what they like about
the products and what they don't like.

00:38:42.926 --> 00:38:48.713
And if you do that, you can take that
answers, uh, and then remap them on a

00:38:48.713 --> 00:38:54.566
score to one to five, uh, basically by
clustering them in the embedding space.

00:38:54.607 --> 00:38:57.520
You can take a sentence and then
you can send it back to ChatGPT,

00:38:57.537 --> 00:39:00.746
you can get an embedding of that
sentence, which means a vector.

00:39:01.185 --> 00:39:04.280
And then you can look at the clusters
in the embedding space of these

00:39:04.310 --> 00:39:07.952
answers called similar answers
that have the similar reasoning

00:39:08.495 --> 00:39:10.735
will be, uh, close together.

00:39:11.443 --> 00:39:17.058
And then you can, uh, aggregate those, uh,
answers instead of with the hard scoring,

00:39:17.064 --> 00:39:19.021
you can use a soft scoring approach.

00:39:19.033 --> 00:39:24.212
So maybe something that a nuanced
answer is something in between, give,

00:39:24.212 --> 00:39:31.918
uh, a score of, uh, 50% to a four
and 30% to a five and 20% to a two.

00:39:32.435 --> 00:39:35.839
It's not that you have a hard
score or you have just a five.

00:39:36.265 --> 00:39:40.772
And that way of aggregating responses,
it's more representative of the nuanced

00:39:40.772 --> 00:39:42.791
world that, uh, humans think about.

00:39:42.865 --> 00:39:47.483
And so represent better the variability
of humans in terms of this KS divergence

00:39:48.409 --> 00:39:48.713
Okay.

00:39:49.037 --> 00:39:51.123
Let me just restate that
to make sure I got it.

00:39:51.146 --> 00:39:52.571
The, um…

00:39:52.656 --> 00:39:56.220
So if you get a full sentence or, or
more than, you know, a pretty good

00:39:56.297 --> 00:40:00.719
explanation from the LLM, that gives
you enough sentiment in there that you

00:40:00.726 --> 00:40:04.128
can map it better onto, you know, a…

00:40:04.188 --> 00:40:07.454
You can, you can give it a better
mapping and create more divergence

00:40:07.456 --> 00:40:11.702
of the responses that, that's
closer, 90% to the 90% of humans.

00:40:12.175 --> 00:40:12.769
That, that's right.

00:40:12.772 --> 00:40:15.102
Actually, th- you, um…

00:40:15.535 --> 00:40:18.168
When we think that the similarity
between this technique and

00:40:18.169 --> 00:40:19.350
sentiment analysis, I…

00:40:19.408 --> 00:40:22.413
It's something that I haven't
thought about, but it's very, uh,

00:40:22.437 --> 00:40:24.112
similar to sentiment analysis.

00:40:24.498 --> 00:40:28.071
So you basically get the sentence from
the LLM, then you map that sentiment

00:40:28.531 --> 00:40:32.507
into this score of one to five, which
is very similar to that technique there.

00:40:33.501 --> 00:40:36.545
If you watch real humans and, you
know, you can imagine if you're just

00:40:36.545 --> 00:40:40.092
seeing them, you know, write one or
five, it's not as good as them saying,

00:40:40.565 --> 00:40:42.118
"Oh, I really love that color."

00:40:42.185 --> 00:40:42.817
And you know.

00:40:43.029 --> 00:40:46.008
So that's gonna be very, that's
a strong five versus maybe they

00:40:46.009 --> 00:40:46.844
would have scored it, yeah.

00:40:47.056 --> 00:40:49.224
Or, you know, it's okay.

00:40:49.395 --> 00:40:53.133
Like that, that's negative
sentiment in there that you

00:40:53.133 --> 00:40:54.825
only really get from a sentence.

00:40:54.833 --> 00:40:58.203
Sometimes you only get it from in-
intonation of voice and body language.

00:40:58.314 --> 00:41:01.324
That's probably where that
last 10% is, would be my guess.

00:41:01.680 --> 00:41:04.812
Uh, just kinda like those little things
we get as being humans that, that are,

00:41:05.134 --> 00:41:06.773
you know, not in the written language.

00:41:07.186 --> 00:41:07.667
Um.

00:41:08.168 --> 00:41:08.481
Yeah.

00:41:08.598 --> 00:41:12.292
And, and then there's other advantage,
by the way, as you point out too, that

00:41:12.487 --> 00:41:16.720
the hu- the LLMs or the humans, if
they answer with an extended answer,

00:41:16.770 --> 00:41:19.829
they can tell you the reasons of why
they like and don't like the product.

00:41:19.958 --> 00:41:23.611
Now, those reasons are valuable
because then you can, uh, um,

00:41:23.631 --> 00:41:27.909
basically analyze them and use them
to improve the product Which is a buy,

00:41:28.104 --> 00:41:29.838
so it's, you get it for free usually

00:41:30.111 --> 00:41:30.374
Okay.

00:41:30.385 --> 00:41:31.061
So,

00:41:31.061 --> 00:41:31.781
so

00:41:31.793 --> 00:41:32.083
you've-

00:41:32.083 --> 00:41:34.660
And I, it's a funny story because,
sorry, this I have to tell you because

00:41:34.661 --> 00:41:35.883
I got the results this morning.

00:41:35.883 --> 00:41:36.789
It's actually really funny.

00:41:36.790 --> 00:41:40.988
We are rebranding our website, and we are
choosing two different variants of the

00:41:40.988 --> 00:41:44.132
colors of the website and the messaging
that we're gonna use on our website.

00:41:44.625 --> 00:41:45.951
And so we did two things.

00:41:45.982 --> 00:41:48.645
We did an internal survey within our team.

00:41:48.646 --> 00:41:50.074
It's about 40-ish people.

00:41:50.622 --> 00:41:53.660
And then we use our synthetic
consumer technology to evaluate

00:41:53.691 --> 00:41:55.363
that, and the results are spot on.

00:41:55.732 --> 00:42:01.106
And both the complaints that our team had
about some of the fonts of the new website

00:42:01.106 --> 00:42:06.760
being too small or some of the messaging
not being great, uh, were exactly the same

00:42:07.338 --> 00:42:10.024
as our synthetic consumers' equivalent.

00:42:10.103 --> 00:42:15.515
And we also, in this case, instead of
using a generic panels, we prompted

00:42:15.515 --> 00:42:20.456
the synthetic consumers to look like a
group of PhD students and professors,

00:42:20.487 --> 00:42:24.499
which is more or less representative
of our, like, uh, component.

00:42:24.561 --> 00:42:26.357
And so they gave a very good answer.

00:42:26.532 --> 00:42:27.427
Does that freak you out?

00:42:27.455 --> 00:42:28.449
Does it sometimes…

00:42:28.753 --> 00:42:32.553
The statisticians are probably already
there, like, uh, but sometimes it does

00:42:32.553 --> 00:42:36.270
freak me out that, like, we're not that
far away from LLMs at the end of the day.

00:42:36.713 --> 00:42:37.001
Yeah.

00:42:37.001 --> 00:42:38.777
It did, it did freak me out a little bit.

00:42:38.794 --> 00:42:41.282
I wasn't expecting
survey so close, Randall.

00:42:42.294 --> 00:42:42.469
Yeah.

00:42:42.469 --> 00:42:43.964
That's the fun part of
this field we're in.

00:42:43.964 --> 00:42:45.118
It's kinda like you, you…

00:42:45.503 --> 00:42:46.834
I try not to think too much about it.

00:42:47.851 --> 00:42:52.069
Again, r- it'd kinda be a bummer
Makes me wanna just go learn

00:42:52.078 --> 00:42:52.931
to play guitar or something.

00:42:53.565 --> 00:42:55.695
Do something very, very
human that an LLM can't do.

00:42:56.269 --> 00:42:58.465
Um, so, so that was one of my questions.

00:42:58.484 --> 00:43:01.757
So you've built this really interesting
framework, and, uh, did I hear

00:43:01.758 --> 00:43:02.788
you say you've open sourced it?

00:43:02.788 --> 00:43:04.035
Is it out on a GitHub somewhere?

00:43:04.236 --> 00:43:08.373
Um, you, you get the code, uh,
for the, uh, for the reproducing

00:43:08.378 --> 00:43:09.394
the results of the paper.

00:43:09.395 --> 00:43:12.810
Of course, there is part of the things
we, we have to keep proprietary given

00:43:12.843 --> 00:43:17.083
the, uh, collaboration with Colgate,
but they, they were really kind

00:43:17.148 --> 00:43:18.966
to let us open source most of it.

00:43:19.846 --> 00:43:20.098
Yeah.

00:43:20.342 --> 00:43:24.358
But you guys now have this black box of
simulated humans, and you can have it…

00:43:24.760 --> 00:43:27.641
You looked at a, a consumer product.

00:43:27.907 --> 00:43:29.644
Now you can have it look at your website.

00:43:29.794 --> 00:43:35.978
Um, and, uh, you know, presumably this
is pretty interesting because it is one

00:43:35.978 --> 00:43:41.589
thing to go and, you know, hire a firm
to go do a $300,000 consumer study and,

00:43:41.614 --> 00:43:43.470
and all the analysis and everything.

00:43:43.884 --> 00:43:46.676
That's, that's the really high end
of this type of consumer research.

00:43:46.784 --> 00:43:49.423
In my digital world, we're
frequently doing A/B test, right?

00:43:49.424 --> 00:43:54.523
So we'll have a, you know, an audience
of 5% will get A and another five will

00:43:54.523 --> 00:43:57.745
get B, and then we look at some metric
like conversion rate, and then we'll…

00:43:58.095 --> 00:44:01.063
There'll, there'll be a, a hero and
a loser, and then the hero takes

00:44:01.063 --> 00:44:03.235
over the last 90% or whatever it is.

00:44:03.624 --> 00:44:05.105
Um, so now we can actually…

00:44:05.106 --> 00:44:09.804
But in that example, you, you kind of
messed up the people that, the humans

00:44:09.847 --> 00:44:13.442
that were the 5% that got the crappy,
the one, the one that didn't win, right?

00:44:13.442 --> 00:44:13.908
The loser.

00:44:14.175 --> 00:44:18.958
Um, so what's interesting is there could
be a world where you run it in a, in a…

00:44:19.263 --> 00:44:20.550
Also, it takes time, right?

00:44:20.551 --> 00:44:24.923
So this may be in the B2B context or even
consumer, sometimes it may take a week

00:44:24.924 --> 00:44:26.731
or two to like figure out your A/B split.

00:44:27.127 --> 00:44:31.172
So you guys do it, uh, you could do it
simulated instantaneously because you

00:44:31.174 --> 00:44:32.951
can always scale up a simulation, right?

00:44:33.322 --> 00:44:37.538
Uh, and then because it's j- you can di-
you can scale up a digital simulation.

00:44:37.852 --> 00:44:43.100
Um, and then you also don't have the
problem of the testing always has

00:44:43.101 --> 00:44:47.400
a, a set of real possible customers
that, that didn't see the best thing.

00:44:47.830 --> 00:44:51.416
Um, so it also eliminates that, whatever
the cost is of those two things.

00:44:51.921 --> 00:44:54.657
Um, you know, so that's,
that's pretty interesting.

00:44:54.663 --> 00:44:57.294
Have you guys thought about
commercializing that or?

00:44:57.823 --> 00:44:58.183
Yeah.

00:44:58.184 --> 00:45:01.497
There, there are several companies
that approached us with the idea

00:45:01.542 --> 00:45:04.916
of doing exactly that, and we're
working with a few of them.

00:45:05.097 --> 00:45:09.790
We also have, um, uh, uh,
implementations that we may wanna

00:45:09.811 --> 00:45:12.552
release, um, a- about these.

00:45:12.579 --> 00:45:14.165
So we are considering releasing.

00:45:14.389 --> 00:45:17.198
Uh, probably it's gonna happen
actually next week or so.

00:45:17.547 --> 00:45:25.814
So the, the, the idea is, um, it's
fascinating- I do think that, um…

00:45:25.839 --> 00:45:26.966
So there is two things.

00:45:26.977 --> 00:45:28.857
Uh, there is a-- There are some caveats.

00:45:29.418 --> 00:45:34.918
If you want to, um, AB tests will
ever remain the, the gold standard

00:45:35.013 --> 00:45:40.437
for proving causality and the-- this
technique of simulations, the best thing

00:45:40.437 --> 00:45:46.996
it can do, it can help you prioritize,
uh, um, like assets for AB testing and

00:45:46.996 --> 00:45:49.368
the, uh, which is not a small thing.

00:45:49.405 --> 00:45:52.910
So if you think about the difference
between the old world and the new

00:45:52.913 --> 00:45:57.272
world, is that with generative AIs,
you can create thousands, millions of

00:45:57.272 --> 00:45:59.627
assets, actually pretty, pretty quickly.

00:45:59.629 --> 00:46:01.898
And so you can't AB test
possibly all of them.

00:46:02.025 --> 00:46:05.079
Plus, as you said, you have
constraints, it's expensive.

00:46:05.440 --> 00:46:09.669
So even just prioritize a small
subset of those, I think it's

00:46:09.672 --> 00:46:11.620
already extremely interesting.

00:46:12.174 --> 00:46:18.364
Now, can you, uh, reconstruct the
exact effect size in the simulations

00:46:18.369 --> 00:46:22.334
that you would have when you run
the AB test, meaning the delta, the

00:46:22.334 --> 00:46:24.374
incremental revenue or whatever?

00:46:24.841 --> 00:46:27.088
Can it be estimated exactly?

00:46:27.465 --> 00:46:33.373
That's probably not because, um,
we already know that inferring

00:46:33.586 --> 00:46:40.190
causality from observational data,
uh, there is tons of theory that

00:46:40.251 --> 00:46:41.750
tells you that's super hard to do.

00:46:42.081 --> 00:46:45.798
But I do think that establishing these
correlations and prioritizing out of

00:46:45.892 --> 00:46:50.428
a very large set of AB test, the top
ten percent or five percent is actually

00:46:50.428 --> 00:46:54.860
probably possible to do, and then you
would have to run the AB test downstream.

00:46:55.790 --> 00:46:57.643
So that's what we want to achieve.

00:46:57.708 --> 00:47:03.243
Now, there are difficulties because the
question is, when you take an AB test

00:47:03.282 --> 00:47:09.923
and you expose your audience online to
that AB test- The customer traffic that

00:47:09.924 --> 00:47:12.132
you collect is not distributed at random.

00:47:12.132 --> 00:47:14.785
It's not generally the
entire US populations.

00:47:15.205 --> 00:47:19.685
So even how you instrument your offline
simulations to be representative

00:47:20.329 --> 00:47:25.526
of, um, the audience that you
will see once you do the AB test

00:47:25.526 --> 00:47:27.497
online is complicated in itself.

00:47:27.527 --> 00:47:31.201
But actually here, we are thinking
about using Bayesian method and

00:47:31.334 --> 00:47:37.628
techniques to up-sample and down-sample
specifically, uh, subsets of the synthetic

00:47:37.628 --> 00:47:43.250
consumers to be as close as possible
to, uh, the, uh, online population.

00:47:43.251 --> 00:47:46.555
So there's a calibration step that's
still we need to figure it out how

00:47:46.587 --> 00:47:47.951
to make it happen, I understand.

00:47:48.864 --> 00:47:49.160
Yeah.

00:47:49.193 --> 00:47:53.760
You know how, um, so, so, so a lot
of the ad systems have a, uh, an

00:47:53.773 --> 00:47:56.826
audience lookalike function where you
can upload your audience, and then

00:47:56.828 --> 00:48:00.762
they have enough data that they can
create a, a, a map of, you know, that.

00:48:01.210 --> 00:48:03.715
They must be doing Bayesian
stuff on their side for that.

00:48:03.715 --> 00:48:08.822
But yeah, um, within this black
box piece, th- this simulation,

00:48:08.824 --> 00:48:12.201
there could be a way where you like
have a map, you know, m- mimic my

00:48:12.201 --> 00:48:13.593
audience functionality down the road.

00:48:14.035 --> 00:48:14.992
That's, that's a very good…

00:48:15.039 --> 00:48:16.961
That's a, that's a very interesting idea.

00:48:17.445 --> 00:48:17.673
Yeah.

00:48:17.766 --> 00:48:20.327
And the idea is that,
um, how do you do that?

00:48:20.878 --> 00:48:22.491
We're still thinking about this, but how-

00:48:22.814 --> 00:48:24.914
If you're Facebook, you, you
have all the data, right?

00:48:24.958 --> 00:48:25.987
That's, that's how they do it.

00:48:26.311 --> 00:48:26.774
Correct.

00:48:26.809 --> 00:48:30.833
And in- indeed there, there are
startup that actually approached us.

00:48:30.845 --> 00:48:33.722
They had a very clever way of
actually getting through this data.

00:48:33.785 --> 00:48:38.120
So, um, we, we are not under NDA with
them, so I can, I can disclose this,

00:48:38.472 --> 00:48:43.173
but basically they're thinking about
gamified systems where you, um, maybe

00:48:43.175 --> 00:48:47.853
as a consumer, you link your personal
data, you get something reward out of

00:48:47.853 --> 00:48:51.656
it, and then this personal data is used
to create a synthetic representation

00:48:51.656 --> 00:48:56.990
of you that the brand can interface
with to get recommend- uh, to recommend

00:48:56.993 --> 00:49:00.091
you personalized products and to ask
you questions about new products.

00:49:00.529 --> 00:49:05.577
And so you will, in the future, I think
you will have a world where I can, uh,

00:49:05.978 --> 00:49:11.549
get some money by sharing my personal data
and have, have a digital avatar of myself

00:49:11.980 --> 00:49:14.415
interact with any brand in the world.

00:49:14.739 --> 00:49:20.375
And the brand can ask me in a privacy
compliant, safe way, questions about my

00:49:20.375 --> 00:49:22.107
preferences, my consumer preferences.

00:49:22.108 --> 00:49:25.742
I get money out of that, and I
get great recommendations, which

00:49:25.742 --> 00:49:26.945
is the best of both worlds.

00:49:27.626 --> 00:49:29.540
Huh, that's crazy.

00:49:29.542 --> 00:49:32.399
So you could create a digital clone
and have it go make money for you?

00:49:33.852 --> 00:49:34.329
I like that.

00:49:34.558 --> 00:49:35.076
Sign me up.

00:49:35.107 --> 00:49:35.806
I'm, I'm, uh…

00:49:35.861 --> 00:49:36.801
Sign me up to the beta.

00:49:36.919 --> 00:49:38.998
Sounds like I could make money
while I sleep, which is always good.

00:49:42.196 --> 00:49:43.321
Um, cool.

00:49:43.322 --> 00:49:43.905
I wanna be…

00:49:43.939 --> 00:49:44.965
I, I know you're busy.

00:49:45.016 --> 00:49:47.557
Um, the…

00:49:48.805 --> 00:49:50.268
All right, so we talked about this.

00:49:50.275 --> 00:49:52.782
It sounds like you've got some
cool stuff coming up next week.

00:49:52.782 --> 00:49:55.453
Anything else about the paper you
wanna share before we wrap up?

00:49:56.520 --> 00:50:01.326
Um, I, I would say that, uh, y-
yeah, I mean, read, read the paper.

00:50:01.794 --> 00:50:05.715
So we still, again, we got submitted
to, uh, one of the big conferences next.

00:50:06.275 --> 00:50:08.342
Uh, check the code, reach out to us.

00:50:08.344 --> 00:50:12.548
I mean, if you are interested in
these topics, uh, um, we are always

00:50:12.571 --> 00:50:17.128
happy to understand new use cases
of s- some of these technologies and

00:50:17.128 --> 00:50:18.645
keep developing these technologies.

00:50:18.661 --> 00:50:23.531
So reach out and, uh, share your
ideas, thoughts, and we can probably

00:50:23.532 --> 00:50:24.721
work on these problems together

00:50:26.556 --> 00:50:26.828
Cool.

00:50:27.041 --> 00:50:32.897
And if, uh, people want to find you
online, is it, um, do you have a place

00:50:32.913 --> 00:50:38.642
you- so the website is, uh, pymc-labs.com,

00:50:38.661 --> 00:50:40.361
and I'll, I'll put that in the show notes.

00:50:40.407 --> 00:50:42.166
Um, so that's the company website.

00:50:42.480 --> 00:50:45.211
Um, but you look like you're
pretty active on LinkedIn.

00:50:45.219 --> 00:50:46.420
Is that a good place to find you?

00:50:46.420 --> 00:50:46.430
Yeah,

00:50:46.430 --> 00:50:46.586
yeah.

00:50:46.591 --> 00:50:50.025
Feel, feel, feel, uh, uh, free to
reach out and DM me on LinkedIn.

00:50:50.340 --> 00:50:51.869
I'll very responsive there.

00:50:52.716 --> 00:50:53.404
Yep, yep.

00:50:53.437 --> 00:50:54.963
And then, um, cool.

00:50:54.964 --> 00:50:57.500
And that's, so when you post new
papers, they'll show up kind of on your

00:50:57.504 --> 00:51:01.845
LinkedIn, and I think it was on, on your
website in a research area, if I recall.

00:51:01.846 --> 00:51:04.609
You guys have a nice little library of
different research you've done there as

00:51:04.610 --> 00:51:04.685
well.

00:51:04.693 --> 00:51:05.117
Correct.

00:51:05.168 --> 00:51:05.763
Uh, correct.

00:51:05.763 --> 00:51:08.687
We have a pretty extensive
research blog, and we publish

00:51:08.691 --> 00:51:10.521
continuously any content or material.

00:51:10.911 --> 00:51:14.157
Uh, we have benchmarks as
well, so we, uh, we publish the

00:51:14.162 --> 00:51:17.334
benchmark about LLMs and pricing.

00:51:17.383 --> 00:51:22.057
So which LLMs are better at predicting
pricing, which LLMs are most strategic

00:51:22.676 --> 00:51:27.052
about pricing competitions, you know,
like the Price Is Right and so on.

00:51:27.081 --> 00:51:30.711
We have a little, uh, simulation
of the setup of, for The

00:51:30.711 --> 00:51:32.668
Price Is Right, the TV show.

00:51:33.274 --> 00:51:38.441
And we, um, as new models come up, we
make them compete into simulations to see

00:51:38.450 --> 00:51:40.333
which one is, we make it a leaderboard.

00:51:40.609 --> 00:51:43.005
So, so Price Is Right has,
like, six different games.

00:51:43.005 --> 00:51:45.857
Are you just doing the closest without
going over the basic Price Is Right

00:51:45.858 --> 00:51:49.174
or like the little, the Himalayan guy
or like wh- which, which one of the-

00:51:49.369 --> 00:51:50.706
It's a, it's a good question.

00:51:50.706 --> 00:51:54.326
Actually, I should, I should ask Alan
who has actually implemented this.

00:51:54.476 --> 00:51:55.046
I, um-

00:51:55.074 --> 00:51:56.447
It's probably closest without going over.

00:51:56.447 --> 00:51:56.786
That- I, I

00:51:56.976 --> 00:51:58.676
think it's- … that's what most
people- I think it's the basic one

00:51:58.677 --> 00:52:00.402
that they have implemented so far.

00:52:00.411 --> 00:52:03.960
But the, I, I mean, like I, I
thought it was a very fun exercise.

00:52:04.010 --> 00:52:08.228
I think benchmark is generally
a big topics in, uh, uh, LLMs.

00:52:08.548 --> 00:52:12.642
I do think if we want LLMs
to become better representing

00:52:12.658 --> 00:52:16.747
humans, uh, then we also need
benchmarks for something like this.

00:52:16.795 --> 00:52:21.454
And publishing this Price Is Right
was a way of helping researcher

00:52:21.471 --> 00:52:24.858
of, uh, traditional models
developing this type of competition.

00:52:25.778 --> 00:52:26.030
Yeah.

00:52:26.069 --> 00:52:29.384
Very relevant to our world of, a lot
of, lot of people in retail are using,

00:52:29.418 --> 00:52:34.008
thinking about using these for different
pricing, you know, systems of, you know,

00:52:34.484 --> 00:52:36.989
how do you, how do you do dynamic pricing?

00:52:36.991 --> 00:52:39.093
How do you think about,
uh, regional pricing?

00:52:39.106 --> 00:52:43.560
You know, like a sweater in, uh, Los
Angeles may be worth less than one

00:52:43.560 --> 00:52:44.987
in New York or something like that.

00:52:46.425 --> 00:52:46.926
Interesting.

00:52:46.960 --> 00:52:50.091
So, uh, so when a new model
comes out, like, y- you guys

00:52:50.333 --> 00:52:53.440
run it through your little, your
battery of different things and…

00:52:53.522 --> 00:52:53.784
Okay.

00:52:54.036 --> 00:52:54.816
So Gemini 3.0

00:52:54.816 --> 00:52:57.292
is rumored to be coming out this week, so
that's gonna be a big week for you guys.

00:52:57.332 --> 00:52:58.230
You, you're, you ready?

00:52:58.977 --> 00:53:00.594
Gonna, gonna run it through the paces?

00:53:00.733 --> 00:53:01.852
Yeah, we're gonna do that.

00:53:02.007 --> 00:53:05.198
We still need to run on GPT-5.1.

00:53:05.321 --> 00:53:07.117
We, we're gonna publish the results soon.

00:53:07.617 --> 00:53:10.253
So yeah, a lot of
exciting stuff coming up.

00:53:10.979 --> 00:53:11.209
Yeah.

00:53:11.267 --> 00:53:14.100
This is what's fun about this
LLM world, a lot of job security.

00:53:14.143 --> 00:53:16.521
Everything's changing basically
every week, all the time.

00:53:16.545 --> 00:53:18.903
So, awesome.

00:53:18.945 --> 00:53:24.109
Luca, thanks for taking the time to come
and, uh, uh, answer my novice questions

00:53:24.138 --> 00:53:26.892
about Likert's and, uh, your paper.

00:53:26.892 --> 00:53:31.329
I really enjoyed it, and look
forward to my digital, uh,

00:53:31.374 --> 00:53:32.885
clone making some extra money.

00:53:33.573 --> 00:53:34.026
For sure.

00:53:34.051 --> 00:53:35.711
Thank you very much,
Scott, for inviting me.

00:53:35.711 --> 00:53:36.314
Have a good day.

00:53:36.530 --> 00:53:36.894
Bye-bye.

00:53:38.216 --> 00:53:44.061
Follow, rate, and share to stay up
to date with Retailgenic, your guide

00:53:44.076 --> 00:53:47.511
to the future of AI shopping agents.