WEBVTT

NOTE
This file was generated by Descript 

00:00:00.000 --> 00:00:03.849
Judith: Welcome to Berry's In the
Interim podcast, where we explore the

00:00:03.849 --> 00:00:07.610
cutting edge of innovative clinical
trial design for the pharmaceutical and

00:00:07.610 --> 00:00:09.869
medical industries, and so much more.

00:00:10.579 --> 00:00:11.260
Let's dive in.

00:00:12.688 --> 00:00:13.198
Scott Berry: All right.

00:00:13.198 --> 00:00:14.218
Welcome everybody.

00:00:14.218 --> 00:00:20.218
Back to, in the interim, uh, I'm your
host for the, maybe the first minute here,

00:00:20.218 --> 00:00:23.218
Scott Berry, and welcome everybody back.

00:00:23.218 --> 00:00:26.968
And I have, uh, an interesting
discussion for you today.

00:00:26.968 --> 00:00:30.418
I'm gonna turn it over to a guest
host and I'll introduce him in a

00:00:30.418 --> 00:00:32.638
minute and I'll introduce the podcast.

00:00:33.208 --> 00:00:37.228
But the, the, the other guests that
have joined me today are Don Berry,

00:00:37.258 --> 00:00:39.328
who you've seen on in the interim.

00:00:39.748 --> 00:00:45.808
And as far as this concerns, Don
has 50 years plus of experience

00:00:45.808 --> 00:00:47.788
of Bayesian clinical trials.

00:00:48.208 --> 00:00:53.968
And Lindsay Berry, who is joining us,
who has been at Berry Consultants now

00:00:53.968 --> 00:00:56.638
for seven years, do I have that right?

00:00:57.113 --> 00:00:57.613
Lindsay Berry: Six years.

00:00:57.898 --> 00:01:04.138
Scott Berry: Six years, uh, feels
like seven, uh, and has gr a great

00:01:04.138 --> 00:01:08.458
deal of experience, uh, in this
time in Bayesian clinical trials.

00:01:08.668 --> 00:01:12.988
And Nick Berry, who's also been
at Berry Consultants now for six

00:01:13.228 --> 00:01:14.548
Nick Berry: Six and a half years.

00:01:14.548 --> 00:01:14.968
Yeah.

00:01:15.028 --> 00:01:19.888
Scott Berry: and six, and has more
experience than Lindsay, as you can

00:01:19.888 --> 00:01:22.048
hear in, in Bayesian clinical trials.

00:01:22.948 --> 00:01:29.038
So this episode is about Bayesian
clinical trials, and the, the theme of

00:01:29.038 --> 00:01:37.048
this is that the siblings and grandson
and my son, Cooper Berry, who is an

00:01:37.048 --> 00:01:42.298
undergraduate student at Pitzer College,
taking a course in the history of

00:01:42.298 --> 00:01:47.548
mathematics, he wants to do a project on.

00:01:47.998 --> 00:01:53.908
The use of Bayesian statistics in
clinical trials and investigate that.

00:01:53.908 --> 00:01:58.528
And he just so happens to have a wealth
of experience within his own family.

00:01:58.828 --> 00:02:01.408
So he is going to conduct this podcast.

00:02:01.408 --> 00:02:06.868
He's gonna be the host and explore
the, the impact and the future

00:02:06.928 --> 00:02:09.148
of Bayesian in clinical trials.

00:02:09.478 --> 00:02:14.428
So Cooper, welcome to in the
interim, and this episode is yours.

00:02:15.358 --> 00:02:15.928
Cooper Berry: Thank you.

00:02:16.558 --> 00:02:17.608
Thank you for having me on.

00:02:17.638 --> 00:02:19.408
Uh, I'm very excited for this.

00:02:19.978 --> 00:02:26.278
Um, and yeah, I, uh, I think we
can start right ahead with, um, uh,

00:02:26.278 --> 00:02:29.758
a few questions for, uh, for Don.

00:02:30.358 --> 00:02:35.548
So Don, you're, um, your
dissertation work, it was with a

00:02:35.548 --> 00:02:37.648
very famous Bayesian statistician.

00:02:38.128 --> 00:02:42.478
Um, I think you referred to him as one
of the founders of Bayesian Statistics.

00:02:42.688 --> 00:02:43.678
Jimmy Savage.

00:02:44.248 --> 00:02:50.398
Um, he was involved in a lot of different
areas, but maybe not clinical trials.

00:02:51.268 --> 00:02:54.748
What would Jimmy Savage's
views of the Bayesian approach

00:02:54.748 --> 00:02:56.638
today in clinical trials be?

00:03:00.488 --> 00:03:05.323
Don Berry: So, uh, he, uh,
first this, this was at Yale.

00:03:05.533 --> 00:03:07.573
He was my advisor.

00:03:08.203 --> 00:03:14.143
I came to him with a problem, uh,
that, uh, I thought was interesting.

00:03:15.313 --> 00:03:20.773
Uh, and it wasn't specific, although
we did discuss the possibility

00:03:20.773 --> 00:03:25.033
and I kept thinking that this is
applicable to clinical trials,

00:03:25.033 --> 00:03:26.773
this is what clinical trials are.

00:03:28.363 --> 00:03:34.873
Um, and then I got ridiculed by the
people who actually run clinical trials.

00:03:36.118 --> 00:03:40.978
For many years, we're talking
about the 1970s and eighties.

00:03:41.698 --> 00:03:46.558
Uh, by the time we got to the nineties,
they had, they were used to me and

00:03:46.558 --> 00:03:48.448
some were beginning to listen to me.

00:03:50.038 --> 00:03:54.178
Um, but Jimmy Savage, what
would Jimmy Savage have said?

00:03:54.178 --> 00:03:59.998
Well, he said a lot in reading my
dissertation, the five versions of it.

00:04:00.988 --> 00:04:07.588
Um, and none of that had anything
specific to do with clinical trials.

00:04:08.428 --> 00:04:15.598
Uh, it was, uh, decision making,
sequential decision making, um, as

00:04:15.598 --> 00:04:18.448
very simple to say what the problem is.

00:04:19.138 --> 00:04:25.588
I suppose you have two coins,
um, and you don't know what the

00:04:25.588 --> 00:04:29.578
probabilities are of heads every,
but you're given the opportunity to

00:04:29.578 --> 00:04:33.268
toss the coin, say a hundred tosses.

00:04:35.083 --> 00:04:40.093
Every time you get
heads, you get a dollar.

00:04:41.563 --> 00:04:45.283
Every time you get tails, you get nothing.

00:04:46.813 --> 00:04:50.593
So think in terms of clinical trials,
it's, uh, you have a treatment,

00:04:50.593 --> 00:04:55.483
you're investigating or you, and
you compare it to another treatment.

00:04:55.483 --> 00:05:01.183
So you have these two coins, and if
you choose a treatment, that's good.

00:05:01.543 --> 00:05:06.013
The patient lives, if you choose the
treatment, that's bad, the treatment,

00:05:06.583 --> 00:05:08.413
the, the patient does not live.

00:05:09.583 --> 00:05:17.983
Um, and what your goal is, is to treat as
many patients as effectively as possible.

00:05:17.983 --> 00:05:21.433
Get as many heads as possible,
make as much money as possible.

00:05:21.913 --> 00:05:29.473
If you're Google, you want to use, uh,
ads that attract when somebody clicks

00:05:29.473 --> 00:05:33.043
on an ad, Google makes money when you.

00:05:33.718 --> 00:05:34.888
Don't click on the ad.

00:05:34.888 --> 00:05:38.578
Google knows you're looking at
it and you didn't click on it.

00:05:39.058 --> 00:05:41.638
So you don't, Google Gut
does not make any money.

00:05:42.748 --> 00:05:48.058
Um, and they wanna make as much money as
possible, but they also want to, and this

00:05:48.058 --> 00:05:56.098
is the critical issue, they also want to
learn about what is the better therapy.

00:05:56.188 --> 00:05:58.858
Not only treat patients effectively,

00:06:01.288 --> 00:06:07.048
but also learn about the
treatments so that we can treat

00:06:07.378 --> 00:06:09.568
patients later effectively.

00:06:10.048 --> 00:06:12.628
Now, this is just to set the context.

00:06:12.658 --> 00:06:20.578
1979, uh, there was a Belmont report in
the United States that was a collection

00:06:20.578 --> 00:06:23.548
of people they met, met Belmont, I guess.

00:06:24.298 --> 00:06:29.128
Um, and they put together
the ethics of clinical trials

00:06:30.058 --> 00:06:32.638
and what they said in this.

00:06:33.133 --> 00:06:37.153
Was very counter to what I just
said about bandit problems.

00:06:38.143 --> 00:06:44.503
They said the purpose of of, uh,
doing clinical trials is only

00:06:44.503 --> 00:06:48.193
to learn not to treat patients.

00:06:48.193 --> 00:06:50.113
And they're very ex explicit about that.

00:06:51.073 --> 00:06:58.663
And so part of my harangue then
and continuing is we can do both.

00:06:58.663 --> 00:07:04.333
We can combine clinical
practice and clinical treatment.

00:07:05.503 --> 00:07:10.183
Now, doing that is surprisingly difficult.

00:07:11.473 --> 00:07:16.543
And the reason it's difficult
is there are two reasons to

00:07:17.653 --> 00:07:20.053
pick a coin over the other coin.

00:07:21.103 --> 00:07:29.023
One is the coin is looking better
in terms of its effectiveness.

00:07:29.323 --> 00:07:30.733
The, the, the.

00:07:31.318 --> 00:07:33.088
Ability to generate ahead.

00:07:34.078 --> 00:07:44.998
Um, and the other is to learn so that
we have information in the long run

00:07:44.998 --> 00:07:47.038
that's gonna help us treat patients.

00:07:47.038 --> 00:07:54.418
In the long run, you know, the last
50 patients would be benefiting of the

00:07:54.418 --> 00:07:57.508
hundred of the a hundred, uh, coin tosses.

00:07:58.048 --> 00:08:02.548
The last 50 are gonna
benefit from the first 50.

00:08:03.508 --> 00:08:07.348
And the first 50 are gonna help
in benefiting the latter 50.

00:08:07.378 --> 00:08:13.408
But there's no 50, there's no, you
know, discreet aspect to the thing.

00:08:13.408 --> 00:08:20.038
It's that you're you at, every time
you're going to toss the coin, you're

00:08:20.038 --> 00:08:22.168
confronted with just that decision.

00:08:22.828 --> 00:08:30.298
But that decision has a ramification on
the rest of the story and the reason.

00:08:30.673 --> 00:08:35.593
It's difficult is because you,
we, I talk about information,

00:08:35.623 --> 00:08:37.243
I talk about treating patients.

00:08:39.013 --> 00:08:46.633
Those two things are not,
um, uh, uh, separable.

00:08:46.873 --> 00:08:49.843
You can't look at one say,
oh, here's the information.

00:08:50.773 --> 00:08:54.553
You can talk about a standard
deviation, for example.

00:08:55.213 --> 00:09:01.843
Um, you can talk, you can talk for
the, um, the, uh, treating patients

00:09:01.843 --> 00:09:05.563
about the mean of the distribution,
of the probability of heads.

00:09:06.583 --> 00:09:09.883
Um, but you can't separate them out.

00:09:11.293 --> 00:09:15.223
Um, and so my dis my dissertation.

00:09:16.933 --> 00:09:20.923
So, so, so picture your problem.

00:09:21.673 --> 00:09:27.283
You are now, you've now treated
20 patients and it turns out that.

00:09:28.183 --> 00:09:34.393
Coin one was used 12 times and
Coin two was used eight times.

00:09:35.293 --> 00:09:42.163
Uh, and of the 12 times there
were, uh, nine successes.

00:09:42.853 --> 00:09:48.763
And of the eight times there
were, uh, two successes.

00:09:50.203 --> 00:09:51.283
Uh, what do you do?

00:09:52.513 --> 00:09:52.993
Well,

00:09:53.143 --> 00:09:56.083
Cooper Berry: when is it worth
taking flipping coin Two, even

00:09:56.083 --> 00:09:58.393
though it's the worst option
to get more information on it,

00:09:59.743 --> 00:10:00.673
Don Berry: say it again,

00:10:01.063 --> 00:10:02.023
Cooper Berry: when is it worth, um,

00:10:04.183 --> 00:10:06.643
flipping coin two, even though
it's had worse data, just

00:10:06.643 --> 00:10:07.663
to get more information on

00:10:07.723 --> 00:10:10.003
Don Berry: that's an excellent question.

00:10:10.903 --> 00:10:12.763
The worth of Coin two.

00:10:13.603 --> 00:10:18.583
Uh, we don't know as much about
Coin Two as we do about Coin One.

00:10:20.413 --> 00:10:25.393
Uh, and it would be good to get
information about, uh, coin.

00:10:25.843 --> 00:10:29.863
Two better to get information
about Coin Two than Coin One, even

00:10:29.863 --> 00:10:32.083
though Coin one is looking better.

00:10:33.763 --> 00:10:40.423
You know, I would want to use Coin One
if I could, but suppose I use Coin One

00:10:40.573 --> 00:10:43.543
and then I never go back to Coin two.

00:10:46.333 --> 00:10:48.103
That would be bad

00:10:49.633 --> 00:10:54.763
because even though the, you know,
the point estimate of, uh, coin uh,

00:10:54.793 --> 00:11:02.173
two was less than Coin one doesn't
mean that the actual probability

00:11:02.173 --> 00:11:04.243
of head zone coin two was less.

00:11:04.753 --> 00:11:09.523
And so if we never go back to
it, that's opportunity lost.

00:11:10.003 --> 00:11:15.073
And so that's a negative,
uh, in this circumstance.

00:11:15.643 --> 00:11:18.613
I I haven't done the, uh, calculations.

00:11:18.613 --> 00:11:21.553
The calculations are
really, uh, difficult.

00:11:22.303 --> 00:11:22.813
Um.

00:11:24.298 --> 00:11:30.268
But, uh, I think that Coin one
is indeed appropriate because the

00:11:30.268 --> 00:11:36.868
information here is not enough to,
uh, make it worthwhile to treat the

00:11:36.868 --> 00:11:39.688
current patients potentially less

00:11:40.198 --> 00:11:40.978
effectively.

00:11:41.623 --> 00:11:45.763
Scott Berry: but that, that, that
depends on your horizon, I assume.

00:11:45.763 --> 00:11:46.153
So if it

00:11:46.153 --> 00:11:50.038
was 20, if it was 21 patients,
the answer's trivial.

00:11:50.038 --> 00:11:50.893
You do one.

00:11:51.193 --> 00:11:54.733
If it's a million patients, you
might do something different.

00:11:54.868 --> 00:11:56.548
Don Berry: that's where
I was going, Scott.

00:11:57.118 --> 00:11:57.448
Uh,

00:11:57.628 --> 00:11:57.918
Scott Berry: Okay.

00:11:58.588 --> 00:12:01.138
Don Berry: and, uh, yes, exactly.

00:12:01.768 --> 00:12:11.008
Um, and with the, if, if it's a
thousand patients, um, I, I remember,

00:12:11.038 --> 00:12:16.468
uh, Herman Chernoff was very
famous statistician who, um, uh,

00:12:16.483 --> 00:12:19.663
Scott Berry: Oh, so for, for
this podcast, was he a Bayesian,

00:12:23.008 --> 00:12:26.188
Don Berry: He was not a Bayesian,

00:12:26.818 --> 00:12:27.018
Scott Berry: Okay.

00:12:27.268 --> 00:12:32.098
Don Berry: he said he, he said
to me that he was not a Bayesian.

00:12:32.693 --> 00:12:32.983
Scott Berry: Okay.

00:12:33.538 --> 00:12:46.918
Don Berry: Um, and using, uh, uh, Jim
Berger as a, uh, famous Bayesian, um, Jim

00:12:46.918 --> 00:12:51.808
said that anybody who says he's a Bayesian
is a Bayesian, although he was telling me

00:12:51.808 --> 00:12:58.138
about this one person he didn't think was
a Bayesian, since my definition is anyone

00:12:58.168 --> 00:13:04.708
is, is a Bayesian, if they say they're a
Bayesian, um, except for this one person.

00:13:05.293 --> 00:13:05.583
Scott Berry: Okay.

00:13:07.198 --> 00:13:12.178
Don Berry: Uh, so Herman Schoff
said that he was not a Bayesian,

00:13:12.898 --> 00:13:17.728
but he couldn't do what he wanted
to do without being a Bayesian.

00:13:18.313 --> 00:13:22.693
Because in the Bayesian setting,
you are able to ask yourself,

00:13:23.083 --> 00:13:29.683
what is the probability that
I get a success on Coin One?

00:13:30.283 --> 00:13:33.613
What is the probability that
I get a success on Coin two?

00:13:34.123 --> 00:13:39.763
And then if I pick one of those and
go to tomorrow where I get to use

00:13:39.763 --> 00:13:43.423
another coin, uh, I update that.

00:13:44.293 --> 00:13:47.383
So every time I make an
observation, I can update it.

00:13:49.543 --> 00:13:52.423
And Chernoff wanted to do that.

00:13:53.173 --> 00:13:57.103
And he, in this one problem he
wanted to calculate, you know, so

00:13:57.103 --> 00:14:00.043
how many times would I use Coin two?

00:14:01.423 --> 00:14:10.003
Um, if it, it is, uh, I've seen two out
of eight, and then I'm going to keep

00:14:10.003 --> 00:14:16.963
using, uh, let's say the horizon is big,
and so it tells me I should use Coin two.

00:14:17.653 --> 00:14:24.163
How many more failures on Coin two should
I accept before I switch back to Coin one?

00:14:25.333 --> 00:14:31.363
And that, you know, he was advertising
to me the calculations that he got.

00:14:32.113 --> 00:14:37.453
Um, but it was precisely the
issue that Scott mentioned that

00:14:37.453 --> 00:14:40.273
it's, it's the horizon anyway.

00:14:40.273 --> 00:14:46.333
So this whole idea was anathema
to the clinical trial community.

00:14:47.203 --> 00:14:55.303
Uh, I was aghast when I actually, uh,
decided that the only thing I could

00:14:55.303 --> 00:14:57.883
do to influence them was to join them.

00:14:58.273 --> 00:14:59.443
And so I did that.

00:15:00.463 --> 00:15:02.683
Um, I, uh,

00:15:03.538 --> 00:15:04.948
Scott Berry: well, let's, let's hang on.

00:15:04.948 --> 00:15:07.738
Let's, let's, let's turn
it back to, to, to coop.

00:15:07.933 --> 00:15:11.288
Don Berry: okay, I, I got, so
what would, what would Savage have

00:15:11.288 --> 00:15:15.163
thought about, uh, randomization?

00:15:16.738 --> 00:15:18.508
And read aways clinical trials.

00:15:19.108 --> 00:15:19.558
Uh,

00:15:21.958 --> 00:15:25.468
he would, he would've liked my approach.

00:15:26.368 --> 00:15:32.668
He did like my approach, uh, uh,
he was mostly interested in, in

00:15:32.668 --> 00:15:38.818
the theorems and, and results in
the mathematics of, of my approach.

00:15:38.818 --> 00:15:39.028
And,

00:15:39.568 --> 00:15:40.378
you know, the First,

00:15:40.768 --> 00:15:47.068
the first draft was 10 pages and
he said, you know, this is great.

00:15:47.698 --> 00:15:48.958
Maybe we can do more.

00:15:49.828 --> 00:15:56.008
And the final draft we, I published
in the analysts of mathematical

00:15:56.008 --> 00:15:59.788
statistics, uh, was 27 journal pages.

00:16:00.658 --> 00:16:04.648
Uh, and if I tell you what the results
were, you would say that, well,

00:16:04.648 --> 00:16:09.598
that sounds trivial, but the trivial
turns out to be very difficult.

00:16:11.038 --> 00:16:13.348
Uh, and, and in terms of.

00:16:14.063 --> 00:16:15.003
I'm, I'm, I'm gonna,

00:16:15.358 --> 00:16:16.063
Scott Berry: so, so, so

00:16:16.063 --> 00:16:16.723
hang on Don.

00:16:16.753 --> 00:16:17.263
Hang on, Don.

00:16:17.353 --> 00:16:18.343
We're, we're gonna run out.

00:16:18.763 --> 00:16:18.973
Yeah.

00:16:18.973 --> 00:16:20.018
Let's, let's turn back to Coop.

00:16:20.728 --> 00:16:25.258
Cooper Berry: And so what
about a clinical trial makes

00:16:25.558 --> 00:16:30.478
Bayesian, why was Bayesian looked
down upon in terms of clinical

00:16:30.598 --> 00:16:32.128
trials, especially in this time?

00:16:34.258 --> 00:16:36.838
Don Berry: so it depends on the person.

00:16:36.838 --> 00:16:45.868
If you are a pharmaceutical company or
if you are a, uh, national Institutes

00:16:45.868 --> 00:16:51.208
of Health and you're trying to do things
efficiently, that that's what they want.

00:16:51.778 --> 00:16:55.768
They want to get answers quickly.

00:16:56.338 --> 00:17:02.638
Even if the answer is, uh, uh, is,
you know, we, we drop, uh, treatment

00:17:02.728 --> 00:17:05.098
too because it wasn't effective.

00:17:05.098 --> 00:17:07.708
You wanna have a compelling
answer to that question.

00:17:08.248 --> 00:17:12.298
So it's all, uh, for the clinical trial.

00:17:12.838 --> 00:17:15.508
Community, uh, efficacy.

00:17:16.048 --> 00:17:22.048
And, uh, they, they actually, even today
they don't worry as much about treating

00:17:22.048 --> 00:17:28.978
patients as effectively and the clinical
trials, but this is coming because the

00:17:29.518 --> 00:17:35.818
clinic, you know, diseases are no longer
the same diseases that they were back

00:17:35.818 --> 00:17:42.388
when they are much more, we have much more
information about what this disease is.

00:17:42.418 --> 00:17:46.828
I mean, there used to be
one kind of breast cancer.

00:17:47.788 --> 00:17:54.268
Now we've decided that there are at least
four and we've separate them out into

00:17:54.268 --> 00:17:57.328
four, but in fact there are hundreds.

00:17:57.928 --> 00:18:05.158
And so as we continue, we've got to be
more and more interested in treating the

00:18:05.158 --> 00:18:07.558
patients effectively as well as learning.

00:18:07.648 --> 00:18:08.218
So back

00:18:08.218 --> 00:18:09.898
to my, back to my

00:18:09.943 --> 00:18:10.123
Cooper Berry: Yeah.

00:18:10.123 --> 00:18:14.893
Are there, are there ethical concerns
in giving a patient a treatment

00:18:14.893 --> 00:18:19.063
with a prior probability that's
lower than a different treatment?

00:18:22.543 --> 00:18:32.323
Don Berry: Yes, and the way the,
um, clinical trial community

00:18:32.323 --> 00:18:35.623
addresses it is to close their eyes.

00:18:37.843 --> 00:18:41.233
So they say, well, we've
got to do randomized trials

00:18:41.893 --> 00:18:46.693
and we've got to, you know, give
e equal number of patients to

00:18:46.723 --> 00:18:48.253
Therapy one and therapy two.

00:18:49.243 --> 00:18:56.353
Um, and, uh, we need a trial big
enough for us to conclude that one

00:18:56.353 --> 00:18:58.093
therapy is better than the other.

00:18:59.413 --> 00:19:08.623
Uh, and we're going to, uh, have a
placebo and randomize to placebo.

00:19:09.403 --> 00:19:14.293
Uh, and nobody's gonna know who
got the placebo and who didn't.

00:19:14.773 --> 00:19:17.383
And so they close their eyes,

00:19:18.763 --> 00:19:27.223
and I don't open my eyes, but I
open an algorithm's eyes, I build

00:19:27.223 --> 00:19:32.653
an algorithm which is looking
and does what is appropriate,

00:19:32.653 --> 00:19:35.143
say in the bandit, uh, approach.

00:19:35.143 --> 00:19:39.343
You know, we're trying to treat as many
patients as effectively as possible.

00:19:40.273 --> 00:19:45.133
There is absolutely ethical
concerns with doing that.

00:19:45.523 --> 00:19:55.933
And, um, it is, you know, it is the big
deal really, uh, in getting, uh, this kind

00:19:55.933 --> 00:20:00.853
of a, uh, of an approach in the clinical
practice, in the clinical trial practice.

00:20:01.603 --> 00:20:01.783
Cooper Berry: Yeah.

00:20:02.323 --> 00:20:03.343
Follow up question.

00:20:03.433 --> 00:20:04.063
So.

00:20:04.573 --> 00:20:10.813
Scott, um, leading into around the
year 2000 when you and Dawn, uh,

00:20:10.843 --> 00:20:15.403
founded Barry, uh, where was your
head at with the I SPY two trial?

00:20:15.403 --> 00:20:19.933
Where was your mind on what was
Bayesian statistics for you then and

00:20:19.933 --> 00:20:23.773
relating to clinical trials and what
was it like after the I SPY two trial?

00:20:25.483 --> 00:20:28.453
Scott Berry: Yeah, the, the, so
I, I'll go to 2000 and sort of

00:20:28.453 --> 00:20:29.983
the timeline of this a little bit.

00:20:29.983 --> 00:20:33.943
So, I I was trained as a Bayesian.

00:20:34.093 --> 00:20:39.793
I went to Carnegie Mellon University
where most of our courses were Bayesian.

00:20:39.793 --> 00:20:43.303
I, I, you know, it's a little
bit like a, a, a language.

00:20:43.388 --> 00:20:45.523
I, I think is a Bayesian I model.

00:20:45.523 --> 00:20:50.353
As a Bayesian, I can translate it
to Frequentist and what, what, you

00:20:50.353 --> 00:20:52.153
know, what does that mean relative so.

00:20:53.228 --> 00:20:55.513
I, I was doing sports problems.

00:20:55.513 --> 00:20:58.903
I was solving problems as a
Bayesian, and that's how I thought.

00:20:58.903 --> 00:21:02.923
And then Don and I started
Berry Consultants in 2000 and we

00:21:02.923 --> 00:21:06.853
started, I started, that was my
introduction to clinical trials.

00:21:07.063 --> 00:21:12.253
And as you see, Don has had 20 plus years
of experience in them ahead of that.

00:21:12.613 --> 00:21:18.643
So I jumped into a number of
projects with Don and it was how

00:21:18.643 --> 00:21:20.473
I modeled it was how I thought.

00:21:20.473 --> 00:21:23.503
And so it was a very natural
thing for me to go in there.

00:21:23.563 --> 00:21:26.863
And I was a little naive,
not understanding that the

00:21:26.863 --> 00:21:28.753
world was very frequentist.

00:21:29.113 --> 00:21:33.973
Uh, it was dominated by doing
frequentist clinical trials.

00:21:34.243 --> 00:21:36.673
And this was a very new thing.

00:21:37.003 --> 00:21:42.133
I Spy twos about 10 years after
that in the development was entirely

00:21:42.133 --> 00:21:44.053
a Bayesian trial within that.

00:21:44.473 --> 00:21:47.953
But we started this
really with the idea of.

00:21:48.553 --> 00:21:51.193
Can we do Bayesian clinical trials?

00:21:51.318 --> 00:21:52.868
And, and Don will tell you
this, I'll tell you this.

00:21:53.233 --> 00:21:55.573
We started this with, this might not work.

00:21:55.573 --> 00:21:57.763
You know, Barry consultants
might last a year.

00:21:58.003 --> 00:22:01.393
We don't know if, if Bayesian
clinical trials will make it.

00:22:01.693 --> 00:22:07.303
And now here we are, 25 plus years
later with more than 30 scientists

00:22:07.303 --> 00:22:10.663
and you've got four of them here
doing Bayesian clinical trials.

00:22:10.663 --> 00:22:14.803
So to some level it, it worked
despite the uphill battle.

00:22:18.463 --> 00:22:18.883
Cooper Berry: Mm-hmm.

00:22:19.768 --> 00:22:20.818
And how was the

00:22:20.908 --> 00:22:22.138
Don Berry: add, can I just add

00:22:22.228 --> 00:22:23.008
Cooper Berry: yeah, absolutely.

00:22:23.128 --> 00:22:23.218
Don Berry: one?

00:22:24.238 --> 00:22:28.498
So in, in doing the trial, a
lot more complicated because.

00:22:29.758 --> 00:22:32.278
You know, you have to
know what the data are.

00:22:32.788 --> 00:22:35.938
You have to build an algorithm
that's based on what the data are

00:22:35.938 --> 00:22:40.978
accumulating, and there's all kinds of
logistical issues associated with that.

00:22:41.428 --> 00:22:46.168
So at the very juncture that you're
talking about, uh, uh, Cooper,

00:22:48.688 --> 00:22:52.528
we had a new arm that
was coming in the trial.

00:22:52.528 --> 00:22:53.518
This is a multi-arm trial.

00:22:53.518 --> 00:22:54.448
It's not two arms.

00:22:54.448 --> 00:22:56.908
It was ended up to be 23 arms.

00:22:57.508 --> 00:23:03.508
Um, uh, a new arm was coming in
the trial and it had some unusual

00:23:03.508 --> 00:23:05.428
aspect that I wanted to modify.

00:23:05.938 --> 00:23:12.178
So I went to the people that I'd worked
with to build the, um, I was at M MD

00:23:12.178 --> 00:23:14.308
Anderson, I still am cancer center.

00:23:14.758 --> 00:23:22.288
And, uh, I worked with people there
to build the trial and, um, it was an

00:23:22.288 --> 00:23:25.468
extensive process of, uh, like two years.

00:23:26.143 --> 00:23:30.223
Because there were all kinds of things,
you know, the, the mathematics of course,

00:23:30.223 --> 00:23:36.223
but the, uh, the, the, the demonstrating
that, uh, we could actually do it.

00:23:36.973 --> 00:23:44.743
Uh, we had some pilot studies at MD
Anderson, um, and, uh, and, uh, more

00:23:44.743 --> 00:23:50.323
generally, and they had lost the code.

00:23:51.733 --> 00:23:54.073
You know, this happens.

00:23:55.213 --> 00:24:03.313
Um, and we, so I, I, I couldn't do
what I wanted to do, so I went to Scott

00:24:04.333 --> 00:24:07.993
and we did this on a pro bono basis.

00:24:08.563 --> 00:24:10.993
Uh, we never got paid for doing it.

00:24:11.173 --> 00:24:17.833
Um, and in view of my, uh, uh,
potential conflicts at MD Anderson,

00:24:17.833 --> 00:24:21.823
that was appropriate, uh, and I
said, Scott, this is what happened.

00:24:23.233 --> 00:24:25.003
He, I said, can you rewrite.

00:24:26.638 --> 00:24:30.058
Of course he had the benefit of
seeing what everybody else had

00:24:30.058 --> 00:24:32.938
done when was writing it, even
though we didn't have the code.

00:24:34.198 --> 00:24:34.528
And

00:24:34.528 --> 00:24:37.678
so he rewrote it, how
long it takes, Scott?

00:24:38.443 --> 00:24:40.243
Scott Berry: Uh, it was a long weekend.

00:24:44.113 --> 00:24:44.353
Yep.

00:24:45.643 --> 00:24:46.003
Yep.

00:24:46.798 --> 00:24:51.058
Don Berry: And so it was, and we've
now come, we've now come to the

00:24:51.058 --> 00:24:56.248
point where point, uh, we can do
that, uh, more and more quickly.

00:24:57.088 --> 00:24:59.518
So the building is not a small thing.

00:24:59.758 --> 00:25:05.008
It's a, it's a complicated process,
and running is a complicated process.

00:25:05.398 --> 00:25:12.568
But as you've heard, uh, we're doing
it, we're running these trials despite

00:25:12.568 --> 00:25:16.258
the logistical issues that,
uh, arise along the way.

00:25:16.258 --> 00:25:18.538
And with each trial, we learn new things.

00:25:20.203 --> 00:25:20.503
Cooper Berry: Yep.

00:25:20.923 --> 00:25:26.563
So when Scott, when you were at
Carnegie Mellon, the, uh, the Markoff

00:25:26.563 --> 00:25:30.733
Chain Monte Carlo methods, um, were
starting to appear, they were created.

00:25:30.733 --> 00:25:33.163
Uh, how did that change
the debate around Bayesian.

00:25:35.113 --> 00:25:37.453
Scott Berry: I, I think it
changed it dramatically actually.

00:25:37.513 --> 00:25:42.343
Um, a very famous statistician, Brad
Efrin, wrote a paper, why isn't everyone

00:25:42.343 --> 00:25:45.703
Bayesian recognizing the benefits of it?

00:25:45.943 --> 00:25:50.533
And largely the papers was, well, you
guys can't actually calculate anything.

00:25:50.893 --> 00:25:52.603
Uh, it's really too hard to do it.

00:25:52.603 --> 00:25:54.853
It's, it has philosophical advantages.

00:25:54.853 --> 00:25:57.193
It's nice, but it's really
hard to calculate things.

00:25:57.733 --> 00:25:58.933
And it was true.

00:25:59.203 --> 00:26:04.483
Um, many of the models we
use now in clinical trials

00:26:04.963 --> 00:26:07.273
in the eighties, we couldn't.

00:26:07.858 --> 00:26:09.928
We couldn't calculate those things.

00:26:10.318 --> 00:26:14.638
I, I set up a model for Don in
one of his trials and it ran in

00:26:14.638 --> 00:26:16.468
our basement for three months.

00:26:16.738 --> 00:26:20.308
Um, this was in 1983
or something like that.

00:26:20.308 --> 00:26:24.538
And we had to hope the electricity never
went out to calculate a very simple model.

00:26:25.048 --> 00:26:28.018
Uh, and so this was
very, very challenging.

00:26:28.018 --> 00:26:32.488
In the 1990s, the Markoff chain
Monte Carlo techniques came out where

00:26:33.148 --> 00:26:37.978
tied to the, the computing power
made the calculation of all of the

00:26:37.978 --> 00:26:43.558
Bayesian quantities very doable in
line with doing frequentist things

00:26:43.558 --> 00:26:48.028
and man, in some ways catching up
and passing frequentist things.

00:26:48.208 --> 00:26:51.868
So the whole debate about it
was kind of, well, yeah, it

00:26:51.868 --> 00:26:53.068
looks good, but you can't do it.

00:26:53.068 --> 00:26:56.098
Now all of a sudden we could do
it and we could do it really well.

00:26:56.338 --> 00:26:59.758
And it changed the debate between
Bayesian and Frequentist for

00:26:59.758 --> 00:27:02.038
sure, in every scientific field.

00:27:03.028 --> 00:27:06.628
Clinical trials a little bit slower,
but it absolutely changed the debate.

00:27:08.203 --> 00:27:09.613
Cooper Berry: Uh, follow up question.

00:27:09.613 --> 00:27:11.443
Why was it slower in clinical trials?

00:27:11.533 --> 00:27:13.933
What specifically about clinical
trials made it less applied?

00:27:16.228 --> 00:27:19.738
Scott Berry: Yeah, it's, uh, I think
you might get four answers if you

00:27:19.738 --> 00:27:24.808
ask us all four, but it, it, it
largely, it was clinical trials were

00:27:24.808 --> 00:27:30.058
done a particular way, and to do
it different is hard to change it.

00:27:30.238 --> 00:27:34.768
SaaS had been developed and so
there were models and calculation

00:27:34.768 --> 00:27:39.418
available in a frequentist way,
and this is the way it was done.

00:27:39.568 --> 00:27:44.308
And anytime you want to change an
industry and do it different, it's,

00:27:44.338 --> 00:27:46.348
there's, there's stasis there.

00:27:46.653 --> 00:27:48.418
It, it, it's challenging.

00:27:48.658 --> 00:27:56.188
So it, it, it, it was really just changing
the, the usual way it's done and a

00:27:56.188 --> 00:27:58.378
lot of people are set up in that way.

00:27:58.618 --> 00:28:02.088
I think that was, that
was a huge challenge to.

00:28:02.088 --> 00:28:02.158
it

00:28:04.973 --> 00:28:05.193
Don Berry: You

00:28:05.198 --> 00:28:06.118
know, the, the, the big,

00:28:06.658 --> 00:28:09.118
the elephant in this
room is the regulators.

00:28:09.688 --> 00:28:15.298
And for many years, the companies
and others would say, what will the

00:28:15.298 --> 00:28:17.278
FDA think, what will the FDA think?

00:28:18.238 --> 00:28:26.248
And so, uh, during the past, uh, 20 or
so years, uh, changed the FDA changed

00:28:26.728 --> 00:28:33.358
their attitudes and some personnel, and
they have become very much positive.

00:28:34.093 --> 00:28:35.173
About doing this.

00:28:35.173 --> 00:28:40.573
And so even in, in so-called phase III
trials, the trials that are pivotal, that

00:28:40.573 --> 00:28:49.243
are used for, for approving drugs, um,
so it, it, it, it, it's, it's changing.

00:28:49.243 --> 00:28:50.623
It's, it's developing.

00:28:51.823 --> 00:28:54.103
Um, back to you,

00:28:54.343 --> 00:28:54.883
Cooper Berry: Yeah.

00:28:54.883 --> 00:28:58.333
The, so it was 2010, is that correct?

00:28:58.333 --> 00:29:02.893
When the FDA established guidelines
for Bayesian clinical trials,

00:29:04.243 --> 00:29:06.043
Don Berry: it's a complicated question.

00:29:06.073 --> 00:29:10.783
Uh, and no, um, it
depends what you mean by

00:29:10.783 --> 00:29:11.443
the FDA.

00:29:11.443 --> 00:29:16.453
The FDA has three distinct pieces.

00:29:17.173 --> 00:29:22.063
Uh, one is the Center for Devices,
another is the Center for Biologics, and,

00:29:22.363 --> 00:29:24.313
and the other is the Center for Drugs.

00:29:25.063 --> 00:29:26.413
And so the center for.

00:29:27.163 --> 00:29:31.033
Devices became very much interested
in the Bayesian approach.

00:29:31.033 --> 00:29:38.108
And they in fact, approached me
in the 1990s to write a, uh, a,

00:29:38.113 --> 00:29:43.663
a draft guidance for how to do
Bayesian things in medical devices.

00:29:44.083 --> 00:29:50.983
And that was eventually, uh, the
eventual guidance was bore little

00:29:51.043 --> 00:29:52.723
resemblance to what I had written.

00:29:52.723 --> 00:29:57.163
But, um, it was about in, in 2010.

00:29:57.253 --> 00:30:07.543
Yes, but the, the, the baying, there
is no, uh, approved, uh, Bayesian

00:30:07.543 --> 00:30:12.493
guidance, I think, for drugs, but
there are guidances for what's called

00:30:13.033 --> 00:30:19.933
complex innovative designs that
specifically includes the Bayesian thing.

00:30:21.138 --> 00:30:24.853
Scott Berry: The, the center
for, uh, biologics and drugs are,

00:30:24.883 --> 00:30:27.013
are supposed to have one out.

00:30:27.013 --> 00:30:30.193
It was supposed to be in September
with the government shutdown.

00:30:30.493 --> 00:30:33.373
Who knows when this comes
out, but it's due any day now

00:30:33.763 --> 00:30:34.903
is a new guidance.

00:30:35.443 --> 00:30:35.713
Yep.

00:30:36.358 --> 00:30:36.688
Don Berry: But in,

00:30:36.688 --> 00:30:42.088
in terms of the susceptibility in the
community, you know, when we build

00:30:42.088 --> 00:30:45.388
these trials, an automaton can run it.

00:30:46.498 --> 00:30:54.328
Um, and so what we do is run the automaton
many times and, and simulate to evaluate,

00:30:54.988 --> 00:30:59.278
you know, how likely is it that if
you have a drug that's not effective.

00:30:59.608 --> 00:31:04.798
And this is what the tradition in
the FDA has been forever, including

00:31:04.798 --> 00:31:11.158
as, uh, this, this Belmont report,
uh, is the false positive rate.

00:31:11.788 --> 00:31:15.448
Suppose I have a drug that's
not effective, what is

00:31:15.448 --> 00:31:16.918
the probability that you.

00:31:18.013 --> 00:31:24.583
Um, can, uh, uh, that your
trial will answer that way.

00:31:25.663 --> 00:31:31.663
Uh, and so the false positive
rate is not a Bayesian quantity.

00:31:32.653 --> 00:31:41.113
Uh, it is rather the standard for, uh, the
frequentist in terms of, uh, uh, whether

00:31:41.113 --> 00:31:43.123
we're doing things, uh, well or not.

00:31:43.663 --> 00:31:45.313
But we can do that with bays.

00:31:45.913 --> 00:31:54.553
And so what we do is we run these trials,
uh, we simulate, uh, the trial many, many

00:31:54.673 --> 00:31:59.713
times, uh, millions of times sometimes,
uh, to get an accurate assessment

00:31:59.713 --> 00:32:01.303
of what is the false positive rate.

00:32:01.303 --> 00:32:07.573
And then that becomes part of the,
uh, protocol and the approval process.

00:32:08.293 --> 00:32:16.183
So we, we do things that are frequentist
in order to get them to be used.

00:32:18.778 --> 00:32:19.078
Cooper Berry: Yep.

00:32:20.548 --> 00:32:24.268
So Nick, I'm gonna turn the
page to you here a little bit.

00:32:25.318 --> 00:32:29.158
When would you consider yourself
being a Bayesian statistician?

00:32:29.158 --> 00:32:29.668
Your career?

00:32:29.668 --> 00:32:30.958
Was it at Iowa State?

00:32:30.958 --> 00:32:32.398
Was it after Iowa State?

00:32:32.403 --> 00:32:35.913
What was the frequencies Bayesian
debate like when you were at Iowa State?

00:32:36.223 --> 00:32:36.703
Nick Berry: Okay.

00:32:36.853 --> 00:32:41.743
So the first question, when did
I become a Bayesian statistician?

00:32:41.743 --> 00:32:46.303
I'll say when I was like
16 years old, probably.

00:32:46.393 --> 00:32:55.363
Um, just growing up around Scott and
doing things like playing board games

00:32:55.363 --> 00:32:59.743
and strategy, and I remember looking
over his shoulder and doing like little

00:32:59.743 --> 00:33:01.783
problem solving things on the computer.

00:33:01.783 --> 00:33:06.373
And, and I think it's like a, a method
of thinking that was sort of ingrained

00:33:06.373 --> 00:33:11.683
in me, um, this, you know, sort of
sequential update idea of Bayesian.

00:33:11.683 --> 00:33:15.583
And so I think I learned a lot
about, well, what's the probability

00:33:15.583 --> 00:33:17.443
this will happen in the board game?

00:33:17.443 --> 00:33:22.483
And can you make a decision
that maximizes some win, uh,

00:33:22.513 --> 00:33:24.853
condition, given that probability?

00:33:24.853 --> 00:33:25.753
And so it's sort of a.

00:33:26.503 --> 00:33:30.883
Introduction to, to Bayesian
statistics through that.

00:33:30.883 --> 00:33:34.783
And then I went to Iowa State
and maybe my Bayesian arc kind of

00:33:34.783 --> 00:33:37.393
took a dip because Iowa State's
not a very Bayesian department.

00:33:37.723 --> 00:33:40.453
So I, I started fast and then leveled out.

00:33:40.453 --> 00:33:43.663
And, uh, I didn't write
a Bayesian dissertation.

00:33:43.663 --> 00:33:45.403
I didn't have a Bayesian advisor.

00:33:45.733 --> 00:33:49.933
Um, at Iowa State, you had to go outta
your way and take electives if you wanted

00:33:49.933 --> 00:33:51.673
to learn about Bayesian statistics.

00:33:51.733 --> 00:33:55.303
Um, not that it was a taboo subject,
just that it wasn't part of the

00:33:55.303 --> 00:33:56.863
core curriculum for the most part.

00:33:57.403 --> 00:33:58.063
Um,

00:33:58.303 --> 00:34:01.393
and so I, I of course did that
and took Bayesian classes.

00:34:01.453 --> 00:34:08.143
Um, but my learning kind of stalled
in the sense that Scott described

00:34:08.143 --> 00:34:12.103
it being a language that he spoke
and it being just innate and, you

00:34:12.103 --> 00:34:15.283
know, pervasive in the department
wasn't really an experience I had.

00:34:15.283 --> 00:34:20.953
It was something that, uh, you talked
about as an alternative to the, the real

00:34:20.953 --> 00:34:22.693
way to do it or something like that.

00:34:22.693 --> 00:34:23.083
And so.

00:34:23.683 --> 00:34:25.123
Um, sort of leveled out.

00:34:25.183 --> 00:34:28.543
I did, and I've always been a
little focused on computation.

00:34:28.543 --> 00:34:33.283
And so one thing I did think about
and, and talk about and, and, and

00:34:33.283 --> 00:34:35.353
research when I was in school was

00:34:35.893 --> 00:34:38.503
MCMC methods, uh, markup
chain, Monte Carlo.

00:34:38.533 --> 00:34:43.093
Um, and I think that set me up really
nicely for my eventual career as a bay

00:34:43.723 --> 00:34:44.443
statistician.

00:34:44.443 --> 00:34:46.933
And I still think of myself
as sort of a computational

00:34:47.173 --> 00:34:49.123
heavy, uh, statistician, but,

00:34:49.453 --> 00:34:50.323
uh, I can't.

00:34:50.563 --> 00:34:56.773
So maybe the modeling aspect of it is
the part that sticks out to me the most.

00:34:56.773 --> 00:35:03.013
Just the ability to estimate these really
general models using this, this kind

00:35:03.013 --> 00:35:06.343
of, uh, all in one toolkit that is MCMC.

00:35:06.343 --> 00:35:10.213
And, um, so that sort of has drawn
me to it as someone that's always

00:35:10.213 --> 00:35:13.573
on the computer, always programming,
is having this MCMC toolkit just

00:35:13.573 --> 00:35:19.903
makes everything so available,
whereas with a frequentist method.

00:35:20.908 --> 00:35:22.948
Is there a package that does this?

00:35:22.978 --> 00:35:25.168
Uh, sort of, what is this?

00:35:25.438 --> 00:35:30.028
Can I ask, can I get a confidence
interval for this combination of

00:35:30.028 --> 00:35:32.428
parameters and things like that, that
I was always thinking about, just

00:35:32.428 --> 00:35:36.598
become inherently natural and it's just
such a smooth way to, to do model one.

00:35:37.588 --> 00:35:40.798
Uh, and so now I've gone up,
as I described at, at Berry.

00:35:40.918 --> 00:35:46.408
I, you know, have been immersed
in Bayesian methods for, for six

00:35:46.408 --> 00:35:48.418
years now, six and a half years.

00:35:48.418 --> 00:35:51.553
And, uh, I'm, I'm back on the way out.

00:35:53.203 --> 00:35:55.543
Don Berry: So if you had two
words to describe the difference

00:35:55.543 --> 00:35:58.753
between Bayes and a non Bayesian,
it would be decision making.

00:36:00.013 --> 00:36:04.273
Um, and if you look at some
definitions of statistics, it says

00:36:04.273 --> 00:36:06.313
decision making under uncertainty.

00:36:07.273 --> 00:36:09.913
Uh, but that's not the
frequentist approach.

00:36:10.633 --> 00:36:14.323
Uh, is the, you can make decisions of
course, in the frequentist approach,

00:36:14.323 --> 00:36:15.943
but it's not specifically part of it.

00:36:16.393 --> 00:36:23.323
When Savage wrote his fundamental, uh,
book, the Foundations of Statistics, um,

00:36:23.353 --> 00:36:29.113
the one thing that came out, of course
with the Bayesian, uh, but the other

00:36:29.113 --> 00:36:32.263
thing that came out of his efforts was.

00:36:33.713 --> 00:36:42.238
He, he wanted to have,
um, a utility function.

00:36:42.628 --> 00:36:44.308
You know, why are we doing this?

00:36:44.368 --> 00:36:46.678
What, what is the purpose?

00:36:47.248 --> 00:36:51.118
Um, and what do we conclude
based on what we've done?

00:36:51.118 --> 00:36:59.368
So it leads to, uh, as part of
the, uh, foundation, putting

00:36:59.368 --> 00:37:05.818
the two together, the decision
making and the, the data analysis.

00:37:06.238 --> 00:37:10.858
Data analysis, without decision
making, uh, the, it's, it's

00:37:10.858 --> 00:37:12.508
not, can't be Bayesian.

00:37:12.598 --> 00:37:19.198
Bayesian has the two of them together and
that's involves modeling very frequently,

00:37:23.828 --> 00:37:24.048
Scott Berry: Oh,

00:37:24.058 --> 00:37:25.078
Don Berry: strategies.

00:37:25.633 --> 00:37:26.233
Scott Berry: so Cooper,

00:37:26.233 --> 00:37:27.343
You're are you muted.

00:37:30.738 --> 00:37:31.348
Cooper Berry: Yeah, yeah, yeah.

00:37:31.808 --> 00:37:32.028
Um.

00:37:32.753 --> 00:37:37.313
So my follow up question is, where are we
at in the history of Bayesian right now?

00:37:37.553 --> 00:37:42.173
When you look at what's going on now
with AI revolutionizing almost every

00:37:42.173 --> 00:37:46.223
field, how does that affect Bayesian
and where are we going with Bayesian?

00:37:46.223 --> 00:37:47.453
Is it on the way down?

00:37:47.453 --> 00:37:48.143
Is it on the way up?

00:37:48.143 --> 00:37:49.013
Is it just starting?

00:37:49.643 --> 00:37:50.843
Uh, I'll start with Nick.

00:37:51.383 --> 00:37:54.623
Um, and then we can go all
the way around, uh, with

00:37:54.888 --> 00:37:55.178
Nick Berry: Yeah.

00:37:55.603 --> 00:37:55.963
Okay.

00:37:55.963 --> 00:38:02.833
So ai, um, so do you know
who Jeffrey Hinton is?

00:38:03.343 --> 00:38:08.653
He's a, a, a computer scientist
that was at the University of

00:38:08.653 --> 00:38:13.333
Toronto for a long time now known
as like the godfather of ai.

00:38:13.543 --> 00:38:19.273
He essentially invented in a way,
neural networks by inventing back

00:38:19.273 --> 00:38:23.623
propagation, which is now used to
like get the, the weights and estimate

00:38:23.623 --> 00:38:26.503
all the parameters that go into
like these billion parameter models.

00:38:26.983 --> 00:38:27.583
Um,

00:38:28.213 --> 00:38:28.933
he

00:38:29.938 --> 00:38:32.548
A Bayesian as a computer scientist.

00:38:32.548 --> 00:38:38.758
He's a, you know, he's talked about
Bayesian statistics and, um, thinks like a

00:38:38.758 --> 00:38:45.478
Bayesian, um, has tweets about subjective
probabilities that the world, the, the

00:38:45.568 --> 00:38:49.288
human race will be extinct because of AI
in the next 30 years and things like that.

00:38:49.558 --> 00:38:51.178
So he thinks like a Bayesian.

00:38:51.178 --> 00:38:52.798
He has students that
have done great things.

00:38:52.798 --> 00:38:57.268
Bradford Neil is a Bayesian, uh, wrote
up his dissertation about Bayesian

00:38:57.268 --> 00:38:58.918
estimation of neural networks.

00:38:59.338 --> 00:39:06.058
Um, and so there's, there are bayesians
doing amazing stuff in AI fitting

00:39:06.058 --> 00:39:07.408
these large language models there.

00:39:07.408 --> 00:39:11.218
They're at the forefront of ai, but
Bayesian methods are not used in

00:39:11.218 --> 00:39:12.838
any of the state-of-the-art models.

00:39:13.378 --> 00:39:17.188
Bayesian ideas probably are, but
they're not estimating posterior

00:39:17.188 --> 00:39:18.538
distributions for these parameters.

00:39:18.568 --> 00:39:21.388
It's impossible just because
of the magnitude of it.

00:39:21.478 --> 00:39:23.518
Even now, we don't have
enough compute power to do it.

00:39:23.518 --> 00:39:24.478
It's just not feasible.

00:39:24.478 --> 00:39:26.128
So I think.

00:39:27.313 --> 00:39:29.593
Bayesian has had a huge impact.

00:39:29.833 --> 00:39:34.363
Bayesian statistics had a huge
impact on people doing deep learning.

00:39:34.363 --> 00:39:37.273
Right now, it's not implemented
in the state-of-the-art models.

00:39:37.483 --> 00:39:41.533
I don't know if in 30 years,
like now we look back and say, we

00:39:41.533 --> 00:39:45.433
couldn't estimate this model that's
now, you know, four lines in Stan.

00:39:45.793 --> 00:39:50.743
But in 40 years are we gonna look
back and say, okay, we can now get

00:39:50.983 --> 00:39:55.363
posterior joint posteriors for these,
for these parameters in the model.

00:39:55.363 --> 00:39:58.243
And we have uncertainty about
these things that we can combine.

00:39:58.453 --> 00:40:01.123
So I think that's, I mean,

00:40:03.163 --> 00:40:07.843
it seems impossible looking at it
now to, to do anything like MCMC

00:40:07.843 --> 00:40:11.473
on those parameters, but maybe some
variational inference or some, um,

00:40:11.503 --> 00:40:14.983
more direct way to sample from these
will, will lead to, to progress.

00:40:17.143 --> 00:40:18.943
Cooper Berry: Um, yeah.

00:40:18.973 --> 00:40:25.663
So Lindsay, you went to Duke,
uh, I understand that it's a

00:40:25.663 --> 00:40:27.703
more Bayesian graduate school.

00:40:28.513 --> 00:40:32.473
What was the frequent, his first
Bayesian debate, like when you were in

00:40:32.473 --> 00:40:35.383
graduate school and starting at Barry?

00:40:35.383 --> 00:40:37.453
What was the, what was
the transition like?

00:40:41.398 --> 00:40:42.328
Lindsay Berry: Um, yes.

00:40:42.328 --> 00:40:48.628
So I guess I, I won't tell
you if I'm a Bayesian, but um,

00:40:48.718 --> 00:40:55.588
the prior is I am Dawn's granddaughter
and I'm I'm Scott's daughter.

00:40:55.948 --> 00:40:58.948
So that's your prior, uh, you
can form your prior from that.

00:40:59.398 --> 00:41:04.648
And then I did my undergrad at UT
Austin in math and I took classes

00:41:04.648 --> 00:41:11.248
from James Scott, who was a Duke Stats
PhD and he taught intro statistics,

00:41:11.248 --> 00:41:13.048
including Bayesian statistics.

00:41:13.048 --> 00:41:18.058
And then I went to Duke
to get my statistics PhD.

00:41:18.058 --> 00:41:21.808
And I was taught by, uh,
Mike West was my advisor.

00:41:23.188 --> 00:41:30.688
Uh, Jim Berger is at Duke, um, David
Dunson, uh, famous Bayesian statisticians.

00:41:31.198 --> 00:41:31.768
Um,

00:41:32.758 --> 00:41:33.328
so

00:41:34.918 --> 00:41:36.988
Scott Berry: anyway, um,

00:41:37.108 --> 00:41:41.188
Lindsay Berry: on that data, you can
update your, your, uh, posterior and,

00:41:41.308 --> 00:41:46.798
and guess whether you think I'm a
Bayesian or not, but, um, yeah, I, I

00:41:46.798 --> 00:41:55.708
mean, I think where I land is that I
do, I agree with Scott when I, when

00:41:55.708 --> 00:42:01.858
it comes to modeling, I think like a
Bayesian and, you know, my research was

00:42:01.858 --> 00:42:05.548
in essentially sequential forecasting.

00:42:05.548 --> 00:42:12.958
So you are getting updated information
every day about sales of a, an item in a

00:42:12.958 --> 00:42:15.958
supermarket, and your goal is forecasting.

00:42:15.958 --> 00:42:19.018
You know, how many items
will we sell tomorrow?

00:42:19.108 --> 00:42:22.228
Uh, well, we sell in two
days and 14 days, and so.

00:42:23.128 --> 00:42:29.038
It's just natural to a attack that problem
from a sequential learning framework of,

00:42:29.128 --> 00:42:30.988
you know, how many did we sell yesterday?

00:42:31.018 --> 00:42:35.998
Um, the next day, the day after
that continuously updating, um,

00:42:36.058 --> 00:42:39.628
your posterior, um, to forecast.

00:42:39.718 --> 00:42:41.128
Um, so it's just natural.

00:42:41.188 --> 00:42:45.088
Um, I, I guess I feel like

00:42:47.308 --> 00:42:50.788
as Dawn said, you know, we have
to speak both languages though.

00:42:50.788 --> 00:42:55.228
You know, we can't, we use frequentist
methods even here at Barry.

00:42:55.378 --> 00:43:03.538
Um, and we speak the frequentist language
and we try to use Bayesian methods

00:43:03.568 --> 00:43:06.658
where they're appropriate and we feel
like they're adding a lot of value.

00:43:07.048 --> 00:43:12.568
Um, so I think that balance is
really important, um, uh, for me.

00:43:12.808 --> 00:43:16.798
Um, and then I, I guess I can talk about.

00:43:17.083 --> 00:43:20.503
My transition here at
Barry was pretty unique.

00:43:20.533 --> 00:43:25.633
And, you know, we talked about some of the
barriers of Bayesian in clinical trials,

00:43:25.633 --> 00:43:29.323
and one of those is just, it's not what's

00:43:29.863 --> 00:43:31.213
traditionally done

00:43:31.213 --> 00:43:33.913
you know, and when you're going
against that status quo, you hit,

00:43:34.513 --> 00:43:38.593
um, you run into those challenges,
getting new ideas across.

00:43:38.653 --> 00:43:45.823
Um, I joined Barry in the summer of
2019 and I had a, a nice, you know,

00:43:45.823 --> 00:43:49.393
six month period of, of learning
how clinical trials are done.

00:43:49.393 --> 00:43:55.243
And then at the end of 2019, uh, the
COVID pandemic started ramping up.

00:43:55.363 --> 00:44:00.463
And I really think that was a
point where we were forced very

00:44:00.463 --> 00:44:03.583
quickly to change how we do things.

00:44:03.613 --> 00:44:10.063
And early in 2020, I was brought on
to a project called remap Cap, and

00:44:10.063 --> 00:44:12.313
this is an adaptive platform trial.

00:44:13.078 --> 00:44:20.548
That had existed already, um, studying
patients with community acquired

00:44:20.548 --> 00:44:26.068
pneumonia and the platform was kind of
designed to be prepared for a future

00:44:26.068 --> 00:44:29.908
pandemic because they thought, you
know, it was, it was very likely a

00:44:29.968 --> 00:44:34.108
respiratory influenza pandemic or some
other pandemic could occur, and they

00:44:34.108 --> 00:44:38.758
wanted to be ready to pivot, um, to
study patients with those diseases.

00:44:38.758 --> 00:44:43.228
So I, I think COVID really forced us.

00:44:43.918 --> 00:44:46.468
We had to be Bayesian,
we had to be adaptive.

00:44:47.038 --> 00:44:50.428
You know, we, we were faced with
so much uncertainty at that time.

00:44:50.428 --> 00:44:51.838
What was this pandemic gonna.

00:44:52.483 --> 00:44:54.313
You know, how far was it gonna spread?

00:44:54.313 --> 00:44:56.263
How serious was it going to be?

00:44:56.623 --> 00:44:58.843
How many patients could
we enroll in a disease?

00:44:58.843 --> 00:45:02.593
There was so much uncertainty
during that design stage that we

00:45:02.593 --> 00:45:07.933
couldn't do the traditional thing
of running one-to-one trials, um,

00:45:08.113 --> 00:45:10.093
you know, fixed power calculations.

00:45:10.093 --> 00:45:14.083
We had to be adaptive, we had
to do frequent interim analyses.

00:45:14.113 --> 00:45:21.433
Um, so I, I think that was a very
unique experience and, uh, it, it's now

00:45:21.433 --> 00:45:26.263
paved the way I think for, for people
to have more acceptance of Bayesian

00:45:26.263 --> 00:45:28.483
methods of adaptive platform trials.

00:45:29.938 --> 00:45:30.028
Cooper Berry: Mm-hmm.

00:45:30.718 --> 00:45:38.578
Do you think in the next pandemic that
a, um, a Bayesian approach will be taken?

00:45:42.013 --> 00:45:45.313
Lindsay Berry: Uh, yes, I think
it, it kind of has to be, I mean,

00:45:45.763 --> 00:45:49.303
I think, you know, one of the

00:45:49.903 --> 00:45:51.043
benefits is just.

00:45:51.508 --> 00:45:55.618
I think you can't just focus on the
trial that you're running, you know,

00:45:55.618 --> 00:45:57.358
things are happening outside the trial.

00:45:57.358 --> 00:46:00.208
You're learning constantly.

00:46:00.208 --> 00:46:06.148
There's external data emerging and um,
you know, being able to either incorporate

00:46:06.148 --> 00:46:11.218
that into the analysis in a formal way
or, you know, to stop a treatment and

00:46:11.218 --> 00:46:13.198
then still have an interpretable result.

00:46:13.333 --> 00:46:17.188
I, I think, yeah, absolutely.

00:46:18.143 --> 00:46:18.523
Cooper Berry: Got it.

00:46:18.838 --> 00:46:19.348
Awesome.

00:46:20.488 --> 00:46:23.848
Um, so now looking into
the future a little bit.

00:46:24.148 --> 00:46:31.138
So Dawn, it's been about over 50 years and
you've seen Bayesian clinical trials grow.

00:46:31.258 --> 00:46:36.418
Uh, you've seen Monte Carlo methods, Mar
uh, Markoff, shame Monte Carlo methods,

00:46:36.868 --> 00:46:42.808
um, and now looking towards ai, what is
Bayesian statistics to you going forward?

00:46:42.808 --> 00:46:45.058
Are you happy with where things are going?

00:46:45.058 --> 00:46:46.258
Are you pessimistic?

00:46:46.258 --> 00:46:47.458
What, where's your head at with it?

00:46:50.533 --> 00:46:55.723
Don Berry: So, uh, um,
depends what you mean by ai.

00:46:55.728 --> 00:47:00.913
There, there are various definitions,
but the, the, the notion of

00:47:00.913 --> 00:47:04.363
learning, uh, is, uh, fundamental.

00:47:04.363 --> 00:47:13.393
You know, you teach something to teach
itself, to teach itself, um, what

00:47:13.393 --> 00:47:17.563
to do in the context of the process.

00:47:18.223 --> 00:47:22.933
Uh, and so it's, you know, is what we do.

00:47:23.293 --> 00:47:28.453
So in a sense are, the algorithm
that I mentioned is ai.

00:47:28.603 --> 00:47:29.653
I mean, it's learning.

00:47:30.133 --> 00:47:33.133
It has many possibilities that it can do.

00:47:33.763 --> 00:47:37.543
And as it's, it's built to learn.

00:47:38.818 --> 00:47:44.278
And as it learns, um, it makes decisions.

00:47:44.908 --> 00:47:51.058
And, uh, and what I've read
about, uh, artificial intelligence

00:47:51.778 --> 00:47:57.298
is it's, it's, it's all in
decisions and making decisions.

00:47:57.298 --> 00:48:01.528
And as I said before, making
decisions means Bayes.

00:48:01.708 --> 00:48:07.198
I mean, you, you, you can't really
do it without Bay Herman Chernoff.

00:48:08.428 --> 00:48:12.208
I'm not Bayesian, but I use
Bayesian because I have to.

00:48:12.208 --> 00:48:14.638
It's the only thing that
lets me do what I want to do.

00:48:15.118 --> 00:48:20.098
And that is a summary of what
happens in artificial intelligence

00:48:20.608 --> 00:48:23.308
and what happens in ai.

00:48:23.308 --> 00:48:30.898
So I, I think the futures is, is
Bays as to, you know, artificial

00:48:30.898 --> 00:48:32.758
intelligence to me is scary.

00:48:33.478 --> 00:48:35.728
Um, and, um.

00:48:37.078 --> 00:48:43.918
I am, I'm, uh, uh, I'm not
convinced that Bayesian analyses

00:48:43.918 --> 00:48:50.458
won't be used for, um, some of the
scary things that might happen.

00:48:50.788 --> 00:48:52.618
So who knows what's going to happen?

00:48:52.708 --> 00:48:57.538
You know, as Yogi Berra said, prediction
is difficult, especially the future.

00:49:02.208 --> 00:49:05.373
Cooper Berry: And so if we, if we pick.

00:49:05.373 --> 00:49:05.703
Sure.

00:49:05.703 --> 00:49:08.673
Bayesian has a sort of
timeline and we predict Sure.

00:49:08.703 --> 00:49:11.133
We, let's start the timeline.

00:49:11.133 --> 00:49:15.663
The first quarter of a, of a game
at you doing bandit problems.

00:49:16.473 --> 00:49:19.323
What, where are we at right
now at this point in time?

00:49:19.323 --> 00:49:20.823
What, what quarter of a football game?

00:49:20.823 --> 00:49:22.923
What, what time period
are we at halftime yet?

00:49:23.223 --> 00:49:24.393
We still in the first quarter.

00:49:24.393 --> 00:49:25.443
Is it almost over?

00:49:26.193 --> 00:49:28.833
Um, in terms of Bayesian methods,

00:49:33.868 --> 00:49:36.193
Scott Berry: Who are you passing
the ball to with that one?

00:49:36.783 --> 00:49:37.503
Cooper Berry: that's a tough one.

00:49:37.503 --> 00:49:38.943
Anyone have any strong.

00:49:40.528 --> 00:49:44.098
Lindsay Berry: I have an answer
and I don't know, I'm just

00:49:44.098 --> 00:49:45.688
kind of making stuff up here.

00:49:45.688 --> 00:49:54.328
So I think right now we do a lot of
Bayesian inference and, um, you know, we

00:49:54.328 --> 00:50:00.808
might run a Bayesian trial and at, at the
end of the trial we report a posterior

00:50:00.808 --> 00:50:04.798
probability of benefit from a treatment.

00:50:05.818 --> 00:50:08.248
I think what's missing is.

00:50:09.238 --> 00:50:12.808
The decision making piece, Dawn was
just describing, you know, a lot

00:50:12.808 --> 00:50:14.098
of times we just leave it there.

00:50:14.098 --> 00:50:15.928
Here's our posterior probability.

00:50:15.928 --> 00:50:18.028
What are we supposed
to do with that though?

00:50:18.448 --> 00:50:22.408
Um, and, you know, how should
a clinician use this treatment?

00:50:22.408 --> 00:50:24.298
Should a regulator approve a drug?

00:50:25.228 --> 00:50:32.248
I think that second piece is
underrepresented in analysis of

00:50:32.248 --> 00:50:34.528
clinical trials and, and design.

00:50:34.528 --> 00:50:39.658
And so I, I think I would
say we're maybe at halftime.

00:50:39.658 --> 00:50:43.618
You know, we are using some
of the tools of Bayesian

00:50:43.618 --> 00:50:45.328
statistics, but not all of them.

00:50:45.358 --> 00:50:50.098
And I think there's a lot of
promise from decision theory that

00:50:51.208 --> 00:50:54.208
is yet to be, um, discovered.

00:50:55.463 --> 00:50:59.668
Scott Berry: I, I, I'll be, I'll be
bold and say that I'm not even sure

00:50:59.668 --> 00:51:04.588
the game has started yet because I
think there's gonna be an incredible

00:51:04.588 --> 00:51:08.608
explosion of data, and we've seen
that a little bit in the 25 years.

00:51:08.608 --> 00:51:11.458
When we, when we, when I started
doing this 25 years ago, there was

00:51:11.458 --> 00:51:13.708
no data outside of a clinical trial.

00:51:14.128 --> 00:51:18.448
And Frequentist is kind of good at
analyzing a single trial result.

00:51:18.448 --> 00:51:21.268
It's, it's, it's what it does now.

00:51:21.268 --> 00:51:25.258
We have these incredible sources
of data and, and in some cases we

00:51:25.258 --> 00:51:29.128
have that now, and we're changing
the way we're doing trials by

00:51:29.128 --> 00:51:34.558
incorporating that data, incorporating
that on a much grander scale about.

00:51:34.803 --> 00:51:37.353
Exactly what Lindsay and
Don said, decision making.

00:51:37.803 --> 00:51:42.243
The only way to synthesize the
incredible amount of data that's

00:51:42.243 --> 00:51:46.743
coming in clinical trials, but outside
of clinical trials will be Bayesian.

00:51:47.073 --> 00:51:50.553
Uh, I, I think we're gonna see this
incredible growth and it'll be an

00:51:50.553 --> 00:51:54.633
interesting thing, the history of
mathematics, the class you're in.

00:51:54.903 --> 00:51:58.023
Looking back at this, at some
point I think we're gonna see this

00:51:58.023 --> 00:52:02.433
huge explosion on it, but I, I
interrupted Nick, your prediction.

00:52:03.163 --> 00:52:05.083
Nick Berry: Yeah, I won't
let your answer swing me.

00:52:05.083 --> 00:52:08.143
I'll stick to what I was gonna say
originally, which is I think we're

00:52:08.143 --> 00:52:12.373
at the two minute timeout at the
end of the first half it feels like.

00:52:12.943 --> 00:52:15.973
We're not at halftime yet, and
we're actually in like, at least

00:52:15.973 --> 00:52:21.793
in the clinical trials context, in
this sort of mad dash, like, uh,

00:52:22.393 --> 00:52:24.073
these things are getting accepted.

00:52:24.073 --> 00:52:29.293
They're being used in, you know, a
lot of trials, regulatory trials.

00:52:29.413 --> 00:52:34.123
Uh, people that aren't in Barry
are designing Bayesian trials.

00:52:34.123 --> 00:52:34.873
It's not just us.

00:52:34.873 --> 00:52:38.293
And so I feel like we're in this
two minute drill to get to halftime.

00:52:38.293 --> 00:52:40.693
We're, we're, we're leading
up to something, but moving at

00:52:40.693 --> 00:52:42.433
a really fast pace right now.

00:52:42.733 --> 00:52:47.833
And maybe we'll get to some stasis in
some short time where, okay, we plateaued.

00:52:47.863 --> 00:52:50.443
There's no questions about
Bayesian clinical trials.

00:52:50.443 --> 00:52:51.133
You do them.

00:52:51.133 --> 00:52:52.153
Here's how they're done.

00:52:52.183 --> 00:52:53.053
Here's the history.

00:52:53.143 --> 00:52:57.403
Like here's a, a backlog of things you
can look at for other people doing them.

00:52:57.403 --> 00:53:01.303
And uh, and then we will see how
the second half goes, which maybe

00:53:01.303 --> 00:53:04.873
is Scott's thing about synthesizing
these massive amounts of data.

00:53:05.203 --> 00:53:07.843
Um, and, and you know, things like that.

00:53:09.073 --> 00:53:13.963
Cooper Berry: Do you think your job, Nick,
uh, specifically facts related, is gonna

00:53:13.963 --> 00:53:16.333
get easier or harder as we get more data?

00:53:19.018 --> 00:53:24.118
Nick Berry: Oh, facts related as in
developing our software, I think it

00:53:24.118 --> 00:53:29.458
probably gets, gets harder, um,
harder in the sense that we, we

00:53:29.578 --> 00:53:34.048
try to create software that is
generally useful to a lot of people.

00:53:34.348 --> 00:53:38.938
And as the problem space gets bigger and
bigger and bigger and bigger and people

00:53:38.938 --> 00:53:43.168
have different types of external data and
they want to use it in different ways, the

00:53:43.228 --> 00:53:48.118
capabilities that we have to provide to
users grow really, really, really fast.

00:53:48.478 --> 00:53:52.108
So I think it probably, you know, it'd
be much easier to write a clinical trial

00:53:52.108 --> 00:53:56.908
design software that just simulated
fixed frequentist clinical trials.

00:53:57.448 --> 00:54:03.268
Um, and so I think as we get more
data, um, more models, more methods

00:54:03.268 --> 00:54:07.018
for combining data, it probably gets
harder, um, but also more useful.

00:54:07.693 --> 00:54:13.363
If we have something that allows people
to do it quickly, um, they don't have to

00:54:13.513 --> 00:54:15.253
know how to program something in stand.

00:54:15.253 --> 00:54:20.293
They don't have to work on hierarchical
priors and stuff like that.

00:54:20.593 --> 00:54:23.953
It can open the door for a lot
more people to use these methods.

00:54:23.953 --> 00:54:26.053
So, um, yeah.

00:54:26.308 --> 00:54:27.148
Cooper Berry: Don, how about you?

00:54:27.148 --> 00:54:30.028
Where are we at in the,
uh, the Bayesian Bowl game?

00:54:31.218 --> 00:54:33.733
Don Berry: Uh, I'm two minute well.

00:54:34.528 --> 00:54:37.378
Like all bayesians I
probability distribution.

00:54:38.518 --> 00:54:43.798
And my probability distribution
has a mean of the two minute mark.

00:54:44.998 --> 00:54:46.348
We're two minutes into the game.

00:54:47.068 --> 00:54:55.558
Uh, but the distribution is,
is really, uh, uh, has a large

00:54:55.558 --> 00:54:57.868
variance, large stem deviation.

00:54:59.038 --> 00:55:07.018
Um, the critical thing to modify what,
uh, Nick was saying and Scott was

00:55:07.018 --> 00:55:10.618
saying is, is something about the data.

00:55:10.708 --> 00:55:11.728
What, what are the data?

00:55:12.328 --> 00:55:18.958
The data are not simply adding more
things and more patients and the,

00:55:19.348 --> 00:55:23.578
you know, but it's the biology.

00:55:25.228 --> 00:55:33.028
Biology today in every disease
is a far cry from biology.

00:55:34.573 --> 00:55:36.223
50, even 20 years ago.

00:55:36.973 --> 00:55:41.923
Um, and we, we learn more about
the diseases and more about the

00:55:41.923 --> 00:55:44.203
individual aspects of the diseases.

00:55:44.203 --> 00:55:47.233
You know, you have two patients
and they look the same.

00:55:48.343 --> 00:55:52.933
Um, let's say they have breast cancer
and they're the same age and they

00:55:52.933 --> 00:55:59.053
have breast cancer and the same
breast and, and, uh, the size of

00:55:59.053 --> 00:56:00.973
the two breast cancer is the same.

00:56:01.273 --> 00:56:10.243
Everything looks the same and one dies and
the other lose and dies of heart disease.

00:56:11.233 --> 00:56:11.713
Um,

00:56:14.323 --> 00:56:17.893
we talk in medicine
about precision medicine.

00:56:17.893 --> 00:56:23.623
We're running more and more about
the, the, the patient characteristics.

00:56:24.103 --> 00:56:29.833
And soon, and it's this thing that I
mentioned about, uh, there are many.

00:56:31.033 --> 00:56:39.793
Uh, categories of any disease,
uh, and every disease is becoming

00:56:39.793 --> 00:56:43.513
more and more, uh, categorized.

00:56:44.473 --> 00:56:51.343
Uh, and it is that kind of data
that we confront when we do

00:56:51.343 --> 00:56:57.463
trials and we ask about, um,
uh, the artificial intelligence.

00:56:57.463 --> 00:57:02.623
We ask, you know, about how's the
therapy doing in this type of patient?

00:57:03.373 --> 00:57:07.063
Or are there, are there, uh,
things that we should be measuring?

00:57:08.053 --> 00:57:12.703
Uh, and if it looks like it's better
in that type of patient, maybe we

00:57:12.703 --> 00:57:19.153
should use the therapy for that, uh,
patient, but not for another patient.

00:57:19.783 --> 00:57:21.823
And so that's the kind
of thing that we did.

00:57:21.823 --> 00:57:26.593
And, and I spy too, but it's,
it really is the future.

00:57:26.788 --> 00:57:26.848
Yeah.

00:57:27.418 --> 00:57:31.498
And, and all of these are
decisions and therefore bays.

00:57:32.338 --> 00:57:36.988
So, um, I think the future is, is bright.

00:57:38.038 --> 00:57:45.058
Um, we, I think all of us worry that
because it seems so bright, there's

00:57:45.058 --> 00:57:47.068
gonna be a lot of people trying to do it.

00:57:47.968 --> 00:57:50.548
And frankly, not everybody can do it.

00:57:51.478 --> 00:57:58.828
Uh, and they need our help, but there
are certainly lots of people out there

00:57:59.398 --> 00:58:01.618
who can help us just as well as we can.

00:58:04.408 --> 00:58:08.353
Cooper Berry: That's great and I'll,
I'll ask one more closing question here.

00:58:08.353 --> 00:58:15.793
So clearly Bayesian statistics has
somewhat of a, a grasp on our family.

00:58:16.363 --> 00:58:21.268
Um, what is it about Bayesian
that makes it so appealing?

00:58:22.063 --> 00:58:29.143
For statisticians, what is the one core
item of it that makes it so appealing?

00:58:33.028 --> 00:58:37.108
Lindsay Berry: I, you said, what
makes it appealing to statisticians?

00:58:37.138 --> 00:58:44.608
I think it's, it's just appealing to
humans because Bayesian statistics

00:58:44.608 --> 00:58:48.478
is the way a human thinks and learns.

00:58:48.538 --> 00:58:53.098
So, you know, you start out, you don't,
maybe you don't know much about something

00:58:53.098 --> 00:58:55.378
and then you gather some data and you.

00:58:56.383 --> 00:58:59.893
You update your beliefs as you do
that, and that's a sequential process.

00:58:59.893 --> 00:59:03.643
I think you, you know, you,
that's how you change your mind.

00:59:03.643 --> 00:59:10.153
And, um, I just, I just think that's
such a fundamental concept that people

00:59:10.153 --> 00:59:14.203
get it, you know, we work with a lot of
clinicians and that's how they think,

00:59:14.203 --> 00:59:20.023
you know, they see one trial result and
another trial result, and they kind of,

00:59:20.143 --> 00:59:23.173
as they go learn how to treat patients.

00:59:23.173 --> 00:59:29.713
So I, I think it's that concept that's
so appealing about it to not just our

00:59:29.713 --> 00:59:31.153
family, but I think a lot of people.

00:59:32.138 --> 00:59:32.218
Cooper Berry: Hmm.

00:59:32.438 --> 00:59:36.033
Is this appeal different than
the appeal of an adaptive trial

00:59:36.033 --> 00:59:38.073
over a non-adaptive trial?

00:59:42.413 --> 00:59:43.853
Scott Berry: So I think
it is different, but

00:59:43.853 --> 00:59:46.973
it utilizes exactly what Lindsay said.

00:59:47.513 --> 00:59:53.543
Once you adopt and recognize the
beauty of, I learn something, how does

00:59:53.543 --> 00:59:56.303
it, how does it update my knowledge?

00:59:57.083 --> 01:00:00.413
Now I want to keep updating my knowledge.

01:00:00.413 --> 01:00:01.343
I wanna learn something.

01:00:01.348 --> 01:00:05.963
And, and the idea of now doing sequential
experiments when we can do sequential

01:00:05.963 --> 01:00:08.993
learning, it's, it's the natural thing.

01:00:08.993 --> 01:00:12.608
The idea that we will wait
three years before we look at

01:00:12.608 --> 01:00:17.123
the data is just completely the
opposite of the Bayesian view.

01:00:17.578 --> 01:00:18.088
Nick.

01:00:21.268 --> 01:00:23.038
Nick Berry: Yeah, I think
I, I agree with that.

01:00:23.038 --> 01:00:24.568
I agree with what Lindsay said, obviously.

01:00:24.568 --> 01:00:28.288
And then for me, it's, it's the toolkit.

01:00:28.348 --> 01:00:34.258
It's the, the modeling, um, the hot,
just, I don't know the ability to.

01:00:36.433 --> 01:00:41.353
To estimate hierarchical models, draw
them as this, you know, these trees

01:00:41.413 --> 01:00:46.183
then have your hyper priors and estimate
these hierarchical models with shrinkage.

01:00:46.213 --> 01:00:51.343
It's like, I, it's such an incredibly
powerful tool that you can't help

01:00:51.343 --> 01:00:54.883
be like drawn to the light by,
you know, like a moss, right?

01:00:54.883 --> 01:00:58.123
You just, you keep seeing it as
you, you're in set statistics, it's

01:00:58.273 --> 01:01:02.533
hierarchical models and you, you can't
help but like be drawn to this Bayesian

01:01:02.533 --> 01:01:08.473
estimation of parameters and the, you
know, borrowing and, uh, hyper priors.

01:01:08.473 --> 01:01:11.083
It just, it's such an
elegant, nice way to do it.

01:01:11.083 --> 01:01:15.103
And now that we can estimate these
parameters so well, it's just that much

01:01:15.103 --> 01:01:21.463
more powerful, um, with, with not a lot
of startup that has to go into this.

01:01:21.463 --> 01:01:23.363
Like there's not a lot of
cost to doing it that way.

01:01:27.643 --> 01:01:30.463
Don Berry: So, for me, you get a story.

01:01:31.183 --> 01:01:35.443
Uh, I agree with, uh, Lindsay.

01:01:35.443 --> 01:01:36.973
I agree with, uh, Scott.

01:01:37.363 --> 01:01:42.793
Um, uh, the Bayesian
approach is, is, is natural.

01:01:43.483 --> 01:01:46.453
Um, the,

01:01:50.593 --> 01:01:56.383
there's a, a very famous statistician,
the most famous statistician in the world

01:01:56.383 --> 01:02:00.223
ever was father of modern statistics.

01:02:01.243 --> 01:02:08.803
Um, uh, Ronald Fisher, he was
a, a, uh, sir Ronald Fisher.

01:02:09.433 --> 01:02:14.743
Uh, he was a statistician at
Rothamsted, uh, agricultural

01:02:14.743 --> 01:02:17.173
Institute and developed, he.

01:02:17.398 --> 01:02:17.618
he.

01:02:20.068 --> 01:02:21.928
Defined P values.

01:02:22.648 --> 01:02:26.398
He, uh, invented randomization.

01:02:27.838 --> 01:02:32.638
He invented factorial designs.

01:02:33.478 --> 01:02:37.528
He's regarded by virtually
every statistician that I

01:02:37.528 --> 01:02:41.038
know, uh, as almost a God.

01:02:42.658 --> 01:02:44.368
So what would he think?

01:02:44.368 --> 01:02:48.688
You asked, uh, Cooper about
what Savage would think.

01:02:49.318 --> 01:02:53.518
Uh, what would he think about
randomization in clinical trials?

01:02:54.238 --> 01:02:56.338
And he was actually asked that question.

01:02:56.338 --> 01:02:58.168
He was, he worked in agriculture.

01:02:59.038 --> 01:03:05.908
Um, and by the way, he hated
people who would, uh, run

01:03:05.908 --> 01:03:08.098
experiments asking one question.

01:03:08.698 --> 01:03:11.218
He said, you have to
ask lots of questions.

01:03:12.058 --> 01:03:13.228
And we do that.

01:03:14.398 --> 01:03:20.968
But, uh, the one question that he
was asked by a, uh, statistician,

01:03:21.778 --> 01:03:25.828
um, because he, that that's the
statistician asking the question, was

01:03:25.828 --> 01:03:33.028
interested in whether, um, the, the,
the god of randomization would think

01:03:33.028 --> 01:03:38.698
that it's appropriate to randomize
patients, um, in clinical trials.

01:03:39.658 --> 01:03:47.428
Uh, and Fisher had no experience
with no knowledge of, uh, clinical

01:03:47.428 --> 01:03:54.148
trials, but he said, I suppose
what you would do is you would, um,

01:03:56.488 --> 01:04:04.678
calculate a p value, um, thinking P values
instead of Ian Statistics, you would

01:04:04.678 --> 01:04:10.738
calculate a P value for your current data,
and then tomorrow you would calculate

01:04:10.738 --> 01:04:12.538
another P value for the current data.

01:04:13.033 --> 01:04:19.573
And use some function of the p value
to assign randomly, but biased,

01:04:19.573 --> 01:04:24.673
randomly biased in favor of, of
the therapy that is doing better.

01:04:25.843 --> 01:04:32.113
And that's the kind of thing that
we do, that we did in, in I Spy too.

01:04:32.773 --> 01:04:34.033
And it's natural.

01:04:34.243 --> 01:04:39.223
I mean, you don't want to ignore,
here's a person faced with this

01:04:39.403 --> 01:04:46.093
question from another statistician
who, um, doesn't wanna ignore the

01:04:46.213 --> 01:04:47.953
randomization, but wants to do it.

01:04:47.953 --> 01:04:48.373
Right.

01:04:50.353 --> 01:04:54.853
Nick asks the question, even in that
circumstance about ethics, yes, there are

01:04:54.913 --> 01:05:00.943
ethics associated with it, but especially
if you could, um, you know, combine

01:05:00.943 --> 01:05:08.683
the, uh, things going on in randomized
trials with Fisher's concern about

01:05:08.923 --> 01:05:10.998
treating patients that are, that are.

01:05:11.893 --> 01:05:13.243
Things are developing.

01:05:14.228 --> 01:05:18.223
The, the ai, the algorithm is
telling us thing, whether it's a

01:05:18.223 --> 01:05:20.473
p value or posterior probability.

01:05:20.773 --> 01:05:26.533
Somebody else, uh, uh, somebody
named Thompson, uh, did a, uh,

01:05:29.173 --> 01:05:37.153
wrote a paper back in 1933, essentially
what Fisher said, but a Bayesian

01:05:37.153 --> 01:05:39.043
calculation instead of a P value.

01:05:40.003 --> 01:05:43.573
It probably doesn't make any difference
or much difference as to what you're

01:05:43.573 --> 01:05:49.783
using, depending on what function
it is, but evaluating with, uh, with

01:05:49.783 --> 01:05:55.573
simulation, and by the way, to Scott,
one of Scott's earlier points, um,

01:05:56.383 --> 01:06:02.713
Fisher could never have simulated
his, uh, proposal, nor could Thompson.

01:06:03.763 --> 01:06:09.763
Um, because it's, uh, complicated, uh,
many simulations that you have to do.

01:06:10.438 --> 01:06:15.328
Many scenarios, and by the way, Fisher
would be asking other questions too.

01:06:15.658 --> 01:06:17.278
So how do you incorporate that?

01:06:18.088 --> 01:06:24.358
Um, and it's the age of the computer
and the ability to do things.

01:06:24.358 --> 01:06:34.978
And, um, so it's, it, it's natural even
to this, this, uh, god of statistics.

01:06:37.363 --> 01:06:38.623
Scott Berry: Okay, so

01:06:38.623 --> 01:06:44.803
I am going to grab back control of the
host duty Cooper, thank you for coming on.

01:06:44.833 --> 01:06:49.933
Uh, uh, a wonderful conversation
and I want to thank everybody

01:06:49.933 --> 01:06:54.283
for, for joining us, and I want to
thank everybody for spending time

01:06:54.283 --> 01:06:57.583
in the interim until the next one.

01:06:58.273 --> 01:06:58.783
Thank you.

01:06:59.428 --> 01:07:00.058
Cooper Berry: Thank you.

01:07:00.163 --> 01:07:00.633
Lindsay Berry: Thank you.

01:07:00.718 --> 01:07:00.938
Scott Berry: Yep.

01:07:01.213 --> 01:07:01.483
Bye.

01:07:01.813 --> 01:07:02.233
Lindsay Berry: Thanks.