WEBVTT

NOTE
This file was generated by Descript 

00:00:00.000 --> 00:00:03.009
Judith: Welcome to Berry's In the
Interim podcast, where we explore the

00:00:03.009 --> 00:00:06.800
cutting edge of innovative clinical
trial design for the pharmaceutical and

00:00:06.800 --> 00:00:08.960
medical industries, and so much more.

00:00:09.720 --> 00:00:10.449
Let's dive in.

00:00:11.251 --> 00:00:17.611
Scott: Welcome everybody back to In The
Interim Uh I am your cohost today Scott

00:00:17.621 --> 00:00:25.511
Berry and I'm joined by my uh uh my common
cohost Kurt Vielli here and we we are

00:00:25.511 --> 00:00:31.541
here to talk about something And and and
Kurt will occasionally say Oh I've got

00:00:31.541 --> 00:00:36.291
a good topic for the podcast and usually
it's something that's uh beneath the

00:00:36.291 --> 00:00:41.391
crawl a little bit maybe uh wi wi within
the the the pet peeve But a really I I

00:00:41.511 --> 00:00:46.601
think a really important topic a really
important misunderstood topic and that is

00:00:46.641 --> 00:00:53.821
bias in early stopping in clinical trials
So welcome back to In The Interim Kurt

00:00:54.313 --> 00:00:54.763
Kert Viele: Thank you

00:00:55.705 --> 00:01:00.025
Scott: All right So so we are in the
interim and we're talking about bias

00:01:00.025 --> 00:01:05.755
of interims Uh sort of an interesting
aspect of this I I thought I'd introduce

00:01:05.755 --> 00:01:15.335
the topic by uh uh telling a a bit of
an old story and um this this happened

00:01:15.715 --> 00:01:21.775
relatively early So I I started designing
trials in 2000 Uh before that I did a

00:01:21.775 --> 00:01:27.505
little bit of clinical design but uh this
was early in Berry Consultants and was

00:01:27.515 --> 00:01:33.535
working with a medical device company
And they they had an estimate of their

00:01:33.535 --> 00:01:37.195
effect size but thought that they could
be quite a bit better than that So an

00:01:37.195 --> 00:01:42.105
adaptive sample size with early stopping
for superiority made a lot of sense for

00:01:42.105 --> 00:01:49.745
the trial So fairly standard two interim
analyses for early success could go to the

00:01:49.745 --> 00:01:57.135
final analysis Medical device uh nothing
controversial in in the trial design We

00:01:57.135 --> 00:02:02.655
can control type one error It's not the
issue Pretty straightforward And so we're

00:02:02.655 --> 00:02:07.745
working on the design and then I got
word from the the company that there's a

00:02:07.745 --> 00:02:15.625
problem um with stopping And the company
was associated with a group that was gonna

00:02:15.625 --> 00:02:21.025
run the trial and they had an issue with
potentially stopping the trial I thought

00:02:21.075 --> 00:02:26.415
Okay that's that that's odd You know
let's let's talk to them about it So the

00:02:26.425 --> 00:02:32.165
the concern was the bias in the estimate
that comes out of the trial and it's

00:02:32.165 --> 00:02:38.235
important for medical decisionmaking to
have good estimates of of the effect And

00:02:38.235 --> 00:02:45.855
the concern was that the early stopping
biases that estimate And so ookay that's

00:02:45.855 --> 00:02:49.615
great and one of the wonderful things
about simulation is we can simulate the

00:02:49.615 --> 00:02:57.465
bias So it was a response outcome uh
success and failure And I simulated the

00:02:57.465 --> 00:03:03.285
trial under a number of potential true
effects and calculated the bias of the

00:03:03.285 --> 00:03:09.795
trial And it was really relatively small
It was approximately one percent in a in

00:03:09.795 --> 00:03:15.215
a rate of success of We're we're looking
for thirty forty fifty percent success

00:03:15.215 --> 00:03:22.175
rates and it was a an s a bias of about
one percent which I thought was a minor

00:03:23.785 --> 00:03:31.075
Thing relative to the benefits of sample
size patients time to get a success should

00:03:31.075 --> 00:03:39.465
the effect size be that large And um they
the individual I was talking to and I

00:03:39.465 --> 00:03:42.575
was actually talking to two people one
one senior member and I'll tell you who

00:03:42.575 --> 00:03:48.085
the senior member was in a bit But uh at
the time I had no idea e either of these

00:03:48.085 --> 00:03:52.935
people Uh and one was a statistician It
turns out the statistician doesn't say

00:03:52.935 --> 00:03:59.405
anything during this conversation Um and
uh the the person says Oh we're seeing

00:03:59.405 --> 00:04:08.365
biases larger than that And um asks What
is the bias if you restrict to trials

00:04:08.375 --> 00:04:16.455
that uh hit the early success And I said
Well but that's not bias Bias is when

00:04:16.455 --> 00:04:22.315
you look at all the trials what is the
average effect relative to the truth If

00:04:22.315 --> 00:04:27.915
you restrict it to trials that hit that
early success that in and of itself is

00:04:27.925 --> 00:04:33.715
is is a bias Surely you're gonna get much
larger effect cause you have to have a

00:04:33.715 --> 00:04:39.095
large effect to win Um and and showed what
that was but you know that's not bias He

00:04:39.095 --> 00:04:45.025
says Well I I don't actually know what
the mathematical definition of bias is but

00:04:45.025 --> 00:04:51.275
that seems really problematic to me And
it was sort of strange uh in that I said

00:04:51.275 --> 00:04:57.115
Well any i if you look at any trial if
you look at a fixed trial and look at only

00:04:57.115 --> 00:05:01.895
those that are successful you're you're
gonna see much higher estimates claiming

00:05:01.895 --> 00:05:07.515
success You know it it has the same sort
of bias What was fascinating about it

00:05:07.515 --> 00:05:14.985
was it got the the the advice and the the
group running the trial was very closely

00:05:14.985 --> 00:05:20.885
associated with the device company and
needed the approval of the group Turns

00:05:20.885 --> 00:05:25.855
out they had to get rid of all the early
successes It was very weird I thought

00:05:25.855 --> 00:05:33.785
th th this is really strange uh uh you
know in in the whole thing So um the the

00:05:33.785 --> 00:05:39.565
individual that I was talking to I met for
the first time was Gordon Guyatt and we're

00:05:39.565 --> 00:05:45.645
gonna come back to that story But Kurt
this is a this is a thing that I think

00:05:45.645 --> 00:05:52.965
a number of people know that this is a
potential issue the bias and think it's a

00:05:53.195 --> 00:05:55.985
big issue with potentially stopping early

00:05:57.229 --> 00:05:59.729
Kert Viele: So I think, I, uh,
you know, one of the key things

00:05:59.739 --> 00:06:04.069
you brought out about that is
what is the definition of bias?

00:06:04.069 --> 00:06:05.429
What are you worried about?

00:06:05.829 --> 00:06:08.659
And, you know, ordinarily, when
we take this in cla- you know, a

00:06:08.659 --> 00:06:12.319
class in undergrad or so on, we
do-- we talk about all trials.

00:06:12.539 --> 00:06:15.049
Everything that could
happen, average them.

00:06:15.169 --> 00:06:18.589
On average, you get the right
answer, the right response rate.

00:06:19.649 --> 00:06:23.339
you cherry-pick, you're going
to generate some kind of bias

00:06:23.369 --> 00:06:24.279
because you're cherry-picking.

00:06:25.069 --> 00:06:30.899
I do have a little more sympathy, I think,
maybe than you do on this, that if I know

00:06:31.959 --> 00:06:38.349
I stopped at the first interim, I stopped
at half the planned sample size, I know

00:06:38.389 --> 00:06:43.239
these are the situations where I might
see the high part of the distribution.

00:06:44.109 --> 00:06:48.129
know, is that a concern, and
how big of a concern is it?

00:06:48.469 --> 00:06:51.009
I certainly can see people
wanting to compute that.

00:06:51.059 --> 00:06:54.009
So this is gonna be a key part
of our story going forward

00:06:54.577 --> 00:06:58.287
Scott: Yeah yeah So so we'll come back
to that And there's no question that

00:06:59.037 --> 00:07:04.607
trials that show statistical significance
overestimate the treatment effect I I

00:07:04.607 --> 00:07:10.407
think we all understand that um uh in
it A trial that stops early is going to

00:07:10.407 --> 00:07:15.517
provide likely an overestimate Um and
it all depends on what you think the the

00:07:15.947 --> 00:07:20.627
potential range of effects are But but
let's come back to the the definition

00:07:20.627 --> 00:07:27.627
I think that that this is so important
the definition of bias um uh in it So

00:07:28.567 --> 00:07:34.497
what we think of it is what's a biased
estimator An unbiased estimator would

00:07:34.507 --> 00:07:41.607
be one that when you run your experiment
and you you have an estimate of what

00:07:41.607 --> 00:07:45.157
it is whatever it is you're trying
to estimate the the success rate of a

00:07:45.157 --> 00:07:54.287
treatment If on average that estimate is
the real answer we call that unbiased So

00:07:54.287 --> 00:08:02.087
the the expected value of the estimator
we're using within our experiment is

00:08:02.127 --> 00:08:06.817
equal to the thing the estimand we're
trying to estimate We call that unbiased

00:08:07.369 --> 00:08:08.809
Kert Viele: And so we're,
we're, we're thinking about

00:08:08.869 --> 00:08:10.279
everything that could happen.

00:08:10.309 --> 00:08:13.509
I mean, this is all-- If I do
an interim, I don't just have

00:08:13.509 --> 00:08:16.859
early stopping for success, I've
got early stopping for futility.

00:08:17.099 --> 00:08:20.959
So I'm averaging, you know, I,
I stopped at fifty for success.

00:08:20.959 --> 00:08:24.819
I stopped at fifty for futility,
stopped at seventy-five success

00:08:24.819 --> 00:08:28.799
and futility, went to the end,
stopped for success or futility.

00:08:28.839 --> 00:08:31.079
All of those things average together.

00:08:31.529 --> 00:08:36.079
If those all average out to the
right answer, we're unbiased.

00:08:36.339 --> 00:08:39.029
And now we're gonna get into, you
know, do we need to look at each

00:08:39.029 --> 00:08:40.519
of those six things separately?

00:08:41.165 --> 00:08:51.225
Scott: Yep Yep Okay So um um and we know
we can calculate that if you do early

00:08:51.235 --> 00:08:57.395
superiority what happens is within the
trials you could run those trials that

00:08:57.395 --> 00:09:05.685
hit early superiority are are depending on
the the the real parameter the real truth

00:09:06.045 --> 00:09:12.185
likely are overperforming within that And
you stop at the point it's overperforming

00:09:12.185 --> 00:09:17.845
and it doesn't go the rest of the trial
and regress towards the parameter So w

00:09:17.855 --> 00:09:24.125
it when those trials stop and you report
that we know that's that's biased By the

00:09:24.125 --> 00:09:28.585
way my pet peeve and I know we generally
have one of Kert's pet peeves but one

00:09:28.585 --> 00:09:35.015
of my pet peeves is bias means you can't
use it That that all of a sudden it's

00:09:35.035 --> 00:09:42.045
it's uh oh it it it's done Bias is okay
Um we can adjust for it We understand it

00:09:42.055 --> 00:09:48.695
We want to know what is the size of the
bias uh in the circumstance It's just

00:09:48.995 --> 00:09:53.405
bias doesn't mean you can't do it And I
think there's there's this Oh bias you

00:09:53.525 --> 00:09:58.655
you can't do it So we understand there's
bias So let's talk a little bit about

00:09:58.675 --> 00:10:03.515
that Kurt Where like you know within a
toy example let's talk about you know

00:10:03.515 --> 00:10:06.285
what what what is the size of this bias

00:10:06.869 --> 00:10:09.849
Kert Viele: Well, and so let, let's
start back up a little bit further, and

00:10:09.935 --> 00:10:10.215
Scott: Okay

00:10:10.249 --> 00:10:14.409
Kert Viele: start for a fixed trial,
just a straightforward I run a

00:10:14.409 --> 00:10:19.869
trial, and my toy example here is
I just have straight normal data.

00:10:19.899 --> 00:10:21.539
I'm estimating a mean.

00:10:21.929 --> 00:10:23.669
I've arranged a trial.

00:10:23.679 --> 00:10:25.339
Sample size is 100.

00:10:25.339 --> 00:10:27.349
I'm gonna run a Z test at the end.

00:10:27.379 --> 00:10:29.389
Everything is as basic as it could be

00:10:29.873 --> 00:10:33.923
Scott: The one randomized
experimental to con placebo

00:10:34.285 --> 00:10:38.085
Kert Viele: Yep, standardized effect
size of one here just to make things

00:10:38.097 --> 00:10:39.077
Scott: Yep Yep

00:10:39.485 --> 00:10:44.885
Kert Viele: so if I, if I take all
trials and the truth is one, so I

00:10:44.915 --> 00:10:46.485
come in with what I powered for.

00:10:46.555 --> 00:10:50.235
The truth's one, I look at all
trials, I get an average of one.

00:10:50.545 --> 00:10:51.455
It's unbiased.

00:10:51.485 --> 00:10:52.025
They're the same.

00:10:53.023 --> 00:10:57.163
Scott: Futility no superiority
100 patients read it out Yep

00:10:57.225 --> 00:10:57.635
Kert Viele: Yep.

00:10:58.475 --> 00:11:02.415
so now if I-- Now all of those
trials that were unbiased as

00:11:02.415 --> 00:11:04.175
a whole, I can split them.

00:11:04.315 --> 00:11:07.395
I can talk about the ones that won,
you know, p less than zero point zero

00:11:07.395 --> 00:11:09.215
two five, and the ones that lost.

00:11:10.385 --> 00:11:15.815
if I look at the ones that won, they have
an average of about one point zero six.

00:11:15.945 --> 00:11:17.665
So they're biased high.

00:11:17.675 --> 00:11:19.175
They're one point zero six.

00:11:19.245 --> 00:11:21.845
They're compensated for
by the ones that lost.

00:11:21.895 --> 00:11:25.195
And so this is-- you have to
have this compensation going on.

00:11:26.005 --> 00:11:31.005
In fact, if you make the p-value the
stronger the evidence, if you make

00:11:31.005 --> 00:11:34.575
the p-value less than zero point
zero zero zero one, for example,

00:11:34.595 --> 00:11:35.925
so you're like, "Oh, look, I won.

00:11:35.925 --> 00:11:36.615
This is great.

00:11:36.615 --> 00:11:38.075
I have a great drug," and so on.

00:11:38.275 --> 00:11:40.895
Somebody could come
back and go, "Oh, wait."

00:11:41.625 --> 00:11:45.305
you focus on the trials with p less
than zero point zero zero zero one,

00:11:46.225 --> 00:11:48.725
average effect is one point three four.

00:11:48.765 --> 00:11:52.495
It's thirty-four percent bias
compared to the truth of one.

00:11:52.895 --> 00:11:56.295
So you want to be careful to go, "Oh,
you know, I can't accept anything that's

00:11:56.295 --> 00:11:59.915
biased," because you're essentially
saying the stronger the evidence,

00:11:59.955 --> 00:12:02.375
the less I believe it or something.

00:12:04.113 --> 00:12:08.073
Scott: Okay so under under the
truth that the the real effect is

00:12:08.073 --> 00:12:13.133
one where your trial is 90 powered
for one so most of the trials win

00:12:13.491 --> 00:12:13.671
Kert Viele: Yeah

00:12:13.863 --> 00:12:20.433
Scott: and only 10 of the trials don't
hit 05 If you restrict a calculation

00:12:20.433 --> 00:12:26.773
of what's the average effect for those
that win you get 106 So there's a 6

00:12:27.253 --> 00:12:32.773
bias just in successful trials in that
scenario Now if you were to do this

00:12:32.793 --> 00:12:36.553
under the null the bias would be enormous

00:12:36.941 --> 00:12:37.331
Kert Viele: Yeah, you

00:12:37.343 --> 00:12:38.693
Scott: for successful trials

00:12:38.711 --> 00:12:40.561
Kert Viele: yeah, you need about 0.6

00:12:40.571 --> 00:12:41.161
to win.

00:12:41.171 --> 00:12:45.111
Usually a 90% powered trial, you
need about 60% of the powered effect.

00:12:45.331 --> 00:12:48.461
So your bias has to be at least 0.6

00:12:48.461 --> 00:12:49.701
in those cases under the null

00:12:50.225 --> 00:12:55.445
Scott: So bias depends on the truth of
of the the the effect If the effect is

00:12:55.455 --> 00:13:01.505
three you you you're unbiased because
every trial wins and there's there's

00:13:01.505 --> 00:13:06.235
no bias to that So it d so so we're
addressing you know power for the effect

00:13:06.235 --> 00:13:09.735
of one uh within that scenario So the

00:13:10.195 --> 00:13:12.915
Kert Viele: and if you go the other way,
sorry to interrupt you, but if you go

00:13:12.915 --> 00:13:15.515
the other way, if the true effect was 1.5

00:13:15.515 --> 00:13:22.415
or two, it's higher, then you have 100%
power and you get no bias whatsoever.

00:13:22.725 --> 00:13:25.195
So it definitely depends
on the true effect

00:13:25.553 --> 00:13:30.033
Scott: Yep What happens if
I uh if I add futility Uh

00:13:30.733 --> 00:13:33.763
Kert Viele: So then you're gonna
end up where, um, now of course,

00:13:33.783 --> 00:13:38.063
futility in a fixed trial, you
just have the winners and losers.

00:13:38.123 --> 00:13:42.443
But if we start to add into the group
sequential, we add futility, that's

00:13:42.443 --> 00:13:46.773
gonna be a compensating mechanism for
the bias, or the upward bias will get

00:13:46.775 --> 00:13:47.055
Scott: Okay

00:13:47.113 --> 00:13:47.193
Kert Viele: get

00:13:47.445 --> 00:13:51.175
Scott: Okay But we have I I forgot
we've done fixed trials All we did

00:13:51.175 --> 00:13:57.425
was restrict in the you have this 6
bias of successful trials Now what

00:13:57.425 --> 00:14:04.035
if I do uh so you added into this
O'BrienFleming stopping at 50 and 75

00:14:04.813 --> 00:14:05.173
Kert Viele: Yep.

00:14:05.273 --> 00:14:11.083
And so s- if, so if I do that and I
don't have any futility, so futility is

00:14:11.083 --> 00:14:16.503
important here 'cause it does compensate,
then I'm gonna end up overall, I'm

00:14:16.503 --> 00:14:22.543
about-- I get an effect about one point
one five as opposed to one, so 15% up.

00:14:22.853 --> 00:14:24.423
So that's certainly meaningful.

00:14:24.833 --> 00:14:30.563
Um, you gotta weigh that against the fact
that I'm stopping early to help patients,

00:14:30.733 --> 00:14:32.403
and I don't know the effect is one.

00:14:32.413 --> 00:14:34.243
It might actually really be good.

00:14:34.863 --> 00:14:41.803
Um, if I, if the true effect is one and
I stop at 50 in that half the sample…

00:14:41.903 --> 00:14:42.393
Yeah

00:14:42.499 --> 00:14:47.589
Scott: make sure I There's this So
so over all possible trials you run

00:14:48.407 --> 00:14:49.447
Kert Viele: 1.15

00:14:49.839 --> 00:14:55.399
Scott: some of them stop at 50 some
stop at 75 some stop at 100 Not on not

00:14:55.399 --> 00:14:57.459
just successful trials but all trials

00:14:57.503 --> 00:14:57.993
Kert Viele: trials

00:14:58.119 --> 00:15:01.099
Scott: All trials that the the average

00:15:01.283 --> 00:15:01.593
Kert Viele: sorry.

00:15:01.623 --> 00:15:03.263
All trials is 1.07,

00:15:03.363 --> 00:15:03.623
so

00:15:03.793 --> 00:15:04.063
Scott: Yeah

00:15:04.123 --> 00:15:04.483
Kert Viele: correcting me

00:15:04.723 --> 00:15:13.073
Scott: okay Oh so so 107 is that So
the bias in a group sequential O'Brien

00:15:13.073 --> 00:15:18.363
Fleming um design there's a 7 bias

00:15:18.797 --> 00:15:19.697
Kert Viele: Yes, with

00:15:19.833 --> 00:15:25.583
Scott: that Yep yep So the question
is iif we just look at that that's

00:15:25.593 --> 00:15:32.163
the bias of adding it into the trial
uh uh within that scenario Now w

00:15:32.409 --> 00:15:34.599
Kert Viele: different than the
fixed trial that we just had.

00:15:34.629 --> 00:15:35.589
That was at six,

00:15:35.599 --> 00:15:36.199
so we're

00:15:36.315 --> 00:15:41.375
Scott: but that was restricting to only
successful trials Yep Uh so the the If

00:15:41.375 --> 00:15:46.825
somebody if the if a FDA says you're
running this group sequential design

00:15:47.245 --> 00:15:52.935
what is the bias of adding in those early
analyses Assuming that your effect size

00:15:52.935 --> 00:15:59.955
is one it would be 107 There would be a
7 uh uh difference from from the truth

00:15:59.965 --> 00:16:04.305
of that uh in that Now we can you can
adjust the estimate for that There are

00:16:04.305 --> 00:16:10.505
statistical ways to do that uh within that
But even that bias now we are we are able

00:16:10.505 --> 00:16:19.415
to suc correctly claim superiority perhaps
halfway through the experiment 75 of the

00:16:19.415 --> 00:16:23.985
way through the experiment The savings
in patients and times to get the right

00:16:23.985 --> 00:16:32.025
answer at the cost of this 7 bias that's
that's the sort of question And usually

00:16:32.475 --> 00:16:39.365
that seems like a nobrainer tradeoff
to me uh uh within that uh in those

00:16:39.365 --> 00:16:49.885
scenarios uh for that Okay But now what
happens if you only look at trials that

00:16:49.885 --> 00:16:54.065
are suc that win in a group sequential

00:16:54.243 --> 00:16:57.483
Kert Viele: so only the
ones that won, you get 1.15.

00:16:57.933 --> 00:16:59.953
So it's gonna be a little
bit higher because you don't

00:17:00.137 --> 00:17:00.157
Scott: Yeah

00:17:00.263 --> 00:17:01.313
Kert Viele: losers at the end

00:17:01.747 --> 00:17:06.617
Scott: And that includes the probability
you win at 50 at 75 or you go to

00:17:06.617 --> 00:17:11.337
the end and you're successful at the
end uh within that And so there is

00:17:11.567 --> 00:17:19.097
that increase Uh winning trials in
a fixed trials were 106 now it's 115

00:17:19.277 --> 00:17:19.697
Kert Viele: Yes

00:17:19.707 --> 00:17:24.797
Scott: that Now what if I only
look at trials that win at 50

00:17:25.463 --> 00:17:26.353
Kert Viele: Now that one's big.

00:17:26.393 --> 00:17:27.523
That's 1.5

00:17:28.113 --> 00:17:34.093
Scott: 15 yep Now that's not bias right

00:17:34.861 --> 00:17:36.461
Kert Viele: It's not
biased by the definition.

00:17:36.491 --> 00:17:39.761
I have a little more sympathy at
kind of a little uncomfortable

00:17:40.059 --> 00:17:48.509
Scott: Good Okay But that so that in that
scenario and of course in the scenario

00:17:48.509 --> 00:17:57.649
where you need to see that larger effect
size uh uh you know on average 15 uh to to

00:17:57.649 --> 00:18:02.769
win that to be statistically significant
in that scenario if you only look at

00:18:02.779 --> 00:18:07.799
those scenarios you get biased because
the truth is one and you have to have a

00:18:07.799 --> 00:18:13.909
value bigger than that Now if the truth
is two that's that bias is gonna be

00:18:14.093 --> 00:18:14.113
Kert Viele: Basically z-

00:18:14.169 --> 00:18:18.589
Scott: almost zero Right Right Yeah So
one of the points I think you would make

00:18:18.589 --> 00:18:22.849
is if you knew the effect was one you
wouldn't do an interim at 50 you know

00:18:22.849 --> 00:18:27.739
in that scenario Or if it's incredibly
unlikely that the scenario's as big

00:18:27.739 --> 00:18:30.919
as 15 you wouldn't do that interim

00:18:31.533 --> 00:18:34.743
Kert Viele: Yeah, we always, when,
when we show this to clients or

00:18:34.743 --> 00:18:39.743
when we do this, we back solve
what effect do you need to stop for

00:18:39.743 --> 00:18:41.703
success and futility at each interim?

00:18:42.013 --> 00:18:47.243
And we often ask clients directly,
especially on the futility side, "Are

00:18:47.243 --> 00:18:49.443
you comfortable stopping the trial here?

00:18:49.673 --> 00:18:52.593
Would you be willing to
basically give up at this point?"

00:18:52.893 --> 00:18:54.443
That applies for success.

00:18:54.463 --> 00:18:56.303
"Would you believe this result?"

00:18:56.733 --> 00:19:01.833
If the answer is no, then I have a lot
of concerns about doing the interim.

00:19:02.153 --> 00:19:06.093
Again, unless there's some public
health, you know, thing we need to have

00:19:06.093 --> 00:19:08.483
this answer as quickly as possible.

00:19:08.843 --> 00:19:11.993
I still trust the drug
works, uh, in that case,

00:19:12.017 --> 00:19:12.307
Scott: Yeah

00:19:12.323 --> 00:19:12.603
Kert Viele: I do

00:19:12.603 --> 00:19:12.933
worry

00:19:12.933 --> 00:19:14.683
that maybe it doesn't work that well

00:19:15.345 --> 00:19:19.965
Scott: Yeah And and so the estimate of
that So a very common situation and the

00:19:19.965 --> 00:19:26.035
common the situation I was in in the
device trial is that y the the company

00:19:26.035 --> 00:19:32.035
might believe in a the 15 is right but
they wanna make sure their trial is

00:19:32.065 --> 00:19:37.865
powered for one because it's clinically
is still a valuable device It's good for

00:19:37.865 --> 00:19:44.575
patients And so they power the trial at
90 at the maximum for the one but they

00:19:44.575 --> 00:19:51.795
really think they their device could be
as good as 15 175 or two in that scenario

00:19:51.795 --> 00:19:57.495
And then these the the bias under those
scenarios is much smaller and it's a

00:19:57.495 --> 00:20:02.105
much more reasonable thing to have a
variable sample size They actually think

00:20:02.115 --> 00:20:09.155
their effect is 15 and they wanna run a
trial that's only 70 but they go to 100

00:20:09.345 --> 00:20:14.005
potentially to still hit a clinically
meaningful effect uh within that scenario

00:20:14.015 --> 00:20:19.995
And then you're talking about biases of
7 by by doing this even under the one

00:20:19.995 --> 00:20:23.175
It's much smaller in the the 15 scenario

00:20:23.693 --> 00:20:26.253
Kert Viele: And I think this is where
you really-- this is where I really

00:20:26.253 --> 00:20:31.223
put on my Bayesian hat, so to speak,
even though I'm computing biases.

00:20:31.343 --> 00:20:35.363
When I see that data,
it's stopped at, at fifty.

00:20:35.553 --> 00:20:38.313
I've got an effect size
around one point five.

00:20:38.943 --> 00:20:43.363
You know, all of these bias calculations
assume that you know the truth.

00:20:43.403 --> 00:20:45.343
When I get the data, I
don't know the truth.

00:20:45.353 --> 00:20:46.413
I gotta figure it out.

00:20:46.803 --> 00:20:51.923
And so if, if I knew in my head it
can't be bigger than one, then of

00:20:51.923 --> 00:20:53.863
course I don't trust the one point five.

00:20:54.243 --> 00:20:57.903
But if I've walked into the trial
where this might not work at all,

00:20:58.143 --> 00:21:03.513
it might be one, it might be two, it
could be any of those, then I start

00:21:03.513 --> 00:21:08.173
thinking to myself, "Okay, if it were
a null, it hardly would ever stop."

00:21:08.263 --> 00:21:09.983
So it just-- I don't see many of these.

00:21:10.043 --> 00:21:12.563
I know they're biased, but they
just don't happen very often.

00:21:12.573 --> 00:21:13.583
Happen very rarely.

00:21:14.033 --> 00:21:16.993
If it's one, it still doesn't
happen all that often.

00:21:16.993 --> 00:21:19.203
I don't stop at fifty a ton.

00:21:19.603 --> 00:21:22.733
If it's two, I stop at fifty all the time.

00:21:22.983 --> 00:21:27.383
So when I see that one point five,
it's more likely to be one of

00:21:27.383 --> 00:21:32.603
the twos and is the right answer
than it's one of the biased ones.

00:21:32.653 --> 00:21:37.513
It's more likely to be an unbiased
truth than one of the biased falsehoods.

00:21:37.943 --> 00:21:39.433
And that, that gives me a lot of comfort

00:21:39.959 --> 00:21:43.859
Scott: Yeah You'd probably make
a small adjustment like 7 I think

00:21:43.859 --> 00:21:49.539
it's probably not quite you know
15 of what was observed yeah

00:21:49.827 --> 00:21:50.097
Kert Viele: I'd,

00:21:50.109 --> 00:21:50.609
Scott: that scenario

00:21:50.757 --> 00:21:51.947
Kert Viele: I'd, adjust by my prior.

00:21:51.987 --> 00:21:52.287
There's a

00:21:52.369 --> 00:21:52.609
Scott: Yep

00:21:52.737 --> 00:21:53.557
Kert Viele: Bayesian answer to

00:21:53.609 --> 00:21:54.369
Scott: Yes Yeah

00:21:54.497 --> 00:21:56.267
Kert Viele: we've had our
frequentist hat on, but

00:21:56.429 --> 00:21:56.709
Scott: Yeah

00:21:57.027 --> 00:21:58.997
Kert Viele: would just compute
the posterior and it works.

00:21:59.027 --> 00:21:59.157
It

00:21:59.447 --> 00:21:59.577
Scott: Yep

00:21:59.637 --> 00:22:00.577
Kert Viele: bias automatically

00:22:00.817 --> 00:22:06.857
Scott: Yep Okay so we can calculate
the bias uh for that And as we do more

00:22:06.857 --> 00:22:12.597
complicated designs and you could add
response adaptive randomization you whi

00:22:12.617 --> 00:22:18.417
which we've done a recent uh episode of
We could add arm dropping in here We could

00:22:18.427 --> 00:22:24.157
add patient changing We can do all this
We can calculate the exact bias under

00:22:24.157 --> 00:22:27.577
a number of scenarios It's one of the
really nice things about simulation So

00:22:27.867 --> 00:22:32.337
we can calculate bias and we know what
it is Mathematically we can calculate it

00:22:34.069 --> 00:22:36.169
Kert Viele: And we should certainly
emphasize this is one of the

00:22:36.169 --> 00:22:39.859
things that in the regulatory
world we've been doing a lot more.

00:22:39.869 --> 00:22:43.119
The last five, six, seven years,
regulators have been paying a lot

00:22:43.119 --> 00:22:45.439
more attention to this, asking for it.

00:22:45.489 --> 00:22:49.849
And so this is-- we, we've been
providing these and for the most

00:22:49.859 --> 00:22:52.999
part, things are going fine, but
we have found a few examples, and

00:22:52.999 --> 00:22:54.729
we've changed the designs in response

00:22:55.983 --> 00:23:01.423
Scott: And and it it the FDA draft
guidance uh the FDA guidance on adaptive

00:23:01.423 --> 00:23:08.193
designs the ICH E20 adaptive designs
uh that's draft They they all mention

00:23:08.253 --> 00:23:18.173
calculating the bias uh in it I uh I
don't know if we have submitted a design

00:23:18.193 --> 00:23:26.243
over 27 years of doing this to the agency
where they've had an issue with the bias

00:23:28.237 --> 00:23:29.467
Kert Viele: That sounds fair to me,

00:23:29.979 --> 00:23:32.789
Scott: I think yes if we were to
do a sample if we were to do an

00:23:32.799 --> 00:23:38.219
interim at 10 patients in your 100
patients example um uh th th there

00:23:38.219 --> 00:23:40.229
could be a an issue with that Yeah

00:23:40.251 --> 00:23:42.591
Kert Viele: cherry-picking because
we don't actually let it get to that

00:23:42.591 --> 00:23:43.181
point or at least

00:23:43.319 --> 00:23:47.009
Scott: So we wouldn't submit that design
when we calculate the bias We say Wow

00:23:47.269 --> 00:23:51.689
if I were a regulator I'd have a problem
with that sort of thing So I it you

00:23:51.689 --> 00:23:55.569
know it's something that they understand
they recognize when they have to write

00:23:55.569 --> 00:24:02.139
an FDA label I think there's comfort
there with it um in it Uh and we can

00:24:02.139 --> 00:24:06.099
calculate that exactly Okay so I keep
saying that over and over again and

00:24:06.099 --> 00:24:10.349
people are w w you know Why why does he
keep saying that We can calculate that

00:24:10.349 --> 00:24:13.689
sort of thing Coming back to my story

00:24:14.837 --> 00:24:17.097
Kert Viele: I think this is, this is
definitely your pet peeve episode.

00:24:17.745 --> 00:24:22.735
Scott: Yeah Okay All right I'll I'll take
that I'll take that Um but you brought

00:24:22.735 --> 00:24:27.405
it on You you said Hey we should do an
episode Oh and I I I remember this story

00:24:27.431 --> 00:24:29.451
Kert Viele: triggering Scott
is a p- is a hobby of mine

00:24:30.175 --> 00:24:36.645
Scott: Yes So um Gordon Guyatt you may not
know care Gordon Guyatt What happened is

00:24:37.145 --> 00:24:46.975
uh within a few months of that interaction
a paper came out uh in JAMA and the

00:24:46.975 --> 00:24:56.185
lead author is uh Bassler Uh and this
is Bassler et al The senior author in it

00:24:56.195 --> 00:25:03.015
is Gordon Guyatt and it's on behalf of
something called the STOPIT Study Group

00:25:03.015 --> 00:25:12.145
STOPIT 2 Study Group And the paper is So
this is uh y uh this comes out in March

00:25:12.465 --> 00:25:22.965
of 2010 and in JAMA and it is Stopping
Randomized Trials Early for Benefit

00:25:23.475 --> 00:25:30.425
and Estimation of Treatment Effects A
Systematic Review and MetaRegression

00:25:30.425 --> 00:25:37.295
Analysis So I'll read you the conclusion
of the paper and then describe a little

00:25:37.295 --> 00:25:42.135
bit about what they do And by the way
I I will say this as I'm I'm describing

00:25:42.135 --> 00:25:48.185
this that uh in in in this many years I
I think this paper is just wrong I think

00:25:48.185 --> 00:25:52.935
it's misleading I think it's JAMA should
have never published it Uh but we'll

00:25:52.955 --> 00:25:57.965
come more to that But just to sort of
set the scenes for it So the conclusion

00:25:57.965 --> 00:26:07.165
is truncated randomized clinical trials
were associated with greater effect sizes

00:26:07.205 --> 00:26:15.655
than randomized clinical trials that's
not stopped early That's the conclusion

00:26:16.035 --> 00:26:22.755
um uh within this But one of the
sentences in the paper says Statistical

00:26:22.755 --> 00:26:29.965
modeling suggest that randomized clinical
trials stopped early for benefit and

00:26:29.965 --> 00:26:34.745
they call these truncated randomized
clinical trials will systematically

00:26:34.765 --> 00:26:41.035
overestimate treatment effects And
empirical data demonstrate that truncated

00:26:41.045 --> 00:26:46.845
randomized clinical trials often show
implausibly large treatment effects

00:26:49.211 --> 00:26:54.791
So a lot of people reading this paper
looking at it come away with You

00:26:54.791 --> 00:27:02.371
shouldn't do interim analyses It creates
implausibly large bias This is bad You

00:27:02.371 --> 00:27:07.011
know the name of their group is STOPIT
Um and I don't think they mean trials

00:27:07.011 --> 00:27:12.421
I think they mean it And he literally
in that example prevented a company

00:27:12.421 --> 00:27:20.351
from doing a group sequential design um
because of this issue Okay so what did

00:27:20.351 --> 00:27:27.431
they do Uh what what A and their goal
was understanding we can mathematically

00:27:27.451 --> 00:27:34.501
calculate bias but what do we actually
see out there for bias So they went out

00:27:34.501 --> 00:27:42.061
and they found through a a a search they
found trials that were stopped early

00:27:42.101 --> 00:27:51.381
truncated early for superiority And you
can go to their their uh CONSORT diagram

00:27:51.761 --> 00:28:00.871
They originally found 195 trials that were
stopped early Then they looked for trials

00:28:00.881 --> 00:28:09.331
of the same question other randomized
trials of the same question and they were

00:28:09.331 --> 00:28:15.521
gonna compare the effect of these other
trials to the ones that were truncated

00:28:16.261 --> 00:28:22.631
early And by the way in their search of
where they had 195 trials stopped early

00:28:23.041 --> 00:28:27.381
and then they looked for other trials
they found some more that were stopped

00:28:27.381 --> 00:28:35.561
early They moved those over to this
calling them truncated And then yeah

00:28:35.823 --> 00:28:39.723
Kert Viele: if you replicated the
result, you moved it back over and

00:28:39.723 --> 00:28:40.993
it didn't count as a replication

00:28:43.213 --> 00:28:43.803
But yeah

00:28:43.873 --> 00:28:46.123
Scott: you got early truncated

00:28:46.459 --> 00:28:46.709
Kert Viele: Yes,

00:28:46.833 --> 00:28:47.503
Scott: then yeah

00:28:47.759 --> 00:28:50.159
Kert Viele: if you repeated the
experiment, it-- you didn't get

00:28:50.159 --> 00:28:53.499
credit for the replication of
it 'cause you truncated it early

00:28:53.937 --> 00:28:57.657
Scott: Yep yep You go into th
they're gonna create a largely a

00:28:57.677 --> 00:29:04.527
treatment group of trials which were
ones that were stopped early and

00:29:04.527 --> 00:29:10.747
then the other group are the same
question that were not stopped early

00:29:11.541 --> 00:29:12.361
Kert Viele: And some
of them could have been

00:29:13.617 --> 00:29:17.487
Scott: Some of them could have been
Um some of them were statistically

00:29:17.487 --> 00:29:19.727
significantly successful some were not

00:29:20.105 --> 00:29:20.595
Kert Viele: And it's wor-

00:29:20.597 --> 00:29:20.967
Scott: within the

00:29:21.065 --> 00:29:24.025
Kert Viele: it's worth mentioning
that if you could stop early and you

00:29:24.045 --> 00:29:26.835
don't, those trials are biased low.

00:29:27.485 --> 00:29:31.035
So you're-- they're-- we mathematically
know they're gonna be a low group

00:29:32.721 --> 00:29:37.211
Scott: So there they and this was their
empirically what happens out there So

00:29:37.211 --> 00:29:44.221
they took this set of trials that were
stopped early and then they compared to

00:29:44.221 --> 00:29:56.161
the same question that were not stopped
early And they found that the difference

00:29:56.181 --> 00:30:03.771
was about a 30 difference in the estimate
of the effect They did a r a relative

00:30:03.771 --> 00:30:11.611
risk and it was 71 So a 29 relative
difference in the effect of those stopped

00:30:11.611 --> 00:30:18.231
early versus those that were not And
they jump up and down and say This is

00:30:18.231 --> 00:30:25.171
enormous A 30 effect So in your particular
example we had that effect of one If we

00:30:25.171 --> 00:30:32.381
did something that created an average
effect of say 13 we might be concerned

00:30:32.381 --> 00:30:39.691
that that experiment provides bias
and that was that was their experiment

00:30:39.711 --> 00:30:44.831
that they did in this comparison and
they published the results of that

00:30:46.613 --> 00:30:48.783
Kert Viele: And we actually,
so we have that result.

00:30:48.783 --> 00:30:53.693
So the, if you look in our-- the example
we gave, the one hundred sample size and

00:30:53.693 --> 00:31:00.223
so on, if you look at the trials that went
to the end and won, average effect is only

00:31:00.233 --> 00:31:02.373
zero point eight seven compared to one.

00:31:02.463 --> 00:31:05.883
They're biased low by a fair
amount, thirteen percent

00:31:06.969 --> 00:31:12.249
Scott: Yep So in a way they they present
this result and think you're you're all

00:31:12.299 --> 00:31:18.119
clinical trial people and you understand
that we we in a clinical trial as simple

00:31:18.119 --> 00:31:23.129
as twoarm trial we have a treatment group
and a control group and we randomize

00:31:23.169 --> 00:31:31.189
people into those groups and we compare
their results They took this group that

00:31:31.269 --> 00:31:37.639
the trial was truncated early and they
call that group one and then they take

00:31:37.649 --> 00:31:42.229
trials that were not stopped early If it
was stopped early they moved it over into

00:31:42.229 --> 00:31:47.849
group one that were not stopped early and
they called that group two And they said

00:31:47.859 --> 00:31:54.049
Wow group one's got a better treatment
effect than group two And but it by the

00:31:54.049 --> 00:31:59.419
way if you ran a clinical trial like this
it would be that you run a trial and you

00:31:59.429 --> 00:32:06.439
have patients and the ones that respond
well you put them in treatment one and if

00:32:06.439 --> 00:32:09.509
they don't respond well you put them in
treatment two and you say Wow treatment

00:32:09.509 --> 00:32:14.739
one is better than treatment two That
would never get into JAMA but somehow

00:32:14.749 --> 00:32:22.729
this got into JAMA Interestingly when this
was published there were many letters to

00:32:22.729 --> 00:32:28.629
the editor Uh I was a part of one of them
with Brad Carlin and Jason Connor where

00:32:28.629 --> 00:32:35.719
we point out that this is this is faulty
and wrong Uh but the names of the people

00:32:35.729 --> 00:32:42.629
that wrote in saying this was wrong was
somewhat impressive Another one was Steve

00:32:42.669 --> 00:32:47.929
Goodman Don Berry My father wrote a a
letter to the editor a different one than

00:32:47.929 --> 00:32:55.509
the one I was on Uh w and Janet Wittes
uh was on were on that one Another one

00:32:55.509 --> 00:33:03.059
of these letters was Ed Corn and Margaret
Mooney Another one was Susan Ellenberg

00:33:03.069 --> 00:33:10.989
Dave Demetz and Tom Fleming also pointing
out the faults of this And so there were

00:33:10.989 --> 00:33:18.169
a lot of people that pointed out that this
was a bad study uh in it but yet still

00:33:18.169 --> 00:33:24.149
to A and by the way think about it that
you go in and you take trials early If

00:33:24.149 --> 00:33:28.489
they would've done the same thing where
they went out and found trials that were

00:33:28.489 --> 00:33:36.883
statistically significant And then they
went out and found trials that a a any

00:33:36.923 --> 00:33:41.433
other trials run even some that were
statistically significant they would

00:33:41.433 --> 00:33:47.153
see that that's biased uh within that
and that's largely the bias that they

00:33:47.153 --> 00:33:52.683
saw in that And we know there's a bias
to stopping early We can calculate it

00:33:52.683 --> 00:34:00.453
mathematically So when their conclusion
was This is implausibly large I think they

00:34:00.453 --> 00:34:09.523
understood the mathematical calculation
of 7 and we're seeing 30 They even

00:34:09.553 --> 00:34:16.223
study things like did the DSMB have some
investment in it You know all the Looking

00:34:16.223 --> 00:34:22.423
for chicanery here or something like that
within this and it wa it's like no this

00:34:22.423 --> 00:34:29.283
had to happen You you know if you if you
ran an experiment where you took the the

00:34:29.293 --> 00:34:37.373
biased sampling that they did you would
see this 30 bias This is this is nothing

00:34:37.949 --> 00:34:41.239
Kert Viele: I, I'm laughing, Scott,
because I, I was truly successful

00:34:41.239 --> 00:34:42.419
about getting you riled up today

00:34:42.801 --> 00:34:47.701
Scott: Yeah yeah So uh th th this paper's
sort of number one on my list You know

00:34:47.701 --> 00:34:53.211
others uh we'll do other podcasts of
other ones uh within the scenario But

00:34:53.401 --> 00:34:58.291
uh and it was kind of amazing at the
time because I I tried with w w with

00:34:58.291 --> 00:35:04.941
fault to explain this to to Dr Guyatt
uh in that I was unsuccessful The paper

00:35:04.941 --> 00:35:09.631
came out and I think there was nobody
that sort of said this is odd He did

00:35:09.631 --> 00:35:16.711
a debate with Steve Goodman after this
paper came out where Goodman explained

00:35:16.711 --> 00:35:21.931
this and I think he largely I'm not sure
he sort of understood the intricacies

00:35:21.931 --> 00:35:27.261
and largely it was like their rebuttal
to these responses that Well boy look

00:35:27.261 --> 00:35:34.011
this is big Yeah it by by definition
you bias sampled in a way that created

00:35:34.021 --> 00:35:37.861
this Nobody should be surprised by that

00:35:38.639 --> 00:35:41.309
Kert Viele: And I, I mean, to
be-- make sure we summarize

00:35:41.309 --> 00:35:44.679
this properly, we certainly are
claiming there's, there's bias.

00:35:44.689 --> 00:35:46.349
So there is some bias.

00:35:46.389 --> 00:35:47.769
We can compute what it is.

00:35:48.249 --> 00:35:53.209
Um, it, that has to be
balanced versus the advantages.

00:35:53.209 --> 00:35:54.739
We stop early for reasons.

00:35:54.779 --> 00:35:56.999
We have-- we want to protect patients.

00:35:56.999 --> 00:36:02.109
The DAWN trial, for example, this
is a trial that thrombectomy, great

00:36:02.109 --> 00:36:06.619
effect, interim at 200 patients,
maximal sample size, I think

00:36:06.629 --> 00:36:08.409
500 Am I remembering that right?

00:36:08.567 --> 00:36:08.657
Scott: Yep

00:36:08.919 --> 00:36:09.709
Kert Viele: So,

00:36:09.889 --> 00:36:10.409
you know, d--

00:36:10.627 --> 00:36:11.987
Scott: stopped at f 200

00:36:12.099 --> 00:36:17.069
Kert Viele: stopped at 200 So enormous
effect that's panned out in practice.

00:36:17.069 --> 00:36:19.559
This is, you know, this is
the standard of care now.

00:36:19.949 --> 00:36:24.629
If that trial doesn't stop
at 200 it years before this.

00:36:24.679 --> 00:36:26.269
There are other trials with thrombectomy.

00:36:26.269 --> 00:36:28.829
I don't want to claim it's years
before that reaches the market,

00:36:29.169 --> 00:36:33.199
but it's years before that evidence
is actually in front of people

00:36:33.905 --> 00:36:38.755
Scott: Yeah Uh and and so there's
there's huge benefits And also by the

00:36:38.755 --> 00:36:44.875
way it allows you to have a wider range
to have a higher powered trial where

00:36:44.875 --> 00:36:50.655
you wouldn't run that single maximum
because you think it's wasted cause

00:36:50.655 --> 00:36:56.065
you don't need that many There's a huge
benefit to having a flexible sample size

00:36:56.365 --> 00:37:03.875
And this painting it Oh it's biased is
uh you shouldn't do it ignores all of

00:37:03.885 --> 00:37:07.355
these important factors to trial design

00:37:07.851 --> 00:37:11.461
Kert Viele: And I do think you have to
incorporate that entire range of evidence,

00:37:11.571 --> 00:37:13.311
the entire range of possibilities.

00:37:13.711 --> 00:37:18.501
I can't just compute bias on suppose the
null is true, suppose this effect is true.

00:37:19.401 --> 00:37:20.181
the range?

00:37:20.211 --> 00:37:21.841
What's the bias for each of these?

00:37:21.861 --> 00:37:24.411
What's the likelihood I'm
actually gonna see these?

00:37:24.861 --> 00:37:27.431
It's like a doctor
getting a diagnostic test.

00:37:27.791 --> 00:37:29.051
Here's what's in front of me.

00:37:29.331 --> 00:37:35.091
Is this a unbiased, basically unbiased
real result, or is this biased high?

00:37:35.161 --> 00:37:40.511
And, uh, if it's plausible, it's probably
a reasonably unbiased real result

00:37:42.289 --> 00:37:44.649
Scott: Yep All right

00:37:48.423 --> 00:37:48.823
I I

00:37:49.069 --> 00:37:49.229
Kert Viele: You

00:37:49.333 --> 00:37:50.193
Scott: I'm trying to make sure

00:37:50.359 --> 00:37:51.379
Kert Viele: You need,
a second to calm down?

00:37:51.793 --> 00:37:53.703
Scott: Yeah I'm calmed
down I'm calmed down

00:37:53.829 --> 00:37:54.119
Kert Viele: All right

00:37:54.153 --> 00:37:59.963
Scott: Yep yep uh in this one All right So
and I I'm not lost the the the bit of the

00:37:59.963 --> 00:38:05.193
irony that uh we're talking about bias of
interims and here we are in the interim

00:38:05.673 --> 00:38:13.453
uh in this scenario But uh yep that's the
the the world we live in here I I I would

00:38:13.453 --> 00:38:20.043
encourage you to go look at the Bassler
paper Um you know if if if you have your

00:38:20.053 --> 00:38:24.733
study groups and read it I think you
should read it and talk about the paper

00:38:24.993 --> 00:38:29.533
whether you think it's reasonable Read
the uh comments as well I think you'll

00:38:29.533 --> 00:38:31.633
you'll enjoy the various people commenting

00:38:32.015 --> 00:38:34.095
Kert Viele: And, look, look at your
interims when you're doing them.

00:38:34.135 --> 00:38:36.915
They're not-- every, every
interim shouldn't be there.

00:38:37.025 --> 00:38:41.515
So feel free to compute these
quantities and to what's reasonable

00:38:43.255 --> 00:38:48.735
Scott: Yep Yep All right Well I I
appreciate you bringing this topic

00:38:48.765 --> 00:38:55.425
to me Kurt and see what other neat
topics we have coming uh uh there And

00:38:55.425 --> 00:38:59.925
we're gonna stop this we're gonna stop
this episode a little early here Kurt

00:39:00.295 --> 00:39:00.665
Kert Viele: Okay

00:39:00.745 --> 00:39:06.785
Scott: I think, that's appropriate Uh it's
a little early and and we are biased here

00:39:07.065 --> 00:39:13.275
uh and we will be biased next time So uh
thank you all for joining us and until

00:39:13.285 --> 00:39:16.105
next time We will be here in the interim

00:39:16.693 --> 00:39:17.193
Kert Viele: Thanks, Scott