Judith: Welcome to Berry's In the Interim podcast, where we explore the
cutting edge of innovative clinical trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott: Welcome everybody back to In The Interim Uh I am your cohost today Scott
Berry and I'm joined by my uh uh my common cohost Kurt Vielli here and we we are
here to talk about something And and and Kurt will occasionally say Oh I've got
a good topic for the podcast and usually it's something that's uh beneath the
crawl a little bit maybe uh wi wi within the the the pet peeve But a really I I
think a really important topic a really important misunderstood topic and that is
bias in early stopping in clinical trials So welcome back to In The Interim Kurt
Kert Viele: Thank you
Scott: All right So so we are in the interim and we're talking about bias
of interims Uh sort of an interesting aspect of this I I thought I'd introduce
the topic by uh uh telling a a bit of an old story and um this this happened
relatively early So I I started designing trials in 2000 Uh before that I did a
little bit of clinical design but uh this was early in Berry Consultants and was
working with a medical device company And they they had an estimate of their
effect size but thought that they could be quite a bit better than that So an
adaptive sample size with early stopping for superiority made a lot of sense for
the trial So fairly standard two interim analyses for early success could go to the
final analysis Medical device uh nothing controversial in in the trial design We
can control type one error It's not the issue Pretty straightforward And so we're
working on the design and then I got word from the the company that there's a
problem um with stopping And the company was associated with a group that was gonna
run the trial and they had an issue with potentially stopping the trial I thought
Okay that's that that's odd You know let's let's talk to them about it So the
the concern was the bias in the estimate that comes out of the trial and it's
important for medical decisionmaking to have good estimates of of the effect And
the concern was that the early stopping biases that estimate And so ookay that's
great and one of the wonderful things about simulation is we can simulate the
bias So it was a response outcome uh success and failure And I simulated the
trial under a number of potential true effects and calculated the bias of the
trial And it was really relatively small It was approximately one percent in a in
a rate of success of We're we're looking for thirty forty fifty percent success
rates and it was a an s a bias of about one percent which I thought was a minor
Thing relative to the benefits of sample size patients time to get a success should
the effect size be that large And um they the individual I was talking to and I
was actually talking to two people one one senior member and I'll tell you who
the senior member was in a bit But uh at the time I had no idea e either of these
people Uh and one was a statistician It turns out the statistician doesn't say
anything during this conversation Um and uh the the person says Oh we're seeing
biases larger than that And um asks What is the bias if you restrict to trials
that uh hit the early success And I said Well but that's not bias Bias is when
you look at all the trials what is the average effect relative to the truth If
you restrict it to trials that hit that early success that in and of itself is
is is a bias Surely you're gonna get much larger effect cause you have to have a
large effect to win Um and and showed what that was but you know that's not bias He
says Well I I don't actually know what the mathematical definition of bias is but
that seems really problematic to me And it was sort of strange uh in that I said
Well any i if you look at any trial if you look at a fixed trial and look at only
those that are successful you're you're gonna see much higher estimates claiming
success You know it it has the same sort of bias What was fascinating about it
was it got the the the advice and the the group running the trial was very closely
associated with the device company and needed the approval of the group Turns
out they had to get rid of all the early successes It was very weird I thought
th th this is really strange uh uh you know in in the whole thing So um the the
individual that I was talking to I met for the first time was Gordon Guyatt and we're
gonna come back to that story But Kurt this is a this is a thing that I think
a number of people know that this is a potential issue the bias and think it's a
big issue with potentially stopping early
Kert Viele: So I think, I, uh, you know, one of the key things
you brought out about that is what is the definition of bias?
What are you worried about?
And, you know, ordinarily, when we take this in cla- you know, a
class in undergrad or so on, we do-- we talk about all trials.
Everything that could happen, average them.
On average, you get the right answer, the right response rate.
you cherry-pick, you're going to generate some kind of bias
because you're cherry-picking.
I do have a little more sympathy, I think, maybe than you do on this, that if I know
I stopped at the first interim, I stopped at half the planned sample size, I know
these are the situations where I might see the high part of the distribution.
know, is that a concern, and how big of a concern is it?
I certainly can see people wanting to compute that.
So this is gonna be a key part of our story going forward
Scott: Yeah yeah So so we'll come back to that And there's no question that
trials that show statistical significance overestimate the treatment effect I I
think we all understand that um uh in it A trial that stops early is going to
provide likely an overestimate Um and it all depends on what you think the the
potential range of effects are But but let's come back to the the definition
I think that that this is so important the definition of bias um uh in it So
what we think of it is what's a biased estimator An unbiased estimator would
be one that when you run your experiment and you you have an estimate of what
it is whatever it is you're trying to estimate the the success rate of a
treatment If on average that estimate is the real answer we call that unbiased So
the the expected value of the estimator we're using within our experiment is
equal to the thing the estimand we're trying to estimate We call that unbiased
Kert Viele: And so we're, we're, we're thinking about
everything that could happen.
I mean, this is all-- If I do an interim, I don't just have
early stopping for success, I've got early stopping for futility.
So I'm averaging, you know, I, I stopped at fifty for success.
I stopped at fifty for futility, stopped at seventy-five success
and futility, went to the end, stopped for success or futility.
All of those things average together.
If those all average out to the right answer, we're unbiased.
And now we're gonna get into, you know, do we need to look at each
of those six things separately?
Scott: Yep Yep Okay So um um and we know we can calculate that if you do early
superiority what happens is within the trials you could run those trials that
hit early superiority are are depending on the the the real parameter the real truth
likely are overperforming within that And you stop at the point it's overperforming
and it doesn't go the rest of the trial and regress towards the parameter So w
it when those trials stop and you report that we know that's that's biased By the
way my pet peeve and I know we generally have one of Kert's pet peeves but one
of my pet peeves is bias means you can't use it That that all of a sudden it's
it's uh oh it it it's done Bias is okay Um we can adjust for it We understand it
We want to know what is the size of the bias uh in the circumstance It's just
bias doesn't mean you can't do it And I think there's there's this Oh bias you
you can't do it So we understand there's bias So let's talk a little bit about
that Kurt Where like you know within a toy example let's talk about you know
what what what is the size of this bias
Kert Viele: Well, and so let, let's start back up a little bit further, and
Scott: Okay
Kert Viele: start for a fixed trial, just a straightforward I run a
trial, and my toy example here is I just have straight normal data.
I'm estimating a mean.
I've arranged a trial.
Sample size is 100.
I'm gonna run a Z test at the end.
Everything is as basic as it could be
Scott: The one randomized experimental to con placebo
Kert Viele: Yep, standardized effect size of one here just to make things
Scott: Yep Yep
Kert Viele: so if I, if I take all trials and the truth is one, so I
come in with what I powered for.
The truth's one, I look at all trials, I get an average of one.
It's unbiased.
They're the same.
Scott: Futility no superiority 100 patients read it out Yep
Kert Viele: Yep.
so now if I-- Now all of those trials that were unbiased as
a whole, I can split them.
I can talk about the ones that won, you know, p less than zero point zero
two five, and the ones that lost.
if I look at the ones that won, they have an average of about one point zero six.
So they're biased high.
They're one point zero six.
They're compensated for by the ones that lost.
And so this is-- you have to have this compensation going on.
In fact, if you make the p-value the stronger the evidence, if you make
the p-value less than zero point zero zero zero one, for example,
so you're like, "Oh, look, I won.
This is great.
I have a great drug," and so on.
Somebody could come back and go, "Oh, wait."
you focus on the trials with p less than zero point zero zero zero one,
average effect is one point three four.
It's thirty-four percent bias compared to the truth of one.
So you want to be careful to go, "Oh, you know, I can't accept anything that's
biased," because you're essentially saying the stronger the evidence,
the less I believe it or something.
Scott: Okay so under under the truth that the the real effect is
one where your trial is 90 powered for one so most of the trials win
Kert Viele: Yeah
Scott: and only 10 of the trials don't hit 05 If you restrict a calculation
of what's the average effect for those that win you get 106 So there's a 6
bias just in successful trials in that scenario Now if you were to do this
under the null the bias would be enormous
Kert Viele: Yeah, you
Scott: for successful trials
Kert Viele: yeah, you need about 0.6
to win.
Usually a 90% powered trial, you need about 60% of the powered effect.
So your bias has to be at least 0.6
in those cases under the null
Scott: So bias depends on the truth of of the the the effect If the effect is
three you you you're unbiased because every trial wins and there's there's
no bias to that So it d so so we're addressing you know power for the effect
of one uh within that scenario So the
Kert Viele: and if you go the other way, sorry to interrupt you, but if you go
the other way, if the true effect was 1.5
or two, it's higher, then you have 100% power and you get no bias whatsoever.
So it definitely depends on the true effect
Scott: Yep What happens if I uh if I add futility Uh
Kert Viele: So then you're gonna end up where, um, now of course,
futility in a fixed trial, you just have the winners and losers.
But if we start to add into the group sequential, we add futility, that's
gonna be a compensating mechanism for the bias, or the upward bias will get
Scott: Okay
Kert Viele: get
Scott: Okay But we have I I forgot we've done fixed trials All we did
was restrict in the you have this 6 bias of successful trials Now what
if I do uh so you added into this O'BrienFleming stopping at 50 and 75
Kert Viele: Yep.
And so s- if, so if I do that and I don't have any futility, so futility is
important here 'cause it does compensate, then I'm gonna end up overall, I'm
about-- I get an effect about one point one five as opposed to one, so 15% up.
So that's certainly meaningful.
Um, you gotta weigh that against the fact that I'm stopping early to help patients,
and I don't know the effect is one.
It might actually really be good.
Um, if I, if the true effect is one and I stop at 50 in that half the sampleâ¦
Yeah
Scott: make sure I There's this So so over all possible trials you run
Kert Viele: 1.15
Scott: some of them stop at 50 some stop at 75 some stop at 100 Not on not
just successful trials but all trials
Kert Viele: trials
Scott: All trials that the the average
Kert Viele: sorry.
All trials is 1.07,
so
Scott: Yeah
Kert Viele: correcting me
Scott: okay Oh so so 107 is that So the bias in a group sequential O'Brien
Fleming um design there's a 7 bias
Kert Viele: Yes, with
Scott: that Yep yep So the question is iif we just look at that that's
the bias of adding it into the trial uh uh within that scenario Now w
Kert Viele: different than the fixed trial that we just had.
That was at six,
so we're
Scott: but that was restricting to only successful trials Yep Uh so the the If
somebody if the if a FDA says you're running this group sequential design
what is the bias of adding in those early analyses Assuming that your effect size
is one it would be 107 There would be a 7 uh uh difference from from the truth
of that uh in that Now we can you can adjust the estimate for that There are
statistical ways to do that uh within that But even that bias now we are we are able
to suc correctly claim superiority perhaps halfway through the experiment 75 of the
way through the experiment The savings in patients and times to get the right
answer at the cost of this 7 bias that's that's the sort of question And usually
that seems like a nobrainer tradeoff to me uh uh within that uh in those
scenarios uh for that Okay But now what happens if you only look at trials that
are suc that win in a group sequential
Kert Viele: so only the ones that won, you get 1.15.
So it's gonna be a little bit higher because you don't
Scott: Yeah
Kert Viele: losers at the end
Scott: And that includes the probability you win at 50 at 75 or you go to
the end and you're successful at the end uh within that And so there is
that increase Uh winning trials in a fixed trials were 106 now it's 115
Kert Viele: Yes
Scott: that Now what if I only look at trials that win at 50
Kert Viele: Now that one's big.
That's 1.5
Scott: 15 yep Now that's not bias right
Kert Viele: It's not biased by the definition.
I have a little more sympathy at kind of a little uncomfortable
Scott: Good Okay But that so that in that scenario and of course in the scenario
where you need to see that larger effect size uh uh you know on average 15 uh to to
win that to be statistically significant in that scenario if you only look at
those scenarios you get biased because the truth is one and you have to have a
value bigger than that Now if the truth is two that's that bias is gonna be
Kert Viele: Basically z-
Scott: almost zero Right Right Yeah So one of the points I think you would make
is if you knew the effect was one you wouldn't do an interim at 50 you know
in that scenario Or if it's incredibly unlikely that the scenario's as big
as 15 you wouldn't do that interim
Kert Viele: Yeah, we always, when, when we show this to clients or
when we do this, we back solve what effect do you need to stop for
success and futility at each interim?
And we often ask clients directly, especially on the futility side, "Are
you comfortable stopping the trial here?
Would you be willing to basically give up at this point?"
That applies for success.
"Would you believe this result?"
If the answer is no, then I have a lot of concerns about doing the interim.
Again, unless there's some public health, you know, thing we need to have
this answer as quickly as possible.
I still trust the drug works, uh, in that case,
Scott: Yeah
Kert Viele: I do
worry
that maybe it doesn't work that well
Scott: Yeah And and so the estimate of that So a very common situation and the
common the situation I was in in the device trial is that y the the company
might believe in a the 15 is right but they wanna make sure their trial is
powered for one because it's clinically is still a valuable device It's good for
patients And so they power the trial at 90 at the maximum for the one but they
really think they their device could be as good as 15 175 or two in that scenario
And then these the the bias under those scenarios is much smaller and it's a
much more reasonable thing to have a variable sample size They actually think
their effect is 15 and they wanna run a trial that's only 70 but they go to 100
potentially to still hit a clinically meaningful effect uh within that scenario
And then you're talking about biases of 7 by by doing this even under the one
It's much smaller in the the 15 scenario
Kert Viele: And I think this is where you really-- this is where I really
put on my Bayesian hat, so to speak, even though I'm computing biases.
When I see that data, it's stopped at, at fifty.
I've got an effect size around one point five.
You know, all of these bias calculations assume that you know the truth.
When I get the data, I don't know the truth.
I gotta figure it out.
And so if, if I knew in my head it can't be bigger than one, then of
course I don't trust the one point five.
But if I've walked into the trial where this might not work at all,
it might be one, it might be two, it could be any of those, then I start
thinking to myself, "Okay, if it were a null, it hardly would ever stop."
So it just-- I don't see many of these.
I know they're biased, but they just don't happen very often.
Happen very rarely.
If it's one, it still doesn't happen all that often.
I don't stop at fifty a ton.
If it's two, I stop at fifty all the time.
So when I see that one point five, it's more likely to be one of
the twos and is the right answer than it's one of the biased ones.
It's more likely to be an unbiased truth than one of the biased falsehoods.
And that, that gives me a lot of comfort
Scott: Yeah You'd probably make a small adjustment like 7 I think
it's probably not quite you know 15 of what was observed yeah
Kert Viele: I'd,
Scott: that scenario
Kert Viele: I'd, adjust by my prior.
There's a
Scott: Yep
Kert Viele: Bayesian answer to
Scott: Yes Yeah
Kert Viele: we've had our frequentist hat on, but
Scott: Yeah
Kert Viele: would just compute the posterior and it works.
It
Scott: Yep
Kert Viele: bias automatically
Scott: Yep Okay so we can calculate the bias uh for that And as we do more
complicated designs and you could add response adaptive randomization you whi
which we've done a recent uh episode of We could add arm dropping in here We could
add patient changing We can do all this We can calculate the exact bias under
a number of scenarios It's one of the really nice things about simulation So
we can calculate bias and we know what it is Mathematically we can calculate it
Kert Viele: And we should certainly emphasize this is one of the
things that in the regulatory world we've been doing a lot more.
The last five, six, seven years, regulators have been paying a lot
more attention to this, asking for it.
And so this is-- we, we've been providing these and for the most
part, things are going fine, but we have found a few examples, and
we've changed the designs in response
Scott: And and it it the FDA draft guidance uh the FDA guidance on adaptive
designs the ICH E20 adaptive designs uh that's draft They they all mention
calculating the bias uh in it I uh I don't know if we have submitted a design
over 27 years of doing this to the agency where they've had an issue with the bias
Kert Viele: That sounds fair to me,
Scott: I think yes if we were to do a sample if we were to do an
interim at 10 patients in your 100 patients example um uh th th there
could be a an issue with that Yeah
Kert Viele: cherry-picking because we don't actually let it get to that
point or at least
Scott: So we wouldn't submit that design when we calculate the bias We say Wow
if I were a regulator I'd have a problem with that sort of thing So I it you
know it's something that they understand they recognize when they have to write
an FDA label I think there's comfort there with it um in it Uh and we can
calculate that exactly Okay so I keep saying that over and over again and
people are w w you know Why why does he keep saying that We can calculate that
sort of thing Coming back to my story
Kert Viele: I think this is, this is definitely your pet peeve episode.
Scott: Yeah Okay All right I'll I'll take that I'll take that Um but you brought
it on You you said Hey we should do an episode Oh and I I I remember this story
Kert Viele: triggering Scott is a p- is a hobby of mine
Scott: Yes So um Gordon Guyatt you may not know care Gordon Guyatt What happened is
uh within a few months of that interaction a paper came out uh in JAMA and the
lead author is uh Bassler Uh and this is Bassler et al The senior author in it
is Gordon Guyatt and it's on behalf of something called the STOPIT Study Group
STOPIT 2 Study Group And the paper is So this is uh y uh this comes out in March
of 2010 and in JAMA and it is Stopping Randomized Trials Early for Benefit
and Estimation of Treatment Effects A Systematic Review and MetaRegression
Analysis So I'll read you the conclusion of the paper and then describe a little
bit about what they do And by the way I I will say this as I'm I'm describing
this that uh in in in this many years I I think this paper is just wrong I think
it's misleading I think it's JAMA should have never published it Uh but we'll
come more to that But just to sort of set the scenes for it So the conclusion
is truncated randomized clinical trials were associated with greater effect sizes
than randomized clinical trials that's not stopped early That's the conclusion
um uh within this But one of the sentences in the paper says Statistical
modeling suggest that randomized clinical trials stopped early for benefit and
they call these truncated randomized clinical trials will systematically
overestimate treatment effects And empirical data demonstrate that truncated
randomized clinical trials often show implausibly large treatment effects
So a lot of people reading this paper looking at it come away with You
shouldn't do interim analyses It creates implausibly large bias This is bad You
know the name of their group is STOPIT Um and I don't think they mean trials
I think they mean it And he literally in that example prevented a company
from doing a group sequential design um because of this issue Okay so what did
they do Uh what what A and their goal was understanding we can mathematically
calculate bias but what do we actually see out there for bias So they went out
and they found through a a a search they found trials that were stopped early
truncated early for superiority And you can go to their their uh CONSORT diagram
They originally found 195 trials that were stopped early Then they looked for trials
of the same question other randomized trials of the same question and they were
gonna compare the effect of these other trials to the ones that were truncated
early And by the way in their search of where they had 195 trials stopped early
and then they looked for other trials they found some more that were stopped
early They moved those over to this calling them truncated And then yeah
Kert Viele: if you replicated the result, you moved it back over and
it didn't count as a replication
But yeah
Scott: you got early truncated
Kert Viele: Yes,
Scott: then yeah
Kert Viele: if you repeated the experiment, it-- you didn't get
credit for the replication of it 'cause you truncated it early
Scott: Yep yep You go into th they're gonna create a largely a
treatment group of trials which were ones that were stopped early and
then the other group are the same question that were not stopped early
Kert Viele: And some of them could have been
Scott: Some of them could have been Um some of them were statistically
significantly successful some were not
Kert Viele: And it's wor-
Scott: within the
Kert Viele: it's worth mentioning that if you could stop early and you
don't, those trials are biased low.
So you're-- they're-- we mathematically know they're gonna be a low group
Scott: So there they and this was their empirically what happens out there So
they took this set of trials that were stopped early and then they compared to
the same question that were not stopped early And they found that the difference
was about a 30 difference in the estimate of the effect They did a r a relative
risk and it was 71 So a 29 relative difference in the effect of those stopped
early versus those that were not And they jump up and down and say This is
enormous A 30 effect So in your particular example we had that effect of one If we
did something that created an average effect of say 13 we might be concerned
that that experiment provides bias and that was that was their experiment
that they did in this comparison and they published the results of that
Kert Viele: And we actually, so we have that result.
So the, if you look in our-- the example we gave, the one hundred sample size and
so on, if you look at the trials that went to the end and won, average effect is only
zero point eight seven compared to one.
They're biased low by a fair amount, thirteen percent
Scott: Yep So in a way they they present this result and think you're you're all
clinical trial people and you understand that we we in a clinical trial as simple
as twoarm trial we have a treatment group and a control group and we randomize
people into those groups and we compare their results They took this group that
the trial was truncated early and they call that group one and then they take
trials that were not stopped early If it was stopped early they moved it over into
group one that were not stopped early and they called that group two And they said
Wow group one's got a better treatment effect than group two And but it by the
way if you ran a clinical trial like this it would be that you run a trial and you
have patients and the ones that respond well you put them in treatment one and if
they don't respond well you put them in treatment two and you say Wow treatment
one is better than treatment two That would never get into JAMA but somehow
this got into JAMA Interestingly when this was published there were many letters to
the editor Uh I was a part of one of them with Brad Carlin and Jason Connor where
we point out that this is this is faulty and wrong Uh but the names of the people
that wrote in saying this was wrong was somewhat impressive Another one was Steve
Goodman Don Berry My father wrote a a letter to the editor a different one than
the one I was on Uh w and Janet Wittes uh was on were on that one Another one
of these letters was Ed Corn and Margaret Mooney Another one was Susan Ellenberg
Dave Demetz and Tom Fleming also pointing out the faults of this And so there were
a lot of people that pointed out that this was a bad study uh in it but yet still
to A and by the way think about it that you go in and you take trials early If
they would've done the same thing where they went out and found trials that were
statistically significant And then they went out and found trials that a a any
other trials run even some that were statistically significant they would
see that that's biased uh within that and that's largely the bias that they
saw in that And we know there's a bias to stopping early We can calculate it
mathematically So when their conclusion was This is implausibly large I think they
understood the mathematical calculation of 7 and we're seeing 30 They even
study things like did the DSMB have some investment in it You know all the Looking
for chicanery here or something like that within this and it wa it's like no this
had to happen You you know if you if you ran an experiment where you took the the
biased sampling that they did you would see this 30 bias This is this is nothing
Kert Viele: I, I'm laughing, Scott, because I, I was truly successful
about getting you riled up today
Scott: Yeah yeah So uh th th this paper's sort of number one on my list You know
others uh we'll do other podcasts of other ones uh within the scenario But
uh and it was kind of amazing at the time because I I tried with w w with
fault to explain this to to Dr Guyatt uh in that I was unsuccessful The paper
came out and I think there was nobody that sort of said this is odd He did
a debate with Steve Goodman after this paper came out where Goodman explained
this and I think he largely I'm not sure he sort of understood the intricacies
and largely it was like their rebuttal to these responses that Well boy look
this is big Yeah it by by definition you bias sampled in a way that created
this Nobody should be surprised by that
Kert Viele: And I, I mean, to be-- make sure we summarize
this properly, we certainly are claiming there's, there's bias.
So there is some bias.
We can compute what it is.
Um, it, that has to be balanced versus the advantages.
We stop early for reasons.
We have-- we want to protect patients.
The DAWN trial, for example, this is a trial that thrombectomy, great
effect, interim at 200 patients, maximal sample size, I think
500 Am I remembering that right?
Scott: Yep
Kert Viele: So,
you know, d--
Scott: stopped at f 200
Kert Viele: stopped at 200 So enormous effect that's panned out in practice.
This is, you know, this is the standard of care now.
If that trial doesn't stop at 200 it years before this.
There are other trials with thrombectomy.
I don't want to claim it's years before that reaches the market,
but it's years before that evidence is actually in front of people
Scott: Yeah Uh and and so there's there's huge benefits And also by the
way it allows you to have a wider range to have a higher powered trial where
you wouldn't run that single maximum because you think it's wasted cause
you don't need that many There's a huge benefit to having a flexible sample size
And this painting it Oh it's biased is uh you shouldn't do it ignores all of
these important factors to trial design
Kert Viele: And I do think you have to incorporate that entire range of evidence,
the entire range of possibilities.
I can't just compute bias on suppose the null is true, suppose this effect is true.
the range?
What's the bias for each of these?
What's the likelihood I'm actually gonna see these?
It's like a doctor getting a diagnostic test.
Here's what's in front of me.
Is this a unbiased, basically unbiased real result, or is this biased high?
And, uh, if it's plausible, it's probably a reasonably unbiased real result
Scott: Yep All right
I I
Kert Viele: You
Scott: I'm trying to make sure
Kert Viele: You need, a second to calm down?
Scott: Yeah I'm calmed down I'm calmed down
Kert Viele: All right
Scott: Yep yep uh in this one All right So and I I'm not lost the the the bit of the
irony that uh we're talking about bias of interims and here we are in the interim
uh in this scenario But uh yep that's the the the world we live in here I I I would
encourage you to go look at the Bassler paper Um you know if if if you have your
study groups and read it I think you should read it and talk about the paper
whether you think it's reasonable Read the uh comments as well I think you'll
you'll enjoy the various people commenting
Kert Viele: And, look, look at your interims when you're doing them.
They're not-- every, every interim shouldn't be there.
So feel free to compute these quantities and to what's reasonable
Scott: Yep Yep All right Well I I appreciate you bringing this topic
to me Kurt and see what other neat topics we have coming uh uh there And
we're gonna stop this we're gonna stop this episode a little early here Kurt
Kert Viele: Okay
Scott: I think, that's appropriate Uh it's a little early and and we are biased here
uh and we will be biased next time So uh thank you all for joining us and until
next time We will be here in the interim
Kert Viele: Thanks, Scott