Bias in Stopping Trials Early

On the latest episode of "In the Interim...", Dr. Scott Berry and Dr. Kert Viele deliver a focused, technical analysis of statistical bias when stopping trials early. This episode clarifies the definition of bias, detailed within the context of interim analyses, emphasizing the empirical consequences of different stopping rules. The discussion addresses common misconceptions around interpretation as well as including the mathematical rationale for averaging across all trial outcomes, and the error of restricting bias estimates to only successful (early-stopped) trials. The hosts present a detailed critique of Bassler et al. (JAMA 2010), highlighting methodological flaws and misinterpretations of comparisons between truncated and non-truncated studies. Simulation is positioned as the primary tool for quantifying bias, with contextual examples illustrating the manageable magnitude of bias. Regulatory expectations are summarized, referencing formal FDA and ICH guidance on adaptive design bias assessment. The DAWN trial is cited as a real-world example where early stopping accelerated patient benefit.

Key Highlights

• Definition and quantification of bias in early-stopped clinical trials
• Mathematical examples demonstrating bias magnitude in fixed and adaptive group sequential designs
• Detailed critique of the methodology and conclusions in Bassler et al. (JAMA 2010)
• Discussion correcting common misunderstandings in bias estimation and selective reporting
• Simulation as a decisive tool for precise bias estimation
• Regulatory context including FDA guidance and ICH E20 draft guidance
• Reference to DAWN trial as evidence of practical benefits of early stopping

For more, visit us at https://www.berryconsultants.com/

Creators and Guests

Host

Scott Berry

President and a Senior Statistical Scientist at Berry Consultants, LLC

What is In the Interim...?

A podcast on statistical science and clinical trials.

Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Scott: Welcome everybody back to In The
Interim Uh I am your cohost today Scott

Berry and I'm joined by my uh uh my common
cohost Kurt Vielli here and we we are

here to talk about something And and and
Kurt will occasionally say Oh I've got

a good topic for the podcast and usually
it's something that's uh beneath the

crawl a little bit maybe uh wi wi within
the the the pet peeve But a really I I

think a really important topic a really
important misunderstood topic and that is

bias in early stopping in clinical trials
So welcome back to In The Interim Kurt

Kert Viele: Thank you

Scott: All right So so we are in the
interim and we're talking about bias

of interims Uh sort of an interesting
aspect of this I I thought I'd introduce

the topic by uh uh telling a a bit of
an old story and um this this happened

relatively early So I I started designing
trials in 2000 Uh before that I did a

little bit of clinical design but uh this
was early in Berry Consultants and was

working with a medical device company
And they they had an estimate of their

effect size but thought that they could
be quite a bit better than that So an

adaptive sample size with early stopping
for superiority made a lot of sense for

the trial So fairly standard two interim
analyses for early success could go to the

final analysis Medical device uh nothing
controversial in in the trial design We

can control type one error It's not the
issue Pretty straightforward And so we're

working on the design and then I got
word from the the company that there's a

problem um with stopping And the company
was associated with a group that was gonna

run the trial and they had an issue with
potentially stopping the trial I thought

Okay that's that that's odd You know
let's let's talk to them about it So the

the concern was the bias in the estimate
that comes out of the trial and it's

important for medical decisionmaking to
have good estimates of of the effect And

the concern was that the early stopping
biases that estimate And so ookay that's

great and one of the wonderful things
about simulation is we can simulate the

bias So it was a response outcome uh
success and failure And I simulated the

trial under a number of potential true
effects and calculated the bias of the

trial And it was really relatively small
It was approximately one percent in a in

a rate of success of We're we're looking
for thirty forty fifty percent success

rates and it was a an s a bias of about
one percent which I thought was a minor

Thing relative to the benefits of sample
size patients time to get a success should

the effect size be that large And um they
the individual I was talking to and I

was actually talking to two people one
one senior member and I'll tell you who

the senior member was in a bit But uh at
the time I had no idea e either of these

people Uh and one was a statistician It
turns out the statistician doesn't say

anything during this conversation Um and
uh the the person says Oh we're seeing

biases larger than that And um asks What
is the bias if you restrict to trials

that uh hit the early success And I said
Well but that's not bias Bias is when

you look at all the trials what is the
average effect relative to the truth If

you restrict it to trials that hit that
early success that in and of itself is

is is a bias Surely you're gonna get much
larger effect cause you have to have a

large effect to win Um and and showed what
that was but you know that's not bias He

says Well I I don't actually know what
the mathematical definition of bias is but

that seems really problematic to me And
it was sort of strange uh in that I said

Well any i if you look at any trial if
you look at a fixed trial and look at only

those that are successful you're you're
gonna see much higher estimates claiming

success You know it it has the same sort
of bias What was fascinating about it

was it got the the the advice and the the
group running the trial was very closely

associated with the device company and
needed the approval of the group Turns

out they had to get rid of all the early
successes It was very weird I thought

th th this is really strange uh uh you
know in in the whole thing So um the the

individual that I was talking to I met for
the first time was Gordon Guyatt and we're

gonna come back to that story But Kurt
this is a this is a thing that I think

a number of people know that this is a
potential issue the bias and think it's a

big issue with potentially stopping early

Kert Viele: So I think, I, uh,
you know, one of the key things

you brought out about that is
what is the definition of bias?

What are you worried about?

And, you know, ordinarily, when
we take this in cla- you know, a

class in undergrad or so on, we
do-- we talk about all trials.

Everything that could
happen, average them.

On average, you get the right
answer, the right response rate.

you cherry-pick, you're going
to generate some kind of bias

because you're cherry-picking.

I do have a little more sympathy, I think,
maybe than you do on this, that if I know

I stopped at the first interim, I stopped
at half the planned sample size, I know

these are the situations where I might
see the high part of the distribution.

know, is that a concern, and
how big of a concern is it?

I certainly can see people
wanting to compute that.

So this is gonna be a key part
of our story going forward

Scott: Yeah yeah So so we'll come back
to that And there's no question that

trials that show statistical significance
overestimate the treatment effect I I

think we all understand that um uh in
it A trial that stops early is going to

provide likely an overestimate Um and
it all depends on what you think the the

potential range of effects are But but
let's come back to the the definition

I think that that this is so important
the definition of bias um uh in it So

what we think of it is what's a biased
estimator An unbiased estimator would

be one that when you run your experiment
and you you have an estimate of what

it is whatever it is you're trying
to estimate the the success rate of a

treatment If on average that estimate is
the real answer we call that unbiased So

the the expected value of the estimator
we're using within our experiment is

equal to the thing the estimand we're
trying to estimate We call that unbiased

Kert Viele: And so we're,
we're, we're thinking about

everything that could happen.

I mean, this is all-- If I do
an interim, I don't just have

early stopping for success, I've
got early stopping for futility.

So I'm averaging, you know, I,
I stopped at fifty for success.

I stopped at fifty for futility,
stopped at seventy-five success

and futility, went to the end,
stopped for success or futility.

All of those things average together.

If those all average out to the
right answer, we're unbiased.

And now we're gonna get into, you
know, do we need to look at each

of those six things separately?

Scott: Yep Yep Okay So um um and we know
we can calculate that if you do early

superiority what happens is within the
trials you could run those trials that

hit early superiority are are depending on
the the the real parameter the real truth

likely are overperforming within that And
you stop at the point it's overperforming

and it doesn't go the rest of the trial
and regress towards the parameter So w

it when those trials stop and you report
that we know that's that's biased By the

way my pet peeve and I know we generally
have one of Kert's pet peeves but one

of my pet peeves is bias means you can't
use it That that all of a sudden it's

it's uh oh it it it's done Bias is okay
Um we can adjust for it We understand it

We want to know what is the size of the
bias uh in the circumstance It's just

bias doesn't mean you can't do it And I
think there's there's this Oh bias you

you can't do it So we understand there's
bias So let's talk a little bit about

that Kurt Where like you know within a
toy example let's talk about you know

what what what is the size of this bias

Kert Viele: Well, and so let, let's
start back up a little bit further, and

Scott: Okay

Kert Viele: start for a fixed trial,
just a straightforward I run a

trial, and my toy example here is
I just have straight normal data.

I'm estimating a mean.

I've arranged a trial.

Sample size is 100.

I'm gonna run a Z test at the end.

Everything is as basic as it could be

Scott: The one randomized
experimental to con placebo

Kert Viele: Yep, standardized effect
size of one here just to make things

Scott: Yep Yep

Kert Viele: so if I, if I take all
trials and the truth is one, so I

come in with what I powered for.

The truth's one, I look at all
trials, I get an average of one.

It's unbiased.

They're the same.

Scott: Futility no superiority
100 patients read it out Yep

Kert Viele: Yep.

so now if I-- Now all of those
trials that were unbiased as

a whole, I can split them.

I can talk about the ones that won,
you know, p less than zero point zero

two five, and the ones that lost.

if I look at the ones that won, they have
an average of about one point zero six.

So they're biased high.

They're one point zero six.

They're compensated for
by the ones that lost.

And so this is-- you have to
have this compensation going on.

In fact, if you make the p-value the
stronger the evidence, if you make

the p-value less than zero point
zero zero zero one, for example,

so you're like, "Oh, look, I won.

This is great.

I have a great drug," and so on.

Somebody could come
back and go, "Oh, wait."

you focus on the trials with p less
than zero point zero zero zero one,

average effect is one point three four.

It's thirty-four percent bias
compared to the truth of one.

So you want to be careful to go, "Oh,
you know, I can't accept anything that's

biased," because you're essentially
saying the stronger the evidence,

the less I believe it or something.

Scott: Okay so under under the
truth that the the real effect is

one where your trial is 90 powered
for one so most of the trials win

Kert Viele: Yeah

Scott: and only 10 of the trials don't
hit 05 If you restrict a calculation

of what's the average effect for those
that win you get 106 So there's a 6

bias just in successful trials in that
scenario Now if you were to do this

under the null the bias would be enormous

Kert Viele: Yeah, you

Scott: for successful trials

Kert Viele: yeah, you need about 0.6

to win.

Usually a 90% powered trial, you
need about 60% of the powered effect.

So your bias has to be at least 0.6

in those cases under the null

Scott: So bias depends on the truth of
of the the the effect If the effect is

three you you you're unbiased because
every trial wins and there's there's

no bias to that So it d so so we're
addressing you know power for the effect

of one uh within that scenario So the

Kert Viele: and if you go the other way,
sorry to interrupt you, but if you go

the other way, if the true effect was 1.5

or two, it's higher, then you have 100%
power and you get no bias whatsoever.

So it definitely depends
on the true effect

Scott: Yep What happens if
I uh if I add futility Uh

Kert Viele: So then you're gonna
end up where, um, now of course,

futility in a fixed trial, you
just have the winners and losers.

But if we start to add into the group
sequential, we add futility, that's

gonna be a compensating mechanism for
the bias, or the upward bias will get

Scott: Okay

Kert Viele: get

Scott: Okay But we have I I forgot
we've done fixed trials All we did

was restrict in the you have this 6
bias of successful trials Now what

if I do uh so you added into this
O'BrienFleming stopping at 50 and 75

Kert Viele: Yep.

And so s- if, so if I do that and I
don't have any futility, so futility is

important here 'cause it does compensate,
then I'm gonna end up overall, I'm

about-- I get an effect about one point
one five as opposed to one, so 15% up.

So that's certainly meaningful.

Um, you gotta weigh that against the fact
that I'm stopping early to help patients,

and I don't know the effect is one.

It might actually really be good.

Um, if I, if the true effect is one and
I stop at 50 in that half the sample…

Yeah

Scott: make sure I There's this So
so over all possible trials you run

Kert Viele: 1.15

Scott: some of them stop at 50 some
stop at 75 some stop at 100 Not on not

just successful trials but all trials

Kert Viele: trials

Scott: All trials that the the average

Kert Viele: sorry.

All trials is 1.07,

so

Scott: Yeah

Kert Viele: correcting me

Scott: okay Oh so so 107 is that So
the bias in a group sequential O'Brien

Fleming um design there's a 7 bias

Kert Viele: Yes, with

Scott: that Yep yep So the question
is iif we just look at that that's

the bias of adding it into the trial
uh uh within that scenario Now w

Kert Viele: different than the
fixed trial that we just had.

That was at six,

so we're

Scott: but that was restricting to only
successful trials Yep Uh so the the If

somebody if the if a FDA says you're
running this group sequential design

what is the bias of adding in those early
analyses Assuming that your effect size

is one it would be 107 There would be a
7 uh uh difference from from the truth

of that uh in that Now we can you can
adjust the estimate for that There are

statistical ways to do that uh within that
But even that bias now we are we are able

to suc correctly claim superiority perhaps
halfway through the experiment 75 of the

way through the experiment The savings
in patients and times to get the right

answer at the cost of this 7 bias that's
that's the sort of question And usually

that seems like a nobrainer tradeoff
to me uh uh within that uh in those

scenarios uh for that Okay But now what
happens if you only look at trials that

are suc that win in a group sequential

Kert Viele: so only the
ones that won, you get 1.15.

So it's gonna be a little
bit higher because you don't

Scott: Yeah

Kert Viele: losers at the end

Scott: And that includes the probability
you win at 50 at 75 or you go to

the end and you're successful at the
end uh within that And so there is

that increase Uh winning trials in
a fixed trials were 106 now it's 115

Kert Viele: Yes

Scott: that Now what if I only
look at trials that win at 50

Kert Viele: Now that one's big.

That's 1.5

Scott: 15 yep Now that's not bias right

Kert Viele: It's not
biased by the definition.

I have a little more sympathy at
kind of a little uncomfortable

Scott: Good Okay But that so that in that
scenario and of course in the scenario

where you need to see that larger effect
size uh uh you know on average 15 uh to to

win that to be statistically significant
in that scenario if you only look at

those scenarios you get biased because
the truth is one and you have to have a

value bigger than that Now if the truth
is two that's that bias is gonna be

Kert Viele: Basically z-

Scott: almost zero Right Right Yeah So
one of the points I think you would make

is if you knew the effect was one you
wouldn't do an interim at 50 you know

in that scenario Or if it's incredibly
unlikely that the scenario's as big

as 15 you wouldn't do that interim

Kert Viele: Yeah, we always, when,
when we show this to clients or

when we do this, we back solve
what effect do you need to stop for

success and futility at each interim?

And we often ask clients directly,
especially on the futility side, "Are

you comfortable stopping the trial here?

Would you be willing to
basically give up at this point?"

That applies for success.

"Would you believe this result?"

If the answer is no, then I have a lot
of concerns about doing the interim.

Again, unless there's some public
health, you know, thing we need to have

this answer as quickly as possible.

I still trust the drug
works, uh, in that case,

Scott: Yeah

Kert Viele: I do

worry

that maybe it doesn't work that well

Scott: Yeah And and so the estimate of
that So a very common situation and the

common the situation I was in in the
device trial is that y the the company

might believe in a the 15 is right but
they wanna make sure their trial is

powered for one because it's clinically
is still a valuable device It's good for

patients And so they power the trial at
90 at the maximum for the one but they

really think they their device could be
as good as 15 175 or two in that scenario

And then these the the bias under those
scenarios is much smaller and it's a

much more reasonable thing to have a
variable sample size They actually think

their effect is 15 and they wanna run a
trial that's only 70 but they go to 100

potentially to still hit a clinically
meaningful effect uh within that scenario

And then you're talking about biases of
7 by by doing this even under the one

It's much smaller in the the 15 scenario

Kert Viele: And I think this is where
you really-- this is where I really

put on my Bayesian hat, so to speak,
even though I'm computing biases.

When I see that data,
it's stopped at, at fifty.

I've got an effect size
around one point five.

You know, all of these bias calculations
assume that you know the truth.

When I get the data, I
don't know the truth.

I gotta figure it out.

And so if, if I knew in my head it
can't be bigger than one, then of

course I don't trust the one point five.

But if I've walked into the trial
where this might not work at all,

it might be one, it might be two, it
could be any of those, then I start

thinking to myself, "Okay, if it were
a null, it hardly would ever stop."

So it just-- I don't see many of these.

I know they're biased, but they
just don't happen very often.

Happen very rarely.

If it's one, it still doesn't
happen all that often.

I don't stop at fifty a ton.

If it's two, I stop at fifty all the time.

So when I see that one point five,
it's more likely to be one of

the twos and is the right answer
than it's one of the biased ones.

It's more likely to be an unbiased
truth than one of the biased falsehoods.

And that, that gives me a lot of comfort

Scott: Yeah You'd probably make
a small adjustment like 7 I think

it's probably not quite you know
15 of what was observed yeah

Kert Viele: I'd,

Scott: that scenario

Kert Viele: I'd, adjust by my prior.

There's a

Scott: Yep

Kert Viele: Bayesian answer to

Scott: Yes Yeah

Kert Viele: we've had our
frequentist hat on, but

Scott: Yeah

Kert Viele: would just compute
the posterior and it works.

It

Scott: Yep

Kert Viele: bias automatically

Scott: Yep Okay so we can calculate
the bias uh for that And as we do more

complicated designs and you could add
response adaptive randomization you whi

which we've done a recent uh episode of
We could add arm dropping in here We could

add patient changing We can do all this
We can calculate the exact bias under

a number of scenarios It's one of the
really nice things about simulation So

we can calculate bias and we know what
it is Mathematically we can calculate it

Kert Viele: And we should certainly
emphasize this is one of the

things that in the regulatory
world we've been doing a lot more.

The last five, six, seven years,
regulators have been paying a lot

more attention to this, asking for it.

And so this is-- we, we've been
providing these and for the most

part, things are going fine, but
we have found a few examples, and

we've changed the designs in response

Scott: And and it it the FDA draft
guidance uh the FDA guidance on adaptive

designs the ICH E20 adaptive designs
uh that's draft They they all mention

calculating the bias uh in it I uh I
don't know if we have submitted a design

over 27 years of doing this to the agency
where they've had an issue with the bias

Kert Viele: That sounds fair to me,

Scott: I think yes if we were to
do a sample if we were to do an

interim at 10 patients in your 100
patients example um uh th th there

could be a an issue with that Yeah

Kert Viele: cherry-picking because
we don't actually let it get to that

point or at least

Scott: So we wouldn't submit that design
when we calculate the bias We say Wow

if I were a regulator I'd have a problem
with that sort of thing So I it you

know it's something that they understand
they recognize when they have to write

an FDA label I think there's comfort
there with it um in it Uh and we can

calculate that exactly Okay so I keep
saying that over and over again and

people are w w you know Why why does he
keep saying that We can calculate that

sort of thing Coming back to my story

Kert Viele: I think this is, this is
definitely your pet peeve episode.

Scott: Yeah Okay All right I'll I'll take
that I'll take that Um but you brought

it on You you said Hey we should do an
episode Oh and I I I remember this story

Kert Viele: triggering Scott
is a p- is a hobby of mine

Scott: Yes So um Gordon Guyatt you may not
know care Gordon Guyatt What happened is

uh within a few months of that interaction
a paper came out uh in JAMA and the

lead author is uh Bassler Uh and this
is Bassler et al The senior author in it

is Gordon Guyatt and it's on behalf of
something called the STOPIT Study Group

STOPIT 2 Study Group And the paper is So
this is uh y uh this comes out in March

of 2010 and in JAMA and it is Stopping
Randomized Trials Early for Benefit

and Estimation of Treatment Effects A
Systematic Review and MetaRegression

Analysis So I'll read you the conclusion
of the paper and then describe a little

bit about what they do And by the way
I I will say this as I'm I'm describing

this that uh in in in this many years I
I think this paper is just wrong I think

it's misleading I think it's JAMA should
have never published it Uh but we'll

come more to that But just to sort of
set the scenes for it So the conclusion

is truncated randomized clinical trials
were associated with greater effect sizes

than randomized clinical trials that's
not stopped early That's the conclusion

um uh within this But one of the
sentences in the paper says Statistical

modeling suggest that randomized clinical
trials stopped early for benefit and

they call these truncated randomized
clinical trials will systematically

overestimate treatment effects And
empirical data demonstrate that truncated

randomized clinical trials often show
implausibly large treatment effects

So a lot of people reading this paper
looking at it come away with You

shouldn't do interim analyses It creates
implausibly large bias This is bad You

know the name of their group is STOPIT
Um and I don't think they mean trials

I think they mean it And he literally
in that example prevented a company

from doing a group sequential design um
because of this issue Okay so what did

they do Uh what what A and their goal
was understanding we can mathematically

calculate bias but what do we actually
see out there for bias So they went out

and they found through a a a search they
found trials that were stopped early

truncated early for superiority And you
can go to their their uh CONSORT diagram

They originally found 195 trials that were
stopped early Then they looked for trials

of the same question other randomized
trials of the same question and they were

gonna compare the effect of these other
trials to the ones that were truncated

early And by the way in their search of
where they had 195 trials stopped early

and then they looked for other trials
they found some more that were stopped

early They moved those over to this
calling them truncated And then yeah

Kert Viele: if you replicated the
result, you moved it back over and

it didn't count as a replication

But yeah

Scott: you got early truncated

Kert Viele: Yes,

Scott: then yeah

Kert Viele: if you repeated the
experiment, it-- you didn't get

credit for the replication of
it 'cause you truncated it early

Scott: Yep yep You go into th
they're gonna create a largely a

treatment group of trials which were
ones that were stopped early and

then the other group are the same
question that were not stopped early

Kert Viele: And some
of them could have been

Scott: Some of them could have been
Um some of them were statistically

significantly successful some were not

Kert Viele: And it's wor-

Scott: within the

Kert Viele: it's worth mentioning
that if you could stop early and you

don't, those trials are biased low.

So you're-- they're-- we mathematically
know they're gonna be a low group

Scott: So there they and this was their
empirically what happens out there So

they took this set of trials that were
stopped early and then they compared to

the same question that were not stopped
early And they found that the difference

was about a 30 difference in the estimate
of the effect They did a r a relative

risk and it was 71 So a 29 relative
difference in the effect of those stopped

early versus those that were not And
they jump up and down and say This is

enormous A 30 effect So in your particular
example we had that effect of one If we

did something that created an average
effect of say 13 we might be concerned

that that experiment provides bias
and that was that was their experiment

that they did in this comparison and
they published the results of that

Kert Viele: And we actually,
so we have that result.

So the, if you look in our-- the example
we gave, the one hundred sample size and

so on, if you look at the trials that went
to the end and won, average effect is only

zero point eight seven compared to one.

They're biased low by a fair
amount, thirteen percent

Scott: Yep So in a way they they present
this result and think you're you're all

clinical trial people and you understand
that we we in a clinical trial as simple

as twoarm trial we have a treatment group
and a control group and we randomize

people into those groups and we compare
their results They took this group that

the trial was truncated early and they
call that group one and then they take

trials that were not stopped early If it
was stopped early they moved it over into

group one that were not stopped early and
they called that group two And they said

Wow group one's got a better treatment
effect than group two And but it by the

way if you ran a clinical trial like this
it would be that you run a trial and you

have patients and the ones that respond
well you put them in treatment one and if

they don't respond well you put them in
treatment two and you say Wow treatment

one is better than treatment two That
would never get into JAMA but somehow

this got into JAMA Interestingly when this
was published there were many letters to

the editor Uh I was a part of one of them
with Brad Carlin and Jason Connor where

we point out that this is this is faulty
and wrong Uh but the names of the people

that wrote in saying this was wrong was
somewhat impressive Another one was Steve

Goodman Don Berry My father wrote a a
letter to the editor a different one than

the one I was on Uh w and Janet Wittes
uh was on were on that one Another one

of these letters was Ed Corn and Margaret
Mooney Another one was Susan Ellenberg

Dave Demetz and Tom Fleming also pointing
out the faults of this And so there were

a lot of people that pointed out that this
was a bad study uh in it but yet still

to A and by the way think about it that
you go in and you take trials early If

they would've done the same thing where
they went out and found trials that were

statistically significant And then they
went out and found trials that a a any

other trials run even some that were
statistically significant they would

see that that's biased uh within that
and that's largely the bias that they

saw in that And we know there's a bias
to stopping early We can calculate it

mathematically So when their conclusion
was This is implausibly large I think they

understood the mathematical calculation
of 7 and we're seeing 30 They even

study things like did the DSMB have some
investment in it You know all the Looking

for chicanery here or something like that
within this and it wa it's like no this

had to happen You you know if you if you
ran an experiment where you took the the

biased sampling that they did you would
see this 30 bias This is this is nothing

Kert Viele: I, I'm laughing, Scott,
because I, I was truly successful

about getting you riled up today

Scott: Yeah yeah So uh th th this paper's
sort of number one on my list You know

others uh we'll do other podcasts of
other ones uh within the scenario But

uh and it was kind of amazing at the
time because I I tried with w w with

fault to explain this to to Dr Guyatt
uh in that I was unsuccessful The paper

came out and I think there was nobody
that sort of said this is odd He did

a debate with Steve Goodman after this
paper came out where Goodman explained

this and I think he largely I'm not sure
he sort of understood the intricacies

and largely it was like their rebuttal
to these responses that Well boy look

this is big Yeah it by by definition
you bias sampled in a way that created

this Nobody should be surprised by that

Kert Viele: And I, I mean, to
be-- make sure we summarize

this properly, we certainly are
claiming there's, there's bias.

So there is some bias.

We can compute what it is.

Um, it, that has to be
balanced versus the advantages.

We stop early for reasons.

We have-- we want to protect patients.

The DAWN trial, for example, this
is a trial that thrombectomy, great

effect, interim at 200 patients,
maximal sample size, I think

500 Am I remembering that right?

Scott: Yep

Kert Viele: So,

you know, d--

Scott: stopped at f 200

Kert Viele: stopped at 200 So enormous
effect that's panned out in practice.

This is, you know, this is
the standard of care now.

If that trial doesn't stop
at 200 it years before this.

There are other trials with thrombectomy.

I don't want to claim it's years
before that reaches the market,

but it's years before that evidence
is actually in front of people

Scott: Yeah Uh and and so there's
there's huge benefits And also by the

way it allows you to have a wider range
to have a higher powered trial where

you wouldn't run that single maximum
because you think it's wasted cause

you don't need that many There's a huge
benefit to having a flexible sample size

And this painting it Oh it's biased is
uh you shouldn't do it ignores all of

these important factors to trial design

Kert Viele: And I do think you have to
incorporate that entire range of evidence,

the entire range of possibilities.

I can't just compute bias on suppose the
null is true, suppose this effect is true.

the range?

What's the bias for each of these?

What's the likelihood I'm
actually gonna see these?

It's like a doctor
getting a diagnostic test.

Here's what's in front of me.

Is this a unbiased, basically unbiased
real result, or is this biased high?

And, uh, if it's plausible, it's probably
a reasonably unbiased real result

Scott: Yep All right

I I

Kert Viele: You

Scott: I'm trying to make sure

Kert Viele: You need,
a second to calm down?

Scott: Yeah I'm calmed
down I'm calmed down

Kert Viele: All right

Scott: Yep yep uh in this one All right So
and I I'm not lost the the the bit of the

irony that uh we're talking about bias of
interims and here we are in the interim

uh in this scenario But uh yep that's the
the the world we live in here I I I would

encourage you to go look at the Bassler
paper Um you know if if if you have your

study groups and read it I think you
should read it and talk about the paper

whether you think it's reasonable Read
the uh comments as well I think you'll

you'll enjoy the various people commenting

Kert Viele: And, look, look at your
interims when you're doing them.

They're not-- every, every
interim shouldn't be there.

So feel free to compute these
quantities and to what's reasonable

Scott: Yep Yep All right Well I I
appreciate you bringing this topic

to me Kurt and see what other neat
topics we have coming uh uh there And

we're gonna stop this we're gonna stop
this episode a little early here Kurt

Kert Viele: Okay

Scott: I think, that's appropriate Uh it's
a little early and and we are biased here

uh and we will be biased next time So uh
thank you all for joining us and until

next time We will be here in the interim

Kert Viele: Thanks, Scott