In the Interim...

In this episode of "In the Interim…", Dr. Kert Viele, Director of Research at Berry Consultants, takes up a requested topic: published results showing that Bayesian borrowing of historical control data cannot increase power when type I error is controlled. The key is the definition of type 1 error control. Borrowing methods can increase power and lower the probability of false positive conclusions when the historical data is near the current trial parameters, but not when the historical data and current trial begin to separate.  If “type 1 error control” means controlling the worst case upper bound on the chance of a false positive conclusion, borrowing indeed can offer no benefit. In a world where all past data was generated to deceive you (the worst case), you shouldn’t use the past data. But, if type 1 error control means the actual probability of a false positive conclusion, borrowing can produce benefits in power and type 1 error commensurate to the match between history and the present trial. As such, these papers revisit an old tension between minimax methods and expected value approaches. When data is informative and when we borrow from historical situations most like our current trial, we can be optimistic the historical data will lead to better inferences.

Key Highlights
  • Why we naturally want to use past data in our current trials.
  • What does type 1 error protect against? Why we can’t directly compute this probability from a frequentist perspective.
  • Type I error as a worst-case upper bound.
  • Why borrowing cannot gain power under standard type I error control.
  • The fundamental tradeoff of historical data, with lower type I error and higher power where history matches, increased error rates where it does not.
  • A review of FDA guidance on considerations for historical data, including recency, comparable sites, and matching inclusion and exclusion criteria.
  • Minimax rules versus expected value, and the cost of guarding against unlikely parameter values.
For more, visit us at https://www.berryconsultants.com/

Creators and Guests

Host
Scott Berry
President and a Senior Statistical Scientist at Berry Consultants, LLC

What is In the Interim...?

A podcast on statistical science and clinical trials.

Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Hi, welcome to In The Interim.

I am not Scott Berry.

I'm Kurt Feely, the director of
research at Berry Consultants.

Um, I'm doing this without
Scott today, so he is off.

Um, so anyway, welcome and
we'll see how this goes.

Um, today I'm gonna be talking
about, uh, a requested topic.

We take requests, uh, and people have
asked when we're doing borrowing, there's

a series of papers which essentially
they're titled, "If You Borrow," um,

there's no way to increase power.

So essentially it leaves that
impression of, of if we borrow,

we can't gain anything by it,
so perhaps we shouldn't borrow.

So I wanna talk about those papers,
and certainly I'm not gonna, not gonna

criticize them, they're all fine.

Uh, but talk about some philosophy
of science and really some, some key

controversies between Bayesian and
frequentist, how we formulate what we're

trying to do in a clinical trial that I
think are, are really relevant to this.

Um, so anyway, Scott, now the
unfortunate thing is Scott says I'm

not allowed to use slides, so, um,
that's always gonna be rough for me.

Uh, but we recognize there are people
in their car, or if you're listening to

this while you're trying to go to sleep,
uh, I consider that a, a high honor.

So, so anyway, we'll see how this goes.

Um- So I wanna start with an example
of, of why we would borrow, an

example I'm gonna use throughout this.

Um, it's a progressive disease, so, uh,
essentially, people get worse over time.

Uh, I may have some
kind of historical data.

I've seen a lot of patients, and generally
speaking, I expect them to drop about five

points in a year on some endpoint scale.

So I have some idea that
I-- that, that's expected.

I'm only borrowing on the control arm.

Talking about treatment arm,
that's a separate issue.

Um, now I can imagine I have
a new drug, I run a trial.

And I get-- suppose I got one set
of results where I-- my control

arm, it dropped five points.

My treatment arm, it only
dropped three points.

I should be incredibly happy here.

So the control arm's performing
exactly as I expected.

I'm getting confirmation this is
right, etc It's, it's in line.

Um, and my drug's doing well.

It slowed the progression
down to 3 points.

People are progressing slower.

That's all great.

2 points of gain.

So this makes me happy.

On the other hand, um, suppose I got
different results where what actually

happened was the control dropped 7
points, but the treatment dropped 5 So

I still have this 2 point difference.

So control's at 7 treatment's
at 5 I still gained 2 points.

That's good.

Um, but at this point, I'm probably--
if I have that historical data, I have

this kind of lingering doubt in my head
that, um, you know, I, I know I expected

the controls to be at 5 I got 7 and 5
here, so maybe I just-- it was randomly

high on the treatment arm, and so what's
happening is the 7's just spurious.

My treatment ended up at 5 You know, maybe
I just ended up seeing some noise here,

and that's certainly gonna be problematic.

So that's gonna make me in
a lot of, a lot of doubt.

So this is the kind of example
that we wanna bring into

why we wanna do borrowing.

Um, and it seems sensible to treat
those two datasets differently

in how we think about them.

So it, it makes sense to hedge our
bets towards the historical data at

five, maybe becoming more confident
when the controls in our current study

really confirm that or are consistent
with that and, um, having some doubt

or hedging our bets towards five when
they're, in this case, higher at seven.

So by hedging our bets, we're
gonna get different performance

in our trial, depending on whether
the controls are actually five or

actually seven, actually three.

So, so we're gonna talk about, well,
what does that hedging of the bets do?

On the one hand, we might think, "Hey,
we're bringing in information, so

shouldn't we just be able to do better?"

And again, we have this, these papers
that say, generally speaking, if you wanna

control type one error, the answer is no.

Um, and this really gets into
what does it mean to control

type one error, so I wanna, to
really focus on that for a second

Okay, so, um, I may have kind of a
strange perspective on type I error

here, uh, maybe a little different.

So anyway, we'll, we'll see how this goes.

Um, but, um, generally speaking, I don't
have a problem with type I error per

se, at least from the-- I know a lot of
Bayesians do, but I'm not against people

computing type I error rates and so on.

I do think what I'm trying to
avoid, so let's think about why

we want a type I error to be low.

Um, we don't want to approve a null drug.

If a drug doesn't work, it potentially
it has, has cost, it exposes people

to side effects, it's not good
to approve something like that.

Uh, it makes it hard to
do studies in the future.

So, um, it messes up research
going forward to have an approved

drug that doesn't actually work.

So lots of reasons we want to avoid this.

As a Bayesian, I want to avoid
this frequentist directly, want

to avoid this as well, so I
don't see any disagreement there.

Um, I will say what I want
to avoid in this case is not

the conditional probability.

What's the probability I reject
the null given that it's true?

Um, what I'm really interested in is
I wanna know the probability that I

reject the null and the null is true.

Uh, I want both of those things
together for it to really be bad.

If I'm rejecting a true null, of
course, I'm not worried about that.

So if I don't think the null
is gonna be true, I'm not as

worried about this whole event.

Um, by way of analogy, I maybe
think of this as a drug allergy.

Uh, I've got a drug where the probability
something bad happens given you

have an allergy is, is pretty high.

Um, obviously, that's a negative, um,
but, you know, it's not bad in and of

itself as long as you can, you know,
avoid the people that have the allergies.

So the thing that's really bad is you give
the drug and the person has the allergy.

Here, I think it's the same thing.

The worry is suppose the null
is true and we approve the drug.

That's what we really want to avoid,
not just the conditional on this.

Okay, so if that's a risk we want
to avoid, um, well, I can think of

a design, and I can go, well, you
know, what's the probability of that?

What's the probability the
null is true and I reject it?

Um, back to basic statistics.

Again, I'm not allowed to use slides.

I had to try to make this
as simple as possible.

Um, so the probability that the null
is true and we reject it, that's

the basic rule for probabilities
of u-- of intersections here.

So the probability we reject
given the null is true times the

probability the null is true.

Probability of A and B is probability
of A given B times the probability

of B Okay, so we immediately run
into problems with this calculation

from a frequentist perspective.

This is something we want probability
to be small, but if we actually

are gonna compute that, we've got
the probability the null is true.

That involves a parameter, so that's
hard to put a pr- a probability on

within a frequentist perspective.

In a Bayesian perspective, it's
natural, but of course, there's

a controversy over the prior.

Obviously, you know, you know,
Barry's opinions on this.

Uh, but anyway, fully accepted
that this is-- it's a, it-- more

difficult when th- you're putting
probabilities on parameters.

So probability null is true.

That's gonna be hard, but at least I know
either the null is true or the null isn't.

So that probability is either
zero or one at least, okay?

Well, narrowed it down a bit, I guess.

Um, the other piece of it, the
probability we reject given the

null is true, well, that sounds
like a probability we can compute.

Um, but this gets in, I, I always have
a pet peeve in all of these talks.

I don't do it on purpose, but I seem to
always have something that upsets me.

Um, one of my pet peeves recently
is, is when people say, um, you

know, "What is type one error?"

And they say, "This is the probability
you falsely reject the null."

And that's not quite true, and,
and the reason it's becoming

a pet peeve is because it's so
relevant to Bayesian inference.

So the probability we reject given the
null is true, that's actually something

we can't compute without thinking of
parameters either because the statement

the null is true, what we're conditioning
on, I've got this progressive disease

that I talked about at the beginning.

When I say the null is true, the
progression, I'm saying it's the same

between the control and the treatment
arm, but the same, it might be both

arms are equal to 5 both arms are
equal to 7 both arms are equal to 3 And

the trial that I design could behave
differently under those conditions.

I could get a different probability
of rejecting when they're both 3

that's a null, or when they're both
5 that's a null, or when they're

both seven, that's a null or anything
in between So this is hard from a

frequentist perspective as well because
we don't know what the parameter is.

Um, but again, we can also think
about, well, we don't know what the

parameter is, but it's something.

So, uh, usually within a frequentist
standpoint, what we do is we put

an upper bound on this probability.

We say, "I'm gonna look at all
the different possibilities

where the null could be."

This is usually called the null
space, so all the different

nulls we need to consider.

And I'm gonna find the
worst case scenario.

So that's an upper bound on that
function, that probability of

rejecting across the entire null space.

So given the fact that I now have
an upper bound on that probability,

I know that the probability the
null is true is zero or one.

So this whole probability that the
null is true and I reject it, I

can now bound it, and that upper
bound is the type one error rate.

So this makes, uh, you can see where
we're going is, is really what's

happened here is this is a natural
thing when we don't wanna put

these probabilities on parameters.

This bound becomes the tool that we have.

And so this does-- this fits into our
perspective about if we can make the

bound low, we can make the probability
low The difficulty is, of course,

that an upper bound is an upper bound.

It's not the actual probability.

And so we have to think of, you know,
situations where the bound's good and

it's close to the actual probability,
and situations where it really

overstates the risk of making this error.

Um, and that's really the, the key
part of, of the Bayesian aspect here.

Um, okay, so let's, let's talk about
these papers again for a second.

So what do they actually say?

Um, so we've got this, um, uh, y- this--
what the statement actually indicates

is that if you're borrowing, so you
have this historical data and you

wanna control type one error, um, then
it's impossible to get a power gain.

So it's, it doesn't matter
that you're borrowing.

And so the reason for this is really
focusing on this worst case scenario.

What Bayesian is gonna do is…

Remember, I'm hedging my bets.

In my example at the beginning,
I'm hedging my bets towards five.

So that might be a good thing.

If the truth is near five, then I'm
gonna get some benefits out of this.

But if the truth isn't near five,
I'm going to ge-generate some

worse inferences when I borrow.

So when I think of this upper bound,
and this is the worst case scenario,

any time that I borrow, I'm gonna end up
inflating the type one error somewhere.

So not around five, that I'm
actually gonna get a benefit.

I'm actually gonna make the type
one error better around five,

but I'm gonna make it worse.

And whether that's worse at seven
or three or eight or one, that

really gets into the nitty-gritty
of the method that you're using.

But somewhere you're going
to inflate type one error.

And so there's this result in frequentist
statistics that says if you say that

the upper bound exists, if you really
are restricting yourself where you

can't inflate it anywhere, then, uh,
all of the standard tests that you use

in frequentist inference, they're i-
they are the ones with the highest power

within that class of you have a strict
upper bound that you have to obey.

Um, and so these papers that are coming
out, they're all perfectly right and

it's a really good observation, um,
which is, you know, these standard

results that we've known for decades,
maybe even a century, probably decades.

Uh, but anyway, these results that have
said, "Hey, these standard frequentist

tests are the highest power that you
can get when you control type one

error," when you control this upper
bound, they aren't changed by borrowing.

So you still, the frequentist test, if
you're controlling type one error, are

still the best thing that you can do

Okay, so, um, that makes sense.

So now the question is, well, um, if
we're-- if we know that the Bayesian

test is gonna inflate type I error
somewhere, is that necessarily bad?

Um, or could it potentially
be a good thing?

And so this really gets at this notion
of the upper bound versus the real

probability of making a mistake.

Um, so let, let's start
with, um, a standard test.

It has two and a half percent type I
error regardless of what the null is.

Both arms could be five, both arms could
be seven, both arms could be three.

It's always two and a half percent.

That's the standard, our comparator test.

So suppose I have a different
test, and it has-- the worst case

scenario is two and a half percent,
but that only happens somewhere.

So it reaches that max of
two and a half percent.

Everywhere else, it's one percent for
the chance of making that mistake.

So I could ask, is that better?

Um, presumably, yes.

Presumably, you want the lower,
the lower type I error there.

Uh, even as, even as, even as a
Bayesian, we want to avoid this risk.

So that notion two and a half percent
everywhere versus two and a half percent

somewhere and one percent everywhere
else, I'd rather have the one percent

everywhere else if I can get it.

Um, and remember, when we define
type I error, what we're saying is

it's only the bound that matters.

So as long as I'm under, as long
as my worst case is under 2.5

I consider all of those things equal.

So under this notion of strict
type I error control, that 2.5

worst case, 1 everywhere else, it's
not a game because those-- that lower

type I error, it doesn't matter.

It's like saying, you know, I, I
wanna do a surgery technique or I

wanna do a kind of surgery and I
want at most a 3 mortality rate.

And I can say, well, I can do it
one way and I get a 3 mortality

rate and I can do it another way
and I get a 1 mortality rate.

You certainly wouldn't respond,
"Oh, well, they're both at most 3 so

they're, they're all the same to me."

You're like, "No, give me the 1 I'd
certainly rather have a less chance of, of

dying in that case Okay, so unfortunately,
the Bayesian is not that simple.

So it would be nice to say, oh,
the Bayesian has a worst case, and

then it's one percent everywhere
else, and it just dominates.

That's not what happens
with a Bayesian test.

So what actually happens with a Bayesian
test is we said it's going to have to

inflate the type one error somewhere,
and that somewhere is gonna be when the

truth of your current trial doesn't match
the historical data that's coming in.

So if I'm expecting five percent
on-- or, or five points on my, um,

uh, a five-point progression in
here, if the actual progression is

seven or the actual progression is
three, things are gonna get worse.

And they're not gonna be
at two and a half percent.

I may inflate the type one error to four
percent, um, maybe even higher than that.

This really depends on the,
the details of the method.

So a lot of the research in borrowing
right now is to try to inflate it as

little as possible and to try to inflate
it over as narrow a range as possible.

Uh, but I'm not really getting into that.

The key point here is you
have to inflate it somewhere.

Um, so there's going to be this
case of, compared to the standard

test, some type one error, part
of this function's gonna be worse.

So for some values, when the
current trial doesn't match

history, it's gonna be worse.

Um, on the other hand, there are
gonna be gains when the historical

data matches the current trial.

And those advantages here are you'll
actually get lower type one error.

Um, the reason that you get lower type
one error when you're borrowing on

controls is if you think about what
could cause a type one error, the two

arms are in truth equal to each other.

You can get a type one error if your
control arm does randomly bad, so it

basically looks worse than it actually is.

Or you could get a type one error if
your treatment looks randomly good.

Either one of those will do it.

If you're borrowing and history
matches, you really limit the chance

that you're gonna be, uh, the spurious
results on the control arm are less

likely to cause a type one error And
so you're less likely to make it.

Um, so you get lower type I
error when history matches than

that two and a half percent.

So the kind of comparison that we're
gonna make when we say, "I need to compare

a frequentist design to a, a Bayesian
design," in this case, is I can say I

can have two and a half percent type I
error, whether the n- there are-- I have a

null at five or a null at three or a null
at seven, or the comparison is, okay, I

could have a type I error that is higher
for certain values, say, when the control

is equal to s- to three, for example.

But I can get a lower type I error
when the, um, when history matches,

when things are actually near five.

So that suddenly sets up a trade-off
that we are gonna have to measure,

and I just wanna keep in mind,
I'm not obeying that upper bound.

So the papers, when they say, "If you con-
if you control type I error, you can't

gain power," it essentially eliminates
the Bayesian designs from consideration.

So it basically says, among frequentist
designs, a frequentist design is the best.

Um, the Bayesian design's not
in the running in that sense.

Okay, so we've got this trade-off.

So what do we do with it?

Um, we know the type one error is
higher for certain values in the null.

We know that the type one error is lower.

Um, it also helps the power tends
to be higher if history matches.

So there's this range of parameter
values where I actually get lower

type one error and higher power.

So again, this is where
I can get higher power.

I'm not-- But I'm not violating these
papers because I didn't obey the upper

bound, so it's a different kind of test.

I have, I have better type one
error, better power in a region

where history is matching.

And then I have worse behavior when, uh,
history doesn't match my current trial.

And so I need to make some decisions
about is that valuable or not.

So sometimes borrowing is better,
sometimes not borrowing is better.

Um, and so this is where we get
into when you're assessing this with

regulators or a sponsor or whatever,
you really have to give some thought

of, okay, I need to look where is the
borrowing getting me a benefit and

where is the borrowing generating a
risk, and I need to think about which of

those parameter values is more likely.

There's no way to avoid it.

There's no…

The, the, the trial just behaves
differently for certain parameter values.

So I need to know, is it more
likely that history is gonna

hold in my current trial or not?

And I accept all of the
difficulties over that.

When you look in the Bayesian
guidance, it spends a lot of time

talking about, well, what needs to
happen to make us confident of that.

Um, you know, uh, his- if history is
more recent in time, if the sites look

the same as the sites that are in my
current trial, uh, if the, uh, population

inclusion, exclusion are the same, all
of the things we do to try to match

history, what that's aimed at is trying
to say, "Hey, look, this region where I

get a benefit is more likely to occur."

And so that's really what we're aiming at
here So, you know, the unfortunate thing,

uh, you know, this gets back into old
debates in the, in the say, 1950s or so

about minimax rules versus expected value.

Um, frequentist is really, it's a, it's
a minimax kind of idea where I say I'm

gonna make the-- I'm gonna look at the
worst case, and I'm gonna make sure

that I'm robust to that, and there is
value to being robust in those cases.

There's also a cost to being robust,
um, and that's often in the situation

where I'm spending a lot of time
controlling or a lot of time, um,

it certainly is time, but it's also
patience and cost and everything else.

I'm spending a lot of cost and
time and effort in order to control

type one error for parameter
values that aren't very likely.

If the controls tend to be five,
do I need to control type one

error for a controlled, uh,
progression of thirteen points?

Maybe not.

So maybe not nine, maybe at seven or
six I need to start worrying, and that's

really where you get into the debates.

You might wanna choose a borrowing method
that is more robust in those cases.

Some borrowing methods have a broader
range of benefit than others, so

all of that's gonna come into play.

Uh, but anyway, this is really this…

If you're reading those papers, this
is really where the distinction is.

Um, when they say, you know, power when
you're controlling type one error, they're

talking about maintaining an upper bound,
which a Bayesian test just can't do.

Um, uh, but that doesn't mean that
the Bayesian test doesn't come

with some benefits, and really
this is a benefit risk idea.

We're not, uh, as Bayesians,
we're not being minimax.

We're computing what is the expected
value, what's actually-- What is--

Basically, what is-- What do we think is
actually that probability, the probability

I reject and the null is true, as
opposed to just give me an upper bound.

And in many cases, we have good reason
to believe that the actual probability

is better when we borrow, and in
those cases, we recommend to do it.

Uh, so anyway, hope that clears this up.

Um, hopefully that, that, um, uh,
you know, there's obviously more

research in this area about assessing
utilities and, and everything else.

So, um, what am I meant to say here?

Scott says I have to say this has been…

We will see you again in the interim,
uh, and thank you for your time.