Adaptive Design Actions Matrix

In this episode of "In the Interim…", Dr. Scott Berry challenges the widely held belief that any interim look at trial data obligates an alpha adjustment. By constructing a two-by-two matrix: interim data (positive/negative) and adaptive action (increase/decrease sample size), Scott demonstrates that the need for statistical correction depends on precisely what actions are prespecified. He emphasizes that the need for adjustment depends on the action and the data. Technical scenarios examined include group sequential designs, “promising zone” sample size re-estimation (citing the formal results of Mehta and Pocock), and response adaptive randomization. Scott stresses that clear prespecification is required for Type I error control and regulatory compliance. He critiques common missteps, such as unnecessary allocation of alpha to futility boundaries when superiority is not planned, and reiterates that it is the adaptive action, and not mere data review, that determines the statistical impact of interim analyses.

Key Highlights

• Dissects alpha adjustment myths and their historical roots.
• Details two-by-two matrix: interim data direction and adaptive action.
• Explores group sequential, futility, promising zone, and response adaptive examples.
• Clarifies when Type I error is truly affected—action and data matter.
• Stresses prespecification’s role in trial validity and regulatory acceptance.
• Identifies pitfalls in common trial design practices.

For more, visit us at https://www.berryconsultants.com/

Creators and Guests

Host

Scott Berry

President and a Senior Statistical Scientist at Berry Consultants, LLC

What is In the Interim...?

A podcast on statistical science and clinical trials.

Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Well, welcome everybody
back to In The Interim.

I'm your host, Scott Berry.

Appreciate you joining me again today.

Uh, In The Interim is a podcast of
all things clinical trial science.

Now, if you go back to some of this,
I, I try, I, I try to create a, a,

a much more welcome intro to each of
the topics, and it might be a story of

something that happens that I feel like
is relevant to introduce the story.

So I, I was, I was talking with, uh,
uh, my family, I was talking to my wife,

Tammy, who she ends up being a fairly
common topic on this, this podcast.

And I was describing the, the topic and
thinking about is, you know, is there

something, there's gotta be something
where, um, r- really the topic is

where something is true in a particular
scenario and people understand that,

but it's not true in another scenario,
but they carry that with them and

there's a sort of myth about it that
this is a, a much more universal thing.

And, um, and not understanding the aspect
of it, and this just carries forward.

And was, was talking about
superstitions, for example.

Uh, we were…

I was, I was talking about we sit at
our, our son's baseball game, and we

might not say a word at the baseball
game, but then when we're home at the TV,

we might not say a word thinking that,
you know, th- th- this is different.

So I was introducing this, we were
thinking about many of this, and I'll

introduce exactly what I mean by this.

And I said, "Well, you know what,
what I'm trying to describe is there's

this myth in clinical trials that
looking at data Causes penalties.

And, uh, I'll, I'll sort
of explain more of this.

And there, there's this,
this thought that, uh, uh…

And it is true in some scenarios.

In some actions you're going to take
in the trial, you have to adjust

alpha because of those actions.

But it depends on the
action you're gonna take.

And in other settings, you, you could do
a million futility analyses and there's no

adjustment to alpha through, through it.

But there's this perception that
looking at data causes penalties

of, you know, what, what's a, a
common thing in asking her ways

that I could introduce this topic.

And she, she was thinking about ways,
and then she said to me, "Haven't you

done that already on your podcast?

Don't you do that, you
know, multiple times?

Are you gonna talk about that again?"

So, uh, and it is true.

And, but it…

I was triggered again today.

I was doing a, a call with a potential
new client, and they brought up the

idea in their trial, uh, enrollment
is slow and, and various issues.

I won't go into it.

I don't, I don't want to give
away any confidential information.

But the topic came up that they
were thinking about doing futility

analysis in the trial, and they didn't
wanna do it because they thought

they would have to adjust alpha.

And this overall perception largely
that looking at data is a bad thing

to do, that that sort of i- is a
corollary to that, and it comes to this.

This is almost an everyday occurrence.

And so when my wife was trying to come
up with me with, with ways to introduce

this topic, in a way, it actually…

The act of trying to come up with
a story is my intro to the topic

today, where, uh, uh, you know,
"Haven't you done this before?"

Uh, and I have done this before.

It's a little bit different today.

I'm gonna come at this
in a different way today.

But I wanna get at this topic and…

of, of adaptive actions in
trials and what that means.

And there's actually beautiful
mathematical aspects to this,

so I wanna get into that.

The adaptive action matrix in trials, and
it's really, it, it's really kind of a

beautiful thing, and I don't think it's
widely understood, that aspect of it.

So let's back up a little bit and
think about how does this myth Uh,

uh, come about and why do people
still believe that looking at data

somehow is a penalty i-i-in the trial?

Historically, really the only
adaptations that were done in

trials were to look for superiority.

Group sequential designs.

So at seventy-five percent or fifty
percent of the way through the trial, we

would do an interim analysis, and if the
data were promising enough, and we would

have to adjust alpha for that because you
can't do a, a two point five percent test

multiple times and still have two point
five percent error in a superiority test.

And so we have to adjust
alpha for the multiple looks.

That was really the only
adaptations that were done.

Simple designs in this, and we had
the really beautiful mathematics of

alpha spending functions, whether
it's, it's Pocock or Brian Fleming, Kim

De Mets, whatever spending function.

We had this beautiful math
around it, th-this adjustment.

And so these are the
only times it was done.

And so we'd be running large phase
three trials, and this is it.

And with that came the notion that,
okay, every time we look at data,

you have to make an adjustment.

And it's not true.

When you do an interim
analysis for superiority, you

have to make an adjustment.

And those were the only times it was done.

And so there was this general perception
that, okay, every time you do that.

And it got attached to looking at
data, not the adaptive action of

claiming superiority at the time,
which does involve an alpha adjustment.

And hence it's still ubiquitous, and
I, I, I heard it today, and they were

surprised when I told them, "Oh, you could
do lots of i-interims for futility and no

adjustment needed, uh, in that setting."

"Oh, that's great.

We, we should do that."

Yeah, you should do that,
um, i-in the setting.

And so it's this, this, this notion that
looking at data means you need to adjust.

There's penalties to it.

It's bad to do it.

That's the other corollary to
do that, that it's bad to do it.

So Tammy is right.

My wife is right.

I did a podcast on this, and she even
says something like spending alpha.

So episode sixteen was spending alpha,
and, uh, it was, last year it was the

most listened to, watched podcast that I
did somewhat for a Bayesian statistician.

Spending alpha was one
of the more common ones.

Um, so you may be interested
in going back to that.

I want to talk about something a
little bit different, uh, in that,

and that is the different adaptive
actions and potential implications

There's a really, really cool aspect
to this, and I'll, I'll go into it.

And, and so I want you
to envision a matrix.

I want you to envision the
matrix where the columns…

Uh, by the way, one fun thing of
doing a podcast, and it's a really

good, um, activity to have to do, is
to do this podcast knowing people are

listening, and I can't use slides.

It's, uh, we, we use slides
become a crutch, a graphic around

that, and they're wonderful.

Uh, if ability to it.

But the ability to have to explain
something without slides, I think is

a, is a good activity to go through in
thinking about how to communicate ideas.

Uh, it, it makes me a
beg- better communicator.

I don't know about a good communicator,
but I, I think it's helpful.

So I want you to think about
the columns here as the data.

So we're gonna do an interim analysis,
and the columns are positive data.

And I'll come back to what does
it mean to be positive data.

So when the data are positive
is the right-hand column.

The left-hand column is when
the data are negative Okay.

And so I'm gonna differentiate
when you do an interim when

the data are in these cases.

And then the, the rows here are the
action, and almost every action you take

at a, at an interim has an implication to
the effective sample size in your trial.

And think about it that you increase
the sample size or you decrease

the sample size at the interim.

And I'll talk at that it's not
explicitly that you change N.

Sometimes you take other actions that
have an implication to your sample size.

So here's our two-by-two matrix.

The columns are positive data on the
right, negative data on the left.

The rows on top are increased
sample size, and on the bottom,

you decrease sample size.

So it's this two-by-two matrix.

The really cool thing about adaptive
actions and do we need to adjust

alpha, does it cause the need to
adjust alpha, is depends on which

of those four quadrants you're in.

So we talked about the historical
scenario where when your data are

good, you decrease the sample size.

So if we're doing a standard group
sequential trial where a hundred is

your maximum sample size, this is a
generic number, and you do an interim

at fifty patients or seventy-five
patients, let's do fifty patients,

and we look to see if the data are,
uh, below a adjusted P value, .001.

If it is, so fifty patients is
when our data are really good, we

decrease from a hundred to fifty.

That's the quadrant where
our data are positive.

We decrease the sample size.

You have to adjust for
actions in that quadrant.

You are inflating type one error
when you are in that quadrant, and

that's the one that every adaptive
design lived in that quadrant.

And everybody thought, "Okay, everything
in all three quadrants, you have to

adjust alpha," but just by looking
at data, and that's not the case.

So, uh, in that, in that, uh, quadrant,
the most common one of these, of

course, is group sequential designs.

And yes, you have to
adjust alpha for that Okay.

And we, we, we have other quadrants.

So let's think about the other most common
quadrant is when your data are negative

and you decrease sample size

So, and we all understand
that that's futility.

So when your data are negative and
you're gonna decrease sample size,

you actually deflate type I error.

You take away the shot that you
might have won had you gone to 100.

So at 50 patients, when our data
aren't very good, we stop for futility

unlikely, and the trial loses.

You only lessen the chance of success,
and under the null hypothesis,

you lessen the chance of getting
an incorrect positive result.

That's type I error.

Um, in that case, we lessen it.

We actually deflate type I error.

So in this quadrant, the lower right, when
the data are positive and we decrease the

sample size, we increase type I error.

But when you move over a quadrant to
the left of that, when the, when the

data are negative and y- your action
is to decrease the sample size, you

deflate type I error In that setting.

Now, there's-- it's controversial
as to whether by doing futility,

we get to actually buy back alpha,
and that a, a, a different topic

for a different, different day.

Um, uh, in that setting, generally,
that, that's generally not done, but

in some s- cases, it might be done.

But that's, that's the result of it.

So there's a circumstance where the advice
I gave today was you can do interims

for futility over and over and over and
over again, and you can't inflate type

I error because the only action you can
take in that scenario is to decrease the

sample size when the data are negative.

Now, the important thing is, and I,
I, I described that we would talk

about, what does it mean for the
data to be positive or negative?

Now, it can depend by the action, but
largely when the data, when the data

are being negative, meaning at that
point, we do not have superiority.

And so that's-- doesn't have
to be all that negative.

The data can still be
on the positive side.

So for these two a- two adaptive actions,
stopping early for supe- superiority

or stopping for futility, the threshold
is actually do we have significance

at that time point, uh, within that?

I mean, that's kind of the, the,
the break point of this as to what

data are negative and positive.

Fairly straightforward, um,
actions in these settings.

Now, it gets really interesting when
you go to the other two quadrants.

So let's think about the quadrant
where the data are positive,

and you increase the sample size

So, so what, what, what
kinds of actions do that?

In a phase III trial, an action that
does that is the promising zone.

So I want to spend a little bit of
time on that 'cause I think that's

the beauty of the mathematics of it.

But other actions at that time
are one common adaptive action is

response adaptive randomization.

So suppose you have two doses and a
control, and we're gonna-- when the

data's positive on one of the two doses,
we increase the randomization to it.

We're increasing its
effective sample size.

When its data are positive, we
increase its effective sample size.

You decrease type I error.

In this matrix, the bottom right
was, uh, increasing type I error.

As you move from bottom right to upper
right, you decrease type I error.

You're gonna see that in the bottom
right, we increase type I error,

in the upper left, we increase
type I error, and then the two

off-diagonals, we decrease type I error.

Futility decreases type I error.

Response adaptive randomization when
the data are positive on the dose,

and we increase its allocation for
the r- for the next stage or moving

forward, we decrease type I error.

Response adaptive randomization
decreases type I error.

Now, there are details to that about
what, what are we, what are we testing

in that scenario, but if we're testing
are one of the doses better than

placebo and we're increasing the rate
to that, we decrease type I error.

You don't have to i-in-- change your
alpha level to do response adaptive

randomization a hundred times in a trial.

No adjustment needed.

We've had special protocol assessments
with the FDA where they agree, and we,

we did twenty interim analyses to do
response adaptive randomization, and

there's no inflation of type I error.

error And so that's a really neat
thing that you don't necessarily

think about in adaptive designs
and what are the actions in that.

Other things of even
dropping arms in a trial.

Suppose you have a h- a hundred
patients in a trial, and when you

get to fifty, you have two doses,
we're doing one-to-one-to-one

randomization, and we drop a dose.

And the remaining fifty patients, we
don't change the overall sample size,

the remaining fifty patients that would
have been one-to-one-to-one are now

one-to-one 'cause we dropped a dose.

We increase the sample
size on the remaining dose.

By dropping that arm, we deflate
type I error in that scenario.

So r-- and by the way, arm dropping is a
form of response adaptive randomization

By making its probability zero in that
we're changing its randomization based

on the data, uh, in that scenario.

So tho- that, that's kind of a weird
place within this matrix that we don't

spend a lot of time talking about.

I wanna go back to the promising
zone 'cause it brings out really

interesting mathematics of this, where
the, the-- what is the promising zone?

The promising zone is an, is a adaptive
sample size trial that, um, uh,

uh, I'll give you the basics of it.

We're going for, we're…

for a sample size of a hundred
one-to-one randomized trial.

The planned sample size is a hundred.

We're gonna do an interim
before a hundred at fifty, say.

And based on the data at fifty, we might
continue on to the planned one hundred.

But at that time, we might increase the
sample size to be bigger than a hundred.

Eh, this is a kind of sample size
re-estimation technique, this special one.

There's a really beautiful result
mathematically, and this comes from

Mehta and Pocock paper, where when you
do the look at fifty, if the conditional

power for a hundred is above 50

You can increase the sample size
above a hundred to anything you want,

and you don't increase type I error.

You actually decrease type I error

So that's the beautiful result where I
describe-- I, I said when the data are

positive, you increase the sample size.

This mathematical result, so when you're
sitting there at 50, if your data are

trending to winning at 100, conditional
power of 50 means you're right on the ef-

the effect size you need to win at 100.

If you continue on with that
effect size, you're gonna win.

If you get a little bit
worse, you're gonna lose.

Anything better, you're gonna win.

That's conditional power of 50.

Right now, we're at the
threshold we need to win.

That's mathematically the definition
of positive data when you're sitting

at 50 and you're planning 100.

You could increase it
to anything you want.

So this design allows you, based
on as long as your conditional

power is above 50, you can increase
your sample size to get 80% power.

You can increase it to 200 to get
80% power within the scenario.

Notice a couple things of this.

In this matrix of positive data and
increasing your sample size, if you're

right on the line of positive data,
so you're at 50% conditional power,

you don't lose any alpha in that.

But if your conditional power is above
that, you're actually now spending alpha.

I, I'm sorry, you're losing alpha,
like futility, that the promising

zone design loses alpha, uh, i-
in that because of this action.

Because at 50, you're increasing to a
sample size, but you can only do it in

a very narrow region, uh, of the design.

So a separate podcast, my Tammy may
be listening at this and say, "Ooh,

I've heard this somewhere before."

By the way, she does listen
to each of the podcasts.

Um, I, I did a podcast, episode
31, the not so promising zone

design It's, it's commonly done,
and it doesn't work very well.

So please go into that one.

So the math is beautiful, fantastic.

The design and the actions are not.

It doesn't work very well.

If you compare it to other adaptive
approaches, it doesn't work very well.

Spend alpha and do a group sequential.

That's the spending alpha episode 16.

So it all kind of ties in together.

I got…

You know, so I don't
wanna go deep into those.

Please join those episodes if,
if, if any of that is interesting.

So the promising zone design lives in
this new quadrant that we haven't been

doing many adaptive designs in there.

Now, the, the interesting thing is they've
created ways that you can solve exactly.

Sometimes you can push that
fifty percent even lower.

You know, forty percent conditional
power thirty-seven without inflating

type one error, uh, in that scenario.

So you can calculate the math, and
then if you go below that, where your

data is defined to be negative, we
actually move the quadrant from the

upper right to the upper left, where
our data at that time are negative,

and we increase the sample size.

That increases type one error.

So now we've got this
matrix of actions and data.

And re-remind us where we are.

The, the, the right-hand
column is positive data.

If you decrease the sample size or
you do an action that decreases sample

size, you increase type one error
If you do something that increases

the sample size when you have good
data, that decreases the type one

error in those several actions.

Moving over to the left-hand
column are negative data.

And so now to win at 100, we're actually
in this conditional power less than 50%.

We're, we're, we- we're, we're not
trending to a successful trial.

Our data are negative.

What if we wanna push the sample
size to 200 or, or to 1,000?

Now, when the data are negative, if you
increase your sample size, you're almost

saying, "Oh, just give me a new trial.

Give me 1,000 patients.

Give me bigger numbers."

You're almost taking multiple
shots on goal by doing that.

When it's negative, I'll just collect
enough data that will, will, will,

will weigh out this initial negative.

You're, you're doing kind of the
opposite of group sequential, but

you have the same effect of it, that
you're getting multiple shots on goal.

You increase type one error by doing that.

This is not understand- stood very
well that the, the-- this circumstance.

There are a lot of people that I see
designs where, oh, we'll just do an

interim at the design, and we'll push the
sample size to be whatever sample size

gives us ninety percent conditional power.

That trial will inflate type
one error i- in the scenario.

You, you, you don't get this free lunch.

So if you're, if you're stuck on
type one error, you live in that

quadrant there of doing that.

A group sequential design, you
could almost think of group

sequential as the opposite of we
have 100 and we might stop at 50.

A group sequential design is we're
gonna go to 50 and see if we win.

If we don't win, we're gonna increase
the sample size to 75 and see if we win.

And oh, if our data are negative,
we're gonna go to 100 and look at it.

You can rewrite a group sequential design
to live in this quadrant in the upper

left, where we're taking actions on
negative data to increase our sample size.

Multiple shots on goal, you're doing
things that increase type one error

Okay.

So now this is the, the,
this is the circumstance.

Now overall, the general understanding
of this is because we're in

the lower right-hand quadrant,
that looking at data is bad.

There are huge tools throughout here to
create a better design, and going back to

this episode, uh, 16 on spending alpha,
spending alpha is a good thing to do.

Recognizing your, your actions here
and when it happens in the data

in order to create a better design
to get better answers within the

constraints of it is a good thing to do.

So I hope envisioning all of this
as your actions, the important

part of it is looking at data
doesn't increase type one error.

It's the action you may take.

Really important part of adaptive designs,
this is why it's absolutely critical

you have a prospective adaptive design.

You get a great deal of flexibility in
the design as long as it's pre-specified.

If you try to go with an adaptive
design as, "Oh, I'll just look at

the data and then decide what to
do," you can't argue you're in this

quadrant, you're in that quadrant,
these are the actions I'm taking.

You can't even control type one error
from such a design because you haven't

defined the actions at that time from it
The other, the other thing that I, I find

particularly annoying, uh, my pet peeve of
the day, Kurt Veily has pet peeves of the

day, is I'll see where somebody's doing
a futility analysis, and they spend .0001

alpha as a-- But are you
gonna do a superiority test?

No, no, we don't wanna
do superiority at all.

But they're throwing alpha to the gods
because they almost think they have to.

And it's almost disingenuous to do that
because you don't wanna talk for-- stop

for superiority, but yet you allocate it
as though you might do something else.

What, what, what's the design?

I think it's a circumstance where you
just be very clear, be prospective.

You can have a great deal of flexibility.

You can design better trials by
being fully prospective in it.

A great deal of flexibility
in the sample size.

Uh, you can accomplish what you're
trying to accomplish with, with

efficient trial designs, but it has to
be prospectively set up, and you can

take advantage of some of the math here.

Um, going back to introducing
this that, uh, Tammy may say,

"I've heard all that before."

Uh, hoping this matrix idea of thinking
about it, data positive, data negative,

and the action being increasing effective
sample size or decreasing effective sample

size might be something new for you to
think about the world of adaptive designs.

I, I was going to go into this
whole aspect of in golf, knowing the

rules in golf, you can actually drop
your ball in certain situations.

You get a great deal of flexibility
under the rules of golf.

Um, and it's not always you can't
touch your ball, you can't do it.

Knowing the rules helps
you to be a better golfer.

I was gonna enter with that, but I
thought I would enter with, with my

wife saying, "You've done this before."

And yes, I've done it before,
and it's in part because every

day in my day job, I hear these
kinds of things from statisticians

and clinical trialists alike.

So know the rules, design better
trials, use them to your advantage.

Efficient, good trial designs out there.

And until the meantime, we will be here,
maybe spending alpha, maybe not spending

alpha, but we'll be here in the interim.