Judith: Welcome to Berry's In the Interim podcast, where we explore the
cutting edge of innovative clinical trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Well, welcome everybody back to In The Interim.
I'm your host, Scott Berry.
Appreciate you joining me again today.
Uh, In The Interim is a podcast of all things clinical trial science.
Now, if you go back to some of this, I, I try, I, I try to create a, a,
a much more welcome intro to each of the topics, and it might be a story of
something that happens that I feel like is relevant to introduce the story.
So I, I was, I was talking with, uh, uh, my family, I was talking to my wife,
Tammy, who she ends up being a fairly common topic on this, this podcast.
And I was describing the, the topic and thinking about is, you know, is there
something, there's gotta be something where, um, r- really the topic is
where something is true in a particular scenario and people understand that,
but it's not true in another scenario, but they carry that with them and
there's a sort of myth about it that this is a, a much more universal thing.
And, um, and not understanding the aspect of it, and this just carries forward.
And was, was talking about superstitions, for example.
Uh, we wereâ¦
I was, I was talking about we sit at our, our son's baseball game, and we
might not say a word at the baseball game, but then when we're home at the TV,
we might not say a word thinking that, you know, th- th- this is different.
So I was introducing this, we were thinking about many of this, and I'll
introduce exactly what I mean by this.
And I said, "Well, you know what, what I'm trying to describe is there's
this myth in clinical trials that looking at data Causes penalties.
And, uh, I'll, I'll sort of explain more of this.
And there, there's this, this thought that, uh, uhâ¦
And it is true in some scenarios.
In some actions you're going to take in the trial, you have to adjust
alpha because of those actions.
But it depends on the action you're gonna take.
And in other settings, you, you could do a million futility analyses and there's no
adjustment to alpha through, through it.
But there's this perception that looking at data causes penalties
of, you know, what, what's a, a common thing in asking her ways
that I could introduce this topic.
And she, she was thinking about ways, and then she said to me, "Haven't you
done that already on your podcast?
Don't you do that, you know, multiple times?
Are you gonna talk about that again?"
So, uh, and it is true.
And, but itâ¦
I was triggered again today.
I was doing a, a call with a potential new client, and they brought up the
idea in their trial, uh, enrollment is slow and, and various issues.
I won't go into it.
I don't, I don't want to give away any confidential information.
But the topic came up that they were thinking about doing futility
analysis in the trial, and they didn't wanna do it because they thought
they would have to adjust alpha.
And this overall perception largely that looking at data is a bad thing
to do, that that sort of i- is a corollary to that, and it comes to this.
This is almost an everyday occurrence.
And so when my wife was trying to come up with me with, with ways to introduce
this topic, in a way, it actuallyâ¦
The act of trying to come up with a story is my intro to the topic
today, where, uh, uh, you know, "Haven't you done this before?"
Uh, and I have done this before.
It's a little bit different today.
I'm gonna come at this in a different way today.
But I wanna get at this topic andâ¦
of, of adaptive actions in trials and what that means.
And there's actually beautiful mathematical aspects to this,
so I wanna get into that.
The adaptive action matrix in trials, and it's really, it, it's really kind of a
beautiful thing, and I don't think it's widely understood, that aspect of it.
So let's back up a little bit and think about how does this myth Uh,
uh, come about and why do people still believe that looking at data
somehow is a penalty i-i-in the trial?
Historically, really the only adaptations that were done in
trials were to look for superiority.
Group sequential designs.
So at seventy-five percent or fifty percent of the way through the trial, we
would do an interim analysis, and if the data were promising enough, and we would
have to adjust alpha for that because you can't do a, a two point five percent test
multiple times and still have two point five percent error in a superiority test.
And so we have to adjust alpha for the multiple looks.
That was really the only adaptations that were done.
Simple designs in this, and we had the really beautiful mathematics of
alpha spending functions, whether it's, it's Pocock or Brian Fleming, Kim
De Mets, whatever spending function.
We had this beautiful math around it, th-this adjustment.
And so these are the only times it was done.
And so we'd be running large phase three trials, and this is it.
And with that came the notion that, okay, every time we look at data,
you have to make an adjustment.
And it's not true.
When you do an interim analysis for superiority, you
have to make an adjustment.
And those were the only times it was done.
And so there was this general perception that, okay, every time you do that.
And it got attached to looking at data, not the adaptive action of
claiming superiority at the time, which does involve an alpha adjustment.
And hence it's still ubiquitous, and I, I, I heard it today, and they were
surprised when I told them, "Oh, you could do lots of i-interims for futility and no
adjustment needed, uh, in that setting."
"Oh, that's great.
We, we should do that."
Yeah, you should do that, um, i-in the setting.
And so it's this, this, this notion that looking at data means you need to adjust.
There's penalties to it.
It's bad to do it.
That's the other corollary to do that, that it's bad to do it.
So Tammy is right.
My wife is right.
I did a podcast on this, and she even says something like spending alpha.
So episode sixteen was spending alpha, and, uh, it was, last year it was the
most listened to, watched podcast that I did somewhat for a Bayesian statistician.
Spending alpha was one of the more common ones.
Um, so you may be interested in going back to that.
I want to talk about something a little bit different, uh, in that,
and that is the different adaptive actions and potential implications
There's a really, really cool aspect to this, and I'll, I'll go into it.
And, and so I want you to envision a matrix.
I want you to envision the matrix where the columnsâ¦
Uh, by the way, one fun thing of doing a podcast, and it's a really
good, um, activity to have to do, is to do this podcast knowing people are
listening, and I can't use slides.
It's, uh, we, we use slides become a crutch, a graphic around
that, and they're wonderful.
Uh, if ability to it.
But the ability to have to explain something without slides, I think is
a, is a good activity to go through in thinking about how to communicate ideas.
Uh, it, it makes me a beg- better communicator.
I don't know about a good communicator, but I, I think it's helpful.
So I want you to think about the columns here as the data.
So we're gonna do an interim analysis, and the columns are positive data.
And I'll come back to what does it mean to be positive data.
So when the data are positive is the right-hand column.
The left-hand column is when the data are negative Okay.
And so I'm gonna differentiate when you do an interim when
the data are in these cases.
And then the, the rows here are the action, and almost every action you take
at a, at an interim has an implication to the effective sample size in your trial.
And think about it that you increase the sample size or you decrease
the sample size at the interim.
And I'll talk at that it's not explicitly that you change N.
Sometimes you take other actions that have an implication to your sample size.
So here's our two-by-two matrix.
The columns are positive data on the right, negative data on the left.
The rows on top are increased sample size, and on the bottom,
you decrease sample size.
So it's this two-by-two matrix.
The really cool thing about adaptive actions and do we need to adjust
alpha, does it cause the need to adjust alpha, is depends on which
of those four quadrants you're in.
So we talked about the historical scenario where when your data are
good, you decrease the sample size.
So if we're doing a standard group sequential trial where a hundred is
your maximum sample size, this is a generic number, and you do an interim
at fifty patients or seventy-five patients, let's do fifty patients,
and we look to see if the data are, uh, below a adjusted P value, .001.
If it is, so fifty patients is when our data are really good, we
decrease from a hundred to fifty.
That's the quadrant where our data are positive.
We decrease the sample size.
You have to adjust for actions in that quadrant.
You are inflating type one error when you are in that quadrant, and
that's the one that every adaptive design lived in that quadrant.
And everybody thought, "Okay, everything in all three quadrants, you have to
adjust alpha," but just by looking at data, and that's not the case.
So, uh, in that, in that, uh, quadrant, the most common one of these, of
course, is group sequential designs.
And yes, you have to adjust alpha for that Okay.
And we, we, we have other quadrants.
So let's think about the other most common quadrant is when your data are negative
and you decrease sample size
So, and we all understand that that's futility.
So when your data are negative and you're gonna decrease sample size,
you actually deflate type I error.
You take away the shot that you might have won had you gone to 100.
So at 50 patients, when our data aren't very good, we stop for futility
unlikely, and the trial loses.
You only lessen the chance of success, and under the null hypothesis,
you lessen the chance of getting an incorrect positive result.
That's type I error.
Um, in that case, we lessen it.
We actually deflate type I error.
So in this quadrant, the lower right, when the data are positive and we decrease the
sample size, we increase type I error.
But when you move over a quadrant to the left of that, when the, when the
data are negative and y- your action is to decrease the sample size, you
deflate type I error In that setting.
Now, there's-- it's controversial as to whether by doing futility,
we get to actually buy back alpha, and that a, a, a different topic
for a different, different day.
Um, uh, in that setting, generally, that, that's generally not done, but
in some s- cases, it might be done.
But that's, that's the result of it.
So there's a circumstance where the advice I gave today was you can do interims
for futility over and over and over and over again, and you can't inflate type
I error because the only action you can take in that scenario is to decrease the
sample size when the data are negative.
Now, the important thing is, and I, I, I described that we would talk
about, what does it mean for the data to be positive or negative?
Now, it can depend by the action, but largely when the data, when the data
are being negative, meaning at that point, we do not have superiority.
And so that's-- doesn't have to be all that negative.
The data can still be on the positive side.
So for these two a- two adaptive actions, stopping early for supe- superiority
or stopping for futility, the threshold is actually do we have significance
at that time point, uh, within that?
I mean, that's kind of the, the, the break point of this as to what
data are negative and positive.
Fairly straightforward, um, actions in these settings.
Now, it gets really interesting when you go to the other two quadrants.
So let's think about the quadrant where the data are positive,
and you increase the sample size
So, so what, what, what kinds of actions do that?
In a phase III trial, an action that does that is the promising zone.
So I want to spend a little bit of time on that 'cause I think that's
the beauty of the mathematics of it.
But other actions at that time are one common adaptive action is
response adaptive randomization.
So suppose you have two doses and a control, and we're gonna-- when the
data's positive on one of the two doses, we increase the randomization to it.
We're increasing its effective sample size.
When its data are positive, we increase its effective sample size.
You decrease type I error.
In this matrix, the bottom right was, uh, increasing type I error.
As you move from bottom right to upper right, you decrease type I error.
You're gonna see that in the bottom right, we increase type I error,
in the upper left, we increase type I error, and then the two
off-diagonals, we decrease type I error.
Futility decreases type I error.
Response adaptive randomization when the data are positive on the dose,
and we increase its allocation for the r- for the next stage or moving
forward, we decrease type I error.
Response adaptive randomization decreases type I error.
Now, there are details to that about what, what are we, what are we testing
in that scenario, but if we're testing are one of the doses better than
placebo and we're increasing the rate to that, we decrease type I error.
You don't have to i-in-- change your alpha level to do response adaptive
randomization a hundred times in a trial.
No adjustment needed.
We've had special protocol assessments with the FDA where they agree, and we,
we did twenty interim analyses to do response adaptive randomization, and
there's no inflation of type I error.
error And so that's a really neat thing that you don't necessarily
think about in adaptive designs and what are the actions in that.
Other things of even dropping arms in a trial.
Suppose you have a h- a hundred patients in a trial, and when you
get to fifty, you have two doses, we're doing one-to-one-to-one
randomization, and we drop a dose.
And the remaining fifty patients, we don't change the overall sample size,
the remaining fifty patients that would have been one-to-one-to-one are now
one-to-one 'cause we dropped a dose.
We increase the sample size on the remaining dose.
By dropping that arm, we deflate type I error in that scenario.
So r-- and by the way, arm dropping is a form of response adaptive randomization
By making its probability zero in that we're changing its randomization based
on the data, uh, in that scenario.
So tho- that, that's kind of a weird place within this matrix that we don't
spend a lot of time talking about.
I wanna go back to the promising zone 'cause it brings out really
interesting mathematics of this, where the, the-- what is the promising zone?
The promising zone is an, is a adaptive sample size trial that, um, uh,
uh, I'll give you the basics of it.
We're going for, we'reâ¦
for a sample size of a hundred one-to-one randomized trial.
The planned sample size is a hundred.
We're gonna do an interim before a hundred at fifty, say.
And based on the data at fifty, we might continue on to the planned one hundred.
But at that time, we might increase the sample size to be bigger than a hundred.
Eh, this is a kind of sample size re-estimation technique, this special one.
There's a really beautiful result mathematically, and this comes from
Mehta and Pocock paper, where when you do the look at fifty, if the conditional
power for a hundred is above 50
You can increase the sample size above a hundred to anything you want,
and you don't increase type I error.
You actually decrease type I error
So that's the beautiful result where I describe-- I, I said when the data are
positive, you increase the sample size.
This mathematical result, so when you're sitting there at 50, if your data are
trending to winning at 100, conditional power of 50 means you're right on the ef-
the effect size you need to win at 100.
If you continue on with that effect size, you're gonna win.
If you get a little bit worse, you're gonna lose.
Anything better, you're gonna win.
That's conditional power of 50.
Right now, we're at the threshold we need to win.
That's mathematically the definition of positive data when you're sitting
at 50 and you're planning 100.
You could increase it to anything you want.
So this design allows you, based on as long as your conditional
power is above 50, you can increase your sample size to get 80% power.
You can increase it to 200 to get 80% power within the scenario.
Notice a couple things of this.
In this matrix of positive data and increasing your sample size, if you're
right on the line of positive data, so you're at 50% conditional power,
you don't lose any alpha in that.
But if your conditional power is above that, you're actually now spending alpha.
I, I'm sorry, you're losing alpha, like futility, that the promising
zone design loses alpha, uh, i- in that because of this action.
Because at 50, you're increasing to a sample size, but you can only do it in
a very narrow region, uh, of the design.
So a separate podcast, my Tammy may be listening at this and say, "Ooh,
I've heard this somewhere before."
By the way, she does listen to each of the podcasts.
Um, I, I did a podcast, episode 31, the not so promising zone
design It's, it's commonly done, and it doesn't work very well.
So please go into that one.
So the math is beautiful, fantastic.
The design and the actions are not.
It doesn't work very well.
If you compare it to other adaptive approaches, it doesn't work very well.
Spend alpha and do a group sequential.
That's the spending alpha episode 16.
So it all kind of ties in together.
I gotâ¦
You know, so I don't wanna go deep into those.
Please join those episodes if, if, if any of that is interesting.
So the promising zone design lives in this new quadrant that we haven't been
doing many adaptive designs in there.
Now, the, the interesting thing is they've created ways that you can solve exactly.
Sometimes you can push that fifty percent even lower.
You know, forty percent conditional power thirty-seven without inflating
type one error, uh, in that scenario.
So you can calculate the math, and then if you go below that, where your
data is defined to be negative, we actually move the quadrant from the
upper right to the upper left, where our data at that time are negative,
and we increase the sample size.
That increases type one error.
So now we've got this matrix of actions and data.
And re-remind us where we are.
The, the, the right-hand column is positive data.
If you decrease the sample size or you do an action that decreases sample
size, you increase type one error If you do something that increases
the sample size when you have good data, that decreases the type one
error in those several actions.
Moving over to the left-hand column are negative data.
And so now to win at 100, we're actually in this conditional power less than 50%.
We're, we're, we- we're, we're not trending to a successful trial.
Our data are negative.
What if we wanna push the sample size to 200 or, or to 1,000?
Now, when the data are negative, if you increase your sample size, you're almost
saying, "Oh, just give me a new trial.
Give me 1,000 patients.
Give me bigger numbers."
You're almost taking multiple shots on goal by doing that.
When it's negative, I'll just collect enough data that will, will, will,
will weigh out this initial negative.
You're, you're doing kind of the opposite of group sequential, but
you have the same effect of it, that you're getting multiple shots on goal.
You increase type one error by doing that.
This is not understand- stood very well that the, the-- this circumstance.
There are a lot of people that I see designs where, oh, we'll just do an
interim at the design, and we'll push the sample size to be whatever sample size
gives us ninety percent conditional power.
That trial will inflate type one error i- in the scenario.
You, you, you don't get this free lunch.
So if you're, if you're stuck on type one error, you live in that
quadrant there of doing that.
A group sequential design, you could almost think of group
sequential as the opposite of we have 100 and we might stop at 50.
A group sequential design is we're gonna go to 50 and see if we win.
If we don't win, we're gonna increase the sample size to 75 and see if we win.
And oh, if our data are negative, we're gonna go to 100 and look at it.
You can rewrite a group sequential design to live in this quadrant in the upper
left, where we're taking actions on negative data to increase our sample size.
Multiple shots on goal, you're doing things that increase type one error
Okay.
So now this is the, the, this is the circumstance.
Now overall, the general understanding of this is because we're in
the lower right-hand quadrant, that looking at data is bad.
There are huge tools throughout here to create a better design, and going back to
this episode, uh, 16 on spending alpha, spending alpha is a good thing to do.
Recognizing your, your actions here and when it happens in the data
in order to create a better design to get better answers within the
constraints of it is a good thing to do.
So I hope envisioning all of this as your actions, the important
part of it is looking at data doesn't increase type one error.
It's the action you may take.
Really important part of adaptive designs, this is why it's absolutely critical
you have a prospective adaptive design.
You get a great deal of flexibility in the design as long as it's pre-specified.
If you try to go with an adaptive design as, "Oh, I'll just look at
the data and then decide what to do," you can't argue you're in this
quadrant, you're in that quadrant, these are the actions I'm taking.
You can't even control type one error from such a design because you haven't
defined the actions at that time from it The other, the other thing that I, I find
particularly annoying, uh, my pet peeve of the day, Kurt Veily has pet peeves of the
day, is I'll see where somebody's doing a futility analysis, and they spend .0001
alpha as a-- But are you gonna do a superiority test?
No, no, we don't wanna do superiority at all.
But they're throwing alpha to the gods because they almost think they have to.
And it's almost disingenuous to do that because you don't wanna talk for-- stop
for superiority, but yet you allocate it as though you might do something else.
What, what, what's the design?
I think it's a circumstance where you just be very clear, be prospective.
You can have a great deal of flexibility.
You can design better trials by being fully prospective in it.
A great deal of flexibility in the sample size.
Uh, you can accomplish what you're trying to accomplish with, with
efficient trial designs, but it has to be prospectively set up, and you can
take advantage of some of the math here.
Um, going back to introducing this that, uh, Tammy may say,
"I've heard all that before."
Uh, hoping this matrix idea of thinking about it, data positive, data negative,
and the action being increasing effective sample size or decreasing effective sample
size might be something new for you to think about the world of adaptive designs.
I, I was going to go into this whole aspect of in golf, knowing the
rules in golf, you can actually drop your ball in certain situations.
You get a great deal of flexibility under the rules of golf.
Um, and it's not always you can't touch your ball, you can't do it.
Knowing the rules helps you to be a better golfer.
I was gonna enter with that, but I thought I would enter with, with my
wife saying, "You've done this before."
And yes, I've done it before, and it's in part because every
day in my day job, I hear these kinds of things from statisticians
and clinical trialists alike.
So know the rules, design better trials, use them to your advantage.
Efficient, good trial designs out there.
And until the meantime, we will be here, maybe spending alpha, maybe not spending
alpha, but we'll be here in the interim.