Subscribe
Share
Share
Embed
A podcast on statistical science and clinical trials.
Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.
Judith: Welcome to Berry's In the
Interim podcast, where we explore the
cutting edge of innovative clinical
trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott Berry: Well, welcome back to In the
Interim, a podcast from Barry Consultants.
I'm your host, Scott Berry,
and I am going solo today.
What, what are we gonna talk about today?
I, I occasionally get the urge to do these
podcasts, uh, based on my, my, my day
job, which is designing clinical trials.
And the name in the interim,
of course, comes from the fact
that we do a lot of adaptive.
I.
Designs which use interim analyses or
adaptive analyses during the course of a
trial, and hence the name in the interim.
We are spending some
time in the interim here.
So I was, uh, I was working
on a project last week.
I.
And this is something that was, was
said to me, which, uh, has said to me,
has been said to me hundreds of times
in the 25 years that I've been doing
this, where they were interested in
potential adaptations in a clinical trial.
And when I mentioned potential
adaptations, their first co comment
was, will this cost us alpha?
Okay, so the title of this
podcast, spending Alpha,
we're gonna talk about Alpha.
Uh, I will try to explain what Alpha is.
I will try to set it up.
Many of you know what Alpha is, but that
comment, uh, it, it, it, it's such a
challenging comment because in almost
all circumstances you don't lose alpha.
So what is Alpha?
Let's back up.
Typically, in a a clinical trial,
you're testing, and we're gonna
talk about a superiority trial.
There are other kinds of trials,
comparative effectiveness,
non-inferiority trials.
But a trial, a phase three trial that's
trying to demonstrate a particular
investigational therapy is better than
the control, let's call it the placebo.
At, at the center of this, when the trial
is over, the primary analysis of most
of these trials is trying to demonstrate
that the treatment is superior to placebo.
We do that through hypothesis testing,
and many of you are familiar with
the structure of a hypothesis test.
We refer to the null hypothesis
as, as the thing we're trying to
disprove, and typically that's the, the
treatment and the placebo are equal.
That the treatment is
not better than placebo.
You can phrase it that the treatments,
uh, know better, potentially worse,
and the alternative is that the
treatment is superior to placebo.
If we're able to collect data in the
trial that demonstrates that data
is unlikely under the null, under
equality of treatment and placebo.
We can, we, we would claim with
that hypothesis test that the
treatment is superior to placebo.
That's a standard hypothesis test
and a standard phase three trial.
We, we typically draw a line and say,
how unlikely does that data have to
be under the null hypothesis, assuming
the null in order to reject the null
and claim that treatment is superior.
We, uh, you'll, you'll frequently
hear 5% alpha, uh, in it.
And it's such a, that this is not the
topic of this conversation, but it's
something else that, um, I, I've always
struggled with is that in those trials,
it's a, it's a, it's a, uh, it's a one
directional null hypo, uh, alternative
hypothesis that the treatment is superior.
In that scenario, you really get 2.5%.
You get half of that alpha
to demonstrate superiority.
It, it's interesting because I taught
at Texas a and m for five years before
I got into clinical trials, and in a
basic stat course, if you phrase that
problem of hypothesis testing, and
you said set up the hypothesis test
and tell me about the type one error.
They would refer to it as a greater
than, the treatment is greater
than the placebo, and there's 2.5%
alpha.
If you read most protocols of a
superiority trial, they still talk
about two-sided alpha, that you get 5%.
You don't get 5%, you get 2.5.
Most of the protocols would be marked
wrong in a basic stat course, and there's
this weird tradition about still talking
about two-sided 5% alpha, which which
doesn't really make sense in that setting.
So.
Throughout.
Throughout our discussion,
I'm gonna talk about 2.5%,
and that's the standard
for a superiority trial.
That if you can demonstrate with a 2.5%
type 1 error that the treatment is
superior to the placebo, that's typically
referring to it as a successful trial,
and you've demonstrated in that trial
that treatment is superior to placebo.
Now that quantity, that 2.5%
is usually labeled it with
the Greek letter alpha.
α So that's the typical standard of a
phase three trial that you get this 2.5%
type 1 error.
Now, what does that mean?
Why, how does that 2.5%,
just to back up a little bit
more, is that if the treatment
and placebo are exactly equal.
And you run the experiment.
I'll talk about a 400 patient trial.
You run the, the trial, the probability,
assuming they're exactly equal,
that you would get data as extreme.
As what we need to see,
assuming they're equal, is 2.5%
or less, and they refer
to it as a type one error.
If they're equal, the probability
we make a mistake and say the
treatment is superior is 2.5%
in that trial.
And that's a, that's standard
for a phase three trial.
Now before we go on, many of you
will recognize me as a a, a Bayesian.
I, I am a practicing
Bayesian statistician.
By the way, for those of you out
there, you don't actually have to
declare your allegiance to one side
or the other, but I think like a
Bayesian, I was trained as a Bayesian.
Uh, it's the way I set up models.
I, I much prefer to do a Bayesian
approach, which is really quite.
Opposite of the frequentist
approach and type one error, and
it uses posterior probabilities
that the treatment is superior.
Uh, but I live in this world
of clinical trials where the,
the standard is frequentist.
Hypothesis testing.
So let's talk about that and let's,
let's live in that space for now.
Even though I spend a good bit of
time doing Bayesian trials, even
doing Bayesian analysis, taking
advantage of frequentist approaches.
But let's stick to, to standard
frequentist, uh, phase three trials, 2.5%
Type one error Now.
Historically, the f the the most
trials were fixed sample size.
Let's enroll to the end
and do a single analysis.
When you do that single analysis, if
the P value, which is the probability
of seeing data as extreme, that the
treatment would be, would, would beat
the placebo by as much as it does in the
trial if they're equal, is the P value.
And when that's less than 2.5,
we say that we demonstrated
the treatment is superior.
Now we, in a traditional trial,
you do a single time point.
You do that analysis and you get 2.5%
alpha straightforward.
Traditionally, the first types of
adaptive designs that were done.
We're looking at potential early analyses.
So before that final time point, you do
an analysis and maybe we've demonstrated
superior before, superiority before that.
So let's take a trial that originally
a fixed sample size of 400.
Now we might do interim analysis
at 200 patients before the 400
and at 300 and then at 400.
This is sort of the first
type of adaptive trial.
It's now called a group sequential trial.
It's probably always been called
a group sequential trial now.
Now, because you do an analysis
for superiority at 200.
Then you do another one at 300,
and you do another one at 400.
You're really taking three shots on
goal for showing potential superiority.
If you were to do that, using
the same alpha at each test.
We reject if the P value's less
than alpha at 200, at 300 or 400,
the experiment itself would have
a higher type one error than 2.5%
because you took three shots at getting
superiority so that that would not
satisfy the experiment having 2.5%
type one error.
It.
So what we do in those trials
is we adjust the alpha at
each of the three time points.
We lower it from 2.5
when we create three different alphas
referred to as the nominal alpha.
At that test, we have an alpha at
200 and alpha at 300, an alpha at
400, and their beautiful theory.
Uh, O'Brien, Fleming and group
sequential theory, many ways to
allocate different Alpha at the three
interims so that the experiment has 2.5%
type one error.
For example, if you did O'Brien Fleming's
spending boundaries at 200, 300 and
400, the Alpha 200 would be 0.0031.
At 300 would be 0.0092,
and then at 400 would be 0.0213.
So all of those are less than 0.025.
And you can show that if the treatment
doesn't work, that there's a 2.5%
chance that you would be successful.
At least one of those three analyses.
And that's group sequential testing.
And this is historically the first
type of adaptive trials that were done.
Now what notice at the final
analysis at 400, the, the, the alpha
value, the nominal alphas 0.0213.
If you didn't do the two
interims, it would be 0.0
2 5, 0.
It would be bigger.
It because you do those two interims.
The final nominal alpha
is less than 0.025.
That historically has
been called a penalty.
That's the penalty for
the interim analysis now.
This is where I start
to struggle with that.
I hate the term penalty.
I, I think it's actually caused
a lot of misunderstanding, and
I'll describe the sort of first
misunderstanding to that is you get 2.5%
type 1 error.
You've just allocated it
over the three analyses.
You haven't lost anything.
That experiment gives you that same 2.5.
You've just allocated it at three points.
Yes, you've lowered the number at 400, but
you had a bigger number than zero, which
you had in the fixed trial at 200 and 300.
So you've just spread it out.
You've distributed across three analyses.
And there can be benefits
to that in the experiment.
The trial, the, the treatment could be
highly effective and you learn it at
200, you save time and patients, you get
the treatment out to patients earlier.
So there can be large benefits to that.
But because this, these were really
the first time adaptive trials
were done, this notion went to.
When you do those interims,
you have to adjust alpha,
that there's a penalty for it.
And it became that there's a
penalty for looking at data
and it, it's still pervasive today
that the notion is looking at data
is bad and you pay a penalty for it.
And neither one of those is true.
Looking at data is not bad and
you don't pay a penalty for it.
You just spread your alpha at
three different time points.
So hence in introducing it in
adaptive design and a a clinician
or a trial design sponsor says.
Does it cost me alpha?
It's such a hard thing to respond
to because the answer's no.
But yes, you have to distribute it.
So it's one of those things
you have to be careful.
I, I have to be careful how I answer that.
Now
let me, let me walk
through a couple of these.
What I mean by it's not bad
and you haven't lost anything.
So, for example, if we are running
a fixed trial of 400 patients.
And we have, uh, continuous outcomes.
So we're doing a t-test
at the end of the day.
We're testing a, um, we're testing
a, a functional rating scale or a
clinical outcome, or an exercise
test or a biological measurement
of something within a patient.
And most tests.
Are essentially normal.
This could be a test of, of
of, of dichotomous outcomes.
The rate of mortality that if that 400
patient, that 400 patient trial, if we
have what's called an effect size of 0.3,
so the, the, the treatment has a
benefit relative to placebo, that's
30% of the standard deviation.
So if we're doing a six minute walk
test and the standard deviation in
the six minute walk is 50 meters
and our treatment has an advantage,
uh, of that of 15 meters, that's 0.3
15 divided by 30 is an effect size of 0.3.
That would be in the trial of 400
patients, it would be 85% power.
And you could run a fixed trial
and you'd have an 85% chance of
demonstrating superiority that your p
value at the end is less than 0.025.
Now that's, and, and you
get, you, you, you do 0.025
at the end because you're
doing a single interim.
Now, suppose we do interims at 200, 304,
and we do the final at 400 using exactly
the alpha values that I just gave you.
That trial has overall less power.
The power goes from 85 to 84.
It's essentially exactly a one
percentage reduction in power.
Now that happens, you haven't lost
alpha, you don't lose power because you.
You, you, you pay a penalty in Alpha.
It's because you distributed some of
that alpha to smaller sample sizes.
So you took a shot on goal at 200,
but it's smaller than 400, and hence,
your power goes down a little bit.
From that, it goes down
a very small amount.
1%.
For that 1% you save on
average in that same 0.3
effect size, you save.
87 patients on average, on a 400 patient
trial, the average is 313 patients.
That can be a huge advantage.
The time to enroll those patients,
uh, the, the, the number of patients
enrolled to demonstrate superiority.
So you're dropping power by 1%.
You're saving 87 patients
in a 400 patient trial now.
That's a different trial design,
and you get to decide as the
sponsor, as the constructor of
that trial, would you prefer that?
Now there's no penalty to that.
You've distributed it just
like in that 400 patient trial.
You didn't save any
alpha for 500 patients.
You've spent it all at 400, but
we don't talk about the penalty.
For a sample size of 500,
you left zero to that.
So partly what happens is this, this
notion and this, this, this penalty and
it, there really is this thought that
looking at data is bad and it's not bad.
It actually restricts us from
constructing better trial designs.
So for example.
Let's do interims at 200, 300
and at 400 and allow the trial
to potentially go to 500.
So let's not spend all our alpha at 400.
Let's leave some leftover for 500.
So again, I do O'Brien Fleming
boundaries at 200, 300, 400, 500.
That same effect size that had 85% power.
With 400 patients, every trial is
400 and it's 84 per 85% powered.
This new trial that allows it
to go to 500 is now 91% powered.
I've increased the power from 85 to 91.
I've reduced the number of failed
trials when my treatment works.
It was 15% of the trials
times the trial fails.
Now it's now it's 9% of
the time the trial fails.
I, I've gotten rid, rid of 40% of
my failed trials with that, and
my average sample size is 368.
So by allowing flexibility, my average
sample size is smaller than 400.
My power is greater, goes from 85 to 91.
In many ways, this is a much better
trial design, and I'd much rather see
that trial design getting over 90% power
with a smaller, average sample size.
If we were to do this trial over and over
and over again, instead of the 400, we
take less time, we spend less patients,
and we get the right answer more often,
and our type one error is no bigger.
But we don't do that a lot of times
because there's this thought that
it's bad to look at data and I'm
spending alpha and I don't wanna
spend my alpha, I only have 2.5.
Again, you're not spending alpha, you're
distributing over different time points.
The name matters.
And let me give you another example of,
uh, uh, of the name meaning something
and it, it actually hurting it.
So the.
The, when you're driving down an
interstate in the United States and you
drive in the left hand lane that lane,
many people in this country will just sort
of refer to that lane as the fast lane.
I, that's a harmful name for the
left hand lane on that, because the
notion is, oh, that's the fast lane.
If the speed limit is 70.
And I'm driving 73.
It's okay to be in that lane.
It's the fast lane.
The name, the, the lane
is not a fast lane.
It's a passing lane.
I.
The name of that left hand lane is a
passing lane, and there are road signs
all throughout this country that say,
left hand lane is a passing lane.
That means it doesn't matter how
fast you're driving, if you're not
passing somebody, you shouldn't
be in that left hand lane.
If you're driving 90 miles an hour
in that lane and you're not passing
anybody, you're, you're violating
street signs what they tell you to do.
But everybody refers to it as the
fast lane, so it's okay to drive.
75 or 80 if I'm not passing somebody.
It's the same thing here.
By calling it a penalty, you
get people not wanting to do it.
Who wants to, to take a penalty in this?
So looking at data's not bad.
Uh, my father, Don Berry, of course, is
a, a statistician, as many of you know,
and during a particular election, he
made the reference that he turned his
TV on and off many, many times, hoping
to affect the outcome of the election.
Spending his alpha over and
over again hoping to change the
particular outcome of the trial,
which of course it didn't change it.
And looking at the data doesn't change it.
You can do this incredibly well.
Uh, we won't get into operational
bias, we won't get into all of that.
Implementing trials.
We did an episode with Anna McLaughlin
and Michelle Dery talking about
implementing adaptive trials, and
this is something we worry about
there, but this general notion is.
Spending Alpha is bad.
Distributing alpha allows you to create
better trial designs, and that's something
that, uh, this, this weird name in this
thought process prevents us from doing.
Okay, so let's, let's, let me
describe a few examples of this.
So.
What, how does it happen then?
The, and and it re there this wide belief
that when you look at the data you've
spent alpha and it's absolutely untrue.
Just like Don turning the TV on and off.
He didn't change the election.
Looking at data doesn't cost alpha.
It do, it doesn't cause you to distribute
alpha actions you take at the interim.
Are what are critical for understanding
their potential to type one error and
the potential to have to adjust the,
uh, the most straightforward example
of this is a futility analysis.
Suppose you plan afu futility
analysis in that trial at 400 I,
I'm sorry, the 400 patient trial.
You do a futility at 200.
And you write in the protocol,
no success boundary will be done.
No success stopping will be done,
but we'll do a futility analysis.
So if the treatment is doing worse than
the, the, the placebo, at that point you
might, might wanna stop the trial because
it's very unlikely to be successful.
Now you can, you can figure
out the right way to do that.
You can simulate the trial, figure out
good futility, stopping boundaries.
You look at the data.
A a, an independent stat
group looks at the data.
There's no alpha spend for that.
You don't have to adjust
your final alpha for that.
Doing futility actually decreases
the type one error of the experiment
because the only thing you do is
stop the trial for futility, and
hence, you don't claim superiority.
That can't increase the probability
that the trial demonstrates superiority.
It's the ap, it's the opposite action.
You could do a hundred interims
for futility in your trial, and
your final analysis is 0.025.
Now a whole separate topic
is by doing futility.
Should I get a higher alpha at
the end for that and binding
futility and I'm buying back alpha.
I actually am not a big believer
in that violates the likelihood
principle and other things.
So we're not gonna go there.
So we could do a hundred futility
analysis, retain our 0.025
at the end, and we don't have to
adjust it in any way because no
action we took during the trial.
Change the probability of of,
of making a type one error.
That's critical.
Now that we write down exactly
what we're gonna do at interims.
Because the action you could take
might affect type one error, and it's
clear to lay that out prospectively.
So the experiment is completely
prospectively set up.
If the predictive probability of
trial success at 400 drops below 5%
at 200, we're gonna stop for futility.
Otherwise, we're gonna go to
300 and do a success interim.
Then we'll go to 400.
If that's not successful, we would
have to adjust at 300 because
we're taking a shot on goal.
We might wave a flag and say the trial
that the treatment was successful,
sure we would adjust alpha for that.
Again, not bad, you haven't
lost any alpha in the trial.
It doesn't squeeze away, um,
that, that you pay a tax to it.
Now there are pharmaceutical companies
out there, big ones that when they
do a futility analysis, they write
in that they're going to, they're
gonna reduce their alpha by 0.001.
It, Dr.
It it?
Yes.
It drives me nuts.
It's almost disingenuous a little bit.
Because if I were a DSMB member and I saw
that, uh, okay, are you taking a success?
Look, if I see data at that
time point that you're doing
futility and it meets 0.001,
should I stop the trial?
Well, no.
Then you, then you write in that
you could stop the trial for
superiority, but by throwing 0.001
alpha.
In the wastebasket when your action
does not in increase type 1 error,
it's really sort of disingenuous and
it, it's sort of lacking understanding
of what, what alpha is in the trial.
Now, there are other things that
are done at interim analysis.
What, what could those be?
You could expand the sample size.
There's the promising zone design
that may be in that design of 400.
When I get to 200, I could say
I want to expand it to 500.
The expansion of the sample size in some
cases means you have to adjust alpha.
In other cases, not, um, by the
way, it's, it's not a good design.
I'd rather go to 400 in design.
Should I go to 500 than
making that decision at 200?
Yeah.
Other things you can do at interims,
you can do response adaptive
randomization on your experimental arms.
Maybe we have two arms in the trial
and two doses, and we're doing
response adaptive randomization
On those doses, I do an interim, I
update the randomization of those.
The, the, the extreme of that
is I might drop a dose at that.
There's no alpha spend to response.
Adaptive randomization.
No.
That, that, that's a
topic for a different day.
But if you write down, here's
the action that's gonna happen.
There's no success stopping,
we're gonna do response.
Adaptive randomization, you
don't adjust alpha for that.
It's actually, uh, it's kind of a
neat little, uh, thing that by doing
response, adaptive randomization to
the arm, you increase its sample size.
So when the data are good,
you increase it sample size.
It's the exact opposite of the group.
Sequential stopping for superiority,
where when the data are good, you stop.
You quickly stop and say, we won here.
You actually increase the
effective sample size.
So we have, uh, the Sepsis Act trial
was, uh, it conducted several years ago.
Faring was the sponsor.
You can go on to clinical trials.gov
and look this up.
It was a, it was a treatment
cell oppressant for the treatment
of sepsis, septic shock.
The trial was a phase two three
seamless trial, meaning it started
in a phase two setting where it
had three doses in a placebo.
And from 200 to 800 patients, we did
interims Every month in the trial,
it adjusted the randomization to the
three doses, and the trial could shift
to phase three and pick a single dose.
And then in phase three, it would
test only the selected dose and
it would go to 1800 patients.
Trial design's, public,
you can look it up.
More than 20 interim analyses
were planned in that trial.
It all depended on enrollment
rate in the phase two part,
the expansion into phase three.
The final alpha at the end
of phase three was 0.025.
And that was reviewed by the FDA.
It was given a special protocol
assessment, an SPA, meaning that they
dug into all the details of this.
So here's a trial that did more than
20 interim analysis, looking at the
data, and yet not a single action
in that trial would increase the
probability of a type one error and 0.025
was given at the end.
That's a demonstration that
it's the action you take.
It's not the fact that somebody looked at
the data in the adjustment of the alpha.
Now this question of do I lose
alpha if I have to distribute it?
I showed you an example of
a group sequential trial.
We can build a hot, more highly
powered trial with a smaller, average
sample size by distributing alpha.
Let's talk about a
seamless two three trial.
So we, we worked on a project where
in phase two there were two doses and
a placebo, relatively rare condition,
uh, a relatively, uh, uh, bad
condition to have a severe condition.
Rare condition.
The phase two part was
gonna enroll 30, 30, and 30.
And at that point, the trial could
stop for futility if neither dose
was doing very well, or a single one
of the two doses could be selected.
And at that point, it would
move to a one-to-one randomized
trial of 90 versus 90.
So the entire trial is 30 30, 30.
That's 90 total patients.
And then the second part of the trial,
one to one randomizes 180 patients.
So with that design, you
could run a separate phase two
and a separate phase three.
By doing that, you use
all your alpha at 0.025
on the 90 versus 90 part.
And in the expression we,
we get full alpha at that.
An alternative, you could do an
inferentially seamless trial where the
30 patients that were on the dose from in
phase two could be included at the end of
the second part as is, as are the placebo.
So rather than the 90 versus 90.025,
we could do 120 versus 120.
Now that action of picking a dose
and bringing its data forward on that
dose and not bringing forward the
data on the other dose that does.
Increase the probability of a type one
error because you can pick the better
of two arms, and so that act of keeping
those 30 patients mean you have to
adjust your final alpha at the end.
In that example, by keeping the
30 and 30 on the two arms, keeping
those 60 patients in the primary
analysis at the end, in addition to
the 180, our final alpha is 0.01693.
So it's lower than the 0.025
in, in the, in the term I
hate, but I'm gonna use it.
We've paid this penalty 0.001693.
We've dropped by almost 0.009
that we dropped by almost a third of that.
You have higher power.
You, you pay, you, you
distribute alpha in that way.
Because you're including those
30, those 60 patients, you have
higher power by including those
60 patients and doing 0.01693
than you do by analyzing
the 90 versus 90 at 0.025.
So when I'm asked, and I, I'll
be asked that question this
week, does it cost us alpha?
The, you know, the technical answer
is you have to adjust your alpha.
But your power's higher.
And so the question I don't even
think the question really is, I think
they want to know, do I lose power?
And they ask it, do I
do, do I adjust my alpha?
Do I, do I pay a penalty?
It's an example where, uh, you
want to, you want to pay that tax.
It increases your power.
It increases your power to an extent
that you could make the second
part of that trial, 60 versus 60.
So that the total sample size
on the dose you carry forward
is 90 and the placebo is 90.
That's what the original second part was.
In that case, you pay a bigger
penalty 'cause it's only 60 versus 60.
It's 0.0
1 6, 1 6.
Again, you adjust your alpha at the end.
That trial has higher power
than the 90 versus 90 by itself.
You, your trial ends up 30 on the dose.
You don't pick 90 versus 90.
You've saved 60 patients versus
running a separate 90, a separate
180, and you have higher power.
Now, now that I, I should back up.
I should be, I should be clear about
that you have higher power as long
as your two doses are both effective.
If you're taking a shot on one of those
doses in that, and it's ineffective, you
might actually have slightly lower power.
But if your, if your second dose has
at least 50% of the effect of the other
dose, you actually have higher power.
So adjusting alpha can be good.
Looking at data can be good.
It can make a more efficient
trial, a better trial design.
Uh, it can accomplish the goals
easier in those circumstances.
So
let, let, let's sort of summarize this.
Looking at data doesn't mean
you have to adjust your alpha.
It doesn't mean that at all.
The actions you take at that
interim can affect type one error.
That hence the incredibly important
role of a prospective design laying out
exactly what's done at every interim.
And following that design.
So looking at data doesn't
cause alpha adjustment, but
alpha adjustment is not bad.
It's an incredibly powerful tool
to make an adaptive design more
efficient than a one look single shot.
Look at 400 patients or look at whatever.
You can be much more efficient in
your trial designs by distributing
alpha across the different interims.
So.
Can we get rid of referring to a
superiority trial as a two-sided 5% alpha?
Let's get rid of that and let's
stop using the term penalty.
Let's use something much more positive
because it's a good thing to do as
distributing alpha, uh, uh, uh, within it.
Um, it's a much more positive
term than paying a penalty,
which nobody wants to do.
All right.
Well, I appreciate everybody
joining us here in the interim.
Until next time, uh, try to
keep all your alpha and I'll see
you next time in the interim.