Subscribe
Share
Share
Embed
In this episode of "In the Interim…", Dr. Scott Berry and Dr. Kert Viele examine Bayesian borrowing in Phase 3 clinical trials, focusing on statistical handling of prior information and real-world FDA interactions. The episode opens with an analogy, comparing prior probability in Bayesian analysis to interpreting a home pregnancy test, succinctly demonstrating the effect of prior knowledge on trial interpretation. The discussion addresses technical challenges—how borrowing inflates Type I errors and why this is addressed differently under Bayesian operating characteristics. Concrete examples include dynamic versus static borrowing approaches, and formal integration of prior evidence in regulatory submissions. Case studies center on the WATCHMAN device (PROTECT AF and PREVAIL trials) and REBYOTA, illustrating FDA engagement, relevant trial design tactics, and published outcomes. The episode also critiques common pitfalls such as selective data use and improper prior construction, emphasizing the FDA’s focus on comprehensive and unbiased historical sources.
Key Highlights
Key Highlights
- Pregnancy test analogy used to clarify prior probability in trial interpretation.
- Bayesian borrowing’s effects on Type I error and statistical thresholds.
- Case studies: WATCHMAN device (PROTECT AF, PREVAIL) and REBYOTA approvals.
- Dynamic borrowing versus static borrowing strategies in regulatory settings.
- Risks of cherry-picking and importance of unbiased, relevant prior data.
- FDA guidance and review procedures for Bayesian trials.
For more, visit us at https://www.berryconsultants.com/
Creators and Guests
What is In the Interim...?
A podcast on statistical science and clinical trials.
Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.
Judith: Welcome to Berry's In the
Interim podcast, where we explore the
cutting edge of innovative clinical
trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott Berry: All right.
Welcome everybody.
Back to In the interim, I am your
host, Scott Berry, and I'm joined
today by my co-host, uh, Kurt Veley.
Dr.
Kurt Veley is here, and, uh, Kurt
needs no introduction, uh, as
he is a, a regular on the show.
Uh, we have a interesting topic
and, and the topic I'll, I'll,
I'll introduce the topic a little
bit by telling you a true story.
Uh, and Kurt is related to this story.
For those that didn't know,
Kurt and I were fellow graduate
students at Carnegie Mellon.
And, um, when I was a 23-year-old,
first year graduate student at
Carnegie Mellon, I, uh, called my dad.
And of course, um, the, the
aspect I called him is I
called him and informed them.
Him that my wife had taken a
pregnancy test and it was positive.
And uh, I don't know if many of you in
the world have similar interactions with
your father in when such a thing happens,
but
Kert Viele: I think,
the answer is no, Scott.
Scott Berry: I think the answer's no.
Uh, the, the question, the first
thing he said after I said, hi, I said
that, uh, by the way, I did say, uh,
we took the pregnancy test and the
pregnancy test says it's 99% accurate.
And I tell him this, and his
first question to me was, what
was your prior probability?
Honestly, that's, that's what he asked me.
And, um, uh, I, I, my response was,
and uh, and, and Kurt knows this
well, we, we were, I was a married
graduate student making a stipend.
I said the test was $15.
Dad.
The prob, the prior probability
was pretty high, and his next
comment was, oh, congratulations.
That's fantastic.
Uh, and he was excited, uh,
for, for this whole thing.
So they, the, the point to that is
if our prior probability is low.
The 99% means something different
about the the, at the end of the
day, the question is, is Tammy,
is my wife pregnant or not?
Um, in the, in that, based on this
experiment where the prior probability
makes a huge difference, and many
of you have probably seen the rare
disease examples where if you have
a low prior probability of something
and you take a diagnostic test
and it says positive, you're still
probably unlikely to have the disease.
Uh, so in this circumstance, if, if,
if I had a 1% chance that she was
pregnant and the test was 99% effective.
Meaning if she's pregnant, 99% chance it
says yes, if not 99% chance it says no.
I think they're actually
more accurate than that.
But, um, if I had a 1% chance,
I still, it's only 50 50.
She's pregnant in that circumstance
where if I started with a
10% chance, it was over 90%.
She's positive.
The point to that is the prior probability
of that changes my interpretation
of the result of the experiment.
So the topic for today is
using Bayesian borrowing.
And that's a case of Bayesian
borrowing of what my dad did faced
with the result of a clinical trial.
Uh, and the diagnostic
test is a clinical trial.
A clinical trial is the same
thing based with that information.
The prior probability was a huge part to
his interpretation of, uh, of the result.
There are cases now, and the Bayesian
guidance is out where we use Bayesian.
Probabilities going into the trial
to interpret the result of, of the
clinical trial working with the FDA.
And today's topic is doing that.
What does it look like?
What are the characteristics?
What's the behavior?
Have we done this before?
So that's the topic for today, Kurt.
Kert Viele: All right.
Sounds like a good topic and it
sounds like we're talking about, um.
So we, we've done this in a couple
contexts, and it sounds like you're
gonna talk about the second one.
We've done this in context
where we're only borrowing on,
you know, a clinical trial.
There are two arms, there's a
control, there's novel treatments.
We've done situations where we've
borrowed only on the control arm.
So you're comparing to an existing drug.
There's lots of data on the existing
drug, and you're trying to bring that in.
That's one problem you might solve.
Um.
It sounds like today we're gonna talk
about the, um, what we'll actually
consider the more complex version.
Kind of a newer version of we're
borrowing on both arms where we have
some idea about what the treatment
effect might be based on history.
And now the question is, what does our
current trial have to do in order to.
You know, this is a journey from we
know nothing in order to, we know the
drug works if we're partially there
based on what we've already seen.
Um, what else do we need to know?
How much more evidence do we
need to get over that bar?
And I know that we've seen
this in the context of.
You know, completely external trials.
We've seen it in the context of, you know,
people run one trial and get p equal 0.06.
And now the question is what do we do?
Um, so where, where do you
wanna head from here, Scott?
Scott Berry: Yeah.
Yeah.
Uh, good.
So first of all, just to, uh, and, and by
the way, this is a, this is a shout out
to anybody who, uh, listens, watches, uh,
in the interim, if you have questions.
Throw 'em our way.
We're, we're, we're looking
for interesting topics.
This was something that somebody, uh,
contacted me and asked us, have we ever
gone to FDA using Bayesian, borrowing?
And it was a little bit like, whoa.
Yeah, this is kind of what we do.
Uh, and I was a little, almost
surprised by the question, but then
it, it made a ton of sense that, that
now with the Bayesian guidance out
and they were interested in the, this
interaction and what does it look like?
The, the other part to this is the
Bayesian guidance make, ref makes
reference to Bayesian trials where
you're controlling type one error or
you're not controlling type one error.
Her and I was asked by somebody else, a
brilliant researcher, actually, uh, Dr.
Bijoy Manan about, uh, he's a
stroke neurologist in Calgary.
Uh, what's the difference?
Uh, that sort of thing.
So I think we can tackle
all of those things today.
So maybe we should set it up with
a case where this may make sense or
cases where we've done this before.
Uh, just to, just to give
people an example of, of it.
Now first maybe we should set up what is
done in a Frequentist standard clinical
trial is you calculate for that trial
the probability of a type one error.
Um, uh, and it's only data in that
trial that goes into that calculation.
And we say, if the probability of
achieving the data in that trial.
Given the null is less than 2.5%,
we're gonna call it a successful trial.
By the way, in the circumstance of the
pregnancy test, the assuming she's not
pregnant, the probability of it saying
pregnant is 1% statistically significant.
In that case now, that that
might be how we interpret it.
Now, here's a circumstance and we're, it's
not uncommon that a company, a sponsor,
has run a trial and the trial is, uh,
not convincing for approval,
but has some positivity to it.
Maybe it's a p value of 0.07.
Uh, some cases it's actually
been a p value of 0.025,
but yet FDA is not convinced.
Regulators are not convinced.
So let's take a case where there's
positive data and a a p value
of point, uh, a one-sided 0.04,
Kert Viele: And I, I think Scott,
Scott, to this, we should add
another situation this happens
is somebody runs a phase two.
And gets really positive results,
but it's not quite there.
And their question is, Hey, haven't
we already fulfilled at least
part of the bar for approval?
Same kind of idea.
We're partway there, but
we're not fully there.
Scott Berry: So company A has run a trial
that's not statistically significant,
but it's close, and maybe they even went
to FDA and say, will you approve us?
Now, I've never been at the FDA, but
I can imagine them thinking hard about
it, actually thinking the drug probably
works, but it hasn't met the thing.
We can't approve it yet.
So company A is in that situation.
Do you tell them you have to go
and run a trial ignoring all of
the things we know from that first
trial and you have to jump a 0.025
hurdle and we, we ignore it.
Company B has no data and they have
a treatment for the same disease.
They wanna run the same trial.
They're asked to run a trial with a 2.5%
type one error.
Um, are, are the companies asked
to do exactly the same thing?
Because a ton of scientific sense that
that first company a can utilize the
information from that trial and maybe
they don't need as large of a trial
to demonstrate that the drug works.
So that's a circumstance that we're
talking about today where you have prior
probability going into a new trial that
the treatment's effective and you want
to use that to ana analyze the new trial.
Uh, a and it could be that first
trial is a phase two trial.
It could be that first trial, uh, you
know, was a biomarker trial that just
happened to show clinical benefit there.
There's a lot of ways in which
that information comes about.
Another circumstance and we did a complex
innovative design trial where this was a,
a public presentation of a circumstance
where a company had demonstrated
benefit in two types of epilepsy, and
they wanted to investigate a third.
There's a commonality in the diseases.
There's a commonality in the endpoint,
the mechanism of the disease.
You're taking a treatment that's
been shown effective in a related
but slightly different disease,
and IT benefits, clearly benefits.
These other two, and now you're going
into a third, should the standard be.
A trial in that subtype of disease 0.025
type one error, or should there be
some sense of borrowing from the others
and, and for example, in this case, it
was showing about a 30% reduction in
seizures in both of the other types.
If all of a sudden relatively early
in this third type you're seeing a
30% reduction in seizures, you're
pretty, pretty sure the drug works
because of that prior probability.
Yes, these are cases.
The third case, uh, talks a lot about
it is you might have a conclusive proof
of benefit in an adult population.
There are rare circumstances where
pediatrics have the same disease.
It's very rare.
It's hard to run a trial.
Do we just ignore them?
You might run a trial and borrow
from the adult in some way.
So there's a lot of circumstances that you
go into a phase three confirmatory trial
with prior information about the efficacy
of the treatment that's in that trial.
Kert Viele: And I, you know, one thing
to add to this, you know, these kind
of situations, they've come up in the
past and there are often ways that
regulators have made the bar a little
bit easier given this prior information.
And that may be, you know,
one trial versus two.
It may be, hey.
We'll change something about
the, you know, what you have to
do in your phase three trial.
What we're really talking about today
is trying to formalize this process.
So the question is, if you wanna
apply all the statistical rigor
that we usually apply to a trial.
You know, ordinarily this
is, you gotta do 0.025
all the time.
Is there a middle ground where we
could basically formalize and say,
here is how we're gonna analyze it.
We're still gonna keep all the statistical
rigor, but we're gonna recognize that
you do have this, your partway there.
Scott Berry: Okay, so suppose
we have this prior probability.
And we have a prior
distribution about a treatment.
And, um, we're going in and we're running
a new experiment now, and we're gonna
collect data in the new experiment.
And the Bayesian approach is, we're gonna
use this prior in some way, and we'll talk
about different priors in that dynamic
discounting and the, those aspects of it.
But we're gonna use that prior
in some way with the new data.
Now explain.
Why that can inflate type one error
and what type one error means and why.
It's a little bit disconnected in the,
the way we're going about doing this.
Kert Viele: So what's gonna happen
here is suppose that, you know, my,
my, whatever my prior evidence says
is I think I have a 10% effect, 10%
beneficial effect on something when
I come in and compute type one error.
What I am saying is, let's assume
that the control and treatment arms
are exactly equal to each other.
Tell me
Scott Berry: In in the new experiment,
Kert Viele: In the new experiment.
Yeah.
Yep.
So we got our historical
evidence on one side.
We got our current trial on the other,
which I'll refer to it if there's
no effect in the current trial.
Now, this is something
we don't think is true.
We have prior evidence, so we at least
are less likely to think it's true,
is the more precise way to say it.
But if it is true, what's gonna
happen is that prior evidence,
it's gonna bias things upward.
It's gonna say, Hey, if I didn't see
an effect in the current trial, I'm
gonna average this in some way with
the beneficial effect from the past.
I'm gonna.
Pull things up a little bit, I'm
gonna basically say, you know,
maybe I have a two or 3% effect.
The net result of that is that little
bit of bias makes you more likely
to claim efficacy if, if, if, if
all of this is, if you don't have
an effect in the current trial.
So the key issue is.
What is the value of that assumption?
And this is where we get into
the guidance and you know,
do we control type one error?
Do we not?
We will often do that calculation
and say, you know, if this is
true, here is the type one error.
The the results of the type one error
for various kinds of designs, and they're
gonna be hired two point half percent
because of this bias from the past.
The argument that you make is this
assumption that I'm worried about.
The null hypothesis is true.
I've already partially disproven it.
So the value of that type one error
calculation is less in that case.
And that's where we get into,
I know where you want to go.
Discussions of with and
without type one error.
Scott Berry: Hmm.
So if, uh, and.
If you are gonna use that prior
distribution, uh, to it, and,
and what might that look like is
we're gonna use the prior, use the
new data and say we need a 97.5%
probability that the treatment's
beneficial after the combination of the
two, that that's gonna be successful.
We may go to FDA, and
that's the criterion.
Now, if we weren't using
prior distributions.
That probably controls
type one error at 2.5%
for using a flat prior
that that Bayesian analysis is
similar to a frequentist analysis.
Uh, and so the type one error
would be controlled for the new
experiment, and that would satisfy
in the guidance where it says you
are controlling type one error.
Because really all we're
using is the new experiment.
Uh, because we're not bringing
any prior, once we bring that prior
and we combine them together now,
uh, as, as Kurt says, to get 97.5%
with the combination, the new
trial doesn't have to be 97.5
by itself.
That might be 93% is enough.
Combined with the prior to say
overall combined, we're 97.5.
If you evaluate only the new
experiment, it might have a 7% type
one error in that circumstance,
Kert Viele: Yep.
Scott Berry: and that's
where the FDA might agree.
Okay?
The data you're using is reliable,
it's relevant, it's appropriate.
We agree that you can use that
for approval as a combination.
Then they're approving within only
this modular new experiment, the type
one errors above the traditional 2.5
and maybe it's 7% in that circumstance.
So that's kind of the difference between
a Bayesian analysis that controls
type one error for the new modular
experiment and one that doesn't,
Kert Viele: Yeah, and I, I think we're.
Scott Berry: about a lot.
Yep.
Kert Viele: we're, we're
talking about the combination.
I, I always tend to view this if,
if I were doing things from scratch
and I'm gonna do two trials, um,
is that combination good enough?
And the question is, you know,
I'm basically designing my new
experiment when the original, the
historical data is historical.
It's in the past so I can condition on it.
Scott Berry: So let's, let's, um, and, and
we'll, we'll leave as a separate topic.
Where we're borrowing
only on the control arm.
Yes, that can inflate type one error.
It's a different kind of thing about the
new temporal differences in the new trial.
And so we're, we're, we're talking
about the circumstance where we're
going in with an informative prior
about the treatment effect, and you're
having discussions with FDA about that.
Kert Viele: So I'm gonna ask you,
I'm gonna derail this slightly, and
I haven't told you I'm about to ask
this, but what about situations, let's
Alzheimer's sepsis, where nothing has
worked in the past that potentially
should give us a negative prior?
How should people view that?
Scott Berry: Uh, I think it's appropriate
and I think it's, it's been an
unspoken thing probably at the agency.
And there was a time not long
ago where there were 25 straight
failed phase three trials.
Kert Viele: Which is unlucky, by the way.
They should have won one by
Scott Berry: Somebody should have.
Yes.
Uh, and that, by the way, the 26
trial, the FDA may be weary of
that, that if all of a sudden the
26 trials shows P value of 0.02,
maybe they're the lucky one.
Uh, and you can imagine a,
a skeptical prior, which is
talked about in the guidance,
in that circumstance.
Uh, and maybe they even ask for a higher
threshold of that, uh, circumstance.
So that's kind of a different prior
going in where historically the
scenario is, is, is, has been nothing
works, uh, uh, in that scenario.
Kert Viele: Yep.
Scott Berry: so you carry the burden
of the disease with you as you,
as you go into the agency, which I
think is quite, quite reasonable,
Kert Viele: And you don't wanna discourage
research in these areas and so on.
It's just a really hard problem.
I wasn't expecting an answer for it,
but we, we should be honest about it
exists in the other direction too.
Scott Berry: yep.
Okay.
So what this may look like,
um, in the interaction.
So we're in a circumstance, and by the
way, we get the challenges of this.
So, for example, we might get sponsors
that come to us that ran a phase 3
trial, and it was a P value of 0.3.
Barely observed positive differences.
They find a subset of 30%
of the population that, wow,
it's, it's nearly significant.
And they want to take that subset
and ignore everything else and go
to the FDA and borrow from that
subset in a new trial, and it gives
them a jumpstart for approval.
Okay.
And now they've cherry picked
from the previous trial, and by
the way, they can explain why.
Of course, that's the relevant subgroup
and all of that, and they want to
go to FDA and borrow from that.
And there are circumstances
where you and I, they come to us.
We'd say if we were at FDA,
we wouldn't accept that.
Because we believe this is largely
about multiplicity, and now you
can't pull that out and create a
prior that's only based on that.
If you wanna use a prior, based on the
entire trial, that might be something
to discuss and, and have that.
But to self-select data and only
use that data going to the FDA,
you're probably gonna get a no.
And by the way, historically that's
happened to companies and they come
back and say, FDA doesn't do Bayes
Kert Viele: Yeah, that's
always frustrating.
Scott Berry: right, right.
And it's had nothing to do with Bayes
It's, you're, you're misusing it in ways
that we, as Bayesians would say, no, we,
we, I would not accept that circumstance.
So there's a huge part of the data.
You're going into the FDA.
What are you using?
Uh, has it been, you know, self-selected?
Is it all of the data out there?
Is it relevant for the Nutra?
So there's a huge part
of this, of the science.
Science is hard and, and you go in.
So if we go in with a circumstance
where we believe we're bringing
to them a, a relevant prior.
Based on the previous information
that becomes part of the
discussion with the FDA.
Is this reasonable,
Kert Viele: Well, and it becomes
a, it becomes a major part of
the discussion with the FDA.
I think we spend as much time on data as
we do on the methodology at this point.
It certainly is the main reason.
As you said, FDA says no.
Um, a lot of times we're not
told where this subset came from.
It's, Hey, I want to do a trial in
this population, and this is the
data I have on this population.
And, you know, at some level
that's a red flag at this point
that I need to ask the question.
You know, you said this is the
data we have on this population.
What's the data?
What's the, what's the.
What's the world actually look
like so that I can assess whether
this makes, whether this, you
know, is it cherry picked?
How much is it cherry picked?
All of these things come into play.
Scott Berry: Hmm.
Yeah.
The, the dog that didn't bark.
What are you not telling us?
Uh, because we get told
the rosy part of it.
And what are you not telling us and
what, what data are we not using?
Um.
And so for example, in the case of
the complex innovative design program
where we've got these two other
indications and they don't have a
third indication that failed, for
example, if they did, you'd want to
include that in a hierarchical model.
Or there's a, a large phase three and
we're using the entire estimate that comes
from that to borrow for the new trial.
And we go to the agency
and we're presenting them
how we're doing the borrow.
So I'll give you one example of a
case that the device was approved.
The Watchman device is a left
atrial appendage device that's
been approved by CDRH, uh, at FDA.
And they or originally ran a trial
that was inconclusive and the name
of that trial, the original trial,
uh, I'm gonna get this wrong.
Um, the original trial was.
Uh, protect af.
And then they ran a second
confirmatory trial called Prevail.
And the Prevail trial borrowed
where we discounted the first
trial by 50% in a static way.
We'll just discount it 50%.
And by the way, this, this
went through FDA, uh, went
through Bram Zuckerman's group.
Uh, Bram is.
Is a brilliant scientist and
understands what it means to borrow
the implications for the new trial.
More than once, these types of things have
gone through the cardiovascular device
division at the FDA absolutely brilliant
scientist who was involved in this
particular case in, in that borrowing.
So we go in and we propose that
level of borrowing and show.
What happens in the new trial?
The most important thing to show, and
I had a I, I had a client discussion
yesterday with the same thing is what
example trials of the new trial, when
is it successful with the borrowing?
And then how does that compare
to if we didn't borrow?
So for example, if we're, we, we've got a.
Uh, time to event analysis
where we're borrowing data.
If we don't borrow, we
need a hazard ratio of 0.7
or better to win with borrowing.
We show that if it's between 0.7
and 0.77
with the borrowing from the
historical data that showed 0.7,
those are still successful.
Kert Viele: The field goal.
Scott Berry: That, yes,
the Bayesian field goal.
And so you show them the, I think
the most of we, sometimes we, we
revert to operating characteristics.
If the truth is 0.7,
our trial's 93% powered.
We, we, we would tell the FDA or we
would say, if the device doesn't work,
the new trial has an 8% type one error.
And we, you, we talked about what that
means is that additional trials are being
demonstrated with the final analysis at
the end being Bayesian as a combination
of the trials being successful.
But I, I think that's hard for the
clinicians at the FDA That's hard for a,
a proxy for Bram Zuckerman to understand.
Okay, what am I agreeing to?
In that show 'em a example of
a final trial that the final
trial shows an estimate of 0.76.
The Bayesian analysis
set gives amine of 0.72
and a confidence interval that's less than
one, that this would now be approvable.
They can see exactly what
they're agreeing to that.
So that's one of the things we show
them, is depending on the result, we
also wanna show them if the new trial
shows a hazard ratio of one, it doesn't
say, oh, the the treatment works.
They would not, they, they would
say, wait a minute, wait a minute.
You know, we, we don't agree to that.
We don't want this Bayesian
thing to flip that.
So what is it flipping?
Kert Viele: So we've had a couple
situations, I can think of one in
oncology with rare cancers where this
was like your epilepsy example where
we're trying to extend the label.
And so we're borrowing across indications.
And one of the things the FDA
was particularly concerned about.
Is, remember, these are all rare.
We may not enroll very many.
They never wanted a situation,
and I should say ultra rare.
I mean, we're gonna get a handful,
but they didn't want us coming in
and saying, you know, we got zero
responses in four patients, and
we're gonna say you should approve.
On the basis of the prior, we
need some, we need it to work
somewhere, which makes perfect sense.
We need some evidence
that it's doing something.
And so we've had those discussions.
You know, show us the
minimum that we've had.
I remember one a while back it
was, we were borrowing adult data
to pediatrics and again, we were
gonna get small sample sizes.
And the FDA actually said explicitly.
We would be willing if you got 15
responses out of, and I don't remember
how many, uh, kids we were gonna be
able to enroll, but basically if you
can get 15 out of 25, we'll be happy to
accept that, that we think that's enough
given the adult data and everything.
And so we basically back solved,
you know, what would this do, and
then showed that this had reasonable
operating characteristics as well.
But that was the way the
interaction went in terms of.
Here's the data that we think
would be convincing to us.
Scott Berry: What, and and so
through this interaction, we're
showing them the new power.
We're showing them the
type one era of this new.
Modular experiment with the Bayesian
analysis that happens at the end.
We're showing them example trials.
We've had people at the FDA say,
okay, your new experiment has a
8% type one error rather than 2.5.
In that, what if we just say you
can run a a, a new phase three
trial without Bayesian borrowing
and you get an alpha of 0.08.
That, that, that we go to, that, you
know, in, in many circumstances, that
gives the same success and failure of
approval that the Bayesian analysis
creates at the end of the day.
Kert Viele: Okay, you're
asking me a trick question.
So this, this one gets
a little nuanced here.
Um, and I think the, the answer does
depend on are you borrowing only control?
Are you borrowing control and treatment?
Are you borrowing dynamically
or are you borrowing statically?
Um, I'm gonna, I'm gonna certainly
focus on the dynamic borrowing,
which I think we prefer to do.
Um, and I'll have to talk about
what that is 'cause I don't think
we've really brought that up.
The dynamic borrowing the idea is
if you're walking in with prior
information from a previous trial,
suppose your current trial, it's
ongoing and it just looks different.
So the rates are different than what
you saw in history, the treatment
effects, what you saw in history.
The notion is, you know, should I continue
to believe this prior or should I go, Hey,
something different is going on there.
And if you got into the math, what
you're really saying is you have a
prior where maybe the current trial
matches history and maybe it doesn't.
Is there, there's an aspect
of that to your prior.
So what happens when things don't match?
You would borrow less.
So you start to give less
weight to history, and this
is a mitigation strategy.
When you have mismatch, you
borrow less, you're less prone
to make these false conclusions.
Doing that is far better than just
increasing the alpha because increasing
the alpha, basically you're saying, I'm
always willing to take this extra error
regardless of what happens and you don't
have any of these mitigation strategies.
So I think it's really important that
you go, you know, I, I'm, I'm changing
the alpha because I have evidence
that it works and I should be able to
discount that evidence appropriately.
If the new data says it's not right.
Scott Berry: The, the other nice
thing that come, well, two parts
that comes from the Bayesian part
that's a little bit different
than just saying, oh, you get 0.08
alpha, is that there was an explicit
incorporation of the strength of the
previous data that generated that eight.
Kert Viele: Yep.
Scott Berry: what you got in
the new experiment that gave
you a final answer of 97.5%
or 99, whatever that is.
So it's not arbitrarily, oh, we
think you get 8%, you get 11%.
It's the combined data that tells
you what that implied value is.
But the other part is if you
just take 8%, one sided, your
estimate at the end of the day.
Is not utilizing that information.
The confidence interval doesn't match the
result of the trial, uh, in that,
and the estimate can be as important
as the final result in the trial,
which is a combined estimate.
So there, there are, I, I, I believe
there's lots of advantages of
using the Bayesian approach to, to
generate the final combined analysis
of all of the data that we know.
Jumping a regulatory hurdle.
Kert Viele: And a lot of times that
can be even borrowing good information.
You may actually pull
the point estimates down.
We did that actually, it was
the same rare cancer trial.
It had nine groups in it.
One of the early data sets it,
it's a blockbuster success,
but it estimated a 60% rate.
And you know, the, the control
was estimated to be about 10.
So this is a huge treatment effect.
But when we did all the borrowing.
It didn't raise the 60% our prior.
It certainly moved it.
It.
It was positive information, but it
only estimated it to be about 48.
'cause it pulled it back and said,
look, we don't think it's that good.
There were also aspects, it was the
highest group among many, and we've
talked about that in other podcasts.
But anyway, what was really cool
about that is they continued to
do follow up and they continue
to enroll a few more patients.
And lo and behold, the final data
came out pretty close to 48%.
So I mean, it, it, it
will pull things down.
Scott Berry: Okay.
So, uh, uh, we've touched
on a number of these issues.
Uh, I'll give you an example.
Of a case where this happened.
Now, we, we have a number of examples
of trials running right now at
the FDA where there's been agreed
upon Bayesian analysis of this.
We we're going to them, uh,
I'll give you an example of one
where there was this agreement.
The trial ran, and actually the
data's been publicly disclosed,
um, uh, because it went through
an advisory committee meeting.
So it, the, the treatment is called
REBYOTA® And it was a Ferring
company, Rebiotix And the treatment
is for c Diff infection, and this
REBYOTA® is now approved by Seaberg.
In it and they had run a phase two trial.
All of this information is public.
Uh, it's a wonderful thing about
the advisory committee meetings
is that a huge amount of this is
made public and we're doing less
advisory committee meetings now.
Now, and this is sort of negative
of that, a whole separate topic.
Um, but so the information's available.
In the public register on this.
So they ran a phase 2 trial,
and this is a treatment for
recurrent c difficile infection.
This happens when individuals
might be treating for cancer
or a different disease.
Um, they take antibiotics.
It kind of destroys the gut biome.
And c diff, which might live in all of
us, takes over and, and now flourishes.
Is, and it causes it, it
causes debilitating GI issues.
Diarrhea can actually be fatal, uh, in it.
Now what they might do is give
you another course of antibiotics
and hope it sort of cures it.
The recurrent aspects is, it doesn't.
And the, and it's, this is a
horrible, a horrible disease.
The treatment is actually
a fantastic treatment.
It's a fecal matter transplant.
They, they take feces from a healthy
gut biome and through an enema.
They, they give it to the
individual, and the hope is that
this restores the GI tracted healthy.
Probiotics, healthy gut biome to
to fight off the c diff infection.
So they ran a phase 2 trial that
was not statistically significant,
showed a benefit of about 15%
in preventing recurrent disease.
This is a fantastic trial where
there's only one endpoint.
Uh, sure Mortality's looked
at, but the only endpoint is
does the disease come back?
Yeah.
You know there, there isn't a
vast scale, there's nothing.
There's one endpoint in the disease.
Does the disease come back and the
proportion of time it comes back,
and if it doesn't come back within
two months, you're in great shape.
It's very unlikely to come back.
It's almost a new circumstance
of it, but if it does come back,
then you're dealing with this.
This case, they ran a phase
two trial and showed benefit.
They're now running a phase three
trial is a rare disease and.
These.
Interestingly, it became very
challenging to enroll this trial
because there were outfits using FMT
Fecal Matter, transplant Open Biome,
for example, that patients could get,
and this was not being shut down.
The FDA could have shut down all
these things, but they decided to,
at their discretion, not, uh, but it
never had never been approved before.
So there's a circumstance where
there's a belief this is beneficial.
This phase two trial is showing now
they're trying to run a phase three
trial and challenging to get to 0.025
in this circumstance where I
think largely the scientific
community believes this works.
So they agreed to borrow
from the phase 2 trial.
In the phase 3 trial, and have a combined
analysis of those two trials meeting 97.5
with the combination of the two
trials, the phase three trial
read out, and it by itself, if you
analyze only the phase three trial
would've been about 93% probability.
The treatment works in and of itself.
That experiment didn't jump.
The 0.025
hurdle combined with the phase 2
trial in the pre-specified analysis,
pre-specified to the phase 3 trial,
but not phase 2 Um, said 99.1%
probability of the treatment is beneficial
when you combine together the two trials.
In the way that we worked with the
FDA, um, uh, in creating this combined
prior with the phase three was 99.1,
which had jumped the 97.5
hurdle.
That's the standard for
clinical development.
It given the rareness of the disease.
It's not a two trial hurdle or the hurdle.
We, we don't know whether we're in the
one trial, two trial world of FDA in this
circumstance, but given the rare disease,
it very much fit into the, the one trial
level of, of, of ev, of evidence there.
This went to panel.
Uh, public disclosure.
And of course, the, the interesting thing
is the statisticians were a bit of the
hurdle as to whether this is appropriate.
The statisticians split their vote one
to one for it, but it, it was voted 13
to four that it demonstrated efficacy.
Uh, in this circumstance, the
treatment has been approved
by FDA, it's on the market.
And an interesting thing about it is that
the FDA label for REBYOTA® Gives the 99.1
that came from the Bayesian analysis.
There are no P values in the FDA
label, but it has Bayesian posterior
probabilities that it describes the
combination of phase two and phase
three as the estimate of efficacy
that comes out of that, the treatment
effect, relative difference estimate.
Um, is based on that.
And interestingly, at the FDA advisory
committee meeting, the FDA statistician
presented the posterior distribution
based on the combined, showed a picture
of the posterior and talked about
the probability it's above zero, the
probability it's at least 5% better,
the probability it's at least 7% better.
And this, this was.
Based on FDA's analysis, the best summary
of efficacy we have to present to you.
Advisory committee meeting.
And that picture of, of that stood on
the screen for, uh, uh, a long time
as here's what we're trying to, to.
To make a decision on today, which was,
which was sort of cool, so you can go
forward and they do make reference in
the new draft guidance of Rebi OTA as
a case study of that, and the data's
publicly disclosed for your scrutiny.
Kert Viele: And the
publications available.
Scott Berry: And the
publications available.
It spends more time talking
about, uh, other aspects of it
than the, the pure Bayesian part.
Uh, the adaptive design reports in there,
the Bayesian, uh, uh, models in there.
All of the pieces available, Uh,
for it, which.
Kert Viele: a lot of our best stuff is
in the supplementary material, right?
Scott Berry: Yes.
Yeah, exactly.
I would've loved to have stood up
and, and explained to them, uh,
uh, all of the Bayesian machinery,
but, uh, that was not deemed
to be the right thing to
do, uh, in the circumstance.
Uh, by, by the way, we're, uh, it'll
be another podcast that we talk
about, that particular example, a
lot of, of, uh, Anna McLaughlin, Joe
Marion here at Berry Consultants.
Uh, Lindy Banky at, at rebi was the, the,
the clinical lead of this and the one who
spoke to the advisory committee meeting
about she's fantastic, uh, sort of thing.
So lots of people involved in, in
that particular case, um, uh, with it.
Okay, so have we, we missed
any parts to what it looks like
going to FDA good circumstances.
Circumstances by the way
that we say no to a client.
No, we're not, we're not gonna
go present that we, we would
say no if we were at the FDA.
We're not gonna present a
Bayesian analysis of this.
Uh, but we've done this
probably couple dozen times.
Um, over the years,
probably more than that.
We've been told no by the FDA by the way.
Uh, we've presented what we
thought was a very reasonable
scientific argument for that.
We've been told no, we've been told yes.
We've been told, uh, can you change it
in the following ways, uh, in that, so
interaction is not uncommon in those,
Kert Viele: We, we've been told to
do it when we didn't propose it.
Scott Berry: Yeah.
Yeah.
Uh, the rebi actually circumstances
was a suggestion of the FDA,
uh, Bram has made that suggestions.
Other, other, uh, uh, scientists
at the FDA had have suggested,
oh, you should explore Bayesian
methods, um, in the circumstance.
Of course, going back to the
original introduction of this,
the, the home pregnancy test,
uh, is now one of your colleagues
that was, uh, that was Nick Berry,
who is now a 34-year-old scientist
here at Berry Co Consultants.
So, uh, the
Kert Viele: Thank you for
making me feel old, Scott.
Appreciate that.
Scott Berry: I know, I know.
Uh, as you, you, you
babysat the youngster,
Kert Viele: so so we should, I think the
new podcast, we should always start with
a story of view of Brad from grad school.
I
Scott Berry: Yeah, we
have some of you too.
So, uh, turnabout is, is fair play,
uh, uh, in all of this.
Yes.
Alright, well we hope you enjoyed this
little, uh, uh, Bayesian interlude
here, which we hope is a more
common thing with the FDA guidance.
And until next time, we will be here.
In the interim.
Kert Viele: Thank you, Scott.