Judith: Welcome to Berry's In the Interim podcast, where we explore the
cutting edge of innovative clinical trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott Berry: All right.
Welcome everybody.
Back to In the interim, I am your host, Scott Berry, and I'm joined
today by my co-host, uh, Kurt Veley.
Dr.
Kurt Veley is here, and, uh, Kurt needs no introduction, uh, as
he is a, a regular on the show.
Uh, we have a interesting topic and, and the topic I'll, I'll,
I'll introduce the topic a little bit by telling you a true story.
Uh, and Kurt is related to this story.
For those that didn't know, Kurt and I were fellow graduate
students at Carnegie Mellon.
And, um, when I was a 23-year-old, first year graduate student at
Carnegie Mellon, I, uh, called my dad.
And of course, um, the, the aspect I called him is I
called him and informed them.
Him that my wife had taken a pregnancy test and it was positive.
And uh, I don't know if many of you in the world have similar interactions with
your father in when such a thing happens,
but
Kert Viele: I think, the answer is no, Scott.
Scott Berry: I think the answer's no.
Uh, the, the question, the first thing he said after I said, hi, I said
that, uh, by the way, I did say, uh, we took the pregnancy test and the
pregnancy test says it's 99% accurate.
And I tell him this, and his first question to me was, what
was your prior probability?
Honestly, that's, that's what he asked me.
And, um, uh, I, I, my response was, and uh, and, and Kurt knows this
well, we, we were, I was a married graduate student making a stipend.
I said the test was $15.
Dad.
The prob, the prior probability was pretty high, and his next
comment was, oh, congratulations.
That's fantastic.
Uh, and he was excited, uh, for, for this whole thing.
So they, the, the point to that is if our prior probability is low.
The 99% means something different about the the, at the end of the
day, the question is, is Tammy, is my wife pregnant or not?
Um, in the, in that, based on this experiment where the prior probability
makes a huge difference, and many of you have probably seen the rare
disease examples where if you have a low prior probability of something
and you take a diagnostic test and it says positive, you're still
probably unlikely to have the disease.
Uh, so in this circumstance, if, if, if I had a 1% chance that she was
pregnant and the test was 99% effective.
Meaning if she's pregnant, 99% chance it says yes, if not 99% chance it says no.
I think they're actually more accurate than that.
But, um, if I had a 1% chance, I still, it's only 50 50.
She's pregnant in that circumstance where if I started with a
10% chance, it was over 90%.
She's positive.
The point to that is the prior probability of that changes my interpretation
of the result of the experiment.
So the topic for today is using Bayesian borrowing.
And that's a case of Bayesian borrowing of what my dad did faced
with the result of a clinical trial.
Uh, and the diagnostic test is a clinical trial.
A clinical trial is the same thing based with that information.
The prior probability was a huge part to his interpretation of, uh, of the result.
There are cases now, and the Bayesian guidance is out where we use Bayesian.
Probabilities going into the trial to interpret the result of, of the
clinical trial working with the FDA.
And today's topic is doing that.
What does it look like?
What are the characteristics?
What's the behavior?
Have we done this before?
So that's the topic for today, Kurt.
Kert Viele: All right.
Sounds like a good topic and it sounds like we're talking about, um.
So we, we've done this in a couple contexts, and it sounds like you're
gonna talk about the second one.
We've done this in context where we're only borrowing on,
you know, a clinical trial.
There are two arms, there's a control, there's novel treatments.
We've done situations where we've borrowed only on the control arm.
So you're comparing to an existing drug.
There's lots of data on the existing drug, and you're trying to bring that in.
That's one problem you might solve.
Um.
It sounds like today we're gonna talk about the, um, what we'll actually
consider the more complex version.
Kind of a newer version of we're borrowing on both arms where we have
some idea about what the treatment effect might be based on history.
And now the question is, what does our current trial have to do in order to.
You know, this is a journey from we know nothing in order to, we know the
drug works if we're partially there based on what we've already seen.
Um, what else do we need to know?
How much more evidence do we need to get over that bar?
And I know that we've seen this in the context of.
You know, completely external trials.
We've seen it in the context of, you know, people run one trial and get p equal 0.06.
And now the question is what do we do?
Um, so where, where do you wanna head from here, Scott?
Scott Berry: Yeah.
Yeah.
Uh, good.
So first of all, just to, uh, and, and by the way, this is a, this is a shout out
to anybody who, uh, listens, watches, uh, in the interim, if you have questions.
Throw 'em our way.
We're, we're, we're looking for interesting topics.
This was something that somebody, uh, contacted me and asked us, have we ever
gone to FDA using Bayesian, borrowing?
And it was a little bit like, whoa.
Yeah, this is kind of what we do.
Uh, and I was a little, almost surprised by the question, but then
it, it made a ton of sense that, that now with the Bayesian guidance out
and they were interested in the, this interaction and what does it look like?
The, the other part to this is the Bayesian guidance make, ref makes
reference to Bayesian trials where you're controlling type one error or
you're not controlling type one error.
Her and I was asked by somebody else, a brilliant researcher, actually, uh, Dr.
Bijoy Manan about, uh, he's a stroke neurologist in Calgary.
Uh, what's the difference?
Uh, that sort of thing.
So I think we can tackle all of those things today.
So maybe we should set it up with a case where this may make sense or
cases where we've done this before.
Uh, just to, just to give people an example of, of it.
Now first maybe we should set up what is done in a Frequentist standard clinical
trial is you calculate for that trial the probability of a type one error.
Um, uh, and it's only data in that trial that goes into that calculation.
And we say, if the probability of achieving the data in that trial.
Given the null is less than 2.5%,
we're gonna call it a successful trial.
By the way, in the circumstance of the pregnancy test, the assuming she's not
pregnant, the probability of it saying pregnant is 1% statistically significant.
In that case now, that that might be how we interpret it.
Now, here's a circumstance and we're, it's not uncommon that a company, a sponsor,
has run a trial and the trial is, uh,
not convincing for approval, but has some positivity to it.
Maybe it's a p value of 0.07.
Uh, some cases it's actually been a p value of 0.025,
but yet FDA is not convinced.
Regulators are not convinced.
So let's take a case where there's positive data and a a p value
of point, uh, a one-sided 0.04,
Kert Viele: And I, I think Scott, Scott, to this, we should add
another situation this happens is somebody runs a phase two.
And gets really positive results, but it's not quite there.
And their question is, Hey, haven't we already fulfilled at least
part of the bar for approval?
Same kind of idea.
We're partway there, but we're not fully there.
Scott Berry: So company A has run a trial that's not statistically significant,
but it's close, and maybe they even went to FDA and say, will you approve us?
Now, I've never been at the FDA, but I can imagine them thinking hard about
it, actually thinking the drug probably works, but it hasn't met the thing.
We can't approve it yet.
So company A is in that situation.
Do you tell them you have to go and run a trial ignoring all of
the things we know from that first trial and you have to jump a 0.025
hurdle and we, we ignore it.
Company B has no data and they have a treatment for the same disease.
They wanna run the same trial.
They're asked to run a trial with a 2.5%
type one error.
Um, are, are the companies asked to do exactly the same thing?
Because a ton of scientific sense that that first company a can utilize the
information from that trial and maybe they don't need as large of a trial
to demonstrate that the drug works.
So that's a circumstance that we're talking about today where you have prior
probability going into a new trial that the treatment's effective and you want
to use that to ana analyze the new trial.
Uh, a and it could be that first trial is a phase two trial.
It could be that first trial, uh, you know, was a biomarker trial that just
happened to show clinical benefit there.
There's a lot of ways in which that information comes about.
Another circumstance and we did a complex innovative design trial where this was a,
a public presentation of a circumstance where a company had demonstrated
benefit in two types of epilepsy, and they wanted to investigate a third.
There's a commonality in the diseases.
There's a commonality in the endpoint, the mechanism of the disease.
You're taking a treatment that's been shown effective in a related
but slightly different disease, and IT benefits, clearly benefits.
These other two, and now you're going into a third, should the standard be.
A trial in that subtype of disease 0.025
type one error, or should there be some sense of borrowing from the others
and, and for example, in this case, it was showing about a 30% reduction in
seizures in both of the other types.
If all of a sudden relatively early in this third type you're seeing a
30% reduction in seizures, you're pretty, pretty sure the drug works
because of that prior probability.
Yes, these are cases.
The third case, uh, talks a lot about it is you might have a conclusive proof
of benefit in an adult population.
There are rare circumstances where pediatrics have the same disease.
It's very rare.
It's hard to run a trial.
Do we just ignore them?
You might run a trial and borrow from the adult in some way.
So there's a lot of circumstances that you go into a phase three confirmatory trial
with prior information about the efficacy of the treatment that's in that trial.
Kert Viele: And I, you know, one thing to add to this, you know, these kind
of situations, they've come up in the past and there are often ways that
regulators have made the bar a little bit easier given this prior information.
And that may be, you know, one trial versus two.
It may be, hey.
We'll change something about the, you know, what you have to
do in your phase three trial.
What we're really talking about today is trying to formalize this process.
So the question is, if you wanna apply all the statistical rigor
that we usually apply to a trial.
You know, ordinarily this is, you gotta do 0.025
all the time.
Is there a middle ground where we could basically formalize and say,
here is how we're gonna analyze it.
We're still gonna keep all the statistical rigor, but we're gonna recognize that
you do have this, your partway there.
Scott Berry: Okay, so suppose we have this prior probability.
And we have a prior distribution about a treatment.
And, um, we're going in and we're running a new experiment now, and we're gonna
collect data in the new experiment.
And the Bayesian approach is, we're gonna use this prior in some way, and we'll talk
about different priors in that dynamic discounting and the, those aspects of it.
But we're gonna use that prior in some way with the new data.
Now explain.
Why that can inflate type one error and what type one error means and why.
It's a little bit disconnected in the, the way we're going about doing this.
Kert Viele: So what's gonna happen here is suppose that, you know, my,
my, whatever my prior evidence says is I think I have a 10% effect, 10%
beneficial effect on something when I come in and compute type one error.
What I am saying is, let's assume that the control and treatment arms
are exactly equal to each other.
Tell me
Scott Berry: In in the new experiment,
Kert Viele: In the new experiment.
Yeah.
Yep.
So we got our historical evidence on one side.
We got our current trial on the other, which I'll refer to it if there's
no effect in the current trial.
Now, this is something we don't think is true.
We have prior evidence, so we at least are less likely to think it's true,
is the more precise way to say it.
But if it is true, what's gonna happen is that prior evidence,
it's gonna bias things upward.
It's gonna say, Hey, if I didn't see an effect in the current trial, I'm
gonna average this in some way with the beneficial effect from the past.
I'm gonna.
Pull things up a little bit, I'm gonna basically say, you know,
maybe I have a two or 3% effect.
The net result of that is that little bit of bias makes you more likely
to claim efficacy if, if, if, if all of this is, if you don't have
an effect in the current trial.
So the key issue is.
What is the value of that assumption?
And this is where we get into the guidance and you know,
do we control type one error?
Do we not?
We will often do that calculation and say, you know, if this is
true, here is the type one error.
The the results of the type one error for various kinds of designs, and they're
gonna be hired two point half percent because of this bias from the past.
The argument that you make is this assumption that I'm worried about.
The null hypothesis is true.
I've already partially disproven it.
So the value of that type one error calculation is less in that case.
And that's where we get into, I know where you want to go.
Discussions of with and without type one error.
Scott Berry: Hmm.
So if, uh, and.
If you are gonna use that prior distribution, uh, to it, and,
and what might that look like is we're gonna use the prior, use the
new data and say we need a 97.5%
probability that the treatment's beneficial after the combination of the
two, that that's gonna be successful.
We may go to FDA, and that's the criterion.
Now, if we weren't using prior distributions.
That probably controls type one error at 2.5%
for using a flat prior
that that Bayesian analysis is similar to a frequentist analysis.
Uh, and so the type one error would be controlled for the new
experiment, and that would satisfy in the guidance where it says you
are controlling type one error.
Because really all we're using is the new experiment.
Uh, because we're not bringing
any prior, once we bring that prior
and we combine them together now, uh, as, as Kurt says, to get 97.5%
with the combination, the new trial doesn't have to be 97.5
by itself.
That might be 93% is enough.
Combined with the prior to say overall combined, we're 97.5.
If you evaluate only the new experiment, it might have a 7% type
one error in that circumstance,
Kert Viele: Yep.
Scott Berry: and that's where the FDA might agree.
Okay?
The data you're using is reliable, it's relevant, it's appropriate.
We agree that you can use that for approval as a combination.
Then they're approving within only this modular new experiment, the type
one errors above the traditional 2.5
and maybe it's 7% in that circumstance.
So that's kind of the difference between a Bayesian analysis that controls
type one error for the new modular experiment and one that doesn't,
Kert Viele: Yeah, and I, I think we're.
Scott Berry: about a lot.
Yep.
Kert Viele: we're, we're talking about the combination.
I, I always tend to view this if, if I were doing things from scratch
and I'm gonna do two trials, um, is that combination good enough?
And the question is, you know, I'm basically designing my new
experiment when the original, the historical data is historical.
It's in the past so I can condition on it.
Scott Berry: So let's, let's, um, and, and we'll, we'll leave as a separate topic.
Where we're borrowing only on the control arm.
Yes, that can inflate type one error.
It's a different kind of thing about the new temporal differences in the new trial.
And so we're, we're, we're talking about the circumstance where we're
going in with an informative prior about the treatment effect, and you're
having discussions with FDA about that.
Kert Viele: So I'm gonna ask you, I'm gonna derail this slightly, and
I haven't told you I'm about to ask this, but what about situations, let's
Alzheimer's sepsis, where nothing has worked in the past that potentially
should give us a negative prior?
How should people view that?
Scott Berry: Uh, I think it's appropriate and I think it's, it's been an
unspoken thing probably at the agency.
And there was a time not long ago where there were 25 straight
failed phase three trials.
Kert Viele: Which is unlucky, by the way.
They should have won one by
Scott Berry: Somebody should have.
Yes.
Uh, and that, by the way, the 26 trial, the FDA may be weary of
that, that if all of a sudden the 26 trials shows P value of 0.02,
maybe they're the lucky one.
Uh, and you can imagine a, a skeptical prior, which is
talked about in the guidance,
in that circumstance.
Uh, and maybe they even ask for a higher threshold of that, uh, circumstance.
So that's kind of a different prior going in where historically the
scenario is, is, is, has been nothing works, uh, uh, in that scenario.
Kert Viele: Yep.
Scott Berry: so you carry the burden of the disease with you as you,
as you go into the agency, which I think is quite, quite reasonable,
Kert Viele: And you don't wanna discourage research in these areas and so on.
It's just a really hard problem.
I wasn't expecting an answer for it, but we, we should be honest about it
exists in the other direction too.
Scott Berry: yep.
Okay.
So what this may look like, um, in the interaction.
So we're in a circumstance, and by the way, we get the challenges of this.
So, for example, we might get sponsors that come to us that ran a phase 3
trial, and it was a P value of 0.3.
Barely observed positive differences.
They find a subset of 30% of the population that, wow,
it's, it's nearly significant.
And they want to take that subset and ignore everything else and go
to the FDA and borrow from that subset in a new trial, and it gives
them a jumpstart for approval.
Okay.
And now they've cherry picked from the previous trial, and by
the way, they can explain why.
Of course, that's the relevant subgroup and all of that, and they want to
go to FDA and borrow from that.
And there are circumstances where you and I, they come to us.
We'd say if we were at FDA, we wouldn't accept that.
Because we believe this is largely about multiplicity, and now you
can't pull that out and create a prior that's only based on that.
If you wanna use a prior, based on the entire trial, that might be something
to discuss and, and have that.
But to self-select data and only use that data going to the FDA,
you're probably gonna get a no.
And by the way, historically that's happened to companies and they come
back and say, FDA doesn't do Bayes
Kert Viele: Yeah, that's always frustrating.
Scott Berry: right, right.
And it's had nothing to do with Bayes It's, you're, you're misusing it in ways
that we, as Bayesians would say, no, we, we, I would not accept that circumstance.
So there's a huge part of the data.
You're going into the FDA.
What are you using?
Uh, has it been, you know, self-selected?
Is it all of the data out there?
Is it relevant for the Nutra?
So there's a huge part of this, of the science.
Science is hard and, and you go in.
So if we go in with a circumstance where we believe we're bringing
to them a, a relevant prior.
Based on the previous information that becomes part of the
discussion with the FDA.
Is this reasonable,
Kert Viele: Well, and it becomes a, it becomes a major part of
the discussion with the FDA.
I think we spend as much time on data as we do on the methodology at this point.
It certainly is the main reason.
As you said, FDA says no.
Um, a lot of times we're not told where this subset came from.
It's, Hey, I want to do a trial in this population, and this is the
data I have on this population.
And, you know, at some level that's a red flag at this point
that I need to ask the question.
You know, you said this is the data we have on this population.
What's the data?
What's the, what's the.
What's the world actually look like so that I can assess whether
this makes, whether this, you know, is it cherry picked?
How much is it cherry picked?
All of these things come into play.
Scott Berry: Hmm.
Yeah.
The, the dog that didn't bark.
What are you not telling us?
Uh, because we get told the rosy part of it.
And what are you not telling us and what, what data are we not using?
Um.
And so for example, in the case of the complex innovative design program
where we've got these two other indications and they don't have a
third indication that failed, for example, if they did, you'd want to
include that in a hierarchical model.
Or there's a, a large phase three and we're using the entire estimate that comes
from that to borrow for the new trial.
And we go to the agency and we're presenting them
how we're doing the borrow.
So I'll give you one example of a case that the device was approved.
The Watchman device is a left atrial appendage device that's
been approved by CDRH, uh, at FDA.
And they or originally ran a trial that was inconclusive and the name
of that trial, the original trial, uh, I'm gonna get this wrong.
Um, the original trial was.
Uh, protect af.
And then they ran a second confirmatory trial called Prevail.
And the Prevail trial borrowed where we discounted the first
trial by 50% in a static way.
We'll just discount it 50%.
And by the way, this, this went through FDA, uh, went
through Bram Zuckerman's group.
Uh, Bram is.
Is a brilliant scientist and understands what it means to borrow
the implications for the new trial.
More than once, these types of things have gone through the cardiovascular device
division at the FDA absolutely brilliant scientist who was involved in this
particular case in, in that borrowing.
So we go in and we propose that level of borrowing and show.
What happens in the new trial?
The most important thing to show, and I had a I, I had a client discussion
yesterday with the same thing is what example trials of the new trial, when
is it successful with the borrowing?
And then how does that compare to if we didn't borrow?
So for example, if we're, we, we've got a.
Uh, time to event analysis where we're borrowing data.
If we don't borrow, we need a hazard ratio of 0.7
or better to win with borrowing.
We show that if it's between 0.7
and 0.77
with the borrowing from the historical data that showed 0.7,
those are still successful.
Kert Viele: The field goal.
Scott Berry: That, yes, the Bayesian field goal.
And so you show them the, I think the most of we, sometimes we, we
revert to operating characteristics.
If the truth is 0.7,
our trial's 93% powered.
We, we, we would tell the FDA or we would say, if the device doesn't work,
the new trial has an 8% type one error.
And we, you, we talked about what that means is that additional trials are being
demonstrated with the final analysis at the end being Bayesian as a combination
of the trials being successful.
But I, I think that's hard for the clinicians at the FDA That's hard for a,
a proxy for Bram Zuckerman to understand.
Okay, what am I agreeing to?
In that show 'em a example of a final trial that the final
trial shows an estimate of 0.76.
The Bayesian analysis set gives amine of 0.72
and a confidence interval that's less than one, that this would now be approvable.
They can see exactly what they're agreeing to that.
So that's one of the things we show them, is depending on the result, we
also wanna show them if the new trial shows a hazard ratio of one, it doesn't
say, oh, the the treatment works.
They would not, they, they would say, wait a minute, wait a minute.
You know, we, we don't agree to that.
We don't want this Bayesian thing to flip that.
So what is it flipping?
Kert Viele: So we've had a couple situations, I can think of one in
oncology with rare cancers where this was like your epilepsy example where
we're trying to extend the label.
And so we're borrowing across indications.
And one of the things the FDA was particularly concerned about.
Is, remember, these are all rare.
We may not enroll very many.
They never wanted a situation, and I should say ultra rare.
I mean, we're gonna get a handful, but they didn't want us coming in
and saying, you know, we got zero responses in four patients, and
we're gonna say you should approve.
On the basis of the prior, we need some, we need it to work
somewhere, which makes perfect sense.
We need some evidence that it's doing something.
And so we've had those discussions.
You know, show us the minimum that we've had.
I remember one a while back it was, we were borrowing adult data
to pediatrics and again, we were gonna get small sample sizes.
And the FDA actually said explicitly.
We would be willing if you got 15 responses out of, and I don't remember
how many, uh, kids we were gonna be able to enroll, but basically if you
can get 15 out of 25, we'll be happy to accept that, that we think that's enough
given the adult data and everything.
And so we basically back solved, you know, what would this do, and
then showed that this had reasonable operating characteristics as well.
But that was the way the interaction went in terms of.
Here's the data that we think would be convincing to us.
Scott Berry: What, and and so through this interaction, we're
showing them the new power.
We're showing them the type one era of this new.
Modular experiment with the Bayesian analysis that happens at the end.
We're showing them example trials.
We've had people at the FDA say, okay, your new experiment has a
8% type one error rather than 2.5.
In that, what if we just say you can run a a, a new phase three
trial without Bayesian borrowing and you get an alpha of 0.08.
That, that, that we go to, that, you know, in, in many circumstances, that
gives the same success and failure of approval that the Bayesian analysis
creates at the end of the day.
Kert Viele: Okay, you're asking me a trick question.
So this, this one gets a little nuanced here.
Um, and I think the, the answer does depend on are you borrowing only control?
Are you borrowing control and treatment?
Are you borrowing dynamically or are you borrowing statically?
Um, I'm gonna, I'm gonna certainly focus on the dynamic borrowing,
which I think we prefer to do.
Um, and I'll have to talk about what that is 'cause I don't think
we've really brought that up.
The dynamic borrowing the idea is if you're walking in with prior
information from a previous trial, suppose your current trial, it's
ongoing and it just looks different.
So the rates are different than what you saw in history, the treatment
effects, what you saw in history.
The notion is, you know, should I continue to believe this prior or should I go, Hey,
something different is going on there.
And if you got into the math, what you're really saying is you have a
prior where maybe the current trial matches history and maybe it doesn't.
Is there, there's an aspect of that to your prior.
So what happens when things don't match?
You would borrow less.
So you start to give less weight to history, and this
is a mitigation strategy.
When you have mismatch, you borrow less, you're less prone
to make these false conclusions.
Doing that is far better than just increasing the alpha because increasing
the alpha, basically you're saying, I'm always willing to take this extra error
regardless of what happens and you don't have any of these mitigation strategies.
So I think it's really important that you go, you know, I, I'm, I'm changing
the alpha because I have evidence that it works and I should be able to
discount that evidence appropriately.
If the new data says it's not right.
Scott Berry: The, the other nice thing that come, well, two parts
that comes from the Bayesian part that's a little bit different
than just saying, oh, you get 0.08
alpha, is that there was an explicit incorporation of the strength of the
previous data that generated that eight.
Kert Viele: Yep.
Scott Berry: what you got in the new experiment that gave
you a final answer of 97.5%
or 99, whatever that is.
So it's not arbitrarily, oh, we think you get 8%, you get 11%.
It's the combined data that tells you what that implied value is.
But the other part is if you just take 8%, one sided, your
estimate at the end of the day.
Is not utilizing that information.
The confidence interval doesn't match the
result of the trial, uh, in that, and the estimate can be as important
as the final result in the trial, which is a combined estimate.
So there, there are, I, I, I believe there's lots of advantages of
using the Bayesian approach to, to generate the final combined analysis
of all of the data that we know.
Jumping a regulatory hurdle.
Kert Viele: And a lot of times that can be even borrowing good information.
You may actually pull the point estimates down.
We did that actually, it was the same rare cancer trial.
It had nine groups in it.
One of the early data sets it, it's a blockbuster success,
but it estimated a 60% rate.
And you know, the, the control was estimated to be about 10.
So this is a huge treatment effect.
But when we did all the borrowing.
It didn't raise the 60% our prior.
It certainly moved it.
It.
It was positive information, but it only estimated it to be about 48.
'cause it pulled it back and said, look, we don't think it's that good.
There were also aspects, it was the highest group among many, and we've
talked about that in other podcasts.
But anyway, what was really cool about that is they continued to
do follow up and they continue to enroll a few more patients.
And lo and behold, the final data came out pretty close to 48%.
So I mean, it, it, it will pull things down.
Scott Berry: Okay.
So, uh, uh, we've touched on a number of these issues.
Uh, I'll give you an example.
Of a case where this happened.
Now, we, we have a number of examples of trials running right now at
the FDA where there's been agreed upon Bayesian analysis of this.
We we're going to them, uh, I'll give you an example of one
where there was this agreement.
The trial ran, and actually the data's been publicly disclosed,
um, uh, because it went through an advisory committee meeting.
So it, the, the treatment is called REBYOTAÂ® And it was a Ferring
company, Rebiotix And the treatment is for c Diff infection, and this
REBYOTAÂ® is now approved by Seaberg.
In it and they had run a phase two trial.
All of this information is public.
Uh, it's a wonderful thing about the advisory committee meetings
is that a huge amount of this is made public and we're doing less
advisory committee meetings now.
Now, and this is sort of negative of that, a whole separate topic.
Um, but so the information's available.
In the public register on this.
So they ran a phase 2 trial, and this is a treatment for
recurrent c difficile infection.
This happens when individuals might be treating for cancer
or a different disease.
Um, they take antibiotics.
It kind of destroys the gut biome.
And c diff, which might live in all of us, takes over and, and now flourishes.
Is, and it causes it, it causes debilitating GI issues.
Diarrhea can actually be fatal, uh, in it.
Now what they might do is give you another course of antibiotics
and hope it sort of cures it.
The recurrent aspects is, it doesn't.
And the, and it's, this is a horrible, a horrible disease.
The treatment is actually a fantastic treatment.
It's a fecal matter transplant.
They, they take feces from a healthy gut biome and through an enema.
They, they give it to the individual, and the hope is that
this restores the GI tracted healthy.
Probiotics, healthy gut biome to to fight off the c diff infection.
So they ran a phase 2 trial that was not statistically significant,
showed a benefit of about 15% in preventing recurrent disease.
This is a fantastic trial where there's only one endpoint.
Uh, sure Mortality's looked at, but the only endpoint is
does the disease come back?
Yeah.
You know there, there isn't a vast scale, there's nothing.
There's one endpoint in the disease.
Does the disease come back and the proportion of time it comes back,
and if it doesn't come back within two months, you're in great shape.
It's very unlikely to come back.
It's almost a new circumstance of it, but if it does come back,
then you're dealing with this.
This case, they ran a phase two trial and showed benefit.
They're now running a phase three trial is a rare disease and.
These.
Interestingly, it became very challenging to enroll this trial
because there were outfits using FMT Fecal Matter, transplant Open Biome,
for example, that patients could get, and this was not being shut down.
The FDA could have shut down all these things, but they decided to,
at their discretion, not, uh, but it never had never been approved before.
So there's a circumstance where there's a belief this is beneficial.
This phase two trial is showing now they're trying to run a phase three
trial and challenging to get to 0.025
in this circumstance where I think largely the scientific
community believes this works.
So they agreed to borrow from the phase 2 trial.
In the phase 3 trial, and have a combined analysis of those two trials meeting 97.5
with the combination of the two trials, the phase three trial
read out, and it by itself, if you analyze only the phase three trial
would've been about 93% probability.
The treatment works in and of itself.
That experiment didn't jump.
The 0.025
hurdle combined with the phase 2 trial in the pre-specified analysis,
pre-specified to the phase 3 trial, but not phase 2 Um, said 99.1%
probability of the treatment is beneficial when you combine together the two trials.
In the way that we worked with the FDA, um, uh, in creating this combined
prior with the phase three was 99.1,
which had jumped the 97.5
hurdle.
That's the standard for clinical development.
It given the rareness of the disease.
It's not a two trial hurdle or the hurdle.
We, we don't know whether we're in the one trial, two trial world of FDA in this
circumstance, but given the rare disease, it very much fit into the, the one trial
level of, of, of ev, of evidence there.
This went to panel.
Uh, public disclosure.
And of course, the, the interesting thing is the statisticians were a bit of the
hurdle as to whether this is appropriate.
The statisticians split their vote one to one for it, but it, it was voted 13
to four that it demonstrated efficacy.
Uh, in this circumstance, the treatment has been approved
by FDA, it's on the market.
And an interesting thing about it is that the FDA label for REBYOTAÂ® Gives the 99.1
that came from the Bayesian analysis.
There are no P values in the FDA label, but it has Bayesian posterior
probabilities that it describes the combination of phase two and phase
three as the estimate of efficacy that comes out of that, the treatment
effect, relative difference estimate.
Um, is based on that.
And interestingly, at the FDA advisory committee meeting, the FDA statistician
presented the posterior distribution based on the combined, showed a picture
of the posterior and talked about the probability it's above zero, the
probability it's at least 5% better, the probability it's at least 7% better.
And this, this was.
Based on FDA's analysis, the best summary of efficacy we have to present to you.
Advisory committee meeting.
And that picture of, of that stood on the screen for, uh, uh, a long time
as here's what we're trying to, to.
To make a decision on today, which was, which was sort of cool, so you can go
forward and they do make reference in the new draft guidance of Rebi OTA as
a case study of that, and the data's publicly disclosed for your scrutiny.
Kert Viele: And the publications available.
Scott Berry: And the publications available.
It spends more time talking
about, uh, other aspects of it than the, the pure Bayesian part.
Uh, the adaptive design reports in there, the Bayesian, uh, uh, models in there.
All of the pieces available, Uh,
for it, which.
Kert Viele: a lot of our best stuff is in the supplementary material, right?
Scott Berry: Yes.
Yeah, exactly.
I would've loved to have stood up and, and explained to them, uh,
uh, all of the Bayesian machinery,
but, uh, that was not deemed to be the right thing to
do, uh, in the circumstance.
Uh, by, by the way, we're, uh, it'll be another podcast that we talk
about, that particular example, a lot of, of, uh, Anna McLaughlin, Joe
Marion here at Berry Consultants.
Uh, Lindy Banky at, at rebi was the, the, the clinical lead of this and the one who
spoke to the advisory committee meeting about she's fantastic, uh, sort of thing.
So lots of people involved in, in that particular case, um, uh, with it.
Okay, so have we, we missed any parts to what it looks like
going to FDA good circumstances.
Circumstances by the way that we say no to a client.
No, we're not, we're not gonna go present that we, we would
say no if we were at the FDA.
We're not gonna present a Bayesian analysis of this.
Uh, but we've done this probably couple dozen times.
Um, over the years, probably more than that.
We've been told no by the FDA by the way.
Uh, we've presented what we thought was a very reasonable
scientific argument for that.
We've been told no, we've been told yes.
We've been told, uh, can you change it in the following ways, uh, in that, so
interaction is not uncommon in those,
Kert Viele: We, we've been told to do it when we didn't propose it.
Scott Berry: Yeah.
Yeah.
Uh, the rebi actually circumstances was a suggestion of the FDA,
uh, Bram has made that suggestions.
Other, other, uh, uh, scientists at the FDA had have suggested,
oh, you should explore Bayesian methods, um, in the circumstance.
Of course, going back to the original introduction of this,
the, the home pregnancy test, uh, is now one of your colleagues
that was, uh, that was Nick Berry,
who is now a 34-year-old scientist here at Berry Co Consultants.
So, uh, the
Kert Viele: Thank you for making me feel old, Scott.
Appreciate that.
Scott Berry: I know, I know.
Uh, as you, you, you babysat the youngster,
Kert Viele: so so we should, I think the new podcast, we should always start with
a story of view of Brad from grad school.
I
Scott Berry: Yeah, we have some of you too.
So, uh, turnabout is, is fair play,
uh, uh, in all of this.
Yes.
Alright, well we hope you enjoyed this little, uh, uh, Bayesian interlude
here, which we hope is a more common thing with the FDA guidance.
And until next time, we will be here.
In the interim.
Kert Viele: Thank you, Scott.