Judith: Welcome to Berry's In the Interim podcast, where we explore the
cutting edge of innovative clinical trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott Berry: All right, welcome.
We are back.
Uh, in the interim, this is Barry Consultants podcast of all
things statistical, all things scientific with clinical trials, uh,
medical decision making, uh, and.
We have an interesting podcast today in that I don't know the topic, so
my, my co-host on in the interim, uh, Kurt Veley is here and he has a
surprise topic to talk about today.
Kurt.
Kert Viele: Hey Scott.
Um, alright, so I wanna start with a story.
I went to a conference not that long ago, and standard statistical conference.
People are talking about the data they had, all the methods
that they've been developing to.
Understand it, what they've learned from the data, and essentially all of these.
Talks.
They had a really interesting structure, so they, they get through all their
methods and everything, and at the end of every single talk, it was ended
with, if only I had gotten a chance to design the data, design the experiment,
design the data, do this in advance, everything would've been better.
I would've avoided all these problems and so on.
And so I'm listening to all these topics and when I get to the, my talk, of
course, I started it after listening to all this is I live in utopia.
And then talked about experimental design.
And so I, I get back in the car on the, on the way home, and I'm sitting here
thinking about, so am I really in the good place or am I in the bad place?
Because one thing that's always impressed me about the last 20
years in experimental design.
Is all of these methods that we're developing to understand data,
causal inference, all of these, these aspects of, of how do we make
inferences from data in front of us?
We don't use those in experimental design, so we often have an idea that when we
design a trial, we're to ignore every bit of data that's ever existed on Earth.
If I go to my doctor and I ask, why are you giving me a drug?
They're gonna report lots and lots of studies that say, this drug is good.
If I go into an experiment and say, I want to use this data,
I'm immediately told it's bad.
And that depends on if I'm borrowing information.
I got a drug that's targeted for I.
A specific mutation.
I have data on thyroid cancer.
Can I use that for lung cancer?
Generally, the answer is often no.
If I wanna borrow historical data from old clinical trials, Alzheimer's, we
have hundreds of thousands of patients that have been treated on placebos.
Can I use it?
Well, there's problems with that.
So on real world evidence, so I, I would actually define the standard debate
that we've been having for the last 15 years is essentially, is data good
or is it bad in experimental design?
So does it lead us to a good place or a bad place?
So I'd like to talk about where we are, how we got here, and
where we think we're going.
So I'm gonna leave it up to you.
I've surprised you with a topic and see what your reactions are.
Scott Berry: well, well let's, let's, let's sort of figure out the topic
and the interesting part of this.
So what, when you say is data good or bad?
Do you mean that When I design an experiment, I'm saying I want to collect
the following data and I can analyze just the data in that experiment,
which we've been doing for a long time.
Uh, frequentist, uh, rarely Bayesian, but we, we do that.
You're saying by data, is that outside the experiment?
Is there any room in that experiment?
For external,
Kert Viele: Yes.
So what?
What do we do with essentially the totality of human knowledge?
Prior to the experiment, should that enter into the experiment
in any way and wanna avoid this?
I mean, there's lots of rhetoric we could do with this.
If you want to, if you want to say bad things about this, you talk about biases
and confounding and everything else.
If you wanna say good things, you talk about totality, the evidence, but really
let's get at, you know, what is gonna lead us in good and bad directions?
How do we decide when to do this and when to not?
Scott Berry: Yeah.
So let's, let's, let's lay out why not.
Why?
I mean, why, what are the reasons people give that we shouldn't
use other data in our experiment?
Kert Viele: Obvious answer would be.
It, it could lead us in the wrong direction.
So the, uh, experiment might have been done with a different set
of patients at a different time.
There are key differences between that old data and what my current experiment is.
If I use the new data or use the old data in my new experiment.
I can get, uh, answers that are biased in certain ways.
I can draw wrong conclusions.
I can say drugs work that don't simply because of the information I'm bringing
in rather than the experiment itself.
Scott Berry: So is this, is this a frequentist issue?
Is this that we calculate the operating characteristics of the new experiment?
Type one error is only the new experiment.
And if I use any data outside of my experiment, you can
inflate type one error.
You can get bias, all these bad terms, uh, uh, in it.
Is it a frequentist problem that data outside the experiment
all of a sudden type one error means something very different?
Uh, bias means something very different, uh, that I can't
use stuff and be a frequentist.
Kert Viele: So I don't know if it's a, I think there is a
frequent DYS Bayesian divide.
I think Bayesians are trained with the idea of collect some data, update your
beliefs, collect more data, update your beliefs, and it's a natural thing to do.
I.
To bring things in, but I don't know if, if I were to say that the problem is not
being a frequentist in terms of our, do you care about long-term error rates?
I would lay the problem at the feet of, you have to have this 2.5
type one error.
In the worst case scenario.
So in effect, we're playing minimax against nature.
We have to assume that nature has deceived us for the last 25 years, and then in
that case, we shouldn't use the data.
So, uh, on the one hand, that delivers an immense amount of robustness.
On the other hand, we're reinventing the wheel.
Scott Berry: Yeah.
Uh, so every time, so let, let's, let's talk about a case where we would
use external data, and we have, uh, we've designed trials where we've used
external data and what that looks like and what could be the potential problem.
So, um, suppose.
I've got results from another trial, a phase two trial about the relative
efficacy of a treatment, and I bring it into my next experiment.
And I use that as prior knowledge.
And when I'm done with my new experiment, I combine them together.
I.
Or I bring in previous data on the control arm.
Uh, I'm comparing to an active comparator, something that's approved.
It's standard of care, and I wanna run a new experiment of my new drug.
And I want to compare to standard of care.
There are lots of trials and lots of information about standard of care.
Why do I have to run a one-to-one randomized trial standard of care to my
therapy when I know so much about it?
So I might bring in that information specifically into the trial
and e both of those notions.
Can cause issues in my new experiment essentially.
If it's wrong, I if that is different than my new experiment.
Statisticians are great at saying, well, boy, if that data
was a little bit different.
Now your new experiment.
Has a type one error or bias or, or issues that if I bring in anything from
outside the trial, it can cause issues.
So that might be ways in which I bring in external data.
Kert Viele: are certainly, I mean, there are cases where you are
gonna run into those problems.
I mean, we've talked in examples and antibiotics.
The development of resistance over time, we expect therapies to change.
There are situations where we think that problem exists and we need
to address it or not use the data.
There are also cases where diseases are very stable over time, and those
issues may not be as big a concern.
Scott Berry: Yeah, so I, I mean, I'm, I, I, I'm very much on the side that
we should be using external data.
I, I, uh, I think we go about this in a way that's just so strikingly
conservative that it slows us down, uh, in it, that, that.
That, you know, something may happen.
I, I think it falls under the realm that science is hard, uh, incorporating
this other information, uh, be explicit about it, incorporate it, and the
process of this is hard, but the notion that we don't use any of that
information just seems strikingly wrong.
Kert Viele: So I, it's an interesting question 'cause it
certainly can be wrong on occasion.
So I, I wonder if as a society and you, you ask about frequentist and whether
the error rates are driving this, I think you can be a, a perfectly good and
wonderful frequentist and use historical data, but what you're gonna have to
accept is most of the time prior data is leading you in the right direction.
If it's not, we should just see science altogether.
Because we have worse problems.
But in any case, if it's generally leading us in the right direction, we're
gonna be running a few experiments that have say three or 4% type one error.
And we're gonna be running a lot more experiments that have say one or two.
And in the grand scheme of things, we're still putting a
better mix of drugs on the shelf.
More drugs that work, less drugs that don't.
So I think we have to change our mindset to be frequentist in this kind of world.
Scott Berry: Yeah, it's a, and I'm struck by, I, I'm struck by the idea of using
that data and in and in Bayesian we do
Kert Viele: I.
Scott Berry: borrowing of that information.
We do things that are dynamic.
I'm struck by, there are tons of trials that we use,
objective performance criteria.
That it's a single arm trial and the drug has to jump a particular hurdle.
That's a number.
That number's always based on prior data, and we do that all the
time, and presumably that's okay.
Or you run a trial where your only, your data comes from the new
experiment, and that's your control.
But the idea that we do something halfway between that, that we use
it, it just strikes it, people the wrong way, just do one or the other.
And, and I, I'll, I'll, I'll give an example of this happening to me.
Where, by the way, and, and if you listen to, in the interim a lot, you'll find
that we, we very much respect the FDA.
The it, they're, they're, they're the, um, uh, i I in the world.
They do tremendous good and, and they're wonderful.
It doesn't mean we don't run into hurdles.
So we
presented a
Kert Viele: so just, just to interrupt you there, I mean, we go
to the FDAA hundred times a year and have three major disagreements.
That's much better than with our wives, so.
Scott Berry: Yeah.
Yeah.
So we provide an example that we wanted to go in with a new therapy.
It was oncology and we were, we, the, the new trial.
We didn't want to enroll one to one, to a standard of care.
Uh, it's easier to get patients in if they're more likely to
get experimental treatment.
Uh, we wanted to enroll three to one, three to experimental, one to control.
There's a good bit of information about the behavior of the control.
So we were borrowing from previous trials, and let's assume that,
that the prior data was a 15%, uh, overall, uh, objective response rate.
Make it simple and, but we're gonna enroll three to one.
We're gonna borrow on the control using that, that prior
data that's centered on 15%.
In the new experiment because we're borrowing data that's 15% on the control.
If the new rated, that new experiment is actually higher than that, we're pulling
the control down, making it easier for the treatment to look better, and we
present those operating characteristics.
That was, that was deemed to be unacceptable because you
inflate type one error rate.
The response from the agency was use 15% as an objective
control and don't enroll any new controls, and you have to beat 15%.
Of course, that we generally don't calculate the.
Type one error.
If in fact the real rate in that experiment was 25 or 30%
because we're jumping 15%.
But it's almost the, the, the opposite Bayesian view that we know that answer's
15% for what you're trying to beat.
And if you can beat that, that's good.
It's using a, a, a prior that's all focused on a single value rather
than modeling it with some borrowing and recognizing the uncertainty.
Or enroll a whole control arm, so do one or the other, but
it's entirely this experiment.
But the idea of using borrowing or modeling seems to be harder to accept.
Kert Viele: It's interesting, it's almost you have to cross the
Rubicon of what are you willing to completely make this assumption or
not make this assumption at all?
And you can't test the assumption.
Scott Berry: Right.
Right.
And the other extreme of this, and I know, I know, I know this
one, uh, gets at you a little
Kert Viele: Don't do it.
Don't do it, Scott.
Scott Berry: Yeah.
That we talk about using real world evidence all the time.
Digital twins or real world evidence.
So for, for control, we bring in external data.
And people seem to be almost comfortable that your control is
entirely external data, that's okay.
Um, but if you were to do something like use non concurrent controls
from your own experiment, that seems to be unacceptable despite
the fact that these are unbelievably phenomenal, uh, historical controls.
And better than any real world evidence you're gonna get same protocol.
You have the exact same data, the same quality.
The only difference with them is time.
People seem really uncomfortable with that.
But yet, at your conference you went to, there are probably 10 talks
about using real world evidence.
It.
Kert Viele: Yep.
The, um, I mean, it, it's, there are different legal reasons that real world
evidence is being handled in certain ways right now compared to other.
Um, but, you know, FDA is not a, um, it's not a homogeneous organization.
There are lots of people, the academic community is a, is many
people, industry is many people.
Um, but I think we're gonna, if we're talking about where we're going, one of
the key aspects of this is a hierarchy of what's more or less dangerous.
And I'd actually like to see us spending more time on.
Alzheimer's trials.
We've got dozens, hundreds of these.
So how much do the control arms differ from trial to trial to trial to trial?
If we have hundreds of pieces of trials, we should be able to estimate this.
We're statisticians.
We can estimate a variance parameter.
Let's get an idea about what it is, and that variance parameter translates
into how much we should borrow.
Scott Berry: Yeah, and, and you, you mentioned a key point to that if
you're enrolling some new controls, you actually get a comparison.
You get the, you get the controls in your experiment against the previous
controls, and you can judge some level of, of similarity between those and,
and what's become okay.
Kert Viele: Why don't, why don't you talk about that a little more?
About just the details?
'cause I think there's often a sense of.
the fact that there is a hierarchy between a single arm trial, dynamic
borrowing, static borrowing, get an idea about what those methods are
and how they mitigate those risks
Scott Berry: Yeah, yeah, yeah.
So, so the extremes on this, and I can run an experiment where I only include
data in my new experiment, uh, one-to-one randomized control to treatment.
And I'm only gonna compare those two arms.
Uh, in that I could do the exact ex, uh, very much extreme from that,
where I enroll only my treatment arm.
And I, I want to know, is the treatment arm behaving, you know.
Better than, or is the treatment benefiting these patients?
I would have to figure out a way to create a control arm.
I could use entirely historical data.
In that far extreme.
I have no ability to know whether that control rates I'm using is is reasonable.
If suppose I do something halfway that I enroll two to one or three to one.
I use the controls in that experiment, but I also use external
controls to reinforce the control.
Now I'm getting new controls so I can judge their similarity
to the external controls.
And statistically model that similarity using the external data
to strength if they're similar, and not if they're not, but I do get
evidence about their similarity.
And it's different than this extreme case where I only use controls and I have no
idea whether, whether they're good or not.
Uh, as, as a middling ground to this.
Kert Viele: And I think, you know, that's really what's going on in the
academic literature right now in terms of trying to figure out the best ways
to make that assessment, minimize the risks, amplify the benefits.
Um, we're talking now more about covariate matching between studies.
I.
So not just looking at all the historical data, but which patients
best match our current enrollment.
Um, none of that of course is going to be perfect, but I think it, again,
it's the question of do we generally add information or are we taking a risk
and we should be looking at a society If we do this, these kind of trials
over and over and over again, do we get a better set of drugs on the shelf?
Scott Berry: Yeah, and, and part of this is it.
Part of it is statisticians are really smart and they can figure
out how something can go wrong.
That this, the, the, the historical data's not the same as the new
controls we've got, uh, beha, we've got problems with the analysis there.
We've got potential type one error inflation.
We've got potential bias in the estimate of the treatment effect.
And statisticians can figure this out and say that, you know, this is bad.
But if we have to do experiments where we have to enroll one-to-one,
that experiment is bigger.
It's more patients.
It's more patients on control.
We take less shots on goal.
We learn slower.
It's, it's the whole industry is slowed.
If every time we have a question.
It has to be only in that experiment as opposed to using what we know
scientifically to reinforce that making a more efficient trial we can, we can
look at more treatments, we can treat fewer patients, we can do this cheaper.
So there is a huge societal, I'll call it societal, a whole medical field issue.
If every time we have a question.
The only thing we can do is look at the new experiment.
It seems like there's a huge waste.
Kert Viele: Well, we basically, it's like data.
We design an experiment, data comes into existence.
There is this bright shining moment of about five minutes
where we can do something with it.
And then we're meant to put it aside, at least with regard to future experiments.
Obviously people are gonna reference it in deciding treatments and so
forth in the medium, but it, it's interesting, it worries me as a
statistician, I feel deep-seated guilt actually putting data aside.
Scott Berry: Yeah.
Yeah.
Uh, and, and, and again, it's hard, uh, but there's, there's huge ramifications
to not doing it, uh, in that ways.
And I think there are ways of doing it very well.
You can be prospective in your new experiment.
The analysis plan is completely written down, written down.
It's explicit.
When you present the results you, you show the results.
Just in the experiment, you show what happens using the
modeling of external data.
I think we can do this really well.
I think, by the way, this is changing a little bit.
We're seeing this even a somewhat similar issue where you run basket trials.
There, there's a lot of reticence to a, you know, we have four
kinds of patients and we run an experiment of control to treatment.
Of using the data in one of the subsets of patients to help estimate
another subset of patients that gets people really bothered, and
that's been done more and more.
I think it's a very similar type of issue and it's sort of striking and
we're in trouble if we don't do that.
But it has similar statistical ramifications.
It.
Kert Viele: Well, it also, it, it has tremendous implications towards
design because if you can't borrow, suddenly sponsors have an incredible
incentive to claim that their patient population is homogeneous
so that they can pull everything together and maximize their power.
Scott Berry: Yeah.
Kert Viele: you say, you know, as soon as you say there's two different
groups of people, you have to run twice as many patients, no one's gonna
wanna explore those kind of questions.
Scott Berry: yeah, so it's the same bizarre thing that happens, that we run
trials that are single arm trials where we use a historical control a hundred
percent, or we don't use it at all.
And you're describing exactly what happens in trials.
We run a trial and you do a common estimate across the entire population,
a single estimate, which is this, which is full pooling, it's exact same
mathematically of pooling populations altogether to get a single answer.
Or we run separate trials and we call 'em population A and population B, but we only
estimate those individually in A and B.
Now we want to do something that's a middling thing between that, and by the
way, we actually get to observe in the experiment similarity of effect that we
shrink estimates statistically, uh, that isn't full pooling and it isn't separate.
And that people struggle with that.
It's either you have to do all pooling or completely separate
populations, and it's just, it, it, it strikes me as just bizarre.
Kert Viele: So let me ask you about one other thing, which I think is a
real oddball in this conversation.
Um, there is a situation where we routinely combine data across
different studies, and it's often viewed as the pinnacle of evidence
for clinical trials, which is a
Scott Berry: I, uh, I have no idea.
I have no, oh, meta-analysis.
Okay.
Yep.
Kert Viele: where do you think meta-analysis fits in, in terms
of the, the assumptions and how it fits into this debate?
Scott Berry: Yeah, I, I, I think meta-analyses are incredibly valuable.
Um, the modeling makes a ton of sense and incredibly important.
Um, I, I think people overweight single trials and the, the totality of evidence.
Now, not every meta-analysis is every
meta-analysis.
Uh, there's, and, and, and I think people, I sort of think
meta-analyses are bad because there are some that are clearly biased.
There's publication bias, there's, there's availability of data bias.
Um, uh, so I.
Not every meta-analysis is bringing quality data that's free of these
issues, but there are many circumstances where we have data on essentially every
patient that's been experimented on.
And combining this together I think is hugely valuable.
I By, by the way, I think the idea that we run two separate
independent phase three trials and we analyze them separately is stupid.
I, I, I mean, it's stupid that we would analyze them completely
separately, and we need both to be 0.05.
Combining them together into a single inference is better
in every way.
Kert Viele: is two is two 0.49s
as good as a 0.051
and a 0.001
on the P-values.
Clearly you'd want the latter.
Scott Berry: Yeah.
Yeah.
And, and, and so I think of a meta-analysis the same way I, I,
I would do a meta-analysis of my two phase three trials for what,
what does it tell me to the answer?
So it, you know, those techniques, they tend to be Bayesian.
But, uh, largely the availability of data.
And there are some where we know there's missing data, we know there's issues
with it, and we would not take those at.
Uh, you know, at, at a high scientific degree to that.
Um, and that's where the science parts comes into it.
And by the way, these are abused where people present data, it's bias data.
They're ignoring huge amounts and they call it a meta-analysis.
And we think that's bad science.
So we've got to, at some level, be able to judge.
Good science, quality data, meta-analysis, which I think are more
valuable than individual trials there.
And then there's bad science put together that we, we, we recognize
we wouldn't use that information.
Kert Viele: We would certainly say that on any kind of borrowing, bringing
together data that doesn't belong is bad.
You can borrow in a way that does generate horrible biases and
will generate bad conclusions.
Uh, but we also are talking about what can be done well with experience.
Um, we're running outta time.
Why don't you close us out?
Scott Berry: Uh, so, uh, I, I think this by the way, is the future.
I think as we learn more and more about diseases, we get
more and more subsets of them.
Um, we are getting more and more quality data available from other clinical trials.
The sharing of data from clinical trials, the sharing of other
types of resources, we're gonna have explosion with medical data.
The idea that we don't use that in our new experiments, if we
don't use that in valuable ways, we're running worse experiments.
So a wonderful topic, and I think it's certainly the future.
Uh, and so here we are in the interim.
So, uh, till next time, appreciate it.
Thank you,
Kurt.