Mailbag: Learning Healthcare, Multiplicities, and the Pace of Innovation

In this mailbag episode of "In the Interim…", Dr. Scott Berry answers two listener questions and weighs in on the pace of innovation in clinical trial science. Asked what infrastructure a learning healthcare system would require, Scott defines one through the REMAP structure (randomized, embedded, multifactorial, adaptive platform) that Derek Angus named for REMAP-CAP, and uses REMAP Periop at UPMC to argue that embedding randomization and data collection in routine care keeps the cost low while letting a health system learn from tens of thousands of patients a year. On multiplicities, Scott explains why he interprets subgroups, endpoints, and related trials within a Bayesian hierarchical model rather than by adjusting alpha, with the assumed variability across units (tau) setting how far each estimate shrinks toward the overall effect, and he critiques hierarchical testing as non-Bayesian. The episode closes on the contrast between fast-moving innovation, including the AI-accelerated redesign of FACTS, Berry's clinical trial simulation platform, and ARPA-H's SURPASS call for adaptive platform trials, and the conservatism of the EMA's draft reflection paper on platform trials and ICH E20.

Key Highlights

• REMAP (randomized, embedded, multifactorial, adaptive platform) as a working definition of a learning healthcare system.
• Embedding randomization and data collection in routine care to remove a large share of trial costs.
• REMAP Periop and learning from tens of thousands of elective surgeries a year.
• A Bayesian view of multiplicities: hierarchical models in place of alpha adjustment.
• Tau, shrinkage, and reading a subgroup result against the overall effect.
• AI, digital twins, and ARPA-H's SURPASS call for continuous adaptive platform trials.
• The contrast with the EMA's draft platform trial reflection paper and ICH E20.

For more, visit us at https://www.berryconsultants.com/

Creators and Guests

Host

Scott Berry

President and a Senior Statistical Scientist at Berry Consultants, LLC

What is In the Interim...?

A podcast on statistical science and clinical trials.

Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Well, welcome everybody.

Welcome back to In The Interim.

I'm your host, Scott Berry.

I appreciate you joining me again.

I've, I've-- for, for multiple
episodes, I've been asking and,

uh, uh, in- uh, making inquiries
out there if people had questions.

Um, uh, you know, again, Berry,
Berry Consultants, myself, this

podcast focuses a lot on statistics,
focuses a lot on Bayesian statistics.

Uh, I'm a Bayesian statistician working
in the clinical trial science, uh, uh,

industry, and I, I try to, try to come
up with topics that I think people would

be interested viewing from a Bayesian
perspective or from that experience.

But would love, uh, questions from people,
what, what, what questions they have,

uh, things to address on the podcast.

So I have a couple of those, and
so I'm gonna do three topics today.

I have two, um, email requests,
and I, I will get into those.

I'll, I'll, I'll explain
the email requests.

Two questions, really.

And then the third one is just
I, I, I wanna weigh in on this

incredibly fast-moving world
of innovation in clinical trial

science, healthcare learning, uh,
the, you know, m- medical science.

It's all tied together, and it
is-- it, it's pretty exciting.

So I, I, I wanna talk
about that specifically.

Okay.

So the, the first question I got
was from Elias, uh, Ippourson, and I

probably didn't get the name right.

Sorry for that, uh, Elias.

He sent in a question.

He says, "Hi, Dr.

Berry.

In your most recent podcast,
you asked for suggestions.

I feel like I remember you mentioning
a learning healthcare system before."

And I have talked about a
learning healthcare system.

This is of interest to him.

"I wonder if you could discuss the
practice-- the practical barriers that

would need to be worked out vis-a-vis
infrastructure and analysis outcomes

for this, uh, ever to be a reality."

I feel like I hear many Bayesians talk
about a learning healthcare system, but

I've never heard anyone articulate what
astronomical infrastructure investment

and coordination would be needed to even
start considering something like this.

So I love the question, and I, I, I may
come back to way-- to, to the way he

worded-- he words this, Elias words it.

Um, and that is articulate the
astronomical infrastructure investment.

Okay.

So I, I have talked about, um,
learning healthcare system.

It, it's one of these things.

So much of, of problems in, um,
statistical consulting, in scientific

interactions is that a lot of people
use terms to mean something, and other

people use them to mean something else.

And when somebody says it, everybody
nods, and they all think there's

agreement, and they actually all
mean kind of different things.

Learning healthcare system is
absolutely one of these things.

Everybody believes they know
what a learning healthcare system

is, and a lot of people think
of them as different things.

So my view on this, I, I suspect
is quite different than others.

So I'll go through what, what do I
mean by learning healthcare system.

So wonderful question.

Uh, I touched on this a little.

If you go back to the podcast I
have with Derek Angus, and much of

what I've learned or my inspiration
for this comes from Derek.

Brilliant, brilliant scientist, um, and
at, at University of Pittsburgh, UPMC.

I've worked with a, a number of
projects, in particular REMAP-CAP

And Derek named REMAP-CAP.

Um, uh, uh, originally REMAP-CAP,
the clinical trial, and I

think many of you have heard of
this, had many different names.

It was ADSCAP, it was
Prepare, multiple names.

It eventually became REMAP-CAP, and
Derek came up with the name REMAP.

REMAP is ac-- is an acronym,
and I'll go through it.

And this is gonna be kinda my
beginning and end to this question.

So REMAP is an acronym.

The R is randomized, the E is embedded,
the M is multifactorial, and the A and

P are adaptive platform, and then the
CAP is community-acquired pneumonia.

So this is a trial in
community-acquired pneumonia.

There have been multiple trials, other
trials that have now adopted what

they're saying, the REMAP structure,
the REMAP ideas, and so REMAP is a kind

of trial Now let's go through each one
of these 'cause I think they all play

a really, really important role in what
I mean by a learning healthcare system.

So the first word is randomized,
and I know Derek was very, very

strongly that this word, word be in
here and is in a critical part to

a good learning healthcare system.

And part of it is we see the
world of competition for this.

This idea of learning healthcare, a lot
of it is get big data from healthcare

system and try to make inferences
about which treatments work better.

All observational data.

Hugely, um, uh, difficult to try to make
causal statements, and I know there are

reinforcement learning models out there.

There are different ways to try to
take this huge big data in healthcare

and try to make inferences about
who benefits, who doesn't benefit,

and it, it's, it's incredibly hard
to do in the observational sense.

So REMAP is randomized.

The assignment of interventions
in REMAP are all randomized.

Really, really important part.

Embedded.

Embedded carries a huge weight here.

And embedded means I, I think
really two important things.

One is that the data collected on patients
are already part of, uh, the, the process.

We already collect the data, and imagine
in a healthcare system, we already

collect a great deal of data on patients.

In a clinical trial, what do we do?

We go out, and we hire a CRO, pay them
huge amounts of money to collect different

data than is already being collected.

It's incredibly expensive.

We collect this huge separate database.

We have, uh, uh, monitors that go
through and, and, and collect that.

Huge proportion of costs of clinical
trials are the cost for this part.

Now, I, I wanna be careful not to,
to, to sort of say things about CROs.

I think eventually CROs exist
in a very different way.

The idea that we have to collect
this data in a separate way is a

huge part of the cost to trials.

Now, part of things in clinical
trials, costs aren't gonna go away,

but the embedded nature of this is
that we already collect this data.

Can't we just pull it from
the healthcare system?

It's greatly reduces cost.

It's, it can be very,
very, um, quality data.

But the other part of embedded here is
even the assignment to patients, the

order sheet of when a patient comes in.

Now, the, the REMAP-CAP trial, it's just
every patient with community-acquired

pneumonia that meets criteria, immediately
it comes up as part of the healthcare

system, Epic, Cerner, whatever it is,
comes up and says, "Ah, this patient

is eligible for REMAP-CAP," sh-- uh,
and they're randomized We're doing

this kind of thing in trials, and
I'll, I'll touch on this a little bit

more, that that patient is randomized.

And then the order sheet to the
patient gives three treatments

that came from the trial.

This patient should take an IL-6
receptor antagonist, no steroids,

and they get an antiviral.

You know, whatever the particular
therapies are, that just comes

out and it's part of routine care.

That's embedded within routine
care to assign a patient to a

treatment, to collect their outcome
data, and it was randomized.

Multifactorial, a huge part of
this, and I think the world of

clinical trials is going this way.

Certainly comparative
effectiveness, learning healthcare.

We're doing many, many things to patients.

Many clinical trials change
one thing about a patient.

And on purpose, we only
wanna change one thing.

But this multifactorial aspect of
this, they're being assigned many,

many things that we're investigating.

In REMAP-CAP, one particular patient
was randomized to seven different

domains simultaneously, and they got an
intervention from seven different domains.

We have other multifactorial
trials where th-this is being done.

Now, imagine we're learning from
this patient in seven, ten, twenty

dimensions within this setting.

And then the end, the adaptive platform.

So this is a large platform
that is multifactorial.

It's already embedded within
care, and patients are randomized.

Now, coming back to REMAP-CAP, the
only part of REMAP that in some way

REMAP-CAP doesn't completely live up to
these lofty goals is the embedded part.

It has separate data, uh, within that.

The randomization isn't
done embedded in that.

So that is a, a, a lofty goal we still
have as to how much you can embed that.

Okay.

So all of these things are really, really
important, uh, within that, but that

is a learning healthcare system So this
question, uh, that, that Elias brought up

of the colossal, what was the word used?

The astronomical
infrastructure investment.

Imagine you have a healthcare system
and you bring in a REMAP trial.

So you have this platform of learning,
and you have many points that you don't

know what-- You have multiple options and
you don't know how to treat a patient.

And I-- This is harder for me
because I'm not a clinician

and I, I don't treat patients.

But overwhelmingly, they all say
there are many, many times they are

making decisions without high-level
evidence of, of what to do.

There are many points within, uh, in
an individual's healthcare experience.

Uh, even the, the setup of the site, there
are many things that could be done in a

site-specific cluster way that we learn
how to, how to provide better healthcare.

So there's many touch po- touch points
of patients, and we could identify those,

and I'll give you an example of that.

Maybe I'll actually jump to that, is
in elective surgery, there's a trial

at UPMC, and it's, it's REMAP-SPRY
and, um, uh, within that, it's now

moved to REMAP PeriOp, where when a
patient comes in for elective surgery,

there are many, many things that the
healthcare system doesn't know what to do.

And I, I was sort of struck by
when we were talking about this.

They still don't know whether patients
should drink fluids before surgery.

Can this affect pain?

Can this affect nausea and vomiting?

Can it, can it affect, um, delirium
in ways that, um, will keep a patient

in the ICU, will keep a patient in the
hospital for two or three days longer?

We don't know the impact
of fluids, a-and it varies.

We could, we could randomize
fluid management, what the patient

does 12 hours before surgery.

In PeriOp for elective surgery,
healthcare systems like UPMC do tens of

thousands of elective surgeries a year.

Imagine if we learned
from every one of those.

Pain management, perioperative pain
management, antiemetic management.

Uh, during in theater, there are
many decisions that are made, uh, and

it's not clear the right thing to do.

After surgery, different decisions
that are made on a patient.

This was the whole goal of this
REMAP PeriOp, that this is a platform

within the healthcare system.

So astronomical?

It's actually the opposite.

You already have this healthcare system.

If you embed this within care,
the cost is, is actually minimal.

So you don't build a healthcare system
to run a learning healthcare system.

You import the learning technology into
the healthcare system that already exists.

Not only what is the cost of
this You're gonna save money.

There's no question you're gonna save
money, you're gonna treat patients better.

So let's back up a little bit,
uh, i-i-in this setting before I

get to that sort of conclusion.

So imagine the many places in which, um,
points of care, decisions are made by the

treating physicians, and w- four possible
things and, and you don't know what to do.

Imagine if these are randomized, and
in a multifactorial way, there are many

randomizations to ha- that happen in a
patient, and it's part of the order sheet.

When this elective surgery comes in,
they get these are the things we're

gonna do, and they're randomized.

Then the platform pulls all of
the data, the randomizations, and

all of the data from the data that
already exists in the healthcare.

There's no cost.

There's no CRO.

It's probably 90% of the cost
of trials is right there.

By the way, this is…

REMAP is what, um, internet
technology companies do when

they give you advertisements At
different points, you're getting

advertisements, and they're randomized.

These are-- this is bandit
problem r- uh, uh, things that

they're randomized as to who…

Are you more likely to respond
to this one or this one?

It depends on your characteristics, uh,
your, your, your internet preference,

your cookies, all of those things
that it gives you an advertisement.

And they learn from do
you click on that or not?

Do you respond to that?

Then the next time, they use
that to change how they tr-

how they treat other people.

Now, imagine we do that in healthcare.

We bring this simple
technology to healthcare.

It's almost free.

You do response adaptive randomization.

So as you've got these four different,
uh, things, the four different, uh,

it- actions, you learn which ones are
providing better outcomes for patients.

Invariably, they're gonna be cheaper.

You're gonna save money, and you
assign those more and more to patients.

And as you learn that D
doesn't work, it goes away.

You could add E, you could add F.

Eventually, you're gonna get to the
point that you decide B is the best,

and it might depend on the patient.

It might be in GI surgery, you
should do this, but in orthopedic

surgery, you should do that.

You have tens of thousands of, of
surgeries a year to, to do this.

The response adaptive randomization
that just moves to zero one when

you know the answer to that.

So, um, there, there's a
publication, Holder Murray et al.,

"REMAP-PERIOP: A Randomized Embedded
Multifactorial Adaptive Platform Trial

for peri- uh, Perioperative Medicine to
Determine the Optimal Enhanced Recovery

Pathway Components in Complex Abdominal
Surgery within a US Healthcare System."

Uh, go take a look.

It's fabulous trial, fabulous
paper, and it's what I think a

learning healthcare system is

So it's REMAP, and I don't
think it's expensive.

I think it, it, it also tackles one
of the incredible, I don't know,

frustrating thing about healthcare.

We-- I, I work in clinical trials, and I
design clinical trials, and we run a trial

where we enroll 400 patients, and we learn
an amazing amount from these 400 patients.

There are, you know, 100 times
as many of those patients being

treated out there in healthcare
systems that we don't learn from.

We don't.

We don't, we don't learn
from those patients.

One doctor gets anecdotal
information from three patients.

Another doctor gets anecdotal
information from four patients.

They may be misled.

They're not learning in a formal way.

Imagine if those 40,000 patients went
into a learning system The impact

we would have on that, think of the
impact that a 400-patient trial has.

Now we have 40,000 patients in a random--
randomized learning healthcare system.

Incredibly big visions on this.

Okay.

So that, that's what I mean by a learning
healthcare system, and I know a lot

of people use different definitions.

I've been in meetings where people is it's
how you implement what we already know

works is a learning healthcare system.

Another one is we get the big data,
and we try to make inferences, causal

inferences on, uh, uh, anecdotal
data, which by the way, is incredibly

powerful, and it's happening.

Uh, but imagine if that
data were randomized.

Okay.

So Elias, uh, uh, thanks for the
question, and I don't know if you, you,

uh, you, uh, uh, knew what you were
getting yourself into with that question.

Um, okay.

Question number two,
and I need to apologize.

This question was sent in, and
I actually spent a good bit of

time, uh, trying to find it.

Uh, it was a couple months ago, and I,
I haven't gotten a chance to get to it.

It was on the back burner because
we have a really cool paper coming

out of Barry, which, uh, I'm not
gonna reference on this, but I--

maybe I'll reference it a little bit.

But, um, that we're waiting
for finalization of it

that I think on this topic.

Um, so I, I, I can't, I can't thank
the person who asked the question.

But the question was what do
Bayesians think about multiplicities?

And, um, so it just generally I
think the question is, okay, a

huge, incredibly hard problem in,
in life, um, in clinical trials.

You know, I love sports.

In sports, um, in, in learning in any
setting is dealing with multiplicities.

So what is the Bayesian view on this?

Okay, first of all, multiplicity
is an incredibly hard problem.

Um, but it's science, and it's a
really important part of science.

You can't just ignore this.

So almost all data we get, you know,
and statisticians I think do this pretty

well, it's always how did I get this
data, and what data am I not seeing?

It's just this init- it's in,
this immediate reaction to this.

Um, and then there are many examples
of it, many examples in clinical

trials, many examples in sports.

Uh, it- we're in the middle of, uh, the
NFL season, and you, you turn on ESPN,

you turn on something else, betting
has become huge amount of advertising

on these various betting sites.

By the way, I, I don't think
this is a good thing for-- to,

to have these things, but we do.

Um, these betting sites where you
go there, and they'll say, "Oh, this

football team, you know, they're seven
and three against the spread after they

lost a game, and they lost last week."

And so that seven and three
against the spread after a loss

What does it mean in sports?

And if, if anybody follows
sports, you get this all the time.

And so what-- y- when, when you get that
sort of multiplicities, what do you do?

What I don't do is say, "Oh, let's
adjust alpha, and that seven and three

is not significant, therefore, I don't
believe there's an effect there."

I d- I don't think anybody
learns that way, uh, generally.

We do in clinical trials, and
I'll, I'll, I'll come back to that.

Now, in clinical trials,
there are many multiplicities.

The publication of results,
there's publication bias.

What didn't get published when
a pharmaceutical p- company

didn't publish various things?

What does get published?

What's put in the paper?

Uh, we're really good in clinical
trials, probably better than any science,

of sticking to predetermined rules.

We said we're gonna do this, here it is.

It's such an important part, um, to, to
the, the clinical trial industry, and

journals are really, really good at that.

Um, we have multiple arms in trials.

What happens when one looks good and
the other o-- you know, the middle one

looks pretty good, and the other two
don't look very, uh, as, uh, uh, as well?

They didn't perform as well.

Multiple endpoints.

We analyze five endpoints in a
trial, and boy, this endpoint

looks good, and in this subgroup.

So there's many subgroups we look at by
disease severity, disease classification,

phenotypes of disease, uh, demographics
of disease, old, young, time last known

well, uh, all kinds of characteristics.

And many times we're presented
that, oh, in this subgroup, the

treatment was seven and three.

I am making a reference to seven and three
following a loss against the spread And

the researchers there now with this, "Oh,
in this subgroup, there was an advantage."

What, what, what does it mean?

Okay.

Now, what does it mean to me?

Uh, as, uh, it's incredibly hard problem.

It's trying to identify that are
there multiplicities is a huge part of

our lens for interpreting something.

That seven and three for a
team after a loss against the

spread, yeah, it's meaningless.

I, um, uh, and by the way, I think 90%
of the stuff these, these, uh, betting

sites put out and all this stuff,
it's to get you to bet, uh, in it.

And if you follow those sorts of things,
I think you're being, you're being misled.

Now, how do I handle them?

How do I…

A- and I, it's hard for me to say,
"Oh, this is what Bayesians do."

Uh, as a Bayesian, it's always,
what does that mean in the

larger context of the data?

It's always that.

Everything to me is a hierarchical
model, is a Bayesian hierarchical model.

Now, it, it, it, they're not all
exchangeable type models, but that

idea of consuming information in that
way is how I think I think about it.

It's a very Bayesian concept.

What, what, what does that mean?

When I see multiple subgroups in a
trial And I'm looking at one of them.

It's that one relative to now
what did the other group show?

And if it's a bit of an outlier, no,
I, I don't believe that the facts

is real, but it might be outside…

It, it might be different than the rest.

Depending on the strength of that
result, the, the number of results.

Uh, I'm sorry, I should back up.

The, the, the number of
subgroups within this.

So if it's one of ten groups and
it's the one outlier and the rest,

uh, show no effect, I pull that in
towards the others, and I probably

think there's no effect there at all.

Now, again, depends on the size of that
group, the extremeness of the data.

Imagine making inference
of one of the subgroups.

So you open up a journal article
and it presents you eight subgroups,

and it always gives you in the
forest plot the overall effect.

That's a hierarchical
model, uh, i-in many ways.

And when I make inference about one
of those subgroups, I look at the

subgroup and then I look at the overall
effect within that, and of course,

consistency across the subgroups, and
I largely think the overall effect is

the right answer for each subgroup.

If I'm seeing large variability, now,
uh, I, I, I can have more belief in

the heterogeneity, and I might actually
start to think that one of these

estimates in the subgroup, and I pull
away from the middle effect within that.

But my estimate is a combination of
that individual multiplicity thing and

the larger context of the data always.

In, in me, me, me consuming data
in the world, same thing in a

clinical trial, same thing in sports.

Now, the tau, life of tau, uh, Scott's
life is all driven by tau, is I

have subjective beliefs on how much
variability I think there is across these.

And that tau that drives the regression
towards the overall mean or believing the

individual ones is what, uh, i-in normal
life I've come to have some belief in.

In sports, the likelihood that somebody
on grass on a Tuesday is different than

a Thursday, uh, yeah, I mean, I presented
that in a way that you probably think

tau is really, really small and you
shrink everything, and you look over at

the overall effect in that Within that.

Same thing in clinical trials.

It's the exact same
thing in clinical trials.

Now,

is that different?

Um, uh, and, and let's, you know,
what are multiple things in trials?

There's multiple endpoints, multiple
subgroups, multiple trials presented

on a particular drug, uh, within that.

Multiple drugs in a
class, uh, within that.

What I don't do is I don't say,
"Ah, we're looking at eight…

I, I-- there's eight different
classifications that-- units

that have been presented to me."

I change my alpha to a different number,
and then if it's, if, if the P value is,

is significant at that alpha, I completely
believe the entire estimate of the effect.

But otherwise, I believe there's
no effect and it's all the same.

I, I, I don't think anybody
does that in science.

I don't think anybody does
that in clinical trials.

But we do that in
analyzing clinical trials.

We do this hierarchical testing
thing that we look at, was the result

significant at some alpha level
because we're doing multiple endpoints?

Ah, it is.

We believe the effect at face value.

If I'm running a trial with four
endpoints, this should be analyzed

within a Bayesian hierarchical model.

The estimate on this endpoint should
borrow from the other four endpoints.

And the estimated effect should
vary depending on the others.

Now, this is a really important
part to that, that this idea that we

pick an alpha, and is it significant
on that for four different units

I, uh, the idea that we look at the
individual effect in that group compared

to some alpha is just against this idea
of, well, what are the totality of data?

I'm gonna push that estimate into some
realm of, I believe it's different

than zero, or I believe the effect,
depending on the other results.

So whether or not something is significant
should depend on the other units

looked at in a shrinkage towards them.

I think that's the, a Bayesian
way to think about multiplicity.

So I don't like hierarchical testing.

I d- I think it's a strikingly
strange thing we do in trials.

I think it's strikingly non-Bayesian.

I think the Bayesian does it through
the prior, and my prior is almost

always this hierarchical prior
where the variability tells me

the prior for the individual units

Doesn't matter.

Endpoint subgroups, multiple
trials, different results of

patients, different mutation groups.

Uh, all of these things, I would treat
them all that same way, and I think, I

think scientists do treat them that way.

We have this phrase, the FDA has
this phrase, totality of information.

Rarely is it formalized.

I think the world is moving a
little bit more towards that

Okay.

So, uh, a-again, uh, I, I, I don't have
all the answers in multiplicities or

learning healthcare systems for sure.

That's my view on multiplicities
and, um, you know, comes from,

um, um, the world around us.

Um, and I suspect Bayesians
somewhat share, uh, that philosophy.

Okay.

The last part I, I just-- I've been
struck lately, um, i-i-with the

starkness of, of the different worlds
that I interact with in, in science.

And, um, it, it's, it's about the
pace of innovation and the, the

different pockets of scientists
dealing with this pace of information.

You know, uh, to, to many of
us, AI two years ago was this

odd sort of cute little thing.

Um, and in the last year, it,
it has had massive effects.

Uh, everybody in clinical trial
uses it as an assistant in some way.

Uh, some more than others.

It's completely changed, uh,
many fields, many situations.

It's undeniable its effect.

Um, we're, we're so staring at
that and its role within learning

about treatments, about healthcare
and, and what does it mean?

The idea, the advancements that have been
made this year, think about a year from

now Uh, it, it's incredibly exciting.

Digital twins were something that,
you know, it was kind of odd.

All of a sudden…

A- and, and digital twins is another
thing like, uh, learning healthcare.

It means so many different
things to different people.

The ability to simulate patient outcomes,
say for a control, in whether it's

Alzheimer's, ALS, infectious disease.

Yeah, I'll throw infectious
disease in there.

Uh, every other setting, we're getting
better and better and better at predicting

what would happen to a patient if
you give them a particular therapy.

And the idea of utilizing this is just
becoming, uh, clear that the world is

going to utilize this moving forward.

Clinical trial simulation
to optimize trial designs.

The, the one thing this, this speed
of innovation's gonna do, it's

gonna get people to look at our
clinical trials where we, our data

are blinded for five years, and we
look at the data after five years.

It's just, it can't be
that that's what we do.

Uh, trials have to be continuous learning.

I- interims have to be done,
in the interims have to be

done incredibly frequently.

Uh, we, we, we can't design
those trials anymore.

The, the world of this is
absolutely going that way.

By the way, if the clinical trial
infrastructure doesn't utilize

these tools in the smartest,
best way, we're gonna be lapped.

Absolutely lapped, and other alternatives
to learning, to im- approving drugs,

to develop drugs will be done in
ways we may not-- might not be great.

Um, but th- you know, this-- these
things are all absolutely clearly

affecting the world of clinical trials.

Now, I say innovation.

Innovation doesn't always mean for
the better, um, in this, but it

absolutely is having a massive impact.

Even on-- we…

So Berry Consultants has FACTS, a
clinical trial simulation platform.

It's a, it's amazing piece of software.

The ability to go in there
and simulate different trials.

Well, we, we…

It's really good at going in there,
and you lay out a trial, and it

simulates the results of that trial.

If you wanna look at a different trial,
you have to kinda go in and simulate

that, and now we as, we as statisticians,
you know, put graphs together to

compare them, and, you know, we work.

But the ability to do that is incredibly
powerful for designing better trials.

But we wanted the ability that this is
inherent within FACTS, a cloud-based

software program that has these
incredible capabilities to do this,

in-incredible capabilities in software.

So really about the, the, the
start of this year, we said we're

jumping in and we're redesigning
this, we're reprogramming it.

It's unbelievable how fast.

Now, the software team does an
amazing job, but how AI has made

this incredibly fast, incredibly
powerful, uh, within that.

Now we're talking about bringing
agentic AI into this, where you

go into a AI tool and you say,
"I'd like a trial with four arms.

My endpoint is dichotomous.

We're looking at effect sizes of this.

This is what I'm trying to optimize."

It goes in, it does the simulations,
it creates the information for you, it

creates a summary Of the various trials.

I, I mean, this is all--
this is not future.

This is you know, we, we need this stuff
and it's right, it's right in front of us.

All of these things and the ability
to have a large language model that

you interact with that has the worlds
of information at its fingertips to

use for this, it, it's incredible.

I, I, I wanna make reference
to one other program that is…

It's a, um, RFI that's out there,
RFP that's out there for ARPA-H,

Advanced Research Projects.

And you might have heard of DARPA
if you're unfamiliar with ARPA-H.

DARPA is defense-based, um, uh, advanced
research projects and, uh, I, I'm

sorry, I may get this wrong, but you
know, the, the internet was advanced

in this, GPS was advanced in this.

All kinds of, uh, you know, moonshot
things, uh, risky advanced projects.

They don't all work.

Uh, and I think many of them probably
fail, but some of them work and they make

a massive-- they have massive impact.

So that's what the idea
here is for ARPA-H.

They put out a, a call called
SURPASS for adaptive platform trials.

They want them to be
continuous monitoring, phase

one, phase two, phase three.

They want the whole trial to be a,
a agentic AI driven to writing the,

uh, uh, appendices, the sub-studies,
the master protocol, the SAPs, for,

uh, curating the data, for running
continuous, um, uh, adaptive analyses,

all automated throughout this.

Another part of this, and, and this is
sort of the striking thing that it, it

wants to get rid of this idea that we en-
we, we enroll so many placebos in trials.

We relearn the same thing
over and over and over again.

Yeah, it's the definition of insanity.

So they want, as part of these trials,
that you build a digital twin of a

control, and the controls you start
to enroll, uh, feed that model.

Then the trial starts to utilize that
information and reduces the controls.

I mean, it is…

I- I- it's, it's the future.

Uh, but it's now.

It's incredibly exciting.

It's this idea of the whole wealth of
inno-innovation going that direction.

Now- Incredibly exciting, but
what, what, what's the starkness?

Well, last week, Kurt and I reviewed
the EMA reflection paper on platform

trials, and I, I don't want this
to be a reflection on the EMA, um,

a reflection on the reflection.

But in that…

Go read that.

Go read ICH E20.

The reflection paper, for example, pushes
back on non-concurrent controls, and I, I

know you know that this is something that
triggers me, that the placebos enrolled

in the same platform trial six months
earlier, they are so incredibly valuable,

and the idea that you ignore them in a
platform trial when you have multiple

arms that overlap and you can estimate w-
was anything different at that time, and

you've got bridging completely across.

It's a completely connected network.

It's the least risky thing, uh, out there
in terms of designing better trials.

But they say, "We don't s-- we don't
recommend non-concurrent controls."

Uh, yeah, only in rare
circumstance kind of things.

Contrast that with ARPA-H, where
they're recognizing we've got to do

things different in the world, and the
technology and the innovation is there.

ARPA-H says, "We want response
adaptive randomization.

We want this, this virtual
patient controls, and stop putting

so many patients on placebo."

And then you have this
reflection paper where we don't

support either one of those.

And ICH E20 says complexity is bad.

It, it really lays out that doing
two or three interims in a trial,

uh, well, that's too complex.

That's an exploratory trial now.

Uh, I-- it's just such a stark
thing, this striking conservative,

uh, uh, aspect of part of the world
that I interact with, and then this

strikingly fast-moving innovation,
moving in this exciting direction.

Let's grab a hold of this, let's control
it, let's drive it kind of thing.

And statisticians are right
in the middle of both.

I'm not gonna say anything pejorative
about anyone in either one of them.

I think there are people riding this wave
that I disagree with, and I, I think this

is a bad idea kind of thing, and this is,
you know, th- there's no content there.

It's all hat and no cattle for us
in, uh, in Texas sort of thing.

But there are ideas there that
there is something there, and then

there's this stark conservatism
that I want five-year simple trials,

simple data to make decisions.

And, and the world just can't be that way.

So it's a s- incredibly exciting time.

Uh, and, and I've been doing this for,
uh, 26 and a half years, uh, at Berry

Consultants and, you know, innovating
in adaptive trials and platform

trials and basket trials, uh, using
Bayesian analyses in clinical trials.

And innovation has moved.

In 26 and a half years, we've
done some exciting things.

The world has moved, but now
it's, it's, you know, the second

derivative is really high.

I mean, we are going fast all of a
sudden, and it's incredibly exciting.

You know, it…

The world in a year is gonna, you
know, be the speed of what used to

be 10 or 15, 20 years perhaps in it.

So, um, it, it's, it's really
exciting, and it's sort of this stark,

uh, uh, difference in the pockets
that I interact with in, in my job

All right.

So keep the emails coming.

Shoot me questions,
[e-mail].

Shoot me questions.

Uh, hit, uh, LinkedIn.

I'm on LinkedIn.

I don't know how to reference myself, but,
uh, you can probably find me on LinkedIn.

Send me a LinkedIn question on that.

Love to address the questions.

All right.

So thanks for joining me today.

Thanks for listening, and
thanks for sending in questions.

Until next time, we'll
be here in the interim