A Discussion on Virtual Controls

In this episode of "In the Interim…", Dr. Scott Berry shares the framework he presented at a CSRC regulatory science hackathon on digital twins and synthetic controls in cardiovascular clinical trials. He separates three kinds of models that are often given the same name: population-level statistical models (a historical control rate), patient-level statistical models (propensity matching, Cox models, hot-deck imputation, and black-box machine learning predictions), and system-level digital twins of human biology. Drawing on ARPA-H's CIRCLE program for ICU immune modulation, the Framingham risk model, the EMA-qualified use of AI-generated prognostic scores as covariates, and Novartis's recent Lp(a) outcomes trial, Scott examines what each type of model can and cannot replace in a trial. Dr. Berry argues that virtual controls should provide a full predictive distribution for each patient rather than a single forecast, and that Bayesian hierarchical models with prospectively enrolled controls allow a trial to verify those predictions as it runs. He closes with how much smaller trials could become, and why clinical trial science must adapt as natural history models improve.

Key Highlights

• Reserving "digital twin" for system-level models of biology, not statistical predictions such as the Framingham risk model.
• ARPA-H's CIRCLE program: immune-system digital twins to guide ICU therapy, targeting a 25% reduction in ICU stay.
• Why a validated system-level digital twin would remove the need for a clinical trial.
• Prognostic scores as covariates versus virtual controls, and the counterfactual a control group is meant to supply.
• From a single historical control rate to patient-specific predictions built on patient-level data.
• Predictive distributions, not point forecasts, as the basis for comparing treated patients with virtual controls.
• Bayesian hierarchical models, MAP priors, and prospective controls to verify and calibrate virtual controls during the trial.

For more, visit us at https://www.berryconsultants.com

Creators and Guests

Host

Scott Berry

President and a Senior Statistical Scientist at Berry Consultants, LLC

What is In the Interim...?

A podcast on statistical science and clinical trials.

Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Well, welcome everybody
back to In The Interim.

I'm your host, Scott Berry.

And as, as you know, or i- if you don't
know, this is a podcast, uh, sponsored

by Berry Consultants, where we talk about
all things clinical trial science, medical

science, d- uh, developments in these
areas, and the, the intersection of these.

Um, now, I may…

We, we do this mostly weekly
episodes and a lot of them,

uh, recent topics that come up.

We love to hear…

I'd love to hear from you what topics
you'd like to hear from, from, from me

or a group here at Berry that spend a
lot of time on adaptive trials, Bayesian.

We, we think Bayesian.

We believe in Bayesian.

Uh, we, we, we practice Bayesian.

Uh, by the way, Berry Consultants, uh,
last year we looked at all the trials

we designed, and 50% of them were
Bayesian 50% of them were frequentist.

We design frequentist trials,
um, but we think Bayesian.

So if there are things out there
that you think would be good topics

for a podcast you'd like us to
talk about, that would be great.

By the way, this week I got a,
um, a, a request on LinkedIn

to talk about multiplicities.

What do Bayesians think
about multiplicities?

Uh, which by the way, is a really
hard topic, but, but one that, uh,

I do have one plans coming out.

We have a really neat paper, uh,
here on doing some things with

multiplicities, and really, it, it
touches on the view of, of, of a

Bayesian's view of m- of multiplicities.

So I'd love to hear what, what, what
topics you might find interesting.

Uh, another way to judge this out there
is various conferences or invitations

that we get to speak and, and what's
going on in the, the world of conferences.

I was actually at two
conferences this last week.

One of them was on statistical
innovations in rare diseases, and,

um, we've talked about a lot of the
concepts that was discussed there.

Very, very exciting, um,
uh, conference that went on.

The second one of them
was a conference called…

It was a regulatory science
hackathon, and CSRC hosted this.

Number of people from industry, academia,
uh, from, from, uh, FDA were there.

Now, what was the topic?

Uh, what?

Regulatory science hackathon.

What, what's the topic of that?

The title of it was Digital
Twins and Synthetic Controls in

Cardiovascular Clinical Trials.

So this whole topic and question of
digital twins, synthetic controls, and

how do they work within clinical trials.

So I was asked to give a
little presentation there.

I've, I've interacted
on a number of things.

I, I'm not gonna talk about anything
that was said at the conference.

Part of this hackathon is that what,
what is said there stays, stays there

and was thought to be more productive.

But I'll talk about what, what I said.

I can talk about what I said in this.

But again, wanna go back.

If there are certain topics
you would like discussed, you

know, things we may think…

you know, uh, don't think of
that you would think would be

interesting from that perspective,
would love to, to talk about those.

So My discussion, and I, I, I'm not gonna
go through and, and, and present exactly

as I presented at the conference, but I
thought I would go through the topics.

So I started off, and how I wanna
start off is, is thinking about what

is a digital twin, virtual control,
talk a little bit about these.

Uh, I'll talk a little bit about
things that have been done before

we started talking about something
called the digital twin, uh, virtual

controls, controls, historical controls.

How does this all fit in?

What, what, what are these things?

So a digital twin is a very, uh, specific
term given to a digital model of an,

of a real-world physical product.

So i- if you go to Wikipedia, a digital
twin is a digital model of an intended

or actual real-world physical product,
system, or process I, I'm gonna talk about

this as something that is specifically
around the mechanical parts of this.

So for example, in cardiovascular disease,
a digital twin would be a model of the

human cardiovascular system that you could
go in and alter something in this system.

You could decrease the LDL in that
digital model, and it would tell you the

cardiovascular risk in the next 10 years.

Now, we, we, we know something about this,
um, w- in, in it, and you could imagine

a digital version of a cardiovascular
human system that we alter Lp, or we

alter the weight of the individual.

We alter the characteristics, and it
would tell us something about the system.

And you can imagine in clinical
trials, we'd be very interested in

understanding what is the cardiovascular
risk in the next five years, say,

uh, of that, of that, uh, system.

Now, uh, an example of this,
and I'll, I'll present-- I wanna

differentiate that from other types of
things, but a very mechanical, um…

at, at, at a, at a small level, a
micro level of a particular system.

Now, ARPA-H came out, uh, and ARPA
is the Advanced Research Projects

Agency, and the H is for health.

There's DARPA, which are defense-related
ar- uh, Advanced Research Projects Agency.

There's probably others,
I'm sorry, I don't know.

But ARPA-H is specifically around health,
and this is-- these are projects that

are, I'm, I'm sorry for the term, but
moonshots, or they are somewhat risky,

high risk, high reward type, uh, projects
funded by the, the federal government.

Uh, certainly, uh, could be
funded by multiple organizations.

The one I'm gonna talk about here
is funded by the US Department

of Health and Human Services So
ARPA-H put out a call that they

referred to as the CIRCLE program.

Now, this project specifically
is looking at digital twins for

the treatment of patients in the
intensive care unit, in the ICU.

And they put out a call to build a
digital twin of a patient in the ICU and

test it for its ability to assign immune
modulation therapies for these patients.

The overall goal of this project,
the moonshot, if you will, is can

we reduce ICU stay by 25% or more?

Tied to that, save lives, improve the
outcome of patients in the ICU, and they

specifically talk about a 25% threshold.

So they, the, uh, and this, this--
They announced last week the rewards.

They went through an RFI and an
RFP, and they re- announced the

rewards, uh, September 22nd.

You can go on and find the, the,
um, uh, the announcements of this.

And they funded it up for $145
million total over five years.

So this is a big project, a big advance
project to, to see whether digital

twins of patients' immune system

Can these strategies improve outcomes?

So what does that mean specifically?

What they're looking to do in Circle is
to build a digital twin of a patient's

immune system, and then when a patient
is in the ICU, they would ask the

digital twin, would any of these, say, 15
therapies, steroids, tocilizumab, other

immune modulation therapies, and IL-23,
other things, most of these are already

approved, and we understand what these
therapies do to a human That the digital

twin would have a particular human comes
in and has biological characteristics

that would be measured at baseline
and then serially through their stay.

And you would ask the digital
twin, "Should we give a therapy?

Should we change our therapy?

How should we treat that patient?"

You can imagine a patient, and I, I'm
not an MD, so I'm a statistician, so I,

I apologize if I'm getting things wrong.

But you could imagine a patient has
very high CRP levels or something

within their system that puts them
at risk for potentially advancing to

sepsis or, or, or s- or bad outcomes.

And the digital twin looks at the
15 different potential therapies and

says, "Oh, if you give this therapy
to that patient based on the digital

twin, it thinks their CRP is gonna be
lower, and they're much less at risk

for bad outcomes, and they're going
to have better clinical outcomes."

And the digital twin says, "This
patient will not, uh, uh, g- come

down with sepsis," or, "They will not
have downstream sequelae that mean

they're gonna have worse outcomes."

And that's all within this
digital twin of the immune system.

Now, they're gonna run a clinical
trial to see whether that's true.

First, they're gonna do in silico
simulation, and they're gonna look

at multiple patients and say, "Aha,
if we would've intervened, could

we have had this 25% reduction?"

Entirely in silico, if the digital
twin is correct, does it have the

potential to reduce ICU stay by 25%?

Now, they would then hope to test this
in the last couple years of the project,

where you could imagine a clinical trial
that randomizes a patient to the digital

twin care or standard clinical care of
patients, uh, uh, w- the usual care.

Then we can measure specifically the
randomization to the digital twin.

Does it actually change care by this 25%?

So a lot of things going
on there within that.

But you can see within CIRCLE, that type
of digital twin is a model of the system

Now, you can see within clinical trials,
I come at this as a clinical trialist.

I am not an expert in creating a digital
model of the s- uh, a cardiovascular

system or an immune inflammation
system for a patient in the ICU.

Imagine a patient at risk at sepsis, a
patient that has had an MI, a patient

that has had other ARDS, o- other
things that you might be interested

in modeling their system within that.

I don't have any expertise in that, but
how does it go within clinical trials?

So I gave you a, an instance of how
that could be used in a clinical trial.

Now, there are ma- many
other types of digital twins.

I and we at Berry, we use digital
twins all the time, but it's a

different kind of digital twin.

It's a digital twin of a clinical trial.

So we build, and that's what Berry's
been doing for 26 plus years,

when we build a trial design,
we build a digital twin of that.

It models the entire system.

It models patients being
enrolled in the trial.

It models the outcomes being observed,
adaptations in the trial, whether

it's response adaptive randomization,
sample size selection, sample

size re-estimation, enrichment.

Whatever's happening in that
clinical trial, we build a digital

twin of that, and then we apply
things to that digital twin.

What would happen if enrollment
changes to that real system?

And we measure what would be the outcome.

What would happen if this drug
works in half the patients?

What happens to the real system
when we apply these thi- these,

these perturbations or, or
effects to the digital twin

that we've built of the trial?

We see how frequently does
statistical significance happen.

How frequently does the trial stop early?

How frequently does
each particular thing…

And we measure that.

So we build digital twins.

It's a real digital twin of a,
of a, of a particular system.

Now, there are many other of these.

I, I wear a WHOOP band.

Uh, it's a wearable that
measures various aspects.

It measures my sleep, measures my
exercise, measures my heart rate,

measures other things about me, and
interestingly, it gives me a digital age.

It tells me how-- what my biological
age is, and every week, based on

what I do during that week, it tells
me how my biological age has gone.

For example, it might
say, "You aged at 1.0

during this week," which is the, the
average age of a week is of course 1.0.

Or you average-- you aged only at 0.5

Now, a lot goes into that where it
believes by measuring my biological

parts, it has a system of the body.

It, it is a, a digital model that it
can tell me how much I aged during

that week based on the measurements
that it took during this week.

Now, I have no idea how good this is.

I don't rely on this in some sense
that now because I only aged 0.5

during that week, I'm gonna live 0.5

weeks longer.

I don't believe that, uh, but it's
a, it's a helpful thing to me.

Now, we wanna build digital twins
that do that, and how do they

interact in clinical trials?

Now The most pure digital twin
doesn't need a clinical trial.

By definition, it doesn't
need a clinical trial.

What do I mean by that?

If we were to build a really good system,
a cardiovascular digital twin, in the

sense of what a digital twin is, and
we say, "What happens if we give a drug

to a particular patient, Jane Doe?"

We can l- we, we can measure what
that drug does to its lipids, to

its LP little A, to the weight,
to all these things that…

A, and a much finer detail
of the cardiovascular system.

It'll tell us, is that a good thing?

What is the cardiovascular risk of that
patient now over the next five years?

And if we give a, a, a GLP-1 to a
patient for six months and their

weight goes down by 20%, say, and the,
that digital twin model will say…

And other things happen to the body, other
biometrics change, other characteristics

that the digital twin knows might say,
"You have lowered your cardiovascular

risk in the next five years from 8%
to 3%," I'm making up the numbers.

But that's right.

The answer, it's correct.

That digital twin is so good,
you don't need a clinical trial.

Now, you, need a good digital twin,
uh, to, to that extent, but that's

the, the, the interesting aspect
about a digital twin at the system

level, don't need clinical trials.

That's somewhat the whole
point to the digital twin.

Now, of course, we're gonna
wanna show that this works.

Like in Circle, we might do
one-to-one randomization to the

digital twin or a different form of
care to show the digital twin works.

Now, we might also do trials where
we enroll somebody, and the digital

twin says they have a 4% risk in
the next five years, and we enroll

a trial to see, do they have a
4% risk over the next five years?

To validate the digital twin
in some particular ways.

You can also imagine this being a
feedback loop to the digital twin.

Now, these digital twins are going to
be built very likely on observational

data, where they look to see
somebody that has this particular

characteristic, and then they look to
see what was their cardiovascular risk.

You can imagine we've got incredible,
we, uh, the community has incredible

data out there on GLP-1 trials, other
heart failure trials, uh, it- trials

on liver disease, MASH, many, many
trials out there that we could build

pretty good digital twins based on the
characteristics that are collected.

There's a Framingham risk model
that you can go online and ask for

your risk, and you can change one of
the things you inputted, like your

weight or, uh, are, are you, are you,
uh, what's your cholesterol levels?

Do you have high blood pressure?

All those things you could change
and see what it does to your risk.

If that thing's a really good
digital twin, it'll tell you what

your cardiovascular risk is Okay.

Now this whole idea of feedback loop.

I was at a conference a month or
so ago on reinforcement learning.

So reinforcement learning, and I'm not
an expert, and many of you are gonna know

much more, causal modeling approaches.

Uh, really interesting and trying
to read more and more about this.

And largely, these are models that might
be used to train an algorithm to play

chess or train, uh, uh, automated, uh,
self-driving cars to, to, to drive well,

not to hit things, to behave well, that
this reinforcement modeling is using…

I- is used.

Now they-- there's two really important
parts to this, what's called offline

learning, where you have a large
set of data and you run your causal

models to train something based on
this, and that's offline learning.

And then there's the online learning
part, where you're actively changing

something, and you're seeing the behavior
of it really to learn causal pathways.

If we change Lp, and yes, I'm aware
that several weeks ago, a Novartis

trial that looked at preventing
cardiovascular outcomes for

subjects with high Lp, a secondary
prevention trial, does it reduce it?

And they announced that
the trial was not positive.

I don't know the results of it.

I was not involved in it in any way.

But it brings to question, is altering

Lpsomething that has a causal
effect on cardiovascular events?

Now, it still might.

Might be the wrong patients, too late,
all kinds of interesting things to it.

Now, that online learning could be
fed back into the digital twin, so

it learns much more that not only i-
there's characteristic A and now you

come in and you have characteristic B,
is that a better cardiovascular risk?

But does changing it from A
to B change their outcome?

Which of course, is the critical thing.

And by the way, in the ICU, it's gonna
be the incredibly critical thing.

I don't think the Circle
project's gonna work.

I think patients in the
ICU are much too complex.

Now cardiovascular, I think we probably
have really good models of it, that we

understand this a lot better than the ICU.

Now, eventually we will have
incredible digital twins of

cardiovascular systems and ICU.

When is eventually?

Uh, I, I don't, I don't know.

Now, that's not necessarily
my point to this discussion.

My point is more how do
clinical trials w- do this?

Uh, what, what impact do these
things have on clinical trials?

Now, that, that was focusing
on the sort of system level.

What is a synthetic control So
I'm gonna call a synthetic cr-

control not system modeling.

You could imagine a digital twin of
a system level having John Doe come

in, and it gives a prediction of
what's gonna happen to that patient.

A digital twin could do that.

But there are many, many other things
that I don't think it's appropriate

to call them a digital twin.

I think we should reserve digital
twins for system-level modeling.

Something that's a mathematical
prediction of a patient, I

don't think is a digital twin.

I don't think we should call the
Framingham Risk Model a digital twin.

I think it's a mathematical prediction.

We have a lot of models that do
mathematical prediction, and they're

not system-level detailed modeling.

Now, this is a s-- uh, uh, think
about it as a statistical model, a

mathematical statistical model based on
a, a relatively few covariates, 20, 10.

I think of a digital twin that
might have hundreds if not thousands

of characteristics it's modeling.

So based on that, can we give
a mathematical prediction of

feels, functions, survives?

Now, um, um, there's-- what
could we do with such a thing?

The, the EU, and you can go on and look
for, uh, announcements, the EU, um,

talked about and qualified a framework for
AI-generated prognostic digital twins, and

specifically, this is a company, Unlearn.

I don't have anything to do with Unlearn.

But they got this qualified by the EU.

It is a mathematical prediction.

Now, a lot of it is proprietary, and
they don't tell us, and they make

reference to a deep learning model.

I think they largely have datasets
of demographics, and they're making

predictions of a patient's, whether that's
an Alzheimer's patient, an ALS patient, a

cardiovascular patient, and they generate
a single prediction of that patient.

Now, what do they do with it?

How might we use that in a clinical trial?

That has been qualified to use as a
covariate So if I run a trial that is

one-to-one randomized active against
control, for every patient that comes

in, you use it to get a prediction
of what the 18-month, uh, CDR sum of

box is going to be for that patient.

You use that as a covariate.

You get the actual outcome in the
trial, and you estimate a treatment

effect where you put in a, a coefficient
for that covariate that you estimate.

So you might…

Your, your model might be off
a little bit, but it's equally

adjusted in the two groups.

Now, this is a good thing to do.

If that model's a good prediction
of the outcome, it's gonna

reduce the standard deviation.

It is not being used as a control.

It is not a virtual control.

So it doesn't fit within that
framework, but it's a way in

which it's used in trials.

Now, if we use other covariates like
baseline CDR, maybe some biomarkers,

other things, you might do just as
well as reducing the variability.

So this is not a topic on
whether that's a good thing to

do or a necessary thing to do.

But in some sense, it's
not a virtual control.

Now, r- let's remember, why
do we do clinical trials?

Why do we enroll a control
group in a clinical trial?

We enroll 100 patients
on a particular therapy.

We enroll a control group to
understand the counterfactual.

What would happen to those 100 had
we not given them the treatment?

That's what we'd like to know.

So what do we do?

We enroll 100 patients, and we
don't give them the treatment.

And a, a lot of stuff we do in
trials is to try to get them

to be the counterfactual of the
100 that took the treatment.

We do stratification of baseline
factors, so they have similar age and

severity and all things we think matter.

We also randomize so that their,
the counterfactual doesn't come with

it some systematic bias, that it's
a good 100-patient counterfactual

So that, that's why we
do a randomized control.

Now, what happens if we have a
very, very good prediction of

every one of those 100 patients?

We have a statistical model that
says this patient is likely to, to

glide-- decline nine points over nine
months in an ALS trial on the ALS FRS.

Or this patient is, has over the
next two years, they have a 9%

chance of having a MACE event

Without the treatment on a
reference treatment standard of

care, placebo, whatever it is.

If we have that, we don't
need a control group.

We know the counterfactual.

We have evidence of the counterfactual.

You don't need a control.

So there's a huge potential in
clinical trials that these mathematical

models, virtual controls could have
an enormous impact on clinical trials.

Now, for years, we-- Berry
Consultants has been around 26 years

and started this Bayesian thing.

We've been doing this for 26 years.

Now, the mathematical models we
had may not have been fantastic.

They were not patient-level models.

So not uncommon, there's a
previous trial that ran with that

reference control group, a placebo.

Somebody ran an ALS trial with
12 months exposure on a placebo.

Uh, we're doing cardiovascular.

They ran a placebo group in a five-year
cardiovascular risk trial and saw a

7% MACE event rate at, at two years.

That's on my slide, so let's go with that.

7% MACE e-event rate on two
years for the control group.

We never had patient-level data.

We had a publication that told us that.

That's a mathematical model that
in this new trial I'm going to run,

I'm gonna use that mathematical
model as a 7% control rate.

So if my treatment in that
new trial showed a 3.5%

event rate, we'd say, "Aha, we have a 50%
reduction-- uh, ha-hazard ratio of 0.5."

We reduced events by 50% because we
have a mathematical model of what would

have happened without the treatment

Now, that estimate, that 7%,
doesn't change depending on the

patients enrolled in that trial.

We hope they're similar.

Now, you can't separate the fact that
it came five years later or seven

years later, the temporal differences.

But that's all we had.

Had we had the patient-level data,
we would do different things.

So that is a mathematical model.

Now, at the end of the day, we're gonna
judge how, how, how good are these models?

Are the models good?

And by the way, are they good enough
that you don't need a randomized control?

That's largely the huge question here.

Now, that question of good enough
depends on a lot of things.

I'll, I'll, I'll come back to that
What's changing in the world now--

Uh, by the way, going back to that,
you don't have to accept necessarily

on face value that that 7% is right.

We've been designing trials that have
been unbalanced three to one or four

to one between treatment and control.

Now, that 7%, we can learn
with newly enrolled reference

group controls, placebos.

Do they have a 7% event rate?

If it starts to look like that they
are reproducing that 7%, we wanna

use the mathematical model really
strongly to enfor- to reinforce

the control arm and make the four
to one look like four to four.

And we set up statistical methods
that allow that to happen.

If we're see 3.5%

on the new placebo, we wanna discount
that 7% and don't really wanna use

it much because it appears it's
not a good mathematical model.

So we build clinical trials that have
Bayesian methods, Bayesian hierarchical

models, MAP priors, other things that
use that to the extent they're predictive

and the, the, the-- it's adaptive or
dynamic in learning if it's predictive.

Now, we might even change
the randomization ratio

if it's not predictive.

If we're enrolling four to one and we're
not predicting the controls very well

in that setting, we might need to make
this more like four to four in the trial.

These-- We've been doing these for
years and how that fits in with

this question of virtual controls.

Now, that's something that's
done at a marginal level.

We've always been doing
that at a marginal level.

Why?

Because we never had the datasets.

Twenty years ago, you couldn't get
data in ALS trials, in Duchenne

trials, cardiovascular trials.

Data was all cordoned off.

It was, it was proprietary.

You couldn't go and get the
data from that publication.

It's very rare that those
datasets were available.

It's been something in my time in clinical
trials, the-- my 26 years, it-- there's

been an explosion in the availability of
data, whether it's EHR data, quote unquote

real world data, clinical trial controls,
clinical trial actives, registries.

There's been an explosion
in the availability of data

and patient-level data.

So what changes when we have
patient-level data What it is,

is we now have a mathematical
model that's participant-specific.

It's not predicting
everybody in the trial is 7%.

That's what the old models-- And by
the way, we still do those, uh, now.

Um, but that's what that methodology
assumes, is that every patient is 7%.

Now, what we have is by having
patient-level data, two different

patients that come in will
have a different prediction.

One patient, John Doe, might have a
9% prediction based on their, their,

their variables, covariates, age, risk,
where Jane Doe has a 5% or a 3% risk.

So every patient that comes
in, you attach a risk to them.

Patient-level stuff.

What, what are those?

Propensity matching is exactly this.

If we were to match the three closest
patients in a large database to

John Doe, those three patients form
a prediction for John Doe if they

were taking that reference that
those three patients were taking.

And it might say 9% is the risk for
John Doe based on propensity matching.

That's a mathematical model.

We might do a, a large-scale Cox model
with a great deal of covariates, the

Framingham risk model, that gives
a five-year, two-year risk for this

patient based on their covariates.

Hot-deck imputation, where it is largely
like propensity matching that you,

you match, and you might even match it
based on changes over time of a patient

where you go get the five patients
that look the closest to John Doe,

and you use that as a control group.

That's a mathematical prediction.

That's a statistical model.

I don't think that's a digital twin, but
you might use that as a virtual control.

Now, there's this whole scheme of
deep learning, machine learning, AI,

neural networks, super intelligence.

M- I'll call those black box models
that when John Doe comes in, you

plug in the black box, and it
gives a prediction for John Doe.

And, you know, it's a statistical
model i-in, in, in my, in,

in my understanding of this.

Now, what's really interesting about this
is how do we use these in clinical trials?

It's a lot different than the 7%.

If you've got this, I think we're going
the wrong route with this a little bit.

I-- When that comes in, I don't
want a single forecast of John

Doe of what's gonna happen.

I want a distribution over that.

I want…

What you can do, if you've got a
black box model, give me a probability

distribution It could be 10,000
imputed values from that estimate.

I don't want the w-- a, a
propensity single match.

I want propensity-weighted
probability distributions on that.

Now, it brings about an interesting
thing in a clinical trial.

Suppose we're investigating a
novel therapy, and we were using

this virtual control, and we've
got patient-level predictions.

So John Doe comes in, and I've got a
distribution on how they're gonna change.

And let, let me change it a little bit to
ALS, a forecast of what's gonna happen on

their functional rating score mortality.

I've got a probability distribution.

And I've got 100 patients that I've
treated, and I've got 100 distributions.

That's a little bit different than a
clinical trial where our, our normal

thing is give me a control, give
me 100 controls and 100 treatments.

Now, what I could have is 10,000 x 100
controls, or every patient is evaluated

within the probability distribution.

Where did they fall?

Did they fall at the median?

Did they fall-- Wow, that patient
fell at ninety-nine percentile.

We do disease modeling where we
look at how much have we changed

the forecast of the patient.

And the estimand is a disease slowing or
an endpoint slowing or an endpoint effect

where every patient gives us an estimate
of that, and we evaluate those parameters.

Or we build a parametric model around it.

Or we could simulate ten thousand patients
from the probability distribution, and our

control group now is a hundred times ten
thousand, and we can do more traditional

things where we estimate hazard ratios.

It's a whole new thing in clinical trials.

It's bringing about a mechanism that
we haven't used much but is much more

powerful than something that doesn't
give us an, uh, uh, that uncertainty.

Why does that matter?

One patient may come in and we have
very good forecast of that patient.

And then when they have their
outcome, we can evaluate it

relative to that distribution.

Jane Doe might have huge variability
where their outcome ends up very

far from the predicted outcome.

But in terms of standard
deviation, it's not very far.

The Z-score from it is not very far.

That's hugely valuable in
understanding did the treatment we

give have an effect on the outcome

So really important in a whole new
science of comparing to a probability

distribution rather than 100 controls.

Now, we are still using, in the
clinical trial, Bayesian hierarchical

modeling to evaluate are these
predictions good predictions?

And again, we might do four to one
and the ability to enroll patients on

that reference control to validate,
verify, measure are my-- is my

statistical model predicting well?

Hugely valuable.

First of all, for
everybody to accept this.

If you know the model is
good, you don't need this.

But as we're creating these models in
the virtual controls, imagine going

to a regulator and saying, "You just
have to trust that this model works."

Or saying, "We're gonna enroll prospective
controls and we're gonna evaluate does the

model work, and you're gonna be able to
see that when we say seven percent, it's

seven percent or two percent or, or that."

Now, it also brings out a
really interesting thing.

Do we try to measure,
suppose, a bias in that?

That the model's pretty good, but it
tends to be off by a hazard ratio of .8

in both groups and adjust by that, and it
allows us to have a strong control that's

been temporally adjusted in the trial.

Hugely valuable stuff.

It's a way, by the way, to combine
together perhaps causal modeling

with online randomized learning.

This is where the world is going.

The potential impact of
this is absolutely enormous.

Might cut every trial in half, and it
does more than that because if you have

100 versus 100 in a clinical trial to
power a treatment, and instead you know

the o-- the-- a counterfactual prediction
of those patients, you might only need

70 of those patients because you have
a much more reinforced control arm.

Hundred versus 100 when you have the
variability of the 100 controls, but it

might be 70 versus 10,000 times 100, which
is, you know, uh, uh, a very large number.

10,000--

I can do 10,000 times 100.

I think that's a million.

Maybe I can do that.

Um, that it-- you can
reduce the 100 to 70.

Cutting trials by s-- uh, enormous
fractions across that is hugely

valuable, uh, in that setting.

Now, digital twins at the system
level, even more valuable that we

don't even need clinical trials,
and we're gonna get there Um, and

again, I don't know if that's 15…

By, by the way, we're gonna slowly
get there in different diseases.

We have some of them now.

We're gonna get closer and closer to that.

Um, uh, is it 10 years,
20 years, 50 years?

Imagine the unbelievable modeling
we're gonna have of a natural history

patient in many, many diseases.

It's gonna be so good that our
clinical trials have to change.

Uh, by the way, if the clinical
trial ecosystem doesn't change,

we're gonna be passed up, and we're
gonna be discarded, and it's gonna

be Google, Amazon, Apple, uh, y- you
know, many, many other organizations

that are gonna own the data and the
models that are just gonna pass by.

And so this is an incredible
way to work with that within

our clinical trial ecosystem.

Now, makes the job of a
regulator incredibly hard.

I, I, I…

You know, and it's gonna be harder.

But also, the ability to partner on
approving better treatments, better

patient outcomes for regulators,
developers alike is, is really,

really exciting, uh, going forward.

So I, I sort of stratified things
by population marginal level

statistical models, patient level
statistical models, digital twin

system level models, and how that does.

What does that do to our
clinical trial ecosystem?

I hope you're as excited
about this as I am.

I think the clinical trial
science is gonna keep up.

It's gonna be super exciting in this.

So I, I appreciate you joining
me for this discussion.

It was a great discussion with
all the stakeholders involved.

I think there'll be many more of
these within different diseases

within clinical trial science.

So I appreciate you joining
me here, and until next time,

we'll be here in the interim