A Visit with Michael Harhay

In this episode of "In the Interim…", Dr. Scott Berry speaks with Dr. Michael Harhay, Associate Professor at the University of Pennsylvania and Director of the Center for Clinical Trials Innovation. The conversation explores Dr. Harhay’s progression through neuroscience, philosophy, epidemiology, and statistics, examining how this academic path shapes his work in clinical trial methodology. They discuss the Center’s role in addressing unresolved methodological questions arising from pragmatic, health system-based trials, including challenges with cluster and factorial randomized designs. The episode focuses on statistical and conceptual issues in endpoint selection for critical care, such as the analysis of informatively truncated outcomes, composite endpoints including organ support-free days, and the application of the win ratio. The increasing use of Bayesian methods in trial design is addressed.

Key Highlights

• Dr. Harhay’s academic background and transition into clinical trial methodology at Penn.
• The mission of the Center for Clinical Trials Innovation to support methodologic research and training, particularly among statisticians participating in multi-center health system trials.
• Discussion of hospital-level and provider-level randomization strategies in cluster and factorial designs within health systems.
• Ongoing challenges in analysis of composite and informatively truncated endpoints, especially in critical care, exemplified by ventilator-free and organ support-free days.
• Evaluation of analytic strategies including survival average causal effect, composite endpoints, and the win ratio, with emphasis on the need for clinical rather than purely statistical weighting of outcomes.
• Consideration of the conceptual strengths of Bayesian methods and their integration into modern trial design and decision analysis.

For more, visit us at https://www.berryconsultants.com/

Creators and Guests

Host

Scott Berry

President and a Senior Statistical Scientist at Berry Consultants, LLC

What is In the Interim...?

A podcast on statistical science and clinical trials.

Explore the intricacies of Bayesian statistics and adaptive clinical trials. Uncover methods that push beyond conventional paradigms, ushering in data-driven insights that enhance trial outcomes while ensuring safety and efficacy. Join us as we dive into complex medical challenges and regulatory landscapes, offering innovative solutions tailored for pharma pioneers. Featuring expertise from industry leaders, each episode is crafted to provide clarity, foster debate, and challenge mainstream perspectives, ensuring you remain at the forefront of clinical trial excellence.

Judith: Welcome to Berry's In the
Interim podcast, where we explore the

cutting edge of innovative clinical
trial design for the pharmaceutical and

medical industries, and so much more.

Let's dive in.

Scott Berry: All.

right.

Welcome everybody.

Back to

In the Interim.

I am

your host today

Scott Berry

and

I

have a guest with me today,

and we've got a number
of topics to get to.

But first, let me introduce our
guest for today, Michael Hoge,

who is the As is an associate

professor of Epidemiology

Medicine

and

statistics

and

data.

sciences.

There's a lot of, and in there, and,
we'll, we'll, we'll talk about that at,

By the way, the university

of Pennsylvania at Penn, uh,

Mike Michael has

an interesting history,

at Penn.

Uh, we'll go through that.

he's also the founding

director of

the Center for Clinical Trials Innovation,

which specializes in statistical
design, analysis and interpretation of.

Large randomized trials
and cohort studies.

We'll talk more about that.

But Let's.

first talk about how
you got here, Michael.

Uh, You have a PhD in epidemiology
from Penn as well as a master's

in statistics, uh, uh, as well.

So, uh, you have

multiple degrees from Penn.

So let's talk a little bit about

how

you got here, a little bit about
your history to this point,

and welcome to in the interim.

Michael Harhay: great Scott.

Um, thanks you so much for having me.

It's a real pleasure to be here.

Um, a big fan of the podcast.

Um,

yeah, so I've written a
lot of personal statements.

I've told this story a
lot of times and, um,

I wish it was a little bit more
romantic and straightforward, but.

I've always kind of, uh, believed in
this concept of a calculated drift,

and I think my calculated
drift towards Academia

started, um,

as freshman, sophomore year of college.

I went to a small liberal arts school
outside of Pennsylvania for those, um, not

from the us It's has about 2000 students.

The name was Berg.

Um,

and I just really enjoyed it.

I like being in an academic setting.

Um, I majored there in neuroscience and
philosophy and very much wanted to stay.

And do more academics.

Um, and I like the multidisciplinary,
um, opportunity of being in a university

setting where I could dabble in the
sciences and the humanities and philosophy

and I didn't really know how to do that.

Um, so some Googling I found Penn's
bioethics program, um, which saw

some bio related to neuro, and I saw
some ethics related to philosophy,

and it seemed like a good next step.

Um, so I came to Penn, um,
the year after graduation.

Um, so this is 2006 at the time,
and I quickly found out that you

could pay for your graduate degrees
if you were a full-time employee.

So I got a job

in Penn's pulmonary and
critical care division.

Um, basically working on patient
registries for pulmonary hypertension.

And

I just started taking classes and
learning how medical research worked.

Clinical research worked, um,
alongside a bioethics degree.

I was really hooked on the production
of data and how data was used.

Um, and I didn't quite know how to
transform that, so I just kind of

kept on taking classes over multiple
years until I met a young fellow

who had transitioned onto faculty.

And I joined, um, I started my
PhD in 2013 in epidemiology.

So during that time I picked up a
couple degrees, um, in bioethics

public health and demography, and then.

Ultimately my PhD, but a lot of that
I think was just me experimenting

with different ways of doing human
and human level and population

level research and trying to
figure out what I found attractive.

Scott Berry: uh, interest.

So, so at what point in there.

Did the quantitative side of
that, you had philosophy and

bioethics and it

sounds like initially maybe in

the the pulmonary and

critical care you were doing some data

stuff, at what point

did

the quantitative side

of

this interest you?

Michael Harhay: so I was,

always handing off data sets

and then they would come back from a PhD
level statistician where collaborator and

there are always gaps where I was kind of
asked to do smaller things, um, in Excel.

And I think I increasingly just
started to teach myself how to

code, um, first and formally.

Then I started taking,

um, classes in 2009 where I
was first exposed to data.

Um, I know some users and
listeners may not be a big fan,

but I just really enjoyed coding.

Um, and then I joined an analytics
group in 2011 where I started to really

get to work with large data sets and.

I loved it.

I enjoyed nothing more than
waking up, making coffee and

just coding throughout the day.

And, um, I purchased a bunch of books
back then and I increasingly just

got really interested in transforming
data, extracting things from it.

Um, but very aware that I needed some
type of formal training to really

translate into a set of tools, um, that
I could actually extract things from.

Scott Berry: Interesting.

So, so if, if we're, if we're counting
here, just to make sure I have this right.

So you have four Master's
degrees and a PhD.

Do I have that right?

Michael Harhay: Yes, that's correct.

No more for now.

Stopping.

Scott Berry: Okay.

Okay.

Uh, alright.

We'll never say never in that.

All right.

So I asked you how you got

here

and now.

So let's talk about here.

What, what, what is here.

so you're

a, an associate professor of
Epidemiology and, and you founded,

now, uh, upon reaching

Tenure, Tenure, you founded.

the

Center for Clinical Trials Innovation.

Uh, tell me a little bit

about the center for
Clinical Trials Innovation.

Michael Harhay: Yeah, so it's very much.

A culmination of about 10 years of being
exposed to different topics and trials.

Um, so during my PhD

um, in epidemiology, I
had a little bit more

of an apprenticeship type where
I worked under a physician

scientist and MD PhD, um, who ran

both behavioral trials but also
comparative effectiveness trials.

um,

and uh, behavioral trials tended
to be large multi-arm, uh, trials

with multiple different con contrast.

And then we had a couple.

Critical care based trials

in, um, our healthcare system.

And both of them kind of brought me
into a lot of different questions,

um, that we can return to, but

some of them

being, um, just dealing
with multiple intervention

studies and multiple comparisons and

thinking about what the
risk of false answers was.

And that always kind of bothered me
when We were doing behavioral things.

Um, and then on another side,
uh, we were doing studies in our

healthcare system and there's inherent
clustering, um, inherent time factors.

There's also fixed sample sizes, um,
and you can't really control them.

So you're trying to think about how to do

things best.

And then another area.

of interrelated research that, um, I
spend a lot of my time thinking About

that's become a big driver in the

center is how to deal with, um.

Not fully observed outcomes where we
call it informatively truncated outcomes.

So

in the ICU, we're very interested
in questions like duration and

mechanical ventilation or how long

an individual is in the ICU, but if a

percentage of those people die,
you don't have kind of a clear

treatment effect estimate comparison,
because for some people time

in the ICU is time to death.

And for other ones it's time to discharge.

And at the time I didn't really

have the ICH nine framework for, um,

the.

Intercurrent events.

So a lot of my dissertation was
trying to make sense of that,

um, in a more, um, immature way.

But

after that, I did a, um, K
99 or zero zero award, which

w um, supported my
postdoc and then became my

um, early career faculty award.

Um, it supported me my time
at Penn and during that time I

started to build out the work

on, um, intercurrent events and the s
demand framework became more popular.

And I started to

get involved

in more formal

ways, um, in trials

increasingly as like an MPI in
a trial where I would work, um,

as a kind of methodology
lead with a clinical lead.

And that started to lead to a lot of
research questions that we didn't really

find answers for in the literature.

And I can talk more about
them, but over the last four

or five years as an assistant
professor, I started to feel like

We were trying to answer questions where
the methodologic guidance for how to

answer those questions wasn't clear.

And what I wanted to do with the center

was to kind of create a way
to answer those, but also,

um, to support a training hub

where we could train young
academic trial methodologists.

Um, it tends to Be

mostly statisticians as far

to these real world questions and
to kind of advance their careers

and, um, that's what I'm trying
to put together at the moment.

Scott Berry: Very nice.

So I, I,

want to

come back

to this question of measuring
things, uh, uh, critical care.

Uh, very interested in, in
endpoints, in critical care,

the, uh, about the center now.

So you have

Uh,

dedicated

uh, personnel

to this.

You have other individuals
at Penn that are part of this

center, so this is just a, a

relatively

new center, but you're looking to expand
its impact and its role, it sounds like.

Michael Harhay: yeah.

Um.

I think it's a little bit of, it kind

of existed in a non-formal
way on one hand.

And the other hand is like,
if you build it it will come.

So, um, I have a couple, we have

two methodology grants.

One's an RO one from the NIH and

one pcori, and

that supports like formal statistical

methodology.

We mostly focused on this

question of intercurrent
events and composite

outcomes.

And then

we have several trials

that we support.

Across the continuum.

Some just being in consult, uh,
consultative support and others

where I'm the formal pi and we
have one of someone on my team

serving as a primary statistician.

Um, and these are all
health system based trials.

So I've been using those different

funding mechanisms to
kind of build a base.

So we have a

formal PhD staff,
statistician, two postdocs.

And then a lot of collaborators who are

really into the calls, um, and are

helping me bring on mentees and

also bringing on

their own trainees in different stages.

So

it's starting

to become a little bit of a
critical mass and gets them inertia.

Um, and it, it's kind
of really cool to watch.

I don't

know exactly

what it will become, but there's a
lot of people who want to do more

innovative health system trials, and
that's become very much our focus and.

Increasingly, there's just new
opportunities and new, um, venues

that We can expand ourselves.

And I'm, I'm, very strategically
trying to find cool internal partners.

Um, and it's been a lot of fun to kind
of create and just kind of be creative on

a vision, um, like something like that.

But it's also

to calculate a drift concept.

Scott Berry: Yeah.

Yeah.

Uh, very nice.

So I

I want to, I.

I I am

very interested.

in this idea of healthcare based,
uh, uh, health system based trials.

Tell me a Little

bit about

what that means for Penn.

Do you have a set of hospitals

and you're doing comparative
effectiveness within this?

So

it's part of caring for the
patients, but also learning

what, what, what what is available

for Penn?

Michael Harhay: Yeah,

so we have several trials within
Penn, um, and also we collaborate

with several external partners.

So

let's start with Penn because
it really, I think, highlights.

Where I think the innovation
kind of comes into what we do.

So Penn Medicine,

um, has six

hospitals, uh,

depending on how you

count about 40 exclusive

primary care practices and then
a handful of other practices.

There's obviously outpatient
clinics, all different types of

things in the greater Philadelphia
region, and that is a remarkable.

Opportunity for experimentation.

Um, you can improve primary care
practices, you can improve in

hospital practices, you can improve

ED delivery.

And we have multiple
different ways to experiment.

We also spread across wide
geographic areas with different,

um, patient populations.

We're serving different

patient populations, so going
out towards rural Pennsylvania,

going into more dense, um.

Northern New Jersey area.

Um, and then obviously have
different practice variation.

So we team with

a couple different groups.

At the moment,

it's principally critical care,
primary care practices, and we have a

specific center called the Center for
Healthcare Innovation and Transformation.

That also is collaborating across the
health system, trying to help different

service lines, um, like obstetrics

or um, oncology improve.

Or do things that they believe
will improve the delivery of

their care or just sometimes be,
have a good return on investment.

So I can give you a couple
examples of some projects we've

done that I think are kind of

neat in that vein.

Scott Berry: yeah,

no.

So

that is that I, I think this is actually
the future where if you think about

it from a perspective, our who do we

learn from about,

right ways to give treatment,
what treatments work or not.

We

learn from such a,

small fraction of patients.

Uh, in

Clinical trials,

randomized said,

but

yet we're, we're, these

decisions are

being made every day.

and I think

critical care is

a really nice

area

and

and I'm,

I'm echoing

what other clinicians

tell me.

I'm not an ICU doc,

but

largely

care

is random.

There's, they, they don't
know a great deal about this.

They're individually perhaps learning
at a patient level, but None of this,

uh, there's a

huge

set of things that are done
that we don't learn from that

we

could potentially make this
a learning healthcare system.

We could treat them better.

We could learn, uh, extend those.

I was involved with UPMC, which has a
large extent of, of hospitals west of you,

uh, Pittsburgh based.

But even through that,

Where even in elective surgery,
there's a great deal of things that,

That they

really don't know, uh, about how

to care for nausea and
vomiting, how to, you know, what

should you do before surgery?

all of

these things that could be
learned as you treat in that.

So I'm fascinated.

You said you had

a few examples of

that.

I'm fascinated by

what some of those examples look like.

Michael Harhay: Yeah,

I think some of them are so
elegant and clever and they

have such profound changes.

So,

um, I'll I'll start with them and we can

Go through them in a different
depth if you find 'em interesting.

So we just had one paper, um,
published in JAMA Surgery where the

idea was to change how many opioid
pills the EHR, um, default ordered.

So if you come into the ER

and you have a major injury, the

default

number of opioids may just be 30.

So we changed it to 10.

And then asked how frequently do
people come back for new medications?

How often do people complain?

Um, that it's insufficient.

And we found that, generally speaking,
very few physicians overrode the change

of the default prescription number.

And we didn't have a large number
of complaints to the ability we

were able to follow up patients.

So

opioid um, use is a big challenge
in the Philadelphia region,

and one of the goals is to keep opioids.

Where they should be ideally,
um, and to those who need them.

So simple kind of default
changes in our EHR are something

we try to leverage a lot.

We've

used similar mechanisms to increase
the delivery of palliative care.

Also to make the delivery more targeted.

So it is a finite
resource and you need to,

um, send

the physicians out to where
they can ideally be most,

um, useful and have the largest

impact.

We've used it to

kind of triage people.

Um, in our primary care practices, we
have used it to, if someone's come back

several times and just has a high blood
pressure that's not coming outta control,

maybe we need to identify them as someone
who's high risk for genetic disorder

and should be screened for
a different type of therapy.

So we are

trying to find ways to
improve, and this is not really

me thinking of the clinical ideas.

I'm kind of helping with the design, but.

These are the

questions we get is like, this
is something that we're not

doing in a standardized fashion.

It's kind of an easy win.

If we could do it better, can I test it
and create some evidence base around it?

Um, and people are really clever
clinicians, really see how the

things work, and it's really neat
to hear about the human computer

interface with DHR and ways to modify

it.

And I'm helping kind of put a statistical
paradigm around that to test it.

Scott Berry: So

interesting.

So the, the question of how many
pills is the initial order, the

default order, um, within that,

um, that's

done at a cluster level or

a hospital level, it

sounds like not a

Patient level.

Yeah.

Michael Harhay: Yeah.

So for

a lot of reasons, um, we tend to do
the vast majority of our research at a

cluster level.

It tends to be a practice
level or a hospital level.

Um, so we have six hospitals in our
system, or 40 primary care practices.

So we tend to do cluster
level randomization.

We also do provider level randomization,

though there's some, there can be
challenges of that in different fields.

Um, some just shift,

um, specialties just have the
way the patients Are exposed,

um, can potentially shift.

So

we're tend to mostly randomize
at the cluster level.

We do do some individual level
patient randomization, but

that's not kind of our, um,

It's a, it's a minority
of what we get to do.

Scott Berry: Yeah.

So I'm, I'm, I'm interested as you do this

in, in, a number of trials
that have tried to be,

embedded within the healthcare
system and, and that

there's a whole range of what

embedded means.

But in

that setting, once you're addressing
one question, you're collecting the

data and you're looking at it, it,

becomes.

I don't know, intoxicating is a
pejorative word, but it becomes

intoxicating to say, we've got

a

captive set

of

patients and questions.

let's

investigate,

two things rather than one.

let's do factorial things.

Let's do three

things.

You know, it's almost free to add
a third thing once you're doing two

and, and you know, four sort of thing.

Has

that.

happened within these trials?

Michael Harhay: Yes, because a lot of the
interventions we're doing are not usually

single component interventions.

They tend to be

multiple components.

And there's a couple scientific
questions in the multi-component

question, just by default.

So do all

the components need to be there?

Is it the existence of one?

Can we get a little bit of
a, an effect by adding one?

And then if we add a second one,
do we get kind of an additive

or a multiplicative effect?

Um, so we play a lot in cluster designs,
cluster randomized trial designs.

Just for listeners

who may be unfamiliar,
that means that we're

randomizing.

Groups of people.

So groups of people based on, um, what
primary care practice they come into, what

week they get their critical care, what

emergency department they come
in, in our healthcare system.

Um, so we are playing around with a

couple different component designs,
um, where we are trying to look

at all these different questions.

And

so, yes, a large number of questions tend
to be embedded in a lot of our trials.

Um, and it is largely to try to decompose.

Different, uh, contributions of different,
um, strategies to improve an outcome,

Scott Berry: Yeah.

Very nice.

Uh, well, what's

come up in a number of hours

as well

is.

Combining together cluster, randomized
with individual randomized in a

factorial way.

Have, have you run into that.

Michael Harhay: so we have
our first trial, uh, under

review at NIH where we have.

Cluster level randomization of
clinics and then, um, individual

level randomization within patients.

So we are just kind of

this, um,

two level cluster randomized
cluster randomization.

I Can't take credit for the design.

Um, one of my primary
collaborators and, um, my, uh,

cluster

trial, um, Oracle is fan, Lee
is a biostatistician at Yale.

Um, so he's a big.

Thought leader in

our group, um, and contributes
very heavily in our program.

So he's developed cool designs like that.

So I'm

very hopeful that when NIH funding
kind of comes back online officially,

that will be one of the, um, cool
things we get to really roll out.

Scott Berry: Hmm.

Very.

nice.

Very nice.

I I, you talked about endpoints, I
imagine in all of these endpoints

are, are interesting, but Let's

go back to,

the critical care scenario.

And I bring this up because my call

last night,

with.

we have a global call for.

REMAP cap.

The REMAP cap is a trial
that is a community acquired

pneumonia is the cap part of it.

And so we generally,

have hospitalized patients in, in ICUs.

And we, we, for 10 years now, I've
talked about what's the right endpoint.

Um, and, and it it's
continually the discussion.

So it sounds like you spent a good bit
of your dissertation on this question.

Of, of this And

still addressing this question.

today.

And it's Interesting.

You brought up the role of E nine

and, and,

and, all of this,

So let's

talk about critical care.

endpoints, um, within this.

Michael Harhay: Yeah.

I've spent, I think, uh, at
this point a quarter of my life

trying to solve this question.

So

Scott Berry: yep.

Yep.

We

see a lot.

of this question,

and

I,

I've kind of fought hard

that

even

they'll

ask questions about what is the,

what is the distribution,

of

length of mechanical,

ventilation for those that don't die?

So

let's set up a scenario
where it's intensive care.

You can go back to.

COVID and some individuals come

in and they're on,

me, uh, uh, life

support, let's call it.

And you're interested in
the length of life support.

You're doing interventions to them, and
you're wondering that and some patients

die.

during that process,

if

you're really interested, in the
length of organ support, what do you,

uh, what do you do?

with That patient,

within the setting?

and it's always struck me as

hugely problematic when you sensor

that.

out or you remove patients

that die, and you only look
at the length of organ support

and

you've pulled people, out that in fact are

the, the worst

Outcomes

Within that,

It's, it's a really hard question

without

somehow including mortality.

as part of that.

Michael Harhay: Yes.

I couldn't agree more.

Um,

you want me to respond

Scott Berry: I, I

want you to respond.

Yes.

Yes.

Michael Harhay: Yeah.

So, um,

I

this is, I think this question has, uh,

very profoundly shaped
me as a methodologist.

I think it shapes the way
I think about questions.

It's been very transformative for me, and
I, I can just give you some, uh, cliff

notes about that and you can tell me which

part you wanna jump in

most.

So, as a

PhD student.

At that time was

calls inference.

It was around of course, but I don't
think it was as popular as it was

today.

And the Esti mans framework, especially
in our current events, I, I, didn't

have a formal framework, so at the
time I was just thinking about it as

we have a distribution in front of
us, how do we model that distribution?

And it's, you know, tends to be a
little bit of a negative binomial

distribution kind of gamma, where you have

a bunch of people early who
have, are discharging in a Long

tail.

But then you have people that die.

So this really kind of
complicates the question.

So for people, as I mentioned
before, people who die, it's not

duration, mechanical novation, uh,
mechanical ventilation, it's time

to death.

And for those it's, uh, who survive,
it's their time to discharge.

So I spend

some time

exploring what are options.

And the big one at the time
was competing risk models.

And the challenge of competing risk
models, just like survival models,

is they kind of make this assumption.

About what would've happened if people
didn't have the event of interest.

And this is

counterfactual and obviously
people are not going to not die.

So it didn't really make conceptual
sense and my PhD mentor at the time said,

yeah, competing risk may be a modeling
solution, but it's conceptually invalid.

So then I moved on to this
concept of joint modeling, where

you would kind of model the,

um, longitudinal probability being on
where the daily likelihood of being on a

ventilator.

And then you can also model the,
um, competing events separately

and then pull them together.

And I remember my other mentor said, but
now we're thinking about a counterfactual

probability distribution and being
on a, uh, ventilator that would've

been observed if people never died.

And that was the first time I was
really exposed to this concept of the

hypothetical scenario for estimates.

And I never found it

conceptually, um, digestible.

So then I spent a few years kind
of getting into this estimate

called the survival Average

Causal Effect.

And I find a lot of attraction
here, but it has a limitation.

Um, so I can talk about that for a

Scott Berry: minute

yeah.

Yeah.

So, so let's

like de describe

what that, is.

Um.

Michael Harhay: So

it's

a clever estimate, and the idea is that
at the beginning of a trial, you have

these latent strata of individuals.

So the simplest scenario is you have

Some people that would always
survive to the end of the trial.

And let's just say this
is a 60 day follow up.

You have some people who are very sick and
are gonna die, and then you have these two

middle strata, and these are people who,
I'm sorry, that last strata is two people.

They would die regardless of.

They were exposed to the
intervention where to control arm.

Then you have these two middle
strata and You have some strata

that will only survive if they're
exposed to the intervention.

And then

some people, they'll be
harmed by their intervention.

So they would only survive.

They are, uh, ex um, given to control.

And what you want

to be able to do is identify

these always survivors, this four strata.

And these are these people that would
always survive to end the trial regardless

if they received intervention or control.

And the concept there is that if you can
isolate this strata people or stratum,

um, then you can say this is the average
causal effect among these individuals.

And I

think there's a lot of kind of
conceptual attraction there.

But the obvious kind of issue is
that you're no longer looking at

the full randomized population
and that raises issues of the ITT.

Um, it's

also a much smaller population
and critical care trials are

already small.

So

I like that.

And I spent a lot of my life developing
statistical methods to estimate that

' quantity.

cause I do think it's informative and
I think certain clinicians like it

and

it's.

Viable and attractive.

Um, just as additional information,
I think about it as like a

subgroup type of analysis.

Now

that's informative, but for
the last several years we've

increasingly been working on

composite outcomes, which is the kind of
last third, uh, intercurrent strategy.

And we spend a lot of time

trying to think about What is a
statistically valid, patient, relevant,

and clinically interpretable.

um,

Uh, composite outcome
and my colleague Chris

Scott Berry: Yep.

So, so let's let's
pause here for a second.

Uh,

and then we'll go to, to, to, to Chris.

So a

composite.

You talked

about hypothetical,

and so largely in the ES demand,

we're

trying to identify

what would've happened if this,
this intercurrent event of

mortality wouldn't have happened.

In order to model that, a composite
outcome is saying, We're actually

not gonna even make mortality

a an intercurrent event anymore.

It's

We're gonna make it part of the

endpoint.

That

Somehow we're gonna create a
single endpoint where mortality

is part of that endpoint and, and

that's making it a composite,
uh, uh, within that.

And so that, that's what you mean by that.

Okay.

So,

So, what might that look like

in an, in the

intensive care, what's a

composite strategy of mortality?

and organ support in the ICU?

Michael Harhay: Yeah, so I think
the most traditional one, um,

in,

I know you have used a variation of this

in some of the remap cap
platform and many trials do is

what we call ventilator free
days or a wider capture.

Its organ support free days.

And the concept here is that you
have a distribution of days over a

duration of period of time.

So let's just take 30 days.

And for everybody who survives, you

have the

number of days that those
people didn't require.

Mechanical ventilation, where some
other modality of organ support.

And then for

those who die, they tend
to get a fixed value.

variations of it have treated
that value as zero, like no organ

support free days.

And there's another
derivation that uses it as

negative one to kind of
demarcate that it's a separate,

um, level in this continuum
of the survival, um, the, the

observations that would observed
among survival, those who survived.

Is that a fair summary of it?

Scott Berry: I

I,

think that's a fair summary.

And so you've

got, you've

got,

um,

this

minus one outcome that a patient died.

Zero would

then be that the patient
survived, but They had zero days

free of organ support
for 28 days or 30 days.

They might have been on organ
support the entire time.

Somebody who's a

one.

Survived.

And they had One day,

free of organ support, uh,
and then two and then so on.

Michael Harhay: correct.

Scott Berry: And so

now that endpoint is a, composite.

it

includes mortality.

Uh, in the end point,

it includes us.

by the way.

This shows up in all kinds of

diseases.

We do

a lot of

neurological,

diseases.

You started in, uh, your, your
undergraduate in neurology, but

you can think of Alzheimer's.

where you look at a cognitive test
of an individual, and then you have

patients that die in that process.

And what do you do with

death?

And cognitive decline is, is
really the same sort of issue.

We Have,

this in acute stroke.

we have, individuals and we have,

their neurological status,
and you have some that die.

What?

How do, how do you do this?

so

in many ways

now

you've created a composite endpoint.

what

I've been worrying out a a lot about
is how do you analyze that thing?

So

Michael Harhay: Yep.

Yeah.

no, it's um, it's a numerical
or statistical solution.

That has, um, kind of just really to
me, intractable conceptual challenges.

Like how do you analyze that distribution?

So then it brings back to something
I mentioned a few minutes ago.

You can look at it

and you can say, okay,

we're, is it left skewed?

Right?

Skewed normally, streed skewed.

Then You can think about
like, well, do you want to

prioritize certain, um, shifts?

Do you care about the shift
between negative one to zero?

Um, it's just a lot of
questions that I have not really

been able to fully set on.

Um, and that has always brought me
back to the estimate framework where I

tend to look at these distributions
and say, okay, well now we solved the

issue of what to do with death, but
what do we actually want to extract

from this solution?

Um,

and I've never personally found the
ventilator free day where to organ

support free day to be kind of, um.

that attractive.

So over the last few years, um, we've
been spending a lot of time thinking

about this methodology of the win ratio,
which also has limitations, but I'm happy

to talk about some of that concept and
what I like about it a little bit more.

Not maybe tons more,
but a little bit more.

Scott Berry: Yep.

So, um, uh, my

issue interesting,

the organ support free days.

We use that in remap cap, and we had a

number of results that came out,

of it, and I think it worked very well.

We did a proportional

odds model across that,

and it worked.

very well.

And I'll say that because The
distribution of those on active or

placebo.

when the clinicians looked at it and the
statisticians say, these are different,

they

looked at it,

and they said, that's
clinically meaningful.

Uh, and they, they, they thought

they

could explain the effect,

and largely.

it's because in COVID, it was
a very homogeneous disease.

The effects.

were largely proportion across the scale.

They had the same effect on mortality.

They also shortened organ support.

There were some,

exceptions to that.

Uh,

so I think it worked because

at the end of the day, the
clinicians thought it worked.

Uh, in that,

the

huge thing I'm.

running into now is almost every,
the way everybody analyzes this.

There

is a statistical model that
comes out, whether it's,

the win ratio, whether it's the

proportional odds that that creates
a weighting of those values.

Mortality is weighted relative
to zero through a statistical

assumption, and that bothers me.

I don't

think that,

should be done.

I think

we

need to go all the way.

And this is very Bayesian perhaps.

Maybe Bayesian is not the right word,
but we need to clinically weight those

states and analyze them that way.

I think otherwise you're invariably
gonna end up with results you might not

actually believe are clinically relevant.

Michael Harhay: Yeah.

Um,

first.

I would

like to just at least
modify my last comment.

I may have not

spoken, so I don't, I have no

fundamental

problem of any approach to

composite outcomes.

I, and I agree with you.

They give you.

a net,

they give you a summary of the
net benefit of an intervention.

I find that generally attractive.

I've heard, I've heard you and other
people talk about this weighting

scheme and it's, it's a very

provocative thing.

'cause sometimes

I think we do think that
we do other things like

the win ratio and

circumvent this issue, but
we've just done it differently.

So.

Scott Berry: right,

right.

Michael Harhay: Your, point's well taken.

Um,

yeah, I, I, sorry.

Just thinking about like, all these
things get me really excited, so,

Scott Berry: so,

Yeah, no, it, it's

Michael Harhay: way to jump.

Scott Berry: exciting, right?

I, yeah.

Talk

Michael Harhay: to talk
about poor outcomes, but

I can feel this is like, to me, one of the
cooler and more challenging and like, um.

Such essential problems that
exist in clinical research today.

And as you mentioned, it spans so many
different disease areas and pathologies

and better methods are so important.

Um, and ways to, um, incorporate
stakeholder perspectives and weigh

that in so that we can figure
out weighting schemes that are

really meaningful to the
stakeholder distribution involved

is what Is such a kind of

unicorn and holy grail that, um,
we spend a lot of time thinking

about how to do, but haven't really
quite figured out how to do well.

Scott Berry: Yeah, and I, and I

feel like that's, such

a, a,

as

you started this,

does it get you out bed in the morning?

And I I feel like it's.

It's

where we become,

really scientists and we're using
quantitative things to tie to clinical

outcomes and that, that's exciting.

It's not a, a mathematical thing that we
do back in our lab, and it is really neat,

but it

doesn't really help patients.

Right, right, right.

I, I brought up the, the B

word

in a question.

I ask a

lot of people on here.

Do you

consider

yourself a Bayesian.

Michael Harhay: Yeah, so as I
mentioned to you in our email, like

I didn't get a lot of exposure to
Bayesian thinking before I was a

postdoc and really before COVID.

And it was innovators like you, who
really, I think brought Bayesian very much

out of the shadows into the mainstream.

And

when I started

first reading it, I was just,
I found it so intellectually.

Um.

Appealing.

It just makes sense to
me to think that way.

I like to think on a continuum.

I don't like to be, I realize that
there's a decision theoretical component

where you have to decide whether or
not something's efficacious or not.

enough to move on it.

But I

I like

the probability and the
posterior distribution.

Um, the other thing that I find very
attractive about it, um, this is a long

way a of saying, yeah, I'm a convert.

Um, I still do a fair amount
of like frequentist trials.

And I,

I

appreciate

a place for them and I, I,

I've heard other speakers and I would
like to think of myself as intellectually

nimble enough to kind of choose among
a set of tools for the task at hand.

But I feel like Bayesian fits
a lot of task at hand for me.

So I'm increasingly trying
to push people to do it,

And I also feel like it makes a lot
of sense, and this is where I was, um,

going with my answer, if I can, um,

tie it up, is when I was
working on a lot of behavioral

trials where we were trying to

reduce.

People from smoking by
giving them, um, some type of

financial

incentive or even

in our healthcare system where
we're just trying to nudge someone.

So you're looking at
two or three different,

to me, very low risk interventions.

Like what

is the harm of giving someone
the wrong EHR Alert, We're

giving them too much money

to stop smoking.

So I always thought about like, what

Errors meant

versus just wanting to know
if something's effective.

So I kind of

like the Bayesian framework
because I don't kind of like, I

don't find an intrinsic penalty
of a P value and minimizing P

value in alpha spending to be a

meaningful set of constructs
to guide me when I'm doing

inference in a lot of the questions
I'm asked to help people with.

So I think for a number
of reasons, I have.

circled back to a Bayesian
framework, um, to answer questions.

And I realize there's a lot of discussion
that could go on some of what I just said,

and there's a lot of details
in the new FDA report.

But globally, I find that lens
to be an attractive one to think

about the broad array of questions,
um, in a lot of different areas

that I'm asked to think about.

Scott Berry: Yeah, and especially given

all your work you've done.

in comparative effectiveness,
you know, we, we

have these

phase three

trials for

a new molecular entity and I get the 0.025

and a

a p value Makes sense.

But if you're comparing strategy one
to strategy two to strategy three.

The Whole.

idea of hypothesis testing
doesn't really work.

And

so within all of this setting,
this makes a ton of sense.

Yeah.

Well,

that's fabulous.

The

work you're doing sounds super
exciting, um, and and stuff.

that gets me outta bed in the morning
and very interested in, so I'm, I'm

I'm

glad that

that excites you as well.

I hope as this develops and this center

innovates, you'll come back on and tell

us about all the cool stuff.

You're,

doing, Michael.

Michael Harhay: it'd be my pleasure.

Scott Berry: Wonderful.

Um,

And

thank

you

for joining us here in the
interim everybody, we'll be here

next time on the interim.

Thank you.