WEBVTT

NOTE
This file was generated by Descript 

00:00:00.030 --> 00:00:03.650
Judith: Welcome to Berry's In the
Interim podcast, where we explore the

00:00:03.650 --> 00:00:07.670
cutting edge of innovative clinical
trial design for the pharmaceutical and

00:00:07.670 --> 00:00:09.880
medical industries, and so much more.

00:00:10.720 --> 00:00:11.450
Let's dive in.

00:00:12.360 --> 00:00:17.880
Scott Berry: Well, welcome back to
In the Interim, I'm Scott Berry.

00:00:17.880 --> 00:00:18.720
I'm your host.

00:00:19.500 --> 00:00:26.310
For today of, in the interim, I've a
interesting topic for today, I, I used to

00:00:26.310 --> 00:00:33.120
write a column for CHANCE Magazine and I
would quarterly write a column called the

00:00:33.300 --> 00:00:38.700
A statistician reads the sports pages,
and of course, consuming and talking

00:00:38.700 --> 00:00:40.770
about things that show up in sports.

00:00:40.770 --> 00:00:42.690
The statistical analysis of this.

00:00:42.690 --> 00:00:45.600
I did that for, uh,
about 10 years actually.

00:00:45.900 --> 00:00:46.080
Um.

00:00:47.115 --> 00:00:51.825
Um, in, in that, it was, it
was, um, was very rewarding.

00:00:51.825 --> 00:00:53.865
So I'm gonna do that today.

00:00:53.865 --> 00:00:57.045
And this is a statistician reads jama.

00:00:57.765 --> 00:01:02.595
So I'm gonna tell you about an experience
of opening JAMA and giving a read

00:01:02.595 --> 00:01:08.685
to relatively, uh, a random clinical
trial that I read the results of.

00:01:08.685 --> 00:01:12.255
And it, it, it sparked a number
of things I thought very mu,

00:01:12.255 --> 00:01:13.875
very much worth discussing.

00:01:15.225 --> 00:01:20.055
Um, by the way, i, I, for those of you out
there may, whether you're driving to work

00:01:20.055 --> 00:01:25.335
in the morning, the afternoon, you're out
for your daily run or however you consume.

00:01:25.335 --> 00:01:27.675
In the interim, I'd love to hear from you.

00:01:28.155 --> 00:01:31.790
Let me know what kind of topics
you would like to talk about.

00:01:32.490 --> 00:01:32.710
If.

00:01:33.250 --> 00:01:36.460
Particular people you'd like
on the on, in the interim.

00:01:36.850 --> 00:01:40.780
Love to hear what you'd like to hear
about here at Berry Consultants, we

00:01:40.780 --> 00:01:45.850
have a company of about 35 scientists,
static, mostly statistical scientists.

00:01:46.300 --> 00:01:48.820
We work on clinical trial design.

00:01:48.820 --> 00:01:52.210
We work on implementing,
uh, adaptive trials.

00:01:52.450 --> 00:01:55.240
We have software for simulating trials.

00:01:55.500 --> 00:02:01.710
We do a wide range of therapeutic areas,
and yes, we focus on innovative trial

00:02:01.710 --> 00:02:07.020
designs, adaptive trials, bayesian
statistics, and so we'd, I, I'd love

00:02:07.020 --> 00:02:10.920
to hear what kind of topics you'd
like to hear about on in the interim.

00:02:10.920 --> 00:02:10.950
I.

00:02:12.330 --> 00:02:13.530
So here we go.

00:02:13.530 --> 00:02:15.330
So this really happened.

00:02:15.330 --> 00:02:23.070
I got an email from JAMA and it, they,
in there, they list different, uh,

00:02:23.070 --> 00:02:26.520
articles and results of different trials.

00:02:26.760 --> 00:02:31.380
And unfortunately I don't get to read
these as much as I would like, but one

00:02:31.380 --> 00:02:34.470
showed up and I thought, you know what,
I'm, I'm just gonna read one of these and

00:02:34.470 --> 00:02:37.020
see what, see what these trials look like.

00:02:37.635 --> 00:02:47.205
So this trial is the Fair HF two trial,
and it was published, uh, the, the,

00:02:47.235 --> 00:02:53.625
the primary author is Anchor, uh, so
Anchor Etal, and it says, published

00:02:53.625 --> 00:02:57.165
online, March 30th, 2025 in jama.

00:02:58.290 --> 00:03:00.870
Uh, an or in an original investigation.

00:03:00.870 --> 00:03:05.370
So if, if at all what I talk
about is interesting, yes, please,

00:03:05.370 --> 00:03:07.380
please go check out the, the trial.

00:03:08.790 --> 00:03:11.520
Uh, I was, I'm not involved in the trial.

00:03:11.520 --> 00:03:14.220
I had nothing to do
with it, and I, uh, uh.

00:03:14.565 --> 00:03:18.255
Only vaguely know a couple of the
authors, so I don't, I don't have

00:03:18.255 --> 00:03:20.355
any involvement in this trial.

00:03:21.465 --> 00:03:23.775
So the, what is the question?

00:03:23.775 --> 00:03:24.705
In the trial?

00:03:25.155 --> 00:03:28.455
The trial, and it says
this right in jama, what I.

00:03:29.010 --> 00:03:40.290
Are the efficacy and safety of intravenous
intravenous ferric carbo, carbo x maltose.

00:03:40.680 --> 00:03:46.710
I'm going to refer to this as
intravenous, um, uh, iron supplement.

00:03:46.950 --> 00:03:48.900
Uh, the paper refers to it that way.

00:03:48.900 --> 00:03:54.960
So this is for in, in patients with
heart failure and iron deficiency.

00:03:55.830 --> 00:03:57.570
So is this.

00:03:57.885 --> 00:04:04.545
Intravenous iron supplement for
patients with heart failure and

00:04:04.545 --> 00:04:06.765
they have an iron deficiency.

00:04:07.245 --> 00:04:09.225
Is it effective and safe?

00:04:09.855 --> 00:04:12.195
That's the question in the trial.

00:04:13.275 --> 00:04:15.705
Okay, so very interesting.

00:04:15.705 --> 00:04:21.225
So the design, the settings, and the
participants, it's a multicenter trial.

00:04:21.705 --> 00:04:23.655
It's randomized one-to-one.

00:04:24.345 --> 00:04:27.465
With the iron supplement.

00:04:28.215 --> 00:04:33.435
Uh, in there patients with
heart failure defined as having

00:04:33.435 --> 00:04:39.435
A-L-V-E-F left ventricular ejection
fraction less than or equal to

00:04:39.435 --> 00:04:43.425
45% and having an iron deficiency.

00:04:44.070 --> 00:04:49.240
I'll let you if, if, if you know
what levels of iron deficiency are.

00:04:49.765 --> 00:04:56.815
Uh, I'll let you go to the paper to, to,
to see that by the way, they define a

00:04:56.935 --> 00:05:04.675
particular highly deficient group, and
I'll say something about that as well.

00:05:05.005 --> 00:05:09.115
But, so everybody's iron deficient
and there's a particular subgroup

00:05:09.115 --> 00:05:12.625
that is, is, uh, highly deficient.

00:05:14.190 --> 00:05:19.530
It enrolled at 70 clinical
sites in six European countries.

00:05:20.400 --> 00:05:25.050
Enrolled from March,
2017 to November of 23.

00:05:25.500 --> 00:05:30.930
Median follow-up of patients was
16 months, uh, in, in the trial.

00:05:32.070 --> 00:05:37.830
Okay, so again, any of the details,
please, uh, uh, read the article.

00:05:37.860 --> 00:05:39.780
So sounds very interesting.

00:05:40.110 --> 00:05:46.560
The trial enrolled 1,105 patients,
so rather large trial from that

00:05:46.560 --> 00:05:48.840
perspective, large and long trial.

00:05:51.045 --> 00:05:55.785
The primary endpoint in the trial,
and it's a bit sort of interesting and

00:05:55.785 --> 00:05:57.585
I'll try to lay this out a little bit.

00:05:57.975 --> 00:06:02.955
So the primary endpoint, it's
looking at cardiovascular death

00:06:03.045 --> 00:06:05.535
and heart failure, hospitalization.

00:06:05.625 --> 00:06:11.535
Very common endpoints in heart
failure trials, it, it's going to

00:06:11.535 --> 00:06:14.775
analyze three different primary.

00:06:15.945 --> 00:06:21.345
Endpoints where that's an endpoint
tied to an analysis in the trial.

00:06:21.795 --> 00:06:27.705
So the first one is the time to
first cardiovascular death or

00:06:27.705 --> 00:06:29.625
heart failure hospitalization.

00:06:30.135 --> 00:06:35.505
Again, a very common way to analyze
in heart failure trials is a time to

00:06:35.505 --> 00:06:42.135
event, so they're doing standard time
to event analyses for that endpoint.

00:06:43.125 --> 00:06:48.975
Additionally, they're doing an analysis
of the rate of total hospitalization.

00:06:48.975 --> 00:06:53.235
So this is, a patient could have
multiple heart failure hospitalizations,

00:06:53.355 --> 00:06:58.245
and they're using rather standard
analysis techniques of that for

00:06:58.245 --> 00:07:03.315
count data, negative binomial
analysis for count data, depending

00:07:03.315 --> 00:07:05.595
on the exposure the patients have.

00:07:07.035 --> 00:07:12.405
The third one is analyzing the first
endpoint that I described, which

00:07:12.405 --> 00:07:16.215
is time to first cardiovascular
death or hospitalization.

00:07:16.545 --> 00:07:21.105
But it's restricted to that
subgroup that I, that I de, that I

00:07:21.105 --> 00:07:23.505
described that are highly deficient.

00:07:24.960 --> 00:07:29.670
And that, so those are the three
analyses that are gonna define the

00:07:29.670 --> 00:07:32.310
primary set of analyses in the trial.

00:07:32.670 --> 00:07:33.150
Yes.

00:07:33.150 --> 00:07:37.830
They're gonna control the
overall family-wise, experimental

00:07:37.860 --> 00:07:42.480
error rate by, by analyzing all
three of those, uh, analyses.

00:07:43.560 --> 00:07:48.810
So sort of co-primary, if you
will, in the sense that any one

00:07:48.810 --> 00:07:52.470
of those could potentially be
successful and they adjust for that.

00:07:53.955 --> 00:08:01.275
Okay, so they use a Berg procedure
for analyzing those three,

00:08:02.265 --> 00:08:03.735
uh, I'll call them endpoints.

00:08:03.735 --> 00:08:08.805
And I, I struggle with this a little bit
because I like to think of endpoint as

00:08:09.525 --> 00:08:15.015
heart failure, time to heart failure,
hospitalization, or cardiovascular death.

00:08:16.350 --> 00:08:20.070
And how you analyze it or the
subgroups aren't really the endpoint,

00:08:20.070 --> 00:08:21.630
but I'll, I'll describe it that way.

00:08:21.630 --> 00:08:23.160
I think it'll be easier to describe it.

00:08:23.190 --> 00:08:30.600
So they are analyzing these three analyses
and in the analysis they've set up a

00:08:30.600 --> 00:08:37.115
procedure where they will refer to this
as statistically significant if any one

00:08:37.115 --> 00:08:39.150
of the following three things happen.

00:08:40.200 --> 00:08:44.580
If all three of those are significant.

00:08:46.035 --> 00:08:49.725
At, and I'm going to do
it in the one-sided sense.

00:08:50.200 --> 00:08:52.005
In, in, in one sided.

00:08:52.005 --> 00:08:53.625
This is all about superiority.

00:08:53.625 --> 00:08:55.755
They describe it in the paper two-sided.

00:08:56.325 --> 00:08:57.375
So I I'll do it.

00:08:57.375 --> 00:08:58.275
Two-sided, sorry.

00:08:58.455 --> 00:09:03.075
So if all three are less than 0.05,

00:09:03.285 --> 00:09:09.585
two-sided, then the trial's
statistically significant and

00:09:09.585 --> 00:09:11.265
they've demonstrated superiority.

00:09:13.275 --> 00:09:19.875
The second opportunity is that if
two of them are significant at 0.025,

00:09:20.385 --> 00:09:24.495
so that's half of the original,
but if two of them meet 0.025,

00:09:24.765 --> 00:09:28.455
then the trial demonstrates
statistical significance.

00:09:28.455 --> 00:09:37.290
Yes, if one of them, if any one of them
is significant at the point, at 1.67%,

00:09:37.320 --> 00:09:42.480
two-sided, then it demonstrates
statistical significance.

00:09:42.480 --> 00:09:46.350
So this is sort of three shots
on goal where they're looking

00:09:46.350 --> 00:09:48.450
at three different analyses.

00:09:48.720 --> 00:09:52.590
Again, time to cardiovascular
death or hospitalization.

00:09:53.790 --> 00:10:02.070
The the number of heart failure
hospitalizations and a subgroup where

00:10:02.070 --> 00:10:06.630
they analyze time to first, cardiovascular
death, or heart failure hospitalization.

00:10:07.350 --> 00:10:12.780
Okay, so the SAP is is, uh,
published as part of it.

00:10:13.110 --> 00:10:15.480
It's a very well written SAP.

00:10:15.900 --> 00:10:18.120
The design is reasonably standard.

00:10:18.120 --> 00:10:24.510
I found no evidence of any adaptations in
the trial, so they enrolled 1,105 patients

00:10:24.780 --> 00:10:26.790
and carried out the primary analysis.

00:10:27.030 --> 00:10:30.690
I'm sure there was A-D-S-M-B and
safety was reviewed, things like

00:10:30.690 --> 00:10:33.540
that, but a very traditional trial.

00:10:34.500 --> 00:10:41.430
Um, and, and a good trial, and I, I,
I'm going to talk about the publication

00:10:41.430 --> 00:10:43.170
of this and the results of this.

00:10:43.590 --> 00:10:47.280
The, the, the authors should
be commended on this trial.

00:10:47.280 --> 00:10:51.510
The patients involved in the
trial and they deserve praise.

00:10:51.510 --> 00:10:56.190
So I, I, I hope in, in no way
does this come across negative

00:10:56.490 --> 00:11:00.810
for the people who ran, conducted
and published this, this trial.

00:11:01.505 --> 00:11:03.905
But I want to dive into the science of it.

00:11:04.055 --> 00:11:08.225
I wanna push a little bit on the science
of it, and I want to give my reading

00:11:08.225 --> 00:11:12.965
as a statistician who randomly picked
up this article and, and read it.

00:11:13.505 --> 00:11:15.095
what my reading of it is.

00:11:16.205 --> 00:11:16.775
Okay.

00:11:17.105 --> 00:11:21.215
So the structure set up
what happened in the trial.

00:11:22.890 --> 00:11:27.180
So the trial enrolled 1,105 patients.

00:11:27.180 --> 00:11:35.160
Again, it was randomized, double blind
in the setting, the first analysis

00:11:35.220 --> 00:11:39.420
of time to cardiovascular death
or heart failure hospitalization.

00:11:39.630 --> 00:11:45.660
They report this as the, the, the
number of events per hundred patient

00:11:45.660 --> 00:11:48.300
years as just a way to summarize it.

00:11:48.855 --> 00:11:56.115
Uh, in the paper is in the, in
the treatment group, it is 16.7

00:11:57.225 --> 00:12:00.135
and in the placebo it's 21.9

00:12:00.135 --> 00:12:02.835
per a hundred patient years, so 16.7

00:12:02.835 --> 00:12:04.035
to 21.9,

00:12:04.875 --> 00:12:06.045
140.

00:12:06.045 --> 00:12:11.295
One of the patients of 558 on
the treatment had an event.

00:12:11.940 --> 00:12:15.420
And 166 of 5 47.

00:12:15.420 --> 00:12:19.650
Reasonably similar sample
sizes, so 141 and 166.

00:12:20.550 --> 00:12:25.260
The hazard ratio and the time
to event analysis is 0.79.

00:12:27.030 --> 00:12:32.940
The two sided P values 0.041

00:12:32.940 --> 00:12:34.620
sided would be 0.02

00:12:34.890 --> 00:12:36.000
of superiority.

00:12:36.270 --> 00:12:38.310
The, the hazard ratio of 0.79

00:12:38.310 --> 00:12:39.690
is showing the treatment did better.

00:12:41.340 --> 00:12:46.920
And notice that doesn't meet
that endpoint by itself in the

00:12:46.920 --> 00:12:50.490
setting of, of the Huck procedure.

00:12:51.900 --> 00:12:55.470
If that would've been the loan
primary analysis, it would

00:12:55.470 --> 00:12:57.450
be statistically significant.

00:12:59.775 --> 00:13:01.875
Now what happened to the other endpoints?

00:13:02.475 --> 00:13:04.245
It, it met 0.05.

00:13:04.245 --> 00:13:06.165
So if all three meet 0.05,

00:13:06.165 --> 00:13:07.995
the trial will be considered significant.

00:13:08.715 --> 00:13:17.325
The total heart failure hospitalizations
had 264 in the treatment group and

00:13:17.325 --> 00:13:20.265
320 heart failure hospitalizations.

00:13:20.265 --> 00:13:21.975
Now that's adding across patients.

00:13:22.125 --> 00:13:22.845
It matters.

00:13:23.595 --> 00:13:26.865
Uh, how many patients have 0,
1, 2, and three, and so on?

00:13:27.555 --> 00:13:33.675
The relative risk in
that analysis is 0.80,

00:13:33.945 --> 00:13:35.505
0.80.

00:13:35.595 --> 00:13:37.845
Again, a benefit for the treatment.

00:13:38.445 --> 00:13:44.835
A 20% re relative risk reduction
in heart failure hospitalizations.

00:13:45.255 --> 00:13:48.765
The P value, the
two-sided P value is 0.12.

00:13:50.475 --> 00:13:58.965
In that the third, which analyzed this
subgroup of patients that, um, met this

00:13:58.965 --> 00:14:05.895
high need showed a and this was the,
the end point of cardiovascular death

00:14:05.895 --> 00:14:07.635
or heart failure, hospitalization.

00:14:07.635 --> 00:14:11.295
Time to first event showed
a hazard ratio of 0.79

00:14:12.225 --> 00:14:13.365
Also.

00:14:13.845 --> 00:14:16.695
Exactly the same as the primary analysis.

00:14:16.695 --> 00:14:20.865
The confidence interval's a little
wider and the P value's 0.07.

00:14:22.335 --> 00:14:25.155
So let's think back to
the Hawk Bird procedure.

00:14:25.785 --> 00:14:28.095
Do all of them meet 0.05?

00:14:28.095 --> 00:14:28.905
They don't.

00:14:29.055 --> 00:14:31.695
The first one did the other two didn't.

00:14:32.775 --> 00:14:35.295
Do two of them meet 0.025?

00:14:35.295 --> 00:14:38.715
No, actually none of them meet 0.025

00:14:39.315 --> 00:14:42.975
and none of them met 0.0167.

00:14:44.340 --> 00:14:49.200
So according to the primary analysis
methodology, the controlling of

00:14:49.200 --> 00:14:54.630
the experiment wide type one error
rate, this trial is not significant.

00:14:54.930 --> 00:14:58.170
Statistical significance
Was, was not shown.

00:14:58.230 --> 00:14:58.290
Wow.

00:14:59.460 --> 00:15:05.700
Again, a very interesting result
that time to cardiovascular death

00:15:05.700 --> 00:15:09.450
or heart failure hospitalization
showed a significant P value.

00:15:09.450 --> 00:15:15.450
So for example, the, the credible, the
confidence interval shown goes from 0.63

00:15:15.450 --> 00:15:16.635
to 0.99.

00:15:18.335 --> 00:15:20.940
That's the 95% confidence interval.

00:15:22.320 --> 00:15:26.550
So what is the conclusion in the trial?

00:15:27.090 --> 00:15:31.050
So I'm reading this, I'm looking at
the data, I'm looking at the results,

00:15:31.470 --> 00:15:38.250
the conclusion and relevance for
the the paper in patients with heart

00:15:38.250 --> 00:15:40.560
failure and iron deficiency, I.

00:15:41.128 --> 00:15:47.308
Iron supplement did not significantly
reduce the time to first heart failure,

00:15:47.308 --> 00:15:54.298
hospitalization, or cardiovascular death
in the overall cohort or in patients

00:15:54.298 --> 00:16:01.348
with transference saturation less than
20%, or reduce the total number of heart

00:16:01.348 --> 00:16:03.478
failure hospitalizations for placebo.

00:16:04.390 --> 00:16:09.280
And in the little, uh, uh, figure
they show that talks about the

00:16:09.280 --> 00:16:10.870
population, it's really very nice.

00:16:11.110 --> 00:16:18.220
It's the, the cartoon of the, the
article, the conclusion says that iron

00:16:18.220 --> 00:16:23.140
supplement was well tolerated, but
did not significantly improve outcomes

00:16:23.140 --> 00:16:27.340
compared with placebo in patients with
heart failure and iron deficiency.

00:16:29.680 --> 00:16:30.400
Okay.

00:16:31.645 --> 00:16:36.175
So what, what does a statistician,
and by the way, this is

00:16:36.175 --> 00:16:37.885
just me as the statistician.

00:16:37.885 --> 00:16:41.575
I, I'm, I'm sure other statisticians
have a very different reaction to me.

00:16:42.265 --> 00:16:46.975
So I read this and I really,
really struggled with this

00:16:47.185 --> 00:16:48.505
from, from several points.

00:16:48.505 --> 00:16:50.575
And so let me sort of
dive into the points.

00:16:51.145 --> 00:16:58.345
The first part about this is, it, it just,
the, the scientific struggle I have that.

00:16:58.945 --> 00:17:03.055
We report trials as black and white.

00:17:03.715 --> 00:17:06.205
They're significant or they're not.

00:17:08.515 --> 00:17:15.415
And in this trial where the only reason
it's not significant is because that

00:17:15.415 --> 00:17:20.155
overall cohort on cardiovascular death
and heart failure, hospitalization was

00:17:20.155 --> 00:17:23.335
part of a, a multiple testing procedure.

00:17:23.905 --> 00:17:27.805
So it was, it was incredibly
close to being significant.

00:17:28.495 --> 00:17:33.775
But it wasn't, the conclusion of
this trial is that iron supplement

00:17:33.775 --> 00:17:38.155
doesn't help patients, doesn't change
the clinical outcome of patients.

00:17:39.865 --> 00:17:45.295
A Bayesian analysis say of that primary
endpoint would say there's, assuming a

00:17:45.295 --> 00:17:54.625
non uh, informative prior would be a 98%
probability that iron supplement benefits.

00:17:55.150 --> 00:17:58.750
On time to cardiovascular death
or heart failure hospitalization.

00:18:00.430 --> 00:18:05.320
So any way you want to read
this, there's gray area to this.

00:18:05.320 --> 00:18:10.720
Every trial typically has gray
area to this, but yet when we

00:18:10.720 --> 00:18:13.300
publish it, it's all or nothing.

00:18:14.020 --> 00:18:19.360
In this scenario, the conclusion
is the same as if the data were

00:18:19.360 --> 00:18:24.340
identical in the two groups and the
HA and the hazard ratio was one.

00:18:24.670 --> 00:18:29.170
Even if it showed harm, the
conclusion would be the same, that

00:18:29.170 --> 00:18:34.060
the treatment doesn't benefit and
really co conclusions say one thing,

00:18:34.060 --> 00:18:36.310
the treatment benefited or it didn't.

00:18:37.600 --> 00:18:45.220
I think it's a gross simplification
of a six year trial of 1100 patients,

00:18:47.290 --> 00:18:49.180
but I understand how we got there.

00:18:49.180 --> 00:18:54.580
And by the way, statisticians share
some blame in in how we got there.

00:18:54.760 --> 00:19:02.440
We reinforce that hypothesis testing
and type one error of 5% and you can't

00:19:02.440 --> 00:19:05.170
say anything if you don't reject a no.

00:19:06.730 --> 00:19:11.410
And that's the way we
analyze trials this trial.

00:19:11.410 --> 00:19:19.000
I bet if you flip three deaths in this
trial, it meets the Hochberg procedure.

00:19:19.900 --> 00:19:28.180
And our conclusion to clinicians reading
this article is that it benefits, it's

00:19:28.180 --> 00:19:30.340
either it doesn't benefit or it benefits.

00:19:30.340 --> 00:19:31.540
Now we hope.

00:19:32.485 --> 00:19:37.405
Anybody reading this dives into the data,
looks at the results, thinks about other

00:19:37.405 --> 00:19:43.375
trials, thinks about the treatment of
this, and makes a decision based on it.

00:19:43.525 --> 00:19:47.515
But I have to believe
the conclusion from jama.

00:19:49.135 --> 00:19:55.285
It makes a huge indent into any
clinician reading this article.

00:19:56.335 --> 00:19:57.205
So.

00:19:57.580 --> 00:20:05.410
The black and whiteness of trials, uh,
I just feel like is scientifically,

00:20:05.650 --> 00:20:09.790
I, I really struggle with it and
I, I accept that as a member of the

00:20:09.790 --> 00:20:13.630
statistical community, we're probably
partly to blame for this dogmatic

00:20:13.630 --> 00:20:16.060
approach to hypothesis testing.

00:20:16.060 --> 00:20:21.820
And its black and white and we're gonna
come down hard on you if you don't

00:20:21.940 --> 00:20:24.970
interpret it any other way than that.

00:20:26.665 --> 00:20:27.235
Okay.

00:20:27.715 --> 00:20:32.965
And now for those statisticians out
there, I I, I have nothing against

00:20:32.965 --> 00:20:36.925
the Hochberg procedure, and I know
that we design trials, we do FDA

00:20:36.925 --> 00:20:42.415
trials where, uh, 5% scenario, 2.5%,

00:20:42.415 --> 00:20:45.805
one-sided tests I is, is the standard.

00:20:45.805 --> 00:20:48.775
And, and we live by that and
we do hochberg procedures.

00:20:49.570 --> 00:20:53.230
So I, I don't have anything to get
that, but I really struggle with the

00:20:53.230 --> 00:20:55.540
likelihood principle aspect of this.

00:20:56.770 --> 00:21:04.330
The data in this scenario is exactly
the same as another scenario.

00:21:04.540 --> 00:21:10.990
The data are identical where this, this
paper says it's statistically significant,

00:21:11.050 --> 00:21:16.270
and this iron supplement benefits patients
with heart failure and iron deficiency.

00:21:17.680 --> 00:21:19.330
This 0.05

00:21:19.630 --> 00:21:24.970
had, uh, the exact same data set and
the likelihood principle that if we

00:21:24.970 --> 00:21:29.680
have exactly the same data in two
different scenarios, our conclusions

00:21:29.920 --> 00:21:33.010
should be the same if you're a Bayesian.

00:21:33.010 --> 00:21:36.760
The posterior probabilities
identical for those two trials.

00:21:38.125 --> 00:21:44.035
It, the Bayesian machinery, uh,
satisfies the likelihood principle by

00:21:44.035 --> 00:21:47.845
the nature of the Bayesian a Bay theor.

00:21:49.435 --> 00:21:54.985
So I really struggle with this
part of it, and I get it from

00:21:54.985 --> 00:21:56.815
a type one error scenario.

00:21:56.815 --> 00:22:01.255
Being a Bayesian, I don't think
type one error is the be all, end

00:22:01.255 --> 00:22:03.565
all, and it flips this result.

00:22:04.150 --> 00:22:09.400
For clinicians reading it, and
I, I really struggle with that.

00:22:10.930 --> 00:22:11.650
Okay.

00:22:12.940 --> 00:22:18.790
Now, given the rules we play by, and
they knew the rules, they played by,

00:22:18.790 --> 00:22:24.580
they wrote them in this, this article,
so the SAP lays out superiority.

00:22:24.880 --> 00:22:29.980
They knew that if they ended up in this
situation, so if somebody simulated

00:22:29.980 --> 00:22:32.320
this trial and showed them this result.

00:22:34.240 --> 00:22:36.820
By the way, that's a
huge value of simulation.

00:22:36.820 --> 00:22:42.460
If they saw that result and said,
good, that's the result we want, great.

00:22:43.360 --> 00:22:47.980
But I'd be really surprised if
that's the result they wanted In

00:22:47.980 --> 00:22:50.890
this trial, there are no adaptation.

00:22:51.070 --> 00:22:53.500
Could this trial have been adaptive?

00:22:53.800 --> 00:22:55.600
Could we have seen that result?

00:22:55.690 --> 00:22:56.890
Could it have been bigger?

00:22:57.100 --> 00:22:59.770
Now, I recognize this
was a six year trial.

00:22:59.770 --> 00:22:59.830
I.

00:23:00.580 --> 00:23:04.180
And it may be the funding of
this, it couldn't have been bigger

00:23:04.180 --> 00:23:05.560
and this is just the way it is.

00:23:05.680 --> 00:23:08.590
And then the, they would say, the
investigators would say, yes, that

00:23:08.590 --> 00:23:10.480
that's, that's the result we want.

00:23:11.020 --> 00:23:15.070
But I think it's one of those scenarios
that had this trial been six months

00:23:15.070 --> 00:23:20.380
longer, had it enrolled 200 patients,
would it have changed clinical practice?

00:23:20.830 --> 00:23:24.760
Does this paper change clinical practice?

00:23:26.110 --> 00:23:27.880
Should it change clinical practice?

00:23:28.270 --> 00:23:31.300
Would it have changed clinical
practice if the trial were six

00:23:31.300 --> 00:23:32.920
months bigger, 200 patients bigger?

00:23:33.220 --> 00:23:35.170
That's a little bit of the struggle here.

00:23:36.250 --> 00:23:44.170
And so, uh, I, I just bring out that
I, I hate to say it, but is 1100

00:23:44.170 --> 00:23:48.430
patients in six years, is it wasted?

00:23:50.470 --> 00:23:53.680
Uh, in, in the way because
we do black and white.

00:23:54.535 --> 00:24:00.475
Now we shouldn't do black and white,
and I'll talk more about what would be

00:24:00.715 --> 00:24:06.715
shades of gray, but these are the rules
that everybody plays by at this point.

00:24:06.715 --> 00:24:07.855
Journals play by this.

00:24:07.855 --> 00:24:09.385
We kind of know the rules going in.

00:24:09.535 --> 00:24:10.675
Could it have been adaptive?

00:24:12.805 --> 00:24:16.375
The other struggle I have is
that this is about science.

00:24:16.375 --> 00:24:19.825
This is about recommending
treatments to patients that

00:24:19.885 --> 00:24:21.745
that could potentially benefit.

00:24:22.255 --> 00:24:26.095
And I would think if the truth
of this is a hazard ratio of 0.8

00:24:26.545 --> 00:24:31.825
on heart failure, hospitalization and
um, uh, cardiovascular death, this

00:24:31.825 --> 00:24:33.865
is a clinically important treatment.

00:24:35.905 --> 00:24:40.165
So we only analyze data in the trial.

00:24:41.365 --> 00:24:44.125
We are stuck on that by the way.

00:24:44.125 --> 00:24:47.365
I think that's reflective
of frequentist approaches.

00:24:47.455 --> 00:24:49.645
We analyze the data in the trial.

00:24:49.645 --> 00:24:53.545
We calculate the probability of the
data is as extreme or more extreme

00:24:53.545 --> 00:24:55.765
than what we saw, assuming the null.

00:24:56.455 --> 00:24:58.255
That's the P value 0.04

00:25:00.085 --> 00:25:00.565
it.

00:25:01.225 --> 00:25:06.385
But this is, there's science
here we we typically know more.

00:25:06.745 --> 00:25:09.295
Part of my struggle is what's next.

00:25:10.150 --> 00:25:14.950
This trial at I I if, if
you're into giving adjectives,

00:25:14.950 --> 00:25:16.750
it's borderline significant.

00:25:16.960 --> 00:25:20.470
It's very close to being
statistically significant.

00:25:21.040 --> 00:25:26.950
Do we need another trial that's six
years in 1100 patients and get the P

00:25:26.950 --> 00:25:30.220
value of that next trial below 0.05

00:25:30.220 --> 00:25:36.340
or below HUCKER stuff,
or would 200 patients.

00:25:36.640 --> 00:25:39.760
Potentially, I said, suppose
that trial was bigger by 200

00:25:39.760 --> 00:25:43.840
patients or 300 patients, would
it change clinical practice?

00:25:43.870 --> 00:25:46.720
But no, we can't do that.

00:25:48.100 --> 00:25:51.640
The next trial designed would
only analyze that trial.

00:25:51.910 --> 00:25:54.670
There's something incredibly
frustrating about that.

00:25:54.730 --> 00:25:58.450
Now I want to give you a
different potential scenario.

00:25:58.450 --> 00:26:01.060
Suppose this was a novel treatment.

00:26:02.095 --> 00:26:08.785
I assume that nobody owns the, the,
the, uh, the, the rights to this.

00:26:09.085 --> 00:26:12.865
Nobody has patent life
on a, a iron supplement.

00:26:13.165 --> 00:26:17.455
And this is all about,
uh, uh, treating patients.

00:26:18.355 --> 00:26:23.245
But suppose this was a novel
treatment in heart failure and a

00:26:23.245 --> 00:26:27.835
company ran this trial exactly this
and, and they don't get significant.

00:26:28.645 --> 00:26:33.985
And they go to a regulatory agency,
they go to ema, they go to PMDA,

00:26:33.985 --> 00:26:39.175
they go to the us FDA, and they say,
you know, we just can't approve it.

00:26:40.075 --> 00:26:41.485
It, it's not enough.

00:26:41.635 --> 00:26:46.225
Based on that trial, does that
company need to run another

00:26:46.315 --> 00:26:48.025
trial of 1500 patients?

00:26:49.615 --> 00:26:52.465
Can we say, look, there's
information there.

00:26:53.305 --> 00:26:55.945
The next trial doesn't need to be as big.

00:26:55.975 --> 00:26:59.185
We're really close to approving
this, but we just can't do it.

00:26:59.695 --> 00:27:01.585
And there are examples of this.

00:27:02.605 --> 00:27:07.555
Um, do we start over it, it seems, I.

00:27:07.945 --> 00:27:09.625
Bad science.

00:27:10.105 --> 00:27:13.915
Now, the FDA is absolutely doing this.

00:27:14.005 --> 00:27:18.025
There are scenarios where they use
the results of one trial combined

00:27:18.025 --> 00:27:19.675
to the results of another trial.

00:27:19.915 --> 00:27:21.565
You can look up Rebi ota.

00:27:21.565 --> 00:27:22.885
It was approved in 2023.

00:27:24.040 --> 00:27:25.420
Fairing pharmaceuticals.

00:27:25.420 --> 00:27:28.510
There are multiple devices that
have been been approved this way.

00:27:28.720 --> 00:27:32.830
There's multiple scenarios that I
know of that we're working with the

00:27:32.830 --> 00:27:37.270
agency or have designed trials where
it, it uses the previous results

00:27:37.600 --> 00:27:41.470
recognizing that, boy, it's really close.

00:27:41.470 --> 00:27:44.855
We shouldn't need 1500 patients
after this for approval.

00:27:45.835 --> 00:27:48.550
So I want you to think about
just the, the medical community.

00:27:49.450 --> 00:27:54.430
In that scenario where we're so
focused on single trials, what about

00:27:54.430 --> 00:27:56.320
combining the results together?

00:27:56.380 --> 00:28:02.290
All of this, my biggest issue with
this as a statistician reading it

00:28:02.350 --> 00:28:04.450
is I don't believe the conclusions.

00:28:06.190 --> 00:28:07.900
I think they're wrong.

00:28:08.140 --> 00:28:13.900
Now, maybe it's just me, but I
think this treatment works, and let

00:28:13.900 --> 00:28:16.090
me give you a little bit of why.

00:28:16.090 --> 00:28:18.100
So when I read the article.

00:28:18.430 --> 00:28:22.870
And I see that first of all, the
statistics is compelling to me.

00:28:23.920 --> 00:28:26.740
That's highly likely
just based on the trial.

00:28:26.740 --> 00:28:31.540
If we're stuck on only this
trial, a 98% probability for

00:28:31.540 --> 00:28:39.010
a clinically really clinically
important event is really valuable.

00:28:39.760 --> 00:28:45.130
In a scenario like this where this isn't
about an FDA approval, which has its own

00:28:45.130 --> 00:28:48.970
regulatory standards, and I know we've,
we've gotta go by this, but this is about

00:28:48.970 --> 00:28:50.980
the next patient that walks in the door.

00:28:51.730 --> 00:28:54.790
If it were me, I want iron supplement.

00:28:55.810 --> 00:28:57.070
I think it works.

00:28:57.100 --> 00:28:58.630
It's highly likely to work.

00:28:59.170 --> 00:29:03.160
But the other thing I thought is okay,
you know, we, we do get type 1 errors.

00:29:03.160 --> 00:29:06.820
We do get scenarios where we get a
hazard ratio like this in a confidence

00:29:06.820 --> 00:29:08.380
interval and the treatment doesn't work.

00:29:09.280 --> 00:29:14.230
So, as a statistician, I wanna know
what other information is out there

00:29:15.760 --> 00:29:17.740
and what do we know about this?

00:29:17.740 --> 00:29:23.020
Well, there have been previous
large trials run and the trial

00:29:23.020 --> 00:29:27.250
nicely, uh, it, it's actually, you
can't really find much about it.

00:29:27.250 --> 00:29:30.460
There's a little bit in the JAMA
article, so I, I, I don't want to.

00:29:30.955 --> 00:29:31.675
Say there isn't.

00:29:32.155 --> 00:29:36.085
Uh, but it talks about the reason
and still the uncertainty about this.

00:29:36.475 --> 00:29:42.835
And so there is a trial, the
heart FID trial, that it looked

00:29:42.835 --> 00:29:46.765
at time to cardiovascular death
or heart failure, hospitalization.

00:29:46.765 --> 00:29:50.515
All three of the trials I'm gonna
tell you about that have already been

00:29:50.515 --> 00:29:56.575
published, use that same endpoint
time to first cardiovascular death

00:29:56.575 --> 00:29:58.135
or heart failure hospitalization.

00:29:59.260 --> 00:30:02.980
And that trial showed it
had a hazard ratio of 0.93

00:30:03.430 --> 00:30:07.990
for that endpoint with a confidence
interval that went up to 1.06.

00:30:08.620 --> 00:30:10.540
So that probably had a P value.

00:30:10.540 --> 00:30:13.781
Something like, uh, uh, one-sided 0.1.

00:30:14.485 --> 00:30:17.845
Uh, two-sided, you know, maybe 0.15

00:30:17.905 --> 00:30:20.185
and maybe a one-sided 0.075.

00:30:20.695 --> 00:30:24.235
I didn't go find the article, but
I, I found the summary of that.

00:30:24.325 --> 00:30:26.425
So high hazard ratio of 0.93.

00:30:26.965 --> 00:30:31.315
The Iron Man trial, a great name for
the trial with an iron supplement,

00:30:31.645 --> 00:30:34.015
had a hazard ratio of 0.84

00:30:34.555 --> 00:30:39.835
for that same endpoint where the upper
bound of the confidence interval is 1.02.

00:30:42.220 --> 00:30:47.620
Borderline significant but not that
trial did not demonstrate clinical

00:30:47.620 --> 00:30:49.630
benefit, but hazard ratio of 0.84,

00:30:49.810 --> 00:30:50.680
so 0.9

00:30:50.680 --> 00:30:52.420
3.84

00:30:53.380 --> 00:30:58.465
and the Affirm A HF trial
had a hazard ratio of 0.80.

00:31:00.245 --> 00:31:03.115
Confidence interval 0.98.

00:31:04.660 --> 00:31:10.090
They also res report heart failure
hospitalizations for three of them.

00:31:10.330 --> 00:31:12.430
I'm sorry, for two of them, 0.80

00:31:12.430 --> 00:31:13.510
and 0.74.

00:31:15.610 --> 00:31:20.290
The other primary endpoint
in the fair HF two trial.

00:31:21.790 --> 00:31:29.170
So walking into this trial, we've got
three other trials that demonstrate 0.9,

00:31:29.170 --> 00:31:29.920
3.8.

00:31:29.920 --> 00:31:31.150
4.80.

00:31:32.035 --> 00:31:35.125
Uh, uh, for hazard ratio
for that primary endpoint.

00:31:35.545 --> 00:31:41.665
All positive, one of them significant,
one of them borderline significant.

00:31:42.535 --> 00:31:45.535
Uh, one of, uh, another one, 1.06

00:31:45.535 --> 00:31:48.745
for the upper bound of this, and
now this trial, the confidence

00:31:48.745 --> 00:31:51.415
interval for that endpoint is 0.79

00:31:51.415 --> 00:31:52.915
with an upper bound of 0.99.

00:31:53.635 --> 00:31:58.525
That information altogether, this
treatment as, as the read of a

00:31:58.525 --> 00:32:00.445
statistician, this treatment works.

00:32:01.240 --> 00:32:04.930
We are sitting there with four
trials all analyzed separately.

00:32:05.170 --> 00:32:07.810
There is a meta-analysis published.

00:32:08.260 --> 00:32:09.940
Does it move clinicians?

00:32:11.320 --> 00:32:13.570
So I don't know the answer to that.

00:32:13.570 --> 00:32:15.130
Other people can tell me that.

00:32:15.850 --> 00:32:22.090
But if I randomly pick up this
article and I read it, it says,

00:32:22.090 --> 00:32:23.440
this treatment doesn't work.

00:32:24.910 --> 00:32:30.430
Boy, I, I, you know, and I know, uh,
but boy and, and jam is an incredible.

00:32:30.715 --> 00:32:31.255
Journal.

00:32:31.675 --> 00:32:33.835
I, I know multiple editors for Jana.

00:32:33.835 --> 00:32:37.195
They jama they do an
incredible job with it.

00:32:37.285 --> 00:32:41.035
And, and I know exactly how we
got here and this is not unique.

00:32:41.575 --> 00:32:44.185
Uh, we, we see this commonly.

00:32:44.860 --> 00:32:47.695
I, I just, I struggle with it.

00:32:48.925 --> 00:32:51.085
I don't think the conclusion is right.

00:32:51.535 --> 00:32:54.655
I think it's actually highly
likely the conclusion is wrong.

00:32:55.735 --> 00:32:57.745
Do people read the conclusions?

00:32:59.065 --> 00:33:00.805
So that, that's my struggle.

00:33:00.805 --> 00:33:04.105
So what could the world be different?

00:33:04.525 --> 00:33:04.915
You know?

00:33:04.915 --> 00:33:08.335
Okay, Scott, so what, what
would be different here?

00:33:08.725 --> 00:33:09.115
What?

00:33:09.175 --> 00:33:12.565
What can we propose
different ways to do this?

00:33:12.565 --> 00:33:19.345
Well, let's suppose we didn't think of
the trial as being this black and white.

00:33:20.590 --> 00:33:26.080
Where we do a significance test, if
it's significance, we all wave flags

00:33:26.080 --> 00:33:30.550
and we all celebrate and we publish
it and it changes clinical practice.

00:33:30.550 --> 00:33:34.780
And if it's not significant, it doesn't
change Clinical practice, you know,

00:33:34.780 --> 00:33:37.360
is, is suppo what would be different?

00:33:37.840 --> 00:33:41.680
What if the trial reported
the posterior probability?

00:33:41.980 --> 00:33:48.970
The treatment is superior to the control
and it doesn't put an adjective on it.

00:33:50.320 --> 00:33:55.390
It doesn't say significant, borderline
significant, highly significant.

00:33:55.480 --> 00:33:57.460
Three asterisks on it.

00:33:59.170 --> 00:34:01.300
We don't have to do that.

00:34:01.660 --> 00:34:04.750
98% is an adjective

00:34:06.850 --> 00:34:12.550
and it allows somebody to consume the data
if they want to only look at that trial.

00:34:12.580 --> 00:34:12.790
Okay?

00:34:12.790 --> 00:34:16.960
98% probability for the primary
endpoint of superiority.

00:34:17.800 --> 00:34:22.270
By the way, that satisfies the
likelihood principle in that scenario,

00:34:22.990 --> 00:34:27.220
and we're overly stuck on type
one error that it's significant.

00:34:27.220 --> 00:34:32.680
What if that's the report and the little
cartoon in the front of JAMA says, this

00:34:32.680 --> 00:34:39.070
trial demonstrated 98% probability that
it benefits time to first cardiovascular

00:34:39.070 --> 00:34:42.490
event and heart failure hospitalization

00:34:45.250 --> 00:34:46.060
Now.

00:34:47.545 --> 00:34:51.475
We, uh, the first thing a frequentist
is gonna say, well, oh yeah, but

00:34:51.475 --> 00:34:53.005
now you've got a prior for that.

00:34:53.035 --> 00:34:56.965
An important part of this, the
important part is where this sits

00:34:56.965 --> 00:35:02.395
in the science, and I think the,
the article in the Journal of the

00:35:02.395 --> 00:35:09.895
American Medical Association shouldn't
just report on that single trial.

00:35:10.405 --> 00:35:15.235
The trial should prospectively define
a relatively non-informative prior.

00:35:15.760 --> 00:35:21.190
So that the data speaks for itself,
that we, we know how to do that.

00:35:21.190 --> 00:35:22.840
It's common that we do that.

00:35:22.930 --> 00:35:26.050
And here that would give a
98% posterior probability.

00:35:26.920 --> 00:35:30.940
We, the trial prospectively
defines a pessimistic prior.

00:35:31.960 --> 00:35:35.440
It prospectively defines
an optimistic prior.

00:35:35.740 --> 00:35:38.050
This would be relatively easy to do.

00:35:39.310 --> 00:35:41.320
It also specifies.

00:35:41.995 --> 00:35:48.805
A prior, based on the current summary
of scientific information, it uses

00:35:48.805 --> 00:35:53.455
the meta-analysis that was published
and it says based on the previous

00:35:53.455 --> 00:35:57.895
data, and it might even have several
of these based only on trial one.

00:35:57.895 --> 00:35:58.945
Here's what you do.

00:35:59.155 --> 00:36:00.205
Other trial two.

00:36:00.385 --> 00:36:04.075
So if you don't like trial three and you
like trial one, here's several priors.

00:36:05.530 --> 00:36:07.450
And somebody that uses those products.

00:36:07.480 --> 00:36:16.420
Here's the probability, and my guess
in this scenario is that this is 99.9

00:36:16.420 --> 00:36:17.150
probability.

00:36:17.150 --> 00:36:18.460
This treatment is beneficial.

00:36:19.120 --> 00:36:24.670
If you use that summary of that
information updated, which Bayesians

00:36:24.670 --> 00:36:31.990
do based on this trial, and allows the
reader to judge it on this and never.

00:36:32.620 --> 00:36:39.160
Never says it statistically
significantly affects the clinical

00:36:39.160 --> 00:36:43.870
outcome, but says there's a 99.3%

00:36:43.870 --> 00:36:49.000
probability this benefits, or nine
that this trial demonstrated a 98%

00:36:49.000 --> 00:36:51.280
probability that it, it benefits.

00:36:52.780 --> 00:36:57.160
A pessimistic prior would be 94%.

00:36:57.400 --> 00:36:59.860
An optimistic prior is 99.3.

00:36:59.980 --> 00:37:04.240
I'm making these numbers up, but I'm
just guessing what they might be.

00:37:04.930 --> 00:37:08.800
And the prior, based on a summary of
the other trials and this one together,

00:37:08.800 --> 00:37:12.880
that's where I think this would be 99.97%

00:37:12.880 --> 00:37:15.040
probability of benefit,
something like that.

00:37:16.360 --> 00:37:20.800
As a statistician reading it, I'd be much
more comfortable when I read this article.

00:37:21.490 --> 00:37:24.520
That, that's providing
this advice to clinicians.

00:37:25.660 --> 00:37:30.760
Where right now, when I read that article,
I really struggle and boy, I hope they

00:37:30.760 --> 00:37:37.030
look at the data and I hope that they,
they consume all of this information.

00:37:37.030 --> 00:37:39.100
It's hard it, the setting,

00:37:42.275 --> 00:37:42.495
so.

00:37:44.365 --> 00:37:51.595
This is my read of a random article
that sort of stuck with me of the

00:37:51.595 --> 00:37:56.665
results of this, and it stuck with
me largely as somebody who spent 25

00:37:56.665 --> 00:38:02.005
years in clinical trials doing publicly
funded trials, privately funded

00:38:02.005 --> 00:38:07.165
sponsor trials, NIH, funded trials,
patient organization funded trials,

00:38:07.165 --> 00:38:10.705
comparative effectiveness trials that.

00:38:11.455 --> 00:38:15.205
Uh, you know, I struggle with
the scientific outcome of this.

00:38:16.015 --> 00:38:22.015
So we are not in the interim here
we are at the end of the trial.

00:38:22.015 --> 00:38:25.915
Thinking of it, maybe this could have been
in the interim and the trial could have

00:38:25.915 --> 00:38:30.955
been adaptive in the world we live in, but
also things about the world we live in.

00:38:30.955 --> 00:38:35.275
Could we do things differently as
we're moving forward, uh, uh, in this

00:38:35.275 --> 00:38:37.465
and we get more and more results?

00:38:38.095 --> 00:38:39.355
Could this look different?

00:38:40.075 --> 00:38:48.625
So I am Scott Berry in the interim,
and until the next interim, thanks.