Judith: Welcome to Berry's In the Interim podcast, where we explore the
cutting edge of innovative clinical trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Scott Berry: Well, welcome everybody.
Welcome to In the Interim, uh, the podcast of, uh, Barry Consultants and
we, we look at, uh, as much as we can into the science, clinical trial science.
We look at clinical trial results.
We talk about statistics, uh, related to medical decision
making, related to clinical trial results, clinical trial design.
That's our focus.
Uh, and so today I'm going to talk about something that turns out
to be incredibly controversial.
So the name of this podcast is Politics, Religion, and Ordinal Outcomes.
Funny name, of course.
My, my reference of this is that you, you should not at the dinner table, you should
not talk about politics and religion.
Turns out you probably shouldn't talk about ordinal outcomes as well.
Everybody has different reactions to them.
Everybody analyzes them differently.
Uh, a passionate feeling about them.
It's kind of interesting because it's, it's ingrained in the
history of clinical trials.
I'm going to essentially describe that almost every endpoint is
ordinal, so you can't escape this.
So what are the controversies?
How do they show up?
What do I think about ordinal endpoints?
You may not think that this is a controversial topic.
It turns out to be really controversial.
So, ordinal endpoints have been around since the start of clinical trials.
Uh, I, I, I'm not a, a historian, uh, but my, my scouring of, of the, the,
of Google and the internet that really the first clinical trial was, uh, 1747.
And it's attributed to James Lind.
He was a Scottish naval surgeon.
He was very, very interested, of course, in 1747 in Scurvy.
The Scurvy.
Uh, and he ran what is attributed as the first clinical trial.
Now, it's not the first randomized clinical trial, and I couldn't
actually figure out exactly how he assigned patients, and that's the,
it's an interesting part of the trial.
So, uh, if you want to read more about this, by the way,
it's really wonderful reading.
You'll find yourself in a black hole reading about, uh, James Lind,
reading about Scurvy, and reading about this particular clinical trial.
, uh, you can go to james lind library.org,
uh, and read all about this.
Uh, so in 1753, he published a treaties on.
of the scurvy.
So I'm, I'm, I'm reading parts of this.
I found it absolutely fascinating.
And of course, in 1747, he writes that, uh, armies have been
supposed to have lost more of their men by sickness than by sword.
And so he makes reference to, uh, that more of the, the English army died from
scurvy than the French and Spanish armies.
Uh, themselves, uh, killed.
And so, huge problem of the time.
Uh, and he was interested in whether this calamity could be prevented and the
danger of this destructive evil obviated.
There were, interestingly at the time, there were tons of, of, of theories
as to What, what could prevent scurvy, what it was, what was causing it.
We have a pretty good idea at this time in it.
Uh, but he was very interested in this.
On May 20th, 1747, he conducted the first clinical trial.
He took, as he describes it, 12 patients in the scurvy, on
board the Salisbury at sea.
Their cases were as similar as could have them.
The, they all had putrid gums, the spots, and lassitude, I don't know what that
means, with weakness of their knees.
So he of course recognized that we want these patients to look the same.
He had strange inclusion, exclusion criteria, obviously not written down,
but he did everything he did, he could to get 12 patients that looked the same.
He gave them, he describes a common diet, he describes what it is, a water
gruel sweetened by sugar in the morning, fresh mutton broth for dinner, and of
course he refers to supper as different than dinner, barley with raisins,
rice and currants, and with wine.
Uh, so he, he, he, he.
did really smart things for the first clinical trial, trying to get patients
to look the same, tried to treat them the same in things except for the
interventions he was interested in.
So he had 12 patients and he had six different treatments.
He gave two of them court of cider.
a day.
He gave two of them 25 guts of elixir vitriol three times a day.
He gave two of them spoonfuls of vinegar three times a day.
Two of them, two, and he describes it interestingly, two of the worst
patients, their tendons in the ham, rigid seawater, a half a pint a day.
He gave two of them, two oranges and one lemon per day.
And two of them, bigness of nutmeg, three times a day.
This was a surgeon recommended therapy.
Again, lots of people were describing what they would do for the treatment of scurvy.
Uh, by the way, that, that, I don't know what that is, but apparently it had
garlic, mustard seed, radish, balsam of Peru, uh, gummer, and cream of tartar.
Uh, and the interesting thing in all of this is he does not
describe how they were assigned.
They were randomized, he assigned them, uh, an interesting part of a
clinical trial that we now think a really important part of the trial.
Uh, and that's really left out of the description, which is a
fascinating aspect of this 1747 trial.
I'm more, much more interested in today's discussion of ordinal endpoints, of how
he classified the outcomes of this trial.
So he writes, Uh, and I'm quoting from his writing, the consequence was the
results, the most sudden and visible good effects were perceived from the use of
the oranges and lemons one patient taken being at the end of six days fit for duty.
The spots were not indeed quite off his body, nor his gum sound.
The other was the best recovered of any of his condition.
And being deemed well was appointed nurse of the rest of the sick.
So, I don't think this happens in current clinical trials.
that A patient did so well that they become a caregiver of the other
patients in the clinical trial.
Um, uh, in that scenario.
But the description of this is the two best patients out of the 12 were
those that were on a single treatment.
The oranges and lemon.
In this scenario.
So his outcome was an ordinal outcome.
Actually resembles what we would today, we might call a door patient.
That are almost a win ratio type thing.
That comparing patients to patients.
These patients did the best.
There's no quantitative aspect that this patient was five times better than
that patient or double that patient.
There wasn't a mean time to recovery and all of this.
It was an ordinal outcome.
By the way, if you calculate a p value for the probability that lemons
and oranges would have the two best patients, it's one out of 66 or 015.
You, of course, would say, well, wait a minute, there's multiplicities here.
There were six different treatments.
What's the probability that any one of them would have had the two best?
Well, that's one out of eleven, uh, in that, which is 091.
So, adjusting for multiplicities, you get 091.
I'm more interested in the, the endpoint.
Now, interestingly, in part of this, and this may resonate with people,
that, This didn't change anything.
We now know this was actually the right treatment.
that it's about vitamin C and lemons and oranges would have prevented the scurvy
and would have had a massive impact.
Uh, it took 47 years beyond this before this became recognized
as a way to treat patients.
Interestingly, the thought was that you give dehydrated, uh, fruit because
it was hard to keep fresh fruit on, on board, uh, ships and all that.
And it turns out that loses much of the vitamin C content and it's not clear
that would have had any actually benefit.
So, CMC issues and PKPD issues could have really helped James Lind in 1747.
So, that was the first clinical trial, and again, you should go read all about this.
It's absolutely fascinating.
Um, the first randomized human clinical trial is attributed
to Austin Bradford Hill.
And he the, he published a paper in 1948 of the first randomized clinical
trial, which was ex experimenting on streptomycin for the treatment
of pulmonary tur tuberculosis.
And this was published in the.
British Medical Journal in 1948.
This trial enrolled 107 patients and the difference between James Lynn's, James
Lynn's trial and this one was that the patients were randomized to the treatment.
And they were randomized to the control group or they were
randomized to streptomycin in it.
Uh, the publication of this is also very interesting.
There's a traditional table one.
Maybe he invented the table one and the table two.
Uh, but table one give it, gave demographics of the trial and
table two gave the results.
So what was the end point of this first randomized clinical trial of humans?
There were randomized, uh, uh, uh, people probably understand there were randomized.
uh, agricultural experiments, but there were not randomized
human experiments before this.
Um, the primary endpoint of this was a more, what you more think of now as
a more traditional ordinal endpoint.
It was a six level ordinal endpoint.
The top level, but was considerable improvement.
The next was moderate or slight improvement, then no material change.
Moderate or slight deterioration, considerable deterioration, and death.
Six, six outcomes.
By the way, they aren't given numbers.
They aren't given, you could consider giving them letters, A, B, C, D, E, F.
F is the worst, A is the best.
So all this is, is, is, it's ordinal in that one is better than
the other one, but no numeric, uh, number is given as how much better.
We think of these as ordinal outcomes, and it's probably what
you know as ordinal outcomes.
So controversially, how did he analyze this?
We would fight today, I mean fight in a good way, that we would fight
about how to analyze this endpoint.
What do you do?
He analyzed it actually in multiple ways.
He, um, he talked about, uh, in this outcome, the first thing he
did was he talked about deaths and deaths were four, I'm sorry, 7.
3 percent in streptomycin and it was 27 percent in control.
And he talked about this being less than one in a hundred
chance that this would happen.
He then said the best.
If you split it in any particular way, the best split for streptomycin for
this was considerable improvement, which was 51 percent to 8%.
One in a million, he talks about the likelihood that
that would happen by chance.
So he's giving p values, he's analyzing it.
We would recognize, ooh, the multiplicity of looking at the
best place and what does that mean?
And that gets everybody in a bob.
Um, now this was an easy analysis where any way you analyze this endpoint,
it looked really, really good.
Uh, and so there's not going to be a whole lot of struggle.
Um, uh, on this end point in there, and I went back and took the data and I
fit a proportional odds model to this, and you get a proportional odds of 5.
43 to the good of streptomycin.
Uh, the lower bound of the 95 percent confidence interval is 2.
64.
This is a massive effect.
P value of that is 6.
3 times 10 to the minus 6.
Awesome data.
Uh, makes it really easy.
Everybody can, you know, look at this and say, this is clinically
really, really important.
Um, in it.
That was, um, that was some 75 years ago.
We still fight about how to analyze orbital endpoints, um, within this.
Um, okay.
So we're still fighting about this.
I'm going to focus to talk more about where are we today, what are we doing in
Ordinal Endpoints, what do I think about Ordinal Endpoints, uh, from this, as I,
I'm going to pick a particular example.
Very similar to, uh, uh, Austin Bradford Hills, Bradford Hills, uh, endpoint,
uh, of, A six level ordinal endpoint.
We're going to talk about the modified Rankin score.
The modified Rankin score is a seven level ordinal outcome.
It's used on the neurological status of patients.
It's quite common in a number of scenarios.
It's quite common in stroke trials, where the endpoint is your modified
Rankin status at 90 days, 180 days.
different points.
Um, the endpoint itself is seven levels.
They do give numeric values to these.
The numeric values that now we'll talk about, are they relevant or not?
Are they just labels?
And they're labels.
You could have labeled them, uh, A, B, C, D, E, F, G.
But I'll give the labels that are there, you may be familiar with them.
Zero is no symptoms.
Perfect neurological stat.
One is no significant disability.
You're able to carry out usual activities.
Uh, zero is better than one.
We can argue about how much better.
We're going to get to that.
Two is slight disability.
You can look after your own affairs, but unable to carry
out all previous activities.
So you have some disability.
Uh, one is better than two.
Zero is better than one.
One is better than two.
Then three is moderate disability.
You need help.
You require help.
Four, moderate to severe disability, unable to attend to your own
bodily needs without assistance.
Again, a higher level of support needed to carry out daily activities.
Five is severe disability, bedridden and requiring constant nursing care.
It's commonly given the label of vegetative state.
Um, uh, clearly a, a worse state than four, worse than zero through three.
And then six is dead.
Uh, no neurological status.
So, these are the seven outcomes.
It's a very, very common outcome in clinical trials.
The question is, how do we analyze this thing?
How do we analyze those seven outcomes in a clinical trial?
There's huge disagreement that goes on about, uh, analyzing this.
There's disagreement about the medical interpretation of these.
There's disagreement about statistically how to analyze them, uh, the relative
meaning of the values of them.
Uh, and different analyses are used.
The, the, the first way to potentially analyze these, and I shudder to say
it, but is to dichotomize these.
Very common in clinical trials, the modified Rankin is dichotomized.
There's even a name to the dichotomization.
Uh, and it's disabled or not.
So 0 through 2.
is considered a responder, a good outcome.
3 through 6 is considered a bad outcome.
And the clinical trial analyzes how many of those are 0 to 2 and
how many of those are 3 to 6.
That's a way to numerically analyze these outcomes.
Now I'm going to walk through this a little bit and I want to start by saying
this is mathematically equivalent.
to giving a weight of 0, 1, and 2, a value of 1, giving 3 through 6 a value of 0.
I could say that's my utility, and analyze that way, and I get
mathematically the exact same answer.
Okay, so.
It is a weighting by doing that dichotomous and you're saying having
a zero is equivalent to having a two.
Having a three is equivalent to dead.
Absolutely, that is the assumption that's being made when you
analyze the trial that way.
I shudder at that.
I shudder at that clinically.
I shudder at it statistically for a couple reasons.
The power of a trial that lumps everybody together, 0-2 3-6 is lower
than a different way to analyze it.
I think it's clinically harder to interpret.
I don't think any person has a utility function that is equal to that.
But it's very commonly done.
Okay, another way to analyze this endpoint, you, you, you can
probably understand what I think about, uh, zero to two or any
dichotomization of that endpoint.
What about a proportional odds model?
So a proportional odds model is a statistical assumption that the effect
of moving patients to a higher level.
is, has a statistical, uh, odds of that happening.
We assume it's the same of moving people from 6 to better than 6,
from 5 and 6 to better than 5 and 6, from 6 to better than 6.
The odds of that is constant across the scale.
There's a strong, I don't want to say strong.
I'm sure I shouldn't say that word.
There's a statistical assumption that comes with that.
That, that odds is the same across the scale.
Now that gets statisticians all ruffled.
It's almost like when you go to graduate school, you have to
question that assumption in any model, or you're not worth your
statistical salt, uh, uh, within this.
It is a, it is an assumption.
Now, and it can be violated.
The interesting thing about it is that, that way to analyze
that ordinal endpoint imposes a utility on each of those outcomes.
Remember, dichotomizing them imposes a utility.
And you can mathematically write it down.
A proportional odds model imposes a utility on it and you can
mathematically write it down.
What is the utility?
The interesting thing is the utility of that is based on
the prevalence of the outcomes.
If you have a lot of 0s and 1s and 10 percent deaths but 90 percent 0s and
1s, your utility you're imposing is highly weighting 0 to 1 differences.
Because that's where your prevalence is.
Death, and above that, is down weighted because they aren't very prevalent.
You can mathematically write it down.
I could give you the utility functions as you're imposing that upon your input.
People don't think about it that way.
It's a little bit uncomfortable.
If, alternatively, you have a lot of 4's, 5's, and 6's and very few 0's and 1's,
but you have some, you're down weighting those because they're not prevalent.
You are imposing a mathematical weight of those outcomes by analyzing that way.
You can't get out of that.
Okay.
Uh, I'm not saying that's bad.
I've used it.
By the way, I've used both of those in clinical trials.
Um, I struggle with the dichotomization, but I've done it.
Um, I've done proportional odds model, and I don't I don't
necessarily shudder at them.
Um, but I, I understand I'm imposing this strange sort of weight on them.
Okay.
There, there are other ways to analyze these endpoints.
We could do a Wilcoxon test.
We could do a Cox model on them.
Um, first of all, I want to point out that there are tons of ordinal outcomes.
This is the one a lot of people are familiar with.
Um, mortality is a ordinal outcome.
Yes, no, mortality.
It's ordinal.
Now, that's a pretty simple dichotomous one.
It's binary.
Time to death, overall survival in a cancer clinical trial, or any
clinical trial, is an ordinal outcome.
A year is better than six months, but you don't say how much.
You just say it's better when you run a Cox proportional
hazards model or a log rank test.
You're analyzing an ordinal outcome On that when you're analyzing infarct
volume or tumor size That's we sometimes people say that's quantitative.
Well, no, it's ordinal
You're imposing upon it the difference between them of one
unit being a numerical difference.
You're imposing upon a quantitative weight of that ordinal scale, but it's ordinal.
Almost everything is ordinal.
The best corrected visual acuity where you're looking at a chart and saying how
many letters can I see, it's ordinal.
We calculate a mean of them.
We make the assumption that one letter is better, no matter where you are in the
scale, that it has a constant difference.
We're imposing a quantitative weight on them.
People are comfortable with that.
When I impose a numerical value to the, you, to the, uh, modified
Rankine, people jump up in arms.
People.
Scientists regulators.
So there have been multiple, there have been several studies that look at patient
preference for the modified Rankin, that look at, um, the economic value of
them and they're actually very similar.
We have taken them and said, okay, we're not going to do a proportional odds model.
We're not going to do dichotomous.
Um, nobody has that weight.
We're going to use this and we're going to call it a utility function.
By that weight, by the way, the weight that is used on that is a full weight of
1 if you're a 0, 91 if you're a 1, 76 0.
65, 0.
33, and then 0 for both 5 and 6.
That those are equally bad.
That's a utility function.
You can analyze it.
Very simple statistical models.
No assumption of proportional odds.
And we can analyze it that way.
Uh, now, people get frustrated by that.
Lots of people in terms of regulators, what does this mean,
and the first criticism is not everybody has that utility weight.
You're using two different studies, but how do we know the people that
you're analyzing that have that weight?
The problem is any way you analyze this endpoint has that same issue.
If you dichotomize them, you're imposing that everybody in your
study has that weight of 111000.
My claim is nobody has that weight.
But yet, it's very commonly done.
But nobody's frustrated by that.
Bothers me because I don't think anybody has that weight.
P, studies, phase 3 clinical trials at FDA use proportional odds model.
There's a comfort in that.
You got to check assumptions, you got to look at proportional odds.
The usual thing we do for Cox models and all that.
You're imposing a weight.
How do we know the people in your study have that weight?
My point is it's a false criticism because you cannot escape it.
You have to weight the endpoints any way you analyze them.
All of these endpoints we impose a weight on them when we go into a clinical trial.
But when we specify it up front and try to justify that this is a good
weighting system, everybody throws arrows at it and, oh, that's bad.
But dichotomous and proportional odds, that's a good way to do it.
Alright, so what do I think of this?
I think it's the really hard thing in a clinical trial.
It's the way as we move forward we're going to get more and more fine
grained understanding of endpoints.
Our endpoints are going to be more strength, more valuable.
We have to do this.
It's hard.
Science is hard.
Medicine is hard.
Hiding behind ad hoc ways to do this, I think, just leads us to bad places.
It disconnects the statistics from the clinical outcomes, from
the doctors, from the clinicians.
This is where we need to go.
We need baby steps.
We need to work there.
We need to show it's valuable, that it's good for regulators, that it's good for
patients, it's good for statisticians.
But it's really the only way to do this.
So I think as our endpoints become more.
More, um, uh, less signal, uh, more signal, less noise, wearables, uh, daily
values, our iPhone, our iWatches, our, our things, calculating things for us.
These are all ordinal outcomes.
That this is the only way forward.
We need to go there.
Let's do baby steps.
Let's analyze the modified Rankine in really, really better ways, smarter
ways, explicit ways, uh, in this.
And please, please, let's not just dichotomize really nice endpoints.
Um, uh, again, I don't think anybody has that weight system.
Alright, well I hope, I hope I didn't defend you like I would if we
talked about religion or politics.
It's a controversial thing, but I think it's a really important
thing in clinical trials.
I hope you're, you, you, you enjoyed the discussion.
And until the next time, we are in the interim.
Yeah.