Judith: Welcome to Berry's In the Interim podcast, where we explore the
cutting edge of innovative clinical trial design for the pharmaceutical and
medical industries, and so much more.
Let's dive in.
Well, welcome back everyone to In The Interim.
I'm your host, Scott Berry.
And today I'm going to touch on multiple topics.
Those of you who join into this, uh, podcast periodically know we like to
delve into the world of sports and how that's related to science and clinical
trials and statistics and learning.
So we're gonna, we're gonna dive back and forth from sports to clinical
trials here, and, uh, with it.
And, and part of, uh, the, the wonderful thing about this podcast
is we've been doing this podcast, uh, weekly, only missed a couple
weeks within a couple years now.
And I always wonder, "Okay, what's-- what, what, what topics are coming next, uh,
in the world of science and statistics?"
And, you know, the, the world brings topics.
Uh, i- it's kind of amazing.
And so I was, I was very struck by this particular story, and I thought
it was related to who I am as a scientist and a statistician, and,
um, see if you find it compelling.
And the, the title of this is Fairness in Soccer and Science.
So I am notâ¦
Uh, and I'll say soccer in a sense, um, uh, within this.
So, uh, so let me get to the story of this.
Uh, those of you who can sort of see where, where I'm, I'm currently located,
I, I spend summers in Minnesota, and beautiful woods, uh, surrounding me.
You can sort of see the trees there.
Gorgeous nature.
Um, part of some of this gorgeous nature is, is, is a negative.
Uh, flying pests, insects, ticks.
Uh, uh, northern Minnesota is, is famous for its mosquitoes.
Uh, i- it's, uh, for, for those of you who have not been here,
um, in the, in the summertime, it can be, it can be really bad.
At dusk, uh, in the evening, it's, it's unbearable actually.
Uh, horseflies.
And so here we have in, in our cabin here, uh, outside of our laundry
room in the window, there is a gorgeous, uh, spiderweb and, uh, a
gorgeous spider that's right outside the window and, and it lives there.
And large spider.
And by the way, they're, they're incredible.
Um, they're incredible.
Um, um, I, I, I should be careful here.
I, I, I don't think they're an insect, but, but they're incredible
beings and to watch them.
So I enjoy watching them.
And, um, I, I- I can help the spider by turning on the light in the laundry room.
So, uh, i- in the evening here, if you flip on the light, of course, that
attracts many insects to the window and creates a, a huge benefit to that
spider, uh, in terms of the flying things.
I, I actually concern there are too many, uh, at times, uh, given Minnesota.
So is it cheating that, that I turn the light on?
And you know what, uh, now the fact that I don't like the flying
insects, um, within that, but inâ¦
A- and maybe that's me as a statistician that, uh, I turn that
on, and is that sort of cheating?
Um, there's a spider elsewhere, um, that doesn't have the light go on
sort of thing in, in, in all of that.
At least the thought crosses my mind.
Yes, I, I do turn the light on, and no, I don't have any issues with,
with, um, um, helping that spider, uh, do its job, uh, within it.
I think it's a productive thing around there.
Now, it, it makes no difference to us as humans to go outside.
We don't notice anything.
But, but I feel like spiders are good things in that.
But that, that thought goes through my mind, um, uh, as to, I, I don't
know, almost fairness, uh, within that.
So what's the story that came up here, uh, triggering the podcast topic?
Um, I have been watching the World Cup.
We are, we are in the throes of the World Cup.
I am not a, a, a football, I- I'll say it that way, the, the
way the rest of the world says it.
The US calls it soccer.
Uh, I'm not a big football fan.
I, I enjoy watching it, but I don't follow it tremendously.
My kids played it, and I, I kind of understand the rules.
As statisticians, we, we love rules.
A- and I love rules, and I understand it.
I'm trying to understand strategy and all that.
And I was actually gonna do a podcast on part of, uh, part of this that
to a statistician and, and rules that I don't like about soccer
that I think could be improved.
And now people, "Well, this is tradition," and, you know, sort of thing.
And an interesting thing is if, if you get a penalty inside the,
the large goal box, it's a penalty kick, which is an enormous penalty.
I don't know, 85% chance of a goal or something like that.
That can happen well away from the goal, uh, right on the very corner of the
box, and if you're one foot away from that, it's a very different penalty.
It's a much less severe penalty.
There's dichotomization in soccer that I think is, is not on the
continuum, and I don't think isâ¦
I, I don't think it's a good rule.
But that's not the topic of today.
Sort of related to, to maybe what's fair and all of that, and I, I think
they don't call the same penalties.
One foot difference, they would call it a penalty, but not
because of the severity of it.
So it's, it's strange to watch, to, to watch that.
So I think that could be improved But what is it?
It, it is, it is the recent decision of FIFA, the governing body of
the World Cup, to suspend the red card of Folarin Balogun, Balogun.
I, I'm probably saying his wr- his name wrong.
He's, uh, the leading scorer for the US, a striker for the US, and in the
earlier game he got a, um, uh, a red card.
During the game he, he had to leave the game and they had to play
actually one, one player short.
And for about 25 minutes at the end, they actually scored while they were
playing short and, uh, they, they gave up a goal but won the game.
That player is then suspended for the next game.
Now the US gets to have their full contingent of players, so they get to
play all 11 players at this next one, but, but the rules are if you get a red
card you're suspended the next game.
So FIFA announced that they're, they are, uh, changing that and they're
allowing him to play in the next game against Belgium, and this, this, this
podcast is, is going to come out after that game, um, in terms of it, but
it's being recorded before that game.
It's actually the morning of that game within it.
Um, and it was announced the day before that he was no longer suspended, and
it was, it was reported that there was contact with, um, United States President
Trump asking them to look into this and switched it, and the FIFA changed this.
It's the first time in over 50 years that this has ever happened.
It's never happened in the World Cup.
So what, whatâ¦
Yeah, it feels corrupt, and it feels corrupt from the standpoint
that had that been a player on Belgium that would have happened,
this would not have been changed.
That player would not, um, be, be, the, the resi- They suspended the
decision, which is awkward because they suspended the suspension, but,
uh, is, is the wording they gave.
That feels like that wouldn't have happened.
It feels like this is related to the relationship of
Trump to the FIFA president.
The, the, the, theâ¦
This is
influence, uh, from that.
Now, I, I understand the red card call was controversial within that, but it
was the call and it was what happened, and it was a video review that created
that, uh, in it, and I think people would argue, and I think generally people
think it was somewhat of a bad call.
But it was the call, and to remove it feels unfair.
Yeah, a little bit corrupt.
It feels that this is not right.
And it's interesting, much of the- many of the reactions are almost
feeling sorry for the US now.
They, they, they're in a no-win situation.
Should he score a goal in this game, um, you know, it, it- we will
always, the, the, the US soccer team will always be felt like
They had an unfair advantage.
Uh, you can call it cheating, call it what you will, that, that, that scarlet
letter will be there no matter what.
Um, it, it, it bothers me in my, my watching of sports, and this is
where this podcast goes, is this issue of fairness and, and me as a, a
statistician and, and what it means.
So for example, I won't be watching the game.
There, there's something about that is, is just so, so negative to me that this
happens, that it feels like this is now not a fair competition within it.
And I, I don't wanna root against the US, uh, in it, uh, but itâ¦
there's just so much of it that I'd, I'd rather not watch it.
I don't watch WWE.
Um, this is wrestling.
Um, allâ¦
I, you know, Iâ¦
as a kid, I called it all-star wrestling.
I, I don't even know where, where this is.
It's scripted, uh, wrestling, and it's entertainment, and Iâ¦
you know, people can enjoy that.
It's fantastic.
They're phenomenal athletes, by the way.
Um, stunning athletes, but it is a performance.
It's, it's Broadway.
It's, it's scripted.
I d- I, I just don't enjoy that.
I, I don't care who wins, you know, and all that, and that's
not really the point of it.
It's, it's a scripted performance.
It's not, it's not sort of sport, and at the center of sport has
to be this level of fairness.
And it's interesting because there was an earlier game that I was watching, and
I, I've been watching quite a bit of it.
I, uh, it's been really, really entertaining.
It's been incredibly good sport.
I, uh, I've, I've gra- gained an appreciation for soccer.
I'm not gonna call it football because that puts me at some level that I,
I, I, I'm now a, a, a knowledgeable person on that, and I, I'm, I'm not.
I don't, I don't wanna be a pretender in that.
But I've r- I've thoroughly enjoyed it to this point.
There was a game I was watching and, and England was playing Croatia, and
their superstar, uh, Harry Kane, had a penalty kick and, um, missed it.
Uh, the goalie saved it.
They did a video review of it, and the goalie left his spot too early, and that
violation, and so on video review, they gave him a second kick and he made it.
And so he s- he scored And to me, that was completely fair.
There's a rule, and, uh, as long as the rule is reinforced, and if it's
reinforced by video, it, it got it right.
And, you know, you could, you could argue that that seems, that seems
detailed or something, but it's the rule.
And I, Iâ¦
And I think that's related to who I am as a statistician, that there
are rules in the, uh, with- within that, and that's completely fine.
Um, and I thought I had no problem with that.
I don't feel that that's unfair.
There's a set of rules, and you follow the rules.
And that's why this bothers me, that he's, the, the striker is not suspended
because there are rules and you follow the rules, and we are not following
the rules This is related to, to golf.
If you watch a golf tournament, you'll see many times a golfer is in a, a, a
sort of bad position, but they're standing on a sprinkler head, or their ball is
in a particular position, there's a camera in the way, and they are legally
allowed to move their ball, and they generally end up in a better position.
It's not cheating, it is the rules, and it's, it's important for every golfer
to completely understand the rules, and there are officials there that
make sure everybody follows the rules.
Golf is incredibly, um, um, uh, rule-conscious in fairness.
Cheating in golf, you're essentially done as a golfer.
I mean, you can't cheat in golf, and they go out of their way to make sure
to protect the field, and everybody, your playing, uh, partner should be
reinforcing the rules, and it's, it's a really important part of golf is,
is sort of fairness, so that Scottie Scheffler, the number one ranked
player in the world, doesn't get an advantage over the 500th best player.
They play under the same rules.
I'll watch that.
I enjoy that.
That's sport, and, and that's fantastic.
Um, American baseball now has video review of pitches.
There's a strike zone, and you can, you can challenge that.
I, I think it's great.
Getting it right, having, having the rules, I think it's actually
made the umpires, who are, who are fantastic at what they do, and they
generally get it right, um, has made them, have made them better.
But all of this has, uh, has been a very positive thing because everybody
plays under the same rules within that.
There's a fairness to it which is critical in sports.
Okay.
So what, what, what does that mean?
I've gone on for 15 minutes about World Cup soccer and my reaction to it.
Um, I, I, I think as a statistician now, I don't know whether it's that I became
a statistician because, uh, I, I saw the rules and, and, and, and part of
it, I almost think it's, it's science.
Or being a scientist and working in clinical trials make me understand and
appreciate the, the importance of it to Sports and clinical tri-- uh, to,
to clinical trials that then relate back to sports, and I have such an
incredibly negative reaction to, um, removing that suspension, um, uh, in
it, and the lack of fairness to it.
We as statisticians in clinical trials, fairness are everything in clinical
trials, and it is, it is science.
We are, we are learning about particular therapies.
Are they safe and effective?
Is A better than B?
There's a lot of parallels between sports and clinical trials.
If there is undue cheating or unfairness in a clinical trial,
it means we don't know the answer.
Is A better than B?
Is, is, is, is this drug safe and effective for a
regulator to make a decision?
Um, and let's leave out of it perhaps influence in, in those decisions, which we
all as clinical trialists, uh, you know, abhor, uh, within that, um, uh, within it.
And so there's such a parallel in what we do in clinical trials to this question
of fairness, and part of why I love it is you can't fake science, uh, within that.
Let's get out of the aspect that you can fake data, and there's a great deal
of, of things done in clinical trials to make sure the data is honest data.
It's reported as collected.
So you could fake it by presenting data that's not right.
Everybody on the treatment lived.
Everybody on the placebo died.
The drug's a, a miracle.
But, but short of that, you can't fake it.
The data and the results of it, the, the, the, the statistical
analysis tied to it, demonstration, demonstrating the likelihood of benefit
of that treatment, uh, within it.
There's this, uh, there, there's this incredible structure and importance to
things we do in clinical trials to create this fairness, to answer these questions.
Maybe that's why this sort of reacted-- I reacted so negatively, um, to that.
It's, it's the, the, the same reaction that I had to the whole
aspect of the acetaminophen and being the cause of autism.
It's because it was not right.
I'm just, justâ¦
Blatantly, it was a violation of everything about science that we do.
It was wanting it to be right rather than it being right within
the rules of science inferences that we made, uh, uh, within it.
It was not supported by the data within it.
Uh, vaccines causing autism are not supported by the data.
There's a want to it.
It is-- It's, it's corruption within it, and there's such a reaction.
We should all react to that as clinical trialists because it lowers
everything about other conclusions that we draw within that setting.
And there is this battle, uh, within it, um, that, that we continually go on
to, to be able to present this as fair, to learn the truth about the therapies
out there that we're looking at.
Now
We, we, the, the, this adjudication of a clinical trial, um,
within it is A better than B.
We run these contests, and there is this sort of analogy to me of sports, and
maybe it's why I, I, I love what I do.
I love sports.
I love clinical trials.
I, I love seeing the results of these trials.
I love seeing the inferences on these trials.
What, what conclusions are we making?
Are we understanding sort of what, what's fair and how to make these conclusions?
A little bit I'll go back, when I'm asked to present to non-science
people, students, a- anybody about statistics, I generally don't go
in and present quantitative things.
My drawâ¦
Two, two exam- two examples I go back to that I think are just so,
so awesome examples, and they're very, very simple examples.
One is Sherlock Holmes, and it's-- He, he, uh, he wr- Arthur Conan
Doyle wrote about Sherlock Holmes.
It's a sor- story, Silver Blaze, 1892 short story, where there's a disappearance
of a valuable racehorse within it.
And, um, the Scotland Yard detective asks, "Is there any other point
to wish to, to which you would wish to draw my attention?"
And Sherlock Holmes says, "To the curious incident of the dog in the nighttime."
And the Scotland Yard detective says, "The dog did nothing in the nighttime."
And Sherlock Holmes says, "That was the curious incident."
And it was the lack of barking by a dog that's out there that
would bark if somebody was there.
That w- the lack of information was the incredible information there,
and that's something as statisticians that it's so important in what we do.
It's not just the data, it's how did we get the data?
What data did we not see?
How, uh, you know, was this prospectively defined?
Was this one of 1,000 things done and presented to us?
That has very different meaning.
The data are the same, very different meaning to us making inferences
about something than if it was the primary analysis in the trial.
These are so critically important within it, but it can be
presented differently to us.
And so that example of making inferences from data, I, I feel like is what is
so attractive to me about science, about what I do as a statistician.
The other example is theâ¦
And this is also a famous example.
Um, by the way, I don't know if this example, you know what I mean, the whole
sense of this is whether this is a, a tall tale of, of it, but it's such a brilliant
example, uh, I'll assume it- it's true.
It's a story of Abraham Wald making, um, looking to reinforce fighter j-
fighter planes in World War II, and the data being collected were the
location of bullet holes in planes when they went out and during a battle.
Within it, you, they, the location, you, they got data on where the holes were
on the plane And the, the fascinating thing was his conclusion was not to
reinforce where the bullet holes were, but to reinforce where they're not.
Why is that?
Because of the survivorship bias that you only see the planes that
survive and come back, and they have bullet holes in certain locations.
What you don't see are the planes that go down.
You'd love to seeâ¦
The best data would be, let me see where the location is of the
bullet holes that took planes down.
That's where you wanna reinforce.
You don't see that data.
You see planes that survive.
There's a survivorship bias.
And so it's almost as though where those bullet holes are didn't take the
plane down, but assuming a uniformity of bullet holes over the plane, and
there, there's an assumption there, it's where they're not, uh, within it.
Such a cool example of inference from data and science, uh, wi- within that scenario.
And I believe these are the kinds of things in clinical trials.
By the way, this survivorship shows up as incredibly important thing in making me-
in medical decision-making within that.
And I think as scientists in clinical trial, not, notâ¦
I, I mean, this is, this is also, it's not just statisticians.
We, weâ¦
This is something we very much deal with, but clinical trial scientists,
clinicians, everybody involved in the enterprise of clinical trials and
medical decision-making, th- this is such an important part, uh, is making
inferences from the data Within that.
So that's-- It's the joy of, of, for me, clinical trials and
science and the importance of it.
And there are a lot of parts of, of clinical trials that, um, i-in some
sense, people maybe think are un- uh, uh, you know, not appropriate.
Uh, many times, uh, running clinical trials, sponsors will look at subsets,
look at other endpoints, and see data that looks really good and believe the result
of that, not understanding the process of looking at many things is a real, uh,
a part about drawing those conclusions.
You're very likely to find something positive which lowers
the inferential strength of that.
Doesn't mean it's wrong, but it lowers the inferential strength of that, uh, in that.
So these post-hoc analyses, i-it's, it's part of the science of these conclusion.
Survivorship bias absolutely shows up in clinical trials.
If you're comparing to a natural history s- database and you have a
clinical trial where people enter the clinical trial, there are people
in the natural history study that wouldn't get in the clinical trial.
Maybe they die before they would get the drug.
And if somebody has the disease and has a period of time before they are randomized
in the trial, and now they take the drug, it creates a survivorship bias for people
that took the drug that somebody in the natural history that had the disease that
died before they'd get the drug would be in the natural history study, but
they wouldn't be in the clinical trial.
And then the clinical trial, uh, uh, people, uh, patients do better in
comparison to the natural history study.
Such a clear absolute survivorship bias.
It shows up in many times about making inferences about data.
It's about fairness i-i-in, in all of this, and it's-- We, we are
the officers of, of this fairness.
Now interestingly, uh, I, I and, and, uh, Berry Consultants many times are involved
in non-standard clinical trial stuff.
We enjoy being involved in hard clinical trials.
In the most sort of standard clinical trial that you enroll two arms to
a fixed sample size, and you do an analysis, no interim analysis.
In some sense, the, the, the, the cleanest data you can have.
Um, and it's awesome.
The, the, these trials are awesome, but we're typically involved in
adaptive designs, Bayesian analyses.
The pod-- The last episode of, of the-- In the Interim talked about
bias in the estimate of the treatment effect when you have futility stopping
or superiority stopping and, and, and the biased estimate of that.
And by the way, it's, it's very different.
It doesn't mean it's wrong.
It has-- This has influence, and when we get the result,
we understand the influence of that, uh, completely within that.
Borrowing from subgroups.
If you borrow from an adult to a pediatric, uh, patient,
it creates bias in that.
Um, if you borrow from group A to group B, you use controls from a different
era, uh, in a platform trial within that.
Is there potential bias in that?
Now, all of these things, you know, maybe it feels strange that I started this off
with this very high level, um, that this feels unfair that this player is allowed
to play in the game W-within the setting, but yet now talking about adaptive
designs and Bayesian and all that.
It, it's, it, it's incredibly, um, important that we can't
keep running those trials.
We miss shots on goal.
There are drugs that are not explored.
We have to be efficient in patient and time and resources.
Uh, otherwise, we get three really good answers, and we didn't
ask 100 questions within that.
There's an efficiency to this.
Now, this can be, and it is science.
Bayesian is science.
Adaptive designs is science.
Pre-specification in these settings is so incredibly important.
Covariate adjustment is science, and it's incredibly important in that.
It's not cheating to adjust for covariates within the setting.
You might look at the data as the raw data of A and B, but a covariate
adjustment gives you a different answer.
Is that cheating?
No.
We understand that as science.
We can have very clean science.
Let me give you another example.
Suppose you were collecting data, and you were comparing to natural history.
We understand, and I just presented how there's a survivorship bias in comparing
those natural history patients to patients in a clinical trial setting.
Does that mean we can't use it, or does that mean we could potentially adjust
and estimate the survivorship bias?
That can be done in very clean scientific ways that are honest and, and is
science, and that's where w-we, we kind of live on this, this border and,
and, and the hard part is that's not cheating, uh, uh, within the setting.
And so, um, adjusting for it, understanding the bias and early stopping
and understand making conclusions from that is still incredibly important, and
it's an incredibly positive thing for clinical trial science, and that's kind of
the s- the space we live in, and I think the importance of us sharing these ideas
and presenting this, uh, uh, within it
Now, it's, it's the evolution of the standard trial that's 300
patients, five years, look at the data, make a conclusion within that.
This can all be done and is fair.
It's a little bit like the advances of adding video review to it, of,
of new rules within the setting.
It's different than it used to be, but it can all be done in a, in a fair way.
There could be an avenue where a red card is reversed as long as it goes
in the rules of how this is done.
And within this particular setting, it, it went outside the boundaries.
There isn't even really a setting for that, uh, within it,
which hence that feels unfair.
So we all as statisticians, clinicians, clinical trial scientists,
regulators, sites, uh, all of this, we're all officers of science, uh,
in this, and science is evolving.
We are making this machinery better and better and better, and it, it improves
the human condition to understand the truth about different treatments,
the, the efficacy and safety of it.
It's, it's all such an important thing, and maybe that's why I reacted to that
particular case in, in such a way.
Or maybe it's because, uh, I became a statistician because I reacted to that.
Uh, I, I, I don't know which one of those it is, but I, I did
have such a reaction to that.
So everybody out there, keep up the incredible important work
you do, um, uh, within this.
And until the next time, we will be here in the interim