Subscribe
Share
Share
Embed
A Mayo Clinic podcast for laboratory professionals, physicians, and students, hosted by Justin Kreuter, M.D., assistant professor of laboratory medicine and pathology at Mayo Clinic, featuring educational topics and insightful takeaways to apply in your practice.
This is Lab Medicine
Rounds, a curate podcast
for physicians, laboratory
professionals and students.
I'm your host, Justin Kreuter,
the Bow Tie, bandit of Blood
a transfusion medicine
pathologist at Mayo Clinic.
Today we're rounding with Dr.
Bradley Erickson, director
of the Mayo Clinic
Artificial Intelligence Lab
and professor of radiology
at Mayo Clinic to talk about working
with artificial intelligence
and also how to train on it.
Thanks for joining us today
Dr. Erickson.
Thank you for inviting me
to talk with you today.
So why don't we kick
stuff off and maybe define
like the importance, why
are computer aided diagnoses
artificial intelligence
important for healthcare?
There? There are a number
of advantages to AI algorithms as long
as they're appropriately implemented.
In a lot of areas of medicine
we humans tend to be more qualitative
and that can be very
good, but in other venues
it's important to be more quantitative
and computers are particularly
good at those sorts
of tasks and, and AI
falls into that category.
For example, measuring the size of tumors
or other disease processes is
a quantitative task that we
humans usually don't
like to do all that much
but computers are very good
at doing it and they can
do it very efficiently.
And so this is one of
those cases where you
get a win-win situation where
the computer does it better
it does it faster, and
it takes gru work off the
the shoulders of, of the physician.
So I think
that there are some areas
where it's a natural win
for AI to help us do our task better.
The other thing is that for
tests like detection and
and making sure that we
don't miss certain things
having AI there watching over
our shoulder can be valuable
particularly when it's
3:00 AM in the morning.
And, and I know I'm not at the top
of my game at that time or you
know, at the end of the day
after looking at hundreds of
thousands of images, again
we humans tend to fatigue
and AI tools don't.
And so having that kind
of extra set of eyes looking
at things saying, Hey
don't forget about this, or
what do you think about this?
Something to make sure that,
that we give as good attention
to the last case as we
gave to the first case
I think is another value of ai.
You know, as you were
talking there, in my head
I'm thinking about checklists, right?
And like in the operating
room there's checklists
and you know, in the aviation
industry checklist is
I guess is there one way or maybe it
is it one facet of AI that's kind of
like the next generation
of a, of a checklist?
Well, so the, the more typical way that we
implement checklists is what's
called structured reporting.
So when I interpret an examination
there will be a number of, you know
what about the such and such?
What about this, what about that?
And particularly then if the
computer prompts me and says
here are the legal answers
that also then can be nice
training data for training in AI.
And now there are already
companies starting to
show AI tools that can
generate structured reports.
And so you have that double advantage
of the computers looking at everything
and it lays it out in a
nice organized fashion.
So you, you mentioned
in your first answer explaining
why is it important talking
about, I kind of caught
your highlight on the
if it's appropriately implemented.
And so it kinda leads me to the question
of what's important for
physicians to understand
about working with
artificial intelligence?
So when I give talks about AI,
I try to emphasize the point
that despite its name,
AI is not intelligent.
The more correct term in the field is to
call it machine learning or deep learning.
And it's learning a pattern.
And so you could feed it whatever you want
and it would figure out the pattern.
And while
for humans who are really
good at memorizing patterns
which is what a lot of medicine is about
we think of that as being intelligent.
And so that, that's kind of
the, the origin of the term.
But the, the computer is ultimately just
doing a pattern matching thing.
And the danger then is this.
Somebody I know actually
took an x-ray of a pickle
and fed that x-ray into a
cancer detection algorithm
and the algorithm said
there's cancer there.
The problem is that
there's no common sense
that we humans would
think of when the AI runs.
It is just saying this most looks
like this and that's a big problem.
And I think
that that then kind of
gets me to the next point
which is we need to think
about confidence levels of AI.
The current generation
that we have basically
says it's probably this, but it
it doesn't give a lot about
the actual probability value.
It just tells you cancer or no cancer.
And the ability to have
it convey a calibrated
probability as well as a confidence value
I think is critical.
If you think about your
interactions with your physician
and you walk in and they say lung cancer.
How much confidence would that give you?
You know, sometimes it
is just about that clear.
But other times and
probably most times it's more
of a differential diagnosis.
And, and that's kind
of where we need to get
with AI is that we get
that list of possibilities
with some sort of indication
of the confidence level and
those technologies
are being developed,
but we're not there today.
And, as they're
getting developed, is there
kind of a standard way that
the community is thinking
about talking about this
confidence and probability?
Is that going to be kind of universal
for the different tools that
are kind of being developed?
Or is it each, each
to their own the way they
kind of try to convey that?
So we're still early
enough on the development
that each is kind of doing
it their own way, you know
and until we have a more clear winner
I'm not sure that people are
gonna put too much effort
into standardizing that.
In some of the structured
reporting technologies
there are fields for putting
in a confidence value
but the precise way to
interpret that is still
not to defined, you know,
that's actually
a big problem when you think
about our language today.
If I read out a chest x-ray
and I say that's probably pneumonia
does that mean I'm 99%
sure 90% sure 56% sure?
Right? What does probability
mean in a quantitative sense?
And that's a big challenge then in
in terms of creating training data, right?
How do we train the algorithm
that this is what a 56% probability means
but also then how do you map
a number back to language that
that we would understand as humans?
So, that's a big challenge that
that we have today is
that language and humans
are not quantitative the
way that algorithms are and
and thinking about what probabilities
and confidence terms mean is a challenge.
Yeah, I, as I hear you
say that, I think in
pathology, you mentioned
the radiology challenge
when you say this is probably. You know
in pathology there's certain aspects
of our practice where we're talking
about something is
suspicious for something is
atypical something you
know cannot rule out.
And and I guess I'm sort
of reflecting now that
we try to convey that
probability exactly like you said
actually as a, as more of a
subjective rather than
you know, qualitative rather
than a quantitative way.
So is the thought then that there's
that'll help us get away from
some of the, the biases in
in our clinical practices?
Yeah, you know, so bias has
several different components.
You know, I have a, an electronics kind
of background and you know, we
always tend to think of bias
as a bad thing, but of course
for those of you who know
electronics bias is what
makes transistors work, right?
So bias if properly used,
can be a good thing.
How does that apply to AI?
Well, in terms of bias
and particularly, you know
underrepresented populations and so on
we know that some races
genders and so on have
different risk profiles.
And so to say I'm going to
be completely blind to race
or sex is probably not the right approach.
You just need to make sure
that you use that information
to provide the best care
for patients.
And so as we then start to again
produce these probability estimates
you know, that information
is hopefully going to
improve the confidence intervals be
because we have that
additional information
about the the sex and
race of the individual.
So in your role as a director
of our artificial intelligence
lab, how do you go about
or how are you thinking about how we train
you know, our trainees,
our residents and fellows
how to use artificial intelligence well?
I imagine that's
starting to kind of enter
into your life and what can
you share with our audience?
So I try to make the points that
that we've already
discussed about the fact
that it's not intelligent,
it's just doing patterning
and that as long as you give
it an input, today's generation
of AI tools always produce an
output, even if it's nonsense.
And so, you know, I think it's critical
that our trainees need to
get at least some exposure
to AI technology to
understand how it works.
And it of course, more
importantly how it fails.
And you know, I draw a lot
of parallels with
statistics that, you know
even back in the dark ages
when I went to medical school
we had to take statistics
and epidemiology.
And I think that that's
a valuable thing, right?
You have to understand
how to read the literature
but also when you're looking at, you know
a BMI that's, you know, at
this value, well how far
off of the population norm is
that and what does it mean?
And I think there needs to be
at least as much time spent on training
about AI tools in medical
school and residency and
and so on, so that they
understand again, the principles
of machine learning, how
it works, how it fails be
because it's probably going to
have even more application
than medical care than than
statistics and epidemiology.
Is, is there a good, yeah
this is a bit of an ignorant question
in that I, I'm not
sure, like, you know, if
if I wanted to get my
residents and fellows exposed
to AI now, like I'm not sure if there's a
going into old school
things thinking is there
is there kind of the
the recommended textbook
on it or is there something, you know
in our current practice
now where I could have
somebody go deliberately
kind of practice with or are
is there some online
tools that, or a place
that somebody can go a digital playground
and get exposure to and,
and come to appreciate these
points you're highlighting for us?
Yeah, so as, as you kind
of suspected, you know
textbooks are pretty much useless.
They get out of date so fast.
Things like chat GBT, you know, didn't
didn't exist at least in the knowledge
of the population three,
four months ago, right?
So unfortunately textbooks
probably don't cut it.
So to address this problem
there's a guy named Jeremy
Howard who has built a number
of what are called Jupiter Notebooks.
It's a way that you can execute code
but it gets the name Notebook because it's
like a scientific notebook
where you also see the output
and you can put in hypertext
markup like a webpage.
And so he actually wrote a
textbook that is all code and
and these Jupiter Notebooks.
So stealing his idea, my lab
and I have created a website focused
on medical image deep learning.
So if people are interested,
that's at midel.org.
And that's something that, you know
because it's web content
it's a lot easier to keep up to date.
We can add a new page when
some new technology comes along
if there is a bug, you know
unlike a textbook where
you have to publish errata,
we can, you know, update
the code pretty easily.
But I think the ability to
actually see the code run
and people say, gee, I wonder
what happens if I do this?
And they change a bit of the code
and they can see the
impact I think is extremely
valuable for, you know,
early to mid-level learning.
There are courses and in
fact Mayo offers a master's
in AI for medical people
and that gives you a more
in depth learning experience
but obviously requires
a, a bigger commitment.
So there are a number
of options, but you know
I think web resources
probably is the way to go.
YouTube is fantastic,
the challenges that most
of YouTube content is
not specific to medical
but in terms of learning
the general concepts of AI
YouTube is kind of my go-to.
It's, it's wonderful to
hear all these resources.
I'm just kind of curious
have you started to have
program directors approach you
in your laboratory to facilitate education
around AI in their department division?
Yes, and I have, you know, gone
and done the typical visiting
professor thing to do that.
But in addition, I'm
part of an informatics
society that has created
what's called NIIC-RAD
the National Imaging Informatics
Curriculum for Radiology.
And that covers a lot more than just AI.
It talks about, you know,
how do you move images around,
how do you do structured
reports and whatnot,
but we've added AI content to that.
And so that is a week long
webinar that is available
to all radiology programs.
It's actually now across the world
not just the US and so
because it's really not feasible
for many of the smaller programs
to have an expert on AI.
And so this is a way that
we can educate, you know
essentially radiology
programs around the world
on AI as it applies to radiology.
And there are discussions
with other societies
like pathology about
doing a similar thing.
Yeah, that's wonderful.
So my, I've heard other
colleagues kind of talk a
about the future of medicine is
is being handed over to the
robots and what's, you know
the role of the physician
is really to still
have maybe that healing touch or comfort.
But as I hear you talk and
and really talk about how best to use ai
I would gather that's not
the future vision that
that you see.
So what do you think the future
of of AI in medical practice looks like?
So I think that, for instance
AI doing more of the quantitative
tasks and doing some
of the grunt work that we
physicians don't like to
do is the sweet spot.
We focus too much on doing the
sexy, it can make a diagnosis
that a human can't do and
it's cool when that works
but I think the payback
for that is relatively small
compared to the investment.
But I think, you know, those
sorts of tools are coming
we and others have
published on the ability to
protect molecular markers
from standard CT's and MR's.
That there's no way a human
can see what the AI is seeing.
I think that the routine
quantitative measurements
of things like body composition,
the amount of visceral fat,
subcutaneous fat, and muscle is valuable
to many clinicians today.
And having a human trace that
out is simply not practical.
But we've already deployed an algorithm
so actually every abdomen CT done
at Mayo has a body
composition available to it.
They don't routinely report
it, but it's available if
if people want to see it.
I think that the generative
technologies is kind of led
by Chat GPT is also going to
change medicine. Now Chat GPT
we all know
about the hallucinations
where it will make up
really plausible sounding
things that is complete garbage.
But there are variants that don't do
that where that there's
what's called the temperature
which is how much you
weight the probability
and how much you want to
right weight randomness.
But also it can say
and this is the document
where I got this idea from.
And so I think for
summarization, you know, going
through the 30,000 pages of
outside records, great task
for some of these generative
technologies where
you say give me a one page summary
of all the hematologic disease
of this patient in their life.
I think it's very feasible in
the not too distant future.
And if we can do that
with text, there are also
some great generative
technologies for images.
And, and this is one
where my lab has done some
work where we can take for
instance a large collection
of hip x-rays or chest x-rays
and if we also know the sex
and the age and the race and the BMI
we can train a model
where you can then say
generate 10,000 x-rays of
the pelvis with this many
from this age range,
this many, this age range
this many, this sex, this
many the other sex, you know
this many with a certain BMI,
this many a certain race.
And so we can completely
sample the population and none
of the x-rays will be
from any one individual
which then gets around privacy concerns.
So I think those sorts
of generative techniques
to improve the training
of other AI algorithms
is really cool thing that
that we're starting to understand better.
Finally, I think there are components
of AI that will go back to the roots.
And so for those of you who are historians
the first real AI application
in medicine was called MYCIN
and was a set of about 500 rules
for making the diagnosis
of blood infections.
And you could ask it, well
what's the best antibiotic to use
for this type of infection?
And, and that ability to
control things as opposed
to being susceptible to
hallucinations is a real problem.
And so if you can define a set
of rules and say, when you see this
do this and then do this and then do this.
We have a lot of workflow challenges
in healthcare where handoffs are dropped
or certain steps aren't done
in time or they're not done according
to the care process that we all agree on.
And I think that that form of workflow
something called process
automation that's used
in manufacturing of cars
it's used in the financial industry
but it's not used in
healthcare for some reason.
And I think that that's actually a form
of AI that probably is
going to start to be applied
in the next five years or so.
Wow. I think all of our
audience right now really keyed
into your, your
predictions for the future.
Because many of us are doing chart reviews
in preparation for frozen
sections the next day.
And, and also there's lots
of times where I have to
comb through a lot of data.
I hadn't even thought about that.
And then the, the medical educator
in me is just thrilled at the, you know
what might be possible with
almost essentially, you know
I know what, what I would love to have
and expose my learners to, but it's going
and grabbing that materials
that that takes so much time.
We've been rounding with Dr.
Erickson on the importance of working
with artificial intelligence
and how to teach it.
And thank you so much Dr.
Erickson for taking
the time with us today.
It's been my pleasure.
It's been great to talk with you all
and I hope that this was
valuable to your audience.
And to our listeners, thank
you for joining us today.
We invite you to share your thoughts
and suggestions via email.
Please direct any suggestions
to MCLeducation@mayo.edu.
If you have enjoyed this podcast
please follow or subscribe.
And until our next rounds together
we encourage you to continue to connect
lab medicine and the clinical practice
through insightful conversations.