Subscribe
Share
Share
Embed
Most training is sold on confidence. Show Me The Evidence is built on data.
In every episode we take a single study, clinical trial, or systematic review and work through what it found, how it was designed, and what it means for the way we teach and assess skill. We focus on metrics-based training and proficiency-based progression, the approach that asks learners to demonstrate measurable competence before moving on, and we trace its results across surgical, medical, and professional education.
This is a podcast for learning professionals and medical educators who want more than opinion. Expect plain-language breakdowns of the research, honest discussion of what the evidence does and does not support, and conversations with the people behind the studies.
If you make decisions about how people are trained, we think you deserve to see the evidence first.
In the late 1990s, Dr. Dwight
Meglan and I met at
Medicine Meets Virtual
Reality
somewhere in California,
I seem to recall.
Dwight's an engineer. Initially,
when I met him, he was
working on physics-based virtual
reality simulation.
He subsequently moved into
surgical robotics.
You're very welcome
this evening,
Dwight, and nice to
talk to you.
So you developed one
of the first
physics-based virtual reality
simulators.
I purchased one in 1999 at
some considerable cost.
So at that time, how did you
envision that the VR
simulation market would
develop?
And has it developed the way
that you thought it would?
I have to be honest, I really didn't
think about it a whole
lot about relative
to a market.
I was more thinking about it
as a technical problem.
I was just interested in the
idea of how do you do
real-time physics of
tool tissue
interaction and replicate the
real-time fluoroscopy
and give haptic feedback
in a way
that was natural to the
procedure itself.
And I was much more focused on
that than anything else.
And I really didn't give it a
lot of thought other than
there's a lot of catheterization.
So I figured it was
probably going
to be useful and people
would use it.
And do you think that the current
developers of virtual
reality simulation pays much attention
to the physics-based
virtual reality simulation
today, as you did back
in the 1990s?
I don't really think many of
the simulators pay much
attention to that, no matter
what it is.
I was kind of obsessed with
trying to mimic the actual
physics of the tool tissue
interaction.
And I figured if I could get
that right, then the
experience for the user would
reflect reality to
some degree.
So I kind of focused on that.
Whereas, as you well know, a
lot of the simulators kind of
focus on the user experience
itself.
I actually didn't really think
about the user experience.
I just figured if I did
the simulation
right, then the user experience
would be right.
So what do you think have been
the drivers of the current
state of virtual reality
simulation today?
Well, over the last 20 years,
20-odd years?
Basically, the device manufacturers.
I mean, I figured
early on, I didn't
really give
that a lot of thought,
to be honest.
I was just thinking about it
relative to build something
that's useful for the
clinicians.
And that would be something
that would be meaningful.
But as I spent more time working
in the area, because I
worked on other simulators
besides endovascular,
I came to understand that really
the only people that are
really paying for this and really
caring about it a whole
lot were the device
manufacturers.
And they kind of drive the way
things work. And that's kind
of what it seems like it
is today to me.
I mean, there's various professional
societies that have
set up standards for learning
and other things.
But as far as I know, like the
thing I was driving towards
years ago, like 20,
25 years ago,
was trying to figure out how to
come up with physics tests
that would allow you
to verify
the correctness of the
simulator.
And those still don't exist for
the most part, as far as I
know, which I'm kind of
surprised at.
Yeah, but one of the things that
always struck me about the
sensation that the or
the perceptions
are that the user actually had
from the simulation. People
have different sensory
thresholds.
Therefore, the sensory
thresholds are
going to impact on what
they perceive.
And one of the one of the interesting
things I find out
over the years was the plastic
surgeons or microsurgeons.
They won't go for a run. They
won't have coffee.
They try to get a good
night's sleep the
night before they operate
because they know
that they can, a slight
nudge can
impact their performance
the following day.
So it's very hard to actually
come up with, you know, what
are the sensory thresholds for
the clinicians to feel?
So how did you approach that
or try and solve it?
Well, so we did do a couple of
things early on because we
were kind of curious about
how would we
know if we were producing
things that the cardiologists
felt.
And what we ended up doing was
we built an instrumented
torquer that the cardiologists
would
hold to actually manipulate
the tools.
And we figured if we could
measure what
they were actually feeling
in real life,
then we could actually, and we
knew what we were feeding
back to them on the
simulator,
that if they were at least somewhat
correlated, that we
were actually doing at least
some level of a good job.
We never went far enough along
with that to actually try
and figure out if there's
different thresholds for
different people,
which of course there is. We
know from haptics that
different people have just
noticeable difference
thresholds.
But we never pursued
that. I mean,
this was never done
academically.
This was all done in the
auspices of a company.
And since I'm not super active
in the field anymore, like
you said earlier, I mostly work
on surgical robots now.
I'm not aware of anybody having
published extensive data
sets on what it is that a
cardiologist feels when they
do catheterisation.
So there's not minimal
data out there
to even test against,
quite frankly.
And we certainly don't know
much about the actual
interaction of the catheters
with the tissue,
which is what I was really headed
toward. I really wanted
to eventually be able
to say that
we're actually generating the
same forces, contact forces
and pressures and
motions that
you see in real life
quantitatively.
Yeah, I think that's still
a real problem today.
Sometimes I had four of the simulators
that you built when
I worked in University
College Cork.
And occasionally the haptics
weren't switched on and the
clinicians would be
telling me
only the haptics in this
are great.
And I'm not sure what's
going on
here. I mean, I know
about that.
I think over the years, I think
it's just a personal
conclusion that I've come to.
Is it the better the
clinician is, the
more attentional capacity
that they have,
the greater awareness
they have to the
subtle aspects of catheter
wire performance?
And I think it's the same in
robotics or actually
there's no.
Well, except for in the DV5
now does have it.
So, and I've never used it, so
I don't know what it's like.
So.
So you think one of the
biggest drivers
of the physics based virtual
reality simulation
was primarily the
manufacturers.
I mean, and the conclusion
or the observation
I've sort of made is they
seem to want it only
for mostly for marketing and
that education and training
seem to come well down the
track after that.
Would you agree with that?
I mean, everything's subservient
to a purpose
when it's a product, right?
I mean, they're trying
to sell stuff
and that's just the
way it is.
They'd like you to
use it well,
or at the very least
they'd like you
to not use it dangerously,
because if nothing else, it's
bad on their reputation
and maybe it can be more
than that,
depending on the situation.
Could get them into liability
issues or whatever.
So they want you to use
it in a way that
they consider you to be competent
at running the device.
But I mean, I honestly, having
worked on the simulators
and on the robots, I mean,
there's a threshold
at which they kind of
draw the line.
They just want you to be able
to use the tool
in the way that they had
designed it to be used,
at least to their best
understanding
of how it designed it
to be used,
but not, they don't,
as you know,
they don't claim any
responsibility
for the practice of medicine.
So they're not trying to teach
you how to do the procedure.
They're just trying to teach
you how to use the device.
So that's the cutoff.
I'm not entirely sure, but
I had a conversation
with a manufacturer
last week,
and it was about the use of
a stapling device.
Okay.
And the stapling device
has a six to 27%
leak and complication rate.
And one third of the patients that
develop a leak will die.
That's huge consequences
for a relatively--
And the conversation that we're
having now saying that
basically what you want to
be able to do is,
now, well, number one, we
have good evidence
that a lot of the very experienced
surgeons are,
I don't know that they're
not able to,
but they certainly don't know how
to use the device safely.
Okay?
And to what I said to the
manufacturer,
I said, you really want to be
able to train the surgeons
to use your device safely
according to the IFU.
And what they were concerned
about was
that what I was proposing was
that they were training
surgical skills and I wasn't.
I was just arguing that they
should be training
to use the device safely.
Yeah, I would agree
with that,
with what you just said on,
I mean, this is all just about
doing the stapling itself.
It's not about doing
the procedure.
And I mean, that's what the
stapler is, right?
So yeah, I can see that.
But I would have thought
that would be
that the manufacturer would
take that in
as that is their
responsibility,
is to train them how to use
the device effectively,
which basically means safely when
you can write down to it.
That's correct.
I mean, and a lot of the
problems that we see
with new devices or even
old devices,
the clinicians will blame
the device
but say that if something
was wrong,
it was the device fault.
And I mean, for the cardiovascular
devices,
the implanting, the cardiac
defibrillator.
See, clinicians, they don't follow
the instruction for use.
For example, when you're
suturing the device
into the packet that
you've created,
some clinicians don't use
any sutures at all.
They don't use the sleeve
that covers the leads.
They use the wrong type
of sutures.
They use to dispose or sorry,
sutures that melt away
rather than, you know,
For a minute.
Surely I would have thought
that the manufacturers
would be concerned
about that.
I would think so.
Yeah, but yes, they're
concerned about it.
And their trainers were
telling them
that maybe some of the
clinicians
that were training
the juniors
were not maybe as good as they
originally perceived.
You know, they had a senior
position.
They had a high volume of
procedures per year
according to their CV.
But when you looked at their
performance,
they really didn't
do that well.
And over the last decade, what
we've been publishing,
you know, for the concert
validity of the metrics
for the different devices,
what we've been observing is
that some very senior
clinicians
perform worse than the
worst trainee.
Now that's problematic.
Yeah, no, that's not good.
Yeah, it's, you know, the one
question there is,
you know, are they really
that bad
or is it because they're
not acclimated
to the device and the technology
never have been,
you never learned how
to use it well.
This wasn't on the device.
This was actually video recording
from actual patients.
Okay, just, okay.
Yeah, and the metrics
weren't made by,
us, they were made by
their peers
and for a straightforward
procedure,
not for a complex procedure.
Yeah, yeah, well, these,
as you know,
when you're doing things
like this,
we were talking about,
these are all gonna be observational
metrics, right?
Because it's surgeons watching
other surgeons
so they don't have any other
thing they can do
other than watch and say
what's going on.
Whereas, you know, like
in a simulator,
you could actually measure
the actual forces
and, you know, things
like that,
you know, like tool tissue
interactions and stuff
and turn those into metrics
if you wanted to,
but then nobody would know what
those metrics need to be
because we don't know what that
data is in real life.
So we're kind of stuck doing
the same thing,
just making it fancy and
quantitative
in terms of like tools moving
around and stuff.
You know, all the economy,
emotional,
the other stuff.
We'll get to that
one another.
So when you were building the
simulator in the 1990s,
what was your view on what the
simulator should measure?
And-
Basically everything.
There's only one thing worse
than having no data.
That's having too much data.
It's too much.
So measuring everything.
Well, no, no, no, no, there's
a reason for that.
So when I did the original
VIST,
I did try to do a harness
on it
that would record literally
everything,
like every force, every
torque, you know,
every contact point, like
everything,
because I figured eventually we
were gonna wanna go through
there and try and make
sense of it
in terms of what would
give us measures
that would be kind of
telling you
whether you did it right,
whatever it was.
And so the original intent
was literally to record
as many things as we could
and then go rifling
through it
and try and figure out what
the essential measurements
would be that would tell you
whether you had actually
learned how.
The, so the simulators were
originally supposed
to be designed around
learning goals.
And we wanna be able to have
a way to actually
a certain that you had learned
whatever the goal was.
And so our intent was that we
would have some kind
of measurement of that.
And since we didn't know
what that would be,
we were like, let's just do
built-in suspenders
and record everything and
then we'll figure out
how to pare it down to something
that makes sense.
Well, to some extent,
we did that
for the mechanical thrombectomy
for acute stroke.
I mean, we developed the metrics
from the human procedure
along with Mentice.
And then what we did was,
myself and the three interventional
neuro radiologists,
we imposed the metrics
well, Mentice did.
We told them what to do
and they imposed the metrics
on the system.
And one of the things that
the simulator didn't do
was when you inject contrast
into the brain
and into the brain neurovascular
structure
through a microcatheter, they
were using syringe with air.
And it just plunged straight
down and--
Wrong flow characteristics.
Absolutely.
And the interventional neuro radiologist
stood the ground.
They basically said, that's
not gonna do.
Yeah. And we're not standing
over this.
So Mentice had to develop and
subsequently patent
a new methodology where they
were injecting a fluid
into a small device that
actually was made
the whole training more
realistic.
And I mean, it works when
you insist on it,
but you have to start off
with the metrics.
So you were trying to measure
everything
in the late 1990s,
early 2000s.
What did you end up,
what sort of metrics did you
end up producing?
Or the simulator?
Well, so I really was not the
one that did that.
That was others that decided
what the metrics would be.
I just put the platform
together
so that the information would
be available.
And then other people made
decisions about
what the actual metrics
derived
from that information
would be.
These were clinicians or clinician
engineers or engineers?
Yeah, my experience was that
it was a little bit
of everything.
It was clinicians in concert
with the companies.
The companies usually worked
with some clinicians
and it was usually a
small number.
And whoever their subject
matter experts was
that they were paying to
consult with them.
And then they would basically
for the most part
do whatever their SMEs said.
And all we did was
implement it.
Which usually meant just
pulling out things
that were already there.
It wasn't that often we had
to do something new.
So it was a combination
of stuff
that was already getting
dumped out
or that was already available
in the simulator.
And then turning those into
some kind of metrics
which were typically,
as you know,
they're observational
metrics.
They're nothing exotic for
the most part.
And
I know that you keep an eye on
the simulation developments.
What's your view on the metrics
that currently exist?
Not just in endovascular
simulators or
interventional cardiology
simulators,
but robotics and so on.
They haven't changed a
lot, basically.
They're kind of the
same things
we came up at the very
beginning
when you come right
down to it.
I mean, I think it's
slightly worse.
Yeah, I'm sure there's...
Okay, you're much more in the
fray than I am these
days, frankly.
I was at the ERA conference
in Paris a few months ago,
and I walked around all the
booths to look at the
simulations.
And almost none of them, I
think only one of them,
had any form of metrics
whatsoever.
So it was...
I mean, I was...
To be honest with you, I was
kind of shocked.
And...
Well, were they just using
them as a marketing tool?
I mean, a lot of the early
simulators we did were just
marketing tools.
You know, you stuck them
in a booth
and it was a way to show
off the system,
you know, whatever the system
was you were trying to sell.
And they really didn't care if
there were metrics or not.
It was just to give people
an experience.
Yeah. Well, I also talked to
some of the manufacturers,
the device manufacturers,
and they
were complaining about
the simulators
and the fact that they
weren't able...
Well, some of the manufacturers,
device manufacturers,
didn't want to measure
things.
Because...
Because if they...
They don't want to know.
Correct. They don't
want to know.
Plausible deniability.
Plausible deniability.
And other manufacturers complained
that the simulation
companies weren't producing
the
metrics that they required. And
they said, where do you
expect them to get the
metrics from?
You really have to derive the
metrics from real clinicians
doing real procedures
in real life
situations. And a straightforward
procedure, not a complex
procedure. And when
they go to the
manufacturer, they say, these
are our metrics, our
validated metrics. If you
can build the
metrics into your simulator,
we'll purchase it. If you
can't, we're not. So
I mean, go on.
That would require people
to actually
say that, that that's what
they want to do.
You know, if people
still buy the
simulators, even if their
metrics aren't there,
you know, what am I supposed
to say? Yeah, well, I think
something's got to
give, because
what we're seeing is that the evidence
is fairly clear now.
The intraoperative performance
of the
clinician is going to impact
on the patient outcomes. I
mean, the data is there
now. And
with a new device, it's more likely
that a clinician who's
not trained, you know,
to a level of
proficiency, they're going to
perform less well than
somebody who's actually
been trained to
efficiency. And that's the data
on that's very clear. So I
don't understand why
there isn't,
you know, a drive by the manufacturer,
the device
manufacturers to insist
on metrics being
implemented in the simulator.
I don't know what to tell
you. I mean, maybe they
don't think that
it's going to doesn't make a
difference on selling their
gadgets in any form. It's
just a cost,
to do this. And so if it's not
going to make them sell
more, why do it? I
don't know.
Well, I mean, the other side
of that is there's devices
that have failed in the
marketplace.
And it wasn't an issue with the
device. It was the user of
the device. It's been
held back
and repeated the introduction
of that device into the
market. And I've been
told that it's
hundreds and hundreds of millions
in some cases. Yeah, I
think a lot of times people
don't think
ahead of like think ahead like
that. Or those or those
stories don't get around
well enough,
right to management at the medical
device companies to
realise, really, you know,
you could. Yeah.
Hmm. So in the early 2000s, she
appeared to move away from
simulation and more
into surgical
robotics. What were the major
drivers of that change for
you? So I basically
kind of felt
hemmed in with surgical
simulation.
You were always constrained
by,
there's not that many things
to be simulated,
at least at that point.
And it was always driven by
the device manufacturers,
whatever it was that we were
gonna be doing.
And quite frankly, I kind
of felt like
it was kind of a lousy
business
because you were inherently
capped
as to how many things you were
gonna be able to sell,
how much money you were gonna
be able to make.
Not that I really cared that
much about it,
but it just didn't seem like it
was a major growth field.
And surgical robotics had really
just started to get going.
And it was interesting,
and there was a whole bunch
of different things
to be done there.
So I just decided to move
over there.
And as it turns out that
surgical robotics
isn't all that different from
surgical simulation
in terms of sort of the systems
engineering stuff,
which is more where
I work at it.
So I like working at the
real-time sensing,
real-time computation,
real-time actuation,
and then potentially
real-time graphics,
whether it's stereo
or whatever.
And simulators are actually
very similar to robots
as far as that goes in terms
of all the different parts
having to go together and
work in real-time.
So I basically had a skillset
already done
because of the simulators that
moved over onto robots.
And so I just started working
on the robots
and kind of been going
ever since.
So you agree with Rick
Satavo then
that the VR simulators and
surgical robotics
have an analogous DNA,
if you like,
as they're both information
processing systems?
Yeah, they're both
interesting.
They're both systems where you're
doing real-time sensing
and computation and actuation
and typically graphics
generation.
The simulators don't nearly
have as high a bar to them
as the robots do.
I mean, the robot can actually
hurt somebody.
The simulator can't directly
hurt somebody.
It could indirectly hurt somebody
by teaching them wrong,
but it's not got a direct
immediate effect
of punching a hole
in someone.
So the robots have a much
higher standard
and there's much more that
has to be done
with verification.
In fact, robots do all kinds of
things with verification,
whereas the simulators
basically don't.
What do you mean they do
all sorts of things
with verification?
You mean within the system
or within the management
of the system
by the manufacturer?
When you're developing them,
they put together all kinds of
bench tests for the robots
to make sure that they have
the right forces,
the right velocities, the
right accelerations
that they hold up over time,
your reliability,
that they're stable, that the
control loops are stable,
all kinds of things
like that.
There's just dozens and
dozens of tests
that the device manufacturers
put together
to verify that the
specifications
that they've made for the robot
are what the robot does.
And I, at least in my
experience,
there's little to
none of that
that happens with the
surgical simulators.
It's, if anything, it's
validation,
which I basically don't
like validation.
Verification is like you did
the engineering correctly.
Validation is like, "I
think it's okay."
It's not precise for me.
That's like the end user
just saying,
"Okay, that worked,
I like it."
But they don't know why they
like it or don't like it.
I'm not sure I agree
with that.
I mean, validation-
This is from an engineering
point.
This has been an engineering
point of view.
Clinically, I mean, it's
not my methodology.
The methodology was agreed
in the 1970s
by all the education and
training establishments
in terms of how you validate,
whether something works or
it doesn't work.
Well, okay, so from an engineering
point of view,
validation is like, "Yeah, okay,
it does something useful.
They're actually learning
or whatever it is."
But I find that to
be imprecise.
From an engineering
point of view,
I'm much more interested
in verification.
I guess the way I look
at it from a,
like from a surgical robot
point of view,
verification and validation
is,
verification is I built what
I said I was gonna build,
and validation is I built
the right thing.
I consider that much
less precise
from an engineering
point of view.
Yeah, but it isn't
engineering.
And I mean, the ultimate test
of the validation
for a robot or a simulator
is,
does the robot do what it's
supposed to do clinically?
For somebody that can it
do the procedure,
can it do the procedure
safely?
And that's not the engineer's
realm, basically.
That's my point of view.
That's for the clinicians
and others,
but the engineer is
to verify.
Yeah, so in the 1990s, laparoscopic
surgery developed,
they extensively investigated
this whole surgical tracking
and path and path length and
motion and so on.
And it really didn't
go anywhere.
And surgical robotics seems
to be heading down
the same road as an approach
to the assessment
of operative performance.
Sure.
Oh, I think there's two
parts to this.
It's basically the only thing
people can think of
for the most part.
And it's gonna get
turbocharged
because of machine learning
in AI.
Yeah.
It's gonna be on steroids
because they're gonna be
looking for patterns
in everything with the data
they collect.
I have a feeling.
Yeah, but I mean,
I mean, it's like looking
for a target
and a very thick fog.
I mean, the whole thing
about science
is that you're supposed
to hypothesise
what it is that you're
looking.
I mean, the tracking stuff,
it seems to be like going on
a fishing expedition
without any--
It is, yep.
I mean, and do you think it's
gonna go anywhere?
I'm sure they'll find
some things.
I mean, this is the thing
with AI coming at it.
It's that it's able to
do so much more
than we could in the past.
But I mean, you really wanna
have a theory of operation
behind things of what is it
we're trying to achieve
and how is it you're
going about
making this thing attain
whatever it is
it's going to attain.
So I would think that you
basically wanna,
you have to have some
kind of concept
that drives why you do
what you do,
which is why you do the
things you do
where you break procedures
down
so you understand that there's
learning goals
and based on those learning
goals
that there's a scaffolding of
skills that support those.
And then from that,
you'd figure out how to
measure things
because those are the things
that support
your understanding of whether
you would obtain
those skills or not.
Yeah, so what strikes me is
that what laparoscopic did
and what robotic surgery
is doing now
and interventional cardiology and
the fluoroscopically,
they measure process and
they take that
as some sort of indication
of skills.
If somebody does lots
of procedures,
they're gonna be better than
somebody that does less.
If you're smooth with
the instrument
or you have a shorter
path length,
you're obviously more skilled
rather than you just jump
complete phases
or steps in the procedure.
And this process seems to be,
I mean, like they're
hanging on to,
this is a strategy for
measurement.
Yeah, well, so if it was
a perfect world,
I'll just use suturing
as an example,
because that's the thing
I tried years ago.
If it was a perfect world,
we'd actually know exactly how
much tissue apposition,
how much of the tissue faces
need to come be in contact
and how much pressure there
needs to be,
and how well distributed
it is.
And that would actually
be the measure
of how well you did suturing.
It wouldn't be an indirect
measure of like,
did you do it smoothly,
quickly, decisively,
all those things,
because that's not really the
measure of the outcome.
We know that the whole idea
with suturing
is you're trying to
get the tissue
to grow back together
and heal, right?
So you'd measure it based
on the metrics
that are reflective
of achieving
the ineffective environment
for tissue regrowth.
But we don't even know
what those are.
Back in the day, at least when
I was working on it,
that wasn't published
information.
Nobody had sat down and
built like a jig
and tried all kinds of different
amounts of pressure
and distribution of the
sutures along it
and the likes to see how well
living tissue would regrow.
So like, nobody knows as
far as I know.
Yeah, well, I mean, I've
worked quite a bit
with suturing as well,
and I haven't seen once
a milliter
that can reasonably, even
reasonably,
approximate the instrument
thread, needle, suture,
or tissue dynamics.
And so what I've reverted to
is actually getting individuals
to suture on tissue
and then a surrogate
measures,
like how well is the tissue
approximated,
did the surgeon drive
the needle
through the tissue
atraumatically,
do their knots hold when
they're supposed to?
Is the tissue pucker?
And those surrogate measures seem
to work reasonably well,
but why hasn't somebody
actually worked out
the actual physics of suturing
and thread dynamics?
Well, the suture and the
tissue, it's not easy.
It's a high computation
activity
and there frankly aren't
that many people
that work in this field.
If this is, so like look
at what's happening
with AI right now.
The entire planet has decided
that having physics,
what they're calling physical
intelligence now
is sort of one of the next
big things for AI.
So there's like thousands, tens
of thousands of people
doing physics models
now for robots,
for humanoid robots and
everything else.
And you're gonna see a tremendous
amount of improvement
in the physics simulation
abilities.
You just don't have that with
surgical simulation.
It's not like the whole
planet's decided
this is how you're gonna
become a billionaire
is by getting physical
intelligence
combined with humanoid
robots,
which is what a whole lot of
people have decided now.
So you're seeing a huge
amount of improvement.
You're just not gonna see
that in simulation,
in surgical simulation.
It's just not, like I said,
that's the reason I got out
of it a long time ago.
It's just not a big market.
So there's not a lot of
drive to do it.
What's gonna happen with
surgical simulation
is they're gonna get the
indirect benefits
of all the work that's happening
for robotics right now.
And you have all these things
that people are doing
where they're using AI combined
with physics models now,
so that you can use a reduced
accuracy physics model
and you use AI to fill
in the blanks.
I think you'll probably see
that showing up
in surgical simulation,
but it's gonna come from
other fields.
It's not gonna come
from surgical.
It's not gonna be developed
in surgical simulation.
It's gonna be developed
in other fields
and adopted in basically.
But if the measurements
of whatever task
you're at let's stick
with suturing,
if the measurements that you have
are measures of process
rather than actual measures
of skill,
I'm not too sure how the AI system's
gonna improve that.
No, yeah, no, no, yeah,
no, that doesn't,
that's such an indirect
measure.
I don't see how that really
is a good reflection
of mastery, if you will.
Yeah, I mean, so you're still
going to need
the performance metrics
which are derived
from experienced clinicians
and to actually teach an AI
system to how to assess.
And I don't see anybody doing
that in the surgical domain.
And I don't know whether they're
actually doing it
in other domains like famous
or infamous, whichever.
Car manufacturer said that
surgeons would be replaced
in what I can remember, was it
two years or three years?
Oh, him, yeah, right.
So what is happening now in
humanoid robotics
is they're trying to teach
all these humanoids.
So they're measuring all
kinds of things
about everyday activities.
So it'll be interesting to
see if some of that stuff
actually maybe becomes learning
points relative
to things that could
be moved over
into surgical simulation,
just the process
by which they're doing it,
not the details of it.
Yeah, but you see--
You are trying to quantify things
like crazy right now
in everyday activities, so.
Yeah, but what I see a lot
of the AI people do
is they're loading data into
the AI system.
And there's no quality, little
or not very good
quality assurance of the data
that the AI system's
learning on.
And the individuals that are
coding the tasks,
I mean, they don't really
know whether the task
could or not.
I mean, but that has to impede
the development of AI
in terms of the assessment
of performance.
Well, in surgery, certainly
it is.
I mean, if you're looking
at all these
humanoid robots desperately
trying
to do everyday activities,
I think the thing
that's happening there is there's
so many people doing it
that it's an aggregate
result.
There's just so many, there's
like 100 humanoid
robotics companies in
China alone.
So you get enough of
them doing it
and they're all piling on
top of each other.
And no matter how much
they're screwing up,
there's enough of them doing
the right thing
that it accumulates.
And you just don't have that effect
with surgery simulation
because there's just not that
many people active in it
when you come right
down to it.
One of the things that has struck
me across the decades
where we've been hanging
out together
is everybody's looking
for an easy way
to actually do something
that's quite difficult.
And science, I mean, doing good
science, it's not easy.
You know, I mean, and I mean,
one of the things
that I'm shocked at is the
use of Likert scales
for the assessment of
performance.
Because again, it's a measure
of process.
Yeah, yeah.
I mean, in robotics, I'm
sorry, go ahead.
No, I was gonna say all those
hand-wavy things, right?
All those hand-wavy. One
to five, right?
One to five.
I mean, my fellows,
well, they
know my views on it,
don't they?
And they initially think that
Tony's got a thing
about Likert scales, and they
very quickly learn,
okay, for the binary metric-based
assessment,
you've got to watch the entire
video and score it.
And you have to learn how
to score it reliably
with somebody else that
you're paired with.
With the Likert scales, you
can watch it, you know,
about two to five minutes
of the video,
and you can score it
in one to five
on the different constructs.
I mean, that's not measurement,
so it's not.
Yep, yep.
So what's your current view
on the status of surgical
robotic simulation
and the metrics used to
assess performance?
I mean, you've worked on
these or you've--
Yeah, I've been around
them, yeah.
You've been around them?
They could be better.
I mean, you know, even doing
the simple things
like ring exercises and
stuff like that,
they still haven't verified that
the performance of the,
you know, like the
exact forces
that you grasp the rings with
and the exact forces
when you pull on the
stretchy ones,
that those actually reflect
what happens in real life.
So it's all based on tweaking
and tuning
to make it so that it's at
least objectionable
to the largest number
of people,
but that doesn't mean that
it's correct.
And I don't think it's ever,
unless they change how
they do things,
it's probably never gonna be,
you're never gonna know
how correct it is
when you come right
down to it.
But surely that should
be of concern
given that you said earlier
that the robot is something
that can actually hurt
a patient relatively easily
if the surgeon doesn't
actually know
how to use it properly
and safely.
Yeah, well, I'm not disagreeing
with you on that.
I mean, but the manufacturer's
goal
is to verify that it won't unintentionally
hurt somebody.
If it intentionally hurts somebody
because of the operator,
that's kind of not their
problem,
is there kind of their
point of view?
Unless of course it happened
so many times
because of something
that's clearly
a human user interface issue,
then that does become
their problem.
And so whose responsibility
is then
for the clinician to
be trained
to a level that allows them
to use the device safely?
Well, I would assume
it would be
the professional societies.
I mean, the ones that are
actually saying
that yes, you are a surgeon.
And if not the societies,
it's the hospitals themselves
giving privileges.
So then that would be sort
of the next level up
because the privilege
granting
is at a whole different level
than the device manufacturers,
right?
The device manufacturer
is just saying,
I taught you how to
use the device
in a way that we certify that
it can be used.
But the granting of privileges
is saying,
you know how to use this
gadget to actually
get a clinical result and
we're gonna take,
we end up having liability
for it
if you don't essentially,
right?
But in the US, you have
a much better,
I'm okay, it's problematic,
but you have a better
privileging system
than we have over in Europe.
I mean, once you're
qualified,
I mean, I was at a PhD viva
by last week
and one of the examiners
was saying
that he works in Sweden.
And once he's qualified as
a surgical specialty,
he can do almost anything.
He also works in New York.
And he said each procedure
that he wants to perform,
he has to be privileged
for it.
That seems like a better
system than here.
Yeah, yeah, no, I would
go with that,
but I think that's sort
of reflective
of sort of the old school,
you know,
doctors are geniuses
kind of thing
from the old days, right?
So, yeah, it's not--
You mean in Europe or the US?
Well, in Europe in that case,
I mean, I don't know if the
experience you said about in
New York, if that's uniform
across the US, I don't really
follow that.
It would be nice if it
was. I suspect
it's not. It's probably
highly variable.
I can only talk about, you
know, I've worked in Yale
and Emory and it was the case
in both of those like 20
odd years ago.
That's good.
But one of the problems, the Chief
of Cardiology, there was
a credentialing going
on for carotid
artery stenting with embolic
protection
and he was saying that he
had to go to one of
the credentialing committees
and I said,
"That sounds, you know,
pretty powerful."
And he said, "No,
not really."
He says the credentialing system
in a hospital is based on
the person who has the lowest
experience in that procedure
on the committee.
So there was again no
objective for it.
Nobody wants to have any kind
of social admittance of
social whatever that they
don't know what's
going on.
Yeah, okay.
I mean, I think that at some point
there needs to be some
objective assessment
of performance,
either at the end of
a course or a
verification, you know, at
a hospital level.
Because one of the things we're
learning is a lot of the
metrics in the past were used
for the assessment of
performance,
like how many procedures
did you do?
Did you do a fellowship?
How long was your fellowship?
These were the performance
metrics that
were used to say, "Okay, that
person's safe to
do the procedure."
But one of the things that we've
learned over the last few
decades since the
introduction
of the laparoscopic surgery,
to some
extent, maybe even to a large
extent, those sorts
of metrics mean nothing.
They're measures of
process again.
Yeah, well, they're social proof,
if anything, when you
come right down to it, right?
I know this person and I
think they're okay.
And that's kind of the
end of it, right?
Yeah, and do you think
that's okay?
No, of course not.
I mean, we fall back to what Rick
Satava and others went on
and on and on about
years ago,
about the way that the pilots
are done, right?
Mandatory training and mandatory
testing, right?
Yes, yes.
And that's that end of story.
And those simulators, just to
throw this in for reference,
those simulators are
verified.
Those simulators are verified
big time, that they give the
right movements, they
give the
right forces, you know, everything
on those simulators is
reflective of what happens in
real life.
But they cost like tens or hundreds
of millions to develop?
Absolutely.
And so why do you think that
the governments and the
departments of health have
not insisted
on the same level of
verification for the surgical
simulators?
Two things.
A surgeon can only kill one
person at a time.
And it's only one person's worth
as opposed to hundreds of
millions of dollars
for a plane
and however many hundreds of people
that are going to sue
the airline to be
really blown
about it.
Yeah.
Yeah.
But I mean, somebody cited in
that PhD, that globally,
there's the number
of deaths due
to surgical misadventure or surgical
skills deficits is
about the equivalent
of one full
jumbo jet going down per day.
Yeah.
And that actually doesn't
surprise me at all.
I mean, since air travel has
gotten so safe these days.
But you know, like I said, even
if it's that many, it's
still one surgeon killing
one person
at a time, widely distributed
as opposed
to 500 people in one
spectacular event.
So unfortunately, it just doesn't
get in people's brains.
Yeah.
I mean, the Bristol case is a
fairly famous case in the
United Kingdom where
there was
a in the Bristol Royal Infirmary,
there were two cardiac,
they were adult cardiac
surgeons
operating on children,
children and their
mortality rate in the Bristol
case report
said that there were there complication,
their mortality
and morbidity was about
twice the
national average.
I'm told privately
that it was
significantly higher
than that.
That went on for the guts
of 10 years.
And there took 10 years,
right?
Yeah.
I mean, their peers knew that
there were issues.
And it wasn't until it hit
the front pages of Sunday
newspapers that the
colleges of
surgeons, the Royal
College of
Surgeons was forced to
take action.
You know, that's a
very like I
said, that was that was
cumulative.
And it took a long time.
And I'm sure that there
was a whole lot
of social stuff going on
at the same time.
Right.
Like, yeah, these people
were known
to other people and blah,
blah, blah.
And you can't rat
them out or
whatever, whatever it's
going to be.
There's all kinds of stuff like,
like I said, back on the
airplane thing, it's
because it
all happens at once and the
stakes are so high.
If it happens one time,
you know, if if a
surgeon managed to kill
dozens of people in
the span of a week,
you know, it
would become a thing,
you know.
Yeah.
Yeah.
There was a wait, wait till
tell it wait till tell us.
Wait till telesurgery goes on
steroids and you have
somebody who's doing
way too many
cases.
They could make a big mess really
quickly, basically.
What do you mean?
Telesurgery goes on steroids.
You mean that there's a lot
of it being performed or?
Yeah.
And you could have a
certain you.
Well, you and I both know from
being in hospitals that, you
know, the head surgeons
bouncing
between operating rooms with the
fellows and the residents
in there doing doing the most
of the work, you know, and you
places where they have like
four, five, six operating
rooms
going at the same time.
But the the lead surgeon is
supposedly the lead surgeon
for all those.
Imagine what happens when you
have telesurgery.
Yeah.
I mean, but in the United States,
OK, that goes on, but
it's really not supposed
to go
on the lead surgeon supposed
to be in the room all
the time.
And they get away with it because
they sign off on the case
or they perform the important
part of the case.
You know, I mean, they wouldn't
get away with that in the
UK and Ireland, I
don't think.
Yeah.
Yeah.
I mean, there's no two systematic
reviews and meta analysis
on proficiency based
progression
training and comparison to
the training outcomes in
comparison to conventionally
trained quality
assured approaches
to training.
And the results are pretty
similar.
They basically show that
if you're trained
in a proficiency based
progression approach,
you're going to make between
60 and 58 percent fewer
objectively assessed
interoperative
errors.
And there's small scale evidence
that basically said and
the differences in clinical
outcomes
is about 60 percent as well.
So why do you think
that the PBP
approach hasn't been hasn't
had wider adoption?
I mean, this is 20 odd years
now, 24 years.
Well, I would guess, as
you well know,
it's a lot of work to do a
curriculum based on
this.
I mean, it's a whole lot
of work trying
to figure out what the
metrics are and what
progression is like in
an objective
manner to actually be able
to lay that out.
And it's a lot of work.
So I think that's a lot
of the problem.
And I think the other thing is
that I suspect that people
don't like the idea of having
to train to some kind of objective
threshold because they
have no idea how long
it's going
to take them to attain
it, frankly.
But the evidence that evidence
is out there now, you can
train somebody to proficiency
in one third of the
time than you
can using a conventional
approach.
And it's about less than
half of the cost
to train somebody using
the proficiency.
And I mean, that data
is published.
Yeah.
OK.
Well, maybe it's just the
words not gotten around.
This is too new.
People aren't listening.
I don't know what to
tell you there.
But it's well, it's not it's too
new in the general sense.
I mean, they've been teaching
people the same way for
hundreds of years,
if you will.
So this is still kind of
working its way.
I don't disagree with you because
I'm a big supporter of
proficiency based
progression, as
you know.
I think maybe one of the other
things is that it needs to
be made more readily
available
in some kind of an automation
way of doing it.
You know, you and I have talked
about that, like how you
could leverage machine
learning
to actually make it so
that you can
actually do it without having
to train people to do
the evaluations and
the likes,
which is one of the negating
factors.
Right.
Yeah.
Well, I mean, that's
something
still in the in the planning
process.
But I mean, I was asked the question
in Australia, I was at
the surgical robotics
meeting in
Melbourne and somebody asked
me, Tony, if
you've got all this, I
was talking about
proficiency based progression
training and robotics.
He said, Tony, if you've
got all this
evidence, why has it not
had wider adoption?
And I hadn't even thought
about the question.
But my answer was out before.
And clearly, I must have had
it in the back of my mind.
I said, a failure of
leadership.
I mean, the reason that
Orsi Academy, the
reason that the Arthroscopic
Association of
North America and to some
extent, now, Medtronic,
cardiovascular seems to
be going down
that road.
The reason is somebody
senior in the
organization said, OK, this
is what we're doing.
And this is not negotiable.
And I mean, but there doesn't
seem to be enough.
ERUS, The European robotic
urological society.
I mean, I showed the data on the
failure rate of the ARIS
fellowship program,
the number
of fellows that weren't
finishing.
And the president, Alberto
Breda, said, OK, we're
changing.
We're going to go and use a proficiency
based progression.
That's what I call
leadership.
I mean, but it's surgeries,
surgery.
They must have to have
more leaders.
And is it something
I'm missing?
I'm only an engineer as
far as that goes.
Yeah, but you've been around
this for a long time,
and I know that you're using
these things.
Yeah, well, you know,
a lot of people,
yeah, no, I'm just assuming
that they
don't think they need it for
whatever reason.
I mean, you've seen, you know,
in a few isolated cases,
you've seen people make
the decision that
they do need it, and that's
that, right?
I mean, it might also be, I
mean, like with flight
simulation,
I mean, that's all government
regulated.
It's mandated, you know,
and so you'd need
something like that, but I don't
know if that'll ever
certainly won't happen
in the United
States anytime soon, put it
mildly. But yeah, I would
think it.
Well, our current environment,
which insists that the
government can't do anything
right,
and all regulations are evil
for some reason, you know.
Yeah, I was in, it was a long
time ago, 2006, I was
in Australia,
giving a talk for the Royal
Australasian College
of Surgeons,
and part of the deal was that
I would talk to the aviation
industry about simulation
based training.
This was 2006, and one
of the leaders
of the aviation simulation
group come up,
he says to me, "Hold on
a second." He
said, "You have all this
validation data
about how effective it is in terms
of skills training, and
you can't, it's not adopted?"
And I said, "No." And he said,
"How come you've got it?" He
says, "We had no evidence
when it was
mandated." Somebody
in government
basically decided, "Okay,
this is a better way."
And I mean, I don't know how we
persuade them. A fellow of
mine said to me, "Tony,
this isn't
data and evidence thing.
This is a matter
of selling the idea to the
powers that be."
So you do a lot of work
and publish on
the developments in robotics
and simulation.
Where do you envision that
simulation based training
and robotics
are heading in the next
10 years or so?
Well, just like everything else,
it's going to all change
completely because of AI,
quite honestly.
I do think you'll see a lot more
enhancements of things,
like the simulators
because of AI
as time moves on. It's definitely
going to happen. I think
it'll become more pervasive
too,
because the threshold to do it...
So I could go out and
build a new simulator
now by myself
because of AI. I'm not going
to do it because it doesn't
really make a lot of sense
economically,
but I'd have the ability to do
it because of how well Claude
and OpenAI Codex and
all these other
things work now. It's going to
dramatically drop the bar.
Robots are a different
issue because you
still got to go through all
the development of the
mechanics and things. It's
not like AI is going
to make more surgical robots
pop out quicker, but you're
going to see a lot more
software enablement
in surgical robotics.
So there'll be
little automations, little
part-task automations
that are going to start showing
up. Same in surgical
simulation, you'll see
a lot more
sort of semi-automated instruction
things are going to
start showing up independent
of the
simulator cores themselves.
Yeah, you're
just going to see a lot more
of this for sure.
And how fast, do you think
that autonomous robotics
performance surgical procedures
will develop?
No, no, just no. So you've
heard me say
this, other people have
heard me say this,
I'm a real downer on this relative
to academics because
they're all wanting to
do this, is that
unless the financial structures
are changed, there's no way
that autonomous surgery
is going
to happen with commercial surgical
robots because they're
not going to allow themselves
to become
financially liable for what
their robot does. As it is
right now, they just
build the tool
and others use the tool and those
other people are the ones
that become liable from
use of that tool.
If they start automating things,
then they're doing
surgery, they're performing
medicine,
and that's a whole different world.
You'll see lots of AI
that is going to be
suggestive
of like, maybe you shouldn't do
that, or maybe you should
notice this other
thing there,
stuff like that. You'll see lots
of that, but close the
wound for me. I don't
think so.
You might have smart things.
Your stapler example is a
really good one. There is
a possibility that
some kind of a supervised stapler
that just performs that
little tiny task and
does it much
more uniformly and precisely
than the human does. You're
not going to see autonomous
procedures.
You could in a crazy situation
like Satava always used to
talk about, like battlefield
medicine,
or you're on your ship to Mars
or whatever it is, then
totally, I get that. But
in a general sense,
well, so there might be things
in the developing world that
this might be a possibility
to,
but I don't see it in
the Western
Europe. All the developed
markets, China, Asia,
Europe, the United States, I
don't see it. I just don't.
Yeah. I'm sure that the
clinicians will
have something to say about
it as well.
I mean, I assume they're particularly
aware of the
perceived threat of
autonomous
operators or do they have a realistic
understanding? This
is probably not going
to happen.
Yeah. Like I said, I think
it's a regulatory and a
financial issue and it's
not a technology
issue. I think they will
figure out how to
do it, but it's just like
with self-driving,
right? The self-driving cars,
they get freaked out by
things they haven't seen
before, right?
And they've been trying
really hard for a
really long time to get it
to all be perfect,
and it's still not perfect. And
you'd have to do the same
thing in surgery, and you
just don't have
the infrastructure that you have
for autonomous driving in
the first place, right?
In terms of
data collection. And as we know,
there's millions of miles
of simulated driving that's
used to train
the autonomous driving systems.
You'd have to do the same
thing for surgical robotics.
And that
would mean that you'd have to
verify the simulators being
accurate so that you know
that the footage
you're synthetically
generating is
actually reflective of what
happens in real life.
So that might be the only way
that we end up with good
surgical simulators. If
somebody decides
they're going to invest a huge
amount of money into
generating millions of hours
of surgical video
through simulation, they'd have
to actually verify the
Dickens out of it. So it
might be the only way
you're ever going to get a properly
verified surgical
simulator. But okay. So
you would have
a map and the physical forces
that are generated by the
different behaviors.
I'm saying the
endovascular again, because
I mean, you're working in a
confined space. And
I suppose the
metrics for that there would
be easier. I'm not saying
easy, but easier to generate.
But you would still need
the metrics
of the performance of
the procedure.
You know what the cardiologist or the
radiologist does what they need to.
Yeah, so you'd be we'd be verified.
We'd be building
physical stand ins for
the for the
for the patient and then
instrumenting the
hell out of them to get like
the forces and
torques and pressures and everything
else. And then we'd be
using those to verify
the actual,
you know, virtual simulations,
the other
computational simulations.
And from there,
you would take those simulators
used by clinicians, and
you'd probably be back
mapping the you'd
have them perform the procedures
people that everybody
agrees are good. And
then you'd be
back mapping the information
you pull out of those
simulators and turning those
into metrics,
basically. So you don't think
that the clinicians have
anything to worry about about
an autonomous
robot anytime soon?
No, I really
don't. I don't see it.
Yeah. Okay.
You will see part task automation, but
autonomous robots, no
Yeah, yeah. I mean, I think
that's entirely feasible.
OK, Dwight, thanks for
joining me and
Yeah Sure!
thanks for a very interesting
conversation.
Yep, absolutely.