Subscribe
Share
Share
Embed
Most training is sold on confidence. Show Me The Evidence is built on data.
In every episode we take a single study, clinical trial, or systematic review and work through what it found, how it was designed, and what it means for the way we teach and assess skill. We focus on metrics-based training and proficiency-based progression, the approach that asks learners to demonstrate measurable competence before moving on, and we trace its results across surgical, medical, and professional education.
This is a podcast for learning professionals and medical educators who want more than opinion. Expect plain-language breakdowns of the research, honest discussion of what the evidence does and does not support, and conversations with the people behind the studies.
If you make decisions about how people are trained, we think you deserve to see the evidence first.
I'm joined today by Dr Rui
Farinha. Rui is a senior
consultant of urology
in Lisbon.
Rui's completed fellowships
around the world in Spain,
Germany and Belgium in laparoscopic
and robotic surgery.
We met in Orsi in 2019
and Rui, I was
impressed, number one, by
his surgical skill.
Number two, about how quiet he
is on most occasions, but
when he actually says
something, it's
something meaningful, which
is always impressive.
You're very welcome Rui. Can
I start out? I mean, your
surgical career has spanned open
robotic, laparoscopic and
robotic surgery is expensive.
So, I mean, as a surgeon who spans
all of those approaches
to surgery, what additional performance
capabilities do you
think the robot affords
the surgeon?
Well Tony, I have to agree
with you that
surgical robots are expensive,
that's true.
And of course, we know that up
until today, at least, they
are not able to replace
the surgeon and
they are not able to operate
independently.
But rather they act as an interface
that translates the
surgeon's movements into a more
controlled instrument, a
more controlled instrument movements
inside the patient.
But their value and their main
value is that they are able
to increase, augment
the human
performance due to several
reasons.
They are able to introduce a greater
precision and movement
stability on the surgeon because
robotic systems, they can
filter physiological
tremor and they
can also scale surgeon's
movements.
For example, a relatively large
movement at the console can
be translated into a
much smaller
movement at the tip of
the instrument.
And this is particularly valuable
during delicate
dissection, during delicate
suturing,
during for example vascular
anastomosis.
And this is very important in
terms of the precision
of the surgery.
Another huge advantage is that
the robotic platforms can
increase the dexterity
of the surgeon.
For example, when we compare with
laparoscopic instruments,
these instruments were
very rigid and
they had a restricted range
of movement.
And the change was that the robotic
instruments can, for
example, articulate at the
tip of the instrument and
therefore they effectively
provide a wrist inside
the patient.
Their additional degrees of freedom
allow the surgeon to
approach tissue from different
angles to rotate the
instruments and to perform complex
manoeuvres that would be
very difficult with the
conventional laparoscopic
instruments.
So this increased dexterity
is especially important in
narrow anatomical spaces like for
example the pelvis, where
urologists for example perform
radical prostatectomies.
Another big improvement that
an advantage that robotic
platforms afford is their improved
visual information.
These platforms, they provide
a magnified high-definition
three-dimensional view of
the operative field.
And this stereoscopic vision improves
the depth perception
and helps the surgeon distinguish
the different tissue
planes, vessels and the different
anatomical structures.
So this is a very important
advantage.
Another very important advantage
is that the surgeon can
also control the camera directly,
providing a very stable
view without relying entirely
on the assistant.
Also there is another advantage,
which is the improved
surgeon ergonomics and the
endurance of the surgeon.
For example, the surgeon can
operate from an adjustable
console rather than, like in the
laparoscopic experience,
rather than standing for several
hours while holding
instruments in physically
demanding positions.
This reduces muscular strain and
reduces the fatigue of the
surgeon and this therefore can
help the surgeon to maintain
its concentration, to maintain
its precision of movements
and the movement consistency
during long procedures.
Another big advantage is that
the robotic platforms are
also digital platforms that allow
further performance in
systems because the surgeon's
actions pass through a
computer-controlled
interface.
And therefore, robotic systems
can potentially record
instrument movements, can analyse
technical performance and
potentially integrate
imaging,
navigation or augmented
reality information.
And also there is an increased
potential for future systems
to allow the identification of
anatomical structures, to
highlight tumours, for example,
to establish a safe
operating boundary or provide
real-time performance
feedback.
So these technical advantages
do not automatically produce
better clinical outcomes.
This
is very important to
highlight.
Performance still depends on surgeon
training, on procedure
selection, on team coordination
and on the appropriate use
of the technology, of course.
Yeah, so in a word you
would say,
yes, the technology is
worth the money.
Yes, it is.
I totally agree with you.
And I also agree with what you
implied that the technology
is a very powerful tool, but it
relies on the surgeon being
able to use that tool safely
and effectively.
I mean Dwight Meglan, I don't
know if you heard the
interview with Dwight
Meglan. Dwight
is not a believer in
autonomous surgery.
He thinks that that's really,
it's going to depend on the
market, it's going to depend
on lots of other things.
And so really we're back to
the surgeon who makes the
decision, who chooses the cases,
but uses the robot as a
tool for the effective performance,
the effective and safe
performance of a surgical
procedure.
So I mean, one of the interesting
things about you, well I
find very interesting about you
Rui is, you have spanned, we
met in Orsi when you were already
a senior consultant and
you've done fellowships in
laparoscopic as well as
robotic surgery.
And so across your experience,
has the approach to learning
to use these ways to perform
surgery changed?
Well, yes, in fact, the
approaches did change.
And I was very fortunate to be
trained in the different
surgical approaches,
initially open and
then laparoscopic and then
robotic approaches.
And the approach in each one
of these training, types of
training changed considerably,
in fact, because from my
experience, these approaches
for training moved from a
predominantly apprenticeship
based model towards a more
structured simulation based
and performance driven
training.
For example, my training in
open surgery followed the
apprenticeship model in which
I observed the experienced
surgeons first, then I assisted
during procedures and
gradually performed increasingly
complex parts of the
operation under supervision,
of course.
This learning took place mainly
in the operating theatre
and relied heavily on direct observation
and imitation of
the surgeons that were teaching
me, on repeated exposure to
clinical cases, on a verbal
feedback from my senior
surgeons, on a gradual increase
in the operative
responsibility and very
importantly on
the supervises, very
subjective judgement
when I was ready to progress.
And I felt that this model of
training allowed me to
develop anatomical knowledge,
allowed me to develop some
tissue handling, handling knowledge,
of course, and develop
clinical knowledge in
a real surgical
environment, which is
a big advantage.
However, I also felt that
this model had several
disadvantages because my progress
depended strongly on case
availability, for example, which
is a big limitation,
dependent also on the quality
of the supervision.
I was lucky because I was supervised
by different surgeons
with different experiences,
but I noticed during my
training in open surgery that
the quality of the
supervision was not
consistent.
And also in this type of training,
the training depends on
individual trainer's
assessment,
which is also a big
limitation.
And then I transitioned to laparoscopy.
And when I started
my training in laparoscopy, I
noticed that the competence
that I acquired in open surgery
did not automatically
translate it into competence in
minimally invasive surgery.
And I had to acquire a different
set of skills. That's why
I did different fellowships
in different places in
laparoscopy, because I
felt this need
to acquire these different
set of skills.
So, for example, I had to learn
how to operate with a
two-dimensional visual information
with a really
significant reduced in-depth
perception.
I had to learn how to work with
the long, rigid instruments
as the laparoscopic
instruments are.
I had to learn how to manage, read
the fulcrum effect, where
the end and the instrument
tip
movements occurred in the
opposite direction.
I also had to perform precise
movements with laparoscopic
instruments that have a very
limited degree of freedom.
And at the same time, I had to
coordinate both hands while
interpreting an indirect image.
So this was really complex.
And at that time, I met several
surgeons that learned their
skills during life procedures
using an
approach similar to the open
surgical training.
And it was very curious that
they told me that it became
clear for them that basic laparoscopic
skills could and
should be learned outside the
operating theatre.
And in fact, this was the time
in which I observed a
greater use of, for example, box
trainers, virtual reality
simulators, dry laboratory and
animal laboratory training,
on the development of standardised
technical exercises and
formal courses and the appearance
of certification programmes
for surgeons to learn how to perform
laparoscopic surgery.
So laparoscopic training represented
an important shift
from learning almost exclusively
through clinical exposure
to deliberate practice
in a very
controlled environment
as lab skills is.
Then I transitioned to robotic
surgery. And when I started
my robotic training in 2019
with Professor Mottrie,
I was introduced to a more structured
training that changed
my learning process
completely.
I had to use a robotic platform,
which I had to learn how
to use it because I had to learn
how to use the console
controls and how to
manage the
different robotic instruments
because the robotic surgeon
controls all the surgical
instruments that are inside
the patient.
I had to learn how to control
the camera and how to clutch
the instruments. I had
to learn how to
efficiently use multiple
robotic arms.
I had to learn how to prevent
internal and external
instrument collisions, how to
communicate with the bedside
assistant, which is very important
in robotic surgery.
I had also to train the different
troubleshooting and
emergency undocking scenarios.
And I also had to learn how
to maintain situational
awareness while
physically separated from
the patient.
I really noticed that robotic
surgery training was much
more structured in the sense
that it included online
modules, simulation, observation,
bedside assistance,
supervised console practice
and procedural prompting.
So it was much more structured
than the
laparoscopic and the open
surgery training.
And so the most important development
in the approach to
learning was the gradual movement
away from training
defined by time served or numbers
of procedures completed.
In these transitions from open
laparoscopic to robotic
surgery, instead of asking how
many cases as the trainees
performed, the question started
to be, can the trainee,
could I perform the procedures
safely and consistently?
To a defined standard.
I noticed a change in
the approach to learning because
robotic surgical training
incorporated clearly defined learning
objectives, included
procedure specific benchmarks,
objective assessment tools.
It incorporated simulation before
operating on patients. It
incorporated deliberate practice
with proximate feedback
and the progression was differently
assessed because it was
based on demonstrating
proficiency.
However, during these transitions
from open laparoscopic to
robotic surgery, there were aspects
that haven't changed
because surgery cannot
be learned
entirely through simulation,
of course.
So I still had to learn skills
like clinical judgement,
decision making under pressure,
recognition of abnormal
anatomy. I still had
to learn how
to manage bleeding and
complications.
I had to learn how to adapt when
the planned procedure did
not proceed as expected. And
I had to learn how to
communicate and lead within the
operating team, especially
in the robotic surgery
because the robotic
surgeon is far away from
the robotic team.
So, I mean, that's what was happening
in Orsi. Can I
also assume that wasn't what
was happening in many other
centres that were training
robotic surgical skills?
Yeah, I think the experience still
is very different from
centre to centre, from department
to department. Of course,
I was lucky to be trained by Professor
Mottrie in Aalst and
to have my lab training in Orsi
because the training is
really systematic and structured
and this, in fact, is not
happening in every department,
unfortunately.
And so do you think the approach
to surgical training
skills has changed then? It sounds
like it has, but I'm
interested in your view because
you're an experienced
surgeon, a very experienced surgeon,
and in my view, I have
the data that you're
a good surgeon.
So you've come from open to laparoscopic
to robotic. So how
much has it changed? What has
changed? Do you think
it has changed?
I think it changed a lot. Although,
once again, it's not,
this change is not happening
at the same time in every
department. There is still some
differences in different
departments around the world.
Although I think the experience
in the different
departments is expanding and
is becoming more and more
consistent. But the surgical
training changed
significantly.
Although the extent of this
change varies between
institutions, of course, between
specialties also, and also
between procedures because
there is still a lot of
heterogeneity. And it is
changing in very different
aspects.
For example, traditionally surgical
skills were learned,
like we said previously, mainly
through apprenticeship. And
even in robotic surgery,
this is
still happening in several
departments.
And in these departments, unfortunately,
the surgeons, the
trainees, still depend on the
exposure to a sufficient
number and variety of cases,
on the availability and the
teaching ability of the individual
trainers, of course.
On the trainees learning through
observation, repetition,
and on these subjective assessments
that the robotic
trainers give to their
trainees.
But especially in robotic surgery,
the fact is that the
apprenticeship model, although
still remains important, of
course, but it's no longer
considered sufficient
on its own.
I totally agree with you.
I also observed another important
change, which was the
increased use of simulation in
robotic surgery. This is a
fact. And it's spreading throughout
all the trainings
around all the departments. Of
course, we still need to
work a little bit on this
increase on the use of
simulation. But this is an
important change.
Yeah, because I think a lot of
people talk about the demise
of the Halstedian training
part time, but I'm not sure
I agree with them. I mean, I think
what simulation does, it
better prepares the trainee
for the operating room.
And so what the consultant
gets to supervise is a
pre-trained novice. They know
what to do. They've
demonstrated that they know
what to do. They're
quantitatively defined
performance level. They
get to the OR.
And they've got the technical skills
to do the procedure to
a quantitatively defined performance
level, but they just
have never done it
on a patient.
But the Halstedian approach
to apprenticeship and the
OR is still very much valid. It's
just that it now is more
efficient or it should
be more
efficient and more effective.
It has the potential to be
more efficient and effective,
yes.
Yeah, Yeah
So Rui, you've recently
completed a series of
studies on robot-assisted partial
nephrectomy. Why did you
choose the RAPN procedure
to investigate?
Well, I chose the robot-assisted
partial nephrectomy
because this is an ideal procedure
for studying surgical
performance for different
reasons.
First, it is a very technically
demanding operation because
a surgeon must remove
the tumour while
preserving as much healthy
kidney as possible.
He needs to control bleeding.
He needs to reconstruct the
kidney. He needs to minimise
warm ischaemia time. So all
these needs require precision,
require efficiency, require
a sound intraoperative
judgement.
The other reason was that this
surgery contains a series of
clearly identifiable procedural
phases and steps, including
the tumour exposure, the vascular
control, the tumour
excision, the renorrhaphy, and
the restoration of
blood flow.
So these steps can be perfectly
observed. They can be
measured. They can be assessed
separately, which makes the
procedure completely suitable
for developing
procedure-specific performance
metrics.
A third reason is that there is
a meaningful variation in
how surgeons perform a RAPN.
A surgeon may complete the
operation successfully, but the
quality, the efficiency,
and the consistency of performance
can differ considerably.
So case numbers alone do not reveal
whether the procedure
was performed to a high
standard.
And another reason was that
the procedure has clear
consequences for the patient.
Technical performance can influence
directly the blood
loss, the rate of complications,
the surgical margins, the
ischaemia time, the preservation
of renal function, and this
creates a direct link between
how the surgeon performs and
the quality of the clinical
outcome.
So for all these reasons, this
procedure provided a strong
model for investigating how surgical
performance can be
measured objectively and how
training might move from
experience-based learning towards
procedure-specific
proficiency-based
progression.
What are the important lessons
you've learned from that
series of studies? Because I
mean, you've done these
studies over, and this is a high-risk
procedure, not too
sure if all of our listeners
would appreciate that, but
it's a high-risk procedure, and
you've done a series of
studies with a very positive
outcome.
So what are the lessons
you've
learned from that series
of studies?
Well, I have to tell you that
I learned different lessons
and very interesting lessons.
I would say that the first
lesson that I learned was that
the surgical performance in
the robot-assisted partial
nephrectomy can be defined
very objectively.
Because, of course, initially
we think that this is a very
complex procedure, very difficult
to define objectively.
But in fact, after my studies,
I realised that it was able
to break down this procedure
into
different observable
components.
So we did a very detailed task
analysis, and this was
completely possible. And we also
did an international, we
obtained an international
export
consensus on all these
definitions.
And in the end, I defined the procedure
in terms of all the
phases, all the steps, all the
errors, and all the critical
errors, each one with a very clear
operational definition.
So what we did, in fact, was
we converted the expert
knowledge, which is often implicit
and very difficult to
teach, and we converted it into
something that could be
observed, could be measured,
and could be reproduced.
I also learned another lesson,
and this was that completing
the surgery was not the same as
performing it proficiently.
So novice inexperienced surgeons
often complete a similar
number of procedural steps. There
is no big difference in
the number of steps that
they complete.
But the important distinction
was how they completed them,
particularly the number of
errors they made.
In the intraoperative study
that I did, I learned that
experienced surgeons made 69%
fewer total errors than
novices. And this is
a significant
difference between their
performances.
And this demonstrated that the
procedure specific metrics
could identify differences
in the quality of the
performance that would have
been missed if they were
assessed only by task completion
or operating time.
This also taught me that the assessment
must capture both
what the surgeon did, but most
importantly, what the
surgeon should not have done.
Another very important lesson
was that experience is
important, of course. We all
agree with that, but
experience is not equivalent
to proficiency.
I was amazed by the variation
within surgeons who had
similar levels of experience.
Some experienced surgeons
performed extremely well, the
real experts, but others
produced error scores
comparable to
those of the better performing
novices,
which was a bit surprising for
this type of surgery and for
the level of experts that we
had. For example, in the
simulation study, we found that
experienced surgeons in the
low error group made 77%
fewer total errors than
experienced surgeons in the
high error group.
And also, this high error experienced
surgeon group
performed at approximately
the same
level as the low error
novice group.
So this was a very surprising finding.
This meant that, and
this might surprise surgeons,
of course, in the surgical
community, that case numbers
and seniority are useful
indicators of exposure, of course,
but they should not be
treated as a proof of competence
or a proof of proficiency.
So we really need to measure actual
performance. And also,
another very important lesson
that I learned was that
procedure-specific metrics are
much, much more informative
than the commonly used broad
global ratings.
So in my studies, I developed
RAPN-specific binary
metrics that achieved high agreement
between independent
assessors, and they
distinguished
different levels of
performance.
And in contrast, in my studies,
I also used the global
GEARS rating scale, and this scale
obtained substantially
poorer inter-rater
reliability.
And the assessor struggled to
distinguish novice from
experienced performance using these
type of scales. So this
taught me a lesson that once
again, and this once again
might surprise the surgical
community, that
global assessments may provide
a general impression of
dexterity of the or efficiency
of movement, but they do not
tell the training proficiency.
They do not tell which step
was omitted. They do not point
to which error occurred,
where the error occurred,
what
must change during the
next attempt.
So this made me realise that
procedure-specific metrics
have greater educational value
because they provide
explicit and actionable feedback.
And also, another very
important lesson was that a
realistic simulator is not
necessarily a validated
training tool.
So for example, in one of my
studies, I did a systematic
review, and this systematic review
showed that many partial
nephrectomy models were considered
realistic and useful, but
the evidence supporting their
educational effectiveness was
very, very limited.
I found no randomised control
trials, no predictive or
skill transfer validation. I found
a considerable variation
on these models and in the tasks
and in the assessment
methods that were used in the
different studies.
And this is a very important distinction.
A training model
does not need only to look and
feel realistic. It must
allow the training to practise
the correct procedure, to
receive reliable feedback,
and it must
allow to demonstrate measurable
improvement.
The simulator, the training model,
should only be seen as a
vehicle. The validated curriculum
and the performance
metrics are what make it an
educational intervention.
Finally, another very important
lesson that I learned was
that training should progress
according to demonstrated
proficiency. This is very important.
We should move away
from training based primarily
on time, on seniority, on
completion of a predetermined
number of cases.
Instead, the trainees should
practise in a simulated
environment, should receive
metric based feedback, and
should progress only after
reaching a predefined
proficiency benchmark. This is
the most important thing.
This means that the number of
practice attempts may vary
between individuals, but the
required standard should
not vary at all.
I have been reporting for about
20 odd years about the
heterogeneity in the performance
of consultants and
attendings across different
disciplines.
The first time that we observed
some of the consultants
were performing extremely poorly
on a fairly simple
laparoscopic task, to be
honest we did not believe
the data.
Now, across disciplines in
cardiology, radiology,
laparoscopic surgery, anaesthesia,
across disciplines what
we have been demonstrating is
that some consultants, when
you assess them, objectively
assess them, perform
quite poorly.
Did those results in your study
did they surprise you?
Did they shock you? Or
anticipated it?
Well, I have to confess that
they surprised me a bit
because we were dealing with
very experienced and expert
surgeons. And at this level,
usually they should perform
very good.
But in fact, there was a big
difference between them.
Of course, throughout all these
years, I was trained and I
worked with different surgeons.
And of course, these
different surgeons were training
me, they were teaching me.
And in fact, they were experts.
But I was able to observe
that not all of them
were that good.
But I learned from all of
them, of course.
But now we have data. That's
the thing.
And the data shows there
is a difference.
Correct. I mean, having an
opinion on something and
quantitatively demonstrating
something that's entirely
different. Like I said, the first
time that we observed it,
we were going, this can't
be right.
There were some of the consultants
performing 20 standard
deviations from the mean. And
we went through everything,
the methodology, everything,
because
we thought we'd done
something wrong.
And we've seen it so many times
now that we're going, OK,
these are genuine
observations.
OK, one of the things that I've
observed over the years
across disciplines and technologies
is that the
investigators, the device manufacturers,
the scientific
societies deploy simulations
that
look like the procedure
to be learned.
Is that the best approach?
No, I don't think so. It's not
the best approach at all. So
there is in fact a kind of
obsession with realism.
But the issue is that this obsession
for realism can become
a huge distraction. That can
become a big problem.
Because a simulator may look
spectacular, may look, the
people that are seeing the simulator,
they may feel that
this is completely
convincing.
The simulator may receive excellent
ratings from experts,
but yet the simulator can
still be a very poor
training tool.
So these are two different
things.
Being real and being a good
training tool.
Resemblance is not evidence of
educational effectiveness.
That's a very important
thing.
We need to take into
account that
realism and validity are
not the same thing.
For example, I did a systematic
review and the majority of
the partial effect of the training
models that we studied,
they were rated as completely
realistic and useful.
But the underlying evidence for
this was very weak. The
studies were very
heterogeneous.
There were no randomised control
trials studying these
different models and none of them
demonstrated predictive
validity or any transfer of skills
into clinical practice.
So this is a big problem. So the
uncomfortable question is
if a simulator looks like
an operation,
but we cannot show that it produces
safe or more proficient
surgeons, what exactly have
we validated?
Often we have validated the
appearance of the model,
not the training.
And then there is another thing
that we need to think that
the objective is not to recreate
the operating theatre, it
is not to recreate the surgical
environment completely.
We do not want to produce the most
impressive limitation of
reality. We want to create a
controlled environment,
of course.
And in that controlled environment,
the training should be
able to practise the correct
procedural steps.
You should be able to make the
errors without arming the
patient. You should receive a specific
feedback, repeat the
task deliberately, not
repetitively.
And you should be able to
demonstrate proficiency
before progressing.
And therefore, a less realistic
model that reliably trains
the critical components of a
procedure may be far more
valuable than an expensive, highly
realistic model with no
validated curriculum
at all and
no objective assessment
system.
So when developing a simulation,
we should focus on the
fact that we should simulate what
determines performance.
For example, in the
case of the
surgery, in the case of RAPN
a surgeon does not become
proficient because the synthetic
kidney has the perfect
colour or the perfect texture.
It doesn't. The surgeon becomes
proficient through correct
tumour identification, through
correct controlled
dissection, appropriate instrument
use, a safe excision,
correct haemostasis,
an accurate
reconstruction of the kidney.
So simulation must reproduce
the functional and the
decision critical elements
of this procedure.
This simulation should not necessarily
reproduce every
visual and every anatomical detail.
This is not needed.
And for example, I developed
a training model, and this
training model was deliberately
developed, deliberately
designed as a vehicle to deliver
a metric based curriculum.
This new model emulates, for example,
eight of the eleven
phases of the human RAPN
procedure.
And its primary purpose was not
to reproduce every feature
of the human operation, but it
was developed with the goal
to allow observable steps, errors,
and critical errors to
be practised and to be
scored reliably.
Also, we need to think that the
developers of this type of
simulation models need to take
into account that high
fidelity can carry a high educational
cost because greater
realism usually brings greater
costs, greater complexity,
and probably a much
restricted access.
And then there is also an important
thing that these
simulation must reduce the number
of repetitions that
trainees can complete. If the model
is very expensive, then
the number of repetitions that
the model will allow will
not be that much because if the
model is very expensive,
the trainees, maybe they
can only perform
the task once, maybe twice,
and that's it.
And therefore, this is not a good
model. So these matters
because proficiency is
not achieved
by one impressive simulation
session.
Proficiency develops through deliberate
practice, through
proximate feedback, and this
also develops from the
correction of specific
errors.
So this is why when I developed
the new partial nephrectomy
training model, I took these
aspects into consideration
because the model I developed uses
readily available animal
tissue costing less than $10.
Yet, it allows trainees to practise
eight phases of the
procedure and receive procedure
specific feedback. So here
the relevant question is not how
realistic the model is, it
is how much high quality
measurable practice does
the model permit.
So where would you advise the
investigators, the device
manufacturers, the
professionals?
Where would you suggest that
they start in terms of
developing a model? What
must they do?
Well, they should start
by defining the
procedure that they want
to characterise.
That's the first step before
they start to develop
anything.
Yeah, I totally agree with you.
I was at the European Heart
Rhythm Association meeting in
Paris a few months ago and I
walked around all the
manufacturers and looked
at the simulations.
They were saying they had physics-based
virtual reality
simulations. None of them
had metrics.
The procedures looked great,
except for one of them had
metrics. And I totally
agree with you
that you really need to start
with the metrics. You really
need to start with the
construct validity
and say, "Okay, where are the
trainees having the most
difficulty in learning and
performing that
part of the procedure?" And that's
where you really need to
concentrate on. So you've
completed a
series of studies on the robot-assisted
partial nephrectomy
and you've gone all
the way to a
prospective randomised and
controlled trial on your
simulation model. What did
you find from that study?
Did it work? Fortunately, I
was able to start the
development of this project
from the real start,
from developing the
metrics, the
procedural metrics for the
partial nephrectomy.
So I learned several lessons
in all these
studies. And of course,
the first lesson,
and I think I already said that
this, the first lesson was
that I learned that it
was possible
to objectively define RAPN performance.
So my first study
demonstrated that a complex
procedure,
such as robot-assisted partial
nephrectomy, can be
completely deconstructed
into observable
phases, observable steps, errors,
and critical errors. And
then it was very interesting
to see
that in the beginning, we gathered
a group of three
international experts to
help me to define
these metrics. But then we submitted
these metrics to an
enlarged panel of
international
experts in the robot-assisted
partial nephrectomy. And it
was very interesting to
see that we were
able to reach 100% consensus
on these procedure-specific
metrics. And this is
very important,
because this provides a common,
explicit definition of what
is an optimal and
a suboptimal
RAPN performance. And then
in the second study that I
did, I learned another
important lesson.
I learned that the metrics that
we developed and that were
face and content validated
by
all these international experts,
that these metrics could
reliably distinguish
different
between the different surgeons.
For example, when we
applied the metrics to unedited
recordings of
real surgeries, independent assessors
achieved very high
agreement with a mean
interactive
reliability of 0.9. And what
we observed was the
experienced surgeons made 69
fewer total errors
which demonstrates that the
metrics could distinguish
levels of expertise,
which is very
important as an assessment tool.
Even more important was
that the greatest distinction
was not simply between
whether the surgeon
completed the operative
steps. It was how
the surgeon performed them and
how many errors were made.
Also, another very important
finding
was that, and I also said that
previously, that experience
alone does not guarantee
proficiency.
This was a very interesting finding,
because there was,
like we said previously,
there was
a considerable variation
within surgeons,
classified at the same level
of experience.
Some experienced surgeons performed
extremely well, while
others performed at a level
comparable to
better performing novices.
And this was a bit
disconcerting. Another important
finding was that
these procedure-specific metrics
outperformed completely
the global rating scales.
And another
important finding was that the
existing RAPN simulators
lacked strong evidence
of training
effectiveness. So once again,
in this systematic review
that I did, we examined
all the different
training models that existed
for partial nephrectomy, the
animal, the three-dimensional
printed
models, the virtual reality
models. And although these
models were generally rated
as realistic
and useful, there were no randomised
control trials. There
was no evidence of predictive
validity,
no reliable transfer of required
skills into clinical
performance. So the conclusion
was that
realism and perceived usefulness
do not by themselves
establish that the simulator
produces
more and more proficient surgeons.
And then there was also
an important finding,
which was
that the developed a validated
model for metric-based
RAPN training, because
one important
finding was that I developed
a validated model for
metric-based RAPN training
by, and this is
very important, I adapted the
human RAPN metrics to the
new procedural training
model.
And the model metrics
distinguished, once
again, novice from experience
performance.
And once again, they also revealed
substantial variation
within the experienced group,
reinforcing that experience
does not always equate to
proficiency. So most
importantly,
was that after all these studies,
I was able to form a
connected integrated programme
of research,
moving from defining
performance to
testing how it should be
measured. And finally,
to translating those
measurements into simulation-based
training.
So are your studies simply a
series of interesting
observations, or do they have
a real world application? Well,
they have a real world
application, of course.
And this
application is very, very real.
And nowadays it's being
used in Orsi to
train the new
generation of surgeons. So these
studies from the start,
they were not designed
merely to
describe how the robot-assisted partial nephrectomy
performance. They
were designed to create
the components
required for a deployable proficiency-based
training
programme. This was the
main objective
from the start. Yeah, I
think a lot of the
studies that are published
in the journals,
my concern about them is that
they're interesting studies,
but they don't seem to
go anywhere.
Whereas what you've done, okay,
it's being deployed in
Orsi, but more
importantly,
I think it's being deployed with
a new fellowship with the
European Robotic Urological
Society,
where they're now implementing
what you've
done in a RAPN specific
training programme.
And I think that's hugely important.
The studies, yes,
they're interesting. And
I think it's
important because what
you have done is,
rather than just develop
a training programme,
it's a training programme that's
been developed and
scientifically underpinned
with quantifiable
performance measurements
that are
published in peer-reviewed
journals,
and it demonstrates that it's
a much more effective
approach to training.
So, okay,
how has your training to simulation-based
training changed
as a result of your experience
in Orsi
Well, my understanding has
changed quite fundamentally
and quite substantially,
because
a few years ago, I tended to think
of simulation mainly as
a way of reproducing
the operating
theater without exposing a patient
to risk. Today, I see it
completely differently. I see
the purpose of simulation
not simply to
create a lookalike surgical
environment,
but to make performance visible,
measurable, and
improvable. So, I move
from thinking
about realism to thinking
about learning.
So, initially, the natural
question was,
"How closely does the simulator
resemble the real
procedure?" And, of course,
that remains relevant.
I'm not saying that it's not
relevant, particularly for
tissue handling, for dissection,
and suturing.
But, however, I became much more
critical of the assumption
that high fidelity automatically
means
high educational value.
It doesn't
mean at all. So, the better
question is,
"What precisely can the trainee
learn, practise, and
demonstrate on a model?"
Also,
I realised that a simulator is
only a vehicle. A simulator
on its own is not a training
programme.
So, without a defined curriculum,
clear objectives,
structured feedback, and a
performance benchmark,
the trainee may simply repeat
and repeat and repeat the
procedure and potentially
repeat
the same errors. So, the real
educational value of the
intervention is a combination
of a
deconstructed procedure, clearly
defined steps, errors, and
critical errors, deliberate
practice,
reliable assessment, and
the proficiency
standard that must be achieved.
So, this is
non-negotiable. So, my robot-assisted partial
nephrectomy work until
now, showed that expert
knowledge
could be converted into observable
and accessible
performance metrics,
of course,
supported by a huge and
enlarged group
of international expert
consensus. So,
that changed my view of simulation
from a piece of
equipment into a delivery
system for a
scientifically designed curriculum.
Also, my view moved
from repetition to deliberate
practice.
Traditional simulation courses
that I did often provided a
fixed amount of practice.
So, I attended
for a day or two or three. I completed
all the exercises,
and then in the end,
I received
immediately a certificate.
But I realised
that completion is not the
same as learning.
So, proficiency-based progression
is completely different.
It allows different trainees
to require
different amounts of practice
while holding
everyone to the same performance
standard.
So, in proficiency-based progression,
trainees receive
explicit feedback on
the errors.
On the errors they made, they
use the next attempt to
correct those specific
errors,
those specific deficits. They
progress only after
demonstrating that required
benchmark.
This is a major conceptual
change
because the number of repetitions
can vary,
but the required standard
should
not vary. It doesn't
vary at all.
Yeah, I totally agree
with you.
That's a huge mind shift
of a change.
Because a lot of what I've
seen in the past is
simulation-based training
is used as an
educational experience. What you're
describing is not. It's
actually a tool for the
delivery of a
metric-based curriculum. The metrics
are derived from the
consultants who genuinely
know how to
do the procedure. It's benchmarked
on the consultants who
genuinely can do the
procedure
and the trainees are given
deliberate
practice feedback based
on the metrics.
And that's an entirely
different approach to using
simulation.
Are you a fan of
proficiency-based progression
training then?
Yes, I'm a percentage fan.
I really like it.
The trainees that you train, do
they see it as positive? Do
they see it as more
difficult?
Do they see it as
challenging?
They see it as positive and
completely different from
what they are used to
experience in terms of training.
Because when they have
contact with this type
of approach
in terms of training, they
really find that they can
progress faster. And
I think they
really appreciate one important
thing that PBP brings to
training, which is the
approximate
feedback that they have and the
very specific and detailed
feedback. This is something
that
they appreciate. And in this type
of proximate feedback,
I also think that the
human contact is
important. Nowadays, we are talking
a lot about having AI
models that can train
people and can
give feedback. But when you have
two humans interacting and
one is training the other and
is giving a very proximate human
and kind and detailed
feedback, this transforms
the training
and makes the training more human
and friendly. And I think
with this type of approach,
trainees
progress faster. And they really
see that trainers are open
to their doubts and to
their needs. And
this brings a new experience
to training.
I totally agree with you.
I mean, COVID,
I mean, one of the lessons that
I learned from it is human
beings not being in
the same room,
not being proximate teacher,
okay, you can get business
done, but not with the
same efficiency
and effectiveness as you can
when you're actually quite
close to somebody. So if
you were to advise
somebody who wanted to develop
a simulation curriculum for
a robotic surgical procedure,
what advice would you
give them? My
first advice would be
very direct.
I would advise them not to start
by buying or building a
simulator. So and this
is usually
the most frequent approach from
somebody that wants to
start to develop a training
programme,
but this is not the best approach.
So I would advise them
to start by defining
the performance
that they want the surgeon to achieve
and to establish what
type of procedure
they want to
train. And because one of the
most common mistakes in
developing a simulation
curriculum is
to acquire and or develop an
impressive simulation model
and then ask what this
simulation model
might be useful for.
And this is
completely wrong. When we use
the PBP approach,
a proficiency based progression
curriculum should be
developed in the opposite
direction.
And this is what I would advise
the developers to do. I
would advise them to
define what is
a safe performance first, and
then to select and construct
the simulation that can train
and can measure it. You would
advise them to start with a
metrics. With the metrics.
So I would advise
them first to define the training
problem precisely. So
they should first specify
the procedure,
the operative approach, the target
learners, of course.
They should define their
required
level skills. The clinical
case is covered by the
curriculum, of course, and
they should define
the level of independence expected
at completion. They
should initially focus
on the standard
straightforward case because
trying to simulate every
anatomical variation, every
complication
from the beginning makes the curriculum
very difficult to
standardise, very
difficult to
validate. And also, a trainee
that cannot perform a
reference straightforward
procedure
safely should not progress to
more complex cases. And
sometimes when people start
to develop a
training problem, they
try to cover all
the scenarios and this
is not correct.
Yeah, no, I totally agree with
you. I mean, that is a
lesson I learned a long,
long time ago,
is that if somebody can't do a
straightforward procedure,
why would you teach
them something
more complex? And it's easier
to get consensus around
what's a straightforward
procedure and so
on. So your advice to somebody
would be start with the
metrics. And would you get
anybody to help you
develop the metrics or do you
have any specific
requirements? Well, they
should start with the
human metrics, with the human procedure.
And of course, in
the beginning, we should
find at least
three good experts, but they
should be experts that are
able to discuss the ideas,
discuss the
metrics, they should be a bit open-minded,
and they should
be able to accept different
opinions.
They should be so and sometimes
this is very difficult to
find these people, especially
between the surgical community.
But there
are still a few surgeons
that are like this.
And so you don't really need experts
that don't accept the
ideas and the way of the
other surgeons
do their things, because you
really need a group that is
able to connect and
to understand
the other surgeons' positions
and to accept that their
opinion might not be
the correct one
and still go with that opinion.
And do we need experts or
do we need very experienced
surgeons
that are good at the procedure
and do it frequently? I
would say that we need very
experienced surgeons,
not being the next person on
the mental. Yeah, I mean,
that's a different
conversation.
Ideally, people that are able
to do surgery and that are
used and able to train
people.
That's the best combination.
Yeah, I mean, I've
characterised quite a number
of procedures in my
career in the different disciplines.
And one of the things
I've encountered is so-called
experts.
When I listen to them, I'm going,
"Okay, how expert are
they?" And when I dig
a bit deeper,
what I find is they haven't
performed the procedure in
years or they weren't
particularly
good at it. My expert for me,
or the expert for me, is a
very experienced clinician
who does the
procedure, who's good at it,
and is good at training,
because they have the verbiage
to be able to
facilitate the definition of
the intraoperative
performance. So, Rui, you're
very experienced in
robotic surgery now. I knew you
proctor and teach and train
in the Skills Lab and all
over the world
in the Skills Lab and in the operating
room. Do you think
that proficiency-based
progression plays
a role in training in the Skills
Lab for the different
robots and in the operating
room
for the different robots? So,
yes, it plays different
roles. So, in fact,
I believe that
proficiency-based progression
has a very important role in
both settings, although
it serves a
slightly different purpose in
each one of the settings.
First, when we teach a
surgeon to use
the robot, the manufacturer's
instructions for use are
essential. But these instructions
for use,
they mainly describe how the technology
should be operated
safely. They do not
necessarily
demonstrate that the surgeon can
operate it reliably under
realistic conditions.
So, and here,
in this aspect, PBP makes
a difference
because it converts the
instructions for use
into observable and measurable
performance. For example,
docking the robot, positioning
the patient,
managing the different instruments,
responding to alarms,
troubleshooting faults,
performing an
emergency and docking procedure.
So, PBP, rather than
completing a fixed number
of training hours,
makes the surgeon progress only
after demonstrating the
required standard. And this
is very important
because familiarity with the controls,
with the different
buttons, should not be
confused with
proficiency in using those controls,
using those buttons.
So, a surgeon may understand
what each
button does, but it might still
be unable to use the system
efficiently and safely
when workload
or complexity increases. Also,
PBP has an equally important
and perhaps even more
important role
inside the operating room because
the skills laboratory can
establish technical
readiness,
but the operating room introduces
factors that simulation
cannot reproduce fully.
For example,
it introduces patient variability,
tissue response, it
introduces communication with
bedside assistants in real time,
it introduces coordination
with anaesthesiologist,
it introduces
the factor of time pressure and
unexpected events. So, PBP
provides a bridge between
laboratory
training and independent clinical
practice. Progression in
the operating room can
be structured
through defined stages, beginning
with the observation and
selected procedural
components,
followed by supervised procedures
and ultimately
independent operating. And
the advancement should
depend on procedure specific
performance metrics, not
simply on the number of
cases completed
or the subjective impression
that the surgeon appears
ready. So, the central
principle is that
the instructions for use teach
the surgeon how the robot
functions. PBP determines
whether the
surgeon can use it proficiently.
And in the operating room,
PBP determines that
proficiency
can be translated into a safe
and effective procedural
performance. So, PBP does
not replace
expert supervision at all, it
does not replace clinical
judgement, neither responsibility
for
patient outcomes, but it makes
the training pathway more
objective, transparent
and defensible,
and reduces the reliance
on case numbers
or time served as proxies
of proficiency.
Yeah, so PBP you would argue is
a, again, it's a tool for
the delivery of a curriculum,
whether it's in the skills lab
or it's in the OR. Yes. And
you think it has a useful
function
across platforms? I mean, it's
not, you know, platform
specific? It's not at
all platform
specific. It can be used and
developed for different
platforms. You can,
like we did in
the partial nephrectomy studies
that we did, we developed a
metric for the procedure, not
for the procedure done by a specific
platform. Of course,
we can use PBP to develop
metrics
for a specific platform, to develop
a specific training
programme to use that
platform. But
you can develop a programme for the
procedure and that can be
performed by different
platforms.
Yeah, yeah. So, yeah, I mean,
listen Rui, that's
fabulous. Thank you very
much. It's always
a pleasure to talk to you
and always very
informative. You are very
welcome. Thank you
so much, Tony, for the
opportunity.