Subscribe
Share
Share
Embed
Most training is sold on confidence. Show Me The Evidence is built on data.
In every episode we take a single study, clinical trial, or systematic review and work through what it found, how it was designed, and what it means for the way we teach and assess skill. We focus on metrics-based training and proficiency-based progression, the approach that asks learners to demonstrate measurable competence before moving on, and we trace its results across surgical, medical, and professional education.
This is a podcast for learning professionals and medical educators who want more than opinion. Expect plain-language breakdowns of the research, honest discussion of what the evidence does and does not support, and conversations with the people behind the studies.
If you make decisions about how people are trained, we think you deserve to see the evidence first.
My guest today is Dr Ruben De Groote. Ruben and I met in 2019 when we were developing and validating the metrics for the robot-assisted radical prostatectomy. Ruben is a consultant urologist at Azorg in Aalst, Belgium, and his practice primarily consists of robotic surgical procedures, mostly oncological. He is the CEO of 4Health and its very successful digital learning platform, SurgeQuest. Ruben recently completed his PhD on the development and evaluation of a proficiency-based progression robotic surgical training programme for three surgical disciplines. Proficiency-based progression and training will form an important part of our conversation today. You’re very welcome, Ruben, to Show Me the Evidence.
Ruben, what are the fundamental limitations of current surgical education models that make a new approach to teaching and learning operative skills not just desirable but necessary? What are your views on that?
It’s not an easy question to answer, but then again it is, because the problem is very, very clear. Surgical education as it was in the past, and as it was until very recently, is not the same anymore as it is now and as it will be in the near future. The problem is that the way surgical residents were trained in the past no longer makes it possible to reach the same stage and level of expertise by the end of residency training.
Before about 100 years ago, surgical residents started training using the Halsted model—the apprenticeship model, the 'see one, do one, teach one' model—and everything stood or fell by exposure. In the ideal way, surgical residents are trained using surgical exposure. In a good training centre, this was actually applied stepwise. You first got exposure by observing cases in the OR, then you might go a little bit closer and be sterile at the operating table to see the procedure better. Then, when you became more experienced and moved up the ladder, you could increase your exposure by performing basic steps of the procedure. You evolved gradually until, by the end of your training, you were supposed to be able to independently perform complex steps and full surgical procedures.
Now, over the last 15 to 20 years, the circumstances of surgical training have changed significantly. Different external factors have caused surgical exposure in the OR to diminish significantly. Residents nowadays have to deal with far more bureaucracy. When I was a surgical resident, and this was not that long ago, I was able to finish all my bureaucracy before 10 o’clock in the morning and then go to the OR to get the exposure I wanted. Nowadays, even very ambitious residents have to spend at least half a day out of the OR to deal with bureaucracy before they can get in. That is problem number one.
Problem number two is the restriction on working hours. Surgical departments are actually punished legally if they allow residents to work more than 60 hours a week on average. This means they get less exposure in the OR and are less trained according to the traditional model. This is a vicious circle. By the end of their five or six years, they are deemed to be independent, but we know based on good publications that almost one out of three surgical residents are no longer independently ready to practise. This is, of course, a hazard for patient safety.
Luckily, most surgical residents are quite smart and realise they are not ready. Rather than going into independent practice, they go to different hospitals to do fellowships to reach the level they desire. These fellowships, which were originally meant for super-specialising—as I did in surgical robotics—are being transformed into an extension of surgical residency to achieve independence. This again leads to less exposure for the residents who are still in training. The cornerstone of the time-based curriculum, exposure, is vanishing very fast. It is no longer justifiable to continue with this way of surgical training in the future.
Can I ask you a naive question? Where did all this bureaucracy come from?
I have been a resident myself and I have seen it changing. This is a change in medicine in general, not only in surgical medicine. It is a phenomenon where people no longer want to take responsibility. When I was a resident, nurses were allowed to accept verbal orders to give medications. Nowadays, they are not allowed to do that because they don’t want to take the risk. It is not their fault; hospital management forces them not to do this because they get punished if they do. The consequence is that even for the simplest task, like prescribing paracetamol, there needs to be a written order. This increases the bureaucratic burden for everyone. While you need some standardisation, we are now living in a time where the smallest thing you do needs to be written down, and this is backlashing extremely.
So what you’re saying is that more process-driven practices are being imposed on surgical trainees, which is impeding on their time and their exposure to the operating room. Are these problems the same or worse for robotic surgical training?
I think robotic surgical training is quite special because it has both pros and cons. It is a challenge because you have to combine complex procedures with the mastery of technology. Most robotic procedures have several steps and different possible errors that can occur. However, robotics is a really good tool for applying new training methodologies. Unlike open surgery or pure laparoscopy, robotic surgery has a computer between your eyes and the patient. This allows you to store video files and surgical data, review the procedure, give formative feedback to residents, and measure kinematics to objectify the performance. Robotics is a fantastic tool to measure surgical quality in order to improve it.
I agree with you about robotics. But is observing somebody using the robot and doing the procedure sufficient? I’m not convinced that exposure on its own is enough; I think you need structured and metricised training. There was a structure in the Halstedian approach imposed by the trainers, and for robotics, I think you absolutely need that structure. Would you agree?
Yes, absolutely. Every technology stands or falls with the use of it. We are living at a time where the added value of robotics is being disputed because some people do not use it as it should be used. Exposure is not enough; it is about how you use it. When Halsted developed the 'see one, do one, teach one' methodology at Johns Hopkins, he was like the chef of a very famous restaurant. There were a few well-known hospitals for surgical training where everyone wanted to go. The chefs there made the laws and their own recipes. However, it was not universal; there was no recipe applicable to every single centre. We are now in an era where we need much more standardisation. You cannot only rely on being in a good centre; there needs to be benchmarking on surgical quality. Not every centre should be allowed to perform training unless they have reached certain benchmarks. The methodology used should be universal and driven by scientific evidence rather than just the experience of the trainer present.
The apprenticeship model has served surgery for more than a century. I’m getting the impression that you think there are more systematic issues than simply a reduction in work hours or exposure. Do you think there are systematic issues that need to be addressed in the training paradigm for procedure-based medicine?
Yes, absolutely. Exposure is one thing, but during my residency and fellowship, I was lucky enough to see different people and different ways of operating. Learning different ways to do a procedure is an added value, but the downside of not having standardised curricula is that different people train in different ways. If you are unlucky and end up with a trainer who is not interested in teaching, you won't get any training at all. You are extremely dependent on the goodwill of your trainer.
During my fellowship, there were six fellows and nine different consultants performing radical prostatectomies. One week you were trained by one consultant, and the next by another. By the end of a week, once you had learnt a certain way to do a part of the procedure, the next consultant would say, 'What are you doing? You’re doing it completely wrong; you should do it like this.' This is crazy. There are different ways, but it is not that one way is right and the other is wrong. There is a complete lack of standardisation, even in the biggest UK cancer centre where I did my fellowship. This lack of standardisation is a real problem and it is only increasing. Surgical safety is being compromised because people are not ready to operate independently at the end of their training. Patients are being used as training models for consultants who weren't trained well themselves. We need to move away from a time-based curriculum to a competency-based one. Some people will become proficient in four years, while others might need seven or eight. That’s okay, as long as we have safe surgeons when they start.
I totally agree with you, Ruben. At Yale, when we did the VR-to-OR study, one resident took the longest to demonstrate the proficiency benchmark, but once she got there, she performed the best in the operating room. Trainees need that space to hone their skills. But who should be policing and imposing those standards? Industry worries about this too, but they can't tell a surgeon they aren't ready. The European Robotic and Urological Society seems to have grasped this with their fellowships. Who do you think should be setting and policing those standards?
Setting the standards needs to be done by scientific research; it cannot just be an opinion. PBP is a very good example of how to set the bar. It is the result of a rigorous scientific process to get to a very exact description of a surgical procedure with definitions of steps and errors. Scientific societies should set the benchmark, but the authorities should make them mandatory. If I am allowed to prescribe medication after seven years of study, there should be a parallel for micro-credentialing for certain technologies and procedures. It is not because you are a urologist that you are all of a sudden a good robotic surgeon. You should be able to prove that you can use the technology for the benefit of the patient. Authorities, such as governments and universities, should control who can do what, driven by scientific research.
I don’t think industry should have a large role in that because they have a conflict of interest; they are driven by sales. Many commercial institutions don’t really care as long as they have sold their system. I am asked from time to time to do proctoring, often for complete novices. I really don’t like to do that because you go into an OR where a person is assumed to do the whole procedure under your supervision, and you are very limited in your ability to take over. This goes completely against PBP principles. It is endorsed and maintained by industry because you are getting paid by them to proctor. When I train my own fellows according to PBP principles, they aren't allowed to do a nerve-spare if they can't properly do a bladder neck dissection. But as a proctor paid by a company, you are forced to guide someone through a complex nerve-spare even if they struggled with the earlier steps. This goes completely against the principles of quality-assured training.
Frequently devices fail in the marketplace not because of the device itself, but because the individuals using them are not prepared. You have published work on teaching multidisciplinary surgical trainees. The data you've published seems to offer a compelling reason for reimagining surgical training. Why is this shift now an imperative for healthcare education more broadly?
We conducted a randomised controlled trial comparing the apprenticeship model to the PBP methodology. The PBP group significantly outperformed the traditionally trained group; 67 per cent of PBP trainees were proficient by the end of the day versus 17 per cent in the apprenticeship group. We deliberately did this as a multi-specialty trial including urologists, general surgeons, and gynaecologists. This was driven by the opinion in surgical medicine that some specialties are less skilled than others. We showed that this was not the case. At baseline, all three specialties performed equally well—or badly. By the end of training, they all performed as well as each other. The only difference was that the PBP group outperformed the apprenticeship model in every single specialty. The methodology, not the specialty, predicted the skill set. We often lack crossover between specialties, but I benefited from doing cardiovascular and thoracic surgery during my first year. I learnt an appreciation for standardisation, systematics, and teamwork. We should build crossovers into surgical training to learn from different specialities and reduce the differences in skills between them.
My interpretation of that study was that the systematic, evidence-based approach led to better performance. To what extent is it possible to reliably quantify intraoperative surgical performance? What are the key methodological and practical challenges involved, particularly in developing validated metrics?
I’ll give you an example. I was once asked to assess the skills of more than 100 senior trainees on a kidney transplant model using the GEARS score, which is a Likert scale-based tool. I quickly noticed that my scoring of a procedure depended on the one I saw before it; I was continuously comparing people. A 'three out of five' at the start of the day might not be the same as a 'three out of five' after ten videos. It is very subjective. Subjectivity is a real problem with these Likert scales, especially if the trainer is not blinded. You might score someone lower just because you don't like them.
Furthermore, there is a big issue with inter-reader disagreement. To be valid, you need an inter-reader agreement of 80 per cent or higher. In our study, GEARS only had 30 per cent agreement. By default, this makes the tool invalid for assessing surgical quality. Finally, GEARS is very generic. I eventually found that watching just two minutes of a video was enough for me to make an assessment, which is unfair to the trainee. I believe the way forward is using binary metrics. They are procedure-specific and force you to watch the whole procedure to make an assessment. It is zero or one—either a step was performed or an error was made according to a very specific definition. There is no way to cheat, and it is not just an opinion. It is the application of evidence-based medicine to a surgical procedure. It makes it fair and transparent because you know exactly what was done wrong, allowing for fast improvement.
So the binary metrics are not just for assessment, but for giving the trainee explicit formative feedback. I assume that the advice is pretty explicit.
In the hospital where I work in Aalst, we have many fellows. Every Thursday, we come together as a group and one or two fellows show a part of a procedure they performed. We use a split screen: the surgical procedure on one side and the validated operative metrics on the other. I act as a facilitator, translating those metrics to the procedure. We go over it and I point out exactly where they made an error. This is extremely valuable because they get formative feedback that isn't just my subjective feeling. The people sitting around the table also learn from it. This is the clinical translation of how PBP should be done. Eminence-based medicine is often ruled by people with a 'God complex' who think their way is always best, but PBP rules that out. Expert surgeons who are deemed high-volume sometimes score extremely poorly when measured by metrics. With PBP, you can't hide; it’s transparent and fair.
So, do you think proficiency-based progression is a better way to train surgical and procedural skills?
Yes, absolutely. There is no doubt. If a methodology shows a 60 per cent reduction in intraoperative errors compared to the apprenticeship model, you have to accept that it is better. This also translates into better outcomes. If you avoid errors, your outcomes will be better. Surgical medicine still goes by big names, but it should be about surgical quality. PBP is transparent and fair.
At what point in your career did you come to this conclusion? Did learning PBP and the metricised approach impact you personally?
I was initially trained well by a mentor who operated in a very systematic way and I had a huge amount of exposure. I was very happy with my training until I learnt the principles of PBP. There were two distinct moments when I noticed PBP works. The first was when I was involved in the construct validity study of the radical prostatectomy metrics. By scoring many videos of novices and experts, I learnt these metrics by heart. Since then, there is no way back for me. I cannot imagine teaching a procedure any other way because there is no alternative as good as a validated 'cooking recipe'. I can measure my own performance and see how I improve. The second moment was during the trial we conducted at Orsi Academy. Within 30 seconds of watching a blinded video, I can tell you which trainee is PBP-trained. They think as they operate because they are thinking about how to avoid errors. They move smoothly and efficiently. They look slow, but they are fast because they make no unnecessary steps. In the apprenticeship model, trainees make errors repeatedly because nobody points them out. PBP gets a trainee to a surgical level so much faster.
Is PBP only for residents, or for consultants as well? And is it only for technical skills?
It can be applied to everyone, residents, novice consultants, even experienced ones. It can be for nurses, too. PBP is about standardisation and efficiency. There is a very good paper by Dorothy Breen which showed that applying PBP metrics to patient handovers on an ICU using the ISBAR system made the process far more efficient. Information wasn't handed over twice and unuseful information was filtered out. It just makes sense. The basis of PBP is starting with experts to work out what they do and don't do, creating a standardised recipe. It drives and accelerates the learning curve because there is no ambiguity. It's not just exposure; it's instruction.
What changes would you like to see introduced into procedure-based medicine training?
If we continue the way we are doing now and neglect the changing bureaucratic, legal, and financial circumstances, we are being stupid. We are driving a generation of trainees into independent practice before they are ready and exposing patients to hazards. It is like allowing pilots to fly jumbo jets without showing they are proficient on a simulator. Years ago, the Institute of Medicine in the US proposed moving away from time-based training towards competency-based training. We need strong, unbiased leaders who are open to the results of scientific papers and who are strong enough to force the application of these methodologies without compromise.
I totally agree with you. In Melbourne a couple of years ago, someone asked me why PBP hasn't been more widely adopted given the evidence, and I said it was a failure of leadership. Why was Halsted able to impose his way of training for a century?
Halsted was a very strong leader with a big name who made revolutions in surgical medicine. He was able to change something with his power. But now circumstances have forced the community to move away from that paradigm, and there is a clear answer as to which one to choose instead.
One thing that concerns me is the development of wisdom. PBP trains people for a standardised approach on a straightforward case, but clinicians need to build expertise for more complex situations. Do you have any ideas about how one would go about helping clinicians acquire wisdom within the PBP paradigm?
Proficiency means someone is able to perform a standard procedure in a safe way. Going from proficiency to a higher level of surgical expertise is the move towards wisdom. Part of wisdom is expertise by experience, which requires volume. You need volume to continue improving your skills. Another part of wisdom is the ability to deal safely with unexpected or complex situations. I was once proctoring a surgeon who thought he could start independently after 20 cases. I told him he was crazy, because the day will come when he has an extremely difficult case with heavy bleeding or challenging tissues, and he won't be able to manage it. Achieving wisdom requires having someone experienced supporting you. I achieved wisdom because I worked in a department with extremely experienced surgeons; I knew if I got into trouble, someone was there to help.
Finally, to grow towards wisdom, you need good examples. We founded Surgquest, a digital learning platform that provides an overview of global experts showing not only standard procedures but also very challenging ones. By seeing how an expert tackles a complex situation, such as an IVC thrombus, you learn how to do it yourself. This gives you wings as a trainer, too, because you can give your fellows access to the robotic console more easily, knowing that if they make a mistake, you have the wisdom to repair it. That, in my opinion, is wisdom.
I think the Surgquest platform is a fantastic example of helping the surgical community acquire the information that builds wisdom. Thank you, Ruben.
Thank you, Tony.
Thank you, Ruben, for coming on to the Show Me the Evidence podcast. I'm sure our viewers appreciate the experience you've imparted here today and your views on a more scientific and evidence-based approach to surgical training.