Special Topic: The thinking is the job

Show Notes

A new paper says medical training should teach doctors when to defer to a machine. It has the order backwards, and almost nobody is counting what the deferring is worth.

Read the special topic and its sources: https://physicianintheloop.org/specials/mastery-and-the-machine/

The voice on this episode is generated by AI. It reads a special topic from Physician in the Loop as written; every fact in it links to its source.

What is Physician in the Loop?

How technological innovation is changing medicine for patients and the people who practice on the front lines.

This is a Physician in the Loop special topic.

The thinking is the job

A new paper says medical training should teach doctors when to defer to a machine. It has the order backwards, and almost nobody is counting what the deferring is worth.

Four emergency physicians at the University of Texas Southwestern Medical Center published a perspective on Oct. 13 with a line that should stop any doctor cold. One day, they write, the most dangerous actor in the room may not be the machine making a recommendation. It may be “the human who insists on overruling it simply because being in the loop still feels like being in charge.”

They may be right about that doctor. The trouble is the paper they wrote to get there. It asks medical training to stop prizing struggle for its own sake and refocus it from recalling facts toward “interrogating machine reasoning, detecting failure modes” and knowing when a person should step in.

Deference is not a skill of its own. It sits on top of one. The same paper says so. A trainee who has never had to defend a differential diagnosis “may be poorly prepared to recognize when an AI system is incomplete, overconfident, or wrong”. You cannot teach a young doctor to tell a good recommendation from a confident wrong one until she can tell those apart on her own. The struggle is not a teaching method we picked and could swap for a better one. It is the thing being taught.

Which raises the question almost nobody is answering. If the doctor in the loop is the safeguard, how much does it catch? A systematic review of these systems in primary care, published this month, looked at eleven studies. Not one reported whether the clinician agreed with the machine or overrode it. Three of the perspective's six references are commentary, and the five that can be read follow no patient past the moment a doctor accepted or refused a machine's advice.

The one randomized trial among them ran at the end of 2023. 50 physicians, nearly half of them residents, got an hour and up to six written cases, with ChatGPT or their usual references. Blinded scorers graded the reasoning they wrote down. With the model, a median of 76 percent per case. Without it, 74. Two points, and the trial could not separate that from chance. The model working the same cases alone scored 16 points higher. Now read what that trial is not. Nobody took a history. Nobody examined anyone. Nobody found out what happened to a single patient. The model's score came from a prompt a researcher wrote and ran three times, in what the authors call a secondary exploratory analysis. They say the rubric score is a signal and that “broader clinical trials are necessary to assess for meaningful differences in downstream clinical impact.”

Deskilling and automation bias are visible wherever someone bothers to measure them. 27 radiologists read 50 mammograms with ratings attached that they were told came from AI. When the suggestion was wrong, the least experienced readers went from getting about four in five right to one in five. The most experienced dropped below half. In four Polish endoscopy centres, adenoma detection in colonoscopies done without the software fell from 28.4 percent to 22.4 percent in the three months after it arrived. Its authors call that a possible effect of continuous exposure. Not forgetting how to find a polyp. Not looking as hard. One arrangement has held up in real care. A Swedish trial screened nearly 106,000 women. Software sorted the mammograms and flagged findings. Radiologists still decided whom to call back. Detection rose from five cancers per thousand to six, with no more false positives, reading work almost halved and cancers surfacing between screens no worse. Public money and the Swedish Cancer Society paid for it. What was tested there was the arrangement. Not the override.

Rachael Bedard, a geriatrician who uses OpenEvidence every day, wrote in a New York Times guest essay on Sept. 25 that it has made her a better doctor and may be making her trainees worse, and argued for teaching them to think critically with a machine in the loop. Adam Rodman, quoted in the same piece, put the line where it belongs. A little deskilling in experienced hands probably does not matter much. In someone with far less skill, “the cognitive offloading can cause never-skilling.” One of those is a bad habit. The other is a doctor who never arrives.

The blunt case for getting out of the way is not Bedard's. In JAMA in August, Ezekiel Emanuel, Abe Baker-Butler and Neal and Vinod Khosla of Curai Health and Khosla Ventures asked whether autonomous AI will beat AI-aided physicians. Where that turns out to be true, a doctor standing in the way costs patients something real, and the perspective is right that a profession should not dress up its own indispensability as a moral principle. But look at the job it offers instead. Orchestration, it says. Among other things, routing requests, reconciling contradictory outputs, watching for deterioration. That is the most automatable work in the building.

So count it. A hospital that puts a doctor between a model and a patient should record what the doctor did and what followed, and how those patients fared against the rest. A monthly chart sample is within reach of a small hospital. Short of a trial, it is the only way to learn whether the reviewer is a safeguard or a signature. Put a plausible, wrong recommendation in front of a trainee and grade the catch, as the perspective itself asks. The Association of American Medical Colleges has AI competencies targeted for this fall, and that belongs in them. Funders should pay for the prospective trials these authors keep asking for.

Teach the foundations first. They are the only thing that separates a doctor from a layman with a search box, and a profession that hands them over will not be asked to stay. The perspective may be right about the doctor who overrules a better system out of habit. Nobody can say how often, because almost nobody is counting. We are not defending mastery. We are not yielding wisely either. We are guessing, and telling students it is a curriculum.

This special topic was read by an AI voice. Its sources are linked at physicianintheloop.org.