Physician in the Loop

Safety prompt cuts harmful AI choices in 19 of 20 models; Oracle breach tally nears 20 million; device coverage comments close Oct. 13; Mayo AI grades side effects

Show Notes

Safety prompt cuts harmful AI choices in 19 of 20 models; Oracle breach tally nears 20 million; device coverage comments close Oct. 13; Mayo AI grades side effects

The voices on this episode are generated, and so is the script, from the day's post at Physician in the Loop. Every fact in the post links to its source.

Read the post and its sources

Get the free email briefing

A short safety reminder cut potentially harmful clinical choices by 20 open-weight language models from 16.6% to 10.1% across more than 10 million responses, Mount Sinai researchers reported in Communications Medicine.

The models still picked a harmful option in about 1.18 million responses, and the reminder helped least on cases drawn from real hospital records, where harmful choices fell from 46.6% to 24.5%, the authors reported. They called the work a proof of concept that does not estimate how often harm would occur in deployed systems.

Elsewhere, filings with state attorneys general put the toll of Oracle Health's 2025 breach of legacy Cerner servers at nearly 20 million people, according to Fierce Healthcare and SecurityWeek. Comments on Medicare's proposed fast coverage pathway for breakthrough devices close Tuesday. The weekend was otherwise quiet, and the post runs shorter.

In this episode

What is Physician in the Loop?

How technological innovation is changing medicine for patients and the people who practice on the front lines.

Voice 1: This is Physician in the Loop, the daily briefing on AI in medicine. It's Sunday, October 11th, 2026.

Voice 1: A short safety reminder cut harmful clinical choices by twenty AI language models from about 17% down to about 10%, across more than 10 million responses that Mount Sinai researchers logged.

Voice 2: What does that reminder actually say to the model?

Voice 1: It tells the system to verify or escalate an unsafe request. Researchers at the Icahn School of Medicine at Mount Sinai tested it on 601 clinical scenarios, 100 of them pulled from real hospital records, each one nudging toward something like skipping follow-up blood tests or stopping antibiotics early.

Voice 2: Did it work the same everywhere, or worse on the real cases?

Voice 1: Much worse on the real cases. Harmful choices there fell from nearly 47% to about 25%, even with the reminder in place. Across the whole study, the models still chose a harmful option in more than 1 million responses.

Voice 2: So this is a patch, not a fix.

Voice 1: That is how the first author put it. Dr. Mahmud Omar said a reminder should be viewed as one safeguard, not a substitute for clinical oversight. The authors also said the study does not show how often harm would occur in actual practice.

Voice 2: A different kind of risk showed up in Oracle Health's breach filings this week.

Voice 1: State filings now put the toll of Oracle Health's 2025 breach of legacy Cerner servers at nearly 20 million people. Texas alone counts nearly 3 million residents affected, according to its own attorney general filing.

Voice 2: What kind of data are we talking about?

Voice 1: A sample notice filed in California lists names, Social Security numbers, diagnoses, medicines, test results and images. Oracle has not disclosed a total and did not comment.

Voice 2: Has anyone been held responsible yet?

Voice 1: A federal judge in Missouri ruled in June that a consolidated class action can proceed against Oracle and eight health systems. The ruling held that providers cannot be “absolved of liability” for protecting patient data by contracting it out.

Voice 2: So outsourcing the server doesn't outsource the responsibility.

Voice 1: That's the judge's read on it, at least for now.

Voice 2: Regulators also have a clock running on breakthrough devices.

Voice 1: Comments close Tuesday on Medicare's proposed RAPID pathway. Under it, CMS would post a proposed national coverage decision the same day the FDA authorizes an eligible breakthrough device, with a final call about 60 days later for Class II devices and about 90 days later for Class III.

Voice 2: Any device qualifying, or are there limits?

Voice 1: Devices have to enter at the pre-investigational stage and run a study measuring outcomes in Medicare beneficiaries. Diagnostics are excluded, and the earlier TCET pathway is paused for new candidates.

Voice 2: AdvaMed must have something to say about that.

Voice 1: The device trade group asked CMS on Thursday to widen eligibility, admit diagnostics and set firm timelines. Its chief executive, Scott Whitaker, said Medicare beneficiaries shouldn't have to wait for FDA-authorized breakthrough technologies.

Voice 2: Does the notice say anything about AI-driven devices specifically?

Voice 1: No. The notice does not mention software or artificial intelligence at all.

Voice 2: Mayo Clinic put a grading tool to work on a much more personal kind of data, recorded conversations with cancer patients.

Voice 1: What was it actually grading?

Voice 2: Radiation side effects. Mayo combined speech recognition with language models to score urinary and bowel toxicities on the National Cancer Institute's standard scale, using recorded visits with 31 men getting radiotherapy for prostate cancer, across 39 visits.

Voice 1: How close did it come to expert judgment?

Voice 2: About 83% exact agreement with the expert reference grade, with a weighted kappa of 0.717. Expert raters agreed with each other at 0.681, so the tool landed right around that range.

Voice 1: That sounds strong. Is there a catch?

Voice 2: Most assessments were grade zero, which inflates the raw accuracy number. For grades 1 and 2, the F1 scores dropped to about 0.6, and about 32% of positive flags turned out to be false.

Voice 1: So it's better at saying nothing is wrong than at catching something real.

Voice 2: The authors said it themselves: accuracy at grade 2 or higher is still unproven.

Voice 1: Sources and corrections are at physicianintheloop.org, where you can sign up for the free email briefing. This briefing was read by AI voices.