This is a Physician in the Loop special topic.

Cleared is not the same as proven

An FDA clearance shows that software met a regulator's floor; physicians should ask for evidence that it works on patients like theirs before they trust it.

On Oct. 1 two of Medicare's regional contractors proposed not to pay, across 20 states, for software that measures the brain on MRI. The products they named, among them NeuroQuant and icobrain, are cleared by the FDA. "At this time, there is insufficient evidence of clinical utility or validity," the proposals say, and the technology is "considered investigational." The contractor for Kentucky and Ohio stopped covering it in January, and another has proposed the same for 10 more states.

The contractors have put plainly what physicians who use AI should already assume. An FDA clearance or a European CE mark shows that a product met a regulator's floor for a stated use. It does not show that the product works on a given hospital's patients, machines and habits, and the evidence that it would is often thin, private or missing. Authorization should be where a buyer's questions begin, and physicians and hospitals should insist on answers before they let the output steer care.

Almost every AI device reaches the American market by showing that it is substantially equivalent to one already sold: 1,376 of the 1,430 authorized through 2025 took that route, according to an analysis reported by AuntMinnie. Showing equivalence often requires no clinical study. Of 521 AI devices the FDA authorized through 2022, 226 had no published clinical validation, and of the 148 tested on patients as they came in, 22 went through a randomized trial. An earlier review found that 97% of 130 devices authorized from 2015 to 2020 had been evaluated only on past data, and most did not say whether they had been tested at more than one site. Much of this was legal: many of these products did not need validation data, as the lead author of the 2024 study said.

In Europe the gap can be wider. A review published on Oct. 5 by researchers in Helsinki and Split found public evidence of performance for only 30 of 77 authorized AI products in pathology and in the microscope analysis of blood and marrow cells. All eight that also held American authorization had public data; of the 69 authorized only in Europe, fewer than a third did. Where evidence existed it came in 15 different measures, which makes products hard to compare. The authors note that performance results do not have to be published, and they describe "a gap between evidence availability and regulatory authorization."

Even good evidence describes someone else's hospital. Software that flags broken necks on CT reported a sensitivity of 91.7% in its FDA disclosure; at one academic hospital it found 54.9% of fractures, and most of the scans it flagged had no fracture, though it wrongly flagged a smaller share of fracture-free scans than the disclosure implied. Prediction models built into record systems behave the same way. Epic's sepsis model missed two-thirds of sepsis cases at the University of Michigan while alerting on nearly a fifth of all inpatients, and across nine hospitals in one network its ability to tell patients who developed sepsis from those who did not ranged from little better than a coin toss to near the developer's own figures, falling where sepsis was most common. In another study, chest X-ray models could tell which hospital system an image came from almost every time, and a model trained to find pneumonia did worse at a hospital it had not learned from.

When cleared products fail, it tends to be early and among those with the least evidence. Of 950 AI devices, 60 were recalled, 182 times in all; 43% of the recalls came within a year of clearance, and devices with no reported clinical validation were more likely to be recalled. The study's authors suggested requiring prospective evaluation, or clearances that lapse without confirming data.

Nor is the floor about to rise. The FDA's guidance on AI device software has been a draft since January 2025; the agency says it intends to finalize it in the fiscal year that began on Oct. 1. A federal requirement that record-system vendors disclose how their predictive tools were built and tested would be dropped under a proposal published in December. Europe has postponed its AI Act's high-risk rules for AI in medical devices to August 2028. Until regulators ask for more, buyers will have to.

The objection is that this asks too much. Published summaries are not the whole file: the FDA says its summaries leave out most of what companies submit, so missing data is not the same as missing review. Recalls are uncommon, touching about 6% of AI devices. Some authorized tools work in ordinary practice: in an observational study of more than 460,000 women screened in Germany with a CE-marked product, radiologists reading with AI support found more cancers without calling back more women. And small hospitals cannot run trials. Those with fewer resources buy products "off the shelf," as one researcher put it, and a preprint found that rural hospitals in one network lacked the patients needed to recalibrate a model locally. All of that is true, and none of it argues against asking. The vendor holds the evidence the regulator saw, and handing it over costs little. Where good evidence exists, as in Germany, it is easy to show. A hospital too small to test a tool is the one that most needs the seller to have done the testing.

Hospitals should therefore treat authorization as the start of a purchase rather than the end of one. The first question is exactly what was cleared and for what use, since outside that use even the regulator's floor does not apply. Then comes the full validation report rather than the brochure: how many sites and patients, checked against what, with results by age, sex, race and scanner, and how often a positive result is right at a rate of disease like the hospital's own. Who paid for the studies matters too, and any claim that a tool improves care should rest on a prospective or randomized study; this publication's reviews of 99 clinical AI tools found randomized trials showing a benefit in routine care for eight of them. The answers belong in the contract. A vendor that will not put them in writing has answered anyway.

Even good answers describe someone else's patients, so a tool should first run silently on recent local cases, its output compared with what clinicians decided and what happened to the patients; a small hospital can share that work with others that use the same product. Once it is live, someone named should watch its alerts, overrides and misses, the vendor should report every update and recall, including changes made under a plan the FDA authorized in advance, and the hospital should keep the right to switch it off. And a hospital should check coverage before it buys. Medicare's contractors already refuse to pay for some cleared brain MRI software in two states and have proposed the same in 30 more.

Medicare's contractors have started asking cleared software for proof that it helps patients. Physicians should ask before the software reaches them, not after the claim is denied.

This special topic was read by an AI voice. Its sources are linked at physicianintheloop.org.