This is a Physician in the Loop special topic.

Aim for better, not just sooner

AI health products tend to report how many people they reached and how quickly. The test that matters is whether those patients did better, and buyers should ask before they scale.

In Dresden, neurosurgery patients who got a checked, plain-language AI summary of their findings before their first visit had shorter visits and rated them higher. In a randomized trial of 300 adults published on Oct. 15, the median consultation ran 25 minutes with the summary and 33 without. The last author co-founded the company whose software wrote the summaries, though the university says he took no part in collecting or analyzing the data. Their abstract ends on the sentence that matters. Effects on clinical outcomes and on doctors' workload, they write, "require confirmation before routine adoption."

That sentence is the right test for AI in medicine. Shorter visits, shorter waits, more patients seen and more questions answered all have value. Success is patients who do better, with fewer deaths, complications, admissions and missed diagnoses. Insurers and health systems should ask for that evidence before they spread a tool, not after.

Trials of AI mostly ask something else. A 2024 review of 86 randomized trials of clinical AI found that four in five reported a positive main result, mostly on how much the software found or how well it performed. Of 4,667 studies of medical AI gathered from a year of systematic reviews, about one in forty was randomized. Utah's office of AI policy says its pilot with Doctronic, a company whose software renews prescriptions, could "expand patient access." It began by collecting safety data, with economic and public health outcomes planned.

A randomized trial in Kenya shows the gap. This summer Vyonne Njeri, a clinical officer in Nairobi, told NPR about an infant she had taken for a cold until the software reading her notes prompted her to check his heart. "That's something I would have missed on any other day," she said. Her story came after the trial. Across the trial's 16 clinics in 2025, officers with the software wrote better notes and recorded an appropriate diagnosis more often. By the main test, their patients did no better. Treatment failure, anything from a return visit with symptoms unresolved to a missed diagnosis or death, struck about two patients in a hundred in each group within two weeks. The assistant was safe, the authors concluded, but "did not reduce treatment failure within 14 days and any benefit, if present, is probably modest."

An earlier study at Penda Health, the chain that runs those clinics, found the same split. It was a quality improvement study, paid for by OpenAI, which took part in the analysis. Clinicians with the tool made 16% fewer diagnostic errors and 13% fewer treatment errors, as judged by doctors reviewing the records blind. "There were no statistical differences in patient-reported outcomes," the authors wrote. The Kenyan trial's authors suggest such gains may reach patients eventually, if the theories behind them hold. Neither study could show that they had.

Sometimes the answer is yes. In a randomized trial of nearly 16,000 hospital patients by researchers in Taipei, software read each patient's electrocardiogram and warned doctors about those at high risk of dying. Within three months, about seven fewer patients in every thousand had died in the group whose doctors got the warnings. In a randomized trial across 74 hospital units at two health systems, software that read nurses' charting for signs of decline lowered the moment-to-moment risk of dying in the hospital by about a third, and more patients were moved unexpectedly to intensive care, the kind of early escalation its authors say can improve survival.

Speed has a serious defense. By OpenAI's own analysis, ChatGPT averaged more than 580,000 health messages a week in late 2025 from places over half an hour's drive from a general hospital. For someone who cannot get an appointment at all, an answer tonight beats a perfect trial in five years, and demanding proof can keep people waiting. The wait does real harm. But an answer is worth what the patient does with it. In a randomized study run in 2024, in which 1,298 adults in Britain worked through medical scenarios written by doctors, those using a chatbot named a relevant condition in about a third of cases at best, less often than people using any source they liked. Tested alone, the chatbots almost always named one. Even OpenAI's report says AI "will not, on its own, reopen a shuttered hospital."

So ask where asking is cheapest, before a pilot becomes a contract. Proof need not be slow, though it may need to be large. The Kenyan trial enrolled 9,691 patients in under three months, and its authors report that showing a modest effect on rare events such as hospitalization and death would take more than 100,000. A health system can switch a tool on for some clinicians and not others, as that trial did, and follow one patient result beside the access figure, such as admissions or missed diagnoses. Insurers covering more than 165 million Americans have pledged to offer outcomes-based payment for technology-supported care by Jan. 1, 2028, in line with a Medicare model that bases payment on the share of patients meeting their outcome targets. They should hold every AI tool and AI-run service to the same rule, and health systems every tool they buy. Utah should have the outcomes it plans to add before any group of Doctronic's drugs reaches the phase in which the software sends renewals straight to the pharmacist.

The Dresden visits were eight minutes shorter, and patients felt better informed. Whether any of them recovered better is a question the trial was not built to answer, and its authors said so. Faster care is a means to that end. Until a tool shows that its patients did better, its speed is only a promise, and nobody should buy it at scale.

This special topic was read by an AI voice. Its sources are linked at physicianintheloop.org.