This is a Physician in the Loop special topic.

Notes before codes

Most studies find that AI scribes raise the bill. They should have to show what they do for patients, starting with the accuracy of their notes.

The software that listens to a doctor's visit and drafts the note is moving toward the bill. On Oct. 8 Abridge announced an agreement to expand the use of its scribe across the University of Texas System. In September it announced a product that checks hospitals' inpatient claims against the record before they are sent, so that discrepancies can be resolved before they become denials. The product is built, the company says, "on the same platform and clinical record already used at the point of care," and the hospital's own staff decide what to change.

Scribes may ease doctors' paperwork and improve patients' satisfaction, and most studies of their effect on billing find increases. A tool that raises the bill should have to show what it gains for patients, beginning with whether the record it writes is right. One whose notes get worse should be fixed or withdrawn quickly, whatever it earns.

The gains for doctors are real but small. In an observational study of outpatient clinicians at five American academic medical centers, using a scribe was associated with about a quarter of an hour less documentation for every eight hours of scheduled visits, no significant change in after-hours time in the electronic record, and extra visits that brought in little money. In a randomized trial at UCLA, one product cut the time doctors spent on notes by about a tenth and the other made no significant difference, though both showed signs of easing burnout. In a retrospective study of emergency visits at Mass General Brigham, AI scribes saved less time per note than human scribes did and produced longer notes, though in a primary care preprint, note length did not change. None of these studies measured whether patients fared better.

Tested blind against clinicians on simulated visits, the scribes' notes fell short. Researchers at the Veterans Health Administration had 11 AI scribes and 18 clinicians write notes from recordings of primary care visits acted out by standardized patients, and raters who did not know who wrote each note scored them on a standard measure of quality. "In every case, human notes scored higher, in some cases dramatically so," the American College of Physicians said in announcing the study. The clinicians did not write under the pressures of a real clinic, which may have flattered them. In a randomized trial in everyday practice at the University of Wisconsin, by contrast, the authors concluded that documentation time fell "without compromising diagnosis, billing compliance, or note quality."

On billing, the broadest evidence is also the weakest. Trilliant Health, a health care market-intelligence firm, analyzed claims from six health systems that had announced AI scribes and found that from 2018 to 2024 the share of new patients' visits billed at the two highest levels rose by roughly 12 to 20 percentage points. But it had no comparison group or dates for when the scribes arrived, and the rise coincided with a 2021 rewrite of the coding rules. Studies with a comparison group are few and point both ways: a coding audit found higher visit levels too, though an emergency study at Mass General Brigham found no difference in the physician work billed per shift hour. In private, those who pay and those who are paid sound surer. Investors, health plans and providers at a roundtable held by the Peterson Health Technology Institute, a nonprofit that evaluates new technologies, agreed, as its executive director recalled, that "it's quite clear scribes are increasing coding intensity."

Somebody pays for higher codes. Higher-intensity visits carry higher copays for patients, and Hannah Neprash, a University of Minnesota researcher studying the question, warns of surprise bills: a scribe could turn a preventive visit into a problem visit, "which the patient may not learn about until they see a higher-than-expected bill." Experts expect insurers eventually to lower their rates, hurting doctors and hospitals that did not adopt the software. As KFF Health News has reported, Sister Irene Kraus, once chief executive of the Daughters of Charity National Health System, was famous for coining the phrase "no margin, no mission": a hospital needs money to care for anyone. When the mission becomes the margin, the record is written for the claim, and patients can pay more for the same visit.

The scribes' defense is that the higher codes are the right ones, because the note now records work that was always done. Doctors have long under-coded their visits, those who make it say, and a scribe simply follows the rules "that much more accurately and consistently" than people did, as Trilliant's research chief put it. Its best evidence is a blinded audit at an academic orthopaedic practice, where an independent coder also rated visits documented with a scribe at higher levels than dictated ones, though overall coding accuracy was lower with the scribe, 68% against 74%. Some of the rise may well be correction. But an audit checks the note against the coding rules, not against what happened in the room, and none of these studies measured whether patients did better, the gain that would justify a bigger bill. Nor does relieving burnout, the best case for the scribe, depend on a higher code.

Discharge summaries, written for the doctors who take over a patient's care, are where the technology has done best. In a retrospective study at Stony Brook published on Oct. 2, hospitalists and primary care doctors preferred the AI's summaries of long hospital stays in 19 of 20 cases; the AI left out less, and the estimated risk of harm did not differ significantly. Omission is still the error to watch: the primary care doctors, who act on the summaries, flagged more omissions overall than the hospitalists did and judged harm more likely. In a Dutch study, blinded clinicians rated AI summaries as good overall as physicians' own but less complete, and trusted them less often; in a pilot at Stanford, hospitalists flagged omissions in 25 of the 100 drafts on which they gave feedback. Hospitals that let AI draft the summary should have the discharging physician check what it leaves out, and judge the tool by whether patients get their follow-up, not by whether doctors prefer it.

For the scribe in the clinic, the first test is the note. Before a hospital buys a scribe, and at intervals afterward, clinicians with no stake in the decision should compare a sample of its notes with the visits they record and report the error rate to the medical staff; the authors of the blinded test wrote that "independent, vendor-neutral evaluations of note quality are essential before large-scale clinical deployment." The codes deserve the same scrutiny. A medical staff that sees each doctor's mix of visit levels before and after the scribe arrived, and checks a sample of the highest against the visit as well as the chart, will know whether the scribe is correcting old under-coding or overstating complexity; physicians should sign no level they would not defend to an auditor. What matters most is what happens next. A tool whose notes get worse, or whose codes rise while the care recorded stays the same, should be fixed or switched off at once, not when the contract ends.

Medicare, which holds the claims, is best placed to tell correction from inflation. Dr. Mehmet Oz, who runs the Centers for Medicare and Medicaid Services, said in September that in the short term AI "is going to be inflationary because it's going to turbocharge the ability of the current billing systems to work more effectively." The Minnesota team's study of Medicare claims before and after scribes arrived is scheduled to be completed in December. Medicare should then find out which practices use scribes and audit samples of their high-level visits, so that correction is paid for and inflation refused, practice by practice, not through rate cuts that fall on everyone. Scribe makers, some of which advertise the ability to "maximize the number and severity of diagnoses recorded," according to Dr. Neprash, should publish independent tests of what their notes get wrong.

The software that writes the note is now reading the claim. Abridge describes its products as following a case "from bedside to bill," and that is the right order. A scribe should first prove that it writes a record the next doctor can act on; what it does for the bill should come after.

This special topic was read by an AI voice. Its sources are linked at physicianintheloop.org.