Special Topic: Who grades the answers

Show Notes

American clinicians asked one AI reference tool 271 million questions in two and a half years. How often such tools get the answer right, on the questions clinicians actually ask, remains largely unmeasured.

Read the special topic and its sources: https://physicianintheloop.org/specials/curbside-at-scale/

The voice on this episode is generated by AI. It reads a special topic from Physician in the Loop as written; every fact in it links to its source.

What is Physician in the Loop?

How technological innovation is changing medicine for patients and the people who practice on the front lines.

This is a Physician in the Loop special topic.

Who grades the answers

American clinicians asked one AI reference tool 271 million questions in two and a half years. How often such tools get the answer right, on the questions clinicians actually ask, remains largely unmeasured.

You know the question. It comes at the end of a shift, about a drug interaction or a dose in renal failure, and it used to go to your colleague standing in the corridor. Now it often goes to a machine. Between January 2024 and June 2026, clinicians in the United States put 271 million of those questions to a single piece of software. A cross-sectional study of its query logs, published in JAMA on Oct. 15, counted 952,008 clinicians with verified provider numbers on OpenEvidence, which answers clinical questions and shows its citations.

The study did not measure whether a single one of those answers was right.

That is the curbside consult at industrial scale. How these tools perform on the treatment and medication questions clinicians actually put to them “remains largely unmeasured,” the study's authors write, which limits any clinician's ability to judge reliability where the tools get used most. That is not a research question. It is an audit. It belongs to the company, because a product asked this many questions should be graded on the questions it is asked, by clinicians it does not pay.

Independent evaluation is scarce, in the Nature Medicine team's word, and its verdicts are mixed. The team's own is unflattering. Writing in June, a team led from NYU Langone Health tested OpenEvidence and Wolters Kluwer's UpToDate Expert AI against three general-purpose models from OpenAI, Google and Anthropic, using two public benchmarks, one of them exam-style questions, and 100 real queries physicians had typed at one health system. Twelve clinicians reviewed the answers blind. That produced 1,800 ratings. The general-purpose models won all three comparisons, and on the real queries the clinical tools did about as well as Google's automatic search summaries. These were answers scored by clinicians, not care delivered to patients.

OpenEvidence rejected the finding. Becker's Hospital Review reported that in a June 15 letter to the journal the company asked for a retraction, a public apology and an independent review. The company argued the paper used flawed methods and that the public benchmarks may have sat in the models' training data. The letter has not been published, this publication has not seen it, and this account rests on that reporting. On the benchmark point the paper's authors agree with them. The models “may have been exposed to MedQA or HealthBench during training,” they write, and they treat the blinded review of real queries, free of that problem, as their primary evidence. By the same account the senior author said the team stands by the paper, and the journal pointed to its process for formal rebuttals without addressing the retraction demand directly. In September the journal published a critique of the paper, by researchers at the University of Chicago and the University of California, San Diego, with a reply from its authors.

The company has a study on its side, by academic authors who report no affiliation with it. A preprint it helped design, whose survey it administered and whose physicians it paid, in which 149 physicians rated answers to 620 real questions from its platform and preferred its own by 25 to 39 percentage points. It has not been peer reviewed. Among the other evaluations the company pointed to, by Becker's account, was a peer-reviewed study of five patients. Mayo Clinic physicians put questions to the tool from five retrospective cases, and four independent physicians rated the answers accurate and evidence-based in all five. Five cases cannot tell anyone how often a tool is right across 271 million answers.

Where the tested models go wrong is not random either. In Annals of Internal Medicine this month, researchers in Israel built 225 scenarios from 15 Choosing Wisely recommendations, the specialty societies' own lists of low-value tests and treatments, and put them to six commercial models for 4,050 answers. The models called for the discouraged test or treatment in 5.4% of answers and withheld an appropriate one in 0.6%. Most of that came from two of the 15 recommendations. On the larger one, a small thyroid nodule found by accident, more than two in five answers called for an ultrasound, where the society guidance, the authors note, is openly contested.

Doctors who reach for the tool a dozen times a shift will say these tests measure the wrong thing. The objection deserves a hearing. The Annals paper calls itself a benchmark rather than real-world use, built on single prompts and scored by another model with physicians adjudicating. The Nature Medicine queries came from physicians using a general-purpose chatbot. And the honest comparison is not a perfect library. It is what the question would otherwise have got, which is often a half-remembered guideline and a colleague's guess. Every one of those points holds. Not one of them argues against measuring. Many of the tasks used to evaluate clinical AI “may not match the tasks for which physicians actually use it,” the usage study concludes. The company holds both the questions and the answers.

So publish the audit only the company can run. Take a random sample of real answers. Report how often the answer is correct, and how often the source beneath it supports the claim. Have clinicians it neither employs nor pays do the judging. Release the method so somebody else can repeat it. The publishers licensing their journals to it hold a lever, and the JAMA Network is among them. An audit is a fair price for lending a journal's authority to a machine's answer. Until then, read the company's own terms, which say it plainly. “OpenEvidence Content is not intended to provide medical advice, diagnosis or treatment.” A curbside consult generally shifts no responsibility to the colleague who gives it, and the citation under each answer is there to be opened.

Counting the questions was the easy part. Grading the answers is the part that would tell a doctor something, and the study that did the counting says that work remains largely undone.

This special topic was read by an AI voice. Its sources are linked at physicianintheloop.org.