This is a Physician in the Loop special topic. Built for the buyer Much clinical AI is designed for the health systems that pay for it. The doctors who use it should get a vote before it is bought and an off switch for when it slows care. The American Medical Association warned in October that doctors may answer for software they never chose. Its brief on liability for clinical AI says physicians “may be expected to rely on AI systems they were not informed were in use or had no authority to select or decline.” It wants them free to refuse unproven tools without being penalized. It also says the AI duties of hospitals deploying such tools are still unsettled. The trouble starts long before anyone is sued. A doctor may use a clinical AI tool all day, but most of the money spent last year on generative AI built for health care came from health systems. Much of that software is built around what the buyer wants, a clean claim, a full schedule and a better margin. The doctor's day comes second. Physicians should get a formal vote before a clinical AI tool is bought, and a way to switch it off when it slows care. Ask doctors what they want and they describe their chores. When Doximity, which has AI tools of its own, surveyed its physician members, more than four in five wanted help searching the literature, writing letters for patients, educating patients and handling insurance paperwork. Their commonest worry was accuracy. In the AMA's survey early this year, the top requirements for adopting AI included proof of safety and efficacy from a trusted body and “a designated feedback channel when issues arise.” Most wanted a direct role. In all, 85% wanted to be consulted on bringing AI into their practice, or to be responsible for it. What gets bought looks different. Menlo Ventures, a venture firm that invests in health care AI, surveyed about 700 health care executives last year. It estimated that the biggest spending category was scribes, the software that listens to a visit and drafts the note. Next came automated coding and billing. The firm says scribes reduce burnout and coding software recovers revenue lost to errors and denials. Health systems lead, it wrote, because their need is greatest and the return obvious. In 2024 Microsoft said physicians at Northwestern Medicine who used its scribe in at least half their visits could see about 11 more patients a month on average, which it presented as better access to care. Some products are built around improving the bottom line. Ambience Healthcare markets a scribe with “coding awareness as a core feature.” It surfaces billing codes for the clinician to review at the point of care, and the company claims an average of $4.50 more revenue per visit. Ambience presents this as compliance and accurate coding. Perhaps it is. The review still falls to the doctor, in the doctor's time. Scribes change the visit in other ways. Dina Capalongo, a primary care physician at Penn, told KFF Health News that she now says aloud what she is doing and finding during a physical exam, so the AI scribe can record it. Patients find it very interesting, she said. Genevieve Melton-Meaux, chief health informatics and AI officer at Fairview Health Services, said that for a frightened patient, hearing worrying things during an uncomfortable exam “does not help the situation and honestly is insensitive to what the patient is going through.” Only 8% of physicians in Doximity's survey said their institution's process for deciding on AI was clear and that they understood its policies. Nearly half called it “still evolving.” Health systems do switch tools off. Baptist Health South Florida retired a virtual care navigator that fewer than 100 patients a month used, and that suggested urgent or emergency care in nine cases out of ten when better options existed. The question is who pulls the switch, and on what evidence. Health systems have a fair answer. They pay for these tools, sign the contracts and carry the risk on thin margins. Their biggest purchase is the scribe, which doctors wanted. Robert Wachter, chair of medicine at the University of California, San Francisco, told KFF Health News that many systems that might once have run a hard-nosed calculation of return on investment are softening, and “realizing that the cost of recruiting and retaining doctors is pretty high.” Doctors in the AMA's survey worried less about privacy with tools their institution supplied. A vote on every purchase sounds like one more committee. Where buyer and doctor want the same thing, the vote will pass in one meeting. It matters where they part, as with billing prompts inside the visit or a scribe sold on extra patients a month. And some problems appear only in use, as Baptist Health's navigator showed. That is why the switch matters as much as the vote. Medical staffs that have not done so should write both into their bylaws. A clinical AI tool should go live only after the physicians who will use it have seen its evidence and approved it, much as a hospital evaluates a drug for its formulary. The AMA's brief leaves selection to those who deploy a tool, though its House of Delegates has said since 2024 that the organized medical staff should be “an integral part at the outset of choosing” such tools in hospital care. Physicians should share it. The contract should then let the medical staff turn the tool off without penalty if, after a trial in the physicians' own clinics, it misses targets set in advance. Minutes per note, time to answer a patient and errors in a sample of charts would do. The brief already asks deployers for processes to discontinue a tool “when warranted.” The doctors using it should help decide when that is. The AMA's brief warns of doctors relying on software they had “no authority to select or decline.” They still sign the note. A vote before the purchase and a switch after it would give them a say in the tools that write it. Hospitals can start on both now. This special topic was read by an AI voice. Its sources are linked at physicianintheloop.org.