LLM.co

Healthcare AI teams can't afford to improvise compliance. This episode lays out the exact technical and administrative decisions needed to build a HIPAA-compliant private LLM — from BAA scoping to FIPS encryption and RAG access controls.

Show Notes

With healthcare data breaches averaging $7.42 million in 2025 and regulators tightening the screws on electronic PHI, deploying an LLM in a clinical environment is no longer a product decision — it's a compliance architecture problem. This episode of LLM.co cuts past the "should you do it?" debate and gets into the actual build, drawing directly from the full HIPAA private LLM specification that technical leads need to get it right the first time.

The episode walks through the layered decisions that determine whether a private LLM deployment survives a regulatory review or becomes an incident report, covering:

  • Business Associate Agreements at every layer: HIPAA liability follows the data, meaning your vector store, embedding model, log pipeline, key manager, and identity provider all need BAAs — or must be infrastructure you own outright.
  • Why open-weight, locally hosted models win compliance reviews: When the model provider never touches inference traffic, the compliance perimeter shrinks to auditable infrastructure — the core advantage behind the LLM appliance approach.
  • Model selection and the hallucination trade-off: Llama 3 70B via vLLM is the practical on-premises default; medical fine-tunes like MedGemma and BioMistral show promise but can hallucinate more than strong general-purpose base models on clinical tasks.
  • The 2025 HIPAA Security Rule rewrite: HHS's proposed rule would make encryption at rest/in transit and MFA mandatory rather than "addressable," with a 240-day compliance window after finalization — and FIPS 140-3 validated modules, not just AES-256, are the line that matters in an audit.
  • Audit logging as a required control: Prompt text, retrieved document IDs, model version, user identity, completion hashes, and timestamps must flow to tamper-evident, append-only storage — and be reviewed on a schedule, not only after an incident.
  • Applying the minimum necessary standard to RAG: Retrieval systems must filter by user role, not just query text; de-identification of training and analytics corpora must happen upstream, before fine-tuning, because models can memorize and reproduce sensitive content.

The episode also addresses the administrative and physical safeguard layer — where most Office for Civil Rights findings actually originate — and why workforce training now has to include prompt hygiene alongside traditional access controls. Organizations exploring HIPAA-compliant AI for healthcare will find this a practical framework for scoping what a compliant build actually requires. For teams thinking about infrastructure scale, the related episode How Many GPUs Do You Actually Need? A Practical Sizing Guide covers the GPU concurrency and context-length math in depth.

LLM.co

hardware.co

What is LLM.co?

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.

Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.

Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co