Choosing the right NVIDIA GPU for on-prem LLMs comes down to three numbers: weight memory, KV cache cost per user, and overhead. This episode breaks down the arithmetic so GPU sizing decisions stop feeling like guesswork.
GPU selection for on-premises AI is widely treated as a hardware preference question, when it's really a memory budget problem with a straightforward formula. This episode of LLM.co walks through the GPU requirements guide for open-weight LLMs published by hardware.co, turning the model-by-model reference into an internalized framework any team can apply before they ever read a spec sheet.
The episode covers the full sizing methodology and then applies it to the models teams are actually running in production today:
Teams unsure where their workload falls on this spectrum can use the AI Hardware Estimator to calculate VRAM requirements across models, quantizations, and user counts — and compare configurations against hardware.co's rack and cluster offerings. If the right hardware tier turns out to be a multi-card server build, the AI server configuration comparison page maps out four size options side by side. For more on the retrieval infrastructure that often sits alongside these models in regulated environments, the episode Pinecone, Qdrant, Weaviate, and pgvector for Regulated Enterprise RAG is a natural companion listen.
Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.
Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.
Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.
Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co