LLM.co

Choosing the right NVIDIA GPU for on-prem LLMs comes down to three numbers: weight memory, KV cache cost per user, and overhead. This episode breaks down the arithmetic so GPU sizing decisions stop feeling like guesswork.

Show Notes

GPU selection for on-premises AI is widely treated as a hardware preference question, when it's really a memory budget problem with a straightforward formula. This episode of LLM.co walks through the GPU requirements guide for open-weight LLMs published by hardware.co, turning the model-by-model reference into an internalized framework any team can apply before they ever read a spec sheet.

The episode covers the full sizing methodology and then applies it to the models teams are actually running in production today:

  • The three-number formula: Weight memory at your chosen quantization, KV cache cost multiplied by your concurrent user count, and a 10–15% overhead buffer — add them up and your shortlist nearly writes itself.
  • Quantization trade-offs in practice: FP8 is the most robust option across task types; 70B-scale models typically hold strong accuracy at four bits; smaller models can degrade noticeably at INT4.
  • KV cache as the underweighted variable: A dense 70B model can consume 5 GB per request at 8K context or 80 GB at 128K — meaning KV cache, not weight size, often sets the real memory floor for multi-user deployments.
  • Tier-by-tier GPU mapping: 7B–32B models on one or two L40S or RTX 6000 Ada cards; 70B class (e.g., Llama 3.3 70B) on two to four H100s for department-scale concurrency; flagship MoE models like DeepSeek V3/R1 and Qwen3-235B on 8× H100 or H200 nodes.
  • MoE sizing traps: Only a fraction of parameters activate per token, but every weight must still be resident in VRAM — teams must plan for total parameter count, not the active slice.
  • Notable outlier — gpt-oss-120b: OpenAI's Apache 2.0 MoE model, optimized for native MXFP4 on a single 80 GB GPU, is seeing fast on-prem uptake precisely because it breaks the usual multi-card requirement at 120B parameters.

Teams unsure where their workload falls on this spectrum can use the AI Hardware Estimator to calculate VRAM requirements across models, quantizations, and user counts — and compare configurations against hardware.co's rack and cluster offerings. If the right hardware tier turns out to be a multi-card server build, the AI server configuration comparison page maps out four size options side by side. For more on the retrieval infrastructure that often sits alongside these models in regulated environments, the episode Pinecone, Qdrant, Weaviate, and pgvector for Regulated Enterprise RAG is a natural companion listen.

hardware.co

What is LLM.co?

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.

Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.

Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co