LLM.co

Before signing off on a rack full of GPUs, there's a short calculation you should run first. This episode breaks down the four numbers that actually determine your GPU count — and why vendor reference architectures almost always overshoot.

Show Notes

GPU sizing for on-premises AI rarely starts with the right question. Vendors lead with reference architectures built for worst-case scenarios, and organizations end up over-provisioned before a single workload requirement has been defined. This episode of LLM.co works through a practical, math-first approach to figuring out how much GPU memory and how many cards a real inference workload actually needs — drawing on the practical GPU sizing guide from hardware.co as its foundation.

The episode walks through each layer of the sizing calculation in sequence, from model weights to concurrency planning, covering:

  • Weight memory as the baseline: why the two-gigabytes-per-billion-parameter rule in FP16 is your starting point, and how quantization (FP8, INT4) can cut that figure dramatically before you ever consider adding hardware.
  • The VRAM floor vs. ceiling distinction: weights only occupy 60–70% of usable VRAM in a well-planned deployment — the rest goes to KV cache, framework overhead, and serving headroom.
  • Why context length is the most common sizing mistake: at long contexts and high concurrency, the KV cache can dwarf model weights entirely, and three software-side choices (grouped-query attention, capped context, paged attention) address it before any hardware purchase.
  • Separating throughput from latency: aggregate tokens-per-second and per-user time-to-first-token are different constraints that drive different decisions — and the episode shows how to multiply through both to arrive at a defensible GPU count.
  • Memory bandwidth over TFLOPS: why LLM inference is memory-bandwidth-bound, and what that means for reading GPU spec sheets.
  • Sizing to the second-worst normal hour, not the theoretical peak: reference architectures assume your worst day is every day; real workloads — whether bursty document review or steady clinical summarization — are not shaped that way.

The five numbers to write down before requesting any quote — target model and quantization, p95 token lengths, peak concurrent users, acceptable token rate, and time-to-first-token — are the same inputs that inform decisions about comparing AI server configurations across different deployment sizes. Teams exploring on-premises deployment may also want to look at fine-tuning and model training services, where GPU memory requirements shift significantly from inference workloads. For more on keeping production LLM deployments stable during infrastructure changes, the episode How to Replace a Public LLM API Without Breaking Production covers the operational side of that transition.

hardware.co

What is LLM.co?

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.

Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.

Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co