LLM.co

Buying the wrong GPU server for your open-weight LLM isn't just wasteful — it can kill performance entirely. This episode breaks down exactly how to match hardware specs to model size, from single-card workstations to eight-GPU clusters.

Show Notes

Choosing the right GPU server for an open-weight LLM isn't as simple as headcount or budget — it comes down to model size, interconnect bandwidth, KV cache math, and concurrency requirements that only become visible once hardware is on a purchase order. This episode walks through the detailed guidance in the GPU server configuration guide for open-weight LLMs published on hardware.co, translating the key trade-offs into practical decision-making across four distinct build tiers.

Here's what the episode covers:

  • Start with the model, not the chassis. Four parameter bands — 7–14B, 24–34B, 70B dense, and large MoE — each carry specific VRAM, RAM, and interconnect requirements that should drive the server spec, not the other way around.
  • VRAM and KV cache are the two numbers that matter most. Weights run ~2 bytes per parameter at BF16 and drop to ~0.5 bytes at INT4; KV cache scales with context length and concurrency and can easily exceed model weight storage at long-context serving loads.
  • Single-GPU workstations are a legitimate production tier. An 80 GB card can serve a 70B model at 4-bit, a 32B at 8-bit, or any 7–14B at full precision — making it the right landing spot for most pilots and small-team deployments. Teams planning this kind of build can estimate their exact VRAM needs with the AI Hardware Estimator.
  • Two-GPU builds live or die by their interconnect. Whether to run dual independent replicas, tensor-parallel a larger model, or dedicate one card to embeddings and reranking depends almost entirely on whether the cards share NVLink or are limited to PCIe peer-to-peer — a gap that can represent an order-of-magnitude difference in cross-GPU bandwidth.
  • Four GPUs is the honest tier for 70B-class dense models and medium MoE. Mixtral's MoE architecture means you pay for ~46.7B parameters in VRAM but compute at roughly 13B-dense speeds — a useful planning target that explains why four-GPU configs suit MoE workloads so well.
  • Eight-GPU nodes unlock frontier MoE and serious fine-tuning — but DeepSeek-V3's 671B total parameters require multi-node InfiniBand or RoCE setups, a parallel filesystem, and 8–12 kW of power capacity even before accounting for KV cache. For teams scaling to this level, a purpose-built GPU cluster for large models and high concurrency is worth evaluating.

The episode makes clear that GPU count is the headline spec, but interconnect, system RAM, and NVMe storage are often where deployments succeed or fail. For more on the cost and security trade-offs of running models on your own hardware versus a managed service, check out the related episode Private LLM vs ChatGPT Enterprise: Cost, Security, and Performance.

hardware.co

What is LLM.co?

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.

Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.

Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co