LLM.co

When self-hosted LLMs buckle under concurrent users, the KV cache is almost always the hidden bottleneck. This episode breaks down the optimization levers — from PagedAttention and prefix caching to KV quantization and model architecture — that let you serve far more requests without reaching for bigger hardware.

Show Notes

Running a language model in-house is one thing; keeping it responsive when dozens or hundreds of users hit it simultaneously is another challenge entirely. This episode of LLM.co digs into the specific mechanics of KV cache pressure — why it fills GPU memory faster than almost any other factor, and what engineering teams can actually do about it. The discussion is grounded in the KV cache optimization deep-dive for self-hosted LLMs that underpins the episode.

Here's what the episode covers:

  • What the KV cache is and why it scales so aggressively — key-value tensors grow with layer count, head count, context length, and batch size, and naive allocators waste 60–80% of reserved memory to fragmentation before a single request completes.
  • PagedAttention — borrowed from OS virtual memory design, it maps fixed-size cache blocks to physical GPU pages on demand rather than pre-reserving contiguous chunks, routinely doubling concurrent request capacity at a given context length.
  • Prefix caching and RadixAttention — shared system prompts, retrieved documents, and chain-of-thought scaffolding can be computed once and reused across many requests; SGLang's RadixAttention approach has shown up to 5× throughput improvements on structured generation workloads, making it especially relevant for secure enterprise agentic deployments that reuse large prompt fragments.
  • KV cache quantization — dropping tensors from BF16 to FP8 halves memory with negligible quality loss on most models, but gains are entirely engine-dependent; TensorRT-LLM showed up to 45% higher throughput while vLLM showed almost none, because the inference engine must be able to consume the quantized cache.
  • Architectural choices that reduce KV footprint by design — Grouped-Query Attention (as in Llama 3 70B's 8 KV heads vs. 64 query heads) shrinks cache size 8×; DeepSeek's Multi-Head Latent Attention compresses further still, reporting a 93% KV reduction and a 5.76× throughput gain; sliding-window attention caps cache at a fixed context window for chat and short-RAG workloads. These gains matter particularly for teams deploying on-premises custom AI chatbots at scale.
  • CPU offload and speculative decoding — spilling cold KV blocks to host DRAM and using a small draft model to propose tokens in parallel are two tactics for pushing throughput higher when memory is still constrained after everything else has been optimized.

The episode closes with a practical order of operations: architectural choices first, PagedAttention and prefix caching second, quantization third (with engine compatibility verified), and offload or speculative decoding last. For more on the hardware side of self-hosted inference, check out the companion episode Which NVIDIA GPU Do You Actually Need for On-Prem LLMs?

LLM.co

hardware.co

What is LLM.co?

Private and custom large language models — the build, the boundaries and the bill. Fine-tuning versus retrieval, running models in your own environment, evaluation you can actually trust, data governance, and the questions to ask before a vendor answers them for you.

Each episode takes one decision a team is facing — whether your problem needs a custom model at all, how to evaluate output without fooling yourself, what "private" has to mean contractually — and works it through concretely. Written for engineering and data leaders putting a model into production. Five or six minutes, one idea, no demos.

Topics include fine-tuning versus retrieval, self-hosted and private deployment, evaluation you can trust, prompt and context design, data governance and retention, cost and latency tradeoffs, and what "private" has to mean contractually.

Produced by LLM.co, private and custom large language models. Full details, services and further reading at https://llm.co