The Short Answer
A KV cache stores the Key (K) and Value (V) tensors computed for previous tokens during autoregressive generation, so the model doesn't recompute them for every new token. It trades memory for speed — turning what would be O(n²) repeated work into O(n).
Why It Exists
LLMs generate text one token at a time. To produce the next token, the attention mechanism needs the Keys and Values of all previous tokens.
Without a cache, every time you generate a new token, you'd recompute the K and V projections for the entire sequence so far. For a 1000-token output, you'd redo the work for tokens 1-999 again and again — hugely wasteful.
With a KV cache:
- When token
iis processed, its K and V vectors are computed once and stored. - For token
i+1, you only compute K/V for the new token and reuse the cached values for all prior tokens.
The Trade-off (this is what interviewers probe)
- Speed: Massive. You go from recomputing everything each step to only computing the new token.
- Memory cost: The cache grows linearly with sequence length AND batch size. Memory ≈
2 × num_layers × num_heads × head_dim × seq_len × batch × dtype_size. For long contexts, the KV cache can consume more GPU memory than the model weights themselves.
This memory pressure is exactly why techniques like Multi-Query Attention (MQA), Grouped-Query Attention (GQA), and PagedAttention (vLLM) exist — they reduce or better manage KV cache memory.
Why Interviewers Ask
They want to see if you understand:
- The autoregressive nature of LLM inference
- The compute-vs-memory trade-off
- Awareness of why long-context models are expensive to serve
Follow-ups You Should Prepare
- Why does the KV cache NOT help during training (only inference)? (training processes the full sequence in parallel)
- How do MQA and GQA reduce KV cache size?
- What is PagedAttention and how does vLLM use it to manage KV cache memory?
- How does KV cache size scale with context length — and why is a 128K context model so memory-hungry?