Kv-Cache
Why KV Cache Works in LLM Inference
· ☕ 8 min read · ✍ī¸ k4i
why the key-value cache avoids redundant computation in autoregressive decoding, and the memory/compute tradeoffs it introduces.
Why KV Cache Works in LLM Inference