LLM Inference Internals: Core Mechanisms for Serving Engines
· â 1 min read · âī¸ k4i
A series index for core LLM serving mechanisms: prefill/decode, KV cache, PagedAttention, continuous batching, prefix caching, and disaggregated prefill.
Kv-Cache
Prefix Caching: Reusing KV Cache Across Requests
· â 8 min read · âī¸ k4i
When thousands of requests share the same system prompt, recomputing its KV cache each time is pure waste. Prefix caching stores and reuses those vectors, cutting TTFT by up to 97% in common deployments.
Why KV Cache Works in LLM Inference
· â 8 min read · âī¸ k4i
why the key-value cache avoids redundant computation in autoregressive decoding, and the memory/compute tradeoffs it introduces.
