https://k4i.top/tags/caching/Recent content in Caching on k4i's blogHugo -- gohugo.ioensky_io@outlook.com (K4i)sky_io@outlook.com (K4i)All content is subject to the license of <a rel="license noopener" href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> .Wed, 22 Apr 2026 11:30:00 +0800https://k4i.top/posts/prefix-caching/Wed, 22 Apr 2026 11:30:00 +0800sky_io@outlook.com (K4i)Sun, 26 Apr 2026 16:08:06 +0800https://k4i.top/posts/prefix-caching/<h2 id="problem">the repeated-prefix problem</h2> <p>the <a href="https://k4i.top/posts/kv-cache/">KV cache</a> eliminates redundant computation <em>within</em> a single request. but in production, a different kind of redundancy is pervasive: <strong>many requests share the same prefix</strong>.</p> <p>three scenarios account for the majority of production traffic:</p> <p><strong>system prompts.</strong> every request to a code assistant, agent, or customer-facing chatbot begins with the same multi-kilotoken system prompt. on each new request, the server re-runs prefill over those identical tokens — then throws away the resulting KV cache when the request ends.</p>K4ifeatured imagellminferencesystemscachingkv-cacheAILLM Inference Internals