https://k4i.top/tags/latency/Recent content in Latency on k4i's blogHugo -- gohugo.ioensky_io@outlook.com (K4i)sky_io@outlook.com (K4i)All content is subject to the license of <a rel="license noopener" href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> .Wed, 22 Apr 2026 11:00:00 +0800https://k4i.top/posts/chunked-prefill/Wed, 22 Apr 2026 11:00:00 +0800sky_io@outlook.com (K4i)Sun, 26 Apr 2026 16:08:06 +0800https://k4i.top/posts/chunked-prefill/<h2 id="interference">the interference problem</h2> <p><a href="https://k4i.top/posts/continuous-batching/">continuous batching</a> keeps the GPU busy by scheduling at iteration granularity. but one edge case breaks the latency story: <strong>long prefills</strong>.</p> <p>when a request arrives with a 2048-token prompt, the scheduler runs it through prefill in a single iteration. on an A100, a 2048-token prefill for a 7B model takes roughly 200 ms. all the decode requests already in the batch are blocked for the entire duration.</p>K4ifeatured imagellminferencesystemslatencyschedulingAILLM Inference Internals