https://k4i.top/zh/tags/latency/Recent content in Latency on k4i's blogHugo -- gohugo.iozhsky_io@outlook.com (K4i)sky_io@outlook.com (K4i)All content is subject to the license of <a rel="license noopener" href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> .Wed, 22 Apr 2026 11:00:00 +0800https://k4i.top/zh/posts/chunked-prefill/Wed, 22 Apr 2026 11:00:00 +0800sky_io@outlook.com (K4i)Sat, 30 May 2026 00:04:33 +0800https://k4i.top/zh/posts/chunked-prefill/<h2 id="interference">干扰问题</h2> <p><a href="https://k4i.top/zh/posts/continuous-batching/">continuous batching</a> 通过按迭代粒度调度请求，让 GPU 尽量保持忙碌。但它有一个很容易破坏延迟体验的边界情况：<strong>很长的 prefill</strong>。</p> <p>当一个带有 2048-token prompt 的请求到达时，朴素调度器会在一次迭代里把整个 prompt 跑完 prefill。以 A100 上的 7B 模型为例，2048-token prefill 大约需要 200 ms。在这 200 ms 里，当前 batch 里已经在流式输出的 decode 请求都要等待。</p>K4ifeatured imagellm推理systemslatencyschedulingAILLM Inference Internals