https://k4i.top/tags/distributed/Recent content in Distributed on k4i's blogHugo -- gohugo.ioensky_io@outlook.com (K4i)sky_io@outlook.com (K4i)All content is subject to the license of <a rel="license noopener" href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> .Wed, 22 Apr 2026 12:00:00 +0800https://k4i.top/posts/disaggregated-prefill/Wed, 22 Apr 2026 12:00:00 +0800sky_io@outlook.com (K4i)Sun, 26 Apr 2026 16:08:06 +0800https://k4i.top/posts/disaggregated-prefill/<h2 id="ceiling">why same-GPU coexistence has a ceiling</h2> <p><a href="https://k4i.top/posts/chunked-prefill/">chunked prefill</a> makes prefill-decode coexistence more tolerable by slicing the prefill into small pieces. but even with perfect chunking, prefill and decode are still <em>sharing the same GPU</em>. they compete for:</p> <ul> <li><strong>HBM bandwidth</strong> — both need to read from and write to GPU memory each iteration</li> <li><strong>compute units</strong> — prefill&rsquo;s GEMM and decode&rsquo;s GEMV contend for the same tensor cores</li> <li><strong>KV cache space</strong> — prefill temporarily occupies blocks that could serve decode requests</li> </ul> <p>at moderate scale, this coexistence is acceptable. at large scale — thousands of requests/second, strict SLOs, multi-GPU clusters — the competition becomes a bottleneck that chunking alone cannot resolve.</p>K4ifeatured imagellminferencesystemsdistributedschedulingAILLM Inference Internals