https://k4i.top/tags/batching/Recent content in Batching on k4i's blogHugo -- gohugo.ioensky_io@outlook.com (K4i)sky_io@outlook.com (K4i)All content is subject to the license of <a rel="license noopener" href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank">CC BY-NC-SA 4.0</a> .Wed, 22 Apr 2026 10:30:00 +0800https://k4i.top/posts/continuous-batching/Wed, 22 Apr 2026 10:30:00 +0800sky_io@outlook.com (K4i)Sun, 26 Apr 2026 16:08:06 +0800https://k4i.top/posts/continuous-batching/<h2 id="static-batching">the static batching problem</h2> <p>before continuous batching existed, LLM serving systems used <strong>static batching</strong>: collect a batch of requests, run them all through the model together, and wait until every request in the batch finishes generating before accepting the next batch.</p> <p>this sounds reasonable — batching is how you saturate a GPU — but it has a fatal flaw.</p> <p>different requests produce outputs of wildly different lengths. a request asking &ldquo;what is 2+2?&rdquo; might finish in 5 tokens. a request asking for a short story might need 800. in a static batch, the short request finishes early and then&hellip; does nothing. the GPU keeps crunching for the long request while the short request&rsquo;s slot sits idle.</p>K4ifeatured imagellminferencesystemsbatchingschedulingAILLM Inference Internals