A back-of-the-envelope framework for estimating LLM training FLOPs, inference FLOPs, weight memory, KV cache, and training memory.

Routing prefill and decode to separate GPU pools eliminates interference entirely, enabling independent scaling and optimal latency — at the cost of KV cache migration across machines.

When thousands of requests share the same system prompt, recomputing its KV cache each time is pure waste. Prefix caching stores and reuses those vectors, cutting TTFT by up to 97% in common deployments.

Splitting a long prefill across multiple iterations keeps decode requests from stalling, with no extra FLOPs and negligible IO overhead.

How iteration-level scheduling eliminates GPU idle time by inserting new requests the moment a slot opens, and the math behind mixing prefill and decode in a single forward pass.

How vLLM borrows the OS paging idea to eliminate KV cache memory fragmentation, pushing GPU utilization from ~30% to ~96%.

how online softmax extends the fused kernel to handle rows that exceed sram capacity, using a numerically stable 2-pass tiling algorithm.

why the key-value cache avoids redundant computation in autoregressive decoding, and the memory/compute tradeoffs it introduces.

how to write a fused softmax kernel in triton that eliminates redundant memory accesses and outperforms pytorch's native implementation.

A practical guide to SSH local and remote port forwarding, with examples, comparison, and persistent configuration via ~/.ssh/config.