LLM Attention Kernels and GPU Primitives
LLM Attention Kernels and GPU Primitives
· ☕ 1 min read · ✍ī¸ k4i
A series index for LLM attention kernels and GPU primitives: fused softmax, online softmax, FlashAttention, PagedAttention kernels, Triton/CUDA, and memory-access optimization.
LLM Attention Kernels and GPU Primitives
Online Softmax: Tiling for Arbitrarily Large Rows
· ☕ 6 min read · ✍ī¸ k4i
how online softmax extends the fused kernel to handle rows that exceed sram capacity, using a numerically stable 2-pass tiling algorithm.
Online Softmax: Tiling for Arbitrarily Large Rows
Fused Softmax in Triton
· ☕ 7 min read · ✍ī¸ k4i
how to write a fused softmax kernel in triton that eliminates redundant memory accesses and outperforms pytorch's native implementation.
Fused Softmax in Triton