LLM Quantization and Low-Precision Serving
· ☕ 1 min read · ✍️ k4i
A series index for LLM quantization and low-precision serving: INT8/INT4, GPTQ, AWQ, SmoothQuant, NF4, AQLM, KV cache quantization, FP8 serving, and quality/speed/memory tradeoffs.
Quantization
A Survey of LLM Quantization: From Linear Quantization to Codebooks
· ☕ 34 min read · ✍️ k4i
A practical survey of LLM quantization, covering linear quantization, codebook quantization, LLM.int8(), SmoothQuant, GPTQ, AWQ, NF4, AQLM, KV cache quantization, and FP8.
