Why KL Divergence Is Not A Distance: Direction Changes The Question

sky_io@outlook.com (K4i) — Tue, 07 Jul 2026 10:00:00 +0800

KL divergence is often described as a distance between two distributions. That is half useful and half dangerous. It compares two distributions, but it is not a distance because direction matters. More precisely, KL is nonnegative and vanishes only when $P=Q$, but it fails the symmetry and triangle-inequality requirements of a metric.

The formula is:

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}$$

Do not read this as “the distance between P and Q.” A better reading is:

Rlhf on k4i's blog

Why KL Divergence Is Not A Distance: Direction Changes The Question