Entropy, Cross Entropy, And KL Divergence: A Coding-Cost View

sky_io@outlook.com (K4i) — Tue, 14 Jul 2026 10:00:00 +0800

Entropy, cross entropy, and KL divergence often look like three separate formulas:

entropy: $H(P)$
cross entropy: $H(P,Q)$
KL divergence: $D_{\mathrm{KL}}(P\Vert Q)$

These letters are conventions rather than variables derived from the formulas. Shannon used $H$ for entropy in his foundational information-theory paper;¹ there is no single agreed explanation for why he chose that letter, so it is safest to treat it as historical notation. Cross entropy still measures an average coding cost and generalizes entropy, hence $H(P,Q)$: the first argument is the true distribution and the second is the coding distribution. KL uses $D$ to emphasize divergence between distributions. A divergence is not necessarily a metric: KL is asymmetric and does not satisfy the triangle inequality.

Why KL Divergence Is Not A Distance: Direction Changes The Question

sky_io@outlook.com (K4i) — Tue, 07 Jul 2026 10:00:00 +0800

KL divergence is often described as a distance between two distributions. That is half useful and half dangerous. It compares two distributions, but it is not a distance because direction matters. More precisely, KL is nonnegative and vanishes only when $P=Q$, but it fails the symmetry and triangle-inequality requirements of a metric.

The formula is:

$$D_{\mathrm{KL}}(P\Vert Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}$$

Do not read this as “the distance between P and Q.” A better reading is:

Information-Theory on k4i's blog

Entropy, Cross Entropy, And KL Divergence: A Coding-Cost View

Why KL Divergence Is Not A Distance: Direction Changes The Question