Knowledge note

Perplexity

The exponential of a causal language model's average next-token negative log likelihood on a specified tokenized corpus.

Updated September 16, 2026Confidence: High

Perplexity

Perplexity is the exponential of a causal language model’s average next-token negative log likelihood on a specified tokenized corpus.

Why It Exists

Sequence probabilities multiply and quickly become tiny as text gets longer. Negative log likelihood makes the evidence additive, averaging removes the direct effect of sequence length, and exponentiation returns the result to an effective-choice scale that is easier to read.

How It Works

For target tokens $x_1,\ldots,x_N$:

\[\operatorname{PPL}=\exp\left(-\frac{1}{N}\sum_{t=1}^{N}\log p_\theta(x_t\mid x_{<t})\right).\]

Lower PPL means the model assigned higher probability to the observed tokens, but only under the same tokenizer, evaluation corpus, context window, stride, masking policy, and log base. The common “number of choices” explanation is an effective uniform-choice analogy, not a literal count of candidates.

Tradeoffs

PPL is fast and reference-free, which makes it useful for language-modeling regressions and pretraining comparisons. It does not measure factuality, instruction following, reasoning, or human preference, and PPL values across different tokenizers or corpora are not directly comparable.

Common Confusions

  • PPL vs. accuracy: Accuracy checks only whether the top prediction is correct; PPL rewards assigning calibrated probability to the observed token even when it is not top-1.
  • PPL vs. generation quality: A fluent continuation can have low PPL while being false or unhelpful.
  • PPL vs. cross-entropy: For one-hot causal labels, PPL is the exponential of the average cross-entropy/NLL.
  • PPL vs. token count: A PPL of 20 does not mean the vocabulary has 20 tokens; it is an average effective branching factor under the evaluation distribution.

Where It Appears

  • Perplexity (PPL): From Next-Token Loss to Reliable Evaluation — Derives the metric, works through a three-token example, and explains fixed-length sliding-window evaluation.
  • GPT-1 — Reports language-modeling perplexity on BooksCorpus during the original pretraining-plus-finetuning study.
  • GPT-2 — Uses held-out perplexity to show that the scaled model continues improving on WebText.