Perplexity and intelligence: what LLM scores really measure

8 steps · about 5 h 10 min · Intermediate

  1. Entropy, Cross-Entropy and KL-Divergence (video, Aurélien Géron (YouTube), 10 min)

    Perplexity is built from cross-entropy, so this short video comes first.

  2. Intro to Information Theory (video, YouTube, 15 min)

    This adds Shannon's ideas of entropy and compression, including the entropy of English, which is the basis for scoring text prediction.

  3. Evaluating Perplexity on Language Models (article, MachineLearningMastery.com, 20 min)

    This shows the range from 1 to vocabulary size and why scores depend on the tokenizer.

  4. Perplexity of fixed-length models (doc, Hugging Face, 30 min)

    A hands-on sliding-window calculation of GPT-2 perplexity on WikiText-2, which shows the practical caveats.

  5. Neural Networks: Zero To Hero (tutorial, Andrej Karpathy, 2 h)

    Build a small language model and watch the loss, the log of perplexity, fall as it learns.

  6. Language Modeling Is Compression (article, alphaXiv (Delétang et al.), 25 min)

    This explains why minimizing next-token loss is mathematically the same as lossless compression.

  7. Compression Represents Intelligence Linearly (article, arXiv (Huang et al.), 30 min)

    Evidence that better compression of text tracks higher benchmark scores across 31 LLMs.

  8. On the Measure of Intelligence (article, François Chollet, arXiv, 1 h)

    A counterpoint that defines intelligence as skill-acquisition efficiency, so you can judge what perplexity does not capture.