Perplexity and intelligence: what LLM scores really measure
8 steps · about 5 h 10 min · Intermediate
- Entropy, Cross-Entropy and KL-Divergence (video, Aurélien Géron (YouTube), 10 min)
Perplexity is built from cross-entropy, so this short video comes first.
- Intro to Information Theory (video, YouTube, 15 min)
This adds Shannon's ideas of entropy and compression, including the entropy of English, which is the basis for scoring text prediction.
- Evaluating Perplexity on Language Models (article, MachineLearningMastery.com, 20 min)
This shows the range from 1 to vocabulary size and why scores depend on the tokenizer.
- Perplexity of fixed-length models (doc, Hugging Face, 30 min)
A hands-on sliding-window calculation of GPT-2 perplexity on WikiText-2, which shows the practical caveats.
- Neural Networks: Zero To Hero (tutorial, Andrej Karpathy, 2 h)
Build a small language model and watch the loss, the log of perplexity, fall as it learns.
- Language Modeling Is Compression (article, alphaXiv (Delétang et al.), 25 min)
This explains why minimizing next-token loss is mathematically the same as lossless compression.
- Compression Represents Intelligence Linearly (article, arXiv (Huang et al.), 30 min)
Evidence that better compression of text tracks higher benchmark scores across 31 LLMs.
- On the Measure of Intelligence (article, François Chollet, arXiv, 1 h)
A counterpoint that defines intelligence as skill-acquisition efficiency, so you can judge what perplexity does not capture.