Tokens, attention and transformers: how LLMs work
10 steps · about 9 h 10 min · Beginner
- Large Language Models explained briefly (video, 3Blue1Brown, 8 min)
A seven-minute overview of next-word prediction, parameters and pretraining gives you the big picture before any detail.
- Transformers, the tech behind LLMs | Deep Learning Chapter 5 (video, 3Blue1Brown, 27 min)
Shows how tokens become embeddings and flow through a transformer to produce next-word probabilities.
- Transformer Explainer: LLM Transformer Model Visually Explained (interactive, Georgia Tech Polo Club, 25 min)
Lets you type your own text into a live GPT-2 in the browser and watch tokens, attention and predictions change.
- Attention in transformers, step-by-step | Deep Learning Chapter 6 (video, 3Blue1Brown, 26 min)
Now that you've seen attention in action, this explains queries, keys, values, masking and multiple heads.
- The Illustrated Transformer (article, Jay Alammar, 45 min)
A diagram-by-diagram read that fixes the full architecture in your head, including positional encoding and layer normalization.
- Let's build the GPT Tokenizer (video, Andrej Karpathy, 2 h 14 min)
Goes deep on the first step, Byte Pair Encoding, and explains odd LLM behaviors such as poor arithmetic and string reversal.
- karpathy/minbpe: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm (repo, Andrej Karpathy (GitHub), 45 min)
Read the short reference tokenizer and try the exercise to build your own.
- Let's build GPT: from scratch, in code, spelled out. (video, Andrej Karpathy, 2 h)
You code self-attention and a small transformer in PyTorch, which makes the earlier ideas concrete.
- Neural Networks: Zero to Hero (repo, Andrej Karpathy (GitHub), 1 h)
Has the notebooks and exercises for the GPT and tokenizer lectures, plus earlier lectures if you need the neural-network basics.
- Attention Is All You Need (reference, arXiv (Vaswani et al.), 1 h)
With the intuition and code behind you, the original transformer paper is readable and shows where it all started.