Tokens, attention and transformers: how LLMs work

10 steps · about 9 h 10 min · Beginner

  1. Large Language Models explained briefly (video, 3Blue1Brown, 8 min)

    A seven-minute overview of next-word prediction, parameters and pretraining gives you the big picture before any detail.

  2. Transformers, the tech behind LLMs | Deep Learning Chapter 5 (video, 3Blue1Brown, 27 min)

    Shows how tokens become embeddings and flow through a transformer to produce next-word probabilities.

  3. Transformer Explainer: LLM Transformer Model Visually Explained (interactive, Georgia Tech Polo Club, 25 min)

    Lets you type your own text into a live GPT-2 in the browser and watch tokens, attention and predictions change.

  4. Attention in transformers, step-by-step | Deep Learning Chapter 6 (video, 3Blue1Brown, 26 min)

    Now that you've seen attention in action, this explains queries, keys, values, masking and multiple heads.

  5. The Illustrated Transformer (article, Jay Alammar, 45 min)

    A diagram-by-diagram read that fixes the full architecture in your head, including positional encoding and layer normalization.

  6. Let's build the GPT Tokenizer (video, Andrej Karpathy, 2 h 14 min)

    Goes deep on the first step, Byte Pair Encoding, and explains odd LLM behaviors such as poor arithmetic and string reversal.

  7. karpathy/minbpe: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm (repo, Andrej Karpathy (GitHub), 45 min)

    Read the short reference tokenizer and try the exercise to build your own.

  8. Let's build GPT: from scratch, in code, spelled out. (video, Andrej Karpathy, 2 h)

    You code self-attention and a small transformer in PyTorch, which makes the earlier ideas concrete.

  9. Neural Networks: Zero to Hero (repo, Andrej Karpathy (GitHub), 1 h)

    Has the notebooks and exercises for the GPT and tokenizer lectures, plus earlier lectures if you need the neural-network basics.

  10. Attention Is All You Need (reference, arXiv (Vaswani et al.), 1 h)

    With the intuition and code behind you, the original transformer paper is readable and shows where it all started.