Steering LLMs in realtime: how text generation works

10 steps · about 7 h 20 min · Intermediate

  1. Large Language Models, Explained Briefly (video, 3Blue1Brown, 8 min)

    A short animated overview of next-word prediction, which is the loop that Spaceship lets you steer.

  2. The Illustrated Transformer (article, Jay Alammar, 35 min)

    This gives you the picture of attention and the transformer architecture before you look at generation details.

  3. The Illustrated GPT-2 (Visualizing Transformer Language Models) (article, Jay Alammar, 35 min)

    It shows how a decoder-only model produces one token at a time, which is the setting realtime steering works in.

  4. How do temperature, top-k, and top-p sampling differ? (article, Sebastian Raschka, 10 min)

    Sampling settings decide which token comes next, so you need them before you can reason about steering output.

  5. What is a KV cache, and why does it make LLM inference faster? (article, Sebastian Raschka, 12 min)

    This explains why each new token is cheap to compute, which is what makes interactive frame-rate generation possible.

  6. Speculative Decoding for 2x Faster Whisper Inference (article, Hugging Face, 20 min)

    It introduces the draft-and-verify trick for faster generation with working code.

  7. Faster Text Generation with Self-Speculative Decoding (article, Hugging Face, 20 min)

    It extends speculative decoding with early exit, which shows how far the speed-up idea can go.

  8. Spaceship IDE simulator (interactive, Chuanqi Sun, 15 min)

    You can try steering text generation like a pilot, using the simulator without any API keys.

  9. chuanqisun/spaceship: Steer ideas like a space pilot! (repo, Chuanqi Sun (GitHub), 45 min)

    Reading the source of the demo ties the concepts together and shows how a realtime LLM interface is built.

  10. Neural Networks: Zero To Hero (guide, Andrej Karpathy, 4 h)

    Build a small GPT yourself in code to go deeper once you understand the pieces.