Steering LLMs in realtime: how text generation works
10 steps · about 7 h 20 min · Intermediate
- Large Language Models, Explained Briefly (video, 3Blue1Brown, 8 min)
A short animated overview of next-word prediction, which is the loop that Spaceship lets you steer.
- The Illustrated Transformer (article, Jay Alammar, 35 min)
This gives you the picture of attention and the transformer architecture before you look at generation details.
- The Illustrated GPT-2 (Visualizing Transformer Language Models) (article, Jay Alammar, 35 min)
It shows how a decoder-only model produces one token at a time, which is the setting realtime steering works in.
- How do temperature, top-k, and top-p sampling differ? (article, Sebastian Raschka, 10 min)
Sampling settings decide which token comes next, so you need them before you can reason about steering output.
- What is a KV cache, and why does it make LLM inference faster? (article, Sebastian Raschka, 12 min)
This explains why each new token is cheap to compute, which is what makes interactive frame-rate generation possible.
- Speculative Decoding for 2x Faster Whisper Inference (article, Hugging Face, 20 min)
It introduces the draft-and-verify trick for faster generation with working code.
- Faster Text Generation with Self-Speculative Decoding (article, Hugging Face, 20 min)
It extends speculative decoding with early exit, which shows how far the speed-up idea can go.
- Spaceship IDE simulator (interactive, Chuanqi Sun, 15 min)
You can try steering text generation like a pilot, using the simulator without any API keys.
- chuanqisun/spaceship: Steer ideas like a space pilot! (repo, Chuanqi Sun (GitHub), 45 min)
Reading the source of the demo ties the concepts together and shows how a realtime LLM interface is built.
- Neural Networks: Zero To Hero (guide, Andrej Karpathy, 4 h)
Build a small GPT yourself in code to go deeper once you understand the pieces.