Build your own small language model, from a tiny GPT to a fine-tuned chat model
33 steps · about 36 h 20 min · Beginner
- Know what an LLM really is before you build one (video, YouTube, 1 h)
A plain talk on what a language model is: a next-token predictor trained on text, then tuned to be an assistant.
- See how a transformer turns words into predictions (video, YouTube, 30 min)
Clear animations of embeddings, attention and the final probabilities make the code later much easier to follow.
- Understand attention step by step (video, YouTube, 25 min)
Attention is the one new idea in a GPT; this is the clearest picture of what the query, key and value matrices do.
- Read the illustrated transformer (article, Jay Alammar, 30 min)
A diagram-first walk through the whole architecture to keep open while you write your own.
- Build a tiny autograd engine to learn backpropagation (video, YouTube, 2 h 30 min)
Training is just gradients; writing micrograd by hand removes the magic before you lean on PyTorch.
- Build your first character-level language model (video, YouTube, 2 h)
A bigram model on a list of names teaches sampling, loss and training loops at a size you can reason about.
- Play with a tokenizer and watch text become numbers (interactive, Tiktokenizer, 10 min)
Seeing how real models split words, numbers and code shows why the tokenizer shapes everything a model can do.
- Build a byte-pair encoding (BPE) tokenizer (video, YouTube, 2 h 10 min)
You write the same kind of tokenizer GPT models use, and learn where many odd model behaviours come from.
- Check your tokenizer against minbpe (repo, GitHub, 30 min)
Small, readable BPE code with exercises, so you can compare your version and train one on your own text.
- Build a tiny GPT from scratch in code (video, YouTube, 2 h)
The core lesson: a working GPT in a few hundred lines of PyTorch, trained on Shakespeare on one GPU or a laptop.
- Train a small GPT with nanoGPT (repo, GitHub, 1 h)
The tidy version of that code with configs for a character model on a laptop CPU or a free GPU.
- Get a free GPU in Google Colab for training runs (interactive, Google Colab, 15 min)
When your laptop is too slow, a free Colab notebook with a GPU runtime is enough for small models.
- Pick a small dataset: TinyStories (reference, Hugging Face, 15 min)
Simple short stories that let a model of a few million parameters write fluent English; perfect for a first run.
- Learn why tiny models can still write well (article, arXiv, 30 min)
The TinyStories paper shows how a narrow, clean dataset lets very small models learn grammar and storytelling.
- Go deeper: build, pretrain and fine-tune a GPT step by step (repo, GitHub, 4 h)
Chapter-by-chapter notebooks cover data loading, attention, pretraining, loading real weights and instruction tuning.
- Code along a full LLM-from-the-ground-up workshop (video, YouTube, 3 h)
One sitting that ties tokenizing, the model, pretraining and fine-tuning together with the same code.
- Scale up: reproduce GPT-2 (124M) (video, YouTube, 4 h)
Shows what changes at real size: efficient training, mixed precision, learning-rate schedules and evaluation.
- See a complete small ChatGPT pipeline end to end (repo, GitHub, 45 min)
Tokenizer, pretraining, chat fine-tuning, evaluation and a web chat in one small codebase you can run on rented GPUs.
- Take the Hugging Face LLM course basics (tutorial, Hugging Face, 2 h)
Learn the Transformers library you will use to load, train and share open models instead of only your own code.
- Train a tokenizer for your own data with Hugging Face (tutorial, Hugging Face, 1 h)
The fast tokenizers library trains a BPE tokenizer on your text in seconds, ready for a real model.
- Start from a small open model instead of from zero (doc, Hugging Face, 10 min)
Fine-tuning a ready small model like SmolLM2 gets far better results than pretraining your own on a budget.
- Format chat data with chat templates (doc, Hugging Face docs, 15 min)
Training and chatting with the same template is what makes a fine-tuned model answer properly.
- Fine-tune a small model on your own examples (SFT) (tutorial, Hugging Face, 1 h 30 min)
Supervised fine-tuning teaches the model your task or style from a few thousand example conversations.
- Fine-tune cheaply with LoRA (doc, Hugging Face docs, 20 min)
LoRA trains a small add-on instead of every weight, so fine-tuning fits on one consumer or free GPU.
- Understand QLoRA: fine-tune on a 4-bit model (article, arXiv, 30 min)
Shows how quantizing the base model cuts memory again, which is why fine-tuning on a free GPU works at all.
- Run the training with TRL's SFTTrainer (doc, Hugging Face docs, 30 min)
The standard trainer for chat fine-tuning: datasets in, LoRA on, a trained adapter out.
- Or fine-tune faster with Unsloth's free notebooks (doc, Unsloth docs, 45 min)
Ready Colab notebooks that fine-tune popular small models with less memory and time, step by step.
- Practise the whole fine-tuning workflow with the smol course (repo, GitHub, 2 h)
Hands-on exercises on small models: instruction tuning, preference tuning, evaluation, all on modest hardware.
- Learn how to judge whether your model got better (repo, GitHub, 30 min)
Practical advice on benchmarks, human checks and model-as-judge so you measure instead of guess.
- Score your model on standard benchmarks (repo, GitHub, 30 min)
The common tool for running the same benchmark tests others report, before and after your training.
- Export your model to GGUF (doc, Unsloth docs, 15 min)
GGUF is the file format llama.cpp, Ollama and LM Studio run, so this makes your model portable.
- Run your own model in Ollama (doc, Ollama docs, 15 min)
Import the GGUF or adapter, then chat with it and call it through the same local API as any other model.
- Serve it to your apps with llama-server (doc, GitHub, 20 min)
An OpenAI-compatible endpoint for your model; the 'Run your own LLM server cheaply' list covers hosting it well.