Run your own LLM server cheaply, on your laptop or a small server
26 steps · about 5 h 54 min · Beginner
- See the whole picture: hosting a model with Ollama and llama.cpp (video, YouTube, 15 min)
A quick tour of both tools first makes it clear that most local runners share one engine underneath.
- Install Ollama and chat with your first model (doc, Ollama docs, 10 min)
The fastest path from nothing to a working local model, on Mac, Windows or Linux.
- Call your local model with the OpenAI API format (doc, Ollama docs, 10 min)
Existing apps and SDKs work unchanged once you point their base URL at your own server.
- Expose Ollama on your network and keep models loaded (doc, Ollama docs, 10 min)
Binding address, keep-alive and concurrency settings are what turn a desktop toy into a small server.
- Package a model with your own prompt and settings (doc, Ollama docs, 15 min)
A Modelfile pins the system prompt, context size and temperature so every client gets the same behaviour.
- Host a small model on CPU only (video, YouTube, 12 min)
Shows that a 1B model answers usefully on a machine with no GPU at all, which is the cheapest setup.
- Understand quantization: why a 4-bit model fits in your RAM (article, Maarten Grootendorst, 30 min)
Quantization is the trick behind cheap hosting; knowing what Q4 and Q8 trade away helps you pick model files.
- Learn the GGUF file format and its quantization types (doc, Hugging Face docs, 10 min)
GGUF is the format llama.cpp, Ollama and LM Studio load; the type table tells you size against quality.
- Run any GGUF model from Hugging Face in Ollama (doc, Hugging Face docs, 5 min)
Opens up thousands of community models beyond the Ollama library with one command.
- Meet llama.cpp, the engine under most local runners (repo, GitHub, 15 min)
Running the engine directly gives you every knob: threads, context, GPU layers and the exact model file.
- Build llama.cpp for your CPU or GPU (doc, llama.cpp docs, 20 min)
A build matched to your hardware (Metal, CUDA, Vulkan or plain AVX2) is often the biggest free speed-up.
- Serve a model with llama-server (doc, llama.cpp docs, 25 min)
llama-server is a lightweight OpenAI-compatible HTTP server with parallel slots, JSON output and a web UI.
- Deploy llama-server for several users at once (video, YouTube, 15 min)
Walks through slots and batching, the settings that decide how many people one box can serve.
- Run llama.cpp in Docker on a server (doc, llama.cpp docs, 10 min)
Ready-made images make a reproducible server install that you can restart and move between machines.
- Use the llm command-line tool for prompts and scripts (doc, llm docs, 15 min)
Simon Willison's llm CLI logs every prompt to SQLite and pipes model output into shell scripts.
- Point the llm CLI at your local Ollama models (repo, GitHub, 5 min)
One plugin makes every model on your own server available to llm, with no cloud key.
- Connect llm to any OpenAI-compatible endpoint (doc, llm docs, 10 min)
The same setup works for llama-server or vLLM, so scripts survive a change of engine.
- Try LM Studio's local server if you prefer a desktop app (doc, LM Studio docs, 10 min)
A point-and-click way to download models and run an OpenAI-style server, handy for comparing models.
- Ship a model as one file with llamafile (repo, GitHub, 10 min)
A single executable that runs on most operating systems is the simplest way to hand a model to someone else.
- Start vLLM when you have a GPU (doc, vLLM docs, 20 min)
vLLM's paged attention and continuous batching give the highest throughput when many users share one GPU.
- Run vLLM's OpenAI-compatible server (doc, vLLM docs, 20 min)
The vllm serve command, its API key flag and its sampling options are what a production endpoint needs.
- Watch a vLLM server come up in one command (video, YouTube, 12 min)
Seeing the server start, load weights and answer requests makes the docs easier to follow.
- Check what vLLM needs on CPU before you try it (doc, vLLM docs, 10 min)
vLLM's CPU backend needs modern vector instructions; on an older or budget CPU, llama.cpp is the better pick.
- Deploy vLLM with Docker (doc, vLLM docs, 10 min)
The official image is the usual way to run vLLM on a rented GPU server without fighting CUDA versions.
- Put a chat UI in front of your server (video, YouTube, 15 min)
Pairing Ollama with Open WebUI in Docker gives your team a private chat site on your own box.
- Set up Open WebUI users and connections (doc, Open WebUI docs, 15 min)
Accounts, roles and multiple backends let one cheap server serve a family or a small team.