Run your own LLM server cheaply, on your laptop or a small server

26 steps · about 5 h 54 min · Beginner

  1. See the whole picture: hosting a model with Ollama and llama.cpp (video, YouTube, 15 min)

    A quick tour of both tools first makes it clear that most local runners share one engine underneath.

  2. Install Ollama and chat with your first model (doc, Ollama docs, 10 min)

    The fastest path from nothing to a working local model, on Mac, Windows or Linux.

  3. Call your local model with the OpenAI API format (doc, Ollama docs, 10 min)

    Existing apps and SDKs work unchanged once you point their base URL at your own server.

  4. Expose Ollama on your network and keep models loaded (doc, Ollama docs, 10 min)

    Binding address, keep-alive and concurrency settings are what turn a desktop toy into a small server.

  5. Package a model with your own prompt and settings (doc, Ollama docs, 15 min)

    A Modelfile pins the system prompt, context size and temperature so every client gets the same behaviour.

  6. Host a small model on CPU only (video, YouTube, 12 min)

    Shows that a 1B model answers usefully on a machine with no GPU at all, which is the cheapest setup.

  7. Understand quantization: why a 4-bit model fits in your RAM (article, Maarten Grootendorst, 30 min)

    Quantization is the trick behind cheap hosting; knowing what Q4 and Q8 trade away helps you pick model files.

  8. Learn the GGUF file format and its quantization types (doc, Hugging Face docs, 10 min)

    GGUF is the format llama.cpp, Ollama and LM Studio load; the type table tells you size against quality.

  9. Run any GGUF model from Hugging Face in Ollama (doc, Hugging Face docs, 5 min)

    Opens up thousands of community models beyond the Ollama library with one command.

  10. Meet llama.cpp, the engine under most local runners (repo, GitHub, 15 min)

    Running the engine directly gives you every knob: threads, context, GPU layers and the exact model file.

  11. Build llama.cpp for your CPU or GPU (doc, llama.cpp docs, 20 min)

    A build matched to your hardware (Metal, CUDA, Vulkan or plain AVX2) is often the biggest free speed-up.

  12. Serve a model with llama-server (doc, llama.cpp docs, 25 min)

    llama-server is a lightweight OpenAI-compatible HTTP server with parallel slots, JSON output and a web UI.

  13. Deploy llama-server for several users at once (video, YouTube, 15 min)

    Walks through slots and batching, the settings that decide how many people one box can serve.

  14. Run llama.cpp in Docker on a server (doc, llama.cpp docs, 10 min)

    Ready-made images make a reproducible server install that you can restart and move between machines.

  15. Use the llm command-line tool for prompts and scripts (doc, llm docs, 15 min)

    Simon Willison's llm CLI logs every prompt to SQLite and pipes model output into shell scripts.

  16. Point the llm CLI at your local Ollama models (repo, GitHub, 5 min)

    One plugin makes every model on your own server available to llm, with no cloud key.

  17. Connect llm to any OpenAI-compatible endpoint (doc, llm docs, 10 min)

    The same setup works for llama-server or vLLM, so scripts survive a change of engine.

  18. Try LM Studio's local server if you prefer a desktop app (doc, LM Studio docs, 10 min)

    A point-and-click way to download models and run an OpenAI-style server, handy for comparing models.

  19. Ship a model as one file with llamafile (repo, GitHub, 10 min)

    A single executable that runs on most operating systems is the simplest way to hand a model to someone else.

  20. Start vLLM when you have a GPU (doc, vLLM docs, 20 min)

    vLLM's paged attention and continuous batching give the highest throughput when many users share one GPU.

  21. Run vLLM's OpenAI-compatible server (doc, vLLM docs, 20 min)

    The vllm serve command, its API key flag and its sampling options are what a production endpoint needs.

  22. Watch a vLLM server come up in one command (video, YouTube, 12 min)

    Seeing the server start, load weights and answer requests makes the docs easier to follow.

  23. Check what vLLM needs on CPU before you try it (doc, vLLM docs, 10 min)

    vLLM's CPU backend needs modern vector instructions; on an older or budget CPU, llama.cpp is the better pick.

  24. Deploy vLLM with Docker (doc, vLLM docs, 10 min)

    The official image is the usual way to run vLLM on a rented GPU server without fighting CUDA versions.

  25. Put a chat UI in front of your server (video, YouTube, 15 min)

    Pairing Ollama with Open WebUI in Docker gives your team a private chat site on your own box.

  26. Set up Open WebUI users and connections (doc, Open WebUI docs, 15 min)

    Accounts, roles and multiple backends let one cheap server serve a family or a small team.