Running a small language model on your own hardware

Written By: BrainSoft In AI & Research

You have a task that a language model would handle well, but the data cannot leave your network, or the API bill for a high-volume job adds up to more than a spare GPU. So you start wondering whether a small model on your own hardware is good enough. Usually it is, for narrow jobs like classification, extraction, summarising support tickets, or drafting replies from a fixed template.

This walks through setting up a small open-weight model on a single machine: picking the model and quantisation, installing a runtime, testing it from the command line, serving it over HTTP, and measuring whether it holds up. The whole thing takes an afternoon on a workstation with a decent GPU, or a bit longer on CPU only.

1. Check what hardware you actually have. A model that fits in VRAM runs an order of magnitude faster than one that spills to system memory. Run nvidia-smi on a GPU box or check your RAM on a laptop. As a rough rule, a 7-8B parameter model at 4-bit quantisation needs around 5-6 GB of VRAM; a 3B model needs roughly 2-3 GB. Leave headroom for the KV cache, which grows with context length.
2. Pick a small model and a quantised build. Start with something in the 3B to 8B range that has a permissive licence and is well supported by the runtime you plan to use. On Hugging Face, look for GGUF files (for llama.cpp and Ollama) or AWQ/GPTQ builds (for vLLM). Q4_K_M is the usual sweet spot for GGUF: noticeably smaller than the full model with little quality loss on most tasks.
3. Install a runtime. Ollama is the quickest path for a single machine and gives you a server for free. llama.cpp is leaner if you want to embed inference in your own process. vLLM is worth it only if you need concurrent throughput. For a first run, Ollama:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:7b-instruct-q4_K_M

4. Run a prompt and read the output critically. Do not stop at one friendly answer. Give it the messy inputs your real users produce, and check whether it follows format instructions.

ollama run qwen2.5:7b-instruct-q4_K_M \
  "Extract the order number and complaint from: 'Hi, order #4471 arrived cracked.' Reply as JSON."

5. Serve it over HTTP and call it from code. Ollama listens on port 11434 by default and speaks an OpenAI-compatible API, so your existing client code mostly works with a base URL change.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:7b-instruct-q4_K_M",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

6. Measure latency and quality on your own data. Time a batch of real requests and score the outputs against a hand-labelled set of a few dozen examples. Track tokens per second and failure cases. If the model is too slow, drop to a smaller size or a lower quantisation before you buy a bigger GPU.
7. Harden the setup before it touches production. Bind the server to localhost or a private interface, put authentication in front of it, cap concurrency, and set a request timeout. Model servers are not designed to face the internet. If this ends up as part of a larger system, our services cover the integration and deployment side.

Frequently asked questions

#AI & Research