I decided to set up a locally hosted AI server over the summer to better understand how LLMs work, how they are deployed, and the capabilities of smaller LLMs for personal use. I decided that the supplement to my spend on Claude, ChatGPT, etc. would be a cost savings measure, but it was not quite so simple.
In the end, I had an AI workflow across multiple machines on my local network, a desktop server doubling as a space heater, and a fast and capable private AI server.
The hardware
Three machines total, each with a specific role.
| Machine | Hardware | Role |
|---|---|---|
| AI Server | Dual AMD AI PRO R9700 (gfx1201) | vLLM inference, llama-swap model routing — the workhorse |
| Tower | RTX 3080 Ti 12GB, Ryzen 9 8945HX | Local Ollama (multi-model), ComfyUI image generation |
| Unraid | — | Docker host: Agent Zero, RAG pipeline, monitoring, DNS |
The AI server does the heavy lifting. Two R9700s running a 27B model in a 4-bit quantization (MXFP4), with tensor-parallel across both cards. The tower handles smaller models on a single consumer GPU, including embedding models. Unraid doubles as my media and LAN application server, so it was a natural landing spot for my UI containers.
The software stack
What runs where, and where the code lives:
| Component | What it does | Repo |
|---|---|---|
| vLLM | The inference engine on the AMD box | vllm-project/vllm |
| llama-swap | Routes between resident models on the AI server | mostlygeek/llama-swap |
| Ollama | Multi-model local LLMs on the tower | ollama/ollama |
| ComfyUI | Local image generation (Z-Image-Turbo) | comfy-org/comfyui |
| vLLM (Radiance fork) | Patched vLLM image tuned for R9700 RDNA4 performance | magiccodingman/vllm-radiance |
The models resident on the AI server right now: Qwen3.8 in both MXFP4 and NVFP4 quantizations, Gemma4-31B and Muse-Glimmer-30B in FP8, plus a 35B MoE (A3B) for parallel/background work. llama-swap swaps them in and out of VRAM as needed.
The performance
The production baseline is Qwen3.8 27B in MXFP4, tensor-parallel across both R9700s:
| Metric | Value | What it means |
|---|---|---|
| Decode speed | 185.8 tok/s | Sustained token generation |
| TTFT | 84 ms | Time to first token (prompt processing) |
| Prompt processing | 3,841–4,215 tok/s | How fast a long context is ingested |
| GPU power (load) | 146 / 151 W | Per-card, under active decode |
| GPU power (idle) | 47 / 50 W | Per-card, at rest |
A few experiments that shaped the config (each tested, each with a verdict):
| Experiment | Change | Result | Verdict |
|---|---|---|---|
| E01 | Disable fused attention | −22% decode | Fail — keep attention on |
| E05 | Single GPU (TP1) | −57% decode | Fail — TP2 wins everywhere |
| M4 | MTP-8 spec decoding | −47% decode | Fail — dflash-7 stays |
| FP8 | FP8 vs MXFP4 | −61% decode | MXFP4 wins decisively |
| M7 | Disable prefix caching | −7% decode | Fail — prefix caching stays on |
The pattern: most "obvious" optimizations made things worse on this hardware. The hand-written attention kernel, 4-bit quantization, and prefix caching are the three that actually moved the needle.
The cost
This is the part that actually matters. I have measured per-GPU power, and I'll be honest about where I'm estimating.
Measured (from the experiment tracker): - Under load: ~146W + ~151W = ~297W across both GPUs - At idle: ~47W + ~50W = ~97W across both GPUs
Estimated (system total, clearly labeled): adding the CPU, RAM, drives, fans, and ~90% PSU efficiency, I estimate the whole box draws ~400W under load and ~175W at idle. That's the number I use below — if your system draw differs, scale proportionally.
Cost per million tokens (decode at 185.8 tok/s)
| Electricity rate | Per 1M tokens | |---|---|---| | $0.10 / kWh | ~$0.06 | | $0.16 / kWh | ~$0.10 | | $0.30 / kWh | ~$0.18 |
How: 1,000 tokens takes 5.38s. At 0.4 kW that's 0.0006 kWh per 1,000 tokens. × rate, × 1,000 for a million.
For comparison, a 27B-class model on a commercial API typically runs $0.50–$3.00 per million tokens. So local inference is roughly 5–30× cheaper at scale — and the marginal cost of a token is basically the electricity it takes to generate it. Larger (and more capable) models can run as high as $50 per million tokens.
The idle cost (the box is always on)
| Electricity rate | Idle draw | Monthly (730 hrs) |
|---|---|---|
| $0.10 / kWh | ~175W | ~$13 |
| $0.16 / kWh | ~175W | ~$20 |
| $0.30 / kWh | ~175W | ~$38 |
The idle cost is the real price of convenience — it's what you pay to keep the model warm. If you only run the box when you need it, the idle cost drops to near zero and the per-token cost is all you pay.
What it's actually used for
The local AI stack I am running has been very useful as a personal assistant of sorts. The model has streamlined management of my LAN assets, checked for security vulnerabilities on my own network, and even published this website. The capabilities are in line with the permissions it is given, so I highly recommend sandboxing and following a standard dev -> stage -> prod process.
The whole point: a 27B model running at ~200 tok/s, on hardware I own, for the cost of the electricity it uses.
What I'd do differently
- Start with the 4-bit quantization. FP8 looked like the obvious choice and was 61% slower. MXFP4 won decisively on speed — though I was careful not to assume that meant a quality win, since the two runs differed in more than just the quantization. A clean same-spec quality comparison is still an open experiment, which is exactly why the next lesson matters.
- Trust the benchmarks over the claims. Most "optimizations" in the vLLM/ROCm world made things worse on this hardware.
- Tensor-parallel from day one. Single-GPU was 57% slower. The two cards working together is the whole win.
- Track power from the start. The cost story only works because I logged wattage on every experiment. That data is what made the electricity math possible.