Measured improvement Published on 22 September 2026 · 1U · HERMES

qwen3:8b on an 8 GB GPU: from 17 to 73 tokens per second without changing model

My local assistant took 141 seconds for one question and failed cases that passed in the morning. It wasn't the model or the prompt: only 66 % of the model fitted on the GPU, and the rest ran on the CPU.

LLM locales (Ollama, Qdrant, RAG)
Linux (Debian, Alpine, Arch)

Where it started

Hermes, my assistant, runs qwen3:8b quantised to 4 bits with Ollama on my PC's GPU, an 8 GB RTX. In the afternoon a simple question took 141 seconds and a test case that passed 5 out of 5 in the morning failed 8 out of 8.

The first metric I added was one log line per model call: input and output tokens, load time, speed and the GPU/CPU split. With that the problem jumped out:

prompt=5282tok output=371tok load=2.8s total=44.9s speed=10.6tok/s split=6.12GiB 66%GPU

Who was eating the VRAM

With a game and the browser open, only 66 % of the model fitted on the GPU. Loading took 2.8 s: the least of it. The cost was generating at 10 tokens per second with a third of the model on the CPU.

nvidia-smi (the process table, not --query-compute-apps, which leaves out graphics processes) showed who held the memory: Firefox 805 MiB, Steam 228, the desktop about 300, VS Code 145, Element 88. The bottleneck wasn't the model: it was the desktop.

The changes

Four Ollama variables in its systemd service and closing what wasn't needed. The GPU/CPU split is decided when the model loads, so you have to force a reload after freeing memory.

OLLAMA_FLASH_ATTENTION=1      # needed to quantise the KV cache
OLLAMA_KV_CACHE_TYPE=q8_0     # the context cache takes half the space
OLLAMA_NUM_PARALLEL=1         # each parallel request adds its own cache
OLLAMA_MAX_LOADED_MODELS=1    # don't keep another model in VRAM

Result

Normal desktop, f16 cache: 78 % on GPU, about 20 tokens/s.
Flash attention and q8_0 cache: 84 %, 33 tokens/s.
Plus browser and Steam closed: 100 % on GPU and 73 tokens/s, 4.2 times faster than at the start.
The suite of 40 cases × 3 seeds went from 106 to 114 out of 120, with the median from 12.2 s to 4.7 s.

Why it got more right too

The quantised cache didn't hurt quality: it improved it. Not because it makes the model smarter, but because without the CPU split the maths stop varying and borderline cases stop flipping.