1U · HERMESqwen3:8b · 8 GB VRAM · 24 tools In production

Slot 15 of 25 in the rack · In production

1U unit · Personal project in production

Hermes · local AI assistant

117/120 on the test suite, 23/23 on real questions

A chat assistant that knows the state of my servers, my calendar and my documentation. The model runs on an 8 GB GPU in my PC: nothing leaves my network. Because the model is small, accuracy isn't asked of the model: it is built around it.

The rule: Whatever can be computed, is computed; the model is only asked what needs understanding. Every rule left in its hands (dates, weekdays, sums) failed sooner or later, and stopped failing once moved into code.
Where it runs: The bot, the connectors and the indexes live in an always-on container on the server. The language model, voice and embeddings run on my PC's GPU: if it's off, questions wait in a queue.
What it can't do: By my choice, it doesn't start or stop servers, write notes, send email or see official documents. A test case checks it on every run.

4.2×faster with the whole model on the GPU
17.5 → 73tokens per second, freeing desktop VRAM
4.7 → 0.7 swriting without reasoning once the data is there
24tools: status, calendar, email and memory
0.09 mswhat the output guards cost
~110tests without the model, plus 48 cases × 3 seeds

Its code, in the open Public version of the repo: no IPs or secrets.

A chat turn from the inside

Every question goes through several phases before it reaches you. Pick a flow to see when the model thinks, when the code decides and what gets checked before sending.

Flows Pick one and follow the packet.

One question, phase by phase
A search that comes back empty
The PC is off
A voice note
Searching the three memories
An email with a booking
A watcher finds something

One question, phase by phase

Me, on Element — I ask from my phone: «Are there any alerts on the server?»
Matrix (assistant room) — The message reaches my Matrix server, in the assistant's room.
Hermes (the bot) — Hermes picks it up. The first thing isn't calling the model: it's preparing what the model needs to know.
Phase 1 · Context set by code — Phase 1: the code adds today's date, a table of the next 7 days and the trimmed history. Without it, the model thinks it's the year it was trained.
Phase 2 · Hints from the question — Phase 2: the question doesn't match any special pattern, so no hint is added. Less text, better choices.
Phase 3 · Reason only where it helps — Phase 3: no tool has run yet, so the model THINKS: it has to choose one and its arguments.
Language model (Ollama) — The model, on my PC's GPU, decides to call the server health tool.
Tools — The code checks the arguments against the schema and runs the tool. The model never touches anything directly.
Server status — The active alerts come back, each value tagged with its age.
Phase 3 · Reason only where it helps — Phase 3 again: the result has data and no warnings, so now the model does NOT think. It only has to write.
Language model (Ollama) — It writes the answer in 0.7 s instead of 4.7 s, with the same quality.
Phase 4 · Output guards — Phase 4: the guards compare the answer with the sources. No made-up IPs or amounts: go ahead.
Formatting — Markdown becomes HTML for Element, with a plain-text copy for the notification.
Matrix (assistant room) — It's sent to the room and indexed in conversation memory.
Me, on Element — I get the answer, with the data and how old it is.

A search that comes back empty

Hermes (the bot) — A two-step question: «How much was the ticket for what I have this week?»
Phase 3 · Reason only where it helps — A tool has to be chosen: the model thinks.
Language model (Ollama) — It chooses to search the memories for «ticket».
Tools — The code runs the search.
Qdrant (vector database) — Qdrant finds nothing useful with those words.
Phase 3 · Reason only where it helps — Empty result: reasoning switches back ON. Without this, the model gave up with «I found nothing» while the answer was one call away.
Language model (Ollama) — Even thinking, an 8B model failed this case 3 times out of 3.
Code relay — So the code takes over: it knows «what I have this week» is an event.
Calendar — It checks the calendar, filtered with the question itself, and takes the event's exact title.
Email (read-only) — It searches the email for that title and finds the confirmation with the amount.
Language model (Ollama) — The model gets the data already gathered and just has to read it.
Phase 4 · Output guards — The guards check the amount in the answer appears in the email (or is a sum of amounts that do).
Matrix (assistant room) — Right answer. Filtering the calendar with the question cut the time from 60-120 s to 31-41 s.

The PC is off

Me, on Element — I ask something at three in the morning.
Matrix (assistant room) — Matrix gets it fine: the server is always on.
Hermes (the bot) — Hermes checks whether the model is available. It isn't: my PC is off.
Persistent queue — The question goes into a persistent queue on disk. It isn't lost or answered badly.
The 8 GB GPU — In the morning I turn the PC on and the GPU is back.
Language model (Ollama) — Hermes sees the model responding and drains the queue in order.
Hermes (the bot) — Each question runs the full turn, with the current date, not the one when it was asked.
Matrix (assistant room) — The answer reaches the room, marked as a queued reply.
Me, on Element — I get everything I asked during the night.

A voice note

Me, on Element — I send a voice note from my phone.
Matrix (assistant room) — It reaches Matrix as an audio file.
Hermes (the bot) — Hermes sees it's audio and sends it to be transcribed.
Voice (Whisper + speech) — Whisper turns it into text on the PC's GPU.
Hermes (the bot) — With the text, the turn continues as if I'd typed it.
Language model (Ollama) — The model answers with its usual phases.
Hermes (the bot) — Hermes prepares the answer as text.
Voice (Whisper + speech) — The local speech synthesiser turns it into audio.
Matrix (assistant room) — It's sent to the room as a voice message, with the text alongside.
Me, on Element — I listen on my phone. The transcript also goes into conversation memory.

Searching the three memories

Tools — A search tool gets «what happened to the disk after the power cut?».
Embeddings — The question becomes a vector.
Qdrant (vector database) — Qdrant searches by meaning, not exact words.
My documentation — In the documentation, the blackout postmortem shows up.
Conversation memory — In the conversations, the night we talked about it shows up.
Qdrant (vector database) — The code merges the results from the three memories and labels where each comes from. The model used to mix up the three searches and once said it had no semantic database.
Language model (Ollama) — The model gets the fragments with their source and summarises them.
Formatting — The answer comes with a link to the note.

An email with a booking

Email (read-only) — A booking confirmation email arrives.
Embeddings — It's indexed straight away, including the text of the attached PDF.
Qdrant (vector database) — It stays in the vector database for good.
Hermes (the bot) — The email watcher hands it to Hermes without anyone asking.
Language model (Ollama) — The model answers two closed questions, with no tools: is it important? is it a booking with a date and time? It interprets the text, but decides nothing.
Code relay — The code decides: it checks whether it overlaps with something that already exists. That check stopped 4 real duplicates.
Calendar — No overlap: the event is created with the place in its own field and a tidy description. An instruction written inside an email can't create or delete anything.
Matrix (assistant room) — A notice in the events room, with a 🗑️ to undo it.
Me, on Element — I find out without opening the email.

A watcher finds something

The seven watchers — One of the seven watchers checks its topic; certificates, say.
Server status — It checks the real state: expiry dates of certificates and domains.
The seven watchers — It finds one expiring soon. The dates are worked out by code: the model once mixed up a certificate with a domain registration.
Formatting — It prepares a structured notice: what's happening, when and what to do.
Matrix (assistant room) — It posts it in its own room. If everything is fine, it says nothing.
Me, on Element — I only hear about it when there's something to look at.

Language model (Ollama)

An 8-billion-parameter model quantised to 4 bits: about 5 GB. It only loads when asked and unloads after 5 minutes, leaving the GPU free for the rest of the PC.
- LLM locales (Ollama, Qdrant, RAG)
Voice (Whisper + speech)

Whisper turns voice notes into text and a local speech synthesiser answers out loud. Also on the PC's GPU.
- LLM locales (Ollama, Qdrant, RAG)
The 8 GB GPU

The limit behind the whole design. If the model doesn't fit, part of it runs on the CPU and speed drops from 73 to 17 tokens per second. The simulator below shows it with real measurements.
Me, on Element

I ask by text or voice note from the chat, wherever I am.
Email (read-only)

Indexed the moment it arrives (IMAP IDLE), including the text of attached PDFs. Primary inbox only: no promotions or social.
Calendar

The only thing Hermes writes to: it creates events from booking emails and leaves a 🗑️ to undo them with a reaction.
Matrix (assistant room)

My own messaging server. Hermes has its own user and verifies its device on its own: no «unverified source» warnings.
Hermes (the bot)

The Python program that runs the turn: it gets the message, prepares the context, calls the model, runs tools and checks the answer before sending it.
- Python
Persistent queue

If the GPU PC is off, the question isn't lost: it is saved to disk and answered on its own as soon as the PC is back.
- Python
Formatting

Turns Markdown into HTML for Element and keeps a plain-text version for notifications. Anything from outside (subjects, titles) is escaped: it's data, not markup.
- Python
Phase 1 · Context set by code

Instructions, today's date with a table of the next 7 days, and history trimmed to question and answer. The model doesn't know what day it is: unless told, it reasons with its training date.
- Python
Phase 2 · Hints from the question

If the question matches a pattern («what happened with…», «how do I get into…»), one sentence is added to that turn only. Never to the global prompt: the more general text, the worse an 8B model chooses.
- Python
Phase 3 · Reason only where it helps

Thinking makes a call six times slower. It's on to pick a tool and work out arguments, and when a result comes back empty or with a warning; it's off to write with the data in front of it.
- LLM locales (Ollama, Qdrant, RAG)
- Python
Phase 4 · Output guards

Before sending, the code compares the answer with the turn's sources: an IP or an amount that appears in none of them goes back to the model to fix. If it insists, it's sent with a warning.
- Python
Tools

Server status, calendar, email and three memories. The code validates arguments before running («7» becomes 7) and, if the model names a tool without calling it, asks once that it call it.
- Python
Server status

Monitoring and security snapshots, container inventory, ZFS and disks. Every value carries its age: if it's over 30 minutes old, the model knows.
- Python
The seven watchers

Backups, certificates, storage, updates, unusual usage, failed logins and network. Each named after Greek mythology, each with its own room: they only speak when they find something odd.
- Python
Code relay

When a question is really two («how much was the ticket for what I have on Thursday?»), the model failed 3 out of 3. Now the code takes the second step: it checks the calendar, takes the title and searches the email for it.
- Python
Embeddings

Turn each text into a vector of numbers so search works by meaning, not just by words.
- LLM locales (Ollama, Qdrant, RAG)
Qdrant (vector database)

Stores the vectors of my notes, emails, the documents I send it and our conversations. The code searches all three memories at once so the model doesn't have to guess which.
- LLM locales (Ollama, Qdrant, RAG)
My documentation

Homelab procedures, incidents and postmortems. Hermes reads them but never writes to them.
Conversation memory

Every conversation, including transcribed voice notes, is indexed: «what did I do last week with…?» has an answer.
- LLM locales (Ollama, Qdrant, RAG)

What I use from my skill tree

Pick a skill: the parts where I use it light up on the map and I tell you what for.

Python: The whole bot: the turn loop, argument validation, the guards, the relay for two-step questions, the watchers and about 110 tests with no model or network.
LLM locales (Ollama, Qdrant, RAG): Ollama with qwen3:8b for chat, an embedding model to search by meaning, Qdrant as the vector database and Whisper for voice. All local, tuned to fit in 8 GB of VRAM.
Linux (Debian, Alpine, Arch): systemd services with minimal permissions (read-only paths, no privileges) in a dedicated container; the GPU in a Linux PC.
Ansible: The role that deploys the bot, its services and its configuration; secrets are encrypted with Ansible Vault.

Does the model fit on the GPU?

With 8 GB every megabyte counts: the desktop, the browser or Steam take GPU memory, and whatever part of the model doesn't fit runs on the much slower CPU. Change the setup and see what happens, using real measurements.

Turn on JavaScript to try the simulator. In short: with the desktop open only 78 % of the model fitted (20 tokens/s); with the KV cache in q8_0 and the browser and Steam closed, 100 % (73 tokens/s).

Measured on 22-09-2026 with nvidia-smi and the Ollama API. Speed is interpolated between the measured points (71 %, 78 %, 84 % and 100 % of the model on the GPU); the tokens for writing without thinking are an estimate.

Incident log

Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

#1

«The year 2026 is inconsistent»

September 2026 · calendar

Asked to create an event «next Thursday», Hermes said the date made no sense and that 2026 was «inconsistent». Another day it said «Tuesday the 28th» when the 28th was a Monday.
What do you think was going on?
Show the fix

Cause
A language model has no clock. Without a date in its context it reasons with the date of its training data, and working out weekdays isn't its strength either.

Fix
On every turn the code adds the real date and time and a table of the next 7 days, already worked out. The weekday of each event is computed by code too.

Lesson
Dates aren't asked of the model: they're handed to it ready-made.
#2

After one tool, no more

September 2026 · conversation

After a question that used a tool, the following questions in the same conversation used none, even when asked by name. Strengthening the instructions scored 0 out of 4.
What do you think was going on?
Show the fix

Cause
The history kept the intermediate tool results, JSON with many fields. For the 8B model those blocks changed its behaviour in later turns.

Fix
The history of past turns is trimmed to just the question and final answer, without the intermediate steps. Checked: 3 out of 3 on the real conversation that had failed.

Lesson
When more instructions don't fix something, the cause is usually structural: change what the model receives, not what you ask of it.
#3

141 seconds for one question

22-09-2026 · performance

In the afternoon a simple question took 141 s, and a test case that passed 5 out of 5 in the morning failed 8 out of 8 with the same code.
What do you think was going on?
Show the fix

Cause
With a game and the browser open, only 66 % of the model fitted on the GPU; the rest ran on the CPU at 10 tokens per second. On top of that, splitting changes the maths slightly and borderline cases flip.

Fix
KV cache quantised to q8_0, flash attention, one request and one loaded model at a time. With the browser closed the model fits at 100 %: from 17.5 to 73 tokens per second, and the suite went from 106 to 114 out of 120.

Lesson
Before blaming the model, check where it runs. Every run now records the Ollama version and how much of the model was on the GPU.
#4

A made-up public IP

23-09-2026 · guards

I asked for my home public IP and it answered with a perfectly formatted address… that wasn't mine. The right one was in the result of the tool it had just used.
What do you think was going on?
Show the fix

Cause
Small models sometimes fill in with something that «sounds right». A well-formatted IP is exactly that: there's no way to spot it by eye.

Fix
An output guard looks for every IP and amount in the answer in the turn's sources. If it isn't there (or isn't a sum of amounts that are), it asks for a fix; if the model insists, it's sent with a warning. It costs 0.09 ms.

Lesson
Guards must not fire on good data: a warning that fires too often teaches you to ignore it. That's why legitimate sums are accepted, with no limit on items.
#5

An empty day that wasn't

25-09-2026 · daily digest

The morning digest said I had nothing on. I did.
What do you think was going on?
Show the fix

Cause
The calendar authorisation was revoked overnight. The query failed, and the digest treated the failure as «no events».

Fix
Re-authorised, and a failing source is now reported as a failure: «I couldn't read the calendar». If the model writes «I found nothing» after an error, the code adds the warning.

Lesson
«There's nothing» and «I couldn't check» aren't the same: the first says the data doesn't exist, the second says the system is broken.

Told in full in an article:

qwen3:8b on an 8 GB GPU: from 17 to 73 tokens per second without changing model · 22 September 2026
Reason only where it helps: a local 8B model, from 4.7 to 0.7 seconds when writing · 22 September 2026

How it was built

22-09-2026

Phase 0 · Measure better
- One log line per model call: input and output tokens, load time, speed and how much of the model was on the GPU.
- A «narration» detector: when the model says «use tool X» instead of calling it.
- The first real measurement exposed the problem: 141 s for one question, 954 reasoning tokens just to write.
22-09-2026

Phase 1 · Make the model fit in 8 GB
- Flash attention and a q8_0 KV cache: the context takes half the space.
- One request and one loaded model at a time: each parallel request adds its own cache.
- Finding the bottleneck wasn't the model but the desktop: the browser was taking 805 MiB of VRAM.
- Result: 4.2× faster and 8 fewer failures on the suite.
22-09-2026

Phase 2 · Less text per turn (dropped)
- Merging tools to send less text: from 24 to 16.
- The prompt shrank, but so did the score: from 118 to 112 out of 120.
- Lesson: for an 8B model a tool's name weighs more than its description, and hiding capabilities behind a parameter makes them invisible.
22-23-09-2026

Phase 3 · Robustness through code
- Argument validation against the schema and readable errors the model can fix.
- Adaptive reasoning: think to choose, not to write (from 4.7 to 0.7 s).
- Output guards for IPs and amounts, a code relay for two-step questions and warnings when a source is down.
Pending

Phases 4 and 5 · Model and where it runs
- Try the same family with more bits (Q5_K_M) only if it fits with the same context.
- Move the model to the server once it has its own GPU: 1U high and 70 W from the slot. The candidate is a passive 16 GB card.

Decisions (and why)

Where the model runs
Locally, on my GPU
Nothing leaves my network: the assistant reads my email and calendar. I tried a 14B model and it didn't fit in 8 GB: same failure and 4 to 8 times slower. The server CPU has no AVX2: it would be minutes per answer.
How to get accuracy from a small model
Move rules from the prompt into code
Adding text to the prompt made the suite worse (96 vs 105 out of 111) and examples made it copy the shape of a sentence, not its meaning. What worked was having the code compute dates, weekdays, expiries, sums and types, and leaving the model to understand and write.
How a change gets accepted
Measured with 3 seeds, case by case
Each change is measured on its own and only shipped with no regressions. Besides the suite there are tests against production and real questions, because the suite uses synthetic data and can't see a broken tool.
What the model can touch
Read almost everything, write almost nothing
The model interprets and the code decides whether anything gets written. That's why an instruction inside an email can't create or delete anything, and automatic retries never apply to write tools.

What I learned

With a small model, accuracy doesn't come from the model: it comes from what the code does around it.
The more general text an 8B model receives, the worse it chooses. Isolating a task fixes it.
GPU memory is part of the design: an open browser changed the speed fourfold and the suite's result.
Measure with several seeds and with real questions: half the important failures never showed up in the suite.

What's next

A GPU of its own for the server, so Hermes doesn't depend on my PC being on.
Review the suite's criteria: some cases measured wording rather than substance.
Fix questions about past incidents, which still sometimes go to the live-status tools.

Shall we talk?

I can walk through any of these decisions in detail in an interview.

Download CV Get in touch ← Back to the rack

Rebuilt from my Obsidian documentation. No addresses, machine names or personal data: the example answers are made up.

Hermes · local AI assistant

A chat turn from the inside

Language model (Ollama)

Voice (Whisper + speech)

The 8 GB GPU

Me, on Element

Email (read-only)

Calendar

Matrix (assistant room)

Hermes (the bot)

Persistent queue

Formatting

Phase 1 · Context set by code

Phase 2 · Hints from the question

Phase 3 · Reason only where it helps

Phase 4 · Output guards

Tools

Server status

The seven watchers

Code relay

Embeddings

Qdrant (vector database)

My documentation

Conversation memory

What I use from my skill tree

Does the model fit on the GPU?

Incident log

How it was built

Phase 0 · Measure better

Phase 1 · Make the model fit in 8 GB

Phase 2 · Less text per turn (dropped)

Phase 3 · Robustness through code

Phases 4 and 5 · Model and where it runs

Decisions (and why)

Locally, on my GPU

Move rules from the prompt into code

Measured with 3 seeds, case by case

Read almost everything, write almost nothing

What I learned

What's next

Shall we talk?