TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · ZABBIXZabbix 7 LTS · PSK · 4 channels In production

Slot 16 of 25 in the rack · In production

1U unit · Observability

Zabbix · metrics and alerts

From 178 doubled alerts to only the serious ones

Every container and the server itself run an agent sending its metrics encrypted with its own key: CPU, memory, disks, RAID, ZFS, SMART health, Docker containers. Zabbix decides what's a problem, alerts on the right channel for the severity and alerts again when it's fixed.

How it's isolated
Unprivileged container, SSH stopped and masked, inbound dropped except from agents and the internal door. Its API only listens inwards.
What it can't do
Agents don't accept remote commands or have access to the Docker socket: read-only collectors leave the data as JSON.
Alerts
One alert per incident plus its recovery, no hourly repeats. By severity: Discord from average up, email and Element only for high.
  • 82 %of alerts came from one badly framed trigger
  • 1 PSKper agent, stored encrypted
  • 1 commandto register a new container
  • 90 daysof events and trends

Its code, in the open Public version of the repo: no IPs or secrets.

Inside the monitoring

Every machine sends its metrics; here it's decided what's a problem and who gets told. Pick a flow.

Flows Pick one and follow the packet.

Outside Zabbix container · Proxmox unprivileged, no SSH On every machine The server Alerts The fleet Me Discord Element Router Internal door Hermes Agent PSK Collectors No commands Zabbix Triggers History API Severity Email Discord+Element Recovery Registration Vault Firewall This site
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      A disk starts to fail
      1. Read-only collectors — A collector sees a SMART attribute getting worse.
      2. Zabbix agent — The agent sends it.
      3. Encryption with its own key — Encrypted with its key.
      4. Zabbix server — The server receives it.
      5. Problem rules — A trigger turns it into a problem.
      6. Routing by severity — By its severity, it goes to chat.
      7. Alerts room (Matrix) — To the alerts room, encrypted.
      8. Me — It reaches my phone.
      9. Recovery notice — When it's fixed, the recovery notice arrives.
      A new container joins the fleet
      1. Registration script — I run the registration script with the container's name.
      2. Zabbix API — Through the hypervisor, it creates the host in the API.
      3. Zabbix server — The server now knows it.
      4. Keys in the vault — The new key goes into the encrypted vault.
      5. Zabbix agent — Ansible installs the agent with that key.
      6. Server firewall — Its address is added to the server firewall.
      7. Zabbix server — And data starts arriving.
      How the noise was switched off
      1. Problem rules — Before touching anything, I measured a week of alerts: 178, each one twice.
      2. Routing by severity — Everything from «warning» severity, by email and Discord at the same time.
      3. Email — Email now only for high.
      4. Discord channels — Discord, from average up.
      5. Me — And I read it without stress.
      6. Problem rules — The trigger behind 82 % is disabled where it measured nothing real.
      From Zabbix to this site
      1. History — The history says which agents reported each hour.
      2. Hermes — Hermes reads it to answer.
      3. This site — And this site publishes availability as coverage: hours with data over expected hours.
      I open the panel
      1. Me — From home or the VPN.
      2. Internal door — Internal door with two-factor.
      3. Zabbix server — Then the Zabbix login.
      4. History — To look at the history.

      1. Me

        The one who gets the alerts and has to believe them.

      2. Discord channels

        One channel per severity and another for recoveries.

      3. Alerts room (Matrix)

        High severity only, end-to-end encrypted.

      4. Router

        Watched by ping and HTTPS, with no admin credentials.

        • MikroTik

      5. Internal door

        The panel, only from home or the VPN with two-factor.

      6. Hermes

        Reads a status snapshot to answer questions.

      7. Zabbix agent

        In active mode: the agent is the one sending data. Passive listening only on loopback, with CPU capped at 10 %.

        • Zabbix

      8. Encryption with its own key

        Each agent has its pre-shared key. Without it, the server won't accept its data.

        • Zabbix
        • Linux / SSH hardening

      9. Read-only collectors

        Scripts that check ZFS, RAID, SMART without waking sleeping disks, and Docker health, leaving a JSON the agent reads.

        • Zabbix

      10. No remote execution

        The agent's remote commands are forbidden: compromising the monitoring server gives no control over the others.

        • Linux / SSH hardening

      11. Zabbix server

        Receives, stores and evaluates. LTS release.

        • Zabbix

      12. Problem rules

        When a metric is a problem and when it stops being one. Almost all the noise lived here.

        • Zabbix

      13. History

        14 days of detail, 90 of trends and events.

        • Zabbix

      14. Zabbix API

        Only reachable from inside the container. Registering new hosts goes through the hypervisor.

        • Zabbix

      15. Routing by severity

        Each severity goes to its channel. Minor stuff doesn't reach the phone.

        • Zabbix

      16. Email

        High severity only, with verified TLS.

        • Zabbix

      17. Chat

        Discord from average up; Element only for high.

        • Zabbix

      18. Recovery notice

        Every problem also notifies when fixed. Without that you don't know if it's still going on.

        • Zabbix

      19. Registration script

        Adds a new container to Zabbix and Wazuh at once, with a new key, in a single command.

        • Ansible

      20. Keys in the vault

        The script writes the agent's key straight into the encrypted Ansible vault.

        • Ansible

      21. Server firewall

        Only accepts agents from known addresses. A new container has to be added here too.

        • Linux / SSH hardening

      22. This site

        The lab's LEDs and availability come from here, as aggregates.

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      Wazuh
      Shares the registration script: a new container joins both tools at once.
      Zabbix
      Server and active agents with PSK, collectors for ZFS, RAID, SMART and Docker, tuned triggers and alerts by severity with recovery.
      Linux / SSH hardening
      Unprivileged container with no SSH, inbound dropped, no remote execution and no Docker socket access.
      MikroTik
      The router is watched by ping and HTTPS, with no admin credentials.
      VLANs
      Agents on every segment report to the same server, which only accepts known addresses.
      WireGuard
      From the VPN you reach the panel as from home, behind the internal door.
      Ansible
      Installs the agents with their key from the vault; the registration script writes it there.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        Eight containers alerting at once

        05-10-2026 · noise

        Every few days, eight containers alerted at once about «load per CPU above 3 for 15 minutes». None of them was doing anything.

        What do you think was going on?

        Show the fix
        Cause
        In these containers, /proc/loadavg shows the server's load. During the monthly RAID check the hypervisor hit 50, and divided by each container's 2 CPUs, they all «were» at 25.
        Fix
        Disable that trigger on the containers and keep it only on the hypervisor, where it measures something real. Each container's CPU is still watched by usage.
        Lesson
        An alert that fires on something that can't be measured that way teaches you to ignore alerts. Check what the metric really measures in that environment.

      2. #2

        Disk latency «too high»

        08-09-2026 · disks

        Constant alerts about read and write latency above 20 ms on one of the system disks.

        What do you think was going on?

        Show the fix
        Cause
        SMART clean and no kernel errors. It's one of the two spinning disks in the system RAID, under real load: 15-28 ms is expected.
        Fix
        Raise the threshold to 50 ms for that disk and its mirror partner, not for all of them.
        Lesson
        Default thresholds are for a generic disk. They're tuned with data from the actual disk, and for both mirror members at once.

      3. #3

        Agents with no sign of life

        16-09-2026 · registration

        After registering several new containers, only two started sending data. The others stayed at «trying to connect», with no error.

        What do you think was going on?

        Show the fix
        Cause
        The two that worked reused addresses already on the server firewall's list. The new ones were silently dropped.
        Fix
        Add the addresses to the firewall and make that step part of the registration itself: a new container can no longer be left out by oversight.
        Lesson
        A failure that happens twice in a day isn't bad luck: it's a step missing from the procedure.

      Told in full in an article:

      How it was built

      1. 06-09-2026

        Isolated install

        • Zabbix and Wazuh in their own unprivileged containers with no SSH.
        • Agents with their own key across the fleet, no remote commands.
        • Alert channels by severity.

      2. 16-09-2026

        Registration in one command

        • A script that adds a container to Zabbix and Wazuh and stores the key in the vault.
        • Server firewall included in the registration.

      3. September and October 2026

        Less noise

        • Disk thresholds tuned to the real disk.
        • A week of alerts measured before changing anything.
        • Email only for high and the misleading trigger gone.

      Decisions (and why)

      • Who starts the connection

        The agent, in active mode

        That way the server doesn't need to reach every machine: only the machines reach the server. One less surface.

      • How Docker is checked

        Read-only collectors, no socket

        Giving the Docker socket to the monitoring user is giving it the server. A script with minimal permissions leaves the state in a file.

      • How to decide what to alert on

        Measure first

        Lowering thresholds by eye hides real problems. Counting what arrived, how much and from where said what was surplus.

      What I learned

      • An alert nobody believes is worse than none: measure and tune.
      • Check what a metric means inside a container.
      • Whatever fails twice goes into the procedure.

      What's next

      • Repeat the alert measurement after a few weeks to confirm the effect.
      • Also check each service answers its users, not just that its agent reports.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my documentation: no addresses, machine names or credentials.