TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · PROXMOXPVE 9.2 · 48 threads · ~250 GB RAM In production

Slot 24 of 25 in the rack · In production

1U unit · Hypervisor

Proxmox VE · holding everything up

Proxmox 9.2, up from 8.4

A real rack server, a 1U Dell R620 with two Xeons and about 250 GB of RAM, plus an external 24-disk shelf. Proxmox splits that machine into about twenty LXC containers and the lab VMs, with its own firewall and a GPU for Plex.

Why Proxmox
Containers and VMs in one place, a full API for Terraform and ZFS built in. No licences.
The weak spot
The system and every container's disks live on a mirror of two spinning disks. It shows on every update and every power cut.
Security
Hypervisor firewall dropping inbound by default, a firewall per container, auditing of account files and spare services switched off.
  • ~20LXC containers, almost all unprivileged
  • 48threads across two processors
  • 65-90 minof disk checking after a power cut
  • 11containers whose firewall wasn't applied

Its code, in the open Public version of the repo: no IPs or secrets.

Inside the hypervisor

Everything else runs here. Pick a flow to see how a container is born or what happens after a power cut.

Flows Pick one and follow the packet.

Outside Dell PowerEdge R620 · Proxmox VE plus a 24-disk shelf Management and security Guests System disks Hardware My laptop Control node Power cut Zabbix Web console API Firewall Audit Containers VMs GPU Auto start RAID1 md0 LVM-thin ZFS pools Boot 2× Xeon ~250 GB RAM Fans 24-bay shelf
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      A container is born
      1. Control node — The control node asks for a new container.
      2. Proxmox API — Through the API, with least privilege.
      3. LXC containers — Proxmox creates it.
      4. LVM-thin — Its disk, on LVM-thin.
      5. System RAID1 — On top of the system mirror.
      6. ZFS pools — Its large data, mounted from ZFS.
      7. Hypervisor firewall — Its firewall, with its options section enabled: without it, rules aren't applied.
      8. LXC containers — And it starts.
      The power goes
      1. The mains — The power goes and the server switches off abruptly. It comes back and boots on its own.
      2. Hypervisor boot — The hypervisor boots.
      3. System RAID1 — The mirror is checked.
      4. LVM-thin — LVM-thin needs to check its metadata: over an hour on spinning disks.
      5. Automatic start — Meanwhile, the containers' automatic start fails, and isn't retried.
      6. LXC containers — Once storage is ready, they have to be started.
      7. Monitoring — Monitoring confirms everything comes back.
      I manage the server
      1. My laptop — I open the console from home.
      2. Hypervisor firewall — The hypervisor firewall only lets in the admin network and the VPN.
      3. Proxmox web console — The web console.
      4. Virtual machines — I check the lab VMs.
      5. GPU for Plex — And the GPU's state.
      The temperature rises
      1. Two 12-core Xeons — The CPU heats up.
      2. Fan control — The fan script raises their speed over IPMI.
      3. Two 12-core Xeons — The CPU cools down.
      4. 24-disk shelf — The disk shelf has its own cooling, which the script doesn't control.
      5. Monitoring — And monitoring keeps an eye on disk temperatures.
      From Proxmox 8.4 to 9.2
      1. Control node — I prepare the upgrade from the control node.
      2. Proxmox web console — The hypervisor is upgraded.
      3. Hypervisor boot — Boot moves to systemd-boot: kernel parameters change place.
      4. GPU for Plex — The GPU driver has to build again against the new kernel.
      5. Auditing — And auditing reviews what changed.

      1. My laptop

        The web console, only from home or the VPN.

      2. Control node

        Terraform and Ansible, which talk to the API.

        • Terraform
        • Ansible

      3. The mains

        With no UPS, every cut switches the server off abruptly.

      4. Monitoring

        Hypervisor load, RAID, SMART, temperature and real storage usage.

        • Zabbix

      5. Proxmox web console

        Only from the admin network and the VPN: not published on the internet.

        • Proxmox VE

      6. Proxmox API

        Terraform uses it with a least-privilege user and token.

        • Proxmox VE

      7. Hypervisor firewall

        Inbound to the hypervisor dropped by default, and a rules file per container.

        • Proxmox VE
        • Linux / SSH hardening

      8. Auditing

        auditd watches accounts, sudo and SSH; the security agent, the rest.

        • Wazuh
        • Linux / SSH hardening

      9. LXC containers

        One service per container. Lighter than a VM and, unprivileged, their root isn't the server's.

        • LXC

      10. Virtual machines

        For what needs its own kernel: the lab's OKD cluster.

        • Proxmox VE

      11. GPU for Plex

        The GTX 1050 Ti, with its own IOMMU group and the driver on the hypervisor.

        • Proxmox VE
        • Linux (Debian, Alpine, Arch)

      12. Automatic start

        Each container starts on its own, in order, when the hypervisor boots. It's only tried once.

        • Proxmox VE

      13. System RAID1

        Two spinning disks in a software mirror. The system and every container's disks, here.

        • Linux (Debian, Alpine, Arch)

      14. LVM-thin

        Container disks, reserving only what they use. After an abrupt shutdown it needs a long check.

        • Linux (Debian, Alpine, Arch)

      15. ZFS pools

        Large data, on their own pools. They have their own unit.

        • ZFS

      16. Hypervisor boot

        systemd-boot since 9.2, with the kernel parameters for IOMMU.

        • Linux (Debian, Alpine, Arch)

      17. Two 12-core Xeons

        48 threads, strong in parallel but without AVX2: that's why the AI doesn't run here.

      18. Memory

        Plenty for the containers; ZFS uses the rest as cache.

      19. Fan control

        A script sets the chassis fans over IPMI based on CPU temperature. The disk shelf has its own.

        • Linux (Debian, Alpine, Arch)

      20. 24-disk shelf

        Connected over SAS, with its own cooling.

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      Terraform
      Creates containers and VMs through the API, with a least-privilege token.
      Ansible
      Configures the hypervisor: firewall, DNS, auditing and updates.
      Proxmox VE
      Proxmox VE 9.2: containers and VMs, an API for Terraform, hypervisor and per-container firewalls, GPU with IOMMU.
      LXC
      About twenty containers, almost all unprivileged, one per service.
      ZFS
      The large data pools, managed from the hypervisor itself.
      Linux (Debian, Alpine, Arch)
      Software RAID, LVM-thin, systemd-boot, auditd and the IPMI fan script.
      Zabbix
      Watches load, RAID, SMART, temperature and real storage usage.
      Wazuh
      The hypervisor's agent watches accounts, sudo, SSH and services.
      Linux / SSH hardening
      Inbound dropped by default, auditing of critical files and spare services switched off.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        No container would start after a power cut

        07-08-2026 · storage

        After a power cut the server booted on its own, but no container or VM started. The documentation couldn't be opened either: it was inside too.

        What do you think was going on?

        Show the fix
        Cause
        The abrupt shutdown left LVM-thin flagged for checking. The check took 65 to 90 minutes on spinning disks and left its internal parts active separately, which prevented activating the whole.
        Fix
        Wait for the check, deactivate the two loose parts by hand and activate the whole. Then start every container by hand: automatic start is only attempted once.
        Lesson
        Write the postmortem with the exact step: next time, going straight to deactivating the loose parts saves another full check. And a UPS would avoid all of it.

      2. #2

        Firewall rules that did nothing

        22-09-2026 · firewall

        I had restricted the containers' SSH to my machines and the VPN, and blocked the vector database's port. From another container you could still reach both.

        What do you think was going on?

        Show the fix
        Cause
        The containers that did filter had an options section in their file with the firewall enabled. The rest only had rules: Proxmox stores them but doesn't apply them, even if the interface has the firewall ticked.
        Fix
        Enable the options section on the 11 affected containers and check from outside, trying to connect to what should be closed.
        Lesson
        A security rule is tested by attacking it. Otherwise it can sit there written down, doing nothing, for days.

      3. #3

        Terraform lost a permission after upgrading

        September 2026 · upgrade

        After moving to Proxmox 9.2, creating Terraform's permission role failed: a privilege on the list no longer existed.

        What do you think was going on?

        Show the fix
        Cause
        Proxmox 9.2 retired a monitoring privilege that the Terraform provider's docs still asked for.
        Fix
        Drop that privilege from the role and check with a plan that Terraform could still do everything it needed.
        Lesson
        Permission guides age with every release. Better to grant exactly what's needed and test it than copy a list.

      Told in full in an article:

      How it was built

      1. Until summer 2026

        Proxmox 8.4

        • Containers created by hand, hypervisor firewall disabled.

      2. Summer 2026

        Proxmox 9.2

        • Upgrade to 9.2, needed to expand the disk pool.
        • New boot method and GPU driver rebuilt.

      3. September and October 2026

        Lock down and measure

        • Hypervisor firewall dropping by default, with automatic rollback after 5 minutes.
        • Container firewalls truly enabled and tested.
        • Write cache on the system disks and the hypervisor's DNS through the home DNS.

      Decisions (and why)

      • Containers or VMs

        Containers, except what needs its own kernel

        A container starts in seconds and uses almost nothing. Only the lab cluster needs VMs.

      • Privileged or not

        Unprivileged whenever possible

        Unprivileged, the container's root isn't the server's. Plex moved to unprivileged even with the GPU inside.

      • Firewall changes

        With automatic rollback

        A mistake in the hypervisor firewall locks you out of the hypervisor. Each change had a timer that undid it unless I cancelled it.

      What I learned

      • A server without a UPS turns every power cut into an hour-long check.
      • Security rules get tested from outside.
      • Writing down the exact step that fixes something halves the time next time.

      What's next

      • A UPS with coordinated shutdown.
      • Move the system and the container disks to SSD.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my documentation: no addresses, ports or internal names.