TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · STORAGEZFS · RAIDZ1 · mirrors · RAID1 In production

Slot 25 of 25 in the rack · In production

1U unit · Storage

ZFS and RAID · the disks

Writes ×13 without buying anything

Large data lives on a ZFS RAIDZ1 pool of five 10 TB disks in an external shelf; the game worlds, on a mirror of four SAS disks; the system, on a software RAID1. ZFS checks every block as it reads it and a huge RAM cache serves almost all reads.

Why ZFS
Checksums on every block, compression, snapshots and live RAIDZ expansion. And the server's spare RAM becomes cache.
Which pool for what
RAIDZ for lots of space with one disk of margin; mirrors for small, fast writes; software RAID1 for the system.
What it isn't
Neither RAIDZ nor mirrors are a backup: they survive a disk dying, not a deletion. That's in the backups unit.
  • 95 → 1,237fsync writes per second on the big pool
  • 5 × 10 TBin RAIDZ1, grown from 4 to 5 live
  • 0 Brepaired in the scrub after expansion
  • > 98 %of game reads served from the RAM cache

Inside the storage

Several pools for different jobs and an external disk shelf. Pick a flow.

Flows Pick one and follow the packet.

Connections Server and shelf disks ZFS managed by the hypervisor Pools Tuning Disks Health SAS HBA 24-bay shelf Containers Zabbix Proxmox 9.2 RAIDZ1 pool Mirror pool System RAID1 Loose disks ARC cache recordsize 1M atime off Scheduler Write cache On boot Expansion XFS UUID Scrub SMART Usage fio
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      Someone watches a film
      1. Containers — Plex asks for a file.
      2. Big pool (RAIDZ1) — From the big pool, which checks each block's checksum as it reads.
      3. RAM cache — What's read often stays in the RAM cache.
      4. Containers — And it reaches the container, verified.
      I expand the pool live
      1. 24-disk shelf — I put a fifth disk in the shelf.
      2. SAS controller — The controller sees it.
      3. Hypervisor upgrade — First, upgrade the hypervisor: expanding a RAIDZ needs a newer ZFS.
      4. Expanding the RAIDZ — The expansion targets the disk group, not a disk.
      5. Big pool (RAIDZ1) — The pool stays in use while the data is spread out.
      6. Scrub — When done, a scrub verifies everything: 0 errors.
      Thirteen times more writes
      1. Measuring with fio — I measure with fio: 95 fsync writes per second.
      2. Disk write cache — I turn on the disks' write cache.
      3. Re-enable at boot — And a service that turns it on again on every boot.
      4. «none» scheduler — With the scheduler on «none».
      5. Measuring with fio — I measure again: 1,237.
      Watching the disks
      1. SMART — Each disk reports its SMART.
      2. Monitoring — Monitoring collects it without waking sleeping disks.
      3. Scrub — Regular scrubs check every block.
      4. Usage — And each pool's usage.
      5. Monitoring — If something gets worse, an alert.
      A disk going back in
      1. Loose disks — I pull a disk from its bay and put it back.
      2. A disk's identifier — It won't mount: the system thinks the same disk is already mounted.
      3. Loose disks — The file system's old reference is cleared and it mounts.
      4. Containers — And its data is available again.

      1. SAS controller

        A controller in HBA mode: ZFS sees the disks as they are, with no hardware RAID in between.

      2. 24-disk shelf

        A second-hand disk shelf, connected over SAS, where the big pool lives.

      3. Containers

        They mount the datasets they need.

        • LXC

      4. Monitoring

        Pool and RAID state and SMART for each disk, without waking the sleeping ones.

        • Zabbix

      5. Hypervisor upgrade

        Expanding a RAIDZ needs ZFS 2.3 or later: Proxmox had to be upgraded first.

        • Proxmox VE

      6. Big pool (RAIDZ1)

        Five 10 TB disks: room for the media library, with one disk of margin.

        • ZFS

      7. Games pool (mirrors)

        Four SAS disks in two mirrors: small, frequent writes, like saving a world.

        • ZFS

      8. System RAID1

        Two disks in a software mirror for the system and the containers' disks.

        • Linux (Debian, Alpine, Arch)

      9. Loose disks

        A few standalone XFS disks, waiting to join the pool.

        • Linux (Debian, Alpine, Arch)

      10. RAM cache

        ZFS uses free RAM as a read cache. On the games pool it serves over 98 %.

        • ZFS

      11. Block size

        1 MB blocks for video, which is read start to finish. Normal blocks where there are small files.

        • ZFS

      12. No access time

        Don't write every time a file is read.

        • ZFS

      13. «none» scheduler

        ZFS already orders and groups I/O: the kernel scheduler just gets in the way.

        • Linux (Debian, Alpine, Arch)

      14. Disk write cache

        It was off on several disks. With ZFS it's safe: it flushes that cache on every transaction.

        • Linux (Debian, Alpine, Arch)

      15. Re-enable at boot

        On the disks behind the shelf the setting doesn't always survive a shutdown: a service sets it again on every boot.

        • Linux (Debian, Alpine, Arch)

      16. Expanding the RAIDZ

        Adding a fifth disk to a four-disk RAIDZ, live, without moving the data elsewhere.

        • ZFS

      17. A disk's identifier

        A disk pulled out and put back can clash with itself.

        • Linux (Debian, Alpine, Arch)

      18. Scrub

        Reads the whole pool and checks every block against its checksum. After the expansion: 0 errors.

        • ZFS

      19. SMART

        The health each disk reports about itself: reallocated sectors, temperature, errors.

        • Linux (Debian, Alpine, Arch)

      20. Usage

        Above 80-85 %, ZFS writes slower. It needs watching before you get there.

        • ZFS

      21. Measuring with fio

        On a temporary uncompressed dataset, before and after each change, deleted afterwards.

        • Linux (Debian, Alpine, Arch)

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      Proxmox VE
      ZFS managed from the hypervisor; upgraded to allow RAIDZ expansion.
      LXC
      Containers mount the datasets they need.
      ZFS
      RAIDZ1 expanded live, mirrors for games, ARC cache, compression, recordsize and scrubs.
      Linux (Debian, Alpine, Arch)
      Software RAID for the system, disk write cache, I/O scheduler and loose XFS disks.
      Zabbix
      Watches pools, RAID and SMART without waking sleeping disks.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        An expansion that did nothing

        July 2026 · pool

        The script to grow the pool with a new disk finished, but the pool stayed the same size. No clear error.

        What do you think was going on?

        Show the fix
        Cause
        The command targeted one disk in the pool, as for a mirror, instead of the whole RAIDZ group. And the installed ZFS version didn't yet allow RAIDZ expansion.
        Fix
        Upgrade the hypervisor to a version with ZFS 2.3 or later and run the expansion against the disk group. It finished in a day and the scrub afterwards repaired 0 bytes.
        Lesson
        A script that «finishes» isn't a script that works: check the result. And read which version allows what before launching something like that.

      2. #2

        «wrong fs type, bad superblock»

        14-07-2026 · disk

        I pulled a disk from its bay and put it back. It wouldn't mount, and the error talked about a damaged file system.

        What do you think was going on?

        Show the fix
        Cause
        Pulled live, the kernel shut the file system but didn't fully forget it. Coming back under another device name, its identifier clashed with the one still listed as mounted.
        Fix
        Clear the old reference and mount it again, without touching its data.
        Lesson
        The error message doesn't always give the cause: «bad superblock» meant «I already know you». The kernel log said it plainly.

      3. #3

        Slow disks out of the box

        06-10-2026 · performance

        The big pool managed only 95 fsync writes per second, with five disks.

        What do you think was going on?

        Show the fix
        Cause
        The disks had write cache disabled, and the ones behind the shelf didn't even accept the normal command to enable it.
        Fix
        Enable it with the command they do accept, save it on the disk and add a service that enables it again on every boot. From 95 to 1,237 writes per second.
        Lesson
        Factory settings aren't the best for every use. Measure, change one thing, measure again.

      Told in full in an article:

      How it was built

      1. July and August 2026

        Pool expansion

        • Proxmox upgraded to get a ZFS version that can expand RAIDZ.
        • Fifth disk added live and a clean scrub afterwards.

      2. September and October 2026

        Measured tuning

        • No access time and large blocks for video.
        • Write cache and scheduler on every pool, measured with fio before and after.

      Decisions (and why)

      • Hardware RAID or ZFS

        ZFS with the controller in HBA mode

        ZFS needs to see the real disks to check every block and repair what it finds. Hardware RAID would hide them.

      • One pool or several

        One per usage pattern

        Lots of space and sequential reads don't want the same as many small writes. Separating them lets each be tuned.

      What I learned

      • Check the result of a disk operation, not that the script finished.
      • Measure each change before and after.
      • RAID isn't a backup.

      What's next

      • Bring the loose disks into the pool.
      • An SSD for the system and the databases.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my documentation: no paths, disk names or content.