TPTito Pinardo Gutiérrez
← Articles
Español

Build Published on · 1U · BACKUP

From «no backups» to encrypted backups the server itself can't delete

For weeks my homelab had an acknowledged gap: the code had two backups, but the data had no regular copies. In one day I set up Proxmox Backup Server with a disk of its own, encrypted backups of everything every night, a permission that can't delete and a test restore. Along the way, two surprises: a permission that wasn't enough and 158 GB that weren't in any backup.

  • Proxmox Backup Server
  • Proxmox VE
  • ZFS
  • Linux / SSH hardening

Where it started

The important data lives on mirrors and RAIDZ, which protects against a dying disk. But a deletion, a mistake or corruption reaches every disk at once: RAID isn't a backup. I had it written down as pending, in red, and that's what this site's backups unit said too.

What I set up

  • Proxmox Backup Server in its own container, with a 4.5 TB disk just for it. If the disk isn't mounted, the container doesn't start: it never writes backups to the system disk.
  • Every night at 2:30, every container, documents included. Incremental and deduplicated: only what changed travels.
  • Encrypted on the server before leaving. The key has a copy in my password manager; whoever takes the disk reads nothing.
  • Pruning on the backup server itself: 7 daily, 4 weekly and 6 monthly. Verification every Sunday: it rereads every copy.
  • Failure-only alerts, in the incidents room. If nothing arrives, it went fine.

Who copies doesn't delete

The main server gets into the backup server with a token that can back up and restore, but not delete or prune: only the backup server does that. If someone took over the main server, the first thing they'd try is deleting the backups, and they couldn't.

First surprise: with the permission on the token, Proxmox said the datastore didn't exist. What a token can do is the intersection of its permissions and its user's, and the user had none. With the same permission on both, it worked. «Doesn't exist» meant «you can't see it».

158 GB outside the backup

Second surprise, going over what was included: the game worlds weren't in their container's backup. They live on the server's storage and are mounted inside, and Proxmox backs up the container's disk, not what's mounted.

Now they have their own backup at 3:30: a storage snapshot, so a consistent state is copied even if someone's playing, encrypted and to the same backup server. And a written rule: if a container keeps something irreplaceable on a mount, it goes on that list.

The proof

An unrestored backup proves nothing. I restored the messaging client's into a new container, without starting it so it wouldn't clash with the original, and compared file by file: 1,381 of 1,381, identical.

What's left out, on purpose: movies, series and Steam games (tens of terabytes that can be downloaded again) and the lab cluster, which is rebuilt from code.

What's still missing

It's all at home: the backup disk is plugged into the same server. It protects against deletions, corruption and dying disks, which is what has actually happened, but not against theft or fire. The next step is an off-site copy: a rotated disk or a second backup server somewhere else. It stays written in red until then.

Three days later: the test earns its keep

Since 15-10, every Sunday a batch of containers restores itself and is compared with the originals. The first Sunday it failed on all of them with «Permission denied». The backups were fine: that morning, a deployment of the download manager had changed the owner of everything on the media disks, and the backups live on one of them. 96,672 chunks went to another user.

Giving them their owner back fixed it, and the role now skips folders that aren't media. What matters is when I found out: the same day, thanks to the test, not the day I'd have needed a backup.