TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · BACKUPPBS · Git ×2 · rollback In production

Slot 21 of 25 in the rack · In production

1U unit · Backups

Real backups, and what they don't cover yet

Encrypted nightly backups · restore tested

For months, this unit said «data: pending». Now every container on the server gets a copy every night on Proxmox Backup Server, encrypted before it leaves, with retention the server itself can't touch. It's verified weekly and restores have been tested file by file. The code keeps its two backups and every delicate change, its way back. What's missing, an off-site copy, is still written down as missing.

What's copied
Every container, documents included, and the game worlds separately. Movies and Steam games aren't: they can be downloaded again.
Who can delete
Nobody but the backup server itself, which prunes on its own schedule. If the main server fell into the wrong hands, it couldn't delete its backups.
What's missing
It's all at home, on a disk plugged into the same server. It protects against deletions, corruption and dying disks; not against theft or fire.
  • 02:30every night, a copy of every container
  • 7 · 4 · 6daily, weekly and monthly copies kept
  • 1,381files identical in the first test restore; now it repeats itself every Sunday
  • 0backups the main server can delete

Its code, in the open Public version of the repo: no IPs or secrets.

Backups, inside

What's copied every night, who can delete it, how you know it works and what it doesn't cover yet. Pick a walkthrough.

Flows Pick one and follow the packet.

Outside Backups · a backup server and what's around it Proxmox Backup Server in its container, with its own disk Code and documentation Every change, with a way back Data backups Checking they work My laptop Forgejo Private GitHub Password mgr Me Element Separate disk Hook Backup repo git-shell Documentation Way back 5 minutes Dated copies Snapshot PBS Encrypted No deleting Worlds Verification Last commit Restore Alerts
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      Every night
      1. Proxmox Backup Server — At 2:30, Proxmox copies every container to the backup server.
      2. Encrypted before leaving — Encrypted before leaving, with a key I also keep off the server.
      3. Proxmox Backup Server — Incremental: only what changed since the night before travels.
      4. Who copies doesn't delete — The main server can copy but not delete. The backup server prunes on its own: 7 daily, 4 weekly and 6 monthly.
      5. The backup disk — Everything goes to a separate disk, just for this.
      6. Weekly verification — On Sundays, verification rereads every copy.
      7. Failure alerts — If anything fails, an alert goes out.
      8. Element — And reaches the incidents room. If nothing arrives, it went fine.
      The game worlds
      1. The game worlds — The worlds (158 GB) are mounted from outside the games container: its copy doesn't include them.
      2. Snapshot — At 3:30, a storage snapshot, to copy a consistent state even if someone's playing.
      3. Encrypted before leaving — Encrypted like everything else.
      4. Proxmox Backup Server — And to the same backup server, incrementally.
      Every Sunday, a test restore
      1. Restoring for real — Every Sunday a group of containers is restored into new containers, without starting them so they don't clash with the originals.
      2. Proxmox Backup Server — The backup server hands over their copies.
      3. Encrypted before leaving — They're decrypted with the key.
      4. My password manager — The same one I keep in my password manager: if the server died, without that copy there'd be nothing to restore.
      5. Restoring for real — Files unchanged since the backup are compared with the originals, and the temporary containers are deleted. The first time, 1,381 of 1,381 identical.
      6. Me — The summary reaches my phone; if anything fails, the incidents room. I no longer hope it works: I know, every week.
      I make a commit
      1. My laptop — I change something in the infrastructure.
      2. Forgejo, the origin — Commit, and push to Forgejo, the origin.
      3. post-commit hook — The hook fires on its own.
      4. Backup repository — It pushes to the backup repository, on another disk.
      5. Git-only user — With a user that can only do Git.
      6. Backup repository — And the backup is up to date, without me doing anything.
      A delicate change
      1. Dated copies — I'm about to change the hypervisor firewall. Ansible keeps a copy of what it replaces.
      2. Snapshot — And a container snapshot, if it's an update.
      3. Rollback script — The script that puts everything back is saved.
      4. Scheduled rollback — A 5-minute timer starts.
      5. Rollback script — If I'm locked out, it reverts on its own. If all goes well, I cancel it.
      The documentation
      1. Documentation backup — The notes copy themselves every 10 minutes.
      2. Private GitHub repository — To the private repository.
      3. Date of the last commit — But one day the last commit was 13 days old.
      4. Me — I found out by looking at the date, not because anything warned me.
      5. Documentation backup — Now it's the first thing I check.

      1. My laptop

        I edit the repository from here or from the control node.

      2. Forgejo, the origin

        Where the work happens: the repository's origin, with CI. It has its own unit.

        • Git

      3. Private GitHub repository

        Where the documentation is pushed, outside the house.

        • Git

      4. My password manager

        Holds a copy of the backups' encryption key. Without it nothing can be restored, and it's off the server.

      5. Me

        Who decides what's in, what's out and what's missing.

      6. Element

        The incidents room: only failures arrive.

      7. The backup disk

        A 4.5 TB disk just for this. If it isn't mounted, the backup server doesn't start: it never writes to the system disk.

      8. post-commit hook

        Versioned in the repository itself: every commit is pushed to the backup on its own, in the background.

        • Git

      9. Backup repository

        A bare repository in a tiny container, on a different disk from the working one.

        • Git
        • Linux (Debian, Alpine, Arch)

      10. Git-only user

        The backup user can only push and pull: not even a shell. With its own key.

        • Linux / SSH hardening
        • Linux (Debian, Alpine, Arch)

      11. Documentation backup

        The notes vault is pushed every 10 minutes. It has its own unit.

        • Git

      12. Rollback script

        Before a delicate change, the state is saved along with a script that puts everything back.

        • Linux (Debian, Alpine, Arch)

      13. Scheduled rollback

        If the change locks me out, it undoes itself after 5 minutes. If all goes well, I cancel the timer.

        • Linux (Debian, Alpine, Arch)

      14. Dated copies

        Ansible leaves a dated copy of every file it replaces.

        • Ansible

      15. Snapshot

        A frozen picture of the disk before a change or a backup, so a consistent state gets copied. Deleted afterwards.

        • Proxmox VE

      16. Proxmox Backup Server

        In its own container. Every night it gets a copy of every container, incremental and deduplicated: only what changed travels.

        • Proxmox Backup Server
        • LXC

      17. Encrypted before leaving

        The main server encrypts them before sending. Whoever took the backup disk couldn't read anything.

        • Proxmox Backup Server
        • Linux / SSH hardening

      18. Who copies doesn't delete

        The main server gets in with a permission that allows backing up and restoring, not deleting. Only the backup server prunes: 7 daily, 4 weekly and 6 monthly.

        • Proxmox Backup Server
        • Linux / SSH hardening

      19. The game worlds

        They live outside the game panel's container, so they're not in its copy. They have their own, at 3:30.

        • ZFS
        • Proxmox Backup Server

      20. Weekly verification

        Every Sunday, the backup server rereads every copy and checks it's whole.

        • Proxmox Backup Server

      21. Date of the last commit

        The simplest way to tell whether a Git backup is still alive.

        • Git

      22. Restoring for real

        The only proof a backup works, and it's automatic: every Sunday a group of containers is restored on the side and compared file by file. Each container gets its turn once a month.

        • Proxmox VE
        • Proxmox Backup Server

      23. Failure alerts

        If a backup fails, Proxmox posts to the incidents room. If all goes well, silence.

        • Proxmox VE

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      Ansible
      The roles that set up the backup server and the code backup, and the dated copies left every time a file is replaced.
      Proxmox VE
      The nightly job, snapshots, restores and failure alerts.
      LXC
      The backup server and the code backup each live in their own container.
      ZFS
      Snapshots to copy the game worlds in a consistent state. And RAID, which survives a dead disk, isn't a backup.
      Proxmox Backup Server
      Incremental nightly backups of every container, encrypted at the source, pruning and verification on the backup server itself, and a permission that can't delete.
      Git
      A versioned hook that pushes every commit to a second repository, and the documentation to a private repository.
      Linux (Debian, Alpine, Arch)
      A git-shell user, rollback scripts with a timer and a backup folder only its service can read.
      Linux / SSH hardening
      Encrypted backups, a permission that can't delete and a backup user that can only do Git.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        A permission that wasn't enough

        15-10-2026 · backups

        The token the main server uses on the backup server had its backup permission on the datastore. Even so, Proxmox said the datastore didn't exist.

        What do you think was going on?

        Show the fix
        Cause
        On the backup server, a token's effective permissions are the intersection of its own and its user's. The user had none, so neither did the token, and the error presented it as a missing datastore.
        Fix
        Give the same backup-only permission to the user and the token. It still can't delete.
        Lesson
        «Doesn't exist» can mean «you can't see it». Faced with an error about something that does exist, check permissions first.

      2. #2

        158 GB that weren't in any backup

        15-10-2026 · backups

        Going over what went into the backups, the game worlds weren't in the games container's copy, even though from inside they look like just another folder.

        What do you think was going on?

        Show the fix
        Cause
        The worlds live on the server's storage and are mounted into the container. Proxmox backs up the container's disk, not what's mounted.
        Fix
        A separate copy of that storage, with a snapshot so the state is consistent, encrypted and to the same backup server. And a written rule: if any container keeps something irreplaceable on a mount, it goes on that list.
        Lesson
        «It's inside the container» isn't the same as «it's in the container's backup». Check where every folder comes from.

      3. #3

        A backup on the same disk

        17-09-2026 · code

        The infrastructure had its «backup»: a bare repository next to the working one.

        What do you think was going on?

        Show the fix
        Cause
        Both copies lived on the same disk of the same PC. And the laptop only synced by hand and over VPN.
        Fix
        A second repository in a container of its own on the server, with a user that can only do Git, and a versioned hook pushing every commit on its own.
        Lesson
        A backup in the same place as the original doesn't protect against what matters most. Ask what would destroy it along with the original.

      4. #4

        Ansible couldn't create the backup

        17-09-2026 · Alpine

        The backup role failed on the container: first when assigning the key folder's group, then with «sudo: not found».

        What do you think was going on?

        Show the fix
        Cause
        The container is a minimal Alpine: creating a user doesn't create its group, and there's no sudo to switch users.
        Fix
        Create the group explicitly and create the repository as root, then fix the owner with the file module.
        Lesson
        A role written for Debian isn't a role for any Linux. The system's assumptions have to be written down.

      5. #5

        «I have RAID, so I have a backup»

        06-09-2026 · decision

        All the important data is mirrored or on RAIDZ. Why more?

        What do you think was going on?

        Show the fix
        Cause
        RAID protects against a disk dying. Anything else (a wrong command, a corrupted file, ransomware) reaches every disk at once.
        Fix
        For weeks it stayed written down as pending, not hidden. On 15-10, nightly backups on Proxmox Backup Server, encrypted, which the main server can't delete, and a test restore.
        Lesson
        An acknowledged gap beats a false sense of safety. And when it was solved, the proof was a restore, not a file.

      6. #6

        Sunday's test failed on every container

        18-10-2026 · backups

        The test restore failed on all seven containers in the group and on the game worlds, every one with «Permission denied». The day before it was fine.

        What do you think was going on?

        Show the fix
        Cause
        That morning, a deployment of the download manager changed the owner of everything on the media disks, as it always does. The backups live in a folder on one of those disks: their 96,672 chunks went to another user and the backup server could no longer read them. The backups were intact.
        Fix
        Give the backups folder its owner back and rerun the test. So it doesn't happen again: the role skips folders that aren't media and only touches files with the wrong owner, which is also much faster.
        Lesson
        The weekly test caught it the same day. Without it, I'd have found out the day I needed a backup.

      Told in full in an article:

      How it was built

      1. September 2026

        Rollbacks

        • Every security change with its rollback script and a 5-minute timer.
        • Backups of the game panel's accounts, and their fix when they turned out empty.

      2. 17-09-2026

        Code backup

        • A second repository in its own container, with a Git-only user.
        • A versioned hook that pushes every commit on its own.

      3. 15-10-2026

        Real backups

        • Proxmox Backup Server in its container, with a disk just for it.
        • Every container nightly and the game worlds separately, encrypted, with pruning only the backup server can do.
        • Weekly verification, failure-only alerts and a test restore compared file by file.
        • The old ad hoc copies and some old unencrypted ones, gone.

      4. 15 to 18-10-2026

        Restoring every week

        • An automatic test restore every Sunday, with a summary on my phone.
        • Its first real failure, caught the same day: the test told me, not an emergency.
        • The game worlds are checked by mounting the copy and reading only what's needed: seconds instead of hours.

      Decisions (and why)

      • What gets copied

        Everything that can't be downloaded again

        Every container, documents included, and the game worlds. Out: movies, series and Steam games (tens of terabytes that can be downloaded again) and the lab cluster, which is rebuilt from code.

      • Encryption

        On the server, before they leave

        The backup disk could get lost or taken. The key has a copy in my password manager: without it nothing can be restored, not even by me.

      • Who can delete

        Only the backup server

        If someone took over the main server, the first thing they'd try is deleting the backups. With a permission that only allows backing up and restoring, they can't.

      • Where

        At home, for now

        A disk on the same server protects against deletions, corruption and dying disks, which is what has actually happened. It doesn't protect against theft or fire. It's written down as the next step.

      • How what's missing is reported

        In writing

        A red status in the documentation stopped anyone, me included, from believing for weeks that there was something that wasn't there. Now what's red is the off-site copy.

      What I learned

      • A backup that's never been restored is a hope.
      • RAID isn't a backup, and what's mounted from outside isn't in the container's backup.
      • What's missing is written down as missing, even when almost nothing is.

      What's next

      • An off-site copy: a rotated disk or a second backup server somewhere else.
      • A UPS, so power cuts stop being the cause of half the problems.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my own documentation: no addresses, paths or internal names. What's missing is written down as missing.