TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · BAOOpenBao · keys · seal In production

Slot 20 of 25 in the rack · In production

1U unit · Secrets

Secrets, out of the repository and off the disk

Secrets in the repository: zero

The whole infrastructure's secrets lived encrypted in the repository, with a single password kept on the control node's disk next to the Terraform credentials. Now they all live in OpenBao, the open version of Vault, and only their names are left in the repository. Every deployment asks for them with a read-only role, gets a 15-minute token and uses them in memory. After every restart, OpenBao waits sealed until I open it. And along the way it signs the internal services' certificates.

What it holds
Over 90 secrets from every service, split by group, the Terraform credentials and the authority that signs internal certificates. The repository only has the names.
Who asks for them
Only the control node, with a read-only role that works from nowhere else. I log in with the same single sign-on as everywhere else. Every read is audited.
After a restart
Sealed until I open it with a key that isn't on any of the server's disks. Services keep running; what stops is change.
  • 0secret files in the repository
  • 15 minhow long each token lasts
  • 90+secrets moved out of the repository and checked one by one by fingerprint
  • 1key to unseal, and it's not on the server

The keys, inside

Where the whole infrastructure's secrets come from, who can ask for them and what happens when the server restarts. Pick a walkthrough.

Flows Pick one and follow the packet.

Outside OpenBao · container on Proxmox one CPU, half a gigabyte of memory and nothing else to do OpenBao Who may ask Control node: who asks Around it Me Password mgr Element Repository Proxmox API Separate disk Own TLS Storage Seal Audit AppRole Only from there 15-min token Read-only Per group tf Deploy Write role Watcher Web with 2FA Daily copy Home CA
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      A deployment with the keys off the disk
      1. The deploy runner — Time to deploy.
      2. Deployment AppRole — The runner identifies itself with its AppRole.
      3. Bound to one origin — OpenBao checks the request comes from the control node.
      4. Read-only policy — And that it only asks for what its policy allows: reading the two keys.
      5. Short tokens — It gets a 15-minute token.
      6. Secrets per group — Ansible asks for each group's secrets with that token.
      7. The repository — From the repository it only takes their names.
      8. Terraform with borrowed keys — Terraform gets its credentials in memory.
      9. Proxmox API — And talks to the Proxmox API.
      10. Audit log — Every read lands in the audit log.
      After a restart
      1. The seal — The server boots and OpenBao stays sealed.
      2. Seal watcher — The watcher sees it within 10 minutes.
      3. Element — It posts to Element: «OpenBao is sealed».
      4. Me — I see it.
      5. My password manager — I take the key from my password manager.
      6. Unseal from the browser — I paste it into the web page, behind the two-factor gate.
      7. The seal — Unsealed. «Unsealed again» arrives.
      8. The deploy runner — And deployments carry on where they were.
      Changing a secret
      1. Me — I need to change a service's password.
      2. Unseal from the browser — I log into OpenBao with my usual sign-on, with two-factor.
      3. Integrated storage — I create a new version of the secret in its group. The previous one is kept in case I need to go back.
      4. Audit log — The change lands in the audit log.
      5. The deploy runner — That service's next deployment already uses the new one.
      6. The repository — Nothing changes in the repository: it only has the name.
      Certificates for what doesn't face the internet
      1. The deploy runner — Ansible installs the home authority's root on every machine.
      2. Certificate authority — The authority lives in OpenBao and speaks ACME, the same protocol as Let's Encrypt, only for internal names and only from home.
      3. Unseal from the browser — The internal gate asks it for each panel's certificates and renews them on its own, every 90 days at most.
      4. Me — And my browsers trust them without warnings.
      Getting admin access back
      1. Me — No root token is stored anywhere, on purpose.
      2. My password manager — When one is needed, a temporary one is generated with the unseal key.
      3. The seal — OpenBao ships with that path closed: it's opened briefly in the configuration and restarted, which seals it again.
      4. Read-only policy — With the temporary token I make the change, and nothing else.
      5. Audit log — I revoke it: the audit log keeps the record.
      6. The seal — And I close the path again. I check it: it must refuse once more.
      If OpenBao is lost entirely
      1. Backup disk — The container breaks beyond repair.
      2. Daily copy — It's restored from the daily copy.
      3. Integrated storage — Its data is still encrypted.
      4. The seal — It's unsealed with the same key as always.
      5. My password manager — And if both the copy and the key were lost, the two keys have their emergency copy in the password manager: it's rebuilt from scratch.

      1. Me

        The only one who can unseal it.

      2. My password manager

        Where the unseal key lives, plus an emergency copy of both keys. Off the server.

      3. Element

        The alerts room: «sealed» and «unsealed again».

      4. The repository

        It only keeps the names of each group's secrets. The values are in OpenBao.

        • Git
        • Ansible

      5. Proxmox API

        Terraform gets in with a token that now comes from OpenBao.

        • Proxmox VE

      6. Backup disk

        The container's daily copy, on another disk.

      7. Own TLS

        Its own certificate, pinned on the client: nobody can pretend to be OpenBao.

        • OpenBao · Vault
        • Linux / SSH hardening

      8. Integrated storage

        Data is encrypted on disk: without unsealing it can't be read, not even from the copy.

        • OpenBao · Vault

      9. The seal

        After every start, OpenBao hands out nothing until someone enters the key.

        • OpenBao · Vault

      10. Audit log

        Who asked for what and when, every time. Declared in the configuration.

        • OpenBao · Vault
        • Linux / SSH hardening

      11. Deployment AppRole

        The identity the control node uses to ask for the keys.

        • OpenBao · Vault

      12. Bound to one origin

        That role only works from the control node. Copied to another machine, it's useless.

        • Linux / SSH hardening

      13. Short tokens

        Every token expires in 15 minutes. If one leaked, it would be worth little and from one place only.

        • OpenBao · Vault

      14. Read-only policy

        It can read the two keys and nothing else: no writing, no listing other paths.

        • OpenBao · Vault
        • Linux / SSH hardening

      15. Secrets per group

        Each Ansible group reads its own from OpenBao when deploying. There are no encrypted files left, nor a password to open them.

        • Ansible

      16. Terraform with borrowed keys

        A Terraform wrapper that fetches the credentials from OpenBao and passes them in memory.

        • Terraform

      17. The deploy runner

        Asks for the keys before every plan and every apply.

        • Terraform
        • Ansible

      18. Write role

        Scripts that create secrets use another role: it can only create and change the groups' ones, and only from the control node. It can't read the Terraform credentials.

        • OpenBao · Vault

      19. Seal watcher

        Every 10 minutes it checks whether OpenBao is sealed or not answering, and reports changes.

      20. Unseal from the browser

        Only from home or the VPN, with two-factor. I log in with the same single sign-on as the other panels.

        • Linux / SSH hardening

      21. Daily copy

        The whole container, every night, on the backup server. The data is already encrypted.

        • LXC

      22. Certificate authority

        Signs the internal services' certificates over ACME, like Let's Encrypt. Its root is installed on every machine.

        • OpenBao · Vault
        • Linux / SSH hardening

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      Terraform
      A wrapper that passes it the credentials from OpenBao in memory, in PR deploys too.
      Ansible
      Each group reads its secrets from OpenBao when deploying; the inventory only has their names.
      Proxmox VE
      Its API token comes from OpenBao on every plan.
      LXC
      An unprivileged container just for this, copied daily to another disk.
      Git
      The repository only has the secrets' names: no values, no encrypted files.
      Linux (Debian, Alpine, Arch)
      Debian with the official package and its checksum verified.
      Linux / SSH hardening
      A firewall that closes everything but the essentials, a certificate pinned on the client, the web page only behind two-factor and no stored root token.
      OpenBao · Vault
      Master keys behind a read-only AppRole bound to its origin, 15-minute tokens, auditing, its own TLS and manual unsealing.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        The audit log that wouldn't switch on

        15-10-2026 · OpenBao

        The setup script enabled auditing through the API, as half of Vault's documentation says. OpenBao rejected it.

        What do you think was going on?

        Show the fix
        Cause
        OpenBao 2.x no longer lets you enable auditing through the API: it's a security measure in that branch, and it's declared in the configuration file.
        Fix
        The audit log goes in the configuration file Ansible manages, and out of the setup script.
        Lesson
        A fork isn't a clone: what works for Vault has to be checked on OpenBao. And what goes in the configuration is versioned with everything else.

      2. #2

        No root token: how do you change a password?

        15-10-2026 · OpenBao

        I had to change my admin user's password. The root token had been revoked after installing, and the usual way to generate another answered «not allowed».

        What do you think was going on?

        Show the fix
        Cause
        OpenBao 2.x ships with the unauthenticated paths for generating a root token closed by default. That way nobody can try it from the network.
        Fix
        Open it briefly in the configuration, generate the token with the unseal key, make the change, revoke it and close it again. Check that it refuses once more. It's written down step by step.
        Lesson
        Dangerous things shipping closed is good, as long as the way to open them is written down before you need it.

      How it was built

      1. 15-10-2026

        The keys leave the disk

        • Its own container with the official package and pinned checksum, its own TLS, a firewall that closes everything but the essentials, and a daily copy.
        • Both keys inside, checked identical to the old ones; then the plain-text files deleted and the root token revoked.
        • The control node asks with a read-only role and 15-minute tokens.

      2. 15-10-2026

        The seal, by hand

        • A seal watcher with alerts in Element.
        • Unsealing from the browser, behind the internal gate with two-factor.
        • Monitoring and security, like the rest of the fleet.

      3. 15 to 18-10-2026

        All the secrets inside

        • Over 90 secrets from every group move from the repository to OpenBao, checked one by one by fingerprint. Their names stay in the repository.
        • No encrypted files left, nor a password to open them.
        • Logging in with the home sign-on, and a write role only for creating secrets.
        • A certificate authority of my own, with ACME, for internal services.

      4. 15-10-2026

        First rekey

        • A new password for every encrypted secret, with the content checked.
        • Getting admin access back, written down step by step.

      Decisions (and why)

      • Unsealing

        By hand, always

        Auto-unseal without outside services means keeping the key on the server itself, and then OpenBao protects against nothing with disk access. The price: after a power cut there are no deployments until I open it. Services keep running.

      • What goes in first

        The master keys, then everything

        They open everything else. Moving every service's secrets at once would have been a huge change: first a test group, then the rest, checking that each value read from OpenBao was identical to the old one.

      • How they're requested

        Read-only, from one place, for 15 minutes

        If something leaked from the control node, an expired token or one copied to another machine is useless, and the audit log says what was read.

      • The root token

        None stored

        It's generated when needed, with the unseal key, and revoked afterwards. A stored root is the key to all keys.

      • OpenBao or Vault

        OpenBao

        It's the open version of Vault, openly licensed and maintained under the Linux Foundation. For what I use it's the same.

      What I learned

      • A key on the disk of the machine that uses it isn't a key kept safe.
      • What the seal stops is change, not services: a good price.
      • The emergency procedure is written before the emergency.

      What's next

      • An off-server recovery kit with the keys that must never be lost.
      • Logging in with passkeys, no password.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my own documentation: no addresses, paths or internal names, and of course no secrets at all.