TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · GITForgejo · CI · deploy In production

Slot 19 of 25 in the rack · In production

1U unit · Code & delivery

My own Git, with CI and deployment

From push to the site: hands off

The infrastructure lived in a repository with no interface and nobody reviewed anything before it went in: a badly formatted file or a plain-text secret was copied as it was. Now everything goes through Forgejo. A runner with no secrets checks every push, every infrastructure change comes in by PR with its plan in view, and this site deploys itself when its CI goes green.

Where the code lives
At home. GitHub is the shop window: the sanitised copy of the infrastructure and this site, which only gets what's already deployed.
The rule
Nothing goes in without green CI, and no infrastructure is applied without seeing the plan first. I approve the last step with one click.
The alerts
Every run, good or bad, reaches Element encrypted. A failure is posted as soon as the first job fails.
  • 0secrets on the CI runner
  • 1 clickto apply an infrastructure change, with its plan
  • 8 of 8alerts delivered to Element on day one
  • ~1.3 GBof memory to build this site in CI

Where the code starts, inside

Every change to the infrastructure and to this site comes in here, and here it's decided whether it goes out. Pick a walkthrough.

Flows Pick one and follow the packet.

Outside Forgejo and its runners · containers on Proxmox the Git server, a runner with no secrets and one that deploys Forgejo CI runner: checks Deploy runner: applies Around it My machines Element Backup This site GitHub Git over SSH Web with 2FA PRs, branches Webhooks CI runner TF + lint Rust + PG gitleaks PR plan One click Saved plan Leak check Alerts Encrypted relay Renovate Daily copy
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      An infrastructure change, by PR
      1. My machines — From any of my machines I create a branch: the main one takes no direct pushes.
      2. Git over SSH — I push it to Forgejo over SSH.
      3. PRs and a protected branch — I open the PR.
      4. CI runner — The CI runner picks it up: it has no secrets and no access to the homelab.
      5. Terraform, lint and Trivy — Terraform formatting and validation, and ansible-lint.
      6. gitleaks — gitleaks over the whole history and a check that every vault is encrypted.
      7. The PR's plan — Meanwhile, the deploy runner works out the plan: what Terraform touches and which playbooks, in dry-run.
      8. PRs and a protected branch — The plan is posted as a PR comment: I see what's touched before merging.
      9. Approve with one click — I merge. The main branch's plan comes out and I approve it with one click.
      10. Saved plan — That saved plan is applied. If anything changed in between, Terraform refuses.
      11. Alert receiver — The result goes out through the webhook.
      12. Matrix relay — Encrypted, to the CI/CD room.
      13. Element — And reaches my phone.
      This site, from a push to the internet
      1. My machines — I push a change to this site.
      2. Git over SSH — It lands on Forgejo, the repository's origin.
      3. CI runner — The CI runner picks it up.
      4. Rust and PostgreSQL — Formatting, clippy, tests against PostgreSQL, the build, valid content and valid minified JS.
      5. gitleaks — And gitleaks over the whole history.
      6. Leak check — On green, the deploy runner reviews what isn't public yet.
      7. This site — It builds, deploys with Ansible and waits for the site to answer.
      8. GitHub, the public copy — Only then does it push that commit to GitHub.
      9. Alert receiver — The webhook reports the result.
      10. Element — And I see it on my phone.
      Two commits in the red
      1. My machines — I add my machines' SSH keys to the configuration.
      2. Git over SSH — Push.
      3. CI runner — CI picks it up.
      4. Terraform, lint and Trivy — ansible-lint fails: lines over 160 characters. It's the keys.
      5. Alert receiver — The alert goes out as soon as the first job fails, without waiting for the rest.
      6. Element — ❌ in Element, with the commit and the link to the run.
      7. My machines — I fix it: an exception for those lines and only those.
      8. CI runner — Green.
      9. Element — ✅ in Element. Eight alerts went out that day and all eight arrived.
      Versions bump themselves, but don't apply themselves
      1. Renovate — Monday morning: Renovate finds a new Traefik version.
      2. PRs and a protected branch — It opens a PR with the new number and its SHA-256, taken from the official checksum file. If the release doesn't publish one, there's no PR.
      3. CI runner — CI checks it like any other.
      4. Terraform, lint and Trivy — Formatting, lint and secrets.
      5. The PR's plan — The PR's plan says which playbook would run and how many tasks would change.
      6. Approve with one click — I merge and approve with one click.
      7. Saved plan — That plan is applied.
      8. Alert receiver — The result goes out through the webhook.
      9. Element — And reaches Element. Major versions wait for my approval in an issue before a PR is even opened.
      Every night: did anyone change something by hand?
      1. Saved plan — In the early hours, the deploy runner starts from the last deployment.
      2. The PR's plan — It works out the Terraform plan. If anything shows up, someone changed something outside the code: real drift.
      3. Alert receiver — It alerts. Also if the main branch has undeployed changes.
      4. Element — And if everything matches, silence: nothing arrives.
      What's copied and what's rebuilt
      1. Git over SSH — The repositories and their history live in Forgejo.
      2. Daily copy — Every night, a copy of the container to a separate disk.
      3. CI runner — The CI runner isn't copied: it keeps nothing.
      4. Terraform, lint and Trivy — It's rebuilt with its playbook, tools at pinned versions with checked checksums.
      5. Rust and PostgreSQL — And the same goes for Rust and PostgreSQL for this site.

      1. My machines

        The PC, the laptop and the control node: the only ones with a key to push.

      2. Element

        A room of its own for CI/CD, apart from incidents: one message per run, with the commit and the link.

      3. Backup repository

        The same as always: every machine still pushes each commit to it. It has its own unit.

        • Git

      4. This site

        A Rust binary: what the runner deploys when everything goes green.

        • Rust

      5. GitHub, the public copy

        It only gets commits that have already passed CI, the leak check and the deployment.

        • Git

      6. Git over SSH

        Its own port for Git, separate from the admin SSH. Only registered keys get in.

        • Git
        • Forgejo · Actions

      7. Web interface

        Only from home or the VPN, behind the internal gate with two-factor and then the Forgejo login.

        • Linux / SSH hardening

      8. PRs and a protected branch

        The infrastructure's main branch takes no pushes: only PRs with green CI.

        • Forgejo · Actions

      9. Webhooks

        When each run ends, Forgejo calls out with a signed request.

        • Forgejo · Actions

      10. CI runner

        In its own container, without a single secret or any access to the homelab. If it has to read a vault, it reads a deliberately fake one.

        • Forgejo · Actions
        • LXC

      11. Terraform, lint and Trivy

        Terraform formatting and validation with no state or credentials, tflint, ansible-lint (only fails on new warnings) and Trivy, which looks for serious flaws in configuration and dependencies.

        • Terraform
        • Ansible

      12. Rust and PostgreSQL

        For this site: formatting, clippy, tests against a real PostgreSQL, the build and valid content.

        • Rust
        • PostgreSQL

      13. gitleaks

        Over the whole history, not just the last commit: what was deleted is still in the repository.

        • Linux / SSH hardening
        • Git

      14. The PR's plan

        Terraform plan and the affected playbooks in dry-run. It's posted as a PR comment, before merging.

        • Terraform
        • Ansible

      15. Approve with one click

        Merging applies nothing: the main branch's plan comes out and I approve it.

        • Forgejo · Actions

      16. Saved plan

        Exactly that plan is applied. If the state changed afterwards, Terraform refuses instead of doing something else.

        • Terraform

      17. Leak check

        Before anything leaves home: gitleaks, only my public identity as author and no addresses or internal names.

        • Linux / SSH hardening
        • Git

      18. Alert receiver

        Checks the webhook's signature and sums up the run: repository, branch, commit, duration and link.

      19. Matrix relay

        Posts the alert end-to-end encrypted to the CI/CD room.

      20. Renovate

        Every Monday it looks for new versions of everything pinned and opens one PR for each, with its official SHA-256.

        • Renovate

      21. Daily copy

        The Forgejo container, every night to a separate disk: seven dailies and four weeklies.

        • LXC

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      Terraform
      Formatting and validation in CI with no state; the PR's plan and a saved plan applied as is.
      Ansible
      ansible-lint with a baseline, affected playbooks in dry-run and this site deployed with its role.
      LXC
      Forgejo and the runner, each in its own unprivileged container; the runner is rebuilt from scratch.
      Forgejo · Actions
      Git server with PRs, protected branches, Actions and webhooks; runners in host mode, each with just what it needs, and fleet updates at the press of a button.
      Renovate
      Weekly PRs for new versions with their official checksum, grouped when they must go together and majors with prior approval.
      Git
      Origin at home, a backup on every commit, the old central read-only and GitHub as the public copy.
      Linux (Debian, Alpine, Arch)
      Debian containers with pinned versions and checked checksums for every binary.
      Linux / SSH hardening
      A runner with no secrets, gitleaks over the whole history and a leak check before publishing.
      PostgreSQL
      A socket-only PostgreSQL on the runner: each test creates its own database.
      Rust
      This site's CI: formatting, clippy, tests and the build, cached between runs.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        Jobs that never started

        14-10-2026 · Forgejo

        The runner was connected with no errors, but CI jobs sat «waiting» forever.

        What do you think was going on?

        Show the fix
        Cause
        The runner registered with no labels, so it matched no job. Adding them changed nothing: it only asks for work again when the server's task version changes, and the jobs were already queued.
        Fix
        Register it with its label from the playbook and restart it once. It's written down in the docs.
        Lesson
        «Waiting» with no errors is nearly always a matching problem, not a connection one. Labels go in from the first registration.

      2. #2

        The first CI run went red, as expected

        14-10-2026 · Terraform

        terraform validate failed on the runner but not in my terminal. It uses no state or credentials.

        What do you think was going on?

        Show the fix
        Cause
        The provider reads the user's public key with file(), and validate evaluates that function. On the runner, which has no secrets, the file didn't exist.
        Fix
        A deliberately fake file on the runner: validation passes and it opens nothing. Same for the vault password ansible-lint asks for.
        Lesson
        A CI with no secrets uncovers everything the code assumes about the machine. Better to find it in the red than keep it hidden.

      3. #3

        A secret for something switched off

        14-10-2026 · Forgejo

        Forgejo wouldn't start: it wanted an OAuth2 secret, and OAuth2 was disabled.

        What do you think was going on?

        Show the fix
        Cause
        The installed version checks that secret at startup even with the feature switched off.
        Fix
        Generate it inside the container the first time, next to the other internal secrets. It never leaves.
        Lesson
        Internal secrets are generated on the machine that uses them and never travel.

      4. #4

        A warning my terminal didn't show

        14-10-2026 · ansible-lint

        On the runner, ansible-lint raised a new warning in a role that passed clean on the control node. Same version.

        What do you think was going on?

        Show the fix
        Cause
        The control node didn't have the Kubernetes collection that role uses, and ansible-lint skipped it silently. The runner installs everything in the requirements file.
        Fix
        Fix the warning. Since then, CI is the reference: it starts from a clean install.
        Lesson
        «Works on my machine» isn't enough if my machine has gaps. That's why CI starts from scratch.

      5. #5

        A version bump with zero changes

        15-10-2026 · Renovate

        Renovate opened the PR for a new Qdrant version. CI went green, but the plan said the playbook wouldn't change a single task.

        What do you think was going on?

        Show the fix
        Cause
        The role downloaded the binary to a fixed temporary path and skipped the download if one was already there. With the previous version's binary in place, the new one was never fetched. Traefik had the same problem, and the control node's Ansible tool didn't upgrade either if one was already installed.
        Fix
        Each version in its own directory, as other roles already did, and the Ansible tool comparing the installed version with the requested one. After that, the plans showed the changes.
        Lesson
        A version bump that changes nothing is a clue, not good news. Until Renovate, those numbers were bumped by hand and nobody had noticed.

      6. #6

        Renovate saw no versions at all

        15-10-2026 · Renovate

        Renovate's first runs found no new versions of anything coming from GitHub, though there were some.

        What do you think was going on?

        Show the fix
        Cause
        Renovate only queries GitHub releases without a token when it runs on GitHub. At home, those lookups failed.
        Fix
        Use the official Docker images and PyPI packages as sources, which publish the same versions. No extra tokens to keep.
        Lesson
        Before adding a token, look for a source that doesn't need one.

      7. #7

        The first thing Trivy found

        15-10-2026 · CI

        I added Trivy to CI to watch for dependencies with known flaws. The first thing it flagged as serious wasn't a dependency at all.

        What do you think was going on?

        Show the fix
        Cause
        The API that opens a headless browser to read third-party pages ran as root. If one of those pages managed to exploit the browser, it would have root in the container.
        Fix
        Run it as an unprivileged user, the same one that already owned its folders. Checked that the browser starts just the same.
        Lesson
        A new scanner finds old debts. Better it finds them than someone else.

      Told in full in an article:

      How it was built

      1. 14-10-2026 · morning

        Forgejo and CI

        • Forgejo in its container, with its SQLite database and the interface behind the internal gate.
        • A runner with no secrets: Terraform, ansible-lint, encrypted vaults and gitleaks.
        • Every run reported to Element, encrypted.

      2. 14-10-2026 · afternoon

        Forgejo as the origin

        • Machines push to Forgejo; the old central becomes read-only.
        • Monitoring and security like the rest of the fleet, and a daily copy.
        • The full Terraform plan, clean for the first time.

      3. 14-10-2026 · night

        Deploying by PR

        • Protected main branch: only PRs with green CI.
        • The PR's plan as a comment and a saved plan on merge.
        • One-click apply, tested end to end.

      4. 14-10-2026 · night

        This site, too

        • Its origin moves to Forgejo and its CI to the same runner: Rust, PostgreSQL and Node.
        • Continuous deployment with a leak check.
        • GitHub, the public copy that only gets what's already deployed.

      5. 15-10-2026

        Versions and drift

        • Renovate: one PR per new version, with its official checksum; majors only if I approve them.
        • The first PRs uncovered three roles that didn't really upgrade, now fixed.
        • Terraform drift every night and an alert for undeployed changes.
        • The master keys leave the disk and move to OpenBao, with its own unit.

      6. 15 to 18-10-2026

        More checks, less terminal

        • tflint and Trivy in CI: it fails on any serious finding.
        • Updating the fleet without opening a terminal: a button in Forgejo, and packages on their own every Wednesday before dawn.
        • A runner of its own for the game network's plugins, with tests against real servers.

      Decisions (and why)

      • Where the code lives

        At home; GitHub as the shop window

        Internal things have no reason to leave. GitHub shows what I want to show: a sanitised copy and this site, always behind what already works.

      • Which Git server

        Forgejo

        One binary and SQLite: it runs in a gigabyte of memory. A full GitLab needs several for the same job, and memory here has other uses.

      • The runners

        Two, split by what they can touch

        The one that checks everything has nothing to steal: no secrets, no access. The one that deploys lives on the control node, which could already touch everything, and only serves its own repository.

      • Applying infrastructure

        One click, and the exact plan

        A plan that recreates a container with data gets read first. Once there's trust, update-only plans could go on their own; destructive ones, never.

      • Versions

        Automatic PRs, one-click apply

        Renovate takes away the searching and the checksums, but every bump goes through the same plan and approval as one of my changes. Major versions wait for my approval, and databases stay out: upgrading them means migrating data.

      • This site

        Continuous deployment

        If something goes wrong, it's back to the previous commit in minutes and no data is at stake. The leak check runs before deploying, not after.

      What I learned

      • A CI with no secrets uncovers what the code took for granted.
      • Seeing the plan before merging changes how you review.
      • What's public goes behind what's deployed, never ahead.

      What's next

      • Let update-only plans apply on their own.
      • Renovate for this site's dependencies too.
      • Real backups for the whole server, not just Forgejo.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my own documentation: no addresses, ports or internal names.