TPTito Pinardo Gutiérrez
← Back to the rack
Español
1U · OKDOKD 4.19 · 3 nodes · 33 operators Lab

Slot 22 of 25 in the rack · Lab

1U unit · Lab

OKD cluster (OpenShift)

3/3 nodes · 33/33 operators

To practise OpenShift for real: three control-plane nodes that also run workloads, installed with the Agent-Based Installer on VMs created by Terraform. Getting to 3 of 3 nodes ready and 33 of 33 healthy operators took a whole session of failures, almost all because of spinning disks.

Why
OpenShift is what many companies run on top of Kubernetes. Installing it yourself teaches what the installer does underneath.
How
Terraform creates the VMs; the installer generates an ISO with each node's network configuration and the cluster assembles itself from there.
The challenge
Everything is designed for SSD. On spinning disks, three times a default timeout fell short.
  • 10configurations tried before blaming the ISO
  • 16 → 32 GBof RAM per node for the install
  • 6.6 setcd writes, against 200 ms
  • 33/33healthy operators at the end

Its code, in the open Public version of the repo: no IPs or secrets.

Inside the cluster

Three virtual machines acting as both control plane and workers. Pick a flow to see the install or an application.

Flows Pick one and follow the packet.

Outside the cluster Three VMs on Proxmox compact cluster Prepare Nodes Control plane Applications My laptop Home DNS Installer Registry HDDs Configuration ISO Terraform VMs Boot Node 0 Node 1 Node 2 Live env etcd API VIP Operators rpm-ostree Apps VIP Console Test app Route
Hermesvisiting the unit

Click a part or pick a flow. If you came from a skill, the parts where I use it are already lit.

      The install
      1. My laptop — I write the cluster configuration.
      2. Cluster configuration — With each node's network.
      3. openshift-install — The installer puts it into an ISO.
      4. Installer ISO — The boot image is ready.
      5. VMs created with Terraform — Terraform creates the three VMs.
      6. Boot order — They boot from the ISO.
      7. Node 0 (rendezvous) — Node 0 coordinates.
      8. Image registry — Nodes download the system.
      9. Node 1 — It's written to node 1's disk.
      10. Node 2 — And node 2's.
      11. etcd — etcd forms its group.
      12. API virtual address — The API virtual address comes up.
      13. Cluster operators — And the 33 operators end up healthy.
      An ISO that wouldn't boot
      1. Installer ISO — The newest version's ISO hangs while loading the boot.
      2. Boot order — I try firmware, CD buses, CPU types, manual and Terraform configuration.
      3. Spinning disks — Even plain QEMU, no Proxmox and no acceleration: same failure.
      4. Boot order — It isn't my server: it's the ISO.
      5. openshift-install — I switch to the previous stable version.
      6. Installer ISO — I generate the new ISO with the same configuration.
      7. Node 0 (rendezvous) — And the node boots.
      I open the test app
      1. My laptop — From home I open the lab application.
      2. Home DNS — The home DNS resolves its name to the applications virtual address.
      3. Applications virtual address — The cluster's ingress receives the request.
      4. Route — The route sends it to the service.
      5. Test application — And the dashboard answers from the cluster.
      I deploy to the cluster
      1. My laptop — Ansible prepares the application's resources.
      2. Test application — The application is a service dashboard.
      3. API virtual address — They're sent to the cluster API.
      4. Cluster operators — The operators do the rest.
      5. Test application — And the application is published.
      What a spinning disk costs
      1. Spinning disks — The disks take a while to confirm each write.
      2. etcd — etcd complains: writes of up to 6.6 seconds.
      3. Immutable system — Swapping the system image takes longer than systemd waits.
      4. Node 0 (rendezvous) — A node is left on the old image.
      5. Cluster operators — And the operators take a while to agree. With patience and longer timeouts, they end up healthy.

      1. My laptop

        Everything is generated and the install launched from here.

      2. Home DNS

        A zone of its own for the lab: the cluster's API and applications.

        • Technitium DNS

      3. openshift-install

        Generates the installer ISO and follows the install from outside.

        • OpenShift (OKD)

      4. Image registry

        Where nodes download the system and components.

      5. Spinning disks

        The bottleneck of the whole install.

        • Proxmox VE

      6. Cluster configuration

        Name, networks, virtual addresses and each node's network, in two files.

        • OpenShift (OKD)

      7. Installer ISO

        A boot image with all the configuration inside.

        • OpenShift (OKD)

      8. VMs created with Terraform

        Three VMs with host CPU, 32 GB of RAM and the ISO as first boot.

        • Terraform
        • Proxmox VE

      9. Boot order

        ISO first, then disk. If the ISO is still first when it finishes, the node installs itself again.

        • Proxmox VE

      10. Node 0 (rendezvous)

        Coordinates the other two's install and then installs itself.

        • OpenShift (OKD)
        • Kubernetes / K3s

      11. Node 1

        Control plane and workloads.

        • OpenShift (OKD)
        • Kubernetes / K3s

      12. Node 2

        Control plane and workloads.

        • OpenShift (OKD)
        • Kubernetes / K3s

      13. Install environment

        While installing, each node runs from the ISO with a temporary file system in RAM.

        • OpenShift (OKD)

      14. etcd

        The cluster's database. Very sensitive to how long the disk takes to confirm each write.

        • Kubernetes / K3s

      15. API virtual address

        An address that moves from node to node: not another machine.

        • OpenShift (OKD)

      16. Cluster operators

        The 33 components that make up OpenShift. The cluster is healthy when all say available and not degraded.

        • OpenShift (OKD)

      17. Immutable system

        Each node swaps its whole system image on reboot. If the swap doesn't finish in time, it's discarded.

        • OpenShift (OKD)

      18. Applications virtual address

        The way into everything published on the cluster.

        • OpenShift (OKD)

      19. Web console

        The OpenShift console.

        • OpenShift (OKD)

      20. Test application

        A service dashboard deployed on the cluster with Ansible, to check it can really be used.

        • Kubernetes / K3s
        • Ansible

      21. Route

        OpenShift's way of publishing a service by name.

        • OpenShift (OKD)

      What I use from my skill tree

      Pick a skill: the parts where I use it light up on the map and I tell you what for.

      OpenShift (OKD)
      OKD installed with the Agent-Based Installer: ISO with network configuration, virtual addresses for API and applications, operators and routes.
      Kubernetes / K3s
      Under OpenShift: etcd, kubelet and the test app's resources.
      Terraform
      Creates the three VMs with their CPU, RAM, disks and boot order.
      Ansible
      Deploys the test application on the cluster.
      Proxmox VE
      The cluster's three VMs, on the server's disks.
      Technitium DNS
      A lab zone of its own in the home DNS for the API and applications.

      Incident log

      Real failures. Read the symptom, pick the cause you think it was and see how it was fixed.

      1. #1

        The VMs wouldn't boot the ISO

        14-09-2026 · install

        The VMs always hung at the same point while loading the boot from the installer ISO.

        What do you think was going on?

        Show the fix
        Cause
        It reproduced with 10 different configurations, including plain QEMU with no Proxmox and no KVM, while another small ISO booted fine. That version's ISO had a bug in its boot catalogue.
        Fix
        Use the installer's previous stable version, with the same configuration. It booted first time with host CPU.
        Lesson
        Before blaming something external, rule out your own side one variable at a time. And once all of that's ruled out, change version instead of trying more.

      2. #2

        «No space left on device» on a node with free disk

        14-09-2026 · install

        After writing the system to all three disks, the install went silent. The API virtual address never came up.

        What do you think was going on?

        Show the fix
        Cause
        The coordinating node stays in the ISO environment until the end, keeping everything in a temporary RAM file system half the size of memory. With 16 GB that was 8 GB, and the downloaded images exceeded 17.
        Fix
        Raise the VMs to 32 GB of RAM with Terraform and reboot.
        Lesson
        Minimum requirements are for normal operation, not the install's peak. And on slow disks the peak lasts longer.

      3. #3

        etcd in a restart loop

        14-09-2026 · control plane

        One etcd member had been unhealthy for over 37 minutes, and the rest of the control plane with it.

        What do you think was going on?

        Show the fix
        Cause
        etcd confirms each write on disk before carrying on. With every operator starting at once on spinning disks, the wait exceeded what it tolerates.
        Fix
        With a clear diagnosis, give it one more window before treating it as a hardware limit. It stabilised on its own and didn't restart again.
        Lesson
        Tell «broken» from «slow»: the logs said exactly what was slow and by how much. Waiting was a decision, not a hope.

      4. #4

        A node that wouldn't join

        14-09-2026 · nodes

        Two nodes were ready, but the third wouldn't finish joining: its network wouldn't start and the Kubernetes service failed in a loop.

        What do you think was going on?

        Show the fix
        Cause
        On reboot each node moves to the new system image. Copying the configuration took about 5 minutes, and that service's maximum stop time was exactly 5 minutes: it was discarded milliseconds before finishing.
        Fix
        Raise that time to 20 minutes and redo the image swap by hand with the exact same identifier. The node joined.
        Lesson
        Third time in a day a default timeout meant for SSD fell short. When something «hangs» on a slow disk, check the timeouts first.

      Told in full in an article:

      How it was built

      1. September 2026

        Preparation

        • A DNS zone for the lab and VMs with Terraform.
        • Cluster configuration with static networking per node.

      2. 14-09-2026

        The install

        • The new version's ISO ruled out after ten tests; the previous stable one boots.
        • RAM, boot order, slow etcd and a systemd timeout, solved one by one.
        • 3/3 nodes ready, 33/33 healthy operators and the console reachable.

      3. 14-09-2026

        Using it

        • A test application deployed with Ansible and published with a route.

      Decisions (and why)

      • Which topology

        Compact: three nodes doing everything

        With three VMs you get a control plane with quorum and room for workloads. More nodes won't fit on these disks.

      • Which version

        The previous stable one, not the latest

        The latest had a boot bug in its ISO. To learn OpenShift, the version that works beats the newest one.

      • How the VMs are created

        With Terraform, like everything else

        So the whole cluster can be destroyed and recreated with no manual steps, and RAM or boot order change with a commit.

      What I learned

      • Rule out your own side, variable by variable, before blaming anything external.
      • On slow disks, default timeouts are the first suspect.
      • Tell «broken» from «slow» by reading the logs.

      What's next

      • Fix a node left degraded after its image was fixed by hand.
      • SSDs for etcd: the only real fix.

      Shall we talk?

      I can walk through any of these decisions in detail in an interview.

      Download CV Get in touch ← Back to the rack

      Rebuilt from my documentation: no addresses, domains or internal names.