Proxmox starts no containers after a power cut: LVM-thin left half-activated

Symptom

The power went and came back. The server booted on its own, as it should, but every container and virtual machine stayed stopped. Not even the documentation could be opened: it lived in one of those containers too.

In the log, every ten seconds, the same message:

activating LV 'pve/data' failed: Activation of logical volume pve/data is prohibited while logical volume pve/data_tmeta is active.

What was going on

The containers' disks live on an LVM thin pool. An abrupt shutdown leaves it flagged as «needs check», and on the next boot thin_check runs over its metadata.

That check, over 8 GB of metadata on a mirror of spinning disks, took 65 to 90 minutes: random reads at 50-120 KB/s. And when it finished it left the pool's two internal volumes (data and metadata) active separately, which prevents activating the whole pool.

To top it off: Proxmox attempts the containers' automatic start just once, when the host boots. Since the storage wasn't there at that moment, they failed and aren't retried.

How to spot it

• journalctl -b 0 | grep -i 'activating LV' → the message repeating.
• lvs -a -o+lv_active → data_tdata and data_tmeta active, but data (the pool) isn't.
• ps aux | grep thin_check → if a check is running, let it finish (and watch /proc/<pid>/io to see it's still reading).
• pct list and qm list → everything «stopped».

Fix

Retrying the pool activation doesn't work: it keeps failing even after the check has finished. First deactivate the two loose parts so LVM mounts them back inside the pool:

lvchange -an pve/data_tmeta
lvchange -an pve/data_tdata
lvchange -ay pve/data
lvs -a -o+lv_active   # data must show as active

Then, start them by hand

Even with automatic start enabled on every container, they have to be started one by one (pct start / qm start): the service that starts them only tries once per host boot.