Storage resilience: alerting, backups, and the two-node Longhorn bottleneck #39

New issue

Open

opened 2026-08-15 10:40:16 +00:00 by andre · 0 comments

andre commented

2026-08-15 10:40:16 +00:00

Owner

Background: what actually happened

2026-08-02 / 08-04 — quartz64's AHCI controller (0000:01:00.0) logged ATA link errors. 1 event, then 4. Nobody saw them.
2026-08-13 12:27 — the controller dropped off the PCIe bus (SErr 0xffffffff, AHCI controller unavailable!). Both its disks went with it; SMART on both SSDs was PASSED, so this is the adapter, not the media. Two volumes with numberOfReplicas: 1 went faulted, the other 14 went degraded.
2026-08-15 ~07:28 — quartz64 came back and Longhorn began rebuilding all degraded replicas onto it at once (concurrent-replica-rebuild-per-node-limit: 5, no v1 bandwidth limit). During that, bigal lost its iSCSI session to atlas. Since bigal's root filesystem is that LUN, EXT4 aborted its journal and remounted read-only, kubelet could no longer create pod directories, and the Postgres primary had nowhere to run — the create-default-disk=config nodeSelector plus required anti-affinity left exactly two eligible nodes, one broken and one occupied.

1. No alerting

There is no Alertmanager in the cluster. prometheus-server has alerting_rules.yml: {}, and the Loki ruler runs without rule storage, so log-based alerts cannot fire either.

The 08-02 and 08-04 link errors were sitting in Loki the whole time. Eleven days of warning went unread.

Deploy Alertmanager with a delivery route that actually reaches a phone
Loki ruler alert on {transport="kernel"} |~ "SError|AHCI controller unavailable|Aborting journal|critical medium error|Remounting filesystem read-only"
Longhorn alerts on longhorn_volume_robustness (2 = degraded, 3 = faulted) and longhorn_disk_status
Standing warning for any volume with numberOfReplicas: 1 — that is the difference between degraded and faulted

2. No backups

spec.backup on the CNPG cluster is empty, there are no ScheduledBackups, and the Longhorn backuptargets/default has an empty URL and AVAILABLE: false. There is no WAL archiving.

Consequence during the incident: the replica had been down 42 h, the WAL segment it needed (0000000900000018000000ED) was long recycled, and it could never catch up. It had to be rebuilt from scratch. Had the primary's volume also been lost, everything was gone.

A manual pg_dumpall was taken at 2026-08-15T10:13Z and verified (81 MB, all 6 databases, all 9 roles). That is a one-off snapshot, not a process.

Configure a Longhorn backup target (off-cluster / off-site)
Enable CNPG WAL archiving + ScheduledBackup
Verify a restore actually works — an untested backup is a guess

3. Longhorn storage lives on only two nodes

bigal       2 disks, 1967 GB
quartz64    2 disks, 1455 GB
atlas       0 disks
feather     0 disks
orangepi5   0 disks

Every replica in the cluster is on bigal or quartz64, and both have the same class of hardware (a PCIe AHCI adapter on an ARM SBC). Losing one degrades everything; losing both loses everything.

infrastructure/cloudnative-pg/cluster/cluster.yaml already documents the consequence for Postgres and names the fix: "widen it by giving another node a disk, which also gives it this label."

Give feather or orangepi5 a Longhorn disk
Raise numberOfReplicas above 1 for volumes that matter (homelab-postgres-* are still at 1)
Throttle rebuilds so recovery cannot saturate the network again — concurrent-replica-rebuild-per-node-limit, and a v1 replica-rebuilding-bandwidth-limit (currently only set for v2)

Also worth a look

quartz64's AHCI controller is unchanged and still suspect — the disks came back, the adapter's history did not
Paperless restarts in a loop: its startup probe is a celery inspect ping that times out while the single worker is busy with process_mail_accounts (~187 s at startup) — unrelated to this outage, but it is down
Orphaned PV pvc-f413d6be (40 Gi, Released, Retain) still holds the old replica's data on quartz64

Follow-up from the outage on 2026-08-15, which took down Forgejo, Immich, Paperless and Vikunja for roughly two hours. Three gaps made it both possible and hard to diagnose. Each is independently actionable. ## Background: what actually happened - **2026-08-02 / 08-04** — quartz64's AHCI controller (`0000:01:00.0`) logged ATA link errors. 1 event, then 4. Nobody saw them. - **2026-08-13 12:27** — the controller dropped off the PCIe bus (`SErr 0xffffffff`, `AHCI controller unavailable!`). Both its disks went with it; SMART on both SSDs was `PASSED`, so this is the adapter, not the media. Two volumes with `numberOfReplicas: 1` went `faulted`, the other 14 went `degraded`. - **2026-08-15 ~07:28** — quartz64 came back and Longhorn began rebuilding all degraded replicas onto it at once (`concurrent-replica-rebuild-per-node-limit: 5`, no v1 bandwidth limit). During that, bigal lost its iSCSI session to atlas. Since bigal's root filesystem *is* that LUN, EXT4 aborted its journal and remounted read-only, kubelet could no longer create pod directories, and the Postgres primary had nowhere to run — the `create-default-disk=config` nodeSelector plus required anti-affinity left exactly two eligible nodes, one broken and one occupied. ## 1. No alerting There is no Alertmanager in the cluster. `prometheus-server` has `alerting_rules.yml: {}`, and the Loki ruler runs without rule storage, so log-based alerts cannot fire either. The 08-02 and 08-04 link errors were sitting in Loki the whole time. Eleven days of warning went unread. - [ ] Deploy Alertmanager with a delivery route that actually reaches a phone - [ ] Loki ruler alert on `{transport="kernel"} |~ "SError|AHCI controller unavailable|Aborting journal|critical medium error|Remounting filesystem read-only"` - [ ] Longhorn alerts on `longhorn_volume_robustness` (2 = degraded, 3 = faulted) and `longhorn_disk_status` - [ ] Standing warning for any volume with `numberOfReplicas: 1` — that is the difference between `degraded` and `faulted` ## 2. No backups `spec.backup` on the CNPG cluster is empty, there are no `ScheduledBackups`, and the Longhorn `backuptargets/default` has an empty URL and `AVAILABLE: false`. There is no WAL archiving. Consequence during the incident: the replica had been down 42 h, the WAL segment it needed (`0000000900000018000000ED`) was long recycled, and it could never catch up. It had to be rebuilt from scratch. Had the primary's volume also been lost, everything was gone. A manual `pg_dumpall` was taken at 2026-08-15T10:13Z and verified (81 MB, all 6 databases, all 9 roles). That is a one-off snapshot, not a process. - [ ] Configure a Longhorn backup target (off-cluster / off-site) - [ ] Enable CNPG WAL archiving + `ScheduledBackup` - [ ] Verify a restore actually works — an untested backup is a guess ## 3. Longhorn storage lives on only two nodes ``` bigal 2 disks, 1967 GB quartz64 2 disks, 1455 GB atlas 0 disks feather 0 disks orangepi5 0 disks ``` Every replica in the cluster is on bigal or quartz64, and both have the same class of hardware (a PCIe AHCI adapter on an ARM SBC). Losing one degrades everything; losing both loses everything. `infrastructure/cloudnative-pg/cluster/cluster.yaml` already documents the consequence for Postgres and names the fix: *"widen it by giving another node a disk, which also gives it this label."* - [ ] Give feather or orangepi5 a Longhorn disk - [ ] Raise `numberOfReplicas` above 1 for volumes that matter (`homelab-postgres-*` are still at 1) - [ ] Throttle rebuilds so recovery cannot saturate the network again — `concurrent-replica-rebuild-per-node-limit`, and a v1 `replica-rebuilding-bandwidth-limit` (currently only set for v2) ## Also worth a look - quartz64's AHCI controller is unchanged and still suspect — the disks came back, the adapter's history did not - Paperless restarts in a loop: its startup probe is a `celery inspect ping` that times out while the single worker is busy with `process_mail_accounts` (~187 s at startup) — unrelated to this outage, but it is down - Orphaned PV `pvc-f413d6be` (40 Gi, `Released`, `Retain`) still holds the old replica's data on quartz64

andre referenced this issue from a commit

2026-08-15 10:45:10 +00:00

Record the 2026-08-15 outage against the nodeSelector pin

andre referenced this issue

2026-08-15 10:45:19 +00:00

Record the 2026-08-15 outage against the nodeSelector pin #40

andre referenced this issue

2026-08-15 14:01:13 +00:00

Design and implement a backup strategy #41

andre referenced this issue

2026-08-15 14:30:20 +00:00

Design and implement a backup strategy #41