Record the 2026-08-15 outage against the nodeSelector pin #40

Merged

andre merged 1 commit from document-storage-outage-in-cluster-nodeselector into main

2026-08-16 18:20:47 +00:00

andre commented

2026-08-15 10:45:19 +00:00

Owner

Comment-only change to infrastructure/cloudnative-pg/cluster/cluster.yaml. No behaviour change — the node.longhorn.io/create-default-disk: config pin stays exactly as it was.

The comment above that label already warned that pinning both instances to the only two disk-carrying nodes leaves "no room to reschedule if one is lost". On 2026-08-15 that came due and cost roughly two hours of downtime for Forgejo, Immich, Paperless and Vikunja. The warning was right, so it seemed worth recording what the failure actually looked like instead of leaving it hypothetical.

Two details cost real time during the incident and are the reason this note exists:

bigal's root filesystem is an iSCSI LUN served by atlas, not a local disk. Its EXT4 journal abort and read-only remount were a dropped session, not dying hardware. I initially read it as a failing SSD and went as far as comparing SMART data against quartz64's genuinely broken AHCI controller before spotting it.
The failure mode was not "Postgres broke". kubelet on bigal could no longer create pod directories, and combined with the required anti-affinity against the other instance, that left the primary with nowhere to run at all.

Dropping the label is written down as a deliberate emergency lever together with its price (the Scheduled=False flag and a volume that refuses to expand), so the next person reaching for it knows what they are trading. It was used during the incident and has since been restored.

Refs #39, which tracks the underlying gaps: no alerting, no backups, and Longhorn storage on only two nodes.

🤖 Generated with Claude Code

Comment-only change to `infrastructure/cloudnative-pg/cluster/cluster.yaml`. No behaviour change — the `node.longhorn.io/create-default-disk: config` pin stays exactly as it was. The comment above that label already warned that pinning both instances to the only two disk-carrying nodes leaves *"no room to reschedule if one is lost"*. On 2026-08-15 that came due and cost roughly two hours of downtime for Forgejo, Immich, Paperless and Vikunja. The warning was right, so it seemed worth recording what the failure actually looked like instead of leaving it hypothetical. Two details cost real time during the incident and are the reason this note exists: - **bigal's root filesystem is an iSCSI LUN served by atlas, not a local disk.** Its EXT4 journal abort and read-only remount were a dropped session, not dying hardware. I initially read it as a failing SSD and went as far as comparing SMART data against quartz64's genuinely broken AHCI controller before spotting it. - **The failure mode was not "Postgres broke".** kubelet on bigal could no longer create pod directories, and combined with the required anti-affinity against the other instance, that left the primary with nowhere to run at all. Dropping the label is written down as a deliberate emergency lever together with its price (the `Scheduled=False` flag and a volume that refuses to expand), so the next person reaching for it knows what they are trading. It was used during the incident and has since been restored. Refs #39, which tracks the underlying gaps: no alerting, no backups, and Longhorn storage on only two nodes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

andre added 1 commit

2026-08-15 10:45:19 +00:00

Record the 2026-08-15 outage against the nodeSelector pin

validate / manifests (push) Successful in 1m11s

Details

validate / manifests (pull_request) Successful in 1m48s

Details

f6635eb3cc

The comment above this label already warned that pinning both instances to
the only two disk-carrying nodes leaves "no room to reschedule if one is
lost". On 2026-08-15 that came due and cost roughly two hours of downtime
for forgejo, immich, paperless and vikunja, so it is worth writing down what
the failure actually looked like rather than leaving it as a hypothetical.

Two details were not obvious under time pressure and are the reason this
note exists. bigal's root filesystem is an iSCSI LUN served by atlas, not a
local disk, so its EXT4 journal abort and read-only remount were a dropped
session rather than dying hardware — an easy and costly misdiagnosis. And
the resulting failure was not that Postgres broke, but that kubelet could
no longer create pod directories there, which combined with the required
anti-affinity left the primary with nowhere to run at all.

Dropping the label is recorded as a deliberate emergency lever, along with
its price, so the next person reaching for it knows what they are trading.

Refs #39

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

andre merged commit d25e5f61f5 into main

2026-08-16 18:20:47 +00:00

andre referenced this pull request from a commit

2026-08-16 18:20:48 +00:00

Merge pull request 'Record the 2026-08-15 outage against the nodeSelector pin' (#40) from document-storage-outage-in-cluster-nodeselector into main

andre deleted branch document-storage-outage-in-cluster-nodeselector

2026-08-16 18:20:53 +00:00