Stop the CNPG probes from killing instances that are merely busy #43

Merged

andre merged 2 commits from tune-cnpg-probes into main 2026-08-16 18:22:24 +00:00

Conversation 0 Commits 2 Files changed 2 +54 -2

andre commented 2026-08-16 12:58:37 +00:00

Owner

Copy link

Adds spec.probes to the Postgres cluster. Two probes were tuned in opposite directions, because they mean opposite things.

Liveness: it killed a healthy primary

On 2026-08-15 the primary was shut down by its own liveness probe. The isolationCheck lets a primary step down when it can reach neither the API server nor its replicas — a good property, since it prevents split-brain writes during a real partition. But the defaults are connectionTimeout: 1000 / requestTimeout: 1000, and during the Longhorn rebuild storm the API server simply took longer than a second to answer. The only replica was already broken, so there was no second signal to weigh against. The instance concluded it was isolated and stopped. Nothing was partitioned.

At instances: 2 this check hangs on a single signal any time the replica is unavailable — which is exactly when you least want the primary to remove itself.

isolationCheck:
  enabled: true          # kept — it is worth having
  connectionTimeout: 5000
  requestTimeout: 5000
periodSeconds: 30
failureThreshold: 6      # ~3 min sustained, instead of seconds

Startup: it prevents a replica from ever catching up

An instance replaying WAL answers the database system is not yet accepting connections. The default pg_isready strategy counts that as a failure, so a tight threshold makes kubelet kill the pod, recovery restarts from the beginning, and a slow replay turns into a loop that never converges.

startup:
  periodSeconds: 10
  failureThreshold: 60   # 10 minutes of catch-up

Readiness: deliberately untouched

Readiness is the one probe that should stay strict, because failing it only pulls an instance out of the read services rather than ending it.

readiness.type: streaming with a maximumLag would express "connected but lagging" far more precisely than pg_isready, and it is the semantically right answer. It is not in this PR because I could not confirm how the streaming strategy behaves on a primary, and if it ever marks the primary not-ready the homelab-postgres-rw service loses its endpoints — an outage produced by the change meant to prevent outages. Worth doing, worth verifying on a scratch cluster first.

Validation

kubectl apply --dry-run=server   → cluster.postgresql.cnpg.io/homelab-postgres configured
kubectl diff                     → only the probe fields change, nothing else

Merging triggers a rolling restart of both instances, so it is worth landing at a deliberate moment rather than alongside other work.

Refs #39

🤖 Generated with Claude Code

Adds `spec.probes` to the Postgres cluster. Two probes were tuned in opposite directions, because they mean opposite things. ## Liveness: it killed a healthy primary On 2026-08-15 the primary was shut down by its own liveness probe. The `isolationCheck` lets a primary step down when it can reach neither the API server nor its replicas — a good property, since it prevents split-brain writes during a real partition. But the defaults are `connectionTimeout: 1000` / `requestTimeout: 1000`, and during the Longhorn rebuild storm the API server simply took longer than a second to answer. The only replica was already broken, so there was no second signal to weigh against. The instance concluded it was isolated and stopped. Nothing was partitioned. At `instances: 2` this check hangs on a single signal any time the replica is unavailable — which is exactly when you least want the primary to remove itself. ```yaml isolationCheck: enabled: true # kept — it is worth having connectionTimeout: 5000 requestTimeout: 5000 periodSeconds: 30 failureThreshold: 6 # ~3 min sustained, instead of seconds ``` ## Startup: it prevents a replica from ever catching up An instance replaying WAL answers `the database system is not yet accepting connections`. The default `pg_isready` strategy counts that as a failure, so a tight threshold makes kubelet kill the pod, recovery restarts from the beginning, and a slow replay turns into a loop that never converges. ```yaml startup: periodSeconds: 10 failureThreshold: 60 # 10 minutes of catch-up ``` ## Readiness: deliberately untouched Readiness is the one probe that should stay strict, because failing it only pulls an instance out of the read services rather than ending it. `readiness.type: streaming` with a `maximumLag` would express "connected but lagging" far more precisely than `pg_isready`, and it is the semantically right answer. It is not in this PR because I could not confirm how the `streaming` strategy behaves on a **primary**, and if it ever marks the primary not-ready the `homelab-postgres-rw` service loses its endpoints — an outage produced by the change meant to prevent outages. Worth doing, worth verifying on a scratch cluster first. ## Validation ``` kubectl apply --dry-run=server → cluster.postgresql.cnpg.io/homelab-postgres configured kubectl diff → only the probe fields change, nothing else ``` Merging triggers a rolling restart of both instances, so it is worth landing at a deliberate moment rather than alongside other work. Refs #39 🤖 Generated with [Claude Code](https://claude.com/claude-code)

andre added 1 commit 2026-08-16 12:58:37 +00:00

Stop the probes from killing instances that are merely busy ... 47881cc79c

On 2026-08-15 the liveness isolationCheck shut down a healthy primary. A
Longhorn rebuild storm pushed API server responses past the 1000 ms default
timeout, and because the only replica was already broken the instance had no
second signal to check against, so it concluded it was isolated and stepped
down. Nothing was actually partitioned. The check is worth keeping — it is
what prevents split-brain writes during a real partition — but at
instances: 2 it depends on a single signal whenever the replica is down, and
a second of latency on a loaded ARM cluster is not evidence of a partition.
Timeouts go to 5000 ms and it now takes roughly three minutes of sustained
failure rather than seconds.

The startup probe has the opposite problem. An instance replaying WAL
answers "the database system is not yet accepting connections", which
pg_isready counts as a failure; a tight threshold then kills the pod and
recovery restarts from the beginning, so an instance that is slow to catch
up can never catch up at all. It now gets ten minutes.

Readiness is deliberately left alone. It is the one probe that should stay
strict, because failing it only removes an instance from the read services
instead of ending it. Switching it to the streaming strategy with a
maximumLag would express replica lag more precisely, but its behaviour on a
primary needs verifying on a scratch cluster first — getting it wrong empties
the -rw service, which is the very outage this is meant to prevent.

Refs #39

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

andre added 1 commit 2026-08-16 18:15:47 +00:00

Let the instances reach each other's status server ... bcf5b6ca71

The isolationCheck the previous commit tunes asks whether a primary can
still reach the API server *and* its replicas before deciding it is
partitioned. Its peer half has never worked here: the policy allowed 5432
between instances but never 8000, in either direction, so the replica probe
could only ever time out. The check has therefore always been a single
signal wearing the costume of two.

That is the missing half of the 2026-08-15 story. Raising the timeouts stops
a slow API server from being mistaken for a partition, but as long as the
peer check cannot succeed, one slow signal is still the whole decision.

Found while chasing "Instance connectivity error" logged every ten seconds
by an instance that had been unable to rejoin for six hours; a direct test
from one instance to the other confirmed 5432 open and 8000 blocked.

Scoped to pods carrying cnpg.io/cluster rather than to the namespace, so the
pgbench and fio jobs that share it gain nothing. Both the ingress and egress
halves are needed — one without the other changes nothing at all.

Refs #39

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

andre merged commit e6c13cfdf9 into main 2026-08-16 18:22:24 +00:00

andre referenced this pull request from a commit 2026-08-16 18:22:24 +00:00

Merge pull request 'Stop the CNPG probes from killing instances that are merely busy' (#43) from tune-cnpg-probes into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

andre/homelab-gitops!43

No description provided.