andre/homelab-gitops

Fork 0

Design and implement a backup strategy #41

New issue

Open

opened 2026-08-15 14:01:13 +00:00 by andre · 1 comment

andre commented

2026-08-15 14:01:13 +00:00

Owner

This issue covers designing the strategy and implementing it. Split into follow-ups once the design decisions below are settled.

Current state


Longhorn `backuptargets/default`	URL empty, `available: false`
Longhorn `RecurringJobs`	none
CNPG `spec.backup`	empty
WAL archiving	not configured
barman-cloud plugin	not installed (operator is `cloudnative-pg:1.29.2`)
Off-cluster destination	none — atlas has 38 G free on its only disk, no NFS/CIFS mounts anywhere

What actually needs protecting

299.5 G is in use across 22 volumes, but a large part of it is worthless to back up.

Tier 1 — irreplaceable, ~172 G

PVC	Used	Note
`immich/immich-library`	153.9 G	dominates everything else
`forgejo/forgejo-data`	9.9 G	git repos, this repo included
`postgres` (logical)	~4.4 G	6 databases; needs logical/WAL backup, not a volume snapshot
`paperless/paperless-data`	1.5 G
`paperless/paperless-media`	1.3 G	scanned documents
`scanservjs/scanservjs-data`	0.4 G
`vault/data-vault-0`	0.2 G	small but losing it is its own category of bad
`vikunja/vikunja-files`	0.1 G

Tier 2 — recreatable with effort, ~1 G: observability/grafana (dashboards), magellan-staging-*

Tier 3 — derivable, explicitly exclude, ~123 G

PVC	Used
`forgejo-runner/data-forgejo-runner-{0,1,2}`	47.2 + 42.5 + 28.9 G
`observability/storage-loki-0`	2.0 G
`observability/prometheus-server`	1.5 G
`immich/immich-model-cache`	1.0 G
`paperless/paperless-{redis,consume,gpt}`	0.3 G

Backing up Tier 3 would make the job 2.5× larger for no recovery value. The runner caches alone are 40 % of all used space.

Design decisions to make first

RPO/RTO per tier. Losing a day of Immich uploads is annoying; losing a day of Forgejo commits or Paperless documents is not. These probably do not want the same schedule.
Destination. Nothing local has room for 172 G. Object storage off-site (S3/B2/Wasabi) is natively supported by both Longhorn and CNPG's barman-cloud and costs roughly a euro a month at this size — likely the cheapest answer, and it satisfies off-site by construction. A local NAS would be faster to restore from but is hardware that does not exist yet. A hybrid (local for speed, off-site for disaster) is the textbook answer if the budget allows.
3-2-1, honestly. Longhorn replicas are not a backup — they live on the same two nodes, and #39 tracks fixing that. Decide what genuinely counts as a second copy here.
Encryption at rest and key custody. If backups go off-site they must be encrypted, and the key must survive the cluster — a key that only exists in Vault, whose own volume is in the backup, is a circular dependency.
Retention. How far back, and what does that cost at 172 G.

Implementation

Configure the Longhorn backup target and verify available: true
RecurringJobs per tier, with volumes selected by label rather than by name so new PVCs are covered by default and Tier 3 has to opt out explicitly
Postgres: continuous WAL archiving + scheduled base backups. Note CNPG 1.29 deprecates the in-tree barmanObjectStore in favour of the barman-cloud plugin — confirm which path applies before writing manifests
Vault: decide whether the volume snapshot is sufficient or whether it needs its own seal/unseal-aware procedure
Monitoring: alert on backup age, not just backup failure. A job that silently stopped running produces no failures at all. Ties into the alerting work in #39

Verification

This is the part that usually gets skipped, and it is the only part that proves any of the above works.

Restore Postgres into a scratch namespace from backup alone and diff against the primary
Restore one Tier 1 volume and mount it
Write down the actual measured restore time for the 154 G Immich library — that number decides whether the RTO above is fiction
Schedule a recurring restore drill

Side observation

Several volumes report actualSize larger than their provisioned size (data-forgejo-runner-0: 47.2 G used on a 40 G volume). That is accumulated Longhorn snapshots, and it suggests snapshot cleanup is not running anywhere. Worth understanding before adding scheduled snapshots on top.

There are currently no backups of anything in this cluster. The 2026-08-15 outage (#39) made that concrete: when the Postgres replica fell 42 h behind and its WAL was recycled, the only remaining copy of every database was a single Longhorn replica on one node. A `pg_dumpall` was taken by hand that day and verified — that is a snapshot someone typed, not a strategy. This issue covers designing the strategy and implementing it. Split into follow-ups once the design decisions below are settled. ## Current state | | | |---|---| | Longhorn `backuptargets/default` | URL empty, `available: false` | | Longhorn `RecurringJobs` | none | | CNPG `spec.backup` | empty | | WAL archiving | not configured | | barman-cloud plugin | not installed (operator is `cloudnative-pg:1.29.2`) | | Off-cluster destination | none — atlas has 38 G free on its only disk, no NFS/CIFS mounts anywhere | ## What actually needs protecting 299.5 G is in use across 22 volumes, but a large part of it is worthless to back up. **Tier 1 — irreplaceable, ~172 G** | PVC | Used | Note | |---|---|---| | `immich/immich-library` | 153.9 G | dominates everything else | | `forgejo/forgejo-data` | 9.9 G | git repos, this repo included | | `postgres` (logical) | ~4.4 G | 6 databases; needs logical/WAL backup, not a volume snapshot | | `paperless/paperless-data` | 1.5 G | | | `paperless/paperless-media` | 1.3 G | scanned documents | | `scanservjs/scanservjs-data` | 0.4 G | | | `vault/data-vault-0` | 0.2 G | small but losing it is its own category of bad | | `vikunja/vikunja-files` | 0.1 G | | **Tier 2 — recreatable with effort, ~1 G:** `observability/grafana` (dashboards), `magellan-staging-*` **Tier 3 — derivable, explicitly exclude, ~123 G** | PVC | Used | |---|---| | `forgejo-runner/data-forgejo-runner-{0,1,2}` | 47.2 + 42.5 + 28.9 G | | `observability/storage-loki-0` | 2.0 G | | `observability/prometheus-server` | 1.5 G | | `immich/immich-model-cache` | 1.0 G | | `paperless/paperless-{redis,consume,gpt}` | 0.3 G | Backing up Tier 3 would make the job 2.5× larger for no recovery value. The runner caches alone are 40 % of all used space. ## Design decisions to make first - [ ] **RPO/RTO per tier.** Losing a day of Immich uploads is annoying; losing a day of Forgejo commits or Paperless documents is not. These probably do not want the same schedule. - [ ] **Destination.** Nothing local has room for 172 G. Object storage off-site (S3/B2/Wasabi) is natively supported by both Longhorn and CNPG's barman-cloud and costs roughly a euro a month at this size — likely the cheapest answer, and it satisfies off-site by construction. A local NAS would be faster to restore from but is hardware that does not exist yet. A hybrid (local for speed, off-site for disaster) is the textbook answer if the budget allows. - [ ] **3-2-1, honestly.** Longhorn replicas are *not* a backup — they live on the same two nodes, and #39 tracks fixing that. Decide what genuinely counts as a second copy here. - [ ] **Encryption at rest and key custody.** If backups go off-site they must be encrypted, and the key must survive the cluster — a key that only exists in Vault, whose own volume is in the backup, is a circular dependency. - [ ] **Retention.** How far back, and what does that cost at 172 G. ## Implementation - [ ] Configure the Longhorn backup target and verify `available: true` - [ ] `RecurringJob`s per tier, with volumes selected by label rather than by name so new PVCs are covered by default and Tier 3 has to opt *out* explicitly - [ ] Postgres: continuous WAL archiving + scheduled base backups. Note CNPG 1.29 deprecates the in-tree `barmanObjectStore` in favour of the barman-cloud plugin — confirm which path applies before writing manifests - [ ] Vault: decide whether the volume snapshot is sufficient or whether it needs its own seal/unseal-aware procedure - [ ] Monitoring: alert on *backup age*, not just backup failure. A job that silently stopped running produces no failures at all. Ties into the alerting work in #39 ## Verification This is the part that usually gets skipped, and it is the only part that proves any of the above works. - [ ] Restore Postgres into a scratch namespace from backup alone and diff against the primary - [ ] Restore one Tier 1 volume and mount it - [ ] Write down the actual measured restore time for the 154 G Immich library — that number decides whether the RTO above is fiction - [ ] Schedule a recurring restore drill ## Side observation Several volumes report `actualSize` larger than their provisioned size (`data-forgejo-runner-0`: 47.2 G used on a 40 G volume). That is accumulated Longhorn snapshots, and it suggests snapshot cleanup is not running anywhere. Worth understanding before adding scheduled snapshots on top.

andre commented

2026-08-15 14:30:20 +00:00

Author

Owner

Destination decided

The "no local capacity" premise in the issue body is wrong — there is a NAS with 2 × 10 TB in RAID-1 (10 TB usable) and a Hetzner Storage Box. That makes real 3-2-1 possible without buying anything.

cluster ──NFS──► NAS (onsite, 10 TB usable)  ──BorgBackup/SFTP──►  Hetzner Storage Box (offsite)

At ~172 G of Tier 1 data into 10 TB, capacity is a non-issue onsite — retention can be generous, and the constraint moves to the Storage Box quota instead. Borg's deduplication and client-side encryption cover both the offsite retention and the encryption-at-rest requirement in one step, and Hetzner supports Borg with a dedicated endpoint. NFS clients are already present and healthy on all five nodes (NFSClientInstalled=True), so the Longhorn side needs no host preparation — just confirm the export is NFSv4, which is what Longhorn expects.

The one thing this does not solve: Postgres

The NAS speaks NFS/SMB and no S3. CNPG's barman-cloud cannot write to NFS — it only supports S3-compatible, Azure Blob and GCS. So the topology above covers every volume except the one that mattered most on 2026-08-15.

Three ways out, none of them free:

A. MinIO in-cluster, backed by an NFS share from the NAS. Gives CNPG the S3 endpoint it wants, keeps all data on the NAS. Caveat worth stating plainly: MinIO does not officially support NFS as its backend — the failure modes are locking-related and it is explicitly discouraged upstream. At this scale it will very likely work, but it is not a configuration anyone will support if it misbehaves.

B. MinIO in-cluster on a Longhorn volume. Officially supported storage for MinIO, but circular: the Postgres backup would live on the same Longhorn storage the backup exists to protect, on the same two nodes tracked in #39. Only acceptable if the Borg sync to Hetzner runs often enough to be the real backup.

C. Skip PITR. A CronJob running pg_dumpall to the NFS share, plus Longhorn volume backups of the Postgres PVCs. No MinIO, no extra moving parts in the recovery path. The cost is real: RPO becomes the dump interval instead of seconds, and restore is a full logical restore. At 4.4 G of database that restore is minutes, not hours.

Worth weighing against what actually happened: continuous WAL archiving is precisely what would have let the stale replica catch up instead of requiring a rebuild. That argues for A or B. Against that, every additional component in the recovery path is one more thing that can be broken at the exact moment it is needed — which is also precisely what happened on 08-15.

Decide between A, B and C

Where Borg runs

The NAS was described as NFS/SMB only, so it probably cannot run Borg itself. Candidates: a CronJob in-cluster mounting the NFS export, or a systemd timer on atlas. In-cluster keeps it in GitOps and visible to the alerting from #39; on atlas it survives the cluster being down, which is the scenario backups exist for.

Decide where the Borg job runs
Borg repokey must live somewhere that survives losing both the cluster and the NAS — see the key-custody note in the issue body, it applies here directly

Revised task list

Export a dataset on the NAS for Longhorn, NFSv4, and point backuptargets/default at it
Verify available: true, then take one manual backup and restore it before building anything on top
RecurringJobs per tier (labels, not names)
Postgres per the A/B/C decision above
Borg repo on the Storage Box, initialised with repokey-blake2, pruning policy matching the retention decision
Alert on backup age and on Borg sync age — two stages now, two things that can silently stop

## Destination decided The "no local capacity" premise in the issue body is wrong — there is a NAS with 2 × 10 TB in RAID-1 (10 TB usable) and a Hetzner Storage Box. That makes real 3-2-1 possible without buying anything. ``` cluster ──NFS──► NAS (onsite, 10 TB usable) ──BorgBackup/SFTP──► Hetzner Storage Box (offsite) ``` At ~172 G of Tier 1 data into 10 TB, capacity is a non-issue onsite — retention can be generous, and the constraint moves to the Storage Box quota instead. Borg's deduplication and client-side encryption cover both the offsite retention and the encryption-at-rest requirement in one step, and Hetzner supports Borg with a dedicated endpoint. NFS clients are already present and healthy on all five nodes (`NFSClientInstalled=True`), so the Longhorn side needs no host preparation — just confirm the export is NFSv4, which is what Longhorn expects. ## The one thing this does not solve: Postgres The NAS speaks NFS/SMB and no S3. **CNPG's barman-cloud cannot write to NFS** — it only supports S3-compatible, Azure Blob and GCS. So the topology above covers every volume *except* the one that mattered most on 2026-08-15. Three ways out, none of them free: **A. MinIO in-cluster, backed by an NFS share from the NAS.** Gives CNPG the S3 endpoint it wants, keeps all data on the NAS. Caveat worth stating plainly: MinIO does not officially support NFS as its backend — the failure modes are locking-related and it is explicitly discouraged upstream. At this scale it will very likely work, but it is not a configuration anyone will support if it misbehaves. **B. MinIO in-cluster on a Longhorn volume.** Officially supported storage for MinIO, but circular: the Postgres backup would live on the same Longhorn storage the backup exists to protect, on the same two nodes tracked in #39. Only acceptable if the Borg sync to Hetzner runs often enough to be the real backup. **C. Skip PITR.** A `CronJob` running `pg_dumpall` to the NFS share, plus Longhorn volume backups of the Postgres PVCs. No MinIO, no extra moving parts in the recovery path. The cost is real: RPO becomes the dump interval instead of seconds, and restore is a full logical restore. At 4.4 G of database that restore is minutes, not hours. Worth weighing against what actually happened: continuous WAL archiving is precisely what would have let the stale replica catch up instead of requiring a rebuild. That argues for A or B. Against that, every additional component in the recovery path is one more thing that can be broken at the exact moment it is needed — which is also precisely what happened on 08-15. - [ ] Decide between A, B and C ## Where Borg runs The NAS was described as NFS/SMB only, so it probably cannot run Borg itself. Candidates: a `CronJob` in-cluster mounting the NFS export, or a systemd timer on atlas. In-cluster keeps it in GitOps and visible to the alerting from #39; on atlas it survives the cluster being down, which is the scenario backups exist for. - [ ] Decide where the Borg job runs - [ ] Borg repokey must live somewhere that survives losing both the cluster and the NAS — see the key-custody note in the issue body, it applies here directly ## Revised task list - [ ] Export a dataset on the NAS for Longhorn, NFSv4, and point `backuptargets/default` at it - [ ] Verify `available: true`, then take one manual backup and restore it before building anything on top - [ ] `RecurringJob`s per tier (labels, not names) - [ ] Postgres per the A/B/C decision above - [ ] Borg repo on the Storage Box, initialised with `repokey-blake2`, pruning policy matching the retention decision - [ ] Alert on backup age *and* on Borg sync age — two stages now, two things that can silently stop

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

andre/homelab-gitops#41

No description provided.

Rows
Columns