Homelab GitOps

This repository is the single source of truth for Kubernetes applications in the Homelab K3s cluster. Argo CD continuously reconciles main; changes made directly with kubectl, Helm, or the old Ansible deployment roles are drift and will be reverted.

Ownership boundary

Git owns:

• Argo CD, cert-manager, MetalLB, Traefik, Longhorn, Vault and the Vault Secrets Operator
• the CloudNativePG operator and PostgreSQL cluster
• Forgejo, Forgejo Actions runners, Vikunja, Immich, Paperless-ngx and both MCP services
• ingresses, policies, storage classes and all non-secret Kubernetes config

The homelab repository still owns:

• K3s and host provisioning, node labels/taints and physical Longhorn disks
• OpenWrt DNS, firewall and ingress forwarding
• Vault initialization/unseal, policies, auth roles and secret values
• one-time Forgejo runner registration and database bootstrap grants
• the initial Argo CD installation and this repository's root Application

K3s-managed components such as CoreDNS and metrics-server are intentionally not duplicated here.

Ingress availability

Traefik runs twice, with required hostname anti-affinity and a PodDisruptionBudget, on two distinct worker nodes. Its LoadBalancer Service uses externalTrafficPolicy: Local; MetalLB therefore advertises the VIP only from a node with a local ready Traefik endpoint. The L2Advertisement allows Atlas' enp2s0 and the workers' end0, so the announcement can move between worker nodes after a pod or node failure.

Storage placement

Only bigal and quartz64 have Longhorn disks, and the label node.longhorn.io/create-default-disk: config marks them. Longhorn serves a volume to any node over the network regardless, so a pod on a diskless node runs perfectly well — but the longhorn and longhorn-ssd-single-replica classes ask for a local replica on a best-effort basis, and when that placement can never succeed Longhorn leaves the volume Scheduled=False and then refuses to expand it.

So a workload with a Longhorn volume needs one of two things: the node selector above, or the longhorn-remote class, which asks for no local replica at all. Pick the selector when local I/O is worth having, as with PostgreSQL; pick the class when the pod has to sit on a diskless node anyway, as Immich's machine learning does.

Vault prerequisites for applications

Argo CD can create these applications, but their pods cannot become ready until the following values exist in Vault, which this repository cannot create:

KV v2 path (secret/)	Keys
immich-database	username (immich), password
paperless-database	username (paperless), password
paperless-service	secret — Django signing key
paperless-gpt	mistral-api-token, paperless-api-token
rosenhammer-staging-database	username (rosenhammer_staging), password
rosenhammer-staging-service	secret_key_base — Phoenix signing key
rosenhammer-staging-smtp	username, password — contact form mail relay, entered by hand
magellan-staging-service	auth_token — bearer token for the MCP HTTP endpoint
scanservjs	paperless-api-token — optional, for automatic upload to Paperless
container-registry	Shared CONTAINER_REGISTRY_TOKEN, registry metadata, and .dockerconfigjson for every private Forgejo image consumer
argocd-oidc	client_id, client_secret — the Forgejo OAuth2 application backing Argo CD single sign-on

Dedicated Kubernetes auth roles are also required for immich, paperless, rosenhammer_staging, magellan_staging and scanservjs, each bound to its application's ServiceAccount and namespace.

deploy-scansrvjs The paperless role's policy has to cover paperless-gpt as well; the Vault Secrets Operator reads all three Paperless paths through that one role. Private-image consumers additionally receive the shared container-registry-read policy. The existing cnpg-app-credentials role additionally needs its policy widened to read all three database paths, otherwise CloudNativePG never receives the passwords it is supposed to apply to the new PostgreSQL roles.

Rosenhammer uses the same provisioning playbook. It generates the database password and Phoenix signing key, reconciles the shared registry credential supplied through CONTAINER_REGISTRY_TOKEN, and creates the rosenhammer_staging Kubernetes auth role. No credential is committed to this repository.

The SMTP credentials are the exception the playbook does not generate: they belong to a mail account outside this cluster, so rosenhammer-staging-smtp has to be written once by hand. Until it exists the pods still start and the site stays up — only the contact form reports a delivery error and logs :no_recipient_configured.

Argo CD single sign-on

Argo CD authenticates against Forgejo's OIDC provider directly; the bundled Dex stays disabled because Forgejo already serves discovery at /.well-known/openid-configuration and Vault authenticates browsers the same way. Two things live outside this repository and must exist before the login button works.

First, a confidential OAuth2 application in Forgejo, under Settings → Applications, with these redirect URIs:

• https://argocd.chief-detektor.dev/auth/callback — the web UI
• http://localhost:8085/auth/callback — argocd login --sso from a terminal

Second, its generated credentials in Vault as argocd-oidc/client_id and argocd-oidc/client_secret. Forgejo shows the client secret exactly once, at creation. The Vault Secrets Operator renders both into the argocd-oidc Secret, which carries the app.kubernetes.io/part-of: argocd label so that argocd-cm will resolve the $argocd-oidc:... references.

Authorisation is by email claim, not by group: Forgejo has one user and no organisations, so its groups claim arrives empty. policy.default is deliberately empty, which means any other Forgejo account that authenticates successfully still gets no permissions until policy.csv names it.

The local admin account stays enabled on purpose. Argo CD deploys the Forgejo that authenticates it, so an SSO-only configuration would lock the cluster's operator out of the one tool needed to repair a broken Forgejo. Its password remains in argocd-secret, which the Application's ignoreDifferences already excludes from reconciliation.

Paperless-ngx state that paperless-gpt depends on

paperless-gpt reads documents out of Paperless-ngx over its REST API and writes recognised text, title, tags, correspondent and document type back. Three things it needs live in the Paperless-ngx database, not in a manifest, so they have to be created once in the web interface:

1. An API token, under the profile menu of a superuser. It goes into Vault as paperless-gpt/paperless-api-token; the value is never committed here.
2. The two trigger tags, paperless-gpt-auto (title, tags, correspondent, document type) and paperless-gpt-ocr-auto (Mistral OCR). paperless-gpt only auto-creates paperless-gpt-failed on startup, so these two must exist before a workflow can reference them.
3. A workflow with trigger type Document added that assigns both tags. Every consumed document then queues itself; paperless-gpt removes the tag again when it is done and marks the result with paperless-gpt-ocr-complete.

CREATE_NEW_TAGS is off, so the model may only pick from tags that already exist — new categories stay a deliberate act.

No Paperless-ngx setting has to change for the API access itself. ALLOWED_HOSTS defaults to * and PAPERLESS_URL only appends to it, so the in-cluster request to http://paperless is accepted as it is.

OCR runs twice by design: Paperless-ngx keeps producing the archive PDF with Tesseract, and paperless-gpt replaces only the searchable content field with Mistral's result. PDF_UPLOAD stays off, so the original file is never touched and a bad recognition run costs nothing but a re-tag.

Repository layout

bootstrap/                 one manually applied root Application
clusters/homelab/          AppProject and child Applications
infrastructure/            operators, platform config and database
apps/                      user-facing workloads

Every external chart and image version is pinned. Kubernetes Secrets with credentials are generated from Vault by the Vault Secrets Operator; secret values must never be committed.

Change workflow

1. Create a branch.

2. Change manifests or a pinned version.

3. Validate locally:

   ./scripts/validate.sh
   

4. Open and review a pull request in Forgejo.

5. Merge to main. Argo CD then syncs, self-heals and prunes automatically.

Do not run kubectl apply, helm upgrade, or an application deployment playbook for resources represented here. Emergency live changes must be backported to Git immediately or Argo CD will revert them.

Bootstrap

The root Application is the only resource applied outside Argo CD:

k3s kubectl apply -f bootstrap/root-application.yaml

After bootstrap, use the Argo CD UI to verify that homelab-root and every child Application are Synced and Healthy.