Runbook index — OPNsense home-lab build

Discrete, sequential runbooks. Do them in order; validate each before moving on (every runbook ends with a Validation section). Design lives in ../CLAUDE.md.

#	Runbook	Layer
01	01-provision.md — Provision: Proxmox foundation + all VMs (bridges, pool, template, clone fleet, OPNsense VM)	Proxmox
02	02-opnsense-install-and-base.md — install, base, SSH, plugins, tunings	OPNsense
03	03-interfaces-and-vlans.md — the 6 zone VLANs + aliases	OPNsense
04	04-dhcp-and-dns.md — Unbound, Kea DHCP, IoT DNS intercept	OPNsense
05	05-protonvpn-egress.md — wg client, gateways, policy routing, kill-switch	OPNsense
06	06-firewall-rules.md — per-zone allow-lists, default deny	OPNsense
07	07-roadwarrior-and-mgmt-access.md — WG dial-in + WG-only MGMT	OPNsense
08	08-mdns-reflector.md — cross-VLAN discovery (WORKSTATIONS ⇄ IOT)	OPNsense/AP
09	09-public-services-caddy.md — origin Caddy (internal TLS-terminate) + hidden Bind primary	DMZ/origin
10	10-freeipa-server.md — identity: LDAP/Kerberos/DNS/CA	Server
11	11-zfs-nfs-storage.md — storage: AlmaLinux ZFS (OpenZFS kmod) + NFSv4 sec=sys home/media (NVMe passthrough)	Server
12	12-client-enrollment-automount.md — IPA enroll + /home/$USER	Client
13	13-authelia-sso.md — Authelia SSO/MFA + Caddy forward-auth + OIDC (FreeIPA-backed; replaced Authentik 2026-07-06)	Server
14	14-validation.md — full end-to-end test matrix	Verify
15	15-control-node.md — control node: dedicated EL9 Ansible control node (pinned ansible-core; reaches the fleet via a CA-signed control cert; lab.kdbx; vault password TPM-sealed); CA signing split to the admin laptop	Config/Ops
16	16-ansible-config-and-maintenance.md — Ansible: common baseline, DB backups, coordinated updates	Config/Ops
17	17-edge-vps.md — edge: remote VPS Caddy-L4 (SNI passthrough) + Bind secondary + home-dialed WG tunnel	Edge/VPS
18	18-observability.md — obs: metrics/logs/alerts (VictoriaMetrics+Loki+Grafana+Alloy), appliance feeds, ntfy alerts	Config/Ops
19	19-fmd-find-my-device.md — fmd: self-hosted phone locate/wipe (public via edge, device API open + web UI behind Authelia, UnifiedPush via ntfy)	Server/Edge
20	20-ssh-ca.md — SSH CA: user + host CAs (YubiKey custody), cert-based root, Ansible trust distribution, KRL revocation, PVE/OPNsense integration; device operations split to rb45	Security/Ops
21	21-media-and-haos.md — media + HA: Jellyfin/tvheadend/Invidious (EL9 quadlets; USB tuner passthrough) + Home Assistant (HAOS, SERVERS)	Server
23	23-host-firewall.md — host firewall: fleet-wide firewalld (default-deny inbound, SSH + declared service ports) via the host_firewall role in common; defense-in-depth behind OPNsense for east-west containment; ports as inventory data	Security/Ops
24	24-tang1-nbde-keyserver.md — NBDE keyserver: tang1 Raspberry Pi (Pi OS, MGMT VLAN), single-function Tang server for Clevis network-bound auto-unlock of pve's encrypted vmpool; appliance (out of the fleet Ansible runs), provisioned by provision/tang1-setup.sh	Storage/crypto
25	25-vmpool-encryption.md — encrypted vmpool: SSD vmpool/enc (aes-256-gcm) + Clevis/Tang self-healing boot unlock (fail-closed, manual break-glass), 3-way key escrow, guest provisioning onto it (VM+LXC), and a full disaster-recovery section	Storage/crypto
26	26-host-ca-automation.md — automated host-CA: split the SSH CAs (user-CA stays YubiKey/manual; host-CA goes online, Clevis/Tang-sealed to tang1 on ctrl, auto-renews short-lived host certs via a timer), ntfy on every sign + central tang access logging; user-CA = access stays offline	Security/Ops
27	27-failure-alerting.md — failure alerting: ntfy on ANY crashed service/container/host — alloy node_exporter systemd collector → node_systemd_unit_state{state="failed"} + a Grafana "Systemd unit failed" rule, plus a ctrl-side dead-man's-switch (Grafana can't alert on its own death); includes the API-validate-before-deploy checklist	Config/Ops
28	28-agent-operating-model.md — AI agent ops: run Claude as the IT admin from a PAW on ctrl (tmux, dedicated claude-agent account) holding its OWN YubiKey-signed cert (not the automation control cert; reaches all zones except the identity core); cutoff = expiry/KRL/live-kill; the four-tier credential custody model	Security/Ops
29	29-caddy-container-migration.md — Caddy → container: one shared digest-pinned image (rfc2136 + caddy-l4) built in CI to the internal zot registry (registry.${domain}), replacing the per-host native xcaddy builds; staged canary-first conversion of the four Caddy planes (mgmt→apps→origin→edges) to podman quadlets	Config/Ops
30	30-registry-cache.md — registry + pull-through cache: one internal zot (SERVERS LXC) that hosts our images AND caches all upstreams (docker.io/ghcr.io/gitlab/codeberg); fleet registries.conf mirrors + split-horizon registry.${domain}; fixes the B9 internal pull path	Config/Ops
31	31-static-sites-oci.md — static sites as OCI images: retire sftp-to-/var/www (the outage-class bug); build site content into FROM scratch images in zot, served by the containerized origin via --mount type=image (versioned, rollback-able, no state to lose). Mechanism piloted	Config/Ops
32	32-opencode-qwen36-tuning.md — opencode + qwen36 tuning: how the opencode coding agent on the GPU host is optimally configured against the local vLLM (the served model is a point-in-time record — Qwen3.6-27B-FP8 at time of writing) — serve flags (tool-call parser, YaRN context, FP8 KV cache, reasoning) + opencode's provider/sampling (Qwen3 official params). GPU-host-agent reference	Config/Ops
33	33-repo-parameterization.md — repo parameterization / generic-master: clone → set your values → build; lab.yml carries generic defaults and the real values live in the zz-local.yml instance overlay, so the tracked tree holds no instance identity. Scope + phased plan + progress	Config/Ops
34	34-appliance-update-checks.md — update checks (the two layers that don't self-update): read-only probes of the manually-updated appliances (OPNsense/pve/tang1/UniFi/HA) and of every pinned container image (list derived from the role defaults) → ntfy on lab-alerts when something is behind. Never auto-updates. The image leg is the compensating control for pinning + masked podman-auto-update, added after a stale pin took invidious down unnoticed (rb21)	Config/Ops
35	35-forgejo-split.md — split the git server: two Forgejo instances — forgejo becomes the private control-plane SCM (git.${domain}, Tier-2, holds myos+CI, wiring unchanged) and a new forgejo-pub takes the public repos (git.${domain_alt}, edge-exposed); migrate public repos out (service: gitea), demote control-plane git off the edge; cutover + rollback	Config/Ops
36	36-keepass-admin-access.md — KeePass admin access: the two-vault model (lab.kdbx infra/break-glass on ctrl + a personal desktop DB), password-only unlock (the old shared-YubiKey-CR factor retired 2026-07-18 → rb47 presence-unlock), SSHFS + KeePassXC-Browser autofill (no web service), and the full admin-GUI/URL checklist	Security/Ops
37	37-rpm-mirror.md — pinned RPM snapshot mirror (supply-chain resilience): a stateless mirror VM reposyncs AlmaLinux+EPEL+ZFS into the fileserver ZFS (over NFS) and serves it over its own caddy; the fleet pins to a blessed snapshot (use_rpm_mirror) and can roll back even if upstream is poisoned. Blessing = zfs snapshot on the fileserver (mirror host has no snapshot authority); auto-bless after an N-day soak (auto-bless-mirrors.timer, nfs_mirror_soak_days=3; manual bless = emergency fast-track); gpgcheck enforced	Security/Ops
38	38-zot-on-nfs.md — zot registry blobs on ZFS/NFS (DESIGN): proposal to snapshot/rollback the registry like the RPM mirror; surfaces two blockers (NFS refused in unprivileged LXC → registry needs VM conversion; OCI-on-NFS/boltdb validation) and weighs a lighter skopeo-to-ZFS backup of our own images. Decision needed	Security/Ops
39	39-deb-mirror.md — pinned Debian (.deb) snapshot mirror (supply-chain resilience): the .deb analog of rb37 — the mirror VM runs debmirror in a Debian container (base pulled through zot) into fileserver ZFS; full trixie+updates+security, all sections; blessed snapshot + rollback; apt pins via use_deb_mirror. Same host/caddy/bless as the RPM mirror	Security/Ops
40	40-hdd-backup.md — backups: the two SSD pools → the HDD: a daily vzdump job (guests vmpool → rpool) + nightly syncoid (pve pulls the fileserver's supernas → rpool, least-privilege delegated) so a single SSD failure loses nothing; same machine, different disk. Off-box (second machine) is the remaining step	Security/Ops
41	41-seafile-file-sync.md — RETIRED 2026-08-31 (kept for history; superseded by runbook 52's syncthing+shelf — libraries exported to supernas/sync/casey/misc/from-seafile/, sha256 zero-diff verified. Its § Radicale (dav) appendix is NOT retired and stays current). Was: Seafile file-sync (adopted keeper): the files DMZ VM (seafile-mc + MariaDB + memcached) as the lab's file-sync service — Tier-2 via apps-proxy, IPA SSO via Authelia OIDC (both URL roots pinned https; client_secret_basic; DMZ→origin-Caddy hairpin), phone Camera Upload over the always-on WG (targeted WG_PHONES → apps-proxy:443 rule), 256 GB disk, and two-layer DR (vzdump image + nightly MariaDB dump). Graduated from trial 2026-07-09; now stateful	Config/Ops
42	42-alerting-coverage-audit.md — alerting coverage audit (2026-07-12): the full failure-mode-vs-what-pages matrix for every service, from four read-only sweeps; cross-cutting findings (pve monitoring dark spot, gating probes with no consumer rule, dead syslog gateway, CI observability loss, edge failover designed-not-realized, DNSSEC unanchored) + the prioritized P1–P3 fix program. Point-in-time record — fixes update it	Security/Ops
43	43-lab-secrets.md — lab-secrets store: the instance's ansible-vault-encrypted *.vault.yml + lab.kdbx live in a dedicated Tier-2 lab-secrets Forgejo repo (kept out of the public myos repo). A control-role oneshot (load-lab-secrets.sh) pulls it with a read-only deploy key and overlays the files into group_vars/all/ before any converge. Vault password stays TPM-sealed (repo read = ciphertext only); deploy key is escrowed bootstrap material	Security/Ops
44	44-assistant-research-stack.md — assistant + research stack (the GPU host): Open WebUI (assistant.${domain}) + local-deep-research (research.${domain}) + litellm sidecar as the lab's general assistant / deep-research plane — Tier-1 via origin Caddy + Authelia trusted-header SSO (casey-only), shared local vLLM + SearXNG (JSON), cloud-fallback via escrowed Anthropic key, host-networking + ufw origin-only invariant, monthly pin-bump discipline. opencode (rb32) stays the coding agent	Config/Ops
45	45-yubikey-operations.md — YubiKey device operations: the token itself — the applets (PIV/FIDO2/OTP; OATH/OpenPGP unused), every PIN/PUK/management-key and what each gates, the two-key redundancy model (independent per-token 9c/9d/9a; the old shared OTP challenge-response lab.kdbx factor is retired — the DB is password-only, armed via rb47), a command reference per lab operation, web-GUI login (WebAuthn is per-token and non-exportable → register both keys everywhere; no lab site uses it today), enrolling and retiring a token, and the non-destructive + destructive drill suites. rb20 remains the SSH-CA manual	Security/Ops
46	46-key-rotation.md — rotating the lab's keys (dependency-ordered): answers "is a full key rotation even possible?" — yes, but leaves-first and make-before-break, never as one operation. The five root secrets, the bootstrap dependency graph, the six circular dependencies and how each is broken, a Phase 0 escrow pre-flight, a 7-phase rotation order, and a catalogue of lockout orderings (L1: rotating tang1's keys before re-encrypting the JWEs can take the whole lab down). Hygiene, not incident response	Security/Ops
47	47-keepass-presence-unlock.md — KeePass presence-unlock (YubiKey-gated lab.kdbx): unlock the vault only when the operator is present, no master-password typing. The YubiKey on atomic decrypts a per-session master-password (age-plugin, multi-recipient to both tokens) and ships it over SSH to a short-TTL tmpfs credential on ctrl (arm-kdbx/kdbx-get/disarm); 9a/9c/9d untouched; manual kdbx-unlock stays break-glass	Security/Ops
48	48-jitsi-meet.md — Jitsi Meet (self-hosted video, E2EE): a fourth standalone public plane — docker-jitsi-meet podman quadlets (web/prosody/jicofo/jvb) on the box's OWN public IP + UDP/10000 media, not on the edge/origin planes. Terminates its own TLS (LE HTTP-01); E2EE on by default so the box can't read media; room creation JWT-gated — any IPA user via the Tier-2 meet-start issuer (meet_issuer on sso, Authelia forward-auth; meet only verifies — no route to the identity core; guests → lobby). Home tunnel = mirror-pin + obs /32s only; on central obs. Interim on-prem pve VM (dies with a home outage) standing in for off-prem edge3 (jitsi + meet_issuer roles)	Config/Ops
50	50-someonetoldme.md — someonetoldme.org: the FIRST fleet app built from our own source rather than a pinned upstream image. Four quadlets from one image (postgres + one-shot migrate + web + worker); built on runner-pub and pushed to zot as apps/someonetoldme by the scoped ci-pub account, pinned by digest (resolve it from the REGISTRY — podman reports a different one). Records the traps it actually hit: ALLOWED_HOSTS needing 127.0.0.1 for the container's own healthcheck, an image with neither wget nor curl, and db_backups running in a separate play	Apps
49	49-forgejo-upgrade.md — crossing Forgejo majors (both instances): the migrations are one-way, so step one major at a time with a restore point before every start — never skip. Staging with -e (branch protection forbids per-step commits) and the downgrade trap that creates; the zot pre-pull guard (a wedged registry hangs, so pull every tag up front); public instance first as the migration canary; and the control-plane's integration surface — Authelia OIDC/PKCE, CI runner, and GPG merge-signing, whose failure silently freezes the host-CA signer. Forgejo is lock_major in the rb34 image check on purpose — a newer major is reported as a note, never as "stale" — so these bumps stay operator-initiated	Config/Ops
51	51-operating-interface-and-approval-tiers.md — the operating interface + approval tiers: what the operator is for, now that the hub fronts the repo, the infrastructure and ntfy alarm triage — four touchpoints, all master-key or approval-click. Splits the one "operator approves every PR" gate by path, default-deny: T2 (anything unlisted, plus ansible/**, scripts/**, .forgejo/**, provision/**, CLAUDE.md, …) stays operator-approved and operator-merged, because the host-CA signer executes those paths (rb26); T0 (docs/**) is delegated to an agent-reviewer identity. Three enforcement layers with the client-side one honestly labelled a router, not a wall; the three-category boundary map + its five blurs; the fresh-context review workflow; the project intake channel; pre-registered shadow-mode exit criteria; and N1–N11 negative tests	Security/Ops
52	52-file-sync.md — File sync (adopted keeper, 2026-08-31): Syncthing + shelf on the fileserver — "phone photos as plain files on the storage tier", no application database in the data path. It superseded Seafile (rb41) and Nextcloud, whose libraries were migrated in — so supernas/sync is now the only copy of that data, which is what turned the deferred items into shipped ones. Records the placement rationale (an unprivileged LXC refuses NFS, so anywhere else orphans the laptop-mount story), the tree convention (camera/ machine-owned receive-only inbox, photos/YYYY/ durable human library, docs/ — and why promotion must COPY, never move, plus the promotion job deliberately deferred as a design decision owed), the interactive pairing code cannot do (device IDs are generated; send-only phone / receive-only server, and why sync is not backup), backups (sanoid + the three-place bizon wiring), the NFS export in the role default rather than a host_vars shadow that would rot, the per-tenant shelf cross-ref (rb55 §9), and the freshness alarm — the failure a green unit never reports: an emitter whose tree list is derived from shelf_tenants, which excludes syncthing's own .stversions (filled by phone-side DELETIONS, so measuring it reports fresh while nothing arrives), emits a timestamp not an age, emits 0 for an empty tree so a wiped phone pages hard, and emits nothing when it could not read a tree — with the standing detector-cannot-run third rule. Plus the adoption record (Seafile exported sha256-zero-diff then retired; shelf is already our own program while syncthing stays a pinned upstream binary, and why) and the traps: filebrowser proxy-auth trusted the header and auto-created the account — the reason :8081 is source-scoped, and load-bearing again under shelf-map, which trusts Remote-User to pick a tenant (filebrowser itself UNINSTALLED at adoption), ProtectSystem=strict + ReadWritePaths= silently defeating a mountpoint guard (bind mount inside the namespace — hence the + prefix), Authelia's default_policy: deny login→403 loop, syncthing having no offline config CLI, and EPEL9 not packaging it	Apps/Storage
53	53-xmpp-control-channel.md — the lab control channel: a contact list on the phone where each contact is a lab function; messaging one dispatches a deterministic command (first contact: Downloader — share a link, the file lands in the media library). Prosody on a Tier-2 chat LXC with local accounts (identity-independent by design, same rule as tvheadend/ntfy) and federation OFF — a boundary, not a tweak, since s2s would let any remote server message the command bots. The load-bearing property: the bus holds no privilege and executes nothing — bots run beside the privilege they expose and dial outbound, so compromising chat buys the ability to message bots that refuse you. Three independent gates (transport / bare-JID allowlist / full-match URL grammar, argv never a shell string), an empty allowlist refusing to start, self-signed TLS pinned by the client (deliberately not the DNS-01 wildcard: a control channel must not depend on the ACME/IPA machinery it exists to help repair), and no AI in the dispatch path — an agent is a separate, explicitly-invoked contact. Covers the Android client, phone setup, the negative tests each gate must pass, and why the vocabulary stays narrow	Ops/Apps
54	54-pve-upgrade.md — upgrading the hypervisor across majors (PVE 8 → 9, and with it Debian bookworm → trixie): the one procedure whose blast radius is the whole lab, because opnsense and ctrl are guests on the host being upgraded — so the network drops with it and the agent cannot drive its own recovery; the reboot half is operator-run from the IPMI console. Records the gates that make it safe rather than hopeful: run apt in a multiplexer on pve (not from ctrl, a guest of the host); apt dist-upgrade -s proving proxmox-ve is upgraded not removed (a single leftover bookworm repo inverts that); deriving the --force-confold decision from the actually-modified conffile set (six here, all wanted — incl. three proxmox-boot-tool hooks Debian's versions would break); and proving the Clevis+Tang vmpool unseal before the reboot rather than discovering it locked when every guest is down. Plus the two prohibitions that keep rollback alive — no zpool upgrade (burns the retained-kernel fallback and the bizon send compatibility), no apt autoremove	Proxmox
55	55-tenant-plane.md — the multi-tenant service prototype: running services for paying tenants who are not the household, on the same hardware, without a tenant ever being one mistake away from the lab's own identity, data or admin surfaces. Records the three separations and why each is load-bearing rather than tidy — lldap instead of FreeIPA (which governs who can root the fleet, so a customer in it is one group-membership mistake from the infrastructure), a tenant-only Caddy that defines no admin vhost at all (isolation that is structural, not a correctly-written bind), and a separate WireGuard instance whose interface-scoped rules make the tenant grant ONE rule for every tenant forever, so onboarding adds no firewall rule and no repo change. Then the storage model: one ZFS dataset per tenant with both quotas (refquota excludes snapshots, so a write-and-delete-heavy tenant stays under their advertised space while used grows unbounded — and hitting quota is a tenant-visible ENOSPC caused by our retention), a 30-day tenant retention template on both ends so "deleted tenant data is unrecoverable after 30 days" is true, and the rule that coverage must exist before the dataset (all three replication layers were non-recursive, so a child dataset was silently unsnapshotted, unreplicated and unpruned while every job reported rc=0). Plus the migration procedure actually run — zfs create over a populated path mounts over it and the tenant reads empty; prove equivalence with an itemised rsync -n --delete requiring zero differences rather than matching counts; and re-read the source at swap time, because a baseline written down hours earlier makes growth look like loss	Tenant
56	56-personal-appliance.md — DRAFT decision record + phase-1 design (2026-09-12; nothing built): consolidating the household personal hub and the tenant plane onto one pve guest (appliance, VM, DMZ), retiring dav + tenant-proxy + tenant-dir + tenant-sso and taking shelf off the fileserver; the GPU workstation becomes a client like the phone and the laptop. Draws the boundary by audience, not by service: two doors on one box — the HOUSEHOLD on FreeIPA/lab-Authelia, the TENANTS on lldap + their own Authelia + their own WireGuard — with rb55 §1's load-bearing invariant carried across intact, so the tenant face stays a separate Caddy instance that was never given the household vhost list to inherit (two vhost lists in one config is exactly the correctly-written-bind arrangement that invariant was written against). Records the operator's Tier-1 override for the hub (public via the origin behind Authelia, against the lab's own default-Tier-2 rule) with its honest price — the trust anchor moves from the network to identity — and the finding that CalDAV cannot take that door at all (no native client traverses an SSO redirect). Picks VM (an unprivileged LXC refuses NFS and cannot own a pool, so the form does not churn between phases) and DMZ (origin→appliance becomes intra-zone, so the Tier-1 door costs no firewall rule, unlike radio's seq-645 relay). Phase 1 keeps ZFS authority on the fileserver over NFS rather than moving the datasets, because rb55 §3.3's coverage before the dataset rule and a three-place backup lineage do not belong in the same change as four service moves — at the price of one host-scoped DMZ→SERVERS :2049 grant, and a new numeric uid/gid parity invariant that fails silently and open. Plus the step-by-step migration with a rollback per step, what is STATE (Radicale collections, the lldap DB, Authelia's SQLite, syncthing identities) and must be moved rather than rebuilt, and eight numbered open decisions each with a recommendation	Tenant/Apps

Experimental (not part of the clean build): 22-bootc-workstations.md — image-mode Fedora Atomic workstations pilot (built in CI → registry → bootc; the GPU host stays Ubuntu/OEM; servers out of scope). Parked; revisit later.

Soup-to-nuts: provision first (runbook 01 = provision/, qm clone+cloud-init), build through the middle, Ansible config & maintenance (runbook 16 = ansible/, boring-&-pinned roles + playbooks; baseline applied at runbook 09).

Edge/origin build order (see ../CLAUDE.md): - Origin (home), runbooks 01–16: the full lab, inbound-silent — the origin Caddy + hidden Bind primary (runbook 09) are internal-only; the only WAN inbound is the WG admin endpoint (.69:51820). - First edge, runbook 17: one VPS (Caddy-L4 passthrough + Bind secondary + WG tunnel home dials out); delegate one site/NS; verify end-to-end. - Multi-vendor, runbook 17 repeated: 2–3 diverse-vendor edges for NS/serving redundancy; home stays inbound-silent (only WG admin).

Notes: - GUI steps target OPNsense 26.1 (e.g. NAT "Port Forward" is now Destination NAT; NAT rules need an explicit Filter rule = Pass/Register/Manual; the Rules GUI was redesigned; Kea DHCP puts gateway/DNS under Option data). Labels can drift by point release. - Do them in order; each ends with a validation gate — don't proceed until it passes. - Several runbooks have "Facts to confirm" placeholders (bridge names, host IPs, ISP/MAC behavior, AP capabilities) — fill those in for your environment. - Generate fresh secrets for every service (WireGuard keys, passwords, TSIG, vault items) — never reuse keys from anywhere. See the secrets note in ../CLAUDE.md.