Runbook index — OPNsense home-lab build

Discrete, sequential runbooks. Do them in order; validate each before moving on (every runbook ends with a Validation section). Design lives in ../CLAUDE.md.

#	Runbook	Layer
01	01-provision.md — Provision: Proxmox foundation + all VMs (bridges, pool, template, clone fleet, OPNsense VM)	Proxmox
02	02-opnsense-install-and-base.md — install, base, SSH, plugins, tunings	OPNsense
03	03-interfaces-and-vlans.md — the 6 zone VLANs + aliases	OPNsense
04	04-dhcp-and-dns.md — Unbound, Kea DHCP, IoT DNS intercept	OPNsense
05	05-protonvpn-egress.md — wg client, gateways, policy routing, kill-switch	OPNsense
06	06-firewall-rules.md — per-zone allow-lists, default deny	OPNsense
07	07-roadwarrior-and-mgmt-access.md — WG dial-in + WG-only MGMT	OPNsense
08	08-mdns-reflector.md — cross-VLAN discovery (WORKSTATIONS ⇄ IOT)	OPNsense/AP
09	09-public-services-caddy.md — origin Caddy (internal TLS-terminate) + hidden Bind primary	DMZ/origin
10	10-freeipa-server.md — identity: LDAP/Kerberos/DNS/CA	Server
11	11-zfs-nfs-storage.md — storage: AlmaLinux ZFS (OpenZFS kmod) + NFSv4 sec=sys home/media (NVMe passthrough)	Server
12	12-client-enrollment-automount.md — IPA enroll + /home/$USER	Client
13	13-authelia-sso.md — Authelia SSO/MFA + Caddy forward-auth + OIDC (FreeIPA-backed; replaced Authentik 2026-07-06)	Server
14	14-validation.md — full end-to-end test matrix	Verify
15	15-control-node.md — control node: dedicated EL9 Ansible control node (pinned ansible-core; reaches the fleet via a CA-signed control cert; lab.kdbx; vault password TPM-sealed); CA signing split to the admin laptop	Config/Ops
16	16-ansible-config-and-maintenance.md — Ansible: common baseline, DB backups, coordinated updates	Config/Ops
17	17-edge-vps.md — edge: remote VPS Caddy-L4 (SNI passthrough) + Bind secondary + home-dialed WG tunnel	Edge/VPS
18	18-observability.md — obs: metrics/logs/alerts (VictoriaMetrics+Loki+Grafana+Alloy), appliance feeds, ntfy alerts	Config/Ops
19	19-fmd-find-my-device.md — fmd: self-hosted phone locate/wipe (public via edge, device API open + web UI behind Authelia, UnifiedPush via ntfy)	Server/Edge
20	20-ssh-ca.md — SSH CA: user + host CAs (YubiKey custody), cert-based root, Ansible trust distribution, KRL revocation, PVE/OPNsense integration; device operations split to rb45	Security/Ops
21	21-media-and-haos.md — media + HA: Jellyfin/tvheadend/Invidious (EL9 quadlets; USB tuner passthrough) + Home Assistant (HAOS, SERVERS)	Server
23	23-host-firewall.md — host firewall: fleet-wide firewalld (default-deny inbound, SSH + declared service ports) via the host_firewall role in common; defense-in-depth behind OPNsense for east-west containment; ports as inventory data	Security/Ops
24	24-tang1-nbde-keyserver.md — NBDE keyserver: tang1 Raspberry Pi (Pi OS, MGMT VLAN), single-function Tang server for Clevis network-bound auto-unlock of pve's encrypted vmpool; appliance (out of the fleet Ansible runs), provisioned by provision/tang1-setup.sh	Storage/crypto
25	25-vmpool-encryption.md — encrypted vmpool: SSD vmpool/enc (aes-256-gcm) + Clevis/Tang self-healing boot unlock (fail-closed, manual break-glass), 3-way key escrow, guest provisioning onto it (VM+LXC), and a full disaster-recovery section	Storage/crypto
26	26-host-ca-automation.md — automated host-CA: split the SSH CAs (user-CA stays YubiKey/manual; host-CA goes online, Clevis/Tang-sealed to tang1 on ctrl, auto-renews short-lived host certs via a timer), ntfy on every sign + central tang access logging; user-CA = access stays offline	Security/Ops
27	27-failure-alerting.md — failure alerting: ntfy on ANY crashed service/container/host — alloy node_exporter systemd collector → node_systemd_unit_state{state="failed"} + a Grafana "Systemd unit failed" rule, plus a ctrl-side dead-man's-switch (Grafana can't alert on its own death); includes the API-validate-before-deploy checklist	Config/Ops
28	28-agent-operating-model.md — AI agent ops: run Claude as the IT admin from a PAW on ctrl (tmux, dedicated claude-agent account) holding its OWN YubiKey-signed cert (not the automation control cert; reaches all zones except the identity core); cutoff = expiry/KRL/live-kill; the four-tier credential custody model	Security/Ops
29	29-caddy-container-migration.md — Caddy → container: one shared digest-pinned image (rfc2136 + caddy-l4) built in CI to the internal zot registry (registry.${domain}), replacing the per-host native xcaddy builds; staged canary-first conversion of the four Caddy planes (mgmt→apps→origin→edges) to podman quadlets	Config/Ops
30	30-registry-cache.md — registry + pull-through cache: one internal zot (SERVERS LXC) that hosts our images AND caches all upstreams (docker.io/ghcr.io/gitlab/codeberg); fleet registries.conf mirrors + split-horizon registry.${domain}; fixes the B9 internal pull path	Config/Ops
31	31-static-sites-oci.md — static sites as OCI images: retire sftp-to-/var/www (the outage-class bug); build site content into FROM scratch images in zot, served by the containerized origin via --mount type=image (versioned, rollback-able, no state to lose). Mechanism piloted	Config/Ops
32	32-opencode-qwen36-tuning.md — opencode + qwen36 tuning: how the opencode coding agent on the GPU host is optimally configured against the local vLLM (the served model is a point-in-time record — Qwen3.6-27B-FP8 at time of writing) — serve flags (tool-call parser, YaRN context, FP8 KV cache, reasoning) + opencode's provider/sampling (Qwen3 official params). GPU-host-agent reference	Config/Ops
33	33-repo-parameterization.md — repo parameterization / generic-master: clone → set your values → build; lab.yml carries generic defaults and the real values live in the zz-local.yml instance overlay, so the tracked tree holds no instance identity. Scope + phased plan + progress	Config/Ops
34	34-appliance-update-checks.md — update checks (the two layers that don't self-update): read-only probes of the manually-updated appliances (OPNsense/pve/tang1/UniFi/HA/Nextcloud) and of every pinned container image (list derived from the role defaults) → ntfy on lab-alerts when something is behind. Never auto-updates. The image leg is the compensating control for pinning + masked podman-auto-update, added after a stale pin took invidious down unnoticed (rb21)	Config/Ops
35	35-forgejo-split.md — split the git server: two Forgejo instances — forgejo becomes the private control-plane SCM (git.${domain}, Tier-2, holds myos+CI, wiring unchanged) and a new forgejo-pub takes the public repos (git.${domain_alt}, edge-exposed); migrate public repos out (service: gitea), demote control-plane git off the edge; cutover + rollback	Config/Ops
36	36-keepass-admin-access.md — KeePass admin access: the two-vault model (lab.kdbx infra/break-glass on ctrl + a personal desktop DB), password-only unlock (the old shared-YubiKey-CR factor retired 2026-07-18 → rb47 presence-unlock), SSHFS + KeePassXC-Browser autofill (no web service), and the full admin-GUI/URL checklist	Security/Ops
37	37-rpm-mirror.md — pinned RPM snapshot mirror (supply-chain resilience): a stateless mirror VM reposyncs AlmaLinux+EPEL+ZFS into the fileserver ZFS (over NFS) and serves it over its own caddy; the fleet pins to a blessed snapshot (use_rpm_mirror) and can roll back even if upstream is poisoned. Blessing = zfs snapshot on the fileserver (mirror host has no snapshot authority); auto-bless after an N-day soak (auto-bless-mirrors.timer, nfs_mirror_soak_days=3; manual bless = emergency fast-track); gpgcheck enforced	Security/Ops
38	38-zot-on-nfs.md — zot registry blobs on ZFS/NFS (DESIGN): proposal to snapshot/rollback the registry like the RPM mirror; surfaces two blockers (NFS refused in unprivileged LXC → registry needs VM conversion; OCI-on-NFS/boltdb validation) and weighs a lighter skopeo-to-ZFS backup of our own images. Decision needed	Security/Ops
39	39-deb-mirror.md — pinned Debian (.deb) snapshot mirror (supply-chain resilience): the .deb analog of rb37 — the mirror VM runs debmirror in a Debian container (base pulled through zot) into fileserver ZFS; full trixie+updates+security, all sections; blessed snapshot + rollback; apt pins via use_deb_mirror. Same host/caddy/bless as the RPM mirror	Security/Ops

| 40 | 40-hdd-backup.md — backups: the two SSD pools → the HDD: a daily vzdump job (guests vmpool → rpool) + nightly syncoid (pve pulls the fileserver's supernas → rpool, least-privilege delegated) so a single SSD failure loses nothing; same machine, different disk. Off-box (second machine) is the remaining step | Security/Ops |

| 41 | 41-seafile-file-sync.md — Seafile file-sync (adopted keeper): the files DMZ VM (seafile-mc + MariaDB + memcached) as the lab's file-sync service — Tier-2 via apps-proxy, IPA SSO via Authelia OIDC (both URL roots pinned https; client_secret_basic; DMZ→origin-Caddy hairpin), phone Camera Upload over the always-on WG (targeted WG_PHONES → apps-proxy:443 rule), 256 GB disk, and two-layer DR (vzdump image + nightly MariaDB dump). Graduated from trial 2026-07-09; now stateful | Config/Ops |

| 42 | 42-alerting-coverage-audit.md — alerting coverage audit (2026-07-12): the full failure-mode-vs-what-pages matrix for every service, from four read-only sweeps; cross-cutting findings (pve monitoring dark spot, gating probes with no consumer rule, dead syslog gateway, CI observability loss, edge failover designed-not-realized, DNSSEC unanchored) + the prioritized P1–P3 fix program. Point-in-time record — fixes update it | Security/Ops | | 43 | 43-lab-secrets.md — lab-secrets store: the instance's ansible-vault-encrypted *.vault.yml + lab.kdbx live in a dedicated Tier-2 lab-secrets Forgejo repo (kept out of the public myos repo). A control-role oneshot (load-lab-secrets.sh) pulls it with a read-only deploy key and overlays the files into group_vars/all/ before any converge. Vault password stays TPM-sealed (repo read = ciphertext only); deploy key is escrowed bootstrap material | Security/Ops |

| 44 | 44-assistant-research-stack.md — assistant + research stack (the GPU host): Open WebUI (assistant.${domain}) + local-deep-research (research.${domain}) + litellm sidecar as the lab's general assistant / deep-research plane — Tier-1 via origin Caddy + Authelia trusted-header SSO (casey-only), shared local vLLM + SearXNG (JSON), cloud-fallback via escrowed Anthropic key, host-networking + ufw origin-only invariant, monthly pin-bump discipline. opencode (rb32) stays the coding agent | Config/Ops | | 45 | 45-yubikey-operations.md — YubiKey device operations: the token itself — the applets (PIV/FIDO2/OTP; OATH/OpenPGP unused), every PIN/PUK/management-key and what each gates, the two-key redundancy model (independent per-token 9c/9d/9a; the old shared OTP challenge-response lab.kdbx factor is retired — the DB is password-only, armed via rb47), a command reference per lab operation, web-GUI login (WebAuthn is per-token and non-exportable → register both keys everywhere; no lab site uses it today), enrolling and retiring a token, and the non-destructive + destructive drill suites. rb20 remains the SSH-CA manual | Security/Ops |

| 46 | 46-key-rotation.md — rotating the lab's keys (dependency-ordered): answers "is a full key rotation even possible?" — yes, but leaves-first and make-before-break, never as one operation. The five root secrets, the bootstrap dependency graph, the six circular dependencies and how each is broken, a Phase 0 escrow pre-flight, a 7-phase rotation order, and a catalogue of lockout orderings (L1: rotating tang1's keys before re-encrypting the JWEs can take the whole lab down). Hygiene, not incident response | Security/Ops | | 47 | 47-keepass-presence-unlock.md — KeePass presence-unlock (YubiKey-gated lab.kdbx): unlock the vault only when the operator is present, no master-password typing. The YubiKey on atomic decrypts a per-session master-password (age-plugin, multi-recipient to both tokens) and ships it over SSH to a short-TTL tmpfs credential on ctrl (arm-kdbx/kdbx-get/disarm); 9a/9c/9d untouched; manual kdbx-unlock stays break-glass | Security/Ops | | 48 | 48-jitsi-meet.md — Jitsi Meet (self-hosted video, E2EE) on a dedicated public VPS: a fourth standalone public plane — docker-jitsi-meet podman quadlets (web/prosody/jicofo/jvb + an OIDC→JWT adapter) on the box's OWN public IP + UDP/10000 media, not on the edge/origin planes and with no home tunnel. Terminates its own TLS (LE HTTP-01); E2EE on by default so the vendor can't read media; Authelia OIDC gates room creation (guests → lobby). Survives a home/Comcast outage; deliberately off central obs (jitsi role) | Config/Ops | | 50 | 50-someonetoldme.md — someonetoldme.org: the FIRST fleet app built from our own source rather than a pinned upstream image. Four quadlets from one image (postgres + one-shot migrate + web + worker); built on runner-pub and pushed to zot as apps/someonetoldme by the scoped ci-pub account, pinned by digest (resolve it from the REGISTRY — podman reports a different one). Records the traps it actually hit: ALLOWED_HOSTS needing 127.0.0.1 for the container's own healthcheck, an image with neither wget nor curl, and db_backups running in a separate play | Apps | | 49 | 49-forgejo-upgrade.md — crossing Forgejo majors (both instances): the migrations are one-way, so step one major at a time with a restore point before every start — never skip. Staging with -e (branch protection forbids per-step commits) and the downgrade trap that creates; the zot pre-pull guard (a wedged registry hangs, so pull every tag up front); public instance first as the migration canary; and the control-plane's integration surface — Authelia OIDC/PKCE, CI runner, and GPG merge-signing, whose failure silently freezes the host-CA signer. Forgejo is lock_major in the rb34 image check on purpose — a newer major is reported as a note, never as "stale" — so these bumps stay operator-initiated | Config/Ops |

Experimental (not part of the clean build): 22-bootc-workstations.md — image-mode Fedora Atomic workstations pilot (built in CI → registry → bootc; the GPU host stays Ubuntu/OEM; servers out of scope). Parked; revisit later.

Soup-to-nuts: provision first (runbook 01 = provision/, qm clone+cloud-init), build through the middle, Ansible config & maintenance (runbook 16 = ansible/, boring-&-pinned roles + playbooks; baseline applied at runbook 09).

Edge/origin build order (see ../CLAUDE.md): - Origin (home), runbooks 01–16: the full lab, inbound-silent — the origin Caddy + hidden Bind primary (runbook 09) are internal-only; the only WAN inbound is the WG admin endpoint (.69:51820). - First edge, runbook 17: one VPS (Caddy-L4 passthrough + Bind secondary + WG tunnel home dials out); delegate one site/NS; verify end-to-end. - Multi-vendor, runbook 17 repeated: 2–3 diverse-vendor edges for NS/serving redundancy; home stays inbound-silent (only WG admin).

Notes: - GUI steps target OPNsense 26.1 (e.g. NAT "Port Forward" is now Destination NAT; NAT rules need an explicit Filter rule = Pass/Register/Manual; the Rules GUI was redesigned; Kea DHCP puts gateway/DNS under Option data). Labels can drift by point release. - Do them in order; each ends with a validation gate — don't proceed until it passes. - Several runbooks have "Facts to confirm" placeholders (bridge names, host IPs, ISP/MAC behavior, AP capabilities) — fill those in for your environment. - Generate fresh secrets for every service (WireGuard keys, passwords, TSIG, vault items) — never reuse keys from anywhere. See the secrets note in ../CLAUDE.md.