CLAUDE.md — ${domain} home lab

What this repo is

The infrastructure-as-code and runbooks for the ${domain} home lab — a segmented home network/firewall (OPNsense) plus a small AlmaLinux 9 server fleet, built to be reproducible from a clean install. The pieces:

• provision/ — qm clone + cloud-init shell scripts that define and create the VMs (Proxmox CLI only).
• ansible/ — configuration (built-in modules only; pinned ansible-core).
• runbooks/ — ordered, step-by-step build/validate guides; see runbooks/00-index.md.
• docs/ — the public, reproducible-for-others guides (mkdocs → myos.io).

⚠️ Secrets. Never commit plaintext keys or passwords. Secrets live in the Ansible vault (ansible/group_vars/all/vault.yml); WireGuard and host keys are generated on build, not stored in the repo. Don't paste secrets into chat or commits.

The goal

A fully robust home lab, not just a firewall: trust-based network segmentation, an edge/origin split that serves public sites from remote multi-vendor VPSes while TLS terminates at home (the ISP and the VPS vendors are treated as adversaries), WireGuard-only management access, a road-warrior VPN, and central identity (FreeIPA) backing NFS home directories — on the latest stable OPNsense and a single AlmaLinux 9 fleet, kept low-churn and rebuildable.

What "adversary" means here. Throughout this doc, "adversary" / "threat model" refers to the mundane, pervasive misuse of data, not (or not only) the sexy, scary stuff — rogue states, targeted international espionage. The thing being defended against is the slow death by a thousand cuts: ISPs, VPS vendors, data brokers, and ad-tech monetizing your traffic, building behavioral/advertising profiles, and tricking or nudging users. So calling the ISP and the VPS vendors "adversaries" means simply that they are positioned to harvest, sell, or profile your traffic — and the design denies them that by default. It is privacy/anti-profiling hygiene, not spy-vs-spy.

Design decisions (the why)

Grouped by theme; each detail is expanded in the matching Target design subsection.

Threat model & egress — the ISP (Comcast Business) is the adversary

• Threat model: the ISP (Comcast Business) is the adversary — hence all egress (incl. servers) via Proton, MSS clamping on the tunnel path (~1380), Unbound recursion bound to the Proton tunnels (no DNS leak to Comcast), and the OPNsense VM sized for VPN throughput (8 vCPU + multiqueue NICs, kernel WG is CPU-bound).
• Everything via ProtonVPN for all outbound (clients + servers) — never local traffic, never inbound. The VPN_BYPASS alias is a safety valve (the rule hook is in every zone) but is empty by default, so by policy everything egresses via Proton; you add a destination only when Proton's exit IP gets blocked there, sending just that out the real WAN.
• ProtonVPN uses a load-balanced gateway group — all tunnels in Tier 1, load-balanced per-connection ("sticky connections" on). They all ride the same Comcast WAN, so a standby tier buys nothing. Gateway monitoring is disabled on the Proton tunnels — ICMP monitoring over Proton WireGuard is unreliable (dpinger can't get replies through the tunnel; see runbook 05), so the working setup turns it off; the tradeoff is no automatic per-tunnel failover (a dead Proton server keeps its share of connections until disabled by hand, or add the optional endpoint-monitor). Key Proton gotcha: every Proton config uses the same client address 10.2.0.2, but OPNsense needs unique interface IPs — so each tunnel gets a unique /28 address and outbound NAT rewrites the source back to 10.2.0.2 so Proton accepts all four (runbook 05 §5). Tier 2 is reserved for a genuinely independent uplink (a cellular modem / second ISP running its own Proton tunnel) — the only thing that survives a Comcast outage. Not bandwidth bonding — a single stream still rides one tunnel.
• ProtonVPN runs on OPNsense (gateway group, not a separate container).
• Break-glass: a disabled floating rule (internal → WAN_GW) you manually enable only when all of Proton is down — never an automatic WAN failover tier (that would silently leak to the ISP).

Inbound posture (edge/origin) — the ISP threat extended to ingress

• Edge/origin inbound posture (the ISP threat extended to ingress): home serves no public services on the Comcast link — the only inbound exception is the WG admin endpoint on .69:51820. The public-facing layer lives on thin, stateless, multi-vendor mini-VPS "edges" doing TLS-SNI passthrough (caddy-l4) to the origin Caddy at home, which terminates TLS — so no VPS vendor ever sees plaintext. Home dials out the edge WireGuard tunnels (no static needed for them). edge public block roles: .67/.68 are the edge servers (public ns1/ns2 + caddy-l4 SNI passthrough); .69 is the WG-admin endpoint — WireGuard + sshd ONLY (no web/DNS on it); .65 and .66 are reserve. Hidden Bind primary at home → public Bind secondaries on the edges (registrar delegates to the edges). See runbook 17; phased (origin → 1 edge → multi-vendor).
• WG-only (strict) management: zero standing access to MGMT.

Network shape

• IP space: the lab lives in 10.20.0.0/16 — zones at 10.20.{10,20,30,40,60,80,90}.0/24, VLAN-tagged on the LAN trunk.
• Zone assignment by exposure/trust, not medium: WORKSTATIONS holds all trusted stationary devices that never leave the house — wired desktops and stationary wifi (same trust → one zone; it's also the no-tunnel fallback at home). Anything that leaves the house (laptop/phone) lives in GUEST and reaches internal services only via always-on WireGuard — per-peer, identical at home and away, nothing to toggle (see Trust zones).
• Keep a locked-down IOT zone (discovery solved via mDNS reflector).
• IPv4-only build; IPv6 cleanly OFF (WAN v6 = none, no RA/DHCPv6 on zones). No internal benefit to v6 here, and dual-stack would double the firewall rules and risk IPv6 leaking around ProtonVPN (a v6 kill-switch would be required). Keeping v6 consistently off (not half-on) is what avoids the "apps try v6, hang, need -4" problem. Revisit only as a deliberate dual-stack phase later (Proton supports v6 egress). Optional fleet hardening: prefer-IPv4 via gai.conf.

Identity & per-service auth

• FreeIPA for central identity (LDAP + Kerberos + DNS + CA).
• Per-service auth = local break-glass admin + IPA-backed users. Every server app keeps a local admin (username/password in the KeePass DB, not an IPA account) so an admin is never locked out when IPA is down/unreachable; user logins go through IPA wherever the app supports it (LDAP, or Authelia-OIDC which is itself IPA-backed) so people are centrally managed (add/remove/group/HBAC). This is already the shape of the fleet: SSH uses the SSH-CA almalinux local cert (HBAC-immune) + IPA for everything else; Jellyfin = local jfadmin + IPA users via the LDAP plugin; Authelia (sso, replaced Authentik 2026-07-06) = config-as-code + IPA users via its LDAP→FreeIPA source (forward-auth for search/fmd + OIDC for Grafana); the control-plane Forgejo (git.${domain}) = local admin + IPA users via Authelia OIDC (a DMZ box that can't reach the identity core, so the token-exchange hairpins through the origin Caddy; the public forgejo-pub stays local-only to keep the internet-facing instance off identity — runbooks 13 + 35). Appliances that can't do IPA user logins (e.g. tvheadend, and ntfy — no LDAP/OIDC backend) stay local-only but still get a real named local admin (no anonymous-admin). ntfy runs auth-default-access: deny-all + a fixed ntfyadmin + per-user/per-app tokens (the grafana publish token is write-only to lab-alerts); users/tokens live in its persisted auth.db, managed out-of-band (podman exec ntfy ntfy user|access|token). The Ansible vault holds what automation needs; KeePass is the human break-glass record (some admin passwords live in both). Both the vault files (vault.yml + *.vault.yml) and lab.kdbx live encrypted in a dedicated lab-secrets Forgejo repo (Tier-2 private), kept out of the public myos repo; a control-role oneshot overlays them into group_vars/all/ before any converge with a read-only deploy key — the vault password stays TPM-sealed, so a repo read yields only ciphertext (runbook 43).

Platform, provisioning & backups (low-churn, rebuildable)

• Latest stable OPNsense; configuration built clean.
• One OS family: AlmaLinux 9 for the whole Linux fleet (Caddy, Bind, Authelia host (sso), FreeIPA, fileserver). Chosen for: FreeIPA's native platform, ~10-yr lifecycle (fewer upgrades = less churn), SELinux-enforcing, and killing mixed-shop complexity. Rocky is a 1:1 substitute. (Workstations stay whatever they are — they just enroll as IPA clients. tvheadend is the lone Debian exception, for its in-kernel DVB drivers.)
• Proxmox pool lab: all VMs go in a Pool (Datacenter → Pools) for visual/permission grouping; the provision script sets --pool on clone.
• IaC, boring & pinned (low-churn is a hard requirement): provisioning is a qm clone/pct create + cloud-init shell script (provision/lab-provision.sh) using only Proxmox's CLI — no OpenTofu/Terraform, no third-party providers. Each vms.conf row's TYPE (vm|lxc) picks the guest form (taxonomy below); all guests land on the encrypted SSD vmpool-enc (runbook 25). Config is Ansible with built-in modules only (ansible/), installed as a pinned ansible-core (not the big ansible collection bundle). Appliances (OPNsense, FreeIPA server, Home Assistant) stay as runbooks / manual installs; the Linux fleet is provisioned + configured by code — including storage (NFS is served by an AlmaLinux 9 VM running OpenZFS, the nfs_server role — see "Identity + NFS"). This combo gives full destroy-and-rebuild with minimal moving parts.
• Guest form = VM or LXC, by need; per-app segmentation unchanged. Every app keeps its own IP/zone/firewall (consolidation onto shared hosts is out — it would collapse the trust model). VM for own-kernel / PCI+USB passthrough / appliance / vTPM guests: OPNsense, Home Assistant, tvheadend, fileserver, FreeIPA, ctrl (vTPM-sealed vault), runner (executes arbitrary CI job containers/builds via rootful podman — needs a real kernel boundary for isolation + build/overlayfs sanity, a poor LXC fit). LXC (unprivileged, nesting=1) for the lighter native single-app guests (the 3 Caddy proxies, bind, jump, sso (Authelia), registry (zot) — these run as LXC today) + — as the end-state, not yet realized — the fixed-quadlet podman/docker hosts (obs, ntfy, forgejo, searxng, fmd, jellyfin, invidious, umami). Those 8 currently run as VMs in provision/vms.conf on purpose: per the statefulness caveat below they were disk-moved with their data, and converting each to LXC is gated on escrowing its state in the vault first — so the guest-form list is the target, and the VM→LXC conversion is a deliberate, per-host follow-up, not a claim about the current fleet. The stock CT lacks the sshd/almalinux/CA-trust the VM cloud-image baked, so lab-provision.sh reproduces them post-create; common then takes over (it skips the VM-only qemu-guest-agent on containers). SSD alone fixed the boot storm, so LXC is a lightness refinement, not a requirement.
• Runtime form = who wrote the software, not what feels modern (operator, 2026-08-22). The guest form above (VM vs LXC) says where a workload lives; this says how it runs inside it, and the three cases do not overlap:
• Our own programs → a plain hardened systemd unit. shelf, alloy, syncthing, caddy, bind. We already own the source or the config; wrapping them in an image buys nothing and costs a supply chain, a registry pin and a second thing to debug. What bounds them is the unit — ProtectSystem=strict, an empty CapabilityBoundingSet, a seccomp filter, and where isolation must be structural, a mount namespace (TemporaryFileSystem= + BindPaths=) so other tenants' paths are absent rather than merely unreadable (runbook 55 §9.1a).
• Upstream's software → podman + OCI, because the image supply chain is the thing being bought. Authelia, Forgejo, Jellyfin, lldap, umami, Invidious. We are not going to build or patch these, so what we actually want from the vendor is a reproducible, digest-pinnable artifact — and having taken it, it belongs behind a container boundary, tag/digest-pinned, with podman-auto-update masked and staleness watched by the rb34 probe.
• A full userspace → LXC, which is already the guest tier above. Nothing extra to decide.

systemd-nspawn is deliberately absent. It is a container manager for booting OS trees: for one service it means running and debugging a second supervision system to obtain isolation systemd's own directives already give, and for a whole userspace we have LXC. The case it would serve is the case neither of the other two leaves open.

• Statefulness caveat (learned the hard way — the vmpool migration). "Stateless" means config-as-code with no local data/identity. The fixed-quadlet podman hosts above hold app data (DBs, repos, metrics history) and bind (DNSSEC keys + zone files) / origin-caddy (its WireGuard identity key) hold state peers or restores depend on — so in practice these were disk-moved onto vmpool-enc (data preserved), not rebuilt (a fresh rebuild loses that state and breaks downstream — e.g. the edge tunnels / DNSSEC). Convert such a guest to LXC only after escrowing its state in the vault (zone files + DNSSEC keys for bind; the WG private key for origin-caddy). Truly stateless rebuilds (jump, the Caddy proxies, bind with escrow) are safe. One LXC gotcha: a container that forwards over a WireGuard tunnel (origin-caddy: edge↔DMZ AXFR/NOTIFY) must have wg0 in a firewalld zone (public) or firewalld (nftables) silently drops the new tunnel→DMZ flows — NAT/masquerade itself works fine in a nesting LXC. (Codified in the origin_caddy role.)
• Guest storage = encrypted vmpool-enc (ZFS-native aes-256-gcm), Tang-unlocked. The SSD vmpool/enc auto-unlocks at boot via Clevis + Tang (NBDE) against tang1 — key in RAM only, fail-closed off-LAN/stolen, self-healing (a timer retries until tang1 is reachable), with a manual passphrase break-glass and 3-way key escrow (KeePass + offline + the gitignored vmpool-enc-key.vault.yml). The fileserver/supernas pool gets the same treatment in a later milestone. See runbook 25 (build + full DR).
• Backups: the real sources → an 18 TB USB pool on bizon, direct (runbook 40). The pve box has two SSDs — vmpool (NVMe, encrypted guest disks) + the fileserver's passed-through NVMe (supernas, data + the RPM/DEB blessed mirror) — plus a 2.7 TB HDD (rpool) carrying the Proxmox OS. rpool used to be the on-box backup target for both SSD pools, but that Seagate started throwing uncorrectable errors while being the intermediary for every backup, so the design was reworked (2026-07-25) to send the three sources straight to bizon's wd18tb2311/lab-backup, keeping backups alive through a rpool failure (disk replacement + a mirrored boot pool is the pending hardware step):
• Guests (vmpool/enc/vmdata + rpool/data) + pve OS (rpool/ROOT) → bizon, pve PUSHES (MGMT→SERVERS; bizon can't reach MGMT): syncoid -r --no-sync-snap rides sanoid hourly crash-consistent autosnaps (sanoid now covers all three trees), hourly at :20 (guest-repl-bizon). App-consistent DB dumps still ride along via db_backups. (vzdump + the var-lib-vz push were the last rpool-dependent tier — retired 2026-07-25* after a 4-cycle soak, so rpool is now out of every backup path.)
• supernas → bizon, bizon PULLS direct from the fileserver (SERVERS→SERVERS): a dedicated bizon key on the restricted, zfs-delegated syncoid user (source-locked), daily 04:45 (labbackup-pull-supernas) — fully pve/rpool-independent. Carries home/media/recordings + the blessed mirror-current/deb-mirror-current w/ snapshots.
• bizon target invariants (unchanged): a genuinely separate machine, receive-only delegated (create/receive/mount/hold, no destroy — a compromised pve can't erase history), own encryption root (/root/.zfs-labbackup.key on bizon's internal disk, escrowed in KeePass; non-raw sends re-encrypt at rest under it). Off-site is the remaining step (fire/theft; the nfs_*_replication_target hooks). pve also gained a ZED→ntfy zedlet (rb27/42) so ZFS data/state errors page (they didn't before). Code rebuilds the boxes; these backups restore the data. See runbooks 40 + 11.
• Supply-chain: a pinned RPM snapshot mirror (the mirror host, runbook 37). The threat model extends to the package supply chain — a poisoned upstream RPM. Defense in three layers: (1) gpgcheck=1 enforced fleet-wide + signed metadata preserved on sync; (2) a local, immutable, point-in-time mirror — a stateless mirror VM reposyncs AlmaLinux+EPEL+ZFS into the fileserver's ZFS (over NFS) and the fleet pins to a blessed snapshot (use_rpm_mirror), so it can roll back to a known-good set even if upstream is poisoned/removed — self-sufficient, no upstream needed to restore; (3) a time-delayed "bless" (soak) — the fileserver snapshots each clean nightly sync and a timer auto-promotes the snapshot that has soaked N days (nfs_mirror_soak_days, default 3), so the fleet installs "upstream as of N days ago" and a pkg caught poisoned within that window never reaches the fleet — no manual gate, with bless-mirror.sh/bless-deb-mirror.sh kept as an emergency fast-track (rb37/39). Blessing = zfs snapshot ON the fileserver; the internet-facing mirror host is an NFS client with no snapshot authority, so a compromised sync can't rewrite the blessed history (the risky sync is separated from the immutable retention). Same instinct as the edge/origin split: isolate the thing that touches the outside from the thing that holds the trust. (VM, not LXC: NFS is refused in an unprivileged LXC; a VM also hardens the boundary for an internet-pulling box.) The same host + bless model covers Debian for the lone Debian box (tvheadend): debmirror runs in a Debian container (its base image pulled through zot) mirroring the full trixie archive into a parallel ZFS dataset, served at mirror.${domain}/debian, pinned via use_deb_mirror (deb_mirror role, runbook 39).

Target design

Trust zones (default-deny between all zones; gateway = .1 in each)

VLANs are tagged on the LAN NIC (vtnet0); the Proxmox LAN bridge must be VLAN-aware/trunk to carry them.

A zone is a trust boundary — not a way to categorise devices by medium or role. The seven zones group into four trust tiers, and that's what justifies each one (it's also why wired + stationary-wifi desktops share one zone: same trust, only the medium differs):

• Isolated / high-value — MGMT (the infrastructure) and DMZ (the edge/ public-facing layer): the two highest-impact blast radii, kept furthest from everything.
• Trusted-internal — SERVERS (the data/identity tier) and WORKSTATIONS (trusted stationary clients): separated from each other on an allow-list, both walled off from the untrusted + edge tiers.
• Untrusted-devices — IOT (appliances) and GUEST (roamers + visitors): blocked from all RFC1918; internet only (roamers reach internal via always-on WG).
• Remote — the WG road-warrior tunnel.

New zone? Ask "what trust boundary does it draw?" — if the answer is "none, it just organises devices," it shouldn't be a zone.

Zone	VLAN	Subnet	Contents	Internet path
MGMT	10	10.20.10.0/24	Firewall, Proxmox host, switch, APs, IPMI	Proton (bypass→WAN)
SERVERS	20	10.20.20.0/24	FreeIPA, Authelia/sso (SSO), fileserver (ZFS/NFS), app backends	Proton out; WAN in
DMZ (edge-facing)	30	10.20.30.0/24	internal origin Caddy (TLS terminator) + hidden Bind primary; reached only via the edge WG tunnels	Proton out; in via edge tunnels
WORKSTATIONS	40	10.20.40.0/24	Trusted stationary clients — wired desktops + stationary wifi (NFS home, SSO)	Proton (bypass→WAN)
IOT	60	10.20.60.0/24	Roku, LibreELEC/Kodi box (KODI_BOX .100, scoped tvheadend + Invidious-backend allows — untrusted plugins keep it out of WORKSTATIONS, rb06 §IOT), wifi printer, smart home	Proton; YouTube→WAN
GUEST	90	10.20.90.0/24	Roaming devices (laptop/phone) + visitors; client isolation; internal via always-on WG	Proton (or WAN)
WG road-warrior	—	10.20.80.0/24	Remote dial-in peers	n/a

Zone assignment is by exposure, not medium. A device that leaves the house is higher-risk (it touches untrusted networks), so it is treated as untrusted: laptops and phones live in GUEST and reach internal services only over always-on WireGuard, exactly like any remote peer — so the access model is identical at home and away. Stationary devices that stay home — wired desktops and stationary wifi — are trusted and share the one WORKSTATIONS zone, which doubles as the no-tunnel fallback: if WG is down at home you can join it for direct access.

Host & IP allocation (single source of truth)

Gateway = .1 in every zone. DHCP pools .100–.199; static/servers .2–.99. Cloned VMs and their resources are defined in provision/vms.conf; this table is the human-readable cross-reference. Provisioning class in the Notes: clone+Ansible (the default), clone+manual (VM cloned, app set up by hand), appliance (manually installed, not cloned), external.

Host	Zone	IP	Notes
OPNsense (per-zone gw)	all	10.20.<vlan>.1	firewall; appliance (runbooks 01–08)
Proxmox host (pve)	MGMT	10.20.10.5	hypervisor (:8006); appliance. Major upgrades = runbook 54 — the one procedure whose blast radius is the whole lab (opnsense + ctrl are guests here, so the network drops with it and the agent can't drive its own recovery; the reboot half is operator-run from the IPMI console)
mgmt-proxy	MGMT	10.20.10.6	internal MGMT Caddy (wildcard cert; infra UIs; WG-admin only)
ctrl	MGMT	10.20.10.8	dedicated Ansible control node (pinned ansible-core; reaches the fleet via a CA-signed control cert; lab.kdbx; vault password TPM-sealed); runbook 15. Also runs the automated host-CA signer (runbook 26) + the obs dead-man's-switch (curls obs health → ntfy directly, since Grafana can't alert on its own death — runbook 27) + the 04:00 daily digest (one proof-of-life + fleet-summary ntfy/day, derived from live VM queries, incl. every standing Grafana alarm — the notification policy pages each alarm ONCE (repeat_interval 120h, 2026-09-06) and the digest carries it until it clears, untagged so it never wakes the agent; its absence is the alarm — runbook 27) + the daily fw-conformance run (03:30, real-packet zone probes incl. the iot-canary vantage; transition-only ntfy + digest line — runbook 18) + the weekly freeipa baseline converge (Sun 05:30, between the signer's /opt/myos advance and the drift check: the agent PAW cert cannot reach the identity core, so merged baseline changes reach freeipa only via this control-identity converge — runbook 15 §6b)) + the alert-scan doorbell (2-min poll of lab-alerts; spawns a headless triage agent only when a triage-tagged alarm is pending — no idle agent session; runbook 28)
tang1	MGMT	10.20.10.11	NBDE/Tang keyserver for Clevis network-bound auto-unlock of pve's encrypted vmpool; single-function Raspberry Pi (Pi OS); appliance (physical; out of the fleet Ansible runs — provision/tang1-setup.sh + runbook 24). The lab's first physical infra host; on MGMT (not VLAN 20) to avoid a circular boot dependency. Static IP, NOT DHCP (DHCP/Kea runs on OPNsense, a VM on pve — a DHCP-dependent keyserver would deadlock pve's boot-unlock; a dormant Kea reservation on its MAC mirrors .11 as backup). .12 stays reserved, but a second tang2 geo node was declined in favour of plaintext key escrow (rb24 decision record, 2026-07-10) — tang1 is the sole keyserver by choice.
UniFi controller	MGMT	10.20.10.101	network controller (APs/switch); appliance
FreeIPA	SERVERS	10.20.20.2	LDAP/Kerberos/DNS/CA; clone+manual (server installed per runbook 10)
apps-proxy	SERVERS	10.20.20.6 (+ .7)	internal user-apps Caddy (Tier 2 private: HA + Jellyfin + Invidious + umami GUI + git (Forgejo internal front-door) + search (searxng) + sync (shelf — the file-sync front door, rb52; DUAL-PLANE since 2026-09-13 like radio: this Tier-2 face is unchanged and stays the LAN/WG path, while a Tier-1 origin face behind the same Authelia serves it from the internet — origin → fileserver :8081 direct, DMZ→SERVERS host-scoped seq 546, no bound listener needed because the backend is in SERVERS) + dav (Radicale — deliberately NOT promoted: a CalDAV client cannot traverse an SSO redirect, rb56 §7 O2) + auth (Authelia) + ntfy + meet-start (Jitsi JWT issuer) + radio (OpenWebRX+ SDR on adac — the first WORKSTATIONS backend, scoped apps-proxy→adac:8073 allow; DUAL-PLANE: this Tier-2 face is network-anchored with NO forward-auth so the native Android app works — the Jellyfin exception — while the Tier-1 origin face is Authelia-gated for web-from-anywhere, relayed via the radio-only bound listener .8); split-horizon DNS) — vhost list = ansible/group_vars/apps. .7 = the invidious BOUND front door (a /32 alias on eth0 — role-managed, deliberately NOT a second NIC; exact-host Caddy site pinned to it, wildcard plane pinned to .6, other names get an explicit 421): the one Tier-2 name granted to the untrusted Kodi box, expressible as an L3 rule only because the name has its own IP (rb06 §IOT; fw-conformance asserts .7 serves nothing else). .8 = the radio origin-relay bound listener (same own-IP trick, 2026-09-02): the Tier-1 radio path is origin → .8:443 (DMZ→SERVERS host-scoped, seq 645) → the existing SERVERS→WKS allow → adac:8073 — because DMZ→WORKSTATIONS stays closed
fileserver (AlmaLinux ZFS)	SERVERS	10.20.20.10	OpenZFS on a passed-through NVMe; NFSv4 sec=sys home/media/recordings/sync. Also the file-sync service since 2026-08-31 (file_sync role, runbook 52): syncthing (phone camera roll) + shelf (sync.${domain} via apps-proxy — dual-plane since 2026-09-13: the same :8081 shelf-map socket is also served publicly by the origin Caddy behind Authelia, so the household hub works without WG; three layers grant that door and all are required — shelf_trusted_proxies, host_firewall_source_ports, fw-automation seq 546; per-tenant since rb55 §9) writing plain files onto the storage tier — no application database in the data path, which is the whole point. It superseded Seafile (rb41) and Nextcloud, whose libraries were migrated here, so supernas/sync is now the only copy of that data: it carries sanoid + the bizon pull, and a sync-freshness metric pages when a tree stops receiving (the failure a green unit never reports) + the notes surface (rb52, shelf SPEC §15; the operator's notes at supernas/sync/casey/notes, workstation-edited over syncthing — shelf serves and backs them up, it is never the editor). shelf is PUSH-TO-DEPLOY since 2026-09-13 (rb52 §Push-to-deploy): the public CI SSHes in as the deploy-shelf account — a force-command cert, one program, one validated tag, host-scoped to runner-pub (fw-automation seq 547) — and the installer verifies the release's signed merge against a pinned instance-key fingerprint before switching a versioned symlink; the myos pin is now only the floor a rebuild starts from
registry	SERVERS	10.20.20.15	internal OCI registry + pull-through cache (zot); hosts our images + caches upstreams; registry.${domain} via apps-proxy (runbook 30)
mirror	SERVERS	10.20.20.16	pinned RPM + DEB snapshot mirror (supply-chain resilience); reposyncs AlmaLinux+EPEL+ZFS and debmirrors Debian (in a container, base via zot) into the fileserver ZFS (over NFS), served at mirror.${domain} via its own caddy. Blessing = zfs snapshot on the fileserver (this host = NFS client, no snapshot authority). VM (NFS refused in unprivileged LXC). rpm_mirror + deb_mirror roles (runbooks 37, 39)
Jellyfin	SERVERS	10.20.20.30	media server (EL9 podman quadlet); LDAP→FreeIPA login; library via NFS
tvheadend	SERVERS	10.20.20.31	DVR/tuner backend (Debian; USB tuner passthrough); feeds Jellyfin Live TV
sso	SERVERS	10.20.20.41	Authelia — SSO/forward-auth + OIDC plane (LXC podman quadlet; LDAP→FreeIPA via svc-ldap; SQLite state). Replaced Authentik 2026-07-06: auth.${domain} portal, forward-auth for search/fmd, OIDC for Grafana + control-plane Forgejo. authelia role. Also hosts meet-start (:9092, meet_issuer role) — the Jitsi meeting-start JWT issuer behind its forward-auth (runbook 48 §Auth)
Home Assistant	SERVERS	10.20.20.45	HAOS appliance (Zigbee USB), installed by hand; controls IOT via allow-rule + mDNS
obs	SERVERS	10.20.20.50	observability sink (Grafana + VictoriaMetrics + Loki + Alloy); failure alerting = a ZFS permanent-data-error rule (a pool reports ONLINE with all device counters 0 while zpool status says N data errors — found on pve's rpool where every collected signal said healthy, because the quantity was measured nowhere; now emitted by both the fleet role and pve's own pusher), alloy systemd collector → node_systemd_unit_state{state="failed"} + a Grafana "Systemd unit failed" rule → ntfy on ANY crashed service/quadlet-container, a stale-timer rule (scheduled job silently stopped — expectations derived from each host's own timers, no hand-kept list), log-pattern rules (Loki; app fails while its unit stays green, e.g. invidious PO-token), (a restart-loop rule was tried and withdrawn 2026-08-02 — fleet-unit-failed matches state="failed", which an auto-restarting unit NEVER enters, so a 233k-restart crash loop went unseen; but no available metric separates that from a legitimately long Type=oneshot, and the real fix needs node_exporter's restarts counter — runbook 27), a rule-drift check (Grafana provisioning only ever ADDS — a rule deleted from the template keeps firing forever — so a timer compares the rendered file against Grafana's own DB in both directions, plus a third rule for "the checker itself cannot run", since a broken detector reporting no problems found is worse than none), a pending-reboot rule (the DETECT half of the OS update layer: dnf-automatic installs a kernel but it is inert until reboot, and update.yml — which reboots correctly — is operator-run, so between runs a host sits on a superseded kernel while units/probes/packages all read healthy; found on both public edges 2026-08-19. The grace window comes from a host-side since stamp, not Grafana's for:, which re-arms on any scrape gap or unrelated reboot; and when the detector cannot answer it emits no condition value rather than a 0, with a companion "detector cannot run" rule — same reasoning as the drift checker's third rule), plus a ctrl-side dead-man's-switch + 04:00 daily digest for obs/liveness itself (runbook 27). Notification policy = one page per alarm (repeat_interval 120h, Grafana's nflog ceiling; 2026-09-06): a standing condition pages at onset and is then CAPTURED, not repeated — obs exports the live firing set as grafana_alert_firing (a 5-min textfile timer reading the alertmanager API on localhost, zero new secrets) and the digest lists it daily; a new instance joining a group still pages
ntfy	SERVERS	10.20.20.60	notification bus (UnifiedPush distributor); token-auth (deny-all), local users + ntfyadmin; lab-alerts = Grafana alert topic. Also hosts the ntfy→XMPP bridge bots (ntfy_xmpp_bridge role, rb53 §bridge): alerts@ relays lab-alerts into the chat app (read-only token), triage@ republishes a forwarded alert tagged triage to wake the agent (write-only token) and relays the triage-status topic back into the conversation (the working agent's progress lines — request → progress → outcome in one thread, rb53 §bridge) — the bots live beside the bus's privilege and dial outbound to chat; ntfy stays the alarm bus, XMPP is a view of it
chat	SERVERS	10.20.20.61	Prosody XMPP — the lab control channel (runbook 53). A contact list on the phone where each contact is a lab function; messaging one dispatches a deterministic command (Downloader → media-fetch on the fileserver; alerts@/triage@ = the ntfy bridge view — alarms in, forwarded-alert = wake the agent; notepad contacts (rb53 §Notepads) = per-interest JIDs whose messages a fileserver bot files into ~owner/notes/<contact>.md transcripts — receipt-only ack, names are instance identity in zz-local). Tier-2 private (WORKSTATIONS/WG-RW only; no edge, no origin vhost, no public A). Local accounts, deliberately not IPA/Authelia — a control channel must keep working when identity is degraded (the tvheadend/ntfy rule). Federation OFF (no s2s configured or bound): a boundary, not a hardening tweak, since s2s would let any remote server message the command bots. Transport only — holds no privilege, executes nothing: the bots run beside the privilege they expose and dial outbound, so compromising this guest buys the ability to message bots that refuse you (same instinct as edge/origin). Self-signed TLS pinned by the client, deliberately not the DNS-01 wildcard — the control channel must not depend on the ACME/IPA machinery it exists to help repair. No AI in the dispatch path; an agent is a separate, explicitly-invoked contact. LXC; xmpp_server role
bizon	SERVERS	10.20.20.70	GPU/vLLM host — local LLM inference for the agent/assistant plane (opencode rb32; assistant/research rb44; dsh coding agent coder.${domain} → :3081, rb44 §coder — dsh has NO own auth, Authelia + origin-only ufw are the whole gate) + ComfyUI image generation (artist.${domain}, rb44 §artist: :8188, systemd-socket-activated, GPU-timeshared with vLLM — never point a probe/health-check at :8188 or :8001, a bare TCP connect preempts the other stack) + the off-pve backup target (18 TB USB wd18tb2311/lab-backup, rb40). Ubuntu/OEM box, workstation-role-managed (not in all_linux; SSH-CA trust + host-cert autosign via admin-workstations; host_vars/bizon.yml)
jump	DMZ	10.20.30.5	SSH jump host (fleet access via edge :443 relay)
Origin Caddy	DMZ	10.20.30.10	internal TLS terminator + forward-auth (no public inbound)
forgejo	DMZ	10.20.30.30	git server (podman quadlet) — control-plane SCM (the lab_org/myos infra repo + CI/runner + registry-push target); canonical git.${domain}, Tier-2 private (apps-proxy/WG only, not on the edge)
forgejo-pub	DMZ	10.20.30.31	git server (podman quadlet) — public Forgejo, canonical git.${domain_alt}; edge-exposed via origin Caddy; public + public-related repos only, no control-plane (signs its UI merges since 2026-09-13 — the deploy signature for shelf's release-on-merge, meaningful only with operator-only branch protection on that repo, rb35)
Bind (hidden primary)	DMZ	10.20.30.40	internal authoritative primary; AXFR → edge secondaries; not delegated
searxng	DMZ	10.20.30.50	metasearch (podman quadlet)
runner	DMZ	10.20.30.60	Forgejo Actions runner (ci_runner role) for the control-plane forgejo only; offers the host label (jobs run directly on the VM) and holds the origin deploy key + zot push creds
runner-pub	DMZ	10.20.30.61	Forgejo Actions runner for the public forgejo-pub. A runner serves exactly one instance, and this is a separate VM — sharing runner would let the edge-exposed public git dispatch code onto the box holding the control-plane deploy credentials. Container-only labels (no host); outbound-only except one grant — since 2026-09-13 it holds the shelf deploy cert and may reach fileserver:22 (seq 547) to ring the push-to-deploy doorbell; that is a forced command over a cert whose principal exists on no other host, and the fileserver verifies the release itself, so the two-instance separation is unchanged (rb35 §8). ci_runner role + host_vars/runner-pub.yml (runbook 35 §8)
Invidious	DMZ	10.20.30.70	privacy YouTube frontend; Tier-2 private, served via apps-proxy at https://invidious.${domain} (like Jellyfin) — split-horizon override → apps-proxy, which reverse-proxies to this backend :3000 (needs the SERVERS→DMZ allow, runbook 06); no edge/Authelia. domain/external_port/https_only set to https+443 so generated URLs match the proxy name
fmd	DMZ	10.20.30.80	FindMyDevice server (podman quadlet)
umami	DMZ	10.20.30.90	cookieless web analytics (podman quadlet + Postgres); split stats.${domain} — collection PUBLIC (edge origin-Caddy allowlists only /script.js + /api/send* → umami; everything else 404s), GUI/dashboard PRIVATE (Tier-2 via apps-proxy split-horizon → umami :3000, needs the SERVERS→DMZ allow); Postgres → db_backups
dav	DMZ	10.20.30.110	Radicale CalDAV/CardDAV — adopted keeper (graduated from trial 2026-07-15; the Nextcloud-cal/contacts-alternative eval landed here, and Nextcloud was retired 2026-08-31 with its contacts verified content-identical here). Single podman quadlet; backend_app=radicale; Tier-2 private via apps-proxy (dav.${domain}, split-horizon); auth = the TENANT directory (lldap on tenant-dir over pinned LDAPS; htpasswd retired — rb55); also served on the tenant plane (dav.<public apex> via tenant-proxy, rb55 §11); collections on disk
someonetoldme	DMZ	10.20.30.120	someonetoldme.org Django site (LXC) — the first fleet app built from our own source rather than a pinned upstream image. Podman quadlets: postgres 16 + a one-shot migrate + web (gunicorn) + a django-q2 media worker, all from ONE image so they cannot drift apart on deps/migrations (backend_app=someonetoldme). Image is built on runner-pub from the public repo and pushed to our zot as apps/someonetoldme by the scoped ci-pub account (apps/, not sites/ — the origin's refresh-site.sh treats sites/<name> as mountable content images), then pinned by digest. Public via the edge → origin Caddy, which terminates TLS; the app serves plain HTTP, trusts X-Forwarded-* only from the origin, and ALLOWED_HOSTS must carry the public name plus its own IP plus 127.0.0.1 (its container healthcheck probes loopback). Nightly pg_dump → db_backups — the rows are guest-book entries and card records that physically etched cards point at, so they get the logical-dump tier, not snapshots alone. Media is stored LOCALLY on a 100 G disk and R2 was dropped — a vendor holding user-uploaded video in plaintext inverts the lab's own threat model, and NFS was rejected on a hard blocker (refused in an unprivileged LXC, the reason mirror is a VM) plus a runtime dependency on the fileserver. So the guest is stateful by design (the shape files/Seafile had before it was retired): media rides the hourly vmpool-enc → bizon replication (crash-consistent is fine for immutable blobs) and the DB gets the logical dump on top. PUBLIC since 2026-08-02 (Tier 0, no Authelia — a public guest-book site that etched cards point at): origin vhost → :8000 plus the edge SNI passthrough entry, both in the overlay; apex only, www has no public A record. Its podman network subnet is pinned and declared as a firewalld trusted source — without that grant the containers cannot resolve each other, and the app serves pages perfectly while unable to reach Postgres, with all three units green (runbook 50 §3). Runbook 50
tenant-proxy	DMZ	10.20.30.130	tenant-facing Caddy — the third internal_proxy instance (mgmt/apps/tenant). Serves ONLY tenant vhosts (dav today) and defines no admin vhost at all, which is what makes tenant isolation structural rather than a correctly-written bind on the shared apps-proxy. Reached only from the tenant WireGuard instance + the owner's own tiers. group_vars/tenant_proxy; runbook 55
tenant-dir	DMZ	10.20.30.140	lldap — the TENANT directory (podman quadlet; LDAPS 6360, base DN dc=tenants,dc=local; admin web UI loopback-only, reached via ssh -L, since it mints users and resets passwords). Deliberately NOT FreeIPA: FreeIPA governs who can root the fleet, so a paying customer in it is one group-membership mistake from the infrastructure — and DMZ→SERVERS identity ports are denied by design anyway. Backs Radicale auth over LDAPS. Runbook 55
iot-canary	IOT	10.20.60.9	fw-conformance probe LXC — the IOT zone's empirical test vantage (tests/fw-conformance; added after the 2026-08 intercept outage went 2 months unseen because IOT had no probe). Baseline-only, and deliberately ungranted (must represent the zone): no alloy (obs unreachable from IOT by design), no mirror pin, no IPA (host_vars/iot-canary.yml); SSH via admin-mgmt per-host principal (a mgmt-plane instrument, principals.yml)
Workstations	WORKSTATIONS	10.20.40.x	wired + stationary-wifi trusted desktops; IPA-enrolled by hand, workstation role for SSH-CA + autofs
adac	WORKSTATIONS	10.20.40.100	Pi 4 Bluetooth/network audio receiver + SDR host (Raspberry Pi OS trixie; PipeWire A2DP sink) — workstations stream audio to it (same-zone flow, its zone rationale; deliberately NO NFS/autofs). Onboard wifi+BT firmware-disabled since 2026-09-02; BT runs on a USB adapter. Also hosts OpenWebRX+ (RTL-SDR Blog V4; rootful podman quadlet, as-built like its audio pipeline) served as radio.${domain} DUAL-PLANE (the plane's first WORKSTATIONS backend): Tier-2 via apps-proxy with no forward-auth (native app; Jellyfin exception) + Tier-1 via the origin behind Authelia (web from anywhere, relayed through the apps-proxy .8 bound listener), mutually exclusive with an on-demand rtl-tcp unit. workstation role (fleet SSH-CA, no autofs) + bt_audio role: pairing closed by default, on-demand 3-min window via systemctl start bt-pairing. .40.100 pinned by a Kea reservation (2026-09-02); NOT IPA-enrolled. Recovered + onboarded 2026-08-05
WG admin peer	WG-RW	10.20.80.10	→ MGMT
WG user peers	WG-RW	10.20.80.21+	→ SERVERS (e.g. Jellyfin; NFS home shares — per-peer /32 in the fileserver export)
Edge tunnels	WG	10.20.200.0/24	per-VPS /32s (home dials out to each edge)
Edge VPSes	(external)	vendor public IPs	Caddy-L4 (SNI passthrough) + Bind secondary; multi-vendor; external (vps_edge role)
Jitsi (meet)	(public segment)	own public IP	public video plane — docker-jitsi-meet podman quadlets (web/prosody/jicofo/jvb); E2EE on by default; own TLS (LE HTTP-01, renewed in-container). Holds its own public IP on its own NIC (vms.conf VLAN col = wan, untagged on the WAN bridge) like the edge VMs — not behind a firewall VIP/NAT, which is unreachable from inside and forces split-horizon + carve-outs. On-prem interim for a future off-prem box; egress = ISP not Proton; outside OPNsense inbound filtering. room creation is JWT-gated: any IPA user starts/moderates via the Tier-2 meet-start.${domain} issuer (meet_issuer role on sso, Authelia forward-auth); meet only verifies tokens (shared secret, no route to the identity core — the tvheadend/ntfy rule holds), guests follow a plain link to the lobby (runbook 48 §Auth; superseded the per-person Prosody-internal roster 2026-08-15). clone+Ansible (jitsi role, runbook 48)

Most home Linux VMs clone the AlmaLinux 9 template (provision/vms.conf) and are configured by Ansible; tvheadend clones the Debian template (in-kernel DVB drivers). Manually-installed appliances (not cloned/Ansible): OPNsense (GUI runbooks), Home Assistant (standalone HAOS image). FreeIPA is a cloned VM whose application is installed by hand (runbook 10), not by an Ansible role. The edge VPSes are vendor-provisioned + configured by the vps_edge role.

Control plane. Ansible runs from a dedicated control-node VM (ctrl, MGMT) — not a personal workstation — so the credential that reaches the whole fleet (a CA-signed control cert: its own key signed by the SSH user CA with principals admin-{mgmt,servers,dmz,freeipa,workstations} — the canonical list is codified in yubikey-tool.sh sign-control (workstations added 2026-08-27 so the host-CA autosign and the rb34 pin-drift gather reach the workstation boxes) — accepted everywhere via TrustedUserCAKeys) and lab.kdbx live on one single-purpose, common-managed, rebuildable box; its vault password is TPM-sealed (systemd-creds, no plaintext at rest). ctrl runs Ansible as the almalinux user (never root — it only connects out). A cert (not a raw authorized_keys entry) is the fleet credential because it expires and is KRL-revocable; so fleet_transport_pubkeys carries only root@pve (the pre-cert provisioning bootstrap). CA signing is split by what the cert grants (runbook 26): the user CA — which grants access (admin certs → root) — stays YubiKey-gated on the admin laptop (ctrl's own control cert, new admins, etc. are signed there and committed to git), so the access root is offline and unreachable by any online compromise. The host CA — which grants only host identity — is now an online, automated CA on ctrl: its key is Clevis+Tang-sealed to tang1 and decrypted per-signature, so a weekly host-ca-autosign.timer renews short-lived host certs hands-off while a stolen/off-LAN ctrl fails closed (no tang1 → no signing). Trust is additive (the YubiKey host CA stays trusted as break-glass); every sign is pushed to ntfy and tang1 ships all access to Loki (chokepoint audit) — see runbooks 26 + 15.

Why segment: a flat network lets one compromised device (a phone, a guest, a hacked public website) reach everything. Segmentation contains blast radius. The two highest-impact zones — DMZ (the public sites) and MGMT (the infrastructure) — are isolated, not flat, and the most-exposed devices (roamers, IoT, guests) are kept furthest from them.

Credential custody model (four tiers + the AI-agent PAW)

Credentials are segmented by who wields them, and custody strength scales with blast radius — the same instinct as the SSH-CA split. Four tiers:

• T1 — automation (machine): what the fleet/automation uses unattended — the almalinux control cert (Tang-gated, loaded by load-control-key.sh) + the TPM-sealed vault on ctrl. Used by the timers (host-CA autosign, obs dead-man's-switch) and by ansible.
• T2 — the AI agent (Claude): the agent runs as a Privileged Access Workstation on ctrl (tmux, a dedicated claude-agent account) and reaches the fleet with its OWN keypair (Tang- sealed) + a short-lived YubiKey-signed cert (key-id claude-agent, principals admin-mgmt,admin-servers,admin-dmz,admin-workstations — not admin-freeipa: every zone incl. DMZ + the workstations (standing since 2026-07-12), but not the identity/CA core nor ctrl itself). It gets root on the fleet it services (become there) but its standing cert can't root its own PAW — claude-agent has no sudo on ctrl, and ctrl is a crown-jewel (admin-ctrl) so the standing fleet cert can't hop to almalinux@ctrl. Elevation is just-in-time, not denied: the operator signs a task-scoped cert with exactly the principals a task needs (incl. crown jewels), short-TTL — scripts/yubikey-tool.sh sign-agent-task. The ctrl-root path is an admin-ctrl task cert → ssh almalinux@ctrl → sudo -i (almalinux sudo codified; PermitRootLogin no, so root SSH stays off / console-only). The hypervisor-root path is the same shape for pve: an admin-hv task cert → ssh root@pve — a DEDICATED principal that maps to root@pve and nothing else (runbook 20 appliance table), so the agent can do JIT provisioning (lab-provision.sh, qm, pct) without the god-mode admin; sharpest jewel (pve root = full fleet), so task-only + +4h. The standing cert is signed on demand (sign-agent, +1w); cutoff (both standing + task) = expiry / KRL (scripts/push-krl.sh) / live-kill (scripts/agent-killswitch.sh). The agent's API keys live in ctrl's vault. Its access is a deliberate, hardware-granted, revocable grant — never the automation control cert. (Runbook 28.)
• T3 — human web logins: admin passwords in KeePass lab.kdbx, YubiKey-gated; keep a second YubiKey + offline recovery codes (no single-key lockout).

• T4 — human WebAuthn: FIDO2/WebAuthn on the YubiKey wherever an admin site supports it; T3 is the fallback for sites that don't.

• A fifth, narrow class — the CI deploy cert (2026-09-13, runbook 26 §A third certificate class). The same offline user CA signs it, but it grants no shell and no zone: principal deploy-shelf, mapped on exactly one host (the fileserver) to a non-admin account, with force-command pinning it to one program that takes one validated argument — a release tag. It lives in the public Forgejo's Actions secrets so a shelf release deploys itself (runbook 52 §Push-to-deploy), and the reason that is acceptable is that the credential is a doorbell, not a deploy: what gets installed is decided on the fileserver, from the release API, against a pinned instance-key signature and published checksums. It expires (+52w) and is KRL-revocable like every other cert, and the forced command is enforced server-side too, so a cert re-signed without the option cannot widen it. Custody (2026-09-13): minted from the YubiKey user-CA by the tool (scripts/yubikey-tool.sh sign-deploy <name>, never a hand-typed ssh-keygen -s), with a fresh keypair every run — so re-running it is the rotation; the private half lives only in that repo's Actions secrets (never on ctrl, never in the vault or KeePass — it is reissuable, not recoverable), and its public half is registered on ctrl (/etc/lab/deploy-certs/) purely so the 04:00 digest can watch the expiry and page once before a year-old credential silently stops a deploy.

The offline user-CA (YubiKey 9c) signs all access — both your 9a login cert and the agent cert; the host-CA (online, Tang-sealed) grants only host identity; the control cert is the automation transport. No online access-CA exists (that would break the split-CA invariant), and the agent never borrows automation's credential (T1/T2 stay separate). Deliberately rejected for the agent: enrolling DMZ in IPA (breaks DMZ isolation) and a pure-IPA agent identity (cert auth is HBAC-immune, and the cutoff must cover DMZ) — the PAW + own-cert model gives a uniform, operator- controlled kill switch across all managed computers without either.

Decision (2026-07-07): the agent stays cert + PAW + JIT — no dedicated IPA identity (declined "C4"). We revisited giving claude-agent a named IPA user + HBAC + IPA sudo — the same IPA-as-PAM model we'd use for a human operator ("treat the agent like an employee"). Declined, because the human model breaks precisely where an AI differs from a human: (1) Injectability — an agent manipulable by the data it reads must minimise standing privilege, and standing IPA sudo (even command-scoped) is a bigger injection blast radius than "lands as almalinux, elevate only via an operator-signed JIT task cert." (2) Uniform, instant, fail-closed cutoff — one lever (cert expiry / KRL / killswitch / Tang-seal) beats a split cutoff (ipa user-disable on enrolled hosts while the cert still lands on DMZ + as break-glass). (3) IPA-independence — the agent must administer the fleet including when IPA is degraded; cert access is HBAC-immune/IPA-independent by design (same reason almalinux SSH is cert-based), whereas an IPA-dependent agent could be locked out of fixing IPA. (4) DMZ is never IPA-enrolled, so an IPA identity is a second access system, not a replacement. The one real gain — attribution — is already met by the cert key-id claude-agent logged on every sshd login (address further, if needed, with a distinct login user or auth-log alerting — not a standing identity). IPA-as-PAM is reserved for humans (the identity-posture work): if a person ever fills the operator seat, they enroll as an IPA user with HBAC + sudo at that point. See runbook 28.

Approval tiers — who must approve a change to this repo (runbook 51). The single "operator approves every PR" gate was doing two unrelated jobs — holding the host-CA trust chain and spell-checking the docs site — so it is split by path, default-deny: T2 (scripts/**, .forgejo/**, ansible/**, provision/**, tests/**, CLAUDE.md, .claude/**, .gitignore, and anything unlisted) is approved and merged by the operator in the Forgejo UI; T0 (docs/**, README.md, TODO.md) and T1 (empty for now) may be approved by the delegated agent-reviewer identity and merged by the agent. All of ansible/** is T2 because the host-CA signer executes it (site.yml --tags sshca, runbook 26) — delegated tiers never contain a path the signer runs, and that is the only reason Forgejo stays in the signer's trust store. CLAUDE.md stays T2 permanently: an agent must not approve edits to its own governing policy.

Inter-zone policy (allow-list; everything else denied)

• MGMT — reachable ONLY from the WG admin peer (+ Proxmox console fallback); MGMT may reach all zones for administration.
• SERVERS — receives from WORKSTATIONS: NFSv4 (2049 only), LDAP (389/636), Kerberos (88/464), DNS (53), IPA UI (HTTP/S), plus Jellyfin/HA/tvheadend for media. Receives from DMZ only on the single app→backend port. Does not freely initiate into client zones (one codified, host+port-scoped exception: apps-proxy → adac:8073 for the radio.${domain} Tier-2 front-door — emitted by apps-frontdoor-sync.py, scoping asserted by fw-conformance). Reaches the firewall itself only for DNS/NTP — not the OPNsense web GUI (:443): SERVERS is a data/identity tier, not an admin zone, so a firewall-automation block denies SERVERS → (self):443 on every firewall IP (the SERVERS → firewall self-rule otherwise allows 443; codified in provision/opnsense/fw-automation-rules.json, asserted by the fw-conformance harness).
• DMZ (edge-facing/internal) — inbound only via the edge WireGuard tunnels (origin Caddy + hidden Bind primary live here; no WAN inbound); may reach SERVERS only on its specific backend port; must not reach MGMT/WORKSTATIONS/IOT.
• WORKSTATIONS — → internet via Proton; → SERVERS (identity/NFS/DNS + media); → IOT for cast/print. Not → MGMT, not → DMZ-internal. (A meet carve-out briefly existed here and was removed on 2026-07-29 by giving that box its own public IP instead — see the Jitsi note below for why publishing a service on a firewall VIP forces carve-outs.)
• WORKSTATIONS ⇄ IOT — mDNS reflector + unicast allow rules (see below).
• IOT — DNS/NTP to firewall only; blocked from all RFC1918; per-device internet only if needed.
• GUEST — internet only; blocked from all RFC1918; AP client isolation. Also hosts roaming devices (laptops/phones that leave the house): they reach internal services exactly like a remote peer — over always-on WireGuard (per-peer rules) — so a roamer in GUEST is just a public client for everything else. (If WG is down at home, move the device onto WORKSTATIONS — the trusted wifi — for direct access.)
• WG road-warrior — per-peer rules keyed on tunnel IP: admin peer → MGMT; user peers → SERVERS (e.g. Jellyfin) only.

ProtonVPN egress (on OPNsense, multi-endpoint failover)

"Everything via Proton" = all outbound (clients AND servers) rides a failover group of N WireGuard tunnels; only local traffic, a curated bypass list, and inbound services use WAN.

• N WireGuard client tunnels to different Proton servers, each its own interface + gateway, combined in a gateway group PROTONVPN_GRP (tiered failover; Trigger = member down). Failover, not bandwidth bonding (Proton can't bond; a single stream = one tunnel).
• System default stays WAN_GW (a default must be a single gateway); the group is applied via policy routing on each zone's internet rule.
• Per-zone rule order: fw services → local/RFC1918 (default gw) → inter-zone blocks → VPN_BYPASS → WAN_GW → any → PROTONVPN_GRP → block (kill).
• VPN_BYPASS alias is empty by default (safety valve). When a destination's Proton exit IP gets blocked, add it (CIDRs/FQDNs, or a nested URL table like Google's ranges for YouTube) and it exits the real WAN — that destination then sees your real IP. The same rule hook is on every zone incl. IOT (e.g. if the Roku's YouTube ever needs it).
• Per-endpoint static route <proton-endpoint>/32 → WAN_GW so the tunnel transport never loops into the group.
• Fail-closed: "skip rules when gateway down" OFF + final block = no leak if the whole group drops.
• Inbound is only the WG admin endpoint (.69:51820) on WAN — it replies via WAN automatically (pf reply-to), independent of the Proton default. Public services arrive over the edge tunnels, not WAN (see edge/origin below). (A separate VPN container was considered and rejected: it still needs OPNsense's real WAN + policy routing for inbound and adds double-NAT.)

Identity enrollment scope (HBAC reach vs zone isolation)

Not every host is an IPA client — and that's deliberate, governed by zone and OS:

• Enrollable (SERVERS/MGMT): these reach the IPA server (10.20.20.2) on the Kerberos/LDAP ports, so the common role's ipa-client-install succeeds. Enrolled: sso (Authelia), fileserver, obs, ntfy, mgmt-proxy, apps-proxy, jellyfin, ctrl (the control node, MGMT). The freeipa host is the server itself — common never enrolls it.
• DMZ hosts are NOT IPA-enrolled — by design (LOCKED). The inter-zone policy denies DMZ → SERVERS on the IPA ports (88/389/464/636/53); DMZ is the restricted edge-facing zone. The common role has a reachability gate that silently skips enrollment when IPA is unreachable (never fails the play) — which is why caddy/bind/forgejo/searxng/runner/fmd/jump/invidious get the rest of the baseline (Alloy, SSH-CA hardening) but are not IPA clients. Decision: keep DMZ on the cert→almalinux local path; HBAC governs SERVERS/MGMT/control only. The local almalinux account (SSH-CA cert) is not an SSSD/IPA user, so it is HBAC-immune and keeps working regardless — and "the identity core is unreachable from the most-exposed zone" is the correct posture, not a gap. (Rejected alternative: a DMZ identity-proxy — jump host doing KKDCP Kerberos-over-443 + an L4 LDAPS/636 passthrough to IPA — would shrink the hole to one host on two TLS ports, but a proxy relocates the DMZ→identity trust path, it doesn't remove it; not worth the per-host SSSD/krb5 complexity here.)
• tvheadend (Debian) is fully managed but deliberately NOT IPA-enrolled. It is the lone Debian box (it needs Debian's in-kernel DVB drivers — see provision/templates.md). The common + alloy roles are OS-agnostic (dnf and apt — package names, the SSH service unit, and the CA-trust path/command resolve per OS family; EPEL/dnf-automatic are gated to EL), so tvheadend does get the full baseline (hardening, fail2ban, qemu-guest-agent, security updates via unattended-upgrades, SSH-CA host/user trust) and an Alloy agent. But it stays NOT IPA-enrolled (lab_ipa_enroll: false in its host_vars) — a special case of the fleet's core admin login is always local policy: tvheadend is an appliance whose users can't do IPA logins anyway, so it is local-only end to end (named local app admin + the SSH-CA almalinux cert for OS login, which is HBAC-immune and IPA-independent). Enrolling would add the lone-Debian freeipa-client/SSSD layer for no user-login benefit. Flip lab_ipa_enroll: true (the role's apt install freeipa-client path works) to join it.

Identity + NFS home directories

• fileserver is the NFS/storage host in SERVERS (10.20.20.10) — an AlmaLinux 9 VM running OpenZFS that holds the home + media datasets and serves NFSv4. Chosen over an appliance because it's a single NVMe (no array to manage), we need NFS only, and an IPA-enrolled EL box folds into the fleet's Ansible/identity/observability and keeps the krb5 path open. The data NVMe (PCI 0000:10:00.0) is passed through; OpenZFS uses the kABI-tracking kmod + dnf versionlock so a kernel update can't break the pool. Redundancy = sanoid snapshots + zfs send replication to a second box, not local RAID.
• FreeIPA provides identity — LDAP + Kerberos + DNS + CA → consistent UID/GID, SSO, and host/service certs from IPA's CA. Enroll workstations and the fileserver as native IPA clients (the common role).
• NFSv4 export of the home dataset; clients automount /home/$USER. Ownership is identity-mapped by FreeIPA UID/GID (e.g. operator = 640400004 on every host).
• NFS security = sec=sys (the standard, boring, reliable choice). Served over the firewalled internal SERVERS/WORKSTATIONS VLANs — consistent with the threat model (the ISP and edge vendors are the adversaries, not the internal LAN). Identity still maps correctly because both ends read the same FreeIPA UIDs. Roaming workstations (laptops that live in GUEST) mount the same shares over the road-warrior WireGuard tunnel — the fileserver sees the peer's tunnel IP (peers are routed, not NATed), so the export ACL lists each as its own /32 in the WG-RW subnet (nfs_clients in host_vars/fileserver.yml), mirroring the per-peer firewall model.

  Why sec=sys, krb5 deferred: FreeIPA mints aes-sha2 tickets while the kernel NFS RPCSEC_GSS only supports aes-sha1 (krb5 would need an enctype-restricted nfs/ keytab + a client krb5.conf drop-in). sec=sys on a firewalled, trusted LAN is the simpler, reliable choice. HARDEN LATER (optional): ipa service-add nfs/fileserver…, set the home export to sec=krb5, add the aes-sha1 keytab + drop-in (runbook 11 §7).

DNS architecture (hidden primary + public secondaries)

• Public authoritative = the edge. Bind secondaries run on the remote VPS edges (public IPs); the registrar's NS records delegate to them (≥2 diverse vendors). The world only ever queries the edges.
• Hidden authoritative primary = home. Bind at home (internal, not in NS records, not publicly reachable) is the source of truth: it serves AXFR/IXFR (TSIG) to the edge secondaries over the edge WireGuard tunnels, and answers the origin Caddy's RFC2136 ACME updates.
• Public A records (cloud.${domain}, …) point at the edge public IPs (clients reach the edge Caddy-L4).
• Internal resolution: Unbound on OPNsense is the resolver all clients use (DNSSEC, IoT intercept); it forwards ${ipa_domain} → FreeIPA and recurses the rest (recursion bound to the Proton tunnels — no leak to the ISP). FreeIPA integrated DNS is authoritative for the internal IPA realm only.

Edge/origin: public edge on remote VPSes, TLS terminates at home

Threat model extends to the VPS vendors too, so no vendor (or the ISP) ever sees plaintext — the edge does dumb L4/SNI passthrough; TLS terminates at home.

• Edge (each VPS, multi-vendor): Caddy-L4 (caddy-l4) listens :443, matches TLS SNI, and forward_tcps the encrypted stream over a WireGuard tunnel to the origin Caddy. No TLS termination, no certs, no plaintext on the VPS. A Bind secondary answers public DNS. Edge egress is direct (public boxes — not Proton). Vendor-provisioned + configured by the Ansible vps_edge role.
• Origin Caddy (home, internal): terminates TLS, manages certs via ACME DNS-01 (RFC2136) to the local hidden Bind primary, runs Authelia forward-auth (SSO/MFA, FreeIPA-backed), and reverse-proxies to the backends. It has no public inbound — reached only via the edge tunnels (and internally).
• WireGuard edge tunnels: home dials out to each VPS (VPS = stable public endpoint; home behind Comcast initiates + keepalive). Inbound flow: client → edge:443 → SNI match → tunnel → origin Caddy (TLS) → forward-auth → backend → reply returns statefully. Home is never directly contacted by the public.
• Authelia (SERVERS, sso 10.20.20.41) — forward-auth stays at the origin Caddy (where Authelia lives), not the edge. Apps that speak OIDC use Authelia directly (Grafana, the control-plane Forgejo); others via forward_auth.
• Admin/SSH: via the road-warrior WireGuard on home (.69:51820) — the one deliberate inbound exception on the Comcast link.

Exposure model: three tiers, three Caddy planes

Every web service is classified into one of three exposure tiers (the choice is per service, not global):

• Tier 0 — Public, no auth: intentionally public (blog, static sites).
• Tier 1 — Public + Authelia: internet-reachable via the edge, gated by Authelia forward-auth (or the app's own auth). Trust anchor = identity.
• Tier 2 — Private: never internet-reachable; only WORKSTATIONS/WG-RW. Trust anchor = the network. Smallest attack surface (the service isn't reachable, so its own pre-auth bugs aren't exposed).

Decision rule (tilted by the always-on WG posture): default everything to Tier 2 (private) — always-on WG on roaming devices + WG/LAN at home makes private nearly free. Promote to Tier 1 (public+Authelia) only when a service must work on a device that won't have WG (e.g. fmd device API — a stolen phone with its WG peer cert revoked must still phone home) or is shared with non-VPN people. (ntfy was Tier-1 for push, but with always-on WG on every device its bus rides the tunnel, so it was demoted to Tier-2/WG-only on 2026-07-02 — one fewer public listener.) The control-plane git (git.${domain}, the forgejo host holding the infra repo + CI) was likewise demoted to Tier-2/WG-only on 2026-07-04 when the git server was split into two instances — the internet-facing git moved to a separate forgejo-pub host serving only public repos (git.${domain_alt}), so the lab's own SCM no longer has any public listener (runbook 35). High-value/juicy-target services (Home Assistant — controls the house; admin UIs) stay Tier 2 regardless.

Distributed-domain policy (2026-08-15). Public exposure and public distribution are different things: a name is distributed when it is handed to third parties and lives in places the lab does not control (meeting invitations, beacon snippets embedded in public pages, share links). Distributed names live on the dedicated public apex (someonetoldme.xyz, lab-served DNS via the edges); ${domain} is never distributed — it stays internal-and-own-device (fmd deliberately stays on ${domain}: its only audience is the household's own phones). First movers: the umami collection endpoint (stats.someonetoldme.xyz, dual-served during beacon migration) and Jitsi (meet.someonetoldme.xyz, hard cutover — its cert and XMPP domain are single-valued). The Tier-2 planes are untouched by this policy (internal names, never distributed by definition).

These map to three Caddy planes:

Plane	Host	Tier(s)	Reachable from	DNS
origin Caddy	DMZ 10.20.30.10	0 + 1	internet (via edge)	public
apps-proxy	SERVERS 10.20.20.6	2 (user apps: HA, Jellyfin, Invidious, umami GUI, git, search, sync (also Tier-1 via origin — dual-plane, 2026-09-13), dav, auth, ntfy, radio (also Tier-1 via origin — dual-plane) — see group_vars/apps)	WS/WG-RW	internal split-horizon (Unbound override; not public)
mgmt-proxy	MGMT 10.20.10.6	2-admin (infra UIs)	wg-admin only	internal split-horizon

Tier-2 names (ha/jellyfin.${domain}) still get a publicly-trusted wildcard cert via DNS-01/RFC2136 (the cert needs only the TXT record, not a public A record), so https://ha.${domain} shows a green lock while resolving only internally. Moving a service between tiers = move its vhost between Caddies + flip its Unbound override.

Public domains served (origin plane). The origin Caddy isn't ${domain}-only — it also terminates several unrelated public apexes (real LE certs via the same DNS-01/RFC2136 path; content placed at /var/www/<name>). Current set (see origin_caddy defaults): ${domain} services (auth/search/fmd/stats/radio — the SDR's Tier-1 web face, Authelia-gated, relayed via the apps-proxy .8 bound listener since DMZ→WORKSTATIONS stays closed — and sync, the household file hub's Tier-1 face (2026-09-13, rb52 + rb56 §7 O1): Authelia-gated, straight to the fileserver's shelf-map :8081 over a host-scoped DMZ→SERVERS grant (seq 546) rather than a bound listener, because that backend already lives in SERVERS; dav is deliberately NOT promoted alongside it, since a CalDAV client cannot traverse an SSO redirect — plus the Tier-1 agent/assistant plane on the GPU host: opencode rb32 + assistant/research/artist/coder rb44, all Authelia-gated per-user), plus git/metronome/zddc.example.net (the public git.${domain_alt} is served by the forgejo-pub instance; the control-plane git.${domain} is Tier-2 private and not on this origin plane — see the two-instance split, runbook 35), chidot.example, example.org (Tier-0 static sites, except example.org/media-trust which is Authelia-gated), and myos.io — the infra docs site, now Tier-0 PUBLIC (the intent-first rewrite, un-gated 2026-06-29). The published pages read myos.io: the authored sections use ${domain} placeholders (reader-personalizable via a Personalize page + localStorage), and the Reference appendix (the symlinked CLAUDE.md + runbooks) now carries only ${domain}/${realm}/${edge.*} tokens — scrubbed of literal instance identity (A3) and resolved client-side by placeholders.js — so the earlier build-time domain/IP-masking sed (provision/sites/myos/Containerfile) has been retired: the master tree carries no identity to mask. A build-time leak gate greps the built site for any residual private token and fails the build if one survives — now the sole standing regression guard. This completes the interim bridge to full repo parameterization (a cloner sets one domain var to stand up their own).

(Media clients are an exception to forward-auth: native apps — Jellyfin clients, Kodi — can't traverse an SSO redirect, so Jellyfin authenticates against FreeIPA via LDAP and is network-gated Tier 2, rather than sitting behind Authelia forward-auth.)

Two ways to count visitors, chosen per site (analytics posture). umami (stats.${domain}) is the in-page option — cookieless, but it still needs a beacon in the page, so it can't serve a site whose product claim is "this page never phones home." For those, the origin Caddy counts server-side instead: an opt-in access_log: true on that one vhost logs the request the browser already made — client IP masked to /24 in the encoder, the whole request-header map (UA, Referer, Cookie) deleted before anything is written — to stdout → journald → Alloy → Loki → the Site visits Grafana dashboard (runbook 09 §9). No new software, no log file, no beacon; the durable record is a tally, not a profile, and the page stays byte-identical to a saved copy of itself. It is off for every vhost by default, and only appropriate for sites whose per-user state rides the URL fragment (never sent) rather than the query string (logged in uri). Two retention horizons, deliberately split: the raw lines are request records, so they expire with Loki at 30 days; the count is promoted to a metric (site_http_requests_total{vhost,class}, via alloy_access_log_metrics) kept 13 months in VictoriaMetrics — so a multi-year "how many people came" costs an aggregate, never a longer retention on request records. The path is never a metric label (bots would make it unbounded), and distinct-visitor counts stay log-only/30-day since they need a distinct-count. First user: the metronome static site (2026-07-26).

A separate TENANT plane (runbook 55). Everything above serves the household. The multi-tenant plane serves paying tenants who are not the household — the prototype for a product — and shares none of the lab's identity, front door or VPN. Three separations, each load-bearing rather than tidy: lldap (tenant-dir) instead of FreeIPA, because FreeIPA governs who can root the fleet; a tenant-only Caddy (tenant-proxy) that defines no admin vhost, so no Caddy mistake can expose one; and a separate WireGuard instance (wg5/opt5/WGTENANTS, udp 51821 on .69 — the second inbound exception on the Comcast link) whose interface-scoped rules make the tenant grant ONE rule for every tenant forever, so onboarding a tenant adds no firewall rule and no repo change. Tenant data is one ZFS dataset per tenant under supernas/sync (a legacy name kept deliberately — renaming it would strand the backup lineage), with both refquota and quota set, and a 30-day retention template on both ends rather than the lab's sixyear, so "deleted tenant data is unrecoverable after 30 days" is a true sentence. ${domain} is never distributed, so tenant names must move to the public apex before any name reaches a non-family tenant.

A fourth public plane — Jitsi Meet (meet.${domain}, runbook 48). Real-time video is the one service that doesn't fit any of the three Caddy planes: a WebRTC SFU needs a public UDP media port the TCP-SNI edges can't carry, and routing meeting media through the home edges would load the residential uplink. So Jitsi lives off the origin/edge planes — it terminates its own TLS and serves its own UDP/10000 media, not on the origin_caddy apex list. It breaks two edge invariants (a box holds a cert; a public UDP port is open), acceptable because Jitsi E2EE is on by default (insertable streams): the videobridge forwards media it cannot decrypt. Auth is JWT gated by Authelia OIDC (an OIDC→JWT adapter) so only authenticated users can create/moderate a room; guests land in the lobby.

INTERIM posture (rev. 2026-07-29): on-prem, but with its OWN public IP. Providers TBD, so meet is a pve VM — attached to the public segment holding .65 on its own NIC, exactly like the edge VMs, not behind a NAT. It was first deployed behind an OPNsense WAN VIP (.66 → DMZ .15), and that had to be undone: inbound NAT applies only to traffic arriving on WAN, so internal clients resolving the public name reached the firewall's GUI (its DNS-rebind guard) instead of Jitsi. Propping it up took split-horizon DNS, a WORKSTATIONS→DMZ carve-out and dual ICE candidates — four mechanisms for one wrong attachment, all deleted once it owned the address. Lesson: don't publish a service on a firewall VIP. Consequences that remain: it dies with a home/Comcast outage; its egress is the ISP, not Proton (as with the edges); it sits outside OPNsense's inbound filtering (protected by the common baseline — cert-only SSH, fail2ban, auditd); and the "vendor can't read media" rationale is moot on trusted pve (E2EE kept anyway). Being on-network it does rejoin central obs (alloy) — but only because its tunnel AllowedIPs carries the obs /32; without that the agent runs and ships NOTHING while looking perfectly healthy (found 2026-08-01, mute for as long as the tunnel had existed). The lab's own public addresses — meet + the two edges, listed individually, NOT the containing /29 — sit in VPN_BYPASS, so internal→our-own-public-IPs egresses the real WAN rather than a Proton exit. The WG road-warrior endpoint (.69) is deliberately EXCLUDED and must stay on Proton egress: it is a VIP on the WAN interface, so bypassing NATs an internal peer's handshake to the private transit address and loops it back into "Block private networks from WAN" — the tunnel then comes up but passes nothing (rx: 0), for home clients only. Bypassing the whole /29 did exactly that on 2026-07-31 (runbooks 05 §6 + 48). End-state: the same role off-prem — only where the box lives changes, since the network shape already matches.

Its cert is the container's own Let's Encrypt, HTTP-01 — deliberately NOT DNS-01. This is the one plane that is a genuinely standalone web server, so it validates over the public :80 it already has open for the 80→443 redirect: the image runs acme.sh in --standalone from cont-init (before nginx binds :80, with hooks that stop/restart nginx around issuance) and installs its own renewal cron under a supervised cron service. So obtain and renew are in-container — no RFC2136 to the home primary, no TSIG secret on a public box, no timer of ours — and the same setting works unchanged off-prem. Pin the CA (LETSENCRYPT_ACME_SERVER): acme.sh defaults to ZeroSSL, and because issuance holds nginx down, a stalled ZeroSSL order takes the whole site off the air, not just the renewal. It is still blackbox-probed for TLS expiry, being the one cert no Caddy manages.

IoT discovery (mDNS) — the Roku/printer problem

Roku/Chromecast/AirPlay/printers advertise over mDNS/Bonjour, which is link-local multicast (224.0.0.251:5353) and does not cross VLANs. So a firewall allow rule alone won't let a phone find the Roku. Fix = an mDNS reflector (OPNsense Avahi-based repeater) bridging WORKSTATIONS⇄IOT announcements, plus unicast allow rules for the actual media/print streams. This keeps IoT isolated while casting/printing "just works." (Note: mDNS does not cross WireGuard, so roaming devices in GUEST cannot cast — they use the target app's own remote/play.)

Key firewall design points

• Single default gateway + explicit policy routing — the system default is one gateway (WAN_GW); Proton egress is applied per-zone via policy routing. Never two defaultgw at once (that breaks routing).
• Kill-switch — an explicit fail-closed block so a VPN drop never leaks to WAN.
• Neutral WAN monitor IP (e.g. 9.9.9.9) — WAN health is decoupled from any Proton server.
• SSH — key/cert-only, no root password, mgmt-scoped (no password auth).
• Clean DNS — Unbound only, no duplicate/stale resolvers; recursion bound to the Proton tunnels.
• DMZ and MGMT are first-class isolated zones — the two highest-impact segments.
• Host firewalls (optional defense-in-depth) — every fleet host runs firewalld in default-deny inbound (SSH + its declared service ports only), applied uniformly by the host_firewall role via common. This is the "harden later" layer, not a replacement for the trust model: OPNsense's inter-zone allow-list still owns the trust boundary (the LAN is trusted; the adversary is the ISP/vendors). What it adds is closing every unused inbound listener + a reboot-safe, codified posture. Plain ports by design (no per-host source-scoping — that would duplicate OPNsense). Ports are inventory data (group_vars/host_vars host_firewall_extra_ports). See runbook 23.

OPNsense VM notes

• Two virtio NICs: vtnet0 = LAN trunk (VLANs 10–90), vtnet1 = WAN.
• Required virtio tunings: disable checksum / segmentation / large-receive offloading (needed for stable virtio under KVM/Proxmox).
• Sized for VPN throughput: 8 vCPU + multiqueue NICs (kernel WireGuard is CPU-bound).
• Plugins: os-acme-client (if used internally), os-wireguard/kernel WG, os-qemu-guest-agent; Unbound on, dnsmasq off.

Build sequence (clean install)

Environment: Proxmox node pve; WAN bridge vmbr0, LAN trunk vmbr1; VM storage local-zfs, ISO storage local. WAN: the ISP router stays inline and routes the static public block to OPNsense — WAN takes a private DHCP transit lease (gateway WAN_DHCP, kept stable via a router DHCP reservation), and the public IPs are assigned on WAN as IP Alias VIPs. VMIDs in 91xx: AlmaLinux template 9190, OPNsense 9140, fleet 9100–9130. An SSH CA is trusted fleet-wide (CA pubkey baked into the template + TrustedUserCAKeys in the Ansible hardening role). Ansible-core via dnf on an EL control node. FreeIPA: realm ${realm}, domain ${ipa_domain}.

Edge policy: out = ProtonVPN (every zone → the failover group over WAN; kill-switch fail-closed). in = via the edge tunnels only — home is inbound-silent except the WG admin endpoint on .69:51820. VPN_BYPASS is an empty safety valve (default is pure all-via-Proton).

VM base images: the Linux fleet clones a single AlmaLinux 9 template (provision/vms.conf); the fileserver is the same template + OpenZFS (runbook 11). Provisioning is runbook 01 (provision/lab-provision.sh); the Ansible config & maintenance layer is runbook 16 (the common baseline role, db_backups, and update.yml serial: 1 coordinated upgrades on top of per-host dnf-automatic). App-layer (Authelia image, FreeIPA major versions) updated deliberately, separate from the OS.

Update posture = three layers, and each needs BOTH an apply story and a detect story. Only the OS layer applies by itself; the other two are deliberate-by-design, which is correct — but "deliberate" silently degrades into "forgotten" without detection, so each gets a read-only probe → ntfy (runbook 34): appliances via appliance-update-check.yml, and every pinned artifact — container images AND the roles' version-pinned binaries (alloy, forgejo-runner, sanoid, the jellyfin LDAP plugin) — via the one image-update-check.yml, whose list is derived from the role defaults (never hand-kept) so a new pin is covered automatically; only the upstream release source of a binary is declared (BINARY_SOURCES), and a pin with none reports unchecked, never silently dropped. Images are digest/tag-pinned with podman-auto-update masked fleet-wide; that pinning is deliberate, and this probe is its compensating control. Added 2026-07-30 after a pin 18 days stale took invidious fully down — units green, its own alert rule correctly silent, found by a user (runbook 21); the binary half followed 2026-08-14 after forgejo-runner was found 8 releases (2 months) stale by hand, covered by neither probe. One script/timer/report, not a parallel one — a second checker is a second thing that can silently stop. Weekly, with the summary folded into the 04:00 digest so the weekly check has daily proof-of-life. A pin that is ABI-coupled to another (the jellyfin LDAP plugin ↔ the jellyfin server) reports its own lockstep status — reported, never paged as a routine bump, because bumping it alone is the outage. A fourth question the first three never ask (added 2026-08-27): is the host actually RUNNING its declared pin? Those probes all compare a declared pin to upstream; none compares the declared pin to the running container. A pin bumped in git and a container actually replaced are separate events, and the gap between them was unwatched — found when an Open WebUI security bump merged 2026-08-25 and the converge never followed, leaving the GPU host on the superseded image for two more days with the container healthy, its unit green, and the staleness probe correctly reporting the pin current. The gap is structural, not a lapse: pin-autobump opens a PR and never merges or converges, approval is a human step, converging is another. So image-drift-check.py compares each host's running images against the pins on main — same playbook, timer, ntfy topic and digest line as the staleness check, never a second timer. Matching is by image repository, so there is no hand-kept host→image table. A host that cannot be read reports unknown and pages on its own: drift there is unmeasured, not clean — the detector saying "I could not answer" is itself the alert, the same rule as the obs rule-drift checker. With one discipline on top: hosts the check is expected not to reach (the workstations, since the control cert has no admin-workstations; a roaming laptop) are declared by inventory group, counted in the state line and never paged — an alarm that fires every week by construction is one the operator learns to dismiss, which silences the findings that matter. That suppression is built to rot loudly: a structurally-suppressed host that turns out to be readable is reported by name so the entry gets pruned rather than masking that host forever. Anything that runs on ctrl or the fleet targets python 3.9 (AlmaLinux 9's system interpreter) — a newer workstation interpreter will run syntax that dies there, which cost the drift check its first acceptance run. Binary pins are covered the same way (added 2026-08-27): the installed binary is asked its own version and compared to the pin, using the same containment rule the roles themselves use to decide whether to upgrade — so a finding means exactly "the next converge would upgrade this", and the check and the converge cannot disagree about what current means. The binary's path is derived (X_version → X_bin in the same role's defaults) for 6 of the 8 pins; only the two whose path breaks the convention are declared — and a pin with neither reports unchecked. (An argv-only override, for a binary that hides its version behind a subcommand rather than --version, is supported too; its one user was filebrowser, uninstalled at rb52 adoption 2026-08-31.) Presence of the binary is what says the pin applies to that host — so, as with matching images by repository, there is no hand-kept host→pin table. First run found three real drifts on the GPU host, including a forgejo-runner eight releases behind that had been bumped in git and never converged (runbook 34).

One artifact now goes the other way, and only one (2026-09-13): shelf FOLLOWS its release. Everything above is pin-and-bless — we choose when to accept someone else's change. shelf is the one fleet program the lab writes, released from the public Forgejo by an operator merge that the instance signs, so requiring a second approval (a myos pin bump) was asking the same question twice. The release workflow now SSHes to the fileserver with a force-command-scoped deploy cert and names a tag; the installer there verifies that signature against a pinned instance-key fingerprint, checksums both assets, installs into a versioned directory, switches a symlink atomically, health-checks, and rolls back on failure (runbooks 52 §Push-to-deploy + 26). shelf_version in the role defaults became a floor — the version a rebuilt fileserver starts from, never a downgrade for a host CI has moved forward — and rb34 learned a follows-release class for exactly this shape: the staleness half stops nagging about a floor that is supposed to lag, and the drift half changes the question to "is the host on the latest release?", paging when it has been behind for more than a day (the CI deploy did not land) or is below the floor. This does not generalise to upstream images, where the pin is how we choose when to take a vendor's change and the soak stands in for trust we do not have; every one of those stays pinned with podman-auto-update masked (rb56 §7 O5).

The ordered steps live in runbooks/00-index.md — runbooks 01–17, in sequence, each ending in its own Validation section: Proxmox + provisioning → OPNsense base → the zone VLANs → Proton egress + road-warrior → Unbound/DHCP/mDNS → per-zone firewall rules → FreeIPA + NFS + origin Caddy + Authelia → the edge VPSes → end-to-end validation. The blockquote above carries the policy those steps must satisfy; the runbooks carry the steps themselves — don't restate them here, or the two drift.

Deliverables & format

Work is delivered as a sequence of discrete step-by-step runbooks under runbooks/, dialed in one at a time (build → validate → next). The OPNsense layer is delivered as GUI runbooks (click-through + precise rule/zone spec); the server layer (FreeIPA, ZFS/NFS storage, clients) as plain command runbooks. See runbooks/00-index.md for the ordered sequence and status.

The runbooks render as plain Markdown (runbooks/*.md); the public docs/ guides build with mkdocs (→ myos.io) for reproducibility by others.

Docs are a first-class, non-optional deliverable — keep them in lockstep with the system (a hard completion gate). Any change that affects how the lab is built or operated updates the matching record in the same change: the relevant runbooks/* step, this CLAUDE.md (the design source of truth), the host/IP allocation table, and any role README. A change isn't "done" until its docs are. The myos.io site is auto-published on push to main by a Forgejo Actions workflow (.forgejo/workflows/myos-image.yml: mkdocs build on the runner → a FROM-scratch OCI content image pushed to the internal zot registry → the origin Caddy serves it via Mount=type=image and refresh-site pulls the new tag; runbook 31). It is path-filtered — it fires only on docs/**, mkdocs.yml, requirements-docs.txt, runbooks/**, CLAUDE.md, the site Containerfile, or the workflow itself, so a main push that touches only ansible/ publishes nothing. (The older rsync-to-/var/www/myos docs-deploy.yml was retired once the origin cut over to the image; the OCI path replaced it.) What's published always reflects main — which is exactly why the docs must not drift from reality. The GUI master index on the docs landing page (docs/index.md) is sourced from the three Caddy planes' vhost lists; add a service there and mirror its row. See ansible/ci/README.md for the static-site/docs CD model.

Operating role: you (Claude) are the network administrator. You connect directly to the Proxmox host, firewall, and servers as required to configure, administer, and troubleshoot them — you do the work, then test and validate it is correct (don't just hand back steps for someone else to run). The runbooks are your reference and the record of the design, not a wall you stop at.

The division of labor: the user holds the master keys (vault password, YubiKey SSH-CA signing, KeePass, root/break-glass credentials) and authorizes hard-to-reverse or outward-facing actions; you do the hands-on administration and verification. Keep the user educated and informed — explain what you're doing and why, surface tradeoffs and risks, and report results faithfully (including failures) — but you are the one executing and proving the change, not delegating it back. When you genuinely lack an access path or credential, say exactly what you need rather than defaulting to "the user will run it."

Working notes

• This file is the source of truth for the design.