Skip to content

CLAUDE.md — ${domain} home lab

What this repo is

The infrastructure-as-code and runbooks for the ${domain} home lab — a segmented home network/firewall (OPNsense) plus a small AlmaLinux 9 server fleet, built to be reproducible from a clean install. The pieces:

  • provision/ — qm clone + cloud-init shell scripts that define and create the VMs (Proxmox CLI only).
  • ansible/ — configuration (built-in modules only; pinned ansible-core).
  • runbooks/ — ordered, step-by-step build/validate guides; see runbooks/00-index.md.
  • docs/ — the public, reproducible-for-others guides (mkdocs → myos.io).

⚠️ Secrets. Never commit plaintext keys or passwords. Secrets live in the Ansible vault (ansible/group_vars/all/vault.yml); WireGuard and host keys are generated on build, not stored in the repo. Don't paste secrets into chat or commits.

The goal

A fully robust home lab, not just a firewall: trust-based network segmentation, an edge/origin split that serves public sites from remote multi-vendor VPSes while TLS terminates at home (the ISP and the VPS vendors are treated as adversaries), WireGuard-only management access, a road-warrior VPN, and central identity (FreeIPA) backing NFS home directories — on the latest stable OPNsense and a single AlmaLinux 9 fleet, kept low-churn and rebuildable.

What "adversary" means here. Throughout this doc, "adversary" / "threat model" refers to the mundane, pervasive misuse of data, not (or not only) the sexy, scary stuff — rogue states, targeted international espionage. The thing being defended against is the slow death by a thousand cuts: ISPs, VPS vendors, data brokers, and ad-tech monetizing your traffic, building behavioral/advertising profiles, and tricking or nudging users. So calling the ISP and the VPS vendors "adversaries" means simply that they are positioned to harvest, sell, or profile your traffic — and the design denies them that by default. It is privacy/anti-profiling hygiene, not spy-vs-spy.

Design decisions (the why)

Grouped by theme; each detail is expanded in the matching Target design subsection.

Threat model & egress — the ISP (Comcast Business) is the adversary

  • Threat model: the ISP (Comcast Business) is the adversary — hence all egress (incl. servers) via Proton, MSS clamping on the tunnel path (~1380), Unbound recursion bound to the Proton tunnels (no DNS leak to Comcast), and the OPNsense VM sized for VPN throughput (8 vCPU + multiqueue NICs, kernel WG is CPU-bound).
  • Everything via ProtonVPN for all outbound (clients + servers) — never local traffic, never inbound. The VPN_BYPASS alias is a safety valve (the rule hook is in every zone) but is empty by default, so by policy everything egresses via Proton; you add a destination only when Proton's exit IP gets blocked there, sending just that out the real WAN.
  • ProtonVPN uses a load-balanced gateway group — all tunnels in Tier 1, load-balanced per-connection ("sticky connections" on). They all ride the same Comcast WAN, so a standby tier buys nothing. Gateway monitoring is disabled on the Proton tunnels — ICMP monitoring over Proton WireGuard is unreliable (dpinger can't get replies through the tunnel; see runbook 05), so the working setup turns it off; the tradeoff is no automatic per-tunnel failover (a dead Proton server keeps its share of connections until disabled by hand, or add the optional endpoint-monitor). Key Proton gotcha: every Proton config uses the same client address 10.2.0.2, but OPNsense needs unique interface IPs — so each tunnel gets a unique /28 address and outbound NAT rewrites the source back to 10.2.0.2 so Proton accepts all four (runbook 05 §5). Tier 2 is reserved for a genuinely independent uplink (a cellular modem / second ISP running its own Proton tunnel) — the only thing that survives a Comcast outage. Not bandwidth bonding — a single stream still rides one tunnel.
  • ProtonVPN runs on OPNsense (gateway group, not a separate container).
  • Break-glass: a disabled floating rule (internal → WAN_GW) you manually enable only when all of Proton is down — never an automatic WAN failover tier (that would silently leak to the ISP).

Inbound posture (edge/origin) — the ISP threat extended to ingress

  • Edge/origin inbound posture (the ISP threat extended to ingress): home serves no public services on the Comcast link — the only inbound exception is the WG admin endpoint on .69:51820. The public-facing layer lives on thin, stateless, multi-vendor mini-VPS "edges" doing TLS-SNI passthrough (caddy-l4) to the origin Caddy at home, which terminates TLS — so no VPS vendor ever sees plaintext. Home dials out the edge WireGuard tunnels (no static needed for them). edge public block roles: .67/.68 are the edge servers (public ns1/ns2 + caddy-l4 SNI passthrough); .69 is the WG-admin endpoint — WireGuard + sshd ONLY (no web/DNS on it); .65 and .66 are reserve. Hidden Bind primary at home → public Bind secondaries on the edges (registrar delegates to the edges). See runbook 17; phased (origin → 1 edge → multi-vendor).
  • WG-only (strict) management: zero standing access to MGMT.

Network shape

  • IP space: the lab lives in 10.20.0.0/16 — zones at 10.20.{10,20,30,40,60,80,90}.0/24, VLAN-tagged on the LAN trunk.
  • Zone assignment by exposure/trust, not medium: WORKSTATIONS holds all trusted stationary devices that never leave the house — wired desktops and stationary wifi (same trust → one zone; it's also the no-tunnel fallback at home). Anything that leaves the house (laptop/phone) lives in GUEST and reaches internal services only via always-on WireGuard — per-peer, identical at home and away, nothing to toggle (see Trust zones).
  • Keep a locked-down IOT zone (discovery solved via mDNS reflector).
  • IPv4-only build; IPv6 cleanly OFF (WAN v6 = none, no RA/DHCPv6 on zones). No internal benefit to v6 here, and dual-stack would double the firewall rules and risk IPv6 leaking around ProtonVPN (a v6 kill-switch would be required). Keeping v6 consistently off (not half-on) is what avoids the "apps try v6, hang, need -4" problem. Revisit only as a deliberate dual-stack phase later (Proton supports v6 egress). Optional fleet hardening: prefer-IPv4 via gai.conf.

Identity & per-service auth

  • FreeIPA for central identity (LDAP + Kerberos + DNS + CA).
  • Per-service auth = local break-glass admin + IPA-backed users. Every server app keeps a local admin (username/password in the KeePass DB, not an IPA account) so an admin is never locked out when IPA is down/unreachable; user logins go through IPA wherever the app supports it (LDAP, or Authelia-OIDC which is itself IPA-backed) so people are centrally managed (add/remove/group/HBAC). This is already the shape of the fleet: SSH uses the SSH-CA almalinux local cert (HBAC-immune) + IPA for everything else; Jellyfin = local jfadmin + IPA users via the LDAP plugin; Authelia (sso, replaced Authentik 2026-07-06) = config-as-code + IPA users via its LDAP→FreeIPA source (forward-auth for search/fmd + OIDC for Grafana); the control-plane Forgejo (git.${domain}) = local admin + IPA users via Authelia OIDC (a DMZ box that can't reach the identity core, so the token-exchange hairpins through the origin Caddy; the public forgejo-pub stays local-only to keep the internet-facing instance off identity — runbooks 13 + 35). Appliances that can't do IPA user logins (e.g. tvheadend, and ntfy — no LDAP/OIDC backend) stay local-only but still get a real named local admin (no anonymous-admin). ntfy runs auth-default-access: deny-all + a fixed ntfyadmin + per-user/per-app tokens (the grafana publish token is write-only to lab-alerts); users/tokens live in its persisted auth.db, managed out-of-band (podman exec ntfy ntfy user|access|token). The Ansible vault holds what automation needs; KeePass is the human break-glass record (some admin passwords live in both). Both the vault files (vault.yml + *.vault.yml) and lab.kdbx live encrypted in a dedicated lab-secrets Forgejo repo (Tier-2 private), kept out of the public myos repo; a control-role oneshot overlays them into group_vars/all/ before any converge with a read-only deploy key — the vault password stays TPM-sealed, so a repo read yields only ciphertext (runbook 43).

Platform, provisioning & backups (low-churn, rebuildable)

  • Latest stable OPNsense; configuration built clean.
  • One OS family: AlmaLinux 9 for the whole Linux fleet (Caddy, Bind, Authelia host (sso), FreeIPA, fileserver). Chosen for: FreeIPA's native platform, ~10-yr lifecycle (fewer upgrades = less churn), SELinux-enforcing, and killing mixed-shop complexity. Rocky is a 1:1 substitute. (Workstations stay whatever they are — they just enroll as IPA clients. tvheadend is the lone Debian exception, for its in-kernel DVB drivers.)
  • Proxmox pool lab: all VMs go in a Pool (Datacenter → Pools) for visual/permission grouping; the provision script sets --pool on clone.
  • IaC, boring & pinned (low-churn is a hard requirement): provisioning is a qm clone/pct create + cloud-init shell script (provision/lab-provision.sh) using only Proxmox's CLI — no OpenTofu/Terraform, no third-party providers. Each vms.conf row's TYPE (vm|lxc) picks the guest form (taxonomy below); all guests land on the encrypted SSD vmpool-enc (runbook 25). Config is Ansible with built-in modules only (ansible/), installed as a pinned ansible-core (not the big ansible collection bundle). Appliances (OPNsense, FreeIPA server, Home Assistant, Nextcloud) stay as runbooks / manual installs; the Linux fleet is provisioned + configured by code — including storage (NFS is served by an AlmaLinux 9 VM running OpenZFS, the nfs_server role — see "Identity + NFS"). This combo gives full destroy-and-rebuild with minimal moving parts.
  • Guest form = VM or LXC, by need; per-app segmentation unchanged. Every app keeps its own IP/zone/firewall (consolidation onto shared hosts is out — it would collapse the trust model). VM for own-kernel / PCI+USB passthrough / appliance / vTPM guests: OPNsense, Home Assistant, tvheadend, fileserver, FreeIPA, Nextcloud, ctrl (vTPM-sealed vault), runner (executes arbitrary CI job containers/builds via rootful podman — needs a real kernel boundary for isolation + build/overlayfs sanity, a poor LXC fit). LXC (unprivileged, nesting=1) for the lighter native single-app guests (the 3 Caddy proxies, bind, jump, sso (Authelia), registry (zot) — these run as LXC today) + — as the end-state, not yet realized — the fixed-quadlet podman/docker hosts (obs, ntfy, forgejo, searxng, fmd, jellyfin, invidious, umami). Those 8 currently run as VMs in provision/vms.conf on purpose: per the statefulness caveat below they were disk-moved with their data, and converting each to LXC is gated on escrowing its state in the vault first — so the guest-form list is the target, and the VM→LXC conversion is a deliberate, per-host follow-up, not a claim about the current fleet. The stock CT lacks the sshd/almalinux/CA-trust the VM cloud-image baked, so lab-provision.sh reproduces them post-create; common then takes over (it skips the VM-only qemu-guest-agent on containers). SSD alone fixed the boot storm, so LXC is a lightness refinement, not a requirement.
  • Statefulness caveat (learned the hard way — the vmpool migration). "Stateless" means config-as-code with no local data/identity. The fixed-quadlet podman hosts above hold app data (DBs, repos, metrics history) and bind (DNSSEC keys + zone files) / origin-caddy (its WireGuard identity key) hold state peers or restores depend on — so in practice these were disk-moved onto vmpool-enc (data preserved), not rebuilt (a fresh rebuild loses that state and breaks downstream — e.g. the edge tunnels / DNSSEC). Convert such a guest to LXC only after escrowing its state in the vault (zone files + DNSSEC keys for bind; the WG private key for origin-caddy). Truly stateless rebuilds (jump, the Caddy proxies, bind with escrow) are safe. One LXC gotcha: a container that forwards over a WireGuard tunnel (origin-caddy: edge↔DMZ AXFR/NOTIFY) must have wg0 in a firewalld zone (public) or firewalld (nftables) silently drops the new tunnel→DMZ flows — NAT/masquerade itself works fine in a nesting LXC. (Codified in the origin_caddy role.)
  • Guest storage = encrypted vmpool-enc (ZFS-native aes-256-gcm), Tang-unlocked. The SSD vmpool/enc auto-unlocks at boot via Clevis + Tang (NBDE) against tang1 — key in RAM only, fail-closed off-LAN/stolen, self-healing (a timer retries until tang1 is reachable), with a manual passphrase break-glass and 3-way key escrow (KeePass + offline + the gitignored vmpool-enc-key.vault.yml). The fileserver/supernas pool gets the same treatment in a later milestone. See runbook 25 (build + full DR).
  • Backups: the real sources → an 18 TB USB pool on bizon, direct (runbook 40). The pve box has two SSDs — vmpool (NVMe, encrypted guest disks) + the fileserver's passed-through NVMe (supernas, data + the RPM/DEB blessed mirror) — plus a 2.7 TB HDD (rpool) carrying the Proxmox OS. rpool used to be the on-box backup target for both SSD pools, but that Seagate started throwing uncorrectable errors while being the intermediary for every backup, so the design was reworked (2026-07-25) to send the three sources straight to bizon's wd18tb2311/lab-backup, keeping backups alive through a rpool failure (disk replacement + a mirrored boot pool is the pending hardware step):
  • Guests (vmpool/enc/vmdata + rpool/data) + pve OS (rpool/ROOT) → bizon, pve PUSHES (MGMT→SERVERS; bizon can't reach MGMT): syncoid -r --no-sync-snap rides sanoid hourly crash-consistent autosnaps (sanoid now covers all three trees), hourly at :20 (guest-repl-bizon). App-consistent DB dumps still ride along via db_backups. (vzdump + the var-lib-vz push were the last rpool-dependent tier — retired 2026-07-25* after a 4-cycle soak, so rpool is now out of every backup path.)
  • supernas → bizon, bizon PULLS direct from the fileserver (SERVERS→SERVERS): a dedicated bizon key on the restricted, zfs-delegated syncoid user (source-locked), daily 04:45 (labbackup-pull-supernas) — fully pve/rpool-independent. Carries home/media/recordings + the blessed mirror-current/deb-mirror-current w/ snapshots.
  • bizon target invariants (unchanged): a genuinely separate machine, receive-only delegated (create/receive/mount/hold, no destroy — a compromised pve can't erase history), own encryption root (/root/.zfs-labbackup.key on bizon's internal disk, escrowed in KeePass; non-raw sends re-encrypt at rest under it). Off-site is the remaining step (fire/theft; the nfs_*_replication_target hooks). pve also gained a ZED→ntfy zedlet (rb27/42) so ZFS data/state errors page (they didn't before). Code rebuilds the boxes; these backups restore the data. See runbooks 40 + 11.
  • Supply-chain: a pinned RPM snapshot mirror (the mirror host, runbook 37). The threat model extends to the package supply chain — a poisoned upstream RPM. Defense in three layers: (1) gpgcheck=1 enforced fleet-wide + signed metadata preserved on sync; (2) a local, immutable, point-in-time mirror — a stateless mirror VM reposyncs AlmaLinux+EPEL+ZFS into the fileserver's ZFS (over NFS) and the fleet pins to a blessed snapshot (use_rpm_mirror), so it can roll back to a known-good set even if upstream is poisoned/removed — self-sufficient, no upstream needed to restore; (3) a time-delayed "bless" (soak) — the fileserver snapshots each clean nightly sync and a timer auto-promotes the snapshot that has soaked N days (nfs_mirror_soak_days, default 3), so the fleet installs "upstream as of N days ago" and a pkg caught poisoned within that window never reaches the fleet — no manual gate, with bless-mirror.sh/bless-deb-mirror.sh kept as an emergency fast-track (rb37/39). Blessing = zfs snapshot ON the fileserver; the internet-facing mirror host is an NFS client with no snapshot authority, so a compromised sync can't rewrite the blessed history (the risky sync is separated from the immutable retention). Same instinct as the edge/origin split: isolate the thing that touches the outside from the thing that holds the trust. (VM, not LXC: NFS is refused in an unprivileged LXC; a VM also hardens the boundary for an internet-pulling box.) The same host + bless model covers Debian for the lone Debian box (tvheadend): debmirror runs in a Debian container (its base image pulled through zot) mirroring the full trixie archive into a parallel ZFS dataset, served at mirror.${domain}/debian, pinned via use_deb_mirror (deb_mirror role, runbook 39).

Target design

Trust zones (default-deny between all zones; gateway = .1 in each)

VLANs are tagged on the LAN NIC (vtnet0); the Proxmox LAN bridge must be VLAN-aware/trunk to carry them.

A zone is a trust boundary — not a way to categorise devices by medium or role. The seven zones group into four trust tiers, and that's what justifies each one (it's also why wired + stationary-wifi desktops share one zone: same trust, only the medium differs):

  • Isolated / high-value — MGMT (the infrastructure) and DMZ (the edge/ public-facing layer): the two highest-impact blast radii, kept furthest from everything.
  • Trusted-internal — SERVERS (the data/identity tier) and WORKSTATIONS (trusted stationary clients): separated from each other on an allow-list, both walled off from the untrusted + edge tiers.
  • Untrusted-devices — IOT (appliances) and GUEST (roamers + visitors): blocked from all RFC1918; internet only (roamers reach internal via always-on WG).
  • Remote — the WG road-warrior tunnel.

New zone? Ask "what trust boundary does it draw?" — if the answer is "none, it just organises devices," it shouldn't be a zone.

Zone VLAN Subnet Contents Internet path
MGMT 10 10.20.10.0/24 Firewall, Proxmox host, switch, APs, IPMI Proton (bypass→WAN)
SERVERS 20 10.20.20.0/24 FreeIPA, Authelia/sso (SSO), fileserver (ZFS/NFS), app backends Proton out; WAN in
DMZ (edge-facing) 30 10.20.30.0/24 internal origin Caddy (TLS terminator) + hidden Bind primary; reached only via the edge WG tunnels Proton out; in via edge tunnels
WORKSTATIONS 40 10.20.40.0/24 Trusted stationary clients — wired desktops + stationary wifi (NFS home, SSO) Proton (bypass→WAN)
IOT 60 10.20.60.0/24 Roku, wifi printer, smart home Proton; YouTube→WAN
GUEST 90 10.20.90.0/24 Roaming devices (laptop/phone) + visitors; client isolation; internal via always-on WG Proton (or WAN)
WG road-warrior — 10.20.80.0/24 Remote dial-in peers n/a

Zone assignment is by exposure, not medium. A device that leaves the house is higher-risk (it touches untrusted networks), so it is treated as untrusted: laptops and phones live in GUEST and reach internal services only over always-on WireGuard, exactly like any remote peer — so the access model is identical at home and away. Stationary devices that stay home — wired desktops and stationary wifi — are trusted and share the one WORKSTATIONS zone, which doubles as the no-tunnel fallback: if WG is down at home you can join it for direct access.

Host & IP allocation (single source of truth)

Gateway = .1 in every zone. DHCP pools .100–.199; static/servers .2–.99. Cloned VMs and their resources are defined in provision/vms.conf; this table is the human-readable cross-reference. Provisioning class in the Notes: clone+Ansible (the default), clone+manual (VM cloned, app set up by hand), appliance (manually installed, not cloned), external.

Host Zone IP Notes
OPNsense (per-zone gw) all 10.20.<vlan>.1 firewall; appliance (runbooks 01–08)
Proxmox host (pve) MGMT 10.20.10.5 hypervisor (:8006); appliance
mgmt-proxy MGMT 10.20.10.6 internal MGMT Caddy (wildcard cert; infra UIs; WG-admin only)
ctrl MGMT 10.20.10.8 dedicated Ansible control node (pinned ansible-core; reaches the fleet via a CA-signed control cert; lab.kdbx; vault password TPM-sealed); runbook 15. Also runs the automated host-CA signer (runbook 26) + the obs dead-man's-switch (curls obs health → ntfy directly, since Grafana can't alert on its own death — runbook 27) + the 04:00 daily digest (one proof-of-life + fleet-summary ntfy/day, derived from live VM queries; its absence is the alarm — runbook 27)
tang1 MGMT 10.20.10.11 NBDE/Tang keyserver for Clevis network-bound auto-unlock of pve's encrypted vmpool; single-function Raspberry Pi (Pi OS); appliance (physical; out of the fleet Ansible runs — provision/tang1-setup.sh + runbook 24). The lab's first physical infra host; on MGMT (not VLAN 20) to avoid a circular boot dependency. Static IP, NOT DHCP (DHCP/Kea runs on OPNsense, a VM on pve — a DHCP-dependent keyserver would deadlock pve's boot-unlock; a dormant Kea reservation on its MAC mirrors .11 as backup). .12 stays reserved, but a second tang2 geo node was declined in favour of plaintext key escrow (rb24 decision record, 2026-07-10) — tang1 is the sole keyserver by choice.
UniFi controller MGMT 10.20.10.101 network controller (APs/switch); appliance
FreeIPA SERVERS 10.20.20.2 LDAP/Kerberos/DNS/CA; clone+manual (server installed per runbook 10)
apps-proxy SERVERS 10.20.20.6 internal user-apps Caddy (Tier 2 private: HA + Jellyfin + Invidious + umami GUI + git (Forgejo internal front-door) + search (searxng) + cloud (Nextcloud) + files (Seafile) + dav (Radicale) + auth (Authelia) + ntfy; split-horizon DNS) — vhost list = ansible/group_vars/apps
fileserver (AlmaLinux ZFS) SERVERS 10.20.20.10 OpenZFS on a passed-through NVMe; NFSv4 sec=sys home/media/recordings
registry SERVERS 10.20.20.15 internal OCI registry + pull-through cache (zot); hosts our images + caches upstreams; registry.${domain} via apps-proxy (runbook 30)
mirror SERVERS 10.20.20.16 pinned RPM + DEB snapshot mirror (supply-chain resilience); reposyncs AlmaLinux+EPEL+ZFS and debmirrors Debian (in a container, base via zot) into the fileserver ZFS (over NFS), served at mirror.${domain} via its own caddy. Blessing = zfs snapshot on the fileserver (this host = NFS client, no snapshot authority). VM (NFS refused in unprivileged LXC). rpm_mirror + deb_mirror roles (runbooks 37, 39)
Jellyfin SERVERS 10.20.20.30 media server (EL9 podman quadlet); LDAP→FreeIPA login; library via NFS
tvheadend SERVERS 10.20.20.31 DVR/tuner backend (Debian; USB tuner passthrough); feeds Jellyfin Live TV
sso SERVERS 10.20.20.41 Authelia — SSO/forward-auth + OIDC plane (LXC podman quadlet; LDAP→FreeIPA via svc-ldap; SQLite state). Replaced Authentik 2026-07-06: auth.${domain} portal, forward-auth for search/fmd, OIDC for Grafana + control-plane Forgejo. authelia role
Home Assistant SERVERS 10.20.20.45 HAOS appliance (Zigbee USB), installed by hand; controls IOT via allow-rule + mDNS
obs SERVERS 10.20.20.50 observability sink (Grafana + VictoriaMetrics + Loki + Alloy); failure alerting = alloy systemd collector → node_systemd_unit_state{state="failed"} + a Grafana "Systemd unit failed" rule → ntfy on ANY crashed service/quadlet-container, a stale-timer rule (scheduled job silently stopped — expectations derived from each host's own timers, no hand-kept list), log-pattern rules (Loki; app fails while its unit stays green, e.g. invidious PO-token), a restart-loop rule (fleet-unit-failed matches state="failed", which an auto-restarting unit NEVER enters — a vendor service crash-looped 233k times unseen, so a companion rule alerts on time spent activating), a rule-drift check (Grafana provisioning only ever ADDS — a rule deleted from the template keeps firing forever — so a timer compares the rendered file against Grafana's own DB in both directions, plus a third rule for "the checker itself cannot run", since a broken detector reporting no problems found is worse than none), plus a ctrl-side dead-man's-switch + 04:00 daily digest for obs/liveness itself (runbook 27)
ntfy SERVERS 10.20.20.60 notification bus (UnifiedPush distributor); token-auth (deny-all), local users + ntfyadmin; lab-alerts = Grafana alert topic
bizon SERVERS 10.20.20.70 GPU/vLLM host — local LLM inference for the agent/assistant plane (opencode rb32; assistant/research rb44) + the off-pve backup target (18 TB USB wd18tb2311/lab-backup, rb40). Ubuntu/OEM box, workstation-role-managed (not in all_linux; SSH-CA trust + host-cert autosign via admin-workstations; host_vars/bizon.yml)
jump DMZ 10.20.30.5 SSH jump host (fleet access via edge :443 relay)
Origin Caddy DMZ 10.20.30.10 internal TLS terminator + forward-auth (no public inbound)
Nextcloud DMZ 10.20.30.20 Nextcloud AIO web backend (behind origin Caddy); clone+manual (AIO installed by hand; +250G data disk)
forgejo DMZ 10.20.30.30 git server (podman quadlet) — control-plane SCM (the lab_org/myos infra repo + CI/runner + registry-push target); canonical git.${domain}, Tier-2 private (apps-proxy/WG only, not on the edge)
forgejo-pub DMZ 10.20.30.31 git server (podman quadlet) — public Forgejo, canonical git.${domain_alt}; edge-exposed via origin Caddy; public + public-related repos only, no control-plane
Bind (hidden primary) DMZ 10.20.30.40 internal authoritative primary; AXFR → edge secondaries; not delegated
searxng DMZ 10.20.30.50 metasearch (podman quadlet)
runner DMZ 10.20.30.60 Forgejo Actions runner (ci_runner role) for the control-plane forgejo only; offers the host label (jobs run directly on the VM) and holds the origin deploy key + zot push creds
runner-pub DMZ 10.20.30.61 Forgejo Actions runner for the public forgejo-pub. A runner serves exactly one instance, and this is a separate VM — sharing runner would let the edge-exposed public git dispatch code onto the box holding the control-plane deploy credentials. Container-only labels (no host); outbound-only. ci_runner role + host_vars/runner-pub.yml (runbook 35 §8)
Invidious DMZ 10.20.30.70 privacy YouTube frontend; Tier-2 private, served via apps-proxy at https://invidious.${domain} (like Jellyfin) — split-horizon override → apps-proxy, which reverse-proxies to this backend :3000 (needs the SERVERS→DMZ allow, runbook 06); no edge/Authelia. domain/external_port/https_only set to https+443 so generated URLs match the proxy name
fmd DMZ 10.20.30.80 FindMyDevice server (podman quadlet)
umami DMZ 10.20.30.90 cookieless web analytics (podman quadlet + Postgres); split stats.${domain} — collection PUBLIC (edge origin-Caddy allowlists only /script.js + /api/send* → umami; everything else 404s), GUI/dashboard PRIVATE (Tier-2 via apps-proxy split-horizon → umami :3000, needs the SERVERS→DMZ allow); Postgres → db_backups
files DMZ 10.20.30.100 Seafile CE file-sync — adopted keeper (graduated from trial 2026-07-09; the Nextcloud-files-alternative eval landed here). Podman quadlets: seafile-mc + MariaDB + memcached (backend_app=seafile); Tier-2 private via apps-proxy (files.${domain}, split-horizon → SERVERS→DMZ allow); nightly MariaDB → db_backups (deployed). IPA users via Authelia OIDC (seafile_oidc_enabled; web + sync-client browser login; client_secret_basic; both URL roots pinned https) — DMZ box, so the token-exchange hairpins through the origin Caddy (AddHost auth.${domain}→caddy), like the control-plane Forgejo; local seafile_admin stays break-glass. Stateful (holds phone Camera-Upload media; 256 GB disk) — back up, don't casually rebuild. Runbook 41
dav DMZ 10.20.30.110 Radicale CalDAV/CardDAV — adopted keeper (graduated from trial 2026-07-15; the Nextcloud-cal/contacts-alternative eval landed here). Single podman quadlet; backend_app=radicale; Tier-2 private via apps-proxy (dav.${domain}, split-horizon); htpasswd (bcrypt) auth; collections on disk
someonetoldme DMZ 10.20.30.120 someonetoldme.org Django site (LXC) — the first fleet app built from our own source rather than a pinned upstream image. Podman quadlets: postgres 16 + a one-shot migrate + web (gunicorn) + a django-q2 media worker, all from ONE image so they cannot drift apart on deps/migrations (backend_app=someonetoldme). Image is built on runner-pub from the public repo and pushed to our zot as apps/someonetoldme by the scoped ci-pub account (apps/, not sites/ — the origin's refresh-site.sh treats sites/<name> as mountable content images), then pinned by digest. Public via the edge → origin Caddy, which terminates TLS; the app serves plain HTTP, trusts X-Forwarded-* only from the origin, and ALLOWED_HOSTS must carry the public name plus its own IP plus 127.0.0.1 (its container healthcheck probes loopback). Nightly pg_dump → db_backups — the rows are guest-book entries and card records that physically etched cards point at, so they get the logical-dump tier, not snapshots alone. Media is stored LOCALLY on a 100 G disk and R2 was dropped — a vendor holding user-uploaded video in plaintext inverts the lab's own threat model, and NFS was rejected on a hard blocker (refused in an unprivileged LXC, the reason mirror is a VM) plus a runtime dependency on the fileserver. So the guest is stateful by design, like files: media rides the hourly vmpool-enc → bizon replication (crash-consistent is fine for immutable blobs) and the DB gets the logical dump on top. Not yet public: no origin vhost. Runbook 50
Workstations WORKSTATIONS 10.20.40.x wired + stationary-wifi trusted desktops; IPA-enrolled by hand, workstation role for SSH-CA + autofs
WG admin peer WG-RW 10.20.80.10 → MGMT
WG user peers WG-RW 10.20.80.21+ → SERVERS (e.g. Jellyfin; NFS home shares — per-peer /32 in the fileserver export)
Edge tunnels WG 10.20.200.0/24 per-VPS /32s (home dials out to each edge)
Edge VPSes (external) vendor public IPs Caddy-L4 (SNI passthrough) + Bind secondary; multi-vendor; external (vps_edge role)
Jitsi (meet) (public segment) own public IP public video plane — docker-jitsi-meet podman quadlets (web/prosody/jicofo/jvb + OIDC→JWT adapter); E2EE on by default; own TLS (LE HTTP-01, renewed in-container). Holds its own public IP on its own NIC (vms.conf VLAN col = wan, untagged on the WAN bridge) like the edge VMs — not behind a firewall VIP/NAT, which is unreachable from inside and forces split-horizon + carve-outs. On-prem interim for a future off-prem box; egress = ISP not Proton; outside OPNsense inbound filtering. room creation is gated by a named Prosody-internal moderator (password in KeePass), guests to the lobby — not SSO: this box has no route to the identity core, same rule as tvheadend/ntfy (runbook 48 §Auth). clone+Ansible (jitsi role, runbook 48)

Most home Linux VMs clone the AlmaLinux 9 template (provision/vms.conf) and are configured by Ansible; tvheadend clones the Debian template (in-kernel DVB drivers). Manually-installed appliances (not cloned/Ansible): OPNsense (GUI runbooks), Home Assistant (standalone HAOS image). FreeIPA and Nextcloud are cloned VMs whose applications are installed by hand (runbooks 10 and 09/Nextcloud AIO), not by an Ansible role. The edge VPSes are vendor-provisioned + configured by the vps_edge role.

Control plane. Ansible runs from a dedicated control-node VM (ctrl, MGMT) — not a personal workstation — so the credential that reaches the whole fleet (a CA-signed control cert: its own key signed by the SSH user CA with principals admin-{mgmt,servers,dmz,freeipa}, accepted everywhere via TrustedUserCAKeys) and lab.kdbx live on one single-purpose, common-managed, rebuildable box; its vault password is TPM-sealed (systemd-creds, no plaintext at rest). ctrl runs Ansible as the almalinux user (never root — it only connects out). A cert (not a raw authorized_keys entry) is the fleet credential because it expires and is KRL-revocable; so fleet_transport_pubkeys carries only root@pve (the pre-cert provisioning bootstrap). CA signing is split by what the cert grants (runbook 26): the user CA — which grants access (admin certs → root) — stays YubiKey-gated on the admin laptop (ctrl's own control cert, new admins, etc. are signed there and committed to git), so the access root is offline and unreachable by any online compromise. The host CA — which grants only host identity — is now an online, automated CA on ctrl: its key is Clevis+Tang-sealed to tang1 and decrypted per-signature, so a weekly host-ca-autosign.timer renews short-lived host certs hands-off while a stolen/off-LAN ctrl fails closed (no tang1 → no signing). Trust is additive (the YubiKey host CA stays trusted as break-glass); every sign is pushed to ntfy and tang1 ships all access to Loki (chokepoint audit) — see runbooks 26 + 15.

Why segment: a flat network lets one compromised device (a phone, a guest, a hacked public website) reach everything. Segmentation contains blast radius. The two highest-impact zones — DMZ (the public sites) and MGMT (the infrastructure) — are isolated, not flat, and the most-exposed devices (roamers, IoT, guests) are kept furthest from them.

Credential custody model (four tiers + the AI-agent PAW)

Credentials are segmented by who wields them, and custody strength scales with blast radius — the same instinct as the SSH-CA split. Four tiers:

  • T1 — automation (machine): what the fleet/automation uses unattended — the almalinux control cert (Tang-gated, loaded by load-control-key.sh) + the TPM-sealed vault on ctrl. Used by the timers (host-CA autosign, obs dead-man's-switch) and by ansible.
  • T2 — the AI agent (Claude): the agent runs as a Privileged Access Workstation on ctrl (tmux, a dedicated claude-agent account) and reaches the fleet with its OWN keypair (Tang- sealed) + a short-lived YubiKey-signed cert (key-id claude-agent, principals admin-mgmt,admin-servers,admin-dmz,admin-workstations — not admin-freeipa: every zone incl. DMZ + the workstations (standing since 2026-07-12), but not the identity/CA core nor ctrl itself). It gets root on the fleet it services (become there) but its standing cert can't root its own PAW — claude-agent has no sudo on ctrl, and ctrl is a crown-jewel (admin-ctrl) so the standing fleet cert can't hop to almalinux@ctrl. Elevation is just-in-time, not denied: the operator signs a task-scoped cert with exactly the principals a task needs (incl. crown jewels), short-TTL — scripts/yubikey-tool.sh sign-agent-task. The ctrl-root path is an admin-ctrl task cert → ssh almalinux@ctrl → sudo -i (almalinux sudo codified; PermitRootLogin no, so root SSH stays off / console-only). The hypervisor-root path is the same shape for pve: an admin-hv task cert → ssh root@pve — a DEDICATED principal that maps to root@pve and nothing else (runbook 20 appliance table), so the agent can do JIT provisioning (lab-provision.sh, qm, pct) without the god-mode admin; sharpest jewel (pve root = full fleet), so task-only + +4h. The standing cert is signed on demand (sign-agent, +1w); cutoff (both standing + task) = expiry / KRL (scripts/push-krl.sh) / live-kill (scripts/agent-killswitch.sh). The agent's API keys live in ctrl's vault. Its access is a deliberate, hardware-granted, revocable grant — never the automation control cert. (Runbook 28.)
  • T3 — human web logins: admin passwords in KeePass lab.kdbx, YubiKey-gated; keep a second YubiKey + offline recovery codes (no single-key lockout).
  • T4 — human WebAuthn: FIDO2/WebAuthn on the YubiKey wherever an admin site supports it; T3 is the fallback for sites that don't.

The offline user-CA (YubiKey 9c) signs all access — both your 9a login cert and the agent cert; the host-CA (online, Tang-sealed) grants only host identity; the control cert is the automation transport. No online access-CA exists (that would break the split-CA invariant), and the agent never borrows automation's credential (T1/T2 stay separate). Deliberately rejected for the agent: enrolling DMZ in IPA (breaks DMZ isolation) and a pure-IPA agent identity (cert auth is HBAC-immune, and the cutoff must cover DMZ) — the PAW + own-cert model gives a uniform, operator- controlled kill switch across all managed computers without either.

Decision (2026-07-07): the agent stays cert + PAW + JIT — no dedicated IPA identity (declined "C4"). We revisited giving claude-agent a named IPA user + HBAC + IPA sudo — the same IPA-as-PAM model we'd use for a human operator ("treat the agent like an employee"). Declined, because the human model breaks precisely where an AI differs from a human: (1) Injectability — an agent manipulable by the data it reads must minimise standing privilege, and standing IPA sudo (even command-scoped) is a bigger injection blast radius than "lands as almalinux, elevate only via an operator-signed JIT task cert." (2) Uniform, instant, fail-closed cutoff — one lever (cert expiry / KRL / killswitch / Tang-seal) beats a split cutoff (ipa user-disable on enrolled hosts while the cert still lands on DMZ + as break-glass). (3) IPA-independence — the agent must administer the fleet including when IPA is degraded; cert access is HBAC-immune/IPA-independent by design (same reason almalinux SSH is cert-based), whereas an IPA-dependent agent could be locked out of fixing IPA. (4) DMZ is never IPA-enrolled, so an IPA identity is a second access system, not a replacement. The one real gain — attribution — is already met by the cert key-id claude-agent logged on every sshd login (address further, if needed, with a distinct login user or auth-log alerting — not a standing identity). IPA-as-PAM is reserved for humans (the identity-posture work): if a person ever fills the operator seat, they enroll as an IPA user with HBAC + sudo at that point. See runbook 28.

Inter-zone policy (allow-list; everything else denied)

  • MGMT — reachable ONLY from the WG admin peer (+ Proxmox console fallback); MGMT may reach all zones for administration.
  • SERVERS — receives from WORKSTATIONS: NFSv4 (2049 only), LDAP (389/636), Kerberos (88/464), DNS (53), IPA UI (HTTP/S), plus Jellyfin/HA/tvheadend for media. Receives from DMZ only on the single app→backend port. Does not freely initiate into client zones. Reaches the firewall itself only for DNS/NTP — not the OPNsense web GUI (:443): SERVERS is a data/identity tier, not an admin zone, so a firewall-automation block denies SERVERS → (self):443 on every firewall IP (the SERVERS → firewall self-rule otherwise allows 443; codified in provision/opnsense/fw-automation-rules.json, asserted by the fw-conformance harness).
  • DMZ (edge-facing/internal) — inbound only via the edge WireGuard tunnels (origin Caddy + hidden Bind primary live here; no WAN inbound); may reach SERVERS only on its specific backend port; must not reach MGMT/WORKSTATIONS/IOT.
  • WORKSTATIONS — → internet via Proton; → SERVERS (identity/NFS/DNS + media); → IOT for cast/print. Not → MGMT, not → DMZ-internal. (A meet carve-out briefly existed here and was removed on 2026-07-29 by giving that box its own public IP instead — see the Jitsi note below for why publishing a service on a firewall VIP forces carve-outs.)
  • WORKSTATIONS ⇄ IOT — mDNS reflector + unicast allow rules (see below).
  • IOT — DNS/NTP to firewall only; blocked from all RFC1918; per-device internet only if needed.
  • GUEST — internet only; blocked from all RFC1918; AP client isolation. Also hosts roaming devices (laptops/phones that leave the house): they reach internal services exactly like a remote peer — over always-on WireGuard (per-peer rules) — so a roamer in GUEST is just a public client for everything else. (If WG is down at home, move the device onto WORKSTATIONS — the trusted wifi — for direct access.)
  • WG road-warrior — per-peer rules keyed on tunnel IP: admin peer → MGMT; user peers → SERVERS (e.g. Jellyfin) only.

ProtonVPN egress (on OPNsense, multi-endpoint failover)

"Everything via Proton" = all outbound (clients AND servers) rides a failover group of N WireGuard tunnels; only local traffic, a curated bypass list, and inbound services use WAN.

  • N WireGuard client tunnels to different Proton servers, each its own interface + gateway, combined in a gateway group PROTONVPN_GRP (tiered failover; Trigger = member down). Failover, not bandwidth bonding (Proton can't bond; a single stream = one tunnel).
  • System default stays WAN_GW (a default must be a single gateway); the group is applied via policy routing on each zone's internet rule.
  • Per-zone rule order: fw services → local/RFC1918 (default gw) → inter-zone blocks → VPN_BYPASS → WAN_GW → any → PROTONVPN_GRP → block (kill).
  • VPN_BYPASS alias is empty by default (safety valve). When a destination's Proton exit IP gets blocked, add it (CIDRs/FQDNs, or a nested URL table like Google's ranges for YouTube) and it exits the real WAN — that destination then sees your real IP. The same rule hook is on every zone incl. IOT (e.g. if the Roku's YouTube ever needs it).
  • Per-endpoint static route <proton-endpoint>/32 → WAN_GW so the tunnel transport never loops into the group.
  • Fail-closed: "skip rules when gateway down" OFF + final block = no leak if the whole group drops.
  • Inbound is only the WG admin endpoint (.69:51820) on WAN — it replies via WAN automatically (pf reply-to), independent of the Proton default. Public services arrive over the edge tunnels, not WAN (see edge/origin below). (A separate VPN container was considered and rejected: it still needs OPNsense's real WAN + policy routing for inbound and adds double-NAT.)

Identity enrollment scope (HBAC reach vs zone isolation)

Not every host is an IPA client — and that's deliberate, governed by zone and OS:

  • Enrollable (SERVERS/MGMT): these reach the IPA server (10.20.20.2) on the Kerberos/LDAP ports, so the common role's ipa-client-install succeeds. Enrolled: sso (Authelia), fileserver, obs, ntfy, mgmt-proxy, apps-proxy, jellyfin, ctrl (the control node, MGMT). The freeipa host is the server itself — common never enrolls it.
  • DMZ hosts are NOT IPA-enrolled — by design (LOCKED). The inter-zone policy denies DMZ → SERVERS on the IPA ports (88/389/464/636/53); DMZ is the restricted edge-facing zone. The common role has a reachability gate that silently skips enrollment when IPA is unreachable (never fails the play) — which is why caddy/bind/forgejo/searxng/runner/fmd/jump/invidious get the rest of the baseline (Alloy, SSH-CA hardening) but are not IPA clients. Decision: keep DMZ on the cert→almalinux local path; HBAC governs SERVERS/MGMT/control only. The local almalinux account (SSH-CA cert) is not an SSSD/IPA user, so it is HBAC-immune and keeps working regardless — and "the identity core is unreachable from the most-exposed zone" is the correct posture, not a gap. (Rejected alternative: a DMZ identity-proxy — jump host doing KKDCP Kerberos-over-443 + an L4 LDAPS/636 passthrough to IPA — would shrink the hole to one host on two TLS ports, but a proxy relocates the DMZ→identity trust path, it doesn't remove it; not worth the per-host SSSD/krb5 complexity here.)
  • tvheadend (Debian) is fully managed but deliberately NOT IPA-enrolled. It is the lone Debian box (it needs Debian's in-kernel DVB drivers — see provision/templates.md). The common + alloy roles are OS-agnostic (dnf and apt — package names, the SSH service unit, and the CA-trust path/command resolve per OS family; EPEL/dnf-automatic are gated to EL), so tvheadend does get the full baseline (hardening, fail2ban, qemu-guest-agent, security updates via unattended-upgrades, SSH-CA host/user trust) and an Alloy agent. But it stays NOT IPA-enrolled (lab_ipa_enroll: false in its host_vars) — a special case of the fleet's core admin login is always local policy: tvheadend is an appliance whose users can't do IPA logins anyway, so it is local-only end to end (named local app admin + the SSH-CA almalinux cert for OS login, which is HBAC-immune and IPA-independent). Enrolling would add the lone-Debian freeipa-client/SSSD layer for no user-login benefit. Flip lab_ipa_enroll: true (the role's apt install freeipa-client path works) to join it.

Identity + NFS home directories

  • fileserver is the NFS/storage host in SERVERS (10.20.20.10) — an AlmaLinux 9 VM running OpenZFS that holds the home + media datasets and serves NFSv4. Chosen over an appliance because it's a single NVMe (no array to manage), we need NFS only, and an IPA-enrolled EL box folds into the fleet's Ansible/identity/observability and keeps the krb5 path open. The data NVMe (PCI 0000:10:00.0) is passed through; OpenZFS uses the kABI-tracking kmod + dnf versionlock so a kernel update can't break the pool. Redundancy = sanoid snapshots + zfs send replication to a second box, not local RAID.
  • FreeIPA provides identity — LDAP + Kerberos + DNS + CA → consistent UID/GID, SSO, and host/service certs from IPA's CA. Enroll workstations and the fileserver as native IPA clients (the common role).
  • NFSv4 export of the home dataset; clients automount /home/$USER. Ownership is identity-mapped by FreeIPA UID/GID (e.g. operator = 640400004 on every host).
  • NFS security = sec=sys (the standard, boring, reliable choice). Served over the firewalled internal SERVERS/WORKSTATIONS VLANs — consistent with the threat model (the ISP and edge vendors are the adversaries, not the internal LAN). Identity still maps correctly because both ends read the same FreeIPA UIDs. Roaming workstations (laptops that live in GUEST) mount the same shares over the road-warrior WireGuard tunnel — the fileserver sees the peer's tunnel IP (peers are routed, not NATed), so the export ACL lists each as its own /32 in the WG-RW subnet (nfs_clients in host_vars/fileserver.yml), mirroring the per-peer firewall model.

    Why sec=sys, krb5 deferred: FreeIPA mints aes-sha2 tickets while the kernel NFS RPCSEC_GSS only supports aes-sha1 (krb5 would need an enctype-restricted nfs/ keytab + a client krb5.conf drop-in). sec=sys on a firewalled, trusted LAN is the simpler, reliable choice. HARDEN LATER (optional): ipa service-add nfs/fileserver…, set the home export to sec=krb5, add the aes-sha1 keytab + drop-in (runbook 11 §7).

DNS architecture (hidden primary + public secondaries)

  • Public authoritative = the edge. Bind secondaries run on the remote VPS edges (public IPs); the registrar's NS records delegate to them (≥2 diverse vendors). The world only ever queries the edges.
  • Hidden authoritative primary = home. Bind at home (internal, not in NS records, not publicly reachable) is the source of truth: it serves AXFR/IXFR (TSIG) to the edge secondaries over the edge WireGuard tunnels, and answers the origin Caddy's RFC2136 ACME updates.
  • Public A records (cloud.${domain}, …) point at the edge public IPs (clients reach the edge Caddy-L4).
  • Internal resolution: Unbound on OPNsense is the resolver all clients use (DNSSEC, IoT intercept); it forwards ${ipa_domain} → FreeIPA and recurses the rest (recursion bound to the Proton tunnels — no leak to the ISP). FreeIPA integrated DNS is authoritative for the internal IPA realm only.

Edge/origin: public edge on remote VPSes, TLS terminates at home

Threat model extends to the VPS vendors too, so no vendor (or the ISP) ever sees plaintext — the edge does dumb L4/SNI passthrough; TLS terminates at home.

  • Edge (each VPS, multi-vendor): Caddy-L4 (caddy-l4) listens :443, matches TLS SNI, and forward_tcps the encrypted stream over a WireGuard tunnel to the origin Caddy. No TLS termination, no certs, no plaintext on the VPS. A Bind secondary answers public DNS. Edge egress is direct (public boxes — not Proton). Vendor-provisioned + configured by the Ansible vps_edge role.
  • Origin Caddy (home, internal): terminates TLS, manages certs via ACME DNS-01 (RFC2136) to the local hidden Bind primary, runs Authelia forward-auth (SSO/MFA, FreeIPA-backed), and reverse-proxies to the backends. It has no public inbound — reached only via the edge tunnels (and internally).
  • WireGuard edge tunnels: home dials out to each VPS (VPS = stable public endpoint; home behind Comcast initiates + keepalive). Inbound flow: client → edge:443 → SNI match → tunnel → origin Caddy (TLS) → forward-auth → backend → reply returns statefully. Home is never directly contacted by the public.
  • Authelia (SERVERS, sso 10.20.20.41) — forward-auth stays at the origin Caddy (where Authelia lives), not the edge. Apps that speak OIDC use Authelia directly (Grafana, the control-plane Forgejo); others via forward_auth.
  • Admin/SSH: via the road-warrior WireGuard on home (.69:51820) — the one deliberate inbound exception on the Comcast link.

Exposure model: three tiers, three Caddy planes

Every web service is classified into one of three exposure tiers (the choice is per service, not global):

  • Tier 0 — Public, no auth: intentionally public (blog, static sites).
  • Tier 1 — Public + Authelia: internet-reachable via the edge, gated by Authelia forward-auth (or the app's own auth). Trust anchor = identity.
  • Tier 2 — Private: never internet-reachable; only WORKSTATIONS/WG-RW. Trust anchor = the network. Smallest attack surface (the service isn't reachable, so its own pre-auth bugs aren't exposed).

Decision rule (tilted by the always-on WG posture): default everything to Tier 2 (private) — always-on WG on roaming devices + WG/LAN at home makes private nearly free. Promote to Tier 1 (public+Authelia) only when a service must work on a device that won't have WG (e.g. fmd device API — a stolen phone with its WG peer cert revoked must still phone home) or is shared with non-VPN people. (ntfy was Tier-1 for push, but with always-on WG on every device its bus rides the tunnel, so it was demoted to Tier-2/WG-only on 2026-07-02 — one fewer public listener.) The control-plane git (git.${domain}, the forgejo host holding the infra repo + CI) was likewise demoted to Tier-2/WG-only on 2026-07-04 when the git server was split into two instances — the internet-facing git moved to a separate forgejo-pub host serving only public repos (git.${domain_alt}), so the lab's own SCM no longer has any public listener (runbook 35). High-value/juicy-target services (Home Assistant — controls the house; admin UIs) stay Tier 2 regardless.

These map to three Caddy planes:

Plane Host Tier(s) Reachable from DNS
origin Caddy DMZ 10.20.30.10 0 + 1 internet (via edge) public
apps-proxy SERVERS 10.20.20.6 2 (user apps: HA, Jellyfin, Invidious, umami GUI, git, search, cloud, files, dav, auth, ntfy — see group_vars/apps) WS/WG-RW internal split-horizon (Unbound override; not public)
mgmt-proxy MGMT 10.20.10.6 2-admin (infra UIs) wg-admin only internal split-horizon

Tier-2 names (ha/jellyfin.${domain}) still get a publicly-trusted wildcard cert via DNS-01/RFC2136 (the cert needs only the TXT record, not a public A record), so https://ha.${domain} shows a green lock while resolving only internally. Moving a service between tiers = move its vhost between Caddies + flip its Unbound override.

Public domains served (origin plane). The origin Caddy isn't ${domain}-only — it also terminates several unrelated public apexes (real LE certs via the same DNS-01/RFC2136 path; content placed at /var/www/<name>). Current set (see origin_caddy defaults): ${domain} services (auth/search/fmd/cloud/stats, plus the Tier-1 agent/assistant plane on the GPU host: opencode rb32 + assistant/research rb44, all Authelia-gated per-user), plus git/metronome/zddc.example.net (the public git.${domain_alt} is served by the forgejo-pub instance; the control-plane git.${domain} is Tier-2 private and not on this origin plane — see the two-instance split, runbook 35), chidot.example, example.org (Tier-0 static sites, except example.org/media-trust which is Authelia-gated), and myos.io — the infra docs site, now Tier-0 PUBLIC (the intent-first rewrite, un-gated 2026-06-29). The published pages read myos.io: the authored sections use ${domain} placeholders (reader-personalizable via a Personalize page + localStorage), and the Reference appendix (the symlinked CLAUDE.md + runbooks) now carries only ${domain}/${realm}/${edge.*} tokens — scrubbed of literal instance identity (A3) and resolved client-side by placeholders.js — so the earlier build-time domain/IP-masking sed (provision/sites/myos/Containerfile) has been retired: the master tree carries no identity to mask. A build-time leak gate greps the built site for any residual private token and fails the build if one survives — now the sole standing regression guard. This completes the interim bridge to full repo parameterization (a cloner sets one domain var to stand up their own).

(Media clients are an exception to forward-auth: native apps — Jellyfin clients, Kodi — can't traverse an SSO redirect, so Jellyfin authenticates against FreeIPA via LDAP and is network-gated Tier 2, rather than sitting behind Authelia forward-auth.)

Two ways to count visitors, chosen per site (analytics posture). umami (stats.${domain}) is the in-page option — cookieless, but it still needs a beacon in the page, so it can't serve a site whose product claim is "this page never phones home." For those, the origin Caddy counts server-side instead: an opt-in access_log: true on that one vhost logs the request the browser already made — client IP masked to /24 in the encoder, the whole request-header map (UA, Referer, Cookie) deleted before anything is written — to stdout → journald → Alloy → Loki → the Site visits Grafana dashboard (runbook 09 §9). No new software, no log file, no beacon; the durable record is a tally, not a profile, and the page stays byte-identical to a saved copy of itself. It is off for every vhost by default, and only appropriate for sites whose per-user state rides the URL fragment (never sent) rather than the query string (logged in uri). Two retention horizons, deliberately split: the raw lines are request records, so they expire with Loki at 30 days; the count is promoted to a metric (site_http_requests_total{vhost,class}, via alloy_access_log_metrics) kept 13 months in VictoriaMetrics — so a multi-year "how many people came" costs an aggregate, never a longer retention on request records. The path is never a metric label (bots would make it unbounded), and distinct-visitor counts stay log-only/30-day since they need a distinct-count. First user: the metronome static site (2026-07-26).

A fourth public plane — Jitsi Meet (meet.${domain}, runbook 48). Real-time video is the one service that doesn't fit any of the three Caddy planes: a WebRTC SFU needs a public UDP media port the TCP-SNI edges can't carry, and routing meeting media through the home edges would load the residential uplink. So Jitsi lives off the origin/edge planes — it terminates its own TLS and serves its own UDP/10000 media, not on the origin_caddy apex list. It breaks two edge invariants (a box holds a cert; a public UDP port is open), acceptable because Jitsi E2EE is on by default (insertable streams): the videobridge forwards media it cannot decrypt. Auth is JWT gated by Authelia OIDC (an OIDC→JWT adapter) so only authenticated users can create/moderate a room; guests land in the lobby.

INTERIM posture (rev. 2026-07-29): on-prem, but with its OWN public IP. Providers TBD, so meet is a pve VM — attached to the public segment holding .65 on its own NIC, exactly like the edge VMs, not behind a NAT. It was first deployed behind an OPNsense WAN VIP (.66 → DMZ .15), and that had to be undone: inbound NAT applies only to traffic arriving on WAN, so internal clients resolving the public name reached the firewall's GUI (its DNS-rebind guard) instead of Jitsi. Propping it up took split-horizon DNS, a WORKSTATIONS→DMZ carve-out and dual ICE candidates — four mechanisms for one wrong attachment, all deleted once it owned the address. Lesson: don't publish a service on a firewall VIP. Consequences that remain: it dies with a home/Comcast outage; its egress is the ISP, not Proton (as with the edges); it sits outside OPNsense's inbound filtering (protected by the common baseline — cert-only SSH, fail2ban, auditd); and the "vendor can't read media" rationale is moot on trusted pve (E2EE kept anyway). Being on-network it does rejoin central obs (alloy) — but only because its tunnel AllowedIPs carries the obs /32; without that the agent runs and ships NOTHING while looking perfectly healthy (found 2026-08-01, mute for as long as the tunnel had existed). The lab's own public addresses — meet + the two edges, listed individually, NOT the containing /29 — sit in VPN_BYPASS, so internal→our-own-public-IPs egresses the real WAN rather than a Proton exit. The WG road-warrior endpoint (.69) is deliberately EXCLUDED and must stay on Proton egress: it is a VIP on the WAN interface, so bypassing NATs an internal peer's handshake to the private transit address and loops it back into "Block private networks from WAN" — the tunnel then comes up but passes nothing (rx: 0), for home clients only. Bypassing the whole /29 did exactly that on 2026-07-31 (runbooks 05 §6 + 48). End-state: the same role off-prem — only where the box lives changes, since the network shape already matches.

Its cert is the container's own Let's Encrypt, HTTP-01 — deliberately NOT DNS-01. This is the one plane that is a genuinely standalone web server, so it validates over the public :80 it already has open for the 80→443 redirect: the image runs acme.sh in --standalone from cont-init (before nginx binds :80, with hooks that stop/restart nginx around issuance) and installs its own renewal cron under a supervised cron service. So obtain and renew are in-container — no RFC2136 to the home primary, no TSIG secret on a public box, no timer of ours — and the same setting works unchanged off-prem. Pin the CA (LETSENCRYPT_ACME_SERVER): acme.sh defaults to ZeroSSL, and because issuance holds nginx down, a stalled ZeroSSL order takes the whole site off the air, not just the renewal. It is still blackbox-probed for TLS expiry, being the one cert no Caddy manages.

IoT discovery (mDNS) — the Roku/printer problem

Roku/Chromecast/AirPlay/printers advertise over mDNS/Bonjour, which is link-local multicast (224.0.0.251:5353) and does not cross VLANs. So a firewall allow rule alone won't let a phone find the Roku. Fix = an mDNS reflector (OPNsense Avahi-based repeater) bridging WORKSTATIONS⇄IOT announcements, plus unicast allow rules for the actual media/print streams. This keeps IoT isolated while casting/printing "just works." (Note: mDNS does not cross WireGuard, so roaming devices in GUEST cannot cast — they use the target app's own remote/play.)

Key firewall design points

  • Single default gateway + explicit policy routing — the system default is one gateway (WAN_GW); Proton egress is applied per-zone via policy routing. Never two defaultgw at once (that breaks routing).
  • Kill-switch — an explicit fail-closed block so a VPN drop never leaks to WAN.
  • Neutral WAN monitor IP (e.g. 9.9.9.9) — WAN health is decoupled from any Proton server.
  • SSH — key/cert-only, no root password, mgmt-scoped (no password auth).
  • Clean DNS — Unbound only, no duplicate/stale resolvers; recursion bound to the Proton tunnels.
  • DMZ and MGMT are first-class isolated zones — the two highest-impact segments.
  • Host firewalls (optional defense-in-depth) — every fleet host runs firewalld in default-deny inbound (SSH + its declared service ports only), applied uniformly by the host_firewall role via common. This is the "harden later" layer, not a replacement for the trust model: OPNsense's inter-zone allow-list still owns the trust boundary (the LAN is trusted; the adversary is the ISP/vendors). What it adds is closing every unused inbound listener + a reboot-safe, codified posture. Plain ports by design (no per-host source-scoping — that would duplicate OPNsense). Ports are inventory data (group_vars/host_vars host_firewall_extra_ports). See runbook 23.

OPNsense VM notes

  • Two virtio NICs: vtnet0 = LAN trunk (VLANs 10–90), vtnet1 = WAN.
  • Required virtio tunings: disable checksum / segmentation / large-receive offloading (needed for stable virtio under KVM/Proxmox).
  • Sized for VPN throughput: 8 vCPU + multiqueue NICs (kernel WireGuard is CPU-bound).
  • Plugins: os-acme-client (if used internally), os-wireguard/kernel WG, os-qemu-guest-agent; Unbound on, dnsmasq off.

Build sequence (clean install)

Environment: Proxmox node pve; WAN bridge vmbr0, LAN trunk vmbr1; VM storage local-zfs, ISO storage local. WAN: the ISP router stays inline and routes the static public block to OPNsense — WAN takes a private DHCP transit lease (gateway WAN_DHCP, kept stable via a router DHCP reservation), and the public IPs are assigned on WAN as IP Alias VIPs. VMIDs in 91xx: AlmaLinux template 9190, OPNsense 9140, fleet 9100–9130. An SSH CA is trusted fleet-wide (CA pubkey baked into the template + TrustedUserCAKeys in the Ansible hardening role). Ansible-core via dnf on an EL control node. FreeIPA: realm ${realm}, domain ${ipa_domain}.

Edge policy: out = ProtonVPN (every zone → the failover group over WAN; kill-switch fail-closed). in = via the edge tunnels only — home is inbound-silent except the WG admin endpoint on .69:51820. VPN_BYPASS is an empty safety valve (default is pure all-via-Proton).

VM base images: the Linux fleet clones a single AlmaLinux 9 template (provision/vms.conf); the fileserver is the same template + OpenZFS (runbook 11). Provisioning is runbook 01 (provision/lab-provision.sh); the Ansible config & maintenance layer is runbook 16 (the common baseline role, db_backups, and update.yml serial: 1 coordinated upgrades on top of per-host dnf-automatic). App-layer (Authelia image, FreeIPA major versions) updated deliberately, separate from the OS.

Update posture = three layers, and each needs BOTH an apply story and a detect story. Only the OS layer applies by itself; the other two are deliberate-by-design, which is correct — but "deliberate" silently degrades into "forgotten" without detection, so each gets a read-only probe → ntfy (runbook 34): appliances via appliance-update-check.yml, and every pinned container image via image-update-check.yml, whose list is derived from the role defaults (never hand-kept) so a new image is covered automatically. Images are digest/tag-pinned with podman-auto-update masked fleet-wide; that pinning is deliberate, and this probe is its compensating control. Added 2026-07-30 after a pin 18 days stale took invidious fully down — units green, its own alert rule correctly silent, found by a user (runbook 21). Weekly, with the summary folded into the 04:00 digest so the weekly check has daily proof-of-life.

  1. Proxmox: OPNsense VM, latest stable, qemu-guest-agent; two virtio NICs (NIC0→LAN bridge, NIC1→WAN bridge so vtnet0=LAN, vtnet1=WAN); LAN bridge VLAN-aware/trunk for tags 10/20/30/40/60/90.
  2. Base config: MGMT interface at 10.20.10.1, WAN DHCP transit; hostname/domain/NTP/timezone.
  3. Interfaces/VLANs: create the zone VLANs + assignments at their .1 gateways.
  4. Routing/VPN: Proton multi-tunnel failover group, road-warrior server (enroll peers), WAN default + policy-route to the group, RFC1918 + VPN_BYPASS rules, fail-closed rule, per-endpoint /32 static routes.
  5. Services (firewall): Unbound (+ IoT DNS intercept), mDNS reflector, DHCP. No public port-forwards — the only WAN inbound is the WG admin endpoint (.69:51820, runbook 07).
  6. Identity/storage/origin: FreeIPA + NFSv4 home exports; enroll clients; automount + sec=sys; origin Caddy (internal — TLS-terminate/HTTP3, DNS-01/RFC2136 to the local hidden primary); Authelia SSO/MFA (FreeIPA-backed)
  7. Caddy forward_auth; Unbound forwards the IPA zone to FreeIPA; hidden Bind primary (internal, AXFR to the edge secondaries). 6b. Edge (runbook 17): stand up the VPS edges (Caddy-L4 SNI passthrough + Bind secondary + home-dialed WG tunnel), delegate registrar NS/A records to them, prove end-to-end.
  8. Firewall rules: per-zone, default-deny, documented, fail-closed.
  9. Validate: Proton egress (clients+servers) + failover + VPN_BYPASS + kill-switch; RFC1918 bypass (NFS stays local/fast); inter-zone matrix; mDNS cast/print across WORKSTATIONS↔IOT; road-warrior dial-in + admin→MGMT; public sites via Caddy (HTTP/3) with Authelia SSO+MFA; FreeIPA login + NFS home; media (Jellyfin LDAP login, tvheadend Live TV).

Deliverables & format

Work is delivered as a sequence of discrete step-by-step runbooks under runbooks/, dialed in one at a time (build → validate → next). The OPNsense layer is delivered as GUI runbooks (click-through + precise rule/zone spec); the server layer (FreeIPA, ZFS/NFS storage, clients) as plain command runbooks. See runbooks/00-index.md for the ordered sequence and status.

The runbooks render as plain Markdown (runbooks/*.md); the public docs/ guides build with mkdocs (→ myos.io) for reproducibility by others.

Docs are a first-class, non-optional deliverable — keep them in lockstep with the system (a hard completion gate). Any change that affects how the lab is built or operated updates the matching record in the same change: the relevant runbooks/* step, this CLAUDE.md (the design source of truth), the host/IP allocation table, and any role README. A change isn't "done" until its docs are. The myos.io site is auto-published on push to main by a Forgejo Actions workflow (.forgejo/workflows/myos-image.yml: mkdocs build on the runner → a FROM-scratch OCI content image pushed to the internal zot registry → the origin Caddy serves it via Mount=type=image and refresh-site pulls the new tag; runbook 31). It is path-filtered — it fires only on docs/**, mkdocs.yml, requirements-docs.txt, runbooks/**, CLAUDE.md, the site Containerfile, or the workflow itself, so a main push that touches only ansible/ publishes nothing. (The older rsync-to-/var/www/myos docs-deploy.yml was retired once the origin cut over to the image; the OCI path replaced it.) What's published always reflects main — which is exactly why the docs must not drift from reality. The GUI master index on the docs landing page (docs/index.md) is sourced from the three Caddy planes' vhost lists; add a service there and mirror its row. See ansible/ci/README.md for the static-site/docs CD model.

Operating role: you (Claude) are the network administrator. You connect directly to the Proxmox host, firewall, and servers as required to configure, administer, and troubleshoot them — you do the work, then test and validate it is correct (don't just hand back steps for someone else to run). The runbooks are your reference and the record of the design, not a wall you stop at.

The division of labor: the user holds the master keys (vault password, YubiKey SSH-CA signing, KeePass, root/break-glass credentials) and authorizes hard-to-reverse or outward-facing actions; you do the hands-on administration and verification. Keep the user educated and informed — explain what you're doing and why, surface tradeoffs and risks, and report results faithfully (including failures) — but you are the one executing and proving the change, not delegating it back. When you genuinely lack an access path or credential, say exactly what you need rather than defaulting to "the user will run it."

Working notes

  • This file is the source of truth for the design.