Skip to content

vps_edge

Configures a public edge (remote multi-vendor VPS) for the edge/origin split:

  • caddy-l4 — TLS-SNI passthrough on :443, run as the shared B9 container image (caddy-l4 baked into the image, not a per-host xcaddy build — retired 2026-06-28, runbook 29). The edge can't reach the internal registry, so the role ships the image over SSH (skopeo-export on the registry host → control node → podman load on the edge) and runs it as the caddy-proxy quadlet via caddy_container. It peeks at the ClientHello SNI and forwards the still-encrypted stream over the WireGuard tunnel to the origin Caddy. No TLS termination, no certs, no plaintext on the edge.
  • WireGuard — the edge LISTENS; home dials out to it (origin is the WG client + keepalive). No Endpoint on the edge side.
  • Bind secondary — public authoritative; AXFRs the signed zones from the hidden primary via the edge-xfr TSIG key. recursion no, allow-query any.
  • firewalld — opens 443/tcp, 53/tcp+udp, and the WG port.

Built-in modules only. Apply with playbooks/edge.yml (serial: 1 — never blank the public layer all at once).

Behavior

End to end: a client → edge :443 (SNI match) → tunnel → origin Caddy → backend returns 200, and the cert the client sees is the origin's (the edge never terminates TLS — it is pure SNI passthrough); an unknown SNI is dropped; the Bind secondary serves the signed zones via edge-xfr AXFR.

Required vars (per environment)

Var Where Notes
edge_home_wg_pubkey group_vars/vps_edges (vault ok) origin Caddy's WG public key
edge_xfr_key_secret vault (vault_bind_tsig_edge_xfr) SAME secret as the primary's edge-xfr
edge_tunnel_ip host_vars/<edge> this edge's tunnel /32 (e.g. 10.20.200.2)
ansible_host inventory the edge's public IP (or tunnel IP once up)

Home side (NOT in this role — do once on the origin)

The tunnel terminates on the origin Caddy (the WG client that dials out to each edge). On the origin:

  1. WG client /etc/wireguard/wg0.conf — Address 10.20.200.1/24, [Peer] = each edge (PublicKey, Endpoint <edge-public-ip>:51820, AllowedIPs <edge_tunnel_ip>/32, PersistentKeepalive 25). Home dials out; the transport rides the WAN/Proton egress (the edge sees a Proton IP, not home).
  2. Origin Caddy already listens :443 on all interfaces → it answers on wg0 (10.20.200.1:443). That is edge_origin_caddy — no home routing needed for the TLS forward.
  3. For AXFR only: the edge pulls zones from the hidden primary 10.20.30.40 (a different host than the tunnel endpoint). Enable net.ipv4.ip_forward=1 on the origin Caddy and ensure tunnel→10.20.30.40:53 is routed; the primary's allow-query already covers 10.20.0.0/16 and allow-transfer is the edge-xfr key. (Alternatively give the primary its own WG peer.)

Rolling out an edge

  1. Vendor-provision the VPS (AlmaLinux 9), put its public IP as ansible_host in [vps_edges], add host_vars/<edge>.yml with edge_tunnel_ip.
  2. Set the vault secrets; add the edge as a [Peer] on the origin's WG client.
  3. ansible-playbook -i inventory.ini playbooks/edge.yml --limit <edge>.
  4. The role prints the edge's WG pubkey — add it to the origin peer if not already.
  5. Delegate registrar NS to the edge (keep ns1/ns2.${edge.ns_domain}), prove end-to-end, then enable origin ACME (acme key) for real LE certs.

Gotchas

  • systemctl enable --now <svc> 2>&1 | tail -0 silently swallows the start — start services without piping through tail -0. (This role uses the service module, which is fine.)
  • caddy-l4 is v0.x (no stable releases) — its version is pinned in the shared B9 image (provision/caddy/Containerfile); bump it there + rebuild the image deliberately, never auto-update. (There is no per-host edge_caddy_l4_version var anymore — the module ships inside the container, runbook 29.)
  • AlmaLinux is SELinux-enforcing — the Bind slave dir /var/named/slaves needs the correct context (named_cache_t on slave files; the package defaults + restorecon cover it). The caddy-l4 runtime is now a container, so no native binary to label.

Edge hardening

The public edges get, beyond the common baseline (host firewall default-deny, fail2ban, SSH-CA, security updates): Bind RRL (rate-limit in named.conf.j2 — caps identical responses per client prefix, slip 2 so legit clients retry over TCP) to blunt DNS-amplification abuse of the public NS, and auditd (kernel audit trail of logins/sudo/auth on the most-exposed hosts). Also version-hiding (version/hostname/server-id "none" on the public NS) and AIDE file-integrity (baseline DB + a daily check-then-absorb timer: aide --update finds drift → one AIDE-DRIFT alert, full diff in the journal + /var/lib/aide/last-report.txt, then the fresh DB becomes the baseline). AIDE runs against a scope-narrowed config (edge_aide_config, {{ edge_aide_config }}): it @@includes the stock /etc/aide.conf then !-excludes high-churn / zero-attacker-value trees (edge_aide_excludes — golang stdlib source, .build-id, man/doc/locale, tzdata), so a routine package update no longer floods thousands of changes and false-fires the alert (a golang bump alone was ~7 000 files). Binaries (/usr/bin, /sbin, …), libraries, /etc, /root, /boot stay watched — tampering there still trips the alarm; a more-specific ! overrides the broad /usr NORMAL (stock's own !/usr/src mechanism). Validated on edge1: 98 818 → 67 883 entries, golang source gone, go/gofmt binaries + all security paths retained. The config also overrides the vfat ESP (/boot/efi) to ESPR = NORMAL-i: vfat inode numbers are minted in-memory and change on cache eviction, so checking i there false-fires at random (edge1 2026-07-16, fbx64.efi inode-only change) — content/perm tampering still alarms, the unstable attribute is dropped (verified: forced inode churn stays green). A config change re-inits the baseline (the reinit aide baseline handler). RRL is tunable via edge_rrl_*. (Deliberately NOT a heavyweight EDR agent — proportionate, self-hosted only.) Since 2026-07-10 the edges also run the fleet Alloy agent (edge.yml): metrics + journald ship to obs over the WG tunnel (obs /32 in edge_peer_allowed_ips, masqueraded by the origin — origin_tunnel_masq_dests — so it arrives as DMZ→obs on the existing port-scoped allows).

How the AIDE alarm is raised (changed 2026-08-01). Drift now exits 0 and emits a single AIDE-DRIFT line, which the Grafana rule "Edge AIDE file-integrity drift" pages on over a 2h window — one page per drift event. It previously propagated aide's nonzero rc and rode the generic "Systemd unit failed" rule, but a failed Type=oneshot holds failed until its next run, and this timer is daily: measured against node_systemd_unit_state, the 07-26 and 07-30 drift days each sat failed for 24.0h, and at repeat_interval: 4h that is ~6 pages for one package update. "One loud day" had become "one loud day, six times over" — the same signal-death the check-then-absorb design exists to prevent. A real aide error (rc > 7 — bad config, unreadable DB, nothing checked) still exits nonzero and still rides "Systemd unit failed"; "the timer stopped firing" is covered by the stale-timer rule. So exiting 0 on drift narrows what is loud, it does not open a silent hole. Two ordering details, both found by self-test rather than reasoning: the status line is emitted before the report (a routine drift is ~3 600 journal lines against journald's 10 000/30s burst, so a trailing line is dropped exactly when drift is largest), and the unit sets LogRateLimitIntervalSec=0 so the forensic diff itself survives to Loki intact. Logic + full rationale live in templates/aide-check.sh.j2; run it by hand for a one-off check.