Runbook 17 — Public edge: two diverse VPSes (Caddy-L4 passthrough + Bind secondary)

Goal: push the public-facing layer off the Comcast link onto two thin, stateless mini-VPS "edges" on different providers/regions. Each runs TLS-SNI passthrough (caddy-l4) to the origin Caddy at home — it never terminates TLS, holds no cert, never sees plaintext — plus a public Bind secondary the registrar delegates to. Home reaches each edge by dialing out a WireGuard tunnel.

Prereqs: the origin works internally (runbook 09 — origin Caddy terminates TLS via DNS-01; hidden Bind primary serves the zone with an AXFR TSIG key). A registered domain you control NS records for.

YOUR ENVIRONMENT: ${domain} is delegated to two public authoritative nameservers you own — ns1.${edge.ns_domain} and ns2.${edge.ns_domain}. The edges run on off-prem cloud VPSes (different vendors — see "Picking the two VPSes"), each carrying caddy-l4 SNI-passthrough + a home-dialed WG tunnel + a Bind secondary of the hidden primary (10.20.30.40, AXFR over the tunnel). The registrar's NS records (ns1/ns2.${edge.ns_domain}) delegate to the two edge public IPs; only those A/glue IPs are vendor-specific, not the NS names.

The home WAN block (the edge public block) carries no public web/DNS — its only inbound is .69 = the WG admin endpoint; .67/.68 are reserved, .65/.66 free. All public serving lives on the off-prem edges, never on the Comcast link.

What "load-balance" means here — read first. Both edges forward to the same origin at home, so two edges buy redundancy + provider/region diversity + ingress proximity, not backend scaling (home is the single backend either way). So: no anycast/BGP — this is DNS round-robin across two front doors with the origin behind them. Keep that framing; it's why a $5 VPS is plenty.

Build one edge at a time. Stand up edge1 alone and verify the whole chain end-to-end before cloning the pattern to edge2 (different vendor) + DNS diversity + optional auto-failover. Two unknowns at once is a lost weekend.

---

Picking the two VPSes

• Different parent companies / ASNs — that's the entire point. e.g. DigitalOcean + Vultr, or Hetzner + Linode. Not two regions of one provider (one provider outage would take both).
• Geographically apart is fine (e.g. an east + a west region) but helps resilience more than latency — every request still funnels to home, so home's RTT dominates. Don't over-optimize geo.
• $5 / 1 vCPU / 1 GB / AlmaLinux 9 is ample — SNI passthrough is just TCP shoveling and a Bind secondary is tiny. The real thing to check is the monthly bandwidth allowance (all your site traffic transits the edge → 1–2 TB tiers are common and fine for personal sites). The role is bandwidth-bound, not CPU-bound — don't pay for CPU, and ARM is fine (caddy-l4/BIND/WireGuard all run native on ARM).

Recommended pairing (researched, 2026): - edge1 = Hetzner CAX11 (ARM) — ~$5 all-in, 20 TB traffic included, NVMe, DDoS protection + firewall at no charge, US (Ashburn/Hillsboro) or EU. The bandwidth value winner. (Small IPv4 add-on fee ≈ €0.50/mo; you need the public IPv4 for the A records.) - edge2 (different parent) = Oracle Cloud "Always Free" Ampere ARM (~$0, ~10 TB/mo egress) — caveat: Oracle can reclaim idle free instances, so keep it serving / monitored. Prefer a paid, guaranteed-yours box? OVHcloud (~$5, unmetered @ ~500 Mbps) or BuyVM (~$3.50, generous bandwidth + DDoS). - Avoid Vultr/Linode/DigitalOcean for this role — stingy transfer + pricey overage ($0.01/GB); they're priced for CPU, which you don't need.

Both run the same vps_edge role (ARM or x86; xcaddy build + dnf packages are arch-native). Pick edge2's region on the opposite coast from edge1.

---

Addressing (both edges)

Thing	Value
Edge tunnel subnet	10.20.200.0/24 (one /32 per edge + the home end)
Home tunnel IP	10.20.200.1 (origin side of every edge tunnel)
edge1 tunnel IP	10.20.200.2 · public IP <vendor-A>
edge2 tunnel IP	10.20.200.3 · public IP <vendor-B>
Origin Caddy	10.20.30.10:443 (TLS terminator, reached over the tunnel)
Hidden primary	10.20.30.40:53 (AXFR source, reached over the tunnel)
Home WGEDGE instance	listen 51821 (distinct from roadwarrior 51820 + Proton)

Per edge, increment the tunnel IP (.2, .3, …) and add a separate Peer on the home WGEDGE instance. Home is always 10.20.200.1 on its side.

---

Per-edge build (do edge1 fully, prove it, then repeat for edge2)

Everything in §1–§5 is per-edge. The vps_edge Ansible role (runbook 16 / ansible/roles/vps_edge) automates all of it; this is the explanatory/manual form.

Build order vs. the origin: §1–§2 (host + WG tunnel) are self-contained and can be built before the origin exists — they de-risk the trickiest part. But §3 (Caddy-L4) and §4 (Bind secondary) need the origin (origin Caddy + hidden Bind primary, runbook 09) — they have nothing to forward to / transfer from until then. So: provision + tunnel now, pause, finish §3–§4 once rb09 is done.

1. The host

Vendor-provision AlmaLinux 9 with a persistent public IP (see "Picking the two VPSes"), SSH key-only. On the host: dnf -y install wireguard-tools bind bind-utils golang firewalld, harden (SSH key-only, fail2ban, dnf-automatic — the common Ansible role applies), do §5's firewalld now, then generate the tunnel key:

umask 077; wg genkey | tee edge.key | wg pubkey > edge.pub

golang is there to build caddy-l4 (xcaddy compiles the layer4 SNI-passthrough plugin from source). Its stdlib source tree (/usr/lib/golang/src, ~7 000 files) is rewritten on every Go update by dnf-automatic, which would flood AIDE — so the edge AIDE config excludes it (and man/doc/locale/build-id) while still watching the compiled caddy binary and all of /usr/bin, /etc, /root, /boot. See the vps_edge role README (edge_aide_excludes); a golang bump false-fired aide-check on 2026-07-14 before this was added.

AIDE drift pages ONCE, and the check exits 0 to make that true. The daily aide --update is check-then-absorb: drift alarms for one run, then becomes the new baseline. That alarm used to be the oneshot's nonzero exit riding the generic "Systemd unit failed" rule — but a failed Type=oneshot stays failed until its next run, and this timer is daily. Measured on node_systemd_unit_state, the 2026-07-26 and 07-30 drift days each sat failed 24.0h; at repeat_interval: 4h one package update paged ~6 times. Since 2026-08-01 drift emits a single AIDE-DRIFT line and exits 0, and the Grafana rule "Edge AIDE file-integrity drift" (Loki, 2h window) pages once per event. Look at it when it fires — the change-set is absorbed either way, so this is your only notice. A real aide error (rc > 7: bad config, unreadable DB — i.e. nothing was checked) still exits nonzero and still pages via "Systemd unit failed"; a stopped timer is caught by the stale-timer rule. Full exit policy: roles/vps_edge/templates/aide-check.sh.j2.

/boot/efi (the vfat ESP) is watched without the inode attribute. UEFI requires FAT for the ESP, and the Linux vfat driver mints inode numbers from an in-memory counter — they change whenever the cached inode is evicted (drop_caches reproduces it), so stock's /boot NORMAL (which checks i) false-fires at random on ESP files (edge1, 2026-07-16: fbx64.efi, inode-only "change", content hash identical). The edge AIDE config overrides the ESP subtree with ESPR = NORMAL-i — content/perm/owner tampering (boot-kit) still trips the alarm; the unstable attribute doesn't. Verified: forcing inode churn renumbers all nine ESP files and the check stays green.

Optional — local stand-in for development. To exercise the whole chain before committing to a VPS, stand in a local AlmaLinux 9 VM: clone the template (9190) → NIC on vmbr0 (WAN bridge, no VLAN tag) → a static public IP on your routed block (gateway = your business router), DNS 9.9.9.9/1.1.1.1. It egresses directly (a public box, not behind Proton) and sits outside OPNsense's internal zones, so it reaches home only via the WG tunnel (§2) — same isolation as a real edge, no WAN inbound rule (home dials out). It's a functional test, not a resilience one (same power/ISP/Proxmox); the off-prem VPSes are the production path.

2. WireGuard — home dials out to this edge

The VPS has a stable public IP, so home initiates (Comcast can't be dialed into beyond .69). The VPS listens; home sets the VPS as a peer Endpoint with PersistentKeepalive so the tunnel stays up through NAT.

2a. On the VPS (wg-quick, listener)

umask 077; wg genkey | tee edge.key | wg pubkey > edge.pub

/etc/wireguard/wg0.conf:

[Interface]
Address    = 10.20.200.2/24      # edge2 = 10.20.200.3
ListenPort = 51820
PrivateKey = <edge.key>

[Peer]                            # home (origin) — no Endpoint; home dials in
PublicKey  = <home-WGEDGE-pubkey>
AllowedIPs = 10.20.200.1/32, 10.20.30.0/24

AllowedIPs includes DMZ so the edge can reach the origin Caddy + hidden primary over the tunnel. systemctl enable --now wg-quick@wg0.

2b. On home OPNsense — one WGEDGE instance, one Peer per edge

VPN → WireGuard → Peers → + (one per edge): - Public key <edge.pub> · Endpoint = the VPS public IP / 51820 · Allowed IPs 10.20.200.2/32 (edge2 → .3/32) · Keepalive 25 (home is behind NAT).

VPN → WireGuard → Instances → + (the home side, created once): - Name edge · new private key (its pubkey = <home-WGEDGE-pubkey> above) - Listen port 51821 · Tunnel address 10.20.200.1/24 · Peers = all edge peers. Assign the interface → WGEDGE.

No gateway/default-route change — this carries only 10.20.200.0/24 + service traffic, never internet egress. No WAN inbound rule is needed (home dials out).

2c. WGEDGE firewall rules (home side) — per edge, source-restricted

Firewall → Rules → WGEDGE: 1. Pass TCP: src 10.20.200.2/32 (edge1) → 10.20.30.10 (origin Caddy) 443 2. Pass TCP/UDP: src 10.20.200.2/32 → 10.20.30.40 (hidden primary) 53 (AXFR/NOTIFY) 3. (repeat 1–2 for edge2 10.20.200.3/32) 4. Default deny — an edge can reach nothing else inside home.

3. Caddy-L4 — TLS-SNI passthrough

caddy-l4 ships as a container, not a native xcaddy build (retired 2026-06-28; see runbook 29). The edge runs the shared B9 image (one digest-pinned image with caddy-l4 and rfc2136 baked in — the module version is pinned in provision/caddy/Containerfile, not per-host) as a podman quadlet via the caddy_container role. Edges can't reach the internal registry, so the vps_edge role ships the image over SSH: skopeo exports it on the registry host → control node → podman load on the edge → run as the caddy-proxy quadlet. wg0 stays host-level. There is no native build or edge_caddy_l4_version var anymore — bump the module in the Containerfile + rebuild the image.

/etc/caddy/Caddyfile — match by SNI, proxy the encrypted stream over the tunnel to the origin Caddy (no tls on the upstream → no termination):

{
    layer4 {
        :443 {
            @web tls sni cloud.${domain} auth.${domain}
            route @web {
                proxy 10.20.30.10:443        # origin Caddy (DMZ), over the WG tunnel
            }
        }
    }
}

The cert is presented by the origin Caddy at home; this box holds no key. caddy validate then systemctl enable --now caddy.

4. Bind secondary — public authoritative

/etc/named.conf (secondary of the hidden primary, transfer over the tunnel):

key "edge-xfr" { algorithm hmac-sha256; secret "<same secret as the primary>"; };
options {
    listen-on   { 127.0.0.1; <edge-public-ip>; };
    allow-query { any; };          // public authoritative
    recursion   no;                // never an open resolver
    allow-transfer { none; };
};
zone "${domain}" {
    type secondary;
    primaries { 10.20.200.1 key edge-xfr; };   // home primary, over the tunnel
    file "/var/named/slaves/${domain}.zone";
};

On the hidden primary (runbook 09 §4): allow-transfer { key edge-xfr; } and also-notify both edge tunnel IPs (10.20.200.2, 10.20.200.3) so changes push instantly. systemctl enable --now named.

5. firewalld (edge host)

firewall-cmd --permanent --add-port=443/tcp --add-port=53/tcp --add-port=53/udp \
             --add-port=51820/udp --add-service=ssh
firewall-cmd --reload

Edge egress is direct (public boxes — not Proton).

---

Two-edge specifics

6. DNS — delegate to both edges + round-robin (this is the load-balancing)

All public DNS is self-hosted (hidden primary at home → both edge secondaries). The load-balance/failover lives entirely in the zone + the registrar:

1. Registrar NS → both edges (≥2 diverse vendors):
2. ${domain}. NS ns1.${domain}.
3. ${domain}. NS ns2.${domain}.
4. Glue: ns1.${domain} → edge1 public IP, ns2.${domain} → edge2 public IP. Public DNS is now redundant — a dead edge still leaves the other answering NS.
5. In the hidden-primary zone (${domain}):
6. ns1 / ns2 A records → edge1 / edge2 public IPs.
7. Service records round-robin both edges — e.g.

   cloud   60  IN A  <edge1>
   cloud   60  IN A  <edge2>
   auth    60  IN A  <edge1>
   auth    60  IN A  <edge2>
   

8. Keep TTL ~60–300s so failover/changes propagate without hammering the edges.
9. Clients land on either edge (round-robin), and browsers do "happy eyeballs" — they retry the other IP on connection failure within ~a second. That's your soft load-balance + soft failover, needing nothing but two A records each.

7. (Optional) Automatic failover — health-check on the hidden primary

Soft failover (browser retry) is enough for most personal setups. If you want a dead edge's A record pulled automatically (so even non-retrying clients/apps don't hit it), run a tiny health-check on the hidden Bind primary at home — fully self-hosted, no third party. It uses the same RFC2136/nsupdate path the origin Caddy already uses, and only updates when the healthy set changes (no serial churn):

/usr/local/bin/edge-healthcheck.sh:

#!/usr/bin/env bash
set -euo pipefail
ZONE=${domain}
KEY=/etc/bind/healthcheck.key            # TSIG key; primary allow-update for the LB names
STATE=/run/edge-health.state
declare -A EDGES=( [edge1]=<edge1-public-ip> [edge2]=<edge2-public-ip> )
NAMES=(cloud auth)                       # the round-robin hostnames

# 1) which edges answer :443 right now
up=()
for n in "${!EDGES[@]}"; do
  timeout 4 bash -c "</dev/tcp/${EDGES[$n]}/443" 2>/dev/null && up+=("$n")
done
cur="${up[*]:-NONE}"
[ -f "$STATE" ] && [ "$(cat "$STATE")" = "$cur" ] && exit 0   # no change → no update
# never blackhole: if all are down, leave records as-is
[ "${#up[@]}" -eq 0 ] && { echo "$cur" >"$STATE"; logger "edge-healthcheck: all down, kept records"; exit 0; }

# 2) rewrite the LB A records to only the healthy edges
{ echo "server 127.0.0.1"; echo "zone $ZONE"
  for name in "${NAMES[@]}"; do
    echo "update delete $name.$ZONE A"
    for n in "${up[@]}"; do echo "update add $name.$ZONE 60 A ${EDGES[$n]}"; done
  done
  echo "send"; } | nsupdate -k "$KEY"
echo "$cur" >"$STATE"
logger "edge-healthcheck: healthy set -> $cur (zone updated, NOTIFY to secondaries)"

nsupdate bumps the SOA serial and NOTIFYs → AXFR/IXFR to the surviving secondary automatically. Drive it with a systemd timer every ~30 s:

# /etc/systemd/system/edge-healthcheck.timer  → OnUnitActiveSec=30s, Persistent=true
# /etc/systemd/system/edge-healthcheck.service → ExecStart=/usr/local/bin/edge-healthcheck.sh

Notes: the primary needs an allow-update { key healthcheck; }; scoped to the LB names (separate from the edge-xfr transfer key and the acme. challenge key). Failover is TTL-bound (clients cache the A for its TTL → keep it 60s). NS-level failure (a whole edge incl. its Bind) is covered by step 6's two-NS delegation, which resolvers handle by trying the other NS.

8. Giving back (optional, in-keeping)

The lab leans on non-commercial infra (root servers, NLnet Labs' Unbound/BIND, Caddy, AlmaLinux). The edges are spare, always-on public boxes — the natural place to reciprocate: run a public NTP pool node (chrony + join pool.ntp.org), and sponsor the projects you depend on (NLnet Labs first, then ISC/RIPE NCC/Quad9). (Trim freely — recorded because it came up while choosing monitor targets.)

9. Off-site home watchdog — the alert that survives a whole-home outage

Every home-side alert path (Grafana → ntfy, the ctrl dead-man's-switch, the 04:00 digest) dies with home — a whole-home outage (power, Comcast, OPNsense, pve) takes the pager down with it and the phone just goes quiet. This was runbook 27's §Limits and runbook 42's one "structural" open item. The edges are the lab's only off-site machines with direct (non-Proton, non-home) egress, so the fix lives here, at zero new infrastructure: each edge runs a 5-minute watchdog timer (edge-home-watch, codified in vps_edge) that probes home two ways over the tunnel —

1. WG handshake age (wg show wg0 latest-handshakes): home dials out with persistent-keepalive, so a fresh handshake is the steady state; stale > 5 min means the tunnel — and by implication home — is unreachable.
2. HTTPS to the origin Caddy's tunnel endpoint (--resolve the docs-site SNI to 10.20.200.1): any HTTP response counts as alive — content/404 correctness stays the home-side blackbox prober's job (runbook 27).

Either signal succeeding = home is up. Both failing 3 consecutive runs (~15 min) posts ONE transition alert — to the public ntfy.sh fallback topic (vault_deadman_fallback_url, the same second channel the ctrl deadman uses, which the phone already subscribes to), because the lab's own ntfy is on the wrong side of the outage by definition. Content is deliberately generic ("home unreachable from edge1") — a third-party service never carries internal detail. Recovery posts once.

Signal semantics: ONE edge firing = suspect that edge's tunnel/VPS. BOTH edges firing = home is actually dark. Who watches the watchdog: the edges run the fleet Alloy, so the timer is auto-covered by the derived fleet-timer-stale rule — which works exactly when home (obs) is up, the complementary case. Tradeoff, accepted: the topic URL now also lives on the two most-exposed hosts (/etc/lab/edge-home-watch.url, 0600) — an edge compromise could read it and spam/spoof generic lines at the phone; rotating the topic (re-vault + converge ctrl and the edges, re-subscribe the phone) evicts a leaked name.

The watchdog is required-on by default (edge_home_watch_enabled), and the role fails fast if the fallback URL is unset rather than deploying mute — the runbook 42 shakedown lesson: a watchdog that warn-and-skips its channel looks like coverage and pages nobody, discovered only during the outage it existed for.

---

Standing up edge2

1. Provision edge2 on the other vendor/region; run §1–§5 with edge2's values (tunnel 10.20.200.3, its own keypair/public IP).
2. Home: add edge2 as a second Peer on the existing WGEDGE instance, and add the §2c rule pair for 10.20.200.3/32.
3. Hidden primary: add edge2 to also-notify; add edge2's IP to the cloud/auth round-robin and the ns2 glue.
4. Registrar: add the second NS (ns2 → edge2).
5. ansible-playbook -i inventory.ini playbooks/edge.yml (it's serial: 1, so it never blanks both edges at once).

---

Validation

Edge1, from the public internet: - [ ] dig @<edge1-ip> ${domain} SOA answers from the secondary with the primary's serial (AXFR over the tunnel); dig @<edge1-ip> google.com → REFUSED (no recursion). - [ ] curl -v https://cloud.${domain} --resolve cloud.${domain}:443:<edge1-ip> returns the site and the cert is the home origin's (passthrough — the edge holds no key). - [ ] Home VPN → WireGuard → Status shows a live handshake to edge1; WGEDGE rules show passes to .30.10:443 / .30.40:53 only. - [ ] Home stays silent: external scan of the Comcast IP still shows only UDP 51820 on .69.

Both edges + failover: - [ ] Registrar lists ≥2 NS on diverse vendors; cloud.${domain} returns both edge IPs (round-robin). - [ ] Kill edge1 (stop caddy/named, or firewall-cmd block): the site + DNS still serve via edge2; browsers fail over within ~a second. - [ ] Health-check fires: within ~30 s the hidden primary drops edge1's A record (check dig cloud.${domain} → only edge2); restore edge1 → it returns. - [ ] Each edge box reaches only 10.20.30.10:443 + 10.20.30.40:53 inside home; home Comcast scan unchanged (only WG). - [ ] Each edge reproducible from the vps_edge role (clean second-run no-op).

---

Documentation followed

• caddy-l4 (SNI matcher + proxy, Caddyfile layer4): github.com/mholt/caddy-l4, built via caddyserver/xcaddy.
• BIND 9 secondary, TSIG, AXFR/IXFR, also-notify, dynamic update (nsupdate/ RFC2136): bind9.readthedocs.io.
• OPNsense WireGuard instances/peers (peer-first; Endpoint = address + port; keepalive): docs.opnsense.org/manual/wireguard.html.

No-VPN fleet access via the two-tier jump path

A client with no WireGuard reaches the fleet over the edges' :443 SSH relay. Both jump tiers are forward-only accounts that authorize ONLY the dedicated jump transit principal (no shell, no admin) — access is gated at the target, not the relays:

laptop (internet/GUEST) → edge:443 (SSH banner → edge `jump` sshd, PermitOpen 10.20.30.5:22)
                        → DMZ jump 10.20.30.5 (`jump`, firewall-restricted to fleet:22)
                        → target:22

Command (e.g. to the GPU host at 10.20.20.70):

ssh -J jump@${edge.ns1_public}:443,jump@10.20.30.5 <you>@10.20.20.70   # .67=edge1, .68=edge2

~/.ssh/config:

Host edge-jump
    HostName ${edge.ns1_public}   # or ${edge.ns2_public}
    Port 443
    User jump
Host gpu-host
    HostName 10.20.20.70
    User <you>
    ProxyJump edge-jump,jump@10.20.30.5

Your cert carries jump (to traverse both relays) PLUS your way onto the target — the target's own principal (e.g. jump,admin-servers for a SERVERS box), your IPA identity (SSSD-served pubkey, no cert needed to land), or a raw authorized_keys entry on the target. You never need admin to traverse — admin is for landing, not transit. Sign a login cert with jump,<target-principal> (the yubikey-tool.sh prompt hints at this).

Client gotchas (debugged 2026-06-29)

When ssh -J jump@<edge>:443,jump@10.20.30.5 <you>@<target> fails, it's almost always client-side (the relays are simple). From a real -v:

• Transit vs landing principal. Your cert needs jump to traverse both relays PLUS the target's own access to land — e.g. sign jump,admin-servers (a SERVERS box), or carry jump alone and land via your IPA identity / a raw authorized_keys on the target. You never sign admin to traverse. Both relays are jump-only (no shell).
• The DMZ hop user is jump@10.20.30.5 (the forward-only transit account) — not almalinux@ or any shell account.
• Agent overrides bite the inner hop. If ~/.ssh/config pins IdentityAgent <path> + IdentitiesOnly yes for 10.20.*, the nested ProxyJump ssh to 10.20.30.5 uses that — and if the socket/file is missing or holds the wrong key, you get Permission denied there while the edge hop (which used your default agent) succeeded. Symptom in -v: get_agent_identities: ssh_get_authentication_socket: No such file or directory. Fix: let the hops use the agent that actually holds your jump cert (don't override IdentityAgent to a non-existent socket; don't IdentitiesOnly away the good cert).
• Stale target IP. Hosts move zones — e.g. the GPU host is 10.20.20.70 (SERVERS), not its old 10.20.10.7 (MGMT control-node) address. Check the host/IP table in CLAUDE.md.

DNSSEC anchoring — DS publication (pending operator, runbook 42 C8)

The zones are signed (single CSK alg-13 per zone, dnssec-policy default, inline-signing on the hidden primary) but unanchored: no DS record exists at any parent, so validating resolvers treat every zone as insecure and the signatures protect nobody. Publishing DS completes the design — and turns RRSIG expiry into a total-outage class, which is why the fleet-rrsig-expiring rule (dns-watch on the primary, <5 days runway pages; runbook 42) went live first.

Escrow gate — CLEARED 2026-07-13. DS publication also raises the stakes on the CSKs themselves: once a DS is pinned at the parent, losing the keys (bind is a single un-rebuildable VM) takes the zone dark until a registrar-side DS change. The full /var/named/dynamic state (keys + zones) is now escrowed in the vault, verified round-trip — see runbook 09 §Escrow. Re-take it after any rollover; KeePass copy = operator.

DS records (SHA-256): generate the live table from the CSKs on the hidden primary — one row per zone (regenerate after any key change):

for k in /var/named/dynamic/K*.key; do dnssec-dsfromkey -2 "$k"; done
# → per zone:   <zone>.   IN DS <keytag> 13 2 <sha256-hex-digest>

The literal per-zone table is instance identity — kept out of this public tree (the build-time leak gate rejects it). The current instance's table lives in the gitignored TODO.md; regenerate with the command above whenever in doubt.

Procedure (operator — registrar web UIs are T3 credentials): 1. Canary first: publish ONE low-stakes zone's DS (giide.xyz — parked, serves nothing) at its registrar. Most registrars want (keytag=e.g. 65021, algorithm=13, digest-type=2, digest=). 2. Wait for the parent to publish (minutes–hours): dig DS giide.xyz +short @9.9.9.9. 3. Validate the chain: delv @9.9.9.9 giide.xyz SOA must say ; fully validated (and https://dnsviz.net/d/giide.xyz/analyze/ should be all-green). A validation failure here breaks only a parked zone — remove the DS and diagnose. 4. Soak ~48 h (dns-watch keeps proving edges in sync + RRSIG runway ≥5 d), then roll the remaining zones the same way. 5. After each key change (should be never under dnssec-policy default — no scheduled rolls), the DS must be re-generated + re-published BEFORE the old key retires.

Standing invariants once DS is live: never let the primary's named stay broken past the RRSIG runway (fleet-rrsig-expiring pages at <5 d); never restore the zones from a backup that predates a key change; fleet-dns-out-of-sync continues to prove the edges serve what the primary signs.